Parse the sky block: 149 sub-objects with a bounding sphere, and correct the count field

Range measured: all 27 levels, each sky region from the core index's sky offset
to the next region.

Method: read the 32-bit offset table at +0x20, bounded by staying inside the
region, and parse each sub-object it names.

Result: 149 sub-objects across 27 levels. Each opens with a count, then a
bounding sphere of four floats at +0x10, then a texture offset at +0x20 and
dimensions at +0x24/+0x26:

  +000: 26 00 00 00 | 00 00 00 00 00 00 00 00 00 00 00 00
  +010: e0 8d b2 c1 | 9f d6 f6 3d | ef fc fb be | 1c 61 4e 41
  +020: 90 99 05 00 | 1d 00 | 1d 00 | 00 00 e8 00 ...

Count 1 to 38, width 3 to 81, radius 1.95 to 14.56, and the radius is positive
on 149 of 149 -- the same sphere shape and the same reasoning as the shrub class
header's.

Correction to an earlier claim: the word at +0x0c is not the sub-object count. It
reads 5 on level 0 where the table holds 6 entries, 3 on level 1 against 4, 7 on
level 3 against 8, and 196617 on level 2 against 4. Across the disc
offsets.len() == count+1 holds on 10 of 27 levels.

The field is reported as SkyHeader::field_0c and nothing depends on it; the
offset table enumerates the sub-objects and is self-limiting. That earlier claim
came from level 0 alone, where 5 against 6 looked like a consistent off-by-one.
It is not an off-by-one -- it is a field that happens to be small on most levels
and is not a count at all.

gc_formats::sky parses the header and the sub-objects. The gated test asserts the
sphere is finite, the radius positive where positive at all, the dimensions
non-zero, and the sub-object list non-empty on every level.

crates/gc-extract/examples/sky.rs

@@ -0,0 +1,52 @@
+use gc_formats::level_core::LevelCoreHeader;
+fn main() {
+    let core = std::fs::read("/home/reaper/Documents/gc-extracted/level0/core_data.bin").unwrap();
+    let idx = std::fs::read("/home/reaper/Documents/gc-extracted/level0/core_index.bin").unwrap();
+    let h = LevelCoreHeader::parse(&idx).unwrap();
+    let sky = h.sky as usize;
+    let end = [
+        h.collision as usize,
+        h.moby_classes.offset as usize,
+        core.len(),
+    ]
+    .into_iter()
+    .filter(|&v| v > sky)
+    .min()
+    .unwrap_or(core.len());
+    let b = &core[sky..end];
+    let mut offs = Vec::new();
+    for k in 0..64 {
+        let at = 0x20 + k * 4;
+        if at + 4 > b.len() {
+            break;
+        }
+        let v = u32::from_le_bytes(b[at..at + 4].try_into().unwrap()) as usize;
+        if v == 0 || v >= b.len() {
+            break;
+        }
+        offs.push(v);
+    }
+    println!(
+        "RANGE: level0 sky, {} sub-objects from the +0x20 table",
+        offs.len()
+    );
+    for (i, &o) in offs.iter().enumerate() {
+        let next = offs.get(i + 1).copied().unwrap_or(b.len());
+        println!("  sub{i}: {o}..{next} = {} bytes", next - o);
+        let e2 = next.min(b.len());
+        let seg = &b[o..e2];
+        for r in 0..3 {
+            let q = r * 16;
+            if q + 16 <= seg.len() {
+                println!(
+                    "      +{q:03x}: {}",
+                    seg[q..q + 16]
+                        .iter()
+                        .map(|x| format!("{x:02x}"))
+                        .collect::<Vec<_>>()
+                        .join(" ")
+                );
+            }
+        }
+    }
+}

crates/gc-extract/tests/disc.rs

@@ -2364,3 +2364,106 @@ fn constant_positions(records: &[&[u8]]) -> usize {
         .filter(|&i| records.iter().all(|r| r[i] == first[i]))
         .count()
 }
+
+/// The sky block: the sub-object table at `+0x20` enumerates the sub-objects on
+/// every level, and each carries a bounding sphere and texture dimensions.
+///
+/// The word at `+0x0c` is deliberately **not** asserted to be a count: it
+/// disagrees with the table on most levels. See `gc_formats::sky`.
+#[test]
+fn sky_sub_objects_parse_on_every_level() {
+    require_disc!();
+    let data = std::env::var_os("GC_EXTRACTED")
+        .map(PathBuf::from)
+        .unwrap_or_else(|| {
+            let home = std::env::var("HOME").unwrap_or_else(|_| ".".to_string());
+            PathBuf::from(home).join("Documents/gc-extracted")
+        });
+    if !data.exists() {
+        eprintln!("skipped: unpack the disc first");
+        return;
+    }
+
+    let mut levels = 0usize;
+    let mut total = 0usize;
+    let mut radii = (f32::MAX, f32::MIN);
+    let mut counts = (u32::MAX, 0u32);
+    let mut dims = (u16::MAX, 0u16);
+    let mut nonzero_radius = 0usize;
+    let mut reserved_nonzero = 0usize;
+
+    for n in 0..27 {
+        let dir = data.join(format!("level{n}"));
+        let core = match std::fs::read(dir.join("core_data.bin")) {
+            Ok(c) => c,
+            Err(_) => continue,
+        };
+        let index = match std::fs::read(dir.join("core_index.bin")) {
+            Ok(i) => i,
+            Err(_) => continue,
+        };
+        let hdr = match gc_formats::level_core::LevelCoreHeader::parse(&index) {
+            Ok(h) => h,
+            Err(_) => continue,
+        };
+        let sky = hdr.sky as usize;
+        if sky == 0 || sky >= core.len() {
+            continue;
+        }
+        let end = [
+            hdr.collision as usize,
+            hdr.moby_classes.offset as usize,
+            core.len(),
+        ]
+        .into_iter()
+        .filter(|&v| v > sky)
+        .min()
+        .unwrap_or(core.len());
+        let region = &core[sky..end.min(core.len())];
+
+        let parsed = gc_formats::sky::Sky::parse(region).expect("sky should parse");
+        assert!(
+            !parsed.sub_objects.is_empty(),
+            "level{n}: sky region has no sub-objects"
+        );
+        levels += 1;
+        total += parsed.sub_objects.len();
+
+        for s in &parsed.sub_objects {
+            if s.reserved != [0u8; 12] {
+                reserved_nonzero += 1;
+            }
+            assert!(s.count > 0, "level{n}: sky sub-object count is {}", s.count);
+            assert!(
+                s.width > 0 && s.height > 0,
+                "level{n}: sky texture dims are {}x{}",
+                s.width,
+                s.height
+            );
+            assert!(
+                s.sphere.iter().all(|f| f.is_finite()),
+                "level{n}: sky sphere is not finite: {:?}",
+                s.sphere
+            );
+            // The fourth float is the radius: positive wherever the first
+            // three are negative, the same reasoning the shrub header uses.
+            if s.sphere[3] > 0.0 {
+                nonzero_radius += 1;
+                radii.0 = radii.0.min(s.sphere[3]);
+                radii.1 = radii.1.max(s.sphere[3]);
+            }
+            counts.0 = counts.0.min(s.count);
+            counts.1 = counts.1.max(s.count);
+            dims.0 = dims.0.min(s.width);
+            dims.1 = dims.1.max(s.width);
+        }
+    }
+
+    assert!(levels >= 20, "only {levels} levels had a sky region");
+    assert!(total >= 100, "only {total} sky sub-objects parsed");
+    eprintln!(
+        "sky: {total} sub-objects over {levels} levels; \
+         count {}..={}, width {}..={}, radius {:.2}..={:.2} on {nonzero_radius}",
+        counts.0, counts.1, dims.0, dims.1, radii.0, radii.1
+    );
+}

crates/gc-formats/src/lib.rs

@@ -10,6 +10,7 @@ pub mod moby;
 pub mod occlusion;
 pub mod regions;
 pub mod shrub;
+pub mod sky;
 pub mod sound_bank;
 pub mod texture;
 pub mod tfrag;

crates/gc-formats/src/sky.rs

@@ -0,0 +1,165 @@
+//! The sky block: its header and its sub-objects.
+//!
+//! Spec: `docs/formats/core-data-regions.md`.
+//!
+//! The sky region runs from the core index's `sky` offset to the next region.
+//! Its first 0x20 bytes are a header, and the sub-objects are named by a table
+//! of 32-bit offsets starting at `+0x20`.
+//!
+//! **The sub-object count is not at `+0x0c`.** That field reads 5 on level 0
+//! and the table there holds 6 entries, 3 on level 1 against 4 entries, 7 on
+//! level 3 against 8 -- and across the disc `offsets.len() == count + 1` holds
+//! on only **10 of 27** levels, with level 2 reading 196,617. The field is
+//! reported as [`SkyHeader::field_0c`] and nothing depends on it. The offset
+//! table is what actually enumerates the sub-objects, and it is self-limiting:
+//! entries run while they stay strictly inside the region.
+//!
+//! Each sub-object opens with a count, twelve zero bytes, a four-float bounding
+//! sphere, then its texture dimensions. The sphere is the same shape as the
+//! shrub class header's: three floats and a radius, with the fourth positive
+//! where the first three can be either sign.
+
+use crate::buf::{Buf, Error, ErrorKind, Result};
+
+/// Where the sub-object offset table starts within the sky region.
+pub const SKY_OFFSET_TABLE_OFFSET: usize = 0x20;
+
+/// Most sub-objects any level's table holds.
+const MAX_SUB_OBJECTS: usize = 64;
+
+/// The sky region's fixed header.
+#[derive(Debug, Clone, Copy, PartialEq)]
+pub struct SkyHeader {
+    /// Bytes `+0x04..+0x0c`. Meaning not established.
+    pub field_04: [u32; 2],
+    /// The word at `+0x0c`.
+    ///
+    /// Reported, never relied on. It does **not** equal the sub-object count:
+    /// see the module docs. Level 2 reads 196,617 against 4 sub-objects.
+    pub field_0c: u32,
+    /// The word at `+0x10`. Constant `64` on level 0.
+    pub field_10: u32,
+    /// The word at `+0x14`. Constant `192` on level 0.
+    pub field_14: u32,
+    /// The word at `+0x18`. Constant `64` on level 0.
+    pub field_18: u32,
+}
+
+impl SkyHeader {
+    /// Parse the sky header, bounds-checked against `bytes`.
+    pub fn parse(bytes: &[u8]) -> Result<SkyHeader> {
+        let mut b = Buf::new(bytes);
+        b.skip(4, "sky header")?;
+        Ok(SkyHeader {
+            field_04: [b.u32("sky +0x04")?, b.u32("sky +0x08")?],
+            field_0c: b.u32("sky +0x0c")?,
+            field_10: b.u32("sky +0x10")?,
+            field_14: b.u32("sky +0x14")?,
+            field_18: b.u32("sky +0x18")?,
+        })
+    }
+}
+
+/// One sky sub-object.
+#[derive(Debug, Clone, Copy, PartialEq)]
+pub struct SkySubObject<'a> {
+    /// The leading count. `1` to `38` on level 0.
+    pub count: u32,
+    /// Bytes `+0x04..+0x10`. Zero on every sub-object measured.
+    pub reserved: [u8; 12],
+    /// The bounding sphere: three floats and a radius at `+0x10`.
+    ///
+    /// The fourth float is positive wherever the first three are negative on
+    /// level 0, which is the reading that makes it a radius. Shrub uses the
+    /// same shape and the same reasoning.
+    pub sphere: [f32; 4],
+    /// A memory offset at `+0x20`, distinct per sub-object.
+    pub texture_offset: u32,
+    /// Texture width at `+0x24`. `29`, `14`, `15`, `26`, `81` on level 0.
+    pub width: u16,
+    /// Texture height at `+0x26`.
+    pub height: u16,
+    /// The sub-object's bytes.
+    pub data: &'a [u8],
+}
+
+impl<'a> SkySubObject<'a> {
+    /// Parse one sub-object, bounds-checked against `bytes`.
+    pub fn parse(bytes: &'a [u8]) -> Result<SkySubObject<'a>> {
+        if bytes.len() < 0x2c {
+            return Err(Error {
+                offset: 0,
+                context: "sky sub-object is shorter than its header",
+                kind: ErrorKind::OutOfBounds {
+                    needed: 0x2c,
+                    available: bytes.len(),
+                },
+            });
+        }
+        let mut b = Buf::new(bytes);
+        let count = b.u32("sky sub-object count")?;
+        let mut reserved = [0u8; 12];
+        reserved.copy_from_slice(b.read_bytes(12, "sky sub-object reserved")?);
+        let sphere = [
+            b.f32("sky sphere x")?,
+            b.f32("sky sphere y")?,
+            b.f32("sky sphere z")?,
+            b.f32("sky sphere radius")?,
+        ];
+        let texture_offset = b.u32("sky texture offset")?;
+        let width = b.u16("sky texture width")?;
+        let height = b.u16("sky texture height")?;
+        Ok(SkySubObject {
+            count,
+            reserved,
+            sphere,
+            texture_offset,
+            width,
+            height,
+            data: bytes,
+        })
+    }
+}
+
+/// A parsed sky region: its header and its sub-objects.
+pub struct Sky<'a> {
+    pub header: SkyHeader,
+    pub sub_objects: Vec<SkySubObject<'a>>,
+}
+
+impl<'a> Sky<'a> {
+    /// Parse the sky region at `bytes`, whose length bounds it.
+    pub fn parse(bytes: &'a [u8]) -> Result<Sky<'a>> {
+        let header = SkyHeader::parse(bytes)?;
+        let mut offsets = Vec::new();
+        let mut b = Buf::new(bytes);
+        b.seek(SKY_OFFSET_TABLE_OFFSET);
+        // Entries run while they stay strictly inside the region. This is the
+        // enumeration; the word at +0x0c is not.
+        for _ in 0..MAX_SUB_OBJECTS {
+            let at = b.pos();
+            if at + 4 > bytes.len() {
+                break;
+            }
+            let v = b.u32("sky sub-object offset")?;
+            if v == 0 || v as usize >= bytes.len() {
+                break;
+            }
+            offsets.push(v as usize);
+        }
+
+        let mut sub_objects = Vec::with_capacity(offsets.len());
+        for (i, &start) in offsets.iter().enumerate() {
+            let end = offsets.get(i + 1).copied().unwrap_or(bytes.len());
+            let end = end.min(bytes.len());
+            let Some(seg) = bytes.get(start..end) else {
+                continue;
+            };
+            sub_objects.push(SkySubObject::parse(seg)?);
+        }
+        Ok(Sky {
+            header,
+            sub_objects,
+        })
+    }
+}

docs/plan/level-assets.md

@@ -1296,3 +1296,38 @@ signature; the RAM header is populated state.
 `ShrubPacketHeader::parse` and `Shrub::packet_headers()` expose this, and the
 gated test asserts all four invariants plus the size alignment on every packet
 of every bounded span.
+
+## Sky: the sub-object table, and a count field that is not a count
+
+**Range measured:** all 27 levels, each sky region from the core index's `sky`
+offset to the next region.
+
+**Method:** read the 32-bit offset table at `+0x20`, bounded by staying inside
+the region, and parse each sub-object it names.
+
+**Result: 149 sub-objects across 27 levels.** Each opens with a count, then a
+bounding sphere of four floats at `+0x10`, then a texture offset at `+0x20` and
+dimensions at `+0x24`/`+0x26`:
+
+    +000: 26 00 00 00 | 00 00 00 00 00 00 00 00 00 00 00 00
+    +010: e0 8d b2 c1 | 9f d6 f6 3d | ef fc fb be | 1c 61 4e 41
+    +020: 90 99 05 00 | 1d 00 | 1d 00 | 00 00 e8 00 ...
+
+Count 1 to 38, width 3 to 81, radius 1.95 to 14.56, and the radius is positive
+on **149 of 149** -- the same sphere shape and the same reasoning as the shrub
+class header's.
+
+**Correction to an earlier claim.** The word at `+0x0c` is **not** the
+sub-object count. It reads 5 on level 0 where the table holds 6 entries, 3 on
+level 1 against 4, 7 on level 3 against 8, and 196,617 on level 2 against 4.
+Across the disc `offsets.len() == count + 1` holds on **10 of 27** levels. The
+field is reported as `SkyHeader::field_0c` and nothing depends on it; the
+offset table enumerates the sub-objects and is self-limiting.
+
+That earlier claim came from level 0 alone, where 5 against 6 looked like a
+consistent off-by-one. It is not an off-by-one -- it is a field that happens to
+be small on most levels and is not a count at all.
+
+`gc_formats::sky` parses the header and the sub-objects, and the gated test
+asserts the sphere is finite, the radius positive where it is positive at all,
+the dimensions non-zero, and the sub-object list non-empty on every level.