Parse the level core class tables: moby, tie and shrub

These were ruled out for want of an entry size. The size is knowable
exactly, because each table ends precisely where the next begins:

  moby  238 entries at   192, 32 bytes:  192 + 238*32 =  7808 = tie's
  tie    74 entries at  7808, 32 bytes: 7808 +  74*32 = 10176 = shrub's
  shrub   7 entries at 10176, 48 bytes: 10176 +   7*48 = 10512 = textures

That closes on 27 of 27 levels, which is what makes it evidence rather
than a coincidence: any wrong entry size breaks the chain immediately.
It is the same "does this hold on every level" discriminator the project
already uses for the container pair table.

Across the disc: 5,796 moby, 1,864 tie and 358 shrub classes, plus the
1,595 tfrag textures. Class entries are 32 bytes -- asset offset, class
number, two unknown words and 16 bytes of texture slots -- and a shrub
entry adds 16 bytes of billboard slots for 48.

The header now decodes through to the texture tables rather than
skipping 0x18..0x30 wholesale, which also recovers the moby, tie and
shrub texture ranges. Two more pairs sit between shrub textures and the
textures base offset.

Two things worth recording. ClassEntry::parse read its 16 texture bytes
before its four i32s, because the struct literal that uses them comes
after the loop that advances the cursor -- so every entry was parsed
backwards. Struct literal field order does not reorder the reads that
already happened. And every shrub class on level 0 carries all-zero
billboard slots, so the dimensions are optional rather than something
worth asserting on.

This is the third instance of the same failure shape. The gameplay
extent was "an unidentified compression" because it was parsed
compressed; the tfrag record framing was "unrecoverable" because the
entry size was never known. Both looked like properties of the format
and were properties of the question.

Also records that point lights are a runtime feature rather than missing
work: every one of the 18,102 patches has point_lights == 0xffff and the
eight-slot bank is filled by gameplay, so there is nothing for them to
light in a static level render.

And that PCSX2 is not installed, so the comparison loop cannot be built
yet -- which is why lighting magnitudes stay unverified.

crates/gc-extract/tests/disc.rs

@@ -970,3 +970,125 @@ fn tfrag_lod_parents_resolve_for_every_entry() {
          {lod01} LOD-01 and {lod0} LOD-0, all resolved"
     );
 }
+
+/// The level core index's class tables.
+///
+/// These are what put the environment on top of the terrain: tie instances,
+/// moby actors and shrub foliage. They were ruled out earlier for want of an
+/// entry size, which turned out to be knowable exactly.
+///
+/// The check is the chain: each table must end precisely where the next one
+/// begins, moby (32 bytes/entry) into tie (32) into shrub (48) into the tfrag
+/// texture table. That closes on all 27 levels and would not close if any
+/// entry size were wrong, so it is the evidence rather than a coincidence.
+#[test]
+fn level_core_class_tables_tile_the_index() {
+    require_disc!();
+    gc_formats::iso9660::IsoImage::open(std::path::Path::new(
+        &std::env::var("GC_DISC").expect("GC_DISC is set by require_disc!"),
+    ))
+    .expect("ISO should open");
+    let data = std::env::var_os("GC_EXTRACTED")
+        .map(PathBuf::from)
+        .unwrap_or_else(|| {
+            let home = std::env::var("HOME").unwrap_or_else(|_| ".".to_string());
+            PathBuf::from(home).join("Documents/gc-extracted")
+        });
+    if !data.exists() {
+        eprintln!("skipped: unpack the disc first");
+        return;
+    }
+
+    use gc_formats::level_core as lc;
+    let mut levels = 0usize;
+    let (mut moby, mut tie, mut shrub, mut textures) = (0usize, 0usize, 0usize, 0usize);
+
+    for n in 0..27 {
+        let dir = data.join(format!("level{n}"));
+        let Ok(index) = std::fs::read(dir.join("core_index.bin")) else {
+            continue;
+        };
+        let h = lc::LevelCoreHeader::parse(&index).expect("core header");
+
+        let end = |r: lc::ArrayRange, size: usize| r.offset as usize + r.count as usize * size;
+        assert_eq!(
+            end(h.moby_classes, lc::MOBY_CLASS_SIZE),
+            h.tie_classes.offset as usize,
+            "level{n}: moby -> tie"
+        );
+        assert_eq!(
+            end(h.tie_classes, lc::TIE_CLASS_SIZE),
+            h.shrub_classes.offset as usize,
+            "level{n}: tie -> shrub"
+        );
+        assert_eq!(
+            end(h.shrub_classes, lc::SHRUB_CLASS_SIZE),
+            h.tfrag_textures.offset as usize,
+            "level{n}: shrub -> textures"
+        );
+
+        // Every entry parses, and its asset offset lands inside the level.
+        if n == 0 {
+            eprintln!(
+                "hdr moby={:?} tie={:?} shrub={:?} tex={:?} base={}",
+                h.moby_classes, h.tie_classes, h.shrub_classes, h.tfrag_textures, h.textures_base
+            );
+            eprintln!("index[192..200] = {:?}", &index[192..200]);
+        }
+        let moby_v = lc::class_table(
+            &index,
+            h.moby_classes,
+            lc::MOBY_CLASS_SIZE,
+            lc::ClassEntry::parse,
+        )
+        .expect("moby classes");
+        let tie_v = lc::class_table(
+            &index,
+            h.tie_classes,
+            lc::TIE_CLASS_SIZE,
+            lc::ClassEntry::parse,
+        )
+        .expect("tie classes");
+        let shrub_v = lc::class_table(
+            &index,
+            h.shrub_classes,
+            lc::SHRUB_CLASS_SIZE,
+            lc::ShrubClassEntry::parse,
+        )
+        .expect("shrub classes");
+        let tex_v = lc::texture_table(&index, h.tfrag_textures).expect("tfrag textures");
+
+        assert_eq!(moby_v.len(), h.moby_classes.count as usize);
+        for (i, c) in moby_v.iter().chain(tie_v.iter()).enumerate() {
+            assert!(
+                c.offset_in_asset_wad >= 0,
+                "level{n} entry {i}: asset offset {}",
+                c.offset_in_asset_wad
+            );
+        }
+        // Billboard slots are optional: every shrub class on level 0 carries
+        // them all zero, so only their sign is meaningful here.
+        for s in &shrub_v {
+            assert!(s.base.offset_in_asset_wad >= 0);
+            assert!(
+                s.palette_offset >= -1 && s.texture_offset >= -1,
+                "level{n}: shrub billboard offsets below the -1 sentinel"
+            );
+        }
+        moby += moby_v.len();
+        tie += tie_v.len();
+        shrub += shrub_v.len();
+        textures += tex_v.len();
+        levels += 1;
+    }
+
+    assert_eq!(levels, 27, "expected all 27 levels, got {levels}");
+    assert!(
+        moby + tie + shrub > 1000,
+        "only {moby}+{tie}+{shrub} class entries"
+    );
+    eprintln!(
+        "level core index: {moby} moby, {tie} tie, {shrub} shrub classes, \
+         {textures} tfrag textures across {levels} levels"
+    );
+}

crates/gc-formats/src/level_core.rs

@@ -21,6 +21,76 @@ pub struct ArrayRange {
     pub offset: i32,
 }
 
+/// One entry in the moby, tie or shrub class table (32 bytes).
+///
+/// These name the environment geometry sitting on top of the terrain -- the
+/// props, foliage and actors. `offset_in_asset_wad` points into the level's
+/// asset WAD, not into this index.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct ClassEntry {
+    pub offset_in_asset_wad: i32,
+    pub o_class: i32,
+    pub unknown_8: i32,
+    pub unknown_c: i32,
+    pub textures: [u8; 16],
+}
+
+impl ClassEntry {
+    pub fn parse(bytes: &[u8]) -> Result<ClassEntry> {
+        let mut b = Buf::new(bytes);
+        // Order matters: these reads advance the cursor, so the four words have
+        // to come before the trailing texture bytes, not after.
+        let offset_in_asset_wad = b.i32("class asset offset")?;
+        let o_class = b.i32("class number")?;
+        let unknown_8 = b.i32("class unknown_8")?;
+        let unknown_c = b.i32("class unknown_c")?;
+        let mut textures = [0u8; 16];
+        for slot in textures.iter_mut() {
+            *slot = b.u8("class textures")?;
+        }
+        Ok(ClassEntry {
+            offset_in_asset_wad,
+            o_class,
+            unknown_8,
+            unknown_c,
+            textures,
+        })
+    }
+}
+
+/// One shrub class entry: a class entry plus the billboard's texture slots (48
+/// bytes).
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct ShrubClassEntry {
+    pub base: ClassEntry,
+    pub billboard_width: i16,
+    pub billboard_height: i16,
+    pub max_mip: i16,
+    pub palette_offset: i16,
+    pub texture_offset: i16,
+    pub mip1: i16,
+    pub mip2: i16,
+    pub mip3: i16,
+}
+
+impl ShrubClassEntry {
+    pub fn parse(bytes: &[u8]) -> Result<ShrubClassEntry> {
+        let base = ClassEntry::parse(bytes)?;
+        let mut b = Buf::new(&bytes[32..]);
+        Ok(ShrubClassEntry {
+            base,
+            billboard_width: b.i16("shrub billboard width")?,
+            billboard_height: b.i16("shrub billboard height")?,
+            max_mip: b.i16("shrub max mip")?,
+            palette_offset: b.i16("shrub palette offset")?,
+            texture_offset: b.i16("shrub texture offset")?,
+            mip1: b.i16("shrub mip1")?,
+            mip2: b.i16("shrub mip2")?,
+            mip3: b.i16("shrub mip3")?,
+        })
+    }
+}
+
 /// One texture in the level's texture table (16 bytes).
 #[derive(Debug, Clone, Copy, PartialEq, Eq)]
 pub struct TextureEntry {
@@ -66,6 +136,48 @@ impl TextureEntry {
     }
 }
 
+/// Bytes per class-table entry, by table.
+///
+/// These sizes are not stated anywhere in the reference; they are fixed by the
+/// fact that each table ends exactly where the next one begins. That chain
+/// closes on all 27 levels, which is the only thing that makes it trustworthy.
+pub const MOBY_CLASS_SIZE: usize = 32;
+pub const TIE_CLASS_SIZE: usize = 32;
+pub const SHRUB_CLASS_SIZE: usize = 48;
+
+/// Parse a class table.
+pub fn class_table<T, F>(index: &[u8], range: ArrayRange, size: usize, parse: F) -> Result<Vec<T>>
+where
+    F: Fn(&[u8]) -> Result<T>,
+{
+    let count = range.count.max(0) as usize;
+    let offset = range.offset.max(0) as usize;
+    let need = count
+        .checked_mul(size)
+        .and_then(|n| offset.checked_add(n))
+        .ok_or(Error {
+            offset,
+            context: "class table range overflows",
+            kind: ErrorKind::InvalidLength("class table"),
+        })?;
+    if need > index.len() {
+        return Err(Error {
+            offset,
+            context: "class table runs past the index",
+            kind: ErrorKind::OutOfBounds {
+                needed: need,
+                available: index.len(),
+            },
+        });
+    }
+    let mut out = Vec::with_capacity(count);
+    for i in 0..count {
+        let at = offset + i * size;
+        out.push(parse(&index[at..at + size])?);
+    }
+    Ok(out)
+}
+
 /// Parse a texture table.
 pub fn texture_table(index: &[u8], range: ArrayRange) -> Result<Vec<TextureEntry>> {
     let count = range.count.max(0) as usize;
@@ -114,8 +226,15 @@ pub struct LevelCoreHeader {
     pub sky: i32,
     /// Byte offset of the collision block.
     pub collision: i32,
+    /// Class tables: the environment geometry on top of the terrain.
+    pub moby_classes: ArrayRange,
+    pub tie_classes: ArrayRange,
+    pub shrub_classes: ArrayRange,
     /// The table tfrag ad-gifs index into.
     pub tfrag_textures: ArrayRange,
+    pub moby_textures: ArrayRange,
+    pub tie_textures: ArrayRange,
+    pub shrub_textures: ArrayRange,
     /// Byte offset of texture pixel data within `core_data`.
     pub textures_base: i32,
 }
@@ -133,10 +252,15 @@ impl LevelCoreHeader {
         let occlusion = b.i32("occlusion")?;
         let sky = b.i32("sky")?;
         let collision = b.i32("collision")?;
-        // 0x18..0x30: moby/tie/shrub/gadget class tables, not decoded yet.
+        let moby_classes = pair(&mut b, "moby classes")?;
-        b.skip(0x30 - 0x18, "class tables")?;
+        let tie_classes = pair(&mut b, "tie classes")?;
+        let shrub_classes = pair(&mut b, "shrub classes")?;
         let tfrag_textures = pair(&mut b, "tfrag textures")?;
-        b.skip(0x60 - 0x38, "texture tables")?;
+        let moby_textures = pair(&mut b, "moby textures")?;
+        let tie_textures = pair(&mut b, "tie textures")?;
+        let shrub_textures = pair(&mut b, "shrub textures")?;
+        // 0x50..0x60: particle and effect texture tables, then the banks.
+        b.skip(0x60 - 0x50, "particle texture tables")?;
         let textures_base = b.i32("textures base")?;
         Ok(LevelCoreHeader {
             gs_ram,
@@ -144,7 +268,13 @@ impl LevelCoreHeader {
             occlusion,
             sky,
             collision,
+            moby_classes,
+            tie_classes,
+            shrub_classes,
             tfrag_textures,
+            moby_textures,
+            tie_textures,
+            shrub_textures,
             textures_base,
         })
     }

docs/plan/level-assets.md

@@ -516,6 +516,52 @@ light bank. The three inputs were:
 * the **light bank** from the level's `gameplay` extent -- **not working**,
   see below.
 
+### Point lights are a runtime feature, not a level-load one
+
+Every patch on every level has `point_lights == 0xffff` (none) and
+`dir_lights_one == -1`, checked across all 18,102 patches. The eight-slot point
+light bank is filled by gameplay at run time, not read from the disc.
+
+So `point_light_vertex` is not missing work for a static level render -- it
+would light nothing. The directional pass *is* the complete lighting for a
+level-load frame, and the point-light path belongs with the hero, explosions
+and the other things that spawn light.
+
+`dir_lights_one == -1` on every patch also means the per-vertex `select` word
+is what selects a light set, never the header field -- which is what the
+implementation uses.
+
+### The class tables were not unreadable, only unsized
+
+`core_index` holds moby, tie and shrub class tables at 0x18, 0x20 and 0x28,
+followed by the texture tables. They had been ruled out for want of an entry
+size, and every field order and stride tried.
+
+The sizes are knowable exactly, because **each table ends precisely where the
+next begins**:
+
+- moby 238 entries at 192, 32 bytes each: 192 + 238x32 = 7808 = tie's offset
+- tie 74 at 7808, 32 bytes: 7808 + 74x32 = 10176 = shrub's offset
+- shrub 7 at 10176, 48 bytes: 10176 + 7x48 = 10512 = the tfrag texture table
+- tfrag textures 93 at 10512, 16 bytes: 10512 + 93x16 = 12000 = moby textures
+
+That closes on **27 of 27 levels**, which is what makes it evidence rather
+than a coincidence: any wrong entry size would break the chain immediately.
+
+The class entries are 32 bytes -- `{asset offset, class number, two unknown
+words, 16 bytes of texture slots}` -- and a shrub entry adds 16 bytes of
+billboard slots for 48. Across all 27 levels: **5,796 moby, 1,864 tie and 358
+shrub classes**, and 1,595 tfrag textures.
+
+Every shrub class on level 0 carries all-zero billboard slots, so the billboard
+dimensions are optional rather than something to assert on.
+
+The failure here is the third instance of the same shape. The gameplay extent
+was "an unidentified compression" because it was being parsed compressed; the
+tfrag record framing was "unrecoverable" because the entry size was never
+known. Both looked like properties of the format and were properties of the
+question.
+
 ### Why the frame is still dark: the light bank does not parse on RAC2
 
 The reference frames the bank as "a u32 offset at +0x04, then an `i32` count
@@ -613,9 +659,12 @@ the fidelity rule.
 - **Lighting magnitudes.** The pass runs and the output is plausible; whether
   it matches the game needs comparison against reference frames, which needs
   the PCSX2 loop that is still to come. Nothing in this file establishes it.
-- **The `core_index` class tables**, still unread after every field order and
+- **tie, moby and shrub geometry.** The class tables now parse (below), but
-  stride tried. tfrag, texture and lighting data came from identifying each
+  nothing reads the geometry they name yet. `offset_in_asset_wad` points into
-  structure on its own terms instead.
+  the level's asset WAD, which is a separate container.
+- **Point lights** are spawned by gameplay at runtime and are absent from the
+  disc -- every one of the 18,102 patches has `point_lights == 0xffff` -- so
+  there is nothing for them to light until the game runs.
 
 A tfrag can now be turned into positioned, UV-mapped, textured, lit triangles,
 told which LOD to draw, and morphed toward its parents at the right rate.

docs/roadmap.md

@@ -142,14 +142,12 @@ Phase 1 hands over the extents; this phase works out what each one contains.
 
 - [x] `data` — the extent header locates `core_index` and `core_data`; the
       geometry decompresses and is verified on level 0.
-- [ ] `core_index` class tables — the header base is confirmed (the compressed
+- [x] `core_index` class tables — moby, tie and shrub. They were ruled out
-      and decompressed sizes of `core_data` sit at the same offsets in all 27
+      for want of an entry size, which is knowable exactly: each table ends
-      levels), and every offset lands inside `core_data`. But the class-table
+      precisely where the next begins, and that chain closes on 27 of 27
-      fields do not decode under any field order or stride tried. Ruled out and
+      levels. 32 bytes per moby/tie entry, 48 per shrub. Across the disc:
-      written up in `plan/level-assets.md`. Identifying `tfrag` by its own
+      **5,796 moby, 1,864 tie, 358 shrub classes**. Identifying `tfrag` by its
-      structure instead, which is what the note suggested, worked: tfrag,
+      own structure is what led here.
-      texture and lighting data are all parsed. The class tables themselves
-      remain unread.
 - [x] `occlusion` and `sky` offsets located and their first bytes read.
 - [x] Region layout of `core_data` established: tfrag, occlusion, sky, collision
       and `textures_base` are offsets, strictly increasing on all 27 levels.