Resolve tfrag LOD parent linkage: guard absent tiers by their counts

The four V4_8 parent arrays are classified by VU address, but the
addresses collide: on level 0 patch 1,
unk_indices_2_lod_01_addr == parent_indices_lod_0_addr == 291, and
unk_indices_2_lod_0_addr == indices_addr == 294. Matching on address
alone cannot separate them, and it fails silently -- the arrays still
parse, they just hold the wrong numbers.

Two corrections, and the first one is worth recording because it looked
like a data problem rather than a code one:

Each V4_8 element is 4 bytes and holds four indices, so an array of N
entries is count*4 bytes, not count. An earlier analysis compared the
unpack's element count against the position counts and concluded that
15,000 patches could not hold enough parents. That was the wrong unit;
the data was fine.

unk_06 and unk_0a suppress an empty tier. When the extra vertex-info
count is zero there is no extra array and its address is simply the next
tier's, so guarding those branches on the count lets the address fall
through to the array that exists. That one change took resolved LOD-0
entries from 28,739 to 341,306.

Result: of 886,841 vertex-info entries across all 27 levels, 516,492 are
common and have no link by design, and all 370,349 LOD entries resolve --
29,043 LOD-01 and 341,306 LOD-0 -- with zero failures. Each resolves to a
real first parent, a LOD-01 parent that is always a common entry, and
three positions inside the patch. A common entry never claims a parent.

Morphing can now be completed: a patch knows which LOD to draw, how far
to blend each vertex toward its parents, and where those parents are.

crates/gc-extract/tests/disc.rs

@@ -837,10 +837,10 @@ fn tfrag_lod_parents_resolve_for_every_entry() {
     let mut entries = 0usize;
     let mut lod01 = 0usize;
     let mut lod0 = 0usize;
-    let mut lod01_missing = 0usize;
-    let mut lod0_missing = 0usize;
     let mut resolved_common = 0usize;
     let mut common_total = 0usize;
+    let mut primary_failed = 0usize;
+    let mut extras_unresolved = 0usize;
 
     for n in 0..27 {
         let dir = data.join(format!("level{n}"));
@@ -875,6 +875,18 @@ fn tfrag_lod_parents_resolve_for_every_entry() {
                 .unwrap_or_else(|e| panic!("level{n} patch {i}: {e}"));
 
             common_total += g.vinfo_tier[0];
+            if n == 0 && patches < 5 {
+                eprintln!(
+                    "p{i}: tier={:?} n01={} n0={} p01={}b x01={}b p0={}b x0={}b",
+                    g.vinfo_tier,
+                    g.vu_positions_lod_01_count,
+                    g.vu_positions_lod_0_count,
+                    g.parent_indices_lod01.len(),
+                    g.unk_indices_2_lod01.len(),
+                    g.parent_indices_lod0.len(),
+                    g.unk_indices_2_lod0.len()
+                );
+            }
             for v in 0..g.vertex_info.len() {
                 entries += 1;
                 match gc_formats::tfrag::tfrag_lod_link(&g, v) {
@@ -905,10 +917,23 @@ fn tfrag_lod_parents_resolve_for_every_entry() {
                         }
                     }
                     None => {
-                        if v < g.vinfo_tier[1] {
+                        // Distinguish the two kinds of LOD entry. A *primary*
-                            lod01_missing += 1;
+                        // entry has a real position and a recorded parent and
-                        } else if v < g.vinfo_tier[1] + g.vinfo_tier[2] {
+                        // must always resolve; an *extra* entry is a
-                            lod0_missing += 1;
+                        // non-position bookkeeping record whose array address
+                        // can collide with the strip index array.
+                        let c = g.vinfo_tier[0];
+                        let kind = if v < c {
+                            None
+                        } else if v < c + g.vinfo_tier[1] {
+                            Some((g.vu_positions_lod_01_count, v - c, true))
+                        } else {
+                            Some((g.vu_positions_lod_0_count, v - c - g.vinfo_tier[1], false))
+                        };
+                        match kind {
+                            None => {}
+                            Some((prim, k, _)) if k < prim => primary_failed += 1,
+                            Some(_) => extras_unresolved += 1,
                         }
                     }
                 }
@@ -929,15 +954,19 @@ fn tfrag_lod_parents_resolve_for_every_entry() {
         lod01 > 0 && lod0 > 0,
         "both LOD tiers should occur in the data"
     );
+    assert_eq!(
+        primary_failed, 0,
+        "{primary_failed} primary LOD entries did not resolve"
+    );
+    assert_eq!(
+        extras_unresolved, 0,
+        "{extras_unresolved} extra LOD entries did not resolve"
+    );
+    // Common entries have no link; every other entry does.
+    assert_eq!(entries, common_total + lod01 + lod0);
 
-    // RAC2 does not record a parent for every LOD entry: the parent arrays are
-    // shorter than the VU header's position counts on most patches, so many
-    // LOD entries have no recorded parent. That is a property of the data, not
-    // a parse failure -- what matters is that every *recorded* parent resolves
-    // to real vertex info and real positions, which is asserted above.
     eprintln!(
         "{entries} vertex-info entries over {patches} patches: {common_total} common, \
-         {lod01} + {lod0} LOD entries with a recorded parent, \
+         {lod01} LOD-01 and {lod0} LOD-0, all resolved"
-         {lod01_missing} LOD-01 and {lod0_missing} LOD-0 entries without one"
     );
 }

crates/gc-formats/src/tfrag.rs

@@ -389,7 +389,12 @@ fn list_ranges(h: &TfragHeader) -> [(Option<usize>, ListRange); 5] {
 pub struct TfragVuHeader {
     pub positions_common_count: u16,
     pub positions_lod_01_count: u16,
+    /// Extra (non-position) LOD-01 vertex-info count. Zero means there is no
+    /// extra tier, and the "unknown indices 2" array is absent.
+    pub unk_06: u16,
     pub positions_lod_0_count: u16,
+    /// Extra (non-position) LOD-0 vertex-info count.
+    pub unk_0a: u16,
     pub positions_common_addr: u16,
     pub vertex_info_common_addr: u16,
     pub vertex_info_lod_01_addr: u16,
@@ -412,7 +417,9 @@ impl TfragVuHeader {
         Ok(TfragVuHeader {
             positions_common_count: w[0],
             positions_lod_01_count: w[2],
+            unk_06: w[3],
             positions_lod_0_count: w[4],
+            unk_0a: w[5],
             positions_common_addr: w[6],
             vertex_info_common_addr: w[7],
             vertex_info_lod_01_addr: w[9],
@@ -811,6 +818,9 @@ pub fn parse_tfrag(block: &[u8], header: &TfragHeader) -> Result<TfragGeometry>
                         }
                     }
                     (3, 2) => {
+                        // Each V4_8 element is 4 bytes and each element holds
+                        // four indices, so an array of N entries is N * 4
+                        // bytes long -- not N bytes.
                         let bytes = &payload[..v.count() * 4];
                         if !v.usn() && addr == vu.strips_addr {
                             strips = read_strips(payload, v.count())?;
@@ -820,13 +830,13 @@ pub fn parse_tfrag(block: &[u8], header: &TfragHeader) -> Result<TfragGeometry>
                             && vu.positions_lod_01_count != 0
                         {
                             g.parent_indices_lod01 = bytes.to_vec();
-                        } else if addr == vu.unk_indices_2_lod_01_addr {
+                        } else if addr == vu.unk_indices_2_lod_01_addr && vu.unk_06 != 0 {
                             g.unk_indices_2_lod01 = bytes.to_vec();
                         } else if addr == vu.parent_indices_lod_0_addr
                             && vu.positions_lod_0_count != 0
                         {
                             g.parent_indices_lod0 = bytes.to_vec();
-                        } else if addr == vu.unk_indices_2_lod_0_addr {
+                        } else if addr == vu.unk_indices_2_lod_0_addr && vu.unk_0a != 0 {
                             g.unk_indices_2_lod0 = bytes.to_vec();
                         } else {
                             return Err(Error {
@@ -1920,3 +1930,76 @@ mod lod_tests {
         assert_eq!(TfragMorphTier::Lod0.collapse_distance(d), 128.0);
     }
 }
+
+#[cfg(test)]
+mod parent_tests {
+    use super::*;
+
+    /// A patch with `common` + `lod01` + `lod0` vertex-info entries, one
+    /// position each, and parent arrays of the given lengths.
+    fn linked(common: usize, lod01: usize, lod0: usize, p01: usize, p0: usize) -> TfragGeometry {
+        let n = common + lod01 + lod0;
+        let mut g = TfragGeometry {
+            positions: (0..n).map(|i| [i as i32, 0, 0]).collect(),
+            vertex_info: (0..n)
+                .map(|i| TfragVertexInfo {
+                    s: 0,
+                    t: 0,
+                    parent: 0,
+                    vertex: (i * 2) as i16,
+                })
+                .collect(),
+            vinfo_tier: [common, lod01, lod0],
+            vu_positions_lod_01_count: lod01,
+            vu_positions_lod_0_count: lod0,
+            parent_indices_lod01: (0..p01).map(|i| (i % 4) as u8).collect(),
+            parent_indices_lod0: (0..p0).map(|i| (i % 4) as u8).collect(),
+            ..TfragGeometry::default()
+        };
+        // Every LOD entry's first parent is a common entry, as on disc.
+        for i in 0..n {
+            g.vertex_info[i].parent = 0;
+        }
+        g
+    }
+
+    #[test]
+    fn a_common_entry_has_no_link() {
+        let g = linked(4, 2, 2, 2, 2);
+        assert!(tfrag_lod_link(&g, 0).is_none());
+        assert!(tfrag_lod_link(&g, 3).is_none());
+        // Out of range too.
+        assert!(tfrag_lod_link(&g, 99).is_none());
+    }
+
+    #[test]
+    fn lod_entries_resolve_through_the_parent_arrays() {
+        let g = linked(4, 2, 2, 2, 2);
+        let l01 = tfrag_lod_link(&g, 4).expect("LOD-01 entry 0");
+        assert_eq!(l01.tier, TfragMorphTier::Lod01);
+        assert!(l01.morphs);
+        assert_eq!(l01.own_position, 4);
+        assert_eq!(l01.parent1_position, l01.parent1_vinfo);
+
+        let l0 = tfrag_lod_link(&g, 6).expect("LOD-0 entry 0");
+        assert_eq!(l0.tier, TfragMorphTier::Lod0);
+        assert!(l0.morphs);
+        assert_eq!(l0.own_position, 6);
+    }
+
+    #[test]
+    fn an_entry_beyond_its_parent_array_has_no_link() {
+        // Two LOD-0 entries but only one parent index recorded.
+        let g = linked(4, 0, 2, 0, 1);
+        assert!(tfrag_lod_link(&g, 4).is_some());
+        assert!(tfrag_lod_link(&g, 5).is_none());
+    }
+
+    #[test]
+    fn morph_parents_are_only_reported_for_entries_that_morph() {
+        let g = linked(4, 0, 2, 0, 2);
+        assert!(tfrag_morph_parents(&g, 4).is_some());
+        // A common entry never morphs.
+        assert!(tfrag_morph_parents(&g, 0).is_none());
+    }
+}

docs/plan/level-assets.md

@@ -451,38 +451,50 @@ A morph weight runs 0 to 1 across its tier's band: LOD-01 between D1 and D0,
 LOD-0 between D2 and D1. Past the far edge of the band an entry is not morphed
 but *replaced* by its first parent — a collapse, not a blend.
 
-### What is still unresolved: the parent-index arrays
+### Parent indices: settled, but only after two corrections
 
-The four V4_8 parent arrays (`parent_indices_lod01`, `unk_indices_2_lod01`,
+The four V4_8 arrays (`parent_indices_lod01`, `unk_indices_2_lod01`,
-`parent_indices_lod0`, `unk_indices_2_lod0`) are classified by VU address, but
+`parent_indices_lod0`, `unk_indices_2_lod0`) are classified by VU address,
-**the addresses are not unique**. On level 0 patch 1,
+and **the addresses are not unique**. On level 0 patch 1,
-`unk_indices_2_lod_01_addr == parent_indices_lod_0_addr == 291`, and the VU
+`unk_indices_2_lod_01_addr == parent_indices_lod_0_addr == 291`, and
-header's `unk_06` and `unk_0a` are both zero — there are no extra entries at
+`unk_indices_2_lod_0_addr == indices_addr == 294`. So address matching alone
-all, so nothing but the order the unpacks appear in distinguishes the two.
+cannot separate them, and getting it wrong is silent: the arrays still parse,
+they just hold the wrong numbers.
 
-The consequence is measurable. Across all 27 levels, of 886,841 vertex-info
+Two things were needed to fix it, and the first attempt at getting it wrong is
-entries, 516,492 are common and 57,782 LOD entries resolve to a parent. But
+worth recording because it looked like a data problem rather than a code one.
-316,548 LOD-0 entries have no recorded parent against only 28,739 that do, and
-only 3,274 of 18,102 patches carry enough V4_8 bytes to cover
-`positions_lod_01_count + positions_lod_0_count + extras`. The asymmetry is too
-large to be genuine, so one of two things is true: the parent arrays are being
-misassigned where addresses collide, or RAC2's VU header means something
-different by `positions_lod_0_count` than RAC1's does.
 
-The vertex-info tiers themselves *are* confirmed — the addresses at +9 and +11
+**Each V4_8 element is 4 bytes and holds four indices**, so an array of N
-do match the observed V4_16 unpacks, and the three tier counts sum to
+entries is `count * 4` bytes, not `count`. An early analysis compared the
-`vert_count` on every patch. So the header layout is right; it is specifically
+unpack's element count against the position counts and concluded that 15,000
-the parent arrays that are unresolved.
+patches could not possibly hold enough parents. That was the wrong unit; the
+data was fine.
 
-Ruled out for now: address-only classification (collides), and assuming the
+**`unk_06` and `unk_0a` suppress an empty tier.** When the extra vertex-info
-arrays are one entry per position (fails on 15,000 patches). Not yet tried:
+count is zero there is no extra array, and its VU address is simply the next
-ordering by VU address rather than by declaration, and using `unk_06`/`unk_0a`
+tier's. Guarding those two branches on `unk_06 != 0` / `unk_0a != 0` lets the
-to suppress an empty tier before falling through to the next candidate.
+address fall through to the array that actually exists. This one change took
+resolved LOD-0 entries from 28,739 to 341,306.
+
+### Verification
+
+Every vertex-info entry across all 27 levels is now accounted for: of 886,841
+entries, 516,492 are common and have no link by design, and **all 370,349 LOD
+entries resolve** — 29,043 LOD-01 and 341,306 LOD-0 — with zero failures. Each
+resolves to a first parent that is real vertex info, a LOD-01 parent that is
+always a common entry, and three positions inside the patch. A common entry
+never claims a parent.
+
+Ruled out along the way: address-only classification (addresses collide), and
+one parent array entry per unpack element (wrong unit).
 
 ### Still open
 
-The parent arrays above, and the GS register values the level load builds from
+The GS register values the level load builds from the texture entry (TBP0/TBP1
-the texture entry (TBP0/TBP1 and so on) for a renderer that emulates the GS
+and so on), which a renderer needs if it emulates the GS rather than resolving
-rather than resolving addresses itself. A tfrag can be turned into positioned,
+texture addresses itself, and the guard-band clip test that sends patches
-UV-mapped, textured triangles and told which LOD to draw; morphing cannot be
+crossing the guard band to the clipping program.
-completed until the parent arrays resolve.
+
+A tfrag can now be turned into positioned, UV-mapped, textured triangles,
+told which LOD to draw, and morphed toward its parents at the right rate.
+Everything needed for a first frame is parsed and verified.

docs/roadmap.md

@@ -20,7 +20,8 @@ early, while they are still cheap to retire.
       header, and the `data` extent is parsed as a container of byte ranges. The
       geometry decompresses: level 0 yields 16,969,296 bytes of level content
       from a 9.7 MB stream.
-- [ ] Next: tfrag parent indices, then the first drawable frame.
+- [ ] Next: the first drawable frame — build the GS register set and a
+      Bevy fly-camera over level 0's tfrags.
 
 The disc being the exact build the reference projects pin is the single
 biggest favourable fact so far: no disc conversion, no regional timing work,
@@ -147,11 +148,12 @@ Phase 1 hands over the extents; this phase works out what each one contains.
       the coarser mode and distance never moves back to finer detail. Morph
       weights run 0..1 across each tier's band, and collapse replaces an entry
       with its first parent rather than blending it.
-- [ ] tfrag parent indices: the four V4_8 arrays are classified by VU address,
+- [x] **tfrag parent indices.** The four V4_8 arrays are classified by VU
-      but the addresses collide (`unk_06` and `unk_0a` are zero, so an absent
+      address, but the addresses collide, and each V4_8 element holds four
-      tier shares the next one's address). 57,782 of 374,349 LOD entries
+      indices so an array is `count * 4` bytes rather than `count`. With
-      resolve; the 316,548 unresolved LOD-0 entries are far too lopsided to be
+      `unk_06`/`unk_0a` suppressing absent tiers, all 370,349 LOD entries
-      genuine. Tried and ruled out in `plan/level-assets.md`.
+      across the 27 levels resolve: 29,043 LOD-01 and 341,306 LOD-0, zero
+      failures. See `plan/level-assets.md` for the two corrections.
 - [ ] `gameplay` — begins `WAD` + a per-level byte, and does *not* decode with
       the WAD LZ77 decoder. All 27 levels fail identically, so the input is
       wrong rather than the decoder. Ruled out: bit order, header length, the