Parse tfrag terrain blocks and decode all vertex positions

Adds gc-formats::tfrag and a gated real-disc test that decodes every
terrain patch on all 27 levels.

Two rules decide whether this works at all, and both were wrong on the
first attempt:

The vector format is encoded in the VIF command byte, not the immediate.
Bits 2-3 are vn (component count minus one) and bits 0-1 are vl, giving
element_size = ((32 >> vl) * (vn + 1)) / 8. Terrain positions are vn=2,
vl=1 (command 0x69) at 6 bytes. Reading the format from the immediate's
low nibble -- the more commonly quoted layout -- yields 12 or 16 bytes
and misaligns everything after it.

Unpack payloads pad to 4 bytes, and several non-unpack commands carry
inline data: STROW/STCOL 16, STMASK 4, MPG num*8, DIRECT/DIRECTHL
imm*16. Ignoring those desynchronises the walk, and a desynchronised
walk still finds code words that look like positions.

Positions are also not all in one list. vert_count is the total across
three tiers, the LOD-2 list precedes the common one in offset order, and
the LOD-0-only list starts at shared_ofs + lod_1_size*16 rather than at
an offset of its own. Scanning only the common list matches about a
quarter of patches, each exactly -- which is what made it convincing.

The invariant is that every patch yields exactly its own vert_count. It
caught both errors, so tfrag_positions returns None on a mismatch rather
than partial geometry. All 27 levels: 18,102 patches, 821,439 vertices,
zero mismatches.

Still open: strips, triangle indices and the RGBA vertex-info arrays.
Positions alone are not drawable.
This commit is contained in:
2026-10-10 02:07:02 +02:00
parent 3ec548efe8
commit 36641b4cf8
5 changed files with 782 additions and 5 deletions
+80 -1
View File
@@ -205,6 +205,9 @@ Next: rather than guess the header further, identify `tfrag` by its own
structure. Terrain is the largest subsystem and the one that decides whether
anything draws, so it is the right next target even though it is the harder one.
*(Done — see "tfrag terrain" below. Identifying tfrag by its own structure
worked where the class-table guess did not.)*
## `data`: the real target, untouched
6,484 sectors on level 0, down to 2,102 on level 24 — 4.2 to 21.9 MB. This is
@@ -237,4 +240,80 @@ hypothesis, test it against **all 27 levels at once**, and require an invariant
that holds everywhere. The sector/byte ambiguity in Phase 1 survived because it
happened to be correct on the one container checked by hand. The level header
was fixed in a single pass because the tiling check failed immediately on
levels 1, 2, 4, 7, 8, 11, 19 and 20.
levels 1, 2, 4, 7, 8, 11, 19 and 20.
## tfrag terrain: block structure and vertex positions
Settled. `gc-formats::tfrag` parses the block, and a gated test decodes every
patch on all 27 levels.
### Block and patch framing
The tfrags block opens with a 12-byte header: `table_offset` (always `0x40`),
`tfrag_count`, and an `f32` LOD distance (`32.0`). The patch table follows at
`table_offset`, `tfrag_count` records of `0x40` bytes each. On level 0 that is
833 patches.
A patch header's `data` offset is **relative to `table_offset`**, not to the
start of the block. Getting this wrong happens to work for the first patch and
silently produces garbage afterwards.
### Positions live in three lists, not one
`vert_count` is the *total* across all three position tiers. Within a patch's
data block the VIF lists do not run in ascending offset order — the LOD-2 list
sits *before* the common list — and the LOD-0-only list has no offset of its
own, starting instead at `shared_ofs + lod_1_size * 16`. The ranges, in transfer
order, are:
| List | Range (from the patch data block) | Holds |
|---|---|---|
| common | `shared_ofs .. lod_1_ofs` | VU header, ad-gifs, vertex info, origin, positions |
| LOD 2 | `lod_2_ofs .. shared_ofs` | strips, indices |
| LOD 1 | `lod_1_ofs .. lod_0_ofs` | strips, indices |
| LOD 0+1 | `lod_0_ofs .. shared_ofs + lod_1_size*16` | parents, vertex info, positions |
| LOD 0 | `shared_ofs + lod_1_size*16 .. rgba_ofs` | positions, strips, indices, parents |
Scanning only the common list and expecting `num == vert_count` finds positions
in roughly a quarter of patches. The other three quarters keep most of their
positions in the LOD lists. That was the first wrong turn and it looked
plausible: every patch it did match matched *exactly*.
### The VIF1 code word
Two rules had to be right before any of this worked, and both were wrong on the
first attempt.
**The vector format lives in the command byte, not the immediate.** Bits 2-3 of
the command are `vn` (component count minus one) and bits 0-1 are `vl` (0 =
32-bit, 1 = 16-bit, 2 = 8-bit, 3 = 5-bit), so
element_size = ((32 >> vl) * (vn + 1)) / 8
Terrain positions are `vn=2, vl=1` — command `0x69` — at 6 bytes each. Reading
the format out of the immediate's low nibble, which is the more commonly quoted
layout, yields 12 or 16 bytes and quietly misaligns everything after it.
**Unpack payloads pad to 4 bytes, and several non-unpack commands carry inline
data.** An unpack advances `(count * element_size + 3) & !3`. `STROW`/`STCOL`
carry 16 bytes, `STMASK` 4, `MPG` `num * 8`, `DIRECT`/`DIRECTHL` `imm * 16`;
every other non-unpack is a bare 4-byte code word. Ignoring the first four
desynchronises the walk, and a desynchronised walk still finds code words that
look like positions — it just finds the wrong number of them.
The 4-byte padding is easy to misread. With 75 elements the payload is 450
bytes and the following word lands at +452, where `0x01000404` (STCYCL) reads
perfectly plausibly. The word at +450 is `0x04040000` (ITOP), which is the real
one. Taking the plausible-looking reading is how this was briefly wrong.
### Verification
Requiring every patch to yield exactly its own `vert_count` is the invariant
that caught both errors, and it is the check the parser is built around: a
mismatch means the walk desynchronised, so `tfrag_positions` returns `None`
rather than partial geometry. Across all 27 levels: 18,102 patches,
821,439 vertices, zero mismatches.
### Still open
Strips and triangle indices, the RGBA vertex-info arrays, texture binding, and
the LOD/morph/cull rules in `TfragProc`. Positions alone are not drawable.