Parse tfrag terrain blocks and decode all vertex positions
Adds gc-formats::tfrag and a gated real-disc test that decodes every terrain patch on all 27 levels. Two rules decide whether this works at all, and both were wrong on the first attempt: The vector format is encoded in the VIF command byte, not the immediate. Bits 2-3 are vn (component count minus one) and bits 0-1 are vl, giving element_size = ((32 >> vl) * (vn + 1)) / 8. Terrain positions are vn=2, vl=1 (command 0x69) at 6 bytes. Reading the format from the immediate's low nibble -- the more commonly quoted layout -- yields 12 or 16 bytes and misaligns everything after it. Unpack payloads pad to 4 bytes, and several non-unpack commands carry inline data: STROW/STCOL 16, STMASK 4, MPG num*8, DIRECT/DIRECTHL imm*16. Ignoring those desynchronises the walk, and a desynchronised walk still finds code words that look like positions. Positions are also not all in one list. vert_count is the total across three tiers, the LOD-2 list precedes the common one in offset order, and the LOD-0-only list starts at shared_ofs + lod_1_size*16 rather than at an offset of its own. Scanning only the common list matches about a quarter of patches, each exactly -- which is what made it convincing. The invariant is that every patch yields exactly its own vert_count. It caught both errors, so tfrag_positions returns None on a mismatch rather than partial geometry. All 27 levels: 18,102 patches, 821,439 vertices, zero mismatches. Still open: strips, triangle indices and the RGBA vertex-info arrays. Positions alone are not drawable.
This commit is contained in:
@@ -205,6 +205,9 @@ Next: rather than guess the header further, identify `tfrag` by its own
|
||||
structure. Terrain is the largest subsystem and the one that decides whether
|
||||
anything draws, so it is the right next target even though it is the harder one.
|
||||
|
||||
*(Done — see "tfrag terrain" below. Identifying tfrag by its own structure
|
||||
worked where the class-table guess did not.)*
|
||||
|
||||
## `data`: the real target, untouched
|
||||
|
||||
6,484 sectors on level 0, down to 2,102 on level 24 — 4.2 to 21.9 MB. This is
|
||||
@@ -237,4 +240,80 @@ hypothesis, test it against **all 27 levels at once**, and require an invariant
|
||||
that holds everywhere. The sector/byte ambiguity in Phase 1 survived because it
|
||||
happened to be correct on the one container checked by hand. The level header
|
||||
was fixed in a single pass because the tiling check failed immediately on
|
||||
levels 1, 2, 4, 7, 8, 11, 19 and 20.
|
||||
levels 1, 2, 4, 7, 8, 11, 19 and 20.
|
||||
## tfrag terrain: block structure and vertex positions
|
||||
|
||||
Settled. `gc-formats::tfrag` parses the block, and a gated test decodes every
|
||||
patch on all 27 levels.
|
||||
|
||||
### Block and patch framing
|
||||
|
||||
The tfrags block opens with a 12-byte header: `table_offset` (always `0x40`),
|
||||
`tfrag_count`, and an `f32` LOD distance (`32.0`). The patch table follows at
|
||||
`table_offset`, `tfrag_count` records of `0x40` bytes each. On level 0 that is
|
||||
833 patches.
|
||||
|
||||
A patch header's `data` offset is **relative to `table_offset`**, not to the
|
||||
start of the block. Getting this wrong happens to work for the first patch and
|
||||
silently produces garbage afterwards.
|
||||
|
||||
### Positions live in three lists, not one
|
||||
|
||||
`vert_count` is the *total* across all three position tiers. Within a patch's
|
||||
data block the VIF lists do not run in ascending offset order — the LOD-2 list
|
||||
sits *before* the common list — and the LOD-0-only list has no offset of its
|
||||
own, starting instead at `shared_ofs + lod_1_size * 16`. The ranges, in transfer
|
||||
order, are:
|
||||
|
||||
| List | Range (from the patch data block) | Holds |
|
||||
|---|---|---|
|
||||
| common | `shared_ofs .. lod_1_ofs` | VU header, ad-gifs, vertex info, origin, positions |
|
||||
| LOD 2 | `lod_2_ofs .. shared_ofs` | strips, indices |
|
||||
| LOD 1 | `lod_1_ofs .. lod_0_ofs` | strips, indices |
|
||||
| LOD 0+1 | `lod_0_ofs .. shared_ofs + lod_1_size*16` | parents, vertex info, positions |
|
||||
| LOD 0 | `shared_ofs + lod_1_size*16 .. rgba_ofs` | positions, strips, indices, parents |
|
||||
|
||||
Scanning only the common list and expecting `num == vert_count` finds positions
|
||||
in roughly a quarter of patches. The other three quarters keep most of their
|
||||
positions in the LOD lists. That was the first wrong turn and it looked
|
||||
plausible: every patch it did match matched *exactly*.
|
||||
|
||||
### The VIF1 code word
|
||||
|
||||
Two rules had to be right before any of this worked, and both were wrong on the
|
||||
first attempt.
|
||||
|
||||
**The vector format lives in the command byte, not the immediate.** Bits 2-3 of
|
||||
the command are `vn` (component count minus one) and bits 0-1 are `vl` (0 =
|
||||
32-bit, 1 = 16-bit, 2 = 8-bit, 3 = 5-bit), so
|
||||
|
||||
element_size = ((32 >> vl) * (vn + 1)) / 8
|
||||
|
||||
Terrain positions are `vn=2, vl=1` — command `0x69` — at 6 bytes each. Reading
|
||||
the format out of the immediate's low nibble, which is the more commonly quoted
|
||||
layout, yields 12 or 16 bytes and quietly misaligns everything after it.
|
||||
|
||||
**Unpack payloads pad to 4 bytes, and several non-unpack commands carry inline
|
||||
data.** An unpack advances `(count * element_size + 3) & !3`. `STROW`/`STCOL`
|
||||
carry 16 bytes, `STMASK` 4, `MPG` `num * 8`, `DIRECT`/`DIRECTHL` `imm * 16`;
|
||||
every other non-unpack is a bare 4-byte code word. Ignoring the first four
|
||||
desynchronises the walk, and a desynchronised walk still finds code words that
|
||||
look like positions — it just finds the wrong number of them.
|
||||
|
||||
The 4-byte padding is easy to misread. With 75 elements the payload is 450
|
||||
bytes and the following word lands at +452, where `0x01000404` (STCYCL) reads
|
||||
perfectly plausibly. The word at +450 is `0x04040000` (ITOP), which is the real
|
||||
one. Taking the plausible-looking reading is how this was briefly wrong.
|
||||
|
||||
### Verification
|
||||
|
||||
Requiring every patch to yield exactly its own `vert_count` is the invariant
|
||||
that caught both errors, and it is the check the parser is built around: a
|
||||
mismatch means the walk desynchronised, so `tfrag_positions` returns `None`
|
||||
rather than partial geometry. Across all 27 levels: 18,102 patches,
|
||||
821,439 vertices, zero mismatches.
|
||||
|
||||
### Still open
|
||||
|
||||
Strips and triangle indices, the RGBA vertex-info arrays, texture binding, and
|
||||
the LOD/morph/cull rules in `TfragProc`. Positions alone are not drawable.
|
||||
|
||||
+11
-4
@@ -20,7 +20,7 @@ early, while they are still cheap to retire.
|
||||
header, and the `data` extent is parsed as a container of byte ranges. The
|
||||
geometry decompresses: level 0 yields 16,969,296 bytes of level content
|
||||
from a 9.7 MB stream.
|
||||
- [ ] Next: the geometry directory's class tables, then `tfrag` terrain.
|
||||
- [ ] Next: tfrag triangle topology, then the geometry directory's class tables.
|
||||
|
||||
The disc being the exact build the reference projects pin is the single
|
||||
biggest favourable fact so far: no disc conversion, no regional timing work,
|
||||
@@ -114,9 +114,16 @@ Phase 1 hands over the extents; this phase works out what each one contains.
|
||||
- [x] Region layout of `core_data` established: tfrag, occlusion, sky, collision
|
||||
and `textures_base` are offsets, strictly increasing on all 27 levels.
|
||||
Terrain is `[0, occlusion)` and carries inline texels.
|
||||
- [ ] **tfrag vertex format: not recovered.** The 64-byte record framing and the
|
||||
scale-8 fixed-point reading were both tested and disproven; see
|
||||
`plan/level-assets.md` for what was ruled out.
|
||||
- [x] **tfrag vertex format: recovered.** The tfrags block header gives a table
|
||||
offset (`0x40`) and a patch count; each patch is a 0x40-byte header whose
|
||||
`data` offset is relative to the table, not to the block. Positions are
|
||||
V3_16 payloads inside a VIF1 command list, split across three lists per
|
||||
patch (common, LOD 0+1, LOD 0). The whole vector format is encoded in the
|
||||
command byte and every element size follows from it. Verified on all 27
|
||||
levels: **18,102 patches, 821,439 vertices**, each patch yielding exactly
|
||||
its own `vert_count`. See `plan/level-assets.md` and `gc-formats::tfrag`.
|
||||
- [ ] tfrag topology: strips, triangle indices and the RGBA vertex-info arrays,
|
||||
which is what turns positions into drawable geometry.
|
||||
- [ ] `gameplay` — begins `WAD` + a per-level byte, and does *not* decode with
|
||||
the WAD LZ77 decoder. All 27 levels fail identically, so the input is
|
||||
wrong rather than the decoder. Ruled out: bit order, header length, the
|
||||
|
||||
Reference in New Issue
Block a user