Reconcile roadmap.md with the parsers that actually exist

Findings were going to docs/plan/level-assets.md and docs/formats/*.md while
roadmap.md kept listing work as open that has been done for some time. The worst
case was occlusion appearing twice: open at one line with "2 to 25 sectors, small
enough to read whole", and done further down with 412521 cells over 140939 masks.
The open line is deleted.

Also corrected:

- tfrag was listed flatly open while six tfrag entries beneath it were checked.
  Now partial: format, topology, textures, LOD, normals and colour are done;
  morph targets and strip/chain bookkeeping are not.
- sky was lumped in with volumes as open. Sky is done -- 149 sub-objects across 27
  levels -- so it has its own entry, keeping the word at +0x0c documented as not
  a count, since it disagrees with the table on most levels and reads 196617 on
  level 2 against 4 sub-objects.
- level_overlay was open. It now has a parser and a 27/27 test, so it is partial
  rather than done: header, constant block and level data parse, +0x10..+0x1220 is
  identical on every level, and the word at +0x0c falls below the file length on 17
  levels and above on 10. The header words are still unnamed and the Phase 3
  seven-section parser is untouched.
- sound_bank was flatly open. It is partial: container framing parses and bounds
  are verified on 27/27, but the SBlk record fields are not decoded and DAW shows
  no stride, consistent with ADPCM.
- tie now records what is actually known, including the two indirection chains
  that have not been walked: class table entries point into core_data.wad, and the
  header's packets[0..3] resolve against the lump base. Level 0 has 74 tie, 238
  moby and 7 shrub classes.
- collision now separates the header, which is decoded, from the cell layout,
  which is not -- including the 17-entry table at +5932 against a 75-cell grid and
  the disproved 64-byte RAM stride.
- moby_anim records the negative result: per-chunk-type record counts are
  level-independent, so nothing in the chunk set is keyed by moby class.

The "Where we are" section now states plainly what has parsers and that what
remains open is naming rather than locating, and names the five types with no
decoded fields. Reading that list is how you would have known to ask.

Phase 2 remains in progress. Volumes and moby_anim have no located structure and
SBlk records, tie vertices and collision cells have located structure without
decoded fields.
This commit is contained in:
2026-10-12 03:43:44 +02:00
parent d8b8ccc6d4
commit e5d267a810
+79 -13
View File
@@ -21,6 +21,19 @@ early, while they are still cheap to retire.
header, and the `data` extent is parsed as a container of byte ranges. The
geometry decompresses: level 0 yields 16,969,296 bytes of level content
from a 9.7 MB stream.
- [x] **Parsers with real-disc tests, per asset type.** `level_wad`, `data`,
`core_index` class tables, `tfrag` (format, topology, textures, LOD,
normals, colour), `occlusion`, `sky`, `moby`, `collision` framing and
header, `shrub` (blob, packets, palette, billboard), the post-shrub
`chunks` container, `sound_bank` framing and `level_overlay`. 227 tests
green against the real disc. **What is open is naming, not locating** --
see the Phase 2 list for the four types that still have no decoded
fields: `tie` vertices, collision cells, `SBlk` records, `volumes` and
`moby_anim`.
- [x] **Lighting verified against the running game.** PCSX2 2.8.2 with savestates
for level 0 and level 1. The tfrag lighting pass is byte-exact on both
against the game's own RAM. Setup and method in
`plan/pcsx2-verification.md`.
- [x] **First drawable frame.** `gc-engine` loads a level, applies LOD selection
and morphing for the camera, groups triangles by texture and draws them
under a Bevy fly-camera. Level 0 renders as 26,758 textured triangles
@@ -215,15 +228,24 @@ Phase 1 hands over the extents; this phase works out what each one contains.
with the WAD LZ77 decoder, bit order and header length ruled out" note
was wrong about the cause: the decoder was right and the input was not
what it looked like. Corrected in `plan/level-assets.md`.
- [ ] `occlusion` — 2 to 25 sectors, small enough to read whole.
- [ ] `sound_bank` — `SBlk` and `DAW` signatures found in all 27.
- [~] `sound_bank` — container framing parsed (`SoundBankHeader::parse`,
`parse_sound_bank`); `SBlk` and `DAW` bounds verified on 27/27. The
**record fields inside `SBlk` are not decoded**, and `DAW` shows no
detectable stride, consistent with ADPCM. See the note under
`level_overlay` below for what the game's own strings say about loading.
- [x] `data` — **not geometry.** It is the level's file descriptor: its
first table carries the exact byte lengths of `overlay.bin`,
`core_index.bin` and `gs_ram.bin` at fixed `u32` indices 1, 3 and 5 on
every level checked. What follows the descriptor is still unknown.
- [ ] `tfrag` — terrain: LOD chains, strips, morph targets
- [ ] `tie` — instanced environment geometry, fat vertices
- [~] `tfrag` — vertex format, topology, texture bindings, per-vertex colour
and normals, LOD selection and parent indices are all recovered and
verified. **Still open:** morph targets and the strip/chain bookkeeping.
- [ ] `tie` — class table and the 0x48-byte header agree with R&C1; the
**vertex layout is not decoded**. The class table entries are pointers
into `core_data.wad`, and the header's `packets[0..3]` LOD offsets are
resolved against the lump base -- neither chain has been walked end to
end yet. Level 0 has 74 tie classes, 238 moby and 7 shrub.
- [~] `shrub` — class table done (48-byte entries, not 32; reading shrub at
32 still gives 231 sane-looking entries of 358, which is the hazard).
Geometry header parsed and checked field by field on all 358 spans:
@@ -250,13 +272,30 @@ Phase 1 hands over the extents; this phase works out what each one contains.
makes the record size fall out of the data rather than being assumed.
1,270,802 vertices over 4,560 class spans; 116 spans carry no geometry.
Vertices are 16-byte VIF-packed half-float quads, handed back raw.
- [ ] `moby_anim` — skeletal animation, still open
- [x] `collision` — container framing. A 0x40-byte header self-describes: it
carries its own size and the payload size, and `header + payload` stays
inside `core_data` on all 27 levels. 41,481,520 payload bytes across 19
levels; 8 levels declare zero length. The cell layout is **not** decoded
-- the payload does not walk as fixed-stride cells, and variable-length
faces stop partway into the first cell rather than tiling it.
- [ ] `moby_anim` — **still open, and the obvious reading is ruled out.** An
instance table would have to scale with the level's moby class count, but
**per-chunk-type record counts are level-independent** -- the same types
report 90, 77, 50 and 47 on every level tried, and the largest type
totals 2,430 = 90 x 27. Nothing in the chunk set is keyed by moby class.
The types also pair up (two types both reporting 90 on every level,
another pair both 77), which looks like two views of one fixed table
rather than two independent tables.
- [x] `collision` — container framing and header fields. A 0x40-byte header
self-describes: it carries its own size and the payload size, and
`header + payload` stays inside `core_data` on all 27 levels. 41,481,520
payload bytes across 19 levels; 8 declare zero length. Header bytes `+8`
reads zero on 22 levels and a large value on 7/9/12/13/14; the payload's
two leading `u16` are non-zero on all 19 non-empty levels (1-74 and
13-112) and are reported as `CollisionHeader::first`/`::second`.
- [ ] The collision **cell layout is not decoded.** The payload does not walk
as fixed-stride cells. On level 4 a table starts at payload `+5932` and
runs **17 entries** -- deltas rise then fall -- against a 75-cell grid,
with no arithmetic relating the two. RAM is reshaped on load: level 1's
`24 x 16` grid dimensions appear in memory, level 0's `11 x 33` appear
nowhere, so disc-side decoding has to work against the reshaped form.
A 64-byte RAM stride looked promising and is **disproved** -- the longest
such run is 27 records of identity matrices, one per level. See
`plan/pcsx2-verification.md`.
- [x] `occlusion` — the PVS grid. A z/y/x tree of u16 node offsets over 4x4x4-unit
cells, populated cells naming 128-byte masks. **412,521 cells over
140,939 masks** across the 24 levels that have one; the mask array ends
@@ -264,8 +303,35 @@ Phase 1 hands over the extents; this phase works out what each one contains.
data, and an offset of 0 means *absent* rather than "at the start" — the
latter parses as a grid whose mask array happens to fit, which is how it
nearly went wrong.
- [ ] `sky`, `volumes`; textures are done (CSM1 CLUT + PS2 alpha)
- [ ] `level_overlay` — the seven-section parser and relocation from Phase 3
- [x] `sky` — 149 sub-objects across all 27 levels, each with a count, a
four-float bounding sphere and texture dimensions. The offset table at
`+0x20` is self-limiting: entries run while strictly inside the region.
**The word at `+0x0c` is not a count** -- it disagrees with the table on
most levels and reads 196,617 on level 2 against 4 sub-objects.
- [ ] `volumes` — **not located.** Nothing examined so far presents itself as a
volume: `core_data`, the chunk region and `gameplay.bin` all yield
structures that are accounted for by something else.
- [~] `level_overlay` — header, constant block and level data parsed
(`gc_formats::overlay`, 27/27). `+0x10..+0x1220` -- 4,624 bytes -- is
byte-identical on every level; the word at `+0x0c` is **not** the file
length, falling below it on 17 levels and above on 10. The debug string
pool it opens with carries `Camera_CollPrimTest WARNING! - grid out of
bounds! (RAR)`, independent support for reading the collision payload's
two leading `u16`s as grid dimensions. **The header words are not named
and the level-specific tail is not structured yet**; the Phase 3
seven-section parser and relocation are untouched.
- [x] **The post-shrub chunk container.** A 16-byte header, a 10-byte zero gap,
then offsets tagged `0xF0000000 | low24` resolving to `chunk_start + low`;
header byte 14 is the record count, equal to `offsets.len()` on 1,123 of
1,123 chunks across all 27 levels. **77 distinct chunk types, 1,123
chunks, 19,546 records, 50 strides.** Moby class IDs enrich at 9,493 hits
against 246.1 expected with contiguous per-type arrays -- but the arrays
land in a different place for each type, so no single pooled scan finds
them. The record *fields* are not decoded.
- [x] **Moby class arrays are contiguous per type, and the pooled scan was
wrong.** See rule 3a in `AGENTS.md`: the original claim of a `u16` array at
slots 129-140 was the largest-stride type dominating an aggregate. Per
type, every one of the 26 places its array somewhere else.
**Gate:** every parser has a golden test against bytes from the real disc, and
reports mismatched header counts as failures rather than warnings.