Reconcile roadmap.md with the parsers that actually exist
Findings were going to docs/plan/level-assets.md and docs/formats/*.md while roadmap.md kept listing work as open that has been done for some time. The worst case was occlusion appearing twice: open at one line with "2 to 25 sectors, small enough to read whole", and done further down with 412521 cells over 140939 masks. The open line is deleted. Also corrected: - tfrag was listed flatly open while six tfrag entries beneath it were checked. Now partial: format, topology, textures, LOD, normals and colour are done; morph targets and strip/chain bookkeeping are not. - sky was lumped in with volumes as open. Sky is done -- 149 sub-objects across 27 levels -- so it has its own entry, keeping the word at +0x0c documented as not a count, since it disagrees with the table on most levels and reads 196617 on level 2 against 4 sub-objects. - level_overlay was open. It now has a parser and a 27/27 test, so it is partial rather than done: header, constant block and level data parse, +0x10..+0x1220 is identical on every level, and the word at +0x0c falls below the file length on 17 levels and above on 10. The header words are still unnamed and the Phase 3 seven-section parser is untouched. - sound_bank was flatly open. It is partial: container framing parses and bounds are verified on 27/27, but the SBlk record fields are not decoded and DAW shows no stride, consistent with ADPCM. - tie now records what is actually known, including the two indirection chains that have not been walked: class table entries point into core_data.wad, and the header's packets[0..3] resolve against the lump base. Level 0 has 74 tie, 238 moby and 7 shrub classes. - collision now separates the header, which is decoded, from the cell layout, which is not -- including the 17-entry table at +5932 against a 75-cell grid and the disproved 64-byte RAM stride. - moby_anim records the negative result: per-chunk-type record counts are level-independent, so nothing in the chunk set is keyed by moby class. The "Where we are" section now states plainly what has parsers and that what remains open is naming rather than locating, and names the five types with no decoded fields. Reading that list is how you would have known to ask. Phase 2 remains in progress. Volumes and moby_anim have no located structure and SBlk records, tie vertices and collision cells have located structure without decoded fields.
This commit is contained in:
+79
-13
@@ -21,6 +21,19 @@ early, while they are still cheap to retire.
|
||||
header, and the `data` extent is parsed as a container of byte ranges. The
|
||||
geometry decompresses: level 0 yields 16,969,296 bytes of level content
|
||||
from a 9.7 MB stream.
|
||||
- [x] **Parsers with real-disc tests, per asset type.** `level_wad`, `data`,
|
||||
`core_index` class tables, `tfrag` (format, topology, textures, LOD,
|
||||
normals, colour), `occlusion`, `sky`, `moby`, `collision` framing and
|
||||
header, `shrub` (blob, packets, palette, billboard), the post-shrub
|
||||
`chunks` container, `sound_bank` framing and `level_overlay`. 227 tests
|
||||
green against the real disc. **What is open is naming, not locating** --
|
||||
see the Phase 2 list for the four types that still have no decoded
|
||||
fields: `tie` vertices, collision cells, `SBlk` records, `volumes` and
|
||||
`moby_anim`.
|
||||
- [x] **Lighting verified against the running game.** PCSX2 2.8.2 with savestates
|
||||
for level 0 and level 1. The tfrag lighting pass is byte-exact on both
|
||||
against the game's own RAM. Setup and method in
|
||||
`plan/pcsx2-verification.md`.
|
||||
- [x] **First drawable frame.** `gc-engine` loads a level, applies LOD selection
|
||||
and morphing for the camera, groups triangles by texture and draws them
|
||||
under a Bevy fly-camera. Level 0 renders as 26,758 textured triangles
|
||||
@@ -215,15 +228,24 @@ Phase 1 hands over the extents; this phase works out what each one contains.
|
||||
with the WAD LZ77 decoder, bit order and header length ruled out" note
|
||||
was wrong about the cause: the decoder was right and the input was not
|
||||
what it looked like. Corrected in `plan/level-assets.md`.
|
||||
- [ ] `occlusion` — 2 to 25 sectors, small enough to read whole.
|
||||
- [ ] `sound_bank` — `SBlk` and `DAW` signatures found in all 27.
|
||||
- [~] `sound_bank` — container framing parsed (`SoundBankHeader::parse`,
|
||||
`parse_sound_bank`); `SBlk` and `DAW` bounds verified on 27/27. The
|
||||
**record fields inside `SBlk` are not decoded**, and `DAW` shows no
|
||||
detectable stride, consistent with ADPCM. See the note under
|
||||
`level_overlay` below for what the game's own strings say about loading.
|
||||
- [x] `data` — **not geometry.** It is the level's file descriptor: its
|
||||
first table carries the exact byte lengths of `overlay.bin`,
|
||||
`core_index.bin` and `gs_ram.bin` at fixed `u32` indices 1, 3 and 5 on
|
||||
every level checked. What follows the descriptor is still unknown.
|
||||
|
||||
- [ ] `tfrag` — terrain: LOD chains, strips, morph targets
|
||||
- [ ] `tie` — instanced environment geometry, fat vertices
|
||||
- [~] `tfrag` — vertex format, topology, texture bindings, per-vertex colour
|
||||
and normals, LOD selection and parent indices are all recovered and
|
||||
verified. **Still open:** morph targets and the strip/chain bookkeeping.
|
||||
- [ ] `tie` — class table and the 0x48-byte header agree with R&C1; the
|
||||
**vertex layout is not decoded**. The class table entries are pointers
|
||||
into `core_data.wad`, and the header's `packets[0..3]` LOD offsets are
|
||||
resolved against the lump base -- neither chain has been walked end to
|
||||
end yet. Level 0 has 74 tie classes, 238 moby and 7 shrub.
|
||||
- [~] `shrub` — class table done (48-byte entries, not 32; reading shrub at
|
||||
32 still gives 231 sane-looking entries of 358, which is the hazard).
|
||||
Geometry header parsed and checked field by field on all 358 spans:
|
||||
@@ -250,13 +272,30 @@ Phase 1 hands over the extents; this phase works out what each one contains.
|
||||
makes the record size fall out of the data rather than being assumed.
|
||||
1,270,802 vertices over 4,560 class spans; 116 spans carry no geometry.
|
||||
Vertices are 16-byte VIF-packed half-float quads, handed back raw.
|
||||
- [ ] `moby_anim` — skeletal animation, still open
|
||||
- [x] `collision` — container framing. A 0x40-byte header self-describes: it
|
||||
carries its own size and the payload size, and `header + payload` stays
|
||||
inside `core_data` on all 27 levels. 41,481,520 payload bytes across 19
|
||||
levels; 8 levels declare zero length. The cell layout is **not** decoded
|
||||
-- the payload does not walk as fixed-stride cells, and variable-length
|
||||
faces stop partway into the first cell rather than tiling it.
|
||||
- [ ] `moby_anim` — **still open, and the obvious reading is ruled out.** An
|
||||
instance table would have to scale with the level's moby class count, but
|
||||
**per-chunk-type record counts are level-independent** -- the same types
|
||||
report 90, 77, 50 and 47 on every level tried, and the largest type
|
||||
totals 2,430 = 90 x 27. Nothing in the chunk set is keyed by moby class.
|
||||
The types also pair up (two types both reporting 90 on every level,
|
||||
another pair both 77), which looks like two views of one fixed table
|
||||
rather than two independent tables.
|
||||
- [x] `collision` — container framing and header fields. A 0x40-byte header
|
||||
self-describes: it carries its own size and the payload size, and
|
||||
`header + payload` stays inside `core_data` on all 27 levels. 41,481,520
|
||||
payload bytes across 19 levels; 8 declare zero length. Header bytes `+8`
|
||||
reads zero on 22 levels and a large value on 7/9/12/13/14; the payload's
|
||||
two leading `u16` are non-zero on all 19 non-empty levels (1-74 and
|
||||
13-112) and are reported as `CollisionHeader::first`/`::second`.
|
||||
- [ ] The collision **cell layout is not decoded.** The payload does not walk
|
||||
as fixed-stride cells. On level 4 a table starts at payload `+5932` and
|
||||
runs **17 entries** -- deltas rise then fall -- against a 75-cell grid,
|
||||
with no arithmetic relating the two. RAM is reshaped on load: level 1's
|
||||
`24 x 16` grid dimensions appear in memory, level 0's `11 x 33` appear
|
||||
nowhere, so disc-side decoding has to work against the reshaped form.
|
||||
A 64-byte RAM stride looked promising and is **disproved** -- the longest
|
||||
such run is 27 records of identity matrices, one per level. See
|
||||
`plan/pcsx2-verification.md`.
|
||||
- [x] `occlusion` — the PVS grid. A z/y/x tree of u16 node offsets over 4x4x4-unit
|
||||
cells, populated cells naming 128-byte masks. **412,521 cells over
|
||||
140,939 masks** across the 24 levels that have one; the mask array ends
|
||||
@@ -264,8 +303,35 @@ Phase 1 hands over the extents; this phase works out what each one contains.
|
||||
data, and an offset of 0 means *absent* rather than "at the start" — the
|
||||
latter parses as a grid whose mask array happens to fit, which is how it
|
||||
nearly went wrong.
|
||||
- [ ] `sky`, `volumes`; textures are done (CSM1 CLUT + PS2 alpha)
|
||||
- [ ] `level_overlay` — the seven-section parser and relocation from Phase 3
|
||||
- [x] `sky` — 149 sub-objects across all 27 levels, each with a count, a
|
||||
four-float bounding sphere and texture dimensions. The offset table at
|
||||
`+0x20` is self-limiting: entries run while strictly inside the region.
|
||||
**The word at `+0x0c` is not a count** -- it disagrees with the table on
|
||||
most levels and reads 196,617 on level 2 against 4 sub-objects.
|
||||
- [ ] `volumes` — **not located.** Nothing examined so far presents itself as a
|
||||
volume: `core_data`, the chunk region and `gameplay.bin` all yield
|
||||
structures that are accounted for by something else.
|
||||
- [~] `level_overlay` — header, constant block and level data parsed
|
||||
(`gc_formats::overlay`, 27/27). `+0x10..+0x1220` -- 4,624 bytes -- is
|
||||
byte-identical on every level; the word at `+0x0c` is **not** the file
|
||||
length, falling below it on 17 levels and above on 10. The debug string
|
||||
pool it opens with carries `Camera_CollPrimTest WARNING! - grid out of
|
||||
bounds! (RAR)`, independent support for reading the collision payload's
|
||||
two leading `u16`s as grid dimensions. **The header words are not named
|
||||
and the level-specific tail is not structured yet**; the Phase 3
|
||||
seven-section parser and relocation are untouched.
|
||||
- [x] **The post-shrub chunk container.** A 16-byte header, a 10-byte zero gap,
|
||||
then offsets tagged `0xF0000000 | low24` resolving to `chunk_start + low`;
|
||||
header byte 14 is the record count, equal to `offsets.len()` on 1,123 of
|
||||
1,123 chunks across all 27 levels. **77 distinct chunk types, 1,123
|
||||
chunks, 19,546 records, 50 strides.** Moby class IDs enrich at 9,493 hits
|
||||
against 246.1 expected with contiguous per-type arrays -- but the arrays
|
||||
land in a different place for each type, so no single pooled scan finds
|
||||
them. The record *fields* are not decoded.
|
||||
- [x] **Moby class arrays are contiguous per type, and the pooled scan was
|
||||
wrong.** See rule 3a in `AGENTS.md`: the original claim of a `u16` array at
|
||||
slots 129-140 was the largest-stride type dominating an aggregate. Per
|
||||
type, every one of the 26 places its array somewhere else.
|
||||
|
||||
**Gate:** every parser has a golden test against bytes from the real disc, and
|
||||
reports mismatched header counts as failures rather than warnings.
|
||||
|
||||
Reference in New Issue
Block a user