ReaperOfVeriod
a9e43e39c1
The first typed structure read from real data. Every one of the 27 level containers begins with the same 0x60-byte header, which turns the generic pair table into named assets: data, sound_bank, gameplay and occlusion, plus three chunk descriptors. Two things about this header are worth stating because both are silent failures rather than errors. Sizes here are sector counts, not byte counts. The generic G/ pair table on the same disc uses byte counts, so both readings type-check and the wrong one is off by a factor of 2048. The tiling check settles it: read as sectors, the four extents tile every one of the 27 containers exactly; read as bytes they overlap each other. Level id is not container order. LEVEL21.WAD reports id 30, so a level was cut from the disc and its number left a gap. Anything keying off file order -- a save slot, the planet select screen -- would mislabel that planet. There is also a second trap in the same area: the ISO filesystem sorts names alphabetically, so LEVEL10 precedes LEVEL2, and any tool needing order has to sort numerically. The header's field at 0x04 is zero on all 27 levels. It is not the container size despite looking like one, so the code reads it as a runtime field and takes the container length from the ISO entry instead. Measured across all 27 levels, three invariants now hold in tests: extents tile in file order; extent coverage plus chunks reaches at least the end of the container (21 levels land exactly on its length, 6 leave 134 to 689 sectors of trailing data); and exactly one level's id differs from its container index. The coverage assertion is >= rather than == on purpose -- 21 levels end one sector past the file end because the header occupies sector 0, and asserting equality would truncate a level's tail. gameplay is left unparsed and that is recorded rather than hidden. It begins "WAD" plus a per-level byte and does not decode with the WAD LZ77 decoder: all 27 levels produce plausible output for the first packet and then fail on a rule the format's own encoder never breaks, which means the input is wrong rather than the decoder. docs/plan/level-assets.md records what is ruled out and what to try next. 66 unit tests, plus 7 integration tests against the real disc. The unpack hash table is unchanged, so this adds parsing without altering the pipeline.
117 lines
4.9 KiB
Markdown
117 lines
4.9 KiB
Markdown
# Level assets
|
|
|
|
Phase 2 work record. What has been established about the four extents inside
|
|
each `LEVEL<n>.WAD`, and what remains.
|
|
|
|
## Done: the container header
|
|
|
|
`crates/gc-formats/src/level_wad.rs` parses the 0x60-byte header on all 27
|
|
levels and names the four extents. See
|
|
[`../formats/level-wad.md`](../formats/level-wad.md) for the layout and the
|
|
measured invariants.
|
|
|
|
Two facts worth carrying forward:
|
|
|
|
- **Sizes in this header are sector counts**, unlike the generic `G/*.WAD` pair
|
|
table where they are byte counts. Both are on the same disc.
|
|
- **`LEVEL21.WAD` reports level id 30.** Container order is not level identity.
|
|
|
|
## `gameplay`: a compression we have not identified
|
|
|
|
This is the one genuinely puzzling result so far, and it is worth recording in
|
|
full because the next attempt will otherwise repeat the work.
|
|
|
|
The extent begins:
|
|
|
|
```text
|
|
offset 0 57 41 44 "WAD"
|
|
offset 3 79 0d 0a 0d <- varies per level
|
|
offset 4+ <payload>
|
|
```
|
|
|
|
The variation is the tell. For the LZ77 WAD container the fourth byte is part of
|
|
a 32-bit little-endian length; here byte 3 differs per level (0x79, 0x3a, 0xd8,
|
|
0xa9, ...) while byte 4 is a small count (13, 88, 13, 105, 163, ...) and bytes 6
|
|
and 7 are always zero.
|
|
|
|
**Attempt: treat it as the known WAD LZ77 stream.** Decoded all 27 levels with
|
|
the decoder in `wad.rs`, starting after an 8-byte header. Every one produces
|
|
plausible output for the first packet — `0000 0000 0000 0750 0100 0020 2b0c...`
|
|
looks like real level data — and then fails on all 27 with "adjacent literal
|
|
packets", which the format's encoder never emits.
|
|
|
|
A uniform failure on all 27 inputs means the input is wrong, not the decoder.
|
|
Ruled out:
|
|
|
|
- **Bit order.** The WAD container is a flag-byte packet stream, not a bit
|
|
stream, so no ordering applies. Both orderings fail identically.
|
|
- **Header length.** Tried 8 bytes. A different skip would not fix a failure
|
|
that occurs uniformly in the first few packets.
|
|
- **The WAD format at all.** The first packet (flag `0x00`) decodes cleanly as a
|
|
literal run of 18 bytes, which is *why* it looks plausible, but the run
|
|
immediately after breaks the format's own rule. Whatever this is, it is not
|
|
WAD LZ77.
|
|
|
|
Next to try, in order of cheapness:
|
|
|
|
1. Whether the payload is uncompressed and the header is a table. `(len - 8)`
|
|
is 196..497 across the levels and divides by neither 8 nor a plausible stride,
|
|
which argues against a flat pair table — but a *variable-length* table
|
|
(counts of counts) fits that range.
|
|
2. Whether byte 4 is an entry count and bytes 5.. are records of a fixed but
|
|
non-obvious width.
|
|
3. Whether the LZ77 variant differs only in the "adjacent literals" rule.
|
|
|
|
None of these is expensive to test; each is a few lines against all 27 levels at
|
|
once. The right move is to write a throwaway script that tries many hypotheses
|
|
across all 27 and reports which produce self-consistent results — the approach
|
|
that worked for the container pair table, where "does this hold on every level"
|
|
was the discriminator that the sector/byte ambiguity failed.
|
|
|
|
## `sound_bank`: signatures found, structure unknown
|
|
|
|
```text
|
|
+0x18 "SBlk" present in all 27 levels
|
|
+0x24 "DAW" present in all 27 levels
|
|
```
|
|
|
|
Sizes 458..833 bytes. Too small to be the level's audio; likely an index into
|
|
the global `AUDIO.WAD`.
|
|
|
|
## `occlusion`: smallest, and unread
|
|
|
|
2 to 25 sectors. Small enough to dump whole and stare at.
|
|
|
|
## `data`: the real target, untouched
|
|
|
|
6,484 sectors on level 0, down to 2,102 on level 24 — 4.2 to 21.9 MB. This is
|
|
the level's geometry and the largest thing in the container. Next target.
|
|
|
|
What is known from the community reference and from Insomniac's engine family:
|
|
it splits into four geometry subsystems — `tfrag` (terrain fragments),
|
|
`tie` (instanced environment structures), `shrub` (foliage) and `moby` (dynamic
|
|
actors) — plus sky shells, collision data and textures. ReRAC's documentation of
|
|
the equivalent structures in the first game is the closest reference available,
|
|
though R&C2's spherical worlds change how they are used.
|
|
|
|
## Order of work
|
|
|
|
1. `gameplay` — smallest, fully bounded, and the format is identified enough to
|
|
make progress. A win here validates the approach before the large effort.
|
|
2. `occlusion` — tiny.
|
|
3. `sound_bank` — small.
|
|
4. `data` — the large one, and the one that decides whether anything renders.
|
|
|
|
Doing the small extents first is not just convenience: each one is a contained
|
|
reverse-engineering problem that either resolves or tells us something, whereas
|
|
`data` will keep producing ambiguous readings until we have something to check
|
|
them against.
|
|
|
|
## Method note
|
|
|
|
Every format question this phase has been settled the same way: write the
|
|
hypothesis, test it against **all 27 levels at once**, and require an invariant
|
|
that holds everywhere. The sector/byte ambiguity in Phase 1 survived because it
|
|
happened to be correct on the one container checked by hand. The level header
|
|
was fixed in a single pass because the tiling check failed immediately on
|
|
levels 1, 2, 4, 7, 8, 11, 19 and 20. |