ReaperOfVeriod/Going-Commando
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Parse the level container header, naming each level's four extents

Browse Source 
The first typed structure read from real data. Every one of the 27 level
containers begins with the same 0x60-byte header, which turns the generic pair
table into named assets: data, sound_bank, gameplay and occlusion, plus three
chunk descriptors.

Two things about this header are worth stating because both are silent failures
rather than errors.

Sizes here are sector counts, not byte counts. The generic G/ pair table on the
same disc uses byte counts, so both readings type-check and the wrong one is off
by a factor of 2048. The tiling check settles it: read as sectors, the four
extents tile every one of the 27 containers exactly; read as bytes they overlap
each other.

Level id is not container order. LEVEL21.WAD reports id 30, so a level was cut
from the disc and its number left a gap. Anything keying off file order -- a
save slot, the planet select screen -- would mislabel that planet. There is also
a second trap in the same area: the ISO filesystem sorts names alphabetically, so
LEVEL10 precedes LEVEL2, and any tool needing order has to sort numerically.

The header's field at 0x04 is zero on all 27 levels. It is not the container
size despite looking like one, so the code reads it as a runtime field and takes
the container length from the ISO entry instead.

Measured across all 27 levels, three invariants now hold in tests: extents tile
in file order; extent coverage plus chunks reaches at least the end of the
container (21 levels land exactly on its length, 6 leave 134 to 689 sectors of
trailing data); and exactly one level's id differs from its container index. The
coverage assertion is >= rather than == on purpose -- 21 levels end one sector
past the file end because the header occupies sector 0, and asserting equality
would truncate a level's tail.

gameplay is left unparsed and that is recorded rather than hidden. It begins
"WAD" plus a per-level byte and does not decode with the WAD LZ77 decoder: all
27 levels produce plausible output for the first packet and then fail on a rule
the format's own encoder never breaks, which means the input is wrong rather
than the decoder. docs/plan/level-assets.md records what is ruled out and what
to try next.

66 unit tests, plus 7 integration tests against the real disc. The unpack hash
table is unchanged, so this adds parsing without altering the pipeline.
This commit is contained in:
ReaperOfVeriod
2026-10-09 16:20:25 +02:00
parent 027373e65b
commit a9e43e39c1
6 changed files with 753 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/formats/level-wad.md
+139
View File
@@ -0,0 +1,139 @@
# The level container (`LEVEL<n>.WAD`)
Target: `SCUS_972.68` v1.01. 27 containers.
The first typed structure in this project, and the one that turns the generic
container table of [`global-wad.md`](global-wad.md) into named, individually
parseable assets. Implementation: `crates/gc-formats/src/level_wad.rs`.
## Header
All 27 level containers begin with the same 0x60-byte header:
```text
offset size field
0x00 4 header size, 0x60 on every level
0x04 4 zero on every level; written by the game at load time
0x08 4 level id
0x0c 4 reverb configuration index
0x10 8 data geometry: tfrag, tie, shrub, moby, sky, textures
0x18 8 sound_bank
0x20 8 gameplay
0x28 8 occlusion
0x30 48 three {offset, size} chunk descriptors
```
Sizes here are **sector counts**. This is the opposite of the generic pair table
in the `G/` containers, where sizes are byte counts. Both appear on the same
disc, so reading one as the other produces extents that are wrong by a factor of
2048 and that still *look* plausible. See "Two fields, two meanings" below.
## Extents
The four named extents appear in the file in an order that is **not** the order
of the fields in the header:
```text
sound_bank data gameplay occlusion
```
Level 0, measured:
| Part | Offset | Sectors | Bytes | Unpacked to |
|---|---:|---:|---:|---|
| sound_bank | 1 | 734 | 1.50 MB | `level0/0002.bin` |
| data | 735 | 6,484 | 13.28 MB | `level0/0001.bin` |
| gameplay | 7,219 | 322 | 0.66 MB | `level0/0003.bin` |
| occlusion | 7,541 | 12 | 0.02 MB | `level0/0004.bin` |
The numbering of the unpacked files follows the header's field order, so the
`data` extent is `0001.bin` while it is physically second in the container. Both
orderings are worth keeping straight; the typed header exists so callers never
have to.
`data` is where the level's visible content lives and is the target of the next
phase.
## Two fields, two meanings
This is the trap in this format, and it is worth stating plainly because both
readings type-check.
| Where | Size field is |
|---|---|
| `G/*.WAD` generic pair table | **bytes** |
| `LEVEL<n>.WAD` named extents | **sectors** |
Measured across all 92 indexed pair containers, the byte reading places every
entry inside its file while the sector reading does not. Measured across all 27
level headers, the sector reading tiles each container exactly.
If the level header's sizes were read as bytes, `sound_bank` on level 0 would
claim 734 bytes starting at sector 1, and `data` would start at byte offset 735
— inside the sound bank. The extents would overlap, which is the symptom to
look for.
## Invariants, as measured on all 27 levels
These are the checks the parser is validated against, and each holds everywhere
rather than on one sample:
1. **Extent order and tiling.** Each extent's end is the next extent's start.
27 of 27.
2. **The runtime sector field at `0x04` is zero.** 27 of 27. It is not the
container size and must not be used as one — the container's length comes
from the ISO directory entry.
3. **Extent coverage.** `sound_bank + data + gameplay + occlusion + chunks`
always reaches at least the end of the container. 21 of 27 land on exactly
the container's sector length; 6 leave between 134 and 689 sectors of trailing
data past the last named extent (levels 1, 2, 4, 11, 19, 20).
The 21-level case ends *one sector past* the file end. The header occupies
sector 0 and the extents begin at sector 1, so this is ordinary sector
rounding rather than an error. The correct assertion is therefore `>=`, not
equality — asserting equality would truncate the tail of a level.
4. **Level id equals container index on 26 of 27 levels.** `LEVEL21.WAD`
reports **id 30**.
## Level ids are not file order
Point 4 matters enough to state separately. `LEVEL21.WAD` has `id = 30`, so the
container's position in the directory is not the level's identity. Some level
was cut from the disc and its number is not reused, leaving a gap.
Anything that keys off file order — a save slot, a planet select screen, the
moby class table — would silently mislabel that planet. Use the header's `id`.
The ISO filesystem also sorts names alphabetically, so `LEVEL10.WAD` precedes
`LEVEL2.WAD`. Tests and tools that care about order must sort numerically.
## Chunk descriptors
`0x30..0x60` holds three `{offset, size}` pairs in sector counts, following the
named extents. They are populated on 6 levels (1, 2, 4, 7, 8, 11, 19, 20 — of
which some are the trailing-data levels) and zero elsewhere. Their contents are
not yet identified.
## Not yet parsed
- **`data`** — the geometry. This is the next target and where the visible
level comes from: tfrag terrain, tie instances, shrub foliage, moby classes,
sky shells and textures.
- **`gameplay`** — begins `WAD` + a per-level id byte + a count, and behaves
like an LZ77 stream but does **not** decode with the WAD decoder (all 27
levels fail with "adjacent literal packets", which the format forbids). It is
a different compression, or a different framing. Recorded in
[`../plan/level-assets.md`](../plan/level-assets.md).
- **`sound_bank`** — begins with `SBlk` at `+24` and `DAW` at `+36`; both
signatures are present in all 27 levels.
- **`occlusion`** — 2 to 25 sectors. Small enough to read fully.
## Provenance
The header layout matches the `GcUyaLevelWadHeader` structure documented by the
Wrench project (<https://github.com/chaoticgd/wrench>, GPLv3). **No Wrench code
is copied** — see `PROVENANCE.md`. The structure here is written from the
specification and every field is verified against all 27 levels on this disc.
The two facts that the documentation does not state — the sector-count sizes and
the `LEVEL21` id divergence — were measured here.
docs/plan/level-assets.md
+117
View File
@@ -0,0 +1,117 @@
# Level assets
Phase 2 work record. What has been established about the four extents inside
each `LEVEL<n>.WAD`, and what remains.
## Done: the container header
`crates/gc-formats/src/level_wad.rs` parses the 0x60-byte header on all 27
levels and names the four extents. See
[`../formats/level-wad.md`](../formats/level-wad.md) for the layout and the
measured invariants.
Two facts worth carrying forward:
- **Sizes in this header are sector counts**, unlike the generic `G/*.WAD` pair
table where they are byte counts. Both are on the same disc.
- **`LEVEL21.WAD` reports level id 30.** Container order is not level identity.
## `gameplay`: a compression we have not identified
This is the one genuinely puzzling result so far, and it is worth recording in
full because the next attempt will otherwise repeat the work.
The extent begins:
```text
offset 0 57 41 44 "WAD"
offset 3 79 0d 0a 0d <- varies per level
offset 4+ <payload>
```
The variation is the tell. For the LZ77 WAD container the fourth byte is part of
a 32-bit little-endian length; here byte 3 differs per level (0x79, 0x3a, 0xd8,
0xa9, ...) while byte 4 is a small count (13, 88, 13, 105, 163, ...) and bytes 6
and 7 are always zero.
**Attempt: treat it as the known WAD LZ77 stream.** Decoded all 27 levels with
the decoder in `wad.rs`, starting after an 8-byte header. Every one produces
plausible output for the first packet — `0000 0000 0000 0750 0100 0020 2b0c...`
looks like real level data — and then fails on all 27 with "adjacent literal
packets", which the format's encoder never emits.
A uniform failure on all 27 inputs means the input is wrong, not the decoder.
Ruled out:
- **Bit order.** The WAD container is a flag-byte packet stream, not a bit
stream, so no ordering applies. Both orderings fail identically.
- **Header length.** Tried 8 bytes. A different skip would not fix a failure
that occurs uniformly in the first few packets.
- **The WAD format at all.** The first packet (flag `0x00`) decodes cleanly as a
literal run of 18 bytes, which is *why* it looks plausible, but the run
immediately after breaks the format's own rule. Whatever this is, it is not
WAD LZ77.
Next to try, in order of cheapness:
1. Whether the payload is uncompressed and the header is a table. `(len - 8)`
is 196..497 across the levels and divides by neither 8 nor a plausible stride,
which argues against a flat pair table — but a *variable-length* table
(counts of counts) fits that range.
2. Whether byte 4 is an entry count and bytes 5.. are records of a fixed but
non-obvious width.
3. Whether the LZ77 variant differs only in the "adjacent literals" rule.
None of these is expensive to test; each is a few lines against all 27 levels at
once. The right move is to write a throwaway script that tries many hypotheses
across all 27 and reports which produce self-consistent results — the approach
that worked for the container pair table, where "does this hold on every level"
was the discriminator that the sector/byte ambiguity failed.
## `sound_bank`: signatures found, structure unknown
```text
+0x18 "SBlk" present in all 27 levels
+0x24 "DAW" present in all 27 levels
```
Sizes 458..833 bytes. Too small to be the level's audio; likely an index into
the global `AUDIO.WAD`.
## `occlusion`: smallest, and unread
2 to 25 sectors. Small enough to dump whole and stare at.
## `data`: the real target, untouched
6,484 sectors on level 0, down to 2,102 on level 24 — 4.2 to 21.9 MB. This is
the level's geometry and the largest thing in the container. Next target.
What is known from the community reference and from Insomniac's engine family:
it splits into four geometry subsystems — `tfrag` (terrain fragments),
`tie` (instanced environment structures), `shrub` (foliage) and `moby` (dynamic
actors) — plus sky shells, collision data and textures. ReRAC's documentation of
the equivalent structures in the first game is the closest reference available,
though R&C2's spherical worlds change how they are used.
## Order of work
1. `gameplay` — smallest, fully bounded, and the format is identified enough to
make progress. A win here validates the approach before the large effort.
2. `occlusion` — tiny.
3. `sound_bank` — small.
4. `data` — the large one, and the one that decides whether anything renders.
Doing the small extents first is not just convenience: each one is a contained
reverse-engineering problem that either resolves or tells us something, whereas
`data` will keep producing ambiguous readings until we have something to check
them against.
## Method note
Every format question this phase has been settled the same way: write the
hypothesis, test it against **all 27 levels at once**, and require an invariant
that holds everywhere. The sector/byte ambiguity in Phase 1 survived because it
happened to be correct on the one container checked by hand. The level header
was fixed in a single pass because the tiling check failed immediately on
levels 1, 2, 4, 7, 8, 11, 19 and 20.
docs/roadmap.md
+24 -3
View File
@@ -16,7 +16,10 @@ early, while they are still cheap to retire.
- [x] **Phase 1 complete.** The disc pipeline runs end to end and is verified
against a committed hash table: 5,135 files, 4,500,106,960 bytes, every
one of the 92 asset containers parsing.
- [ ] Phase 2 next: the asset format parsers.
- [x] **Phase 2 started.** The level container header is parsed and typed, which
names the `data`, `sound_bank`, `gameplay` and `occlusion` extents inside
each of the 27 level containers.
- [ ] Phase 2 next: the `gameplay` extent, then `data`.
The disc being the exact build the reference projects pin is the single
biggest favourable fact so far: no disc conversion, no regional timing work,
@@ -85,8 +88,26 @@ failing, which is the failure mode worth guarding against.
disc data.
Phase 1 hands over the extents; this phase works out what each one contains.
`global_wad.md` records which pair in each container corresponds to which asset,
and that table starts empty.
**Done so far:**
- [x] `level_wad.rs` — the 0x60-byte header on all 27 level containers, naming
four extents plus three chunk descriptors. Verified: extents tile each
container in file order (27 of 27).
- [x] Two facts the format documentation does not state, both measured here:
sizes in this header are **sector counts** where the generic `G/` table
uses byte counts, and **`LEVEL21.WAD` reports level id 30**, so container
order is not level identity.
**In progress:**
- [ ] `gameplay` — begins `WAD` + a per-level byte, and does *not* decode with
the WAD LZ77 decoder. All 27 levels fail identically, so the input is
wrong rather than the decoder. Ruled out: bit order, header length, the
LZ77 framing. Hypotheses queued in `plan/level-assets.md`.
- [ ] `occlusion` — 2 to 25 sectors, small enough to read whole.
- [ ] `sound_bank` — `SBlk` and `DAW` signatures found in all 27.
- [ ] `data` — the large one: 4.2 to 21.9 MB of geometry per level.
- [ ] `tfrag` — terrain: LOD chains, strips, morph targets
- [ ] `tie` — instanced environment geometry, fat vertices