Add the disc pipeline: identify, unpack and verify

Phase 1 of the roadmap. The disc is now turned into verified asset files, and
the result is checked against a committed hash table rather than against
whatever happens to be on disk.

  gc-extract identify disc.iso        SCUS_972.68 v1.01 NTSC, ELF SHA-1 matches
  gc-extract unpack   disc.iso out/   5135 files, 4,500,106,960 bytes, 10.8s
  gc-extract verify   out/ table.tsv  OK: 5135 files match

All 92 G/*.WAD containers parse. MPEG1 and MPEG2 are raw video and pass
through whole; the other 90 are indexed containers whose extents are extracted
individually.

Formats recovered and written up in docs/formats/:

- iso9660.rs. R&C2 lists its assets in the filesystem, unlike R&C1, so a
  directory walk is the primary asset index. Three details that all fail
  quietly rather than loudly: directory records store every 32-bit field twice
  (little-endian then big-endian), the CD001 signature is at sector offset 1
  rather than 0x8001, and the directory flag at record offset 25 is
  authoritative where the identifier's trailing slash is not.

- global_wad.rs. Container headers hold a table of extents. All 92 indexed
  containers use {offset in sectors, size in bytes}. Measured, not assumed: the
  sector-count reading places every entry of all 27 LEVEL<n> and all 27
  AUDIO<n> containers out of range, while the byte reading places all of them
  correctly. SCENE<n> is the one exception, a flat sector list whose extents
  are sized by the next value.

- wad.rs. Insomniac's LZ77 stream: four flag-selected packet classes, with 0-3
  trailing literal bytes per packet carried in unused low bits. This is not a
  bit stream, which is why it resists the usual LZ77 reading.

- buf.rs. A bounds-checked reader, so untrusted 2003 disc data produces typed
  errors rather than panics.

gc-extract carries the three stages and a SHA-1 implementation with FIPS 180-4
test vectors, so identification works without pulling in a crypto dependency.

Verification: 54 unit tests, plus four integration tests that run only when
GC_DISC names a disc. Those four check identification against the recorded
values, cross-check the Rust identifier against tools/identify-disc.py, parse
all 92 containers, and confirm a full unpack matches the committed table. The
committed table is 316 KB of sizes and SHA-1s for all 5,135 files: hashes only,
no game content, so it is safe to version and it turns "matches retail" into
something a test can assert.

docs/plan/disc-pipeline.md records two wrong turns, because both are easy to
repeat and both failed by producing plausible wrong output rather than an
error: the container table was read backwards, and the LZ77 stream was modelled
as a bit stream. The first survived because it happened to be correct on the
one container used for manual checking.
This commit is contained in:
2026-10-09 16:08:30 +02:00
parent a02f79a6d1
commit 027373e65b
21 changed files with 8372 additions and 97 deletions
+31 -13
View File
@@ -13,7 +13,10 @@ early, while they are still cheap to retire.
- [x] Workspace builds (`cargo check --workspace`)
- [x] Repository hygiene in place (pre-commit guard, `.gitignore`, licence
record)
- [ ] Nothing else implemented yet
- [x] **Phase 1 complete.** The disc pipeline runs end to end and is verified
against a committed hash table: 5,135 files, 4,500,106,960 bytes, every
one of the 92 asset containers parsing.
- [ ] Phase 2 next: the asset format parsers.
The disc being the exact build the reference projects pin is the single
biggest favourable fact so far: no disc conversion, no regional timing work,
@@ -46,22 +49,33 @@ and no re-derivation of the reference baseline.
---
## Phase 1 — Disc pipeline
## Phase 1 — Disc pipeline ✅
**Goal:** turn the disc image into verified, unpacked asset files.
**Done.** Gate met: every extracted file matches the committed hash table.
- [ ] `iso9660` reader (R&C2 lists assets as real files, unlike R&C1)
- [ ] `RC2.HDR` table of contents parser (408,444 bytes at disc root)
- [ ] WAD LZ77 decompression
- [ ] Unpack global WADs (`G/*.WAD`) and per-level WADs
- [ ] Commit a size+SHA-1 table per target file (hashes only, never content)
- [ ] `verify` stage
- [x] `iso9660` reader (R&C2 lists assets as real files, unlike R&C1)
- [x] `G/*.WAD` container header parsing — 92 containers, three table shapes
- [x] WAD LZ77 decompression (`wad.rs`, four packet classes)
- [x] Unpack every container, bounds-checking each extent
- [x] SHA-1 table for all 5,135 files (hashes only, never content)
- [x] `verify` stage, plus integration tests against a real disc
- [x] Format specs written: `disc-layout.md`, `global-wad.md`, `wad.md`
- [x] Work record with the negative results: `plan/disc-pipeline.md`
**Gate:** every extracted file matches the committed hash table, and the
decompressed size of every WAD stream matches its header.
```text
gc-extract identify disc.iso -> SCUS_972.68 v1.01 NTSC, ELF SHA-1 matches
gc-extract unpack disc.iso out/ -> 5135 files, 4,500,106,960 bytes, 10.8s
gc-extract verify out/ table.tsv -> OK: 5135 files match
```
**Note:** the three MPEG WADs total roughly 2.4 GB of the 3.8 GB disc. Video is
a data problem before it is a decoding problem — plan to stream.
**Not done, deliberately:** the table of contents at sector 1001 is documented
but not read. Enumerating the ISO filesystem is simpler and sufficient; the ToC
matters when per-level overlay loading is understood in Phase 3.
**Two wrong turns are recorded** in `plan/disc-pipeline.md`, because both would
be easy to repeat: the container pair table read backwards, and the LZ77 stream
mistaken for a bit stream. Both produced plausible-looking wrong output before
failing, which is the failure mode worth guarding against.
---
@@ -70,6 +84,10 @@ a data problem before it is a decoding problem — plan to stream.
**Goal:** every asset type parses into plain Rust values, tested against real
disc data.
Phase 1 hands over the extents; this phase works out what each one contains.
`global_wad.md` records which pair in each container corresponds to which asset,
and that table starts empty.
- [ ] `tfrag` — terrain: LOD chains, strips, morph targets
- [ ] `tie` — instanced environment geometry, fat vertices
- [ ] `shrub` — foliage and decorative meshes