Add the disc pipeline: identify, unpack and verify

Phase 1 of the roadmap. The disc is now turned into verified asset files, and
the result is checked against a committed hash table rather than against
whatever happens to be on disk.

  gc-extract identify disc.iso        SCUS_972.68 v1.01 NTSC, ELF SHA-1 matches
  gc-extract unpack   disc.iso out/   5135 files, 4,500,106,960 bytes, 10.8s
  gc-extract verify   out/ table.tsv  OK: 5135 files match

All 92 G/*.WAD containers parse. MPEG1 and MPEG2 are raw video and pass
through whole; the other 90 are indexed containers whose extents are extracted
individually.

Formats recovered and written up in docs/formats/:

- iso9660.rs. R&C2 lists its assets in the filesystem, unlike R&C1, so a
  directory walk is the primary asset index. Three details that all fail
  quietly rather than loudly: directory records store every 32-bit field twice
  (little-endian then big-endian), the CD001 signature is at sector offset 1
  rather than 0x8001, and the directory flag at record offset 25 is
  authoritative where the identifier's trailing slash is not.

- global_wad.rs. Container headers hold a table of extents. All 92 indexed
  containers use {offset in sectors, size in bytes}. Measured, not assumed: the
  sector-count reading places every entry of all 27 LEVEL<n> and all 27
  AUDIO<n> containers out of range, while the byte reading places all of them
  correctly. SCENE<n> is the one exception, a flat sector list whose extents
  are sized by the next value.

- wad.rs. Insomniac's LZ77 stream: four flag-selected packet classes, with 0-3
  trailing literal bytes per packet carried in unused low bits. This is not a
  bit stream, which is why it resists the usual LZ77 reading.

- buf.rs. A bounds-checked reader, so untrusted 2003 disc data produces typed
  errors rather than panics.

gc-extract carries the three stages and a SHA-1 implementation with FIPS 180-4
test vectors, so identification works without pulling in a crypto dependency.

Verification: 54 unit tests, plus four integration tests that run only when
GC_DISC names a disc. Those four check identification against the recorded
values, cross-check the Rust identifier against tools/identify-disc.py, parse
all 92 containers, and confirm a full unpack matches the committed table. The
committed table is 316 KB of sizes and SHA-1s for all 5,135 files: hashes only,
no game content, so it is safe to version and it turns "matches retail" into
something a test can assert.

docs/plan/disc-pipeline.md records two wrong turns, because both are easy to
repeat and both failed by producing plausible wrong output rather than an
error: the container table was read backwards, and the LZ77 stream was modelled
as a bit stream. The first survived because it happened to be correct on the
one container used for manual checking.
This commit is contained in:
2026-10-09 16:08:30 +02:00
parent a02f79a6d1
commit 027373e65b
21 changed files with 8372 additions and 97 deletions
+96
View File
@@ -0,0 +1,96 @@
# The WAD LZ77 container
Target: `SCUS_972.68` v1.01.
Insomniac's LZ77 stream. A 16-byte header followed by a sequence of packets.
Each packet begins with a **flag byte** selecting one of four classes.
This format is **not** the `G/*.WAD` container described in
[`global-wad.md`](global-wad.md), despite the shared name and extension. WAD
streams are compressed and appear inside level cores and MPEG data; the `G/`
containers are uncompressed tables. Confusing the two produces garbage.
## Header
```text
offset size field
0x00 3 magic, "WAD"
0x03 4 total stream length in bytes, including this header
0x07 9 unused
```
`compressed_size` at `0x03` is the authoritative length. A stream whose declared
size exceeds the buffer is rejected rather than truncated.
## Packets
| Flag | Class | Displacement bits | Length |
|---|---|---|---|
| `0x00..0x0f` | literal run | — | flag + 3 (flag 0 escapes to a byte + 18) |
| `0x10..0x1f` | back-reference | 15 | 3 bits in flag, else a byte + 7 |
| `0x20..0x3f` | back-reference | 14 | 5 bits in flag, else a byte + 0x1f |
| `0x40..0xff` | back-reference | 11 | 3 bits in flag |
Every packet ends with 0–3 extra literal bytes carried in unused low bits of the
packet, so literal and back-reference packets interleave without alignment.
This is why the format does not read as a bit stream, and why a first attempt
that modelled it as one will appear to produce plausible output for a few
kilobytes before failing.
### Literal run
Flag `< 0x10`. The count is `flag + 3`, except flag `0x00` which escapes to a
following byte giving `byte + 18`. The bytes follow immediately.
Two literal packets in a row are invalid: the encoder merges them. The decoder
rejects the stream rather than accept a misparse.
### Back-reference
A back-reference copies `len` bytes from `displacement` positions back in the
already-produced output.
**Overlapping copies expand as run-length.** The source index advances with the
destination, so `displacement = 1` repeats a single byte. This is used
intentionally by the encoder.
A displacement of zero, or one that reaches before the start of the output, is
invalid.
### Pad and dummy packets
In the `0x10..0x1f` class, a packet whose displacement field is entirely zero is
special:
- length field not 1 → a **pad packet**: the reader skips forward to the next
`0x1000` boundary. These exist so compressed blocks align with something on
the disc.
- length field 1 → a **dummy packet**: it carries only its trailing literal
bytes.
## Worked example
The first stream in this game's `MISC.WAD` begins:
```text
05 80 00 00 00 10 | 00 0d ...
```
- Flag `0x05` is `< 0x10`, so it is a literal run of `5 + 3 = 8` bytes:
`80 00 00 00 10 00 0d 00`.
- The next flag byte starts the following packet.
## Implementation
`crates/gc-formats/src/wad.rs`. Parses through the same bounds-checked reader as
everything else in the crate, so a truncated stream produces a typed error
rather than a panic. A `BadMagic`, an out-of-bounds read, and a displacement
before the output are three distinct error kinds.
## Provenance
The packet structure was recovered from behavioural analysis of the retail
data, guided by the format description in the Wrench project
(<https://github.com/chaoticgd/wrench>, GPLv3). **No Wrench code is copied into
this repository** — see `PROVENANCE.md`. The decoder here is written from the
specification and verified against this disc's own streams.