ReaperOfVeriod
027373e65b
Phase 1 of the roadmap. The disc is now turned into verified asset files, and
the result is checked against a committed hash table rather than against
whatever happens to be on disk.
gc-extract identify disc.iso SCUS_972.68 v1.01 NTSC, ELF SHA-1 matches
gc-extract unpack disc.iso out/ 5135 files, 4,500,106,960 bytes, 10.8s
gc-extract verify out/ table.tsv OK: 5135 files match
All 92 G/*.WAD containers parse. MPEG1 and MPEG2 are raw video and pass
through whole; the other 90 are indexed containers whose extents are extracted
individually.
Formats recovered and written up in docs/formats/:
- iso9660.rs. R&C2 lists its assets in the filesystem, unlike R&C1, so a
directory walk is the primary asset index. Three details that all fail
quietly rather than loudly: directory records store every 32-bit field twice
(little-endian then big-endian), the CD001 signature is at sector offset 1
rather than 0x8001, and the directory flag at record offset 25 is
authoritative where the identifier's trailing slash is not.
- global_wad.rs. Container headers hold a table of extents. All 92 indexed
containers use {offset in sectors, size in bytes}. Measured, not assumed: the
sector-count reading places every entry of all 27 LEVEL<n> and all 27
AUDIO<n> containers out of range, while the byte reading places all of them
correctly. SCENE<n> is the one exception, a flat sector list whose extents
are sized by the next value.
- wad.rs. Insomniac's LZ77 stream: four flag-selected packet classes, with 0-3
trailing literal bytes per packet carried in unused low bits. This is not a
bit stream, which is why it resists the usual LZ77 reading.
- buf.rs. A bounds-checked reader, so untrusted 2003 disc data produces typed
errors rather than panics.
gc-extract carries the three stages and a SHA-1 implementation with FIPS 180-4
test vectors, so identification works without pulling in a crypto dependency.
Verification: 54 unit tests, plus four integration tests that run only when
GC_DISC names a disc. Those four check identification against the recorded
values, cross-check the Rust identifier against tools/identify-disc.py, parse
all 92 containers, and confirm a full unpack matches the committed table. The
committed table is 316 KB of sizes and SHA-1s for all 5,135 files: hashes only,
no game content, so it is safe to version and it turns "matches retail" into
something a test can assert.
docs/plan/disc-pipeline.md records two wrong turns, because both are easy to
repeat and both failed by producing plausible wrong output rather than an
error: the container table was read backwards, and the LZ77 stream was modelled
as a bit stream. The first survived because it happened to be correct on the
one container used for manual checking.
3.4 KiB
The WAD LZ77 container
Target: SCUS_972.68 v1.01.
Insomniac's LZ77 stream. A 16-byte header followed by a sequence of packets. Each packet begins with a flag byte selecting one of four classes.
This format is not the G/*.WAD container described in
global-wad.md, despite the shared name and extension. WAD
streams are compressed and appear inside level cores and MPEG data; the G/
containers are uncompressed tables. Confusing the two produces garbage.
Header
offset size field
0x00 3 magic, "WAD"
0x03 4 total stream length in bytes, including this header
0x07 9 unused
compressed_size at 0x03 is the authoritative length. A stream whose declared
size exceeds the buffer is rejected rather than truncated.
Packets
| Flag | Class | Displacement bits | Length |
|---|---|---|---|
0x00..0x0f |
literal run | — | flag + 3 (flag 0 escapes to a byte + 18) |
0x10..0x1f |
back-reference | 15 | 3 bits in flag, else a byte + 7 |
0x20..0x3f |
back-reference | 14 | 5 bits in flag, else a byte + 0x1f |
0x40..0xff |
back-reference | 11 | 3 bits in flag |
Every packet ends with 0–3 extra literal bytes carried in unused low bits of the packet, so literal and back-reference packets interleave without alignment. This is why the format does not read as a bit stream, and why a first attempt that modelled it as one will appear to produce plausible output for a few kilobytes before failing.
Literal run
Flag < 0x10. The count is flag + 3, except flag 0x00 which escapes to a
following byte giving byte + 18. The bytes follow immediately.
Two literal packets in a row are invalid: the encoder merges them. The decoder rejects the stream rather than accept a misparse.
Back-reference
A back-reference copies len bytes from displacement positions back in the
already-produced output.
Overlapping copies expand as run-length. The source index advances with the
destination, so displacement = 1 repeats a single byte. This is used
intentionally by the encoder.
A displacement of zero, or one that reaches before the start of the output, is invalid.
Pad and dummy packets
In the 0x10..0x1f class, a packet whose displacement field is entirely zero is
special:
- length field not 1 → a pad packet: the reader skips forward to the next
0x1000boundary. These exist so compressed blocks align with something on the disc. - length field 1 → a dummy packet: it carries only its trailing literal bytes.
Worked example
The first stream in this game's MISC.WAD begins:
05 80 00 00 00 10 | 00 0d ...
- Flag
0x05is< 0x10, so it is a literal run of5 + 3 = 8bytes:80 00 00 00 10 00 0d 00. - The next flag byte starts the following packet.
Implementation
crates/gc-formats/src/wad.rs. Parses through the same bounds-checked reader as
everything else in the crate, so a truncated stream produces a typed error
rather than a panic. A BadMagic, an out-of-bounds read, and a displacement
before the output are three distinct error kinds.
Provenance
The packet structure was recovered from behavioural analysis of the retail
data, guided by the format description in the Wrench project
(https://github.com/chaoticgd/wrench, GPLv3). No Wrench code is copied into
this repository — see PROVENANCE.md. The decoder here is written from the
specification and verified against this disc's own streams.