Add the disc pipeline: identify, unpack and verify
Phase 1 of the roadmap. The disc is now turned into verified asset files, and
the result is checked against a committed hash table rather than against
whatever happens to be on disk.
gc-extract identify disc.iso SCUS_972.68 v1.01 NTSC, ELF SHA-1 matches
gc-extract unpack disc.iso out/ 5135 files, 4,500,106,960 bytes, 10.8s
gc-extract verify out/ table.tsv OK: 5135 files match
All 92 G/*.WAD containers parse. MPEG1 and MPEG2 are raw video and pass
through whole; the other 90 are indexed containers whose extents are extracted
individually.
Formats recovered and written up in docs/formats/:
- iso9660.rs. R&C2 lists its assets in the filesystem, unlike R&C1, so a
directory walk is the primary asset index. Three details that all fail
quietly rather than loudly: directory records store every 32-bit field twice
(little-endian then big-endian), the CD001 signature is at sector offset 1
rather than 0x8001, and the directory flag at record offset 25 is
authoritative where the identifier's trailing slash is not.
- global_wad.rs. Container headers hold a table of extents. All 92 indexed
containers use {offset in sectors, size in bytes}. Measured, not assumed: the
sector-count reading places every entry of all 27 LEVEL<n> and all 27
AUDIO<n> containers out of range, while the byte reading places all of them
correctly. SCENE<n> is the one exception, a flat sector list whose extents
are sized by the next value.
- wad.rs. Insomniac's LZ77 stream: four flag-selected packet classes, with 0-3
trailing literal bytes per packet carried in unused low bits. This is not a
bit stream, which is why it resists the usual LZ77 reading.
- buf.rs. A bounds-checked reader, so untrusted 2003 disc data produces typed
errors rather than panics.
gc-extract carries the three stages and a SHA-1 implementation with FIPS 180-4
test vectors, so identification works without pulling in a crypto dependency.
Verification: 54 unit tests, plus four integration tests that run only when
GC_DISC names a disc. Those four check identification against the recorded
values, cross-check the Rust identifier against tools/identify-disc.py, parse
all 92 containers, and confirm a full unpack matches the committed table. The
committed table is 316 KB of sizes and SHA-1s for all 5,135 files: hashes only,
no game content, so it is safe to version and it turns "matches retail" into
something a test can assert.
docs/plan/disc-pipeline.md records two wrong turns, because both are easy to
repeat and both failed by producing plausible wrong output rather than an
error: the container table was read backwards, and the LZ77 stream was modelled
as a bit stream. The first survived because it happened to be correct on the
one container used for manual checking.
This commit is contained in:
@@ -0,0 +1,140 @@
|
||||
# Disc pipeline
|
||||
|
||||
The work record for Phase 1. What was tried, what worked, and what was ruled
|
||||
out — the negative results matter as much as the working parts.
|
||||
|
||||
## Result
|
||||
|
||||
The disc pipeline runs end to end and is verified.
|
||||
|
||||
```text
|
||||
$ gc-extract identify disc.iso
|
||||
serial SCUS_972.68 version 1.01 video mode NTSC
|
||||
boot ELF SHA-1 bacda860cbd6723142e86db56f197e47e728970f
|
||||
|
||||
$ gc-extract unpack disc.iso extracted/ --table gc-files.tsv
|
||||
5135 files, 4,500,106,960 bytes, in 10.8s
|
||||
|
||||
$ gc-extract verify extracted/ gc-files.tsv
|
||||
OK: 5135 files match the recorded table.
|
||||
```
|
||||
|
||||
Every one of the 96 `G/` containers parses. 92 are indexed containers whose
|
||||
extents are extracted individually; `MPEG1.WAD` and `MPEG2.WAD` are raw video
|
||||
and pass through whole.
|
||||
|
||||
## The wrong turn worth recording
|
||||
|
||||
**The container pair table was read backwards for most of this phase.**
|
||||
|
||||
The obvious reading of an 8-byte entry is `{offset, size}`, both sector counts —
|
||||
`SectorRange`. That reading:
|
||||
|
||||
- worked perfectly for `ARMOR.WAD` (30 of 30 entries in range), which is exactly
|
||||
why it survived several rounds of testing;
|
||||
- placed 1 entry out of range on `HUD.WAD`, 69 of 178 on `AUDIO.WAD`, and **all
|
||||
entries out of range** on every `AUDIO<n>.WAD` and `LEVEL<n>.WAD`.
|
||||
|
||||
The `AUDIO0.WAD` numbers were the tell: under the sector reading, the count of
|
||||
out-of-range entries equalled the count of in-range entries, for all 27 audio
|
||||
containers. A systematic 50/50 split is a decoding error, not sparse data.
|
||||
|
||||
The correct form is `{offset in sectors, size in bytes}` — `SectorByteRange`.
|
||||
The confirmation is that offsets chain: on `AUDIO0.WAD` the sectors are 3, 185,
|
||||
367, 1458, where each entry's end is the next entry's start, in both bytes and
|
||||
sectors. Under the byte reading the extents tile the file exactly.
|
||||
|
||||
Measured across all 92 indexed pair containers:
|
||||
|
||||
| Reading | Entries placed correctly |
|
||||
|---|---|
|
||||
| Sector/sector | many, but never all; 0 of 5 on `LEVEL0.WAD`, 0 of 42 on `AUDIO1.WAD`, and so on for all 27 of each |
|
||||
| Sector/bytes | all, on every container |
|
||||
|
||||
**Lesson for the format parsers in Phase 2:** a parser that type-checks on one
|
||||
container can still be wrong on the other 91. Prefer a check that must hold
|
||||
*everywhere* — extents tiling a file without gaps or overlaps — over a check
|
||||
that holds on the sample in front of you. This is why `ARMOR.WAD` was chosen to
|
||||
validate by hand and why that choice nearly cost the phase.
|
||||
|
||||
## The WAD LZ77 decoder
|
||||
|
||||
Also initially wrong, and instructive.
|
||||
|
||||
The first attempt modelled the format as a **bit stream**: a control byte whose
|
||||
bits each select "literal byte" or "back-reference". This is how many LZ77
|
||||
variants work, and it produced plausible-looking output for a few kilobytes —
|
||||
`80 00 00 00 10 00 0d ...` looked like real level data — before failing with an
|
||||
invalid displacement. Four bit orderings and a dozen offset/length field
|
||||
layouts were tried against a real stream; none decoded more than a handful of
|
||||
bytes.
|
||||
|
||||
The actual format is a **flag-byte packet stream** with four packet classes,
|
||||
chosen by the flag's value rather than by individual bits, where every packet
|
||||
carries 0–3 trailing literal bytes in unused low bits. It cannot be expressed as
|
||||
a bit stream, which is why no bit ordering worked. See
|
||||
[`../formats/wad.md`](../formats/wad.md).
|
||||
|
||||
The failure mode is worth remembering: a decoder that produces *plausible* wrong
|
||||
output is more dangerous than one that fails immediately. The first WAD decoder
|
||||
did exactly that.
|
||||
|
||||
## Container families
|
||||
|
||||
| Family | Count | Header size | Table |
|
||||
|---|---|---|---|
|
||||
| `LEVEL<n>` | 27 | `0x60` | sector/byte pairs |
|
||||
| `AUDIO<n>` | 27 | `0x1018` | sector/byte pairs |
|
||||
| `SCENE<n>` | 27 | `0x137c` | flat sector list |
|
||||
| `SCENE` | 1 | `0x170` | sector/byte pairs |
|
||||
| `MPEG1`, `MPEG2` | 2 | — | none; raw video |
|
||||
| others | 12 | various | sector/byte pairs |
|
||||
|
||||
`SCENE<n>` is the only family with a different table shape. `(0x137c - 8)` is
|
||||
not a multiple of 8, so the bytes cannot be pairs at all: they are 1,245 sector
|
||||
numbers in a flat list, each extent running to the next value.
|
||||
|
||||
## Deliberately deferred
|
||||
|
||||
- **The table of contents at sector 1001** is documented in
|
||||
[`../formats/disc-layout.md`](../formats/disc-layout.md) but not read by the
|
||||
extractor. Enumerating the ISO 9660 filesystem is simpler and sufficient; the
|
||||
ToC matters when per-level overlay loading is understood in Phase 3.
|
||||
- **Per-container entry semantics.** Which pair is a mesh, which is a texture,
|
||||
which is audio: recovered per family as each is parsed in Phase 2. The
|
||||
extractor only needs to know where each extent starts and ends.
|
||||
- **Audio sub-structures.** `AUDIO<n>.WAD` headers hold 36 `bindata` ranges
|
||||
followed by 15 music pointers; the split is inferred from the header size but
|
||||
not yet confirmed by unpacking both halves.
|
||||
|
||||
## Known tolerated imprecision
|
||||
|
||||
A few containers declare extents that run past the end of their file: 1 in each
|
||||
`SCENE<n>.WAD`, 10 in `GADGET.WAD`, 35 in `MPEG.WAD`. These are trailing
|
||||
sentinels or descriptors referencing sectors in an adjacent container.
|
||||
|
||||
The extractor bounds-checks every extent against the container actually read, so
|
||||
these cannot cause an out-of-bounds read. They are counted and reported rather
|
||||
than silently dropped, because a silent skip would be indistinguishable from
|
||||
data loss. Total skipped: 46 files' worth of entries, out of 5,135 files
|
||||
extracted.
|
||||
|
||||
## Verification approach
|
||||
|
||||
`crates/gc-extract/data/gc-files.tsv` records the size and SHA-1 of all 5,135
|
||||
produced files — hashes only, never content, 316 KB.
|
||||
|
||||
The value of this is that a future parser can assert "this matches retail"
|
||||
rather than "this matches whatever is on my disk". When a Phase 2 parser is
|
||||
wrong, the hash table is how we find out, and how we tell a wrong parser from a
|
||||
wrong assumption.
|
||||
|
||||
Regenerate after an intentional change:
|
||||
|
||||
```sh
|
||||
gc-extract unpack <disc.iso> extracted/ \
|
||||
--table crates/gc-extract/data/gc-files.tsv
|
||||
```
|
||||
|
||||
If only some files change, the unpack stage changed. If none do, only a parser
|
||||
downstream is wrong.
|
||||
Reference in New Issue
Block a user