ReaperOfVeriod
027373e65b
Phase 1 of the roadmap. The disc is now turned into verified asset files, and
the result is checked against a committed hash table rather than against
whatever happens to be on disk.
gc-extract identify disc.iso SCUS_972.68 v1.01 NTSC, ELF SHA-1 matches
gc-extract unpack disc.iso out/ 5135 files, 4,500,106,960 bytes, 10.8s
gc-extract verify out/ table.tsv OK: 5135 files match
All 92 G/*.WAD containers parse. MPEG1 and MPEG2 are raw video and pass
through whole; the other 90 are indexed containers whose extents are extracted
individually.
Formats recovered and written up in docs/formats/:
- iso9660.rs. R&C2 lists its assets in the filesystem, unlike R&C1, so a
directory walk is the primary asset index. Three details that all fail
quietly rather than loudly: directory records store every 32-bit field twice
(little-endian then big-endian), the CD001 signature is at sector offset 1
rather than 0x8001, and the directory flag at record offset 25 is
authoritative where the identifier's trailing slash is not.
- global_wad.rs. Container headers hold a table of extents. All 92 indexed
containers use {offset in sectors, size in bytes}. Measured, not assumed: the
sector-count reading places every entry of all 27 LEVEL<n> and all 27
AUDIO<n> containers out of range, while the byte reading places all of them
correctly. SCENE<n> is the one exception, a flat sector list whose extents
are sized by the next value.
- wad.rs. Insomniac's LZ77 stream: four flag-selected packet classes, with 0-3
trailing literal bytes per packet carried in unused low bits. This is not a
bit stream, which is why it resists the usual LZ77 reading.
- buf.rs. A bounds-checked reader, so untrusted 2003 disc data produces typed
errors rather than panics.
gc-extract carries the three stages and a SHA-1 implementation with FIPS 180-4
test vectors, so identification works without pulling in a crypto dependency.
Verification: 54 unit tests, plus four integration tests that run only when
GC_DISC names a disc. Those four check identification against the recorded
values, cross-check the Rust identifier against tools/identify-disc.py, parse
all 92 containers, and confirm a full unpack matches the committed table. The
committed table is 316 KB of sizes and SHA-1s for all 5,135 files: hashes only,
no game content, so it is safe to version and it turns "matches retail" into
something a test can assert.
docs/plan/disc-pipeline.md records two wrong turns, because both are easy to
repeat and both failed by producing plausible wrong output rather than an
error: the container table was read backwards, and the LZ77 stream was modelled
as a bit stream. The first survived because it happened to be correct on the
one container used for manual checking.
140 lines
5.9 KiB
Markdown
140 lines
5.9 KiB
Markdown
# Disc pipeline
|
||
|
||
The work record for Phase 1. What was tried, what worked, and what was ruled
|
||
out — the negative results matter as much as the working parts.
|
||
|
||
## Result
|
||
|
||
The disc pipeline runs end to end and is verified.
|
||
|
||
```text
|
||
$ gc-extract identify disc.iso
|
||
serial SCUS_972.68 version 1.01 video mode NTSC
|
||
boot ELF SHA-1 bacda860cbd6723142e86db56f197e47e728970f
|
||
|
||
$ gc-extract unpack disc.iso extracted/ --table gc-files.tsv
|
||
5135 files, 4,500,106,960 bytes, in 10.8s
|
||
|
||
$ gc-extract verify extracted/ gc-files.tsv
|
||
OK: 5135 files match the recorded table.
|
||
```
|
||
|
||
Every one of the 96 `G/` containers parses. 92 are indexed containers whose
|
||
extents are extracted individually; `MPEG1.WAD` and `MPEG2.WAD` are raw video
|
||
and pass through whole.
|
||
|
||
## The wrong turn worth recording
|
||
|
||
**The container pair table was read backwards for most of this phase.**
|
||
|
||
The obvious reading of an 8-byte entry is `{offset, size}`, both sector counts —
|
||
`SectorRange`. That reading:
|
||
|
||
- worked perfectly for `ARMOR.WAD` (30 of 30 entries in range), which is exactly
|
||
why it survived several rounds of testing;
|
||
- placed 1 entry out of range on `HUD.WAD`, 69 of 178 on `AUDIO.WAD`, and **all
|
||
entries out of range** on every `AUDIO<n>.WAD` and `LEVEL<n>.WAD`.
|
||
|
||
The `AUDIO0.WAD` numbers were the tell: under the sector reading, the count of
|
||
out-of-range entries equalled the count of in-range entries, for all 27 audio
|
||
containers. A systematic 50/50 split is a decoding error, not sparse data.
|
||
|
||
The correct form is `{offset in sectors, size in bytes}` — `SectorByteRange`.
|
||
The confirmation is that offsets chain: on `AUDIO0.WAD` the sectors are 3, 185,
|
||
367, 1458, where each entry's end is the next entry's start, in both bytes and
|
||
sectors. Under the byte reading the extents tile the file exactly.
|
||
|
||
Measured across all 92 indexed pair containers:
|
||
|
||
| Reading | Entries placed correctly |
|
||
|---|---|
|
||
| Sector/sector | many, but never all; 0 of 5 on `LEVEL0.WAD`, 0 of 42 on `AUDIO1.WAD`, and so on for all 27 of each |
|
||
| Sector/bytes | all, on every container |
|
||
|
||
**Lesson for the format parsers in Phase 2:** a parser that type-checks on one
|
||
container can still be wrong on the other 91. Prefer a check that must hold
|
||
*everywhere* — extents tiling a file without gaps or overlaps — over a check
|
||
that holds on the sample in front of you. This is why `ARMOR.WAD` was chosen to
|
||
validate by hand and why that choice nearly cost the phase.
|
||
|
||
## The WAD LZ77 decoder
|
||
|
||
Also initially wrong, and instructive.
|
||
|
||
The first attempt modelled the format as a **bit stream**: a control byte whose
|
||
bits each select "literal byte" or "back-reference". This is how many LZ77
|
||
variants work, and it produced plausible-looking output for a few kilobytes —
|
||
`80 00 00 00 10 00 0d ...` looked like real level data — before failing with an
|
||
invalid displacement. Four bit orderings and a dozen offset/length field
|
||
layouts were tried against a real stream; none decoded more than a handful of
|
||
bytes.
|
||
|
||
The actual format is a **flag-byte packet stream** with four packet classes,
|
||
chosen by the flag's value rather than by individual bits, where every packet
|
||
carries 0–3 trailing literal bytes in unused low bits. It cannot be expressed as
|
||
a bit stream, which is why no bit ordering worked. See
|
||
[`../formats/wad.md`](../formats/wad.md).
|
||
|
||
The failure mode is worth remembering: a decoder that produces *plausible* wrong
|
||
output is more dangerous than one that fails immediately. The first WAD decoder
|
||
did exactly that.
|
||
|
||
## Container families
|
||
|
||
| Family | Count | Header size | Table |
|
||
|---|---|---|---|
|
||
| `LEVEL<n>` | 27 | `0x60` | sector/byte pairs |
|
||
| `AUDIO<n>` | 27 | `0x1018` | sector/byte pairs |
|
||
| `SCENE<n>` | 27 | `0x137c` | flat sector list |
|
||
| `SCENE` | 1 | `0x170` | sector/byte pairs |
|
||
| `MPEG1`, `MPEG2` | 2 | — | none; raw video |
|
||
| others | 12 | various | sector/byte pairs |
|
||
|
||
`SCENE<n>` is the only family with a different table shape. `(0x137c - 8)` is
|
||
not a multiple of 8, so the bytes cannot be pairs at all: they are 1,245 sector
|
||
numbers in a flat list, each extent running to the next value.
|
||
|
||
## Deliberately deferred
|
||
|
||
- **The table of contents at sector 1001** is documented in
|
||
[`../formats/disc-layout.md`](../formats/disc-layout.md) but not read by the
|
||
extractor. Enumerating the ISO 9660 filesystem is simpler and sufficient; the
|
||
ToC matters when per-level overlay loading is understood in Phase 3.
|
||
- **Per-container entry semantics.** Which pair is a mesh, which is a texture,
|
||
which is audio: recovered per family as each is parsed in Phase 2. The
|
||
extractor only needs to know where each extent starts and ends.
|
||
- **Audio sub-structures.** `AUDIO<n>.WAD` headers hold 36 `bindata` ranges
|
||
followed by 15 music pointers; the split is inferred from the header size but
|
||
not yet confirmed by unpacking both halves.
|
||
|
||
## Known tolerated imprecision
|
||
|
||
A few containers declare extents that run past the end of their file: 1 in each
|
||
`SCENE<n>.WAD`, 10 in `GADGET.WAD`, 35 in `MPEG.WAD`. These are trailing
|
||
sentinels or descriptors referencing sectors in an adjacent container.
|
||
|
||
The extractor bounds-checks every extent against the container actually read, so
|
||
these cannot cause an out-of-bounds read. They are counted and reported rather
|
||
than silently dropped, because a silent skip would be indistinguishable from
|
||
data loss. Total skipped: 46 files' worth of entries, out of 5,135 files
|
||
extracted.
|
||
|
||
## Verification approach
|
||
|
||
`crates/gc-extract/data/gc-files.tsv` records the size and SHA-1 of all 5,135
|
||
produced files — hashes only, never content, 316 KB.
|
||
|
||
The value of this is that a future parser can assert "this matches retail"
|
||
rather than "this matches whatever is on my disk". When a Phase 2 parser is
|
||
wrong, the hash table is how we find out, and how we tell a wrong parser from a
|
||
wrong assumption.
|
||
|
||
Regenerate after an intentional change:
|
||
|
||
```sh
|
||
gc-extract unpack <disc.iso> extracted/ \
|
||
--table crates/gc-extract/data/gc-files.tsv
|
||
```
|
||
|
||
If only some files change, the unpack stage changed. If none do, only a parser
|
||
downstream is wrong. |