Going-Commando/docs/plan/disc-pipeline.md

Disc pipeline

The work record for Phase 1. What was tried, what worked, and what was ruled out — the negative results matter as much as the working parts.

Result

The disc pipeline runs end to end and is verified.

$ gc-extract identify disc.iso
  serial  SCUS_972.68   version 1.01   video mode NTSC
  boot ELF SHA-1 bacda860cbd6723142e86db56f197e47e728970f

$ gc-extract unpack disc.iso extracted/ --table gc-files.tsv
  5135 files, 4,500,106,960 bytes, in 10.8s

$ gc-extract verify extracted/ gc-files.tsv
  OK: 5135 files match the recorded table.

Every one of the 96 G/ containers parses. 92 are indexed containers whose extents are extracted individually; MPEG1.WAD and MPEG2.WAD are raw video and pass through whole.

The wrong turn worth recording

The container pair table was read backwards for most of this phase.

The obvious reading of an 8-byte entry is {offset, size}, both sector counts — SectorRange. That reading:

• worked perfectly for ARMOR.WAD (30 of 30 entries in range), which is exactly why it survived several rounds of testing;
• placed 1 entry out of range on HUD.WAD, 69 of 178 on AUDIO.WAD, and all entries out of range on every AUDIO<n>.WAD and LEVEL<n>.WAD.

The AUDIO0.WAD numbers were the tell: under the sector reading, the count of out-of-range entries equalled the count of in-range entries, for all 27 audio containers. A systematic 50/50 split is a decoding error, not sparse data.

The correct form is {offset in sectors, size in bytes} — SectorByteRange. The confirmation is that offsets chain: on AUDIO0.WAD the sectors are 3, 185, 367, 1458, where each entry's end is the next entry's start, in both bytes and sectors. Under the byte reading the extents tile the file exactly.

Measured across all 92 indexed pair containers:

Reading	Entries placed correctly
Sector/sector	many, but never all; 0 of 5 on LEVEL0.WAD, 0 of 42 on AUDIO1.WAD, and so on for all 27 of each
Sector/bytes	all, on every container

Lesson for the format parsers in Phase 2: a parser that type-checks on one container can still be wrong on the other 91. Prefer a check that must hold everywhere — extents tiling a file without gaps or overlaps — over a check that holds on the sample in front of you. This is why ARMOR.WAD was chosen to validate by hand and why that choice nearly cost the phase.

The WAD LZ77 decoder

Also initially wrong, and instructive.

The first attempt modelled the format as a bit stream: a control byte whose bits each select "literal byte" or "back-reference". This is how many LZ77 variants work, and it produced plausible-looking output for a few kilobytes — 80 00 00 00 10 00 0d ... looked like real level data — before failing with an invalid displacement. Four bit orderings and a dozen offset/length field layouts were tried against a real stream; none decoded more than a handful of bytes.

The actual format is a flag-byte packet stream with four packet classes, chosen by the flag's value rather than by individual bits, where every packet carries 0–3 trailing literal bytes in unused low bits. It cannot be expressed as a bit stream, which is why no bit ordering worked. See ../formats/wad.md.

The failure mode is worth remembering: a decoder that produces plausible wrong output is more dangerous than one that fails immediately. The first WAD decoder did exactly that.

Container families

Family	Count	Header size	Table
LEVEL<n>	27	0x60	sector/byte pairs
AUDIO<n>	27	0x1018	sector/byte pairs
SCENE<n>	27	0x137c	flat sector list
SCENE	1	0x170	sector/byte pairs
MPEG1, MPEG2	2	—	none; raw video
others	12	various	sector/byte pairs

SCENE<n> is the only family with a different table shape. (0x137c - 8) is not a multiple of 8, so the bytes cannot be pairs at all: they are 1,245 sector numbers in a flat list, each extent running to the next value.

Deliberately deferred

• The table of contents at sector 1001 is documented in ../formats/disc-layout.md but not read by the extractor. Enumerating the ISO 9660 filesystem is simpler and sufficient; the ToC matters when per-level overlay loading is understood in Phase 3.
• Per-container entry semantics. Which pair is a mesh, which is a texture, which is audio: recovered per family as each is parsed in Phase 2. The extractor only needs to know where each extent starts and ends.
• Audio sub-structures. AUDIO<n>.WAD headers hold 36 bindata ranges followed by 15 music pointers; the split is inferred from the header size but not yet confirmed by unpacking both halves.

Known tolerated imprecision

A few containers declare extents that run past the end of their file: 1 in each SCENE<n>.WAD, 10 in GADGET.WAD, 35 in MPEG.WAD. These are trailing sentinels or descriptors referencing sectors in an adjacent container.

The extractor bounds-checks every extent against the container actually read, so these cannot cause an out-of-bounds read. They are counted and reported rather than silently dropped, because a silent skip would be indistinguishable from data loss. Total skipped: 46 files' worth of entries, out of 5,135 files extracted.

Verification approach

crates/gc-extract/data/gc-files.tsv records the size and SHA-1 of all 5,135 produced files — hashes only, never content, 316 KB.

The value of this is that a future parser can assert "this matches retail" rather than "this matches whatever is on my disk". When a Phase 2 parser is wrong, the hash table is how we find out, and how we tell a wrong parser from a wrong assumption.

Regenerate after an intentional change:

gc-extract unpack <disc.iso> extracted/ \
  --table crates/gc-extract/data/gc-files.tsv

If only some files change, the unpack stage changed. If none do, only a parser downstream is wrong.