# Disc pipeline The work record for Phase 1. What was tried, what worked, and what was ruled out — the negative results matter as much as the working parts. ## Result The disc pipeline runs end to end and is verified. ```text $ gc-extract identify disc.iso serial SCUS_972.68 version 1.01 video mode NTSC boot ELF SHA-1 bacda860cbd6723142e86db56f197e47e728970f $ gc-extract unpack disc.iso extracted/ --table gc-files.tsv 5135 files, 4,500,106,960 bytes, in 10.8s $ gc-extract verify extracted/ gc-files.tsv OK: 5135 files match the recorded table. ``` Every one of the 96 `G/` containers parses. 92 are indexed containers whose extents are extracted individually; `MPEG1.WAD` and `MPEG2.WAD` are raw video and pass through whole. ## The wrong turn worth recording **The container pair table was read backwards for most of this phase.** The obvious reading of an 8-byte entry is `{offset, size}`, both sector counts — `SectorRange`. That reading: - worked perfectly for `ARMOR.WAD` (30 of 30 entries in range), which is exactly why it survived several rounds of testing; - placed 1 entry out of range on `HUD.WAD`, 69 of 178 on `AUDIO.WAD`, and **all entries out of range** on every `AUDIO.WAD` and `LEVEL.WAD`. The `AUDIO0.WAD` numbers were the tell: under the sector reading, the count of out-of-range entries equalled the count of in-range entries, for all 27 audio containers. A systematic 50/50 split is a decoding error, not sparse data. The correct form is `{offset in sectors, size in bytes}` — `SectorByteRange`. The confirmation is that offsets chain: on `AUDIO0.WAD` the sectors are 3, 185, 367, 1458, where each entry's end is the next entry's start, in both bytes and sectors. Under the byte reading the extents tile the file exactly. Measured across all 92 indexed pair containers: | Reading | Entries placed correctly | |---|---| | Sector/sector | many, but never all; 0 of 5 on `LEVEL0.WAD`, 0 of 42 on `AUDIO1.WAD`, and so on for all 27 of each | | Sector/bytes | all, on every container | **Lesson for the format parsers in Phase 2:** a parser that type-checks on one container can still be wrong on the other 91. Prefer a check that must hold *everywhere* — extents tiling a file without gaps or overlaps — over a check that holds on the sample in front of you. This is why `ARMOR.WAD` was chosen to validate by hand and why that choice nearly cost the phase. ## The WAD LZ77 decoder Also initially wrong, and instructive. The first attempt modelled the format as a **bit stream**: a control byte whose bits each select "literal byte" or "back-reference". This is how many LZ77 variants work, and it produced plausible-looking output for a few kilobytes — `80 00 00 00 10 00 0d ...` looked like real level data — before failing with an invalid displacement. Four bit orderings and a dozen offset/length field layouts were tried against a real stream; none decoded more than a handful of bytes. The actual format is a **flag-byte packet stream** with four packet classes, chosen by the flag's value rather than by individual bits, where every packet carries 0–3 trailing literal bytes in unused low bits. It cannot be expressed as a bit stream, which is why no bit ordering worked. See [`../formats/wad.md`](../formats/wad.md). The failure mode is worth remembering: a decoder that produces *plausible* wrong output is more dangerous than one that fails immediately. The first WAD decoder did exactly that. ## Container families | Family | Count | Header size | Table | |---|---|---|---| | `LEVEL` | 27 | `0x60` | sector/byte pairs | | `AUDIO` | 27 | `0x1018` | sector/byte pairs | | `SCENE` | 27 | `0x137c` | flat sector list | | `SCENE` | 1 | `0x170` | sector/byte pairs | | `MPEG1`, `MPEG2` | 2 | — | none; raw video | | others | 12 | various | sector/byte pairs | `SCENE` is the only family with a different table shape. `(0x137c - 8)` is not a multiple of 8, so the bytes cannot be pairs at all: they are 1,245 sector numbers in a flat list, each extent running to the next value. ## Deliberately deferred - **The table of contents at sector 1001** is documented in [`../formats/disc-layout.md`](../formats/disc-layout.md) but not read by the extractor. Enumerating the ISO 9660 filesystem is simpler and sufficient; the ToC matters when per-level overlay loading is understood in Phase 3. - **Per-container entry semantics.** Which pair is a mesh, which is a texture, which is audio: recovered per family as each is parsed in Phase 2. The extractor only needs to know where each extent starts and ends. - **Audio sub-structures.** `AUDIO.WAD` headers hold 36 `bindata` ranges followed by 15 music pointers; the split is inferred from the header size but not yet confirmed by unpacking both halves. ## Known tolerated imprecision A few containers declare extents that run past the end of their file: 1 in each `SCENE.WAD`, 10 in `GADGET.WAD`, 35 in `MPEG.WAD`. These are trailing sentinels or descriptors referencing sectors in an adjacent container. The extractor bounds-checks every extent against the container actually read, so these cannot cause an out-of-bounds read. They are counted and reported rather than silently dropped, because a silent skip would be indistinguishable from data loss. Total skipped: 46 files' worth of entries, out of 5,135 files extracted. ## Verification approach `crates/gc-extract/data/gc-files.tsv` records the size and SHA-1 of all 5,135 produced files — hashes only, never content, 316 KB. The value of this is that a future parser can assert "this matches retail" rather than "this matches whatever is on my disk". When a Phase 2 parser is wrong, the hash table is how we find out, and how we tell a wrong parser from a wrong assumption. Regenerate after an intentional change: ```sh gc-extract unpack extracted/ \ --table crates/gc-extract/data/gc-files.tsv ``` If only some files change, the unpack stage changed. If none do, only a parser downstream is wrong.