Files

T

ReaperOfVeriod 027373e65b Add the disc pipeline: identify, unpack and verify

Phase 1 of the roadmap. The disc is now turned into verified asset files, and
the result is checked against a committed hash table rather than against
whatever happens to be on disk.

  gc-extract identify disc.iso        SCUS_972.68 v1.01 NTSC, ELF SHA-1 matches
  gc-extract unpack   disc.iso out/   5135 files, 4,500,106,960 bytes, 10.8s
  gc-extract verify   out/ table.tsv  OK: 5135 files match

All 92 G/*.WAD containers parse. MPEG1 and MPEG2 are raw video and pass
through whole; the other 90 are indexed containers whose extents are extracted
individually.

Formats recovered and written up in docs/formats/:

- iso9660.rs. R&C2 lists its assets in the filesystem, unlike R&C1, so a
  directory walk is the primary asset index. Three details that all fail
  quietly rather than loudly: directory records store every 32-bit field twice
  (little-endian then big-endian), the CD001 signature is at sector offset 1
  rather than 0x8001, and the directory flag at record offset 25 is
  authoritative where the identifier's trailing slash is not.

- global_wad.rs. Container headers hold a table of extents. All 92 indexed
  containers use {offset in sectors, size in bytes}. Measured, not assumed: the
  sector-count reading places every entry of all 27 LEVEL<n> and all 27
  AUDIO<n> containers out of range, while the byte reading places all of them
  correctly. SCENE<n> is the one exception, a flat sector list whose extents
  are sized by the next value.

- wad.rs. Insomniac's LZ77 stream: four flag-selected packet classes, with 0-3
  trailing literal bytes per packet carried in unused low bits. This is not a
  bit stream, which is why it resists the usual LZ77 reading.

- buf.rs. A bounds-checked reader, so untrusted 2003 disc data produces typed
  errors rather than panics.

gc-extract carries the three stages and a SHA-1 implementation with FIPS 180-4
test vectors, so identification works without pulling in a crypto dependency.

Verification: 54 unit tests, plus four integration tests that run only when
GC_DISC names a disc. Those four check identification against the recorded
values, cross-check the Rust identifier against tools/identify-disc.py, parse
all 92 containers, and confirm a full unpack matches the committed table. The
committed table is 316 KB of sizes and SHA-1s for all 5,135 files: hashes only,
no game content, so it is safe to version and it turns "matches retail" into
something a test can assert.

docs/plan/disc-pipeline.md records two wrong turns, because both are easy to
repeat and both failed by producing plausible wrong output rather than an
error: the container table was read backwards, and the LZ77 stream was modelled
as a bit stream. The first survived because it happened to be correct on the
one container used for manual checking.

2026-10-09 16:08:30 +02:00

6.5 KiB

Raw Permalink Blame History

The `G/*.WAD` asset containers

Target: SCUS_972.68 v1.01 (USA Black Label, NTSC).

Everything except the boot ELF, SYSTEM.CNF, RC2.HDR and the IOP image lives in a container under G/ on the disc's ISO 9660 filesystem. R&C2 lists these as real files, unlike R&C1 whose filesystem is vestigial, so enumeration is a directory walk rather than a hunt for a table at a fixed sector.

Container list

96 containers. The names follow the asset family; n runs 0..26 for the per-level containers.

File	Count	Bytes (disc)	Role
`ARMOR.WAD`	1	2,480,432	Armor and wrench meshes plus textures
`AUDIO.WAD`	1	35,283,296	Global audio banks and music
`AUDIO<n>.WAD`	27	3,825,304 .. 46,747,520	Per-level audio
`BONUS.WAD`	1	31,762,085	Collectibles, unlockables
`GADGET.WAD`	1	4,972,992	Gadget definitions
`HUD.WAD`	1	37,927,695	HUD elements and text
`LEVEL<n>.WAD`	27	8,350,423 .. 25,119,448	Level core, gameplay, occlusion
`MISC.WAD`	1	14,840,496	Miscellaneous assets
`MPEG.WAD`	1	1,075,267,584	Indexed video and subtitle streams
`MPEG1.WAD`	1	1,296,228,352	Raw concatenated video, not indexed
`MPEG2.WAD`	1	114,769,924	Raw concatenated video, not indexed
`SCENE.WAD`	1	4,466,256	Global cutscene data
`SCENE<n>.WAD`	27	4,988 .. 14,964,240	Per-level cutscene data
`SPACE.WAD`	1	10,977,664	Space level assets

The three MPEG WADs are 2.5 GB of the 3.8 GB disc. Video is a data problem before it is a decoding problem; the pipeline reads them in sectors rather than slurping whole files.

Header

Every indexed container begins with:

offset  size  field
0x00    4     header size in bytes
0x04    4     sector count of the whole container
0x08..  ...   table, interpreted per container kind (below)

The header size is the reliable discriminator: it is the one field every container agrees on, and its value identifies the container family.

Header size	Containers	Table kind
`0x0040`	`MISC`	sector/byte pairs
`0x0060`	`LEVEL<n>`	sector/byte pairs
`0x00f8`	`ARMOR`	sector/byte pairs
`0x0170`	`SCENE`	sector/byte pairs
`0x0328`	`MPEG`	sector/byte pairs
`0x03c8`	`GADGET`	sector/byte pairs
`0x0a48`	`BONUS`	sector/byte pairs
`0x0ba8`	`SPACE`	sector/byte pairs
`0x1018`	`AUDIO<n>`	sector/byte pairs
`0x137c`	`SCENE<n>`	flat sector list
`0x1800`	`AUDIO`	sector/byte pairs
`0x1870`	`HUD`	sector/byte pairs

Sector/byte pairs

The common form. From 0x08 to header_size, a run of 8-byte entries:

offset  size  field
0x00    4     first sector of the extent
0x04    4     length of the extent **in bytes**

The offset is in sectors, the size is in bytes. This is the single most important thing to get right, and it is easy to get backwards because both fields are 32-bit integers in a structure where either reading type-checks.

This was measured, not assumed. For all 92 indexed pair containers:

Reading the size as bytes places every entry inside its file.
Reading the size as a sector count places entries past the end of the file, and on LEVEL<n> and AUDIO<n> places none of them correctly — 0 of 5 on LEVEL0.WAD, 0 of 42 on AUDIO1.WAD, and so on for all 27 of each.

Offsets chain, which is the confirmation that the reading is right. On AUDIO0.WAD: sectors 3, 185, 367, 1458, where 185 = 3 + 182 sectors and each entry's end is the next entry's start.

Payload is not compressed

The bytes behind a pair are raw. Insomniac's LZ77 WAD container (see wad.md) is used elsewhere on the disc — inside level cores and MPEG streams — but these global containers store their sections uncompressed and let the table do the addressing. A decompressor applied here produces garbage.

Worked example: ARMOR.WAD

header_size = 0xf8, so (0xf8 - 8) / 8 = 30 entries. Reading the pairs:

entry  0: sector    1, 372,272 bytes
entry  1: sector  124, 163,840 bytes   (= 73 + 51 sectors earlier)
...
entry 29: sector 1200,      24,576 bytes

The published structure is nine armor entries then six wrench entries, each a {mesh, textures} pair, which is 15 pairs — but the generic parser reads 30 entries because the header also declares the textures of the nine armors as their own ranges. Entry 0 is armor 0's mesh and entry 1 is armor 0's textures; entries 2 and 3 are armor 1's, and so on. Which pair means what is per-container knowledge, recovered when that asset family is parsed.

Flat sector list (`SCENE<n>.WAD`)

The 27 per-level scene containers are the exception. header_size = 0x137c, and 0x137c - 8 = 4980 is not a multiple of 8, so the bytes cannot be pairs at all. They are a flat run of 1,245 u32 sector numbers, 30 scenes of 83 values each.

Each value is the first sector of one extent. The extent's length is the difference to the next value; the last runs to the end of the container.

sectors[0] = 3, sectors[1] = 7   ->  extent 0 is sectors 3..7  (4 sectors)

The global SCENE.WAD is not one of these: it uses the pair form.

Raw streams

MPEG1.WAD and MPEG2.WAD have no table. Their first word is video payload, not a plausible header size (7864320 and a negative value respectively), so they are copied through whole and split by the video phase.

Known trailing sentinels

A few containers declare entries whose extent runs past the end of the file: 1 in every SCENE<n>.WAD, 10 in GADGET.WAD, 35 in MPEG.WAD. These are trailing sentinels or descriptors that reference sectors belonging to an adjacent container. The extractor bounds-checks every extent and skips these with a count, rather than reading past the end of the buffer or trusting the header.

Verification

Unpacking the disc produces 5,135 files totalling 4,500,106,960 bytes. Every file's size and SHA-1 are recorded in crates/gc-extract/data/gc-files.tsv (hashes only, no content). See ../plan/disc-pipeline.md for how that gate is run.

What is not yet known

Which pair in each container corresponds to which asset. Needed before any asset can be parsed; the geometry is recovered per family in Phase 2.
The bindata sub-structures inside audio and scene containers.
How MPEG.WAD's indexed streams relate to the raw MPEG1/MPEG2 files. Two descriptions of the same video data exist on the disc; which the game reads is not yet established.

6.5 KiB Raw Permalink Blame History

The G/*.WAD asset containers