ReaperOfVeriod
027373e65b
Phase 1 of the roadmap. The disc is now turned into verified asset files, and
the result is checked against a committed hash table rather than against
whatever happens to be on disk.
gc-extract identify disc.iso SCUS_972.68 v1.01 NTSC, ELF SHA-1 matches
gc-extract unpack disc.iso out/ 5135 files, 4,500,106,960 bytes, 10.8s
gc-extract verify out/ table.tsv OK: 5135 files match
All 92 G/*.WAD containers parse. MPEG1 and MPEG2 are raw video and pass
through whole; the other 90 are indexed containers whose extents are extracted
individually.
Formats recovered and written up in docs/formats/:
- iso9660.rs. R&C2 lists its assets in the filesystem, unlike R&C1, so a
directory walk is the primary asset index. Three details that all fail
quietly rather than loudly: directory records store every 32-bit field twice
(little-endian then big-endian), the CD001 signature is at sector offset 1
rather than 0x8001, and the directory flag at record offset 25 is
authoritative where the identifier's trailing slash is not.
- global_wad.rs. Container headers hold a table of extents. All 92 indexed
containers use {offset in sectors, size in bytes}. Measured, not assumed: the
sector-count reading places every entry of all 27 LEVEL<n> and all 27
AUDIO<n> containers out of range, while the byte reading places all of them
correctly. SCENE<n> is the one exception, a flat sector list whose extents
are sized by the next value.
- wad.rs. Insomniac's LZ77 stream: four flag-selected packet classes, with 0-3
trailing literal bytes per packet carried in unused low bits. This is not a
bit stream, which is why it resists the usual LZ77 reading.
- buf.rs. A bounds-checked reader, so untrusted 2003 disc data produces typed
errors rather than panics.
gc-extract carries the three stages and a SHA-1 implementation with FIPS 180-4
test vectors, so identification works without pulling in a crypto dependency.
Verification: 54 unit tests, plus four integration tests that run only when
GC_DISC names a disc. Those four check identification against the recorded
values, cross-check the Rust identifier against tools/identify-disc.py, parse
all 92 containers, and confirm a full unpack matches the committed table. The
committed table is 316 KB of sizes and SHA-1s for all 5,135 files: hashes only,
no game content, so it is safe to version and it turns "matches retail" into
something a test can assert.
docs/plan/disc-pipeline.md records two wrong turns, because both are easy to
repeat and both failed by producing plausible wrong output rather than an
error: the container table was read backwards, and the LZ77 stream was modelled
as a bit stream. The first survived because it happened to be correct on the
one container used for manual checking.
160 lines
6.5 KiB
Markdown
160 lines
6.5 KiB
Markdown
# The `G/*.WAD` asset containers
|
|
|
|
Target: `SCUS_972.68` v1.01 (USA Black Label, NTSC).
|
|
|
|
Everything except the boot ELF, `SYSTEM.CNF`, `RC2.HDR` and the IOP image
|
|
lives in a container under `G/` on the disc's ISO 9660 filesystem. R&C2 lists
|
|
these as real files, unlike R&C1 whose filesystem is vestigial, so enumeration
|
|
is a directory walk rather than a hunt for a table at a fixed sector.
|
|
|
|
## Container list
|
|
|
|
96 containers. The names follow the asset family; `n` runs 0..26 for the
|
|
per-level containers.
|
|
|
|
| File | Count | Bytes (disc) | Role |
|
|
|---|---|---|---|
|
|
| `ARMOR.WAD` | 1 | 2,480,432 | Armor and wrench meshes plus textures |
|
|
| `AUDIO.WAD` | 1 | 35,283,296 | Global audio banks and music |
|
|
| `AUDIO<n>.WAD` | 27 | 3,825,304 .. 46,747,520 | Per-level audio |
|
|
| `BONUS.WAD` | 1 | 31,762,085 | Collectibles, unlockables |
|
|
| `GADGET.WAD` | 1 | 4,972,992 | Gadget definitions |
|
|
| `HUD.WAD` | 1 | 37,927,695 | HUD elements and text |
|
|
| `LEVEL<n>.WAD` | 27 | 8,350,423 .. 25,119,448 | Level core, gameplay, occlusion |
|
|
| `MISC.WAD` | 1 | 14,840,496 | Miscellaneous assets |
|
|
| `MPEG.WAD` | 1 | 1,075,267,584 | Indexed video and subtitle streams |
|
|
| `MPEG1.WAD` | 1 | 1,296,228,352 | Raw concatenated video, not indexed |
|
|
| `MPEG2.WAD` | 1 | 114,769,924 | Raw concatenated video, not indexed |
|
|
| `SCENE.WAD` | 1 | 4,466,256 | Global cutscene data |
|
|
| `SCENE<n>.WAD` | 27 | 4,988 .. 14,964,240 | Per-level cutscene data |
|
|
| `SPACE.WAD` | 1 | 10,977,664 | Space level assets |
|
|
|
|
The three MPEG WADs are 2.5 GB of the 3.8 GB disc. Video is a data problem
|
|
before it is a decoding problem; the pipeline reads them in sectors rather than
|
|
slurping whole files.
|
|
|
|
## Header
|
|
|
|
Every indexed container begins with:
|
|
|
|
```text
|
|
offset size field
|
|
0x00 4 header size in bytes
|
|
0x04 4 sector count of the whole container
|
|
0x08.. ... table, interpreted per container kind (below)
|
|
```
|
|
|
|
The header size is the reliable discriminator: it is the one field every
|
|
container agrees on, and its value identifies the container family.
|
|
|
|
| Header size | Containers | Table kind |
|
|
|---|---|---|
|
|
| `0x0040` | `MISC` | sector/byte pairs |
|
|
| `0x0060` | `LEVEL<n>` | sector/byte pairs |
|
|
| `0x00f8` | `ARMOR` | sector/byte pairs |
|
|
| `0x0170` | `SCENE` | sector/byte pairs |
|
|
| `0x0328` | `MPEG` | sector/byte pairs |
|
|
| `0x03c8` | `GADGET` | sector/byte pairs |
|
|
| `0x0a48` | `BONUS` | sector/byte pairs |
|
|
| `0x0ba8` | `SPACE` | sector/byte pairs |
|
|
| `0x1018` | `AUDIO<n>` | sector/byte pairs |
|
|
| `0x137c` | `SCENE<n>` | **flat sector list** |
|
|
| `0x1800` | `AUDIO` | sector/byte pairs |
|
|
| `0x1870` | `HUD` | sector/byte pairs |
|
|
|
|
## Sector/byte pairs
|
|
|
|
The common form. From `0x08` to `header_size`, a run of 8-byte entries:
|
|
|
|
```text
|
|
offset size field
|
|
0x00 4 first sector of the extent
|
|
0x04 4 length of the extent **in bytes**
|
|
```
|
|
|
|
The offset is in **sectors**, the size is in **bytes**. This is the single most
|
|
important thing to get right, and it is easy to get backwards because both
|
|
fields are 32-bit integers in a structure where either reading type-checks.
|
|
|
|
**This was measured, not assumed.** For all 92 indexed pair containers:
|
|
|
|
- Reading the size as bytes places every entry inside its file.
|
|
- Reading the size as a sector count places entries past the end of the file,
|
|
and on `LEVEL<n>` and `AUDIO<n>` places *none* of them correctly — 0 of 5 on
|
|
`LEVEL0.WAD`, 0 of 42 on `AUDIO1.WAD`, and so on for all 27 of each.
|
|
|
|
Offsets chain, which is the confirmation that the reading is right. On
|
|
`AUDIO0.WAD`: sectors 3, 185, 367, 1458, where `185 = 3 + 182` sectors and each
|
|
entry's end is the next entry's start.
|
|
|
|
### Payload is not compressed
|
|
|
|
The bytes behind a pair are raw. Insomniac's LZ77 WAD container (see
|
|
[`wad.md`](wad.md)) is used elsewhere on the disc — inside level cores and MPEG
|
|
streams — but these global containers store their sections uncompressed and let
|
|
the table do the addressing. A decompressor applied here produces garbage.
|
|
|
|
### Worked example: ARMOR.WAD
|
|
|
|
`header_size = 0xf8`, so `(0xf8 - 8) / 8 = 30` entries. Reading the pairs:
|
|
|
|
```text
|
|
entry 0: sector 1, 372,272 bytes
|
|
entry 1: sector 124, 163,840 bytes (= 73 + 51 sectors earlier)
|
|
...
|
|
entry 29: sector 1200, 24,576 bytes
|
|
```
|
|
|
|
The published structure is nine armor entries then six wrench entries, each a
|
|
`{mesh, textures}` pair, which is 15 pairs — but the generic parser reads 30
|
|
entries because the header also declares the textures of the nine armors as
|
|
their own ranges. Entry 0 is armor 0's mesh and entry 1 is armor 0's textures;
|
|
entries 2 and 3 are armor 1's, and so on. Which pair means what is per-container
|
|
knowledge, recovered when that asset family is parsed.
|
|
|
|
## Flat sector list (`SCENE<n>.WAD`)
|
|
|
|
The 27 per-level scene containers are the exception. `header_size = 0x137c`,
|
|
and `0x137c - 8 = 4980` is **not** a multiple of 8, so the bytes cannot be
|
|
pairs at all. They are a flat run of 1,245 `u32` sector numbers, 30 scenes of
|
|
83 values each.
|
|
|
|
Each value is the first sector of one extent. The extent's length is the
|
|
difference to the next value; the last runs to the end of the container.
|
|
|
|
```text
|
|
sectors[0] = 3, sectors[1] = 7 -> extent 0 is sectors 3..7 (4 sectors)
|
|
```
|
|
|
|
The global `SCENE.WAD` is not one of these: it uses the pair form.
|
|
|
|
## Raw streams
|
|
|
|
`MPEG1.WAD` and `MPEG2.WAD` have no table. Their first word is video payload,
|
|
not a plausible header size (`7864320` and a negative value respectively), so
|
|
they are copied through whole and split by the video phase.
|
|
|
|
## Known trailing sentinels
|
|
|
|
A few containers declare entries whose extent runs past the end of the file:
|
|
1 in every `SCENE<n>.WAD`, 10 in `GADGET.WAD`, 35 in `MPEG.WAD`. These are
|
|
trailing sentinels or descriptors that reference sectors belonging to an
|
|
adjacent container. The extractor bounds-checks every extent and skips these
|
|
with a count, rather than reading past the end of the buffer or trusting the
|
|
header.
|
|
|
|
## Verification
|
|
|
|
Unpacking the disc produces 5,135 files totalling 4,500,106,960 bytes. Every
|
|
file's size and SHA-1 are recorded in
|
|
`crates/gc-extract/data/gc-files.tsv` (hashes only, no content). See
|
|
[`../plan/disc-pipeline.md`](../plan/disc-pipeline.md) for how that gate is run.
|
|
|
|
## What is not yet known
|
|
|
|
- Which pair in each container corresponds to which asset. Needed before any
|
|
asset can be parsed; the geometry is recovered per family in Phase 2.
|
|
- The `bindata` sub-structures inside audio and scene containers.
|
|
- How `MPEG.WAD`'s indexed streams relate to the raw `MPEG1`/`MPEG2` files.
|
|
Two descriptions of the same video data exist on the disc; which the game
|
|
reads is not yet established. |