Add the disc pipeline: identify, unpack and verify
Phase 1 of the roadmap. The disc is now turned into verified asset files, and
the result is checked against a committed hash table rather than against
whatever happens to be on disk.
gc-extract identify disc.iso SCUS_972.68 v1.01 NTSC, ELF SHA-1 matches
gc-extract unpack disc.iso out/ 5135 files, 4,500,106,960 bytes, 10.8s
gc-extract verify out/ table.tsv OK: 5135 files match
All 92 G/*.WAD containers parse. MPEG1 and MPEG2 are raw video and pass
through whole; the other 90 are indexed containers whose extents are extracted
individually.
Formats recovered and written up in docs/formats/:
- iso9660.rs. R&C2 lists its assets in the filesystem, unlike R&C1, so a
directory walk is the primary asset index. Three details that all fail
quietly rather than loudly: directory records store every 32-bit field twice
(little-endian then big-endian), the CD001 signature is at sector offset 1
rather than 0x8001, and the directory flag at record offset 25 is
authoritative where the identifier's trailing slash is not.
- global_wad.rs. Container headers hold a table of extents. All 92 indexed
containers use {offset in sectors, size in bytes}. Measured, not assumed: the
sector-count reading places every entry of all 27 LEVEL<n> and all 27
AUDIO<n> containers out of range, while the byte reading places all of them
correctly. SCENE<n> is the one exception, a flat sector list whose extents
are sized by the next value.
- wad.rs. Insomniac's LZ77 stream: four flag-selected packet classes, with 0-3
trailing literal bytes per packet carried in unused low bits. This is not a
bit stream, which is why it resists the usual LZ77 reading.
- buf.rs. A bounds-checked reader, so untrusted 2003 disc data produces typed
errors rather than panics.
gc-extract carries the three stages and a SHA-1 implementation with FIPS 180-4
test vectors, so identification works without pulling in a crypto dependency.
Verification: 54 unit tests, plus four integration tests that run only when
GC_DISC names a disc. Those four check identification against the recorded
values, cross-check the Rust identifier against tools/identify-disc.py, parse
all 92 containers, and confirm a full unpack matches the committed table. The
committed table is 316 KB of sizes and SHA-1s for all 5,135 files: hashes only,
no game content, so it is safe to version and it turns "matches retail" into
something a test can assert.
docs/plan/disc-pipeline.md records two wrong turns, because both are easy to
repeat and both failed by producing plausible wrong output rather than an
error: the container table was read backwards, and the LZ77 stream was modelled
as a bit stream. The first survived because it happened to be correct on the
one container used for manual checking.
This commit is contained in:
@@ -0,0 +1,110 @@
|
||||
# Disc layout
|
||||
|
||||
Target: `SCUS_972.68` v1.01 (USA Black Label, NTSC 60 Hz).
|
||||
|
||||
All integers are little-endian. Sector size is `0x800` (2048) bytes throughout;
|
||||
everything on the disc is sector-aligned.
|
||||
|
||||
## Root directory
|
||||
|
||||
Observed by enumerating the ISO 9660 filesystem:
|
||||
|
||||
```text
|
||||
SYSTEM.CNF 60 bytes
|
||||
RC2.HDR 408,444 the table of contents, as a named file
|
||||
SCUS_972.68 2,618,684 the boot ELF
|
||||
IOPRP255.IMG 264,449 IOP modules
|
||||
Z6TAIL.DUP 0 ISO 9660 padding
|
||||
G/... 96 asset containers (see global-wad.md)
|
||||
```
|
||||
|
||||
`RC2.HDR` being present as a real file is convenient. R&C1's equivalent is not a
|
||||
named file and sits at sector 1500.
|
||||
|
||||
## ISO 9660 usage
|
||||
|
||||
The filesystem is load-bearing for this game, unlike R&C1 whose filesystem is
|
||||
nearly vestigial. Everything the engine needs is either a file here or a sector
|
||||
range declared by a container header.
|
||||
|
||||
**Consequence for the extractor:** `G/*` can be enumerated by walking the
|
||||
directory. There is no need to parse a table of contents to find the assets.
|
||||
|
||||
### Details that will bite
|
||||
|
||||
- **Directory records store each 32-bit field twice**, little-endian then
|
||||
big-endian. Reading 4 bytes where the record has 8 shifts every subsequent
|
||||
field and yields an empty or nonsense listing with no error.
|
||||
- **The `CD001` signature is at sector offset `0x0001`**, not `0x8001`. The
|
||||
latter is its absolute file offset and lies outside a 2048-byte sector, so a
|
||||
bounds-checked read there fails rather than returning empty bytes.
|
||||
- **The directory flag is at record offset 25**, bit 1. Do not rely on the
|
||||
identifier's trailing `/`: not every writer sets it, and the flag is
|
||||
authoritative. Getting this wrong writes a subdirectory's *contents* to a file
|
||||
named after the directory.
|
||||
- Records never span a logical block; a zero record length means "advance to the
|
||||
next block".
|
||||
|
||||
## Table of contents (`RC2.HDR`)
|
||||
|
||||
The game also maintains a table of contents, and a second copy lives at sector
|
||||
**1001** (`SYSTEM.CNF` sits at sector 1000 on this disc). The ToC begins with a
|
||||
chain of global container headers, each `{header_size, sector}`, running from
|
||||
offset 0 until the level records begin.
|
||||
|
||||
Verified on this disc:
|
||||
|
||||
```text
|
||||
offset 0x0000 header_size 0x0328 sector 2610 MPEG
|
||||
offset 0x0328 header_size 0x0040 sector 1216608 MISC
|
||||
offset 0x0368 header_size 0x1870 sector 1223855 HUD
|
||||
offset 0x1bd8 header_size 0x0a48 sector 1242375 BONUS
|
||||
offset 0x2620 header_size 0x1800 sector 1257884 AUDIO
|
||||
offset 0x3e20 header_size 0x0ba8 sector 1275113 SPACE
|
||||
offset 0x49c8 header_size 0x0170 sector 1280474 SCENE
|
||||
offset 0x4b38 header_size 0x03c8 sector 1282655 GADGET
|
||||
offset 0x4f00 header_size 0x00f8 sector 1285084 ARMOR
|
||||
```
|
||||
|
||||
The nine header sizes match the container table in
|
||||
[`global-wad.md`](global-wad.md) exactly, which cross-confirms both readings.
|
||||
|
||||
### Level records
|
||||
|
||||
Level records follow the globals on a **`0x2000`-byte stride**, each beginning
|
||||
with `{header_size, sector}` where the sector is an **absolute** disc sector:
|
||||
|
||||
```text
|
||||
+0x00000 LEVEL 0x0060 sector 1286296
|
||||
+0x02000 SCENE 0x137c sector 1296509
|
||||
+0x04000 AUDIO 0x1018 sector 1309652
|
||||
+0x06000 <zero>
|
||||
...
|
||||
```
|
||||
|
||||
Levels appear as `LEVEL`/`SCENE`/`AUDIO` triples separated by gaps, and every
|
||||
level's sector cross-checks against the `G/LEVEL<n>.WAD` entry found by walking
|
||||
the filesystem. That agreement between two independent routes is what confirms
|
||||
the layout; a single route agreeing with itself proves nothing.
|
||||
|
||||
**Not yet implemented.** The extractor does not read the ToC. It enumerates the
|
||||
filesystem instead, which is sufficient and simpler. The ToC becomes necessary
|
||||
when the game needs to resolve a sector at runtime, or when per-level overlay
|
||||
loading has to be understood (Phase 3).
|
||||
|
||||
## Boot executable
|
||||
|
||||
`SCUS_972.68`, 2,618,684 bytes, SHA-1
|
||||
`bacda860cbd6723142e86db56f197e47e728970f`.
|
||||
|
||||
ELF32, little-endian, MIPS (EM_MIPS = 8), entry point `0x00131AE8`, `$gp`
|
||||
`0x001AEFF0`. The entry point independently matches the value recorded by the
|
||||
`rac2-gc-decomp` community reference, which cross-confirms the disc
|
||||
identification.
|
||||
|
||||
## Provenance
|
||||
|
||||
Container layout knowledge was guided by the Wrench project's format
|
||||
descriptions (<https://github.com/chaoticgd/wrench>, GPLv3). **No Wrench code is
|
||||
copied** — see `PROVENANCE.md`. Structures here are written from the
|
||||
specification and verified against this disc.
|
||||
Reference in New Issue
Block a user