ReaperOfVeriod/Going-Commando
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add the disc pipeline: identify, unpack and verify

Browse Source 
Phase 1 of the roadmap. The disc is now turned into verified asset files, and
the result is checked against a committed hash table rather than against
whatever happens to be on disk.

  gc-extract identify disc.iso        SCUS_972.68 v1.01 NTSC, ELF SHA-1 matches
  gc-extract unpack   disc.iso out/   5135 files, 4,500,106,960 bytes, 10.8s
  gc-extract verify   out/ table.tsv  OK: 5135 files match

All 92 G/*.WAD containers parse. MPEG1 and MPEG2 are raw video and pass
through whole; the other 90 are indexed containers whose extents are extracted
individually.

Formats recovered and written up in docs/formats/:

- iso9660.rs. R&C2 lists its assets in the filesystem, unlike R&C1, so a
  directory walk is the primary asset index. Three details that all fail
  quietly rather than loudly: directory records store every 32-bit field twice
  (little-endian then big-endian), the CD001 signature is at sector offset 1
  rather than 0x8001, and the directory flag at record offset 25 is
  authoritative where the identifier's trailing slash is not.

- global_wad.rs. Container headers hold a table of extents. All 92 indexed
  containers use {offset in sectors, size in bytes}. Measured, not assumed: the
  sector-count reading places every entry of all 27 LEVEL<n> and all 27
  AUDIO<n> containers out of range, while the byte reading places all of them
  correctly. SCENE<n> is the one exception, a flat sector list whose extents
  are sized by the next value.

- wad.rs. Insomniac's LZ77 stream: four flag-selected packet classes, with 0-3
  trailing literal bytes per packet carried in unused low bits. This is not a
  bit stream, which is why it resists the usual LZ77 reading.

- buf.rs. A bounds-checked reader, so untrusted 2003 disc data produces typed
  errors rather than panics.

gc-extract carries the three stages and a SHA-1 implementation with FIPS 180-4
test vectors, so identification works without pulling in a crypto dependency.

Verification: 54 unit tests, plus four integration tests that run only when
GC_DISC names a disc. Those four check identification against the recorded
values, cross-check the Rust identifier against tools/identify-disc.py, parse
all 92 containers, and confirm a full unpack matches the committed table. The
committed table is 316 KB of sizes and SHA-1s for all 5,135 files: hashes only,
no game content, so it is safe to version and it turns "matches retail" into
something a test can assert.

docs/plan/disc-pipeline.md records two wrong turns, because both are easy to
repeat and both failed by producing plausible wrong output rather than an
error: the container table was read backwards, and the LZ77 stream was modelled
as a bit stream. The first survived because it happened to be correct on the
one container used for manual checking.
This commit is contained in:
ReaperOfVeriod
2026-10-09 16:08:30 +02:00
parent a02f79a6d1
commit 027373e65b
21 changed files with 8372 additions and 97 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/gc-extract/src/unpack.rs
+222
View File
@@ -0,0 +1,222 @@
//! Stage 2: unpack the disc into asset files.
//!
//! R&C2 lists its assets in the ISO 9660 filesystem, unlike R&C1, so
//! enumeration is straightforward: every entry under `G/` is a container whose
//! header holds a table of sector ranges, and each range is written out as a
//! file.
//!
//! Output goes outside the repository. `AGENTS.md` explains why, and the
//! pre-commit hook enforces it.
use std::collections::BTreeMap;
use std::path::{Path, PathBuf};
use gc_formats::global_wad::{GlobalWad, RangeKind};
use gc_formats::iso9660::IsoImage;
/// One unpacked asset, with the identity we record for verification.
pub struct Extracted {
/// Path relative to the output directory, e.g. `armor/0000.bin`.
pub path: String,
pub size: u64,
pub sha1: String,
}
/// Unpack `iso_path` into `out_dir`.
///
/// Returns every file written, sorted by path, so the caller can build a hash
/// table or verify against one.
pub fn unpack(iso_path: &Path, out_dir: &Path) -> Result<Vec<Extracted>, String> {
let iso = IsoImage::open(iso_path).map_err(|e| format!("cannot read the disc: {e}"))?;
std::fs::create_dir_all(out_dir)
.map_err(|e| format!("cannot create {}: {e}", out_dir.display()))?;
let mut written: BTreeMap<String, Extracted> = BTreeMap::new();
let mut skipped: Vec<(String, String)> = Vec::new();
for entry in &iso.files {
if entry.is_dir {
continue;
}
// `Z6TAIL.DUP` is zero-length padding the ISO 9660 filesystem needs.
if entry.name.starts_with("Z6TAIL") {
continue;
}
let relative = entry.name.trim_end_matches('/').to_string();
if !relative.starts_with("G/") {
// Not a container: the boot ELF, SYSTEM.CNF, RC2.HDR and the IOP
// image are copied through verbatim.
let data = gc_formats::iso9660::read_at(iso_path, entry.lba, entry.size)
.map_err(|e| format!("cannot read {}: {e}", entry.name))?;
write_file(out_dir, &relative, &data, &mut written);
continue;
}
// A container: write out each range its header declares.
let raw = gc_formats::iso9660::read_at(iso_path, entry.lba, entry.size)
.map_err(|e| format!("cannot read {}: {e}", entry.name))?;
let stem = relative
.trim_start_matches("G/")
.trim_end_matches(".WAD")
.to_lowercase();
// MPEG1 and MPEG2 are raw concatenated video, not indexed containers:
// their first word is video payload, not a header size. Copy them
// through; the video phase splits them properly.
if stem.starts_with("mpeg1") || stem.starts_with("mpeg2") {
write_file(out_dir, &format!("{stem}.bin"), &raw, &mut written);
println!(
" {:<16} {} bytes, raw video stream (not indexed)",
stem,
raw.len()
);
continue;
}
let container = match GlobalWad::parse_with(&raw, RangeKind::for_container(&relative)) {
Ok(c) => c,
Err(e) => {
// Some containers have a header this generic parser does not
// model (the scene containers nest further). Copy the file
// through untouched rather than failing the whole run: the
// bytes are preserved and a typed parser can be added later.
skipped.push((relative.clone(), e.to_string()));
write_file(out_dir, &relative, &raw, &mut written);
continue;
}
};
let present = container.present().count();
let mut out_of_range = 0usize;
for (index, offset, byte_len) in container.present() {
let len = byte_len as usize;
// Bounds-check against the container we actually read, so a corrupt
// header cannot make us read past the end of the file.
if offset + len as u64 > raw.len() as u64 {
out_of_range += 1;
continue;
}
let data = &raw[offset as usize..offset as usize + len];
write_file(
out_dir,
&format!("{stem}/{index:04}.bin"),
data,
&mut written,
);
}
if out_of_range > 0 {
eprintln!(
" {:<16} {} bytes, {} entries -> {} present, {out_of_range} out of range (header describes \
sectors beyond the file; see docs/plan/disc-pipeline.md)",
stem, raw.len(), container.entries.len(), present
);
} else {
println!(
" {:<16} {} bytes, {} entries -> {} present",
stem,
raw.len(),
container.entries.len(),
present
);
}
}
for (name, why) in &skipped {
eprintln!(" {name}: not an indexed container ({why}); copied verbatim");
}
Ok(written.into_values().collect())
}
fn write_file(
out_dir: &Path,
relative: &str,
data: &[u8],
written: &mut BTreeMap<String, Extracted>,
) {
use crate::identify::sha1_hex;
let dest = out_dir.join(relative);
if let Some(parent) = dest.parent() {
if let Err(e) = std::fs::create_dir_all(parent) {
eprintln!("warning: cannot create {}: {e}", parent.display());
return;
}
}
if let Err(e) = std::fs::write(&dest, data) {
eprintln!("warning: cannot write {}: {e}", dest.display());
return;
}
written.insert(
relative.to_string(),
Extracted {
path: relative.to_string(),
size: data.len() as u64,
sha1: sha1_hex(data),
},
);
}
/// Default output directory: `extracted/` beside the repository root.
pub fn default_out_dir() -> PathBuf {
PathBuf::from("extracted")
}
#[cfg(test)]
mod tests {
use super::*;
/// A container with three entries, the second of which is absent.
fn container_bytes() -> Vec<u8> {
let header_size = 8 + 3 * 8;
let mut v = Vec::new();
v.extend_from_slice(&(header_size as u32).to_le_bytes());
v.extend_from_slice(&10u32.to_le_bytes());
v.extend_from_slice(&1i32.to_le_bytes());
v.extend_from_slice(&1i32.to_le_bytes());
v.extend_from_slice(&2i32.to_le_bytes());
v.extend_from_slice(&0i32.to_le_bytes()); // absent
v.extend_from_slice(&2i32.to_le_bytes());
v.extend_from_slice(&1i32.to_le_bytes());
v.resize(header_size + 3 * 2048, 0xAB);
v
}
#[test]
fn parses_a_container_and_lists_present_entries() {
let raw = container_bytes();
let c = GlobalWad::parse(&raw).unwrap();
assert_eq!(c.entries.len(), 3);
let present: Vec<usize> = c.present().map(|(i, _, _)| i).collect();
assert_eq!(present, vec![0, 2]);
// The default reading treats the second field as a byte count, so an
// entry of size 1 is one byte.
assert_eq!(c.present().next().unwrap().2, 1);
}
#[test]
fn every_declared_range_is_inside_the_container() {
let raw = container_bytes();
let c = GlobalWad::parse(&raw).unwrap();
for (_, offset, len) in c.present() {
assert!(
offset + len <= raw.len() as u64,
"range at {offset} runs past the end"
);
}
}
#[test]
fn a_range_past_the_end_would_be_caught() {
let mut raw = container_bytes();
// Push the first entry's offset far beyond the file.
raw[8..12].copy_from_slice(&9999i32.to_le_bytes());
let c = GlobalWad::parse(&raw).unwrap();
let (_, offset, len) = c.present().next().unwrap();
assert!(offset + len > raw.len() as u64);
}
}