Add the disc pipeline: identify, unpack and verify
Phase 1 of the roadmap. The disc is now turned into verified asset files, and
the result is checked against a committed hash table rather than against
whatever happens to be on disk.
gc-extract identify disc.iso SCUS_972.68 v1.01 NTSC, ELF SHA-1 matches
gc-extract unpack disc.iso out/ 5135 files, 4,500,106,960 bytes, 10.8s
gc-extract verify out/ table.tsv OK: 5135 files match
All 92 G/*.WAD containers parse. MPEG1 and MPEG2 are raw video and pass
through whole; the other 90 are indexed containers whose extents are extracted
individually.
Formats recovered and written up in docs/formats/:
- iso9660.rs. R&C2 lists its assets in the filesystem, unlike R&C1, so a
directory walk is the primary asset index. Three details that all fail
quietly rather than loudly: directory records store every 32-bit field twice
(little-endian then big-endian), the CD001 signature is at sector offset 1
rather than 0x8001, and the directory flag at record offset 25 is
authoritative where the identifier's trailing slash is not.
- global_wad.rs. Container headers hold a table of extents. All 92 indexed
containers use {offset in sectors, size in bytes}. Measured, not assumed: the
sector-count reading places every entry of all 27 LEVEL<n> and all 27
AUDIO<n> containers out of range, while the byte reading places all of them
correctly. SCENE<n> is the one exception, a flat sector list whose extents
are sized by the next value.
- wad.rs. Insomniac's LZ77 stream: four flag-selected packet classes, with 0-3
trailing literal bytes per packet carried in unused low bits. This is not a
bit stream, which is why it resists the usual LZ77 reading.
- buf.rs. A bounds-checked reader, so untrusted 2003 disc data produces typed
errors rather than panics.
gc-extract carries the three stages and a SHA-1 implementation with FIPS 180-4
test vectors, so identification works without pulling in a crypto dependency.
Verification: 54 unit tests, plus four integration tests that run only when
GC_DISC names a disc. Those four check identification against the recorded
values, cross-check the Rust identifier against tools/identify-disc.py, parse
all 92 containers, and confirm a full unpack matches the committed table. The
committed table is 316 KB of sizes and SHA-1s for all 5,135 files: hashes only,
no game content, so it is safe to version and it turns "matches retail" into
something a test can assert.
docs/plan/disc-pipeline.md records two wrong turns, because both are easy to
repeat and both failed by producing plausible wrong output rather than an
error: the container table was read backwards, and the LZ77 stream was modelled
as a bit stream. The first survived because it happened to be correct on the
one container used for manual checking.
This commit is contained in:
@@ -0,0 +1,184 @@
|
||||
//! Integration tests against a real disc image.
|
||||
//!
|
||||
//! These are skipped unless `GC_DISC` points at a legally obtained disc, since
|
||||
//! the repository must not contain game data. When it is set they check the
|
||||
//! identification and unpack results against the committed hash table.
|
||||
//!
|
||||
//! ```sh
|
||||
//! GC_DISC="path/to/disc.iso" cargo test -p gc-extract --test disc
|
||||
//! ```
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
/// The disc under test, if the environment names one.
|
||||
fn disc() -> Option<PathBuf> {
|
||||
let p = std::env::var_os("GC_DISC")?;
|
||||
let p = PathBuf::from(p);
|
||||
p.is_file().then_some(p)
|
||||
}
|
||||
|
||||
/// Skip a test with a clear reason rather than failing.
|
||||
macro_rules! require_disc {
|
||||
() => {
|
||||
match disc() {
|
||||
Some(p) => p,
|
||||
None => {
|
||||
eprintln!("skipped: set GC_DISC to a Going Commando disc image to run this");
|
||||
return;
|
||||
}
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn identifies_the_supported_build() {
|
||||
let path = require_disc!();
|
||||
let info = gc_extract::identify::identify(&path).expect("disc should identify");
|
||||
|
||||
assert_eq!(info.serial, "SCUS_972.68");
|
||||
assert_eq!(info.volume_id, "RATCHETANDCLANK2");
|
||||
assert!(
|
||||
info.is_target(),
|
||||
"expected the supported build, got: {}",
|
||||
info.verdict()
|
||||
);
|
||||
assert_eq!(
|
||||
info.boot_elf_sha1,
|
||||
gc_extract::identify::TARGET_BOOT_ELF_SHA1,
|
||||
"boot ELF SHA-1 differs from the recorded retail value"
|
||||
);
|
||||
assert_eq!(info.boot_elf_size, 2_618_684);
|
||||
// This disc is NTSC; the movement constants assume 60 Hz.
|
||||
assert!(!info.is_pal(), "test disc should be NTSC");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn agrees_with_the_python_reference_identifier() {
|
||||
let path = require_disc!();
|
||||
let info = gc_extract::identify::identify(&path).expect("disc should identify");
|
||||
|
||||
// tools/identify-disc.py is the reference implementation and prints these
|
||||
// two values. If they ever disagree, one of them has a bug.
|
||||
// Run from the workspace root so the relative path to the tool resolves.
|
||||
let root = std::path::Path::new(env!("CARGO_MANIFEST_DIR"))
|
||||
.parent()
|
||||
.and_then(|p| p.parent())
|
||||
.expect("the crate should live two levels below the workspace root")
|
||||
.to_path_buf();
|
||||
let py = std::process::Command::new("python3")
|
||||
.arg(root.join("tools/identify-disc.py"))
|
||||
.arg(&path)
|
||||
.current_dir(&root)
|
||||
.output()
|
||||
.expect("python3 should be available to run the reference identifier");
|
||||
assert!(
|
||||
py.status.success(),
|
||||
"the reference identifier failed:\n{}",
|
||||
String::from_utf8_lossy(&py.stderr)
|
||||
);
|
||||
let text = String::from_utf8_lossy(&py.stdout);
|
||||
|
||||
assert!(
|
||||
text.contains(&format!("serial {}", info.serial)),
|
||||
"serial disagrees with the reference identifier:\n{text}"
|
||||
);
|
||||
assert!(
|
||||
text.contains(&format!("SHA-1 {}", info.boot_elf_sha1)),
|
||||
"boot ELF SHA-1 disagrees with the reference identifier:\n{text}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unpacks_and_matches_the_committed_table() {
|
||||
let path = require_disc!();
|
||||
|
||||
let table_path = Path::new(env!("CARGO_MANIFEST_DIR")).join("data/gc-files.tsv");
|
||||
let table_text =
|
||||
std::fs::read_to_string(&table_path).expect("the hash table should be committed");
|
||||
let expected = gc_extract::verify::parse_table(&table_text);
|
||||
assert!(
|
||||
expected.len() > 5000,
|
||||
"expected a full table, got {} rows",
|
||||
expected.len()
|
||||
);
|
||||
|
||||
// Unpack into a temporary directory so the test leaves nothing behind.
|
||||
let tmp = std::env::temp_dir().join(format!("gc-unpack-{}", std::process::id()));
|
||||
let _ = std::fs::remove_dir_all(&tmp);
|
||||
std::fs::create_dir_all(&tmp).expect("should create a temp directory");
|
||||
|
||||
let files = gc_extract::unpack::unpack(&path, &tmp).expect("unpack should succeed");
|
||||
|
||||
// Every unpacked file must match the recorded size and hash.
|
||||
let report = gc_extract::verify::verify(&tmp, &expected);
|
||||
let _ = std::fs::remove_dir_all(&tmp);
|
||||
|
||||
assert!(
|
||||
report.ok(),
|
||||
"unpacked data does not match the committed table:\n{}",
|
||||
report.summary()
|
||||
);
|
||||
assert_eq!(
|
||||
files.len(),
|
||||
expected.len(),
|
||||
"unpack produced a different file count"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn every_container_parses() {
|
||||
let path = require_disc!();
|
||||
|
||||
let iso = gc_formats::iso9660::IsoImage::open(&path).expect("ISO should open");
|
||||
let containers: Vec<_> = iso
|
||||
.files
|
||||
.iter()
|
||||
.filter(|e| e.name.starts_with("G/") && e.name.ends_with(".WAD"))
|
||||
.collect();
|
||||
|
||||
// 92 `G/*.WAD` containers: 27 LEVEL, 27 AUDIO, 27 SCENE, 11 globals.
|
||||
assert_eq!(containers.len(), 92, "expected 92 asset containers");
|
||||
|
||||
// MPEG1 and MPEG2 are raw video streams with no table; the other 90 must
|
||||
// all parse as indexed containers.
|
||||
let raw_video: Vec<&str> = containers
|
||||
.iter()
|
||||
.map(|c| c.name.as_str())
|
||||
.filter(|n| *n == "G/MPEG1.WAD" || *n == "G/MPEG2.WAD")
|
||||
.collect();
|
||||
assert_eq!(
|
||||
raw_video.len(),
|
||||
2,
|
||||
"expected MPEG1 and MPEG2 to be the raw streams"
|
||||
);
|
||||
|
||||
// The per-level families must all be present, at 27 each.
|
||||
for family in ["LEVEL", "AUDIO", "SCENE"] {
|
||||
let n = containers
|
||||
.iter()
|
||||
.filter(|c| {
|
||||
let stem = c.name.trim_start_matches("G/").trim_end_matches(".WAD");
|
||||
stem.strip_prefix(family)
|
||||
.map(|rest| !rest.is_empty() && rest.chars().all(|ch| ch.is_ascii_digit()))
|
||||
.unwrap_or(false)
|
||||
})
|
||||
.count();
|
||||
assert_eq!(n, 27, "expected 27 {family}<n>.WAD containers, found {n}");
|
||||
}
|
||||
|
||||
for entry in &containers {
|
||||
if raw_video.contains(&entry.name.as_str()) {
|
||||
continue;
|
||||
}
|
||||
let data = gc_formats::iso9660::read_at(&path, entry.lba, entry.size)
|
||||
.expect("read should succeed");
|
||||
let kind = gc_formats::global_wad::RangeKind::for_container(&entry.name);
|
||||
let container = gc_formats::global_wad::GlobalWad::parse_with(&data, kind)
|
||||
.unwrap_or_else(|e| panic!("{} failed to parse: {e}", entry.name));
|
||||
assert!(
|
||||
!container.entries.is_empty(),
|
||||
"{} parsed but had no entries",
|
||||
entry.name
|
||||
);
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user