Add the disc pipeline: identify, unpack and verify
Phase 1 of the roadmap. The disc is now turned into verified asset files, and
the result is checked against a committed hash table rather than against
whatever happens to be on disk.
gc-extract identify disc.iso SCUS_972.68 v1.01 NTSC, ELF SHA-1 matches
gc-extract unpack disc.iso out/ 5135 files, 4,500,106,960 bytes, 10.8s
gc-extract verify out/ table.tsv OK: 5135 files match
All 92 G/*.WAD containers parse. MPEG1 and MPEG2 are raw video and pass
through whole; the other 90 are indexed containers whose extents are extracted
individually.
Formats recovered and written up in docs/formats/:
- iso9660.rs. R&C2 lists its assets in the filesystem, unlike R&C1, so a
directory walk is the primary asset index. Three details that all fail
quietly rather than loudly: directory records store every 32-bit field twice
(little-endian then big-endian), the CD001 signature is at sector offset 1
rather than 0x8001, and the directory flag at record offset 25 is
authoritative where the identifier's trailing slash is not.
- global_wad.rs. Container headers hold a table of extents. All 92 indexed
containers use {offset in sectors, size in bytes}. Measured, not assumed: the
sector-count reading places every entry of all 27 LEVEL<n> and all 27
AUDIO<n> containers out of range, while the byte reading places all of them
correctly. SCENE<n> is the one exception, a flat sector list whose extents
are sized by the next value.
- wad.rs. Insomniac's LZ77 stream: four flag-selected packet classes, with 0-3
trailing literal bytes per packet carried in unused low bits. This is not a
bit stream, which is why it resists the usual LZ77 reading.
- buf.rs. A bounds-checked reader, so untrusted 2003 disc data produces typed
errors rather than panics.
gc-extract carries the three stages and a SHA-1 implementation with FIPS 180-4
test vectors, so identification works without pulling in a crypto dependency.
Verification: 54 unit tests, plus four integration tests that run only when
GC_DISC names a disc. Those four check identification against the recorded
values, cross-check the Rust identifier against tools/identify-disc.py, parse
all 92 containers, and confirm a full unpack matches the committed table. The
committed table is 316 KB of sizes and SHA-1s for all 5,135 files: hashes only,
no game content, so it is safe to version and it turns "matches retail" into
something a test can assert.
docs/plan/disc-pipeline.md records two wrong turns, because both are easy to
repeat and both failed by producing plausible wrong output rather than an
error: the container table was read backwards, and the LZ77 stream was modelled
as a bit stream. The first survived because it happened to be correct on the
one container used for manual checking.
This commit is contained in:
@@ -0,0 +1,215 @@
|
||||
//! `gc-extract` — the disc pipeline command line.
|
||||
//!
|
||||
//! Three subcommands, in the order they are normally used:
|
||||
//!
|
||||
//! ```text
|
||||
//! gc-extract identify <disc.iso>
|
||||
//! gc-extract unpack <disc.iso> <out-dir> [--table <table.tsv>]
|
||||
//! gc-extract verify <out-dir> <table.tsv>
|
||||
//! ```
|
||||
//!
|
||||
//! Only your own legally obtained disc is read. Nothing is written inside the
|
||||
//! repository.
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::process::ExitCode;
|
||||
|
||||
use gc_extract::{identify, unpack, verify};
|
||||
|
||||
fn main() -> ExitCode {
|
||||
let args: Vec<String> = std::env::args().skip(1).collect();
|
||||
match args.first().map(String::as_str) {
|
||||
Some("identify") => run_identify(&args[1..]),
|
||||
Some("unpack") => run_unpack(&args[1..]),
|
||||
Some("verify") => run_verify(&args[1..]),
|
||||
Some("--help") | Some("-h") | Some("help") | None => {
|
||||
usage();
|
||||
ExitCode::SUCCESS
|
||||
}
|
||||
Some(other) => {
|
||||
eprintln!("unknown subcommand `{other}`\n");
|
||||
usage();
|
||||
ExitCode::FAILURE
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Group digits for readability. Rust's format syntax has no thousands
|
||||
/// separator, so do it here rather than sprinkle the logic through the prints.
|
||||
fn thousands(n: u64) -> String {
|
||||
let s = n.to_string();
|
||||
let mut out = String::with_capacity(s.len() + s.len() / 3);
|
||||
for (i, c) in s.chars().enumerate() {
|
||||
if i > 0 && (s.len() - i) % 3 == 0 {
|
||||
out.push('_');
|
||||
}
|
||||
out.push(c);
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
fn usage() {
|
||||
println!(
|
||||
"gc-extract — Ratchet & Clank: Going Commando disc pipeline
|
||||
|
||||
identify <disc.iso>
|
||||
Report serial, version and video mode, and hash the boot ELF.
|
||||
|
||||
unpack <disc.iso> <out-dir> [--table <file.tsv>]
|
||||
Decompress every WAD on the disc into <out-dir>.
|
||||
With --table, also write a SHA-1 table (sizes and hashes only).
|
||||
|
||||
verify <out-dir> <table.tsv>
|
||||
Check <out-dir> against a previously written table.
|
||||
|
||||
You supply the disc. Nothing here is redistributed with the project."
|
||||
);
|
||||
}
|
||||
|
||||
fn run_identify(args: &[String]) -> ExitCode {
|
||||
if args.is_empty() {
|
||||
eprintln!("usage: gc-extract identify <disc.iso>");
|
||||
return ExitCode::FAILURE;
|
||||
}
|
||||
let path = Path::new(&args[0]);
|
||||
match identify(path) {
|
||||
Ok(info) => {
|
||||
println!("Disc identification");
|
||||
println!("{}", "=".repeat(64));
|
||||
println!(" file {}", info.path);
|
||||
println!(" size {} bytes", thousands(info.image_size));
|
||||
println!(" volume id {}", info.volume_id);
|
||||
println!(" serial {}", info.serial);
|
||||
println!(
|
||||
" version {}",
|
||||
if info.version.is_empty() {
|
||||
"(absent)"
|
||||
} else {
|
||||
&info.version
|
||||
}
|
||||
);
|
||||
println!(
|
||||
" video mode {}",
|
||||
if info.vmode.is_empty() {
|
||||
"(absent)"
|
||||
} else {
|
||||
&info.vmode
|
||||
}
|
||||
);
|
||||
println!();
|
||||
println!("Boot executable");
|
||||
println!("{}", "-".repeat(64));
|
||||
println!(
|
||||
" size {} bytes",
|
||||
thousands(info.boot_elf_size as u64)
|
||||
);
|
||||
println!(" SHA-1 {}", info.boot_elf_sha1);
|
||||
println!();
|
||||
println!("{}", "=".repeat(64));
|
||||
println!(" {}", info.verdict());
|
||||
ExitCode::SUCCESS
|
||||
}
|
||||
Err(e) => {
|
||||
eprintln!("error: {e}");
|
||||
ExitCode::FAILURE
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn run_unpack(args: &[String]) -> ExitCode {
|
||||
if args.len() < 2 {
|
||||
eprintln!("usage: gc-extract unpack <disc.iso> <out-dir> [--table <file.tsv>]");
|
||||
return ExitCode::FAILURE;
|
||||
}
|
||||
let iso = PathBuf::from(&args[0]);
|
||||
let out = PathBuf::from(&args[1]);
|
||||
let mut table_path: Option<PathBuf> = None;
|
||||
let mut i = 2;
|
||||
while i < args.len() {
|
||||
match args[i].as_str() {
|
||||
"--table" => {
|
||||
i += 1;
|
||||
match args.get(i) {
|
||||
Some(p) => table_path = Some(PathBuf::from(p)),
|
||||
None => {
|
||||
eprintln!("--table needs a path");
|
||||
return ExitCode::FAILURE;
|
||||
}
|
||||
}
|
||||
}
|
||||
other => {
|
||||
eprintln!("unexpected argument `{other}`");
|
||||
return ExitCode::FAILURE;
|
||||
}
|
||||
}
|
||||
i += 1;
|
||||
}
|
||||
|
||||
let ident = match identify::identify(&iso) {
|
||||
Ok(i) => i,
|
||||
Err(e) => {
|
||||
eprintln!("error: {e}");
|
||||
return ExitCode::FAILURE;
|
||||
}
|
||||
};
|
||||
println!("Disc: {} {} ({})", ident.serial, ident.version, ident.vmode);
|
||||
|
||||
let started = std::time::Instant::now();
|
||||
println!("Unpacking into {} ...", out.display());
|
||||
match unpack::unpack(&iso, &out) {
|
||||
Ok(files) => {
|
||||
let total: u64 = files.iter().map(|f| f.size).sum();
|
||||
println!(
|
||||
" {} files, {} bytes, in {:.1}s",
|
||||
files.len(),
|
||||
thousands(total),
|
||||
started.elapsed().as_secs_f64()
|
||||
);
|
||||
|
||||
if let Some(tp) = table_path {
|
||||
let entries: Vec<(String, u64, String)> = files
|
||||
.iter()
|
||||
.map(|f| (f.path.clone(), f.size, f.sha1.clone()))
|
||||
.collect();
|
||||
let text = verify::render_table(&entries);
|
||||
if let Err(e) = std::fs::write(&tp, text) {
|
||||
eprintln!("error: cannot write {}: {e}", tp.display());
|
||||
return ExitCode::FAILURE;
|
||||
}
|
||||
println!(" table written to {}", tp.display());
|
||||
}
|
||||
ExitCode::SUCCESS
|
||||
}
|
||||
Err(e) => {
|
||||
eprintln!("error: {e}");
|
||||
ExitCode::FAILURE
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn run_verify(args: &[String]) -> ExitCode {
|
||||
if args.len() < 2 {
|
||||
eprintln!("usage: gc-extract verify <out-dir> <table.tsv>");
|
||||
return ExitCode::FAILURE;
|
||||
}
|
||||
let dir = PathBuf::from(&args[0]);
|
||||
let table = match std::fs::read_to_string(&args[1]) {
|
||||
Ok(t) => t,
|
||||
Err(e) => {
|
||||
eprintln!("error: cannot read {}: {e}", args[1]);
|
||||
return ExitCode::FAILURE;
|
||||
}
|
||||
};
|
||||
let expected = verify::parse_table(&table);
|
||||
if expected.is_empty() {
|
||||
eprintln!("error: {} contained no usable rows", args[1]);
|
||||
return ExitCode::FAILURE;
|
||||
}
|
||||
let report = verify::verify(&dir, &expected);
|
||||
println!("{}", report.summary());
|
||||
if report.ok() {
|
||||
ExitCode::SUCCESS
|
||||
} else {
|
||||
ExitCode::FAILURE
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user