Add the disc pipeline: identify, unpack and verify
Phase 1 of the roadmap. The disc is now turned into verified asset files, and
the result is checked against a committed hash table rather than against
whatever happens to be on disk.
gc-extract identify disc.iso SCUS_972.68 v1.01 NTSC, ELF SHA-1 matches
gc-extract unpack disc.iso out/ 5135 files, 4,500,106,960 bytes, 10.8s
gc-extract verify out/ table.tsv OK: 5135 files match
All 92 G/*.WAD containers parse. MPEG1 and MPEG2 are raw video and pass
through whole; the other 90 are indexed containers whose extents are extracted
individually.
Formats recovered and written up in docs/formats/:
- iso9660.rs. R&C2 lists its assets in the filesystem, unlike R&C1, so a
directory walk is the primary asset index. Three details that all fail
quietly rather than loudly: directory records store every 32-bit field twice
(little-endian then big-endian), the CD001 signature is at sector offset 1
rather than 0x8001, and the directory flag at record offset 25 is
authoritative where the identifier's trailing slash is not.
- global_wad.rs. Container headers hold a table of extents. All 92 indexed
containers use {offset in sectors, size in bytes}. Measured, not assumed: the
sector-count reading places every entry of all 27 LEVEL<n> and all 27
AUDIO<n> containers out of range, while the byte reading places all of them
correctly. SCENE<n> is the one exception, a flat sector list whose extents
are sized by the next value.
- wad.rs. Insomniac's LZ77 stream: four flag-selected packet classes, with 0-3
trailing literal bytes per packet carried in unused low bits. This is not a
bit stream, which is why it resists the usual LZ77 reading.
- buf.rs. A bounds-checked reader, so untrusted 2003 disc data produces typed
errors rather than panics.
gc-extract carries the three stages and a SHA-1 implementation with FIPS 180-4
test vectors, so identification works without pulling in a crypto dependency.
Verification: 54 unit tests, plus four integration tests that run only when
GC_DISC names a disc. Those four check identification against the recorded
values, cross-check the Rust identifier against tools/identify-disc.py, parse
all 92 containers, and confirm a full unpack matches the committed table. The
committed table is 316 KB of sizes and SHA-1s for all 5,135 files: hashes only,
no game content, so it is safe to version and it turns "matches retail" into
something a test can assert.
docs/plan/disc-pipeline.md records two wrong turns, because both are easy to
repeat and both failed by producing plausible wrong output rather than an
error: the container table was read backwards, and the LZ77 stream was modelled
as a bit stream. The first survived because it happened to be correct on the
one container used for manual checking.
This commit is contained in:
@@ -0,0 +1,388 @@
|
||||
//! Stage 1: identify the disc.
|
||||
//!
|
||||
//! Reads only the ISO 9660 structures, `SYSTEM.CNF` and the boot ELF — a few
|
||||
//! megabytes of seeks — so it is near-instant on a local image.
|
||||
//!
|
||||
//! The reference implementation of this stage is `tools/identify-disc.py`; the
|
||||
//! two are checked against each other in `tests/disc.rs`.
|
||||
|
||||
use std::path::Path;
|
||||
|
||||
use gc_formats::iso9660::IsoImage;
|
||||
|
||||
/// The build this project targets.
|
||||
pub const TARGET_SERIAL: &str = "SCUS_972.68";
|
||||
pub const TARGET_VERSION: &str = "1.01";
|
||||
/// Measured from a retail Black Label v1.01 image. See `docs/disc-identification.md`.
|
||||
pub const TARGET_BOOT_ELF_SHA1: &str = "bacda860cbd6723142e86db56f197e47e728970f";
|
||||
|
||||
/// What we learned about a disc.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct DiscInfo {
|
||||
pub path: String,
|
||||
pub image_size: u64,
|
||||
pub volume_id: String,
|
||||
/// Product serial, i.e. the boot executable's file name.
|
||||
pub serial: String,
|
||||
/// `VER` from `SYSTEM.CNF`.
|
||||
pub version: String,
|
||||
/// `VMODE` from `SYSTEM.CNF`: `NTSC` or `PAL`, possibly empty.
|
||||
pub vmode: String,
|
||||
pub boot_elf_size: u32,
|
||||
pub boot_elf_sha1: String,
|
||||
}
|
||||
|
||||
impl DiscInfo {
|
||||
/// Is this the exact build the reference material describes?
|
||||
pub fn is_target(&self) -> bool {
|
||||
self.serial == TARGET_SERIAL && self.version == TARGET_VERSION
|
||||
}
|
||||
|
||||
/// True when the video mode makes the game run at 50 Hz.
|
||||
///
|
||||
/// This matters more than a label difference: a PAL disc runs the game at
|
||||
/// 50 Hz, which puts speed-sensitive movement such as the chargeboots at
|
||||
/// 5/6 of their intended rate.
|
||||
pub fn is_pal(&self) -> bool {
|
||||
self.vmode.eq_ignore_ascii_case("PAL")
|
||||
}
|
||||
|
||||
/// A short human-readable summary of whether this disc can be used.
|
||||
///
|
||||
/// The video mode is reported for every disc, not just the target: a 50 Hz
|
||||
/// PAL disc changes movement timing, which is the kind of detail that is
|
||||
/// easy to forget and hard to notice later.
|
||||
pub fn verdict(&self) -> String {
|
||||
let rate = if self.is_pal() {
|
||||
" 50 Hz: speed-sensitive movement runs at 5/6 speed, so timing constants differ."
|
||||
} else if self.vmode.is_empty() {
|
||||
""
|
||||
} else {
|
||||
" 60 Hz."
|
||||
};
|
||||
|
||||
if self.is_target() && self.boot_elf_sha1 == TARGET_BOOT_ELF_SHA1 {
|
||||
return format!(
|
||||
"USA Black Label v1.01, NTSC 60 Hz: the build we target. Boot ELF SHA-1 matches.{rate}"
|
||||
);
|
||||
}
|
||||
if self.serial == "SCUS_972.68GH" {
|
||||
return format!(
|
||||
"Greatest Hits v2.00. This is a different build: the chargeboots were patched, \
|
||||
and reference material built for v1.01 will not match.{rate}"
|
||||
);
|
||||
}
|
||||
if self.serial == TARGET_SERIAL {
|
||||
return format!(
|
||||
"Serial matches but version is {} (we target {TARGET_VERSION}).{rate}",
|
||||
if self.version.is_empty() {
|
||||
"unknown"
|
||||
} else {
|
||||
&self.version
|
||||
}
|
||||
);
|
||||
}
|
||||
format!("Not the target build. We target {TARGET_SERIAL} v{TARGET_VERSION}.{rate}")
|
||||
}
|
||||
}
|
||||
|
||||
/// Extract `KEY = value` from `SYSTEM.CNF`, stopping at `;`, CR or LF.
|
||||
fn cnf_value(cnf: &str, key: &str) -> String {
|
||||
for line in cnf.lines() {
|
||||
if let Some((k, v)) = line.split_once('=') {
|
||||
if k.trim() == key {
|
||||
return v.trim().split(';').next().unwrap_or("").trim().to_string();
|
||||
}
|
||||
}
|
||||
}
|
||||
String::new()
|
||||
}
|
||||
|
||||
/// Identify a disc image, or explain why it cannot be used.
|
||||
pub fn identify(path: &Path) -> Result<DiscInfo, String> {
|
||||
let iso = IsoImage::open(path).map_err(|e| format!("cannot read {}: {e}", path.display()))?;
|
||||
|
||||
let cnf_entry = iso
|
||||
.find("SYSTEM.CNF")
|
||||
.ok_or_else(|| "no SYSTEM.CNF: not a PlayStation 2 game disc".to_string())?;
|
||||
let cnf = gc_formats::iso9660::read_at(path, cnf_entry.lba, cnf_entry.size)
|
||||
.map_err(|e| format!("cannot read SYSTEM.CNF: {e}"))?;
|
||||
let cnf = String::from_utf8_lossy(&cnf);
|
||||
|
||||
let boot = cnf_value(&cnf, "BOOT2");
|
||||
if boot.is_empty() {
|
||||
return Err("SYSTEM.CNF has no BOOT2 line: not a PlayStation 2 game disc".into());
|
||||
}
|
||||
let boot_path = boot
|
||||
.strip_prefix("cdrom0:")
|
||||
.unwrap_or(&boot)
|
||||
.replace('\\', "/")
|
||||
// SYSTEM.CNF writes the boot path rooted, e.g. `cdrom0:\SCUS_972.68`.
|
||||
// Strip any leading slash so it matches an ISO directory name.
|
||||
.trim_start_matches('/')
|
||||
.to_string();
|
||||
let serial = boot_path.rsplit('/').next().unwrap_or("").to_string();
|
||||
|
||||
let entry = iso
|
||||
.find(&boot_path)
|
||||
.ok_or_else(|| format!("SYSTEM.CNF boots {boot_path}, which is not on the disc"))?;
|
||||
let elf = gc_formats::iso9660::read_at(path, entry.lba, entry.size)
|
||||
.map_err(|e| format!("cannot read the boot executable {serial}: {e}"))?;
|
||||
|
||||
if elf.len() < 52 || &elf[..4] != b"\x7fELF" {
|
||||
return Err(format!("{serial} is not an ELF executable"));
|
||||
}
|
||||
|
||||
Ok(DiscInfo {
|
||||
path: path.display().to_string(),
|
||||
image_size: std::fs::metadata(path).map(|m| m.len()).unwrap_or(0),
|
||||
volume_id: iso.volume_id.clone(),
|
||||
serial,
|
||||
version: cnf_value(&cnf, "VER"),
|
||||
vmode: cnf_value(&cnf, "VMODE"),
|
||||
boot_elf_size: entry.size,
|
||||
boot_elf_sha1: sha1_hex(&elf),
|
||||
})
|
||||
}
|
||||
|
||||
/// SHA-1 of `data`, lowercase hex.
|
||||
///
|
||||
/// Implemented here rather than pulled in as a dependency so that `identify`
|
||||
/// stays usable as a standalone stage with a minimal dependency set.
|
||||
pub fn sha1_hex(data: &[u8]) -> String {
|
||||
let mut h = Sha1::new();
|
||||
h.update(data);
|
||||
h.hex()
|
||||
}
|
||||
|
||||
/// Minimal SHA-1 (FIPS 180-4).
|
||||
pub struct Sha1 {
|
||||
state: [u32; 5],
|
||||
buffer: [u8; 64],
|
||||
buffered: usize,
|
||||
length: u64,
|
||||
}
|
||||
|
||||
impl Default for Sha1 {
|
||||
fn default() -> Self {
|
||||
Self::new()
|
||||
}
|
||||
}
|
||||
|
||||
impl Sha1 {
|
||||
pub fn new() -> Self {
|
||||
Self {
|
||||
state: [
|
||||
0x6745_2301,
|
||||
0xEFCD_AB89,
|
||||
0x98BA_DCFE,
|
||||
0x1032_5476,
|
||||
0xC3D2_E1F0,
|
||||
],
|
||||
buffer: [0u8; 64],
|
||||
buffered: 0,
|
||||
length: 0,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn update(&mut self, mut data: &[u8]) {
|
||||
self.length = self.length.wrapping_add(data.len() as u64);
|
||||
if self.buffered > 0 {
|
||||
let take = (64 - self.buffered).min(data.len());
|
||||
self.buffer[self.buffered..self.buffered + take].copy_from_slice(&data[..take]);
|
||||
self.buffered += take;
|
||||
data = &data[take..];
|
||||
if self.buffered == 64 {
|
||||
let block = self.buffer;
|
||||
self.compress(&block);
|
||||
self.buffered = 0;
|
||||
}
|
||||
}
|
||||
while data.len() >= 64 {
|
||||
let mut block = [0u8; 64];
|
||||
block.copy_from_slice(&data[..64]);
|
||||
self.compress(&block);
|
||||
data = &data[64..];
|
||||
}
|
||||
if !data.is_empty() {
|
||||
self.buffer[..data.len()].copy_from_slice(data);
|
||||
self.buffered = data.len();
|
||||
}
|
||||
}
|
||||
|
||||
fn compress(&mut self, block: &[u8; 64]) {
|
||||
let mut w = [0u32; 80];
|
||||
for i in 0..16 {
|
||||
w[i] = u32::from_be_bytes([
|
||||
block[i * 4],
|
||||
block[i * 4 + 1],
|
||||
block[i * 4 + 2],
|
||||
block[i * 4 + 3],
|
||||
]);
|
||||
}
|
||||
for i in 16..80 {
|
||||
w[i] = (w[i - 3] ^ w[i - 8] ^ w[i - 14] ^ w[i - 16]).rotate_left(1);
|
||||
}
|
||||
let (mut a, mut b, mut c, mut d, mut e) = (
|
||||
self.state[0],
|
||||
self.state[1],
|
||||
self.state[2],
|
||||
self.state[3],
|
||||
self.state[4],
|
||||
);
|
||||
for (i, &wi) in w.iter().enumerate() {
|
||||
let (f, k) = match i {
|
||||
0..=19 => ((b & c) | ((!b) & d), 0x5A82_7999u32),
|
||||
20..=39 => (b ^ c ^ d, 0x6ED9_EBA1),
|
||||
40..=59 => ((b & c) | (b & d) | (c & d), 0x8F1B_BCDC),
|
||||
_ => (b ^ c ^ d, 0xCA62_C1D6),
|
||||
};
|
||||
let temp = a
|
||||
.rotate_left(5)
|
||||
.wrapping_add(f)
|
||||
.wrapping_add(e)
|
||||
.wrapping_add(k)
|
||||
.wrapping_add(wi);
|
||||
e = d;
|
||||
d = c;
|
||||
c = b.rotate_left(30);
|
||||
b = a;
|
||||
a = temp;
|
||||
}
|
||||
self.state[0] = self.state[0].wrapping_add(a);
|
||||
self.state[1] = self.state[1].wrapping_add(b);
|
||||
self.state[2] = self.state[2].wrapping_add(c);
|
||||
self.state[3] = self.state[3].wrapping_add(d);
|
||||
self.state[4] = self.state[4].wrapping_add(e);
|
||||
}
|
||||
|
||||
pub fn hex(mut self) -> String {
|
||||
let bit_len = self.length.wrapping_mul(8);
|
||||
self.update_raw(&[0x80]);
|
||||
while self.buffered != 56 {
|
||||
self.update_raw(&[0x00]);
|
||||
}
|
||||
let len_bytes = bit_len.to_be_bytes();
|
||||
self.update_raw(&len_bytes);
|
||||
let mut out = String::with_capacity(40);
|
||||
for word in self.state {
|
||||
out.push_str(&format!("{word:08x}"));
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// Feed padding without changing the recorded message length.
|
||||
fn update_raw(&mut self, data: &[u8]) {
|
||||
for &byte in data {
|
||||
self.buffer[self.buffered] = byte;
|
||||
self.buffered += 1;
|
||||
if self.buffered == 64 {
|
||||
let block = self.buffer;
|
||||
self.compress(&block);
|
||||
self.buffered = 0;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn sha1_matches_known_vectors() {
|
||||
assert_eq!(sha1_hex(b""), "da39a3ee5e6b4b0d3255bfef95601890afd80709");
|
||||
assert_eq!(sha1_hex(b"abc"), "a9993e364706816aba3e25717850c26c9cd0d89d");
|
||||
assert_eq!(
|
||||
sha1_hex(b"abcdbcdecdefdefgefghfghighijhijkijkljklmklmnlmnomnopnopq"),
|
||||
"84983e441c3bd26ebaae4aa1f95129e5e54670f1"
|
||||
);
|
||||
// Exactly one block.
|
||||
assert_eq!(
|
||||
sha1_hex(&[b'a'; 64]),
|
||||
"0098ba824b5c16427bd7a1122a5a442a25ec644d"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sha1_handles_multi_block_input() {
|
||||
let million: Vec<u8> = std::iter::repeat_n(b'a', 1_000_000).collect();
|
||||
assert_eq!(
|
||||
sha1_hex(&million),
|
||||
"34aa973cd4c4daa4f61eeb2bdbad27316534016f"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cnf_parsing() {
|
||||
let cnf = "BOOT2 = cdrom0:\\SCUS_972.68;1\r\nVER = 1.01\r\nVMODE = NTSC\r\n\r\n";
|
||||
assert_eq!(cnf_value(cnf, "BOOT2"), "cdrom0:\\SCUS_972.68");
|
||||
assert_eq!(cnf_value(cnf, "VER"), "1.01");
|
||||
assert_eq!(cnf_value(cnf, "VMODE"), "NTSC");
|
||||
assert_eq!(cnf_value(cnf, "MISSING"), "");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn ntsc_verdict_mentions_sixty_hertz() {
|
||||
let d = DiscInfo {
|
||||
path: "x.iso".into(),
|
||||
image_size: 0,
|
||||
volume_id: String::new(),
|
||||
serial: TARGET_SERIAL.into(),
|
||||
version: TARGET_VERSION.into(),
|
||||
vmode: "NTSC".into(),
|
||||
boot_elf_size: 0,
|
||||
boot_elf_sha1: TARGET_BOOT_ELF_SHA1.into(),
|
||||
};
|
||||
assert!(d.verdict().contains("60 Hz"), "{}", d.verdict());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn target_verdict_for_the_supported_build() {
|
||||
let d = DiscInfo {
|
||||
path: "x.iso".into(),
|
||||
image_size: 0,
|
||||
volume_id: "RATCHETANDCLANK2".into(),
|
||||
serial: TARGET_SERIAL.into(),
|
||||
version: TARGET_VERSION.into(),
|
||||
vmode: "NTSC".into(),
|
||||
boot_elf_size: 0,
|
||||
boot_elf_sha1: TARGET_BOOT_ELF_SHA1.into(),
|
||||
};
|
||||
assert!(d.is_target());
|
||||
assert!(!d.is_pal());
|
||||
assert!(d.verdict().contains("the build we target"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn greatest_hits_is_called_out_as_a_different_build() {
|
||||
let d = DiscInfo {
|
||||
path: "x.iso".into(),
|
||||
image_size: 0,
|
||||
volume_id: String::new(),
|
||||
serial: "SCUS_972.68GH".into(),
|
||||
version: "2.00".into(),
|
||||
vmode: "NTSC".into(),
|
||||
boot_elf_size: 0,
|
||||
boot_elf_sha1: String::new(),
|
||||
};
|
||||
assert!(!d.is_target());
|
||||
assert!(d.verdict().contains("different build"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pal_is_flagged_as_fifty_hertz() {
|
||||
let d = DiscInfo {
|
||||
path: "x.iso".into(),
|
||||
image_size: 0,
|
||||
volume_id: String::new(),
|
||||
serial: "SCES-516.07".into(),
|
||||
version: "1.00".into(),
|
||||
vmode: "PAL".into(),
|
||||
boot_elf_size: 0,
|
||||
boot_elf_sha1: String::new(),
|
||||
};
|
||||
assert!(d.is_pal());
|
||||
assert!(d.verdict().contains("50 Hz"), "{}", d.verdict());
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user