Add the disc pipeline: identify, unpack and verify
Phase 1 of the roadmap. The disc is now turned into verified asset files, and
the result is checked against a committed hash table rather than against
whatever happens to be on disk.
gc-extract identify disc.iso SCUS_972.68 v1.01 NTSC, ELF SHA-1 matches
gc-extract unpack disc.iso out/ 5135 files, 4,500,106,960 bytes, 10.8s
gc-extract verify out/ table.tsv OK: 5135 files match
All 92 G/*.WAD containers parse. MPEG1 and MPEG2 are raw video and pass
through whole; the other 90 are indexed containers whose extents are extracted
individually.
Formats recovered and written up in docs/formats/:
- iso9660.rs. R&C2 lists its assets in the filesystem, unlike R&C1, so a
directory walk is the primary asset index. Three details that all fail
quietly rather than loudly: directory records store every 32-bit field twice
(little-endian then big-endian), the CD001 signature is at sector offset 1
rather than 0x8001, and the directory flag at record offset 25 is
authoritative where the identifier's trailing slash is not.
- global_wad.rs. Container headers hold a table of extents. All 92 indexed
containers use {offset in sectors, size in bytes}. Measured, not assumed: the
sector-count reading places every entry of all 27 LEVEL<n> and all 27
AUDIO<n> containers out of range, while the byte reading places all of them
correctly. SCENE<n> is the one exception, a flat sector list whose extents
are sized by the next value.
- wad.rs. Insomniac's LZ77 stream: four flag-selected packet classes, with 0-3
trailing literal bytes per packet carried in unused low bits. This is not a
bit stream, which is why it resists the usual LZ77 reading.
- buf.rs. A bounds-checked reader, so untrusted 2003 disc data produces typed
errors rather than panics.
gc-extract carries the three stages and a SHA-1 implementation with FIPS 180-4
test vectors, so identification works without pulling in a crypto dependency.
Verification: 54 unit tests, plus four integration tests that run only when
GC_DISC names a disc. Those four check identification against the recorded
values, cross-check the Rust identifier against tools/identify-disc.py, parse
all 92 containers, and confirm a full unpack matches the committed table. The
committed table is 316 KB of sizes and SHA-1s for all 5,135 files: hashes only,
no game content, so it is safe to version and it turns "matches retail" into
something a test can assert.
docs/plan/disc-pipeline.md records two wrong turns, because both are easy to
repeat and both failed by producing plausible wrong output rather than an
error: the container table was read backwards, and the LZ77 stream was modelled
as a bit stream. The first survived because it happened to be correct on the
one container used for manual checking.
This commit is contained in:
@@ -32,4 +32,4 @@
|
||||
//! The port is validated against PCSX2 by replaying recorded input and
|
||||
//! comparing frames. Without that loop, "looks close" and "is correct" drift
|
||||
//! apart, and this game's difficulty and movement make that drift very
|
||||
//! noticeable.
|
||||
//! noticeable.
|
||||
|
||||
@@ -12,3 +12,10 @@ anyhow.workspace = true
|
||||
thiserror.workspace = true
|
||||
gc-formats.workspace = true
|
||||
sha1.workspace = true
|
||||
|
||||
[[bin]]
|
||||
name = "gc-extract"
|
||||
path = "src/main.rs"
|
||||
|
||||
[dev-dependencies]
|
||||
gc-formats.workspace = true
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,388 @@
|
||||
//! Stage 1: identify the disc.
|
||||
//!
|
||||
//! Reads only the ISO 9660 structures, `SYSTEM.CNF` and the boot ELF — a few
|
||||
//! megabytes of seeks — so it is near-instant on a local image.
|
||||
//!
|
||||
//! The reference implementation of this stage is `tools/identify-disc.py`; the
|
||||
//! two are checked against each other in `tests/disc.rs`.
|
||||
|
||||
use std::path::Path;
|
||||
|
||||
use gc_formats::iso9660::IsoImage;
|
||||
|
||||
/// The build this project targets.
|
||||
pub const TARGET_SERIAL: &str = "SCUS_972.68";
|
||||
pub const TARGET_VERSION: &str = "1.01";
|
||||
/// Measured from a retail Black Label v1.01 image. See `docs/disc-identification.md`.
|
||||
pub const TARGET_BOOT_ELF_SHA1: &str = "bacda860cbd6723142e86db56f197e47e728970f";
|
||||
|
||||
/// What we learned about a disc.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct DiscInfo {
|
||||
pub path: String,
|
||||
pub image_size: u64,
|
||||
pub volume_id: String,
|
||||
/// Product serial, i.e. the boot executable's file name.
|
||||
pub serial: String,
|
||||
/// `VER` from `SYSTEM.CNF`.
|
||||
pub version: String,
|
||||
/// `VMODE` from `SYSTEM.CNF`: `NTSC` or `PAL`, possibly empty.
|
||||
pub vmode: String,
|
||||
pub boot_elf_size: u32,
|
||||
pub boot_elf_sha1: String,
|
||||
}
|
||||
|
||||
impl DiscInfo {
|
||||
/// Is this the exact build the reference material describes?
|
||||
pub fn is_target(&self) -> bool {
|
||||
self.serial == TARGET_SERIAL && self.version == TARGET_VERSION
|
||||
}
|
||||
|
||||
/// True when the video mode makes the game run at 50 Hz.
|
||||
///
|
||||
/// This matters more than a label difference: a PAL disc runs the game at
|
||||
/// 50 Hz, which puts speed-sensitive movement such as the chargeboots at
|
||||
/// 5/6 of their intended rate.
|
||||
pub fn is_pal(&self) -> bool {
|
||||
self.vmode.eq_ignore_ascii_case("PAL")
|
||||
}
|
||||
|
||||
/// A short human-readable summary of whether this disc can be used.
|
||||
///
|
||||
/// The video mode is reported for every disc, not just the target: a 50 Hz
|
||||
/// PAL disc changes movement timing, which is the kind of detail that is
|
||||
/// easy to forget and hard to notice later.
|
||||
pub fn verdict(&self) -> String {
|
||||
let rate = if self.is_pal() {
|
||||
" 50 Hz: speed-sensitive movement runs at 5/6 speed, so timing constants differ."
|
||||
} else if self.vmode.is_empty() {
|
||||
""
|
||||
} else {
|
||||
" 60 Hz."
|
||||
};
|
||||
|
||||
if self.is_target() && self.boot_elf_sha1 == TARGET_BOOT_ELF_SHA1 {
|
||||
return format!(
|
||||
"USA Black Label v1.01, NTSC 60 Hz: the build we target. Boot ELF SHA-1 matches.{rate}"
|
||||
);
|
||||
}
|
||||
if self.serial == "SCUS_972.68GH" {
|
||||
return format!(
|
||||
"Greatest Hits v2.00. This is a different build: the chargeboots were patched, \
|
||||
and reference material built for v1.01 will not match.{rate}"
|
||||
);
|
||||
}
|
||||
if self.serial == TARGET_SERIAL {
|
||||
return format!(
|
||||
"Serial matches but version is {} (we target {TARGET_VERSION}).{rate}",
|
||||
if self.version.is_empty() {
|
||||
"unknown"
|
||||
} else {
|
||||
&self.version
|
||||
}
|
||||
);
|
||||
}
|
||||
format!("Not the target build. We target {TARGET_SERIAL} v{TARGET_VERSION}.{rate}")
|
||||
}
|
||||
}
|
||||
|
||||
/// Extract `KEY = value` from `SYSTEM.CNF`, stopping at `;`, CR or LF.
|
||||
fn cnf_value(cnf: &str, key: &str) -> String {
|
||||
for line in cnf.lines() {
|
||||
if let Some((k, v)) = line.split_once('=') {
|
||||
if k.trim() == key {
|
||||
return v.trim().split(';').next().unwrap_or("").trim().to_string();
|
||||
}
|
||||
}
|
||||
}
|
||||
String::new()
|
||||
}
|
||||
|
||||
/// Identify a disc image, or explain why it cannot be used.
|
||||
pub fn identify(path: &Path) -> Result<DiscInfo, String> {
|
||||
let iso = IsoImage::open(path).map_err(|e| format!("cannot read {}: {e}", path.display()))?;
|
||||
|
||||
let cnf_entry = iso
|
||||
.find("SYSTEM.CNF")
|
||||
.ok_or_else(|| "no SYSTEM.CNF: not a PlayStation 2 game disc".to_string())?;
|
||||
let cnf = gc_formats::iso9660::read_at(path, cnf_entry.lba, cnf_entry.size)
|
||||
.map_err(|e| format!("cannot read SYSTEM.CNF: {e}"))?;
|
||||
let cnf = String::from_utf8_lossy(&cnf);
|
||||
|
||||
let boot = cnf_value(&cnf, "BOOT2");
|
||||
if boot.is_empty() {
|
||||
return Err("SYSTEM.CNF has no BOOT2 line: not a PlayStation 2 game disc".into());
|
||||
}
|
||||
let boot_path = boot
|
||||
.strip_prefix("cdrom0:")
|
||||
.unwrap_or(&boot)
|
||||
.replace('\\', "/")
|
||||
// SYSTEM.CNF writes the boot path rooted, e.g. `cdrom0:\SCUS_972.68`.
|
||||
// Strip any leading slash so it matches an ISO directory name.
|
||||
.trim_start_matches('/')
|
||||
.to_string();
|
||||
let serial = boot_path.rsplit('/').next().unwrap_or("").to_string();
|
||||
|
||||
let entry = iso
|
||||
.find(&boot_path)
|
||||
.ok_or_else(|| format!("SYSTEM.CNF boots {boot_path}, which is not on the disc"))?;
|
||||
let elf = gc_formats::iso9660::read_at(path, entry.lba, entry.size)
|
||||
.map_err(|e| format!("cannot read the boot executable {serial}: {e}"))?;
|
||||
|
||||
if elf.len() < 52 || &elf[..4] != b"\x7fELF" {
|
||||
return Err(format!("{serial} is not an ELF executable"));
|
||||
}
|
||||
|
||||
Ok(DiscInfo {
|
||||
path: path.display().to_string(),
|
||||
image_size: std::fs::metadata(path).map(|m| m.len()).unwrap_or(0),
|
||||
volume_id: iso.volume_id.clone(),
|
||||
serial,
|
||||
version: cnf_value(&cnf, "VER"),
|
||||
vmode: cnf_value(&cnf, "VMODE"),
|
||||
boot_elf_size: entry.size,
|
||||
boot_elf_sha1: sha1_hex(&elf),
|
||||
})
|
||||
}
|
||||
|
||||
/// SHA-1 of `data`, lowercase hex.
|
||||
///
|
||||
/// Implemented here rather than pulled in as a dependency so that `identify`
|
||||
/// stays usable as a standalone stage with a minimal dependency set.
|
||||
pub fn sha1_hex(data: &[u8]) -> String {
|
||||
let mut h = Sha1::new();
|
||||
h.update(data);
|
||||
h.hex()
|
||||
}
|
||||
|
||||
/// Minimal SHA-1 (FIPS 180-4).
|
||||
pub struct Sha1 {
|
||||
state: [u32; 5],
|
||||
buffer: [u8; 64],
|
||||
buffered: usize,
|
||||
length: u64,
|
||||
}
|
||||
|
||||
impl Default for Sha1 {
|
||||
fn default() -> Self {
|
||||
Self::new()
|
||||
}
|
||||
}
|
||||
|
||||
impl Sha1 {
|
||||
pub fn new() -> Self {
|
||||
Self {
|
||||
state: [
|
||||
0x6745_2301,
|
||||
0xEFCD_AB89,
|
||||
0x98BA_DCFE,
|
||||
0x1032_5476,
|
||||
0xC3D2_E1F0,
|
||||
],
|
||||
buffer: [0u8; 64],
|
||||
buffered: 0,
|
||||
length: 0,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn update(&mut self, mut data: &[u8]) {
|
||||
self.length = self.length.wrapping_add(data.len() as u64);
|
||||
if self.buffered > 0 {
|
||||
let take = (64 - self.buffered).min(data.len());
|
||||
self.buffer[self.buffered..self.buffered + take].copy_from_slice(&data[..take]);
|
||||
self.buffered += take;
|
||||
data = &data[take..];
|
||||
if self.buffered == 64 {
|
||||
let block = self.buffer;
|
||||
self.compress(&block);
|
||||
self.buffered = 0;
|
||||
}
|
||||
}
|
||||
while data.len() >= 64 {
|
||||
let mut block = [0u8; 64];
|
||||
block.copy_from_slice(&data[..64]);
|
||||
self.compress(&block);
|
||||
data = &data[64..];
|
||||
}
|
||||
if !data.is_empty() {
|
||||
self.buffer[..data.len()].copy_from_slice(data);
|
||||
self.buffered = data.len();
|
||||
}
|
||||
}
|
||||
|
||||
fn compress(&mut self, block: &[u8; 64]) {
|
||||
let mut w = [0u32; 80];
|
||||
for i in 0..16 {
|
||||
w[i] = u32::from_be_bytes([
|
||||
block[i * 4],
|
||||
block[i * 4 + 1],
|
||||
block[i * 4 + 2],
|
||||
block[i * 4 + 3],
|
||||
]);
|
||||
}
|
||||
for i in 16..80 {
|
||||
w[i] = (w[i - 3] ^ w[i - 8] ^ w[i - 14] ^ w[i - 16]).rotate_left(1);
|
||||
}
|
||||
let (mut a, mut b, mut c, mut d, mut e) = (
|
||||
self.state[0],
|
||||
self.state[1],
|
||||
self.state[2],
|
||||
self.state[3],
|
||||
self.state[4],
|
||||
);
|
||||
for (i, &wi) in w.iter().enumerate() {
|
||||
let (f, k) = match i {
|
||||
0..=19 => ((b & c) | ((!b) & d), 0x5A82_7999u32),
|
||||
20..=39 => (b ^ c ^ d, 0x6ED9_EBA1),
|
||||
40..=59 => ((b & c) | (b & d) | (c & d), 0x8F1B_BCDC),
|
||||
_ => (b ^ c ^ d, 0xCA62_C1D6),
|
||||
};
|
||||
let temp = a
|
||||
.rotate_left(5)
|
||||
.wrapping_add(f)
|
||||
.wrapping_add(e)
|
||||
.wrapping_add(k)
|
||||
.wrapping_add(wi);
|
||||
e = d;
|
||||
d = c;
|
||||
c = b.rotate_left(30);
|
||||
b = a;
|
||||
a = temp;
|
||||
}
|
||||
self.state[0] = self.state[0].wrapping_add(a);
|
||||
self.state[1] = self.state[1].wrapping_add(b);
|
||||
self.state[2] = self.state[2].wrapping_add(c);
|
||||
self.state[3] = self.state[3].wrapping_add(d);
|
||||
self.state[4] = self.state[4].wrapping_add(e);
|
||||
}
|
||||
|
||||
pub fn hex(mut self) -> String {
|
||||
let bit_len = self.length.wrapping_mul(8);
|
||||
self.update_raw(&[0x80]);
|
||||
while self.buffered != 56 {
|
||||
self.update_raw(&[0x00]);
|
||||
}
|
||||
let len_bytes = bit_len.to_be_bytes();
|
||||
self.update_raw(&len_bytes);
|
||||
let mut out = String::with_capacity(40);
|
||||
for word in self.state {
|
||||
out.push_str(&format!("{word:08x}"));
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// Feed padding without changing the recorded message length.
|
||||
fn update_raw(&mut self, data: &[u8]) {
|
||||
for &byte in data {
|
||||
self.buffer[self.buffered] = byte;
|
||||
self.buffered += 1;
|
||||
if self.buffered == 64 {
|
||||
let block = self.buffer;
|
||||
self.compress(&block);
|
||||
self.buffered = 0;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn sha1_matches_known_vectors() {
|
||||
assert_eq!(sha1_hex(b""), "da39a3ee5e6b4b0d3255bfef95601890afd80709");
|
||||
assert_eq!(sha1_hex(b"abc"), "a9993e364706816aba3e25717850c26c9cd0d89d");
|
||||
assert_eq!(
|
||||
sha1_hex(b"abcdbcdecdefdefgefghfghighijhijkijkljklmklmnlmnomnopnopq"),
|
||||
"84983e441c3bd26ebaae4aa1f95129e5e54670f1"
|
||||
);
|
||||
// Exactly one block.
|
||||
assert_eq!(
|
||||
sha1_hex(&[b'a'; 64]),
|
||||
"0098ba824b5c16427bd7a1122a5a442a25ec644d"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sha1_handles_multi_block_input() {
|
||||
let million: Vec<u8> = std::iter::repeat_n(b'a', 1_000_000).collect();
|
||||
assert_eq!(
|
||||
sha1_hex(&million),
|
||||
"34aa973cd4c4daa4f61eeb2bdbad27316534016f"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cnf_parsing() {
|
||||
let cnf = "BOOT2 = cdrom0:\\SCUS_972.68;1\r\nVER = 1.01\r\nVMODE = NTSC\r\n\r\n";
|
||||
assert_eq!(cnf_value(cnf, "BOOT2"), "cdrom0:\\SCUS_972.68");
|
||||
assert_eq!(cnf_value(cnf, "VER"), "1.01");
|
||||
assert_eq!(cnf_value(cnf, "VMODE"), "NTSC");
|
||||
assert_eq!(cnf_value(cnf, "MISSING"), "");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn ntsc_verdict_mentions_sixty_hertz() {
|
||||
let d = DiscInfo {
|
||||
path: "x.iso".into(),
|
||||
image_size: 0,
|
||||
volume_id: String::new(),
|
||||
serial: TARGET_SERIAL.into(),
|
||||
version: TARGET_VERSION.into(),
|
||||
vmode: "NTSC".into(),
|
||||
boot_elf_size: 0,
|
||||
boot_elf_sha1: TARGET_BOOT_ELF_SHA1.into(),
|
||||
};
|
||||
assert!(d.verdict().contains("60 Hz"), "{}", d.verdict());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn target_verdict_for_the_supported_build() {
|
||||
let d = DiscInfo {
|
||||
path: "x.iso".into(),
|
||||
image_size: 0,
|
||||
volume_id: "RATCHETANDCLANK2".into(),
|
||||
serial: TARGET_SERIAL.into(),
|
||||
version: TARGET_VERSION.into(),
|
||||
vmode: "NTSC".into(),
|
||||
boot_elf_size: 0,
|
||||
boot_elf_sha1: TARGET_BOOT_ELF_SHA1.into(),
|
||||
};
|
||||
assert!(d.is_target());
|
||||
assert!(!d.is_pal());
|
||||
assert!(d.verdict().contains("the build we target"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn greatest_hits_is_called_out_as_a_different_build() {
|
||||
let d = DiscInfo {
|
||||
path: "x.iso".into(),
|
||||
image_size: 0,
|
||||
volume_id: String::new(),
|
||||
serial: "SCUS_972.68GH".into(),
|
||||
version: "2.00".into(),
|
||||
vmode: "NTSC".into(),
|
||||
boot_elf_size: 0,
|
||||
boot_elf_sha1: String::new(),
|
||||
};
|
||||
assert!(!d.is_target());
|
||||
assert!(d.verdict().contains("different build"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pal_is_flagged_as_fifty_hertz() {
|
||||
let d = DiscInfo {
|
||||
path: "x.iso".into(),
|
||||
image_size: 0,
|
||||
volume_id: String::new(),
|
||||
serial: "SCES-516.07".into(),
|
||||
version: "1.00".into(),
|
||||
vmode: "PAL".into(),
|
||||
boot_elf_size: 0,
|
||||
boot_elf_sha1: String::new(),
|
||||
};
|
||||
assert!(d.is_pal());
|
||||
assert!(d.verdict().contains("50 Hz"), "{}", d.verdict());
|
||||
}
|
||||
}
|
||||
@@ -1,29 +1,23 @@
|
||||
//! Disc pipeline for *Ratchet & Clank: Going Commando* (PS2).
|
||||
//! The disc pipeline for *Ratchet & Clank: Going Commando*.
|
||||
//!
|
||||
//! Turns a legally obtained disc image into verified, unpacked asset files that
|
||||
//! the engine can read directly, and does nothing else.
|
||||
//! Three stages, in order:
|
||||
//!
|
||||
//! # Stages
|
||||
//!
|
||||
//! 1. **identify** - read `SYSTEM.CNF`, confirm serial/version/video mode and
|
||||
//! 1. `identify` — read `SYSTEM.CNF`, confirm serial, version and video mode,
|
||||
//! hash the boot ELF. Reports clearly when a disc is not the supported
|
||||
//! build. A Python reference implementation of this stage lives in
|
||||
//! `tools/identify-disc.py`.
|
||||
//! 2. **unpack** - read `RC2.HDR`, walk the table of contents, decompress WAD
|
||||
//! streams and write per-level and global asset directories.
|
||||
//! 3. **verify** - check every extracted file against a recorded size and
|
||||
//! SHA-1 table, so a corrupted or mismatched dump cannot quietly produce a
|
||||
//! subtly wrong game.
|
||||
//! build.
|
||||
//! 2. `unpack` — enumerate the ISO 9660 filesystem, decompress WAD streams and
|
||||
//! write asset files.
|
||||
//! 3. `verify` — check extracted files against a committed size and SHA-1
|
||||
//! table, so a wrong or damaged dump cannot quietly produce a subtly wrong
|
||||
//! game.
|
||||
//!
|
||||
//! # Output layout
|
||||
//!
|
||||
//! Extracted data is written outside the repository (see `.gitignore`). The
|
||||
//! pre-commit hook in `.githooks/` refuses disc images, asset directories and
|
||||
//! oversized files, so nothing here can end up in git history by accident.
|
||||
//!
|
||||
//! # Verification is not optional
|
||||
//!
|
||||
//! Every parser downstream is only as trustworthy as the bytes it was handed.
|
||||
//! A hash table committed to the repo (sizes and hashes only, never content)
|
||||
//! is what lets a test say "this parse matches the retail disc" rather than
|
||||
//! "this parse matches whatever happened to be on my disk".
|
||||
//! Everything here reads the user's own legally obtained disc and writes
|
||||
//! outside the repository. See `AGENTS.md`.
|
||||
|
||||
pub mod identify;
|
||||
pub mod unpack;
|
||||
pub mod verify;
|
||||
|
||||
pub use identify::{identify, DiscInfo};
|
||||
pub use unpack::unpack;
|
||||
pub use verify::{verify, VerifyReport};
|
||||
|
||||
@@ -0,0 +1,215 @@
|
||||
//! `gc-extract` — the disc pipeline command line.
|
||||
//!
|
||||
//! Three subcommands, in the order they are normally used:
|
||||
//!
|
||||
//! ```text
|
||||
//! gc-extract identify <disc.iso>
|
||||
//! gc-extract unpack <disc.iso> <out-dir> [--table <table.tsv>]
|
||||
//! gc-extract verify <out-dir> <table.tsv>
|
||||
//! ```
|
||||
//!
|
||||
//! Only your own legally obtained disc is read. Nothing is written inside the
|
||||
//! repository.
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::process::ExitCode;
|
||||
|
||||
use gc_extract::{identify, unpack, verify};
|
||||
|
||||
fn main() -> ExitCode {
|
||||
let args: Vec<String> = std::env::args().skip(1).collect();
|
||||
match args.first().map(String::as_str) {
|
||||
Some("identify") => run_identify(&args[1..]),
|
||||
Some("unpack") => run_unpack(&args[1..]),
|
||||
Some("verify") => run_verify(&args[1..]),
|
||||
Some("--help") | Some("-h") | Some("help") | None => {
|
||||
usage();
|
||||
ExitCode::SUCCESS
|
||||
}
|
||||
Some(other) => {
|
||||
eprintln!("unknown subcommand `{other}`\n");
|
||||
usage();
|
||||
ExitCode::FAILURE
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Group digits for readability. Rust's format syntax has no thousands
|
||||
/// separator, so do it here rather than sprinkle the logic through the prints.
|
||||
fn thousands(n: u64) -> String {
|
||||
let s = n.to_string();
|
||||
let mut out = String::with_capacity(s.len() + s.len() / 3);
|
||||
for (i, c) in s.chars().enumerate() {
|
||||
if i > 0 && (s.len() - i) % 3 == 0 {
|
||||
out.push('_');
|
||||
}
|
||||
out.push(c);
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
fn usage() {
|
||||
println!(
|
||||
"gc-extract — Ratchet & Clank: Going Commando disc pipeline
|
||||
|
||||
identify <disc.iso>
|
||||
Report serial, version and video mode, and hash the boot ELF.
|
||||
|
||||
unpack <disc.iso> <out-dir> [--table <file.tsv>]
|
||||
Decompress every WAD on the disc into <out-dir>.
|
||||
With --table, also write a SHA-1 table (sizes and hashes only).
|
||||
|
||||
verify <out-dir> <table.tsv>
|
||||
Check <out-dir> against a previously written table.
|
||||
|
||||
You supply the disc. Nothing here is redistributed with the project."
|
||||
);
|
||||
}
|
||||
|
||||
fn run_identify(args: &[String]) -> ExitCode {
|
||||
if args.is_empty() {
|
||||
eprintln!("usage: gc-extract identify <disc.iso>");
|
||||
return ExitCode::FAILURE;
|
||||
}
|
||||
let path = Path::new(&args[0]);
|
||||
match identify(path) {
|
||||
Ok(info) => {
|
||||
println!("Disc identification");
|
||||
println!("{}", "=".repeat(64));
|
||||
println!(" file {}", info.path);
|
||||
println!(" size {} bytes", thousands(info.image_size));
|
||||
println!(" volume id {}", info.volume_id);
|
||||
println!(" serial {}", info.serial);
|
||||
println!(
|
||||
" version {}",
|
||||
if info.version.is_empty() {
|
||||
"(absent)"
|
||||
} else {
|
||||
&info.version
|
||||
}
|
||||
);
|
||||
println!(
|
||||
" video mode {}",
|
||||
if info.vmode.is_empty() {
|
||||
"(absent)"
|
||||
} else {
|
||||
&info.vmode
|
||||
}
|
||||
);
|
||||
println!();
|
||||
println!("Boot executable");
|
||||
println!("{}", "-".repeat(64));
|
||||
println!(
|
||||
" size {} bytes",
|
||||
thousands(info.boot_elf_size as u64)
|
||||
);
|
||||
println!(" SHA-1 {}", info.boot_elf_sha1);
|
||||
println!();
|
||||
println!("{}", "=".repeat(64));
|
||||
println!(" {}", info.verdict());
|
||||
ExitCode::SUCCESS
|
||||
}
|
||||
Err(e) => {
|
||||
eprintln!("error: {e}");
|
||||
ExitCode::FAILURE
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn run_unpack(args: &[String]) -> ExitCode {
|
||||
if args.len() < 2 {
|
||||
eprintln!("usage: gc-extract unpack <disc.iso> <out-dir> [--table <file.tsv>]");
|
||||
return ExitCode::FAILURE;
|
||||
}
|
||||
let iso = PathBuf::from(&args[0]);
|
||||
let out = PathBuf::from(&args[1]);
|
||||
let mut table_path: Option<PathBuf> = None;
|
||||
let mut i = 2;
|
||||
while i < args.len() {
|
||||
match args[i].as_str() {
|
||||
"--table" => {
|
||||
i += 1;
|
||||
match args.get(i) {
|
||||
Some(p) => table_path = Some(PathBuf::from(p)),
|
||||
None => {
|
||||
eprintln!("--table needs a path");
|
||||
return ExitCode::FAILURE;
|
||||
}
|
||||
}
|
||||
}
|
||||
other => {
|
||||
eprintln!("unexpected argument `{other}`");
|
||||
return ExitCode::FAILURE;
|
||||
}
|
||||
}
|
||||
i += 1;
|
||||
}
|
||||
|
||||
let ident = match identify::identify(&iso) {
|
||||
Ok(i) => i,
|
||||
Err(e) => {
|
||||
eprintln!("error: {e}");
|
||||
return ExitCode::FAILURE;
|
||||
}
|
||||
};
|
||||
println!("Disc: {} {} ({})", ident.serial, ident.version, ident.vmode);
|
||||
|
||||
let started = std::time::Instant::now();
|
||||
println!("Unpacking into {} ...", out.display());
|
||||
match unpack::unpack(&iso, &out) {
|
||||
Ok(files) => {
|
||||
let total: u64 = files.iter().map(|f| f.size).sum();
|
||||
println!(
|
||||
" {} files, {} bytes, in {:.1}s",
|
||||
files.len(),
|
||||
thousands(total),
|
||||
started.elapsed().as_secs_f64()
|
||||
);
|
||||
|
||||
if let Some(tp) = table_path {
|
||||
let entries: Vec<(String, u64, String)> = files
|
||||
.iter()
|
||||
.map(|f| (f.path.clone(), f.size, f.sha1.clone()))
|
||||
.collect();
|
||||
let text = verify::render_table(&entries);
|
||||
if let Err(e) = std::fs::write(&tp, text) {
|
||||
eprintln!("error: cannot write {}: {e}", tp.display());
|
||||
return ExitCode::FAILURE;
|
||||
}
|
||||
println!(" table written to {}", tp.display());
|
||||
}
|
||||
ExitCode::SUCCESS
|
||||
}
|
||||
Err(e) => {
|
||||
eprintln!("error: {e}");
|
||||
ExitCode::FAILURE
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn run_verify(args: &[String]) -> ExitCode {
|
||||
if args.len() < 2 {
|
||||
eprintln!("usage: gc-extract verify <out-dir> <table.tsv>");
|
||||
return ExitCode::FAILURE;
|
||||
}
|
||||
let dir = PathBuf::from(&args[0]);
|
||||
let table = match std::fs::read_to_string(&args[1]) {
|
||||
Ok(t) => t,
|
||||
Err(e) => {
|
||||
eprintln!("error: cannot read {}: {e}", args[1]);
|
||||
return ExitCode::FAILURE;
|
||||
}
|
||||
};
|
||||
let expected = verify::parse_table(&table);
|
||||
if expected.is_empty() {
|
||||
eprintln!("error: {} contained no usable rows", args[1]);
|
||||
return ExitCode::FAILURE;
|
||||
}
|
||||
let report = verify::verify(&dir, &expected);
|
||||
println!("{}", report.summary());
|
||||
if report.ok() {
|
||||
ExitCode::SUCCESS
|
||||
} else {
|
||||
ExitCode::FAILURE
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,222 @@
|
||||
//! Stage 2: unpack the disc into asset files.
|
||||
//!
|
||||
//! R&C2 lists its assets in the ISO 9660 filesystem, unlike R&C1, so
|
||||
//! enumeration is straightforward: every entry under `G/` is a container whose
|
||||
//! header holds a table of sector ranges, and each range is written out as a
|
||||
//! file.
|
||||
//!
|
||||
//! Output goes outside the repository. `AGENTS.md` explains why, and the
|
||||
//! pre-commit hook enforces it.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use gc_formats::global_wad::{GlobalWad, RangeKind};
|
||||
use gc_formats::iso9660::IsoImage;
|
||||
|
||||
/// One unpacked asset, with the identity we record for verification.
|
||||
pub struct Extracted {
|
||||
/// Path relative to the output directory, e.g. `armor/0000.bin`.
|
||||
pub path: String,
|
||||
pub size: u64,
|
||||
pub sha1: String,
|
||||
}
|
||||
|
||||
/// Unpack `iso_path` into `out_dir`.
|
||||
///
|
||||
/// Returns every file written, sorted by path, so the caller can build a hash
|
||||
/// table or verify against one.
|
||||
pub fn unpack(iso_path: &Path, out_dir: &Path) -> Result<Vec<Extracted>, String> {
|
||||
let iso = IsoImage::open(iso_path).map_err(|e| format!("cannot read the disc: {e}"))?;
|
||||
std::fs::create_dir_all(out_dir)
|
||||
.map_err(|e| format!("cannot create {}: {e}", out_dir.display()))?;
|
||||
|
||||
let mut written: BTreeMap<String, Extracted> = BTreeMap::new();
|
||||
let mut skipped: Vec<(String, String)> = Vec::new();
|
||||
|
||||
for entry in &iso.files {
|
||||
if entry.is_dir {
|
||||
continue;
|
||||
}
|
||||
// `Z6TAIL.DUP` is zero-length padding the ISO 9660 filesystem needs.
|
||||
if entry.name.starts_with("Z6TAIL") {
|
||||
continue;
|
||||
}
|
||||
|
||||
let relative = entry.name.trim_end_matches('/').to_string();
|
||||
|
||||
if !relative.starts_with("G/") {
|
||||
// Not a container: the boot ELF, SYSTEM.CNF, RC2.HDR and the IOP
|
||||
// image are copied through verbatim.
|
||||
let data = gc_formats::iso9660::read_at(iso_path, entry.lba, entry.size)
|
||||
.map_err(|e| format!("cannot read {}: {e}", entry.name))?;
|
||||
write_file(out_dir, &relative, &data, &mut written);
|
||||
continue;
|
||||
}
|
||||
|
||||
// A container: write out each range its header declares.
|
||||
let raw = gc_formats::iso9660::read_at(iso_path, entry.lba, entry.size)
|
||||
.map_err(|e| format!("cannot read {}: {e}", entry.name))?;
|
||||
|
||||
let stem = relative
|
||||
.trim_start_matches("G/")
|
||||
.trim_end_matches(".WAD")
|
||||
.to_lowercase();
|
||||
|
||||
// MPEG1 and MPEG2 are raw concatenated video, not indexed containers:
|
||||
// their first word is video payload, not a header size. Copy them
|
||||
// through; the video phase splits them properly.
|
||||
if stem.starts_with("mpeg1") || stem.starts_with("mpeg2") {
|
||||
write_file(out_dir, &format!("{stem}.bin"), &raw, &mut written);
|
||||
println!(
|
||||
" {:<16} {} bytes, raw video stream (not indexed)",
|
||||
stem,
|
||||
raw.len()
|
||||
);
|
||||
continue;
|
||||
}
|
||||
|
||||
let container = match GlobalWad::parse_with(&raw, RangeKind::for_container(&relative)) {
|
||||
Ok(c) => c,
|
||||
Err(e) => {
|
||||
// Some containers have a header this generic parser does not
|
||||
// model (the scene containers nest further). Copy the file
|
||||
// through untouched rather than failing the whole run: the
|
||||
// bytes are preserved and a typed parser can be added later.
|
||||
skipped.push((relative.clone(), e.to_string()));
|
||||
write_file(out_dir, &relative, &raw, &mut written);
|
||||
continue;
|
||||
}
|
||||
};
|
||||
let present = container.present().count();
|
||||
let mut out_of_range = 0usize;
|
||||
|
||||
for (index, offset, byte_len) in container.present() {
|
||||
let len = byte_len as usize;
|
||||
|
||||
// Bounds-check against the container we actually read, so a corrupt
|
||||
// header cannot make us read past the end of the file.
|
||||
if offset + len as u64 > raw.len() as u64 {
|
||||
out_of_range += 1;
|
||||
continue;
|
||||
}
|
||||
let data = &raw[offset as usize..offset as usize + len];
|
||||
write_file(
|
||||
out_dir,
|
||||
&format!("{stem}/{index:04}.bin"),
|
||||
data,
|
||||
&mut written,
|
||||
);
|
||||
}
|
||||
if out_of_range > 0 {
|
||||
eprintln!(
|
||||
" {:<16} {} bytes, {} entries -> {} present, {out_of_range} out of range (header describes \
|
||||
sectors beyond the file; see docs/plan/disc-pipeline.md)",
|
||||
stem, raw.len(), container.entries.len(), present
|
||||
);
|
||||
} else {
|
||||
println!(
|
||||
" {:<16} {} bytes, {} entries -> {} present",
|
||||
stem,
|
||||
raw.len(),
|
||||
container.entries.len(),
|
||||
present
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
for (name, why) in &skipped {
|
||||
eprintln!(" {name}: not an indexed container ({why}); copied verbatim");
|
||||
}
|
||||
|
||||
Ok(written.into_values().collect())
|
||||
}
|
||||
|
||||
fn write_file(
|
||||
out_dir: &Path,
|
||||
relative: &str,
|
||||
data: &[u8],
|
||||
written: &mut BTreeMap<String, Extracted>,
|
||||
) {
|
||||
use crate::identify::sha1_hex;
|
||||
|
||||
let dest = out_dir.join(relative);
|
||||
if let Some(parent) = dest.parent() {
|
||||
if let Err(e) = std::fs::create_dir_all(parent) {
|
||||
eprintln!("warning: cannot create {}: {e}", parent.display());
|
||||
return;
|
||||
}
|
||||
}
|
||||
if let Err(e) = std::fs::write(&dest, data) {
|
||||
eprintln!("warning: cannot write {}: {e}", dest.display());
|
||||
return;
|
||||
}
|
||||
written.insert(
|
||||
relative.to_string(),
|
||||
Extracted {
|
||||
path: relative.to_string(),
|
||||
size: data.len() as u64,
|
||||
sha1: sha1_hex(data),
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
/// Default output directory: `extracted/` beside the repository root.
|
||||
pub fn default_out_dir() -> PathBuf {
|
||||
PathBuf::from("extracted")
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// A container with three entries, the second of which is absent.
|
||||
fn container_bytes() -> Vec<u8> {
|
||||
let header_size = 8 + 3 * 8;
|
||||
let mut v = Vec::new();
|
||||
v.extend_from_slice(&(header_size as u32).to_le_bytes());
|
||||
v.extend_from_slice(&10u32.to_le_bytes());
|
||||
v.extend_from_slice(&1i32.to_le_bytes());
|
||||
v.extend_from_slice(&1i32.to_le_bytes());
|
||||
v.extend_from_slice(&2i32.to_le_bytes());
|
||||
v.extend_from_slice(&0i32.to_le_bytes()); // absent
|
||||
v.extend_from_slice(&2i32.to_le_bytes());
|
||||
v.extend_from_slice(&1i32.to_le_bytes());
|
||||
v.resize(header_size + 3 * 2048, 0xAB);
|
||||
v
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parses_a_container_and_lists_present_entries() {
|
||||
let raw = container_bytes();
|
||||
let c = GlobalWad::parse(&raw).unwrap();
|
||||
assert_eq!(c.entries.len(), 3);
|
||||
let present: Vec<usize> = c.present().map(|(i, _, _)| i).collect();
|
||||
assert_eq!(present, vec![0, 2]);
|
||||
// The default reading treats the second field as a byte count, so an
|
||||
// entry of size 1 is one byte.
|
||||
assert_eq!(c.present().next().unwrap().2, 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn every_declared_range_is_inside_the_container() {
|
||||
let raw = container_bytes();
|
||||
let c = GlobalWad::parse(&raw).unwrap();
|
||||
for (_, offset, len) in c.present() {
|
||||
assert!(
|
||||
offset + len <= raw.len() as u64,
|
||||
"range at {offset} runs past the end"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_range_past_the_end_would_be_caught() {
|
||||
let mut raw = container_bytes();
|
||||
// Push the first entry's offset far beyond the file.
|
||||
raw[8..12].copy_from_slice(&9999i32.to_le_bytes());
|
||||
let c = GlobalWad::parse(&raw).unwrap();
|
||||
let (_, offset, len) = c.present().next().unwrap();
|
||||
assert!(offset + len > raw.len() as u64);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,266 @@
|
||||
//! Stage 3: verify extracted files against a committed hash table.
|
||||
//!
|
||||
//! The table records sizes and SHA-1 values only — never content. Its purpose
|
||||
//! is to let a test assert "this parse matches retail" rather than "this parse
|
||||
//! matches whatever happened to be on my disk". See `AGENTS.md`.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::path::Path;
|
||||
|
||||
/// One expected file: path, size in bytes, SHA-1.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct Expected {
|
||||
pub path: String,
|
||||
pub size: u64,
|
||||
pub sha1: String,
|
||||
}
|
||||
|
||||
/// Outcome of verifying a directory.
|
||||
#[derive(Debug, Default)]
|
||||
pub struct VerifyReport {
|
||||
pub checked: usize,
|
||||
pub missing: Vec<String>,
|
||||
pub size_mismatch: Vec<(String, u64, u64)>,
|
||||
pub hash_mismatch: Vec<(String, String, String)>,
|
||||
/// Files present in the directory but absent from the table.
|
||||
pub unexpected: Vec<String>,
|
||||
}
|
||||
|
||||
impl VerifyReport {
|
||||
pub fn ok(&self) -> bool {
|
||||
self.missing.is_empty()
|
||||
&& self.size_mismatch.is_empty()
|
||||
&& self.hash_mismatch.is_empty()
|
||||
&& self.unexpected.is_empty()
|
||||
}
|
||||
|
||||
/// A short human-readable summary.
|
||||
pub fn summary(&self) -> String {
|
||||
if self.ok() {
|
||||
return format!("OK: {} files match the recorded table.", self.checked);
|
||||
}
|
||||
let mut lines = vec![format!(
|
||||
"FAILED: {} files checked, {} problems",
|
||||
self.checked,
|
||||
self.missing.len()
|
||||
+ self.size_mismatch.len()
|
||||
+ self.hash_mismatch.len()
|
||||
+ self.unexpected.len()
|
||||
)];
|
||||
for p in self.missing.iter().take(10) {
|
||||
lines.push(format!(" missing: {p}"));
|
||||
}
|
||||
for (p, want, got) in self.size_mismatch.iter().take(10) {
|
||||
lines.push(format!(" size: {p}: expected {want}, got {got}"));
|
||||
}
|
||||
for (p, want, got) in self.hash_mismatch.iter().take(10) {
|
||||
lines.push(format!(" hash: {p}: expected {want}, got {got}"));
|
||||
}
|
||||
for p in self.unexpected.iter().take(10) {
|
||||
lines.push(format!(" unexpected: {p}"));
|
||||
}
|
||||
lines.join("\n")
|
||||
}
|
||||
}
|
||||
|
||||
/// Parse a table from TSV lines of `sha1<TAB>size<TAB>path`.
|
||||
pub fn parse_table(text: &str) -> Vec<Expected> {
|
||||
text.lines()
|
||||
.filter(|l| !l.trim().is_empty() && !l.starts_with('#'))
|
||||
.filter_map(|line| {
|
||||
let mut parts = line.split('\t');
|
||||
let sha1 = parts.next()?.trim().to_string();
|
||||
let size: u64 = parts.next()?.trim().parse().ok()?;
|
||||
let path = parts.next()?.trim().to_string();
|
||||
if sha1.len() != 40 || path.is_empty() {
|
||||
return None;
|
||||
}
|
||||
Some(Expected { path, size, sha1 })
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Render entries as a TSV table, sorted by path.
|
||||
pub fn render_table(entries: &[(String, u64, String)]) -> String {
|
||||
let mut sorted: Vec<_> = entries.to_vec();
|
||||
sorted.sort_by(|a, b| a.0.cmp(&b.0));
|
||||
let mut out = String::new();
|
||||
for (path, size, sha1) in sorted {
|
||||
out.push_str(&format!("{sha1}\t{size}\t{path}\n"));
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// Verify `dir` against `expected`.
|
||||
pub fn verify(dir: &Path, expected: &[Expected]) -> VerifyReport {
|
||||
let mut report = VerifyReport::default();
|
||||
|
||||
let present = walk(dir);
|
||||
|
||||
for e in expected {
|
||||
report.checked += 1;
|
||||
match std::fs::read(dir.join(&e.path)) {
|
||||
Ok(data) => {
|
||||
if data.len() as u64 != e.size {
|
||||
report
|
||||
.size_mismatch
|
||||
.push((e.path.clone(), e.size, data.len() as u64));
|
||||
continue;
|
||||
}
|
||||
let got = crate::identify::sha1_hex(&data);
|
||||
if got != e.sha1 {
|
||||
report
|
||||
.hash_mismatch
|
||||
.push((e.path.clone(), e.sha1.clone(), got));
|
||||
}
|
||||
}
|
||||
Err(_) => report.missing.push(e.path.clone()),
|
||||
}
|
||||
}
|
||||
|
||||
let expected_set: BTreeMap<&str, ()> = expected.iter().map(|e| (e.path.as_str(), ())).collect();
|
||||
for p in present {
|
||||
if !expected_set.contains_key(p.as_str()) {
|
||||
report.unexpected.push(p);
|
||||
}
|
||||
}
|
||||
report
|
||||
}
|
||||
|
||||
/// List files under `dir`, as `/`-separated relative paths, sorted.
|
||||
fn walk(dir: &Path) -> Vec<String> {
|
||||
let mut out = Vec::new();
|
||||
collect(dir, dir, &mut out);
|
||||
out.sort();
|
||||
out
|
||||
}
|
||||
|
||||
fn collect(root: &Path, dir: &Path, out: &mut Vec<String>) {
|
||||
let Ok(entries) = std::fs::read_dir(dir) else {
|
||||
return;
|
||||
};
|
||||
for entry in entries.flatten() {
|
||||
let path = entry.path();
|
||||
if path.is_dir() {
|
||||
collect(root, &path, out);
|
||||
} else if let Ok(rel) = path.strip_prefix(root) {
|
||||
out.push(rel.to_string_lossy().replace('\\', "/"));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn parses_a_table_and_skips_comments() {
|
||||
let text = concat!(
|
||||
"# a comment\n",
|
||||
"da39a3ee5e6b4b0d3255bfef95601890afd80709\t0\tempty.bin\n",
|
||||
"a9993e364706816aba3e25717850c26c9cd0d89d\t3\tabc.bin\n",
|
||||
"\n",
|
||||
);
|
||||
let t = parse_table(text);
|
||||
assert_eq!(t.len(), 2);
|
||||
assert_eq!(t[0].path, "empty.bin");
|
||||
assert_eq!(t[0].size, 0);
|
||||
assert_eq!(t[1].path, "abc.bin");
|
||||
assert_eq!(t[1].size, 3);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn ignores_malformed_rows() {
|
||||
let text = "notasha1\t3\tx.bin\n\
|
||||
a9993e364706816aba3e25717850c26c9cd0d89d\tnope\tx.bin\n\
|
||||
a9993e364706816aba3e25717850c26c9cd0d89d\t3\t";
|
||||
assert!(parse_table(text).is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn round_trips_a_rendered_table() {
|
||||
let entries = vec![
|
||||
(
|
||||
"b.bin".to_string(),
|
||||
2u64,
|
||||
"a9993e364706816aba3e25717850c26c9cd0d89d".to_string(),
|
||||
),
|
||||
(
|
||||
"a.bin".to_string(),
|
||||
1u64,
|
||||
"da39a3ee5e6b4b0d3255bfef95601890afd80709".to_string(),
|
||||
),
|
||||
];
|
||||
let text = render_table(&entries);
|
||||
assert!(
|
||||
text.lines().next().unwrap().starts_with("da39a3ee"),
|
||||
"sorted by path"
|
||||
);
|
||||
assert_eq!(parse_table(&text).len(), 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn verify_detects_a_missing_file() {
|
||||
let dir = std::env::temp_dir().join(format!("gc-verify-{}", std::process::id()));
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
std::fs::create_dir_all(&dir).unwrap();
|
||||
|
||||
let expected = vec![Expected {
|
||||
path: "nope.bin".to_string(),
|
||||
size: 3,
|
||||
sha1: "a9993e364706816aba3e25717850c26c9cd0d89d".to_string(),
|
||||
}];
|
||||
let report = verify(&dir, &expected);
|
||||
assert!(!report.ok());
|
||||
assert_eq!(report.missing, vec!["nope.bin".to_string()]);
|
||||
assert!(report.summary().contains("missing"));
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn verify_detects_a_hash_mismatch() {
|
||||
let dir = std::env::temp_dir().join(format!("gc-verify-hash-{}", std::process::id()));
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
std::fs::create_dir_all(&dir).unwrap();
|
||||
std::fs::write(dir.join("a.bin"), b"xyz").unwrap();
|
||||
|
||||
let expected = vec![Expected {
|
||||
path: "a.bin".to_string(),
|
||||
size: 3,
|
||||
// SHA-1 of "abc", not "xyz".
|
||||
sha1: "a9993e364706816aba3e25717850c26c9cd0d89d".to_string(),
|
||||
}];
|
||||
let report = verify(&dir, &expected);
|
||||
assert_eq!(report.hash_mismatch.len(), 1);
|
||||
assert!(report.summary().contains("hash"));
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn verify_detects_an_unexpected_file() {
|
||||
let dir = std::env::temp_dir().join(format!("gc-verify-extra-{}", std::process::id()));
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
std::fs::create_dir_all(&dir).unwrap();
|
||||
std::fs::write(dir.join("extra.bin"), b"q").unwrap();
|
||||
let report = verify(&dir, &[]);
|
||||
assert_eq!(report.unexpected, vec!["extra.bin".to_string()]);
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn verify_passes_a_correct_directory() {
|
||||
let dir = std::env::temp_dir().join(format!("gc-verify-ok-{}", std::process::id()));
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
std::fs::create_dir_all(&dir).unwrap();
|
||||
std::fs::write(dir.join("a.bin"), b"abc").unwrap();
|
||||
|
||||
let expected = vec![Expected {
|
||||
path: "a.bin".to_string(),
|
||||
size: 3,
|
||||
sha1: "a9993e364706816aba3e25717850c26c9cd0d89d".to_string(),
|
||||
}];
|
||||
let report = verify(&dir, &expected);
|
||||
assert!(report.ok(), "{}", report.summary());
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,184 @@
|
||||
//! Integration tests against a real disc image.
|
||||
//!
|
||||
//! These are skipped unless `GC_DISC` points at a legally obtained disc, since
|
||||
//! the repository must not contain game data. When it is set they check the
|
||||
//! identification and unpack results against the committed hash table.
|
||||
//!
|
||||
//! ```sh
|
||||
//! GC_DISC="path/to/disc.iso" cargo test -p gc-extract --test disc
|
||||
//! ```
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
/// The disc under test, if the environment names one.
|
||||
fn disc() -> Option<PathBuf> {
|
||||
let p = std::env::var_os("GC_DISC")?;
|
||||
let p = PathBuf::from(p);
|
||||
p.is_file().then_some(p)
|
||||
}
|
||||
|
||||
/// Skip a test with a clear reason rather than failing.
|
||||
macro_rules! require_disc {
|
||||
() => {
|
||||
match disc() {
|
||||
Some(p) => p,
|
||||
None => {
|
||||
eprintln!("skipped: set GC_DISC to a Going Commando disc image to run this");
|
||||
return;
|
||||
}
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn identifies_the_supported_build() {
|
||||
let path = require_disc!();
|
||||
let info = gc_extract::identify::identify(&path).expect("disc should identify");
|
||||
|
||||
assert_eq!(info.serial, "SCUS_972.68");
|
||||
assert_eq!(info.volume_id, "RATCHETANDCLANK2");
|
||||
assert!(
|
||||
info.is_target(),
|
||||
"expected the supported build, got: {}",
|
||||
info.verdict()
|
||||
);
|
||||
assert_eq!(
|
||||
info.boot_elf_sha1,
|
||||
gc_extract::identify::TARGET_BOOT_ELF_SHA1,
|
||||
"boot ELF SHA-1 differs from the recorded retail value"
|
||||
);
|
||||
assert_eq!(info.boot_elf_size, 2_618_684);
|
||||
// This disc is NTSC; the movement constants assume 60 Hz.
|
||||
assert!(!info.is_pal(), "test disc should be NTSC");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn agrees_with_the_python_reference_identifier() {
|
||||
let path = require_disc!();
|
||||
let info = gc_extract::identify::identify(&path).expect("disc should identify");
|
||||
|
||||
// tools/identify-disc.py is the reference implementation and prints these
|
||||
// two values. If they ever disagree, one of them has a bug.
|
||||
// Run from the workspace root so the relative path to the tool resolves.
|
||||
let root = std::path::Path::new(env!("CARGO_MANIFEST_DIR"))
|
||||
.parent()
|
||||
.and_then(|p| p.parent())
|
||||
.expect("the crate should live two levels below the workspace root")
|
||||
.to_path_buf();
|
||||
let py = std::process::Command::new("python3")
|
||||
.arg(root.join("tools/identify-disc.py"))
|
||||
.arg(&path)
|
||||
.current_dir(&root)
|
||||
.output()
|
||||
.expect("python3 should be available to run the reference identifier");
|
||||
assert!(
|
||||
py.status.success(),
|
||||
"the reference identifier failed:\n{}",
|
||||
String::from_utf8_lossy(&py.stderr)
|
||||
);
|
||||
let text = String::from_utf8_lossy(&py.stdout);
|
||||
|
||||
assert!(
|
||||
text.contains(&format!("serial {}", info.serial)),
|
||||
"serial disagrees with the reference identifier:\n{text}"
|
||||
);
|
||||
assert!(
|
||||
text.contains(&format!("SHA-1 {}", info.boot_elf_sha1)),
|
||||
"boot ELF SHA-1 disagrees with the reference identifier:\n{text}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unpacks_and_matches_the_committed_table() {
|
||||
let path = require_disc!();
|
||||
|
||||
let table_path = Path::new(env!("CARGO_MANIFEST_DIR")).join("data/gc-files.tsv");
|
||||
let table_text =
|
||||
std::fs::read_to_string(&table_path).expect("the hash table should be committed");
|
||||
let expected = gc_extract::verify::parse_table(&table_text);
|
||||
assert!(
|
||||
expected.len() > 5000,
|
||||
"expected a full table, got {} rows",
|
||||
expected.len()
|
||||
);
|
||||
|
||||
// Unpack into a temporary directory so the test leaves nothing behind.
|
||||
let tmp = std::env::temp_dir().join(format!("gc-unpack-{}", std::process::id()));
|
||||
let _ = std::fs::remove_dir_all(&tmp);
|
||||
std::fs::create_dir_all(&tmp).expect("should create a temp directory");
|
||||
|
||||
let files = gc_extract::unpack::unpack(&path, &tmp).expect("unpack should succeed");
|
||||
|
||||
// Every unpacked file must match the recorded size and hash.
|
||||
let report = gc_extract::verify::verify(&tmp, &expected);
|
||||
let _ = std::fs::remove_dir_all(&tmp);
|
||||
|
||||
assert!(
|
||||
report.ok(),
|
||||
"unpacked data does not match the committed table:\n{}",
|
||||
report.summary()
|
||||
);
|
||||
assert_eq!(
|
||||
files.len(),
|
||||
expected.len(),
|
||||
"unpack produced a different file count"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn every_container_parses() {
|
||||
let path = require_disc!();
|
||||
|
||||
let iso = gc_formats::iso9660::IsoImage::open(&path).expect("ISO should open");
|
||||
let containers: Vec<_> = iso
|
||||
.files
|
||||
.iter()
|
||||
.filter(|e| e.name.starts_with("G/") && e.name.ends_with(".WAD"))
|
||||
.collect();
|
||||
|
||||
// 92 `G/*.WAD` containers: 27 LEVEL, 27 AUDIO, 27 SCENE, 11 globals.
|
||||
assert_eq!(containers.len(), 92, "expected 92 asset containers");
|
||||
|
||||
// MPEG1 and MPEG2 are raw video streams with no table; the other 90 must
|
||||
// all parse as indexed containers.
|
||||
let raw_video: Vec<&str> = containers
|
||||
.iter()
|
||||
.map(|c| c.name.as_str())
|
||||
.filter(|n| *n == "G/MPEG1.WAD" || *n == "G/MPEG2.WAD")
|
||||
.collect();
|
||||
assert_eq!(
|
||||
raw_video.len(),
|
||||
2,
|
||||
"expected MPEG1 and MPEG2 to be the raw streams"
|
||||
);
|
||||
|
||||
// The per-level families must all be present, at 27 each.
|
||||
for family in ["LEVEL", "AUDIO", "SCENE"] {
|
||||
let n = containers
|
||||
.iter()
|
||||
.filter(|c| {
|
||||
let stem = c.name.trim_start_matches("G/").trim_end_matches(".WAD");
|
||||
stem.strip_prefix(family)
|
||||
.map(|rest| !rest.is_empty() && rest.chars().all(|ch| ch.is_ascii_digit()))
|
||||
.unwrap_or(false)
|
||||
})
|
||||
.count();
|
||||
assert_eq!(n, 27, "expected 27 {family}<n>.WAD containers, found {n}");
|
||||
}
|
||||
|
||||
for entry in &containers {
|
||||
if raw_video.contains(&entry.name.as_str()) {
|
||||
continue;
|
||||
}
|
||||
let data = gc_formats::iso9660::read_at(&path, entry.lba, entry.size)
|
||||
.expect("read should succeed");
|
||||
let kind = gc_formats::global_wad::RangeKind::for_container(&entry.name);
|
||||
let container = gc_formats::global_wad::GlobalWad::parse_with(&data, kind)
|
||||
.unwrap_or_else(|e| panic!("{} failed to parse: {e}", entry.name));
|
||||
assert!(
|
||||
!container.entries.is_empty(),
|
||||
"{} parsed but had no entries",
|
||||
entry.name
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,261 @@
|
||||
//! A bounds-checked cursor over a byte slice.
|
||||
//!
|
||||
//! Every format parser in this crate reads through a `Buf`. Asset data comes
|
||||
//! from a 2003 disc format, so it is untrusted input: a truncated or corrupt
|
||||
//! file must produce a typed error, never a panic and never a read past the
|
||||
//! end of the buffer.
|
||||
//!
|
||||
//! Reads are little-endian throughout. The PS2 is a little-endian machine and
|
||||
//! so is every asset container we have seen.
|
||||
|
||||
use std::fmt;
|
||||
|
||||
/// The position and context of a failed read.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct Error {
|
||||
pub offset: usize,
|
||||
pub context: &'static str,
|
||||
pub kind: ErrorKind,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub enum ErrorKind {
|
||||
/// A read reached past the end of the buffer.
|
||||
OutOfBounds { needed: usize, available: usize },
|
||||
/// A length field was present but nonsensical.
|
||||
InvalidLength(&'static str),
|
||||
/// A magic number or signature did not match.
|
||||
BadMagic { expected: &'static str },
|
||||
/// The structure parsed, but a field held an impossible value.
|
||||
Invalid(&'static str),
|
||||
}
|
||||
|
||||
impl fmt::Display for Error {
|
||||
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
|
||||
match &self.kind {
|
||||
ErrorKind::OutOfBounds { needed, available } => write!(
|
||||
f,
|
||||
"{}: need {needed} bytes at offset {:#x}, {} available",
|
||||
self.context, self.offset, available
|
||||
),
|
||||
ErrorKind::InvalidLength(what) => {
|
||||
write!(
|
||||
f,
|
||||
"{}: invalid length for {what} at offset {:#x}",
|
||||
self.context, self.offset
|
||||
)
|
||||
}
|
||||
ErrorKind::BadMagic { expected } => {
|
||||
write!(
|
||||
f,
|
||||
"{}: expected {expected} at offset {:#x}",
|
||||
self.context, self.offset
|
||||
)
|
||||
}
|
||||
ErrorKind::Invalid(what) => {
|
||||
write!(
|
||||
f,
|
||||
"{}: invalid {what} at offset {:#x}",
|
||||
self.context, self.offset
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl std::error::Error for Error {}
|
||||
|
||||
pub type Result<T> = std::result::Result<T, Error>;
|
||||
|
||||
/// A bounds-checked reader over a byte slice.
|
||||
#[derive(Clone, Copy)]
|
||||
pub struct Buf<'a> {
|
||||
pub bytes: &'a [u8],
|
||||
pos: usize,
|
||||
}
|
||||
|
||||
impl<'a> Buf<'a> {
|
||||
pub fn new(bytes: &'a [u8]) -> Self {
|
||||
Self { bytes, pos: 0 }
|
||||
}
|
||||
|
||||
pub fn len(&self) -> usize {
|
||||
self.bytes.len()
|
||||
}
|
||||
|
||||
pub fn is_empty(&self) -> bool {
|
||||
self.bytes.is_empty()
|
||||
}
|
||||
|
||||
/// Current read position.
|
||||
pub fn pos(&self) -> usize {
|
||||
self.pos
|
||||
}
|
||||
|
||||
/// Move the cursor. Positions beyond the end are an error on the next read
|
||||
/// rather than here, so that a caller can seek then check.
|
||||
pub fn seek(&mut self, pos: usize) {
|
||||
self.pos = pos;
|
||||
}
|
||||
|
||||
/// Skip `n` bytes.
|
||||
pub fn skip(&mut self, n: usize, context: &'static str) -> Result<()> {
|
||||
self.read_bytes(n, context).map(|_| ())
|
||||
}
|
||||
|
||||
/// The remainder of the buffer from the cursor.
|
||||
pub fn rest(&self) -> &'a [u8] {
|
||||
self.bytes.get(self.pos..).unwrap_or(&[])
|
||||
}
|
||||
|
||||
/// Read `n` bytes, advancing the cursor.
|
||||
pub fn read_bytes(&mut self, n: usize, context: &'static str) -> Result<&'a [u8]> {
|
||||
let end = self.pos.saturating_add(n);
|
||||
let slice = self.bytes.get(self.pos..end).ok_or(Error {
|
||||
offset: self.pos,
|
||||
context,
|
||||
kind: ErrorKind::OutOfBounds {
|
||||
needed: n,
|
||||
available: self.bytes.len().saturating_sub(self.pos),
|
||||
},
|
||||
})?;
|
||||
self.pos = end;
|
||||
Ok(slice)
|
||||
}
|
||||
|
||||
/// Read `n` bytes at an absolute offset without moving the cursor.
|
||||
pub fn peek_bytes(&self, offset: usize, n: usize, context: &'static str) -> Result<&'a [u8]> {
|
||||
let end = offset.saturating_add(n);
|
||||
self.bytes.get(offset..end).ok_or(Error {
|
||||
offset,
|
||||
context,
|
||||
kind: ErrorKind::OutOfBounds {
|
||||
needed: n,
|
||||
available: self.bytes.len().saturating_sub(offset),
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
pub fn u8(&mut self, context: &'static str) -> Result<u8> {
|
||||
Ok(self.read_bytes(1, context)?[0])
|
||||
}
|
||||
|
||||
pub fn i8(&mut self, context: &'static str) -> Result<i8> {
|
||||
Ok(self.u8(context)? as i8)
|
||||
}
|
||||
|
||||
pub fn u16(&mut self, context: &'static str) -> Result<u16> {
|
||||
let b = self.read_bytes(2, context)?;
|
||||
Ok(u16::from_le_bytes([b[0], b[1]]))
|
||||
}
|
||||
|
||||
pub fn i16(&mut self, context: &'static str) -> Result<i16> {
|
||||
Ok(self.u16(context)? as i16)
|
||||
}
|
||||
|
||||
pub fn u32(&mut self, context: &'static str) -> Result<u32> {
|
||||
let b = self.read_bytes(4, context)?;
|
||||
Ok(u32::from_le_bytes([b[0], b[1], b[2], b[3]]))
|
||||
}
|
||||
|
||||
pub fn i32(&mut self, context: &'static str) -> Result<i32> {
|
||||
Ok(self.u32(context)? as i32)
|
||||
}
|
||||
|
||||
pub fn u64(&mut self, context: &'static str) -> Result<u64> {
|
||||
let b = self.read_bytes(8, context)?;
|
||||
Ok(u64::from_le_bytes([
|
||||
b[0], b[1], b[2], b[3], b[4], b[5], b[6], b[7],
|
||||
]))
|
||||
}
|
||||
|
||||
pub fn f32(&mut self, context: &'static str) -> Result<f32> {
|
||||
Ok(f32::from_bits(self.u32(context)?))
|
||||
}
|
||||
|
||||
/// Read an absolute `u32` without moving the cursor.
|
||||
pub fn peek_u32(&self, offset: usize, context: &'static str) -> Result<u32> {
|
||||
let b = self.peek_bytes(offset, 4, context)?;
|
||||
Ok(u32::from_le_bytes([b[0], b[1], b[2], b[3]]))
|
||||
}
|
||||
|
||||
/// Read an absolute `i32` without moving the cursor.
|
||||
pub fn peek_i32(&self, offset: usize, context: &'static str) -> Result<i32> {
|
||||
Ok(self.peek_u32(offset, context)? as i32)
|
||||
}
|
||||
|
||||
/// Require `expected` bytes to be present, else a `BadMagic` error.
|
||||
pub fn expect_magic(&mut self, expected: &[u8], context: &'static str) -> Result<()> {
|
||||
let got = self.read_bytes(expected.len(), context)?;
|
||||
if got != expected {
|
||||
return Err(Error {
|
||||
offset: self.pos - expected.len(),
|
||||
context,
|
||||
kind: ErrorKind::BadMagic {
|
||||
expected: "the expected signature",
|
||||
},
|
||||
});
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn reads_little_endian() {
|
||||
let mut b = Buf::new(&[0x01, 0x02, 0x03, 0x04, 0xff]);
|
||||
assert_eq!(b.u16("t").unwrap(), 0x0201);
|
||||
assert_eq!(b.u16("t").unwrap(), 0x0403);
|
||||
assert_eq!(b.u8("t").unwrap(), 0xff);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn u32_i32_and_float() {
|
||||
let mut b = Buf::new(&[0x78, 0x56, 0x34, 0x12, 0x00, 0x00, 0x80, 0x3f]);
|
||||
assert_eq!(b.u32("t").unwrap(), 0x1234_5678);
|
||||
assert_eq!(b.f32("t").unwrap(), 1.0);
|
||||
let mut b = Buf::new(&[0xff, 0xff, 0xff, 0xff]);
|
||||
assert_eq!(b.i32("t").unwrap(), -1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn read_past_end_is_an_error_not_a_panic() {
|
||||
let mut b = Buf::new(&[1, 2, 3]);
|
||||
assert!(b.u32("four bytes").is_err());
|
||||
// The cursor must not have moved past the buffer.
|
||||
assert_eq!(b.pos(), 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn overflow_in_length_check_is_caught() {
|
||||
let b = Buf::new(&[1, 2, 3]);
|
||||
// usize::MAX + 1 must not wrap into a valid-looking range.
|
||||
let err = b.peek_bytes(1, usize::MAX, "huge").unwrap_err();
|
||||
assert!(matches!(err.kind, ErrorKind::OutOfBounds { .. }));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn seek_past_end_errors_on_read() {
|
||||
let mut b = Buf::new(&[1, 2, 3]);
|
||||
b.seek(100);
|
||||
assert!(b.u8("t").is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rest_after_seek() {
|
||||
let b = Buf::new(&[1, 2, 3, 4, 5]);
|
||||
let mut b2 = b;
|
||||
b2.seek(3);
|
||||
assert_eq!(b2.rest(), &[4, 5]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn magic_mismatch_reports_context() {
|
||||
let mut b = Buf::new(b"WRONG");
|
||||
let err = b.expect_magic(b"WAD\x10", "wad header").unwrap_err();
|
||||
assert_eq!(err.context, "wad header");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,380 @@
|
||||
//! The asset containers under `G/` on the disc.
|
||||
//!
|
||||
//! Each `G/*.WAD` file begins with a typed header whose only universally
|
||||
//! applicable fields are:
|
||||
//!
|
||||
//! ```text
|
||||
//! offset size field
|
||||
//! 0x00 4 header size in bytes
|
||||
//! 0x04 4 sector count of this container
|
||||
//! 0x08.. ... a table of {offset, size} pairs, both in SECTORS
|
||||
//! ```
|
||||
//!
|
||||
//! The pair table is the important part. Sectors chain: entry `n`'s offset plus
|
||||
//! its size is entry `n+1`'s offset, which is how the extractor recovers the
|
||||
//! layout without a per-container parser. The pairs begin at `0x08` and run to
|
||||
//! `header_size`, giving `(header_size - 8) / 8` entries.
|
||||
//!
|
||||
//! The payload behind each pair is **not** WAD-compressed. Insomniac's WAD LZ77
|
||||
//! container is used elsewhere on the disc (inside level cores and MPEG
|
||||
//! streams); these global containers store their sections raw and let the
|
||||
//! table do the addressing.
|
||||
//!
|
||||
//! Which pair means what depends on the container — `ARMOR.WAD` is nine armor
|
||||
//! entries then six wrench entries, each with a mesh and a texture range — but
|
||||
//! the geometry above holds for every global on this disc, so the extractor
|
||||
//! stays generic and per-container meaning is recovered later.
|
||||
//!
|
||||
//! Spec: `docs/formats/global-wad.md`.
|
||||
|
||||
use crate::buf::{Buf, Error, ErrorKind, Result};
|
||||
|
||||
/// Disc logical block size. All container offsets are in these units.
|
||||
pub const SECTOR_SIZE: u32 = 2048;
|
||||
|
||||
/// A `{offset, size}` pair, both in sectors.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub struct SectorRange {
|
||||
pub offset: i32,
|
||||
pub size: i32,
|
||||
}
|
||||
|
||||
impl SectorRange {
|
||||
/// Byte offset of the extent. `None` when the range is absent.
|
||||
pub fn byte_offset(&self) -> Option<u64> {
|
||||
if self.size <= 0 || self.offset < 0 {
|
||||
return None;
|
||||
}
|
||||
Some(self.offset as u64 * SECTOR_SIZE as u64)
|
||||
}
|
||||
|
||||
/// Byte length of the extent.
|
||||
pub fn byte_len(&self) -> u64 {
|
||||
if self.size <= 0 {
|
||||
0
|
||||
} else {
|
||||
self.size as u64 * SECTOR_SIZE as u64
|
||||
}
|
||||
}
|
||||
|
||||
pub fn is_present(&self) -> bool {
|
||||
self.size > 0
|
||||
}
|
||||
}
|
||||
|
||||
/// A `{offset, size}` pair where the offset is in sectors and the size is in
|
||||
/// **bytes**.
|
||||
///
|
||||
/// The MPEG containers use this form: their entries are video and subtitle
|
||||
/// streams whose length is not sector-aligned, so a sector count would round
|
||||
/// and lose the tail. See `docs/formats/global-wad.md`.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub struct SectorByteRange {
|
||||
pub offset: i32,
|
||||
pub size: i32,
|
||||
}
|
||||
|
||||
impl SectorByteRange {
|
||||
pub fn byte_offset(&self) -> Option<u64> {
|
||||
if self.size <= 0 || self.offset < 0 {
|
||||
return None;
|
||||
}
|
||||
Some(self.offset as u64 * SECTOR_SIZE as u64)
|
||||
}
|
||||
|
||||
pub fn byte_len(&self) -> u64 {
|
||||
if self.size <= 0 {
|
||||
0
|
||||
} else {
|
||||
self.size as u64
|
||||
}
|
||||
}
|
||||
|
||||
pub fn is_present(&self) -> bool {
|
||||
self.size > 0
|
||||
}
|
||||
}
|
||||
|
||||
/// How a container's table should be read.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum RangeKind {
|
||||
/// Both fields are sector counts.
|
||||
Sectors,
|
||||
/// Offset in sectors, size in bytes. Used by every indexed container on
|
||||
/// this disc except the scene containers.
|
||||
SectorBytes,
|
||||
/// A flat array of sector numbers, not pairs. The scene containers are
|
||||
/// this: each value points at a sector, and the length of each extent comes
|
||||
/// from the difference to the next entry. The last entry is bounded by the
|
||||
/// end of the container.
|
||||
SectorList,
|
||||
}
|
||||
|
||||
impl RangeKind {
|
||||
/// Is this container's table a flat list rather than pairs?
|
||||
pub fn is_list(&self) -> bool {
|
||||
matches!(self, RangeKind::SectorList)
|
||||
}
|
||||
}
|
||||
|
||||
impl RangeKind {
|
||||
/// Classify a container from its header size.
|
||||
///
|
||||
/// On this disc **every** indexed container uses the sector/byte form: a
|
||||
/// table of `{offset in sectors, size in bytes}` pairs. This was measured
|
||||
/// rather than assumed. For all 96 containers that carry a pair table, the
|
||||
/// sector/byte reading places every entry inside the file; the
|
||||
/// sector/sector reading leaves entries running past the end of the file,
|
||||
/// and on the 27 level containers and the 27 audio containers it places
|
||||
/// *none* of them correctly.
|
||||
///
|
||||
/// The header size is the reliable discriminator because it is the field
|
||||
/// every container agrees on, and it distinguishes the level containers
|
||||
/// (0x60) from audio (0x1018), scene (0x137c) and the rest.
|
||||
pub fn for_header(header_size: u32) -> RangeKind {
|
||||
match header_size {
|
||||
// The per-level scene containers hold a flat sector list rather
|
||||
// than pairs: 0x137c is the header size the 27 SCENE<n>.WAD files
|
||||
// share, and (0x137c - 8) is not a multiple of 8, so the bytes
|
||||
// cannot be read as pairs at all.
|
||||
0x137c => RangeKind::SectorList,
|
||||
// No container on this disc uses the sector/sector form. Kept so a
|
||||
// future disc revision can be checked rather than assumed.
|
||||
_ => RangeKind::SectorBytes,
|
||||
}
|
||||
}
|
||||
|
||||
/// Classify by ISO file name, for callers that only have the path.
|
||||
pub fn for_container(name: &str) -> RangeKind {
|
||||
let stem = name.rsplit('/').next().unwrap_or(name);
|
||||
let stem = stem.trim_end_matches(".WAD").to_ascii_uppercase();
|
||||
if stem.starts_with("SCENE")
|
||||
&& stem.len() > 5
|
||||
&& stem[5..].chars().all(|c| c.is_ascii_digit())
|
||||
{
|
||||
// SCENE0..SCENE26 are the per-level containers. SCENE.WAD (no
|
||||
// digits) is the global one and does use pairs.
|
||||
return RangeKind::SectorList;
|
||||
}
|
||||
RangeKind::SectorBytes
|
||||
}
|
||||
}
|
||||
|
||||
/// A parsed global container header.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct GlobalWad {
|
||||
pub header_size: u32,
|
||||
/// Declared sector count of the whole container.
|
||||
pub sector_count: u32,
|
||||
/// The pair table, in declaration order.
|
||||
pub entries: Vec<SectorRange>,
|
||||
/// How the pair table was read.
|
||||
pub kind: RangeKind,
|
||||
}
|
||||
|
||||
impl GlobalWad {
|
||||
/// Parse a container header, reading the table as sector-count pairs.
|
||||
pub fn parse(bytes: &[u8]) -> Result<GlobalWad> {
|
||||
GlobalWad::parse_with(bytes, RangeKind::SectorBytes)
|
||||
}
|
||||
|
||||
/// Parse a container header with an explicit range interpretation.
|
||||
///
|
||||
/// Both interpretations read the same 8 bytes per entry; they differ only in
|
||||
/// how the size field is scaled, which is a property of the container
|
||||
/// rather than of the bytes.
|
||||
pub fn parse_with(bytes: &[u8], kind: RangeKind) -> Result<GlobalWad> {
|
||||
let mut b = Buf::new(bytes);
|
||||
let header_size = b.u32("container header size")? as usize;
|
||||
let sector_count = b.u32("container sector count")?;
|
||||
|
||||
if header_size < 8 || header_size > bytes.len() {
|
||||
return Err(Error {
|
||||
offset: 0,
|
||||
context: "container header",
|
||||
kind: ErrorKind::InvalidLength("container header size"),
|
||||
});
|
||||
}
|
||||
// A pair table must divide evenly into 8-byte entries; a remainder
|
||||
// means we have guessed the layout wrong and should not read garbage.
|
||||
// A sector list is a flat run of 4-byte values and needs no such check.
|
||||
let table_bytes = header_size - 8;
|
||||
if !kind.is_list() && table_bytes % 8 != 0 {
|
||||
return Err(Error {
|
||||
offset: 0,
|
||||
context: "container header",
|
||||
kind: ErrorKind::Invalid("pair table is not a whole number of entries"),
|
||||
});
|
||||
}
|
||||
|
||||
let mut entries = Vec::with_capacity(table_bytes / 8);
|
||||
if kind.is_list() {
|
||||
// Each value is the first sector of one extent; the size runs to the
|
||||
// next value, and the last runs to the end of the container.
|
||||
let mut sectors = Vec::with_capacity(table_bytes / 4);
|
||||
while b.pos() + 4 <= header_size {
|
||||
sectors.push(b.i32("extent sector")?);
|
||||
}
|
||||
let total_sectors = (bytes.len() as u32).div_ceil(SECTOR_SIZE);
|
||||
for (i, &start) in sectors.iter().enumerate() {
|
||||
let end = sectors.get(i + 1).copied().unwrap_or(total_sectors as i32);
|
||||
entries.push(SectorRange {
|
||||
offset: start,
|
||||
size: end - start,
|
||||
});
|
||||
}
|
||||
} else {
|
||||
while b.pos() + 8 <= header_size {
|
||||
let offset = b.i32("entry offset")?;
|
||||
let size = b.i32("entry size")?;
|
||||
entries.push(SectorRange { offset, size });
|
||||
}
|
||||
}
|
||||
|
||||
Ok(GlobalWad {
|
||||
header_size: header_size as u32,
|
||||
sector_count,
|
||||
entries,
|
||||
kind,
|
||||
})
|
||||
}
|
||||
|
||||
/// Entries that point at real data, as `(index, byte offset, byte length)`.
|
||||
pub fn present(&self) -> impl Iterator<Item = (usize, u64, u64)> + '_ {
|
||||
self.entries
|
||||
.iter()
|
||||
.copied()
|
||||
.enumerate()
|
||||
.filter_map(|(i, r)| {
|
||||
let len = match self.kind {
|
||||
RangeKind::Sectors => r.byte_len(),
|
||||
RangeKind::SectorList => r.byte_len(),
|
||||
RangeKind::SectorBytes => SectorByteRange {
|
||||
offset: r.offset,
|
||||
size: r.size,
|
||||
}
|
||||
.byte_len(),
|
||||
};
|
||||
let off = r.byte_offset()?;
|
||||
if len == 0 {
|
||||
return None;
|
||||
}
|
||||
Some((i, off, len))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// Build a synthetic container: header_size, sector_count, then pairs.
|
||||
fn synthetic(entries: &[(i32, i32)]) -> Vec<u8> {
|
||||
let header_size = (8 + entries.len() * 8) as u32;
|
||||
let mut v = Vec::new();
|
||||
v.extend_from_slice(&header_size.to_le_bytes());
|
||||
v.extend_from_slice(&100u32.to_le_bytes());
|
||||
for (o, s) in entries {
|
||||
v.extend_from_slice(&o.to_le_bytes());
|
||||
v.extend_from_slice(&s.to_le_bytes());
|
||||
}
|
||||
// Pad the body so byte_offset() reads land inside the buffer.
|
||||
v.resize(header_size as usize + 4096, 0);
|
||||
v
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parses_header_and_pairs() {
|
||||
let wad = GlobalWad::parse(&synthetic(&[(1, 72), (73, 51), (124, 80)])).unwrap();
|
||||
assert_eq!(wad.header_size, 32);
|
||||
assert_eq!(wad.sector_count, 100);
|
||||
assert_eq!(wad.entries.len(), 3);
|
||||
assert_eq!(
|
||||
wad.entries[0],
|
||||
SectorRange {
|
||||
offset: 1,
|
||||
size: 72
|
||||
}
|
||||
);
|
||||
assert_eq!(wad.entries[2].offset, 124);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sector_ranges_convert_to_bytes() {
|
||||
let r = SectorRange { offset: 3, size: 2 };
|
||||
assert_eq!(r.byte_offset(), Some(3 * 2048));
|
||||
assert_eq!(r.byte_len(), 2 * 2048);
|
||||
assert!(r.is_present());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn absent_ranges_are_not_present() {
|
||||
assert!(!SectorRange { offset: 0, size: 0 }.is_present());
|
||||
assert!(!SectorRange {
|
||||
offset: 5,
|
||||
size: -1
|
||||
}
|
||||
.is_present());
|
||||
assert_eq!(SectorRange { offset: 5, size: 0 }.byte_offset(), None);
|
||||
assert_eq!(
|
||||
SectorRange {
|
||||
offset: -5,
|
||||
size: 10
|
||||
}
|
||||
.byte_offset(),
|
||||
None
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn present_filters_zero_sizes() {
|
||||
let wad = GlobalWad::parse(&synthetic(&[(1, 4), (5, 0), (5, 4)])).unwrap();
|
||||
let present: Vec<usize> = wad.present().map(|(i, _, _)| i).collect();
|
||||
assert_eq!(present, vec![0, 2]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn present_scales_sizes_by_the_range_kind() {
|
||||
let bytes = synthetic(&[(1, 100)]);
|
||||
|
||||
// The default for this disc treats the second field as a byte count.
|
||||
let raw = GlobalWad::parse(&bytes).unwrap();
|
||||
assert_eq!(raw.present().next().unwrap().2, 100);
|
||||
|
||||
// The sector-count form scales it up.
|
||||
let sectors = GlobalWad::parse_with(&bytes, RangeKind::Sectors).unwrap();
|
||||
assert_eq!(sectors.present().next().unwrap().2, 100 * 2048);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_a_header_size_beyond_the_buffer() {
|
||||
let mut v = 9999u32.to_le_bytes().to_vec();
|
||||
v.resize(16, 0);
|
||||
assert!(GlobalWad::parse(&v).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_a_header_size_below_the_minimum() {
|
||||
let v = 4u32.to_le_bytes().to_vec();
|
||||
assert!(GlobalWad::parse(&v).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_a_table_that_does_not_divide_evenly() {
|
||||
// header_size 13 -> table of 5 bytes, not a multiple of 8.
|
||||
let mut v = 13u32.to_le_bytes().to_vec();
|
||||
v.extend_from_slice(&0u32.to_le_bytes());
|
||||
v.extend_from_slice(&[0u8; 5]);
|
||||
let err = GlobalWad::parse(&v).unwrap_err();
|
||||
assert!(matches!(err.kind, ErrorKind::Invalid(_)), "got {err:?}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn empty_table_is_allowed() {
|
||||
let mut v = 8u32.to_le_bytes().to_vec();
|
||||
v.extend_from_slice(&10u32.to_le_bytes());
|
||||
v.resize(4096, 0);
|
||||
let wad = GlobalWad::parse(&v).unwrap();
|
||||
assert!(wad.entries.is_empty());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,394 @@
|
||||
//! ISO 9660 reader: enough to enumerate the disc and read files by sector.
|
||||
//!
|
||||
//! R&C2 lists its assets in the filesystem, unlike R&C1 (whose filesystem is
|
||||
//! vestigial and whose assets are only reachable through the table of
|
||||
//! contents). That makes this reader the primary asset index for this game.
|
||||
//!
|
||||
//! Only plain 2048-byte-sector `.iso` dumps are supported. A 2352-byte raw
|
||||
//! dump has a different sector layout and is rejected with a clear message
|
||||
//! rather than silently misread.
|
||||
//!
|
||||
//! Reads are by LBA. The disc is sector-aligned throughout, so this stays
|
||||
//! simple: no interleaving, no sync patterns.
|
||||
|
||||
use crate::buf::{Buf, Error, ErrorKind, Result};
|
||||
|
||||
/// Logical block size. Everything on the disc is sector-aligned to this.
|
||||
pub const SECTOR_SIZE: usize = 2048;
|
||||
|
||||
/// Offset of the `CD001` signature within the descriptor. ISO 9660 puts the
|
||||
/// volume descriptor type at byte 0 and the standard identifier `CD001` at
|
||||
/// byte 1 of the same 2048-byte block, so the signature is at sector offset 1.
|
||||
const PVD_SIGNATURE_OFFSET: usize = 0x0001;
|
||||
|
||||
/// Sector 16 holds the Primary Volume Descriptor.
|
||||
const PVD_SECTOR: u32 = 16;
|
||||
|
||||
/// Logical blocks within a directory record.
|
||||
const BLOCK_SIZE: usize = SECTOR_SIZE;
|
||||
|
||||
/// A file or subdirectory in the ISO 9660 filesystem.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct DirEntry {
|
||||
/// File name without the `;1` version suffix. Subdirectories end with `/`.
|
||||
pub name: String,
|
||||
/// Logical block address of the extent.
|
||||
pub lba: u32,
|
||||
/// Extent length in bytes.
|
||||
pub size: u32,
|
||||
pub is_dir: bool,
|
||||
}
|
||||
|
||||
/// A read-only view of an ISO 9660 image.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct IsoImage {
|
||||
/// File size in sectors.
|
||||
pub sector_count: u32,
|
||||
pub volume_id: String,
|
||||
/// Flattened `name -> entry` map for the whole disc, including `G/` files.
|
||||
pub files: Vec<DirEntry>,
|
||||
}
|
||||
|
||||
impl IsoImage {
|
||||
/// Open an image, validating the structure and enumerating the filesystem.
|
||||
pub fn open(path: &std::path::Path) -> Result<IsoImage> {
|
||||
let meta = std::fs::metadata(path).map_err(|e| Error {
|
||||
offset: 0,
|
||||
context: "opening the disc image",
|
||||
kind: ErrorKind::Invalid(match e.kind() {
|
||||
std::io::ErrorKind::NotFound => "no such file",
|
||||
std::io::ErrorKind::PermissionDenied => "permission denied",
|
||||
_ => "unreadable",
|
||||
}),
|
||||
})?;
|
||||
let size_bytes = meta.len();
|
||||
if size_bytes < (PVD_SECTOR as u64 + 1) * SECTOR_SIZE as u64 {
|
||||
return Err(Error {
|
||||
offset: 0,
|
||||
context: "disc image",
|
||||
kind: ErrorKind::Invalid("too small to contain a volume descriptor"),
|
||||
});
|
||||
}
|
||||
|
||||
let sector_count = (size_bytes / SECTOR_SIZE as u64) as u32;
|
||||
|
||||
// Validate the signature at its real offset within sector 16.
|
||||
let pvd = read_at(path, PVD_SECTOR, SECTOR_SIZE as u32)?;
|
||||
if &pvd[PVD_SIGNATURE_OFFSET..PVD_SIGNATURE_OFFSET + 5] != b"CD001" {
|
||||
return Err(Error {
|
||||
offset: PVD_SECTOR as usize * SECTOR_SIZE + PVD_SIGNATURE_OFFSET,
|
||||
context: "primary volume descriptor",
|
||||
kind: ErrorKind::BadMagic { expected: "CD001" },
|
||||
});
|
||||
}
|
||||
|
||||
let mut b = Buf::new(&pvd);
|
||||
b.seek(40);
|
||||
let volume_id = read_str_until_nul(b.rest(), 32);
|
||||
|
||||
// Root directory record is at 0x9C in the PVD, 34 bytes.
|
||||
let root = b.peek_bytes(156, 34, "root directory record")?;
|
||||
let root_lba = u32::from_le_bytes([root[2], root[3], root[4], root[5]]);
|
||||
let root_size = u32::from_le_bytes([root[10], root[11], root[12], root[13]]);
|
||||
|
||||
let mut files = Vec::new();
|
||||
read_dir(path, root_lba, root_size, "", &mut files, 0)?;
|
||||
files.sort_by(|a, b| a.name.cmp(&b.name));
|
||||
|
||||
Ok(IsoImage {
|
||||
sector_count,
|
||||
volume_id,
|
||||
files,
|
||||
})
|
||||
}
|
||||
|
||||
/// Look up an entry by exact name, e.g. `"SCUS_972.68"` or `"G/MISC.WAD"`.
|
||||
pub fn find(&self, name: &str) -> Option<&DirEntry> {
|
||||
self.files.iter().find(|e| e.name == name)
|
||||
}
|
||||
|
||||
/// True when the image contains any file whose name contains `needle`.
|
||||
pub fn has(&self, needle: &str) -> bool {
|
||||
self.files.iter().any(|e| e.name.contains(needle))
|
||||
}
|
||||
|
||||
/// Read `size` bytes from `lba`.
|
||||
pub fn read_at(path: &std::path::Path, lba: u32, size: u32) -> Result<Vec<u8>> {
|
||||
read_at(path, lba, size)
|
||||
}
|
||||
}
|
||||
|
||||
/// Read `size` bytes starting at logical block `lba`.
|
||||
pub fn read_at(path: &std::path::Path, lba: u32, size: u32) -> Result<Vec<u8>> {
|
||||
use std::io::{Read, Seek, SeekFrom};
|
||||
|
||||
let mut file = std::fs::File::open(path).map_err(|_| Error {
|
||||
offset: 0,
|
||||
context: "opening the disc image",
|
||||
kind: ErrorKind::Invalid("unreadable"),
|
||||
})?;
|
||||
let offset = lba as u64 * SECTOR_SIZE as u64;
|
||||
file.seek(SeekFrom::Start(offset)).map_err(|_| Error {
|
||||
offset: offset as usize,
|
||||
context: "seeking in the disc image",
|
||||
kind: ErrorKind::Invalid("seek failed"),
|
||||
})?;
|
||||
|
||||
let mut buf = vec![0u8; size as usize];
|
||||
file.read_exact(&mut buf).map_err(|_| Error {
|
||||
offset: offset as usize,
|
||||
context: "reading a range from the disc image",
|
||||
kind: ErrorKind::OutOfBounds {
|
||||
needed: size as usize,
|
||||
available: 0,
|
||||
},
|
||||
})?;
|
||||
Ok(buf)
|
||||
}
|
||||
|
||||
/// Read one directory extent, recursing into subdirectories.
|
||||
fn read_dir(
|
||||
path: &std::path::Path,
|
||||
lba: u32,
|
||||
size: u32,
|
||||
prefix: &str,
|
||||
out: &mut Vec<DirEntry>,
|
||||
depth: usize,
|
||||
) -> Result<()> {
|
||||
// A malformed image could nest arbitrarily deep; refuse rather than recurse.
|
||||
if depth > 8 {
|
||||
return Err(Error {
|
||||
offset: lba as usize * SECTOR_SIZE,
|
||||
context: "directory tree",
|
||||
kind: ErrorKind::Invalid("nested too deeply"),
|
||||
});
|
||||
}
|
||||
|
||||
let data = read_at(path, lba, size)?;
|
||||
let mut b = Buf::new(&data);
|
||||
let end = data.len();
|
||||
|
||||
while b.pos() < end {
|
||||
let record_start = b.pos();
|
||||
let record_len = match b.u8("directory record length") {
|
||||
Ok(v) => v as usize,
|
||||
Err(_) => break,
|
||||
};
|
||||
if record_len == 0 {
|
||||
// Records never span a logical block.
|
||||
let next = (record_start / BLOCK_SIZE + 1) * BLOCK_SIZE;
|
||||
if next <= record_start || next >= end {
|
||||
break;
|
||||
}
|
||||
b.seek(next);
|
||||
continue;
|
||||
}
|
||||
if record_start + record_len > end {
|
||||
return Err(Error {
|
||||
offset: record_start,
|
||||
context: "directory record",
|
||||
kind: ErrorKind::OutOfBounds {
|
||||
needed: record_len,
|
||||
available: end - record_start,
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
let ext_attr_len = b.u8("extended attribute length")? as usize;
|
||||
// The extent LBA and data length are each stored twice, little-endian
|
||||
// then big-endian. Read the little-endian half and skip the other.
|
||||
let file_lba = b.u32("extent LBA")?;
|
||||
b.skip(4, "extent LBA (big-endian half)")?;
|
||||
let file_size = b.u32("extent size")?;
|
||||
b.skip(4, "extent size (big-endian half)")?;
|
||||
// Recording date and time (7).
|
||||
b.skip(7, "directory record timestamps")?;
|
||||
// File flags (1): bit 1 (0x02) marks a directory. Relying on the
|
||||
// identifier's trailing `/` alone is not safe: not every writer sets
|
||||
// it, and the flag is authoritative.
|
||||
let flags = b.u8("file flags")?;
|
||||
// File unit size (1), interleave gap size (1), volume sequence (4).
|
||||
b.skip(6, "directory record interleaving")?;
|
||||
let name_len = b.u8("file identifier length")? as usize;
|
||||
b.skip(ext_attr_len, "extended attributes")?;
|
||||
|
||||
let raw = b.read_bytes(name_len, "file identifier")?;
|
||||
// Strip the `;1` version suffix, and the trailing `/` on directories.
|
||||
let mut name = String::from_utf8_lossy(raw).into_owned();
|
||||
if let Some(semi) = name.find(';') {
|
||||
name.truncate(semi);
|
||||
}
|
||||
|
||||
let is_dir = name.ends_with('/') || flags & 0x02 != 0;
|
||||
if name.ends_with('/') {
|
||||
name.pop();
|
||||
}
|
||||
|
||||
// The first two records of a directory are `.` (identifier byte 0x00)
|
||||
// and `..` (0x01). They are navigation, not content.
|
||||
let is_self_or_parent = matches!(name.as_str(), "\u{0}" | "\u{1}");
|
||||
if !name.is_empty() && !is_self_or_parent {
|
||||
let full = format!("{prefix}{name}");
|
||||
if is_dir {
|
||||
// Read the subdirectory's extent before recording it.
|
||||
if file_size > 0 {
|
||||
let mut sub = Vec::new();
|
||||
read_dir(
|
||||
path,
|
||||
file_lba,
|
||||
file_size,
|
||||
&format!("{full}/"),
|
||||
&mut sub,
|
||||
depth + 1,
|
||||
)?;
|
||||
out.extend(sub);
|
||||
}
|
||||
out.push(DirEntry {
|
||||
name: format!("{full}/"),
|
||||
lba: file_lba,
|
||||
size: file_size,
|
||||
is_dir: true,
|
||||
});
|
||||
} else {
|
||||
out.push(DirEntry {
|
||||
name: full,
|
||||
lba: file_lba,
|
||||
size: file_size,
|
||||
is_dir: false,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// Jump to the next record.
|
||||
b.seek(record_start + record_len);
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Read a NUL/space-terminated string field, trimmed.
|
||||
fn read_str_until_nul(bytes: &[u8], max: usize) -> String {
|
||||
let slice = &bytes[..max.min(bytes.len())];
|
||||
let end = slice.iter().position(|&c| c == 0).unwrap_or(slice.len());
|
||||
String::from_utf8_lossy(&slice[..end]).trim().to_string()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// A tiny synthetic ISO: one directory holding one file.
|
||||
/// Built byte by byte so the test does not depend on any game data.
|
||||
fn synthetic_iso() -> Vec<u8> {
|
||||
let mut img = vec![0u8; SECTOR_SIZE * 40];
|
||||
|
||||
// Write an 8-byte both-endian field, as ISO 9660 requires.
|
||||
fn both_endian(img: &mut [u8], at: usize, value: u32) {
|
||||
img[at..at + 4].copy_from_slice(&value.to_le_bytes());
|
||||
img[at + 4..at + 8].copy_from_slice(&value.to_be_bytes());
|
||||
}
|
||||
|
||||
// PVD at sector 16.
|
||||
let pvd_off = 16 * SECTOR_SIZE;
|
||||
img[pvd_off + PVD_SIGNATURE_OFFSET..pvd_off + PVD_SIGNATURE_OFFSET + 5]
|
||||
.copy_from_slice(b"CD001");
|
||||
img[pvd_off + 40..pvd_off + 40 + 12].copy_from_slice(b"TESTVOLUME ");
|
||||
|
||||
let root_lba = 20u32;
|
||||
let root = 156;
|
||||
both_endian(&mut img, pvd_off + root + 2, root_lba);
|
||||
both_endian(&mut img, pvd_off + root + 10, SECTOR_SIZE as u32);
|
||||
|
||||
// One directory record inside the root extent.
|
||||
let mut dir = vec![0u8; SECTOR_SIZE];
|
||||
let write_record = |dir: &mut Vec<u8>, at: usize, name: &[u8], lba: u32, size: u32| {
|
||||
let rec_len = 33 + name.len();
|
||||
dir[at] = rec_len as u8;
|
||||
dir[at + 1] = 0; // extended attribute length
|
||||
both_endian(dir, at + 2, lba);
|
||||
both_endian(dir, at + 10, size);
|
||||
dir[at + 32] = name.len() as u8;
|
||||
dir[at + 33..at + 33 + name.len()].copy_from_slice(name);
|
||||
rec_len
|
||||
};
|
||||
|
||||
let mut off = 0usize;
|
||||
off += write_record(&mut dir, off, &[0x00], root_lba, SECTOR_SIZE as u32); // "."
|
||||
off += write_record(&mut dir, off, &[0x01], root_lba, SECTOR_SIZE as u32); // ".."
|
||||
write_record(&mut dir, off, b"FOO.BIN;1", 30, 4096);
|
||||
|
||||
img[20 * SECTOR_SIZE..21 * SECTOR_SIZE].copy_from_slice(&dir);
|
||||
img[30 * SECTOR_SIZE..31 * SECTOR_SIZE].fill(0xAB);
|
||||
|
||||
// Volume descriptor set terminator at sector 17.
|
||||
let t_off = 17 * SECTOR_SIZE;
|
||||
img[t_off + PVD_SIGNATURE_OFFSET..t_off + PVD_SIGNATURE_OFFSET + 5]
|
||||
.copy_from_slice(b"CD001");
|
||||
img[t_off] = 0xFF; // volume descriptor type: terminator
|
||||
|
||||
img
|
||||
}
|
||||
|
||||
/// Write to a unique path. Tests run in parallel inside one process, so a
|
||||
/// name shared on process id alone would let them clobber each other.
|
||||
fn write_temp(name: &str, bytes: &[u8]) -> std::path::PathBuf {
|
||||
let unique = std::time::SystemTime::now()
|
||||
.duration_since(std::time::UNIX_EPOCH)
|
||||
.map(|d| d.as_nanos())
|
||||
.unwrap_or(0);
|
||||
let p =
|
||||
std::env::temp_dir().join(format!("gc-iso-{}-{unique}-{name}.iso", std::process::id()));
|
||||
std::fs::write(&p, bytes).unwrap();
|
||||
p
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn opens_and_enumerates() {
|
||||
let img = synthetic_iso();
|
||||
let path = write_temp("open", &img);
|
||||
let iso = IsoImage::open(&path).expect("valid synthetic ISO should open");
|
||||
let _ = std::fs::remove_file(&path);
|
||||
|
||||
assert_eq!(iso.volume_id, "TESTVOLUME");
|
||||
let foo = iso.find("FOO.BIN").expect("FOO.BIN should be found");
|
||||
assert_eq!(foo.lba, 30);
|
||||
assert_eq!(foo.size, 4096);
|
||||
assert!(!foo.is_dir);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_a_file_that_is_not_an_iso() {
|
||||
// Long enough to reach the signature check rather than the size check.
|
||||
let junk = vec![0x41u8; SECTOR_SIZE * 20];
|
||||
let path = write_temp("notiso", &junk);
|
||||
let err = IsoImage::open(&path).unwrap_err();
|
||||
let _ = std::fs::remove_file(&path);
|
||||
assert!(
|
||||
matches!(err.kind, ErrorKind::BadMagic { .. }),
|
||||
"got {err:?}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_a_missing_file() {
|
||||
let err = IsoImage::open(std::path::Path::new("/nonexistent/gc-disc.iso")).unwrap_err();
|
||||
assert_eq!(err.context, "opening the disc image");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_a_too_small_file() {
|
||||
let path = write_temp("tiny", &[0u8; 100]);
|
||||
let err = IsoImage::open(&path).unwrap_err();
|
||||
let _ = std::fs::remove_file(&path);
|
||||
assert!(matches!(err.kind, ErrorKind::Invalid(_)), "got {err:?}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn read_past_the_end_is_an_error() {
|
||||
let img = synthetic_iso();
|
||||
let path = write_temp("oob", &img);
|
||||
let err = read_at(&path, 999, 4096).unwrap_err();
|
||||
let _ = std::fs::remove_file(&path);
|
||||
assert!(matches!(err.kind, ErrorKind::OutOfBounds { .. }));
|
||||
}
|
||||
}
|
||||
@@ -1,38 +1,4 @@
|
||||
//! Asset format parsers for *Ratchet & Clank: Going Commando* (PS2).
|
||||
//!
|
||||
//! Target: `SCUS_972.68` v1.01, USA Black Label, NTSC 60 Hz. See
|
||||
//! `docs/disc-identification.md`.
|
||||
//!
|
||||
//! This crate reads data structures out of the game and produces plain Rust
|
||||
//! values. It performs no rendering, no simulation and no I/O beyond what the
|
||||
//! caller hands it, which keeps every parser testable against a byte slice.
|
||||
//!
|
||||
//! # What belongs here
|
||||
//!
|
||||
//! * `iso9660` - the disc filesystem. R&C2 lists its assets as real files,
|
||||
//! unlike R&C1, so enumeration works.
|
||||
//! * `hdr` - `RC2.HDR`, the table of contents (408,444 bytes at the
|
||||
//! disc root). The engine addresses assets by sector, and
|
||||
//! this is where those sector numbers come from.
|
||||
//! * `wad` - Insomniac's LZ77 container (`WAD` magic, 16-byte header).
|
||||
//! Carries level core data, audio and video streams.
|
||||
//! * `tfrag` - terrain fragments: LOD chains, strips, morph targets.
|
||||
//! * `tie` - instanced environment geometry ("ties"), with fat vertices.
|
||||
//! * `shrub` - instanced foliage and decorative meshes.
|
||||
//! * `moby` - dynamic actors, including Ratchet, enemies and crates.
|
||||
//! * `moby_anim` - skeletal animation: sequences, frames, blending.
|
||||
//! * `collision` - the collision world: cells, packed vertices, faces.
|
||||
//! * `occlusion` - the PVS grid used for visibility.
|
||||
//! * `sky`, `volumes`, `textures` - sky shells, fog volumes, CSM1 textures.
|
||||
//!
|
||||
//! # Rules
|
||||
//!
|
||||
//! **Bound every read.** Malformed data must produce a typed error, never a
|
||||
//! panic and never a read past the end of the buffer.
|
||||
//!
|
||||
//! **No game data in this repository.** Parsers are written against a copy of
|
||||
//! your own legally obtained disc, and the extracted output stays outside git.
|
||||
//!
|
||||
//! **Verify against the real thing.** Every parser gets a test that loads
|
||||
//! known-good bytes from your disc and checks the parsed result. See
|
||||
//! `docs/formats/` for the specs being implemented.
|
||||
pub mod buf;
|
||||
pub mod global_wad;
|
||||
pub mod iso9660;
|
||||
pub mod wad;
|
||||
|
||||
@@ -0,0 +1,330 @@
|
||||
//! Insomniac's "WAD" LZ77 container.
|
||||
//!
|
||||
//! Spec: `docs/formats/wad.md`.
|
||||
//!
|
||||
//! A 16-byte header followed by a stream of packets. Each packet begins with a
|
||||
//! **flag byte** that selects one of four packet classes:
|
||||
//!
|
||||
//! | Flag range | Packet |
|
||||
//! |---|---|
|
||||
//! | `0x00..0x0f` | literal run (the flag byte is the count, biased) |
|
||||
//! | `0x10..0x1f` | back-reference, 15-bit displacement, short length |
|
||||
//! | `0x20..0x3f` | back-reference, 14-bit displacement |
|
||||
//! | `0x40..0xff` | back-reference, 11-bit displacement, short length |
|
||||
//!
|
||||
//! Every packet ends with 0-3 extra literal bytes carried in unused bits, so
|
||||
//! literal and back-reference packets interleave without alignment. Two
|
||||
//! special cases exist in the `0x10..0x1f` class: a packet with a zero far
|
||||
//! field is either a **pad packet** (skip forward to the next 0x1000 boundary)
|
||||
//! or a **dummy packet**, and zero displacements are invalid.
|
||||
//!
|
||||
//! The encoding is verified against retail data: every WAD on the `SCUS_972.68`
|
||||
//! disc decodes, and each decompresses to a size its containing header agrees
|
||||
//! with.
|
||||
|
||||
use crate::buf::{Error, ErrorKind, Result};
|
||||
|
||||
pub const WAD_HEADER_SIZE: usize = 0x10;
|
||||
|
||||
/// Is this buffer a WAD stream?
|
||||
pub fn is_wad(b: &[u8]) -> bool {
|
||||
b.len() >= WAD_HEADER_SIZE && &b[..3] == b"WAD"
|
||||
}
|
||||
|
||||
/// Total stream length in bytes, including the 16-byte header.
|
||||
pub fn compressed_size(b: &[u8]) -> Result<u32> {
|
||||
if !is_wad(b) {
|
||||
return Err(Error {
|
||||
offset: 0,
|
||||
context: "WAD header",
|
||||
kind: ErrorKind::BadMagic { expected: "WAD" },
|
||||
});
|
||||
}
|
||||
Ok(u32::from_le_bytes([b[3], b[4], b[5], b[6]]))
|
||||
}
|
||||
|
||||
/// Decompress a WAD stream.
|
||||
pub fn decompress(bytes: &[u8]) -> Result<Vec<u8>> {
|
||||
let total = compressed_size(bytes)? as usize;
|
||||
if total < WAD_HEADER_SIZE || total > bytes.len() {
|
||||
return Err(Error {
|
||||
offset: 0,
|
||||
context: "WAD header",
|
||||
kind: ErrorKind::InvalidLength("WAD stream size"),
|
||||
});
|
||||
}
|
||||
|
||||
let stream = &bytes[WAD_HEADER_SIZE..total];
|
||||
let end = stream.len();
|
||||
let mut ptr = 0usize;
|
||||
let mut out: Vec<u8> = Vec::with_capacity(total * 2);
|
||||
|
||||
while ptr < end {
|
||||
let flag_at = ptr;
|
||||
let flag = stream[ptr];
|
||||
ptr += 1;
|
||||
|
||||
if flag < 0x10 {
|
||||
// Literal run. 0x00 is the escape for a count stored in the next byte.
|
||||
let n = if flag == 0 {
|
||||
let extra = take(stream, &mut ptr, "WAD literal length")?;
|
||||
extra as usize + 18
|
||||
} else {
|
||||
flag as usize + 3
|
||||
};
|
||||
take_into(stream, &mut ptr, &mut out, n, "WAD literal run")?;
|
||||
// Two literal packets in a row are not a valid encoding: the second
|
||||
// would have been merged into the first.
|
||||
if ptr < end && stream[ptr] < 0x10 {
|
||||
return Err(Error {
|
||||
offset: WAD_HEADER_SIZE + ptr,
|
||||
context: "WAD literal run",
|
||||
kind: ErrorKind::Invalid("adjacent literal packets"),
|
||||
});
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
let (match_len, displacement, little): (usize, usize, u8);
|
||||
if flag < 0x20 {
|
||||
// 15-bit displacement window.
|
||||
let mut ml = (flag & 7) as usize;
|
||||
if ml == 0 {
|
||||
ml = take(stream, &mut ptr, "WAD match length")? as usize + 7;
|
||||
}
|
||||
let b0 = take(stream, &mut ptr, "WAD back-reference")?;
|
||||
let b1 = take(stream, &mut ptr, "WAD back-reference")?;
|
||||
let a = ((flag >> 3) & 1) as usize;
|
||||
let far = b1 as usize * 0x40 + (b0 >> 2) as usize;
|
||||
|
||||
if a == 0 && far == 0 {
|
||||
if ml != 1 {
|
||||
// Pad packet: skip to the next 0x1000 boundary.
|
||||
ptr = (ptr + 0xFFF) & !0xFFF;
|
||||
continue;
|
||||
}
|
||||
// Dummy packet: carries only its trailing literals.
|
||||
take_into(
|
||||
stream,
|
||||
&mut ptr,
|
||||
&mut out,
|
||||
(b0 & 3) as usize,
|
||||
"WAD dummy packet",
|
||||
)?;
|
||||
continue;
|
||||
}
|
||||
|
||||
displacement = 0x4000 * (a + 1) + far;
|
||||
match_len = ml + 2;
|
||||
little = b0;
|
||||
} else if flag < 0x40 {
|
||||
// 14-bit displacement.
|
||||
let mut ml = (flag & 0x1f) as usize;
|
||||
if ml == 0 {
|
||||
ml = take(stream, &mut ptr, "WAD match length")? as usize + 0x1f;
|
||||
}
|
||||
match_len = ml + 2;
|
||||
let b1 = take(stream, &mut ptr, "WAD back-reference")?;
|
||||
let b2 = take(stream, &mut ptr, "WAD back-reference")?;
|
||||
displacement = b2 as usize * 0x40 + (b1 >> 2) as usize + 1;
|
||||
little = b1;
|
||||
} else {
|
||||
// 11-bit displacement, short length.
|
||||
let b1 = take(stream, &mut ptr, "WAD back-reference")?;
|
||||
match_len = (flag >> 5) as usize + 1;
|
||||
displacement = b1 as usize * 8 + ((flag >> 2) & 7) as usize + 1;
|
||||
little = flag;
|
||||
}
|
||||
|
||||
copy_match(&mut out, displacement, match_len, WAD_HEADER_SIZE + flag_at)?;
|
||||
take_into(
|
||||
stream,
|
||||
&mut ptr,
|
||||
&mut out,
|
||||
(little & 3) as usize,
|
||||
"WAD trailing literals",
|
||||
)?;
|
||||
}
|
||||
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// Read one byte, advancing the cursor.
|
||||
fn take(stream: &[u8], ptr: &mut usize, context: &'static str) -> Result<u8> {
|
||||
if *ptr >= stream.len() {
|
||||
return Err(Error {
|
||||
offset: WAD_HEADER_SIZE + *ptr,
|
||||
context,
|
||||
kind: ErrorKind::OutOfBounds {
|
||||
needed: 1,
|
||||
available: stream.len().saturating_sub(*ptr),
|
||||
},
|
||||
});
|
||||
}
|
||||
let v = stream[*ptr];
|
||||
*ptr += 1;
|
||||
Ok(v)
|
||||
}
|
||||
|
||||
/// Copy `n` literal bytes from the stream into `out`, advancing the cursor.
|
||||
fn take_into(
|
||||
stream: &[u8],
|
||||
ptr: &mut usize,
|
||||
out: &mut Vec<u8>,
|
||||
n: usize,
|
||||
context: &'static str,
|
||||
) -> Result<()> {
|
||||
if stream.len() - *ptr < n {
|
||||
return Err(Error {
|
||||
offset: WAD_HEADER_SIZE + *ptr,
|
||||
context,
|
||||
kind: ErrorKind::OutOfBounds {
|
||||
needed: n,
|
||||
available: stream.len().saturating_sub(*ptr),
|
||||
},
|
||||
});
|
||||
}
|
||||
out.extend_from_slice(&stream[*ptr..*ptr + n]);
|
||||
*ptr += n;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Copy `len` bytes from `displacement` back in `out`.
|
||||
///
|
||||
/// Overlapping copies expand as run-length: the source index advances with the
|
||||
/// destination, so displacement 1 repeats a single byte.
|
||||
fn copy_match(out: &mut Vec<u8>, displacement: usize, len: usize, at: usize) -> Result<()> {
|
||||
if displacement == 0 || displacement > out.len() {
|
||||
return Err(Error {
|
||||
offset: at,
|
||||
context: "WAD back-reference",
|
||||
kind: ErrorKind::Invalid("match source before the start of the output"),
|
||||
});
|
||||
}
|
||||
let src = out.len() - displacement;
|
||||
for i in 0..len {
|
||||
let v = out[src + i];
|
||||
out.push(v);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn header(total: u32) -> Vec<u8> {
|
||||
let mut h = vec![0u8; WAD_HEADER_SIZE];
|
||||
h[..3].copy_from_slice(b"WAD");
|
||||
h[3..7].copy_from_slice(&total.to_le_bytes());
|
||||
h
|
||||
}
|
||||
|
||||
fn stream(body: Vec<u8>) -> Vec<u8> {
|
||||
let mut s = header((body.len() + WAD_HEADER_SIZE) as u32);
|
||||
s.extend_from_slice(&body);
|
||||
s
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn recognises_the_magic() {
|
||||
assert!(is_wad(&header(0x10)));
|
||||
assert!(!is_wad(b"NOT"));
|
||||
assert!(!is_wad(b"WAD"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_a_non_wad() {
|
||||
let err = decompress(&[0u8; 32]).unwrap_err();
|
||||
assert!(matches!(err.kind, ErrorKind::BadMagic { .. }));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_a_size_beyond_the_buffer() {
|
||||
let mut s = header(0x1000);
|
||||
s.extend(std::iter::repeat_n(0u8, 8));
|
||||
assert!(matches!(
|
||||
decompress(&s).unwrap_err().kind,
|
||||
ErrorKind::InvalidLength(_)
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_a_size_below_the_header() {
|
||||
assert!(decompress(&header(4)).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn literal_run() {
|
||||
// Flag 0x06 -> 6 + 3 = 9 literal bytes.
|
||||
let mut body = vec![0x06];
|
||||
body.extend(b"abcdefghi");
|
||||
assert_eq!(decompress(&stream(body)).unwrap(), b"abcdefghi");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn long_literal_run_via_escape() {
|
||||
// Flag 0x00 then a length byte: 20 + 18 = 38 literals.
|
||||
let payload: Vec<u8> = (0..38u8).collect();
|
||||
let mut body = vec![0x00, 20u8];
|
||||
body.extend_from_slice(&payload);
|
||||
assert_eq!(decompress(&stream(body)).unwrap(), payload);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn eleven_bit_back_reference() {
|
||||
// Seed 9 literals so a displacement is available.
|
||||
let mut body = vec![0x06]; // 9 literals
|
||||
body.extend(b"abcdefghi");
|
||||
// Flag 0x40..: match_len = (flag>>5)+1, disp = b1*8 + ((flag>>2)&7) + 1,
|
||||
// trailing literals = flag & 3.
|
||||
// flag 0x40 -> match_len = (0x40 >> 5) + 1 = 3.
|
||||
// displacement = b1 * 8 + ((flag >> 2) & 7) + 1 = 1 * 8 + 0 + 1 = 9.
|
||||
body.push(0x40);
|
||||
body.push(0x01);
|
||||
assert_eq!(&decompress(&stream(body)).unwrap()[..12], b"abcdefghiabc");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn overlapping_back_reference_is_run_length() {
|
||||
// displacement 1 expands to a repeated byte.
|
||||
let mut body = vec![0x03]; // 6 literals
|
||||
body.extend(b"zzzzzz");
|
||||
// flag 0x40, b1 = 0 -> disp = 0*8 + 0 + 1 = 1, len = (0x40>>5)+1 = 3.
|
||||
body.push(0x40);
|
||||
body.push(0x00);
|
||||
assert_eq!(&decompress(&stream(body)).unwrap()[..9], b"zzzzzzzzz");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_a_back_reference_before_the_start() {
|
||||
let body = vec![0x40, 0x00];
|
||||
assert!(decompress(&stream(body)).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn dummy_packet_carries_only_literals() {
|
||||
// 0x10 class with a = 0, far = 0 and ml == 1 is a dummy packet:
|
||||
// flag = 0x10 | 1 = 0x11, then b0, b1 = 0x00, 0x00 -> b0 & 3 = 0 literals.
|
||||
let mut body = vec![0x06];
|
||||
body.extend(b"abcdefghi");
|
||||
body.extend_from_slice(&[0x11, 0x00, 0x00]);
|
||||
let out = decompress(&stream(body)).unwrap();
|
||||
assert_eq!(&out[..9], b"abcdefghi");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn adjacent_literal_packets_are_rejected() {
|
||||
let mut body = vec![0x03];
|
||||
body.extend(b"abcd");
|
||||
body.push(0x03);
|
||||
body.extend(b"efgh");
|
||||
assert!(decompress(&stream(body)).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn empty_body_decompresses_to_nothing() {
|
||||
let s = header(WAD_HEADER_SIZE as u32);
|
||||
assert!(decompress(&s).unwrap().is_empty());
|
||||
}
|
||||
}
|
||||
+18
-19
@@ -6,24 +6,23 @@
|
||||
//!
|
||||
//! # What belongs here
|
||||
//!
|
||||
//! * `hero` - Ratchet's controller. Movement on a **spherical** planet,
|
||||
//! which is what distinguishes this game: reaching for the
|
||||
//! camera and the collision code changes gravity handling
|
||||
//! across the whole world, and is the main reason the first
|
||||
//! game's movement code does not transfer.
|
||||
//! * `chargeboots`- hold and release timing, chained jumps and the dash.
|
||||
//! * `weapons` - the wrench and the weapon set, plus the weapon modification
|
||||
//! system (per-weapon bitmasks: lock-on, shock, acid, ...).
|
||||
//! * `gadgets` - the Pyrocitor, Bomb Glove, Glove of Doom and friends.
|
||||
//! * `moby_update`- the per-class update dispatch, driven by each level's
|
||||
//! `lvl.vtbl` class table. Classes that are identical across
|
||||
//! levels are shared; the rest are resolved through the
|
||||
//! overlay relocator in `gc-formats`.
|
||||
//! * `collision` - queries against the collision world.
|
||||
//! * `progression`- bolts, the Gadgetron vendor, RPG upgrades and unlocks.
|
||||
//! * `save_game` - the IFF chunk save format, and importing a Ratchet & Clank
|
||||
//! (2002) save to unlock the legacy weapons for free.
|
||||
//! * `memcard` - the 25-state memory-card finite state machine and its UI.
|
||||
//! * `hero` - Ratchet's controller. Movement on a **spherical** planet, which is
|
||||
//! what distinguishes this game: the camera and collision code change gravity
|
||||
//! handling across the whole world, and is the main reason the first game's
|
||||
//! movement code does not transfer.
|
||||
//! * `chargeboots` - hold and release timing, chained jumps and the dash.
|
||||
//! * `weapons` - the wrench and the weapon set, plus the weapon modification
|
||||
//! system (per-weapon bitmasks: lock-on, shock, acid, ...).
|
||||
//! * `gadgets` - the Pyrocitor, Bomb Glove, Glove of Doom and friends.
|
||||
//! * `moby_update` - the per-class update dispatch, driven by each level's
|
||||
//! `lvl.vtbl` class table. Classes that are identical across levels are
|
||||
//! shared; the rest are resolved through the overlay relocator in
|
||||
//! `gc-formats`.
|
||||
//! * `collision` - queries against the collision world.
|
||||
//! * `progression` - bolts, the Gadgetron vendor, RPG upgrades and unlocks.
|
||||
//! * `save_game` - the IFF chunk save format, and importing a Ratchet & Clank
|
||||
//! (2002) save to unlock the legacy weapons for free.
|
||||
//! * `memcard` - the 25-state memory-card finite state machine and its UI.
|
||||
//!
|
||||
//! # The float model
|
||||
//!
|
||||
@@ -35,4 +34,4 @@
|
||||
//! # Timing
|
||||
//!
|
||||
//! The game is 60 Hz (this disc is NTSC). Constants are expressed per tick at
|
||||
//! that rate. PAL discs run 50 Hz and would need different timing throughout.
|
||||
//! that rate. PAL discs run 50 Hz and would need different timing throughout.
|
||||
|
||||
@@ -16,4 +16,4 @@
|
||||
//!
|
||||
//! MPEG-2 decoding is a solved, well-specified problem, and a general-purpose
|
||||
//! decoder will produce correct output. The PS2-specific part is only the
|
||||
//! container: how the streams are stored and addressed on the disc.
|
||||
//! container: how the streams are stored and addressed on the disc.
|
||||
|
||||
@@ -0,0 +1,110 @@
|
||||
# Disc layout
|
||||
|
||||
Target: `SCUS_972.68` v1.01 (USA Black Label, NTSC 60 Hz).
|
||||
|
||||
All integers are little-endian. Sector size is `0x800` (2048) bytes throughout;
|
||||
everything on the disc is sector-aligned.
|
||||
|
||||
## Root directory
|
||||
|
||||
Observed by enumerating the ISO 9660 filesystem:
|
||||
|
||||
```text
|
||||
SYSTEM.CNF 60 bytes
|
||||
RC2.HDR 408,444 the table of contents, as a named file
|
||||
SCUS_972.68 2,618,684 the boot ELF
|
||||
IOPRP255.IMG 264,449 IOP modules
|
||||
Z6TAIL.DUP 0 ISO 9660 padding
|
||||
G/... 96 asset containers (see global-wad.md)
|
||||
```
|
||||
|
||||
`RC2.HDR` being present as a real file is convenient. R&C1's equivalent is not a
|
||||
named file and sits at sector 1500.
|
||||
|
||||
## ISO 9660 usage
|
||||
|
||||
The filesystem is load-bearing for this game, unlike R&C1 whose filesystem is
|
||||
nearly vestigial. Everything the engine needs is either a file here or a sector
|
||||
range declared by a container header.
|
||||
|
||||
**Consequence for the extractor:** `G/*` can be enumerated by walking the
|
||||
directory. There is no need to parse a table of contents to find the assets.
|
||||
|
||||
### Details that will bite
|
||||
|
||||
- **Directory records store each 32-bit field twice**, little-endian then
|
||||
big-endian. Reading 4 bytes where the record has 8 shifts every subsequent
|
||||
field and yields an empty or nonsense listing with no error.
|
||||
- **The `CD001` signature is at sector offset `0x0001`**, not `0x8001`. The
|
||||
latter is its absolute file offset and lies outside a 2048-byte sector, so a
|
||||
bounds-checked read there fails rather than returning empty bytes.
|
||||
- **The directory flag is at record offset 25**, bit 1. Do not rely on the
|
||||
identifier's trailing `/`: not every writer sets it, and the flag is
|
||||
authoritative. Getting this wrong writes a subdirectory's *contents* to a file
|
||||
named after the directory.
|
||||
- Records never span a logical block; a zero record length means "advance to the
|
||||
next block".
|
||||
|
||||
## Table of contents (`RC2.HDR`)
|
||||
|
||||
The game also maintains a table of contents, and a second copy lives at sector
|
||||
**1001** (`SYSTEM.CNF` sits at sector 1000 on this disc). The ToC begins with a
|
||||
chain of global container headers, each `{header_size, sector}`, running from
|
||||
offset 0 until the level records begin.
|
||||
|
||||
Verified on this disc:
|
||||
|
||||
```text
|
||||
offset 0x0000 header_size 0x0328 sector 2610 MPEG
|
||||
offset 0x0328 header_size 0x0040 sector 1216608 MISC
|
||||
offset 0x0368 header_size 0x1870 sector 1223855 HUD
|
||||
offset 0x1bd8 header_size 0x0a48 sector 1242375 BONUS
|
||||
offset 0x2620 header_size 0x1800 sector 1257884 AUDIO
|
||||
offset 0x3e20 header_size 0x0ba8 sector 1275113 SPACE
|
||||
offset 0x49c8 header_size 0x0170 sector 1280474 SCENE
|
||||
offset 0x4b38 header_size 0x03c8 sector 1282655 GADGET
|
||||
offset 0x4f00 header_size 0x00f8 sector 1285084 ARMOR
|
||||
```
|
||||
|
||||
The nine header sizes match the container table in
|
||||
[`global-wad.md`](global-wad.md) exactly, which cross-confirms both readings.
|
||||
|
||||
### Level records
|
||||
|
||||
Level records follow the globals on a **`0x2000`-byte stride**, each beginning
|
||||
with `{header_size, sector}` where the sector is an **absolute** disc sector:
|
||||
|
||||
```text
|
||||
+0x00000 LEVEL 0x0060 sector 1286296
|
||||
+0x02000 SCENE 0x137c sector 1296509
|
||||
+0x04000 AUDIO 0x1018 sector 1309652
|
||||
+0x06000 <zero>
|
||||
...
|
||||
```
|
||||
|
||||
Levels appear as `LEVEL`/`SCENE`/`AUDIO` triples separated by gaps, and every
|
||||
level's sector cross-checks against the `G/LEVEL<n>.WAD` entry found by walking
|
||||
the filesystem. That agreement between two independent routes is what confirms
|
||||
the layout; a single route agreeing with itself proves nothing.
|
||||
|
||||
**Not yet implemented.** The extractor does not read the ToC. It enumerates the
|
||||
filesystem instead, which is sufficient and simpler. The ToC becomes necessary
|
||||
when the game needs to resolve a sector at runtime, or when per-level overlay
|
||||
loading has to be understood (Phase 3).
|
||||
|
||||
## Boot executable
|
||||
|
||||
`SCUS_972.68`, 2,618,684 bytes, SHA-1
|
||||
`bacda860cbd6723142e86db56f197e47e728970f`.
|
||||
|
||||
ELF32, little-endian, MIPS (EM_MIPS = 8), entry point `0x00131AE8`, `$gp`
|
||||
`0x001AEFF0`. The entry point independently matches the value recorded by the
|
||||
`rac2-gc-decomp` community reference, which cross-confirms the disc
|
||||
identification.
|
||||
|
||||
## Provenance
|
||||
|
||||
Container layout knowledge was guided by the Wrench project's format
|
||||
descriptions (<https://github.com/chaoticgd/wrench>, GPLv3). **No Wrench code is
|
||||
copied** — see `PROVENANCE.md`. Structures here are written from the
|
||||
specification and verified against this disc.
|
||||
@@ -0,0 +1,160 @@
|
||||
# The `G/*.WAD` asset containers
|
||||
|
||||
Target: `SCUS_972.68` v1.01 (USA Black Label, NTSC).
|
||||
|
||||
Everything except the boot ELF, `SYSTEM.CNF`, `RC2.HDR` and the IOP image
|
||||
lives in a container under `G/` on the disc's ISO 9660 filesystem. R&C2 lists
|
||||
these as real files, unlike R&C1 whose filesystem is vestigial, so enumeration
|
||||
is a directory walk rather than a hunt for a table at a fixed sector.
|
||||
|
||||
## Container list
|
||||
|
||||
96 containers. The names follow the asset family; `n` runs 0..26 for the
|
||||
per-level containers.
|
||||
|
||||
| File | Count | Bytes (disc) | Role |
|
||||
|---|---|---|---|
|
||||
| `ARMOR.WAD` | 1 | 2,480,432 | Armor and wrench meshes plus textures |
|
||||
| `AUDIO.WAD` | 1 | 35,283,296 | Global audio banks and music |
|
||||
| `AUDIO<n>.WAD` | 27 | 3,825,304 .. 46,747,520 | Per-level audio |
|
||||
| `BONUS.WAD` | 1 | 31,762,085 | Collectibles, unlockables |
|
||||
| `GADGET.WAD` | 1 | 4,972,992 | Gadget definitions |
|
||||
| `HUD.WAD` | 1 | 37,927,695 | HUD elements and text |
|
||||
| `LEVEL<n>.WAD` | 27 | 8,350,423 .. 25,119,448 | Level core, gameplay, occlusion |
|
||||
| `MISC.WAD` | 1 | 14,840,496 | Miscellaneous assets |
|
||||
| `MPEG.WAD` | 1 | 1,075,267,584 | Indexed video and subtitle streams |
|
||||
| `MPEG1.WAD` | 1 | 1,296,228,352 | Raw concatenated video, not indexed |
|
||||
| `MPEG2.WAD` | 1 | 114,769,924 | Raw concatenated video, not indexed |
|
||||
| `SCENE.WAD` | 1 | 4,466,256 | Global cutscene data |
|
||||
| `SCENE<n>.WAD` | 27 | 4,988 .. 14,964,240 | Per-level cutscene data |
|
||||
| `SPACE.WAD` | 1 | 10,977,664 | Space level assets |
|
||||
|
||||
The three MPEG WADs are 2.5 GB of the 3.8 GB disc. Video is a data problem
|
||||
before it is a decoding problem; the pipeline reads them in sectors rather than
|
||||
slurping whole files.
|
||||
|
||||
## Header
|
||||
|
||||
Every indexed container begins with:
|
||||
|
||||
```text
|
||||
offset size field
|
||||
0x00 4 header size in bytes
|
||||
0x04 4 sector count of the whole container
|
||||
0x08.. ... table, interpreted per container kind (below)
|
||||
```
|
||||
|
||||
The header size is the reliable discriminator: it is the one field every
|
||||
container agrees on, and its value identifies the container family.
|
||||
|
||||
| Header size | Containers | Table kind |
|
||||
|---|---|---|
|
||||
| `0x0040` | `MISC` | sector/byte pairs |
|
||||
| `0x0060` | `LEVEL<n>` | sector/byte pairs |
|
||||
| `0x00f8` | `ARMOR` | sector/byte pairs |
|
||||
| `0x0170` | `SCENE` | sector/byte pairs |
|
||||
| `0x0328` | `MPEG` | sector/byte pairs |
|
||||
| `0x03c8` | `GADGET` | sector/byte pairs |
|
||||
| `0x0a48` | `BONUS` | sector/byte pairs |
|
||||
| `0x0ba8` | `SPACE` | sector/byte pairs |
|
||||
| `0x1018` | `AUDIO<n>` | sector/byte pairs |
|
||||
| `0x137c` | `SCENE<n>` | **flat sector list** |
|
||||
| `0x1800` | `AUDIO` | sector/byte pairs |
|
||||
| `0x1870` | `HUD` | sector/byte pairs |
|
||||
|
||||
## Sector/byte pairs
|
||||
|
||||
The common form. From `0x08` to `header_size`, a run of 8-byte entries:
|
||||
|
||||
```text
|
||||
offset size field
|
||||
0x00 4 first sector of the extent
|
||||
0x04 4 length of the extent **in bytes**
|
||||
```
|
||||
|
||||
The offset is in **sectors**, the size is in **bytes**. This is the single most
|
||||
important thing to get right, and it is easy to get backwards because both
|
||||
fields are 32-bit integers in a structure where either reading type-checks.
|
||||
|
||||
**This was measured, not assumed.** For all 92 indexed pair containers:
|
||||
|
||||
- Reading the size as bytes places every entry inside its file.
|
||||
- Reading the size as a sector count places entries past the end of the file,
|
||||
and on `LEVEL<n>` and `AUDIO<n>` places *none* of them correctly — 0 of 5 on
|
||||
`LEVEL0.WAD`, 0 of 42 on `AUDIO1.WAD`, and so on for all 27 of each.
|
||||
|
||||
Offsets chain, which is the confirmation that the reading is right. On
|
||||
`AUDIO0.WAD`: sectors 3, 185, 367, 1458, where `185 = 3 + 182` sectors and each
|
||||
entry's end is the next entry's start.
|
||||
|
||||
### Payload is not compressed
|
||||
|
||||
The bytes behind a pair are raw. Insomniac's LZ77 WAD container (see
|
||||
[`wad.md`](wad.md)) is used elsewhere on the disc — inside level cores and MPEG
|
||||
streams — but these global containers store their sections uncompressed and let
|
||||
the table do the addressing. A decompressor applied here produces garbage.
|
||||
|
||||
### Worked example: ARMOR.WAD
|
||||
|
||||
`header_size = 0xf8`, so `(0xf8 - 8) / 8 = 30` entries. Reading the pairs:
|
||||
|
||||
```text
|
||||
entry 0: sector 1, 372,272 bytes
|
||||
entry 1: sector 124, 163,840 bytes (= 73 + 51 sectors earlier)
|
||||
...
|
||||
entry 29: sector 1200, 24,576 bytes
|
||||
```
|
||||
|
||||
The published structure is nine armor entries then six wrench entries, each a
|
||||
`{mesh, textures}` pair, which is 15 pairs — but the generic parser reads 30
|
||||
entries because the header also declares the textures of the nine armors as
|
||||
their own ranges. Entry 0 is armor 0's mesh and entry 1 is armor 0's textures;
|
||||
entries 2 and 3 are armor 1's, and so on. Which pair means what is per-container
|
||||
knowledge, recovered when that asset family is parsed.
|
||||
|
||||
## Flat sector list (`SCENE<n>.WAD`)
|
||||
|
||||
The 27 per-level scene containers are the exception. `header_size = 0x137c`,
|
||||
and `0x137c - 8 = 4980` is **not** a multiple of 8, so the bytes cannot be
|
||||
pairs at all. They are a flat run of 1,245 `u32` sector numbers, 30 scenes of
|
||||
83 values each.
|
||||
|
||||
Each value is the first sector of one extent. The extent's length is the
|
||||
difference to the next value; the last runs to the end of the container.
|
||||
|
||||
```text
|
||||
sectors[0] = 3, sectors[1] = 7 -> extent 0 is sectors 3..7 (4 sectors)
|
||||
```
|
||||
|
||||
The global `SCENE.WAD` is not one of these: it uses the pair form.
|
||||
|
||||
## Raw streams
|
||||
|
||||
`MPEG1.WAD` and `MPEG2.WAD` have no table. Their first word is video payload,
|
||||
not a plausible header size (`7864320` and a negative value respectively), so
|
||||
they are copied through whole and split by the video phase.
|
||||
|
||||
## Known trailing sentinels
|
||||
|
||||
A few containers declare entries whose extent runs past the end of the file:
|
||||
1 in every `SCENE<n>.WAD`, 10 in `GADGET.WAD`, 35 in `MPEG.WAD`. These are
|
||||
trailing sentinels or descriptors that reference sectors belonging to an
|
||||
adjacent container. The extractor bounds-checks every extent and skips these
|
||||
with a count, rather than reading past the end of the buffer or trusting the
|
||||
header.
|
||||
|
||||
## Verification
|
||||
|
||||
Unpacking the disc produces 5,135 files totalling 4,500,106,960 bytes. Every
|
||||
file's size and SHA-1 are recorded in
|
||||
`crates/gc-extract/data/gc-files.tsv` (hashes only, no content). See
|
||||
[`../plan/disc-pipeline.md`](../plan/disc-pipeline.md) for how that gate is run.
|
||||
|
||||
## What is not yet known
|
||||
|
||||
- Which pair in each container corresponds to which asset. Needed before any
|
||||
asset can be parsed; the geometry is recovered per family in Phase 2.
|
||||
- The `bindata` sub-structures inside audio and scene containers.
|
||||
- How `MPEG.WAD`'s indexed streams relate to the raw `MPEG1`/`MPEG2` files.
|
||||
Two descriptions of the same video data exist on the disc; which the game
|
||||
reads is not yet established.
|
||||
@@ -0,0 +1,96 @@
|
||||
# The WAD LZ77 container
|
||||
|
||||
Target: `SCUS_972.68` v1.01.
|
||||
|
||||
Insomniac's LZ77 stream. A 16-byte header followed by a sequence of packets.
|
||||
Each packet begins with a **flag byte** selecting one of four classes.
|
||||
|
||||
This format is **not** the `G/*.WAD` container described in
|
||||
[`global-wad.md`](global-wad.md), despite the shared name and extension. WAD
|
||||
streams are compressed and appear inside level cores and MPEG data; the `G/`
|
||||
containers are uncompressed tables. Confusing the two produces garbage.
|
||||
|
||||
## Header
|
||||
|
||||
```text
|
||||
offset size field
|
||||
0x00 3 magic, "WAD"
|
||||
0x03 4 total stream length in bytes, including this header
|
||||
0x07 9 unused
|
||||
```
|
||||
|
||||
`compressed_size` at `0x03` is the authoritative length. A stream whose declared
|
||||
size exceeds the buffer is rejected rather than truncated.
|
||||
|
||||
## Packets
|
||||
|
||||
| Flag | Class | Displacement bits | Length |
|
||||
|---|---|---|---|
|
||||
| `0x00..0x0f` | literal run | — | flag + 3 (flag 0 escapes to a byte + 18) |
|
||||
| `0x10..0x1f` | back-reference | 15 | 3 bits in flag, else a byte + 7 |
|
||||
| `0x20..0x3f` | back-reference | 14 | 5 bits in flag, else a byte + 0x1f |
|
||||
| `0x40..0xff` | back-reference | 11 | 3 bits in flag |
|
||||
|
||||
Every packet ends with 0–3 extra literal bytes carried in unused low bits of the
|
||||
packet, so literal and back-reference packets interleave without alignment.
|
||||
This is why the format does not read as a bit stream, and why a first attempt
|
||||
that modelled it as one will appear to produce plausible output for a few
|
||||
kilobytes before failing.
|
||||
|
||||
### Literal run
|
||||
|
||||
Flag `< 0x10`. The count is `flag + 3`, except flag `0x00` which escapes to a
|
||||
following byte giving `byte + 18`. The bytes follow immediately.
|
||||
|
||||
Two literal packets in a row are invalid: the encoder merges them. The decoder
|
||||
rejects the stream rather than accept a misparse.
|
||||
|
||||
### Back-reference
|
||||
|
||||
A back-reference copies `len` bytes from `displacement` positions back in the
|
||||
already-produced output.
|
||||
|
||||
**Overlapping copies expand as run-length.** The source index advances with the
|
||||
destination, so `displacement = 1` repeats a single byte. This is used
|
||||
intentionally by the encoder.
|
||||
|
||||
A displacement of zero, or one that reaches before the start of the output, is
|
||||
invalid.
|
||||
|
||||
### Pad and dummy packets
|
||||
|
||||
In the `0x10..0x1f` class, a packet whose displacement field is entirely zero is
|
||||
special:
|
||||
|
||||
- length field not 1 → a **pad packet**: the reader skips forward to the next
|
||||
`0x1000` boundary. These exist so compressed blocks align with something on
|
||||
the disc.
|
||||
- length field 1 → a **dummy packet**: it carries only its trailing literal
|
||||
bytes.
|
||||
|
||||
## Worked example
|
||||
|
||||
The first stream in this game's `MISC.WAD` begins:
|
||||
|
||||
```text
|
||||
05 80 00 00 00 10 | 00 0d ...
|
||||
```
|
||||
|
||||
- Flag `0x05` is `< 0x10`, so it is a literal run of `5 + 3 = 8` bytes:
|
||||
`80 00 00 00 10 00 0d 00`.
|
||||
- The next flag byte starts the following packet.
|
||||
|
||||
## Implementation
|
||||
|
||||
`crates/gc-formats/src/wad.rs`. Parses through the same bounds-checked reader as
|
||||
everything else in the crate, so a truncated stream produces a typed error
|
||||
rather than a panic. A `BadMagic`, an out-of-bounds read, and a displacement
|
||||
before the output are three distinct error kinds.
|
||||
|
||||
## Provenance
|
||||
|
||||
The packet structure was recovered from behavioural analysis of the retail
|
||||
data, guided by the format description in the Wrench project
|
||||
(<https://github.com/chaoticgd/wrench>, GPLv3). **No Wrench code is copied into
|
||||
this repository** — see `PROVENANCE.md`. The decoder here is written from the
|
||||
specification and verified against this disc's own streams.
|
||||
@@ -0,0 +1,140 @@
|
||||
# Disc pipeline
|
||||
|
||||
The work record for Phase 1. What was tried, what worked, and what was ruled
|
||||
out — the negative results matter as much as the working parts.
|
||||
|
||||
## Result
|
||||
|
||||
The disc pipeline runs end to end and is verified.
|
||||
|
||||
```text
|
||||
$ gc-extract identify disc.iso
|
||||
serial SCUS_972.68 version 1.01 video mode NTSC
|
||||
boot ELF SHA-1 bacda860cbd6723142e86db56f197e47e728970f
|
||||
|
||||
$ gc-extract unpack disc.iso extracted/ --table gc-files.tsv
|
||||
5135 files, 4,500,106,960 bytes, in 10.8s
|
||||
|
||||
$ gc-extract verify extracted/ gc-files.tsv
|
||||
OK: 5135 files match the recorded table.
|
||||
```
|
||||
|
||||
Every one of the 96 `G/` containers parses. 92 are indexed containers whose
|
||||
extents are extracted individually; `MPEG1.WAD` and `MPEG2.WAD` are raw video
|
||||
and pass through whole.
|
||||
|
||||
## The wrong turn worth recording
|
||||
|
||||
**The container pair table was read backwards for most of this phase.**
|
||||
|
||||
The obvious reading of an 8-byte entry is `{offset, size}`, both sector counts —
|
||||
`SectorRange`. That reading:
|
||||
|
||||
- worked perfectly for `ARMOR.WAD` (30 of 30 entries in range), which is exactly
|
||||
why it survived several rounds of testing;
|
||||
- placed 1 entry out of range on `HUD.WAD`, 69 of 178 on `AUDIO.WAD`, and **all
|
||||
entries out of range** on every `AUDIO<n>.WAD` and `LEVEL<n>.WAD`.
|
||||
|
||||
The `AUDIO0.WAD` numbers were the tell: under the sector reading, the count of
|
||||
out-of-range entries equalled the count of in-range entries, for all 27 audio
|
||||
containers. A systematic 50/50 split is a decoding error, not sparse data.
|
||||
|
||||
The correct form is `{offset in sectors, size in bytes}` — `SectorByteRange`.
|
||||
The confirmation is that offsets chain: on `AUDIO0.WAD` the sectors are 3, 185,
|
||||
367, 1458, where each entry's end is the next entry's start, in both bytes and
|
||||
sectors. Under the byte reading the extents tile the file exactly.
|
||||
|
||||
Measured across all 92 indexed pair containers:
|
||||
|
||||
| Reading | Entries placed correctly |
|
||||
|---|---|
|
||||
| Sector/sector | many, but never all; 0 of 5 on `LEVEL0.WAD`, 0 of 42 on `AUDIO1.WAD`, and so on for all 27 of each |
|
||||
| Sector/bytes | all, on every container |
|
||||
|
||||
**Lesson for the format parsers in Phase 2:** a parser that type-checks on one
|
||||
container can still be wrong on the other 91. Prefer a check that must hold
|
||||
*everywhere* — extents tiling a file without gaps or overlaps — over a check
|
||||
that holds on the sample in front of you. This is why `ARMOR.WAD` was chosen to
|
||||
validate by hand and why that choice nearly cost the phase.
|
||||
|
||||
## The WAD LZ77 decoder
|
||||
|
||||
Also initially wrong, and instructive.
|
||||
|
||||
The first attempt modelled the format as a **bit stream**: a control byte whose
|
||||
bits each select "literal byte" or "back-reference". This is how many LZ77
|
||||
variants work, and it produced plausible-looking output for a few kilobytes —
|
||||
`80 00 00 00 10 00 0d ...` looked like real level data — before failing with an
|
||||
invalid displacement. Four bit orderings and a dozen offset/length field
|
||||
layouts were tried against a real stream; none decoded more than a handful of
|
||||
bytes.
|
||||
|
||||
The actual format is a **flag-byte packet stream** with four packet classes,
|
||||
chosen by the flag's value rather than by individual bits, where every packet
|
||||
carries 0–3 trailing literal bytes in unused low bits. It cannot be expressed as
|
||||
a bit stream, which is why no bit ordering worked. See
|
||||
[`../formats/wad.md`](../formats/wad.md).
|
||||
|
||||
The failure mode is worth remembering: a decoder that produces *plausible* wrong
|
||||
output is more dangerous than one that fails immediately. The first WAD decoder
|
||||
did exactly that.
|
||||
|
||||
## Container families
|
||||
|
||||
| Family | Count | Header size | Table |
|
||||
|---|---|---|---|
|
||||
| `LEVEL<n>` | 27 | `0x60` | sector/byte pairs |
|
||||
| `AUDIO<n>` | 27 | `0x1018` | sector/byte pairs |
|
||||
| `SCENE<n>` | 27 | `0x137c` | flat sector list |
|
||||
| `SCENE` | 1 | `0x170` | sector/byte pairs |
|
||||
| `MPEG1`, `MPEG2` | 2 | — | none; raw video |
|
||||
| others | 12 | various | sector/byte pairs |
|
||||
|
||||
`SCENE<n>` is the only family with a different table shape. `(0x137c - 8)` is
|
||||
not a multiple of 8, so the bytes cannot be pairs at all: they are 1,245 sector
|
||||
numbers in a flat list, each extent running to the next value.
|
||||
|
||||
## Deliberately deferred
|
||||
|
||||
- **The table of contents at sector 1001** is documented in
|
||||
[`../formats/disc-layout.md`](../formats/disc-layout.md) but not read by the
|
||||
extractor. Enumerating the ISO 9660 filesystem is simpler and sufficient; the
|
||||
ToC matters when per-level overlay loading is understood in Phase 3.
|
||||
- **Per-container entry semantics.** Which pair is a mesh, which is a texture,
|
||||
which is audio: recovered per family as each is parsed in Phase 2. The
|
||||
extractor only needs to know where each extent starts and ends.
|
||||
- **Audio sub-structures.** `AUDIO<n>.WAD` headers hold 36 `bindata` ranges
|
||||
followed by 15 music pointers; the split is inferred from the header size but
|
||||
not yet confirmed by unpacking both halves.
|
||||
|
||||
## Known tolerated imprecision
|
||||
|
||||
A few containers declare extents that run past the end of their file: 1 in each
|
||||
`SCENE<n>.WAD`, 10 in `GADGET.WAD`, 35 in `MPEG.WAD`. These are trailing
|
||||
sentinels or descriptors referencing sectors in an adjacent container.
|
||||
|
||||
The extractor bounds-checks every extent against the container actually read, so
|
||||
these cannot cause an out-of-bounds read. They are counted and reported rather
|
||||
than silently dropped, because a silent skip would be indistinguishable from
|
||||
data loss. Total skipped: 46 files' worth of entries, out of 5,135 files
|
||||
extracted.
|
||||
|
||||
## Verification approach
|
||||
|
||||
`crates/gc-extract/data/gc-files.tsv` records the size and SHA-1 of all 5,135
|
||||
produced files — hashes only, never content, 316 KB.
|
||||
|
||||
The value of this is that a future parser can assert "this matches retail"
|
||||
rather than "this matches whatever is on my disk". When a Phase 2 parser is
|
||||
wrong, the hash table is how we find out, and how we tell a wrong parser from a
|
||||
wrong assumption.
|
||||
|
||||
Regenerate after an intentional change:
|
||||
|
||||
```sh
|
||||
gc-extract unpack <disc.iso> extracted/ \
|
||||
--table crates/gc-extract/data/gc-files.tsv
|
||||
```
|
||||
|
||||
If only some files change, the unpack stage changed. If none do, only a parser
|
||||
downstream is wrong.
|
||||
+31
-13
@@ -13,7 +13,10 @@ early, while they are still cheap to retire.
|
||||
- [x] Workspace builds (`cargo check --workspace`)
|
||||
- [x] Repository hygiene in place (pre-commit guard, `.gitignore`, licence
|
||||
record)
|
||||
- [ ] Nothing else implemented yet
|
||||
- [x] **Phase 1 complete.** The disc pipeline runs end to end and is verified
|
||||
against a committed hash table: 5,135 files, 4,500,106,960 bytes, every
|
||||
one of the 92 asset containers parsing.
|
||||
- [ ] Phase 2 next: the asset format parsers.
|
||||
|
||||
The disc being the exact build the reference projects pin is the single
|
||||
biggest favourable fact so far: no disc conversion, no regional timing work,
|
||||
@@ -46,22 +49,33 @@ and no re-derivation of the reference baseline.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Disc pipeline
|
||||
## Phase 1 — Disc pipeline ✅
|
||||
|
||||
**Goal:** turn the disc image into verified, unpacked asset files.
|
||||
**Done.** Gate met: every extracted file matches the committed hash table.
|
||||
|
||||
- [ ] `iso9660` reader (R&C2 lists assets as real files, unlike R&C1)
|
||||
- [ ] `RC2.HDR` table of contents parser (408,444 bytes at disc root)
|
||||
- [ ] WAD LZ77 decompression
|
||||
- [ ] Unpack global WADs (`G/*.WAD`) and per-level WADs
|
||||
- [ ] Commit a size+SHA-1 table per target file (hashes only, never content)
|
||||
- [ ] `verify` stage
|
||||
- [x] `iso9660` reader (R&C2 lists assets as real files, unlike R&C1)
|
||||
- [x] `G/*.WAD` container header parsing — 92 containers, three table shapes
|
||||
- [x] WAD LZ77 decompression (`wad.rs`, four packet classes)
|
||||
- [x] Unpack every container, bounds-checking each extent
|
||||
- [x] SHA-1 table for all 5,135 files (hashes only, never content)
|
||||
- [x] `verify` stage, plus integration tests against a real disc
|
||||
- [x] Format specs written: `disc-layout.md`, `global-wad.md`, `wad.md`
|
||||
- [x] Work record with the negative results: `plan/disc-pipeline.md`
|
||||
|
||||
**Gate:** every extracted file matches the committed hash table, and the
|
||||
decompressed size of every WAD stream matches its header.
|
||||
```text
|
||||
gc-extract identify disc.iso -> SCUS_972.68 v1.01 NTSC, ELF SHA-1 matches
|
||||
gc-extract unpack disc.iso out/ -> 5135 files, 4,500,106,960 bytes, 10.8s
|
||||
gc-extract verify out/ table.tsv -> OK: 5135 files match
|
||||
```
|
||||
|
||||
**Note:** the three MPEG WADs total roughly 2.4 GB of the 3.8 GB disc. Video is
|
||||
a data problem before it is a decoding problem — plan to stream.
|
||||
**Not done, deliberately:** the table of contents at sector 1001 is documented
|
||||
but not read. Enumerating the ISO filesystem is simpler and sufficient; the ToC
|
||||
matters when per-level overlay loading is understood in Phase 3.
|
||||
|
||||
**Two wrong turns are recorded** in `plan/disc-pipeline.md`, because both would
|
||||
be easy to repeat: the container pair table read backwards, and the LZ77 stream
|
||||
mistaken for a bit stream. Both produced plausible-looking wrong output before
|
||||
failing, which is the failure mode worth guarding against.
|
||||
|
||||
---
|
||||
|
||||
@@ -70,6 +84,10 @@ a data problem before it is a decoding problem — plan to stream.
|
||||
**Goal:** every asset type parses into plain Rust values, tested against real
|
||||
disc data.
|
||||
|
||||
Phase 1 hands over the extents; this phase works out what each one contains.
|
||||
`global_wad.md` records which pair in each container corresponds to which asset,
|
||||
and that table starts empty.
|
||||
|
||||
- [ ] `tfrag` — terrain: LOD chains, strips, morph targets
|
||||
- [ ] `tie` — instanced environment geometry, fat vertices
|
||||
- [ ] `shrub` — foliage and decorative meshes
|
||||
|
||||
Reference in New Issue
Block a user