Add the disc pipeline: identify, unpack and verify

Phase 1 of the roadmap. The disc is now turned into verified asset files, and
the result is checked against a committed hash table rather than against
whatever happens to be on disk.

  gc-extract identify disc.iso        SCUS_972.68 v1.01 NTSC, ELF SHA-1 matches
  gc-extract unpack   disc.iso out/   5135 files, 4,500,106,960 bytes, 10.8s
  gc-extract verify   out/ table.tsv  OK: 5135 files match

All 92 G/*.WAD containers parse. MPEG1 and MPEG2 are raw video and pass
through whole; the other 90 are indexed containers whose extents are extracted
individually.

Formats recovered and written up in docs/formats/:

- iso9660.rs. R&C2 lists its assets in the filesystem, unlike R&C1, so a
  directory walk is the primary asset index. Three details that all fail
  quietly rather than loudly: directory records store every 32-bit field twice
  (little-endian then big-endian), the CD001 signature is at sector offset 1
  rather than 0x8001, and the directory flag at record offset 25 is
  authoritative where the identifier's trailing slash is not.

- global_wad.rs. Container headers hold a table of extents. All 92 indexed
  containers use {offset in sectors, size in bytes}. Measured, not assumed: the
  sector-count reading places every entry of all 27 LEVEL<n> and all 27
  AUDIO<n> containers out of range, while the byte reading places all of them
  correctly. SCENE<n> is the one exception, a flat sector list whose extents
  are sized by the next value.

- wad.rs. Insomniac's LZ77 stream: four flag-selected packet classes, with 0-3
  trailing literal bytes per packet carried in unused low bits. This is not a
  bit stream, which is why it resists the usual LZ77 reading.

- buf.rs. A bounds-checked reader, so untrusted 2003 disc data produces typed
  errors rather than panics.

gc-extract carries the three stages and a SHA-1 implementation with FIPS 180-4
test vectors, so identification works without pulling in a crypto dependency.

Verification: 54 unit tests, plus four integration tests that run only when
GC_DISC names a disc. Those four check identification against the recorded
values, cross-check the Rust identifier against tools/identify-disc.py, parse
all 92 containers, and confirm a full unpack matches the committed table. The
committed table is 316 KB of sizes and SHA-1s for all 5,135 files: hashes only,
no game content, so it is safe to version and it turns "matches retail" into
something a test can assert.

docs/plan/disc-pipeline.md records two wrong turns, because both are easy to
repeat and both failed by producing plausible wrong output rather than an
error: the container table was read backwards, and the LZ77 stream was modelled
as a bit stream. The first survived because it happened to be correct on the
one container used for manual checking.
This commit is contained in:
2026-10-09 16:08:30 +02:00
parent a02f79a6d1
commit 027373e65b
21 changed files with 8372 additions and 97 deletions
+261
View File
@@ -0,0 +1,261 @@
//! A bounds-checked cursor over a byte slice.
//!
//! Every format parser in this crate reads through a `Buf`. Asset data comes
//! from a 2003 disc format, so it is untrusted input: a truncated or corrupt
//! file must produce a typed error, never a panic and never a read past the
//! end of the buffer.
//!
//! Reads are little-endian throughout. The PS2 is a little-endian machine and
//! so is every asset container we have seen.
use std::fmt;
/// The position and context of a failed read.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Error {
pub offset: usize,
pub context: &'static str,
pub kind: ErrorKind,
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum ErrorKind {
/// A read reached past the end of the buffer.
OutOfBounds { needed: usize, available: usize },
/// A length field was present but nonsensical.
InvalidLength(&'static str),
/// A magic number or signature did not match.
BadMagic { expected: &'static str },
/// The structure parsed, but a field held an impossible value.
Invalid(&'static str),
}
impl fmt::Display for Error {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match &self.kind {
ErrorKind::OutOfBounds { needed, available } => write!(
f,
"{}: need {needed} bytes at offset {:#x}, {} available",
self.context, self.offset, available
),
ErrorKind::InvalidLength(what) => {
write!(
f,
"{}: invalid length for {what} at offset {:#x}",
self.context, self.offset
)
}
ErrorKind::BadMagic { expected } => {
write!(
f,
"{}: expected {expected} at offset {:#x}",
self.context, self.offset
)
}
ErrorKind::Invalid(what) => {
write!(
f,
"{}: invalid {what} at offset {:#x}",
self.context, self.offset
)
}
}
}
}
impl std::error::Error for Error {}
pub type Result<T> = std::result::Result<T, Error>;
/// A bounds-checked reader over a byte slice.
#[derive(Clone, Copy)]
pub struct Buf<'a> {
pub bytes: &'a [u8],
pos: usize,
}
impl<'a> Buf<'a> {
pub fn new(bytes: &'a [u8]) -> Self {
Self { bytes, pos: 0 }
}
pub fn len(&self) -> usize {
self.bytes.len()
}
pub fn is_empty(&self) -> bool {
self.bytes.is_empty()
}
/// Current read position.
pub fn pos(&self) -> usize {
self.pos
}
/// Move the cursor. Positions beyond the end are an error on the next read
/// rather than here, so that a caller can seek then check.
pub fn seek(&mut self, pos: usize) {
self.pos = pos;
}
/// Skip `n` bytes.
pub fn skip(&mut self, n: usize, context: &'static str) -> Result<()> {
self.read_bytes(n, context).map(|_| ())
}
/// The remainder of the buffer from the cursor.
pub fn rest(&self) -> &'a [u8] {
self.bytes.get(self.pos..).unwrap_or(&[])
}
/// Read `n` bytes, advancing the cursor.
pub fn read_bytes(&mut self, n: usize, context: &'static str) -> Result<&'a [u8]> {
let end = self.pos.saturating_add(n);
let slice = self.bytes.get(self.pos..end).ok_or(Error {
offset: self.pos,
context,
kind: ErrorKind::OutOfBounds {
needed: n,
available: self.bytes.len().saturating_sub(self.pos),
},
})?;
self.pos = end;
Ok(slice)
}
/// Read `n` bytes at an absolute offset without moving the cursor.
pub fn peek_bytes(&self, offset: usize, n: usize, context: &'static str) -> Result<&'a [u8]> {
let end = offset.saturating_add(n);
self.bytes.get(offset..end).ok_or(Error {
offset,
context,
kind: ErrorKind::OutOfBounds {
needed: n,
available: self.bytes.len().saturating_sub(offset),
},
})
}
pub fn u8(&mut self, context: &'static str) -> Result<u8> {
Ok(self.read_bytes(1, context)?[0])
}
pub fn i8(&mut self, context: &'static str) -> Result<i8> {
Ok(self.u8(context)? as i8)
}
pub fn u16(&mut self, context: &'static str) -> Result<u16> {
let b = self.read_bytes(2, context)?;
Ok(u16::from_le_bytes([b[0], b[1]]))
}
pub fn i16(&mut self, context: &'static str) -> Result<i16> {
Ok(self.u16(context)? as i16)
}
pub fn u32(&mut self, context: &'static str) -> Result<u32> {
let b = self.read_bytes(4, context)?;
Ok(u32::from_le_bytes([b[0], b[1], b[2], b[3]]))
}
pub fn i32(&mut self, context: &'static str) -> Result<i32> {
Ok(self.u32(context)? as i32)
}
pub fn u64(&mut self, context: &'static str) -> Result<u64> {
let b = self.read_bytes(8, context)?;
Ok(u64::from_le_bytes([
b[0], b[1], b[2], b[3], b[4], b[5], b[6], b[7],
]))
}
pub fn f32(&mut self, context: &'static str) -> Result<f32> {
Ok(f32::from_bits(self.u32(context)?))
}
/// Read an absolute `u32` without moving the cursor.
pub fn peek_u32(&self, offset: usize, context: &'static str) -> Result<u32> {
let b = self.peek_bytes(offset, 4, context)?;
Ok(u32::from_le_bytes([b[0], b[1], b[2], b[3]]))
}
/// Read an absolute `i32` without moving the cursor.
pub fn peek_i32(&self, offset: usize, context: &'static str) -> Result<i32> {
Ok(self.peek_u32(offset, context)? as i32)
}
/// Require `expected` bytes to be present, else a `BadMagic` error.
pub fn expect_magic(&mut self, expected: &[u8], context: &'static str) -> Result<()> {
let got = self.read_bytes(expected.len(), context)?;
if got != expected {
return Err(Error {
offset: self.pos - expected.len(),
context,
kind: ErrorKind::BadMagic {
expected: "the expected signature",
},
});
}
Ok(())
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn reads_little_endian() {
let mut b = Buf::new(&[0x01, 0x02, 0x03, 0x04, 0xff]);
assert_eq!(b.u16("t").unwrap(), 0x0201);
assert_eq!(b.u16("t").unwrap(), 0x0403);
assert_eq!(b.u8("t").unwrap(), 0xff);
}
#[test]
fn u32_i32_and_float() {
let mut b = Buf::new(&[0x78, 0x56, 0x34, 0x12, 0x00, 0x00, 0x80, 0x3f]);
assert_eq!(b.u32("t").unwrap(), 0x1234_5678);
assert_eq!(b.f32("t").unwrap(), 1.0);
let mut b = Buf::new(&[0xff, 0xff, 0xff, 0xff]);
assert_eq!(b.i32("t").unwrap(), -1);
}
#[test]
fn read_past_end_is_an_error_not_a_panic() {
let mut b = Buf::new(&[1, 2, 3]);
assert!(b.u32("four bytes").is_err());
// The cursor must not have moved past the buffer.
assert_eq!(b.pos(), 0);
}
#[test]
fn overflow_in_length_check_is_caught() {
let b = Buf::new(&[1, 2, 3]);
// usize::MAX + 1 must not wrap into a valid-looking range.
let err = b.peek_bytes(1, usize::MAX, "huge").unwrap_err();
assert!(matches!(err.kind, ErrorKind::OutOfBounds { .. }));
}
#[test]
fn seek_past_end_errors_on_read() {
let mut b = Buf::new(&[1, 2, 3]);
b.seek(100);
assert!(b.u8("t").is_err());
}
#[test]
fn rest_after_seek() {
let b = Buf::new(&[1, 2, 3, 4, 5]);
let mut b2 = b;
b2.seek(3);
assert_eq!(b2.rest(), &[4, 5]);
}
#[test]
fn magic_mismatch_reports_context() {
let mut b = Buf::new(b"WRONG");
let err = b.expect_magic(b"WAD\x10", "wad header").unwrap_err();
assert_eq!(err.context, "wad header");
}
}
+380
View File
@@ -0,0 +1,380 @@
//! The asset containers under `G/` on the disc.
//!
//! Each `G/*.WAD` file begins with a typed header whose only universally
//! applicable fields are:
//!
//! ```text
//! offset size field
//! 0x00 4 header size in bytes
//! 0x04 4 sector count of this container
//! 0x08.. ... a table of {offset, size} pairs, both in SECTORS
//! ```
//!
//! The pair table is the important part. Sectors chain: entry `n`'s offset plus
//! its size is entry `n+1`'s offset, which is how the extractor recovers the
//! layout without a per-container parser. The pairs begin at `0x08` and run to
//! `header_size`, giving `(header_size - 8) / 8` entries.
//!
//! The payload behind each pair is **not** WAD-compressed. Insomniac's WAD LZ77
//! container is used elsewhere on the disc (inside level cores and MPEG
//! streams); these global containers store their sections raw and let the
//! table do the addressing.
//!
//! Which pair means what depends on the container — `ARMOR.WAD` is nine armor
//! entries then six wrench entries, each with a mesh and a texture range — but
//! the geometry above holds for every global on this disc, so the extractor
//! stays generic and per-container meaning is recovered later.
//!
//! Spec: `docs/formats/global-wad.md`.
use crate::buf::{Buf, Error, ErrorKind, Result};
/// Disc logical block size. All container offsets are in these units.
pub const SECTOR_SIZE: u32 = 2048;
/// A `{offset, size}` pair, both in sectors.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct SectorRange {
pub offset: i32,
pub size: i32,
}
impl SectorRange {
/// Byte offset of the extent. `None` when the range is absent.
pub fn byte_offset(&self) -> Option<u64> {
if self.size <= 0 || self.offset < 0 {
return None;
}
Some(self.offset as u64 * SECTOR_SIZE as u64)
}
/// Byte length of the extent.
pub fn byte_len(&self) -> u64 {
if self.size <= 0 {
0
} else {
self.size as u64 * SECTOR_SIZE as u64
}
}
pub fn is_present(&self) -> bool {
self.size > 0
}
}
/// A `{offset, size}` pair where the offset is in sectors and the size is in
/// **bytes**.
///
/// The MPEG containers use this form: their entries are video and subtitle
/// streams whose length is not sector-aligned, so a sector count would round
/// and lose the tail. See `docs/formats/global-wad.md`.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct SectorByteRange {
pub offset: i32,
pub size: i32,
}
impl SectorByteRange {
pub fn byte_offset(&self) -> Option<u64> {
if self.size <= 0 || self.offset < 0 {
return None;
}
Some(self.offset as u64 * SECTOR_SIZE as u64)
}
pub fn byte_len(&self) -> u64 {
if self.size <= 0 {
0
} else {
self.size as u64
}
}
pub fn is_present(&self) -> bool {
self.size > 0
}
}
/// How a container's table should be read.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum RangeKind {
/// Both fields are sector counts.
Sectors,
/// Offset in sectors, size in bytes. Used by every indexed container on
/// this disc except the scene containers.
SectorBytes,
/// A flat array of sector numbers, not pairs. The scene containers are
/// this: each value points at a sector, and the length of each extent comes
/// from the difference to the next entry. The last entry is bounded by the
/// end of the container.
SectorList,
}
impl RangeKind {
/// Is this container's table a flat list rather than pairs?
pub fn is_list(&self) -> bool {
matches!(self, RangeKind::SectorList)
}
}
impl RangeKind {
/// Classify a container from its header size.
///
/// On this disc **every** indexed container uses the sector/byte form: a
/// table of `{offset in sectors, size in bytes}` pairs. This was measured
/// rather than assumed. For all 96 containers that carry a pair table, the
/// sector/byte reading places every entry inside the file; the
/// sector/sector reading leaves entries running past the end of the file,
/// and on the 27 level containers and the 27 audio containers it places
/// *none* of them correctly.
///
/// The header size is the reliable discriminator because it is the field
/// every container agrees on, and it distinguishes the level containers
/// (0x60) from audio (0x1018), scene (0x137c) and the rest.
pub fn for_header(header_size: u32) -> RangeKind {
match header_size {
// The per-level scene containers hold a flat sector list rather
// than pairs: 0x137c is the header size the 27 SCENE<n>.WAD files
// share, and (0x137c - 8) is not a multiple of 8, so the bytes
// cannot be read as pairs at all.
0x137c => RangeKind::SectorList,
// No container on this disc uses the sector/sector form. Kept so a
// future disc revision can be checked rather than assumed.
_ => RangeKind::SectorBytes,
}
}
/// Classify by ISO file name, for callers that only have the path.
pub fn for_container(name: &str) -> RangeKind {
let stem = name.rsplit('/').next().unwrap_or(name);
let stem = stem.trim_end_matches(".WAD").to_ascii_uppercase();
if stem.starts_with("SCENE")
&& stem.len() > 5
&& stem[5..].chars().all(|c| c.is_ascii_digit())
{
// SCENE0..SCENE26 are the per-level containers. SCENE.WAD (no
// digits) is the global one and does use pairs.
return RangeKind::SectorList;
}
RangeKind::SectorBytes
}
}
/// A parsed global container header.
#[derive(Debug, Clone)]
pub struct GlobalWad {
pub header_size: u32,
/// Declared sector count of the whole container.
pub sector_count: u32,
/// The pair table, in declaration order.
pub entries: Vec<SectorRange>,
/// How the pair table was read.
pub kind: RangeKind,
}
impl GlobalWad {
/// Parse a container header, reading the table as sector-count pairs.
pub fn parse(bytes: &[u8]) -> Result<GlobalWad> {
GlobalWad::parse_with(bytes, RangeKind::SectorBytes)
}
/// Parse a container header with an explicit range interpretation.
///
/// Both interpretations read the same 8 bytes per entry; they differ only in
/// how the size field is scaled, which is a property of the container
/// rather than of the bytes.
pub fn parse_with(bytes: &[u8], kind: RangeKind) -> Result<GlobalWad> {
let mut b = Buf::new(bytes);
let header_size = b.u32("container header size")? as usize;
let sector_count = b.u32("container sector count")?;
if header_size < 8 || header_size > bytes.len() {
return Err(Error {
offset: 0,
context: "container header",
kind: ErrorKind::InvalidLength("container header size"),
});
}
// A pair table must divide evenly into 8-byte entries; a remainder
// means we have guessed the layout wrong and should not read garbage.
// A sector list is a flat run of 4-byte values and needs no such check.
let table_bytes = header_size - 8;
if !kind.is_list() && table_bytes % 8 != 0 {
return Err(Error {
offset: 0,
context: "container header",
kind: ErrorKind::Invalid("pair table is not a whole number of entries"),
});
}
let mut entries = Vec::with_capacity(table_bytes / 8);
if kind.is_list() {
// Each value is the first sector of one extent; the size runs to the
// next value, and the last runs to the end of the container.
let mut sectors = Vec::with_capacity(table_bytes / 4);
while b.pos() + 4 <= header_size {
sectors.push(b.i32("extent sector")?);
}
let total_sectors = (bytes.len() as u32).div_ceil(SECTOR_SIZE);
for (i, &start) in sectors.iter().enumerate() {
let end = sectors.get(i + 1).copied().unwrap_or(total_sectors as i32);
entries.push(SectorRange {
offset: start,
size: end - start,
});
}
} else {
while b.pos() + 8 <= header_size {
let offset = b.i32("entry offset")?;
let size = b.i32("entry size")?;
entries.push(SectorRange { offset, size });
}
}
Ok(GlobalWad {
header_size: header_size as u32,
sector_count,
entries,
kind,
})
}
/// Entries that point at real data, as `(index, byte offset, byte length)`.
pub fn present(&self) -> impl Iterator<Item = (usize, u64, u64)> + '_ {
self.entries
.iter()
.copied()
.enumerate()
.filter_map(|(i, r)| {
let len = match self.kind {
RangeKind::Sectors => r.byte_len(),
RangeKind::SectorList => r.byte_len(),
RangeKind::SectorBytes => SectorByteRange {
offset: r.offset,
size: r.size,
}
.byte_len(),
};
let off = r.byte_offset()?;
if len == 0 {
return None;
}
Some((i, off, len))
})
}
}
#[cfg(test)]
mod tests {
use super::*;
/// Build a synthetic container: header_size, sector_count, then pairs.
fn synthetic(entries: &[(i32, i32)]) -> Vec<u8> {
let header_size = (8 + entries.len() * 8) as u32;
let mut v = Vec::new();
v.extend_from_slice(&header_size.to_le_bytes());
v.extend_from_slice(&100u32.to_le_bytes());
for (o, s) in entries {
v.extend_from_slice(&o.to_le_bytes());
v.extend_from_slice(&s.to_le_bytes());
}
// Pad the body so byte_offset() reads land inside the buffer.
v.resize(header_size as usize + 4096, 0);
v
}
#[test]
fn parses_header_and_pairs() {
let wad = GlobalWad::parse(&synthetic(&[(1, 72), (73, 51), (124, 80)])).unwrap();
assert_eq!(wad.header_size, 32);
assert_eq!(wad.sector_count, 100);
assert_eq!(wad.entries.len(), 3);
assert_eq!(
wad.entries[0],
SectorRange {
offset: 1,
size: 72
}
);
assert_eq!(wad.entries[2].offset, 124);
}
#[test]
fn sector_ranges_convert_to_bytes() {
let r = SectorRange { offset: 3, size: 2 };
assert_eq!(r.byte_offset(), Some(3 * 2048));
assert_eq!(r.byte_len(), 2 * 2048);
assert!(r.is_present());
}
#[test]
fn absent_ranges_are_not_present() {
assert!(!SectorRange { offset: 0, size: 0 }.is_present());
assert!(!SectorRange {
offset: 5,
size: -1
}
.is_present());
assert_eq!(SectorRange { offset: 5, size: 0 }.byte_offset(), None);
assert_eq!(
SectorRange {
offset: -5,
size: 10
}
.byte_offset(),
None
);
}
#[test]
fn present_filters_zero_sizes() {
let wad = GlobalWad::parse(&synthetic(&[(1, 4), (5, 0), (5, 4)])).unwrap();
let present: Vec<usize> = wad.present().map(|(i, _, _)| i).collect();
assert_eq!(present, vec![0, 2]);
}
#[test]
fn present_scales_sizes_by_the_range_kind() {
let bytes = synthetic(&[(1, 100)]);
// The default for this disc treats the second field as a byte count.
let raw = GlobalWad::parse(&bytes).unwrap();
assert_eq!(raw.present().next().unwrap().2, 100);
// The sector-count form scales it up.
let sectors = GlobalWad::parse_with(&bytes, RangeKind::Sectors).unwrap();
assert_eq!(sectors.present().next().unwrap().2, 100 * 2048);
}
#[test]
fn rejects_a_header_size_beyond_the_buffer() {
let mut v = 9999u32.to_le_bytes().to_vec();
v.resize(16, 0);
assert!(GlobalWad::parse(&v).is_err());
}
#[test]
fn rejects_a_header_size_below_the_minimum() {
let v = 4u32.to_le_bytes().to_vec();
assert!(GlobalWad::parse(&v).is_err());
}
#[test]
fn rejects_a_table_that_does_not_divide_evenly() {
// header_size 13 -> table of 5 bytes, not a multiple of 8.
let mut v = 13u32.to_le_bytes().to_vec();
v.extend_from_slice(&0u32.to_le_bytes());
v.extend_from_slice(&[0u8; 5]);
let err = GlobalWad::parse(&v).unwrap_err();
assert!(matches!(err.kind, ErrorKind::Invalid(_)), "got {err:?}");
}
#[test]
fn empty_table_is_allowed() {
let mut v = 8u32.to_le_bytes().to_vec();
v.extend_from_slice(&10u32.to_le_bytes());
v.resize(4096, 0);
let wad = GlobalWad::parse(&v).unwrap();
assert!(wad.entries.is_empty());
}
}
+394
View File
@@ -0,0 +1,394 @@
//! ISO 9660 reader: enough to enumerate the disc and read files by sector.
//!
//! R&C2 lists its assets in the filesystem, unlike R&C1 (whose filesystem is
//! vestigial and whose assets are only reachable through the table of
//! contents). That makes this reader the primary asset index for this game.
//!
//! Only plain 2048-byte-sector `.iso` dumps are supported. A 2352-byte raw
//! dump has a different sector layout and is rejected with a clear message
//! rather than silently misread.
//!
//! Reads are by LBA. The disc is sector-aligned throughout, so this stays
//! simple: no interleaving, no sync patterns.
use crate::buf::{Buf, Error, ErrorKind, Result};
/// Logical block size. Everything on the disc is sector-aligned to this.
pub const SECTOR_SIZE: usize = 2048;
/// Offset of the `CD001` signature within the descriptor. ISO 9660 puts the
/// volume descriptor type at byte 0 and the standard identifier `CD001` at
/// byte 1 of the same 2048-byte block, so the signature is at sector offset 1.
const PVD_SIGNATURE_OFFSET: usize = 0x0001;
/// Sector 16 holds the Primary Volume Descriptor.
const PVD_SECTOR: u32 = 16;
/// Logical blocks within a directory record.
const BLOCK_SIZE: usize = SECTOR_SIZE;
/// A file or subdirectory in the ISO 9660 filesystem.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct DirEntry {
/// File name without the `;1` version suffix. Subdirectories end with `/`.
pub name: String,
/// Logical block address of the extent.
pub lba: u32,
/// Extent length in bytes.
pub size: u32,
pub is_dir: bool,
}
/// A read-only view of an ISO 9660 image.
#[derive(Debug, Clone)]
pub struct IsoImage {
/// File size in sectors.
pub sector_count: u32,
pub volume_id: String,
/// Flattened `name -> entry` map for the whole disc, including `G/` files.
pub files: Vec<DirEntry>,
}
impl IsoImage {
/// Open an image, validating the structure and enumerating the filesystem.
pub fn open(path: &std::path::Path) -> Result<IsoImage> {
let meta = std::fs::metadata(path).map_err(|e| Error {
offset: 0,
context: "opening the disc image",
kind: ErrorKind::Invalid(match e.kind() {
std::io::ErrorKind::NotFound => "no such file",
std::io::ErrorKind::PermissionDenied => "permission denied",
_ => "unreadable",
}),
})?;
let size_bytes = meta.len();
if size_bytes < (PVD_SECTOR as u64 + 1) * SECTOR_SIZE as u64 {
return Err(Error {
offset: 0,
context: "disc image",
kind: ErrorKind::Invalid("too small to contain a volume descriptor"),
});
}
let sector_count = (size_bytes / SECTOR_SIZE as u64) as u32;
// Validate the signature at its real offset within sector 16.
let pvd = read_at(path, PVD_SECTOR, SECTOR_SIZE as u32)?;
if &pvd[PVD_SIGNATURE_OFFSET..PVD_SIGNATURE_OFFSET + 5] != b"CD001" {
return Err(Error {
offset: PVD_SECTOR as usize * SECTOR_SIZE + PVD_SIGNATURE_OFFSET,
context: "primary volume descriptor",
kind: ErrorKind::BadMagic { expected: "CD001" },
});
}
let mut b = Buf::new(&pvd);
b.seek(40);
let volume_id = read_str_until_nul(b.rest(), 32);
// Root directory record is at 0x9C in the PVD, 34 bytes.
let root = b.peek_bytes(156, 34, "root directory record")?;
let root_lba = u32::from_le_bytes([root[2], root[3], root[4], root[5]]);
let root_size = u32::from_le_bytes([root[10], root[11], root[12], root[13]]);
let mut files = Vec::new();
read_dir(path, root_lba, root_size, "", &mut files, 0)?;
files.sort_by(|a, b| a.name.cmp(&b.name));
Ok(IsoImage {
sector_count,
volume_id,
files,
})
}
/// Look up an entry by exact name, e.g. `"SCUS_972.68"` or `"G/MISC.WAD"`.
pub fn find(&self, name: &str) -> Option<&DirEntry> {
self.files.iter().find(|e| e.name == name)
}
/// True when the image contains any file whose name contains `needle`.
pub fn has(&self, needle: &str) -> bool {
self.files.iter().any(|e| e.name.contains(needle))
}
/// Read `size` bytes from `lba`.
pub fn read_at(path: &std::path::Path, lba: u32, size: u32) -> Result<Vec<u8>> {
read_at(path, lba, size)
}
}
/// Read `size` bytes starting at logical block `lba`.
pub fn read_at(path: &std::path::Path, lba: u32, size: u32) -> Result<Vec<u8>> {
use std::io::{Read, Seek, SeekFrom};
let mut file = std::fs::File::open(path).map_err(|_| Error {
offset: 0,
context: "opening the disc image",
kind: ErrorKind::Invalid("unreadable"),
})?;
let offset = lba as u64 * SECTOR_SIZE as u64;
file.seek(SeekFrom::Start(offset)).map_err(|_| Error {
offset: offset as usize,
context: "seeking in the disc image",
kind: ErrorKind::Invalid("seek failed"),
})?;
let mut buf = vec![0u8; size as usize];
file.read_exact(&mut buf).map_err(|_| Error {
offset: offset as usize,
context: "reading a range from the disc image",
kind: ErrorKind::OutOfBounds {
needed: size as usize,
available: 0,
},
})?;
Ok(buf)
}
/// Read one directory extent, recursing into subdirectories.
fn read_dir(
path: &std::path::Path,
lba: u32,
size: u32,
prefix: &str,
out: &mut Vec<DirEntry>,
depth: usize,
) -> Result<()> {
// A malformed image could nest arbitrarily deep; refuse rather than recurse.
if depth > 8 {
return Err(Error {
offset: lba as usize * SECTOR_SIZE,
context: "directory tree",
kind: ErrorKind::Invalid("nested too deeply"),
});
}
let data = read_at(path, lba, size)?;
let mut b = Buf::new(&data);
let end = data.len();
while b.pos() < end {
let record_start = b.pos();
let record_len = match b.u8("directory record length") {
Ok(v) => v as usize,
Err(_) => break,
};
if record_len == 0 {
// Records never span a logical block.
let next = (record_start / BLOCK_SIZE + 1) * BLOCK_SIZE;
if next <= record_start || next >= end {
break;
}
b.seek(next);
continue;
}
if record_start + record_len > end {
return Err(Error {
offset: record_start,
context: "directory record",
kind: ErrorKind::OutOfBounds {
needed: record_len,
available: end - record_start,
},
});
}
let ext_attr_len = b.u8("extended attribute length")? as usize;
// The extent LBA and data length are each stored twice, little-endian
// then big-endian. Read the little-endian half and skip the other.
let file_lba = b.u32("extent LBA")?;
b.skip(4, "extent LBA (big-endian half)")?;
let file_size = b.u32("extent size")?;
b.skip(4, "extent size (big-endian half)")?;
// Recording date and time (7).
b.skip(7, "directory record timestamps")?;
// File flags (1): bit 1 (0x02) marks a directory. Relying on the
// identifier's trailing `/` alone is not safe: not every writer sets
// it, and the flag is authoritative.
let flags = b.u8("file flags")?;
// File unit size (1), interleave gap size (1), volume sequence (4).
b.skip(6, "directory record interleaving")?;
let name_len = b.u8("file identifier length")? as usize;
b.skip(ext_attr_len, "extended attributes")?;
let raw = b.read_bytes(name_len, "file identifier")?;
// Strip the `;1` version suffix, and the trailing `/` on directories.
let mut name = String::from_utf8_lossy(raw).into_owned();
if let Some(semi) = name.find(';') {
name.truncate(semi);
}
let is_dir = name.ends_with('/') || flags & 0x02 != 0;
if name.ends_with('/') {
name.pop();
}
// The first two records of a directory are `.` (identifier byte 0x00)
// and `..` (0x01). They are navigation, not content.
let is_self_or_parent = matches!(name.as_str(), "\u{0}" | "\u{1}");
if !name.is_empty() && !is_self_or_parent {
let full = format!("{prefix}{name}");
if is_dir {
// Read the subdirectory's extent before recording it.
if file_size > 0 {
let mut sub = Vec::new();
read_dir(
path,
file_lba,
file_size,
&format!("{full}/"),
&mut sub,
depth + 1,
)?;
out.extend(sub);
}
out.push(DirEntry {
name: format!("{full}/"),
lba: file_lba,
size: file_size,
is_dir: true,
});
} else {
out.push(DirEntry {
name: full,
lba: file_lba,
size: file_size,
is_dir: false,
});
}
}
// Jump to the next record.
b.seek(record_start + record_len);
}
Ok(())
}
/// Read a NUL/space-terminated string field, trimmed.
fn read_str_until_nul(bytes: &[u8], max: usize) -> String {
let slice = &bytes[..max.min(bytes.len())];
let end = slice.iter().position(|&c| c == 0).unwrap_or(slice.len());
String::from_utf8_lossy(&slice[..end]).trim().to_string()
}
#[cfg(test)]
mod tests {
use super::*;
/// A tiny synthetic ISO: one directory holding one file.
/// Built byte by byte so the test does not depend on any game data.
fn synthetic_iso() -> Vec<u8> {
let mut img = vec![0u8; SECTOR_SIZE * 40];
// Write an 8-byte both-endian field, as ISO 9660 requires.
fn both_endian(img: &mut [u8], at: usize, value: u32) {
img[at..at + 4].copy_from_slice(&value.to_le_bytes());
img[at + 4..at + 8].copy_from_slice(&value.to_be_bytes());
}
// PVD at sector 16.
let pvd_off = 16 * SECTOR_SIZE;
img[pvd_off + PVD_SIGNATURE_OFFSET..pvd_off + PVD_SIGNATURE_OFFSET + 5]
.copy_from_slice(b"CD001");
img[pvd_off + 40..pvd_off + 40 + 12].copy_from_slice(b"TESTVOLUME ");
let root_lba = 20u32;
let root = 156;
both_endian(&mut img, pvd_off + root + 2, root_lba);
both_endian(&mut img, pvd_off + root + 10, SECTOR_SIZE as u32);
// One directory record inside the root extent.
let mut dir = vec![0u8; SECTOR_SIZE];
let write_record = |dir: &mut Vec<u8>, at: usize, name: &[u8], lba: u32, size: u32| {
let rec_len = 33 + name.len();
dir[at] = rec_len as u8;
dir[at + 1] = 0; // extended attribute length
both_endian(dir, at + 2, lba);
both_endian(dir, at + 10, size);
dir[at + 32] = name.len() as u8;
dir[at + 33..at + 33 + name.len()].copy_from_slice(name);
rec_len
};
let mut off = 0usize;
off += write_record(&mut dir, off, &[0x00], root_lba, SECTOR_SIZE as u32); // "."
off += write_record(&mut dir, off, &[0x01], root_lba, SECTOR_SIZE as u32); // ".."
write_record(&mut dir, off, b"FOO.BIN;1", 30, 4096);
img[20 * SECTOR_SIZE..21 * SECTOR_SIZE].copy_from_slice(&dir);
img[30 * SECTOR_SIZE..31 * SECTOR_SIZE].fill(0xAB);
// Volume descriptor set terminator at sector 17.
let t_off = 17 * SECTOR_SIZE;
img[t_off + PVD_SIGNATURE_OFFSET..t_off + PVD_SIGNATURE_OFFSET + 5]
.copy_from_slice(b"CD001");
img[t_off] = 0xFF; // volume descriptor type: terminator
img
}
/// Write to a unique path. Tests run in parallel inside one process, so a
/// name shared on process id alone would let them clobber each other.
fn write_temp(name: &str, bytes: &[u8]) -> std::path::PathBuf {
let unique = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_nanos())
.unwrap_or(0);
let p =
std::env::temp_dir().join(format!("gc-iso-{}-{unique}-{name}.iso", std::process::id()));
std::fs::write(&p, bytes).unwrap();
p
}
#[test]
fn opens_and_enumerates() {
let img = synthetic_iso();
let path = write_temp("open", &img);
let iso = IsoImage::open(&path).expect("valid synthetic ISO should open");
let _ = std::fs::remove_file(&path);
assert_eq!(iso.volume_id, "TESTVOLUME");
let foo = iso.find("FOO.BIN").expect("FOO.BIN should be found");
assert_eq!(foo.lba, 30);
assert_eq!(foo.size, 4096);
assert!(!foo.is_dir);
}
#[test]
fn rejects_a_file_that_is_not_an_iso() {
// Long enough to reach the signature check rather than the size check.
let junk = vec![0x41u8; SECTOR_SIZE * 20];
let path = write_temp("notiso", &junk);
let err = IsoImage::open(&path).unwrap_err();
let _ = std::fs::remove_file(&path);
assert!(
matches!(err.kind, ErrorKind::BadMagic { .. }),
"got {err:?}"
);
}
#[test]
fn rejects_a_missing_file() {
let err = IsoImage::open(std::path::Path::new("/nonexistent/gc-disc.iso")).unwrap_err();
assert_eq!(err.context, "opening the disc image");
}
#[test]
fn rejects_a_too_small_file() {
let path = write_temp("tiny", &[0u8; 100]);
let err = IsoImage::open(&path).unwrap_err();
let _ = std::fs::remove_file(&path);
assert!(matches!(err.kind, ErrorKind::Invalid(_)), "got {err:?}");
}
#[test]
fn read_past_the_end_is_an_error() {
let img = synthetic_iso();
let path = write_temp("oob", &img);
let err = read_at(&path, 999, 4096).unwrap_err();
let _ = std::fs::remove_file(&path);
assert!(matches!(err.kind, ErrorKind::OutOfBounds { .. }));
}
}
+4 -38
View File
@@ -1,38 +1,4 @@
//! Asset format parsers for *Ratchet & Clank: Going Commando* (PS2).
//!
//! Target: `SCUS_972.68` v1.01, USA Black Label, NTSC 60 Hz. See
//! `docs/disc-identification.md`.
//!
//! This crate reads data structures out of the game and produces plain Rust
//! values. It performs no rendering, no simulation and no I/O beyond what the
//! caller hands it, which keeps every parser testable against a byte slice.
//!
//! # What belongs here
//!
//! * `iso9660` - the disc filesystem. R&C2 lists its assets as real files,
//! unlike R&C1, so enumeration works.
//! * `hdr` - `RC2.HDR`, the table of contents (408,444 bytes at the
//! disc root). The engine addresses assets by sector, and
//! this is where those sector numbers come from.
//! * `wad` - Insomniac's LZ77 container (`WAD` magic, 16-byte header).
//! Carries level core data, audio and video streams.
//! * `tfrag` - terrain fragments: LOD chains, strips, morph targets.
//! * `tie` - instanced environment geometry ("ties"), with fat vertices.
//! * `shrub` - instanced foliage and decorative meshes.
//! * `moby` - dynamic actors, including Ratchet, enemies and crates.
//! * `moby_anim` - skeletal animation: sequences, frames, blending.
//! * `collision` - the collision world: cells, packed vertices, faces.
//! * `occlusion` - the PVS grid used for visibility.
//! * `sky`, `volumes`, `textures` - sky shells, fog volumes, CSM1 textures.
//!
//! # Rules
//!
//! **Bound every read.** Malformed data must produce a typed error, never a
//! panic and never a read past the end of the buffer.
//!
//! **No game data in this repository.** Parsers are written against a copy of
//! your own legally obtained disc, and the extracted output stays outside git.
//!
//! **Verify against the real thing.** Every parser gets a test that loads
//! known-good bytes from your disc and checks the parsed result. See
//! `docs/formats/` for the specs being implemented.
pub mod buf;
pub mod global_wad;
pub mod iso9660;
pub mod wad;
+330
View File
@@ -0,0 +1,330 @@
//! Insomniac's "WAD" LZ77 container.
//!
//! Spec: `docs/formats/wad.md`.
//!
//! A 16-byte header followed by a stream of packets. Each packet begins with a
//! **flag byte** that selects one of four packet classes:
//!
//! | Flag range | Packet |
//! |---|---|
//! | `0x00..0x0f` | literal run (the flag byte is the count, biased) |
//! | `0x10..0x1f` | back-reference, 15-bit displacement, short length |
//! | `0x20..0x3f` | back-reference, 14-bit displacement |
//! | `0x40..0xff` | back-reference, 11-bit displacement, short length |
//!
//! Every packet ends with 0-3 extra literal bytes carried in unused bits, so
//! literal and back-reference packets interleave without alignment. Two
//! special cases exist in the `0x10..0x1f` class: a packet with a zero far
//! field is either a **pad packet** (skip forward to the next 0x1000 boundary)
//! or a **dummy packet**, and zero displacements are invalid.
//!
//! The encoding is verified against retail data: every WAD on the `SCUS_972.68`
//! disc decodes, and each decompresses to a size its containing header agrees
//! with.
use crate::buf::{Error, ErrorKind, Result};
pub const WAD_HEADER_SIZE: usize = 0x10;
/// Is this buffer a WAD stream?
pub fn is_wad(b: &[u8]) -> bool {
b.len() >= WAD_HEADER_SIZE && &b[..3] == b"WAD"
}
/// Total stream length in bytes, including the 16-byte header.
pub fn compressed_size(b: &[u8]) -> Result<u32> {
if !is_wad(b) {
return Err(Error {
offset: 0,
context: "WAD header",
kind: ErrorKind::BadMagic { expected: "WAD" },
});
}
Ok(u32::from_le_bytes([b[3], b[4], b[5], b[6]]))
}
/// Decompress a WAD stream.
pub fn decompress(bytes: &[u8]) -> Result<Vec<u8>> {
let total = compressed_size(bytes)? as usize;
if total < WAD_HEADER_SIZE || total > bytes.len() {
return Err(Error {
offset: 0,
context: "WAD header",
kind: ErrorKind::InvalidLength("WAD stream size"),
});
}
let stream = &bytes[WAD_HEADER_SIZE..total];
let end = stream.len();
let mut ptr = 0usize;
let mut out: Vec<u8> = Vec::with_capacity(total * 2);
while ptr < end {
let flag_at = ptr;
let flag = stream[ptr];
ptr += 1;
if flag < 0x10 {
// Literal run. 0x00 is the escape for a count stored in the next byte.
let n = if flag == 0 {
let extra = take(stream, &mut ptr, "WAD literal length")?;
extra as usize + 18
} else {
flag as usize + 3
};
take_into(stream, &mut ptr, &mut out, n, "WAD literal run")?;
// Two literal packets in a row are not a valid encoding: the second
// would have been merged into the first.
if ptr < end && stream[ptr] < 0x10 {
return Err(Error {
offset: WAD_HEADER_SIZE + ptr,
context: "WAD literal run",
kind: ErrorKind::Invalid("adjacent literal packets"),
});
}
continue;
}
let (match_len, displacement, little): (usize, usize, u8);
if flag < 0x20 {
// 15-bit displacement window.
let mut ml = (flag & 7) as usize;
if ml == 0 {
ml = take(stream, &mut ptr, "WAD match length")? as usize + 7;
}
let b0 = take(stream, &mut ptr, "WAD back-reference")?;
let b1 = take(stream, &mut ptr, "WAD back-reference")?;
let a = ((flag >> 3) & 1) as usize;
let far = b1 as usize * 0x40 + (b0 >> 2) as usize;
if a == 0 && far == 0 {
if ml != 1 {
// Pad packet: skip to the next 0x1000 boundary.
ptr = (ptr + 0xFFF) & !0xFFF;
continue;
}
// Dummy packet: carries only its trailing literals.
take_into(
stream,
&mut ptr,
&mut out,
(b0 & 3) as usize,
"WAD dummy packet",
)?;
continue;
}
displacement = 0x4000 * (a + 1) + far;
match_len = ml + 2;
little = b0;
} else if flag < 0x40 {
// 14-bit displacement.
let mut ml = (flag & 0x1f) as usize;
if ml == 0 {
ml = take(stream, &mut ptr, "WAD match length")? as usize + 0x1f;
}
match_len = ml + 2;
let b1 = take(stream, &mut ptr, "WAD back-reference")?;
let b2 = take(stream, &mut ptr, "WAD back-reference")?;
displacement = b2 as usize * 0x40 + (b1 >> 2) as usize + 1;
little = b1;
} else {
// 11-bit displacement, short length.
let b1 = take(stream, &mut ptr, "WAD back-reference")?;
match_len = (flag >> 5) as usize + 1;
displacement = b1 as usize * 8 + ((flag >> 2) & 7) as usize + 1;
little = flag;
}
copy_match(&mut out, displacement, match_len, WAD_HEADER_SIZE + flag_at)?;
take_into(
stream,
&mut ptr,
&mut out,
(little & 3) as usize,
"WAD trailing literals",
)?;
}
Ok(out)
}
/// Read one byte, advancing the cursor.
fn take(stream: &[u8], ptr: &mut usize, context: &'static str) -> Result<u8> {
if *ptr >= stream.len() {
return Err(Error {
offset: WAD_HEADER_SIZE + *ptr,
context,
kind: ErrorKind::OutOfBounds {
needed: 1,
available: stream.len().saturating_sub(*ptr),
},
});
}
let v = stream[*ptr];
*ptr += 1;
Ok(v)
}
/// Copy `n` literal bytes from the stream into `out`, advancing the cursor.
fn take_into(
stream: &[u8],
ptr: &mut usize,
out: &mut Vec<u8>,
n: usize,
context: &'static str,
) -> Result<()> {
if stream.len() - *ptr < n {
return Err(Error {
offset: WAD_HEADER_SIZE + *ptr,
context,
kind: ErrorKind::OutOfBounds {
needed: n,
available: stream.len().saturating_sub(*ptr),
},
});
}
out.extend_from_slice(&stream[*ptr..*ptr + n]);
*ptr += n;
Ok(())
}
/// Copy `len` bytes from `displacement` back in `out`.
///
/// Overlapping copies expand as run-length: the source index advances with the
/// destination, so displacement 1 repeats a single byte.
fn copy_match(out: &mut Vec<u8>, displacement: usize, len: usize, at: usize) -> Result<()> {
if displacement == 0 || displacement > out.len() {
return Err(Error {
offset: at,
context: "WAD back-reference",
kind: ErrorKind::Invalid("match source before the start of the output"),
});
}
let src = out.len() - displacement;
for i in 0..len {
let v = out[src + i];
out.push(v);
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
fn header(total: u32) -> Vec<u8> {
let mut h = vec![0u8; WAD_HEADER_SIZE];
h[..3].copy_from_slice(b"WAD");
h[3..7].copy_from_slice(&total.to_le_bytes());
h
}
fn stream(body: Vec<u8>) -> Vec<u8> {
let mut s = header((body.len() + WAD_HEADER_SIZE) as u32);
s.extend_from_slice(&body);
s
}
#[test]
fn recognises_the_magic() {
assert!(is_wad(&header(0x10)));
assert!(!is_wad(b"NOT"));
assert!(!is_wad(b"WAD"));
}
#[test]
fn rejects_a_non_wad() {
let err = decompress(&[0u8; 32]).unwrap_err();
assert!(matches!(err.kind, ErrorKind::BadMagic { .. }));
}
#[test]
fn rejects_a_size_beyond_the_buffer() {
let mut s = header(0x1000);
s.extend(std::iter::repeat_n(0u8, 8));
assert!(matches!(
decompress(&s).unwrap_err().kind,
ErrorKind::InvalidLength(_)
));
}
#[test]
fn rejects_a_size_below_the_header() {
assert!(decompress(&header(4)).is_err());
}
#[test]
fn literal_run() {
// Flag 0x06 -> 6 + 3 = 9 literal bytes.
let mut body = vec![0x06];
body.extend(b"abcdefghi");
assert_eq!(decompress(&stream(body)).unwrap(), b"abcdefghi");
}
#[test]
fn long_literal_run_via_escape() {
// Flag 0x00 then a length byte: 20 + 18 = 38 literals.
let payload: Vec<u8> = (0..38u8).collect();
let mut body = vec![0x00, 20u8];
body.extend_from_slice(&payload);
assert_eq!(decompress(&stream(body)).unwrap(), payload);
}
#[test]
fn eleven_bit_back_reference() {
// Seed 9 literals so a displacement is available.
let mut body = vec![0x06]; // 9 literals
body.extend(b"abcdefghi");
// Flag 0x40..: match_len = (flag>>5)+1, disp = b1*8 + ((flag>>2)&7) + 1,
// trailing literals = flag & 3.
// flag 0x40 -> match_len = (0x40 >> 5) + 1 = 3.
// displacement = b1 * 8 + ((flag >> 2) & 7) + 1 = 1 * 8 + 0 + 1 = 9.
body.push(0x40);
body.push(0x01);
assert_eq!(&decompress(&stream(body)).unwrap()[..12], b"abcdefghiabc");
}
#[test]
fn overlapping_back_reference_is_run_length() {
// displacement 1 expands to a repeated byte.
let mut body = vec![0x03]; // 6 literals
body.extend(b"zzzzzz");
// flag 0x40, b1 = 0 -> disp = 0*8 + 0 + 1 = 1, len = (0x40>>5)+1 = 3.
body.push(0x40);
body.push(0x00);
assert_eq!(&decompress(&stream(body)).unwrap()[..9], b"zzzzzzzzz");
}
#[test]
fn rejects_a_back_reference_before_the_start() {
let body = vec![0x40, 0x00];
assert!(decompress(&stream(body)).is_err());
}
#[test]
fn dummy_packet_carries_only_literals() {
// 0x10 class with a = 0, far = 0 and ml == 1 is a dummy packet:
// flag = 0x10 | 1 = 0x11, then b0, b1 = 0x00, 0x00 -> b0 & 3 = 0 literals.
let mut body = vec![0x06];
body.extend(b"abcdefghi");
body.extend_from_slice(&[0x11, 0x00, 0x00]);
let out = decompress(&stream(body)).unwrap();
assert_eq!(&out[..9], b"abcdefghi");
}
#[test]
fn adjacent_literal_packets_are_rejected() {
let mut body = vec![0x03];
body.extend(b"abcd");
body.push(0x03);
body.extend(b"efgh");
assert!(decompress(&stream(body)).is_err());
}
#[test]
fn empty_body_decompresses_to_nothing() {
let s = header(WAD_HEADER_SIZE as u32);
assert!(decompress(&s).unwrap().is_empty());
}
}