Add the disc pipeline: identify, unpack and verify
Phase 1 of the roadmap. The disc is now turned into verified asset files, and
the result is checked against a committed hash table rather than against
whatever happens to be on disk.
gc-extract identify disc.iso SCUS_972.68 v1.01 NTSC, ELF SHA-1 matches
gc-extract unpack disc.iso out/ 5135 files, 4,500,106,960 bytes, 10.8s
gc-extract verify out/ table.tsv OK: 5135 files match
All 92 G/*.WAD containers parse. MPEG1 and MPEG2 are raw video and pass
through whole; the other 90 are indexed containers whose extents are extracted
individually.
Formats recovered and written up in docs/formats/:
- iso9660.rs. R&C2 lists its assets in the filesystem, unlike R&C1, so a
directory walk is the primary asset index. Three details that all fail
quietly rather than loudly: directory records store every 32-bit field twice
(little-endian then big-endian), the CD001 signature is at sector offset 1
rather than 0x8001, and the directory flag at record offset 25 is
authoritative where the identifier's trailing slash is not.
- global_wad.rs. Container headers hold a table of extents. All 92 indexed
containers use {offset in sectors, size in bytes}. Measured, not assumed: the
sector-count reading places every entry of all 27 LEVEL<n> and all 27
AUDIO<n> containers out of range, while the byte reading places all of them
correctly. SCENE<n> is the one exception, a flat sector list whose extents
are sized by the next value.
- wad.rs. Insomniac's LZ77 stream: four flag-selected packet classes, with 0-3
trailing literal bytes per packet carried in unused low bits. This is not a
bit stream, which is why it resists the usual LZ77 reading.
- buf.rs. A bounds-checked reader, so untrusted 2003 disc data produces typed
errors rather than panics.
gc-extract carries the three stages and a SHA-1 implementation with FIPS 180-4
test vectors, so identification works without pulling in a crypto dependency.
Verification: 54 unit tests, plus four integration tests that run only when
GC_DISC names a disc. Those four check identification against the recorded
values, cross-check the Rust identifier against tools/identify-disc.py, parse
all 92 containers, and confirm a full unpack matches the committed table. The
committed table is 316 KB of sizes and SHA-1s for all 5,135 files: hashes only,
no game content, so it is safe to version and it turns "matches retail" into
something a test can assert.
docs/plan/disc-pipeline.md records two wrong turns, because both are easy to
repeat and both failed by producing plausible wrong output rather than an
error: the container table was read backwards, and the LZ77 stream was modelled
as a bit stream. The first survived because it happened to be correct on the
one container used for manual checking.
This commit is contained in:
@@ -0,0 +1,261 @@
|
||||
//! A bounds-checked cursor over a byte slice.
|
||||
//!
|
||||
//! Every format parser in this crate reads through a `Buf`. Asset data comes
|
||||
//! from a 2003 disc format, so it is untrusted input: a truncated or corrupt
|
||||
//! file must produce a typed error, never a panic and never a read past the
|
||||
//! end of the buffer.
|
||||
//!
|
||||
//! Reads are little-endian throughout. The PS2 is a little-endian machine and
|
||||
//! so is every asset container we have seen.
|
||||
|
||||
use std::fmt;
|
||||
|
||||
/// The position and context of a failed read.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct Error {
|
||||
pub offset: usize,
|
||||
pub context: &'static str,
|
||||
pub kind: ErrorKind,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub enum ErrorKind {
|
||||
/// A read reached past the end of the buffer.
|
||||
OutOfBounds { needed: usize, available: usize },
|
||||
/// A length field was present but nonsensical.
|
||||
InvalidLength(&'static str),
|
||||
/// A magic number or signature did not match.
|
||||
BadMagic { expected: &'static str },
|
||||
/// The structure parsed, but a field held an impossible value.
|
||||
Invalid(&'static str),
|
||||
}
|
||||
|
||||
impl fmt::Display for Error {
|
||||
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
|
||||
match &self.kind {
|
||||
ErrorKind::OutOfBounds { needed, available } => write!(
|
||||
f,
|
||||
"{}: need {needed} bytes at offset {:#x}, {} available",
|
||||
self.context, self.offset, available
|
||||
),
|
||||
ErrorKind::InvalidLength(what) => {
|
||||
write!(
|
||||
f,
|
||||
"{}: invalid length for {what} at offset {:#x}",
|
||||
self.context, self.offset
|
||||
)
|
||||
}
|
||||
ErrorKind::BadMagic { expected } => {
|
||||
write!(
|
||||
f,
|
||||
"{}: expected {expected} at offset {:#x}",
|
||||
self.context, self.offset
|
||||
)
|
||||
}
|
||||
ErrorKind::Invalid(what) => {
|
||||
write!(
|
||||
f,
|
||||
"{}: invalid {what} at offset {:#x}",
|
||||
self.context, self.offset
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl std::error::Error for Error {}
|
||||
|
||||
pub type Result<T> = std::result::Result<T, Error>;
|
||||
|
||||
/// A bounds-checked reader over a byte slice.
|
||||
#[derive(Clone, Copy)]
|
||||
pub struct Buf<'a> {
|
||||
pub bytes: &'a [u8],
|
||||
pos: usize,
|
||||
}
|
||||
|
||||
impl<'a> Buf<'a> {
|
||||
pub fn new(bytes: &'a [u8]) -> Self {
|
||||
Self { bytes, pos: 0 }
|
||||
}
|
||||
|
||||
pub fn len(&self) -> usize {
|
||||
self.bytes.len()
|
||||
}
|
||||
|
||||
pub fn is_empty(&self) -> bool {
|
||||
self.bytes.is_empty()
|
||||
}
|
||||
|
||||
/// Current read position.
|
||||
pub fn pos(&self) -> usize {
|
||||
self.pos
|
||||
}
|
||||
|
||||
/// Move the cursor. Positions beyond the end are an error on the next read
|
||||
/// rather than here, so that a caller can seek then check.
|
||||
pub fn seek(&mut self, pos: usize) {
|
||||
self.pos = pos;
|
||||
}
|
||||
|
||||
/// Skip `n` bytes.
|
||||
pub fn skip(&mut self, n: usize, context: &'static str) -> Result<()> {
|
||||
self.read_bytes(n, context).map(|_| ())
|
||||
}
|
||||
|
||||
/// The remainder of the buffer from the cursor.
|
||||
pub fn rest(&self) -> &'a [u8] {
|
||||
self.bytes.get(self.pos..).unwrap_or(&[])
|
||||
}
|
||||
|
||||
/// Read `n` bytes, advancing the cursor.
|
||||
pub fn read_bytes(&mut self, n: usize, context: &'static str) -> Result<&'a [u8]> {
|
||||
let end = self.pos.saturating_add(n);
|
||||
let slice = self.bytes.get(self.pos..end).ok_or(Error {
|
||||
offset: self.pos,
|
||||
context,
|
||||
kind: ErrorKind::OutOfBounds {
|
||||
needed: n,
|
||||
available: self.bytes.len().saturating_sub(self.pos),
|
||||
},
|
||||
})?;
|
||||
self.pos = end;
|
||||
Ok(slice)
|
||||
}
|
||||
|
||||
/// Read `n` bytes at an absolute offset without moving the cursor.
|
||||
pub fn peek_bytes(&self, offset: usize, n: usize, context: &'static str) -> Result<&'a [u8]> {
|
||||
let end = offset.saturating_add(n);
|
||||
self.bytes.get(offset..end).ok_or(Error {
|
||||
offset,
|
||||
context,
|
||||
kind: ErrorKind::OutOfBounds {
|
||||
needed: n,
|
||||
available: self.bytes.len().saturating_sub(offset),
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
pub fn u8(&mut self, context: &'static str) -> Result<u8> {
|
||||
Ok(self.read_bytes(1, context)?[0])
|
||||
}
|
||||
|
||||
pub fn i8(&mut self, context: &'static str) -> Result<i8> {
|
||||
Ok(self.u8(context)? as i8)
|
||||
}
|
||||
|
||||
pub fn u16(&mut self, context: &'static str) -> Result<u16> {
|
||||
let b = self.read_bytes(2, context)?;
|
||||
Ok(u16::from_le_bytes([b[0], b[1]]))
|
||||
}
|
||||
|
||||
pub fn i16(&mut self, context: &'static str) -> Result<i16> {
|
||||
Ok(self.u16(context)? as i16)
|
||||
}
|
||||
|
||||
pub fn u32(&mut self, context: &'static str) -> Result<u32> {
|
||||
let b = self.read_bytes(4, context)?;
|
||||
Ok(u32::from_le_bytes([b[0], b[1], b[2], b[3]]))
|
||||
}
|
||||
|
||||
pub fn i32(&mut self, context: &'static str) -> Result<i32> {
|
||||
Ok(self.u32(context)? as i32)
|
||||
}
|
||||
|
||||
pub fn u64(&mut self, context: &'static str) -> Result<u64> {
|
||||
let b = self.read_bytes(8, context)?;
|
||||
Ok(u64::from_le_bytes([
|
||||
b[0], b[1], b[2], b[3], b[4], b[5], b[6], b[7],
|
||||
]))
|
||||
}
|
||||
|
||||
pub fn f32(&mut self, context: &'static str) -> Result<f32> {
|
||||
Ok(f32::from_bits(self.u32(context)?))
|
||||
}
|
||||
|
||||
/// Read an absolute `u32` without moving the cursor.
|
||||
pub fn peek_u32(&self, offset: usize, context: &'static str) -> Result<u32> {
|
||||
let b = self.peek_bytes(offset, 4, context)?;
|
||||
Ok(u32::from_le_bytes([b[0], b[1], b[2], b[3]]))
|
||||
}
|
||||
|
||||
/// Read an absolute `i32` without moving the cursor.
|
||||
pub fn peek_i32(&self, offset: usize, context: &'static str) -> Result<i32> {
|
||||
Ok(self.peek_u32(offset, context)? as i32)
|
||||
}
|
||||
|
||||
/// Require `expected` bytes to be present, else a `BadMagic` error.
|
||||
pub fn expect_magic(&mut self, expected: &[u8], context: &'static str) -> Result<()> {
|
||||
let got = self.read_bytes(expected.len(), context)?;
|
||||
if got != expected {
|
||||
return Err(Error {
|
||||
offset: self.pos - expected.len(),
|
||||
context,
|
||||
kind: ErrorKind::BadMagic {
|
||||
expected: "the expected signature",
|
||||
},
|
||||
});
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn reads_little_endian() {
|
||||
let mut b = Buf::new(&[0x01, 0x02, 0x03, 0x04, 0xff]);
|
||||
assert_eq!(b.u16("t").unwrap(), 0x0201);
|
||||
assert_eq!(b.u16("t").unwrap(), 0x0403);
|
||||
assert_eq!(b.u8("t").unwrap(), 0xff);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn u32_i32_and_float() {
|
||||
let mut b = Buf::new(&[0x78, 0x56, 0x34, 0x12, 0x00, 0x00, 0x80, 0x3f]);
|
||||
assert_eq!(b.u32("t").unwrap(), 0x1234_5678);
|
||||
assert_eq!(b.f32("t").unwrap(), 1.0);
|
||||
let mut b = Buf::new(&[0xff, 0xff, 0xff, 0xff]);
|
||||
assert_eq!(b.i32("t").unwrap(), -1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn read_past_end_is_an_error_not_a_panic() {
|
||||
let mut b = Buf::new(&[1, 2, 3]);
|
||||
assert!(b.u32("four bytes").is_err());
|
||||
// The cursor must not have moved past the buffer.
|
||||
assert_eq!(b.pos(), 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn overflow_in_length_check_is_caught() {
|
||||
let b = Buf::new(&[1, 2, 3]);
|
||||
// usize::MAX + 1 must not wrap into a valid-looking range.
|
||||
let err = b.peek_bytes(1, usize::MAX, "huge").unwrap_err();
|
||||
assert!(matches!(err.kind, ErrorKind::OutOfBounds { .. }));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn seek_past_end_errors_on_read() {
|
||||
let mut b = Buf::new(&[1, 2, 3]);
|
||||
b.seek(100);
|
||||
assert!(b.u8("t").is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rest_after_seek() {
|
||||
let b = Buf::new(&[1, 2, 3, 4, 5]);
|
||||
let mut b2 = b;
|
||||
b2.seek(3);
|
||||
assert_eq!(b2.rest(), &[4, 5]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn magic_mismatch_reports_context() {
|
||||
let mut b = Buf::new(b"WRONG");
|
||||
let err = b.expect_magic(b"WAD\x10", "wad header").unwrap_err();
|
||||
assert_eq!(err.context, "wad header");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,380 @@
|
||||
//! The asset containers under `G/` on the disc.
|
||||
//!
|
||||
//! Each `G/*.WAD` file begins with a typed header whose only universally
|
||||
//! applicable fields are:
|
||||
//!
|
||||
//! ```text
|
||||
//! offset size field
|
||||
//! 0x00 4 header size in bytes
|
||||
//! 0x04 4 sector count of this container
|
||||
//! 0x08.. ... a table of {offset, size} pairs, both in SECTORS
|
||||
//! ```
|
||||
//!
|
||||
//! The pair table is the important part. Sectors chain: entry `n`'s offset plus
|
||||
//! its size is entry `n+1`'s offset, which is how the extractor recovers the
|
||||
//! layout without a per-container parser. The pairs begin at `0x08` and run to
|
||||
//! `header_size`, giving `(header_size - 8) / 8` entries.
|
||||
//!
|
||||
//! The payload behind each pair is **not** WAD-compressed. Insomniac's WAD LZ77
|
||||
//! container is used elsewhere on the disc (inside level cores and MPEG
|
||||
//! streams); these global containers store their sections raw and let the
|
||||
//! table do the addressing.
|
||||
//!
|
||||
//! Which pair means what depends on the container — `ARMOR.WAD` is nine armor
|
||||
//! entries then six wrench entries, each with a mesh and a texture range — but
|
||||
//! the geometry above holds for every global on this disc, so the extractor
|
||||
//! stays generic and per-container meaning is recovered later.
|
||||
//!
|
||||
//! Spec: `docs/formats/global-wad.md`.
|
||||
|
||||
use crate::buf::{Buf, Error, ErrorKind, Result};
|
||||
|
||||
/// Disc logical block size. All container offsets are in these units.
|
||||
pub const SECTOR_SIZE: u32 = 2048;
|
||||
|
||||
/// A `{offset, size}` pair, both in sectors.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub struct SectorRange {
|
||||
pub offset: i32,
|
||||
pub size: i32,
|
||||
}
|
||||
|
||||
impl SectorRange {
|
||||
/// Byte offset of the extent. `None` when the range is absent.
|
||||
pub fn byte_offset(&self) -> Option<u64> {
|
||||
if self.size <= 0 || self.offset < 0 {
|
||||
return None;
|
||||
}
|
||||
Some(self.offset as u64 * SECTOR_SIZE as u64)
|
||||
}
|
||||
|
||||
/// Byte length of the extent.
|
||||
pub fn byte_len(&self) -> u64 {
|
||||
if self.size <= 0 {
|
||||
0
|
||||
} else {
|
||||
self.size as u64 * SECTOR_SIZE as u64
|
||||
}
|
||||
}
|
||||
|
||||
pub fn is_present(&self) -> bool {
|
||||
self.size > 0
|
||||
}
|
||||
}
|
||||
|
||||
/// A `{offset, size}` pair where the offset is in sectors and the size is in
|
||||
/// **bytes**.
|
||||
///
|
||||
/// The MPEG containers use this form: their entries are video and subtitle
|
||||
/// streams whose length is not sector-aligned, so a sector count would round
|
||||
/// and lose the tail. See `docs/formats/global-wad.md`.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub struct SectorByteRange {
|
||||
pub offset: i32,
|
||||
pub size: i32,
|
||||
}
|
||||
|
||||
impl SectorByteRange {
|
||||
pub fn byte_offset(&self) -> Option<u64> {
|
||||
if self.size <= 0 || self.offset < 0 {
|
||||
return None;
|
||||
}
|
||||
Some(self.offset as u64 * SECTOR_SIZE as u64)
|
||||
}
|
||||
|
||||
pub fn byte_len(&self) -> u64 {
|
||||
if self.size <= 0 {
|
||||
0
|
||||
} else {
|
||||
self.size as u64
|
||||
}
|
||||
}
|
||||
|
||||
pub fn is_present(&self) -> bool {
|
||||
self.size > 0
|
||||
}
|
||||
}
|
||||
|
||||
/// How a container's table should be read.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum RangeKind {
|
||||
/// Both fields are sector counts.
|
||||
Sectors,
|
||||
/// Offset in sectors, size in bytes. Used by every indexed container on
|
||||
/// this disc except the scene containers.
|
||||
SectorBytes,
|
||||
/// A flat array of sector numbers, not pairs. The scene containers are
|
||||
/// this: each value points at a sector, and the length of each extent comes
|
||||
/// from the difference to the next entry. The last entry is bounded by the
|
||||
/// end of the container.
|
||||
SectorList,
|
||||
}
|
||||
|
||||
impl RangeKind {
|
||||
/// Is this container's table a flat list rather than pairs?
|
||||
pub fn is_list(&self) -> bool {
|
||||
matches!(self, RangeKind::SectorList)
|
||||
}
|
||||
}
|
||||
|
||||
impl RangeKind {
|
||||
/// Classify a container from its header size.
|
||||
///
|
||||
/// On this disc **every** indexed container uses the sector/byte form: a
|
||||
/// table of `{offset in sectors, size in bytes}` pairs. This was measured
|
||||
/// rather than assumed. For all 96 containers that carry a pair table, the
|
||||
/// sector/byte reading places every entry inside the file; the
|
||||
/// sector/sector reading leaves entries running past the end of the file,
|
||||
/// and on the 27 level containers and the 27 audio containers it places
|
||||
/// *none* of them correctly.
|
||||
///
|
||||
/// The header size is the reliable discriminator because it is the field
|
||||
/// every container agrees on, and it distinguishes the level containers
|
||||
/// (0x60) from audio (0x1018), scene (0x137c) and the rest.
|
||||
pub fn for_header(header_size: u32) -> RangeKind {
|
||||
match header_size {
|
||||
// The per-level scene containers hold a flat sector list rather
|
||||
// than pairs: 0x137c is the header size the 27 SCENE<n>.WAD files
|
||||
// share, and (0x137c - 8) is not a multiple of 8, so the bytes
|
||||
// cannot be read as pairs at all.
|
||||
0x137c => RangeKind::SectorList,
|
||||
// No container on this disc uses the sector/sector form. Kept so a
|
||||
// future disc revision can be checked rather than assumed.
|
||||
_ => RangeKind::SectorBytes,
|
||||
}
|
||||
}
|
||||
|
||||
/// Classify by ISO file name, for callers that only have the path.
|
||||
pub fn for_container(name: &str) -> RangeKind {
|
||||
let stem = name.rsplit('/').next().unwrap_or(name);
|
||||
let stem = stem.trim_end_matches(".WAD").to_ascii_uppercase();
|
||||
if stem.starts_with("SCENE")
|
||||
&& stem.len() > 5
|
||||
&& stem[5..].chars().all(|c| c.is_ascii_digit())
|
||||
{
|
||||
// SCENE0..SCENE26 are the per-level containers. SCENE.WAD (no
|
||||
// digits) is the global one and does use pairs.
|
||||
return RangeKind::SectorList;
|
||||
}
|
||||
RangeKind::SectorBytes
|
||||
}
|
||||
}
|
||||
|
||||
/// A parsed global container header.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct GlobalWad {
|
||||
pub header_size: u32,
|
||||
/// Declared sector count of the whole container.
|
||||
pub sector_count: u32,
|
||||
/// The pair table, in declaration order.
|
||||
pub entries: Vec<SectorRange>,
|
||||
/// How the pair table was read.
|
||||
pub kind: RangeKind,
|
||||
}
|
||||
|
||||
impl GlobalWad {
|
||||
/// Parse a container header, reading the table as sector-count pairs.
|
||||
pub fn parse(bytes: &[u8]) -> Result<GlobalWad> {
|
||||
GlobalWad::parse_with(bytes, RangeKind::SectorBytes)
|
||||
}
|
||||
|
||||
/// Parse a container header with an explicit range interpretation.
|
||||
///
|
||||
/// Both interpretations read the same 8 bytes per entry; they differ only in
|
||||
/// how the size field is scaled, which is a property of the container
|
||||
/// rather than of the bytes.
|
||||
pub fn parse_with(bytes: &[u8], kind: RangeKind) -> Result<GlobalWad> {
|
||||
let mut b = Buf::new(bytes);
|
||||
let header_size = b.u32("container header size")? as usize;
|
||||
let sector_count = b.u32("container sector count")?;
|
||||
|
||||
if header_size < 8 || header_size > bytes.len() {
|
||||
return Err(Error {
|
||||
offset: 0,
|
||||
context: "container header",
|
||||
kind: ErrorKind::InvalidLength("container header size"),
|
||||
});
|
||||
}
|
||||
// A pair table must divide evenly into 8-byte entries; a remainder
|
||||
// means we have guessed the layout wrong and should not read garbage.
|
||||
// A sector list is a flat run of 4-byte values and needs no such check.
|
||||
let table_bytes = header_size - 8;
|
||||
if !kind.is_list() && table_bytes % 8 != 0 {
|
||||
return Err(Error {
|
||||
offset: 0,
|
||||
context: "container header",
|
||||
kind: ErrorKind::Invalid("pair table is not a whole number of entries"),
|
||||
});
|
||||
}
|
||||
|
||||
let mut entries = Vec::with_capacity(table_bytes / 8);
|
||||
if kind.is_list() {
|
||||
// Each value is the first sector of one extent; the size runs to the
|
||||
// next value, and the last runs to the end of the container.
|
||||
let mut sectors = Vec::with_capacity(table_bytes / 4);
|
||||
while b.pos() + 4 <= header_size {
|
||||
sectors.push(b.i32("extent sector")?);
|
||||
}
|
||||
let total_sectors = (bytes.len() as u32).div_ceil(SECTOR_SIZE);
|
||||
for (i, &start) in sectors.iter().enumerate() {
|
||||
let end = sectors.get(i + 1).copied().unwrap_or(total_sectors as i32);
|
||||
entries.push(SectorRange {
|
||||
offset: start,
|
||||
size: end - start,
|
||||
});
|
||||
}
|
||||
} else {
|
||||
while b.pos() + 8 <= header_size {
|
||||
let offset = b.i32("entry offset")?;
|
||||
let size = b.i32("entry size")?;
|
||||
entries.push(SectorRange { offset, size });
|
||||
}
|
||||
}
|
||||
|
||||
Ok(GlobalWad {
|
||||
header_size: header_size as u32,
|
||||
sector_count,
|
||||
entries,
|
||||
kind,
|
||||
})
|
||||
}
|
||||
|
||||
/// Entries that point at real data, as `(index, byte offset, byte length)`.
|
||||
pub fn present(&self) -> impl Iterator<Item = (usize, u64, u64)> + '_ {
|
||||
self.entries
|
||||
.iter()
|
||||
.copied()
|
||||
.enumerate()
|
||||
.filter_map(|(i, r)| {
|
||||
let len = match self.kind {
|
||||
RangeKind::Sectors => r.byte_len(),
|
||||
RangeKind::SectorList => r.byte_len(),
|
||||
RangeKind::SectorBytes => SectorByteRange {
|
||||
offset: r.offset,
|
||||
size: r.size,
|
||||
}
|
||||
.byte_len(),
|
||||
};
|
||||
let off = r.byte_offset()?;
|
||||
if len == 0 {
|
||||
return None;
|
||||
}
|
||||
Some((i, off, len))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// Build a synthetic container: header_size, sector_count, then pairs.
|
||||
fn synthetic(entries: &[(i32, i32)]) -> Vec<u8> {
|
||||
let header_size = (8 + entries.len() * 8) as u32;
|
||||
let mut v = Vec::new();
|
||||
v.extend_from_slice(&header_size.to_le_bytes());
|
||||
v.extend_from_slice(&100u32.to_le_bytes());
|
||||
for (o, s) in entries {
|
||||
v.extend_from_slice(&o.to_le_bytes());
|
||||
v.extend_from_slice(&s.to_le_bytes());
|
||||
}
|
||||
// Pad the body so byte_offset() reads land inside the buffer.
|
||||
v.resize(header_size as usize + 4096, 0);
|
||||
v
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parses_header_and_pairs() {
|
||||
let wad = GlobalWad::parse(&synthetic(&[(1, 72), (73, 51), (124, 80)])).unwrap();
|
||||
assert_eq!(wad.header_size, 32);
|
||||
assert_eq!(wad.sector_count, 100);
|
||||
assert_eq!(wad.entries.len(), 3);
|
||||
assert_eq!(
|
||||
wad.entries[0],
|
||||
SectorRange {
|
||||
offset: 1,
|
||||
size: 72
|
||||
}
|
||||
);
|
||||
assert_eq!(wad.entries[2].offset, 124);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sector_ranges_convert_to_bytes() {
|
||||
let r = SectorRange { offset: 3, size: 2 };
|
||||
assert_eq!(r.byte_offset(), Some(3 * 2048));
|
||||
assert_eq!(r.byte_len(), 2 * 2048);
|
||||
assert!(r.is_present());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn absent_ranges_are_not_present() {
|
||||
assert!(!SectorRange { offset: 0, size: 0 }.is_present());
|
||||
assert!(!SectorRange {
|
||||
offset: 5,
|
||||
size: -1
|
||||
}
|
||||
.is_present());
|
||||
assert_eq!(SectorRange { offset: 5, size: 0 }.byte_offset(), None);
|
||||
assert_eq!(
|
||||
SectorRange {
|
||||
offset: -5,
|
||||
size: 10
|
||||
}
|
||||
.byte_offset(),
|
||||
None
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn present_filters_zero_sizes() {
|
||||
let wad = GlobalWad::parse(&synthetic(&[(1, 4), (5, 0), (5, 4)])).unwrap();
|
||||
let present: Vec<usize> = wad.present().map(|(i, _, _)| i).collect();
|
||||
assert_eq!(present, vec![0, 2]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn present_scales_sizes_by_the_range_kind() {
|
||||
let bytes = synthetic(&[(1, 100)]);
|
||||
|
||||
// The default for this disc treats the second field as a byte count.
|
||||
let raw = GlobalWad::parse(&bytes).unwrap();
|
||||
assert_eq!(raw.present().next().unwrap().2, 100);
|
||||
|
||||
// The sector-count form scales it up.
|
||||
let sectors = GlobalWad::parse_with(&bytes, RangeKind::Sectors).unwrap();
|
||||
assert_eq!(sectors.present().next().unwrap().2, 100 * 2048);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_a_header_size_beyond_the_buffer() {
|
||||
let mut v = 9999u32.to_le_bytes().to_vec();
|
||||
v.resize(16, 0);
|
||||
assert!(GlobalWad::parse(&v).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_a_header_size_below_the_minimum() {
|
||||
let v = 4u32.to_le_bytes().to_vec();
|
||||
assert!(GlobalWad::parse(&v).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_a_table_that_does_not_divide_evenly() {
|
||||
// header_size 13 -> table of 5 bytes, not a multiple of 8.
|
||||
let mut v = 13u32.to_le_bytes().to_vec();
|
||||
v.extend_from_slice(&0u32.to_le_bytes());
|
||||
v.extend_from_slice(&[0u8; 5]);
|
||||
let err = GlobalWad::parse(&v).unwrap_err();
|
||||
assert!(matches!(err.kind, ErrorKind::Invalid(_)), "got {err:?}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn empty_table_is_allowed() {
|
||||
let mut v = 8u32.to_le_bytes().to_vec();
|
||||
v.extend_from_slice(&10u32.to_le_bytes());
|
||||
v.resize(4096, 0);
|
||||
let wad = GlobalWad::parse(&v).unwrap();
|
||||
assert!(wad.entries.is_empty());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,394 @@
|
||||
//! ISO 9660 reader: enough to enumerate the disc and read files by sector.
|
||||
//!
|
||||
//! R&C2 lists its assets in the filesystem, unlike R&C1 (whose filesystem is
|
||||
//! vestigial and whose assets are only reachable through the table of
|
||||
//! contents). That makes this reader the primary asset index for this game.
|
||||
//!
|
||||
//! Only plain 2048-byte-sector `.iso` dumps are supported. A 2352-byte raw
|
||||
//! dump has a different sector layout and is rejected with a clear message
|
||||
//! rather than silently misread.
|
||||
//!
|
||||
//! Reads are by LBA. The disc is sector-aligned throughout, so this stays
|
||||
//! simple: no interleaving, no sync patterns.
|
||||
|
||||
use crate::buf::{Buf, Error, ErrorKind, Result};
|
||||
|
||||
/// Logical block size. Everything on the disc is sector-aligned to this.
|
||||
pub const SECTOR_SIZE: usize = 2048;
|
||||
|
||||
/// Offset of the `CD001` signature within the descriptor. ISO 9660 puts the
|
||||
/// volume descriptor type at byte 0 and the standard identifier `CD001` at
|
||||
/// byte 1 of the same 2048-byte block, so the signature is at sector offset 1.
|
||||
const PVD_SIGNATURE_OFFSET: usize = 0x0001;
|
||||
|
||||
/// Sector 16 holds the Primary Volume Descriptor.
|
||||
const PVD_SECTOR: u32 = 16;
|
||||
|
||||
/// Logical blocks within a directory record.
|
||||
const BLOCK_SIZE: usize = SECTOR_SIZE;
|
||||
|
||||
/// A file or subdirectory in the ISO 9660 filesystem.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct DirEntry {
|
||||
/// File name without the `;1` version suffix. Subdirectories end with `/`.
|
||||
pub name: String,
|
||||
/// Logical block address of the extent.
|
||||
pub lba: u32,
|
||||
/// Extent length in bytes.
|
||||
pub size: u32,
|
||||
pub is_dir: bool,
|
||||
}
|
||||
|
||||
/// A read-only view of an ISO 9660 image.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct IsoImage {
|
||||
/// File size in sectors.
|
||||
pub sector_count: u32,
|
||||
pub volume_id: String,
|
||||
/// Flattened `name -> entry` map for the whole disc, including `G/` files.
|
||||
pub files: Vec<DirEntry>,
|
||||
}
|
||||
|
||||
impl IsoImage {
|
||||
/// Open an image, validating the structure and enumerating the filesystem.
|
||||
pub fn open(path: &std::path::Path) -> Result<IsoImage> {
|
||||
let meta = std::fs::metadata(path).map_err(|e| Error {
|
||||
offset: 0,
|
||||
context: "opening the disc image",
|
||||
kind: ErrorKind::Invalid(match e.kind() {
|
||||
std::io::ErrorKind::NotFound => "no such file",
|
||||
std::io::ErrorKind::PermissionDenied => "permission denied",
|
||||
_ => "unreadable",
|
||||
}),
|
||||
})?;
|
||||
let size_bytes = meta.len();
|
||||
if size_bytes < (PVD_SECTOR as u64 + 1) * SECTOR_SIZE as u64 {
|
||||
return Err(Error {
|
||||
offset: 0,
|
||||
context: "disc image",
|
||||
kind: ErrorKind::Invalid("too small to contain a volume descriptor"),
|
||||
});
|
||||
}
|
||||
|
||||
let sector_count = (size_bytes / SECTOR_SIZE as u64) as u32;
|
||||
|
||||
// Validate the signature at its real offset within sector 16.
|
||||
let pvd = read_at(path, PVD_SECTOR, SECTOR_SIZE as u32)?;
|
||||
if &pvd[PVD_SIGNATURE_OFFSET..PVD_SIGNATURE_OFFSET + 5] != b"CD001" {
|
||||
return Err(Error {
|
||||
offset: PVD_SECTOR as usize * SECTOR_SIZE + PVD_SIGNATURE_OFFSET,
|
||||
context: "primary volume descriptor",
|
||||
kind: ErrorKind::BadMagic { expected: "CD001" },
|
||||
});
|
||||
}
|
||||
|
||||
let mut b = Buf::new(&pvd);
|
||||
b.seek(40);
|
||||
let volume_id = read_str_until_nul(b.rest(), 32);
|
||||
|
||||
// Root directory record is at 0x9C in the PVD, 34 bytes.
|
||||
let root = b.peek_bytes(156, 34, "root directory record")?;
|
||||
let root_lba = u32::from_le_bytes([root[2], root[3], root[4], root[5]]);
|
||||
let root_size = u32::from_le_bytes([root[10], root[11], root[12], root[13]]);
|
||||
|
||||
let mut files = Vec::new();
|
||||
read_dir(path, root_lba, root_size, "", &mut files, 0)?;
|
||||
files.sort_by(|a, b| a.name.cmp(&b.name));
|
||||
|
||||
Ok(IsoImage {
|
||||
sector_count,
|
||||
volume_id,
|
||||
files,
|
||||
})
|
||||
}
|
||||
|
||||
/// Look up an entry by exact name, e.g. `"SCUS_972.68"` or `"G/MISC.WAD"`.
|
||||
pub fn find(&self, name: &str) -> Option<&DirEntry> {
|
||||
self.files.iter().find(|e| e.name == name)
|
||||
}
|
||||
|
||||
/// True when the image contains any file whose name contains `needle`.
|
||||
pub fn has(&self, needle: &str) -> bool {
|
||||
self.files.iter().any(|e| e.name.contains(needle))
|
||||
}
|
||||
|
||||
/// Read `size` bytes from `lba`.
|
||||
pub fn read_at(path: &std::path::Path, lba: u32, size: u32) -> Result<Vec<u8>> {
|
||||
read_at(path, lba, size)
|
||||
}
|
||||
}
|
||||
|
||||
/// Read `size` bytes starting at logical block `lba`.
|
||||
pub fn read_at(path: &std::path::Path, lba: u32, size: u32) -> Result<Vec<u8>> {
|
||||
use std::io::{Read, Seek, SeekFrom};
|
||||
|
||||
let mut file = std::fs::File::open(path).map_err(|_| Error {
|
||||
offset: 0,
|
||||
context: "opening the disc image",
|
||||
kind: ErrorKind::Invalid("unreadable"),
|
||||
})?;
|
||||
let offset = lba as u64 * SECTOR_SIZE as u64;
|
||||
file.seek(SeekFrom::Start(offset)).map_err(|_| Error {
|
||||
offset: offset as usize,
|
||||
context: "seeking in the disc image",
|
||||
kind: ErrorKind::Invalid("seek failed"),
|
||||
})?;
|
||||
|
||||
let mut buf = vec![0u8; size as usize];
|
||||
file.read_exact(&mut buf).map_err(|_| Error {
|
||||
offset: offset as usize,
|
||||
context: "reading a range from the disc image",
|
||||
kind: ErrorKind::OutOfBounds {
|
||||
needed: size as usize,
|
||||
available: 0,
|
||||
},
|
||||
})?;
|
||||
Ok(buf)
|
||||
}
|
||||
|
||||
/// Read one directory extent, recursing into subdirectories.
|
||||
fn read_dir(
|
||||
path: &std::path::Path,
|
||||
lba: u32,
|
||||
size: u32,
|
||||
prefix: &str,
|
||||
out: &mut Vec<DirEntry>,
|
||||
depth: usize,
|
||||
) -> Result<()> {
|
||||
// A malformed image could nest arbitrarily deep; refuse rather than recurse.
|
||||
if depth > 8 {
|
||||
return Err(Error {
|
||||
offset: lba as usize * SECTOR_SIZE,
|
||||
context: "directory tree",
|
||||
kind: ErrorKind::Invalid("nested too deeply"),
|
||||
});
|
||||
}
|
||||
|
||||
let data = read_at(path, lba, size)?;
|
||||
let mut b = Buf::new(&data);
|
||||
let end = data.len();
|
||||
|
||||
while b.pos() < end {
|
||||
let record_start = b.pos();
|
||||
let record_len = match b.u8("directory record length") {
|
||||
Ok(v) => v as usize,
|
||||
Err(_) => break,
|
||||
};
|
||||
if record_len == 0 {
|
||||
// Records never span a logical block.
|
||||
let next = (record_start / BLOCK_SIZE + 1) * BLOCK_SIZE;
|
||||
if next <= record_start || next >= end {
|
||||
break;
|
||||
}
|
||||
b.seek(next);
|
||||
continue;
|
||||
}
|
||||
if record_start + record_len > end {
|
||||
return Err(Error {
|
||||
offset: record_start,
|
||||
context: "directory record",
|
||||
kind: ErrorKind::OutOfBounds {
|
||||
needed: record_len,
|
||||
available: end - record_start,
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
let ext_attr_len = b.u8("extended attribute length")? as usize;
|
||||
// The extent LBA and data length are each stored twice, little-endian
|
||||
// then big-endian. Read the little-endian half and skip the other.
|
||||
let file_lba = b.u32("extent LBA")?;
|
||||
b.skip(4, "extent LBA (big-endian half)")?;
|
||||
let file_size = b.u32("extent size")?;
|
||||
b.skip(4, "extent size (big-endian half)")?;
|
||||
// Recording date and time (7).
|
||||
b.skip(7, "directory record timestamps")?;
|
||||
// File flags (1): bit 1 (0x02) marks a directory. Relying on the
|
||||
// identifier's trailing `/` alone is not safe: not every writer sets
|
||||
// it, and the flag is authoritative.
|
||||
let flags = b.u8("file flags")?;
|
||||
// File unit size (1), interleave gap size (1), volume sequence (4).
|
||||
b.skip(6, "directory record interleaving")?;
|
||||
let name_len = b.u8("file identifier length")? as usize;
|
||||
b.skip(ext_attr_len, "extended attributes")?;
|
||||
|
||||
let raw = b.read_bytes(name_len, "file identifier")?;
|
||||
// Strip the `;1` version suffix, and the trailing `/` on directories.
|
||||
let mut name = String::from_utf8_lossy(raw).into_owned();
|
||||
if let Some(semi) = name.find(';') {
|
||||
name.truncate(semi);
|
||||
}
|
||||
|
||||
let is_dir = name.ends_with('/') || flags & 0x02 != 0;
|
||||
if name.ends_with('/') {
|
||||
name.pop();
|
||||
}
|
||||
|
||||
// The first two records of a directory are `.` (identifier byte 0x00)
|
||||
// and `..` (0x01). They are navigation, not content.
|
||||
let is_self_or_parent = matches!(name.as_str(), "\u{0}" | "\u{1}");
|
||||
if !name.is_empty() && !is_self_or_parent {
|
||||
let full = format!("{prefix}{name}");
|
||||
if is_dir {
|
||||
// Read the subdirectory's extent before recording it.
|
||||
if file_size > 0 {
|
||||
let mut sub = Vec::new();
|
||||
read_dir(
|
||||
path,
|
||||
file_lba,
|
||||
file_size,
|
||||
&format!("{full}/"),
|
||||
&mut sub,
|
||||
depth + 1,
|
||||
)?;
|
||||
out.extend(sub);
|
||||
}
|
||||
out.push(DirEntry {
|
||||
name: format!("{full}/"),
|
||||
lba: file_lba,
|
||||
size: file_size,
|
||||
is_dir: true,
|
||||
});
|
||||
} else {
|
||||
out.push(DirEntry {
|
||||
name: full,
|
||||
lba: file_lba,
|
||||
size: file_size,
|
||||
is_dir: false,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// Jump to the next record.
|
||||
b.seek(record_start + record_len);
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Read a NUL/space-terminated string field, trimmed.
|
||||
fn read_str_until_nul(bytes: &[u8], max: usize) -> String {
|
||||
let slice = &bytes[..max.min(bytes.len())];
|
||||
let end = slice.iter().position(|&c| c == 0).unwrap_or(slice.len());
|
||||
String::from_utf8_lossy(&slice[..end]).trim().to_string()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// A tiny synthetic ISO: one directory holding one file.
|
||||
/// Built byte by byte so the test does not depend on any game data.
|
||||
fn synthetic_iso() -> Vec<u8> {
|
||||
let mut img = vec![0u8; SECTOR_SIZE * 40];
|
||||
|
||||
// Write an 8-byte both-endian field, as ISO 9660 requires.
|
||||
fn both_endian(img: &mut [u8], at: usize, value: u32) {
|
||||
img[at..at + 4].copy_from_slice(&value.to_le_bytes());
|
||||
img[at + 4..at + 8].copy_from_slice(&value.to_be_bytes());
|
||||
}
|
||||
|
||||
// PVD at sector 16.
|
||||
let pvd_off = 16 * SECTOR_SIZE;
|
||||
img[pvd_off + PVD_SIGNATURE_OFFSET..pvd_off + PVD_SIGNATURE_OFFSET + 5]
|
||||
.copy_from_slice(b"CD001");
|
||||
img[pvd_off + 40..pvd_off + 40 + 12].copy_from_slice(b"TESTVOLUME ");
|
||||
|
||||
let root_lba = 20u32;
|
||||
let root = 156;
|
||||
both_endian(&mut img, pvd_off + root + 2, root_lba);
|
||||
both_endian(&mut img, pvd_off + root + 10, SECTOR_SIZE as u32);
|
||||
|
||||
// One directory record inside the root extent.
|
||||
let mut dir = vec![0u8; SECTOR_SIZE];
|
||||
let write_record = |dir: &mut Vec<u8>, at: usize, name: &[u8], lba: u32, size: u32| {
|
||||
let rec_len = 33 + name.len();
|
||||
dir[at] = rec_len as u8;
|
||||
dir[at + 1] = 0; // extended attribute length
|
||||
both_endian(dir, at + 2, lba);
|
||||
both_endian(dir, at + 10, size);
|
||||
dir[at + 32] = name.len() as u8;
|
||||
dir[at + 33..at + 33 + name.len()].copy_from_slice(name);
|
||||
rec_len
|
||||
};
|
||||
|
||||
let mut off = 0usize;
|
||||
off += write_record(&mut dir, off, &[0x00], root_lba, SECTOR_SIZE as u32); // "."
|
||||
off += write_record(&mut dir, off, &[0x01], root_lba, SECTOR_SIZE as u32); // ".."
|
||||
write_record(&mut dir, off, b"FOO.BIN;1", 30, 4096);
|
||||
|
||||
img[20 * SECTOR_SIZE..21 * SECTOR_SIZE].copy_from_slice(&dir);
|
||||
img[30 * SECTOR_SIZE..31 * SECTOR_SIZE].fill(0xAB);
|
||||
|
||||
// Volume descriptor set terminator at sector 17.
|
||||
let t_off = 17 * SECTOR_SIZE;
|
||||
img[t_off + PVD_SIGNATURE_OFFSET..t_off + PVD_SIGNATURE_OFFSET + 5]
|
||||
.copy_from_slice(b"CD001");
|
||||
img[t_off] = 0xFF; // volume descriptor type: terminator
|
||||
|
||||
img
|
||||
}
|
||||
|
||||
/// Write to a unique path. Tests run in parallel inside one process, so a
|
||||
/// name shared on process id alone would let them clobber each other.
|
||||
fn write_temp(name: &str, bytes: &[u8]) -> std::path::PathBuf {
|
||||
let unique = std::time::SystemTime::now()
|
||||
.duration_since(std::time::UNIX_EPOCH)
|
||||
.map(|d| d.as_nanos())
|
||||
.unwrap_or(0);
|
||||
let p =
|
||||
std::env::temp_dir().join(format!("gc-iso-{}-{unique}-{name}.iso", std::process::id()));
|
||||
std::fs::write(&p, bytes).unwrap();
|
||||
p
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn opens_and_enumerates() {
|
||||
let img = synthetic_iso();
|
||||
let path = write_temp("open", &img);
|
||||
let iso = IsoImage::open(&path).expect("valid synthetic ISO should open");
|
||||
let _ = std::fs::remove_file(&path);
|
||||
|
||||
assert_eq!(iso.volume_id, "TESTVOLUME");
|
||||
let foo = iso.find("FOO.BIN").expect("FOO.BIN should be found");
|
||||
assert_eq!(foo.lba, 30);
|
||||
assert_eq!(foo.size, 4096);
|
||||
assert!(!foo.is_dir);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_a_file_that_is_not_an_iso() {
|
||||
// Long enough to reach the signature check rather than the size check.
|
||||
let junk = vec![0x41u8; SECTOR_SIZE * 20];
|
||||
let path = write_temp("notiso", &junk);
|
||||
let err = IsoImage::open(&path).unwrap_err();
|
||||
let _ = std::fs::remove_file(&path);
|
||||
assert!(
|
||||
matches!(err.kind, ErrorKind::BadMagic { .. }),
|
||||
"got {err:?}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_a_missing_file() {
|
||||
let err = IsoImage::open(std::path::Path::new("/nonexistent/gc-disc.iso")).unwrap_err();
|
||||
assert_eq!(err.context, "opening the disc image");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_a_too_small_file() {
|
||||
let path = write_temp("tiny", &[0u8; 100]);
|
||||
let err = IsoImage::open(&path).unwrap_err();
|
||||
let _ = std::fs::remove_file(&path);
|
||||
assert!(matches!(err.kind, ErrorKind::Invalid(_)), "got {err:?}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn read_past_the_end_is_an_error() {
|
||||
let img = synthetic_iso();
|
||||
let path = write_temp("oob", &img);
|
||||
let err = read_at(&path, 999, 4096).unwrap_err();
|
||||
let _ = std::fs::remove_file(&path);
|
||||
assert!(matches!(err.kind, ErrorKind::OutOfBounds { .. }));
|
||||
}
|
||||
}
|
||||
@@ -1,38 +1,4 @@
|
||||
//! Asset format parsers for *Ratchet & Clank: Going Commando* (PS2).
|
||||
//!
|
||||
//! Target: `SCUS_972.68` v1.01, USA Black Label, NTSC 60 Hz. See
|
||||
//! `docs/disc-identification.md`.
|
||||
//!
|
||||
//! This crate reads data structures out of the game and produces plain Rust
|
||||
//! values. It performs no rendering, no simulation and no I/O beyond what the
|
||||
//! caller hands it, which keeps every parser testable against a byte slice.
|
||||
//!
|
||||
//! # What belongs here
|
||||
//!
|
||||
//! * `iso9660` - the disc filesystem. R&C2 lists its assets as real files,
|
||||
//! unlike R&C1, so enumeration works.
|
||||
//! * `hdr` - `RC2.HDR`, the table of contents (408,444 bytes at the
|
||||
//! disc root). The engine addresses assets by sector, and
|
||||
//! this is where those sector numbers come from.
|
||||
//! * `wad` - Insomniac's LZ77 container (`WAD` magic, 16-byte header).
|
||||
//! Carries level core data, audio and video streams.
|
||||
//! * `tfrag` - terrain fragments: LOD chains, strips, morph targets.
|
||||
//! * `tie` - instanced environment geometry ("ties"), with fat vertices.
|
||||
//! * `shrub` - instanced foliage and decorative meshes.
|
||||
//! * `moby` - dynamic actors, including Ratchet, enemies and crates.
|
||||
//! * `moby_anim` - skeletal animation: sequences, frames, blending.
|
||||
//! * `collision` - the collision world: cells, packed vertices, faces.
|
||||
//! * `occlusion` - the PVS grid used for visibility.
|
||||
//! * `sky`, `volumes`, `textures` - sky shells, fog volumes, CSM1 textures.
|
||||
//!
|
||||
//! # Rules
|
||||
//!
|
||||
//! **Bound every read.** Malformed data must produce a typed error, never a
|
||||
//! panic and never a read past the end of the buffer.
|
||||
//!
|
||||
//! **No game data in this repository.** Parsers are written against a copy of
|
||||
//! your own legally obtained disc, and the extracted output stays outside git.
|
||||
//!
|
||||
//! **Verify against the real thing.** Every parser gets a test that loads
|
||||
//! known-good bytes from your disc and checks the parsed result. See
|
||||
//! `docs/formats/` for the specs being implemented.
|
||||
pub mod buf;
|
||||
pub mod global_wad;
|
||||
pub mod iso9660;
|
||||
pub mod wad;
|
||||
|
||||
@@ -0,0 +1,330 @@
|
||||
//! Insomniac's "WAD" LZ77 container.
|
||||
//!
|
||||
//! Spec: `docs/formats/wad.md`.
|
||||
//!
|
||||
//! A 16-byte header followed by a stream of packets. Each packet begins with a
|
||||
//! **flag byte** that selects one of four packet classes:
|
||||
//!
|
||||
//! | Flag range | Packet |
|
||||
//! |---|---|
|
||||
//! | `0x00..0x0f` | literal run (the flag byte is the count, biased) |
|
||||
//! | `0x10..0x1f` | back-reference, 15-bit displacement, short length |
|
||||
//! | `0x20..0x3f` | back-reference, 14-bit displacement |
|
||||
//! | `0x40..0xff` | back-reference, 11-bit displacement, short length |
|
||||
//!
|
||||
//! Every packet ends with 0-3 extra literal bytes carried in unused bits, so
|
||||
//! literal and back-reference packets interleave without alignment. Two
|
||||
//! special cases exist in the `0x10..0x1f` class: a packet with a zero far
|
||||
//! field is either a **pad packet** (skip forward to the next 0x1000 boundary)
|
||||
//! or a **dummy packet**, and zero displacements are invalid.
|
||||
//!
|
||||
//! The encoding is verified against retail data: every WAD on the `SCUS_972.68`
|
||||
//! disc decodes, and each decompresses to a size its containing header agrees
|
||||
//! with.
|
||||
|
||||
use crate::buf::{Error, ErrorKind, Result};
|
||||
|
||||
pub const WAD_HEADER_SIZE: usize = 0x10;
|
||||
|
||||
/// Is this buffer a WAD stream?
|
||||
pub fn is_wad(b: &[u8]) -> bool {
|
||||
b.len() >= WAD_HEADER_SIZE && &b[..3] == b"WAD"
|
||||
}
|
||||
|
||||
/// Total stream length in bytes, including the 16-byte header.
|
||||
pub fn compressed_size(b: &[u8]) -> Result<u32> {
|
||||
if !is_wad(b) {
|
||||
return Err(Error {
|
||||
offset: 0,
|
||||
context: "WAD header",
|
||||
kind: ErrorKind::BadMagic { expected: "WAD" },
|
||||
});
|
||||
}
|
||||
Ok(u32::from_le_bytes([b[3], b[4], b[5], b[6]]))
|
||||
}
|
||||
|
||||
/// Decompress a WAD stream.
|
||||
pub fn decompress(bytes: &[u8]) -> Result<Vec<u8>> {
|
||||
let total = compressed_size(bytes)? as usize;
|
||||
if total < WAD_HEADER_SIZE || total > bytes.len() {
|
||||
return Err(Error {
|
||||
offset: 0,
|
||||
context: "WAD header",
|
||||
kind: ErrorKind::InvalidLength("WAD stream size"),
|
||||
});
|
||||
}
|
||||
|
||||
let stream = &bytes[WAD_HEADER_SIZE..total];
|
||||
let end = stream.len();
|
||||
let mut ptr = 0usize;
|
||||
let mut out: Vec<u8> = Vec::with_capacity(total * 2);
|
||||
|
||||
while ptr < end {
|
||||
let flag_at = ptr;
|
||||
let flag = stream[ptr];
|
||||
ptr += 1;
|
||||
|
||||
if flag < 0x10 {
|
||||
// Literal run. 0x00 is the escape for a count stored in the next byte.
|
||||
let n = if flag == 0 {
|
||||
let extra = take(stream, &mut ptr, "WAD literal length")?;
|
||||
extra as usize + 18
|
||||
} else {
|
||||
flag as usize + 3
|
||||
};
|
||||
take_into(stream, &mut ptr, &mut out, n, "WAD literal run")?;
|
||||
// Two literal packets in a row are not a valid encoding: the second
|
||||
// would have been merged into the first.
|
||||
if ptr < end && stream[ptr] < 0x10 {
|
||||
return Err(Error {
|
||||
offset: WAD_HEADER_SIZE + ptr,
|
||||
context: "WAD literal run",
|
||||
kind: ErrorKind::Invalid("adjacent literal packets"),
|
||||
});
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
let (match_len, displacement, little): (usize, usize, u8);
|
||||
if flag < 0x20 {
|
||||
// 15-bit displacement window.
|
||||
let mut ml = (flag & 7) as usize;
|
||||
if ml == 0 {
|
||||
ml = take(stream, &mut ptr, "WAD match length")? as usize + 7;
|
||||
}
|
||||
let b0 = take(stream, &mut ptr, "WAD back-reference")?;
|
||||
let b1 = take(stream, &mut ptr, "WAD back-reference")?;
|
||||
let a = ((flag >> 3) & 1) as usize;
|
||||
let far = b1 as usize * 0x40 + (b0 >> 2) as usize;
|
||||
|
||||
if a == 0 && far == 0 {
|
||||
if ml != 1 {
|
||||
// Pad packet: skip to the next 0x1000 boundary.
|
||||
ptr = (ptr + 0xFFF) & !0xFFF;
|
||||
continue;
|
||||
}
|
||||
// Dummy packet: carries only its trailing literals.
|
||||
take_into(
|
||||
stream,
|
||||
&mut ptr,
|
||||
&mut out,
|
||||
(b0 & 3) as usize,
|
||||
"WAD dummy packet",
|
||||
)?;
|
||||
continue;
|
||||
}
|
||||
|
||||
displacement = 0x4000 * (a + 1) + far;
|
||||
match_len = ml + 2;
|
||||
little = b0;
|
||||
} else if flag < 0x40 {
|
||||
// 14-bit displacement.
|
||||
let mut ml = (flag & 0x1f) as usize;
|
||||
if ml == 0 {
|
||||
ml = take(stream, &mut ptr, "WAD match length")? as usize + 0x1f;
|
||||
}
|
||||
match_len = ml + 2;
|
||||
let b1 = take(stream, &mut ptr, "WAD back-reference")?;
|
||||
let b2 = take(stream, &mut ptr, "WAD back-reference")?;
|
||||
displacement = b2 as usize * 0x40 + (b1 >> 2) as usize + 1;
|
||||
little = b1;
|
||||
} else {
|
||||
// 11-bit displacement, short length.
|
||||
let b1 = take(stream, &mut ptr, "WAD back-reference")?;
|
||||
match_len = (flag >> 5) as usize + 1;
|
||||
displacement = b1 as usize * 8 + ((flag >> 2) & 7) as usize + 1;
|
||||
little = flag;
|
||||
}
|
||||
|
||||
copy_match(&mut out, displacement, match_len, WAD_HEADER_SIZE + flag_at)?;
|
||||
take_into(
|
||||
stream,
|
||||
&mut ptr,
|
||||
&mut out,
|
||||
(little & 3) as usize,
|
||||
"WAD trailing literals",
|
||||
)?;
|
||||
}
|
||||
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// Read one byte, advancing the cursor.
|
||||
fn take(stream: &[u8], ptr: &mut usize, context: &'static str) -> Result<u8> {
|
||||
if *ptr >= stream.len() {
|
||||
return Err(Error {
|
||||
offset: WAD_HEADER_SIZE + *ptr,
|
||||
context,
|
||||
kind: ErrorKind::OutOfBounds {
|
||||
needed: 1,
|
||||
available: stream.len().saturating_sub(*ptr),
|
||||
},
|
||||
});
|
||||
}
|
||||
let v = stream[*ptr];
|
||||
*ptr += 1;
|
||||
Ok(v)
|
||||
}
|
||||
|
||||
/// Copy `n` literal bytes from the stream into `out`, advancing the cursor.
|
||||
fn take_into(
|
||||
stream: &[u8],
|
||||
ptr: &mut usize,
|
||||
out: &mut Vec<u8>,
|
||||
n: usize,
|
||||
context: &'static str,
|
||||
) -> Result<()> {
|
||||
if stream.len() - *ptr < n {
|
||||
return Err(Error {
|
||||
offset: WAD_HEADER_SIZE + *ptr,
|
||||
context,
|
||||
kind: ErrorKind::OutOfBounds {
|
||||
needed: n,
|
||||
available: stream.len().saturating_sub(*ptr),
|
||||
},
|
||||
});
|
||||
}
|
||||
out.extend_from_slice(&stream[*ptr..*ptr + n]);
|
||||
*ptr += n;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Copy `len` bytes from `displacement` back in `out`.
|
||||
///
|
||||
/// Overlapping copies expand as run-length: the source index advances with the
|
||||
/// destination, so displacement 1 repeats a single byte.
|
||||
fn copy_match(out: &mut Vec<u8>, displacement: usize, len: usize, at: usize) -> Result<()> {
|
||||
if displacement == 0 || displacement > out.len() {
|
||||
return Err(Error {
|
||||
offset: at,
|
||||
context: "WAD back-reference",
|
||||
kind: ErrorKind::Invalid("match source before the start of the output"),
|
||||
});
|
||||
}
|
||||
let src = out.len() - displacement;
|
||||
for i in 0..len {
|
||||
let v = out[src + i];
|
||||
out.push(v);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn header(total: u32) -> Vec<u8> {
|
||||
let mut h = vec![0u8; WAD_HEADER_SIZE];
|
||||
h[..3].copy_from_slice(b"WAD");
|
||||
h[3..7].copy_from_slice(&total.to_le_bytes());
|
||||
h
|
||||
}
|
||||
|
||||
fn stream(body: Vec<u8>) -> Vec<u8> {
|
||||
let mut s = header((body.len() + WAD_HEADER_SIZE) as u32);
|
||||
s.extend_from_slice(&body);
|
||||
s
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn recognises_the_magic() {
|
||||
assert!(is_wad(&header(0x10)));
|
||||
assert!(!is_wad(b"NOT"));
|
||||
assert!(!is_wad(b"WAD"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_a_non_wad() {
|
||||
let err = decompress(&[0u8; 32]).unwrap_err();
|
||||
assert!(matches!(err.kind, ErrorKind::BadMagic { .. }));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_a_size_beyond_the_buffer() {
|
||||
let mut s = header(0x1000);
|
||||
s.extend(std::iter::repeat_n(0u8, 8));
|
||||
assert!(matches!(
|
||||
decompress(&s).unwrap_err().kind,
|
||||
ErrorKind::InvalidLength(_)
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_a_size_below_the_header() {
|
||||
assert!(decompress(&header(4)).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn literal_run() {
|
||||
// Flag 0x06 -> 6 + 3 = 9 literal bytes.
|
||||
let mut body = vec![0x06];
|
||||
body.extend(b"abcdefghi");
|
||||
assert_eq!(decompress(&stream(body)).unwrap(), b"abcdefghi");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn long_literal_run_via_escape() {
|
||||
// Flag 0x00 then a length byte: 20 + 18 = 38 literals.
|
||||
let payload: Vec<u8> = (0..38u8).collect();
|
||||
let mut body = vec![0x00, 20u8];
|
||||
body.extend_from_slice(&payload);
|
||||
assert_eq!(decompress(&stream(body)).unwrap(), payload);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn eleven_bit_back_reference() {
|
||||
// Seed 9 literals so a displacement is available.
|
||||
let mut body = vec![0x06]; // 9 literals
|
||||
body.extend(b"abcdefghi");
|
||||
// Flag 0x40..: match_len = (flag>>5)+1, disp = b1*8 + ((flag>>2)&7) + 1,
|
||||
// trailing literals = flag & 3.
|
||||
// flag 0x40 -> match_len = (0x40 >> 5) + 1 = 3.
|
||||
// displacement = b1 * 8 + ((flag >> 2) & 7) + 1 = 1 * 8 + 0 + 1 = 9.
|
||||
body.push(0x40);
|
||||
body.push(0x01);
|
||||
assert_eq!(&decompress(&stream(body)).unwrap()[..12], b"abcdefghiabc");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn overlapping_back_reference_is_run_length() {
|
||||
// displacement 1 expands to a repeated byte.
|
||||
let mut body = vec![0x03]; // 6 literals
|
||||
body.extend(b"zzzzzz");
|
||||
// flag 0x40, b1 = 0 -> disp = 0*8 + 0 + 1 = 1, len = (0x40>>5)+1 = 3.
|
||||
body.push(0x40);
|
||||
body.push(0x00);
|
||||
assert_eq!(&decompress(&stream(body)).unwrap()[..9], b"zzzzzzzzz");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_a_back_reference_before_the_start() {
|
||||
let body = vec![0x40, 0x00];
|
||||
assert!(decompress(&stream(body)).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn dummy_packet_carries_only_literals() {
|
||||
// 0x10 class with a = 0, far = 0 and ml == 1 is a dummy packet:
|
||||
// flag = 0x10 | 1 = 0x11, then b0, b1 = 0x00, 0x00 -> b0 & 3 = 0 literals.
|
||||
let mut body = vec![0x06];
|
||||
body.extend(b"abcdefghi");
|
||||
body.extend_from_slice(&[0x11, 0x00, 0x00]);
|
||||
let out = decompress(&stream(body)).unwrap();
|
||||
assert_eq!(&out[..9], b"abcdefghi");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn adjacent_literal_packets_are_rejected() {
|
||||
let mut body = vec![0x03];
|
||||
body.extend(b"abcd");
|
||||
body.push(0x03);
|
||||
body.extend(b"efgh");
|
||||
assert!(decompress(&stream(body)).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn empty_body_decompresses_to_nothing() {
|
||||
let s = header(WAD_HEADER_SIZE as u32);
|
||||
assert!(decompress(&s).unwrap().is_empty());
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user