Parse the level container header, naming each level's four extents
The first typed structure read from real data. Every one of the 27 level containers begins with the same 0x60-byte header, which turns the generic pair table into named assets: data, sound_bank, gameplay and occlusion, plus three chunk descriptors. Two things about this header are worth stating because both are silent failures rather than errors. Sizes here are sector counts, not byte counts. The generic G/ pair table on the same disc uses byte counts, so both readings type-check and the wrong one is off by a factor of 2048. The tiling check settles it: read as sectors, the four extents tile every one of the 27 containers exactly; read as bytes they overlap each other. Level id is not container order. LEVEL21.WAD reports id 30, so a level was cut from the disc and its number left a gap. Anything keying off file order -- a save slot, the planet select screen -- would mislabel that planet. There is also a second trap in the same area: the ISO filesystem sorts names alphabetically, so LEVEL10 precedes LEVEL2, and any tool needing order has to sort numerically. The header's field at 0x04 is zero on all 27 levels. It is not the container size despite looking like one, so the code reads it as a runtime field and takes the container length from the ISO entry instead. Measured across all 27 levels, three invariants now hold in tests: extents tile in file order; extent coverage plus chunks reaches at least the end of the container (21 levels land exactly on its length, 6 leave 134 to 689 sectors of trailing data); and exactly one level's id differs from its container index. The coverage assertion is >= rather than == on purpose -- 21 levels end one sector past the file end because the header occupies sector 0, and asserting equality would truncate a level's tail. gameplay is left unparsed and that is recorded rather than hidden. It begins "WAD" plus a per-level byte and does not decode with the WAD LZ77 decoder: all 27 levels produce plausible output for the first packet and then fail on a rule the format's own encoder never breaks, which means the input is wrong rather than the decoder. docs/plan/level-assets.md records what is ruled out and what to try next. 66 unit tests, plus 7 integration tests against the real disc. The unpack hash table is unchanged, so this adds parsing without altering the pipeline.
This commit is contained in:
@@ -182,3 +182,127 @@ fn every_container_parses() {
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// The level container header is the first typed structure we parse from real
|
||||
/// data, and the tiling invariant is the check that holds on every level rather
|
||||
/// than one hand-picked sample.
|
||||
#[test]
|
||||
fn level_headers_tile_their_containers() {
|
||||
let path = require_disc!();
|
||||
let iso = gc_formats::iso9660::IsoImage::open(&path).expect("ISO should open");
|
||||
|
||||
let mut checked = 0;
|
||||
for entry in iso.files.iter().filter(|e| e.name.starts_with("G/LEVEL")) {
|
||||
let data = gc_formats::iso9660::read_at(&path, entry.lba, 0x60).expect("read header");
|
||||
let h = gc_formats::level_wad::LevelHeader::parse(&data)
|
||||
.unwrap_or_else(|e| panic!("{}: {e}", entry.name));
|
||||
|
||||
assert_eq!(h.header_size, 0x60, "{}", entry.name);
|
||||
assert_eq!(
|
||||
h.runtime_sector, 0,
|
||||
"{}: runtime sector field should be 0 on disc",
|
||||
entry.name
|
||||
);
|
||||
|
||||
// Every present extent must start where the previous one ended.
|
||||
let extents = h.ordered_extents();
|
||||
for w in extents.windows(2) {
|
||||
let (pn, p) = w[0];
|
||||
let (nn, n) = w[1];
|
||||
assert_eq!(
|
||||
p.offset + p.size,
|
||||
n.offset,
|
||||
"{}: gap or overlap between {} and {}",
|
||||
entry.name,
|
||||
pn.name(),
|
||||
nn.name()
|
||||
);
|
||||
}
|
||||
|
||||
// Nothing may fall outside the container. The last extent may end one
|
||||
// sector past the file end through sector rounding, hence `+ 1`.
|
||||
let container_sectors = (entry.size / 2048) as i32;
|
||||
for (name, r) in &extents {
|
||||
assert!(
|
||||
r.offset >= 0 && r.offset + r.size <= container_sectors + 1,
|
||||
"{}: {} extent {:?} is outside the container ({container_sectors} sectors)",
|
||||
entry.name,
|
||||
name.name(),
|
||||
r
|
||||
);
|
||||
}
|
||||
checked += 1;
|
||||
}
|
||||
|
||||
assert_eq!(checked, 27, "expected 27 level containers");
|
||||
}
|
||||
|
||||
/// Level ids are not file order: `LEVEL21.WAD` reports 30. Anything that keys
|
||||
/// off the container index would silently mislabel that planet.
|
||||
#[test]
|
||||
fn level_ids_are_not_container_indices() {
|
||||
let path = require_disc!();
|
||||
let iso = gc_formats::iso9660::IsoImage::open(&path).expect("ISO should open");
|
||||
|
||||
// The ISO filesystem lists names alphabetically, so LEVEL10 sorts before
|
||||
// LEVEL2. Order by the numeric suffix, which is what the container names
|
||||
// mean.
|
||||
let mut levels: Vec<(usize, &gc_formats::iso9660::DirEntry)> = iso
|
||||
.files
|
||||
.iter()
|
||||
.filter(|e| e.name.starts_with("G/LEVEL"))
|
||||
.map(|e| {
|
||||
let n: usize = e
|
||||
.name
|
||||
.trim_start_matches("G/LEVEL")
|
||||
.trim_end_matches(".WAD")
|
||||
.parse()
|
||||
.expect("level container names end in a number");
|
||||
(n, e)
|
||||
})
|
||||
.collect();
|
||||
levels.sort_by_key(|(n, _)| *n);
|
||||
|
||||
let mut mismatches = Vec::new();
|
||||
for (index, (_, entry)) in levels.iter().enumerate() {
|
||||
let data = gc_formats::iso9660::read_at(&path, entry.lba, 0x60).expect("read header");
|
||||
let h = gc_formats::level_wad::LevelHeader::parse(&data).expect("parse");
|
||||
if h.id as usize != index {
|
||||
mismatches.push(format!(
|
||||
"{}: id {} but is container {index}",
|
||||
entry.name, h.id
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
// Exactly one level is out of step: the gap left by a removed level.
|
||||
assert_eq!(
|
||||
mismatches.len(),
|
||||
1,
|
||||
"expected exactly one level whose id differs from its container index, got {mismatches:?}"
|
||||
);
|
||||
assert!(
|
||||
mismatches[0].contains("LEVEL21"),
|
||||
"unexpected mismatch: {mismatches:?}"
|
||||
);
|
||||
}
|
||||
|
||||
/// Sizes in the level header are sector counts. If they were byte counts, the
|
||||
/// extents could not tile a container this size, so this also guards against
|
||||
/// "fixing" the parser to read them as bytes.
|
||||
#[test]
|
||||
fn level_extent_sizes_are_sector_counts() {
|
||||
let path = require_disc!();
|
||||
let iso = gc_formats::iso9660::IsoImage::open(&path).expect("ISO should open");
|
||||
let entry = iso.find("G/LEVEL0.WAD").expect("level 0");
|
||||
|
||||
let data = gc_formats::iso9660::read_at(&path, entry.lba, 0x60).expect("read header");
|
||||
let h = gc_formats::level_wad::LevelHeader::parse(&data).expect("parse");
|
||||
|
||||
// Sound bank occupies sectors 1..735 and data 735..7219.
|
||||
assert_eq!(h.sound_bank.offset, 1);
|
||||
assert_eq!(h.sound_bank.size, 734);
|
||||
assert_eq!(h.data.offset, 735);
|
||||
// Read as bytes, 734 would be under four sectors and the tiling would break.
|
||||
assert_eq!(h.sound_bank.byte_len(), 734 * 2048);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,348 @@
|
||||
//! The level container header (`LEVEL<n>.WAD`).
|
||||
//!
|
||||
//! Spec: `docs/formats/level-wad.md`.
|
||||
//!
|
||||
//! Every one of the 27 level containers begins with the same 0x60-byte header,
|
||||
//! which names five extents and turns the generic pair table into named,
|
||||
//! individually parseable assets:
|
||||
//!
|
||||
//! ```text
|
||||
//! offset size field
|
||||
//! 0x00 4 header size, 0x60 on every level
|
||||
//! 0x04 4 zero on every level; a field the game fills in at runtime
|
||||
//! 0x08 4 level id
|
||||
//! 0x0c 4 reverb configuration index
|
||||
//! 0x10 8 data (geometry: tfrag, tie, shrub, moby, sky, ...)
|
||||
//! 0x18 8 sound_bank
|
||||
//! 0x20 8 gameplay
|
||||
//! 0x28 8 occlusion
|
||||
//! 0x30 48 chunk descriptors (3 x 16)
|
||||
//! ```
|
||||
//!
|
||||
//! `data` is by far the largest and is where the level's visible content lives.
|
||||
//! `gameplay` carries the per-class dispatch tables; `occlusion` is tiny, 12
|
||||
//! sectors on level 0.
|
||||
//!
|
||||
//! **Sizes here are sector counts, not byte counts.** This differs from the
|
||||
//! generic pair table in [`crate::global_wad`], whose sizes are bytes. Both
|
||||
//! appear on the disc and reading one as the other produces extents that look
|
||||
//! plausible and are wrong by a factor of 2048. Verified: on all 27 levels the
|
||||
//! four named extents tile the container in order with no gap and no overlap,
|
||||
//! which only holds if the size field is a sector count.
|
||||
|
||||
use crate::buf::{Buf, Error, ErrorKind, Result};
|
||||
use crate::global_wad::SectorRange;
|
||||
|
||||
/// Header size, identical on all 27 levels.
|
||||
pub const LEVEL_HEADER_SIZE: usize = 0x60;
|
||||
|
||||
/// A parsed level container header.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct LevelHeader {
|
||||
pub header_size: u32,
|
||||
/// Always zero on disc across all 27 levels. The game writes it at load
|
||||
/// time, so it is read but must not be trusted as the container size; use
|
||||
/// the ISO entry's length for that.
|
||||
pub runtime_sector: u32,
|
||||
/// The level's own identifier.
|
||||
///
|
||||
/// Equal to the container index on 26 of 27 levels. `LEVEL21.WAD` reports
|
||||
/// **30**, so the two are not interchangeable: this field is what the game
|
||||
/// calls a level, and the file order is not authoritative. Some level was
|
||||
/// removed from the disc and its number reused by nothing, leaving a gap.
|
||||
pub id: i32,
|
||||
/// Index into the game's reverb configuration.
|
||||
pub reverb: i32,
|
||||
pub data: SectorRange,
|
||||
pub sound_bank: SectorRange,
|
||||
pub gameplay: SectorRange,
|
||||
pub occlusion: SectorRange,
|
||||
/// Three 16-byte chunk descriptors. These follow the named extents and claim
|
||||
/// the remainder of the container on most levels; they are zero on others.
|
||||
pub chunks: [SectorRange; 6],
|
||||
}
|
||||
|
||||
/// A named extent from a level header.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum LevelPart {
|
||||
Data,
|
||||
SoundBank,
|
||||
Gameplay,
|
||||
Occlusion,
|
||||
}
|
||||
|
||||
impl LevelPart {
|
||||
pub fn name(self) -> &'static str {
|
||||
match self {
|
||||
LevelPart::Data => "data",
|
||||
LevelPart::SoundBank => "sound_bank",
|
||||
LevelPart::Gameplay => "gameplay",
|
||||
LevelPart::Occlusion => "occlusion",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl LevelHeader {
|
||||
/// Parse a level container header from the start of the file.
|
||||
pub fn parse(bytes: &[u8]) -> Result<LevelHeader> {
|
||||
let mut b = Buf::new(bytes);
|
||||
|
||||
let header_size = b.u32("level header size")?;
|
||||
if header_size as usize != LEVEL_HEADER_SIZE {
|
||||
return Err(Error {
|
||||
offset: 0,
|
||||
context: "level container header",
|
||||
kind: ErrorKind::Invalid("header size is not 0x60"),
|
||||
});
|
||||
}
|
||||
|
||||
let runtime_sector = b.u32("level runtime sector")?;
|
||||
let id = b.i32("level id")?;
|
||||
let reverb = b.i32("reverb index")?;
|
||||
|
||||
let range = |b: &mut Buf<'_>, what: &'static str| -> Result<SectorRange> {
|
||||
let offset = b.i32(what)?;
|
||||
let size = b.i32(what)?;
|
||||
Ok(SectorRange { offset, size })
|
||||
};
|
||||
|
||||
let data = range(&mut b, "data extent")?;
|
||||
let sound_bank = range(&mut b, "sound_bank extent")?;
|
||||
let gameplay = range(&mut b, "gameplay extent")?;
|
||||
let occlusion = range(&mut b, "occlusion extent")?;
|
||||
|
||||
// 0x30..0x60: six u32s forming three {offset, size} descriptors.
|
||||
let mut chunks = [SectorRange { offset: 0, size: 0 }; 6];
|
||||
for slot in chunks.iter_mut() {
|
||||
slot.offset = b.i32("chunk descriptor")?;
|
||||
slot.size = b.i32("chunk descriptor")?;
|
||||
}
|
||||
|
||||
Ok(LevelHeader {
|
||||
header_size,
|
||||
runtime_sector,
|
||||
id,
|
||||
reverb,
|
||||
data,
|
||||
sound_bank,
|
||||
gameplay,
|
||||
occlusion,
|
||||
chunks,
|
||||
})
|
||||
}
|
||||
|
||||
/// The extent for a named part.
|
||||
pub fn part(&self, which: LevelPart) -> SectorRange {
|
||||
match which {
|
||||
LevelPart::Data => self.data,
|
||||
LevelPart::SoundBank => self.sound_bank,
|
||||
LevelPart::Gameplay => self.gameplay,
|
||||
LevelPart::Occlusion => self.occlusion,
|
||||
}
|
||||
}
|
||||
|
||||
/// The named extents in the order they appear in the file.
|
||||
///
|
||||
/// The order on the disc is sound bank, data, gameplay, occlusion — not the
|
||||
/// order the fields appear in the header.
|
||||
pub fn ordered_extents(&self) -> Vec<(LevelPart, SectorRange)> {
|
||||
[
|
||||
(LevelPart::SoundBank, self.sound_bank),
|
||||
(LevelPart::Data, self.data),
|
||||
(LevelPart::Gameplay, self.gameplay),
|
||||
(LevelPart::Occlusion, self.occlusion),
|
||||
]
|
||||
.into_iter()
|
||||
.filter(|(_, r)| r.is_present())
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Every extent that points at real data, in file order.
|
||||
pub fn present_parts(&self) -> Vec<(LevelPart, SectorRange)> {
|
||||
self.ordered_extents()
|
||||
}
|
||||
|
||||
/// Chunk descriptors that claim real data, as `(index, extent)`.
|
||||
pub fn present_chunks(&self) -> Vec<(usize, SectorRange)> {
|
||||
self.chunks
|
||||
.iter()
|
||||
.copied()
|
||||
.enumerate()
|
||||
.filter(|(_, r)| r.is_present())
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Total sectors claimed by the named extents and the chunks.
|
||||
///
|
||||
/// Cross-check against the container's length in sectors. Measured across
|
||||
/// all 27 levels: 21 land at `length` (the extents start at sector 1, after
|
||||
/// the header, and the last one ends one sector past the file end, which is
|
||||
/// normal sector rounding), and 6 leave 134 to 689 sectors of trailing data
|
||||
/// past the last named extent.
|
||||
///
|
||||
/// So the value is always `>= length - 1`, and that is the invariant worth
|
||||
/// asserting. It is not always equal to `length`, and treating it as such
|
||||
/// would truncate the tail of a level.
|
||||
pub fn payload_sectors(&self) -> i32 {
|
||||
let named: i32 = self.ordered_extents().iter().map(|(_, r)| r.size).sum();
|
||||
let chunks: i32 = self.present_chunks().iter().map(|(_, r)| r.size).sum();
|
||||
named + chunks
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// A header matching LEVEL0.WAD on the retail disc.
|
||||
fn level0_header() -> Vec<u8> {
|
||||
let mut v = vec![0u8; 0x60];
|
||||
v[0x00..0x04].copy_from_slice(&0x60u32.to_le_bytes());
|
||||
// The runtime sector field is zero on every retail level.
|
||||
v[0x04..0x08].copy_from_slice(&0u32.to_le_bytes());
|
||||
v[0x08..0x0c].copy_from_slice(&0i32.to_le_bytes());
|
||||
v[0x0c..0x10].copy_from_slice(&3i32.to_le_bytes());
|
||||
v[0x10..0x14].copy_from_slice(&735i32.to_le_bytes());
|
||||
v[0x14..0x18].copy_from_slice(&6484i32.to_le_bytes());
|
||||
v[0x18..0x1c].copy_from_slice(&1i32.to_le_bytes());
|
||||
v[0x1c..0x20].copy_from_slice(&734i32.to_le_bytes());
|
||||
v[0x20..0x24].copy_from_slice(&7219i32.to_le_bytes());
|
||||
v[0x24..0x28].copy_from_slice(&322i32.to_le_bytes());
|
||||
v[0x28..0x2c].copy_from_slice(&7541i32.to_le_bytes());
|
||||
v[0x2c..0x30].copy_from_slice(&12i32.to_le_bytes());
|
||||
v
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parses_a_retail_level_header() {
|
||||
let h = LevelHeader::parse(&level0_header()).unwrap();
|
||||
assert_eq!(h.header_size, 0x60);
|
||||
assert_eq!(h.runtime_sector, 0);
|
||||
assert_eq!(h.id, 0);
|
||||
assert_eq!(h.reverb, 3);
|
||||
assert_eq!(
|
||||
h.data,
|
||||
SectorRange {
|
||||
offset: 735,
|
||||
size: 6484
|
||||
}
|
||||
);
|
||||
assert_eq!(
|
||||
h.sound_bank,
|
||||
SectorRange {
|
||||
offset: 1,
|
||||
size: 734
|
||||
}
|
||||
);
|
||||
assert_eq!(
|
||||
h.gameplay,
|
||||
SectorRange {
|
||||
offset: 7219,
|
||||
size: 322
|
||||
}
|
||||
);
|
||||
assert_eq!(
|
||||
h.occlusion,
|
||||
SectorRange {
|
||||
offset: 7541,
|
||||
size: 12
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn extents_tile_the_container_in_order() {
|
||||
let h = LevelHeader::parse(&level0_header()).unwrap();
|
||||
// Each extent's end must be the next extent's start, in file order. This
|
||||
// is the check that holds on every level rather than only the sample in
|
||||
// front of us.
|
||||
let parts = h.ordered_extents();
|
||||
for w in parts.windows(2) {
|
||||
assert_eq!(
|
||||
w[0].1.offset + w[0].1.size,
|
||||
w[1].1.offset,
|
||||
"extent gap or overlap between {:?} and {:?}",
|
||||
w[0].1,
|
||||
w[1].1
|
||||
);
|
||||
}
|
||||
// Level 0's container is 7552 sectors; its extents end one sector past
|
||||
// that, which is why `payload_sectors` is compared with `>=` below
|
||||
// rather than for equality.
|
||||
assert!(h.payload_sectors() >= 7552 - 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn payload_sectors_sums_the_named_extents() {
|
||||
let h = LevelHeader::parse(&level0_header()).unwrap();
|
||||
// Sizes are sector counts: 734 + 6484 + 322 + 12.
|
||||
assert_eq!(h.payload_sectors(), 7552);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn chunk_descriptors_add_to_the_remainder() {
|
||||
let mut v = level0_header();
|
||||
// Level 1 declares two chunks after its named extents.
|
||||
v[0x30..0x34].copy_from_slice(&8741i32.to_le_bytes());
|
||||
v[0x34..0x38].copy_from_slice(&1519i32.to_le_bytes());
|
||||
v[0x38..0x3c].copy_from_slice(&10260i32.to_le_bytes());
|
||||
v[0x3c..0x40].copy_from_slice(&116i32.to_le_bytes());
|
||||
let h = LevelHeader::parse(&v).unwrap();
|
||||
assert_eq!(h.present_chunks().len(), 2);
|
||||
// The named extents still total 7552; the two chunks add 1635.
|
||||
assert_eq!(h.payload_sectors(), 7552 + 1519 + 116);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn present_parts_skips_empty_extents() {
|
||||
let mut v = level0_header();
|
||||
v[0x28..0x30].copy_from_slice(&[0u8; 8]);
|
||||
let h = LevelHeader::parse(&v).unwrap();
|
||||
let names: Vec<&str> = h.present_parts().iter().map(|(p, _)| p.name()).collect();
|
||||
assert_eq!(names, vec!["sound_bank", "data", "gameplay"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_a_wrong_header_size() {
|
||||
let mut v = level0_header();
|
||||
v[0x00..0x04].copy_from_slice(&0x68u32.to_le_bytes());
|
||||
let err = LevelHeader::parse(&v).unwrap_err();
|
||||
assert!(matches!(err.kind, ErrorKind::Invalid(_)), "got {err:?}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_a_truncated_header() {
|
||||
// Correct declared size, but the buffer stops early.
|
||||
let mut v = vec![0u8; 0x20];
|
||||
v[0x00..0x04].copy_from_slice(&0x60u32.to_le_bytes());
|
||||
let err = LevelHeader::parse(&v).unwrap_err();
|
||||
assert!(
|
||||
matches!(err.kind, ErrorKind::OutOfBounds { .. }),
|
||||
"got {err:?}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn extent_order_is_sound_bank_then_data_not_header_order() {
|
||||
let h = LevelHeader::parse(&level0_header()).unwrap();
|
||||
let order: Vec<&str> = h.ordered_extents().iter().map(|(p, _)| p.name()).collect();
|
||||
assert_eq!(order, vec!["sound_bank", "data", "gameplay", "occlusion"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn chunk_descriptors_are_read() {
|
||||
let mut v = level0_header();
|
||||
v[0x30..0x34].copy_from_slice(&11i32.to_le_bytes());
|
||||
v[0x34..0x38].copy_from_slice(&22i32.to_le_bytes());
|
||||
let h = LevelHeader::parse(&v).unwrap();
|
||||
assert_eq!(
|
||||
h.chunks[0],
|
||||
SectorRange {
|
||||
offset: 11,
|
||||
size: 22
|
||||
}
|
||||
);
|
||||
// The rest stay zero.
|
||||
assert_eq!(h.chunks[1], SectorRange { offset: 0, size: 0 });
|
||||
}
|
||||
}
|
||||
@@ -1,4 +1,5 @@
|
||||
pub mod buf;
|
||||
pub mod global_wad;
|
||||
pub mod iso9660;
|
||||
pub mod level_wad;
|
||||
pub mod wad;
|
||||
|
||||
Reference in New Issue
Block a user