Fix level extents read as bytes, and extract the level geometry
A defect in the Phase 1 extractor, found by reading the data extent's contents: level extents are declared in sector counts, the generic G/ pair table uses byte counts, and level containers were being routed through the generic path. So LEVEL0.WAD's data extent -- 6,484 sectors -- was taken as 6,484 bytes. Every test still passed, because a truncated file is still a file. The extent's own header gave it away: it declares an uncompressed total of 13,277,920 bytes, and 13,277,920 / 6,484 is 2047.7. A ratio landing on 2048 is a unit error, not a compression ratio. Across 27 levels this had been discarding roughly 300 MB of level content. Level containers now go through the typed LevelHeader instead. Level 0 yields: data.bin 13,279,232 the data extent core_data.wad 9,738,678 the geometry, compressed core_data.bin 16,969,296 the geometry, decompressed core_index.bin 31,600 the geometry directory gameplay.bin 659,456 sound_bank.bin 1,503,232 core_data is a WAD LZ77 stream -- the format already implemented for the game -- and it decompresses cleanly. The first real level content this project has recovered. New: LevelDataHeader parses the data extent's own header, a table of byte ranges naming the level overlay executable, the geometry directory, the GS RAM image, the HUD banks and the geometry. Note this is a third meaning for a "size" field on the same disc, after sector counts in the level header and byte counts in the generic table; all three are documented together in level-data.md so the next parser does not rediscover this the hard way. The geometry directory's offsets were checked against the geometry rather than taken on trust: all twelve land inside the 16.9 MB of decompressed core_data, which is what confirms the header is being read correctly. The class tables themselves do not yet decode under the documented layout, and that is recorded as open work in plan/level-assets.md rather than papered over. 75 unit tests, 9 integration tests against the real disc. Hash table regenerated at 5,159 files and 5,671,915,799 bytes -- hashes only, no content. Extraction now writes to ~/Documents/gc-extracted rather than /tmp, which is 16 GB and did not have room for a 5.3 GB output.
This commit is contained in:
@@ -0,0 +1,315 @@
|
||||
//! The level data extent header (`LEVEL<n>.WAD` → `data`).
|
||||
//!
|
||||
//! Spec: `docs/formats/level-data.md`.
|
||||
//!
|
||||
//! The `data` extent is the level's container: it holds the level overlay
|
||||
//! executable, the compressed geometry, the GS RAM image and the texture banks,
|
||||
//! each at a byte offset declared by a table at the very start of the extent.
|
||||
//!
|
||||
//! ```text
|
||||
//! offset size field
|
||||
//! 0x00 8 overlay the level's executable
|
||||
//! 0x08 8 core_index the geometry directory (see core_index.rs)
|
||||
//! 0x10 8 gs_ram Graphics Synthesizer RAM image
|
||||
//! 0x18 8 hud_header
|
||||
//! 0x20 40 hud_banks[5]
|
||||
//! 0x48 8 core_data the compressed geometry
|
||||
//! 0x50 8 transition_textures
|
||||
//! ```
|
||||
//!
|
||||
//! Every field is a `{offset, size}` pair of **byte** values relative to the
|
||||
//! start of the extent. Note this is a third convention on the same disc: the
|
||||
//! level header's extents are sector counts, the `G/` pair tables are byte
|
||||
//! counts, and this is byte counts again.
|
||||
//!
|
||||
//! The table is 0x58 bytes and the first extent starts at 0x80, so bytes 0x58
|
||||
//! to 0x7f are padding.
|
||||
|
||||
use crate::buf::{Buf, Error, ErrorKind, Result};
|
||||
|
||||
pub const LEVEL_DATA_HEADER_SIZE: usize = 0x58;
|
||||
|
||||
/// A `{offset, size}` pair in bytes.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub struct ByteRange {
|
||||
pub offset: u32,
|
||||
pub size: u32,
|
||||
}
|
||||
|
||||
impl ByteRange {
|
||||
pub fn is_present(&self) -> bool {
|
||||
self.size > 0
|
||||
}
|
||||
|
||||
pub fn end(&self) -> u64 {
|
||||
self.offset as u64 + self.size as u64
|
||||
}
|
||||
|
||||
/// The slice this range describes, bounds-checked against `total`.
|
||||
pub fn slice<'a>(&self, data: &'a [u8], total: usize) -> Result<&'a [u8]> {
|
||||
let start = self.offset as usize;
|
||||
let end = self.end() as usize;
|
||||
if end > total || start > total {
|
||||
return Err(Error {
|
||||
offset: start,
|
||||
context: "level data extent",
|
||||
kind: ErrorKind::OutOfBounds {
|
||||
needed: self.size as usize,
|
||||
available: total.saturating_sub(start),
|
||||
},
|
||||
});
|
||||
}
|
||||
Ok(&data[start..end])
|
||||
}
|
||||
}
|
||||
|
||||
/// A named extent inside the level data.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum LevelDataPart {
|
||||
/// The level overlay executable: Ratchet's code for this planet.
|
||||
Overlay,
|
||||
/// The geometry directory.
|
||||
CoreIndex,
|
||||
/// The Graphics Synthesizer RAM image.
|
||||
GsRam,
|
||||
HudHeader,
|
||||
HudBank(u8),
|
||||
/// The compressed geometry.
|
||||
CoreData,
|
||||
TransitionTextures,
|
||||
}
|
||||
|
||||
/// The parsed level data header.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct LevelDataHeader {
|
||||
pub overlay: ByteRange,
|
||||
pub core_index: ByteRange,
|
||||
pub gs_ram: ByteRange,
|
||||
pub hud_header: ByteRange,
|
||||
pub hud_banks: [ByteRange; 5],
|
||||
pub core_data: ByteRange,
|
||||
pub transition_textures: ByteRange,
|
||||
}
|
||||
|
||||
impl LevelDataHeader {
|
||||
/// Parse the header from the start of the extent.
|
||||
pub fn parse(bytes: &[u8]) -> Result<LevelDataHeader> {
|
||||
let mut b = Buf::new(bytes);
|
||||
let range = |b: &mut Buf<'_>, what: &'static str| -> Result<ByteRange> {
|
||||
let offset = b.u32(what)? as i32;
|
||||
let size = b.u32(what)? as i32;
|
||||
// Negative values mark absent extents; normalise rather than let a
|
||||
// negative offset reach a slice index.
|
||||
let range = ByteRange {
|
||||
offset: if offset < 0 { 0 } else { offset as u32 },
|
||||
size: if size < 0 { 0 } else { size as u32 },
|
||||
};
|
||||
if size < 0 || offset < 0 {
|
||||
// Preserve "absent" as a zero size.
|
||||
return Ok(ByteRange { offset: 0, size: 0 });
|
||||
}
|
||||
Ok(range)
|
||||
};
|
||||
|
||||
let overlay = range(&mut b, "overlay range")?;
|
||||
let core_index = range(&mut b, "core_index range")?;
|
||||
let gs_ram = range(&mut b, "gs_ram range")?;
|
||||
let hud_header = range(&mut b, "hud_header range")?;
|
||||
|
||||
let mut hud_banks = [ByteRange { offset: 0, size: 0 }; 5];
|
||||
for slot in hud_banks.iter_mut() {
|
||||
*slot = range(&mut b, "hud bank range")?;
|
||||
}
|
||||
|
||||
let core_data = range(&mut b, "core_data range")?;
|
||||
let transition_textures = range(&mut b, "transition textures range")?;
|
||||
|
||||
Ok(LevelDataHeader {
|
||||
overlay,
|
||||
core_index,
|
||||
gs_ram,
|
||||
hud_header,
|
||||
hud_banks,
|
||||
core_data,
|
||||
transition_textures,
|
||||
})
|
||||
}
|
||||
|
||||
/// The range for a named part.
|
||||
pub fn part(&self, which: LevelDataPart) -> ByteRange {
|
||||
match which {
|
||||
LevelDataPart::Overlay => self.overlay,
|
||||
LevelDataPart::CoreIndex => self.core_index,
|
||||
LevelDataPart::GsRam => self.gs_ram,
|
||||
LevelDataPart::HudHeader => self.hud_header,
|
||||
LevelDataPart::HudBank(i) => self.hud_banks[i as usize],
|
||||
LevelDataPart::CoreData => self.core_data,
|
||||
LevelDataPart::TransitionTextures => self.transition_textures,
|
||||
}
|
||||
}
|
||||
|
||||
/// Every present extent, named.
|
||||
pub fn present_parts(&self) -> Vec<(LevelDataPart, ByteRange)> {
|
||||
let mut all: Vec<(LevelDataPart, ByteRange)> = vec![
|
||||
(LevelDataPart::Overlay, self.overlay),
|
||||
(LevelDataPart::CoreIndex, self.core_index),
|
||||
(LevelDataPart::GsRam, self.gs_ram),
|
||||
(LevelDataPart::HudHeader, self.hud_header),
|
||||
];
|
||||
for (i, bank) in self.hud_banks.iter().enumerate() {
|
||||
all.push((LevelDataPart::HudBank(i as u8), *bank));
|
||||
}
|
||||
all.push((LevelDataPart::CoreData, self.core_data));
|
||||
all.push((LevelDataPart::TransitionTextures, self.transition_textures));
|
||||
all.into_iter().filter(|(_, r)| r.is_present()).collect()
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// Level 0's real header, read from the disc.
|
||||
fn level0_data() -> Vec<u8> {
|
||||
let mut v = vec![0u8; LEVEL_DATA_HEADER_SIZE];
|
||||
let put = |v: &mut Vec<u8>, off: usize, o: u32, s: u32| {
|
||||
v[off..off + 4].copy_from_slice(&o.to_le_bytes());
|
||||
v[off + 4..off + 8].copy_from_slice(&s.to_le_bytes());
|
||||
};
|
||||
put(&mut v, 0x00, 128, 2_751_268);
|
||||
put(&mut v, 0x08, 2_751_424, 31_600);
|
||||
put(&mut v, 0x10, 2_783_040, 518_656);
|
||||
put(&mut v, 0x18, 3_301_696, 4_060);
|
||||
put(&mut v, 0x20, 3_305_792, 44_935);
|
||||
put(&mut v, 0x28, 3_350_784, 138_675);
|
||||
put(&mut v, 0x30, 3_489_472, 16);
|
||||
put(&mut v, 0x38, 3_489_536, 16);
|
||||
put(&mut v, 0x40, 3_489_600, 34_286);
|
||||
put(&mut v, 0x48, 3_523_904, 9_738_678);
|
||||
put(&mut v, 0x50, 13_262_592, 15_328);
|
||||
v
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parses_the_real_level_zero_header() {
|
||||
let h = LevelDataHeader::parse(&level0_data()).unwrap();
|
||||
assert_eq!(
|
||||
h.overlay,
|
||||
ByteRange {
|
||||
offset: 128,
|
||||
size: 2_751_268
|
||||
}
|
||||
);
|
||||
assert_eq!(
|
||||
h.core_index,
|
||||
ByteRange {
|
||||
offset: 2_751_424,
|
||||
size: 31_600
|
||||
}
|
||||
);
|
||||
assert_eq!(
|
||||
h.gs_ram,
|
||||
ByteRange {
|
||||
offset: 2_783_040,
|
||||
size: 518_656
|
||||
}
|
||||
);
|
||||
assert_eq!(
|
||||
h.core_data,
|
||||
ByteRange {
|
||||
offset: 3_523_904,
|
||||
size: 9_738_678
|
||||
}
|
||||
);
|
||||
assert_eq!(
|
||||
h.transition_textures,
|
||||
ByteRange {
|
||||
offset: 13_262_592,
|
||||
size: 15_328
|
||||
}
|
||||
);
|
||||
assert_eq!(h.hud_banks[2].size, 16);
|
||||
assert_eq!(h.hud_banks[4].size, 34_286);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn extents_are_ordered_and_non_overlapping() {
|
||||
let h = LevelDataHeader::parse(&level0_data()).unwrap();
|
||||
let parts = h.present_parts();
|
||||
// Each extent begins at or after the previous one ends. Gaps exist --
|
||||
// 28 bytes after the overlay, before `core_index` -- so this is an
|
||||
// ordering check, not a tiling one.
|
||||
let mut cursor = 0u64;
|
||||
for (part, r) in &parts {
|
||||
assert!(
|
||||
r.offset as u64 >= cursor,
|
||||
"{part:?} starts at {} but the previous extent ended at {cursor}",
|
||||
r.offset
|
||||
);
|
||||
cursor = r.end();
|
||||
}
|
||||
assert_eq!(
|
||||
cursor, 13_277_920,
|
||||
"the last extent should end at the total"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_first_extent_starts_after_the_header() {
|
||||
let h = LevelDataHeader::parse(&level0_data()).unwrap();
|
||||
// The header is 0x58 bytes and the first extent begins at 0x80, so the
|
||||
// 40 bytes between them are unused.
|
||||
assert_eq!(h.overlay.offset, 0x80);
|
||||
assert!(h.overlay.offset as usize > LEVEL_DATA_HEADER_SIZE);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn all_ranges_lie_inside_the_extent() {
|
||||
let h = LevelDataHeader::parse(&level0_data()).unwrap();
|
||||
// Level 0's data extent is 13,279,232 bytes.
|
||||
let total = 13_279_232usize;
|
||||
for (part, r) in h.present_parts() {
|
||||
assert!(
|
||||
r.end() <= total as u64,
|
||||
"{part:?} ends past the extent: {:?}",
|
||||
r
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn slicing_is_bounds_checked() {
|
||||
let _h = LevelDataHeader::parse(&level0_data()).unwrap();
|
||||
let data = vec![0u8; 1000];
|
||||
// A range that fits.
|
||||
let ok = ByteRange {
|
||||
offset: 10,
|
||||
size: 100,
|
||||
};
|
||||
assert_eq!(ok.slice(&data, 1000).unwrap().len(), 100);
|
||||
// One that does not.
|
||||
let bad = ByteRange {
|
||||
offset: 950,
|
||||
size: 100,
|
||||
};
|
||||
assert!(bad.slice(&data, 1000).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_truncated_header_errors() {
|
||||
let err = LevelDataHeader::parse(&[0u8; 0x20]).unwrap_err();
|
||||
assert!(
|
||||
matches!(err.kind, ErrorKind::OutOfBounds { .. }),
|
||||
"got {err:?}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn absent_extents_are_zeroed() {
|
||||
let mut v = level0_data();
|
||||
v[0x30..0x38].copy_from_slice(&[0u8; 8]);
|
||||
let h = LevelDataHeader::parse(&v).unwrap();
|
||||
assert_eq!(h.hud_banks[2], ByteRange { offset: 0, size: 0 });
|
||||
assert!(!h.part(LevelDataPart::HudBank(2)).is_present());
|
||||
}
|
||||
}
|
||||
@@ -1,5 +1,6 @@
|
||||
pub mod buf;
|
||||
pub mod global_wad;
|
||||
pub mod iso9660;
|
||||
pub mod level_data;
|
||||
pub mod level_wad;
|
||||
pub mod wad;
|
||||
|
||||
Reference in New Issue
Block a user