ReaperOfVeriod
ccea53807a
The shrub region's end was the last unlocated bound in core_data. It was not in the core index, and unlike moby->tie it had no successor boundary to chain against, so the last shrub class in every level ran out to the lump's end and the aggregates over it (the "3.5M records / 28MB of shrub body") were artifacts of that oversized span. Range measured: for each level, from the last shrub class's geometry offset to the constant trailer. Nothing outside that range is read. Method: scan forward from the last shrub class for the pair (a) the constant two bytes f6 7c sitting immediately in front of (b) a chunk signature -- a 16-byte header ending 0xff, ten zero bytes, then four or more 0xf0-tagged offsets. The two halves are each weak alone, and the halves are measured separately: the marker pair occurs 25 to 103 times in a 5 to 17 MB lump, so matching it alone picks dozens of wrong offsets; the chunk shape matches random data often enough to need four tagged offsets before it is believed. Result: on all 27 levels the marker sits exactly two bytes before the first chunk header, and the two parts of the signature are never found apart. No chunk signature occurs inside any last shrub body, on any level. The three levels without shrub (10, 25, 26) carry the same marker before the container that closes tie instead, which is what makes the locator a property of the region rather than a shrub rule. Consequences: - All 358 shrub class bodies are bounded and are whole 8-byte records. That was not assertable while the last class in each level ran to the lump's end, and it is the check the boundary earns its keep for. - 9,600 records in total, against the 8,938 previously measurable in successor-bounded spans; 662 of them sit in the 24 last-class bodies. The shrub region totals 2,033,776 bytes across the 24 levels that have one. - The chunk container is 25,775,850 bytes across 27 levels (1,236 chunks, 22,562 records). The container measurements were previously taken over "the 900,000 bytes before the trailer", a window that did not align to its real extent: on level 0 it is 690,302 bytes, and 27 of 1,123 chunks were truncated at that invented edge. chunk_containers_resolve_on_every_level now reads the region from core_layout instead of measuring backwards from the end of the lump. core_layout takes core_data now: the container's start is the one bound that cannot be derived from the index. It gains the chunks region, so every byte of level-specific core_data belongs to a named region. Where the signature is not found the end stays None -- the region is left visible rather than clamped to the tail. Still open: the marker's meaning (constant across 27 levels, at exactly this position, unnamed) and the shrub body payload's encoding.
399 lines
16 KiB
Rust
399 lines
16 KiB
Rust
//! Region bounds for a level's `core_data`.
|
||
//!
|
||
//! Spec: `docs/formats/core-data-regions.md`.
|
||
//!
|
||
//! This exists because of a specific failure. Region spans were being derived
|
||
//! per-format during probing, usually by running from one known offset to the
|
||
//! next, and usually with the last span running to `core_data.len()`. On level
|
||
//! 0 that made the final shrub class a 925,008-byte span absorbing most of the
|
||
//! lump's tail. Aggregates computed over that -- record counts, value
|
||
//! distributions -- were silently wrong, and nothing errored.
|
||
//!
|
||
//! So bounds are computed once, here, from successor boundaries, and every
|
||
//! caller uses these rather than inventing their own. Where a bound is not yet
|
||
//! known it is `None`, not a guess: [`Region::end`] being `None` means
|
||
//! "unresolved", and [`core_layout`] refuses to close a region whose successor
|
||
//! it cannot find.
|
||
//!
|
||
//! Measured on the disc:
|
||
//!
|
||
//! | region | start | end |
|
||
//! |------------|--------------|-------------------------|
|
||
//! | tfrags | 0 | occlusion |
|
||
//! | occlusion | occlusion | sky |
|
||
//! | sky | sky | collision |
|
||
//! | collision | collision | first moby class |
|
||
//! | moby | first moby | first tie class |
|
||
//! | tie | first tie | first shrub class, |
|
||
//! | | | else the chunk container |
|
||
//! | shrub | first shrub | the chunk container |
|
||
//! | chunks | the chunk | lump length − 0x38A10 |
|
||
//! | | container | |
|
||
//!
|
||
//! The chunk container's start is located by signature rather than read from
|
||
//! the core index: the class table that precedes it carries no successor
|
||
//! offset, so the boundary is found in `core_data` itself -- see
|
||
//! [`crate::chunks::chunk_region_start`]. Every level has one: on the 24 levels
|
||
//! with shrub it sits exactly two bytes after the last shrub body, and on the
|
||
//! three levels without shrub it closes tie instead.
|
||
//!
|
||
//! Texture texels do **not** bound anything: they sit before moby, not after
|
||
//! shrub. On level 0 the texel range is 3539328 to 5326208 while the last
|
||
//! shrub class starts at 16044288.
|
||
|
||
use crate::chunks::chunk_region_start;
|
||
use crate::level_core::{ArrayRange, LevelCoreHeader};
|
||
|
||
/// Bytes at the end of `core_data` that are identical across all 27 levels.
|
||
///
|
||
/// Measured by comparing the levels byte-for-byte from the end: the last
|
||
/// 231,952 bytes (0x38A10) of every level's lump are the same. Level-specific
|
||
/// content therefore ends at `len - 231952`, which is a real bound -- unlike
|
||
/// running a last region out to `core_data.len()`.
|
||
///
|
||
/// This is not yet explained. It may be fixed data, or padding from a
|
||
/// fixed-size decompression buffer. Either way it is constant, so it bounds
|
||
/// the tail region without having to guess what is in it.
|
||
pub const CORE_DATA_TRAILER_BYTES: usize = 0x38A10;
|
||
|
||
/// A byte range within `core_data`.
|
||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||
pub struct Region {
|
||
pub name: &'static str,
|
||
pub start: usize,
|
||
/// Exclusive end, or `None` when the successor boundary is not known.
|
||
pub end: Option<usize>,
|
||
}
|
||
|
||
impl Region {
|
||
pub fn len(&self) -> Option<usize> {
|
||
self.end.map(|e| e.saturating_sub(self.start))
|
||
}
|
||
|
||
pub fn is_empty(&self) -> bool {
|
||
self.len() == Some(0)
|
||
}
|
||
}
|
||
|
||
/// The lowest geometry offset in a class table, ignoring absent entries.
|
||
///
|
||
/// Absent entries are zero, and treating zero as an offset would drag the
|
||
/// region's start to zero -- the same "zero means absent, not position" trap
|
||
/// that the occlusion and collision headers both fell into.
|
||
pub fn first_geometry(index: &[u8], range: ArrayRange, stride: usize) -> Option<usize> {
|
||
let count = range.count.max(0) as usize;
|
||
let base = range.offset.max(0) as usize;
|
||
let mut best: Option<usize> = None;
|
||
for i in 0..count {
|
||
let at = base + i * stride;
|
||
let chunk = index.get(at..at + 4)?;
|
||
let off = i32::from_le_bytes(chunk.try_into().expect("4 bytes"));
|
||
if off > 0 {
|
||
best = Some(best.map_or(off as usize, |b: usize| b.min(off as usize)));
|
||
}
|
||
}
|
||
best
|
||
}
|
||
|
||
/// The highest geometry offset in a class table, ignoring absent entries.
|
||
///
|
||
/// This is the *last* class in the region, which is where the search for the
|
||
/// successor region starts. Starting there rather than at the region's head
|
||
/// matters: the search is for the first chunk signature after this offset, so
|
||
/// beginning it inside the region's own class bodies would risk matching one.
|
||
pub fn last_geometry(index: &[u8], range: ArrayRange, stride: usize) -> Option<usize> {
|
||
let count = range.count.max(0) as usize;
|
||
let base = range.offset.max(0) as usize;
|
||
let mut best: Option<usize> = None;
|
||
for i in 0..count {
|
||
let at = base + i * stride;
|
||
let chunk = index.get(at..at + 4)?;
|
||
let off = i32::from_le_bytes(chunk.try_into().expect("4 bytes"));
|
||
if off > 0 {
|
||
best = Some(best.map_or(off as usize, |b: usize| b.max(off as usize)));
|
||
}
|
||
}
|
||
best
|
||
}
|
||
|
||
/// Compute the region's bounds for one level.
|
||
///
|
||
/// `core` is the level's `core_data` lump. It is needed for one thing only: the
|
||
/// class regions end at the chunk container, whose start is located by
|
||
/// signature in `core` because the class table that precedes it carries no
|
||
/// successor offset.
|
||
pub fn core_layout(index: &[u8], core: &[u8], h: &LevelCoreHeader) -> Vec<Region> {
|
||
let at = |v: i32| -> Option<usize> { (v > 0).then_some(v as usize) };
|
||
let first_moby = first_geometry(index, h.moby_classes, 0x20);
|
||
let first_tie = first_geometry(index, h.tie_classes, 0x20);
|
||
let last_tie = last_geometry(index, h.tie_classes, 0x20);
|
||
let first_shrub = first_geometry(index, h.shrub_classes, 0x30);
|
||
let last_shrub = last_geometry(index, h.shrub_classes, 0x30);
|
||
|
||
// Each region ends where the next one begins. `chained` keeps `None`
|
||
// propagating rather than silently closing a region against a gap.
|
||
fn chained(name: &'static str, start: Option<usize>, next: Option<usize>) -> Option<Region> {
|
||
let start = start?;
|
||
Some(Region {
|
||
name,
|
||
start,
|
||
end: next.filter(|&n| n > start),
|
||
})
|
||
}
|
||
|
||
// The chunk container follows the last of the shrub and tie classes.
|
||
// Searching from the last *class* rather than from the region start keeps
|
||
// the signature search out of the class bodies themselves.
|
||
let container = match (last_shrub, last_tie) {
|
||
(Some(shrub), _) => chunk_region_start(core, shrub),
|
||
(None, Some(tie)) => chunk_region_start(core, tie),
|
||
(None, None) => None,
|
||
};
|
||
|
||
// The final region is bounded by the constant trailer rather than by
|
||
// `core_data.len()`, which would let it absorb the trailer and whatever
|
||
// tail follows the last named region.
|
||
let tail_end = core
|
||
.len()
|
||
.checked_sub(CORE_DATA_TRAILER_BYTES)
|
||
.filter(|&e| e > 0);
|
||
|
||
let regions = [
|
||
chained("tfrags", Some(0), at(h.occlusion)),
|
||
chained("occlusion", at(h.occlusion), at(h.sky)),
|
||
chained("sky", at(h.sky), at(h.collision)),
|
||
chained("collision", at(h.collision), first_moby),
|
||
// A region with no successor region ends at the chunk container, which
|
||
// is what follows the last class on every level.
|
||
chained("moby", first_moby, first_tie.or(container)),
|
||
chained("tie", first_tie, first_shrub.or(container)),
|
||
chained("shrub", first_shrub, container),
|
||
chained("chunks", container, tail_end),
|
||
];
|
||
|
||
regions.into_iter().flatten().collect()
|
||
}
|
||
|
||
#[cfg(test)]
|
||
mod tests {
|
||
use super::*;
|
||
|
||
/// One index buffer carrying both the header fields and the class tables,
|
||
/// so a test cannot accidentally pair a header with unrelated class data.
|
||
fn index(occlusion: i32, sky: i32, collision: i32, classes: &[(usize, &[i32])]) -> Vec<u8> {
|
||
let mut v = vec![0u8; 4096];
|
||
v[0x0c..0x10].copy_from_slice(&occlusion.to_le_bytes());
|
||
v[0x10..0x14].copy_from_slice(&sky.to_le_bytes());
|
||
v[0x14..0x18].copy_from_slice(&collision.to_le_bytes());
|
||
let bases = [0x18usize, 0x20, 0x28];
|
||
for (slot, (offset, vals)) in classes.iter().enumerate() {
|
||
v[bases[slot]..bases[slot] + 4].copy_from_slice(&(vals.len() as i32).to_le_bytes());
|
||
v[bases[slot] + 4..bases[slot] + 8].copy_from_slice(&(*offset as i32).to_le_bytes());
|
||
for (i, g) in vals.iter().enumerate() {
|
||
let at = offset + i * 0x20;
|
||
v[at..at + 4].copy_from_slice(&g.to_le_bytes());
|
||
}
|
||
}
|
||
v
|
||
}
|
||
|
||
fn layout(v: &[u8]) -> Vec<Region> {
|
||
let h = crate::level_core::LevelCoreHeader::parse(v).unwrap();
|
||
core_layout(v, &core(v), &h)
|
||
}
|
||
|
||
/// A zeroed lump, optionally carrying a real chunk-container signature just
|
||
/// past the last class -- the layout locates the container in `core`, so a
|
||
/// core of plain zeros locates nothing.
|
||
fn core(v: &[u8]) -> Vec<u8> {
|
||
let h = crate::level_core::LevelCoreHeader::parse(v).unwrap();
|
||
let mut c = vec![0u8; 1 << 24];
|
||
let last_class = [
|
||
last_geometry(v, h.tie_classes, 0x20),
|
||
last_geometry(v, h.shrub_classes, 0x30),
|
||
]
|
||
.into_iter()
|
||
.flatten()
|
||
.max();
|
||
if let Some(last) = last_class {
|
||
plant_chunk(&mut c, last + 100);
|
||
}
|
||
c
|
||
}
|
||
|
||
/// The marker pair plus a chunk signature at `at`, as the disc carries it.
|
||
fn plant_chunk(core: &mut [u8], at: usize) {
|
||
core[at - 2..at].copy_from_slice(&crate::chunks::CHUNK_REGION_MARKER);
|
||
core[at + 14] = 4;
|
||
core[at + 15] = 0xff;
|
||
for k in 0..4 {
|
||
let word = (crate::chunks::CHUNK_TAG as u32) << 24;
|
||
let at =
|
||
at + crate::chunks::CHUNK_HEADER_BYTES + crate::chunks::CHUNK_GAP_BYTES + k * 4;
|
||
core[at..at + 4].copy_from_slice(&word.to_le_bytes());
|
||
}
|
||
}
|
||
|
||
#[test]
|
||
fn regions_chain_and_do_not_overlap() {
|
||
let v = index(
|
||
2_117_888,
|
||
2_192_576,
|
||
2_615_232,
|
||
&[(200, &[3000]), (400, &[4000]), (600, &[5000])],
|
||
);
|
||
let regions = layout(&v);
|
||
let names: Vec<_> = regions.iter().map(|r| r.name).collect();
|
||
assert_eq!(
|
||
names,
|
||
[
|
||
"tfrags",
|
||
"occlusion",
|
||
"sky",
|
||
"collision",
|
||
"moby",
|
||
"tie",
|
||
"shrub",
|
||
"chunks"
|
||
]
|
||
);
|
||
assert_eq!(regions[0].start, 0);
|
||
assert_eq!(regions[1].start, 2_117_888);
|
||
assert_eq!(regions[1].end, Some(2_192_576));
|
||
assert_eq!(regions[4].start, 3000);
|
||
assert_eq!(regions[4].end, Some(4000));
|
||
for pair in regions.windows(2) {
|
||
if let Some(end) = pair[0].end {
|
||
assert!(
|
||
end <= pair[1].start,
|
||
"{:?} ends after {:?} starts",
|
||
pair[0],
|
||
pair[1]
|
||
);
|
||
}
|
||
}
|
||
}
|
||
|
||
#[test]
|
||
fn a_zero_class_offset_is_absent_not_position_zero() {
|
||
let v = index(
|
||
100,
|
||
200,
|
||
300,
|
||
&[(200, &[0, 3000]), (400, &[4000]), (600, &[5000])],
|
||
);
|
||
let moby = layout(&v).into_iter().find(|r| r.name == "moby").unwrap();
|
||
assert_eq!(moby.start, 3000);
|
||
}
|
||
|
||
#[test]
|
||
fn the_class_regions_end_at_the_chunk_container() {
|
||
let core_len = 1 << 24;
|
||
let v = index(
|
||
100,
|
||
200,
|
||
300,
|
||
&[(200, &[1000]), (400, &[2000]), (600, &[3000])],
|
||
);
|
||
let regions = layout(&v);
|
||
// The container sits 100 bytes past the last shrub class, which is
|
||
// where the search for it starts.
|
||
let container = 3000 + 100;
|
||
let shrub = regions.iter().find(|r| r.name == "shrub").unwrap();
|
||
assert_eq!(shrub.start, 3000);
|
||
assert_eq!(shrub.end, Some(container));
|
||
let chunks = regions.iter().find(|r| r.name == "chunks").unwrap();
|
||
assert_eq!(chunks.start, container);
|
||
// Not core_data.len(): the trailer is constant across levels and does
|
||
// not belong to any region.
|
||
assert_eq!(chunks.end, Some(core_len - CORE_DATA_TRAILER_BYTES));
|
||
assert!(chunks.end.unwrap() < core_len);
|
||
// Tie still ends at shrub, not at the container.
|
||
let tie = regions.iter().find(|r| r.name == "tie").unwrap();
|
||
assert_eq!(tie.end, Some(3000));
|
||
}
|
||
|
||
#[test]
|
||
fn tie_ends_at_the_container_when_there_is_no_shrub() {
|
||
// Third class table absent: tie is the last class region, so it is tie
|
||
// the container closes.
|
||
let v = index(100, 200, 300, &[(200, &[1000]), (400, &[2000])]);
|
||
let h = crate::level_core::LevelCoreHeader::parse(&v).unwrap();
|
||
let core = core(&v);
|
||
let container = 2000 + 100;
|
||
let regions = core_layout(&v, &core, &h);
|
||
let tie = regions.iter().find(|r| r.name == "tie").unwrap();
|
||
assert_eq!(tie.end, Some(container));
|
||
assert!(regions.iter().all(|r| r.name != "shrub"));
|
||
let chunks = regions.iter().find(|r| r.name == "chunks").unwrap();
|
||
assert_eq!(chunks.start, container);
|
||
}
|
||
|
||
#[test]
|
||
fn an_unlocated_container_leaves_the_class_end_unresolved() {
|
||
// A core of plain zeros carries no signature, so shrub cannot be
|
||
// closed -- the end stays visible rather than clamped to the lump's
|
||
// tail, and the chunk region does not exist at all.
|
||
let v = index(
|
||
100,
|
||
200,
|
||
300,
|
||
&[(200, &[1000]), (400, &[2000]), (600, &[3000])],
|
||
);
|
||
let h = crate::level_core::LevelCoreHeader::parse(&v).unwrap();
|
||
let regions = core_layout(&v, &[0u8; 4 << 20], &h);
|
||
let shrub = regions.iter().find(|r| r.name == "shrub").unwrap();
|
||
assert_eq!(shrub.end, None);
|
||
assert!(regions.iter().all(|r| r.name != "chunks"));
|
||
}
|
||
|
||
#[test]
|
||
fn a_lump_shorter_than_the_trailer_leaves_the_end_unresolved() {
|
||
let v = index(
|
||
100,
|
||
200,
|
||
300,
|
||
&[(200, &[1000]), (400, &[2000]), (600, &[3000])],
|
||
);
|
||
let h = crate::level_core::LevelCoreHeader::parse(&v).unwrap();
|
||
// Shorter than the trailer, so the container starts but cannot end.
|
||
let mut core = vec![0u8; CORE_DATA_TRAILER_BYTES - 1];
|
||
plant_chunk(&mut core, 3000 + 100);
|
||
let regions = core_layout(&v, &core, &h);
|
||
let shrub = regions.iter().find(|r| r.name == "shrub").unwrap();
|
||
assert_eq!(shrub.end, Some(3100));
|
||
let chunks = regions.iter().find(|r| r.name == "chunks").unwrap();
|
||
assert_eq!(chunks.end, None, "must not clamp a backwards bound");
|
||
}
|
||
|
||
#[test]
|
||
fn an_absent_region_is_omitted_entirely() {
|
||
let v = index(0, 0, 0, &[]);
|
||
let regions = layout(&v);
|
||
// Only tfrags can be placed at all, and only with an unknown end.
|
||
assert_eq!(regions.len(), 1, "{regions:?}");
|
||
assert_eq!(regions[0].name, "tfrags");
|
||
assert_eq!(regions[0].start, 0);
|
||
assert_eq!(regions[0].end, None);
|
||
}
|
||
|
||
#[test]
|
||
fn a_gap_does_not_close_a_region() {
|
||
// Sky starts before occlusion ends: nonsense, so neither may be closed
|
||
// against the other.
|
||
let v = index(
|
||
500,
|
||
400,
|
||
600,
|
||
&[(200, &[3000]), (400, &[4000]), (600, &[5000])],
|
||
);
|
||
let regions = layout(&v);
|
||
// occlusion at 500 cannot be closed against sky at 400, so it stays
|
||
// open rather than being clamped into a backwards range.
|
||
let occ = regions.iter().find(|r| r.name == "occlusion").unwrap();
|
||
assert_eq!(occ.start, 500);
|
||
assert_eq!(occ.end, None);
|
||
assert_eq!(occ.len(), None);
|
||
}
|
||
}
|