ReaperOfVeriod/Going-Commando
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Resolve the shrub region bound: the chunk container closes it

Browse Source 
The shrub region's end was the last unlocated bound in core_data. It was not
in the core index, and unlike moby->tie it had no successor boundary to chain
against, so the last shrub class in every level ran out to the lump's end and
the aggregates over it (the "3.5M records / 28MB of shrub body") were
artifacts of that oversized span.

Range measured: for each level, from the last shrub class's geometry offset to
the constant trailer. Nothing outside that range is read.

Method: scan forward from the last shrub class for the pair (a) the constant
two bytes f6 7c sitting immediately in front of (b) a chunk signature -- a
16-byte header ending 0xff, ten zero bytes, then four or more 0xf0-tagged
offsets. The two halves are each weak alone, and the halves are measured
separately: the marker pair occurs 25 to 103 times in a 5 to 17 MB lump, so
matching it alone picks dozens of wrong offsets; the chunk shape matches
random data often enough to need four tagged offsets before it is believed.

Result: on all 27 levels the marker sits exactly two bytes before the first
chunk header, and the two parts of the signature are never found apart. No
chunk signature occurs inside any last shrub body, on any level. The three
levels without shrub (10, 25, 26) carry the same marker before the container
that closes tie instead, which is what makes the locator a property of the
region rather than a shrub rule.

Consequences:

- All 358 shrub class bodies are bounded and are whole 8-byte records. That
  was not assertable while the last class in each level ran to the lump's end,
  and it is the check the boundary earns its keep for.
- 9,600 records in total, against the 8,938 previously measurable in
  successor-bounded spans; 662 of them sit in the 24 last-class bodies. The
  shrub region totals 2,033,776 bytes across the 24 levels that have one.
- The chunk container is 25,775,850 bytes across 27 levels (1,236 chunks,
  22,562 records). The container measurements were previously taken over "the
  900,000 bytes before the trailer", a window that did not align to its real
  extent: on level 0 it is 690,302 bytes, and 27 of 1,123 chunks were
  truncated at that invented edge. chunk_containers_resolve_on_every_level now
  reads the region from core_layout instead of measuring backwards from the
  end of the lump.

core_layout takes core_data now: the container's start is the one bound that
cannot be derived from the index. It gains the chunks region, so every byte of
level-specific core_data belongs to a named region. Where the signature is not
found the end stays None -- the region is left visible rather than clamped to
the tail.

Still open: the marker's meaning (constant across 27 levels, at exactly this
position, unnamed) and the shrub body payload's encoding.
This commit is contained in:
ReaperOfVeriod
2026-10-11 06:20:48 +02:00
parent 4d1e277a15
commit ccea53807a
8 changed files with 681 additions and 117 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/gc-extract/tests/disc.rs
+290 -74
View File
@@ -1648,7 +1648,6 @@ fn shrub_headers_name_their_own_class_on_every_level() {
let mut denormals = 0usize;
let mut bounded_records = 0usize;
let mut bounded_bodies = 0usize;
let mut last_bodies = 0usize;
for n in 0..27 {
let dir = data.join(format!("level{n}"));
@@ -1661,10 +1660,10 @@ fn shrub_headers_name_their_own_class_on_every_level() {
let h = gc_formats::level_core::LevelCoreHeader::parse(&index).expect("core header");
let range = h.shrub_classes;
// Class spans tile between consecutive geometry offsets, with the last
// one bounded by the start of core_data's constant trailer. Without
// this the last class absorbs the trailer and the body check measures
// its neighbours.
let trailer = gc_formats::regions::CORE_DATA_TRAILER_BYTES;
// one bounded by the chunk container. It is found by signature, since
// the core index carries no successor offset for shrub; without it the
// last class absorbs the container and the body check measures its
// neighbours.
let mut offsets: Vec<usize> = Vec::new();
for i in 0..range.count.max(0) as usize {
let at = range.offset.max(0) as usize + i * SHRUB_CLASS_ENTRY_BYTES;
@@ -1675,48 +1674,45 @@ fn shrub_headers_name_their_own_class_on_every_level() {
}
}
offsets.sort_unstable();
// Levels without shrub geometry have no span to bound. The container
// that follows tie on those levels is checked by
// `chunk_container_closes_the_last_class_region`.
let container = match offsets.last() {
Some(&last) if !offsets.is_empty() => {
gc_formats::chunks::chunk_region_start(&core, last).unwrap_or_else(|| {
panic!("level{n}: no chunk container after the shrub classes")
})
}
_ => core.len(),
};
let mut span_end_of: std::collections::HashMap<usize, usize> =
std::collections::HashMap::new();
for (i, &o) in offsets.iter().enumerate() {
let end = offsets.get(i + 1).copied().unwrap_or(core.len() - trailer);
let end = offsets.get(i + 1).copied().unwrap_or(container);
span_end_of.insert(o, end);
}
for i in 0..range.count.max(0) as usize {
let at = range.offset.max(0) as usize + i * SHRUB_CLASS_ENTRY_BYTES;
let e = ShrubClassEntry::parse(&index[at..at + SHRUB_CLASS_ENTRY_BYTES])
.unwrap_or_else(|e| panic!("level{n}: shrub class {i}: {e}"));
let (span_end, has_successor) = match span_end_of
.get(&(e.base.offset_in_asset_wad.max(0) as usize))
{
Some(&e2)
if offsets.last() != Some(&(e.base.offset_in_asset_wad.max(0) as usize)) =>
{
(e2, true)
}
_ => (core.len(), false),
};
let (span_end, has_successor) =
match span_end_of.get(&(e.base.offset_in_asset_wad.max(0) as usize)) {
Some(&e2) => (e2, true),
None => (core.len(), false),
};
match ShrubHeader::region(&core, e.base.offset_in_asset_wad, e.base.o_class, span_end) {
Ok(Some(mut s)) => {
s.span_end_is_a_successor = has_successor;
// Only a class whose span ends at the *next* class has a
// bounded body. The last class's span runs to the trailer
// and contains the 690 KB region that follows shrub, so its
// body is not wholly records -- that is the open question,
// not a defect, and asserting either way would be wrong.
if s.span_end_is_a_successor {
assert!(
s.body_is_well_formed(),
"level{n} class {i}: bounded shrub body is not whole 8-byte records"
);
bounded_bodies += 1;
}
last_bodies += 1;
// Count records only where the span is actually bounded.
// Summing the unbounded last class too reproduces the
// retracted "3.5 million records" figure.
if s.span_end_is_a_successor {
bounded_records += s.body_record_count();
}
// Every class is bounded now, so every body must be whole
// 8-byte records. That was not assertable while the last
// class ran to the lump's end, and it is the check the
// boundary earns its keep for.
assert!(
s.body_is_well_formed(),
"level{n} class {i}: shrub body is not whole 8-byte records"
);
bounded_bodies += 1;
bounded_records += s.body_record_count();
assert_eq!(
s.header.o_class,
e.base.o_class.max(0) as u32,
@@ -1761,27 +1757,29 @@ fn shrub_headers_name_their_own_class_on_every_level() {
spans + absent
);
assert!(spans > 340, "only {spans} shrub spans with geometry");
// The container bounds every class now, so every body is checkable.
assert_eq!(bounded_bodies, spans, "every shrub class should be bounded");
eprintln!(
"shrub: {spans} headers naming their own class, {absent} without geometry; \
{bounded_bodies}/{last_bodies} class bodies bounded by a successor, \
{bounded_records} 8-byte records in bounded bodies, \
all {bounded_bodies} class bodies bounded by the chunk container, \
{bounded_records} 8-byte records in total, \
{denormals} denormal reserved words"
);
}
/// Region bounds derived from successor boundaries, on every level.
///
/// The point is not that these are correct -- shrub's end is still unknown --
/// but that the unknown is visible. Every region here is closed by the next
/// region's start rather than by `core_data.len()`, so no span silently
/// absorbs the tail of the lump the way the last shrub class was doing.
/// Every region is closed by the next region's start, or -- for the last class
/// region -- by the located chunk container, never by `core_data.len()`, so no
/// span silently absorbs the tail of the lump the way the last shrub class was
/// doing.
#[test]
fn core_regions_chain_on_every_level() {
require_disc!();
let data = std::env::var_os("GC_EXTRACTED")
.map(PathBuf::from)
.unwrap_or_else(|| {
let home = std::env::var("HOME").unwrap_or_else(|_| ".".to_string());
let home = std::env::var("HOME").unwrap_or_else(|_| ".".into());
PathBuf::from(home).join("Documents/gc-extracted")
});
if !data.exists() {
@@ -1790,23 +1788,26 @@ fn core_regions_chain_on_every_level() {
}
use gc_formats::level_core::LevelCoreHeader;
use gc_formats::regions::core_layout;
use gc_formats::regions::{core_layout, CORE_DATA_TRAILER_BYTES};
let mut levels = 0usize;
let mut unresolved_shrub = 0usize;
let mut unresolved_chunks = 0usize;
let mut bounded = 0usize;
let mut shrub_bytes = 0usize;
let mut chunk_bytes = 0usize;
let (mut first, mut last) = (usize::MAX, 0usize);
for n in 0..27 {
let dir = data.join(format!("level{n}"));
let Ok(index) = std::fs::read(dir.join("core_index.bin")) else {
let (Ok(core), Ok(index)) = (
std::fs::read(dir.join("core_data.bin")),
std::fs::read(dir.join("core_index.bin")),
) else {
continue;
};
let core_len = std::fs::metadata(dir.join("core_data.bin"))
.map(|m| m.len() as usize)
.unwrap_or(0);
let core_len = core.len();
let h = LevelCoreHeader::parse(&index).expect("core header");
let regions = core_layout(&index, &h, core_len);
let regions = core_layout(&index, &core, &h);
// Regions appear in order, never overlap, and never run past the lump.
for pair in regions.windows(2) {
@@ -1838,7 +1839,7 @@ fn core_regions_chain_on_every_level() {
// Regions appear in the canonical order, but not every level has every
// region -- level 10 has no occlusion and no shrub at all.
const ORDER: [&str; 7] = [
const ORDER: [&str; 8] = [
"tfrags",
"occlusion",
"sky",
@@ -1846,6 +1847,7 @@ fn core_regions_chain_on_every_level() {
"moby",
"tie",
"shrub",
"chunks",
];
let names: Vec<_> = regions.iter().map(|r| r.name).collect();
let ranks: Vec<usize> = names
@@ -1862,11 +1864,11 @@ fn core_regions_chain_on_every_level() {
"level{n}: regions out of order: {names:?}"
);
if let Some(s) = regions.iter().find(|r| r.name == "shrub") {
// The constant trailer bounds it; assert it is actually inside
// the lump rather than at its end.
match s.end {
Some(e) => {
assert!(e < core_len, "level{n}: shrub ends at the lump's end");
// The container's start, not the trailer and not the lump's
// end: the trailer is the chunk region's business.
assert!(e < core_len - CORE_DATA_TRAILER_BYTES);
shrub_bytes += s.len().unwrap_or(0);
bounded += 1;
}
@@ -1875,20 +1877,33 @@ fn core_regions_chain_on_every_level() {
first = first.min(s.start);
last = last.max(s.start);
}
if let Some(c) = regions.iter().find(|r| r.name == "chunks") {
match c.end {
Some(e) => {
// The trailer is the only thing past the chunk container.
assert_eq!(e, core_len - CORE_DATA_TRAILER_BYTES);
chunk_bytes += c.len().unwrap_or(0);
}
None => unresolved_chunks += 1,
}
}
levels += 1;
}
assert_eq!(levels, 27, "expected all 27 levels, got {levels}");
// Known gap, not a bug: the offset ending the shrub region is not in the
// core index and textures sit before moby, so they cannot bound it.
assert_eq!(
unresolved_shrub, 0,
"every level with shrub should bound it"
"every level with shrub should bound it at the chunk container"
);
assert_eq!(bounded, 24, "expected 24 levels with a shrub region");
assert_eq!(
unresolved_chunks, 0,
"every chunk region should end at the trailer"
);
eprintln!(
"regions: shrub starts {first}..{last}; {shrub_bytes} bytes total, \
bounded by the constant trailer on all {bounded} levels"
"regions: shrub starts {first}..{last}; {shrub_bytes} bytes of shrub and \
{chunk_bytes} bytes of chunk data, shrub closed by the located container on \
all {bounded} levels"
);
}
@@ -1896,6 +1911,15 @@ fn core_regions_chain_on_every_level() {
/// signature on every level, and their offsets resolve to records with a
/// shared field layout.
///
/// The region is bounded properly rather than by the fixed window this test
/// first used: it runs from the marker-paired chunk signature that closes the
/// last class region to the constant trailer. The window approach worked and
/// then over-worked -- a fixed 900,000 bytes from the end reaches past the
/// container on some levels and stops inside it on others, which is how 27
/// chunks came to be truncated at a "window edge" that is not a format
/// boundary. The locator is asserted by
/// `chunk_container_closes_the_last_class_region`.
///
/// The layout facts asserted here are measured, not assumed:
///
/// - every tagged offset has top byte `0xf0`, on all 27 levels;
@@ -1909,7 +1933,7 @@ fn chunk_containers_resolve_on_every_level() {
let data = std::env::var_os("GC_EXTRACTED")
.map(PathBuf::from)
.unwrap_or_else(|| {
let home = std::env::var("HOME").unwrap_or_else(|_| ".".to_string());
let home = std::env::var("HOME").unwrap_or_else(|_| ".".into());
PathBuf::from(home).join("Documents/gc-extracted")
});
if !data.exists() {
@@ -1917,6 +1941,9 @@ fn chunk_containers_resolve_on_every_level() {
return;
}
use gc_formats::level_core::LevelCoreHeader;
use gc_formats::regions::{core_layout, CORE_DATA_TRAILER_BYTES};
let mut levels = 0usize;
let mut chunks = 0usize;
let mut records = 0usize;
@@ -1928,20 +1955,30 @@ fn chunk_containers_resolve_on_every_level() {
for n in 0..27 {
let dir = data.join(format!("level{n}"));
let core = match std::fs::read(dir.join("core_data.bin")) {
Ok(c) => c,
Err(_) => continue,
let (Ok(core), Ok(index)) = (
std::fs::read(dir.join("core_data.bin")),
std::fs::read(dir.join("core_index.bin")),
) else {
continue;
};
// The region runs from after the last shrub geometry to the constant
// trailer, whose size is measured and identical on all 27 levels.
const TRAILER: usize = 0x38a10;
let region_start = core.len().saturating_sub(TRAILER + 900_000);
let region_end = core.len() - TRAILER;
if region_end <= region_start {
// The region runs from the chunk signature that closes the last class
// region to the constant trailer, both measured and identical in size
// on all 27 levels. core_layout locates the start through the same
// signature the chunk parser uses.
let h = LevelCoreHeader::parse(&index).expect("core header");
let Some(region) = core_layout(&index, &core, &h)
.into_iter()
.find(|r| r.name == "chunks")
else {
continue;
}
let region = &core[region_start..region_end];
};
let Some(end) = region.end else {
panic!("level{n}: chunk region is unbounded");
};
assert_eq!(end, core.len() - CORE_DATA_TRAILER_BYTES);
assert!(!region.is_empty(), "level{n}: empty chunk region");
let region = &core[region.start..end];
let found = gc_formats::chunks::parse_all_chunks(region).expect("chunks should parse");
if found.is_empty() {
@@ -1970,9 +2007,9 @@ fn chunk_containers_resolve_on_every_level() {
strides.insert(chunk.chunk.stride);
// Records resolve unless the last one would run past the region
// end -- which happens for chunks sitting at the window edge, since
// the window is a fixed 900000 bytes and does not align to chunk
// boundaries. Truncation is counted rather than asserted away.
// end, which happens for the chunk sitting at the trailer end:
// the container's records stop there, so its final record or two
// fall outside. Truncation is counted rather than asserted away.
assert!(
chunk.records.len() <= chunk.chunk.offsets.len(),
"level{n}: chunk at {} resolved MORE records than offsets",
@@ -1993,7 +2030,7 @@ fn chunk_containers_resolve_on_every_level() {
}
}
assert!(levels >= 20, "only {levels} levels had chunks");
assert_eq!(levels, 27, "expected all 27 levels, got {levels}");
assert!(chunks > 500, "only {chunks} chunks resolved");
assert!(records > 5_000, "only {records} records resolved");
assert!(
@@ -2011,7 +2048,7 @@ fn chunk_containers_resolve_on_every_level() {
eprintln!(
"chunks: {chunks} over {levels} levels, {records} records, {} distinct strides, \
constant {constant_total}/{stride_total} ({:.1}%), per-chunk min {:.1}% max {:.1}%, \
{truncated} chunks truncated at the window edge",
{truncated} chunks truncated at the trailer end",
strides.len(),
mean_frac * 100.0,
frac.iter().cloned().fold(f64::INFINITY, f64::min) * 100.0,
@@ -2019,6 +2056,185 @@ fn chunk_containers_resolve_on_every_level() {
);
}
/// The chunk container closes the last class region, on every level.
///
/// The core index carries no successor offset for shrub -- unlike moby and
/// tie, whose boundaries fall out of the class tables -- so the boundary is
/// located in `core_data`: the constant marker pair sitting immediately in
/// front of a chunk signature. Both halves of that signature are weak alone,
/// which is why the test measures them separately:
///
/// - the marker pair `f6 7c` occurs 25 to 103 times in a 5 to 17 MB lump, so
/// on its own it means nothing;
/// - the chunk shape (16-byte header ending `0xff`, 10 zeros, tagged offsets)
/// matches random data often enough to need four tagged offsets before it
/// means anything.
///
/// Together they are exact: on all 27 levels the marker sits exactly two bytes
/// before the first chunk header. Three checks pin that down per level:
///
/// 1. the located boundary carries the marker;
/// 2. the last shrub body walks to 8-byte records that end exactly where the
/// signature begins, with no chunk signature inside the body itself;
/// 3. the container's records fill the region up to the constant trailer.
///
/// Levels without shrub are the control for the locator: there the container
/// closes tie instead, and the same marker precedes it.
#[test]
fn chunk_container_closes_the_last_class_region() {
require_disc!();
let data = std::env::var_os("GC_EXTRACTED")
.map(PathBuf::from)
.unwrap_or_else(|| {
let home = std::env::var("HOME").unwrap_or_else(|_| ".".into());
PathBuf::from(home).join("Documents/gc-extracted")
});
if !data.exists() {
eprintln!("skipped: unpack the disc first");
return;
}
use gc_formats::chunks::{self, CHUNK_REGION_MARKER};
use gc_formats::level_core::{LevelCoreHeader, ShrubClassEntry};
use gc_formats::regions::{core_layout, last_geometry, CORE_DATA_TRAILER_BYTES};
use gc_formats::shrub::{ShrubHeader, SHRUB_CLASS_ENTRY_BYTES};
let mut levels = 0usize;
let mut with_shrub = 0usize;
let mut marker_wide_max = 0usize;
let mut body_records = 0usize;
let mut container_bytes = 0usize;
let mut fill_gap_max = 0usize;
for n in 0..27 {
let dir = data.join(format!("level{n}"));
let (Ok(core), Ok(index)) = (
std::fs::read(dir.join("core_data.bin")),
std::fs::read(dir.join("core_index.bin")),
) else {
panic!("level{n}: missing core_data or core_index");
};
let trailer = core.len() - CORE_DATA_TRAILER_BYTES;
let h = LevelCoreHeader::parse(&index).expect("core header");
let last_shrub = last_geometry(&index, h.shrub_classes, SHRUB_CLASS_ENTRY_BYTES);
let last_tie = last_geometry(&index, h.tie_classes, 0x20);
// The container follows the last of the shrub and tie classes, so the
// signature search starts after the class bodies rather than inside
// them.
let scan_from = last_shrub
.or(last_tie)
.unwrap_or_else(|| panic!("level{n}: no class geometry to follow"));
let container = chunks::chunk_region_start(&core, scan_from)
.unwrap_or_else(|| panic!("level{n}: no chunk container after {scan_from}"));
// 1. The marker, and how common it is without the signature behind it.
assert_eq!(
&core[container - 2..container],
&CHUNK_REGION_MARKER,
"level{n}: container at {container} is not preceded by the marker"
);
marker_wide_max = marker_wide_max.max(
core.windows(2)
.filter(|w| *w == CHUNK_REGION_MARKER)
.count(),
);
// 2. The last shrub body ends exactly at the marker. This is the check
// that the boundary is the successor of shrub rather than a coincident
// pair inside it.
if let Some(last) = last_shrub {
let o_class = (0..h.shrub_classes.count.max(0) as usize)
.filter_map(|i| {
let at = h.shrub_classes.offset.max(0) as usize + i * SHRUB_CLASS_ENTRY_BYTES;
let e = ShrubClassEntry::parse(
index
.get(at..at + SHRUB_CLASS_ENTRY_BYTES)
.unwrap_or_default(),
)
.ok()?;
(e.base.offset_in_asset_wad == last as i32).then_some(e.base.o_class)
})
.next()
.unwrap_or_else(|| panic!("level{n}: no shrub class at {last}"));
let span = ShrubHeader::region(&core, last as i32, o_class, trailer)
.expect("last shrub span parses")
.expect("last shrub span present");
let data_start = last + span.header.data_offset as usize;
// Walk 8-byte records, bounded by the container so the walk can
// never continue into the chunk region itself.
let mut at = data_start;
while at + 8 <= container && core[at + 6] == 0 && core[at + 7] == 0 {
at += 8;
}
assert_eq!(
at + 2,
container,
"level{n}: shrub body at {data_start} does not end at the marker"
);
assert!(at > data_start, "level{n}: last shrub body is empty");
body_records += (at - data_start) / 8;
// No chunk signature inside the body: the search for the boundary
// must not match one it should have skipped.
let sig_inside = (last..container - 2).any(|off| chunks::is_chunk_start(&core, off, 4));
assert!(
!sig_inside,
"level{n}: chunk signature inside the last shrub body"
);
with_shrub += 1;
} else {
// Without shrub the container closes tie, and the marker precedes
// it the same way -- the locator is not shrub-specific.
let tie = core_layout(&index, &core, &h)
.into_iter()
.find(|r| r.name == "tie")
.unwrap_or_else(|| panic!("level{n}: no tie region to close"));
assert_eq!(
tie.end,
Some(container),
"level{n}: tie must end at the container"
);
}
// 3. The container fills its region up to the constant trailer: the
// furthest record should reach within a few KB of it, never inside it.
let region = &core[container..trailer];
let found = chunks::parse_all_chunks(region).expect("chunks should parse");
let region_ptr = region.as_ptr() as usize;
let furthest = found
.iter()
.flat_map(|c| {
c.records
.iter()
.map(move |r| r.as_ptr() as usize + r.len() - region_ptr)
})
.max()
.unwrap_or(0);
assert!(
region.len() - furthest < 4096,
"level{n}: chunk records stop {} bytes short of the trailer",
region.len() - furthest
);
container_bytes += region.len();
fill_gap_max = fill_gap_max.max(region.len() - furthest);
levels += 1;
}
assert_eq!(levels, 27, "expected all 27 levels, got {levels}");
assert_eq!(
with_shrub, 24,
"expected 24 levels with shrub, got {with_shrub}"
);
eprintln!(
"container: {levels} levels ({with_shrub} closing shrub), \
{body_records} 8-byte records in last-shrub bodies, \
{container_bytes} bytes of chunk data, largest trailer gap {fill_gap_max}, \
marker pair occurs at most {marker_wide_max} times per lump"
);
}
/// Byte positions that hold the same value in every supplied record.
fn constant_positions(records: &[&[u8]]) -> usize {
let Some(first) = records.first() else {