A parser defect: one chunk type is not fixed-stride, and over-reads
[75 42 ef f2 71 b5] -- the signature that is both level-specific and
variable-length -- makes three chunks per level, 81 total:
stride 76 deltas [76, 76, 76, 76, 76, 76, 76, 76] uniform
stride 92 deltas [92, 88, 84, 92, 92, 92, 84, 88] NOT uniform
stride 104 deltas [104, 104, 104, ...] uniform
Identical on all 27 levels: 27 chunks of each shape, three shapes, no others.
So "this signature has strides {76, 92, 104}" is a misdescription. It has three
chunk shapes, one of which is not fixed-stride at all. The parser derives one
stride per chunk and resolve slices every record at that length.
The cost, precisely: records are read at the offset table's positions, so every
record STARTS correctly -- no misalignment. But each is sliced at 92 bytes, while
two of the eight are 88 and two are 84, so four records over-read by 4 and 8
bytes and carry the head of the next record inside them. Bounds checking holds;
this is a data-correctness bug, not a safety one. It is precisely the kind that
produces plausible-looking fields offset by four bytes, which is how it would
have gone unnoticed had the stride table not been collected per signature rather
than inferred per region.
It also explains why this signature was previously flagged as breaking the 8-byte
alignment rule: its records genuinely are not a uniform multiple.
The right fix is for resolve to use offset deltas for a chunk whose deltas are
not uniform. Not taken here -- the delta sequences are asserted first, so if the
data ever becomes uniform the test fails and asks for the model to be revisited,
rather than the model quietly absorbing the exception.
Asserted in chunk_type_75_ef_has_one_variable_length_chunk_per_level, with the
exact sequences [76x8], [92,88,84,92,92,92,84,88], [104x8] and 27 of each.
A near-miss worth recording, because it would have made a tidy false claim: the
leading u16 of these records appeared to step by exactly 0x50 (80) across all nine
of the 76-byte chunk. Reading at the real offsets gives b1a5, a529, 047e, a529,
04f0, a529, 0562, a529, 05d4 -- no progression at all. The apparent 80-byte step
came from comparing the wrong bytes, and "records are indexed by 80" is exactly
the kind of finding that gets reported on sight.
243 tests, clippy clean.
This commit is contained in:
@@ -4414,3 +4414,98 @@ fn most_chunk_signatures_are_level_independent() {
|
||||
assert_eq!(shared, 65, "the shared/level-specific split moved");
|
||||
assert_eq!(specific.len(), 9, "level-specific signature count moved");
|
||||
}
|
||||
|
||||
/// The signature `[75 42 ef f2 71 b5]` produces three chunks per level, and one
|
||||
/// of the three has genuinely variable-length records.
|
||||
///
|
||||
/// The parser takes a single stride per chunk. For the 92-byte chunk that is
|
||||
/// wrong: its record offsets step by `92, 88, 84, 92, 92, 92, 84, 88`. Every
|
||||
/// record *starts* correctly, because `resolve` uses the offset table, but each
|
||||
/// is sliced at 92 bytes -- so the two 88-byte and three 84-byte records
|
||||
/// over-read by 4 and 8 bytes and pick up the head of the next record.
|
||||
///
|
||||
/// So "this signature has strides {76, 92, 104}" is a misdescription. It has
|
||||
/// three chunk shapes, one of which is not fixed-stride at all. Asserted with
|
||||
/// the exact delta sequences so a future single-stride assumption cannot survive
|
||||
/// quietly.
|
||||
#[test]
|
||||
fn chunk_type_75_42_ef_has_one_variable_length_chunk_per_level() {
|
||||
require_disc!();
|
||||
|
||||
let data = std::env::var_os("GC_EXTRACTED")
|
||||
.map(PathBuf::from)
|
||||
.unwrap_or_else(|| {
|
||||
let home = std::env::var("HOME").unwrap_or_else(|_| ".".to_string());
|
||||
PathBuf::from(home).join("Documents/gc-extracted")
|
||||
});
|
||||
if !data.exists() {
|
||||
eprintln!("skipped: unpack the disc first");
|
||||
return;
|
||||
}
|
||||
|
||||
use gc_formats::chunks::{chunk_region_start, parse_all_chunks};
|
||||
use gc_formats::level_core::LevelCoreHeader;
|
||||
use std::collections::BTreeMap;
|
||||
|
||||
let target: [u8; 6] = [0x75, 0x42, 0xef, 0xf2, 0x71, 0xb5];
|
||||
let mut shapes: BTreeMap<Vec<u32>, usize> = BTreeMap::new();
|
||||
let mut mixed = 0usize;
|
||||
let mut chunks = 0usize;
|
||||
for lv in 0..27usize {
|
||||
let (Ok(idx), Ok(core)) = (
|
||||
std::fs::read(data.join(format!("level{lv}/core_index.bin"))),
|
||||
std::fs::read(data.join(format!("level{lv}/core_data.bin"))),
|
||||
) else {
|
||||
continue;
|
||||
};
|
||||
let _ = LevelCoreHeader::parse(&idx).expect("core header");
|
||||
let scan = chunk_region_start(&core, 0)
|
||||
.or_else(|| chunk_region_start(&core, 0x1000))
|
||||
.unwrap_or(0);
|
||||
let Ok(found) = parse_all_chunks(&core[scan..]) else {
|
||||
continue;
|
||||
};
|
||||
for rc in found {
|
||||
if rc.chunk.header[..6] != target[..] {
|
||||
continue;
|
||||
}
|
||||
chunks += 1;
|
||||
let deltas: Vec<u32> = rc.chunk.offsets.windows(2).map(|w| w[1] - w[0]).collect();
|
||||
if !deltas.windows(2).all(|w| w[0] == w[1]) {
|
||||
mixed += 1;
|
||||
}
|
||||
*shapes.entry(deltas).or_default() += 1;
|
||||
}
|
||||
}
|
||||
|
||||
assert_eq!(chunks, 81, "chunk count for this signature moved");
|
||||
assert_eq!(mixed, 27, "the number of variable-length chunks moved");
|
||||
|
||||
let uniform76: Vec<u32> = vec![76; 8];
|
||||
let variable: Vec<u32> = vec![92, 88, 84, 92, 92, 92, 84, 88];
|
||||
let uniform104: Vec<u32> = vec![104; 8];
|
||||
assert_eq!(
|
||||
shapes.get(&uniform76).copied(),
|
||||
Some(27),
|
||||
"the 76-byte chunk shape changed"
|
||||
);
|
||||
assert_eq!(
|
||||
shapes.get(&variable).copied(),
|
||||
Some(27),
|
||||
"the variable-length chunk's delta sequence changed"
|
||||
);
|
||||
assert_eq!(
|
||||
shapes.get(&uniform104).copied(),
|
||||
Some(27),
|
||||
"the 104-byte chunk shape changed"
|
||||
);
|
||||
assert_eq!(shapes.len(), 3, "a fourth chunk shape appeared");
|
||||
|
||||
// the defect, stated as a measurement rather than a comment
|
||||
let over = variable.iter().filter(|&&d| d < 92).count();
|
||||
eprintln!(
|
||||
"chunk [75 42 ef]: 81 chunks; 27 variable-length, deltas {variable:?}; \
|
||||
{over} of 8 records shorter than the 92 the parser slices"
|
||||
);
|
||||
assert_eq!(over, 4, "the count of short records changed");
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user