ReaperOfVeriod/Going-Commando
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix the record boundaries after all, and retract a claim built on the bug

Browse Source 
I reverted the record-boundary fix last commit on the grounds that "fixed-length
records plus inter-record padding" and "variable-length records" produce the same
offset table. That reasoning was wrong, and cheaply so.

stride is the FIRST offset delta. On the 92, 88, 84, 92, 92, 92, 84, 88 chunk,
if every record were stride bytes long then records 1, 2, 6 and 7 would each run
past the START of their own successor. Records are addressed by an explicit
offset table; they cannot overlap. The padding model is not merely less likely
than variable-length, it is incoherent -- I should have checked that before
reverting a correct fix on the strength of a plausible-sounding alternative.

So: resolve now measures each record against its successor again, with the
overlap argument written into the documentation so it cannot be re-litigated
without someone noticing it is arithmetic.

THE RETRACTION, which is the substantive part:

Three tests broke. Two were straightforward, but the third was not.

  * constant_positions indexed records as if they were all the same length. Now
    that they are not, it compared past the short ones. It takes the minimum
    length instead.

  * "1,509 of 1,509 chunks have uniform record length" -- which was true only
    because the parser sliced every record at one stride, so the test was
    confirming the parser's behaviour back to itself. It now asserts 257 uniform
    and 1,252 ragged, and the sum is checked against 1,509 so neither number can
    drift alone.

  * moby_class_ids_sit_in_one_slot_per_chunk_type asserted >= 20 chunk types
    carry a slot of moby class IDs at 38x-289x enrichment. It now finds 2, of
    which 1 clears 100x. RETRACTED. That test scored records that carried 4 to 8
    bytes of the NEXT record, which read as extra fields and lifted every slot's
    apparent hit rate. The general claim -- one class-ID slot per chunk type
    across the container -- does not hold. The test asserts 2, asserts 1 strong,
    and asserts the count has NOT returned to 20, so the retraction cannot quietly
    lapse back into the claim.

That is a committed finding withdrawn on the strength of a parser bug. It is also
the second time in this area that a clean number -- >= 20 types, 1509 uniform --
was really the parser describing itself.

243 tests, clippy clean.
This commit is contained in:
ReaperOfVeriod
2026-10-13 07:59:23 +02:00
parent 5f0396e3d7
commit 4fab4d0359
4 changed files with 167 additions and 101 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/gc-extract/examples/u.rs
+57
View File
@@ -0,0 +1,57 @@
use std::collections::BTreeMap;
/// For each chunk with a non-uniform delta, look at the bytes a uniform stride
/// would swallow past the next record's start. Under "padding" they should be
/// uniform or zero; under "variable-length" they are data and should not be.
fn main() {
let data =
std::path::PathBuf::from(std::env::var("HOME").unwrap()).join("Documents/gc-extracted");
let mut hist: BTreeMap<Vec<u8>, usize> = BTreeMap::new();
let mut total = 0usize;
let mut zero = 0usize;
for lv in 0..6usize {
let Ok(core) = std::fs::read(data.join(format!("level{lv}/core_data.bin"))) else {
continue;
};
for c in gc_formats::chunks::parse_all_chunks(&core).unwrap() {
let s = c.chunk.stride;
let o = &c.chunk.offsets;
for w in o.windows(2) {
let (a, b) = (w[0] as usize, w[1] as usize);
if b - a >= s {
continue;
}
let gap = s - (b - a); // bytes the uniform slice over-reads
// the disputed bytes are the last `gap` of the uniform slice
let lo = c.chunk.offset + a + (b - a);
let Some(tail) = core.get(lo..lo + gap) else {
continue;
};
let mut t = tail.to_vec();
t.sort_unstable();
if t.iter().all(|&x| x == t[0]) {
zero += 1;
}
*hist.entry(tail.to_vec()).or_default() += 1;
total += 1;
}
}
}
println!(
"{total} over-reads examined; {zero} are a single repeated byte ({:.1}%)",
100.0 * zero as f64 / total.max(1) as f64
);
println!("\nmost common disputed byte sequences:");
let mut v: Vec<_> = hist.into_iter().collect();
v.sort_by_key(|(_, n)| std::cmp::Reverse(*n));
for (b, n) in v.iter().take(15) {
println!(
" x{n:<5} {}",
b.iter()
.map(|x| format!("{x:02x}"))
.collect::<Vec<_>>()
.join(" ")
);
}
println!("\ndistinct disputed sequences: {}", v.len());
}
crates/gc-extract/tests/disc.rs
+53 -16
View File
@@ -2577,7 +2577,11 @@ fn constant_positions(records: &[&[u8]]) -> usize {
let Some(first) = records.first() else {
return 0;
};
(0..first.len())
// Records in one chunk are NOT all the same length: the offset deltas vary,
// and `resolve` measures each record against its successor. Compare only the
// span every record has, or this indexes past the short ones.
let len = records.iter().map(|r| r.len()).min().unwrap_or(0);
(0..len)
.filter(|&i| records.iter().all(|r| r[i] == first[i]))
.count()
}
@@ -3151,10 +3155,14 @@ fn chunk_records_start_16_byte_aligned() {
assert!(chunks >= 1500, "only {chunks} chunks parsed");
assert!(records >= 34000, "only {records} chunk records");
// Every chunk has a uniform record length -- length varies between chunks
// of a type, never within one -- and on most chunks the leading byte is a
// per-chunk phase counter rather than data.
// Record length varies BETWEEN chunks of a type, and -- contrary to what
// this test used to assert -- also WITHIN a chunk, on 1,252 of them. The old
// "1,509 of 1,509 uniform" was true only because `resolve` sliced every
// record at the chunk's first delta, which the test was then confirming back
// to itself. On most chunks the leading byte is a per-chunk phase counter
// rather than data.
let mut uniform = 0usize;
let mut ragged_chunks = 0usize;
let mut ruled = 0usize;
let mut indexed = 0usize;
for n in 0..27 {
@@ -3162,6 +3170,8 @@ fn chunk_records_start_16_byte_aligned() {
for c in parse_all_chunks(&core).expect("chunks") {
if c.has_uniform_record_length() {
uniform += 1;
} else {
ragged_chunks += 1;
}
if c.bucket_rule().is_some() {
ruled += 1;
@@ -3171,7 +3181,12 @@ fn chunk_records_start_16_byte_aligned() {
}
}
}
assert_eq!(uniform, 1509, "uniform-record-length count moved off 1509");
assert_eq!(uniform, 257, "uniform-record-length count moved off 257");
assert_eq!(
uniform + ragged_chunks,
1509,
"chunk total moved; add the ragged counter"
);
assert_eq!(ruled, 1262, "bucket-rule count moved off 1262");
assert_eq!(indexed, 1235, "index-rule count moved off 1235");
@@ -3207,9 +3222,20 @@ fn chunk_records_start_16_byte_aligned() {
/// Moby class IDs live in one body slot per chunk type, and the slot varies.
///
/// Measured over all 27 levels: the best slot per type carries 5 to 42 distinct
/// valid moby class IDs at 153x to 289x enrichment over the chance rate. The
/// `distinct >= 4` filter is load-bearing -- without it the top slots are
/// **Largely retracted.** This used to assert that >= 20 chunk types carry a
/// slot of moby class IDs, at 153x to 289x enrichment over chance.
///
/// It was measuring over-read records. `resolve` sliced every record at the
/// chunk's first offset delta, which on 1,252 of 1,509 chunks ran 4 to 8 bytes
/// past a record's real end -- four bytes of the *next* record, glued on. That
/// reads as extra fields and lifts a slot's apparent hit rate. With record
/// boundaries measured against each record's successor, and ragged chunks
/// skipped, only 2 types show a varying class-ID slot.
///
/// What survives is narrow and is asserted as such: 2 chunk types show a slot
/// that varies in the manner of a class ID, and exactly 1 of those still clears
/// 100x enrichment over chance. The
/// `distinct >= 4` filter is still load-bearing -- without it the top slots are
/// constants that read as 305x because any value below the moby count counts as
/// a hit. See `docs/formats/core-data-regions.md`.
#[test]
@@ -3295,16 +3321,27 @@ fn moby_class_ids_sit_in_one_slot_per_chunk_type() {
}
}
assert!(
best.len() >= 20,
"only {} chunk types carry a varying class-ID slot",
best.len()
);
// RETRACTED. This asserted >= 20 types while every record was being sliced
// at the chunk's first delta, which over-read 4 to 8 bytes past the true end
// of a record on 1,252 of 1,509 chunks. Records with four bytes of the next
// one glued to the end score as though they carried extra fields. With the
// boundaries fixed, and this test skipping ragged chunks, only 2 types show
// a varying class-ID slot -- so "one class-ID slot per chunk type" was a
// measurement of the parser's over-read, not of the disc.
let strong: Vec<_> = best.values().filter(|(_, r, _)| *r >= 100.0).collect();
assert!(
strong.len() >= 20,
"only {} types reach 100x enrichment",
strong.len()
!strong.is_empty(),
"no type reaches 100x enrichment -- was 20 before the record-boundary fix"
);
assert_eq!(strong.len(), 1, "strong-enrichment type count moved");
assert!(
best.len() >= 2,
"only {} chunk types carry a varying class-ID slot (was >= 20)",
best.len()
);
assert!(
best.len() < 20,
"class-ID slots are back at >= 20 types; revisit the retraction"
);
let mut all: Vec<_> = best
crates/gc-formats/src/chunks.rs
+32 -20
View File
@@ -103,12 +103,12 @@ pub const CHUNK_REGION_MARKER: [u8; 2] = [0xf6, 0x7c];
/// A chunk header plus the records its offsets address.
///
/// `stride` is the distance between the first two resolved offsets, and nothing
/// more. The parser takes each offset literally rather than assuming a uniform
/// stride, because the deltas are not constant. On level 0 some chunks step
/// 340, 340, 336, 340; on all 27 levels signature `[75 42 ef f2 71 b5]` steps
/// 92, 88, 84, 92, 92, 92, 84, 88, which is a real variation in record length
/// and not a rounding artifact. Record *lengths* come from the deltas, in
/// [`resolve`].
/// more. The deltas within a chunk are **not** constant, and the variation is
/// real: on level 0 some chunks step 340, 340, 336, 340, and on all 27 levels
/// signature `[75 42 ef f2 71 b5]` steps 92, 88, 84, 92, 92, 92, 84, 88. Taking
/// every record at `stride` would overlap four of those eight with their own
/// successors, which an explicit offset table cannot express. Record *lengths*
/// come from the deltas; see [`resolve`].
pub struct Chunk {
/// The 16 opaque header bytes.
pub header: [u8; CHUNK_HEADER_BYTES],
@@ -391,18 +391,26 @@ pub fn parse_chunk(region: &[u8], offset: usize) -> Result<Option<Chunk>> {
/// Resolve every record a chunk addresses, bounds-checked against `region`.
///
/// Each record is `region[chunk.offset + low .. + stride]`, and `stride` is the
/// chunk's first offset delta.
/// A record runs from its own offset to the **next** record's offset, not to
/// `chunk.offset + stride`.
///
/// **This may slice past a record's real end.** 1,252 of the 1,509 chunks have
/// offsets that are not uniformly spaced -- the deviations are word-aligned
/// multiples of 4, mostly +/-4 -- so for those the last few bytes of a slice can
/// be the head of the next record. Whether that means the records are genuinely
/// variable-length, or that the offset table has inter-record padding, **cannot
/// be told from the offsets alone**: both models produce exactly this table.
/// Deciding between them needs a field whose end is independently observable, so
/// until that exists this function takes the uniform reading and the ambiguity
/// is recorded rather than guessed at. See `docs/formats/core-data-regions.md`.
/// This is not a stylistic choice. `stride` is the *first* offset delta, and on
/// 1,252 of the 1,509 measured chunks later deltas are smaller -- the deviations
/// are word-aligned multiples of 4, most exactly +/-4. Slicing every record at
/// the first delta would therefore make some records run past the start of their
/// own successor. On the measured `92, 88, 84, 92, 92, 92, 84, 88` chunk, four
/// of the eight records overlap the next one that way, and records are addressed
/// by an explicit offset table, so they cannot overlap. The only coherent
/// reading is that a record ends where the next begins.
///
/// Slicing at `stride` was not wrong in a way that would error; it handed four
/// records 4 to 8 bytes of the following record, which is the kind of defect
/// that surfaces much later as fields that look meaningful and are quietly off
/// by four bytes.
///
/// The last record has no successor to measure against and falls back to
/// `stride`; on the measured levels the final delta is itself uniform, so this
/// is a fallback rather than a silent guess.
///
/// A record that would run past the end of the region is dropped rather than
/// clamped, so the returned length may be shorter than `chunk.offsets` -- on the
@@ -410,7 +418,7 @@ pub fn parse_chunk(region: &[u8], offset: usize) -> Result<Option<Chunk>> {
/// malformed chunk rather than report it.
pub fn resolve<'r>(chunk: &Chunk, region: &'r [u8]) -> Result<ResolvedChunk<'r>> {
let mut records = Vec::with_capacity(chunk.offsets.len());
for &low in &chunk.offsets {
for (i, &low) in chunk.offsets.iter().enumerate() {
let start = chunk
.offset
.checked_add(low as usize)
@@ -419,10 +427,14 @@ pub fn resolve<'r>(chunk: &Chunk, region: &'r [u8]) -> Result<ResolvedChunk<'r>>
context: "chunk record",
kind: ErrorKind::Invalid("chunk offset overflows usize"),
})?;
if start + chunk.stride > region.len() {
let len = match chunk.offsets.get(i + 1) {
Some(&next) => (next - low) as usize,
None => chunk.stride,
};
if start + len > region.len() {
continue;
}
records.push(&region[start..start + chunk.stride]);
records.push(&region[start..start + len]);
}
Ok(ResolvedChunk {
chunk: Chunk {