ReaperOfVeriod/Going-Commando
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Audit what the record-boundary fix contaminated: one retraction, one survivor

Browse Source 
The fix changed record lengths in 1,252 of 1,509 chunks, so every committed claim
computed over record bytes needed re-checking rather than assuming. All of them:

  claim                                        reads            verdict
  1,509 chunks / 34,491 records                offsets          unaffected
  1,262 chunks with a leading-byte bucket rule record byte 0    unaffected
  1,235 chunks with an index rule              record bytes 1-2 unaffected
  34,432 of 34,491 records 8-byte aligned      record byte 0    unaffected
  55 strides, all multiples of 4               offsets          unaffected
  moby class IDs in one slot per chunk type    record bytes      RETRACTED
  17-21% byte positions constant               record bytes      SURVIVES

Only claims that scanned across a record's BODY were at risk. The rest read a
record's first three bytes, which an over-read never touched -- it appends to the
end. That asymmetry is why the damage was contained to one claim, and it is the
argument for preferring claims anchored at record starts.

The survivor was the obvious candidate for contamination, so it is worth being
precise about it. Re-measured with each record bounded by its successor, and
comparing only the span every record in the chunk has (they are no longer all the
same length):

    lowest   level 8   16.7%
    highest             20.0%

Still inside the documented band. Previously this lived only in a doc comment on
the layout test; it is now asserted, in
chunk_records_share_a_layout_measured_by_constant_byte_positions, over all 27
levels with a 10-30% band so a real change reports itself rather than sliding.

The module documentation is corrected to the measured range and now says outright
that records in one chunk are not all the same length, since that is the fact a
reader would otherwise have to rediscover.

244 tests, clippy clean.
This commit is contained in:
ReaperOfVeriod
2026-10-13 08:02:31 +02:00
parent 4fab4d0359
commit 92aa0fe49f
4 changed files with 118 additions and 61 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/gc-extract/examples/u.rs
-57
View File
@@ -1,57 +0,0 @@
use std::collections::BTreeMap;
/// For each chunk with a non-uniform delta, look at the bytes a uniform stride
/// would swallow past the next record's start. Under "padding" they should be
/// uniform or zero; under "variable-length" they are data and should not be.
fn main() {
let data =
std::path::PathBuf::from(std::env::var("HOME").unwrap()).join("Documents/gc-extracted");
let mut hist: BTreeMap<Vec<u8>, usize> = BTreeMap::new();
let mut total = 0usize;
let mut zero = 0usize;
for lv in 0..6usize {
let Ok(core) = std::fs::read(data.join(format!("level{lv}/core_data.bin"))) else {
continue;
};
for c in gc_formats::chunks::parse_all_chunks(&core).unwrap() {
let s = c.chunk.stride;
let o = &c.chunk.offsets;
for w in o.windows(2) {
let (a, b) = (w[0] as usize, w[1] as usize);
if b - a >= s {
continue;
}
let gap = s - (b - a); // bytes the uniform slice over-reads
// the disputed bytes are the last `gap` of the uniform slice
let lo = c.chunk.offset + a + (b - a);
let Some(tail) = core.get(lo..lo + gap) else {
continue;
};
let mut t = tail.to_vec();
t.sort_unstable();
if t.iter().all(|&x| x == t[0]) {
zero += 1;
}
*hist.entry(tail.to_vec()).or_default() += 1;
total += 1;
}
}
}
println!(
"{total} over-reads examined; {zero} are a single repeated byte ({:.1}%)",
100.0 * zero as f64 / total.max(1) as f64
);
println!("\nmost common disputed byte sequences:");
let mut v: Vec<_> = hist.into_iter().collect();
v.sort_by_key(|(_, n)| std::cmp::Reverse(*n));
for (b, n) in v.iter().take(15) {
println!(
" x{n:<5} {}",
b.iter()
.map(|x| format!("{x:02x}"))
.collect::<Vec<_>>()
.join(" ")
);
}
println!("\ndistinct disputed sequences: {}", v.len());
}