Fix the record boundaries after all, and retract a claim built on the bug

I reverted the record-boundary fix last commit on the grounds that "fixed-length
records plus inter-record padding" and "variable-length records" produce the same
offset table. That reasoning was wrong, and cheaply so.

stride is the FIRST offset delta. On the 92, 88, 84, 92, 92, 92, 84, 88 chunk,
if every record were stride bytes long then records 1, 2, 6 and 7 would each run
past the START of their own successor. Records are addressed by an explicit
offset table; they cannot overlap. The padding model is not merely less likely
than variable-length, it is incoherent -- I should have checked that before
reverting a correct fix on the strength of a plausible-sounding alternative.

So: resolve now measures each record against its successor again, with the
overlap argument written into the documentation so it cannot be re-litigated
without someone noticing it is arithmetic.

THE RETRACTION, which is the substantive part:

Three tests broke. Two were straightforward, but the third was not.

  * constant_positions indexed records as if they were all the same length. Now
    that they are not, it compared past the short ones. It takes the minimum
    length instead.

  * "1,509 of 1,509 chunks have uniform record length" -- which was true only
    because the parser sliced every record at one stride, so the test was
    confirming the parser's behaviour back to itself. It now asserts 257 uniform
    and 1,252 ragged, and the sum is checked against 1,509 so neither number can
    drift alone.

  * moby_class_ids_sit_in_one_slot_per_chunk_type asserted >= 20 chunk types
    carry a slot of moby class IDs at 38x-289x enrichment. It now finds 2, of
    which 1 clears 100x. RETRACTED. That test scored records that carried 4 to 8
    bytes of the NEXT record, which read as extra fields and lifted every slot's
    apparent hit rate. The general claim -- one class-ID slot per chunk type
    across the container -- does not hold. The test asserts 2, asserts 1 strong,
    and asserts the count has NOT returned to 20, so the retraction cannot quietly
    lapse back into the claim.

That is a committed finding withdrawn on the strength of a parser bug. It is also
the second time in this area that a clean number -- >= 20 types, 1509 uniform --
was really the parser describing itself.

243 tests, clippy clean.

crates/gc-extract/examples/u.rs

@@ -0,0 +1,57 @@
+use std::collections::BTreeMap;
+
+/// For each chunk with a non-uniform delta, look at the bytes a uniform stride
+/// would swallow past the next record's start. Under "padding" they should be
+/// uniform or zero; under "variable-length" they are data and should not be.
+fn main() {
+    let data =
+        std::path::PathBuf::from(std::env::var("HOME").unwrap()).join("Documents/gc-extracted");
+    let mut hist: BTreeMap<Vec<u8>, usize> = BTreeMap::new();
+    let mut total = 0usize;
+    let mut zero = 0usize;
+    for lv in 0..6usize {
+        let Ok(core) = std::fs::read(data.join(format!("level{lv}/core_data.bin"))) else {
+            continue;
+        };
+        for c in gc_formats::chunks::parse_all_chunks(&core).unwrap() {
+            let s = c.chunk.stride;
+            let o = &c.chunk.offsets;
+            for w in o.windows(2) {
+                let (a, b) = (w[0] as usize, w[1] as usize);
+                if b - a >= s {
+                    continue;
+                }
+                let gap = s - (b - a); // bytes the uniform slice over-reads
+                                       // the disputed bytes are the last `gap` of the uniform slice
+                let lo = c.chunk.offset + a + (b - a);
+                let Some(tail) = core.get(lo..lo + gap) else {
+                    continue;
+                };
+                let mut t = tail.to_vec();
+                t.sort_unstable();
+                if t.iter().all(|&x| x == t[0]) {
+                    zero += 1;
+                }
+                *hist.entry(tail.to_vec()).or_default() += 1;
+                total += 1;
+            }
+        }
+    }
+    println!(
+        "{total} over-reads examined; {zero} are a single repeated byte ({:.1}%)",
+        100.0 * zero as f64 / total.max(1) as f64
+    );
+    println!("\nmost common disputed byte sequences:");
+    let mut v: Vec<_> = hist.into_iter().collect();
+    v.sort_by_key(|(_, n)| std::cmp::Reverse(*n));
+    for (b, n) in v.iter().take(15) {
+        println!(
+            "  x{n:<5} {}",
+            b.iter()
+                .map(|x| format!("{x:02x}"))
+                .collect::<Vec<_>>()
+                .join(" ")
+        );
+    }
+    println!("\ndistinct disputed sequences: {}", v.len());
+}

crates/gc-extract/tests/disc.rs

@@ -2577,7 +2577,11 @@ fn constant_positions(records: &[&[u8]]) -> usize {
     let Some(first) = records.first() else {
         return 0;
     };
-    (0..first.len())
+    // Records in one chunk are NOT all the same length: the offset deltas vary,
+    // and `resolve` measures each record against its successor. Compare only the
+    // span every record has, or this indexes past the short ones.
+    let len = records.iter().map(|r| r.len()).min().unwrap_or(0);
+    (0..len)
         .filter(|&i| records.iter().all(|r| r[i] == first[i]))
         .count()
 }
@@ -3151,10 +3155,14 @@ fn chunk_records_start_16_byte_aligned() {
 
     assert!(chunks >= 1500, "only {chunks} chunks parsed");
     assert!(records >= 34000, "only {records} chunk records");
-    // Every chunk has a uniform record length -- length varies between chunks
-    // of a type, never within one -- and on most chunks the leading byte is a
-    // per-chunk phase counter rather than data.
+    // Record length varies BETWEEN chunks of a type, and -- contrary to what
+    // this test used to assert -- also WITHIN a chunk, on 1,252 of them. The old
+    // "1,509 of 1,509 uniform" was true only because `resolve` sliced every
+    // record at the chunk's first delta, which the test was then confirming back
+    // to itself. On most chunks the leading byte is a per-chunk phase counter
+    // rather than data.
     let mut uniform = 0usize;
+    let mut ragged_chunks = 0usize;
     let mut ruled = 0usize;
     let mut indexed = 0usize;
     for n in 0..27 {
@@ -3162,6 +3170,8 @@ fn chunk_records_start_16_byte_aligned() {
         for c in parse_all_chunks(&core).expect("chunks") {
             if c.has_uniform_record_length() {
                 uniform += 1;
+            } else {
+                ragged_chunks += 1;
             }
             if c.bucket_rule().is_some() {
                 ruled += 1;
@@ -3171,7 +3181,12 @@ fn chunk_records_start_16_byte_aligned() {
             }
         }
     }
-    assert_eq!(uniform, 1509, "uniform-record-length count moved off 1509");
+    assert_eq!(uniform, 257, "uniform-record-length count moved off 257");
+    assert_eq!(
+        uniform + ragged_chunks,
+        1509,
+        "chunk total moved; add the ragged counter"
+    );
     assert_eq!(ruled, 1262, "bucket-rule count moved off 1262");
     assert_eq!(indexed, 1235, "index-rule count moved off 1235");
 
@@ -3207,9 +3222,20 @@ fn chunk_records_start_16_byte_aligned() {
 
 /// Moby class IDs live in one body slot per chunk type, and the slot varies.
 ///
-/// Measured over all 27 levels: the best slot per type carries 5 to 42 distinct
-/// valid moby class IDs at 153x to 289x enrichment over the chance rate. The
-/// `distinct >= 4` filter is load-bearing -- without it the top slots are
+/// **Largely retracted.** This used to assert that >= 20 chunk types carry a
+/// slot of moby class IDs, at 153x to 289x enrichment over chance.
+///
+/// It was measuring over-read records. `resolve` sliced every record at the
+/// chunk's first offset delta, which on 1,252 of 1,509 chunks ran 4 to 8 bytes
+/// past a record's real end -- four bytes of the *next* record, glued on. That
+/// reads as extra fields and lifts a slot's apparent hit rate. With record
+/// boundaries measured against each record's successor, and ragged chunks
+/// skipped, only 2 types show a varying class-ID slot.
+///
+/// What survives is narrow and is asserted as such: 2 chunk types show a slot
+/// that varies in the manner of a class ID, and exactly 1 of those still clears
+/// 100x enrichment over chance. The
+/// `distinct >= 4` filter is still load-bearing -- without it the top slots are
 /// constants that read as 305x because any value below the moby count counts as
 /// a hit. See `docs/formats/core-data-regions.md`.
 #[test]
@@ -3295,16 +3321,27 @@ fn moby_class_ids_sit_in_one_slot_per_chunk_type() {
         }
     }
 
-    assert!(
-        best.len() >= 20,
-        "only {} chunk types carry a varying class-ID slot",
-        best.len()
-    );
+    // RETRACTED. This asserted >= 20 types while every record was being sliced
+    // at the chunk's first delta, which over-read 4 to 8 bytes past the true end
+    // of a record on 1,252 of 1,509 chunks. Records with four bytes of the next
+    // one glued to the end score as though they carried extra fields. With the
+    // boundaries fixed, and this test skipping ragged chunks, only 2 types show
+    // a varying class-ID slot -- so "one class-ID slot per chunk type" was a
+    // measurement of the parser's over-read, not of the disc.
     let strong: Vec<_> = best.values().filter(|(_, r, _)| *r >= 100.0).collect();
     assert!(
-        strong.len() >= 20,
-        "only {} types reach 100x enrichment",
-        strong.len()
+        !strong.is_empty(),
+        "no type reaches 100x enrichment -- was 20 before the record-boundary fix"
+    );
+    assert_eq!(strong.len(), 1, "strong-enrichment type count moved");
+    assert!(
+        best.len() >= 2,
+        "only {} chunk types carry a varying class-ID slot (was >= 20)",
+        best.len()
+    );
+    assert!(
+        best.len() < 20,
+        "class-ID slots are back at >= 20 types; revisit the retraction"
     );
 
     let mut all: Vec<_> = best