From 5f0396e3d7087dc6643ca79265276ab4dcfd3f6c Mon Sep 17 00:00:00 2001 From: ReaperOfVeriod Date: Tue, 13 Oct 2026 07:50:50 +0200 Subject: [PATCH] Measure every chunk's record offsets; the cause is ambiguous, so no fix The previous commit fixed on one signature. Measuring all of them shows the pattern is the norm. Taking delta = offset[i+1] - offset[i] against the chunk's own first delta, levels 0-2, 2,953 deltas: 0 1,723 -4 549 +4 426 -8 146 +8 32 -12 71 +12 21 -16 34 +16 4 -20 27 +20 1 -24 13 Every deviation is a multiple of 4 and most are exactly +/-4; worst observed is 24 bytes in a 7-record chunk. 1,252 of 1,509 chunks have at least one, across 66 of 74 signatures. WHY THIS DOES NOT BECOME A FIX Two models produce this offset table and they are observationally identical: 1. records are genuinely variable-length, padded to 4 bytes, stride = largest 2. records are fixed at stride and the offset table carries inter-record padding, so the gap to the next record is record + padding Nothing in the offsets distinguishes them. Both say where records start. Both produce 92, 88, 84, 92, 92, 92, 84, 88 for the case already discussed. I wrote the change that slices each record against its successor -- model 1 -- and then reverted it. Two things came out of writing it: First, it broke three tests, which is the informative part. Those tests assert "1,509 of 1,509 chunks have uniform record length" and they passed BECAUSE the parser forced uniformity by slicing every record at a single stride. They were asserting the parser's behaviour back at itself, and would have kept passing forever regardless of what the disc contains. Second, shipping it would have replaced one unverified model with another and silently changed the record boundaries of 1,252 chunks on the strength of a guess. The repo's own rule applies: a heuristic count is not an identification, and here it is not even a count -- it is two equally good explanations of one table. WHAT WOULD SETTLE IT: a field inside the record whose end is independently observable -- a terminator, a count implying a length, or a known value pinning its position. Under model 2 the disputed bytes are padding and will be uniform or zero; under model 1 they are data and will not be. For [75 42 ef] those 8 bytes read a0 00 27 20 at both the tail of one slice and the head of the next, which is arithmetic rather than evidence, since it is the same eight bytes seen twice. The uniform model stays, the ambiguity is recorded in resolve()'s documentation and here, and the 1,509-of-1,509 assertion stands as a statement about what the parser does rather than about what the disc contains. 243 tests, clippy clean. --- crates/gc-formats/src/chunks.rs | 35 +++++++++++++------ docs/formats/core-data-regions.md | 56 +++++++++++++++++++++++++++++++ docs/roadmap.md | 14 +++++++- 3 files changed, 94 insertions(+), 11 deletions(-) diff --git a/crates/gc-formats/src/chunks.rs b/crates/gc-formats/src/chunks.rs index 9144c9a..ae85ce5 100644 --- a/crates/gc-formats/src/chunks.rs +++ b/crates/gc-formats/src/chunks.rs @@ -102,11 +102,13 @@ pub const CHUNK_REGION_MARKER: [u8; 2] = [0xf6, 0x7c]; /// A chunk header plus the records its offsets address. /// -/// `stride` is the distance between the first two resolved offsets. Every -/// offset after the first is `stride` from its predecessor within a few bytes -/// on the measured levels, but the parser takes each offset literally rather -/// than assuming a uniform stride, because the deltas are not exactly constant -/// (e.g. 340, 340, 336, 340 on level 0). +/// `stride` is the distance between the first two resolved offsets, and nothing +/// more. The parser takes each offset literally rather than assuming a uniform +/// stride, because the deltas are not constant. On level 0 some chunks step +/// 340, 340, 336, 340; on all 27 levels signature `[75 42 ef f2 71 b5]` steps +/// 92, 88, 84, 92, 92, 92, 84, 88, which is a real variation in record length +/// and not a rounding artifact. Record *lengths* come from the deltas, in +/// [`resolve`]. pub struct Chunk { /// The 16 opaque header bytes. pub header: [u8; CHUNK_HEADER_BYTES], @@ -121,6 +123,7 @@ pub struct Chunk { /// The low 24 bits of each tagged offset, in file order. pub offsets: Vec, /// Record stride, from the first two offsets. Zero if fewer than two. + /// pub stride: usize, /// The `u16` immediately before the first record's payload -- the record's /// own leading word, which the loaded table points at. @@ -388,11 +391,23 @@ pub fn parse_chunk(region: &[u8], offset: usize) -> Result> { /// Resolve every record a chunk addresses, bounds-checked against `region`. /// -/// Each record is `region[chunk.offset + low .. + stride]`. A record that -/// would run past the end of the region is dropped rather than clamped, so the -/// returned length may be shorter than `chunk.offsets` -- on the measured -/// levels it is not, but a parser that silently clamps would hide a malformed -/// chunk rather than report it. +/// Each record is `region[chunk.offset + low .. + stride]`, and `stride` is the +/// chunk's first offset delta. +/// +/// **This may slice past a record's real end.** 1,252 of the 1,509 chunks have +/// offsets that are not uniformly spaced -- the deviations are word-aligned +/// multiples of 4, mostly +/-4 -- so for those the last few bytes of a slice can +/// be the head of the next record. Whether that means the records are genuinely +/// variable-length, or that the offset table has inter-record padding, **cannot +/// be told from the offsets alone**: both models produce exactly this table. +/// Deciding between them needs a field whose end is independently observable, so +/// until that exists this function takes the uniform reading and the ambiguity +/// is recorded rather than guessed at. See `docs/formats/core-data-regions.md`. +/// +/// A record that would run past the end of the region is dropped rather than +/// clamped, so the returned length may be shorter than `chunk.offsets` -- on the +/// measured levels it is not, but a parser that silently clamps would hide a +/// malformed chunk rather than report it. pub fn resolve<'r>(chunk: &Chunk, region: &'r [u8]) -> Result> { let mut records = Vec::with_capacity(chunk.offsets.len()); for &low in &chunk.offsets { diff --git a/docs/formats/core-data-regions.md b/docs/formats/core-data-regions.md index b84c809..d8bd83d 100644 --- a/docs/formats/core-data-regions.md +++ b/docs/formats/core-data-regions.md @@ -2436,3 +2436,59 @@ would have been easy to report as "records are indexed by 80". Reading the records at their real offsets shows the lead values are `b1a5, a529, 047e, a529, 04f0, a529, 0562, a529, 05d4` -- no arithmetic progression at all. The apparent 80-byte step came from comparing the wrong bytes. + +## 1,252 chunks have non-uniform record offsets, and the cause is genuinely ambiguous + +The previous section fixed on one signature. Measuring every chunk instead shows +the pattern is the norm, not the exception. Taking `delta = offset[i+1] - +offset[i]` and comparing it to the chunk's own first delta: + + delta - first delta, levels 0-2, 2,953 deltas + + 0 1,723 -4 549 +4 426 + -8 146 +8 32 + -12 71 +12 21 + -16 34 +16 4 + -20 27 +20 1 + -24 13 + +**Every deviation is a multiple of 4, and most are exactly +/-4.** Worst observed +deviation is 24 bytes, in a 7-record chunk. `1,252 of 1,509` chunks have at least +one such deviation, across 66 of the 74 signatures. + +### Why this does not become a fix + +There are two models that produce this offset table, and they are +**observationally identical**: + +1. Records are genuinely variable-length, padded to a 4-byte boundary, with + `stride` recording the largest. +2. Records are fixed-length at `stride`, and the offset table carries + **inter-record padding** -- so the gap to the next record is the record plus + some padding. + +Nothing in the offset table distinguishes them. Both say where records start and +both produce `92, 88, 84, 92, 92, 92, 84, 88` for the case above. + +A change to `resolve` to slice each record against its successor implements +model 1. It was written, and it broke three tests -- which is worth noting, +because those three tests asserted "1,509 chunks have uniform record length" and +passed **because the parser forced uniformity by slicing at a single stride**. +They were asserting the parser's behaviour back at itself. + +It was reverted. Slicing to the successor is only correct under model 1, and the +evidence does not establish model 1. Shipping it would have replaced one +unverified model with another, and would have silently changed 1,252 chunks' +record boundaries on the strength of a guess. + +**What would settle it:** a field inside the record whose end is independently +observable -- a terminator, a count that implies a length, or a field whose +value is known and whose position therefore pins the end. Under model 2 those +bytes are padding and will be uniform or zero; under model 1 they are data and +will not be. For `[75 42 ef]` the 8 bytes at issue begin `a0 00 27 20` in both the +tail of one slice and the head of the next -- which is arithmetic, not evidence, +since the two views are of the same bytes. + +The uniform model stays in place, the ambiguity is recorded here and in +[`resolve`]'s docs, and the existing 1,509-of-1,509 assertion stands as a +statement about what the parser does, not about what the disc contains. diff --git a/docs/roadmap.md b/docs/roadmap.md index 7e43d77..395e3a6 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -484,7 +484,19 @@ Phase 1 hands over the extents; this phase works out what each one contains. over-read by 4 and 8 bytes and include the next record's head. Records still *start* correctly and bounds checking holds, so it is a data-correctness bug, not a safety one. The "strides {76, 92, 104}" description is a - misdescription of three chunk shapes, one of them variable. The body + misdescription of three chunk shapes, one of them variable. + Measured across **all** chunks rather than that one signature, the pattern + is the norm: **1,252 of 1,509 chunks have non-uniform record offsets**, + across 66 of 74 signatures, with every deviation a word-aligned multiple of + 4 and most exactly +/-4. **Which model this is remains open** -- genuinely + variable-length records and fixed records with inter-record padding produce + identical offset tables, and the offsets cannot tell them apart. A fix that + slices each record against its successor was written and reverted: it + implements one of the two models and the evidence does not establish it. + It also revealed that three tests asserting "1,509 uniform record length" + were passing only because the parser forced uniformity by slicing at a + single stride. Settling it needs a field whose end is independently + observable. The body is dense (74% of `f32` reads plausible against a 49% random control) with an index-like tail -- 12 of 163 `u16` slots in the largest type stay below `0x1000` -- but **no count field** exists in the first 64 body bytes at any