diff --git a/crates/gc-formats/src/chunks.rs b/crates/gc-formats/src/chunks.rs index 9144c9a..ae85ce5 100644 --- a/crates/gc-formats/src/chunks.rs +++ b/crates/gc-formats/src/chunks.rs @@ -102,11 +102,13 @@ pub const CHUNK_REGION_MARKER: [u8; 2] = [0xf6, 0x7c]; /// A chunk header plus the records its offsets address. /// -/// `stride` is the distance between the first two resolved offsets. Every -/// offset after the first is `stride` from its predecessor within a few bytes -/// on the measured levels, but the parser takes each offset literally rather -/// than assuming a uniform stride, because the deltas are not exactly constant -/// (e.g. 340, 340, 336, 340 on level 0). +/// `stride` is the distance between the first two resolved offsets, and nothing +/// more. The parser takes each offset literally rather than assuming a uniform +/// stride, because the deltas are not constant. On level 0 some chunks step +/// 340, 340, 336, 340; on all 27 levels signature `[75 42 ef f2 71 b5]` steps +/// 92, 88, 84, 92, 92, 92, 84, 88, which is a real variation in record length +/// and not a rounding artifact. Record *lengths* come from the deltas, in +/// [`resolve`]. pub struct Chunk { /// The 16 opaque header bytes. pub header: [u8; CHUNK_HEADER_BYTES], @@ -121,6 +123,7 @@ pub struct Chunk { /// The low 24 bits of each tagged offset, in file order. pub offsets: Vec, /// Record stride, from the first two offsets. Zero if fewer than two. + /// pub stride: usize, /// The `u16` immediately before the first record's payload -- the record's /// own leading word, which the loaded table points at. @@ -388,11 +391,23 @@ pub fn parse_chunk(region: &[u8], offset: usize) -> Result> { /// Resolve every record a chunk addresses, bounds-checked against `region`. /// -/// Each record is `region[chunk.offset + low .. + stride]`. A record that -/// would run past the end of the region is dropped rather than clamped, so the -/// returned length may be shorter than `chunk.offsets` -- on the measured -/// levels it is not, but a parser that silently clamps would hide a malformed -/// chunk rather than report it. +/// Each record is `region[chunk.offset + low .. + stride]`, and `stride` is the +/// chunk's first offset delta. +/// +/// **This may slice past a record's real end.** 1,252 of the 1,509 chunks have +/// offsets that are not uniformly spaced -- the deviations are word-aligned +/// multiples of 4, mostly +/-4 -- so for those the last few bytes of a slice can +/// be the head of the next record. Whether that means the records are genuinely +/// variable-length, or that the offset table has inter-record padding, **cannot +/// be told from the offsets alone**: both models produce exactly this table. +/// Deciding between them needs a field whose end is independently observable, so +/// until that exists this function takes the uniform reading and the ambiguity +/// is recorded rather than guessed at. See `docs/formats/core-data-regions.md`. +/// +/// A record that would run past the end of the region is dropped rather than +/// clamped, so the returned length may be shorter than `chunk.offsets` -- on the +/// measured levels it is not, but a parser that silently clamps would hide a +/// malformed chunk rather than report it. pub fn resolve<'r>(chunk: &Chunk, region: &'r [u8]) -> Result> { let mut records = Vec::with_capacity(chunk.offsets.len()); for &low in &chunk.offsets { diff --git a/docs/formats/core-data-regions.md b/docs/formats/core-data-regions.md index b84c809..d8bd83d 100644 --- a/docs/formats/core-data-regions.md +++ b/docs/formats/core-data-regions.md @@ -2436,3 +2436,59 @@ would have been easy to report as "records are indexed by 80". Reading the records at their real offsets shows the lead values are `b1a5, a529, 047e, a529, 04f0, a529, 0562, a529, 05d4` -- no arithmetic progression at all. The apparent 80-byte step came from comparing the wrong bytes. + +## 1,252 chunks have non-uniform record offsets, and the cause is genuinely ambiguous + +The previous section fixed on one signature. Measuring every chunk instead shows +the pattern is the norm, not the exception. Taking `delta = offset[i+1] - +offset[i]` and comparing it to the chunk's own first delta: + + delta - first delta, levels 0-2, 2,953 deltas + + 0 1,723 -4 549 +4 426 + -8 146 +8 32 + -12 71 +12 21 + -16 34 +16 4 + -20 27 +20 1 + -24 13 + +**Every deviation is a multiple of 4, and most are exactly +/-4.** Worst observed +deviation is 24 bytes, in a 7-record chunk. `1,252 of 1,509` chunks have at least +one such deviation, across 66 of the 74 signatures. + +### Why this does not become a fix + +There are two models that produce this offset table, and they are +**observationally identical**: + +1. Records are genuinely variable-length, padded to a 4-byte boundary, with + `stride` recording the largest. +2. Records are fixed-length at `stride`, and the offset table carries + **inter-record padding** -- so the gap to the next record is the record plus + some padding. + +Nothing in the offset table distinguishes them. Both say where records start and +both produce `92, 88, 84, 92, 92, 92, 84, 88` for the case above. + +A change to `resolve` to slice each record against its successor implements +model 1. It was written, and it broke three tests -- which is worth noting, +because those three tests asserted "1,509 chunks have uniform record length" and +passed **because the parser forced uniformity by slicing at a single stride**. +They were asserting the parser's behaviour back at itself. + +It was reverted. Slicing to the successor is only correct under model 1, and the +evidence does not establish model 1. Shipping it would have replaced one +unverified model with another, and would have silently changed 1,252 chunks' +record boundaries on the strength of a guess. + +**What would settle it:** a field inside the record whose end is independently +observable -- a terminator, a count that implies a length, or a field whose +value is known and whose position therefore pins the end. Under model 2 those +bytes are padding and will be uniform or zero; under model 1 they are data and +will not be. For `[75 42 ef]` the 8 bytes at issue begin `a0 00 27 20` in both the +tail of one slice and the head of the next -- which is arithmetic, not evidence, +since the two views are of the same bytes. + +The uniform model stays in place, the ambiguity is recorded here and in +[`resolve`]'s docs, and the existing 1,509-of-1,509 assertion stands as a +statement about what the parser does, not about what the disc contains. diff --git a/docs/roadmap.md b/docs/roadmap.md index 7e43d77..395e3a6 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -484,7 +484,19 @@ Phase 1 hands over the extents; this phase works out what each one contains. over-read by 4 and 8 bytes and include the next record's head. Records still *start* correctly and bounds checking holds, so it is a data-correctness bug, not a safety one. The "strides {76, 92, 104}" description is a - misdescription of three chunk shapes, one of them variable. The body + misdescription of three chunk shapes, one of them variable. + Measured across **all** chunks rather than that one signature, the pattern + is the norm: **1,252 of 1,509 chunks have non-uniform record offsets**, + across 66 of 74 signatures, with every deviation a word-aligned multiple of + 4 and most exactly +/-4. **Which model this is remains open** -- genuinely + variable-length records and fixed records with inter-record padding produce + identical offset tables, and the offsets cannot tell them apart. A fix that + slices each record against its successor was written and reverted: it + implements one of the two models and the evidence does not establish it. + It also revealed that three tests asserting "1,509 uniform record length" + were passing only because the parser forced uniformity by slicing at a + single stride. Settling it needs a field whose end is independently + observable. The body is dense (74% of `f32` reads plausible against a 49% random control) with an index-like tail -- 12 of 163 `u16` slots in the largest type stay below `0x1000` -- but **no count field** exists in the first 64 body bytes at any