Measure every chunk's record offsets; the cause is ambiguous, so no fix

The previous commit fixed on one signature. Measuring all of them shows the
pattern is the norm. Taking delta = offset[i+1] - offset[i] against the chunk's
own first delta, levels 0-2, 2,953 deltas:

      0  1,723      -4    549      +4    426
     -8    146      +8     32
    -12     71     +12     21
    -16     34     +16      4
    -20     27     +20      1
    -24     13

Every deviation is a multiple of 4 and most are exactly +/-4; worst observed is 24
bytes in a 7-record chunk. 1,252 of 1,509 chunks have at least one, across 66 of
74 signatures.

WHY THIS DOES NOT BECOME A FIX

Two models produce this offset table and they are observationally identical:

  1. records are genuinely variable-length, padded to 4 bytes, stride = largest
  2. records are fixed at stride and the offset table carries inter-record
     padding, so the gap to the next record is record + padding

Nothing in the offsets distinguishes them. Both say where records start. Both
produce 92, 88, 84, 92, 92, 92, 84, 88 for the case already discussed.

I wrote the change that slices each record against its successor -- model 1 -- and
then reverted it. Two things came out of writing it:

First, it broke three tests, which is the informative part. Those tests assert
"1,509 of 1,509 chunks have uniform record length" and they passed BECAUSE the
parser forced uniformity by slicing every record at a single stride. They were
asserting the parser's behaviour back at itself, and would have kept passing
forever regardless of what the disc contains.

Second, shipping it would have replaced one unverified model with another and
silently changed the record boundaries of 1,252 chunks on the strength of a
guess. The repo's own rule applies: a heuristic count is not an identification,
and here it is not even a count -- it is two equally good explanations of one
table.

WHAT WOULD SETTLE IT: a field inside the record whose end is independently
observable -- a terminator, a count implying a length, or a known value pinning
its position. Under model 2 the disputed bytes are padding and will be uniform or
zero; under model 1 they are data and will not be. For [75 42 ef] those 8 bytes
read a0 00 27 20 at both the tail of one slice and the head of the next, which is
arithmetic rather than evidence, since it is the same eight bytes seen twice.

The uniform model stays, the ambiguity is recorded in resolve()'s documentation
and here, and the 1,509-of-1,509 assertion stands as a statement about what the
parser does rather than about what the disc contains. 243 tests, clippy clean.
This commit is contained in:
2026-10-13 07:50:50 +02:00
parent b9e9835639
commit 5f0396e3d7
3 changed files with 94 additions and 11 deletions
+56
View File
@@ -2436,3 +2436,59 @@ would have been easy to report as "records are indexed by 80". Reading the
records at their real offsets shows the lead values are `b1a5, a529, 047e,
a529, 04f0, a529, 0562, a529, 05d4` -- no arithmetic progression at all. The
apparent 80-byte step came from comparing the wrong bytes.
## 1,252 chunks have non-uniform record offsets, and the cause is genuinely ambiguous
The previous section fixed on one signature. Measuring every chunk instead shows
the pattern is the norm, not the exception. Taking `delta = offset[i+1] -
offset[i]` and comparing it to the chunk's own first delta:
delta - first delta, levels 0-2, 2,953 deltas
0 1,723 -4 549 +4 426
-8 146 +8 32
-12 71 +12 21
-16 34 +16 4
-20 27 +20 1
-24 13
**Every deviation is a multiple of 4, and most are exactly +/-4.** Worst observed
deviation is 24 bytes, in a 7-record chunk. `1,252 of 1,509` chunks have at least
one such deviation, across 66 of the 74 signatures.
### Why this does not become a fix
There are two models that produce this offset table, and they are
**observationally identical**:
1. Records are genuinely variable-length, padded to a 4-byte boundary, with
`stride` recording the largest.
2. Records are fixed-length at `stride`, and the offset table carries
**inter-record padding** -- so the gap to the next record is the record plus
some padding.
Nothing in the offset table distinguishes them. Both say where records start and
both produce `92, 88, 84, 92, 92, 92, 84, 88` for the case above.
A change to `resolve` to slice each record against its successor implements
model 1. It was written, and it broke three tests -- which is worth noting,
because those three tests asserted "1,509 chunks have uniform record length" and
passed **because the parser forced uniformity by slicing at a single stride**.
They were asserting the parser's behaviour back at itself.
It was reverted. Slicing to the successor is only correct under model 1, and the
evidence does not establish model 1. Shipping it would have replaced one
unverified model with another, and would have silently changed 1,252 chunks'
record boundaries on the strength of a guess.
**What would settle it:** a field inside the record whose end is independently
observable -- a terminator, a count that implies a length, or a field whose
value is known and whose position therefore pins the end. Under model 2 those
bytes are padding and will be uniform or zero; under model 1 they are data and
will not be. For `[75 42 ef]` the 8 bytes at issue begin `a0 00 27 20` in both the
tail of one slice and the head of the next -- which is arithmetic, not evidence,
since the two views are of the same bytes.
The uniform model stays in place, the ambiguity is recorded here and in
[`resolve`]'s docs, and the existing 1,509-of-1,509 assertion stands as a
statement about what the parser does, not about what the disc contains.