ReaperOfVeriod/Going-Commando
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix the record boundaries after all, and retract a claim built on the bug

Browse Source 
I reverted the record-boundary fix last commit on the grounds that "fixed-length
records plus inter-record padding" and "variable-length records" produce the same
offset table. That reasoning was wrong, and cheaply so.

stride is the FIRST offset delta. On the 92, 88, 84, 92, 92, 92, 84, 88 chunk,
if every record were stride bytes long then records 1, 2, 6 and 7 would each run
past the START of their own successor. Records are addressed by an explicit
offset table; they cannot overlap. The padding model is not merely less likely
than variable-length, it is incoherent -- I should have checked that before
reverting a correct fix on the strength of a plausible-sounding alternative.

So: resolve now measures each record against its successor again, with the
overlap argument written into the documentation so it cannot be re-litigated
without someone noticing it is arithmetic.

THE RETRACTION, which is the substantive part:

Three tests broke. Two were straightforward, but the third was not.

  * constant_positions indexed records as if they were all the same length. Now
    that they are not, it compared past the short ones. It takes the minimum
    length instead.

  * "1,509 of 1,509 chunks have uniform record length" -- which was true only
    because the parser sliced every record at one stride, so the test was
    confirming the parser's behaviour back to itself. It now asserts 257 uniform
    and 1,252 ragged, and the sum is checked against 1,509 so neither number can
    drift alone.

  * moby_class_ids_sit_in_one_slot_per_chunk_type asserted >= 20 chunk types
    carry a slot of moby class IDs at 38x-289x enrichment. It now finds 2, of
    which 1 clears 100x. RETRACTED. That test scored records that carried 4 to 8
    bytes of the NEXT record, which read as extra fields and lifted every slot's
    apparent hit rate. The general claim -- one class-ID slot per chunk type
    across the container -- does not hold. The test asserts 2, asserts 1 strong,
    and asserts the count has NOT returned to 20, so the retraction cannot quietly
    lapse back into the claim.

That is a committed finding withdrawn on the strength of a parser bug. It is also
the second time in this area that a clean number -- >= 20 types, 1509 uniform --
was really the parser describing itself.

243 tests, clippy clean.
This commit is contained in:
ReaperOfVeriod
2026-10-13 07:59:23 +02:00
parent 5f0396e3d7
commit 4fab4d0359
4 changed files with 167 additions and 101 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/formats/core-data-regions.md
+25 -65
View File
@@ -967,77 +967,37 @@ correlating.
**Still open:** what the 64 types mean, which record fields are
index/length/offset, and whether any type is moby_anim.
## Chunk records reference moby class IDs at 38x chance
## RETRACTED: chunk records referencing moby class IDs at 38x chance
**Range measured:** level 0, the 900,000 bytes before the trailer; 28 chunks,
475 records, 67,780 `u16` slots. Reference set: the 238 distinct `o_class`
values from level 0's moby class table, read through `LevelCoreHeader`.
This section previously claimed that chunk records reference moby class IDs, with
one body slot per chunk type, at 38x-289x enrichment over chance, and that >= 20
chunk types showed it.
**Method:** scan every `u16` in every record; count how many are in the moby
class set. Compare against the rate expected if hits were chance, at
238/65536 per slot.
**That was a measurement of the parser, not of the disc.**
**Result: 9,493 hits against 246.1 expected -- 38.6x excess.**
`resolve` sliced every record at the chunk's *first* offset delta. On 1,252 of
1,509 chunks the later deltas are smaller -- word-aligned multiples of 4, most
exactly +/-4 -- so those slices ran past the record's real end and carried **4 to
8 bytes of the following record**. A record with the head of the next one glued
to its tail scores as though it carried extra fields, and a slot scan across
those bytes lifts apparent hit rates without anything being true.
The hits are not spread. They concentrate in a contiguous run of `u16` slots:
With record boundaries measured against each record's successor, and ragged
chunks skipped, what survives is:
| slot | hits | | slot | hits |
|---|---|---|---|---|
| 138 | 188 | | 134 | 173 |
| 135 | 182 | | 136 | 168 |
| 137 | 180 | | 129 | 139 |
| | | | 130 | 131 |
chunk types with a varying class-ID-like slot 2 (was >= 20)
of those, clearing 100x enrichment 1 (was >= 20)
Slots 129 through 140 -- twelve consecutive `u16`, that is bytes 258 through
280 of the record -- carry 105 to 188 hits each, and nothing else in the record
comes close except a separate cluster at slot 38.
So the general claim -- one class-ID slot per chunk type, across the container --
does not hold and is withdrawn. One chunk type still shows a slot that varies in
the manner of a class ID at 100x enrichment over chance. That is asserted, and
the test also asserts the count has *not* returned to 20, so the retraction
cannot quietly lapse back into the claim.
**What this looks like.** A contiguous twelve-value array in which most entries
are moby class identifiers is what a model or instance definition listing the
mobies it references would look like. Every chunk contributes in proportion to
its record count: the stride-340 chunk with 77 records has 1,451 hits, the
stride-368 chunk 1,384, the stride-148 chunk with 10 records has 254.
The `distinct >= 4` filter this section introduced is still load-bearing and is
unaffected: without it the top slots are constants that read as 305x, because any
value below the moby count counts as a hit.
**What this does not show.** Enrichment is not identification. A 38x excess
over chance is strong evidence that these values are moby class IDs rather
than incidental small integers, but it does not establish that the containing
chunk is `moby_anim` -- an instance table, a model list and an animation table
would all reference class IDs. The format is still unnamed, and so is the
meaning of slots 129-140.
**Correction to the above: the slot offsets were a pooling artifact.**
The entry above reported slots 129 through 140. Repeating the scan per chunk
type shows that is wrong: **every chunk type places its class array at a
different offset.**
| type (header prefix) | top hit slots |
|---|---|
| `aac4` | 129-134 |
| `82c4` | 104-109 |
| `4ec4` | 106-111 |
| `b8c4` | 120-125 |
| `9943` | 82-87 |
| `79c4` | 111-115 |
| `3245` | 24-29 |
Slots 129-140 were simply the type with the largest stride, dominating a scan
that pooled all types together. The previous entry flagged exactly this as
unverified, and the check confirmed the flag.
**What survives the correction.** In every one of the 26 types the top hits
form a **contiguous run of consecutive `u16` slots**. That is the real
invariant, and it is per-type: each chunk type has its own class array at its
own offset, of its own length.
So the corrected statement is: chunk records contain a contiguous array of
moby class IDs, and where that array sits depends on the chunk's type header.
This is consistent with each type being a distinct record layout, which is
what the header-implies-one-stride result already indicated -- but the class
array is a per-type field, not a common one at a fixed offset.
The moby reference itself is unaffected: 38.6x excess measured across all
records stands, since it does not depend on which slots were counted.
## Shrub body records on level 0: six varying bytes, two constant
@@ -1825,8 +1785,8 @@ likely to be lost. Audit against `crates/gc-extract/tests/disc.rs`:
packets[0] == 256 on 1,864 of 1,864 same test
chunk record length uniform per chunk tie-classes test + chunk_records_
start_16_byte_aligned
moby class IDs in one body slot per type moby_class_ids_sit_in_one_slot_
per_chunk_type
moby class IDs in one body slot per type RETRACTED -- see above; the test
now asserts 2 types, 1 strong
collision payload essentially verbatim collision_payload_is_verbatim_
in_ram (GC_SSTATE-gated)
tfrag pointers relocate, patches rewrite tfrag_table_pointers_relocate_