e1482abd755b54a223e7806f3225f5ba294a53a9
11
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 ReaperOfVeriod
|
8ccc3ddda4 |
Retract the uniformity claim everywhere it still stood, including the API docs
The record-length retraction had been applied to the tests and to the section
that discovered the problem, but the same claim was still standing in four other
places -- including the rustdoc for has_uniform_record_length(), which is what a
caller would actually read:
crates/gc-formats/src/chunks.rs:206 "Measured: true for 1,509 of 1,509 chunks"
docs/roadmap.md:448 "uniform record length (1,509 of 1,509)"
docs/formats/core-data-regions.md a whole section titled
"Every chunk has a uniform record length"
All three now say 257 of 1,509, and the method's rustdoc explains WHY the old
number was an artefact -- that resolve sliced at the first offset delta, so every
record came out the same length by construction and the predicate was confirming
the parser back to itself.
That section also contained a withdrawal, and the withdrawal was wrong:
"This withdraws the previous commit's variable-length type. That came from
taking minlen/maxlen over all of a type's records at once, pooling across
chunks; rule 3a's pooled claim, third time this session."
The pooling rule fired correctly and was overruled by a measurement with no
independent content. The per-chunk check that should have caught the pooled
signal was itself reporting the parser's own slicing, so it agreed with the wrong
answer for the same reason the original claim did. Two errors stacked: a real
signal dismissed by a rule that had been right twice already, and a predicate
that could not have told the difference.
This is the third time this session that a test was asserting the parser's
behaviour rather than the disc's, and the third time it was the only thing
standing between a claim and a retraction. has_uniform_record_length is the
clearest instance: a predicate named after a property of the data, returning a
confident answer about the code.
245 tests, clippy clean.
|
||
|
ReaperOfVeriod
|
92aa0fe49f |
Audit what the record-boundary fix contaminated: one retraction, one survivor
The fix changed record lengths in 1,252 of 1,509 chunks, so every committed claim
computed over record bytes needed re-checking rather than assuming. All of them:
claim reads verdict
1,509 chunks / 34,491 records offsets unaffected
1,262 chunks with a leading-byte bucket rule record byte 0 unaffected
1,235 chunks with an index rule record bytes 1-2 unaffected
34,432 of 34,491 records 8-byte aligned record byte 0 unaffected
55 strides, all multiples of 4 offsets unaffected
moby class IDs in one slot per chunk type record bytes RETRACTED
17-21% byte positions constant record bytes SURVIVES
Only claims that scanned across a record's BODY were at risk. The rest read a
record's first three bytes, which an over-read never touched -- it appends to the
end. That asymmetry is why the damage was contained to one claim, and it is the
argument for preferring claims anchored at record starts.
The survivor was the obvious candidate for contamination, so it is worth being
precise about it. Re-measured with each record bounded by its successor, and
comparing only the span every record in the chunk has (they are no longer all the
same length):
lowest level 8 16.7%
highest 20.0%
Still inside the documented band. Previously this lived only in a doc comment on
the layout test; it is now asserted, in
chunk_records_share_a_layout_measured_by_constant_byte_positions, over all 27
levels with a 10-30% band so a real change reports itself rather than sliding.
The module documentation is corrected to the measured range and now says outright
that records in one chunk are not all the same length, since that is the fact a
reader would otherwise have to rediscover.
244 tests, clippy clean.
|
||
|
ReaperOfVeriod
|
4fab4d0359 |
Fix the record boundaries after all, and retract a claim built on the bug
I reverted the record-boundary fix last commit on the grounds that "fixed-length
records plus inter-record padding" and "variable-length records" produce the same
offset table. That reasoning was wrong, and cheaply so.
stride is the FIRST offset delta. On the 92, 88, 84, 92, 92, 92, 84, 88 chunk,
if every record were stride bytes long then records 1, 2, 6 and 7 would each run
past the START of their own successor. Records are addressed by an explicit
offset table; they cannot overlap. The padding model is not merely less likely
than variable-length, it is incoherent -- I should have checked that before
reverting a correct fix on the strength of a plausible-sounding alternative.
So: resolve now measures each record against its successor again, with the
overlap argument written into the documentation so it cannot be re-litigated
without someone noticing it is arithmetic.
THE RETRACTION, which is the substantive part:
Three tests broke. Two were straightforward, but the third was not.
* constant_positions indexed records as if they were all the same length. Now
that they are not, it compared past the short ones. It takes the minimum
length instead.
* "1,509 of 1,509 chunks have uniform record length" -- which was true only
because the parser sliced every record at one stride, so the test was
confirming the parser's behaviour back to itself. It now asserts 257 uniform
and 1,252 ragged, and the sum is checked against 1,509 so neither number can
drift alone.
* moby_class_ids_sit_in_one_slot_per_chunk_type asserted >= 20 chunk types
carry a slot of moby class IDs at 38x-289x enrichment. It now finds 2, of
which 1 clears 100x. RETRACTED. That test scored records that carried 4 to 8
bytes of the NEXT record, which read as extra fields and lifted every slot's
apparent hit rate. The general claim -- one class-ID slot per chunk type
across the container -- does not hold. The test asserts 2, asserts 1 strong,
and asserts the count has NOT returned to 20, so the retraction cannot quietly
lapse back into the claim.
That is a committed finding withdrawn on the strength of a parser bug. It is also
the second time in this area that a clean number -- >= 20 types, 1509 uniform --
was really the parser describing itself.
243 tests, clippy clean.
|
||
|
ReaperOfVeriod
|
5f0396e3d7 |
Measure every chunk's record offsets; the cause is ambiguous, so no fix
The previous commit fixed on one signature. Measuring all of them shows the
pattern is the norm. Taking delta = offset[i+1] - offset[i] against the chunk's
own first delta, levels 0-2, 2,953 deltas:
0 1,723 -4 549 +4 426
-8 146 +8 32
-12 71 +12 21
-16 34 +16 4
-20 27 +20 1
-24 13
Every deviation is a multiple of 4 and most are exactly +/-4; worst observed is 24
bytes in a 7-record chunk. 1,252 of 1,509 chunks have at least one, across 66 of
74 signatures.
WHY THIS DOES NOT BECOME A FIX
Two models produce this offset table and they are observationally identical:
1. records are genuinely variable-length, padded to 4 bytes, stride = largest
2. records are fixed at stride and the offset table carries inter-record
padding, so the gap to the next record is record + padding
Nothing in the offsets distinguishes them. Both say where records start. Both
produce 92, 88, 84, 92, 92, 92, 84, 88 for the case already discussed.
I wrote the change that slices each record against its successor -- model 1 -- and
then reverted it. Two things came out of writing it:
First, it broke three tests, which is the informative part. Those tests assert
"1,509 of 1,509 chunks have uniform record length" and they passed BECAUSE the
parser forced uniformity by slicing every record at a single stride. They were
asserting the parser's behaviour back at itself, and would have kept passing
forever regardless of what the disc contains.
Second, shipping it would have replaced one unverified model with another and
silently changed the record boundaries of 1,252 chunks on the strength of a
guess. The repo's own rule applies: a heuristic count is not an identification,
and here it is not even a count -- it is two equally good explanations of one
table.
WHAT WOULD SETTLE IT: a field inside the record whose end is independently
observable -- a terminator, a count implying a length, or a known value pinning
its position. Under model 2 the disputed bytes are padding and will be uniform or
zero; under model 1 they are data and will not be. For [75 42 ef] those 8 bytes
read a0 00 27 20 at both the tail of one slice and the head of the next, which is
arithmetic rather than evidence, since it is the same eight bytes seen twice.
The uniform model stays, the ambiguity is recorded in resolve()'s documentation
and here, and the 1,509-of-1,509 assertion stands as a statement about what the
parser does rather than about what the disc contains. 243 tests, clippy clean.
|
||
|
ReaperOfVeriod
|
c2af0528e3 |
Correct the chunk loader finding and decode a record field
The loader note said every record gains a 02 00 prefix in RAM. It does not: the word is the record's own leading field, already on the disc at every record. What the loader changes is the pointer -- it addresses the word, two bytes earlier than the disc's table -- and that pointer shift is what made the bytes look inserted. Measured, in order: - the two bytes before a payload are identical in core_data and in RAM, record for record: 02 00 on level 0's 90-record chunk at 0x100b302, 08 00 on the level's first chunk. Nothing is inserted; - the RAM table's low 24 bits are the payload's minus 2, consistently, so the record is [leading u16][two index bytes][payload] and the loaded table addresses the leading word. That is also why the index bytes read as +2 from the pointer -- the same +2 the earlier RAM result quotes; - the word is a field, not a pad: uniform across a chunk's records on 1,439 of 1,509 chunks (None on the 70 that vary, reported rather than averaged), fixed within all 71 chunk types, and only ever 1, 2, 4, 6, 8 or 10 -- 554 chunks carry 2, 520 carry 4. Two readings of the value were measured and disproved, so they are recorded as disproved rather than left as candidates: it is not an entry count (the payload divided by it leaves 57 distinct fractional "sizes", and 162 chunks do not divide at all), and it is not the number of distinct leading bytes (agrees on 0 of 1,506 chunks). Chunk gains `prefix: Option<u16>`, and the disc test asserts the uniformity, the per-type fixity and the value set. The scratch example that produced the per-type class-ID table is dropped now that the same scan is a committed test, which also lands here: moby class IDs sit in one body slot per chunk type, with the slot varying by type and the distinct >= 4 filter load-bearing -- without it the top slots are constants that read as 305x because any value below the moby count counts as a hit. |
||
|
ReaperOfVeriod
|
28680b4be3 |
Show the first two chunk-record bytes encode the record's own index
Continuing from the phase counter. Within a chunk: i : 0 1 2 3 4 5 6 7 8 9 10 11 ... 15 16 17 ... byte0: 0 16 32 48 64 80 96 112 128 144 160 176 ... 240 0 16 32 byte1: 0 0 0 0 0 0 0 0 0 0 0 0 ... 0 1 1 1 byte0 is step*(j mod g) and byte1 is (j / g), where g = 256/step and j = index + phase. The pair is the record's index in mixed radix: byte0 carries the low digits scaled by step, byte1 the quotient. Measured over all 27 levels: 1,235 of 1,509 chunks fit this on both bytes. Byte 0 alone fits 1,262, so the 27-chunk difference are chunks whose first byte follows the rule and whose second does not. On 1,235 chunks a record knows its own position and the two leading bytes hold no per-record information. ResolvedChunk::index_rule returns the (step, phase) only when the whole chunk fits on BOTH bytes, so the 274 that do not stay distinguishable and nothing skips a leading byte on a rule that does not apply to it. A number that moved when the implementation changed: a first probe reported 1,181; the committed method reports 1,235. The method is right. The probe accumulated ok0/ok1 flags across candidate phases with an early-exit that could skip a phase which would have matched both bytes, and it also disagreed with bucket_rule on the byte-0 count (1,208 against 1,262), which is what gave it away. Recorded because both numbers came from the same run and only the discrepancy exposed the probe as wrong. The lesson is narrow: when a one-off probe and a committed helper disagree, check the probe first. |
||
|
ReaperOfVeriod
|
79914c420e |
Show the chunk record's leading byte is a per-chunk phase counter; every chunk
has a uniform record length Follows the alignment result and settles what the leading byte is. Within one chunk the leading byte steps by a constant and wraps: offsets: 1420 1748 2072 2392 2716 3044 3372 3700 4028 4352 ... byte0: 0 16 32 48 64 80 96 112 128 144 ... Sweeping every (step, phase) over all 27 levels: fully explained: 1,262 of 1,509 chunks by step: 8 -> 51 chunks, 16 -> 554, 32 -> 520, 64 -> 137 unexplained: 247 chunks across 11 chunk types So on 1,262 chunks the leading byte carries no per-record information: it is a deterministic function of the record's index and the chunk's phase, which is a position marker rather than data. That is why the earlier "16 discrete values, uniform counts" pattern appeared at all. ResolvedChunk::bucket_rule returns the (step, phase) when the whole chunk fits and None otherwise, so the 247 unexplained chunks stay distinguishable and nothing skips a leading byte on the strength of a rule that does not apply to it. Every chunk has a uniform record length: 1,509 of 1,509. Length varies BETWEEN chunks of a type -- one type runs 76 to 104 bytes -- but never within one. This withdraws the previous commit's "variable-length type", which came from pooling minlen/maxlen across all of a type's records at once. That is the pooled-claim error rule 3a exists to prevent, and it is the third time this session a pooled measurement produced a claim that partitioning reversed. The type is not variable-length; the chunk is fixed-length and different chunks differ. |
||
|
ReaperOfVeriod
|
a9bf74fc10 |
Characterise all 77 chunk types: record byte 0 is a bucket index, not data
Characterised every chunk type -- 77 of them, 1,509 chunks, 34,491 records, all 27 levels. Byte 0 of every record is a small aligned index, not record content. It takes 16 discrete values per chunk type (0x00, 0x10, ... 0xf0) with near-uniform counts WITHIN each type. Uniform weighting is the tell: a running byte offset would weight the buckets unevenly, because records differ in size. type_id recs len byte0 0x007d7ec438 2430 328 0x00:162 0x10:162 0x20:162 ... 0xf0:135 0x00429ec43d 2079 368 0x00:162 0x10:135 0x20:135 ... 0xf0:108 0x00d76cc453 810 108 0x00:27 0x08:27 0x10:27 ... 0xe8:27 <- steps by 8 The alignment step is per type, not global. The leading byte is a multiple of 8 on 34,432 of 34,491 records but a multiple of 16 on only 33,349 -- the 1,142-record difference falls in seven types, four of which step by 8. 59 records in three types break even the 8-byte rule, and that is where the variable-length type lives: 0x00f2ef4275, records 76 to 104 bytes, no single stride to align to. What the bucket indexes is NOT named. It is reported as CHUNK_RECORD_ALIGNMENT and record_is_aligned rather than as a parsed field, because naming it would be a guess while the invariant is solid. The test pins the exact exception set -- 59 records across 3 types -- so a change there fails loudly. A measurement I initially got wrong: the first version of the constant claimed a 16-byte multiple on 34,432 records. 34,432 is the multiple-of-8 count; the multiple-of-16 count is 33,349. Right number, wrong label, and the test caught it. That is the second time this session a correct number carried a wrong label, after the tie four-value retraction. Both times the assertion did, which is the argument for putting these numbers in tests rather than only in prose. Still open: what the bucket indexes, and the remaining record fields. |
||
|
ReaperOfVeriod
|
ccea53807a |
Resolve the shrub region bound: the chunk container closes it
The shrub region's end was the last unlocated bound in core_data. It was not in the core index, and unlike moby->tie it had no successor boundary to chain against, so the last shrub class in every level ran out to the lump's end and the aggregates over it (the "3.5M records / 28MB of shrub body") were artifacts of that oversized span. Range measured: for each level, from the last shrub class's geometry offset to the constant trailer. Nothing outside that range is read. Method: scan forward from the last shrub class for the pair (a) the constant two bytes f6 7c sitting immediately in front of (b) a chunk signature -- a 16-byte header ending 0xff, ten zero bytes, then four or more 0xf0-tagged offsets. The two halves are each weak alone, and the halves are measured separately: the marker pair occurs 25 to 103 times in a 5 to 17 MB lump, so matching it alone picks dozens of wrong offsets; the chunk shape matches random data often enough to need four tagged offsets before it is believed. Result: on all 27 levels the marker sits exactly two bytes before the first chunk header, and the two parts of the signature are never found apart. No chunk signature occurs inside any last shrub body, on any level. The three levels without shrub (10, 25, 26) carry the same marker before the container that closes tie instead, which is what makes the locator a property of the region rather than a shrub rule. Consequences: - All 358 shrub class bodies are bounded and are whole 8-byte records. That was not assertable while the last class in each level ran to the lump's end, and it is the check the boundary earns its keep for. - 9,600 records in total, against the 8,938 previously measurable in successor-bounded spans; 662 of them sit in the 24 last-class bodies. The shrub region totals 2,033,776 bytes across the 24 levels that have one. - The chunk container is 25,775,850 bytes across 27 levels (1,236 chunks, 22,562 records). The container measurements were previously taken over "the 900,000 bytes before the trailer", a window that did not align to its real extent: on level 0 it is 690,302 bytes, and 27 of 1,123 chunks were truncated at that invented edge. chunk_containers_resolve_on_every_level now reads the region from core_layout instead of measuring backwards from the end of the lump. core_layout takes core_data now: the container's start is the one bound that cannot be derived from the index. It gains the chunks region, so every byte of level-specific core_data belongs to a named region. Where the signature is not found the end stays None -- the region is left visible rather than clamped to the tail. Still open: the marker's meaning (constant across 27 levels, at exactly this position, unnamed) and the shrub body payload's encoding. |
||
|
ReaperOfVeriod
|
10188416e2 |
Decode the chunk count field: header byte 14 is the record count
Range measured: all 27 levels, 1123 chunks, over the 900000 bytes before the constant trailer. Method: compare header byte 14 against the number of offsets actually resolved for that chunk. Result: byte 14 equals the record count on 1123 of 1123 chunks, zero mismatches. This is the first genuinely decoded field in this container rather than a measured correlation. The parser now truncates the offset table to header byte 14, so a stray tagged word in a chunk's record data cannot silently extend the table and put the count at odds with its own header. Two more header bytes are near-constant and read as ASCII: byte 13 takes only 0x45 (1076 chunks) and 0x46 (47); byte 9 takes 0x44 (26), 0x45 (1084) and 0x46 (13). Those are D, E and F -- a three-letter tag at two byte positions with different distributions. Recorded as suggestive, not established: the other 14 header bytes are high-entropy and read as ASCII at no shift tried, so the D/E/F may be incidental. Worth testing against a known type table. Still open: the 14 opaque header bytes, whether bytes 9 and 13 are type tags, which record fields are index/length/offset, and whether a chunk is moby_anim. |
||
|
ReaperOfVeriod
|
681333535a |
Add a bounds-checked parser for the post-shrub chunk container
gc_formats::chunks parses the container that fills the region after the last shrub class body and before the constant trailer: +0x00 16 bytes, last byte 0xff (opaque, differs per chunk) +0x10 10 zero bytes +0x1a N tagged offsets, top byte 0xf0 Each tagged offset is 0xF0000000 | low24, and a record is read at chunk_start + low. Resolving that way gives records sharing a field layout, which is what distinguishes the container from a coincidental byte pattern. Fields are not named. The 16-byte header is opaque, the 0xf0 tag is unexplained, and it is not established that a chunk is moby_anim -- a 49 x 368-byte table inside this region is byte-identical on all 27 levels, which is evidence against that identification. What the parser provides is chunk and record extraction, which is what the analysis established. The gated test measures, over all 27 levels: 1123 chunks, 19546 records, 50 distinct strides, 12.4% of byte positions constant across records (per-chunk range 2.8% to 75%). Chance gives a constant position probability of 1/256^n. Three things the test got wrong first, recorded because they are the traps: - A per-chunk threshold of 20% constant positions. Individual chunks run from 2.8% to 75%; the aggregate is the invariant, not the chunk. Asserting 20% per chunk was asserting a property the format does not have. - Requiring every offset to resolve. 27 chunks truncate at the window edge, because the 900000-byte window does not align to chunk boundaries. Counted now rather than asserted away. - read_u32 in the chunk signature check was big-endian while Buf::u32 is little-endian. The tag lives in the high byte of a little-endian word, so the mismatch made every level report zero chunks while parse_chunk -- which goes through Buf -- was correct. Worth remembering that the two paths can disagree on a format this young. Tests: 190 -> 191. |