Characterise all 77 chunk types: record byte 0 is a bucket index, not data

Characterised every chunk type -- 77 of them, 1,509 chunks, 34,491 records, all 27
levels.

Byte 0 of every record is a small aligned index, not record content. It takes 16
discrete values per chunk type (0x00, 0x10, ... 0xf0) with near-uniform counts
WITHIN each type. Uniform weighting is the tell: a running byte offset would
weight the buckets unevenly, because records differ in size.

  type_id      recs  len  byte0
  0x007d7ec438 2430  328  0x00:162 0x10:162 0x20:162 ... 0xf0:135
  0x00429ec43d 2079  368  0x00:162 0x10:135 0x20:135 ... 0xf0:108
  0x00d76cc453  810  108  0x00:27  0x08:27  0x10:27  ... 0xe8:27   <- steps by 8

The alignment step is per type, not global. The leading byte is a multiple of 8 on
34,432 of 34,491 records but a multiple of 16 on only 33,349 -- the 1,142-record
difference falls in seven types, four of which step by 8. 59 records in three types
break even the 8-byte rule, and that is where the variable-length type lives:
0x00f2ef4275, records 76 to 104 bytes, no single stride to align to.

What the bucket indexes is NOT named. It is reported as CHUNK_RECORD_ALIGNMENT and
record_is_aligned rather than as a parsed field, because naming it would be a guess
while the invariant is solid. The test pins the exact exception set -- 59 records
across 3 types -- so a change there fails loudly.

A measurement I initially got wrong: the first version of the constant claimed a
16-byte multiple on 34,432 records. 34,432 is the multiple-of-8 count; the
multiple-of-16 count is 33,349. Right number, wrong label, and the test caught it.
That is the second time this session a correct number carried a wrong label, after
the tie four-value retraction. Both times the assertion did, which is the argument
for putting these numbers in tests rather than only in prose.

Still open: what the bucket indexes, and the remaining record fields.
This commit is contained in:
2026-10-12 04:18:55 +02:00
parent 291153a0d9
commit a9bf74fc10
3 changed files with 161 additions and 0 deletions
+33
View File
@@ -55,6 +55,39 @@ pub const CHUNK_TAG: u8 = 0xf0;
/// Not interpreted. Whether it is a terminator, an end count or a tag is not
/// established; what is established is that it is constant across all 27
/// levels and sits at this one place.
/// Alignment of the first byte of a chunk record.
///
/// The leading byte is not decoded -- what it indexes is not established, and
/// naming it would be a guess. What *is* established is that it is a small
/// bucket index, not data: within a chunk type it takes 16 discrete values
/// (0x00, 0x10, ... 0xf0) with near-uniform counts, so it reads as an index into
/// groups rather than a running offset.
///
/// **The alignment step is per type, not global.** Across all 27 levels, over
/// 1,509 chunks and 34,491 records:
///
/// - the leading byte is a multiple of **8** on 34,432 of 34,491;
/// - a multiple of **16** on only 33,349 -- the 1,142-record difference falls in
/// **seven** types, four of which step their buckets by 8 (0x00, 0x08, 0x10,
/// ...) instead of 16;
/// - and 59 records in three types break even the 8-byte rule, which is where the
/// variable-length type lives (0x00f2ef4275, records 76 to 104 bytes).
///
/// So [`record_is_aligned`] is checked at 8, not 16, and a type that steps by 16
/// is not distinguishable from one that steps by 8 without looking at the type.
pub const CHUNK_RECORD_ALIGNMENT: usize = 8;
/// True when a record's leading byte is a multiple of [`CHUNK_RECORD_ALIGNMENT`].
///
/// Measured over all 27 levels: 34,432 of 34,491 records, with the exceptions
/// confined to the seven types above.
pub fn record_is_aligned(record: &[u8]) -> bool {
match record.first() {
Some(&b) => (b as usize) % CHUNK_RECORD_ALIGNMENT == 0,
None => false,
}
}
pub const CHUNK_REGION_MARKER: [u8; 2] = [0xf6, 0x7c];
/// A chunk header plus the records its offsets address.