Show the chunk record's leading byte is a per-chunk phase counter; every chunk

has a uniform record length

Follows the alignment result and settles what the leading byte is.

Within one chunk the leading byte steps by a constant and wraps:

  offsets: 1420 1748 2072 2392 2716 3044 3372 3700 4028 4352 ...
  byte0:      0  16   32   48   64   80   96  112  128  144 ...

Sweeping every (step, phase) over all 27 levels:

  fully explained: 1,262 of 1,509 chunks
  by step: 8 -> 51 chunks, 16 -> 554, 32 -> 520, 64 -> 137
  unexplained:  247 chunks across 11 chunk types

So on 1,262 chunks the leading byte carries no per-record information: it is a
deterministic function of the record's index and the chunk's phase, which is a
position marker rather than data. That is why the earlier "16 discrete values,
uniform counts" pattern appeared at all.

ResolvedChunk::bucket_rule returns the (step, phase) when the whole chunk fits and
None otherwise, so the 247 unexplained chunks stay distinguishable and nothing
skips a leading byte on the strength of a rule that does not apply to it.

Every chunk has a uniform record length: 1,509 of 1,509. Length varies BETWEEN
chunks of a type -- one type runs 76 to 104 bytes -- but never within one. This
withdraws the previous commit's "variable-length type", which came from pooling
minlen/maxlen across all of a type's records at once. That is the pooled-claim
error rule 3a exists to prevent, and it is the third time this session a pooled
measurement produced a claim that partitioning reversed. The type is not
variable-length; the chunk is fixed-length and different chunks differ.
This commit is contained in:
2026-10-12 04:21:32 +02:00
parent a9bf74fc10
commit 79914c420e
3 changed files with 117 additions and 2 deletions
+22 -2
View File
@@ -2920,6 +2920,25 @@ fn chunk_records_start_16_byte_aligned() {
assert!(chunks >= 1500, "only {chunks} chunks parsed");
assert!(records >= 34000, "only {records} chunk records");
// Every chunk has a uniform record length -- length varies between chunks
// of a type, never within one -- and on most chunks the leading byte is a
// per-chunk phase counter rather than data.
let mut uniform = 0usize;
let mut ruled = 0usize;
for n in 0..27 {
let core = std::fs::read(data.join(format!("level{n}/core_data.bin"))).expect("core_data");
for c in parse_all_chunks(&core).expect("chunks") {
if c.has_uniform_record_length() {
uniform += 1;
}
if c.bucket_rule().is_some() {
ruled += 1;
}
}
}
assert_eq!(uniform, 1509, "uniform-record-length count moved off 1509");
assert_eq!(ruled, 1262, "bucket-rule count moved off 1262");
// Not universal. Three chunk types break even the 8-byte rule, all of them
// the variable-length or 8-stepped ones. A parallel probe puts 1,142
// records across seven types outside a *16*-byte rule, which is how the
@@ -2942,8 +2961,9 @@ fn chunk_records_start_16_byte_aligned() {
);
eprintln!(
"chunks: {chunks} chunks, {records} records; leading byte a multiple of 8 on {aligned}, \
{bad} exceptions across {} types (8-byte step and one variable-length type)",
"chunks: {chunks} chunks, {records} records; uniform record length on {uniform}; \
leading byte a multiple of 8 on {aligned}, {bad} exceptions across {} types; \
leading byte fully explained by a per-chunk phase counter on {ruled}",
unaligned.len()
);
}
+59
View File
@@ -122,6 +122,65 @@ pub struct ResolvedChunk<'a> {
pub records: Vec<&'a [u8]>,
}
impl<'a> ResolvedChunk<'a> {
/// The per-chunk phase and step that explain every record's leading byte, or
/// `None` when no `(step, phase)` in that form fits.
///
/// **Measured: 1,262 of 1,509 chunks** across all 27 levels are fully explained
/// by `record[i].leading == step * ((i + phase) mod (256 / step))`, with
/// `step` taking 8 on 51 chunks, 16 on 554, 32 on 520 and 64 on 137. The other
/// 247 chunks, across 11 chunk types, are not explained by it.
///
/// Where this returns `Some`, the leading byte carries **no per-record
/// information**: it is a deterministic function of the record's index and the
/// chunk's phase, so it is a position marker rather than data. The 247 chunks
/// where it returns `None` are the ones where that has *not* been shown, and
/// nothing should skip their leading byte on the strength of this.
pub fn bucket_rule(&self) -> Option<(u8, usize)> {
use std::collections::BTreeSet;
if self.records.is_empty() {
return None;
}
let n = self.records.len();
for step in [8u16, 16, 32, 64] {
let m = (256 / step) as usize;
for phase in 0..m {
let mut ok = true;
for (i, r) in self.records.iter().enumerate() {
let Some(&b) = r.first() else {
ok = false;
break;
};
if b != ((((i + phase) % m) * step as usize % 256) as u8) {
ok = false;
break;
}
}
if ok {
return Some((step as u8, phase));
}
}
}
let _ = n;
let _: Option<BTreeSet<u8>> = None;
None
}
/// True when every record in this chunk has the same length.
///
/// **Measured: true for 1,509 of 1,509 chunks** across all 27 levels. Record
/// length varies *between* chunks of the same type -- one type runs 76 to 104
/// bytes -- but never within one. An earlier claim that some type had
/// variable-length records was pooling across chunks; it is withdrawn.
pub fn has_uniform_record_length(&self) -> bool {
let mut it = self.records.iter().filter(|r| !r.is_empty());
let Some(first) = it.next().map(|r| r.len()) else {
return true;
};
it.all(|r| r.len() == first)
}
}
/// True if `offset` looks like the start of a chunk.
///
/// The signature is the header's last byte being `0xff`, the 10-byte gap
+36
View File
@@ -1316,3 +1316,39 @@ Worth stating plainly: this is the second time this session a correct number
carried a wrong label, after the tie four-value retraction. Both times the test
was the thing that caught it, which is the argument for asserting the number in
a test at all rather than only in prose.
## The chunk record's leading byte is a per-chunk phase counter
Follow-up to the alignment result, and it settles what the leading byte is.
Within one chunk the leading byte steps by a constant and wraps:
offsets: 1420 1748 2072 2392 2716 3044 3372 3700 4028 4352 ...
byte0: 0 16 32 48 64 80 96 112 128 144 ...
i.e. `byte0 == step * ((index + phase) mod (256 / step))`. Sweeping every
`(step, phase)` over all 27 levels:
fully explained: 1,262 of 1,509 chunks
by step: 8 -> 51 chunks, 16 -> 554, 32 -> 520, 64 -> 137
unexplained: 247 chunks across 11 chunk types
**So on 1,262 chunks the leading byte carries no per-record information.** It is
a deterministic function of the record's index and the chunk's phase. That makes
it a position marker, not data -- which is why the earlier "16 discrete values
with uniform counts" pattern showed up.
`ResolvedChunk::bucket_rule` returns the `(step, phase)` when the whole chunk
fits and `None` otherwise, so the 247 unexplained chunks are distinguishable
from the 1,262 explained ones and nothing skips a leading byte on the strength of
a rule that does not apply.
### Every chunk has a uniform record length
**1,509 of 1,509 chunks.** Length varies between chunks of the same type -- one
type runs 76 to 104 bytes -- but never within one chunk.
This **withdraws the previous commit's "variable-length type"**. That came from
taking `minlen`/`maxlen` over all of a type's records at once, pooling across
chunks; rule 3a's pooled claim, third time this session. Within any one chunk the
length is fixed.