Show the chunk record's leading byte is a per-chunk phase counter; every chunk
has a uniform record length Follows the alignment result and settles what the leading byte is. Within one chunk the leading byte steps by a constant and wraps: offsets: 1420 1748 2072 2392 2716 3044 3372 3700 4028 4352 ... byte0: 0 16 32 48 64 80 96 112 128 144 ... Sweeping every (step, phase) over all 27 levels: fully explained: 1,262 of 1,509 chunks by step: 8 -> 51 chunks, 16 -> 554, 32 -> 520, 64 -> 137 unexplained: 247 chunks across 11 chunk types So on 1,262 chunks the leading byte carries no per-record information: it is a deterministic function of the record's index and the chunk's phase, which is a position marker rather than data. That is why the earlier "16 discrete values, uniform counts" pattern appeared at all. ResolvedChunk::bucket_rule returns the (step, phase) when the whole chunk fits and None otherwise, so the 247 unexplained chunks stay distinguishable and nothing skips a leading byte on the strength of a rule that does not apply to it. Every chunk has a uniform record length: 1,509 of 1,509. Length varies BETWEEN chunks of a type -- one type runs 76 to 104 bytes -- but never within one. This withdraws the previous commit's "variable-length type", which came from pooling minlen/maxlen across all of a type's records at once. That is the pooled-claim error rule 3a exists to prevent, and it is the third time this session a pooled measurement produced a claim that partitioning reversed. The type is not variable-length; the chunk is fixed-length and different chunks differ.
This commit is contained in:
@@ -2920,6 +2920,25 @@ fn chunk_records_start_16_byte_aligned() {
|
|||||||
|
|
||||||
assert!(chunks >= 1500, "only {chunks} chunks parsed");
|
assert!(chunks >= 1500, "only {chunks} chunks parsed");
|
||||||
assert!(records >= 34000, "only {records} chunk records");
|
assert!(records >= 34000, "only {records} chunk records");
|
||||||
|
// Every chunk has a uniform record length -- length varies between chunks
|
||||||
|
// of a type, never within one -- and on most chunks the leading byte is a
|
||||||
|
// per-chunk phase counter rather than data.
|
||||||
|
let mut uniform = 0usize;
|
||||||
|
let mut ruled = 0usize;
|
||||||
|
for n in 0..27 {
|
||||||
|
let core = std::fs::read(data.join(format!("level{n}/core_data.bin"))).expect("core_data");
|
||||||
|
for c in parse_all_chunks(&core).expect("chunks") {
|
||||||
|
if c.has_uniform_record_length() {
|
||||||
|
uniform += 1;
|
||||||
|
}
|
||||||
|
if c.bucket_rule().is_some() {
|
||||||
|
ruled += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
assert_eq!(uniform, 1509, "uniform-record-length count moved off 1509");
|
||||||
|
assert_eq!(ruled, 1262, "bucket-rule count moved off 1262");
|
||||||
|
|
||||||
// Not universal. Three chunk types break even the 8-byte rule, all of them
|
// Not universal. Three chunk types break even the 8-byte rule, all of them
|
||||||
// the variable-length or 8-stepped ones. A parallel probe puts 1,142
|
// the variable-length or 8-stepped ones. A parallel probe puts 1,142
|
||||||
// records across seven types outside a *16*-byte rule, which is how the
|
// records across seven types outside a *16*-byte rule, which is how the
|
||||||
@@ -2942,8 +2961,9 @@ fn chunk_records_start_16_byte_aligned() {
|
|||||||
);
|
);
|
||||||
|
|
||||||
eprintln!(
|
eprintln!(
|
||||||
"chunks: {chunks} chunks, {records} records; leading byte a multiple of 8 on {aligned}, \
|
"chunks: {chunks} chunks, {records} records; uniform record length on {uniform}; \
|
||||||
{bad} exceptions across {} types (8-byte step and one variable-length type)",
|
leading byte a multiple of 8 on {aligned}, {bad} exceptions across {} types; \
|
||||||
|
leading byte fully explained by a per-chunk phase counter on {ruled}",
|
||||||
unaligned.len()
|
unaligned.len()
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -122,6 +122,65 @@ pub struct ResolvedChunk<'a> {
|
|||||||
pub records: Vec<&'a [u8]>,
|
pub records: Vec<&'a [u8]>,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
impl<'a> ResolvedChunk<'a> {
|
||||||
|
/// The per-chunk phase and step that explain every record's leading byte, or
|
||||||
|
/// `None` when no `(step, phase)` in that form fits.
|
||||||
|
///
|
||||||
|
/// **Measured: 1,262 of 1,509 chunks** across all 27 levels are fully explained
|
||||||
|
/// by `record[i].leading == step * ((i + phase) mod (256 / step))`, with
|
||||||
|
/// `step` taking 8 on 51 chunks, 16 on 554, 32 on 520 and 64 on 137. The other
|
||||||
|
/// 247 chunks, across 11 chunk types, are not explained by it.
|
||||||
|
///
|
||||||
|
/// Where this returns `Some`, the leading byte carries **no per-record
|
||||||
|
/// information**: it is a deterministic function of the record's index and the
|
||||||
|
/// chunk's phase, so it is a position marker rather than data. The 247 chunks
|
||||||
|
/// where it returns `None` are the ones where that has *not* been shown, and
|
||||||
|
/// nothing should skip their leading byte on the strength of this.
|
||||||
|
pub fn bucket_rule(&self) -> Option<(u8, usize)> {
|
||||||
|
use std::collections::BTreeSet;
|
||||||
|
if self.records.is_empty() {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
let n = self.records.len();
|
||||||
|
for step in [8u16, 16, 32, 64] {
|
||||||
|
let m = (256 / step) as usize;
|
||||||
|
for phase in 0..m {
|
||||||
|
let mut ok = true;
|
||||||
|
for (i, r) in self.records.iter().enumerate() {
|
||||||
|
let Some(&b) = r.first() else {
|
||||||
|
ok = false;
|
||||||
|
break;
|
||||||
|
};
|
||||||
|
if b != ((((i + phase) % m) * step as usize % 256) as u8) {
|
||||||
|
ok = false;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if ok {
|
||||||
|
return Some((step as u8, phase));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let _ = n;
|
||||||
|
let _: Option<BTreeSet<u8>> = None;
|
||||||
|
None
|
||||||
|
}
|
||||||
|
|
||||||
|
/// True when every record in this chunk has the same length.
|
||||||
|
///
|
||||||
|
/// **Measured: true for 1,509 of 1,509 chunks** across all 27 levels. Record
|
||||||
|
/// length varies *between* chunks of the same type -- one type runs 76 to 104
|
||||||
|
/// bytes -- but never within one. An earlier claim that some type had
|
||||||
|
/// variable-length records was pooling across chunks; it is withdrawn.
|
||||||
|
pub fn has_uniform_record_length(&self) -> bool {
|
||||||
|
let mut it = self.records.iter().filter(|r| !r.is_empty());
|
||||||
|
let Some(first) = it.next().map(|r| r.len()) else {
|
||||||
|
return true;
|
||||||
|
};
|
||||||
|
it.all(|r| r.len() == first)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// True if `offset` looks like the start of a chunk.
|
/// True if `offset` looks like the start of a chunk.
|
||||||
///
|
///
|
||||||
/// The signature is the header's last byte being `0xff`, the 10-byte gap
|
/// The signature is the header's last byte being `0xff`, the 10-byte gap
|
||||||
|
|||||||
@@ -1316,3 +1316,39 @@ Worth stating plainly: this is the second time this session a correct number
|
|||||||
carried a wrong label, after the tie four-value retraction. Both times the test
|
carried a wrong label, after the tie four-value retraction. Both times the test
|
||||||
was the thing that caught it, which is the argument for asserting the number in
|
was the thing that caught it, which is the argument for asserting the number in
|
||||||
a test at all rather than only in prose.
|
a test at all rather than only in prose.
|
||||||
|
|
||||||
|
## The chunk record's leading byte is a per-chunk phase counter
|
||||||
|
|
||||||
|
Follow-up to the alignment result, and it settles what the leading byte is.
|
||||||
|
|
||||||
|
Within one chunk the leading byte steps by a constant and wraps:
|
||||||
|
|
||||||
|
offsets: 1420 1748 2072 2392 2716 3044 3372 3700 4028 4352 ...
|
||||||
|
byte0: 0 16 32 48 64 80 96 112 128 144 ...
|
||||||
|
|
||||||
|
i.e. `byte0 == step * ((index + phase) mod (256 / step))`. Sweeping every
|
||||||
|
`(step, phase)` over all 27 levels:
|
||||||
|
|
||||||
|
fully explained: 1,262 of 1,509 chunks
|
||||||
|
by step: 8 -> 51 chunks, 16 -> 554, 32 -> 520, 64 -> 137
|
||||||
|
unexplained: 247 chunks across 11 chunk types
|
||||||
|
|
||||||
|
**So on 1,262 chunks the leading byte carries no per-record information.** It is
|
||||||
|
a deterministic function of the record's index and the chunk's phase. That makes
|
||||||
|
it a position marker, not data -- which is why the earlier "16 discrete values
|
||||||
|
with uniform counts" pattern showed up.
|
||||||
|
|
||||||
|
`ResolvedChunk::bucket_rule` returns the `(step, phase)` when the whole chunk
|
||||||
|
fits and `None` otherwise, so the 247 unexplained chunks are distinguishable
|
||||||
|
from the 1,262 explained ones and nothing skips a leading byte on the strength of
|
||||||
|
a rule that does not apply.
|
||||||
|
|
||||||
|
### Every chunk has a uniform record length
|
||||||
|
|
||||||
|
**1,509 of 1,509 chunks.** Length varies between chunks of the same type -- one
|
||||||
|
type runs 76 to 104 bytes -- but never within one chunk.
|
||||||
|
|
||||||
|
This **withdraws the previous commit's "variable-length type"**. That came from
|
||||||
|
taking `minlen`/`maxlen` over all of a type's records at once, pooling across
|
||||||
|
chunks; rule 3a's pooled claim, third time this session. Within any one chunk the
|
||||||
|
length is fixed.
|
||||||
|
|||||||
Reference in New Issue
Block a user