Decode the chunk count field: header byte 14 is the record count

Range measured: all 27 levels, 1123 chunks, over the 900000 bytes before the
constant trailer.

Method: compare header byte 14 against the number of offsets actually resolved
for that chunk.

Result: byte 14 equals the record count on 1123 of 1123 chunks, zero
mismatches. This is the first genuinely decoded field in this container rather
than a measured correlation.

The parser now truncates the offset table to header byte 14, so a stray
tagged word in a chunk's record data cannot silently extend the table and put
the count at odds with its own header.

Two more header bytes are near-constant and read as ASCII: byte 13 takes only
0x45 (1076 chunks) and 0x46 (47); byte 9 takes 0x44 (26), 0x45 (1084) and
0x46 (13). Those are D, E and F -- a three-letter tag at two byte positions
with different distributions. Recorded as suggestive, not established: the
other 14 header bytes are high-entropy and read as ASCII at no shift tried, so
the D/E/F may be incidental. Worth testing against a known type table.

Still open: the 14 opaque header bytes, whether bytes 9 and 13 are type tags,
which record fields are index/length/offset, and whether a chunk is moby_anim.
This commit is contained in:
2026-10-11 04:26:44 +02:00
parent 681333535a
commit 10188416e2
3 changed files with 55 additions and 0 deletions
+9
View File
@@ -1956,6 +1956,15 @@ fn chunk_containers_resolve_on_every_level() {
chunk.chunk.offset chunk.chunk.offset
); );
assert!(chunk.chunk.stride > 0, "level{n}: zero stride"); assert!(chunk.chunk.stride > 0, "level{n}: zero stride");
// The one decoded header field.
assert_eq!(
chunk.chunk.count as usize,
chunk.chunk.offsets.len(),
"level{n}: chunk at {} header count {} but {} offsets",
chunk.chunk.offset,
chunk.chunk.count,
chunk.chunk.offsets.len()
);
chunks += 1; chunks += 1;
records += chunk.records.len(); records += chunk.records.len();
strides.insert(chunk.chunk.stride); strides.insert(chunk.chunk.stride);
+13
View File
@@ -55,6 +55,12 @@ pub struct Chunk {
pub header: [u8; CHUNK_HEADER_BYTES], pub header: [u8; CHUNK_HEADER_BYTES],
/// Offset of this chunk within the region it was parsed from. /// Offset of this chunk within the region it was parsed from.
pub offset: usize, pub offset: usize,
/// Record count, from header byte 14.
///
/// Measured: equals `offsets.len()` on **1,123 of 1,123** chunks across
/// all 27 levels, zero mismatches. This is the one header field that is
/// decoded rather than merely measured.
pub count: u8,
/// The low 24 bits of each tagged offset, in file order. /// The low 24 bits of each tagged offset, in file order.
pub offsets: Vec<u32>, pub offsets: Vec<u32>,
/// Record stride, from the first two offsets. Zero if fewer than two. /// Record stride, from the first two offsets. Zero if fewer than two.
@@ -124,6 +130,11 @@ pub fn parse_chunk(region: &[u8], offset: usize) -> Result<Option<Chunk>> {
offsets.push(v & 0x00ff_ffff); offsets.push(v & 0x00ff_ffff);
} }
// The offset table runs to the first untagged word, but the header carries
// its own count. Truncating to that count stops a stray tagged word in the
// record data from extending the table.
offsets.truncate(header[14] as usize);
let stride = if offsets.len() >= 2 { let stride = if offsets.len() >= 2 {
(offsets[1] - offsets[0]) as usize (offsets[1] - offsets[0]) as usize
} else { } else {
@@ -131,6 +142,7 @@ pub fn parse_chunk(region: &[u8], offset: usize) -> Result<Option<Chunk>> {
}; };
Ok(Some(Chunk { Ok(Some(Chunk {
count: header[14],
header, header,
offset, offset,
offsets, offsets,
@@ -163,6 +175,7 @@ pub fn resolve<'r>(chunk: &Chunk, region: &'r [u8]) -> Result<ResolvedChunk<'r>>
} }
Ok(ResolvedChunk { Ok(ResolvedChunk {
chunk: Chunk { chunk: Chunk {
count: chunk.count,
header: chunk.header, header: chunk.header,
offset: chunk.offset, offset: chunk.offset,
offsets: chunk.offsets.clone(), offsets: chunk.offsets.clone(),
+33
View File
@@ -799,3 +799,36 @@ the `0xF0` tag, which fields are index/length/offset, and whether a chunk is
moby_anim or something else. A parser can extract chunks and records moby_anim or something else. A parser can extract chunks and records
bounds-checked today; naming fields needs a domain argument this project has bounds-checked today; naming fields needs a domain argument this project has
not yet found. not yet found.
## The chunk header: byte 14 is the record count
**Range measured:** all 27 levels, 1,123 chunks, over the 900,000 bytes
before the constant trailer.
**Method:** compare header byte 14 against the number of offsets actually
resolved for that chunk.
**Result: byte 14 equals the record count on 1,123 of 1,123 chunks, zero
mismatches.** This is the first genuinely *decoded* field in this container
rather than a measured correlation.
The parser now truncates the offset table to header byte 14, so a stray
tagged word appearing in a chunk's record data cannot silently extend the
table -- which would otherwise let the count disagree with its own header.
**Two more header bytes are near-constant and look like ASCII:**
- byte 13: only two values across 1,123 chunks -- `0x45` (1,076) and `0x46`
(47).
- byte 9: three values -- `0x44` (26), `0x45` (1,084), `0x46` (13).
`0x44`, `0x45` and `0x46` are ASCII `D`, `E` and `F`. That three-letter tag
appearing at two different byte positions with different distributions is
suggestive of a type or class field, but it is not established: the remaining
14 header bytes are high-entropy and do not read as ASCII at any shift tried,
so the ASCII in bytes 9 and 13 could be incidental.
**Still open:** the 14 opaque header bytes, whether bytes 9 and 13 are type
tags, which record fields are index/length/offset, and whether a chunk is
moby_anim. The `D`/`E`/`F` values are suggestive enough to be worth testing
against a known type table, but that has not been done.