Decode the chunk count field: header byte 14 is the record count
Range measured: all 27 levels, 1123 chunks, over the 900000 bytes before the constant trailer. Method: compare header byte 14 against the number of offsets actually resolved for that chunk. Result: byte 14 equals the record count on 1123 of 1123 chunks, zero mismatches. This is the first genuinely decoded field in this container rather than a measured correlation. The parser now truncates the offset table to header byte 14, so a stray tagged word in a chunk's record data cannot silently extend the table and put the count at odds with its own header. Two more header bytes are near-constant and read as ASCII: byte 13 takes only 0x45 (1076 chunks) and 0x46 (47); byte 9 takes 0x44 (26), 0x45 (1084) and 0x46 (13). Those are D, E and F -- a three-letter tag at two byte positions with different distributions. Recorded as suggestive, not established: the other 14 header bytes are high-entropy and read as ASCII at no shift tried, so the D/E/F may be incidental. Worth testing against a known type table. Still open: the 14 opaque header bytes, whether bytes 9 and 13 are type tags, which record fields are index/length/offset, and whether a chunk is moby_anim.
This commit is contained in:
@@ -1956,6 +1956,15 @@ fn chunk_containers_resolve_on_every_level() {
|
||||
chunk.chunk.offset
|
||||
);
|
||||
assert!(chunk.chunk.stride > 0, "level{n}: zero stride");
|
||||
// The one decoded header field.
|
||||
assert_eq!(
|
||||
chunk.chunk.count as usize,
|
||||
chunk.chunk.offsets.len(),
|
||||
"level{n}: chunk at {} header count {} but {} offsets",
|
||||
chunk.chunk.offset,
|
||||
chunk.chunk.count,
|
||||
chunk.chunk.offsets.len()
|
||||
);
|
||||
chunks += 1;
|
||||
records += chunk.records.len();
|
||||
strides.insert(chunk.chunk.stride);
|
||||
|
||||
@@ -55,6 +55,12 @@ pub struct Chunk {
|
||||
pub header: [u8; CHUNK_HEADER_BYTES],
|
||||
/// Offset of this chunk within the region it was parsed from.
|
||||
pub offset: usize,
|
||||
/// Record count, from header byte 14.
|
||||
///
|
||||
/// Measured: equals `offsets.len()` on **1,123 of 1,123** chunks across
|
||||
/// all 27 levels, zero mismatches. This is the one header field that is
|
||||
/// decoded rather than merely measured.
|
||||
pub count: u8,
|
||||
/// The low 24 bits of each tagged offset, in file order.
|
||||
pub offsets: Vec<u32>,
|
||||
/// Record stride, from the first two offsets. Zero if fewer than two.
|
||||
@@ -124,6 +130,11 @@ pub fn parse_chunk(region: &[u8], offset: usize) -> Result<Option<Chunk>> {
|
||||
offsets.push(v & 0x00ff_ffff);
|
||||
}
|
||||
|
||||
// The offset table runs to the first untagged word, but the header carries
|
||||
// its own count. Truncating to that count stops a stray tagged word in the
|
||||
// record data from extending the table.
|
||||
offsets.truncate(header[14] as usize);
|
||||
|
||||
let stride = if offsets.len() >= 2 {
|
||||
(offsets[1] - offsets[0]) as usize
|
||||
} else {
|
||||
@@ -131,6 +142,7 @@ pub fn parse_chunk(region: &[u8], offset: usize) -> Result<Option<Chunk>> {
|
||||
};
|
||||
|
||||
Ok(Some(Chunk {
|
||||
count: header[14],
|
||||
header,
|
||||
offset,
|
||||
offsets,
|
||||
@@ -163,6 +175,7 @@ pub fn resolve<'r>(chunk: &Chunk, region: &'r [u8]) -> Result<ResolvedChunk<'r>>
|
||||
}
|
||||
Ok(ResolvedChunk {
|
||||
chunk: Chunk {
|
||||
count: chunk.count,
|
||||
header: chunk.header,
|
||||
offset: chunk.offset,
|
||||
offsets: chunk.offsets.clone(),
|
||||
|
||||
@@ -799,3 +799,36 @@ the `0xF0` tag, which fields are index/length/offset, and whether a chunk is
|
||||
moby_anim or something else. A parser can extract chunks and records
|
||||
bounds-checked today; naming fields needs a domain argument this project has
|
||||
not yet found.
|
||||
|
||||
## The chunk header: byte 14 is the record count
|
||||
|
||||
**Range measured:** all 27 levels, 1,123 chunks, over the 900,000 bytes
|
||||
before the constant trailer.
|
||||
|
||||
**Method:** compare header byte 14 against the number of offsets actually
|
||||
resolved for that chunk.
|
||||
|
||||
**Result: byte 14 equals the record count on 1,123 of 1,123 chunks, zero
|
||||
mismatches.** This is the first genuinely *decoded* field in this container
|
||||
rather than a measured correlation.
|
||||
|
||||
The parser now truncates the offset table to header byte 14, so a stray
|
||||
tagged word appearing in a chunk's record data cannot silently extend the
|
||||
table -- which would otherwise let the count disagree with its own header.
|
||||
|
||||
**Two more header bytes are near-constant and look like ASCII:**
|
||||
|
||||
- byte 13: only two values across 1,123 chunks -- `0x45` (1,076) and `0x46`
|
||||
(47).
|
||||
- byte 9: three values -- `0x44` (26), `0x45` (1,084), `0x46` (13).
|
||||
|
||||
`0x44`, `0x45` and `0x46` are ASCII `D`, `E` and `F`. That three-letter tag
|
||||
appearing at two different byte positions with different distributions is
|
||||
suggestive of a type or class field, but it is not established: the remaining
|
||||
14 header bytes are high-entropy and do not read as ASCII at any shift tried,
|
||||
so the ASCII in bytes 9 and 13 could be incidental.
|
||||
|
||||
**Still open:** the 14 opaque header bytes, whether bytes 9 and 13 are type
|
||||
tags, which record fields are index/length/offset, and whether a chunk is
|
||||
moby_anim. The `D`/`E`/`F` values are suggestive enough to be worth testing
|
||||
against a known type table, but that has not been done.
|
||||
|
||||
Reference in New Issue
Block a user