Decode the chunk count field: header byte 14 is the record count
Range measured: all 27 levels, 1123 chunks, over the 900000 bytes before the constant trailer. Method: compare header byte 14 against the number of offsets actually resolved for that chunk. Result: byte 14 equals the record count on 1123 of 1123 chunks, zero mismatches. This is the first genuinely decoded field in this container rather than a measured correlation. The parser now truncates the offset table to header byte 14, so a stray tagged word in a chunk's record data cannot silently extend the table and put the count at odds with its own header. Two more header bytes are near-constant and read as ASCII: byte 13 takes only 0x45 (1076 chunks) and 0x46 (47); byte 9 takes 0x44 (26), 0x45 (1084) and 0x46 (13). Those are D, E and F -- a three-letter tag at two byte positions with different distributions. Recorded as suggestive, not established: the other 14 header bytes are high-entropy and read as ASCII at no shift tried, so the D/E/F may be incidental. Worth testing against a known type table. Still open: the 14 opaque header bytes, whether bytes 9 and 13 are type tags, which record fields are index/length/offset, and whether a chunk is moby_anim.
This commit is contained in:
@@ -1956,6 +1956,15 @@ fn chunk_containers_resolve_on_every_level() {
|
|||||||
chunk.chunk.offset
|
chunk.chunk.offset
|
||||||
);
|
);
|
||||||
assert!(chunk.chunk.stride > 0, "level{n}: zero stride");
|
assert!(chunk.chunk.stride > 0, "level{n}: zero stride");
|
||||||
|
// The one decoded header field.
|
||||||
|
assert_eq!(
|
||||||
|
chunk.chunk.count as usize,
|
||||||
|
chunk.chunk.offsets.len(),
|
||||||
|
"level{n}: chunk at {} header count {} but {} offsets",
|
||||||
|
chunk.chunk.offset,
|
||||||
|
chunk.chunk.count,
|
||||||
|
chunk.chunk.offsets.len()
|
||||||
|
);
|
||||||
chunks += 1;
|
chunks += 1;
|
||||||
records += chunk.records.len();
|
records += chunk.records.len();
|
||||||
strides.insert(chunk.chunk.stride);
|
strides.insert(chunk.chunk.stride);
|
||||||
|
|||||||
@@ -55,6 +55,12 @@ pub struct Chunk {
|
|||||||
pub header: [u8; CHUNK_HEADER_BYTES],
|
pub header: [u8; CHUNK_HEADER_BYTES],
|
||||||
/// Offset of this chunk within the region it was parsed from.
|
/// Offset of this chunk within the region it was parsed from.
|
||||||
pub offset: usize,
|
pub offset: usize,
|
||||||
|
/// Record count, from header byte 14.
|
||||||
|
///
|
||||||
|
/// Measured: equals `offsets.len()` on **1,123 of 1,123** chunks across
|
||||||
|
/// all 27 levels, zero mismatches. This is the one header field that is
|
||||||
|
/// decoded rather than merely measured.
|
||||||
|
pub count: u8,
|
||||||
/// The low 24 bits of each tagged offset, in file order.
|
/// The low 24 bits of each tagged offset, in file order.
|
||||||
pub offsets: Vec<u32>,
|
pub offsets: Vec<u32>,
|
||||||
/// Record stride, from the first two offsets. Zero if fewer than two.
|
/// Record stride, from the first two offsets. Zero if fewer than two.
|
||||||
@@ -124,6 +130,11 @@ pub fn parse_chunk(region: &[u8], offset: usize) -> Result<Option<Chunk>> {
|
|||||||
offsets.push(v & 0x00ff_ffff);
|
offsets.push(v & 0x00ff_ffff);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// The offset table runs to the first untagged word, but the header carries
|
||||||
|
// its own count. Truncating to that count stops a stray tagged word in the
|
||||||
|
// record data from extending the table.
|
||||||
|
offsets.truncate(header[14] as usize);
|
||||||
|
|
||||||
let stride = if offsets.len() >= 2 {
|
let stride = if offsets.len() >= 2 {
|
||||||
(offsets[1] - offsets[0]) as usize
|
(offsets[1] - offsets[0]) as usize
|
||||||
} else {
|
} else {
|
||||||
@@ -131,6 +142,7 @@ pub fn parse_chunk(region: &[u8], offset: usize) -> Result<Option<Chunk>> {
|
|||||||
};
|
};
|
||||||
|
|
||||||
Ok(Some(Chunk {
|
Ok(Some(Chunk {
|
||||||
|
count: header[14],
|
||||||
header,
|
header,
|
||||||
offset,
|
offset,
|
||||||
offsets,
|
offsets,
|
||||||
@@ -163,6 +175,7 @@ pub fn resolve<'r>(chunk: &Chunk, region: &'r [u8]) -> Result<ResolvedChunk<'r>>
|
|||||||
}
|
}
|
||||||
Ok(ResolvedChunk {
|
Ok(ResolvedChunk {
|
||||||
chunk: Chunk {
|
chunk: Chunk {
|
||||||
|
count: chunk.count,
|
||||||
header: chunk.header,
|
header: chunk.header,
|
||||||
offset: chunk.offset,
|
offset: chunk.offset,
|
||||||
offsets: chunk.offsets.clone(),
|
offsets: chunk.offsets.clone(),
|
||||||
|
|||||||
@@ -799,3 +799,36 @@ the `0xF0` tag, which fields are index/length/offset, and whether a chunk is
|
|||||||
moby_anim or something else. A parser can extract chunks and records
|
moby_anim or something else. A parser can extract chunks and records
|
||||||
bounds-checked today; naming fields needs a domain argument this project has
|
bounds-checked today; naming fields needs a domain argument this project has
|
||||||
not yet found.
|
not yet found.
|
||||||
|
|
||||||
|
## The chunk header: byte 14 is the record count
|
||||||
|
|
||||||
|
**Range measured:** all 27 levels, 1,123 chunks, over the 900,000 bytes
|
||||||
|
before the constant trailer.
|
||||||
|
|
||||||
|
**Method:** compare header byte 14 against the number of offsets actually
|
||||||
|
resolved for that chunk.
|
||||||
|
|
||||||
|
**Result: byte 14 equals the record count on 1,123 of 1,123 chunks, zero
|
||||||
|
mismatches.** This is the first genuinely *decoded* field in this container
|
||||||
|
rather than a measured correlation.
|
||||||
|
|
||||||
|
The parser now truncates the offset table to header byte 14, so a stray
|
||||||
|
tagged word appearing in a chunk's record data cannot silently extend the
|
||||||
|
table -- which would otherwise let the count disagree with its own header.
|
||||||
|
|
||||||
|
**Two more header bytes are near-constant and look like ASCII:**
|
||||||
|
|
||||||
|
- byte 13: only two values across 1,123 chunks -- `0x45` (1,076) and `0x46`
|
||||||
|
(47).
|
||||||
|
- byte 9: three values -- `0x44` (26), `0x45` (1,084), `0x46` (13).
|
||||||
|
|
||||||
|
`0x44`, `0x45` and `0x46` are ASCII `D`, `E` and `F`. That three-letter tag
|
||||||
|
appearing at two different byte positions with different distributions is
|
||||||
|
suggestive of a type or class field, but it is not established: the remaining
|
||||||
|
14 header bytes are high-entropy and do not read as ASCII at any shift tried,
|
||||||
|
so the ASCII in bytes 9 and 13 could be incidental.
|
||||||
|
|
||||||
|
**Still open:** the 14 opaque header bytes, whether bytes 9 and 13 are type
|
||||||
|
tags, which record fields are index/length/offset, and whether a chunk is
|
||||||
|
moby_anim. The `D`/`E`/`F` values are suggestive enough to be worth testing
|
||||||
|
against a known type table, but that has not been done.
|
||||||
|
|||||||
Reference in New Issue
Block a user