The previous commit said "a mean of 6.99 of every 8 bytes are zero". That
is inverted. 6.99 is the mean count of live bytes, and the records are nearly
full rather than sparse. The live-byte histogram is 0:15, 2:1, 4:31, 5:238,
6:461, 7:1030, 8:1114.
That error also invalidated the reasoning built on it in this session's next
message, where the high zero-count was used to explain why the differential
analysis found no constant positions. There was no zero-density to explain.
The differential result stands -- all 8 positions vary -- but for the plain
reason that the records are dense, not because of any artifact.
New finding from re-examining it: byte 7 takes only 14 distinct values where
bytes 0-6 take 142-204. Its common values are 0x00 (1378), 0xff (806) and
0x80 (592), which are the high bytes of signed 16-bit values. That is
consistent with byte 7 being an MSB rather than ordinary data. Unexplained.
Also tried and recorded as not helping: a signed 16-bit reading spans
-32743..32633, near the full range, with no per-position structure (533, 371,
401 and 187 distinct at positions 0-3).
Range measured: level 0, sky region +365760..+388880, 23120 bytes, read as
2890 records of 8. The matched-control autocorrelation puts the peak at
stride 8 (0.217 real, 0.000 shuffled), so 8 is the unit to describe.
Composition: 1185 distinct u32 values across 5780 words; a mean of 6.99 of
every 8 bytes are zero, so most records are sparse; and 549 of 2890 adjacent
pairs, 19%, are exact duplicates of each other. The most common 8-byte record
appears 283 times and is d7d7d780 d7d7d780 -- a 4-byte value stored twice.
Others follow the same shape: 00051980 00051980, 00082580 00082580.
Not decoded. Sparsity with repeated entries is what a keyframe or index table
looks like, but nothing here establishes that, and the duplicated-4-byte shape
may just as easily be two adjacent records sharing a value.
Recorded as a characterisation with its range rather than a finding, because
that is what it is. The useful part is that the next attempt does not have to
re-derive the composition to know whether a decode is plausible.
Range measured: every level's sky region, from the count field at +0x0c
through the sub-object offsets at +0x20, each sub-object over its first
16384 bytes.
Method: peak-stride autocorrelation per sub-object with a matched control --
the same bytes shuffled, seeded per level and index.
Field note: the sub-object count is a u32 at +0x0c, not +0x0a. Reading +0x0a
gives 327,680 on level 0, which is not a count. A brief probe used the wrong
offset and read as though it were a count of 5; it got the right offsets by
accident because the offset table starts at +0x20 either way.
Level 0 has five sub-objects at 365760, 388880, 396432, 406784 and 414032,
of lengths 23120, 7552, 10352, 7248 and 1536, each with structure well clear
of its control.
But the strides do not repeat across levels:
obj 0: 4 (x6), 8 (x13), 32, 288
obj 1: 16 (x3), 32 (x6), 288
obj 2: 8, 16 (x5), 256, 288
obj 3: 8, 16 (x6), 216
obj 4: 8, 16 (x3), 512 (x4)
So the per-index stride table read off level 0 alone was over-fitted to one
level, and is withdrawn. What is consistent is weaker and more honest: every
sub-object sits above its matched control, and most peak at 4, 8, 16 or 32,
suggesting 16- or 32-bit records whose stride varies with content.
Level 2's header is still genuinely different: its count field reads 196,617
and is unusable while every other level yields a plausible count.
The sky autocorrelation turned out to be inflated by a control that did not
match the data, so the 368 result was re-checked the same way rather than
being left on the evidence I now know to be weak.
Range measured: each level's post-shrub region start, first 65536 bytes.
Method: as before, but the control is the same bytes shuffled rather than
uniform random. Shuffling preserves the value distribution and destroys only
the ordering, which is what a control for this statistic needs to do.
Level 0's region is evenly distributed -- 256 distinct byte values, most
common only 15.6% -- so uniform random was the appropriate control here, and
the shuffled control independently confirms it, reading 0.0000 at strides 340,
368 and 4 against the real data's 0.1110 at 368.
Recomputed across all levels with matched controls: 24 of 24 peak at exactly
368, every one with excess above 0.05.
So the 368 period stands, now with a control appropriate to its data. The
contrast is the useful part: the same statistic, asked the same question of
two regions, gave a real answer in one and an inflated one in the other, and
the only thing that separated them was whether the control matched the byte
distribution.
The 256-byte period in the sky region stands, but the evidence I gave for it
was badly chosen and one figure in it was wrong.
The control was uniform random bytes, which read flat at 0.0000 and made the
0.772 match rate look like an enormous signal. It is not: 91.8% of the
region's bytes are the same value, 0x07, so a stride-256 comparison matches
trivially wherever both positions are 0x07.
The correct control is the same bytes shuffled, which keeps the value
distribution and destroys only the ordering. It reads 0.512 at stride 256,
not 0.0000. The real excess is +0.26 -- a genuine signal, and much smaller
than reported.
The extent was wrong for the same reason. Measured against the matched
control, the excess over the value distribution is:
+0 .. +49152 +0.38 .. +0.45
+53248 +0.26
+57344 +0.07
+61440 -0.13
so the periodic data ends between +53248 and +57344, rows 205 to 224, not at
+60160. The earlier number came from a raw threshold with no distribution
control, and a byte value covering 91.8% of the data drags such a threshold
to wherever the dominant run happens to stop.
So: the period is real and cross-checked, the image dimensions are not. Any
"256x236" from the previous commit should be discarded.
The general lesson is the one AGENTS.md asks for and this missed -- a control
has to match the thing being controlled for. Uniform random was the wrong
reference for data with a dominant byte value, and it made a weak result look
like a strong one.
Range measured: level 0, 2192576..2615232, the 422,656-byte sky region.
Periodicity start and end located by sliding an 8 KB window in 256-byte steps
through the region and measuring the 256-stride match rate.
Method: autocorrelation, then a periodicity scan. A random control of the
same size reads flat at 0.0000 for every stride tested.
The peak stride in the first 64 KB is 256 at a 0.772 match rate. Strides 255,
257 and 4 score nearly as high, which is what a genuine period looks like when
sampled rather than assumed -- a useful contrast with the 148-byte "stride"
that held for eight values and then broke.
Across levels the peak is 256 on 21, 512 on 5 and 128 on 1. 512 and 128 are a
harmonic and a sub-multiple of 256, so the fundamental period is 256 bytes.
Within level 0 the periodicity holds from +0 to about +60160 and then stops:
236 rows of 256 bytes, i.e. a 256-wide by 236-tall image at one byte per
pixel. The remaining ~362,240 bytes -- 86% of the region -- are not
256-periodic and stay unidentified. That is presumably what the layer offset
table at +0x20 points into.
The sky region is therefore not one thing. Its first ~60 KB is an image and
is the tractable half; the rest is still open, as is the non-uniform header
where level 2 differs from every other level.
Range measured: level 0, 16047040..16737344. 1875 whole 368-byte records
from the region start, variation checked across the first 60.
Method: the differential analysis that identified the shrub body -- compare
records byte by byte, record which positions differ.
Result: all 368 positions vary. No constant field, no padding, no header
inside a record. The shrub body separated immediately into a varying payload
and a constant two-byte tail; this format has nothing constant to separate.
That is consistent with the autocorrelation peak rather than in tension with
it. Checking which positions carry the 368-stride matches over 20000
positions gives 168 exact 4-byte repeats, distributed evenly at 0.84% across
all four byte offsets. Whole 4-byte sequences recur at exactly 368 while no
individual position is constant -- consistent with records containing entries
that repeat at a fixed relative index, not with a fixed-layout record.
So the 368 stride stands and the records stay unidentified. Worth recording
that the shrub method failed here for a specific and predictable reason: it
needs something constant to latch onto. Applying a method that worked
elsewhere is not the same as it applying here, and saying so is cheaper than
the next person assuming the technique transfers.
Range measured: level 0, 16047040..16737344, sampled as the first 65536
bytes. Cross-checked on every level that has the region, each sampled the
same way from its own region start.
Method: autocorrelation. For each candidate stride, count how often a 4-byte
sequence at offset i repeats at i + stride, as a fraction of comparable
positions. A random control of the same size measured identically reads flat
at 0.0000 for every stride, so the statistic does discriminate.
stride 368 0.1110
stride 340 0.0935
strides 337-339 0.026-0.029 background
random control 0.0000
Stride 368 is the strongest peak. On level 0, 340 is a secondary peak well
clear of the background; the two differ by 28, unexplained.
The cross-check is the part that matters: the peak stride was recomputed
independently for every level with the region, and all 24 peak at exactly
368. One level's peak could be a property of that level's data; 24 of 24
agreeing is a format invariant.
What a 368-byte record is remains unestablished. This is the same position
the shrub body was in before differential analysis identified its records,
and the same method is the obvious next step -- apply it to the 368 stride
rather than continuing to look for headers.
Chosen deliberately over the header search: autocorrelation found this in one
pass with a control, where four header guesses found nothing. The shrub body
only yielded to differential analysis too, and pattern-matching is what
produced this session's four retractions.
Range measured: level 0, 16047040..16737344.
Treating the ten entries' low halves as offsets from the region start lands
on blocks exactly 148 bytes apart, the first nine of which begin 08 00
followed by a 16-bit id stepping by 0x40. That reads as a pointer table and
it is a coincidence.
Walking the same 148-byte stride across the rest of the region, only 14 of
4657 blocks start with 08 00. The first nine agree, then the pattern breaks
completely -- id deltas go from a steady 0x40 to 0x801, 0xf510 and
negative values.
Eight consecutive values at a constant stride is much more than chance
should produce and much less than a structure needs. The lead committed an
entry ago is therefore disconfirmed, and the region remains unidentified:
real, non-zero data, bounded, and unopened beyond its first ~150 bytes.
The open question is unchanged and is where the next attempt should start:
the 690,304 bytes on level 0 between the last shrub class and the constant
trailer.
Range measured: level 0, 16047040..16737344, the 690,304 bytes between the
last shrub class's body and the constant trailer. Nothing outside that range
is read.
Method: reading the bytes at +0x20 as u32 shows ten consecutive entries
whose low halves increase by a constant 0x94 (148), followed by a zero. The
high half is 0xf000 for the first nine and 0x0a40 for the tenth, so the
table is ten entries and ends at +0x48, absolute 16047112.
What the entries mean, what 148 refers to, and why there are ten are all
unestablished. Ten matches no count already known for this level -- 238
moby classes, 74 tie, 7 shrub, 93 tfrag textures -- so it is not yet tied
to anything else in the lump.
Recording it as a lead rather than a finding. The stride is a real
measurement on a bounded range, but a ten-entry table with an unexplained
stride is exactly the shape that looks like a structure while being
something else, and the rule in AGENTS.md is that a pattern gets looked at
before it gets named.
Range measured: each level's last shrub class data_offset field (class +
0x2c) through to the start of the constant trailer. Nothing outside that
range is read.
Method: the last shrub class's body is an array of 8-byte records whose
final two bytes are 00 00. Scanning forward in 8-byte steps for the first
record whose trailing pair differs locates the body's end. That is a
search for a stated pattern rather than a statistic, so it either finds
the boundary or says it did not.
Result: on 24 of 24 levels the boundary lands exactly on an 8-byte record
boundary, with body lengths of 192, 208, 224 or 240 bytes -- the same sizes
seen when shrub classes were grouped and diffed.
Level 0:
last shrub class data 16046800 .. 16047040 240 bytes
next region 16047040 .. 16737344 690,304 bytes
constant trailer 16737344 .. 16969296 231,952 bytes
So 690,304 bytes between the last shrub class and the trailer are a
distinct region rather than shrub data. What is in them is not established.
They are the leading candidate for moby_anim, now that collision..moby has
been ruled out as texture.
Worth recording how a wrong conclusion nearly happened here. An earlier
probe searched for PS2 NaN filler at the start of this range, found none,
and was about to report the body as having zero length. That would have
been wrong: the last shrub class is a real class with real data, and NaN
filler only appears in degenerate ones. Checking the actual byte pattern
instead of concluding from a failed heuristic is what caught it.
Comparing all 27 levels byte-for-byte from the end: the last 231,952 bytes
(0x38A10) of every level's core_data are identical. Level-specific content
therefore ends at len - 0x38A10, which is a real bound for the final region
and is not core_data.len().
This closes shrub. Across the 24 levels that have a shrub region the bound
gives 24,442,928 bytes total, and the disc test asserts on every level that
the region ends strictly inside the lump rather than at its end.
What the trailer is remains unestablished -- fixed data, or padding from a
fixed-size decompression buffer -- but it is constant, which is all a bound
needs.
The data between the last shrub class and the trailer is high entropy: 183 to
256 distinct byte values per 4 KB block, mean 254, where texture data sits
under 200. It is real content of unknown type and is the most likely home for
moby_anim, now that the range between collision and moby has been ruled out
as texture.
A lump shorter than the trailer now yields end: None rather than a backwards
bound, covered by a test.
The "3 of 21 blocks not texture-like" figure from the last commit was an
artifact of sampling every 256 KB and missing everything between. Re-scanned
at 4 KB granularity there are about 100 such blocks, and looking at them
rather than only counting them, they are smooth gradient ramps -- sky and
fog texture. A noisy gradient has a high distinct byte count without being
code, which is exactly the mistake the byte-count heuristic invites.
So collision..moby is entirely texture data. There is no separate moby_anim
region there, and two claims about this range have now been wrong: the
"6.2 MB of unknown data" and the "3 unexplained blocks". Both came from
sizes and counts taken without looking at what they described.
moby_anim is therefore still unlocated, but it is not here. The remaining
candidate is the unresolved tail after the last shrub class -- ~925 KB of
real, non-zero data on level 0 that nothing accounts for.
Recording the two corrections together rather than one at a time, because
the mechanism is the same in both: a heuristic count is not an
identification.
The previous commit claimed ~6.2 MB of unidentified data sat between the
collision header and the first moby class, and speculated it might be
moby_anim. That was wrong, and it came from quoting a size before bounding
the thing it measured.
Looking at the bytes rather than only the offsets: the range is dominated by
8-bit indexed texture data. Byte-value concentration distinguishes them
cleanly -- a texture block draws on a small palette, code and collision data
does not. 18 of 21 sampled 4 KB blocks fall under 200 distinct byte values,
against 249 for tfrags and 209 for collision.
The real layout on level 0:
collision 2615232 .. 3518544 (header plus its declared 903,248 payload)
textures 3518848 .. ~8832192
moby 8832192 ..
The texture table's data_offset values stop at 5326208, which is what made
the gap look genuine. Mip chains extend well past the last entry's offset,
so sizing a region from a table of first-level offsets understates the data
it points at.
Three of the 21 blocks are not texture-like, so something is mixed in that
is still unidentified. It is not 6.2 MB of moby_anim.
This is the third number this session that looked load-bearing and turned
out to rest on an unbounded measurement. The pattern is consistent enough
to be worth stating plainly: sizes derived from tables or offsets are not
evidence until the bytes underneath have been looked at.
Printing the region table next to the header fields makes the gap visible
without having to remember to consult core_layout, and it paid for itself
immediately.
On level 0 the collision region comes out 6216960 bytes, from 2615232 to
the first moby class at 8832192. Collision's own header declares a payload
of 903,248. So roughly 6.2 MB sits between the collision header and the
first moby class, and almost none of it is collision.
The successor rule is correct -- a region does end where the next one
begins -- but "the next" is not yet known there. collision..moby is
really collision + <something unnamed> .. moby. The name is wrong, not
the bound.
Candidates for the space, rough order of likelihood: moby and tie model
animation data, which is what moby_anim would need, then volumes, then
per-class runtime structures. Tfrag texture texels are already placed and
sit earlier in the lump.
Worth resolving before starting moby_anim, since that format is the most
likely tenant. Until then, any probe reading between the collision header
and the first moby class is reading something the region table does not
describe.
Region spans were being re-derived per format while probing, usually
running from one known offset to the next with the last span running to
core_data.len(). On level 0 that gave the final shrub class a 925,008-byte
span absorbing most of the lump's tail, and every aggregate computed over
it -- record counts, value distributions, "this looks random" -- was
silently wrong. Nothing errored, which is what made it survive two
commits and a correction.
core_layout computes the layout once: each region ends where the next one
begins, class regions start at the lowest non-zero geometry offset in their
table, and a region whose successor is unknown gets end: None rather than a
clamped guess. Zero class entries mean absent, not position zero -- the
same trap occlusion and collision each fell into separately.
The disc test asserts what the module is actually for: that the unknown is
visible. Shrub's end is unresolved on the 24 levels that have a shrub
region, and that is asserted rather than hidden.
It also found that not every level has every region -- level 10 has neither
occlusion nor shrub -- so the test checks ordering against the canonical
sequence instead of demanding an identical set.
Still open: where the shrub region ends. Texture texels cannot bound it;
they sit before moby, not after shrub. See docs/formats/core-data-regions.md.
ReaperOfVeriod