The packet table decode left the packet contents open, with a warning that a
VIF reading had already been retracted once and needed a control rather than a
pattern match. This is that measurement.
Range measured: level 0, the 29 packets of the 6 successor-bounded shrub spans,
26640 bytes. Per-packet; nothing pooled.
Method: autocorrelation within the largest packet against a distribution
matched shuffled control, then per-byte distinct counts across all 29 packets.
Result 1: all 29 packet sizes are exact multiples of 16, and the internal stride
scan peaks at 16 (+0.283) with 32, 48, 64, 80 and 128 all elevated behind it --
the signature of a record array, not a coincidental spacing.
Result 2: record 0 is a 16-byte header. Constant across all 29 packets at bytes
0-3 (04 04 00 01), 4-11 (zero), 12-13 (00 80) and 15 (6c). Bytes 14 and 16-23
vary. The 04 04 00 01 prefix looks like a magic or format version.
Result 3: header byte 14 is echoed in the body. It takes values 06, 07, 0e and
11 across packets, and the same value appears at offset 12 of record 1 in every
packet sampled. A real repeated field, not two small values coinciding.
Not established: no header field matches the record count. The packet holds
1056/16 = 66 records and none of the candidate u32 or u16 header offsets at 8,
10, 12, 14, 16, 18 or 20 equals 66, 65 or 67 on more than half the packets. So
the count is either computed from a size field elsewhere in the span, or the
records are variable length and merely 16-byte aligned.
That distinction is unresolved and matters. Fixed 16-byte records make the
payload a flat table; variable records make the 16-byte alignment a coincidence
of every record being a multiple of 16 long. The stride scan cannot separate
the two, because both peak at 16.
The retracted VIF reading is not revived: the tiling is uniform across all 29
packets with no command-shaped tail, and a VIF packet would show a changing
command byte last rather than the constant 6c at header byte 15.
Next: read the record count from the packet table in the span rather than from
the packet header, which is what would separate the two readings.
The open question from the shrub decode was where the mesh lives: the blob
looked like header, palette and optional billboard, with the 333 classes
without a billboard having nothing to draw. Answering it took the running
game, not the disc.
Diffing a loaded shrub class against its disc bytes in the live PCSX2 (level
1, class 1) shows what the loader rewrites: a table at +0x40 of (offset,
size) pairs, whose offsets are span-relative on disc and absolute in RAM with
the sizes unchanged -- all 21 entries matching offset for offset and size for
size -- plus a relocated palette pointer at +0x2c and a pointer at +0x18,
zero on disc, that asserts the parser enforces.
So the blob is header, packet table, packets, [billboard], palette: the mesh
was here the whole time. Why it was missed is the part worth recording --
data_offset points at the END of the span, so reading the body as
data_offset..span_end accounts for the last few hundred bytes and leaves the
20 KB in front of it unexamined. A correct bound and a body small enough to
look plausible are together enough to hide a region.
Range measured: every shrub span on the disc, 358 over 24 levels, each bounded
by its successor class or the located chunk container.
Method: read count at +0x28, then the entries at +0x40, and require them to
tile -- the first packet at align16(0x40 + count*8), every entry exactly size
past the previous, and the last ending at the palette, or 0x40 short of it on
the 25 spans that carry a billboard.
Result: 358 of 358 tile exactly, holding 2,239 packets. That is what pins the
field meanings: count is the packet count (1..55 on the disc), data_offset is
the palette's offset, and the earlier "the body is not sized by count" note
was true for a reason nobody had understood -- the count sizes the packets.
Shrub::packets() exposes them and returns None rather than a partial list
when a table does not tile. Still open: the packet contents (a VIF reading was
already retracted once, so this needs a control rather than a pattern match),
and the 0x20-byte records the loader points at from +0x18.
PCSX2 was set up and working but existed only in this machine's filesystem
and in people's heads, so every comparison looked like a one-off observation
rather than something re-runnable. The setup section in
docs/plan/pcsx2-verification.md now records where the emulator, the BIOS, the
disc, the savestates and the PINE socket are, how a state is captured, and
which command the disc integration tests need -- and AGENTS.md and README.md
point at it from their orientation sections. One operational detail worth
spelling out: the emulator's directory is lowercase (pcsx2), which is not
what people type.
Savestate slot 2 is the second planet, so the comparison runs on a level with
a different geometry extent, light bank and runtime-set population:
level1: 401/401 headers, 401/401 pointers, bank byte-identical
vertices compared: 26470 (105880 colour bytes) -- equal: 100.00%
The live emulator agrees with the savestate byte for byte. Level 1 is a
different planet, so the two fixes in the previous commit are not level-0
coincidences: the normal's z sign and the unlit-record base colour both hold
across 268,280 colour bytes on two levels.
The harness that compares the port against the game's own memory existed but
had never been compiled: tools/trace did not build (27 errors, all in the
argument plumbing), and the tfrag_light.rs light-bank correction it motivated
was unverified. This finishes it and runs it, and the run finds two real bugs.
Result, on level 0 against three inputs -- the live emulator over PINE and two
PCSX2 savestates from the same level, agreeing byte for byte:
headers matching the disc on invariant bytes: 833/833
data pointers relocated as table + disc offset: 833/833
light bank in RAM at 0x50ab20, byte-identical to the disc's
vertices compared: 40600 (162400 colour bytes)
equal: 162400 (100.00%)
The locator finds the level's tfrags block in RAM by tfrag 0's bounding
sphere and accepts a candidate only when every header matches the disc on the
bytes the game leaves alone and every data pointer is the relocated form, so
"the block was found" is itself a verification rather than an assumption. The
bank is searched in RAM by its exact section bytes, which is what settles its
framing: offset at +0x04, an i32 set count, twelve bytes of pad, then the sets.
The previous reading (offset at +0x00, count at +0x04) was wrong.
Two bugs the comparison found, both of which produced plausible output:
1. vertex_normal_bits negated x and y and left z positive, while
NormalTable::normal negates all three. dA came out -0.707 where the game
had +0.707, so max(dA, dA*wA) took the wrong branch and every vertex lit
near its base colour: dark, but not wrong-dark. 95.88% agreement with the
bug, 100.00% without.
2. A record whose select names a set the bank lacks drew transparent black.
The game writes the record's base colour; 3,130 of level 0's 40,600
vertices do this (select 0x000f, set 15 -- a runtime slot the level's
directional bank does not have). The vertex is unlit, not invisible.
Both are now regression tests in the engine. The comparison prints the pass's
per-vertex terms (light_vertex_terms) for the first mismatches, because a
colour mismatch alone cannot be attributed and the comparison is worthless
without attribution -- that is how the second bug was identified from a
three-line read rather than guessed at.
Docs: docs/plan/pcsx2-verification.md records the harness, the result, the
two bugs and what to compare next; roadmap and level-assets.md stop carrying
lighting magnitudes as an open item. Findings reach tests only as numbers --
no test reads a savestate, a dump or a live emulator.
The shrub body's payload encoding was the open question after the span-bound
work, and it answered itself once the R&C1 reference was in view: R&C1's shrub
class header keeps the field offsets R&C2 shows (o_class at 0x24, a count at
0x28, the data offset at 0x2c), which says which bytes to look at, and the disc
says what they hold.
Range measured: all 358 shrub class spans over the 24 levels that have them,
each bounded by its successor class or the located chunk container. Nothing
outside the spans is read.
Result. The body is a palette of 24 unit-length s16 normals -- s16 x, y, z and
a zero pad word per 8-byte record -- followed by all-zero padding records.
Three independent measurements, each of which fails for any other reading:
- all 8,592 live entries have squared length within 65,535 of 32767^2, and
the bound is tight (the largest observed deviation is exactly 65,535). That
is 32768^2 - 32767^2, which is what a unit vector scaled by 32768 produces
when an axis-aligned normal quantises to -32768 and the positive axis clamps
to 32767; components run the full s16 range and 1,765 entries are
axis-aligned. A random s16 triple is off by about a billion, so one entry in
fifteen thousand would pass by chance: all 8,592 pass.
- the padding records are always at index 24 or later and always all zero,
never a live entry. That is what separates the palette from geometry, and it
is the observation the three earlier stride-fitting attempts were missing.
- 17 classes carry a palette of 24 identical (0, 0, 1) entries. A position or
index table cannot produce that; a normal palette for a flat-facing class
must.
The header decodes as: sphere at +0x00 (centre + radius, radius positive on
all 358 spans, 1.72..41.57), size at +0x10, mode_bits and scratch at +0x14,
a zero word at +0x18, the billboard offset at +0x1c, and scale at +0x20. The
parser asserts the two invariants that hold without exception (scratch and the
reserved word are zero on all 358) plus a positive radius and scale, and
reports mode_bits rather than asserting it: it is 4 on 16 of the 358 spans.
The billboard offset is the strongest self-identification after o_class
because it is confirmed from a second file: it is non-zero on exactly the 25
classes whose core-index entry carries billboard texture fields, pairing
correctly both ways. At the offset sits R&C1's ShrubBillboard shape, register
block and all -- fade_distance, then TEX1_1 (K -233..-168, MMIN 4, address
0x14), TEX0_1 (index 1..4, address 0x06) and MIPTBP1_1 (address 0x34), all
three register addresses checked in the disc test.
Two corrections this produces, both recorded in docs/plan/level-assets.md
because the old readings were confidently wrong rather than merely
unverified:
- "reserved is 0 or a denormal" had the two u16 halves the wrong way round:
the 16 non-zero spans are mode_bits = 4 with scratch = 0. As an f32 the word
is still a denormal, so the PS2 reads zero either way -- but the field is a
flag word, not a magnitude, which is exactly the FPU point worth making.
- "bounds" was tested as a min.xyz/max.xyz box, which holds on only 146 of
358 spans, and the field was left unnamed as a result. It is a bounding
sphere: the fourth float is positive on all 358 spans and the centres
cluster near the class origin. The 146/358 figure was a property of the
wrong question.
Still open: the three floats in the billboard record (they repeat
byte-identically across levels -- 16 distinct triples among 25 records, one on
four classes across levels 1, 4, 12 and 21 -- so they are shared art data
rather than dimensions), size, mode_bits, count, and where the shrub mesh
geometry lives. R&C1 puts the mesh packets in this blob; R&C2's blob holds
only the palette, so the 333 classes without a billboard must draw meshes from
somewhere else.
The shrub region's end was the last unlocated bound in core_data. It was not
in the core index, and unlike moby->tie it had no successor boundary to chain
against, so the last shrub class in every level ran out to the lump's end and
the aggregates over it (the "3.5M records / 28MB of shrub body") were
artifacts of that oversized span.
Range measured: for each level, from the last shrub class's geometry offset to
the constant trailer. Nothing outside that range is read.
Method: scan forward from the last shrub class for the pair (a) the constant
two bytes f6 7c sitting immediately in front of (b) a chunk signature -- a
16-byte header ending 0xff, ten zero bytes, then four or more 0xf0-tagged
offsets. The two halves are each weak alone, and the halves are measured
separately: the marker pair occurs 25 to 103 times in a 5 to 17 MB lump, so
matching it alone picks dozens of wrong offsets; the chunk shape matches
random data often enough to need four tagged offsets before it is believed.
Result: on all 27 levels the marker sits exactly two bytes before the first
chunk header, and the two parts of the signature are never found apart. No
chunk signature occurs inside any last shrub body, on any level. The three
levels without shrub (10, 25, 26) carry the same marker before the container
that closes tie instead, which is what makes the locator a property of the
region rather than a shrub rule.
Consequences:
- All 358 shrub class bodies are bounded and are whole 8-byte records. That
was not assertable while the last class in each level ran to the lump's end,
and it is the check the boundary earns its keep for.
- 9,600 records in total, against the 8,938 previously measurable in
successor-bounded spans; 662 of them sit in the 24 last-class bodies. The
shrub region totals 2,033,776 bytes across the 24 levels that have one.
- The chunk container is 25,775,850 bytes across 27 levels (1,236 chunks,
22,562 records). The container measurements were previously taken over "the
900,000 bytes before the trailer", a window that did not align to its real
extent: on level 0 it is 690,302 bytes, and 27 of 1,123 chunks were
truncated at that invented edge. chunk_containers_resolve_on_every_level now
reads the region from core_layout instead of measuring backwards from the
end of the lump.
core_layout takes core_data now: the container's start is the one bound that
cannot be derived from the index. It gains the chunks region, so every byte of
level-specific core_data belongs to a named region. Where the signature is not
found the end stays None -- the region is left visible rather than clamped to
the tail.
Still open: the marker's meaning (constant across 27 levels, at exactly this
position, unnamed) and the shrub body payload's encoding.
Range measured: level 0 shrub bodies, 164 records of 8 bytes, following the
previous commit.
The six varying bytes are not an f32 pair:
f32 @0 158/164 finite, range up to ~1e35. Not positions or normals. The 6
non-finite values (3.7%) are far above the 0.05% random bytes would
give, so the field is structured but is not an f32.
f32 @4 164/164 finite, range 0 to 9.18e-5. Looks promising until you notice
bytes 6 and 7 are always zero, which pins it to a coarse grid and
explains the tiny range. Not independent evidence.
u16 @0 85 distinct, range 0..65534
u16 @2 90 distinct, range 0..65534
u16 @4 80 distinct, range 0..65512
The u16 fields are not small counts or indices; they span essentially the whole
range.
Excluded: two packed u32, an f32 pair, and narrow u16 fields. The six bytes
remain uninterpreted.
Worth noting the f32 @4 case as a trap: a plausible-looking finite non-negative
float range that was entirely an artifact of the two constant zero bytes
pinning it. Checking the constant bytes before believing a float reading is now
the order to work in.
Range measured: level 0, shrub class table, 7 entries at core index offset
10176. Six classes are bounded by a successor; the seventh runs to the end of
core_data and is excluded because its span contains the whole post-shrub
region.
Method: bound each span by the next class geometry offset, take the body from
data_offset to the span end, split into 8-byte records.
Result: 164 records across 6 bounded bodies, lengths 192 twice, 208 twice, 224,
240.
Per byte over those 164 records:
byte 0: 74 distinct, 60 zero byte 4: 72 distinct, 35 zero
byte 1: 58 distinct, 65 zero byte 5: 59 distinct, 38 zero
byte 2: 76 distinct, 43 zero byte 6: 1 distinct, 164 zero
byte 3: 62 distinct, 48 zero byte 7: 1 distinct, 164 zero
Bytes 6 and 7 are zero in every record; bytes 0 to 5 take 58 to 76 distinct
values with only 21 to 39% zero, so uniformly varied rather than sparse.
Read as u32 the fields are not plausible: u32[0] spans 0 to 4294871316 and no
field is monotone across the body. The six bytes are not two packed 32-bit
values.
This settles the six-varying-plus-two-constant shape on level 0 as a per-level
partition, rather than only in aggregate over the disc's 8938 records, which
is what rule 3a asks for. Two trailing zero bytes are padding or a flag pair.
Unresolved: whether the six bytes are six u8, an f32 plus two u16, or
something else. The u32 reading is excluded but no positive reading is
established.
Process note: the first attempt used ClassEntry and a hand-computed o_class
instead of ShrubClassEntry, and every ShrubHeader::region call failed. The
parser was fine; the caller was not.
All 90 two-byte windows of header[0..14] were tested: none maps to a unique
stride across every group. There is no simple two-byte field to read the
record size from.
The stride must therefore come from the offset table, by differencing the
first two tagged offsets, which is what the parser already does.
This validates that choice as more than convenience. Given three chunks sharing
one header with three different strides, a dispatch that assumed "header
implies size" would be wrong for the whole type-ID coverage on at least one
layout. Reading the stride from the offsets is the only interpretation
consistent with every measurement so far.
Negative result recorded deliberately: it rules out a whole family of
hypotheses (stride encoded in the header) rather than narrowing one.
Header 7542eff271b522a79e45348cca4509ff was reported with three strides, 76, 92
and 104. It is not one type with a variable stride, and not a misparse.
It appears three times per level, always with record count 9, always in the
same order at the same offsets within the region:
offset stride count first last
863842 92 9 928 1640
867554 76 9 912 1520
872866 104 9 1000 1832
The offsets are 3712 apart each time and the strides differ, so these are three
separate chunks sharing 16 header bytes.
Consequence for the type ID: 64 distinct headers for 66 distinct record
layouts, not 64 for 64. The header is nearly a type ID but does not uniquely
determine the layout. Not excluded: a distinguishing byte sits outside the 16
captured, or the layout varies within a type.
This does not change "63 of 64 headers imply one stride", which remains true,
but the mapping is not a bijection and should not be used as one. Worth
checking before the parser is extended to dispatch on header alone.
The previous commit reported moby class IDs at u16 slots 129 through 140. That
was a pooling artifact. Repeating the scan per chunk type shows every type
places its class array at a different offset:
aac4 129-134 82c4 104-109 4ec4 106-111
b8c4 120-125 9943 82-87 79c4 111-115
3245 24-29 and so on across all 26 types
Slots 129-140 were simply the type with the largest stride dominating a scan
that pooled every type together. The previous commit flagged this as
unverified; the check confirmed the flag.
What survives is stronger than what it replaced. In every one of the 26 types
the top hits form a contiguous run of consecutive u16 slots. The real
invariant is therefore per-type: each chunk type has its own class array, at
its own offset, of its own length. That fits the header-implies-one-stride
result -- each type is a distinct record layout -- but the class array is a
per-type field, not a common one at a fixed offset.
The moby reference itself is unaffected. The 38.6x excess was measured across
all records and does not depend on which slots were counted.
Range measured: level 0, the 900000 bytes before the trailer; 28 chunks, 475
records, 67780 u16 slots. Reference set: the 238 distinct o_class values from
level 0's moby class table, read through LevelCoreHeader.
Method: scan every u16 in every record, count membership in the moby class set,
and compare against the chance rate of 238/65536 per slot.
Result: 9493 hits against 246.1 expected, a 38.6x excess.
The hits are concentrated, not spread. Slots 129 through 140 -- twelve
consecutive u16, bytes 258 to 280 of the record -- carry 105 to 188 hits each.
Nothing else comes close except a separate cluster at slot 38. Every chunk
contributes in proportion to its record count.
A contiguous twelve-value array that is mostly moby class identifiers is what a
model or instance definition listing referenced mobies would look like.
Enrichment is not identification. A 38x excess over chance is strong evidence
these are moby class IDs rather than incidental small integers, but it does not
establish that the chunk is moby_anim: an instance table, a model list and an
animation table would all reference class IDs. The format and the meaning of
slots 129-140 are both still unnamed.
Also unverified: whether the twelve slots are always the same fields. The
offsets come from the largest records, and this scan pooled all chunk types
together, so a type with a different stride could place them elsewhere.
Process note: the first version of this test extracted the moby class table by
hand and got 67 entries instead of 238, which made every chunk look like a hit
and would have supported any conclusion. Reading it through LevelCoreHeader was
what made the measurement meaningful.
Range measured: all 27 levels, 1123 chunks.
Method: group chunks by their 16-byte header; for each distinct header,
collect the set of strides seen across every level it appears on.
Result: only 64 distinct headers across 1123 chunks. The header is a type
identifier, not a per-chunk descriptor -- the same 16 bytes recur on every
level. And 63 of those 64 imply exactly one stride, so header to record layout
is very nearly a function.
The exception is header 7542eff271b522a79e45348cca4509ff, seen with strides 76,
92 and 104. Either three types share a header or one is a misparse.
This also settles the previous entry's open question about D/E/F being
incidental. Grouping by header bytes 9 and 13:
DE 26 chunks, 2 strides (184, 344)
EE 1050 chunks, 46 strides
EF 34 chunks, 2 strides (268, 640)
FF 13 chunks, 1 stride (520)
Rare tags are tightly coupled to their layouts -- FF is a single stride across
all 13 chunks -- while EE is heterogeneous. That is what a type tag looks
like.
Consequence: a chunk can be dispatched on its header to a record layout, and
63 of 64 headers give that layout's size without inspecting a record. First
point in this container where a field does real work rather than correlating.
Still open: what the 64 types mean, which record fields are
index/length/offset, and whether any type is moby_anim.
Range measured: all 27 levels, 1123 chunks, over the 900000 bytes before the
constant trailer.
Method: compare header byte 14 against the number of offsets actually resolved
for that chunk.
Result: byte 14 equals the record count on 1123 of 1123 chunks, zero
mismatches. This is the first genuinely decoded field in this container rather
than a measured correlation.
The parser now truncates the offset table to header byte 14, so a stray
tagged word in a chunk's record data cannot silently extend the table and put
the count at odds with its own header.
Two more header bytes are near-constant and read as ASCII: byte 13 takes only
0x45 (1076 chunks) and 0x46 (47); byte 9 takes 0x44 (26), 0x45 (1084) and
0x46 (13). Those are D, E and F -- a three-letter tag at two byte positions
with different distributions. Recorded as suggestive, not established: the
other 14 header bytes are high-entropy and read as ASCII at no shift tried, so
the D/E/F may be incidental. Worth testing against a known type table.
Still open: the 14 opaque header bytes, whether bytes 9 and 13 are type tags,
which record fields are index/length/offset, and whether a chunk is moby_anim.
Range measured: each level's post-shrub region, 900000 bytes before the
constant trailer.
Method: per chunk, read tagged offsets as 0xF0000000 | low24, take stride as
the first delta, load one record per offset at chunk_start + low, and count
byte positions constant across that chunk's records.
Result: 1096 chunks and roughly 13000 records resolve on all 27 levels. Every
level shows 17-21% of byte positions constant across its own records. Level 0:
27 chunks, 466 records, 20.8% constant. The stride-368 chunk (77 records) has
26 constant positions of 368; the stride-340 chunk has 25 of 340. Constant
positions cluster at the record end -- 267-276 for stride 368, 258-268 for
stride 340 -- so a footer, not scattered padding.
The tag is uniform: every tagged offset has top byte 0xF0 on all levels.
This settles the previous entry's framing. Resolving at chunk_start + low
yields records with ~20% constant positions, which does not happen by chance,
so the offsets genuinely address records and those records share a layout.
Still not decoded: the 16-byte header, the 0xF0 tag, which fields are
index/length/offset, and whether a chunk is moby_anim. A parser can extract
chunks and records bounds-checked today; naming the fields needs a domain
argument this project has not found.
Range measured: level 0 post-shrub, 16047040..16737344, 690304 bytes.
Method: scan for a 16-byte header ending 0xff, then 10 zero bytes, then a run
of u32 whose top nibble is 0xF.
Result: 28 chunk headers on level 0, present on all 27 levels. Each chunk is
+0x00 16 bytes, last byte 0xff (opaque, differs per chunk)
+0x10 10 zero bytes
+0x1a N tagged offsets, top nibble 0xF
Within a chunk the offsets step by a chunk-specific stride. On level 0:
148, 340, 368, 288, 320, 236, 272, 384, 116, 160, 250, 580, 576. Near-constant
within a chunk, different between chunks.
This resolves two earlier results, both of which were half-right.
The 368-byte period is one chunk's stride, not a region property. That is why
the region-wide autocorrelation is flat: ~15 strides averaging to noise. The
15360-byte window where 368 scored +0.315 is the third chunk at region+30066,
whose stride is 368.
The "ten-entry table" earlier disconfirmed is real. Chunks carry 5 to 30 tagged
offsets, 10 most common on level 0. It was disconfirmed because I tested for a
fixed 148 stride across a whole region; the table exists, but 148 is one
chunk's stride among many.
Both errors were the same one: reporting a local measurement as a regional
property. Twice now. The control methodology caught neither, because a matched
control bounds significance but says nothing about scope.
Not decoded: the 16-byte header, the 0xF tag, where chunk data lives, and what
any chunk means. Framing established, contents not.
The claim that the post-shrub region has a 368-byte period throughout is
wrong. Re-measured over the whole region the curve is flat and near noise:
stride 8 +0.027
stride 12 +0.023
stride 148 +0.009
stride 340 +0.018
stride 368 +0.018
stride 632 +0.020
Nothing peaks at 368, and the 0.1110 recorded earlier does not reproduce under
any sampling convention tried.
Where the structure is: scanning 8 KB windows at 1 KB steps for a 368-excess
above 0.15 finds one substantial region, region+41984..+57344, 15360 bytes,
plus a 4 KB patch at region+650240.
Within that window 368 is unambiguous -- +0.315 against +0.284 at 736, with
364, 372 and 732 all at noise. 41 whole records, mean 16 distinct values per
position, and nine positions exactly constant and zero: 329-336 and 340.
Twenty-five positions have 3 or fewer distinct values, clustered in 325-349,
which is a tail rather than scattered padding.
Corrected statement: a 15360-byte window of the post-shrub region is a table
of 41 records at stride 368, each with a roughly 43-byte tail whose fields are
mostly zero. The surrounding 675 KB is not that table.
What went wrong before: the original measurement was taken over a window where
368 held, then reported as a property of the region. That window holds 41 of
1841 records, and averaging it over the region flattens the peak to the noise
floor. The re-verification checked the stride but not whether it applied
uniformly, so it confirmed the number without testing the claim attached to it.
Range measured: level 0 SBlk, 24..19508, read at 4-byte alignment.
The most common u32 values are 0x0 (2261), 0x1 (369), 0x7f (168),
0x5a000000 (125), 0x1eff7f7f (125), 0x1dfcc (124), 0x19fc0 (93),
0x7c34240d (91), 0x7c3821c8 (91).
That does not look like a table of records with typed fields. A 40-byte layout
of index, length and offset would not produce 0x1eff7f7f and 0x7c34240d a
hundred times each, nor would those values survive a byte shift.
So either the 40-byte period is not a record boundary, or the region is an
encoded stream that happens to have a 40-byte statistical period. The 0x7f
values point at the latter: in a varint encoding 0x7f is the largest value
representable in seven bits, so it appears as a frequent saturated byte --
consistent with a coded stream, inconsistent with fixed-layout records.
Left open. The 40-byte period is solid -- 27 of 27 levels, sharply isolated,
multiples confirming -- and so is the alignment with four constant zero
positions. But those two are in tension in a way nothing else in this project
has been: a confirmed period that is not a record boundary. Someone attacking
this format should not assume the record framing that the autocorrelation
suggests.
Range measured: level 0 SBlk, 24..19508, aligned at +0 -- 487 whole records.
Method: for each of 40 candidate alignments, count byte positions that are
near-constant across records, and keep the best. Alignment +0 wins with 10.
Four positions are exactly constant and all zero: 7, 11, 31 and 35.
Distinct values per position run from 1 to 155, with position 29 the most
variable by a wide margin -- 155 against 21 to 48 elsewhere -- and positions
2, 10, 27, 34 and 39 low at 2 to 8.
The first record carries the SBlk magic and the DAW tag rather than descriptor
data, so it is a header rather than an entry. That matters before reading any
field at a fixed offset.
Not decoded: the layout is established and the constant positions are known,
but the fields are not named and which of the 40 bytes are an index, a length
and an offset is still open.
A fine scan settles it in favour of 40:
stride 40 +0.454
stride 39 +0.266
stride 41 +0.215
stride 38 +0.214
The peak is sharp and isolated, nearly twice its immediate neighbours, and
the multiples confirm it: 80 scores +0.376, 120 +0.394, 160 +0.340, while the
multiples of 12 (24, 48, 96, 144) all sit at 0.21 to 0.24.
So the record is 40 bytes and the 0x7f/0x47f markers at 12-byte spacing are a
nested structure inside those records, not the record boundary. Their spacing
is irregular -- 12, 24 and larger -- which is what one would expect of a field
present in some records and absent in others.
The previous commit's "they disagree" framing was itself wrong in one
respect: the marker spacing never claimed to be the record size. Read as a
claim about markers inside records rather than about the record itself, the
two measurements were never in conflict, and I had set up a false dichotomy
between them.
Range measured: level 0 SBlk, 24..19508, 19484 bytes.
The 0x7f/0x47f marker words sit at 12-byte spacing on 144 of roughly 180
occurrences, with 24-byte and larger gaps accounting for the rest. So there is
a 12-byte structure.
That contradicts the autocorrelation. If records were 12 bytes, stride 12
should peak, and it does not:
stride 12 +0.237
stride 20 +0.175
stride 24 +0.237
stride 40 +0.454
stride 48 +0.239
40 is not a multiple of 12, so the measurements disagree rather than one
explaining the other. Recorded as open rather than resolved in favour of
either: the marker spacing says 12, the period says 40, and 40 being twice as
strong is the more surprising of the two.
Worth resolving before attempting a record layout, because a layout built on
the wrong one will parse plausibly and be wrong.
The part table established vertex_offset + 16 * vertex_count == next_part on
43467 of 43467 parts, which fixes the vertex stride at 16. Autocorrelation
over the spans now reaches the same number without using that result.
Range measured: moby class spans of 16 KB or more on level 0 -- 60 spans,
bounded by consecutive class offsets, over their first 32768 bytes.
Method: peak-stride autocorrelation against a matched control, strides 4 to
64.
stride 16 43 spans
stride 8 8
stride 48 4
stride 32 3
stride 24 2
16 dominates and it is the same 16 the part table produces from a completely
different argument -- a tiling identity over part records rather than a
periodicity measurement over span contents.
Two unrelated methods agreeing is worth more than either alone. The
autocorrelation would be vulnerable to the exact artifacts this project has
already been bitten by: a dominant byte value inflates it, and a period that
happens to divide the measurement window looks identical to a real one.
Neither applies here, and a tiling identity cannot be inflated at all.
This is the strongest form of evidence produced so far in this project -- a
structural invariant confirmed by two methods that do not share a failure
mode.
Range measured: every tie class span of 16 KB or more on levels 0, 3, 6 and
every third level after -- 386 spans, bounded by consecutive class offsets.
Method: peak-stride autocorrelation with a matched control over the first
32768 bytes of each span.
stride 48 285 spans
stride 16 58
stride 64 29
stride 96 11
stride 24 2
stride 32 1
48 bytes dominates but is not universal: 58 spans peak at 16 against 285 at
48, so this is a majority rather than a constant. Every observed peak is a
multiple of 16, suggesting a 16-byte base unit with 48 arising from how those
units group rather than 48 being the record.
This is weaker than the sound bank's 40-byte result and is recorded at the
confidence it deserves. That one was sharply isolated -- 0.454 excess at 40
against 0.21 to 0.28 across strides 36 to 44 -- and identical on all 27
levels. Here the peak is 0.09 to 0.22 excess and moves between levels and
spans, so the two should not be treated as the same kind of claim.
Range measured: each level's SBlk region -- the descriptor table between
sblk_offset and daw_offset, from the sound bank header rather than from
core_data. 17-20 KB per level.
Method: peak-stride autocorrelation with a matched control, the same bytes
shuffled with the same seed on every level.
The peak is at exactly 40 bytes on all 27 levels, every one with excess above
0.2. On level 0 the peak is sharply isolated: 0.454 excess at stride 40
against 0.207 to 0.278 for strides 36 through 44, so it is a period rather
than a broad tendency.
This is the first structural result about the sound bank beyond the container
framing already committed.
Record counts are 487, 443, 428, 494 and 447 whole records on levels 0, 1, 2,
13 and 26, and the region is not an exact multiple of 40 on any level --
19484 leaves 4 bytes over -- so 40 is a period rather than a record size with
no remainder.
Unexplained: which byte positions are constant varies by level. Level 0 has
7, 11, 31 and 35 constant; level 1 has 15, 19, 23, 35 and 39; level 2 has 7,
23, 27 and 31. A fixed 40-byte layout would share its constant positions, so
either the records have a variable tail or 40 is a period in something the
records sit inside.
Also measured: DAW shows no stride above threshold against its control,
consistent with ADPCM nibble-packed audio and with the container framing being
the only part worth decoding. gameplay.bin peaks at 16 bytes. gs_ram.bin has
almost no stride structure but a lag-8 correlation of +0.503, so it is smooth
rather than tabular.
Range measured: sub-objects of the sky region on levels 0, 1, 3, 7, 13 and
26 -- 25 sub-objects, each over its whole length.
Method: correlation at lag against the byte sequence, as before.
lag 8 +0.53 .. +0.79 positive on all 25
lag 64 +0.16 .. +0.63 positive on all 25
lag 512 -0.13 .. +0.42 not universal
lag 1024 -0.05 .. +0.37 not universal
Lag 8 and lag 64 hold on every sub-object of every level tested. The longer
lags do not, so the second scale seen in level 0's first sub-object is a
property of that data rather than of the format.
The sky sub-objects are all the same kind of thing: 8-byte units strongly
related to their neighbours, with a further relation at 64 bytes. That is the
first result about sky that holds everywhere rather than on one level. The
8-byte unit is still unnamed and the values still unidentified.
Range measured: level 0, sky region +365760..+388880, 23120 bytes.
Method: Pearson correlation between the byte sequence and itself at
increasing lag, against a shuffled control. This asks whether consecutive
records are related, which the position-wise distinct-value counts could not
show.
lag 1 +0.050 lag 256 -0.044
lag 4 +0.560 lag 512 +0.304
lag 8 +0.746 lag 1024 +0.247
lag 16 +0.666 lag 2048 +0.079
lag 32 +0.602 shuffled +0.003
lag 64 +0.432
lag 128 +0.140
The profile peaks at lag 8, decays to a zero crossing near 256, then rises
again to a second smaller peak at 512 before decaying. So records are related
within themselves and repeat every 64 records.
This settles the earlier "no constant substructure" reading. The records share
no fixed fields, but they are not independent either, and that was the thing
the distinct-value counts were failing to capture.
Slowly varying values, related within a record, repeating every 64 records, is
the shape of a keyframe or lookup table -- which is what sky data should be,
where a colour or fog parameter is sampled rather than computed per vertex.
Still not decoded: nothing names the values or the sampling. But the shape is
now known well enough for a decode attempt to aim at, rather than to hunt.
The previous commit said "a mean of 6.99 of every 8 bytes are zero". That
is inverted. 6.99 is the mean count of live bytes, and the records are nearly
full rather than sparse. The live-byte histogram is 0:15, 2:1, 4:31, 5:238,
6:461, 7:1030, 8:1114.
That error also invalidated the reasoning built on it in this session's next
message, where the high zero-count was used to explain why the differential
analysis found no constant positions. There was no zero-density to explain.
The differential result stands -- all 8 positions vary -- but for the plain
reason that the records are dense, not because of any artifact.
New finding from re-examining it: byte 7 takes only 14 distinct values where
bytes 0-6 take 142-204. Its common values are 0x00 (1378), 0xff (806) and
0x80 (592), which are the high bytes of signed 16-bit values. That is
consistent with byte 7 being an MSB rather than ordinary data. Unexplained.
Also tried and recorded as not helping: a signed 16-bit reading spans
-32743..32633, near the full range, with no per-position structure (533, 371,
401 and 187 distinct at positions 0-3).
Range measured: level 0, sky region +365760..+388880, 23120 bytes, read as
2890 records of 8. The matched-control autocorrelation puts the peak at
stride 8 (0.217 real, 0.000 shuffled), so 8 is the unit to describe.
Composition: 1185 distinct u32 values across 5780 words; a mean of 6.99 of
every 8 bytes are zero, so most records are sparse; and 549 of 2890 adjacent
pairs, 19%, are exact duplicates of each other. The most common 8-byte record
appears 283 times and is d7d7d780 d7d7d780 -- a 4-byte value stored twice.
Others follow the same shape: 00051980 00051980, 00082580 00082580.
Not decoded. Sparsity with repeated entries is what a keyframe or index table
looks like, but nothing here establishes that, and the duplicated-4-byte shape
may just as easily be two adjacent records sharing a value.
Recorded as a characterisation with its range rather than a finding, because
that is what it is. The useful part is that the next attempt does not have to
re-derive the composition to know whether a decode is plausible.
Range measured: every level's sky region, from the count field at +0x0c
through the sub-object offsets at +0x20, each sub-object over its first
16384 bytes.
Method: peak-stride autocorrelation per sub-object with a matched control --
the same bytes shuffled, seeded per level and index.
Field note: the sub-object count is a u32 at +0x0c, not +0x0a. Reading +0x0a
gives 327,680 on level 0, which is not a count. A brief probe used the wrong
offset and read as though it were a count of 5; it got the right offsets by
accident because the offset table starts at +0x20 either way.
Level 0 has five sub-objects at 365760, 388880, 396432, 406784 and 414032,
of lengths 23120, 7552, 10352, 7248 and 1536, each with structure well clear
of its control.
But the strides do not repeat across levels:
obj 0: 4 (x6), 8 (x13), 32, 288
obj 1: 16 (x3), 32 (x6), 288
obj 2: 8, 16 (x5), 256, 288
obj 3: 8, 16 (x6), 216
obj 4: 8, 16 (x3), 512 (x4)
So the per-index stride table read off level 0 alone was over-fitted to one
level, and is withdrawn. What is consistent is weaker and more honest: every
sub-object sits above its matched control, and most peak at 4, 8, 16 or 32,
suggesting 16- or 32-bit records whose stride varies with content.
Level 2's header is still genuinely different: its count field reads 196,617
and is unusable while every other level yields a plausible count.
The sky autocorrelation turned out to be inflated by a control that did not
match the data, so the 368 result was re-checked the same way rather than
being left on the evidence I now know to be weak.
Range measured: each level's post-shrub region start, first 65536 bytes.
Method: as before, but the control is the same bytes shuffled rather than
uniform random. Shuffling preserves the value distribution and destroys only
the ordering, which is what a control for this statistic needs to do.
Level 0's region is evenly distributed -- 256 distinct byte values, most
common only 15.6% -- so uniform random was the appropriate control here, and
the shuffled control independently confirms it, reading 0.0000 at strides 340,
368 and 4 against the real data's 0.1110 at 368.
Recomputed across all levels with matched controls: 24 of 24 peak at exactly
368, every one with excess above 0.05.
So the 368 period stands, now with a control appropriate to its data. The
contrast is the useful part: the same statistic, asked the same question of
two regions, gave a real answer in one and an inflated one in the other, and
the only thing that separated them was whether the control matched the byte
distribution.
The 256-byte period in the sky region stands, but the evidence I gave for it
was badly chosen and one figure in it was wrong.
The control was uniform random bytes, which read flat at 0.0000 and made the
0.772 match rate look like an enormous signal. It is not: 91.8% of the
region's bytes are the same value, 0x07, so a stride-256 comparison matches
trivially wherever both positions are 0x07.
The correct control is the same bytes shuffled, which keeps the value
distribution and destroys only the ordering. It reads 0.512 at stride 256,
not 0.0000. The real excess is +0.26 -- a genuine signal, and much smaller
than reported.
The extent was wrong for the same reason. Measured against the matched
control, the excess over the value distribution is:
+0 .. +49152 +0.38 .. +0.45
+53248 +0.26
+57344 +0.07
+61440 -0.13
so the periodic data ends between +53248 and +57344, rows 205 to 224, not at
+60160. The earlier number came from a raw threshold with no distribution
control, and a byte value covering 91.8% of the data drags such a threshold
to wherever the dominant run happens to stop.
So: the period is real and cross-checked, the image dimensions are not. Any
"256x236" from the previous commit should be discarded.
The general lesson is the one AGENTS.md asks for and this missed -- a control
has to match the thing being controlled for. Uniform random was the wrong
reference for data with a dominant byte value, and it made a weak result look
like a strong one.
Range measured: level 0, 2192576..2615232, the 422,656-byte sky region.
Periodicity start and end located by sliding an 8 KB window in 256-byte steps
through the region and measuring the 256-stride match rate.
Method: autocorrelation, then a periodicity scan. A random control of the
same size reads flat at 0.0000 for every stride tested.
The peak stride in the first 64 KB is 256 at a 0.772 match rate. Strides 255,
257 and 4 score nearly as high, which is what a genuine period looks like when
sampled rather than assumed -- a useful contrast with the 148-byte "stride"
that held for eight values and then broke.
Across levels the peak is 256 on 21, 512 on 5 and 128 on 1. 512 and 128 are a
harmonic and a sub-multiple of 256, so the fundamental period is 256 bytes.
Within level 0 the periodicity holds from +0 to about +60160 and then stops:
236 rows of 256 bytes, i.e. a 256-wide by 236-tall image at one byte per
pixel. The remaining ~362,240 bytes -- 86% of the region -- are not
256-periodic and stay unidentified. That is presumably what the layer offset
table at +0x20 points into.
The sky region is therefore not one thing. Its first ~60 KB is an image and
is the tractable half; the rest is still open, as is the non-uniform header
where level 2 differs from every other level.
Range measured: level 0, 16047040..16737344. 1875 whole 368-byte records
from the region start, variation checked across the first 60.
Method: the differential analysis that identified the shrub body -- compare
records byte by byte, record which positions differ.
Result: all 368 positions vary. No constant field, no padding, no header
inside a record. The shrub body separated immediately into a varying payload
and a constant two-byte tail; this format has nothing constant to separate.
That is consistent with the autocorrelation peak rather than in tension with
it. Checking which positions carry the 368-stride matches over 20000
positions gives 168 exact 4-byte repeats, distributed evenly at 0.84% across
all four byte offsets. Whole 4-byte sequences recur at exactly 368 while no
individual position is constant -- consistent with records containing entries
that repeat at a fixed relative index, not with a fixed-layout record.
So the 368 stride stands and the records stay unidentified. Worth recording
that the shrub method failed here for a specific and predictable reason: it
needs something constant to latch onto. Applying a method that worked
elsewhere is not the same as it applying here, and saying so is cheaper than
the next person assuming the technique transfers.
Range measured: level 0, 16047040..16737344, sampled as the first 65536
bytes. Cross-checked on every level that has the region, each sampled the
same way from its own region start.
Method: autocorrelation. For each candidate stride, count how often a 4-byte
sequence at offset i repeats at i + stride, as a fraction of comparable
positions. A random control of the same size measured identically reads flat
at 0.0000 for every stride, so the statistic does discriminate.
stride 368 0.1110
stride 340 0.0935
strides 337-339 0.026-0.029 background
random control 0.0000
Stride 368 is the strongest peak. On level 0, 340 is a secondary peak well
clear of the background; the two differ by 28, unexplained.
The cross-check is the part that matters: the peak stride was recomputed
independently for every level with the region, and all 24 peak at exactly
368. One level's peak could be a property of that level's data; 24 of 24
agreeing is a format invariant.
What a 368-byte record is remains unestablished. This is the same position
the shrub body was in before differential analysis identified its records,
and the same method is the obvious next step -- apply it to the 368 stride
rather than continuing to look for headers.
Chosen deliberately over the header search: autocorrelation found this in one
pass with a control, where four header guesses found nothing. The shrub body
only yielded to differential analysis too, and pattern-matching is what
produced this session's four retractions.
Range measured: level 0, 16047040..16737344.
Treating the ten entries' low halves as offsets from the region start lands
on blocks exactly 148 bytes apart, the first nine of which begin 08 00
followed by a 16-bit id stepping by 0x40. That reads as a pointer table and
it is a coincidence.
Walking the same 148-byte stride across the rest of the region, only 14 of
4657 blocks start with 08 00. The first nine agree, then the pattern breaks
completely -- id deltas go from a steady 0x40 to 0x801, 0xf510 and
negative values.
Eight consecutive values at a constant stride is much more than chance
should produce and much less than a structure needs. The lead committed an
entry ago is therefore disconfirmed, and the region remains unidentified:
real, non-zero data, bounded, and unopened beyond its first ~150 bytes.
The open question is unchanged and is where the next attempt should start:
the 690,304 bytes on level 0 between the last shrub class and the constant
trailer.
Range measured: level 0, 16047040..16737344, the 690,304 bytes between the
last shrub class's body and the constant trailer. Nothing outside that range
is read.
Method: reading the bytes at +0x20 as u32 shows ten consecutive entries
whose low halves increase by a constant 0x94 (148), followed by a zero. The
high half is 0xf000 for the first nine and 0x0a40 for the tenth, so the
table is ten entries and ends at +0x48, absolute 16047112.
What the entries mean, what 148 refers to, and why there are ten are all
unestablished. Ten matches no count already known for this level -- 238
moby classes, 74 tie, 7 shrub, 93 tfrag textures -- so it is not yet tied
to anything else in the lump.
Recording it as a lead rather than a finding. The stride is a real
measurement on a bounded range, but a ten-entry table with an unexplained
stride is exactly the shape that looks like a structure while being
something else, and the rule in AGENTS.md is that a pattern gets looked at
before it gets named.
Range measured: each level's last shrub class data_offset field (class +
0x2c) through to the start of the constant trailer. Nothing outside that
range is read.
Method: the last shrub class's body is an array of 8-byte records whose
final two bytes are 00 00. Scanning forward in 8-byte steps for the first
record whose trailing pair differs locates the body's end. That is a
search for a stated pattern rather than a statistic, so it either finds
the boundary or says it did not.
Result: on 24 of 24 levels the boundary lands exactly on an 8-byte record
boundary, with body lengths of 192, 208, 224 or 240 bytes -- the same sizes
seen when shrub classes were grouped and diffed.
Level 0:
last shrub class data 16046800 .. 16047040 240 bytes
next region 16047040 .. 16737344 690,304 bytes
constant trailer 16737344 .. 16969296 231,952 bytes
So 690,304 bytes between the last shrub class and the trailer are a
distinct region rather than shrub data. What is in them is not established.
They are the leading candidate for moby_anim, now that collision..moby has
been ruled out as texture.
Worth recording how a wrong conclusion nearly happened here. An earlier
probe searched for PS2 NaN filler at the start of this range, found none,
and was about to report the body as having zero length. That would have
been wrong: the last shrub class is a real class with real data, and NaN
filler only appears in degenerate ones. Checking the actual byte pattern
instead of concluding from a failed heuristic is what caught it.
Comparing all 27 levels byte-for-byte from the end: the last 231,952 bytes
(0x38A10) of every level's core_data are identical. Level-specific content
therefore ends at len - 0x38A10, which is a real bound for the final region
and is not core_data.len().
This closes shrub. Across the 24 levels that have a shrub region the bound
gives 24,442,928 bytes total, and the disc test asserts on every level that
the region ends strictly inside the lump rather than at its end.
What the trailer is remains unestablished -- fixed data, or padding from a
fixed-size decompression buffer -- but it is constant, which is all a bound
needs.
The data between the last shrub class and the trailer is high entropy: 183 to
256 distinct byte values per 4 KB block, mean 254, where texture data sits
under 200. It is real content of unknown type and is the most likely home for
moby_anim, now that the range between collision and moby has been ruled out
as texture.
A lump shorter than the trailer now yields end: None rather than a backwards
bound, covered by a test.
The "3 of 21 blocks not texture-like" figure from the last commit was an
artifact of sampling every 256 KB and missing everything between. Re-scanned
at 4 KB granularity there are about 100 such blocks, and looking at them
rather than only counting them, they are smooth gradient ramps -- sky and
fog texture. A noisy gradient has a high distinct byte count without being
code, which is exactly the mistake the byte-count heuristic invites.
So collision..moby is entirely texture data. There is no separate moby_anim
region there, and two claims about this range have now been wrong: the
"6.2 MB of unknown data" and the "3 unexplained blocks". Both came from
sizes and counts taken without looking at what they described.
moby_anim is therefore still unlocated, but it is not here. The remaining
candidate is the unresolved tail after the last shrub class -- ~925 KB of
real, non-zero data on level 0 that nothing accounts for.
Recording the two corrections together rather than one at a time, because
the mechanism is the same in both: a heuristic count is not an
identification.
The previous commit claimed ~6.2 MB of unidentified data sat between the
collision header and the first moby class, and speculated it might be
moby_anim. That was wrong, and it came from quoting a size before bounding
the thing it measured.
Looking at the bytes rather than only the offsets: the range is dominated by
8-bit indexed texture data. Byte-value concentration distinguishes them
cleanly -- a texture block draws on a small palette, code and collision data
does not. 18 of 21 sampled 4 KB blocks fall under 200 distinct byte values,
against 249 for tfrags and 209 for collision.
The real layout on level 0:
collision 2615232 .. 3518544 (header plus its declared 903,248 payload)
textures 3518848 .. ~8832192
moby 8832192 ..
The texture table's data_offset values stop at 5326208, which is what made
the gap look genuine. Mip chains extend well past the last entry's offset,
so sizing a region from a table of first-level offsets understates the data
it points at.
Three of the 21 blocks are not texture-like, so something is mixed in that
is still unidentified. It is not 6.2 MB of moby_anim.
This is the third number this session that looked load-bearing and turned
out to rest on an unbounded measurement. The pattern is consistent enough
to be worth stating plainly: sizes derived from tables or offsets are not
evidence until the bytes underneath have been looked at.
Printing the region table next to the header fields makes the gap visible
without having to remember to consult core_layout, and it paid for itself
immediately.
On level 0 the collision region comes out 6216960 bytes, from 2615232 to
the first moby class at 8832192. Collision's own header declares a payload
of 903,248. So roughly 6.2 MB sits between the collision header and the
first moby class, and almost none of it is collision.
The successor rule is correct -- a region does end where the next one
begins -- but "the next" is not yet known there. collision..moby is
really collision + <something unnamed> .. moby. The name is wrong, not
the bound.
Candidates for the space, rough order of likelihood: moby and tie model
animation data, which is what moby_anim would need, then volumes, then
per-class runtime structures. Tfrag texture texels are already placed and
sit earlier in the lump.
Worth resolving before starting moby_anim, since that format is the most
likely tenant. Until then, any probe reading between the collision header
and the first moby class is reading something the region table does not
describe.
Region spans were being re-derived per format while probing, usually
running from one known offset to the next with the last span running to
core_data.len(). On level 0 that gave the final shrub class a 925,008-byte
span absorbing most of the lump's tail, and every aggregate computed over
it -- record counts, value distributions, "this looks random" -- was
silently wrong. Nothing errored, which is what made it survive two
commits and a correction.
core_layout computes the layout once: each region ends where the next one
begins, class regions start at the lowest non-zero geometry offset in their
table, and a region whose successor is unknown gets end: None rather than a
clamped guess. Zero class entries mean absent, not position zero -- the
same trap occlusion and collision each fell into separately.
The disc test asserts what the module is actually for: that the unknown is
visible. Shrub's end is unresolved on the 24 levels that have a shrub
region, and that is asserted rather than hidden.
It also found that not every level has every region -- level 10 has neither
occlusion nor shrub -- so the test checks ordering against the canonical
sequence instead of demanding an identical set.
Still open: where the shrub region ends. Texture texels cannot bound it;
they sit before moby, not after shrub. See docs/formats/core-data-regions.md.
A follow-up measurement contradicted the body size implied by the
differential analysis, and the cause is bad span bounds rather than
anything in the format.
Shrub spans were bounded by the next shrub offset, with the last running
to the end of core_data. On level 0 that gives the final shrub class a
925,008-byte span, absorbing most of the lump's tail. The moby-to-tie
boundary is clean by comparison: last moby offset 13414592, first tie
13416192, exactly 1600 bytes apart.
What that invalidates:
- The 3.5M record / 28 MB body figure is an artifact of the oversized final
span and should not be quoted.
- The "payload looks uniform-random" measurement ran over the same
contaminated range, so it says nothing about shrub geometry either way.
- The differential result itself still stands -- it was computed on groups,
and the groups used had bounded spans. The global aggregates built on it
do not.
What is needed is the offset ending the shrub region. It is not in the core
index, and there is no obvious successor boundary the way moby->tie has one.
This is the second confident-sounding number in this format that turned out
to rest on a bad bound, the first being the VIF reading. Aggregates across
many spans are only as good as the span bounds under them, and bounds are
the easy thing to get wrong because nothing errors when you do.
The previous commit concluded the shrub body's 6-byte payload was
VIF-packed, on the strength of recurring patterns like 7ea5 and 1ab6. That
was pattern matching dressed up as a conclusion, and a control kills it.
Decoding every 32-bit word of every shrub body as a VifCode:
shrub bodies 24.9% unpack, 128 distinct cmds
uniform random 25.1% unpack, 128 distinct cmds
sound_bank (definitely not VIF) 19.0% unpack, 122 distinct cmds
gs_ram (mixed) 13.6% unpack, 128 distinct cmds
moby class 0 span (known VIF) 40.1% unpack, 105 distinct cmds
Shrub sits on the random baseline and exercises all 128 possible command
bytes, which is what unstructured data looks like.
Worth being precise about what this does and does not show. The test is
weak -- even the known-VIF region only reaches 40.1%, well short of a
clean signal -- so this does not prove shrub is not VIF. It proves the
claim was never established, which is the part that was wrong.
Spotting three recurring byte values and naming them a format is the same
mistake as the three stride guesses it was meant to replace, one level
down. A hypothesis needs a discriminating test before it goes in a commit
message, and I skipped that step because the finding was welcome.
What still stands is the part that rests on differential analysis rather
than recognition: 8-byte records, 6 varying and 2 constant, 144 varying
bytes per group regardless of count. The payload's encoding is unknown.
Stride-fitting the shrub body failed three times, so this attacks it from
the other side: group spans by (count, data_offset, tail length) and byte
diff every member of each group. Positions that differ across a group are
geometry; positions that never differ are structure.
The answer is blunt. 144 bytes vary in every group, whether that group has
count 1, 2, 3, 5 or 6 and a tail of 192, 208 or 240 bytes. The varying
data is not sized by count at all, which is why three stride guesses were
fitting the wrong thing -- and why they kept looking nearly plausible.
Those 144 bytes fall in runs of 6, every 8 bytes. The body is an array of
8-byte records: 6 varying bytes and 2 constant.
The payload is not half-floats -- decoding as half3 gives 245.25 and
-168.375, which no shrub has. The patterns (7ea5, 1ab6, 43d9, af80) and
their repetition are VIF-packed data, the same quadruple-word packing moby
and tfrag use. Filler around it is the PS2 NaN 0x7fff0000 rather than
zeros, which is why the body read as noise in the earlier dumps.
So the open question is the VIF code assignment, not the stride, and the
decoder written for tfrag is the obvious place to start.
Reading these formats by unpacking raw offsets out of core_index.bin has
cost real time. The index interleaves single-word region offsets with
count/offset pairs, so miscounting one shifts every field after it: that
produced a page of implausible collision headers on all 27 levels, sent
me chasing a tie-breakoff that turned out not to exist, and nearly had me
"fix" a stride bug in code that was already right.
probe dumps a level's core header and all three class tables through
LevelCoreHeader, so a field name printed is the field name the parser
uses. It also reports how many entries carry geometry inside core_data,
which is what pins the class table strides rather than guessing at them:
shrub reads 7/7 sane at 0x30 against 4/7 read at 0x20.
Reading a structure by hand in a shell still works; this just stops it
being the default, and stops the offset arithmetic being re-derived every
time a new region is investigated.
Checked the obvious hypothesis -- that tie, being the other class-geometry
table, would share moby's part-table format -- and it does not.
The tiling rule vertex_offset + 16 * vertex_count == next_part holds for
0 of 25,024 records across the 1,864 tie spans, against 43,467/43,467 for
moby. Tie's second field also reads as an absolute offset rather than a
span-relative one, which is the first sign the record shape differs.
Past the verified 0x48-byte header the bytes are f32-shaped rather than
the expected VIF marker, but they are not a fixed array either: the run of
consecutive plausible floats varies from 1 to 6 across spans (822 spans
have 6, 455 have 1, 373 have 2). So there is more header past 0x48 and
it is not one array of distances -- the floats after that point are a
different population from the LOD distances before it, including negative
values.
Recorded so the next attempt starts from that rather than re-trying the
moby template.
ReaperOfVeriod