A fine scan settles it in favour of 40:
stride 40 +0.454
stride 39 +0.266
stride 41 +0.215
stride 38 +0.214
The peak is sharp and isolated, nearly twice its immediate neighbours, and
the multiples confirm it: 80 scores +0.376, 120 +0.394, 160 +0.340, while the
multiples of 12 (24, 48, 96, 144) all sit at 0.21 to 0.24.
So the record is 40 bytes and the 0x7f/0x47f markers at 12-byte spacing are a
nested structure inside those records, not the record boundary. Their spacing
is irregular -- 12, 24 and larger -- which is what one would expect of a field
present in some records and absent in others.
The previous commit's "they disagree" framing was itself wrong in one
respect: the marker spacing never claimed to be the record size. Read as a
claim about markers inside records rather than about the record itself, the
two measurements were never in conflict, and I had set up a false dichotomy
between them.
Range measured: level 0 SBlk, 24..19508, 19484 bytes.
The 0x7f/0x47f marker words sit at 12-byte spacing on 144 of roughly 180
occurrences, with 24-byte and larger gaps accounting for the rest. So there is
a 12-byte structure.
That contradicts the autocorrelation. If records were 12 bytes, stride 12
should peak, and it does not:
stride 12 +0.237
stride 20 +0.175
stride 24 +0.237
stride 40 +0.454
stride 48 +0.239
40 is not a multiple of 12, so the measurements disagree rather than one
explaining the other. Recorded as open rather than resolved in favour of
either: the marker spacing says 12, the period says 40, and 40 being twice as
strong is the more surprising of the two.
Worth resolving before attempting a record layout, because a layout built on
the wrong one will parse plausibly and be wrong.
The part table established vertex_offset + 16 * vertex_count == next_part on
43467 of 43467 parts, which fixes the vertex stride at 16. Autocorrelation
over the spans now reaches the same number without using that result.
Range measured: moby class spans of 16 KB or more on level 0 -- 60 spans,
bounded by consecutive class offsets, over their first 32768 bytes.
Method: peak-stride autocorrelation against a matched control, strides 4 to
64.
stride 16 43 spans
stride 8 8
stride 48 4
stride 32 3
stride 24 2
16 dominates and it is the same 16 the part table produces from a completely
different argument -- a tiling identity over part records rather than a
periodicity measurement over span contents.
Two unrelated methods agreeing is worth more than either alone. The
autocorrelation would be vulnerable to the exact artifacts this project has
already been bitten by: a dominant byte value inflates it, and a period that
happens to divide the measurement window looks identical to a real one.
Neither applies here, and a tiling identity cannot be inflated at all.
This is the strongest form of evidence produced so far in this project -- a
structural invariant confirmed by two methods that do not share a failure
mode.
Range measured: every tie class span of 16 KB or more on levels 0, 3, 6 and
every third level after -- 386 spans, bounded by consecutive class offsets.
Method: peak-stride autocorrelation with a matched control over the first
32768 bytes of each span.
stride 48 285 spans
stride 16 58
stride 64 29
stride 96 11
stride 24 2
stride 32 1
48 bytes dominates but is not universal: 58 spans peak at 16 against 285 at
48, so this is a majority rather than a constant. Every observed peak is a
multiple of 16, suggesting a 16-byte base unit with 48 arising from how those
units group rather than 48 being the record.
This is weaker than the sound bank's 40-byte result and is recorded at the
confidence it deserves. That one was sharply isolated -- 0.454 excess at 40
against 0.21 to 0.28 across strides 36 to 44 -- and identical on all 27
levels. Here the peak is 0.09 to 0.22 excess and moves between levels and
spans, so the two should not be treated as the same kind of claim.
Range measured: each level's SBlk region -- the descriptor table between
sblk_offset and daw_offset, from the sound bank header rather than from
core_data. 17-20 KB per level.
Method: peak-stride autocorrelation with a matched control, the same bytes
shuffled with the same seed on every level.
The peak is at exactly 40 bytes on all 27 levels, every one with excess above
0.2. On level 0 the peak is sharply isolated: 0.454 excess at stride 40
against 0.207 to 0.278 for strides 36 through 44, so it is a period rather
than a broad tendency.
This is the first structural result about the sound bank beyond the container
framing already committed.
Record counts are 487, 443, 428, 494 and 447 whole records on levels 0, 1, 2,
13 and 26, and the region is not an exact multiple of 40 on any level --
19484 leaves 4 bytes over -- so 40 is a period rather than a record size with
no remainder.
Unexplained: which byte positions are constant varies by level. Level 0 has
7, 11, 31 and 35 constant; level 1 has 15, 19, 23, 35 and 39; level 2 has 7,
23, 27 and 31. A fixed 40-byte layout would share its constant positions, so
either the records have a variable tail or 40 is a period in something the
records sit inside.
Also measured: DAW shows no stride above threshold against its control,
consistent with ADPCM nibble-packed audio and with the container framing being
the only part worth decoding. gameplay.bin peaks at 16 bytes. gs_ram.bin has
almost no stride structure but a lag-8 correlation of +0.503, so it is smooth
rather than tabular.
Range measured: sub-objects of the sky region on levels 0, 1, 3, 7, 13 and
26 -- 25 sub-objects, each over its whole length.
Method: correlation at lag against the byte sequence, as before.
lag 8 +0.53 .. +0.79 positive on all 25
lag 64 +0.16 .. +0.63 positive on all 25
lag 512 -0.13 .. +0.42 not universal
lag 1024 -0.05 .. +0.37 not universal
Lag 8 and lag 64 hold on every sub-object of every level tested. The longer
lags do not, so the second scale seen in level 0's first sub-object is a
property of that data rather than of the format.
The sky sub-objects are all the same kind of thing: 8-byte units strongly
related to their neighbours, with a further relation at 64 bytes. That is the
first result about sky that holds everywhere rather than on one level. The
8-byte unit is still unnamed and the values still unidentified.
Range measured: level 0, sky region +365760..+388880, 23120 bytes.
Method: Pearson correlation between the byte sequence and itself at
increasing lag, against a shuffled control. This asks whether consecutive
records are related, which the position-wise distinct-value counts could not
show.
lag 1 +0.050 lag 256 -0.044
lag 4 +0.560 lag 512 +0.304
lag 8 +0.746 lag 1024 +0.247
lag 16 +0.666 lag 2048 +0.079
lag 32 +0.602 shuffled +0.003
lag 64 +0.432
lag 128 +0.140
The profile peaks at lag 8, decays to a zero crossing near 256, then rises
again to a second smaller peak at 512 before decaying. So records are related
within themselves and repeat every 64 records.
This settles the earlier "no constant substructure" reading. The records share
no fixed fields, but they are not independent either, and that was the thing
the distinct-value counts were failing to capture.
Slowly varying values, related within a record, repeating every 64 records, is
the shape of a keyframe or lookup table -- which is what sky data should be,
where a colour or fog parameter is sampled rather than computed per vertex.
Still not decoded: nothing names the values or the sampling. But the shape is
now known well enough for a decode attempt to aim at, rather than to hunt.
The previous commit said "a mean of 6.99 of every 8 bytes are zero". That
is inverted. 6.99 is the mean count of live bytes, and the records are nearly
full rather than sparse. The live-byte histogram is 0:15, 2:1, 4:31, 5:238,
6:461, 7:1030, 8:1114.
That error also invalidated the reasoning built on it in this session's next
message, where the high zero-count was used to explain why the differential
analysis found no constant positions. There was no zero-density to explain.
The differential result stands -- all 8 positions vary -- but for the plain
reason that the records are dense, not because of any artifact.
New finding from re-examining it: byte 7 takes only 14 distinct values where
bytes 0-6 take 142-204. Its common values are 0x00 (1378), 0xff (806) and
0x80 (592), which are the high bytes of signed 16-bit values. That is
consistent with byte 7 being an MSB rather than ordinary data. Unexplained.
Also tried and recorded as not helping: a signed 16-bit reading spans
-32743..32633, near the full range, with no per-position structure (533, 371,
401 and 187 distinct at positions 0-3).
Range measured: level 0, sky region +365760..+388880, 23120 bytes, read as
2890 records of 8. The matched-control autocorrelation puts the peak at
stride 8 (0.217 real, 0.000 shuffled), so 8 is the unit to describe.
Composition: 1185 distinct u32 values across 5780 words; a mean of 6.99 of
every 8 bytes are zero, so most records are sparse; and 549 of 2890 adjacent
pairs, 19%, are exact duplicates of each other. The most common 8-byte record
appears 283 times and is d7d7d780 d7d7d780 -- a 4-byte value stored twice.
Others follow the same shape: 00051980 00051980, 00082580 00082580.
Not decoded. Sparsity with repeated entries is what a keyframe or index table
looks like, but nothing here establishes that, and the duplicated-4-byte shape
may just as easily be two adjacent records sharing a value.
Recorded as a characterisation with its range rather than a finding, because
that is what it is. The useful part is that the next attempt does not have to
re-derive the composition to know whether a decode is plausible.
Range measured: every level's sky region, from the count field at +0x0c
through the sub-object offsets at +0x20, each sub-object over its first
16384 bytes.
Method: peak-stride autocorrelation per sub-object with a matched control --
the same bytes shuffled, seeded per level and index.
Field note: the sub-object count is a u32 at +0x0c, not +0x0a. Reading +0x0a
gives 327,680 on level 0, which is not a count. A brief probe used the wrong
offset and read as though it were a count of 5; it got the right offsets by
accident because the offset table starts at +0x20 either way.
Level 0 has five sub-objects at 365760, 388880, 396432, 406784 and 414032,
of lengths 23120, 7552, 10352, 7248 and 1536, each with structure well clear
of its control.
But the strides do not repeat across levels:
obj 0: 4 (x6), 8 (x13), 32, 288
obj 1: 16 (x3), 32 (x6), 288
obj 2: 8, 16 (x5), 256, 288
obj 3: 8, 16 (x6), 216
obj 4: 8, 16 (x3), 512 (x4)
So the per-index stride table read off level 0 alone was over-fitted to one
level, and is withdrawn. What is consistent is weaker and more honest: every
sub-object sits above its matched control, and most peak at 4, 8, 16 or 32,
suggesting 16- or 32-bit records whose stride varies with content.
Level 2's header is still genuinely different: its count field reads 196,617
and is unusable while every other level yields a plausible count.
The sky autocorrelation turned out to be inflated by a control that did not
match the data, so the 368 result was re-checked the same way rather than
being left on the evidence I now know to be weak.
Range measured: each level's post-shrub region start, first 65536 bytes.
Method: as before, but the control is the same bytes shuffled rather than
uniform random. Shuffling preserves the value distribution and destroys only
the ordering, which is what a control for this statistic needs to do.
Level 0's region is evenly distributed -- 256 distinct byte values, most
common only 15.6% -- so uniform random was the appropriate control here, and
the shuffled control independently confirms it, reading 0.0000 at strides 340,
368 and 4 against the real data's 0.1110 at 368.
Recomputed across all levels with matched controls: 24 of 24 peak at exactly
368, every one with excess above 0.05.
So the 368 period stands, now with a control appropriate to its data. The
contrast is the useful part: the same statistic, asked the same question of
two regions, gave a real answer in one and an inflated one in the other, and
the only thing that separated them was whether the control matched the byte
distribution.
The 256-byte period in the sky region stands, but the evidence I gave for it
was badly chosen and one figure in it was wrong.
The control was uniform random bytes, which read flat at 0.0000 and made the
0.772 match rate look like an enormous signal. It is not: 91.8% of the
region's bytes are the same value, 0x07, so a stride-256 comparison matches
trivially wherever both positions are 0x07.
The correct control is the same bytes shuffled, which keeps the value
distribution and destroys only the ordering. It reads 0.512 at stride 256,
not 0.0000. The real excess is +0.26 -- a genuine signal, and much smaller
than reported.
The extent was wrong for the same reason. Measured against the matched
control, the excess over the value distribution is:
+0 .. +49152 +0.38 .. +0.45
+53248 +0.26
+57344 +0.07
+61440 -0.13
so the periodic data ends between +53248 and +57344, rows 205 to 224, not at
+60160. The earlier number came from a raw threshold with no distribution
control, and a byte value covering 91.8% of the data drags such a threshold
to wherever the dominant run happens to stop.
So: the period is real and cross-checked, the image dimensions are not. Any
"256x236" from the previous commit should be discarded.
The general lesson is the one AGENTS.md asks for and this missed -- a control
has to match the thing being controlled for. Uniform random was the wrong
reference for data with a dominant byte value, and it made a weak result look
like a strong one.
Range measured: level 0, 2192576..2615232, the 422,656-byte sky region.
Periodicity start and end located by sliding an 8 KB window in 256-byte steps
through the region and measuring the 256-stride match rate.
Method: autocorrelation, then a periodicity scan. A random control of the
same size reads flat at 0.0000 for every stride tested.
The peak stride in the first 64 KB is 256 at a 0.772 match rate. Strides 255,
257 and 4 score nearly as high, which is what a genuine period looks like when
sampled rather than assumed -- a useful contrast with the 148-byte "stride"
that held for eight values and then broke.
Across levels the peak is 256 on 21, 512 on 5 and 128 on 1. 512 and 128 are a
harmonic and a sub-multiple of 256, so the fundamental period is 256 bytes.
Within level 0 the periodicity holds from +0 to about +60160 and then stops:
236 rows of 256 bytes, i.e. a 256-wide by 236-tall image at one byte per
pixel. The remaining ~362,240 bytes -- 86% of the region -- are not
256-periodic and stay unidentified. That is presumably what the layer offset
table at +0x20 points into.
The sky region is therefore not one thing. Its first ~60 KB is an image and
is the tractable half; the rest is still open, as is the non-uniform header
where level 2 differs from every other level.
Range measured: level 0, 16047040..16737344. 1875 whole 368-byte records
from the region start, variation checked across the first 60.
Method: the differential analysis that identified the shrub body -- compare
records byte by byte, record which positions differ.
Result: all 368 positions vary. No constant field, no padding, no header
inside a record. The shrub body separated immediately into a varying payload
and a constant two-byte tail; this format has nothing constant to separate.
That is consistent with the autocorrelation peak rather than in tension with
it. Checking which positions carry the 368-stride matches over 20000
positions gives 168 exact 4-byte repeats, distributed evenly at 0.84% across
all four byte offsets. Whole 4-byte sequences recur at exactly 368 while no
individual position is constant -- consistent with records containing entries
that repeat at a fixed relative index, not with a fixed-layout record.
So the 368 stride stands and the records stay unidentified. Worth recording
that the shrub method failed here for a specific and predictable reason: it
needs something constant to latch onto. Applying a method that worked
elsewhere is not the same as it applying here, and saying so is cheaper than
the next person assuming the technique transfers.
Range measured: level 0, 16047040..16737344, sampled as the first 65536
bytes. Cross-checked on every level that has the region, each sampled the
same way from its own region start.
Method: autocorrelation. For each candidate stride, count how often a 4-byte
sequence at offset i repeats at i + stride, as a fraction of comparable
positions. A random control of the same size measured identically reads flat
at 0.0000 for every stride, so the statistic does discriminate.
stride 368 0.1110
stride 340 0.0935
strides 337-339 0.026-0.029 background
random control 0.0000
Stride 368 is the strongest peak. On level 0, 340 is a secondary peak well
clear of the background; the two differ by 28, unexplained.
The cross-check is the part that matters: the peak stride was recomputed
independently for every level with the region, and all 24 peak at exactly
368. One level's peak could be a property of that level's data; 24 of 24
agreeing is a format invariant.
What a 368-byte record is remains unestablished. This is the same position
the shrub body was in before differential analysis identified its records,
and the same method is the obvious next step -- apply it to the 368 stride
rather than continuing to look for headers.
Chosen deliberately over the header search: autocorrelation found this in one
pass with a control, where four header guesses found nothing. The shrub body
only yielded to differential analysis too, and pattern-matching is what
produced this session's four retractions.
Range measured: level 0, 16047040..16737344.
Treating the ten entries' low halves as offsets from the region start lands
on blocks exactly 148 bytes apart, the first nine of which begin 08 00
followed by a 16-bit id stepping by 0x40. That reads as a pointer table and
it is a coincidence.
Walking the same 148-byte stride across the rest of the region, only 14 of
4657 blocks start with 08 00. The first nine agree, then the pattern breaks
completely -- id deltas go from a steady 0x40 to 0x801, 0xf510 and
negative values.
Eight consecutive values at a constant stride is much more than chance
should produce and much less than a structure needs. The lead committed an
entry ago is therefore disconfirmed, and the region remains unidentified:
real, non-zero data, bounded, and unopened beyond its first ~150 bytes.
The open question is unchanged and is where the next attempt should start:
the 690,304 bytes on level 0 between the last shrub class and the constant
trailer.
Turns the differential finding into code. Each shrub body record is 6
varying bytes plus 2 constant ones; body_records() hands them back raw and
body_is_well_formed() checks the trailing pair.
ShrubHeader::region now takes an explicit span_end. It has to: without it
the data slice runs to the end of the lump, so the last class in each level
absorbs whatever follows shrub and a per-class check ends up measuring its
neighbours. That is the same unbounded-span mistake as before, caught this
time by a failing assertion rather than by reading the docs afterwards.
The disc test asserts well-formedness only for the 334 of 358 classes whose
span ends at the next class. The last class in each level has no successor,
so its span contains the 690 KB region after shrub and its body is not
wholly records -- that is the open question, and asserting either way would
be wrong. Shrub::span_end_is_a_successor records which case a span is.
Records are summed only across bounded bodies. Summing the last class too
reproduces the "3.5 million records" figure retracted two commits ago, to
within rounding -- which is a useful check on how easy that mistake is to
repeat by accident.
8,938 records across the 334 bounded bodies; 16 denormal reserved words;
358 headers naming their own class.
Range measured: level 0, 16047040..16737344, the 690,304 bytes between the
last shrub class's body and the constant trailer. Nothing outside that range
is read.
Method: reading the bytes at +0x20 as u32 shows ten consecutive entries
whose low halves increase by a constant 0x94 (148), followed by a zero. The
high half is 0xf000 for the first nine and 0x0a40 for the tenth, so the
table is ten entries and ends at +0x48, absolute 16047112.
What the entries mean, what 148 refers to, and why there are ten are all
unestablished. Ten matches no count already known for this level -- 238
moby classes, 74 tie, 7 shrub, 93 tfrag textures -- so it is not yet tied
to anything else in the lump.
Recording it as a lead rather than a finding. The stride is a real
measurement on a bounded range, but a ten-entry table with an unexplained
stride is exactly the shape that looks like a structure while being
something else, and the rule in AGENTS.md is that a pattern gets looked at
before it gets named.
Range measured: each level's last shrub class data_offset field (class +
0x2c) through to the start of the constant trailer. Nothing outside that
range is read.
Method: the last shrub class's body is an array of 8-byte records whose
final two bytes are 00 00. Scanning forward in 8-byte steps for the first
record whose trailing pair differs locates the body's end. That is a
search for a stated pattern rather than a statistic, so it either finds
the boundary or says it did not.
Result: on 24 of 24 levels the boundary lands exactly on an 8-byte record
boundary, with body lengths of 192, 208, 224 or 240 bytes -- the same sizes
seen when shrub classes were grouped and diffed.
Level 0:
last shrub class data 16046800 .. 16047040 240 bytes
next region 16047040 .. 16737344 690,304 bytes
constant trailer 16737344 .. 16969296 231,952 bytes
So 690,304 bytes between the last shrub class and the trailer are a
distinct region rather than shrub data. What is in them is not established.
They are the leading candidate for moby_anim, now that collision..moby has
been ruled out as texture.
Worth recording how a wrong conclusion nearly happened here. An earlier
probe searched for PS2 NaN filler at the start of this range, found none,
and was about to report the body as having zero length. That would have
been wrong: the last shrub class is a real class with real data, and NaN
filler only appears in degenerate ones. Checking the actual byte pattern
instead of concluding from a failed heuristic is what caught it.
Four numbers in one session looked load-bearing and turned out to be
artifacts: a format identified as VIF-packed on the strength of recurring
byte patterns where a control put it on the random baseline, a body size
computed over wrongly bounded spans, a "6.2 MB of unidentified region" that
was texture data, and "3 unexplained blocks" that were a sampling artifact.
All four shared one cause -- measuring before bounding, naming bytes before
looking at them -- and none errored, which is why they survived long enough
to be believed. Two of them needed follow-up commits to undo.
The rule: any size, count or distribution comes with the byte range it
describes and how it was measured. Ranges come from core_layout rather than
from running to core_data.len(). A statistic that cannot separate your format
from uniform random cannot tell you anything. And counting distinct byte
values only tells you which blocks to look at, not what they are.
Worth having in the working agreements rather than in anyone's memory.
Comparing all 27 levels byte-for-byte from the end: the last 231,952 bytes
(0x38A10) of every level's core_data are identical. Level-specific content
therefore ends at len - 0x38A10, which is a real bound for the final region
and is not core_data.len().
This closes shrub. Across the 24 levels that have a shrub region the bound
gives 24,442,928 bytes total, and the disc test asserts on every level that
the region ends strictly inside the lump rather than at its end.
What the trailer is remains unestablished -- fixed data, or padding from a
fixed-size decompression buffer -- but it is constant, which is all a bound
needs.
The data between the last shrub class and the trailer is high entropy: 183 to
256 distinct byte values per 4 KB block, mean 254, where texture data sits
under 200. It is real content of unknown type and is the most likely home for
moby_anim, now that the range between collision and moby has been ruled out
as texture.
A lump shorter than the trailer now yields end: None rather than a backwards
bound, covered by a test.
The "3 of 21 blocks not texture-like" figure from the last commit was an
artifact of sampling every 256 KB and missing everything between. Re-scanned
at 4 KB granularity there are about 100 such blocks, and looking at them
rather than only counting them, they are smooth gradient ramps -- sky and
fog texture. A noisy gradient has a high distinct byte count without being
code, which is exactly the mistake the byte-count heuristic invites.
So collision..moby is entirely texture data. There is no separate moby_anim
region there, and two claims about this range have now been wrong: the
"6.2 MB of unknown data" and the "3 unexplained blocks". Both came from
sizes and counts taken without looking at what they described.
moby_anim is therefore still unlocated, but it is not here. The remaining
candidate is the unresolved tail after the last shrub class -- ~925 KB of
real, non-zero data on level 0 that nothing accounts for.
Recording the two corrections together rather than one at a time, because
the mechanism is the same in both: a heuristic count is not an
identification.
The previous commit claimed ~6.2 MB of unidentified data sat between the
collision header and the first moby class, and speculated it might be
moby_anim. That was wrong, and it came from quoting a size before bounding
the thing it measured.
Looking at the bytes rather than only the offsets: the range is dominated by
8-bit indexed texture data. Byte-value concentration distinguishes them
cleanly -- a texture block draws on a small palette, code and collision data
does not. 18 of 21 sampled 4 KB blocks fall under 200 distinct byte values,
against 249 for tfrags and 209 for collision.
The real layout on level 0:
collision 2615232 .. 3518544 (header plus its declared 903,248 payload)
textures 3518848 .. ~8832192
moby 8832192 ..
The texture table's data_offset values stop at 5326208, which is what made
the gap look genuine. Mip chains extend well past the last entry's offset,
so sizing a region from a table of first-level offsets understates the data
it points at.
Three of the 21 blocks are not texture-like, so something is mixed in that
is still unidentified. It is not 6.2 MB of moby_anim.
This is the third number this session that looked load-bearing and turned
out to rest on an unbounded measurement. The pattern is consistent enough
to be worth stating plainly: sizes derived from tables or offsets are not
evidence until the bytes underneath have been looked at.
Printing the region table next to the header fields makes the gap visible
without having to remember to consult core_layout, and it paid for itself
immediately.
On level 0 the collision region comes out 6216960 bytes, from 2615232 to
the first moby class at 8832192. Collision's own header declares a payload
of 903,248. So roughly 6.2 MB sits between the collision header and the
first moby class, and almost none of it is collision.
The successor rule is correct -- a region does end where the next one
begins -- but "the next" is not yet known there. collision..moby is
really collision + <something unnamed> .. moby. The name is wrong, not
the bound.
Candidates for the space, rough order of likelihood: moby and tie model
animation data, which is what moby_anim would need, then volumes, then
per-class runtime structures. Tfrag texture texels are already placed and
sit earlier in the lump.
Worth resolving before starting moby_anim, since that format is the most
likely tenant. Until then, any probe reading between the collision header
and the first moby class is reading something the region table does not
describe.
Region spans were being re-derived per format while probing, usually
running from one known offset to the next with the last span running to
core_data.len(). On level 0 that gave the final shrub class a 925,008-byte
span absorbing most of the lump's tail, and every aggregate computed over
it -- record counts, value distributions, "this looks random" -- was
silently wrong. Nothing errored, which is what made it survive two
commits and a correction.
core_layout computes the layout once: each region ends where the next one
begins, class regions start at the lowest non-zero geometry offset in their
table, and a region whose successor is unknown gets end: None rather than a
clamped guess. Zero class entries mean absent, not position zero -- the
same trap occlusion and collision each fell into separately.
The disc test asserts what the module is actually for: that the unknown is
visible. Shrub's end is unresolved on the 24 levels that have a shrub
region, and that is asserted rather than hidden.
It also found that not every level has every region -- level 10 has neither
occlusion nor shrub -- so the test checks ordering against the canonical
sequence instead of demanding an identical set.
Still open: where the shrub region ends. Texture texels cannot bound it;
they sit before moby, not after shrub. See docs/formats/core-data-regions.md.
A follow-up measurement contradicted the body size implied by the
differential analysis, and the cause is bad span bounds rather than
anything in the format.
Shrub spans were bounded by the next shrub offset, with the last running
to the end of core_data. On level 0 that gives the final shrub class a
925,008-byte span, absorbing most of the lump's tail. The moby-to-tie
boundary is clean by comparison: last moby offset 13414592, first tie
13416192, exactly 1600 bytes apart.
What that invalidates:
- The 3.5M record / 28 MB body figure is an artifact of the oversized final
span and should not be quoted.
- The "payload looks uniform-random" measurement ran over the same
contaminated range, so it says nothing about shrub geometry either way.
- The differential result itself still stands -- it was computed on groups,
and the groups used had bounded spans. The global aggregates built on it
do not.
What is needed is the offset ending the shrub region. It is not in the core
index, and there is no obvious successor boundary the way moby->tie has one.
This is the second confident-sounding number in this format that turned out
to rest on a bad bound, the first being the VIF reading. Aggregates across
many spans are only as good as the span bounds under them, and bounds are
the easy thing to get wrong because nothing errors when you do.
The previous commit concluded the shrub body's 6-byte payload was
VIF-packed, on the strength of recurring patterns like 7ea5 and 1ab6. That
was pattern matching dressed up as a conclusion, and a control kills it.
Decoding every 32-bit word of every shrub body as a VifCode:
shrub bodies 24.9% unpack, 128 distinct cmds
uniform random 25.1% unpack, 128 distinct cmds
sound_bank (definitely not VIF) 19.0% unpack, 122 distinct cmds
gs_ram (mixed) 13.6% unpack, 128 distinct cmds
moby class 0 span (known VIF) 40.1% unpack, 105 distinct cmds
Shrub sits on the random baseline and exercises all 128 possible command
bytes, which is what unstructured data looks like.
Worth being precise about what this does and does not show. The test is
weak -- even the known-VIF region only reaches 40.1%, well short of a
clean signal -- so this does not prove shrub is not VIF. It proves the
claim was never established, which is the part that was wrong.
Spotting three recurring byte values and naming them a format is the same
mistake as the three stride guesses it was meant to replace, one level
down. A hypothesis needs a discriminating test before it goes in a commit
message, and I skipped that step because the finding was welcome.
What still stands is the part that rests on differential analysis rather
than recognition: 8-byte records, 6 varying and 2 constant, 144 varying
bytes per group regardless of count. The payload's encoding is unknown.
Stride-fitting the shrub body failed three times, so this attacks it from
the other side: group spans by (count, data_offset, tail length) and byte
diff every member of each group. Positions that differ across a group are
geometry; positions that never differ are structure.
The answer is blunt. 144 bytes vary in every group, whether that group has
count 1, 2, 3, 5 or 6 and a tail of 192, 208 or 240 bytes. The varying
data is not sized by count at all, which is why three stride guesses were
fitting the wrong thing -- and why they kept looking nearly plausible.
Those 144 bytes fall in runs of 6, every 8 bytes. The body is an array of
8-byte records: 6 varying bytes and 2 constant.
The payload is not half-floats -- decoding as half3 gives 245.25 and
-168.375, which no shrub has. The patterns (7ea5, 1ab6, 43d9, af80) and
their repetition are VIF-packed data, the same quadruple-word packing moby
and tfrag use. Filler around it is the PS2 NaN 0x7fff0000 rather than
zeros, which is why the body read as noise in the earlier dumps.
So the open question is the VIF code assignment, not the stride, and the
decoder written for tfrag is the obvious place to start.
Extended the shrub header past the two self-identifying words. The first
four words are floats -- finite on all 358 spans -- though they do not
behave like a min/max bounding box, which orders correctly on only 146 of
358, so they are recorded as bounds rather than named. The word at +0x10
is positive everywhere, 3.0 to 20.0, exactly 5.0 on 281 of them.
The word at +0x14 is the interesting one. It is zero on 342 spans and the
bit pattern 04000000 on the other 16, which is a denormal of about 2.8e-45.
The Emotion Engine has no denormals, so hardware reads all 358 as zero.
That is only the second reading. My probe reported the field as 0.000
everywhere because it printed to three decimals, and the test caught it:
level 7 class 7 fails a strict zero check. The lesson is that "exact" in
a measurement means exact in the measured precision -- rounding in the
output made a real distinction invisible.
It also matches AGENTS.md on modelling the FPU rather than correcting it:
this field looks like noise on disc and is a constant to the machine that
reads it, and the parser accepts either rather than picking one.
The shrub body past +0x2c is still not decoded.
Shrub is the third class-geometry family and is not the moby one: all 358
spans on the disc have a leading word of 0 or one too large to index a
table into, so none contain a moby-style part table. The rule that holds
for 43,467 of 43,467 moby parts holds for zero of them.
Two header fields do self-identify, exactly, on all 358 spans: +0x24 equals
the owning class entry's own o_class, and +0x2c is a data offset landing
inside the span. Both are checked rather than merely reported -- a span
naming the wrong class is a different thing, and quietly returning the
mismatched value would hide that.
The body past +0x2c is not decoded. It tiles at no stride consistent with
the count at +0x28 (tails of 240, 240, 192 and 224 bytes against counts of
5, 6, 5 and 3), and the header's leading bytes are packed rather than
float even though +0x10 lands exactly on 5.0f on 281 of 358 spans.
Shrub class entries are 0x30 bytes rather than 0x20, because they carry
billboard texture fields the other two tables lack; the parser records that
so the stride is not re-derived.
Reading these formats by unpacking raw offsets out of core_index.bin has
cost real time. The index interleaves single-word region offsets with
count/offset pairs, so miscounting one shifts every field after it: that
produced a page of implausible collision headers on all 27 levels, sent
me chasing a tie-breakoff that turned out not to exist, and nearly had me
"fix" a stride bug in code that was already right.
probe dumps a level's core header and all three class tables through
LevelCoreHeader, so a field name printed is the field name the parser
uses. It also reports how many entries carry geometry inside core_data,
which is what pins the class table strides rather than guessing at them:
shrub reads 7/7 sane at 0x30 against 4/7 read at 0x20.
Reading a structure by hand in a shell still works; this just stops it
being the default, and stops the offset arithmetic being re-derived every
time a new region is investigated.
Checked the obvious hypothesis -- that tie, being the other class-geometry
table, would share moby's part-table format -- and it does not.
The tiling rule vertex_offset + 16 * vertex_count == next_part holds for
0 of 25,024 records across the 1,864 tie spans, against 43,467/43,467 for
moby. Tie's second field also reads as an absolute offset rather than a
span-relative one, which is the first sign the record shape differs.
Past the verified 0x48-byte header the bytes are f32-shaped rather than
the expected VIF marker, but they are not a fixed array either: the run of
consecutive plausible floats varies from 1 to 6 across spans (822 spans
have 6, 455 have 1, 373 have 2). So there is more header past 0x48 and
it is not one array of distances -- the floats after that point are a
different population from the LOD distances before it, including negative
values.
Recorded so the next attempt starts from that rather than re-trying the
moby template.
The three class tables do not share an entry size. Measured by how many
entries give a geometry offset inside core_data plus a plausible o_class,
over all 27 levels:
moby stride 0x20 5769 / 5796
tie stride 0x20 1864 / 1864
shrub stride 0x30 356 / 358
Reading shrub at 0x20 still yields 231 sane-looking entries out of 358,
which is the hazard: wrong often enough to look right. The committed code
already handles this -- ShrubClassEntry is 48 bytes and carries billboard
texture fields the other two lack -- so this is measurement, not a fix.
Shrub geometry is not moby's part-table format: all 358 spans have a
leading word of 0 or one too large to index a table into, so none contain
a moby-style part table.
Two header fields do self-identify, both exact across all 358 spans:
+0x24 equals the class entry's own o_class, and +0x2c is a data offset
landing inside the span. +0x10 reads 5.0f on 281 of 358, so it is a value
rather than a marker and carries no structural weight.
Rest of the shrub header undecoded.
A moby class span does not start with its geometry. The first u32 is the
offset of a part table, and the table is 16-byte records of vertex
offset, vertex count, an unestablished second count, the next part offset
and flags.
Parts tile: vertex_offset + 16 * vertex_count == next_part. The stride is
never assumed anywhere -- it falls out of that identity, which is what
pins the record size and the field boundaries. The identity holds for
43,467 of 43,467 part records across every moby class on all 27 levels,
and the parser rejects any span that violates it rather than returning a
plausible wrong answer.
Two terminations are needed. A zero vertex_offset ends the run, but 172
records on the disc have a zero vertex_offset and a zero next_part while
still carrying a vertex count. Stopping only on the first leaves 99.6%
instead of 100% and quietly includes a second table as if it were parts.
1,270,802 vertices over 4,560 class spans; 116 spans carry no geometry.
Vertices are 16 bytes of VIF-packed half-float quads cycling through four
distinct words. Handed back raw -- unpacking them is separate work from
finding them, and second_count is recorded but not guessed at.
Three things this session that are cheap to rediscover wrongly and
expensive to rediscover wrongly twice.
The core index interleaves single-word region offsets with count/offset
pairs, so gs_ram and the three class tables are two words each while
tfrags..collision are one. Treating the pairs as single words shifts
every field after them, which is how collision got read as sky and
produced a page of implausible collision headers on all 27 levels.
Sky is not uniform. Most levels share a header shape, but level 2 carries
extra fields and the count word at +0x0a over-reports against the number
of real offset slots, so no single fixed header fits.
Moby geometry starts with zero words rather than a header and then packed
half-float VIF data. Not decoded; recorded so the next attempt starts
from the data instead of re-deriving the layout.
Collision sits inside core_data at the offset the core index gives, behind
a 0x40-byte header that self-describes: it carries its own size and the
payload size. That pairing is the invariant -- the declared header size is
exactly 0x40 on all 27 levels and header + payload stays inside the lump.
41,481,520 payload bytes across the 19 levels that have any; 8 declare
zero length, which is present-but-empty rather than absent, so the two are
counted separately.
The cells inside the payload are not decoded. They do not walk as
fixed-stride cells, and treating faces as variable-length records stops
partway into the first cell on level 0 rather than tiling the payload. The
first cell reads as 11 verts and 33 faces under every hypothesis tried,
which is exactly the kind of agreement that looks like a result and is
not one -- three different strides all produce the same first cell and all
of them are wrong. Left open rather than committed as a guess.
A level's sound extent is two regions: an `SBlk` descriptor table and the
sample data that follows it. Four u32 at the head describe the layout, and
the invariant that establishes it is sblk_offset + sblk_size == daw_offset,
which holds on all 27 levels. That is what pins the field meanings; the
numbers alone are ambiguous about which of them is the offset.
DAW is a format tag inside the SBlk header, not a region signature. It
occurs exactly once in the entire extent, at sblk_offset + 12, naming the
sample format -- the sample data itself starts at daw_offset with no
signature at all, roughly 19 KB away on level 0. Reading it as a region
magic is the obvious reading and it is wrong.
daw_size never reaches the end of the file. The tail is positive on all 27
levels, 232 to 1,848 bytes, so the field describes less than the region
it names; the test asserts that direction rather than inventing a size
that fits.
523,720 SBlk and 37,649,008 DAW bytes across the 27 levels.
The sample format inside DAW is not decoded. It looks like VAG/ADPCM from
the nibble packing, and it has no bearing on drawing anything, so it stays
open rather than getting a half-answer.
visible() documented its argument as `byte << 8 | 1 << bit` -- high byte
selects the mask byte, low byte is already the shifted bit -- then shifted
the low byte a second time as though it were a bit index.
That is wrong twice over. It inverts the meaning of every bit argument,
and for any low byte above 8 it panics in debug, because `1u8 << 32` is
an overflow. Real data reaches it: the folded bits for the upper bytes of
a mask are exactly the values that trip it.
Treat the low byte as the bit mask, which is what the doc says and what
the renderer needs. Added an assertion for a mask index that does not
exist, since that was the case that exposed the mismatch.
A z -> y -> x tree of u16 node offsets over 4x4x4-unit cells, each
populated cell naming one 128-byte mask of 1024 visibility bits that the
renderer fills from the camera's cell each frame.
Two sentinels carry meaning and one of them is easy to miss: a z or y
slot of 0 means "no subtree", and an x slot of 0xffff means "no cell".
Reading the latter as a mask index gives a grid whose highest mask is
65535 on every single level, which is the tell.
The check that makes this real is the mask array. It starts at
masks_offset and runs to (highest_mask + 1) * 0x80, and on every level
with a grid that ends at the block boundary exactly -- widest
unexplained tail across the disc is 0 bytes. A wrong tree shape or cell
size would leave the highest mask index elsewhere entirely.
412,521 populated cells over 140,939 masks, across the 24 levels that
have one.
An offset of 0 means the level has no occlusion data at all, which is
not the same as "at the start of the file" -- that is where the tfrags
block lives, and parsing it as a grid produces a mask array that fits
anyway, with 1600 bytes to spare. That is what made level 10 look
plausible on the first pass; three levels (10, 24, 25) carry none.
Tie is the easy one of the three geometry subsystems -- instanced
static props, no skeleton -- so it is the right place to start.
RAC1's class header is 0x80 bytes. RAC2 agrees with it over the first
0x48 and diverges after. From 0x48 the RAC1 layout does not describe
RAC2 data: lod_info at 0x50 reads back as noise, packet_count and
texture_count are zero on every class examined, and the normals offset
at 0x0c points 117,788,434 bytes into a 16.9 MB lump.
Only the agreed-on prefix is decoded. What makes it trustworthy:
* o_class at 0x44 equals the core index's class number for all 1,864 tie
classes across all 27 levels. A field landing on the right number that
often is not a coincidence, and it pins the header base.
* The LOD distances are ordered 0 < near < mid < far for 1,784 of 1,864.
The 80 exceptions are real, unexplained, and are reported rather than
asserted away.
Level 0's first tie class has distances 12.0 / 25.0 / 40.0, a plausible
prop LOD band, and packet offsets 256, 20,592 and 26,048.
The rest is deliberately not guessed at. A field order that produces
plausible output and is wrong is the specific failure this project keeps
recording, three times now: the gameplay extent, the tfrag record
framing and the class tables were all "reasonable" and all wrong.
assets_compressed_size and assets_decompressed_size at index offsets
0x88 and 0x8c are 9,738,678 and 16,969,296 on level 0 -- exactly the
sizes of core_data before and after decompression. There is no separate
asset container. The "asset WAD" is core_data, and offset_in_asset_wad
indexes the decompressed form directly.
Checked rather than assumed, on all 27 levels: the decompressed size
equals core_data's exact length, every class offset is either 0 or inside
it, and the moby, tie and shrub spans do not overlap each other.
An offset of 0 means the slot has no geometry, not the start of the
lump; level 0's ninth moby class is 4909 at offset 0. Two assumptions
worth discarding while here -- the tables are not sorted by class number
(level 0's run 0, 1, 2, 14 ... 4909) and the offsets within a table are
not monotonic. Neither is needed, but both look true from the first few
entries.
The chunk descriptors after the named extents are not it: they exist on
only 3 of 27 levels, because level 0's four named extents tile the
container exactly. An asset blob there would be missing almost
everywhere.
What sits at the offsets is raw GS/VIF packet data rather than a further
container -- moby[0] begins 00 00 00 00 3f 16 02 55 6f 6f and tie[0]
00 01 00 00 70 50 00 00, which read as packet headers rather than a
magic number.
None of the three is decoded yet and this does not start that. Moby is
skeletal and is the largest single piece left before the rest of a level
can draw.
Also: the pipeline now writes each level's chunk descriptors where they
exist, and LevelCoreHeader decodes the asset lump's two sizes.
These were ruled out for want of an entry size. The size is knowable
exactly, because each table ends precisely where the next begins:
moby 238 entries at 192, 32 bytes: 192 + 238*32 = 7808 = tie's
tie 74 entries at 7808, 32 bytes: 7808 + 74*32 = 10176 = shrub's
shrub 7 entries at 10176, 48 bytes: 10176 + 7*48 = 10512 = textures
That closes on 27 of 27 levels, which is what makes it evidence rather
than a coincidence: any wrong entry size breaks the chain immediately.
It is the same "does this hold on every level" discriminator the project
already uses for the container pair table.
Across the disc: 5,796 moby, 1,864 tie and 358 shrub classes, plus the
1,595 tfrag textures. Class entries are 32 bytes -- asset offset, class
number, two unknown words and 16 bytes of texture slots -- and a shrub
entry adds 16 bytes of billboard slots for 48.
The header now decodes through to the texture tables rather than
skipping 0x18..0x30 wholesale, which also recovers the moby, tie and
shrub texture ranges. Two more pairs sit between shrub textures and the
textures base offset.
Two things worth recording. ClassEntry::parse read its 16 texture bytes
before its four i32s, because the struct literal that uses them comes
after the loop that advances the cursor -- so every entry was parsed
backwards. Struct literal field order does not reorder the reads that
already happened. And every shrub class on level 0 carries all-zero
billboard slots, so the dimensions are optional rather than something
worth asserting on.
This is the third instance of the same failure shape. The gameplay
extent was "an unidentified compression" because it was parsed
compressed; the tfrag record framing was "unrecoverable" because the
entry size was never known. Both looked like properties of the format
and were properties of the question.
Also records that point lights are a runtime feature rather than missing
work: every one of the 18,102 patches has point_lights == 0xffff and the
eight-slot bank is filled by gameplay, so there is nothing for them to
light in a static level render.
And that PCSX2 is not installed, so the comparison loop cannot be built
yet -- which is why lighting magnitudes stay unverified.
No behaviour change. This is the pass over the last stretch of work to
make sure the accumulated result hangs together rather than only each
commit in isolation.
Code. lit_color had grown a `let _ = (depth, lod_distances)` and a dead
branch from repeated edits. The directional pass genuinely does not
depend on distance, so the parameters are gone rather than discarded,
with a note that the point lights -- not implemented -- will need them.
lit_at folded into lit_color, which had become a wrapper around it.
Checked for leftover scaffolding: only two env vars remain, both real
(GC_ENGINE_SHOT and GC_ENGINE_DEBUG_BOUNDS); GC_ENGINE_UNLIT,
GC_ENGINE_DEBUG_COLOR and GC_ENGINE_ABL were never actually written,
because those edits silently failed to apply against rustfmt'd source.
AGENTS.md. The crate table did not say that gc-engine owns hardware
models. ps2.rs is the Emotion Engine's FPU and the lighting pass runs
on it, so the boundary is now stated: gc-formats decodes inputs,
gc-engine reproduces how the hardware computes with them. That is the
distinction rule 2 depends on, and it was implicit.
Docs. Three claims had been overtaken by what we learned and were
actively misleading:
* docs/formats/level-wad.md still said gameplay "does not decode with
the WAD decoder". It does; it is a WAD stream.
* docs/plan/level-assets.md carried a whole section, "a compression we
have not identified", with a ruled-out list including "bit order",
"header length" and "the WAD format at all". All three were ruled out
against the wrong input. Replaced, with the failure mode kept: a
uniform failure across 27 inputs was read as "the decoder is wrong"
when it meant "the data was handed over uncompressed", and the
positive control -- decompress and look -- was never run.
* The same file still said the tfrag "record framing is not recovered".
True, but it was a false lead that has since been superseded by the
VIF1 framing. Kept as a recorded wrong turn rather than deleted.
Also refreshed "Still open" to name what is actually outstanding:
point lights, the GS register set, the guard-band clip test, the
unread core_index class tables, and -- stated plainly -- that lighting
magnitudes are unverified because there is still nothing to compare
against.
Verified after the changes: 148 tests pass, clippy and fmt clean, all
5,213 extracted files match the hash table, and level 0 still loads at
26,758 triangles with the same LOD split and fully lit.
gameplay is a WAD stream, exactly like core_data. The extractor was
writing it compressed, so the light bank framing read the WAD header's
stream length as a light count: level 0's offset at +0x04 was 0xa0d
and the "count" there was 698810368.
The file says so in its first six bytes: 57 41 44 79 0d 0a -- "WADy".
Decompressed, level 0's gameplay extent is 1,872,256 bytes, the offset at
+0x04 is 0xc2b20, and the count there is 5. Five directional light sets.
All 80,274 vertices of level 0 now go through LightTfrags, and the
renderer's fallback line no longer appears. The frame's colour
distribution goes from 295 to 737 distinct sampled colours and takes on
the level's warm cast.
This corrects a recorded negative result that was simply wrong.
docs/roadmap.md carried "gameplay does not decode with the WAD LZ77
decoder, all 27 levels fail identically, so the input is wrong rather
than the decoder", with bit order and header length also ruled out.
That was true of the data being parsed and wrong about the cause: the
decoder was never the problem, it was being handed a file that had not
been through it.
Same lesson as the origin bug, from the other direction. A recorded
failure without a positive control is easy to record and easy to leave;
the control here is trivial and should have been run first -- decompress
and look.
The renderer keeps counting and reporting how many vertices it lit,
because a silent fallback to stored colours looks plausible and is wrong.
Lighting magnitudes remain unverified: that needs comparison against
reference frames via the PCSX2 loop, which is still to come.
Two pieces, and the first is the reason the second can be trusted.
ps2.rs is the EE's FPU on raw u32 bit patterns. It is not f32: there are
no denormals (exponent 0 *is* zero), no NaN and no infinity (exponent 255
is an ordinary value, so MAX reads back in Rust as NaN -- a test asserts
exactly that, because it is the sharpest available statement of why this
module exists), every result truncates toward zero, the adder discards
the small operand's bits more than one below the large one's LSB and
returns the large operand unchanged across a 25-bit exponent gap, and
rsqrt is sqrt-then-divide with truncation between the steps rather than
one rounded operation. The lighting pass accumulates all of these across
every vertex of every patch, so this is a visible difference rather than
a rounding curiosity.
tfrag_light.rs is the pass itself: PEXT5 base colour, the select word
picking one light set or blending two by bits 8..15 / 256, blended
directions re-normalised where unblended ones are not, and
base + A.rgb*max(dA, dA*wA) + B.rgb*max(dB, dB*wB). Lighting indexes
positions, not vertex-info entries, since a common entry and a LOD entry
can share a position.
TfragLight gains a `select` field for the u16 at +6. It was briefly
reconstructed with an unsafe pointer cast over the struct, which is
exactly the sort of thing that breaks silently the moment a field is
reordered; it is now a plainly parsed field.
The pass does not run yet. The light bank framing the reference uses --
a u32 offset at +0x04, an i32 count there -- gives offset 0xa0d and a
count of 698810368 on RAC2 level 0, so RAC2's gameplay extent is not
laid out that way. The renderer now counts the vertices it actually lit
and prints it, because falling back silently to the stored colours
produces a plausible but wrong picture:
lighting: 0 of 80274 vertices shaded by LightTfrags; the rest fall
back to the tfrag's stored colours (no usable light bank)
That line is the current honest state, and the frame has not changed as
a result. Where RAC2 keeps its light bank is the open question.
Normals are per-vertex, and the tfrag's light records store them as an
azimuth and elevation *index* into a 256-entry sine table rather than as
angles. This decodes that table and builds normals for the renderer.
The reference reads the table from an ELF at BOOT_NORMAL_TABLE_VADDR
(0x165500) and notes every level overlay carries a copy. The overlay has
no ELF header -- it is a raw loadable image starting with code -- so there
is no program header to map a virtual address through, and the offset is
not fixed either: level 0 has the table at 0xb650c, level 5 at 0xcca30,
level 26 at 0xb6284.
So it is located by content. Every table starts with exactly (1.0, 0.0),
angle zero, and the scan then requires all 256 entries to match
cos(i*pi/128), sin(i*pi/128). Validating the whole curve is what makes the
search safe: a lone (1.0, 0.0) pair of floats occurs by accident in a
2.7 MB executable, and a test plants one as a decoy. The values are read
from the disc rather than recomputed, since those are the numbers the game
indexes.
A normal is (cos(az)cos(el), sin(az)cos(el), sin(el)) *negated* -- the
stored normals point into the surface -- and then given the same Z-up to
Y-up swap as positions, or the lighting comes out sideways. The azimuth
and elevation fields are stored signed but index 0..=255, so entry 200
must be read as 200, not as -56; there is a test for that.
Also frames and parses the level light bank from the gameplay extent, and
the pipeline now writes overlay.bin. The per-vertex lighting evaluation
is not written yet, so the renderer stays unlit by default. GC_ENGINE_LIT=1
substitutes a host directional light for inspection -- that is a stand-in,
not the game's lighting, and it is deliberately not the default.
The hash table is regenerated for the 27 new overlay.bin files: now 5,213
files and 5,762,497,559 bytes.
The tfrag RGBA array now modulates the texture. It is indexed by
*position*, not by vertex-info: several vertex-info entries can share a
position, and the array is sized vert_count, which is a position count.
Colour follows the same rule as the vertex itself. Morphing blends the
position only -- the game keeps the high-detail colour -- while collapse
replaces the vertex outright, colour included. Getting that the other way
round would tint distant terrain.
The material is unlit, and that is a decision rather than a shortcut.
Level 0's colours really are dark (patch 0 runs (25,26,26) to
(108,144,141)) and the game's lighting is what brightens them, so what is
on screen is the true albedo.
Worth recording: the obvious move -- add a directional light, or crank
ambient brightness -- does nothing at all. With no normals the PBR path
is degenerate, so the brightness change has no effect, and a host light
would have hidden the gap rather than shown it. The colour data was
always reaching the mesh; what is missing is lighting, not a material
flag.
Written up in plan/level-assets.md, including what LightTfrags actually
needs: the light bank from the level's gameplay extent, the 256-entry
normal table at BOOT_NORMAL_TABLE_VADDR (0x165500) which every level
overlay carries, and the already-parsed azimuth/elevation light records.
Adds gc-engine's binary and a level.rs that joins the format layer to the
renderer: LOD selection and morphing per patch, then triangles grouped by
texture so each texture is one draw rather than one per triangle.
The PS2 world is Z-up at 1024 raw units per world unit, so the conversion is
(x, y, z) -> (x, z, -y). The sign flip on the third axis turns the PS2's
right-handed frame into Bevy's left-handed one rather than negating a
determinant, which is what stops the level rendering mirrored.
Morphing and collapse are kept distinct. Morphing blends a vertex toward the
midpoint of its two parents across its tier's band; collapse replaces the
vertex outright, position and UV, once it is past the far edge. Morphing is a
blend, collapse is a cut, and conflating them makes distant terrain
shimmer. Blending has to happen in raw units before the axis swap: mixing a
converted parent with an unconverted child scales the blend by 1024.
Level 0 renders as 26,758 textured triangles over 92 textures, correct UVs,
with LOD selection live (26 patches at LOD 0, 650 at LOD 2 from the start
camera). GC_ENGINE_SHOT writes one frame as a PPM and exits.
Rendering immediately exposed a bug no test could. Positions are i16 offsets
from a per-patch origin, and the obvious rule -- the last non-index STROW row --
is wrong on RAC2, which also uses STROW for per-vertex light records that come
*after* the origin. Taking the last one yields [186, 186, 186, 186] and
collapses the whole level onto the world origin. The entire level then fit in
a 63-unit box and every test still passed: vert_count matched, tri_count
matched, every parent and triangle index resolved, because all of those are
relative to the same broken origin. Only a screenshot showed it. The rule that
works is the STROW immediately preceding the first V3_16 position unpack; with
it, level 0 spans x 86..444, y 46..211, z -477..40, matching the patch
bounding spheres.
That is worth more than the renderer: a parser can be completely
self-consistent and still describe the wrong thing, and where a value is
absolute rather than relative it has to be checked against something
independent.
The frame is flat and dark because there are no normals and no lighting yet.
That is the next task -- the per-vertex light records -- not something to
paper over with an exposure fudge.
ReaperOfVeriod