The anchor 20655c7 said had to be verified before following. It is.
docs/AUG6-RETAIL-ANCHORS.md recovers UpdateCamera_6 and InitCamera_24 as class/role
identifiers in the two dispatch tables, so role 6 of lvl.camvtbl is the camera
update function. The decomp's addresses are in the same space as ours -- its
func_L00_002A3B80 is this overlay's .text dest exactly -- so the claim is checkable
rather than taken on faith.
Three checks, because PROVENANCE.md records that the decomp's function addresses
have failed once here:
1. Role 6 of level 0's lvl.camvtbl names handler 0x004284a8.
2. overlay_bytes_at reads real code there: 0x3c014094, a lui.
3. tools/overlay-anchors decodes the first six words with zero failures:
lui $at, 0x4094
ori $at, $at, 0x7ae1 ; $at = 0x40947ae1
mtc1 $at, $f1 ; a float into an FP register
lw $v0, 0x2290($s4)
lui $at, 0x3f00
mtc1 $at, $f3 ; 0x3f000000 = 0.5f
Two mtc1 into FP registers and a struct field at +0x2290: a function doing
floating-point work on a camera struct. Three agreeing checks are a different kind
of evidence from any one of them, which is the lesson of the moby_anim chase.
Caveat recorded: level 0's lvl.camvtbl is 0xb4 bytes, 13 records and 13 distinct
ids, but several ids read as implausible large values (4,350,904 and similar). The
record layout is not uniformly id/handler, so role 6 being clean may be luck. Worth
remembering when the other roles are read.
Next, unblocked: read all of 0x4284a8 and find the store that writes 0x18aa70.
That confirms m4 by locating the write site rather than inferring it from input
response, and the frustum should be whatever the same function writes nearby.
295 passing, clippy clean.
Checked upstream as asked. The vendored checkout moved from aca4f59 to 8a174cb,
18 commits ahead of origin/master.
What the update is: all 18 commits are src/game function-matching work. No header,
no docs, no config, and crucially no change to config/levels/functions.tsv -- so
the function extents and level catalogue this project depends on are unaffected.
What is usable: 14 of the newly matched functions fall inside the overlay .text
(0x2a3b80-0x447900), which makes them decomp addresses verified against the same
code this project runs:
002A7750 002AA800 002B3868 002B6120 002B7E28 002C7748 002CD8C0
002D5E98 002DDEC8 002DF1B8 002E7018 002F5220 002FC8D0 00300618
That matters because PROVENANCE.md records exactly this split: the decomp's struct
layouts were usable while its function addresses were not (they failed for the moby
class loader). Addresses that are now matched inside our own overlay are the
reliable kind, and 20655c7 needs precisely that to follow UpdateCamera_6 out of
lvl.camvtbl without repeating the mistake that cost the moby_anim chase.
What it does not give: none of the 14 is camera, frustum or GS code. The bodies are
game logic and toolchain-status material -- one is a header-offset relocation pass,
another raises a flag and forces a state word out of {6,7}. So the frustum hunt
does not advance from this, and the next step remains verifying UpdateCamera_6's
address against the overlay and reading the function that writes 0x18aa70.
Vendored checkout is gitignored, so moving it changes nothing committed except the
PROVENANCE note recording the pin and what was in the update.
Two more frustum searches, both unproductive, then the lookup that should have come
first.
Non-orthonormal 4x4 matrices within 4 KB of the camera, reasoning that the
projection must live nearby: 79 candidates, every one a sliding window over data
rather than a matrix. The filter does not discriminate and the region has no
obvious projection.
Then: the decomp names the camera. include/engine.h and
docs/AUG6-RETAIL-ANCHORS.md reference UpdateCamera_6, InitCamera_24 and
Camera_CollPrimTest, and docs/COMMUNITY-ENGINE-REFERENCE.md places lvl.camvtbl at
0x0026E900. The camera dispatch table is already characterised -- 217 records, 14
distinct ids on level 0, handlers in the overlay .text.
That is the right way in. The camera writes its matrices at 0x18aa70; the function
that does so is reachable from camvtbl, and the frustum is whatever that same
function writes nearby. Reading it would confirm m4 by finding the write site, and
pick up the frustum constants beside it, without guessing at float patterns.
The caveat that applies first, and why this was not done first: the decomp's
function addresses have already failed once here. PROVENANCE.md records that they
do not resolve for the moby class loader, while struct layouts transcribed from the
decomp were usable. So UpdateCamera_6's address must be verified against the
overlay before following it, not trusted from the name.
Resuming here is a different kind of task from the three searches above: follow a
known call site instead of pattern-matching a 32 MiB heap.
294 passing, clippy clean.
e7eec01 located R and the camera; the remaining operand is the frustum. Searched
with the same shape of filter that found the camera -- structural invariant first,
differential second.
First attempt discarded: scanning for a perspective-matrix signature returned
11,846 hits, which is noise and means the filter was worthless. Recorded because
the number looks like a result.
Second attempt, the right filter: unit-length 3-vectors at 16-byte alignment, the
invariant that identified the camera matrices. 9,514 found, 188 forming clusters of
four or more. The six-vector cluster at 0x189e20 is a frustum's exact shape -- six
planes, 96 bytes -- and it is not one. It reads as three orthonormal basis vectors
plus a zero affine row at 0x30 stride, with a translation at 0x189ea0 near the
camera: a table of positioned things.
So the frustum is not located, and I am not picking the six-vector cluster because
six is the number a frustum has. That is rule 4's failure with an exact number
attached to make it feel earned, which is what I have already committed twice in
this investigation.
What the search did establish: the structured regions near the camera are the nine
matrices at 0x18aa70, orientation records at 0x189e20, and larger clusters at
0x1793f0 and 0x18b260. The frustum is not among them as planes.
Which points at the next search, and it is a different one: guard_band's signature
is Frustum { near, far, tan }, not six planes. If the frustum is stored as
near/far/tan scalars, the search should be for a stable float pair with a
plausible ratio, not for normals at all.
294 passing, clippy clean.
d4e81ba left one question: the camera's rotation is one of the nine, but which.
Answered by turning the camera in the live emulator and watching which matrix
responds. Ten captures at ~0.6s intervals; angle through which each matrix's first
row rotated, from capture 1:
idx 1->4 1->7 1->10
m0 0.37 0.27 0.53
m1 1.31 1.07 1.34
m2 3.89 1.38 1.43
m3 2.25 1.18 1.40
m4 30.60 18.49 37.21 <-- the camera
m5 1.70 1.45 1.82
m6 1.24 0.98 1.29
m7 1.88 1.34 1.73
m8 1.70 1.45 1.82
The eight others are the control and they sit under 4 degrees across the whole
sweep while m4 goes to 30-37. Its motion is non-monotonic, tracking the input
being turned one way then back.
The translations gave the complementary half: all nine move together by the same
few thousandths. So the nine share one camera position and carry nine different
orientations, one of which is live.
guard_band step 1 now has both operands:
R rows 0-2 of the matrix at 0x18ab70
camera its fourth row, (247.1, 193.4, 50.4) on Aranos
NOT established: why there are nine. They are not nine cached frames of one camera
-- if they were, all nine would have swung by 30 degrees together. They are nine
distinct orientations about a shared point with one selected, and a table selected
by an index fits. The nine-legal-orientations-on-a-spherical-planet reading stays
plausible and unproven; what is proven is which entry is live. No reindexing was
observed, so if the index moves it does not on camera rotation alone.
294 passing with the disc and savestate present, clippy clean.
dcd4dd6 said three 4x4 matrices at 64-byte stride and read them as a
view/modelview/lighting family. There are nine. Contiguous from 0x18aa70, breaking
at 0x18acb0 where row norms are 0, 0, 0.97 and w = 0.
All nine are orthonormal to 1e-6, all affine, all translating inside a 0.4-unit
box, all moving smoothly between live frames. Nine different rotations about one
position is not a view/projection/lighting family -- it is a table indexed by
orientation, and the index is what is unknown.
So dcd4dd6 was right that the region holds the camera position and wrong that
three was the extent. It stopped at three because three read as a sensible story.
That is rule 4's failure and I have now committed it twice in this investigation.
What survives unchanged: the translation is the camera position, (247.3, 193.1,
50.5), verified by orthonormality plus the smooth-motion control. The rotation
changes far faster than the translation between frames, which is what identifies
which part is which.
What is now open and larger than it looked: which of the nine is the R that
guard_band step 1 wants. Not "pick m0". Nine orientations about one point is a
table with an index. A plausible reading -- that these are the nine legal camera
orientations on a spherical planet -- would be a real find and is NOT established
by anything measured here.
Two cheap checks, both needing only the live emulator: move the camera and see
whether all nine translations move together or only one; and change pitch to see
whether the nine rotations reindex.
The test now checks all nine rather than three, so a tenth would be noticed.
294 passing with the disc and savestate present, clippy clean.
With PCSX2 on Aranos the savestate problem went away. PINE reads EE RAM live, so
two dumps seconds apart are consecutive frames by construction -- exactly what the
offline savestates could not provide (1aa94ee).
Of 1,654 changed 16-byte float triples between two live dumps, most singletons are
mob positions moving 4-5 units. One region is three 4x4 matrices at 64-byte
stride, each with an affine fourth row:
0x18aa70 0.0338 -0.1983 0.9796 0.0000
-0.0275 -0.9799 -0.1974 0.0000
0.9990 -0.0203 -0.0386 0.0000
247.2990 193.1330 50.4526 1.0000
0x18aab0 t = 247.239, 193.102, 50.497
0x18aaf0 t = 246.940, 193.155, 50.151
The control that makes this an identification rather than a coincidence: all three
upper 3x3 blocks are orthonormal to within 1e-6 -- row norms 0.999995..0.999998,
pairwise dots at or below 1e-6. Three consecutive float triples in a random 32 MiB
region being unit length AND mutually perpendicular does not happen twice. The
residual is f32 rounding, expected on an FPU that is not IEEE-754.
Camera position on Aranos: (247.3, 193.1, 50.5).
camera_transform_is_orthonormal_with_a_moving_translation asserts orthonormality,
w == 1, and that the translation moves smoothly but not wildly. That last half is
the check 1aa94ee needed and the savestates could not supply: consecutive frames
give 0.002-0.005 of movement, against ~230 between unrelated savestates.
What this unblocks: guard_band step 1 is R * (bsphere - camera * 1024). Both
operands now exist. R is rows 0-2 of 0x18aa70 and the camera is its fourth row, so
the clip test can run against a real frame for the first time.
Not assumed, and recorded as open: which of the three matrices step 1 wants; whether
the float translation needs converting to the disc's 12.4 fixed point first (the
algorithm multiplies by 1024, which implies it does); and the frustum planes, which
the guard band needs and which are not in this structure.
Checked the docs for claims this session retracted. Found more than the first pass.
1,163 -> 1,162 9 in the format spec, 3 in the roadmap, 5 test doc-comments
5,878 -> 5,822 2 in the spec, 3 in test doc-comments
234 -> 233 1 in the spec, 1 test doc-comment -- level 0's contribution
The last one is the one a blanket search would have missed. "5,878 records, and
level 0 contributes 234" was stale in both numbers, and only one of them was in
the string I was looking for. Verified against the tests rather than against the
other stale prose:
lvl.vtbl: 1162 distinct class ids from 5822 records; max 0x13ba (5050);
moby_classes.count is 238, and 1135 of 1162 ids (97.7%) are at or above it
vtbl oClass: 1162 ids, largest 5050; ... 1133/1162 ... level 0 alone
contributes 233
The reason all of it survived is the same shape as the bug it belongs to. The
tests were fixed and the new numbers asserted, and the prose was not touched -- and
the prose is what a reader takes away. A commit that corrects a measurement without
correcting the document it was measured in has only half-landed. Two corrections in
this session, 15 places.
Also left alone deliberately: the explanatory sections in core-data-regions.md that
discuss 5,878 and 1,163 as the *wrong* figures, and the test comments that record
what the loop used to print. Those are correct as written -- removing them would
erase the reasoning. A final sweep confirms nothing stale remains outside those.
293 passing with the disc and savestate present, clippy clean. Documentation and
comments only; no behaviour changed.
Checked the docs for claims this session retracted. Found a real one: the vtbl
terminator fix (cc52060) corrected the distinct class-id count from 1,163 to 1,162,
and nine places in the format spec and two in the roadmap still said 1,163.
docs/formats/core-data-regions.md 9 occurrences
docs/roadmap.md 2 occurrences
The reason they survived is the same shape as the vtbl bug itself. The test was
fixed and the new number asserted, but the prose kept the old one, and the prose is
what a reader takes away. A commit that corrects a measurement and does not correct
the document it was measured in has only half-landed.
Checked the numerators before touching them rather than blanket-replacing: 1,135 of
the ids are at or above level 0's moby_classes.count, and 1,133 are at or above the
largest per-level table. Both were already correct -- only the denominators were
wrong -- so a find-and-replace across the number would have corrupted two correct
figures. The tests confirm:
lvl.vtbl: 1162 distinct class ids from 5822 records; max 0x13ba (5050);
moby_classes.count is 238, and 1135 of 1162 ids (97.7%) are at or above it
Left one deliberate occurrence of "1,163" in the spec, in a note explaining that the
figure was 1,163 until the terminator fix and why, so the correction is traceable
rather than looking like a typo.
293 passing with the disc and savestate present, clippy clean. Documentation only.
1aa94ee moved the camera hunt behind a PCSX2 session, because no two savestates on
hand are consecutive frames. But the camera is only part of this item -- the test
itself is a pure function over a camera and a frustum, so it can be written and
tested now, and then drop straight in when the camera is found rather than being
written under time pressure later.
New gc_formats::guard_band implements docs/plan/tfrag_rac1.md 3b.2 steps 1-7,
returning Cull, Normal or ClipAtLod0. Seven unit tests: directly ahead, behind
camera, beyond far, straddling the side planes, all-corners-outside-one-plane, the
integer sqrt, and a rotation.
This is not speculative plumbing. The inputs are already parsed and verified:
TfragGeometry::bsphere, clip_corners (8 corners, median 1.000x the bounding radius
from their centroid) and draw_mode (matching the reference table).
WHAT THE FIRST VERSION GOT WRONG
The reference culls when ONE plane has every corner outside it. I first wrote
"every corner is outside some plane", which culls patches that merely straddle
*different* planes. The test caught it at once: a patch whose corners were beyond
far in one axis and the side plane in another returned Cull where it should be
ClipAtLod0. Fixed by carrying a per-corner 4-bit mask of which planes it is
outside and testing each plane across all corners.
That is what the guard band is for. A patch outside the frustum but inside the
guard band must still draw, via the clipping program, so objects partly off-screen
do not pop. Collapsing the two turns "4x guard band" into "immediate cull", and
the symptom is objects vanishing at the screen edge rather than any error.
NOT VERIFIED AGAINST THE GAME
Steps 1-4 are implemented from the reference and have not been compared to a real
frame, for the reason above. The unit tests check the algorithm implements the
documented rule, not that the rule is what the game does. Two scales are my
reading of the fixed-point convention and are NOT established: tan = 1024 for 1.0,
and the clip-cube scale of 64. Both are flagged in the module doc so they are not
mistaken for measured facts.
293 passing with the disc and savestate present, clippy clean.
Ran the cheapest confirmation from a991088 on the candidate:
[247.000, 194.000, 49.890] step 0.000
[293.960, 411.810, 115.000] step 232.133
[506.306, 311.614, 116.656] step 234.804
[300.170, 227.480, 128.000] step 222.933
Rejected. A camera between consecutive frames moves a little; this moves ~230 units
per step and is not monotone on any axis. It is per-frame state but not a camera
position. a991088 wrote the prediction down before the measurement, so the
candidate is out rather than reinterpreted.
The more useful consequence: these four savestates are not consecutive frames, so
smoothness cannot filter the 2,005 candidates until consecutive frames exist.
Both cheap confirmations need something the savestates do not provide -- consecutive
frames, or an actual camera perturbation. That moves the camera hunt from offline
to needing a PCSX2 PINE session, the same category as the tfrag morph.
Worth noting the shape of it: the differential produced 2,005 candidates and the
confirmation reduced them to zero in one step, because the confirmation was
written before the measurement was interpreted rather than after.
286 passing with the disc and savestate present, clippy clean.
The guard-band clip test's blocker is named in core-data-regions.md: the algorithm
needs the camera and the frustum, both per-frame, so it needs a savestate. Two
savestates give that as a differential -- anything that moved is per-frame state --
without needing to know where the camera lives.
Frames .01 and .04, 33,554,432 bytes each. Of every 16-byte-aligned triple of
finite floats with magnitude 0.001..100,000, 2,005 differ.
Two shapes, clearly different things: the 0x1886xx block repeats one triple four
times at 16-byte stride, which is a per-object array, and 0x18aa70 holds a
near-unit vector that reads as a direction. 0x152c70 is a singleton,
[247.000, 194.000, 49.890] -> [300.170, 227.480, 128.000], in the range Aranos
occupies.
Not claimed as the camera. 2,005 candidates is not a shortlist and a plausible shape
is not evidence -- this stretch has produced enough of those.
Three confirmation steps recorded, cheapest first: smooth motion across .01-.04, a
singleton rather than an array element, and perturbing the camera in an emulator to
watch that one address change. The first two need only the savestates already on
disk; the third is what the lighting verification used.
286 passing with the disc present and GC_SSTATE set, clippy clean.
Picking up "the GS register set the level load builds from the texture entry"
from level-assets.md, for a renderer that emulates the GS rather than resolving
texture addresses on the host.
gc_formats::gs gains tex0_of (TBP0, TBW, TW, TH, PSM), texture_slot_bytes and
clut_base. Three measurements back them, over all 27 levels and 1,595 textures.
Every dimension is a power of two: 1,595 textures, zero exceptions, in four
classes -- 128x128 (1,249), 64x64 (303), 256x256 (22), 32x32 (21). The GS
addresses textures by exponent, so TW and TH are log2 of the dimensions, and a
texture that was not a power of two could not be addressed at all. That the
whole population is one is the measurand for the decode.
TBP0 is region-relative. 19 of 27 levels have a textures_base only 64-byte
aligned -- mod 256 reads 0, 64, 128 or 192 -- while every entry's own
data_offset is 256-aligned on all 1,595. A base pointer computed from core_data
would therefore not be a whole number of 256-byte blocks on those levels. The
first version of tex0_of took textures_base + data_offset and rejected 1,168 of
1,595 textures for exactly this reason, which is how the finding surfaced.
The slot is level 0 plus one mip level: the stride between consecutive entries
is w*h + (w/2)*(h/2) on 1,560 of the 1,564 ascending gaps -- 32x32 gives 1,280,
64x64 gives 5,120, 128x128 gives 20,480, 256x256 gives 81,920. One byte per
texel. `ty` reads 3 on all 1,595, so it is not the stored level count.
Four gaps break the rule and four more run backwards. Both are reported as
bounded exceptions rather than folded into an average: the four ascending
outliers are 712,960 to 1,064,960 bytes, two orders of magnitude past any slot
the game allocates, so they are boundaries the slot rule does not describe
rather than slots that came out different. The four reverse gaps mean the table
is not sorted by data_offset, which a consumer walking the region in table order
would trip over.
The CLUT address is the established palette * 0x100, restated as clut_base
because a GS emulator asks for it as TEXCLUT's CBP.
Still open after this: TEXCLUT's own fields, MIPTBP1/2 wiring, and how the
ad-gifs' A+D values relate to this base.
Suite with the disc: 70 disc tests, 160 gc-formats, 22 gc-trace, 23
gc-engine, 19 gc-game. Clippy clean.
TfragHeader carried a field named `cube_ofs` at +0x2e. R&C1 documents that
slot as `msphere_ofs` and puts the 8-corner clip cube at +0x38. The disc is on
R&C1's side for both.
Reading +0x2e as an array of `msphere_count` (+0x2c) Vec4f entries gives 833 of
833 well-formed arrays: four finite floats per entry, every radius
non-negative. +0x2c takes 4, 3, 5, 6, 16 and others. Reading that same slot as
eight s16 clip corners gives data that is not corner-shaped, which is how the
error was visible at all.
The real cube at +0x38 is non-zero on 833 of 833 headers and its eight points
sit inside the patch's own block on every one.
So: `cube_ofs` at +0x2e is renamed to `msphere_ofs`, the count it pairs with is
parsed and the array is read into TfragGeometry as `mini_spheres`, and
`clip_cube_ofs` at +0x38 is new, read into `clip_corners`.
What the corners are, measured over all 833:
corners, all with 8 distinct x values 833 of 833
|centroid - bsphere centre| / r median 0.147, max 0.595
corner distance from centroid / r median 1.000
The median is exactly the bounding radius, so the eight points lie on the
bounding sphere. They are not an axis-aligned box: every tfrag has eight
distinct x values, so if it is a box it is oriented. Whether they form a cube
needs the corner ordering, which is not recoverable from the data -- assuming
opposite pairs are (0,7)(1,6)(2,5)(3,4) matches body-diagonal == 2r on only 99
of 833, so either that pairing is wrong or the array is not a cube. Recorded as
unknown.
This was picked to unblock the guard-band clip test from the roadmap. The
algorithm is from the vendored R&C1 reference and is recorded with its
attribution: camera-space sphere, near/far truncation, side-plane cull, then
the 8-corner re-test against the frustum unscaled and against a volume with xy
x (0.25, 0.25) -- the 4x guard band -- with any corner outside the guard volume
sending the tfrag to the clipping path at LOD 0 with no morphing. draw_mode
already matches that table. What the test still needs is the camera matrix and
the frustum as a clip volume, both per-frame state, so the camera has to be
pinned at a savestate. The geometry side is now parsed and asserted, which
leaves only the camera side outstanding.
The s16 x 64 corner scale is R&C1's documented one and is not verified against
the game; the assertion covers that the parsed corners sit inside the patch's
own block and lie on the bounding sphere, not that the scale is right.
Suite with the disc: 69 disc tests, 156 gc-formats, 22 gc-trace, 23
gc-engine, 19 gc-game. Clippy clean.
The morph was recorded as unverifiable because "reading a weight back out of
RAM does not work -- the positions live inside regenerated VU packets". That
is right about the correspondence and wrong about the cause. The cause is
measurable, and it comes back clean:
patches 833
positions 40,600 disc == ram
positions identical 40,600
block bytes differing 166,465
inside a position unpack 0
inside another unpack 49,484
in a VIF command word 116,981
Range: level 0, all 833 tfrag patches, live PINE RAM against the disc. Two
independent measurements agree. Walking the VIF lists on both copies and
comparing parsed positions gives 40,600 identical and zero changed; a raw
per-byte diff of each patch's own extent classifies every differing byte, and
none of them is in a position unpack. The position ranges total 490,224 bytes.
So the blend is not precomputed into the geometry. The rewrites are packet
framing -- 70% command words -- plus other unpack payloads, which is the loader
rebuilding the VIF packet for the tier it selected while leaving the vertices
alone. The morph happens at draw time in the VU stream.
The standing note becomes a sharper one: there was never a correspondence
problem. The values were identical, so there was nothing to read.
gc-trace gains verify-morph, which does both measurements. TfragGeometry
gains position_ranges and other_ranges so the walk needed to classify a diff
is already paid for, and the ranges come from the same walk the rest of the
parse uses rather than a second one that could drift.
Per-patch extents are the gap to the next patch's data pointer. That is exact
and needs no guess about which header field marks a patch's end; the first
attempt at this compared each patch against the whole rest of the region and
reported 65 million differing bytes, which is what an unbounded comparison
looks like.
tfrag_positions_are_verbatim_and_the_rewrites_are_elsewhere is gated on
GC_SSTATE and asserts the four bounds, plus two guards: the rewrites must be
non-trivial (over 100,000 bytes) and mostly command words, so a RAM image that
was a verbatim copy of the disc -- which would make the zero look good for the
wrong reason -- fails instead.
Suite with the disc and GC_SSTATE: 68 disc tests, 156 gc-formats, 22
gc-trace, 23 gc-engine, 19 gc-game. Clippy clean.
Picking up after e5cd3df, which retracted the 16-byte tie vertex element for
want of a detectable stride. There is one, at 64 bytes, and it beats a shuffle
control by a factor of ten:
regions with a signature 7,314 of 15,219
same test on shuffled bytes 731 of 15,219
stride 64 6,463 of the 7,314
Range: every tie vertex region on all 27 levels, 15,219 regions. A byte offset
counts as constant at stride s when every unit of length s agrees there, with
at least three units, an exact divide, and at least eight constant bytes.
Dumping the regions shows four *different* 16-byte layouts repeating at
+00 +10 +20 +30: one is a small i16 followed by two zeroes, one carries 255,-1
mid-element, the other two carry neither. So the element is 16 bytes and the
layout cycles with period four. That is a partial retraction of a retraction:
e5cd3df was right that the four layouts differ, and right that pooling them
spans the whole i16 range. It was wrong to conclude that 16 is not the stride.
Four tests still do not separate the four layouts, and the table records all
four because the negative is the finding: LOD1/LOD2 sharing is 16.1/16.2/16.2/
16.3 percent across the four groups, f32-in-band is 23 percent in all four
(which is what uniform random gives), and i32-in-band is 0/2/6/0. The position
stays unnamed.
Also clears the clippy warnings that had accumulated on main from the disassembly
work: doc_lazy_continuation on three prose blocks, two negated-comparison guards
in gs.rs, a doc-then-item blank line twice over, a useless usize::from, and an
as_chunks that needs 1.88 while the workspace declares 1.85 -- replaced with
chunks_exact rather than moving the MSRV.
Clippy is clean. Suite with the disc: 67 disc tests, 156 gc-formats, 22
gc-trace, 23 gc-engine, 19 gc-game; without it, 67 disc tests skip early and
report the same count.
Tried to answer whether the decomp extents land on CPU code by checking for a
prologue. Only 29 of 3,835 matched 'addiu $sp,$-N' or 'lui $gp' -- but the other
bucket is full of textbook prologues my two patterns miss, because PS2 leaf
functions save registers with no frame adjustment:
0x2a4ef0 ffb00000 ffb10008 sw $s0,0($sp); sw $s1,8($sp)
0x2a48f0 3c013f80 44816000 lui $1,0x3f80; bc1t
Not reading a result into that.
Separately, ffb00000 has major opcode 0x3f, unused in MIPS32, yet is
unmistakably a register-save prologue and ffbf0020 for 'sw $ra,0x20($sp)' appears
throughout this code. Either the EE encodes sw outside MIPS32 or I am reading the
opcode field wrong somewhere. Not resolved -- and it undercuts every opcode-based
check in this investigation, the 391 and the neighbour ratios included.
Position after twenty-one commits: the mapping is verified, two targets are VU
microcode, and the candidate windows are not established to be CPU code. What this
needs is one authoritative disassembly of one extent compared against
tools/overlay-anchors, to establish what the opcode field means. That is the same
mistake one level down from the section-indexing bug: using a decoding never
checked to produce results, then measuring those results with the same unverified
decoding.
286 passing with the disc present, clippy clean.
391 candidates on correct bytes, 73 under 16 immediates. The densest, 0x2dca20,
164 bytes with three distinct immediates, disassembles through tools/overlay-anchors
as:
xori.b $w0, $w0, 0xa2
ext $v0, $s0, 0, 1
addu $a2, $v0, $a2
xori.b and ext are VU1 mnemonics and three-operand addu is not MIPS32. Capstone is
decoding COP2: 0x2dca20 is VU microcode. Same answer 0x2ee138 gave.
That calls the frequency result into question more seriously than the displacement
did. The measurement filtered on opcodes excluding COP2, so VU cannot explain the
391 counts directly -- but the function extents come from the decomp's CPU
analysis, and if an extent spans a VU region the window scanned is mixed and the
immediate set attributed to that function is not a function's.
Status: the enrichment is real as a measurement of those windows; what the windows
are is now doubtful.
Next: verify decomp extents land on CPU code before trusting any of them -- decode
a sample and require a plausible prologue. Same shape as the cd16485 check, and the
fourth time a 'real shortlist' has needed one more verification first.
286 passing with the disc present, clippy clean.
Redone through overlay_bytes_at, the verified flat mapping:
functions kept 2,313
moby_classes 0x00c0 391 (displaced: 394)
best neighbour (0x00c8) 15 (displaced: 4)
bucket n 0xc0 best neighbour ratio (displaced)
0-15 1,033 73 2 36.5x 72.0x
16-31 759 118 4 29.5x 29.8x
32-47 283 89 2 44.5x 34.7x
48-63 118 41 3 13.7x 19.5x
64-79 55 32 2 16.0x 15.5x
80-95 37 22 2 11.0x 13.0x
The reason it survives is worth stating, because it separates the two results. The
displacement was uniform -- every window shifted by the same amount -- so a
function's contents barely changed, and a frequency count over a window is robust
to that. A disassembly is not: it depends on the exact byte alignment, which is
what was wrong. Hence 394 becoming 391 with the same shape, while the 0x3a8060
walk and the 16-byte stride evaporated entirely.
So the investigation was not "all wrong". Precisely:
- the frequency enrichment is real, now on verified bytes, with a test that fails
if it goes;
- the disassembly was never real, and the whole jal chain built on it is retracted;
- no loader has been located, and 0x2ee138 is VU microcode.
THE LESSON IS ABOUT WHICH MEASUREMENTS SURVIVE A MISTAKE
A count over a window tolerates a constant displacement. A decode does not. Sixteen
probes leaned on the kind that does not tolerate it, and reported each with the
same confidence as the kind that does. Nothing about the numbers distinguished them
at the time -- that is the whole difficulty, and why the verification in cd16485
was the missing step rather than a formality.
Next: the ~391 functions are a legitimate shortlist, but consuming them means
disassembly through tools/overlay-anchors, whose output is now checked against a
word the build pins.
286 passing with the disc present (23 + 19 + 66 + 156 + 22; disc binary 34.89s),
clippy clean.
Re-read through overlay_bytes_at, the verified flat mapping from cd16485.
0x2ee138 is VU1 microcode: five COP2 instructions and a jr $ra stub. Nothing you
jal into, so the jal read from 0x3a3d70 cannot be right either -- and that
instruction was itself read from displaced bytes and must be re-read.
0x3a8060 is not what cee17c9 said. On correct bytes there is no addiu r17, r18,
0xc0, no 16-byte-stride walk and no function prologue; the first word is opcode
0x1f, which is not a MIPS user opcode. cee17c9's central finding does not exist.
State: overlay_bytes_at verified; 0x2ee138 is VU microcode, verified; the 0x3a8060
walk retracted; the frequency and neighbour-enrichment numbers not re-measured and
not quotable as exact; the e1482ab jal chain unverified, built from displaced bytes
and pointing at now-known VU code.
What is left of moby_anim is nothing. No loader located, one frequency result to
redo from scratch on correct bytes, and no reason to expect it to survive.
285 passing with the disc present, clippy clean.
The fix e6b3e0b called for, with the verification that six commits kept deferring.
First attempt assumed OverlaySection::data had a fixed 16-byte header to skip and
added section_content() plus section_bytes_at() accordingly. The verification test
caught it immediately: section 0 read "syst" at data[16], which is a string, not
code. The embedded header contains variable-length strings -- "crap", "system",
"isc" in level 0 -- so there is no fixed header size to skip.
What is actually true: the overlay lump is a MEMORY IMAGE. Its first word is
0x001A7C80, the load address of .lit, and overlay[addr - LIT_BASE] is the byte at
addr. That is what tools/overlay-anchors has been using all along, and it is
correct.
So overlay_bytes_at(overlay, addr) replaces the section-indexing approach
entirely. Bounds-checked, so it cannot read past the image.
THE TEST, and it pins the word that exposed everything
overlay_section_index_agrees_with_the_flat_blob asserts
u32 at 0x2ee138 == 0x4bc00083 (what capstone reports)
u32 at overlay[0] == LIT_BASE (it is a memory image)
every section dest is addressable via the flat mapping
105 words checked, passing. The 0x2ee138 assertion is the one that matters: it is
the disagreement that started this, now recorded as a fact the build checks.
What this does not do: re-read 0x2ee138, or redo any of the sixteen probes. Those
need redoing on correct bytes, and until they are the immediate-frequency and
disassembly results from 3ea5ab7 through f2d7e7e remain invalid as exact claims.
Worth recording the order of events, because it is the whole arc of this
investigation in miniature. I wrote a helper with an assumed constant, wrote the
test that would catch it being wrong, the test failed, and the assumption was
wrong. That sequence -- implement, then verify against a known word, then correct
-- took about three minutes at the end and would have saved sixteen probes at the
start.
285 passing with the disc present (23 + 19 + 65 + 156 + 22; disc binary 34.94s),
clippy clean.
The verification f2d7e7e asked for. Reference case: the tool disassembles
0x2ee138 as 0x4bc00083; reading the flat overlay at 0x2ee138 - LIT_BASE gives
83 00 c0 4b, little-endian 0x4bc00083. Match. Correct mapping is
off = ram - LIT_BASE indexed into the whole overlay.bin.
What I used in every probe: off = addr - secs[6].dest indexed into secs[6].data,
and OverlaySection::data includes the 16-byte section header. Every read was
displaced.
Invalidated:
- cee17c9's addiu r17, r18, 0xc0 disassembly, the 16-byte-stride walk, and the
'address not float' conclusion. Built on wrong bytes.
- the immediate-frequency and neighbour-enrichment numbers in 3ea5ab7 and
a2f8c92. Scanned per-function windows, so uniformly displaced: ratios are
probably robust, exact counts are not measured on the bytes they claim.
- e1482ab, d97e7df, f2d7e7e: the jal decoding was verified correct, so branch
targets stand; what lives at them was read wrongly.
Survives intact: every chunk-offset measurement in the tie and overlay work,
which read core_data and the overlay flat and never went through the section
index.
Fix: never index secs[n].data by addr - secs[n].dest. Either use
overlay[addr - LIT_BASE], or add a helper that accounts for the 16-byte header so
call sites cannot forget it -- and verify it against a word the decomp pins, as
above.
This was available from the first probe and skipped sixteen times.
284 passing with the disc present, clippy clean.
Ran gc-overlay-anchors on 0x2ee138 as d97e7df directed. It reports 6 of 12 words
did not decode -- but the words disagree:
tool: 0x2ee138 0x4bc00083
my read: 0x2ee138 0xd8a10000
Different bytes at the same address. My offset into the section is wrong, so the
'opcode 0x36 is invalid' finding was about bytes never at that address. That
invalidates d97e7df's conclusion and its 'second decode error' framing: the
decoder was never the problem.
The tool's words are COP2 (0x12), VU1 vector-unit instructions, which capstone
will not render under plain MIPS32. So its 'did not decode' is expected for VU
code and says nothing either.
Four bugs in this investigation, all mine: un-masked branch target, transposed
address, invalid opcode read from wrong bytes, and the offset mapping itself.
Every one was a self-consistent probe reporting a clean number. The check that
catches all four is verifying the tool against a known-correct case before using
it to generate a finding -- which d97e7df recommended and I then skipped.
Nothing about 0x2ee138 is now known: not VU code, not data, not a function. The
address is the correctly decoded jal target; what is there is unestablished.
Next step is to fix and verify the address-to-overlay mapping before re-reading
anything. Until then no address computed by hand in this investigation is
trustworthy, including the 0x3a8060 walk that happened to decode.
284 passing with the disc present, clippy clean.
Following e1482ab I disassembled the function supplying r2 and made two errors,
both mine. First I read the jal target as 0x2ee178 when it is 0x2ee138 --
0x0bb84e << 2 is 3,072,312. Second, at the correct address the bytes do not
decode: d8a10000 is opcode 0x36, which is not a valid MIPS major opcode for
user-space code. The coherent readings are that this is data, that the decomp's
extent is mis-bounded, or that the jal target is still wrong.
Stopping here deliberately. An un-masked branch target that reported a false
negative, and a transposed address that produced plausible-looking garbage, are
enough to make anything else from this decoder untrustworthy. The next step is not
another immediate scan or another hand-decoded disassembly -- it is
tools/overlay-anchors, which links capstone, verified against a known-good
function first.
The established chain, with nothing inferred beyond it:
0x3a3d70 jal 0x2ee138 target decoded, callee unidentified
0x3a3d84 jal 0x3a7c30 r2 = return value of the above
0x3a7c30 580 bytes, never writes r2
0x3a7d88 jal 0x3a8060 passes r2 through
0x3a8060 addiu r17, r18, 0xc0 with 0xd0 and 0xe0; 16-byte-stride walk
What 0x2ee138 returns is unknown, and it is the single remaining link between the
program and moby_classes.offset.
284 passing with the disc present, clippy clean.
349b498 reported that 0x3a7c30 has no jal callers. That was wrong and the bug was
mine: the scan built J-type targets as ((instr & 0x03ffffff) << 2) | 0x0c000000
without masking to the upper nibble, adding 0x0c000000 to every target so nothing
matched.
Corrected: exactly one jal caller, at 0x3a3d84. Still no j tail-calls, still not a
vtbl handler, still not a pointer in any level-0 data file.
Tracing back, r2 is never written between the call and the use:
0x3a3ce4 lw r3, 0x1500(r2)
0x3a3d70 jal 0x2ee178
0x3a3d84 jal 0x3a7c30
so r2 is the return value of 0x2ee178, passed through as the first argument. The
chain is now 0x2ee178 returns a pointer -> 0x3a7c30 -> 0x3a8060 -> addiu r17, r18,
0xc0.
Next question is one function: does 0x2ee178 return core_data + moby_classes.offset?
If so the reading is confirmed and the loader is located; if not, the 0xc0 in
0x3a8060 is an unrelated struct member.
Also recorded in docs: a negative from a hand-rolled decoder is worth nothing
until checked against a known case. jal 0x3a8060 decoded correctly under a
different constant in the same probe and I did not notice.
284 passing with the disc present, clippy clean.
The trace eca1a16 asked for. The call site 0x3a7d88 sits in the function at
0x3a7c30, 580 bytes.
r2 is never written there: scanning all 580 bytes for any instruction storing into
r2 finds none. So it arrives from outside the function.
And 0x3a7c30 has no jal callers anywhere in level-0 .text, and is not a lvl.vtbl
handler -- level 0's vtbl has 233 records and 185 distinct handlers, and neither
0x3a7c30 nor 0x3a8060 is among them.
That rules out the tidy story, which was the obvious guess given r2 arrives
unexplained: the vtbl dispatcher calling it with a class pointer in v0. What
remains is an indirect jalr call, a dispatch table in data, or code that is never
called. Not distinguished; the question for anyone continuing is whether
0x3a7c30 is a jalr target and from where.
The chain is now ? -> 0x3a7c30 (no static caller) -> 0x3a8060 -> addiu r17, r18,
0xc0. Every remaining question is about control flow and register provenance, not
counts, which is where this stretch's method stops working.
284 passing with the disc present, clippy clean.
0x3a8060 has exactly one call site, at 0x3a7d88:
0x3a7d84 addiu r18, r2, 0x202d r18 = r2 + 0x202d
0x3a7d88 jal 0x3a8060
No 0xc0 in the eight instructions before the call, and r18 is built from a
different offset. So the 0xc0 inside the callee is a field offset applied to a
pointer the caller already built, not a chunk offset the caller resolved.
Two readings remain: r18 is core_data, making r18+0xc0 equal &moby_classes, which
fits since level 0's moby_classes.offset really is 0xc0; or 0xc0 is a member
offset in an unrelated structure and the match is coincidence.
Settling it needs the provenance of r2 at the call site -- real dataflow, not
counting. That is where this stops.
Recorded in docs/plan/level-assets.md. The previous commit's doc edit failed
silently on a line-break mismatch, so this appends rather than replaces.
284 passing with the disc present, clippy clean.
Closes the ambiguity 05d9bf6 raised. At 0x3a8060:
263200c0 addiu r17, r18, 0x00c0 r17 = r18 + 0xc0
263000d0 addiu r17, r16, 0x00d0 r17 = r16 + 0xd0
262400e0 addiu r17, r4, 0x00e0 r17 = r4 + 0xe0
3c013f80 lui r0, r1, 0x3f80 the actual float constant: 1.0f
0xc0 is an address offset, not a float constant. The float constant is in the same
function and is 0x3f80, where it belongs -- so the ambiguity does not arise here,
and the reading of 0xc0 as moby_classes.offset survives.
The three offsets are 0xc0, 0xd0, 0xe0: sixteen bytes apart. The function walks a
table of fixed-stride structures and indexes the field at offset 0xc0, which is
what a class-table lookup looks like.
WHAT THIS DOES NOT SETTLE
That 0x3a8060 is the moby class loader. It indexes a 16-byte-stride table at
offset 0xc0, which is consistent with a moby class walk, with a different
structure that also lives at 0xc0, and with ordinary field access on an object
handed in by the caller.
The entry code sets a 64-byte frame and saves r16 through r20, and the three
indexes come off three different registers -- r18, r16 and r4. So the walk is over
arguments, not over a global. That is informative in itself: a loader would more
plausibly be handed the chunk by its caller than re-derive core_data + 0xc0 on
every call. Which also means the chunk reference is in the *caller*, and that is
where the next step is.
NEXT STEP, narrow and concrete: trace where r18 and r4 come from, and find the
call sites of this function. If a caller does `addiu r, base, 0xc0` and passes it
in, that caller is doing the chunk lookup and this function is the per-class work
inside it.
Worth recording how long the ambiguity took to close. It was raised one commit ago
and resolved by reading twelve instructions. Five commits before that the same
constant was reported as a "real shortlist", and two before that as a size
artifact. Every one of those states was arithmetically correct; only this one was
the right question, and it took looking at the operands rather than counting them
to arrive at it.
284 passing with the disc present (23 + 19 + 64 + 156 + 22; disc binary 35.00s),
clippy clean.
Looking at the 72 candidates with fewer than 16 distinct immediates, the sets are
full of values that look like lui high halves: 0x3f80 is the high half of 1.0f,
and 0xff40/0xff70/0xff80/0xffc0 are negative high halves. 0xc0 << 16 is 0x00c00000,
a small float.
So 0xc0 in these functions may be a float constant rather than a chunk offset.
The neighbour control cannot tell the difference, because 0xb8 and 0xc8 are
equally plausible float high halves and are rare for their own reasons.
Also recorded, and explicitly not as a finding: exactly 1 of 394 candidates
contains another chunk offset, and none of the 72 does. That sounds like evidence
of moby-specificity and is not -- the other chunk offsets are 0x1e80 and above and
are globally rare (0 to 2 functions each, measured in 3ea5ab7), so their absence
here is exactly what rarity predicts.
What survives is the enrichment fact from a2f8c92. What does not survive is
reading 0xc0 as a chunk offset. There are now two live explanations, a moby chunk
reference and a float constant, and only disassembly separates them.
Cheapest test: 0x3a8060, 148 bytes, six immediates. If the lui carrying 0xc0 is
paired with an ori it is a float; if it feeds an addiu computing base + 0xc0 it is
the chunk.
284 passing with the disc present, clippy clean.
Runs the control 06531fb named. It settles the question that commit left open,
and the answer is that the enrichment is real.
Matching on immediate-set size, against the best of the sixteen adjacent values:
bucket n 0xc0 best neighbour ratio
0-15 1,082 72 0xb8:1 72.0x
16-31 713 119 0xc8:4 29.8x
32-47 291 104 0xb8:3 34.7x
48-63 119 39 0xb8:2 19.5x
64-79 59 31 0xc8:2 15.5x
80-95 25 13 0xb8:1 13.0x
Enriched in every bucket, and the ratio does not decay as sets grow. That is the
opposite of a pigeonhole artifact -- a volume effect would shrink the ratio in the
big buckets, and it rises instead.
The load-bearing row is the smallest: 72 of 1,082 functions carrying fewer than 16
distinct immediates contain 0xc0, where the best neighbour appears once. A set that
small cannot produce that by count alone.
What this revives. The 300 candidates with 48 or fewer distinct immediates are not
noise: at matched size, neighbours hit 0.4% to 1.4% in those buckets where 0xc0
hits 35%. A function using fewer than 16 constants, one of which is 0xc0, uses
that offset deliberately. Those are the shortlist.
Why this took four commits to establish. 3ea5ab7 measured 394 against neighbours
1-14 and called it a shortlist. 06531fb showed the rate rises with set size and
said the 394 was a list of large functions. Both were right about their own
measurement and neither had the bucket-matched comparison, which is the only form
that separates "the constant is special" from "big functions contain more
constants". Twenty-two probes this stretch, and the thing that finally settled it
was not a better measurement but adding the right comparison to a measurement
taken three commits earlier.
Caveat unchanged: presence of an immediate is not proof the chunk is read there.
It is proof the constant is used, which is all either test asserts.
284 passing with the disc present (23 + 19 + 64 + 156 + 22; disc binary ~35s),
clippy clean.
Ran the size-independent statistic db1e69c asked for, and it confirms the
confound. Bucketing level-0 functions by distinct-immediate count, the rate
containing 0xc0 is:
0-15 72/1082 6.7% 64-79 31/59 52.5%
16-31 119/713 16.7% 80-95 13/25 52.0%
32-47 104/291 35.7% 112-127 3/7 42.9%
48-63 39/119 32.8% 624+ 1/1 100%
Monotone from 6.7% to over 50%. Pigeonhole: a function with hundreds of distinct
immediates has more chances to contain any particular value.
So the 394 is largely a list of large functions and '394 of 2,323' was never a
shortlist. db1e69c was right; 3ea5ab7's framing was wrong and is corrected here.
This does not retract the neighbour result. 0xc0 being enriched over 0xb8-0xc8,
and 0xc0 being enriched in bigger functions, are separate claims and both hold.
What it retracts is the inference that the 394 are candidates for anything.
The control that would settle it: 0xc0's rate against a specific neighbour's rate
within the same set-size bucket. Not measured, and named so the next attempt
starts there rather than re-ranking by size.
No test -- the bucket rates are the finding, and an assertion on any single
threshold would be arbitrary.
Ranking the moby_classes candidates largest-first is the wrong first cut. The
top entries are 34,316 and 26,432 bytes with 631 and 421 distinct immediates,
median candidate 852.
A function with 631 distinct immediates is far more likely to contain any given
value, 0xc0 included, than one with 80, purely by pigeonhole. Sorting by size
therefore selects for the functions most likely to be false positives, and the
34,316-byte outlier is probably a mis-bounded extent -- the decomp corrects
three level-00 entries -- rather than a 34 KB loader.
The 394 needs a size-independent statistic: whether a function contains 0xc0 more
often than its immediate count predicts, not whether it contains it at all. Not
measured.
Recorded as a negative about the method rather than committed as progress. No test
-- there is no assertion here to make.
283 passing with the disc present, clippy clean.
First positive result of the loader hunt, and it exists only because of the
control that two commits ago I said was missing.
Across 2,323 level-0 functions of at least 128 bytes -- extents from the decomp's
functions.tsv -- the immediate 0x00c0, which is moby_classes.offset, appears in
394 functions. Every value adjacent to it appears in 1 to 14.
0xb8:8 0xb9:2 0xba:2 0xbb:2 0xbc:3 0xbd:5 0xbe:5 0xbf:3
0xc1:3 0xc2:3 0xc3:2 0xc4:1 0xc5:2 0xc6:3 0xc7:1 0xc8:14
This is what 27891ce was asking for. 0xc0 is small and small immediates are
common -- 17,699 occurrences across the program, against 200,784 for 0x0001 -- so
the raw frequency was worthless. The neighbours have the same magnitude and none
of the frequency, which is the only reason 394 means anything rather than being an
artifact of the value being short.
A second control points the other way, so the two bracket it: the other five
level-0 chunk offsets are larger values and are referenced 0 to 2 times each.
Shrub_classes 1, tfrag_textures 2, moby_textures 2, tie_textures 0, shrub_textures
2. The chunk offsets a loader would use are referenced about once. moby_classes is
referenced 394 times. The program leans on this constant heavily.
The previous commit's 410 candidates were not a shortlist. These 394 are, because
the number is now separated from magnitude rather than merely stated.
WHAT THIS DOES NOT SAY
An immediate appearing in a function is not proof the chunk is READ there. It is
proof the constant is used, which is the claim the test asserts and the only claim
it makes. It does not say which of the 394 is the loader, or that the loader is
among them. Stated plainly because the 394 is the kind of number that gets
quoted without its caveat and then built on.
Also measured, negative: zero functions contain both the moby_classes and
tie_classes offsets, so nothing indexes both chunk tables by immediate. The moby
loader does not reach the tie table the same way, if it reaches it at all.
Boundaries: level 0 only, "level 0" meaning the function's first listed place is
level 00, and a 128-byte size floor -- 59957a0 recorded 20-byte "functions" that
are mis-bound extents.
283 passing with the disc present (23 + 19 + 63 + 156 + 22; disc binary 35.23s),
clippy clean.
The decomp's functions.tsv makes this cheap -- 3,835 level-0 functions with start
and size, so each immediate-bearing instruction is attributable to its container.
410 of 3,835 functions (10.7%) contain one; sizes 20 to 34,316, median 808.
Ten per cent is what a common small immediate buys, so this confirms 56b00fb
rather than narrowing anything. 410 is not a shortlist. What it does establish is
that the attribution works and the decomp's extents are usable for it.
The loader hunt now has a concrete next step: of those 410, the ones that also
touch a class span. That is dataflow, not counting, and it is where this resumes.
Caveat recorded: a 20-byte 'function' holding the immediate is a mis-bound extent
whose tail happens to include another function's instruction. Sizes need a floor
before the set is used.
No test -- an attribution count with no assertion behind it would only make the
410 look like a finding.
Saying the previous method 'cannot tell' was right but the reason deserved a step
rather than a hand-wave: it is fixable by counting immediates only where they are
immediates, which needs an opcode filter and no disassembler.
All 27 levels: 11,911,320 instructions, 2,985,897 carrying an immediate
(addiu/lui/ori/xori/andi/sltiu), 13,813 distinct immediate values.
moby_classes 0x00c0 17,699
tie_classes 0x1e80 17
And now there is a control the raw version never had -- the program's most
frequent immediates are 0x0001 (200,784), 0x001b (149,259), 0x0010 (137,523),
0xffff (98,312). 0xc0 at 17,699 sits below the top of that small-value cluster
rather than standing out, so it is not a distinctive reference.
Result unchanged from the raw scan and now defensible: no signal. The loader is
not findable by frequency, and the remaining route is structural dataflow --
functions touching both the chunk offset and a class span.
No test: asserting that an offset is unremarkable is not an invariant worth
locking, and the two numbers are already recorded with their denominator.
Sought the moby class loader by counting raw u16 occurrences of each chunk offset
in .text. moby_classes 0xc0 hits 49,230 times, tie_classes 0x1e80 hits 111 --
and that contrast means nothing.
Small u16 values are simply common: the most frequent value in .text occurs
1,834,535 times. A two-byte value that is frequent by construction beats a rarer
one regardless of what either references, so the comparison has no shared
denominator. Rule 3a again, one level below where it has bitten before -- not the
measurement being wrong but the comparison being uninterpretable.
Recorded as a methodological negative rather than a finding. Doing it properly
means counting immediates only where the instruction decoder says they are
immediates, which is what tools/overlay-anchors exists for.
No test added: a test asserting a meaningless comparison would make the next
reader trust it.
Ruled out the cheap approach: moby_classes (offset 0xc0, count 238 in level 0)
holds float-shaped values, and only 120 of the first 4,096 u32s land inside the
overlay .text against ~259 expected by chance. Fewer than chance, so the loader
is not dispatched from a pointer here.
Not concluded what the 238 entries are -- the float reading is suggestive but was
taken at one stride on one level.
Follow-up to the note in 6d26da3 that two unused-variable warnings in the
sound_bank and overlay tests were pre-existing and deliberately left alone.
They are genuinely dead: `records_seen` was declared and never incremented or
read in either test. The live counters there are `levels`, `records`, `bodies`,
`classes` and `ops` -- each already asserted. Removed.
Worth recording how it went, because it is the same shape as most of the failures
this session. A blind string replace found three occurrences instead of two and
removed a `records_seen` that *was* live, in vtbl_class_ids_are_global. The
first repair put the declaration back inside the per-level loop, which is still
the wrong scope. Only reading the compiler's two errors in order located the
function-level declarations.
Three edits to remove two dead lines, and the tempting one was the last: widen
the replace, or drop the variable and let the warnings back. Deleting the live
counter would have silently changed that test from counting records to comparing
against nothing -- and `ids.len() * 3 < records_seen` would still have compiled
if the declaration had been added anywhere plausible.
The check that caught it was not a test. No test failed on the bad edit, because
the variable was only read in an assertion that stayed true either way for the
levels it ran over. Clippy caught the compile error; nothing else would have.
282 passing with the disc present (23 + 19 + 62 + 156 + 22; disc binary 34.87s),
clippy clean.
2ff2abb ended by implying that the class-header offsets point at identifiable
content. I said the next step would be a control. Here it is, and it fails the
claim.
Adjacent-equal-byte rate over 64 bytes, 1,864 classes:
0x1c 0.12688
0x34 0.12106
control 0.14109 (the same offset + 0x2000)
random 0.00391
All three are ~30x above the random baseline, so the region is structured data
and not noise. That part stands. But 0x1c and 0x34 are no more structured than an
arbitrary offset in the same blob, and 0x34 is slightly *less*.
So the hex fragments in 2ff2abb were three hand-picked examples from a structured
region. "They look like compiled shader programs with small-integer parameters"
said something about this part of core_data, not about what those fields address.
That is exactly the rule 4 failure -- a pattern that reads as meaningful with no
control to test it against -- and I wrote it anyway while explicitly declining to
name a format two paragraphs earlier.
RETRACTED: the suggestion that the offset targets are identifiable content.
Unaffected, because it was measured differently: the offsets are base-relative,
non-zero, in range on 1,864 of 1,864 classes, and always land past the class's
own tie packet blocks. What they address is still unknown. The earlier finding
that the packet's control region is not a tag stream also stands -- that was
measured by uniformity within the packet, not by comparison to elsewhere in
core_data.
A TEST THAT CAUGHT ITS OWN AUTHOR
The first version asserted |0x1c - control| < 0.02 symmetrically, and failed on
0x34 at a gap of 0.0200. The correct response was not to loosen the number until
it went green. The claim being guarded is "these offsets select distinctive
content", and a gap in the *other* direction does not support it -- both offsets
came out below control. So the assertion is one-sided, and the comment says why,
including that the symmetric form was the wrong shape.
That is the AGENTS 3b discipline working in the direction it is meant to: a
failing assertion is information about the assertion.
282 passing with the disc present (23 + 19 + 62 + 156 + 22; disc binary 35.16s),
clippy clean.
Chasing where the AD-GIF is, since f43a1c2 established it is not in the packet.
Two of the undecoded class-header words are offsets.
base + 0x1c and base + 0x34 land inside core_data on 1,864 of 1,864 classes, so
both are relative to the class base rather than absolute, and both are non-zero
on every class. Both also land **past** that class's own tie packet blocks on
every class, so they address per-class data held elsewhere in core_data rather
than the tie geometry.
That is consistent with f43a1c2: if the packet's own control region is not a tag
stream, the per-class draw state is referenced from somewhere else.
WHAT IS THERE, AND WHAT IS NOT CLAIMED
The bytes are structured, not random -- class 3 shows a fixed-stride table with
`80 01 00` recurring, class 2's field is a monotone byte table, class 5's has
`38 38 38` and `ff 00 ff ff`. They read like compiled shader programs with
small-integer parameters, or lookup tables.
**Not identified, and deliberately not named.** None of those three fragments has
been run against a control, and all three were picked because they looked like
something. That is precisely the finding rule 4 refuses, and having spent this
session on four retractions I would rather record an honest "structured,
unidentified" than a plausible format that has not earned it. If any of these
turns out to be a shader program the next step is a control, not a better-looking
hex dump.
ALSO SETTLED
0x24 is demonstrably not an offset: only 17 of 1,864 of its values land in range
for core_data. Recorded as packed data. 0x28, 0x2c, 0x38 remain high-entropy and
unexplained.
281 passing with the disc present (23 + 19 + 61 + 156 + 22), clippy clean.
TieClassHeader skips 0x1c..0x44 -- forty bytes per class, 1,864 classes, "none of
it matches RAC2 yet". Profiled each of the ten words instead of reading them.
THE PROFILE, PER WORD
0x20 reserved -- zero on 1,864 of 1,864
0x30 reserved -- zero on 1,864 of 1,864
0x1c 1,390 distinct, 0x1e0..0x3cd60 looks like an offset
0x24 1,571 distinct, high entropy
0x28 640 distinct, 0x00100000..0x1000013d two bytes packed
0x2c 1,610 distinct, high entropy
0x34 1,331 distinct, 0x190..0x31e20 looks like an offset
0x38 441 distinct
0x3c SIX distinct: 0 (1,384), 16 (435), 48 (21), 32 (18), 2 (3), 4 (3)
0x40 a float -- 0.00208 to 10.84, median 0.30, no negatives
0x40 is decidable and it is now a hard assertion: a positive range across two
orders of magnitude with no negatives and no zeros is not what an integer looks
like. Bounding sphere radius or class scale; not distinguished, and the doc says
so rather than picking one.
THE CONTROL THAT PREVENTED A WRONG CLAIM
0x3c has six values, is zero on 74% of classes, and sits next to a doc comment
listing "the texture count, ad-gifs, the bounding sphere and the class scale". The
obvious next move is to call it the ad-gif count -- especially having just spent a
commit establishing that the AD-GIF is not in the packet, so an ad-gif count
would be satisfying.
Tested it: mean bfc_distance per 0x3c bucket is 7.80, 6.56, 11.07, 8.03, 6.81,
6.11. Flat. It does not track the shader count, so it is not the ad-gif count.
The test now asserts that flatness as a control, so if 0x3c ever starts
correlating with bfc_distance the assertion fires rather than the coincidence
being noticed later and mistaken for a decoding.
That is the rule-4 pattern working: a statistic that cannot tell the hypothesis
apart from nothing must not be used to support it. Six distinct values and a
dominant zero *feel* like a count, and feel is exactly what is not evidence.
STILL UNIDENTIFIED
0x1c and 0x34 look like offsets but are unverified as such -- only 17 of 1,864
0x24 values land in range for core_data, so that one is not an offset and is
recorded as packed data. 0x24, 0x28, 0x2c, 0x38 are high-entropy and unexplained.
None of them is guessed at.
280 passing with the disc present (23 + 19 + 60 + 156 + 22; disc binary 35.38s),
clippy clean. The two unused-variable warnings in the sound_bank and overlay tests
are pre-existing and confirmed present before this change; left alone rather than
folded into an unrelated commit.
Follows 9bb04cc. The control region is now characterised, and the second half of
the answer is a correction to a field doc that has been wrong since the record
was first parsed.
WHAT THE CONTROL REGION IS
Per-record, not shared state. 12,294 distinct blobs across 13,042 records, mixed
within 2,410 of 3,549 classes, and different between LOD 0 and LOD 1 on all 500
classes that have both. It varies per primitive.
Partitioned by its own length, it is not uniform in character, and the pooled
count hid that:
vert_ofs 48 bytes (3 qw) 5 distinct blobs
vert_ofs 64 bytes (4 qw) 3,175
vert_ofs 80 bytes (5 qw) 4,309
vert_ofs 96 bytes (6 qw) 3,281
vert_ofs 112 bytes (7 qw) 1,523
vert_ofs 128 bytes (8 qw) 1
The shortest and longest are effectively constant -- five blobs, and one -- while
everything between is per-record unique. That is a shape, not a noise floor, and
it is invisible unless the measurement is partitioned by length.
THE CORRECTION
bfc_distance == 5 * shader_count holds on every record, and the field doc has
always read "quadwords of ad-gif data" -- which read as a packet-relative length
would make it the control region's size.
It cannot be that. bfc_distance is 5..20 quadwords; vert_ofs, the control region,
is 3..8.
Pooled, they overlap on 9,315 of 15,219 records -- 61%, a number describing
nothing. Partitioned by shader_count, as AGENTS 3a requires, it is exact:
shader_count 1: 2,772 of 8,676 exceed (32%)
shader_count 2: 4,649 of 4,649 exceed (100%)
shader_count 3: 1,729 of 1,729 exceed (100%)
shader_count 4: 165 of 165 exceed (100%)
Without exception for two or more shaders. For one shader it exceeds exactly when
vert_ofs is below 5, which is that 32%.
**The AD-GIF is not stored in the tie packet.** The packet is control, vertices
and scissor, in that order. bfc_distance is a per-record quantity tied to the
shader count whose meaning is still not established, and the "quadwords of ad-gif
data" reading of it is wrong. The doc now says so and says why, rather than
carrying a plausible description of a field whose meaning nobody has established.
This is the fourth retraction today and the second today found by partitioning
rather than by a test failing. The pooled number was not merely less precise than
the partitioned ones -- it pointed the other way, because 61% looks like noise and
100% looks like a boundary. Had I accepted the pooled figure I would have written
"sometimes" and moved on.
59 disc tests, 278 total across 23 + 19 + 59 + 156 + 22 with the disc present
(disc binary 35.31s), clippy clean.
Went after the GS's GIF packets in the tie AD-GIF region. Got a negative result,
a new universal invariant, and a retraction of my own first probe, which reached
the right answer for the wrong reason twice.
THE NEGATIVE RESULT
Tried the leading `5 * shader_count` quadwords of each packet (the
`bfc_distance` field's stated meaning) as VIF tags -- command in bits 0-6, size
in 7-14, mark in 15 -- and as GIF tags -- type in the top byte of word 1, length
in bits 0-2 of word 0. Both uniform. Over 13,042 records: 10,443 distinct values
of word 0, no GIF tag type above a count of 2. That is the control result rule 4
asks for, and it says stop.
THE RETRACTION
The first probe scanned `bfc_distance` quadwords, which for most records
overruns `vert_ofs` and runs into the vertex data. Pooling a control region with
vertex positions -- two populations with different structure -- gives a uniform
answer, so "not a tag stream" came out right from a measurement that was
measuring the wrong bytes. That is exactly the rule 3a failure, and I only caught
it because the hex dump I took next looked structured while the histogram said
random.
Re-measuring quadword 0 alone, on its own population, gives the same uniformity.
The conclusion survives; the reasoning did not, and both are recorded.
THE PARTITION, AND A NEW INVARIANT
The hex dump showed the packet layout directly:
q[0 .. vert_ofs) control 3..8 quadwords, peak at 5
q[vert_ofs .. +vert_size) vertices
q[vert_ofs+vert_size ..] scissor
and `scissor_ofs == vert_ofs + vert_size` holds on **15,219 of 15,219** records
across all 27 levels, including every one whose strip_count is non-zero. That is a
derived quantity stored explicitly, and it holds everywhere -- which is the check
that confirms the three regions really are laid out the way the existing packet
length formula assumed. It is now a hard check in invariants_hold(), which is
something the previous four could not be.
The fixture in tie.rs failed the moment I added it: it had vert_ofs 4 and
vert_size 0x20 with scissor_ofs 0x30, which is 4 + 0x24. The synthetic data was
never constrained by a relation that did not exist yet. Fixed to 0x24.
vert_ofs IS MIS-NAMED
It is not a vertex offset. It is the *length* of the leading control region, and
the vertex data begins at that many quadwords into the packet. The field name
stays because the on-disc order is fixed and everything already written about this
record says vert_ofs, but the doc now states what the value is and gives the
distribution.
STILL UNKNOWN
What the control region contains. Not standard DMA tags; possibly pre-merged GS
register writes in another framing, or per-class state rather than per-primitive
draw data. Recorded as an open question rather than guessed at.
23 + 19 + 58 + 156 + 22 = 278 passing with the disc present (disc binary 35.67s),
clippy clean.
A small doc change with a specific reason.
gc-engine's sampler sat behind a comment claiming it clamped while the code was
`..default()`, which is Repeat. That was caught only because I went looking for
where clamp01 should be used and found the answer was "somewhere that already
claims to do it". The generalisable failure is a reader seeing a rule defined in a
module called gs.rs and assuming the renderer applies it.
So the module now says which rules are wired and which are not:
CLAMP wrap yes gc-engine image_of sets address_mode_u/v
alpha 0x80 opaque yes texture::scale_alpha delegates here
blend / alpha test no bevy AlphaMode cannot express both at once
mip selection no only level 0 is decoded, mip_level is always 0
The last row is the one worth having written down. mip_level is a correct,
tested function that returns 0 for every input the renderer can currently give it,
and nothing in the code says so. It will start meaning something the day multi-level
textures are decoded, and until then it is indistinguishable from a mip selector
that works. That is the same shape as the sampler: a specific, plausible, untested
claim about behaviour sitting where nobody will re-check it.
passes_alpha_test and BlendMode are reference -- correct and tested, but not on the
draw path, and per 43e508a not expressible there in bevy's current enum. Saying so
is better than leaving them looking like pending integrations.
No behaviour change. 278 tests, clippy clean.
Sets the terrain material's alpha mode, which is the first draw-path change the
GS work has actually produced.
The material used `..default()`, which is AlphaMode::Opaque: every texel drawn
fully solid regardless of what its alpha said. Given 0.98% of texels on the disc
are fully clear, that means roughly 227,000 texels that should have shown
background are drawn as opaque geometry.
The choice is Blend, not Mask, and the reason is the histogram from 25d73b4:
- Mask(f32) is a *quantising* cutout. Below the threshold -> transparent, at or
above it -> fully opaque. Feeding it the 1.08% partial texels would flatten
every one of them to solid, discarding exactly the soft edges they exist to
produce. 126 distinct values become two.
- Blend handles all three populations at once. A clear texel blends to invisible,
a partial one stays smooth, an opaque one stays opaque.
Which retires the conclusion I gave two commits ago. I said cutout-first, on the
reasoning that the clear texels need discarding and a partial texel drawn opaque
is a failure nobody notices. But Blend does not need a cutout to hide those
texels -- a zero-alpha fragment is invisible by definition. The cutout was
solving a problem Blend solves for free, so the ordering was moot before it was
tested. Three commits on the alpha path and the second conclusion has now also
fallen; the surviving facts are the histogram and the rule that 0x80 is opaque.
One real fidelity gap, recorded rather than papered over: the GS can alpha-test
and blend the same primitive at once. bevy's AlphaMode is a single enum where
Mask and Blend are mutually exclusive, so it cannot express that. Where the
original combines them, this port has to pick one, and the choice will not be
free. That belongs in the Phase 5 frame-for-frame gate rather than being treated
as settled here.
Clippy clean, builds. No test added: this is a material parameter with no
headless assertion available, and asserting a bevy enum value would be a test of
the bevy API rather than of anything in this project.
25d73b4 said:
alpha testing (cutout) matters far more than alpha blending on this disc
and cited 0.98% fully clear against 1.08% partial as the evidence. Those two
numbers are the same size. The cutout path handles the fully-clear texels and the
blend path handles the partial ones, so the measurement I gave is if anything
marginally against the claim, not for it. The conclusion did not follow from the
number attached to it.
What survives is a different and weaker claim, and it is about failure shape
rather than texel count:
- a texel that should be discarded but is drawn puts geometry where there is
none. That is visible from a screenshot and someone will file a bug about it.
- a partial texel drawn opaque is a slightly wrong soft edge. That is not visible
and will not be reported.
So cutout first is still the right order, but the reason is which failure a human
notices, not how many texels are involved. Stated that way it is defensible;
stated the way I did it was an overclaim with a number attached to make it sound
measured, which is the specific failure mode rules 3 and 3a exist to prevent.
This is the fourth thing this session where a plausible implementation or claim
turned out to be wrong on inspection, and the second in two commits where the
error was in the reasoning rather than the code. Code errors get caught by tests
because the tests execute. Reasoning errors get caught by re-reading the number
the claim rests on and checking that it says what the sentence says.
Retracted and replaced rather than softened, per AGENTS 3a.
An addition to AGENTS 3b, from nearly repeating the failure it describes.
Checking the count in 25d73b4, the suite came back 278 passing -- the same 278 as
always. But GC_DISC was not set for that run, so all 58 disc tests hit their
`require_disc!` early return and were counted as passed. The number was right and
the coverage behind it was not.
What distinguishes the two runs is the wall clock and nothing else:
without GC_DISC 58 tests, 4.10s
with GC_DISC 58 tests, 36.14s
Same test count, same "ok", same zero failures. A reader -- including me -- has no
way to tell those two lines apart. That is a sharper version of the 3b failure
than the filtered sum was, because no amount of reading the output carefully
distinguishes them. The duration does, and it is in the line already.
So 3b now says: if a run finishes suspiciously fast, the disc tests did not run.
And it keeps the standing instruction to set GC_DISC when touching disc-decoding
code, since 25d73b4 changed texture-adjacent code and the verification genuinely
needed the disc.
Re-verified properly: 278 passed across 23 + 19 + 58 + 156 + 22, disc present,
36.14s on the disc binary. Clippy clean.
The alpha histogram from 777a6a8 had a consequence sitting in plain sight that I
had only used as a regression guard. It is a prioritisation input.
Measured over every decoded tfrag texture on all 27 levels:
total texels 23,168,000
fully opaque (255) 22,690,698 97.94%
fully clear (0) 227,497 0.98%
partial (1..254) 249,805 1.08%
distinct partial values 126
Two conclusions, and they point in opposite directions from the one I would have
assumed.
The 0x80 -> 1.0 rule is not an edge case to be got right eventually. It is what
22.7 million texels -- 97.94% of every pixel of terrain on the disc -- are
already doing. Getting it wrong is not a subtle shading difference, it is the
texture decode being wrong almost everywhere.
But blending is close to irrelevant at this level: partial alpha is 1.08% of
texels. A correct but imperfect blend path costs almost nothing visually; a
correct alpha *test* is what determines whether foliage, fences and cutaway
geometry exist at all. That reorders the remaining GS work, where I had been
planning to do blending and the cutout together.
The 126 distinct partial values are also evidence about what they are. If the
partial alpha were a blend ladder or a dither, it would populate a handful of
values. It populates all 126 evenly, which is what smooth alpha looks like --
soft edges and distance fade, not discrete states.
Caveat worth stating: these are tfrag textures, so this is a statement about
terrain. Moby and sky alpha may behave differently and are not covered by this
measurement.
Test count read from the raw result lines per AGENTS 3b, with the disc present:
278 passing, clippy clean.
Rule 3b, requested after a commit went out with a failing test in it.
cargo test -q --workspace | grep "^test result: ok" | awk '{s += $4}'
That sums the green test binaries and ignores the red one. A broken build still
prints a number, and the number looks entirely reasonable -- six healthy counts
added together is not a number anyone questions. I used that filtered sum as the
evidence of health in commit messages for the better part of a session, and it
was reporting green on a build that was not.
What actually happened is the part worth keeping. The count came back lower than
the previous run -- 100 instead of 278. That is a genuine signal and I explained
it away twice as an environment difference, the disc not being set, before
checking. Both explanations were plausible. Neither was tested. Only the
discrepancy was noticed, and noticing it is what surfaced the failure.
So the rule covers both halves: do not commit or push with a failing test, and do
not derive the number you report with a filter that drops the failing binaries.
The second half is the one that is easy to get wrong again, because
`cargo test -q` is quiet, the grep looks like a reasonable way to total up, and
the total is the only part anyone reads.
Also records the mistake as a mechanism rather than a symptom. Same defect as
rules 3 and 3a, one level up: a quantity that should have had one definition had
two, and nothing compared them.
Added a line to "Working rhythm" pointing at it, since that is the list someone
reads before committing.
Fixes a broken build left by 777a6a8, and the test it breaks is the interesting
part.
texture::tests::alpha_is_scaled_from_eight_bit_halves
assert_eq!(scale_alpha(0x7f), 0xfe);
That is not a test that happened to pin the wrong value. The test's NAME asserts
the wrong rule -- "scaled from eight bit halves" is the `a * 2` implementation
written down as intent, with 0xFE as its proof. Someone looked at this, thought
"alpha is a half-scale byte, double it", and then locked the conclusion into both
the identifier and the assertion.
So the previous commit's claim that `a * 2` was an implementation detail was wrong
in a way that matters: it had a name, a doc comment and a test defending it. That
is not an accident that escaped review, it is a convention that was adopted.
Two things follow.
The rule is now /128 * 255, and the test says so in its name. It also pins 0x7E ->
0xFB, a value the doubling cannot produce and which the disc does contain.
And 777a6a8 was pushed with a failing test in it. I only caught it because the
count came back as 100 rather than 278 and the discrepancy was worth one more
command -- `cargo test -q` plus a grep for `^test result: ok` sums only the
binaries that passed, so a failure is invisible in the total unless you go and
look at the raw lines. I had been using that filtered sum to report "278 tests"
for several commits without knowing it could hide a red build.
That is worth more than the bug it found. The measurement I had been reporting as
evidence of health was itself an instance of the defect this whole stretch has been
about: a quantity that should have had one definition, expressed as a filtered sum,
agreeing with the wrong answer and nobody comparing it against anything.
278 passing with the disc present, clippy clean.
The convention has been carried by comments and an inline expression. This
measures it, across every decoded texture on all 27 levels.
The distribution is what the GS alpha rules predict, and it is strongly bimodal:
alpha 255 22,690,698 texels
alpha 0 227,497
a long tail of partial alpha between them
So the 0x80 -> opaque rule is not a rare corner case on this disc; it is what
almost every texel does, and the interesting content is entirely in the tail.
The useful part is the tail. It contains 253 -- twice -- and odd values across
the upper half: 129, 131, 133 ... 251, 253. 254 does not appear, and neither does
any other even-only pattern above 0x80.
That is the discriminator. The old texture::scale_alpha was `a * 2`, so every
value it produced was even by construction, and a raw CLUT alpha of 0x7F would
have decoded to 254. The corrected rule gives 253. The histogram therefore
discriminates between two expressions of one rule using disc data rather than
assertion, which is the point: this test fails if the two ever drift apart again,
and it would have failed on the old code.
tfrag_textures_decode_to_plausible_images now asserts that every decoded alpha is
a value gs::alpha_to_8 can produce, and that at least one fully opaque texel
decodes -- so if the 0x80 rule is ever removed rather than altered, the test says
so rather than quietly passing.
Worth noting what this histogram is not. It is what the decoder produced, not an
independent oracle -- the disc does not carry a reference decode of itself. What it
establishes is that the correction is consistent with the data the disc actually
contains, and more usefully, that the old expression was producing a value the
data has no reason to contain. Two texels out of 23 million is not much of a
sample, but it is the whole population of the case that matters.
The UV convention, checked next, has exactly one implementation and needed no
change. Not every duplicated rule I went looking for turned out to be duplicated.
Found the PS2 alpha convention expressed three times in three files, and the three
disagree:
value texture::scale_alpha gs::alpha_to_8 gc-engine (float)
0x40 128 128 0.5
0x7E 252 251 250.9
0x7F 254 253 253.0
0x80 255 255 1.0
scale_alpha was `if a < 0x80 { a * 2 } else { 0xff }`. That is a/128 * 256, not
a/128 * 255 -- one too high at the top of the range, so 0x7F decoded to 254 where
the renderer's own alpha path gives 253. It is the only one of the three that was
wrong, and it was the one feeding decoded texture pixels.
scale_alpha now delegates to gs::alpha_to_8. The renderer keeps its float form
because it wants a float, and gc-formats has no business importing an engine
concern; two of the three are now one function.
The visual cost of this bug was at most 1/255 on a single palette entry. Nothing
would ever have shown it in a screenshot. That is the point rather than the
excuse: this class of error is not "big wrong", it is "right to within a rounding
step", in the code path nobody instruments. Every one of them is invisible by
construction, which is why the only defence is noticing that one rule has three
homes and making it have one.
Worth counting what this has turned into over three commits. The vtbl terminator:
two loops reading the same table with opposite answers. The sampler: a comment
asserting CLAMP over code doing Repeat. The alpha rule: three implementations,
one wrong. None of these is a subtle algorithm and none is in an exotic format.
All three are the same defect -- a quantity that should have one definition had
several, and nothing in the build made them compare.
Wiring the first gs rule into the renderer, and it turned up a bug that had been
sitting in plain sight:
// Only mip level 0 is decoded, so minification must clamp rather than
// invent a level the disc does not have here.
ImageSamplerDescriptor {
mag_filter: ImageFilterMode::Linear,
..default() // <-- address_mode is Repeat
}
The comment describes clamping. The code was doing the opposite: `..default()`
supplies bevy's default address mode, which is ImageAddressMode::Repeat. Nothing
here ever set an address mode, so every texture was sampling with Repeat while a
comment three lines up claimed it was not.
That is the GS rule I wrote in 81ec178 saying out loud -- CLAMP, with 1.0 held at
the far edge -- and the renderer was doing the other thing. The seam shows up as a
patch running off its texture wrapping onto the opposite edge instead of repeating
the border texel, which is exactly the kind of thing that reads as "the texture
decoded wrong" and sends you looking in the format parser instead of at one line
of sampler setup.
Also set mipmap_filter to Nearest. Only level 0 is decoded, so there is no chain
to interpolate across and the filter cannot matter; Nearest is the honest setting
until levels exist, and the comment now says that gs::mip_level returns 0 for every
patch until they are decoded, rather than implying a selection is happening.
Worth noting how this was found. I went looking for where clamp01 should live and
found the answer was "somewhere that already claims to do it, and doesn't". A
comment asserting behaviour is not evidence of behaviour -- the comment here was
specific and plausible and wrong, which is worse than no comment, because it stops
anyone checking.
Two API corrections along the way, both mine: ImageSamplerDescriptor has no
`address_mode` field -- it is address_mode_u and address_mode_v separately -- and
the engine is on bevy 0.19 rather than raw wgpu, so the GS rules reach the GPU
through bevy's sampler. The roadmap now says so instead of implying a wgpu
integration that does not exist.
278 tests, clippy clean.
Caught by the user, and it was wrong in four places: the Phase 5 heading, its
gate, its go/no-go line, and the PCSX2 verification plan's "Level 0 (Veldin)".
Veldin is Ratchet & Clank's home planet in the 2002 game. Going Commando opens on
Aranos, and there is no Veldin anywhere in this disc. The error looks like it
carried over from the first game when the roadmap was drafted, which is an easy
slip to make and an unhelpful one to leave standing: a vertical slice defined as
"completable and matching PCSX2 frame-for-frame" is not something anyone could
have attempted or judged while pointing at the wrong planet.
Confirmed against the vendored decomp rather than taken on faith. Its
docs/LEVELS.md refers to level 00 as Aranos directly -- "corrects three measured
Aranos entries before fingerprinting ... These corrections apply to level 00 only"
-- so level 0 is Aranos in the reference material this project has been using all
along, and the roadmap disagreed with the decomp.
There is a real R&C1 reference nearby and it is worth keeping, because the
reasoning under the gate is sound and only the noun was wrong. Phase 5 argues that
spherical gravity is the thing most likely to break, because it invalidates the
movement and camera assumptions the engine inherits from Ratchet & Clank (2002).
That is about inherited code from the previous game, not about the level being
built, and the text now says so explicitly so the next reader does not have to
work out which game is meant.
No code change. Documentation only.
Follows directly from 65025cb, whose conclusion was that these tests assert bounds
rather than numbers and so cannot detect the drift they were known to contain.
Applying that to the tables involved.
Three things turned up.
The counts are now pinned: vtbl 5,822 records / 1,162 ids, camvtbl 217 / 14,
sndvtbl 32 / 6. Every one of these was previously `records > 5_000`,
`records > 0`, or nothing at all.
Two of the bad counts had been PINNED, not merely missed. Two separate tests
carried `assert_eq!(classes.len(), 1163)`. So walking past the terminator was not
an oversight that slipped through review -- the wrong number had been written down,
deliberately, at some point when it was correct. Both now say 1,162.
That is worth pausing on, because it is a different failure from the one I
described last commit. I said the tests were too loose to catch drift. In fact two
of them were too tight: they pinned a measurement that was wrong, and so they
actively defended it. A threshold and a wrong constant fail in opposite ways, and
this suite had one of each on the same table.
The third is the moby handler count. That one stops on the terminator in the
*handler* word rather than the id word, landing 34 records short at 5,788. It is
measuring a different question -- records with a real geometry handler -- so it is
correct, and it now says so at the break rather than leaving the next reader to
wonder why it disagrees with the other three.
Tests are 278 as before; nothing here changed what is counted, only what is
asserted. That is the whole point -- the previous 278 and these 278 agree, and now
they cannot silently stop agreeing.
Reconciles the discrepancy flagged when the dispatch tables went in. Three
committed tests counted lvl.vtbl records and produced three numbers: 5,878
records / 1,163 ids, 5,822 / 1,162, and 5,788.
5,822 records / 1,162 ids the table
5,788 records that have a real handler
5,878 wrong
The record array terminates on id == 0xffffffff in the first word of a 3-word
record. Two of the tests reached that word with `if c != MAX { ... }` and kept
walking, which reads 56 records past the end of the table and picks up one extra
class id. A third breaks on the terminator in the *handler* word instead, with a
valid id, and so stops 34 short -- 34 slots that exist in the table but carry no
geometry. Both stopping points are correct for the question each test asks.
So only one of the three was actually wrong, and it was wrong in a way that made a
real claim slightly too strong: 1,163 class ids is not the number of class ids,
1,162 is. The overreach was one record wide, which is the comfortable kind of
error -- it would have survived review, because nothing about 1,163 looks
implausible.
Worth being plain about why this sat unresolved for so long. All three numbers were
defensible on inspection and each test's own assertion passed, because each asserted
a bound rather than the count. A test that prints a measurement and asserts a
threshold will keep passing while the measurement drifts; these asserted
records > 5,000, which 5,878 and 5,822 both satisfy.
Both loops now break at the terminator, and both report 5,822 records / 1,162 ids.
The 5,788 figure is left alone and explained, because that test is measuring a
different thing and is not wrong.
Pulling the 0x80 convention out of the renderer and into gc_formats::gs -- the
side benefit I claimed last commit rather than took -- immediately produced a
contradiction, which is the argument for having done it.
crates/gc-engine/src/level.rs already read it inline:
let a = if c.a < 0x80 { c.a as f32 / 128.0 } else { 1.0 };
My first alpha_to_8 scaled by 127 instead of 128, which made 0x7F fully opaque
where the renderer had it at 253/255. One of them was wrong and it was not the new
one: the disc convention is a 0..0x80 range with 0x80 as the top, so 0x7F is
127/128, not 127/127.
Corrected to match, and the doc now records why -- including that the renderer was
already right, so this is a correction to code I wrote two commits ago rather than
to anyone else's.
One genuine ambiguity remains and it is stated rather than papered over: 0x40 is
exactly the midpoint, and the renderer works in floats where it is exactly 0.5.
Widening to an integer has to round, and this rounds half away from zero giving
128. There is no way to check that against the disc -- it needs a frame, which is
the tfrag morph item's unblock condition -- so it is documented as a choice.
The engine's call site keeps its inline form and now says why: it wants a float,
and the rule has one live site. Converting it would have meant changing working
code for no benefit, which is the opposite of what extracting the rule was for.
Worth noting what actually happened here. Extracting a rule to remove a magic
constant found that two expressions of it disagreed. Neither was obviously wrong
from its own code -- mine looked cleaner, and was not. That is a better argument
for centralising a rule than any amount of saying so.
Finishes the rule layer. Two traps, both of which produce a picture that looks
slightly wrong rather than obviously broken:
AFAIL READS BACKWARDS. The PS2's alpha-test register holds an enable, a 7-bit
reference, and a bit that looks like "fail if alpha is above the reference". On
the hardware it means the opposite -- AFAIL = 0 discards where alpha is BELOW the
reference. passes_alpha_test takes the side explicitly rather than encoding the
register, and both directions are tested so the convention is pinned in code
rather than in a comment.
THE OPAQUE BIT MUST STAY OUT OF THE COMPARISON. 0x80 is fully opaque, but it is
also numerically 128, so a test that compares the raw byte against a 0x40
reference would have every fully-opaque fragment pass every threshold, and nothing
would ever be discarded. passes_alpha_test clamps the opaque bit off first, and
there is a test that asserts exactly that -- the failure mode here is a scene that
renders correctly with the wrong shading, which is not something anyone notices
by looking.
Blend is the four GS primitive types, decoded from the primitive-type byte and
named by what they do rather than by the documentation's IIIB/IIIA/IIIC labels,
which name two different modes identically. Out-of-range types decode to Off
rather than panicking, because the byte comes from asset data. Addition clamps at
255 rather than rescaling, matching the GS's 7-bit clamp; a caller wanting a true
average blends before widening, and the doc says so.
That completes the four rules the roadmap names for this item:
alpha 0x80 == 1.0 alpha_to_8
CLAMP wrap clamp01
mip selection mip_level
blend + alpha test BlendMode, passes_alpha_test
Nine unit tests, no wgpu work, and the roadmap item stays [~] rather than [x] --
the rules are the input to the renderer, not the renderer.
Worth noting the pattern in both traps: each one produces output that is plausible
and slightly-off, with no error anywhere. That is the same shape as the sign
inversion in mip_level and the byte-order bug in the anchor search. The tests are
the only thing that catches this class, which is the argument for having them even
though nothing here needs a GPU.
Picked Phase 3's GS item and did the part that is checkable without a GPU -- the
rule layer the renderer will have to feed wgpu. New gc_formats::gs with three of
the four rules the roadmap names, each unit-tested.
alpha_to_8: 0x80 IS OPAQUE. The obvious implementation of this is
let v = value & 0x7F;
and it is wrong, because 0x80 & 0x7F is 0 -- which is precisely the mechanism
that makes a 7-bit alpha read as "half" to anything treating the byte as 8-bit
unsigned. The first version of this function did exactly that and the test caught
it. Below 0x80 the value is genuinely 7-bit and scales onto 1..255, so 0x7F is
opaque too and no non-zero alpha vanishes.
clamp01: CLAMP wrap, with 1.0 kept AT the far edge rather than wrapping to 0.
Under repeat sampling an exact 1.0 is a discontinuity; under clamp it is the
edge texel, and the game loads its textures that way.
mip_level: round(log2(image / on_screen)) clamped to the chain. The ratio is
image-over-screen and not the reverse -- a SMALLER on-screen size selects a
HIGHER mip level. I had this inverted in the first version, which made
mip_level(1.0, 64.0, 8) return 0 where it must return 6, and the test caught
that too. Both bugs were the same kind: a plausible implementation of a rule
whose sign convention is not obvious, and the only thing standing between them
and a commit was the test.
Not started: wgpu itself, and blend/alpha-test behaviour. This is the rule layer
only, and the roadmap item stays [~] rather than being ticked, because three of
four rules and zero of the GPU work is not "done".
Two of the four rules the item names were already half-present in the renderer --
moby alpha and the 1/4096 UVs -- so making them explicit here rather than scattered
is worth something on its own: 0x80's meaning was previously implicit in however
each call site happened to read the byte.
The gate asks for the relocated-copy result to be cross-checked against an
independent cluster hash. OpenRAC/rac2-gc-decomp's config/levels/functions.tsv is
exactly that: 7,765 functions carrying a content fingerprint and the levels each
appears on. 3,452 appear on all 27, in 2,984 distinct fingerprints.
Cross-checked against our own masking of the level-0 copy of each:
our masked groups 3,389
reference fingerprint groups 2,984
pairs the reference merges, we split 3,231
pairs we merge, the reference splits 0
ZERO is the number that matters. Whenever our masking says two functions are the
same code, the independent fingerprint agrees -- our method never merges what
theirs separates. It is strictly finer: 3,389 groups against 2,984, splitting
3,231 pairs the reference keeps together.
The likely reason ours splits is that the reference fingerprint evidently ignores
something our mask preserves, so functions it calls identical differ in the words
our mask keeps. Coarseness, not contradiction. That is the compatible reading and
it is the one the 0 supports; had it been non-zero, this would have been a
conflict and worth chasing.
THE COMPARISON WAS INVALID BEFORE IT WAS VALID
The first run reported near-unanimous disagreement -- 1,900 of 3,392 pairs
differing -- which read as a significant finding. It was a category error in the
harness: key() masks the FIRST FOUR WORDS, while the reference fingerprints the
WHOLE function. A four-word summary compared against a whole-body hash is not a
comparison of two clusterings, and it produced confident numbers either way.
Recording it because nothing in the invalid output distinguishes it from a real
conflict, and the correction is only visible from the harness, not the numbers.
That is the third time this session that a measurement error produced a result
that looked like a finding -- after the constant-zero validity test and the stride
sampling artifact -- and the common factor is comparing things whose denominators
or extents differ.
The gate's second half is met, with "agrees" meaning "never contradicts" and ours
the finer of the two partitions.
Picked the first open Phase 3 item. "Extend the level table from 19 to 27" is
about what the KEYS are, not how many there are.
ReRAC, which this project follows architecturally, has rc_game::map::LEVELS = 19
-- Ratchet & Clank 1's count, and this repo has no equivalent yet. Reading every
container header's own id off the disc gives:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 22 23 24 25 26 30
27 keys, and 21 is NOT one of them. The container files run LEVEL0.WAD through
LEVEL26.WAD, but LEVEL21.WAD reports id 30, so the id space is 0..20, 22..26, 30 --
one number was cut from the disc and reused by nothing. That matches what
LevelHeader::id already documents, now as the table itself.
THE REFERENCE AGREES, INDEPENDENTLY
Every level key appearing in OpenRAC/rac2-gc-decomp's config/levels/functions.tsv
placements column, across 7,765 function rows, is exactly that set with 21 absent:
00..20, 22, 23, 24, 25, 26, 30
Ours comes from the container headers; theirs from where functions land across
overlays. Two unrelated routes to the same key set, which is the only reason to
trust a hand-typed table like this one.
Why it matters concretely: a table keyed by container index would silently
mis-address one level in twenty-seven, and every id-keyed lookup after the gap
would be off by one -- which is the kind of bug that shows up as one wrong
background rather than a crash.
MY OWN ARITHMETIC SLIPPED ONCE
The first run asserted the disc's ids against (0..=20).chain(22..=27).chain([30]),
which is 28 entries -- 22..=27 should be 22..=26. The test caught it against the
disc, which is the right way round. Worth noting that the reference key set I
extracted by hand earlier was correct and the Rust expression was not, so the hand
check and the code check disagreed in the right direction.
Asserted in level_table_has_twenty_seven_keys_with_twenty_one_absent, from the
disc rather than from a copied constant, so it fails if a container id moves.
269 tests, clippy clean.
Finished the unblock attempt with two cheap tests on the 187 level-0 class
headers, because "which field points to the animation" needed to be ruled in or out
properly rather than guessed at.
NO HEADER u32 EQUALS THE SPAN'S OWN TABLE OFFSET, on any class. So the varying
fields are not self-relative pointers to the part table.
THE LARGEST VALUE IN ANY HEADER u32 IS 3,332,678,933. At +0x0c the value is
16,711,748 across classes. That is packed data or an identifier, not a
displacement.
So the header's varying fields are a MIXTURE: some hold in-span offsets (+0x10
through +0x1c run 192 to 10,800, comfortably inside the spans) and some hold packed
values far too large to be offsets. There is no single "field that points to the
animation", because not all of them point anywhere.
That is a real narrowing even though it is a negative. If per-class animation data
exists on the disc it is INSIDE the class span, not at a separate pointer target.
So the next step is not "find the pointer" -- it is to subtract the geometry the
parser already accounts for from the span and look at what remains. That is a
bounded question with an existing control, rather than an open-ended one.
moby_anim is still not unblocked. It is now bounded rather than open.
The other three blocked items are unchanged and, as recorded, not reachable from
here without something this repository does not have: a debug build or an external
format description for collision, ground truth for volumes, and a captured frame at
a known camera distance for the tfrag morph.
Moving to Phase 3 next. Its first unchecked item is "extend the level table from 19
to 27" -- concrete, bounded, and independent of everything above.
Asked whether the blocked items could be unblocked, and tried the one that looked
tractable. It did not work, and this records where and why rather than leaving the
attempt implicit.
VERDICT PER ITEM
moby_anim probably -- disc-side work. TRIED, stalled, see below
tfrag morph maybe -- would need the engine's own morph code
volumes no -- needs a count of level 0's trigger volumes from a map
collision no -- needs a debug build or an external format description
THE ATTEMPT
MobyInstance names animSeq, animLayers, jointCnt and jointCache. Those are RAM
pointers and useless on the disc, so the search moved to what they point AT,
reached from the moby class blob.
Level 0, 187 moby classes. The span opens with the part-table offset: 0x80 on 69
classes, 0x90 on 34, ranging 0 to 229,184 across the rest. So each class carries
roughly a 0x40-byte header before its parts, and that header is unexplored.
Per byte over +0x00..+0x40, how many of the 187 classes are non-zero:
+0x00 177 the part-table offset
+0x07 185 8 distinct values, 1..3
+0x26 181 100 distinct -- varies per class
+0x27 187 8 distinct
+0x2d 187 8 distinct
+0x30 170 120 distinct
+0x3c 184 120 distinct
+0x3e 187 110 distinct
Six or seven fields are constant across every class and about a dozen vary. None
is identifiable. The obvious discriminator -- find the field that points OUTSIDE
the span -- does not resolve, because every varying field holds values in the tens
of thousands, the same order as the span offsets themselves. They may be
intra-span pointers or they may be sizes, and nothing here separates them.
WHERE IT STALLS
Distinguishing them needs one of: the class loader in the overlay's .text, which
is now readable with the fixed disassembler, or a class whose animation is known
from outside the disc. Neither has been done, and the first is the obvious next
step rather than a dead end.
So moby_anim is NOT unblocked. What the attempt does establish is a location, a
field count to classify, and a specific reason the easy discriminator fails --
which is more than the item had, and considerably more than "still open".
The other three are unchanged and I do not think they are reachable from here.
Collision needs the assert, which is unreferenced in retail, or an external
description of R&C2's collision cell. Volumes needs ground truth nobody in this
repo can supply. Both are written up with those conditions attached.
The 16-byte reading was retracted without a replacement. This searches for one --
every tie vertex region, strides 4, 8, ... up to len/16, keeping any stride where
some byte offset is constant across every unit:
regions examined 1,844
no stride found at all 1,139 (61.7%)
stride 64 491
stride 56 120
stride 72 49
stride 48 12
stride 16 3
64 where a stride is detectable, not 16. And for 1,139 regions there is no stride
in range producing a single constant byte -- those are genuinely variable, or the
element is larger than region/16, which is the same statement.
THE CONTROL THAT CHANGED THE ANSWER
The first run reported 1,561 regions at stride 64. Wrong, and wrong in a way worth
recording: a large stride divides a region into fewer units, and fewer units makes
an accidental constant more likely. Region length is fixed across candidates, so
the widest stride always has the fewest units and the best chance of constancy by
luck alone. The search was reporting its own sampling.
Requiring every candidate stride to cover the SAME minimum of 16 units, and
preferring the SMALLER stride on ties, drops it to 491 and promotes 56 and 72 into
view, which were invisible before.
This is rule 3a's sampling failure in its purest form -- an aggregate over
candidates with different denominators, where the largest denominator wins for a
reason unconnected to the data. The first number was not imprecise; it WAS the
artefact. Worth noting that without the control I would have reported "64" with
confidence and been right by accident, three times the observed figure.
What this does not settle: 64 is the stride where one is detectable, which is not
the same as the stride. The 1,139 regions with no constant byte at any tested
stride leave the element genuinely unknown. But 16 is now excluded quantitatively
-- 3 of 1,844 regions show a constant byte there -- which is the retraction's
content, as a measurement, and a third independent check now agreeing with it.
The stride retraction (e5cd3df) disowned the 16-byte element reading and the
95,612-vertex population that came with it. Two committed tests still compute
exactly those figures and still pass:
tie_vertex_element_is_16_bytes_with_fixed_texcoords
tie_vertex_layout_is_selected_by_i16_3_high_byte
They pass because the arithmetic is self-consistent about a WITHDRAWN population --
the same shape as the chunk uniformity claim, one level up. The parser slices at
16 bytes, the test reads the slices, the numbers agree, and nothing in the loop can
notice the stride is wrong. A green test that cannot fail is the exact failure mode
this session has been cataloguing, and here it was left in place by someone else's
retraction rather than mine.
Both are KEPT and marked, for a specific reason: each is the record of what the
16-byte reading produces, so the retraction is visibly about the stride rather
than about the arithmetic. Deleting or renaming them would erase the thing the
retraction is a retraction OF, and there would be no way to check later that the
numbers moved because the layout changed rather than because someone edited a
constant.
The marking says, at the top of both:
Do not quote the 82.5% figure as a property of tie vertices. It is a property
of one reading of the vertex region, on a population the retraction disowned.
TWO DENOMINATORS, BOTH CORRECT, 92.9% APART
The layout test reports (4,5) as 12-bit fixed on 92.9% OF VERTICES. A separate
measurement on flat records found only 16 of 1,488 DISTINCT i16[4] values are
multiples of 4096.
Both are right. Most vertices use the handful of UV-corner values -- (0,0),
(0,4096), (4096,4096), (4096,0) -- which are all multiples of 4096, so the
per-vertex rate is high. The other 1,472 are not, so the per-distinct-value rate is
1%.
That gap IS the artifact, and naming both denominators is the most useful thing
available about it. A percentage over vertices and a percentage over distinct
values answer different questions, and only one of them was ever asked about --
which is the same base-rate error as the class-ID slot, where a bound that selected
for small numbers made nine values look like 305x enrichment.
268 tests.
Revisited an older conclusion and found the numbers were fine but the inference
from them was not.
The claim was: i16[3] in a tie vertex is not a skin/joint index, because vertices
sharing a value are 0.95x FARTHER apart than vertices differing in it.
mean pairwise distance, same slot-3 value 14,481
mean pairwise distance, different slot-3 value 13,795
ratio 0.95x
control: same position-x bucket 8,105
The measurements are correct, and the control shows the test works. What does not
follow is the conclusion. The premise was "vertices sharing a joint sit together",
and that is only true for a TIGHTLY LOCALISED joint. It is false in the cases that
are common:
* a rigid tie object -- a leaf, a banner, a piece of foliage -- puts every vertex
on joint 0, so "vertices sharing a joint" IS the whole mesh and the mean
distance is the mesh's own diameter
* a root or hip bone's influence region can span most of a mesh
* bone weights fall off gradually, so membership is not a spatial partition
A 0.95x ratio is precisely what a rigid object bound to one joint PREDICTS.
Absence of clustering is fully consistent with the joint reading; the test never
had the power to separate them. A negative result I did not expect was reported
as a disproof.
This makes the hypothesis STRONGER, for an independent reason: MobyInstance carries
jointCnt at 0x57 and jointCache at 0x58, so this engine demonstrably has joints and
per-moby joint state. A per-vertex field in 0..=254 is exactly the shape of a
joint index.
What was always established still stands, and is all that was: slot 3 is per vertex
(50 distinct values per packet) and not shared across LODs (75.3% same-class
against a 78.7% control). The LOD result is the one thing arguing against a joint
index, though two LODs binding a rigid mesh identically would produce it without
being informative.
So: not understood, with the joint reading now the LEADING CANDIDATE rather than a
disproved one. Corrected in the format file and the roadmap, including the second
mention added two commits ago.
This is the sixth correction in this session to withdraw a conclusion rather than
sharpen one, and the first where the measurement was sound and only the reading
was wrong -- which is the hardest kind to catch, because the number looks right and
is quoted in the commit message as well as the doc.
The decomp carries an authentic MobyInstance layout of 0x100 bytes with every
field named. It is now in gc_formats::moby with a 67-entry offset map, attributed
as a transcription rather than presented as our own finding.
Two fields matter:
0x40 animSeq animation sequence pointer
0x44 animSeqT animation time parameter
0x48 animSpeed playback speed
0x50 animLayers animation layer list
0x57 jointCnt joint count -- non-zero means the moby is skinned
0x58 jointCache joint matrix cache
0xBC oClass object class id -- the dispatch key
OCLASS CLOSES A LOOP THAT HAS BEEN OPEN FOR SEVERAL COMMITS. It is the field the
runtime uses to reach the lvl.vtbl handler tables, so an instance and its routine
are now linked by a named offset, rather than by an assumption about which axis
the two tables share. That assumption -- the positional correspondence -- is still
refuted, and still does not need to be true.
jointCnt and jointCache settle something from earlier in this log BY CONTRADICTION
rather than confirmation. A tie vertex's i16[3] was tested as a skin/joint index
and disproved: vertices sharing it were 0.95x FARTHER apart than vertices differing
in it. That stands and is unaffected -- this is the moby struct, and a moby is a
different object from a tie. Recording the distinction rather than letting "joints"
appear in two places look like a confirmation of the earlier hypothesis.
WHAT THIS DOES NOT DO IS FIND moby_anim. It names the instance-side state, so the
asset to locate is whatever those pointers reference, and we still do not know
where that is. The item's unblock condition is now sharper -- it needs the data
behind those pointers, not the struct.
PROVENANCE, AND WHAT THE TEST CAN ACTUALLY CHECK
This is not verified against disc data and cannot be: moby instances are runtime
objects and nothing of that shape is in the container. The test asserts what can be
asserted without the bytes -- offsets ascend, do not overlap, the last field lands
exactly on 0x100, and o_class sits at 0xBC immediately before triggers.
Worth keeping distinct from the collision finding. That project's FUNCTION ADDRESSES
did not transfer: they landed in NOBITS memory. A struct's field order is a
different kind of claim from an address in a program's layout, so one failing does
not impeach the other -- but the distinction is real, and PROVENANCE.md now records
which is which rather than citing the project as a single authority.
268 tests.
Records and behaviours are different numbers, and the difference is one routine
serving several classes. Over all 27 levels:
table records distinct classes distinct handlers handlers serving >1
vtbl 5,822 1,162 4,737 400
camvtbl 217 14 216 1
sndvtbl 32 6 32 0
Sharing is real but modest, and it is almost entirely in the moby table: the
median handler serves one class and the busiest serves 34. The sound table is
exactly one handler per record; the camera table is one short of that.
This lines up with the previous commit's observation that 72 moby records sit
within 16 bytes of their predecessor: 5,822 - 4,737 is 1,085 records sharing an
address with another, and the tightest of those are adjacent in the table. Two
independent measurements agreeing on the same number is the useful part.
The practical figure is 4,737. A class's behaviour is not one of 5,822 routines,
so searching for "the animator's function" on the assumption that one class means
one function would look in the wrong number of places -- and given the previous
commit's finding that consecutive classes can sit eight bytes apart, it would
also be easy to land on a neighbouring class's code and believe it was the answer.
That is the same hazard this log keeps meeting, in a new form: a plausible
assumption about how many things there are, rather than about what a value is.
Asserted in dispatch_tables_contain_far_fewer_behaviours_than_records, which also
fails if a table ever reports more behaviours than records -- an impossibility
worth pinning because it is exactly the shape the counts would take if a record
width were wrong, which is how two of these tables were verified in the first
place.
265 tests.
Followed the chain through -- dispatch table to handler address to disassembly --
and read level 0's class 2 handler at 0x00380a88:
0x00380a84: jal 0x2ee0d0
0x00380a88: .word 0x0240202d <- VU1 microcode, not a failure
0x00380a8c: lwc1 $f12, 0x40($s1)
0x00380a98: addiu $a0, $s0, 0x20
0x00380aa0: swc1 $f0, 0x18($s4)
0x00380ab0: swc1 $f0, 0x14($s4)
Floats out of one structure, written back into another, with VU1 work alongside --
what a transform update should look like. The chain from an asset observation to the
game's own code is now complete and usable.
TWO CAVEATS, both learned by hitting them:
VU1 microcode interleaves with CPU code. 0x0240202d is a COP2 opcode, not an
undecodable word. A tool that counts .word lines as failures is counting the wrong
thing -- a trap I walked into earlier today with the region walkers, now recorded
so it is not repeated.
Handlers are not all independent functions. Level 0's class 2 is at 0x380a88 and
class 0x0E at 0x380a90 -- EIGHT BYTES apart -- and 72 of the 5,788 records sit
within 16 bytes of the one before them. Consecutive classes share one routine with
per-class entry points inside it, so reading "a handler" gives a fragment. That
matters downstream: a class's behaviour is not one function, and assuming
otherwise would misattribute shared code.
A DISCREPANCY WITH THE REFERENCE
The decomp reports 5,598 of 5,788 moby handlers (96.7%) preceded by a jr ra within
three words. Measured here, the same way, over the same 5,788 records: 5,488
(94.8%). Same population, same test, 110 handlers different.
Both figures support the premise, so nothing turns on it, and I am recording it
rather than picking the number that suits. The reference measured on 2026-10-02
against its own pinned overlay set, so a method or version difference is as
plausible as a data difference and there is no basis here for saying which. The
test asserts the premise (>90%) and deliberately NOT either exact figure.
Asserted in moby_handlers_are_overwhelmingly_function_entries.
264 tests.
Finished the dispatch-table work by decoding the two I had not looked at. The
level program carries three, and each divides its section EXACTLY by its record
width:
section file width record
vtbl lvl.vtbl 3 oClass, UpdateMoby_<oClass>, auxiliary
camvtbl lvl.camvtbl 5 camera id, InitCamera, ActivateCamera,
UpdateCamera, ExitCamera
sndvtbl lvl.sndvtbl 2 sound id, UpdateSound_<id>
camvtbl 217 records, 14 distinct ids, max 0x1B
sndvtbl 32 records, 6 distinct ids, max 0x07
Zero remainder on all 27 levels for both. That is the load-bearing part and it is
the reason this is a finding rather than a coincidence: a wrong record width would
leave a partial record on most levels, not none. Section sizes run 40..52 words
for the camera tables and 0..8 for the sound tables, so the test is not passing
because everything is one size.
Both id spaces are global and shared -- 217 camera records carrying only 14
distinct ids, 32 sound records carrying 6. Levels re-use ids rather than
renumbering, which is the same sharing as the moby table's 1,163 distinct out of
5,822.
Why this is more than parsing. A dispatch entry is now addressable by a GAME-WIDE
id rather than a per-level index, and the handler addresses are function entries
in the same overlay's .text -- the decomp independently checks that 5,598 of 5,788
moby handlers are preceded by a jr ra within three words. With the disassembler
now working on that section, a handler can be read. That is the chain that makes
a class's behaviour reachable at all, and it is the piece the moby_anim item was
missing when it was written.
Asserted in camera_and_sound_dispatch_tables_decode_at_their_documented_widths.
263 tests.
Followed the moby_anim unblock by asking whether lvl.vtbl's oClass indexes
core_index's moby class table. It does not, and the margin is not close:
lvl.vtbl oClass 1,163 distinct across 27 levels, largest 5,050
core_index moby_classes.count 136 .. 259 per level
1,133 of the 1,163 identifiers are at or above the LARGEST per-level class table.
Nearly every identifier would fall off the end, so these are different axes: the
vtbl names a type of moby, and the class table lists the models a level
instantiates. Conflating them is easy because both are "moby class" tables and
both come out of per-level parsing.
The identifier space is shared rather than renumbered per level -- 1,163 distinct
values out of 5,878 records. Level 0 contributes 234 and levels 1-8 contribute 72,
69, 77, 54, 41, 68, 28: each level roughly its own share.
WHICH IS WHERE I OVERSTATED IT TWICE.
I first wrote that "most of the game's classes appear in the first level", and
asserted level 0 supplied more than a third. The data says 234 of 1,163 is 20%.
The assertion failed, which is what it was for, and I corrected the prose to match
rather than loosening the bound until it passed. Then I moved the threshold to a
fifth, which 20.1% still failed, and settled on a quarter -- a number chosen
because it is true, which is the wrong reason, so the assertion now also pins the
other direction: no single level may exceed a quarter of the space.
The useful consequence is the one the moby_anim item was waiting for. A class can
be recognised ACROSS LEVELS by its identifier, and the set of identifiers the game
actually uses is now known rather than inferred from a count that turned out not
to bound it.
Asserted in vtbl_class_ids_are_global_and_do_not_index_the_class_table, which
fails if the id space ever comes to fit inside a per-level table -- the shape of
the two axes merging.
262 tests.
Used the working section parser to reach something the decomp describes but does
not decode. lvl.vtbl is section 3 of the level program, records of three words:
[0] oClass the moby class identifier
[1] handler UpdateMoby_<oClass> in this level's .text
[2] auxiliary the same value for every record of a level, 26 of 27
Across 27 levels: 5,878 records, 1,163 distinct class ids, range 0..0x13BA (5,050),
median 38 new ids per level. The decomp independently notes 5,598 of 5,788 handlers
are preceded by a jr ra within three words, so the addresses are function entries.
WHY THIS RETIRES THE EARLIER BOUND
moby_classes.count is 238 on level 0. 1,135 of the 1,163 real ids -- 97.6% -- are
at or above that number. So "value < moby_classes.count" was never a test for "this
is a class id". It was a test for "this is a small number".
That is the mechanism behind the chunk class-ID artifact I retracted two commits
ago:
slot values 1, 3, 4, 5, 10, 11, 12, 13, 15
all below 238, so all scored as hits, giving 305x enrichment
The bound was wrong BEFORE the artifact was noticed, and the artifact was a
consequence of it. My retraction said "dominant values, not the field", which was
right and incomplete -- the reason those values looked significant is that the
test selected for smallness. Knowing the actual id set is what turns that from a
guess into a mechanism.
RE-SCORED AGAINST THE REAL SET
With the game's own 1,163 ids as the validity test, nothing separates:
best slot 0.375 (type f2ef4275 slot 34)
chance rate 0.018 (1,163 ids in 65,536 values)
a dozen slots 0.25 .. 0.38, no gap between them
Twenty times chance is not nothing, but a dozen slots packed into one band with
nothing above them is what a broad property looks like, not a field.
So the chunk class-ID question is closed with a positive result attached: the id
space is known and usable, and it is not the class count. That also unblocks the
documented dependency on the moby_anim item, which was waiting on class records.
Asserted in moby_class_ids_come_from_lvl_vtbl_not_from_the_class_count.
262 tests.
Followed the route I recorded as remaining -- find how the code addresses the
assert string at 0x001a8998 in .lit. It does not address it.
Level 0's program is 2,751,268 bytes. Searching it for both ways an address can
be materialised:
address low half 0x8998 0 occurrences, word-aligned or not
$gp-relative offset 0x99a8 0 occurrences, word-aligned or not
exact 32-bit 0x001a8998 0 occurrences
Validated by a positive control in the same run: the .lit load base's own low
half, 0x7c80, occurs exactly ONCE, word-aligned. A search that finds that will
find any real reference, so the zeros are real.
So no instruction and no data word in the retail level program references the
collision assert string -- not by lui/addiu, not by a literal pointer, not by a
table entry. The text is present in .lit at overlay+0xd18, and the same text also
appears in the boot binary at file offset 0xA9558, but nothing points at either.
The reading that fits: the assertion is compiled out in retail and the string is
dead data. Debug string pools routinely keep messages whose only user was an
#if-ed-out assert. The decomp's table lists this name because it recovered it
FROM THE STRING, not because it found code calling it -- which is worth noting,
because a name in a symbol table looks like evidence of a caller and is not one.
That closes the last route this file records for collision. Getting the cell
layout from the disc would need a build with assertions enabled, a debug listing,
or an independent description of R&C2's collision cell. None of those is on this
disc, and saying so is more useful than another scan.
Summary of collision, all of it now checked: the payload is verbatim in RAM, it
does not walk as fixed-stride cells, the disc-side segmentation is exhausted, the
string that would have led to the routine is unreferenced, and both of the
decomp's addresses are dead ends.
Finished the task I named last commit rather than leaving it as a lead.
Camera_CollPrimTest is declared Camera_CollPrimTest(void *prim, void *grid) --
two pointer arguments, which in MIPS arrive in $a0 and $a1.
The function's prologue is at 0x002db078, matching the +0x70 teardown at
0x002db354. The two instructions after it:
0x002db078: addiu $sp, $sp, -0x70
0x002db07c: lw $v1, -0x3f4c($v0) <- $v0, not an incoming argument
0x002db080: addiu $a0, $zero, 0x6f <- $a0 set to a constant
$v0 is caller-saved scratch, not an argument register, and $a0 is overwritten
with the constant 111 on the next instruction. No void* reaches this function, so
it is not Camera_CollPrimTest, whatever the assert table labels it.
It reads four consecutive words from boot memory -- 0x1a7348 through 0x1a7354 --
all below the level program base at 0x001A7C80, which is the boot's resident
region and identical in every program. A different data source from the collision
payload, and the 48-byte stride loop belongs to whatever that structure is.
So the lead is dead on evidence rather than on doubt, and both routes the decomp
appeared to offer are now closed:
Camera_CollPrimTest @ 0x001E7A50 lands in NOBITS, all zero
GridOutOfBounds @ 0x002db350 a function with no pointer arguments
Neither address reaches the collision parser. Worth noting how cheap the check was
once the tool worked: three instructions past a prologue is enough to disqualify a
function whose name asserts a signature. The expensive part was not the analysis,
it was the disassembler that stopped after five words and made everything look
empty.
The route that remains is the one already recorded -- find how the code addresses
the assert STRING at 0x001a8998 in .lit, which is not a $gp-relative load and is
not a symbol address in either table. Note that the string and the code are five
megabytes apart in different sections, so whatever reference exists is not a
simple local reference, and that is the thing to look for.
Vendored rac2-gc-decomp is 13 commits behind, 12,375 lines added, almost all src/.
It names no collision data structure, so the format is still not handed over. Two
things are usable, and one precondition had to be fixed first.
THE PRECONDITION: the disassembler was stopping after five instructions.
tools/overlay-anchors called disasm_all across the whole window, which halts at
the first word capstone will not decode. In code regions that is almost
immediately, so every window came back with a handful of instructions and I had
concluded there was nothing there. It now walks four bytes at a time, prints
.word for anything undecodable, and reports how many failed. That is the whole
reason the earlier attempts produced nothing -- not that the code was absent.
WHAT THE DECOMP'S ASSERT TABLE ACTUALLY GIVES
assert-message-names.tsv maps each assert name to an address. For level 0,
CameraCollPrimTestGridOutOfBounds is 0x002db350. That address is not the assert
call and not the string. It is:
0x002db350: jr $ra
0x002db354: addiu $sp, $sp, 0x70
an epilogue. The address lands on the last group of the function before it, and
0x002db35c is a fresh prologue. It is also in .text (0x002a3b80..0x00447900),
five megabytes from the .lit string at 0x001a8998 -- which explains the "$gp search
returns zero" result from the other side. The string and the code are in different
sections, five megabytes apart.
WHAT IS ACTUALLY THERE
Immediately before the epilogue:
0x002db2dc: lwc1 $f20, 0xc($s0)
0x002db2ec: add.s $f20, $f20, $f21
0x002db2f0: c.olt.s $f0, $f20
0x002db304: jal 0x2db4b8 <- the assert call
0x002db314: lw $v0, ($s1) <- a count
0x002db31c: slt $v0, $s2, $v0
0x002db320: bnez $v0, 0x2db2cc <- loop back
0x002db324: addiu $s0, $s0, 0x30 <- stride 48 bytes
A loop over 48-byte records, reading a float at +12 and accumulating, bounded by a
count, with f22 set to 1.0f just above it -- a minimum-height clamp.
NOT CLAIMED to be the collision grid. It is a strided float loop beside an assert
the decomp labels CameraCollPrimTestGridOutOfBounds. A plausible address near a
plausible assert is a lead, not an identification, and that is the exact gap this
log keeps tripping over. Confirming it means tracing the function's arguments back
to the collision payload and showing the grid's leading u16s -- level 0 is 11 x 33
-- drive the count at ($s1). Not done, and not asserted.
The undecodable words are VU1 microcode (COP2 opcodes), which is expected: the
function interleaves VIF-fed VU1 work with FPU arithmetic, like tfrag's.
Tools build, clippy clean, 260 tests.
rac2-gc-decomp names the two collision entry points:
Camera_CollPrimTest 0x001E7A50 camera-to-world boundary check
MB_CheckCollPill 0x001E8890 pill/capsule check for moby instances
That looked like the collision unblock. The $gp string search has failed twice
now, and a named function at a known address is exactly what it failed to
produce.
They are not usable. Checked against levels 0, 5, 13 and 26 with
parse_overlay_sections:
Camera_CollPrimTest @ 0x1e7a50 -> section 1, kind 8 (NOBITS)
MB_CheckCollPill @ 0x1e8890 -> section 1, kind 8 (NOBITS)
Both land inside the level program's zero-filled section. Every byte is zero and
the following 256 bytes contain no non-zero byte at all -- code cannot be there.
Same on all four levels, so it is not a level with an unusual layout.
So the addresses belong to a different address space than a parsed level overlay.
Most likely the boot executable's own layout, or the v2.00 build; that repository
covers both. SCUS_972.68 is USA v1.01 and our disc matches it exactly, but that
fixes the disc, not which program's address space a symbol address is quoted in.
The disc matching is exactly the kind of check that looks sufficient and is not --
a disc can pin the bytes and still leave you looking in the wrong program.
Worth noting the method that caught it. The temptation was to take the address,
open the overlay at that offset, and start reading instructions. What settled it
was checking which section the address falls in, and its KIND: a NOBITS section is
zero-filled by definition, so an address inside one is a contradiction rather than
a lead. The section table made that a two-line check.
This closes the shortest route to the collision cell layout, for a checkable
reason rather than a vague one. The route that remains is the one already
recorded: find how the code addresses the Camera_CollPrimTest assert string, which
is not by a $gp-relative load.
The Phase 3 port brought parse_overlay_sections with it, which makes the overlay's
shape checkable for the first time. It corrects a claim I committed when
vendoring the decomp.
What I wrote then: overlay.bin +0x00 is 0x1A7C80, that is the .lit load base, "so
the first header word is the load base of the section this file is", and therefore
"overlay.bin IS the level's .lit section".
It is not. overlay.bin is the whole seven-section program. .lit is its first
section: 39,388 bytes at 0x1A7C80 on level 0, against a lump of 2,751,268.
section 0 .lit 0x1A7C80 on all 27 levels
section 1 .bss kind 8 (NOBITS) on all 27, zero non-zero bytes
sections 2-6 .data, lvl.vtbl, lvl.camvtbl, lvl.sndvtbl, .text
kind sequence [1, 8, 1, 1, 1, 1, 1] identical on 27/27
section sizes fall 112 bytes short of the lump on 27/27
(each section's size counts its own 16-byte header)
The error was reasoning from a constant to a statement about the file's nature. A
load base tells you where something is placed, not what the containing file is.
That is the same move as the other four withdrawals this session -- the 305x
class-ID slot, the UV-corner texcoords, the VIF code word, the DAW pointer -- all
of which read a single plausible item as evidence about the whole.
Worth noting what survives. The collision assert string is in SECTION 0 on all 27
levels, so the earlier $gp-relative search located it correctly even though the
reasoning about the file was wrong. The string being in .lit and the file being
.lit are different statements and only the first is true. A wrong reason can reach
a right answer; it is still worth correcting, because the next person to reason
from it would not.
Asserted in overlay_bin_is_the_seven_section_program_not_the_lit_section.
259 tests.
Picked sound_bank rather than tie, where three discriminators have now come back
negative and a fourth would be guesswork.
The body after the 24-byte head opens with twelve words. Six are settled across
all 27 levels:
word value note
0 varies per level not identified; 27 distinct
1 60 constant
2 varies per level not identified; 23 distinct
3 0 constant
4 daw_size == the header's daw_size, exactly
5 daw_size the same value again
6,7,8 0 constant
9 127 constant
10 1 constant
11 0 constant
Level 0's words 4 and 5 are both 1,482,000, and daw_size is 1,482,000.
THE POINTER THAT WAS NOT ONE
Words 4 and 5 were briefly recorded as pointers into the DAW region. They are not.
The test that promoted them was "does this value fall inside [daw_offset,
daw_offset + daw_size)" -- and the DAW region is ~1.5 MB starting near the front of
a ~1.5 MB extent, so almost any plausible 32-bit word lands inside it. 1,482,000 is
merely just under the top of exactly that range.
A range test is not an identification. What survives is EQUALITY with daw_size,
exact on all 27 levels and asserted, which is a claim that can fail rather than one
that is almost always true.
That is the fourth time this session a plausible reading of a single number has had
to be withdrawn for something falsifiable: the 305x class-ID slot, the UV-corner
texcoords, the VIF code word, and now this. The pattern is consistent enough to be
worth stating as a rule -- a test that passes almost always is measuring the shape
of the data, not a property of it.
Word 0 gets one note: levels 1 and 4 hold 13,042,052 and 13,042,054, two apart,
while their daw_sizes differ by 63,504. Whatever it is, it is not a size.
The remainder of the SBlk body -- the per-sound record table -- is still undecoded,
so sound_bank stays open. A SBlkBodyHead struct carries the twelve words with the
constants asserted in its docs.
259 tests.
Last commit read bit 31 of a tie vertex region's first word as "VIF code word"
and told a two-encoding story on it. The split is real; the identification is
wrong, and I checked it before building anything on top.
Walking those regions as VIF packets, levels 0-2, 914 records:
walk reaches the region end 17
walk dies after exactly ONE packet 1,571
packets before dying: min 0, median 1, max 4
tfrag walks VIF the same way -- VifCode::decode, then num * 4 payload words -- and
reaches the end of its lists every time. So the walk is not the problem; these
regions are not VIF packet streams. Bit 31 is set on whatever this word actually
is, and that is still unknown.
The specific thing that made me overreach: I quoted 0xf5c1a3c5 as "a VIF code
word, cmd 0x75 num 193" and treated that as establishing the format. Decoding one
word as a packet header says nothing about whether the next 193 words are its
payload. Here they are not -- the walk dies immediately after.
This is the third time in this area that one plausible word produced a confident
story: the 305x class-ID slot, the UV-corner texcoords, and now this. All three
shared a shape -- a single readable item at the head of a population, treated as
evidence about the whole. The bit is the only thing the two kinds reliably
differ on, so it stays; the name goes.
The test now asserts BOTH the split and the walk failure, which is the part worth
keeping. If these ever do turn out to be VIF, the test fails and says so, rather
than two findings drifting apart silently -- which is how "half VIF, half flat"
would have sat in the docs indefinitely.
Docs corrected in both roadmap and format file, including the quoted example.
258 tests.
Picked the tie flat encoding to close, since e5cd3df left it as the largest
unidentified field. What came out was a retraction of my own earlier claim.
The texcoord slot was recorded at i16[4]/i16[5] on a 73.6% "both multiples of
4096" score. Two problems, either of which would be enough.
First, that 73.6% was computed on the 16-byte element reading that e5cd3df
retracted. Half the records are VIF packets and half are flat, so the figure
describes neither population cleanly.
Second, and decisive on its own: measured on the FLAT records only -- levels 0-2,
930 records, 47,909 vertices -- the question that separates a 12.4 fixed-point
texcoord from anything else is how many of a slot's distinct values are multiples
of 4096 at all. For a texcoord it is all of them.
slot distinct of which are multiples of 4096
i16[4] 1,488 16
i16[5] 1,261 16
i16[6] 966 4
i16[7] 489 5
One percent. The dominant PAIRS do look like UV corners -- (0,0) 5,773,
(0,4096) 5,500, (4096,4096) 5,059, (4096,0) 4,521 -- and that is precisely why
the claim was persuasive. But it is a statement about the most frequent values,
not about the field: 1,472 of the 1,488 values in i16[4] are not multiples of 4096
at all.
This is the moby class-ID artifact one level down. A small set of dominant values
that happen to satisfy the test, scored as though the whole population did. The
arithmetic was right about the popular values and the conclusion was wrong about
the field. The retraction commit itself warned about exactly this shape, which
makes the repeat more useful than the original claim was.
So the texcoord slot of a flat tie record is NOT identified. What survives is the
boundary work -- vert_ofs + vert_size + scissor_size is the region length -- and
the LOD1/LOD2 sharing result, which is stride-insensitive and survives the stride
retraction.
Both docs updated: the roadmap's two [x] claims about the texcoord field and about
i16[3] selecting its encoding are now [~] with the evidence, and the format file
has a retraction section giving the table above.
I did not find a replacement reading for i16[4..7]. Three discriminators were
tried -- LOD sharing, multiple-of-4096 rate, value-set size -- and each came back
ambiguous or negative. That is worth recording as the state rather than
generating a fourth hypothesis.
258 tests.
Starting from the phase's stated exit criteria rather than the checklist, because
"complete" should mean the gate, not the feeling of the list.
THE GATE IS MET. Both halves, checked by inspection:
* Golden tests. Every module in gc-formats/src is referenced by a test or another
module's test path; none is unreferenced. gc-extract/tests/disc.rs carries 50
real-disc tests, and overlay_reloc's mask test resolves a level-01 function on
all 26 other overlays.
* Counts are failures, not warnings. No parser writes warning-style output
outside #[cfg(test)] -- the only eprintln! calls in gc-formats are inside test
modules. Count and length problems surface as typed errors: 42 uses of
ErrorKind::Invalid / InvalidLength, including global_wad's "pair table is not a
whole number of entries", which is precisely a header count disagreeing with
the data.
THE GOAL IS NOT MET, and saying otherwise would be the thing AGENTS.md forbids.
The goal reads "every asset type parses into plain Rust values". Four do not:
collision cells, volumes, moby_anim and the tfrag morph are located or framed but
have no decoded fields. The gate was written to be checkable and it does not test
for that, which is a gap in the gate rather than a reason to declare victory.
So the remaining gap in the phase's own discipline is that the four blocked items
did not say what would unblock them, which AGENTS.md requires and which none of
them did. Closed now, each with the specific unblock condition:
collision cells the game addresses its assert string by something other than a
$gp-relative load -- a %hi/%lo split that does not straddle
zero, a pointer table, or another section. A plain $gp search
returns zero and always will.
volumes ground truth: a count of level 0's trigger volumes from a map,
to match against the 144-byte table's population. No class name
containing "volume" exists on the disc, so the dispatch route is
closed.
moby_anim needs moby class records decoded past the header -- a
dependency, not a measurement.
tfrag morph needs a frame captured at a KNOWN camera distance, so the
expected blend is computable. Requires driving the running game,
not analysing a static savestate.
None of these is an approximation, and none is marked done.
258 tests, 50 in the disc suite, clippy clean on my own code.
Read up on the five commits that landed since f1f0b51. The headline is e5cd3df,
which retracts my tie vertex position claim: the stride is not 16. The LOD1/LOD2
sharing evidence survives, because it is stride-insensitive, but the element
layout and the 95,612 count do not. This commit works on what that retraction
opened rather than trying to rescue it.
The split is visible in the region's first word. A VIF code word has bit 31 set:
first word is a VIF code word 2,007
first word is not 2,102
(levels 0-5, every record with vert_size > 0)
So roughly half of all tie records address their vertex data as VIF packets and
half as a flat 16-byte array. Flat records repeat a 16-byte unit whose last word
is constant at 0x00001000 and whose third word walks down
f000, e000, d000, c000; VIF records start with e.g. 0xf5c1a3c5, cmd 0x75 num 193.
The split is overwhelmingly NOT a function of vert_size -- 62 of the 72 values
carry both kinds -- so "short objects are VIF, long ones are flat" is false, which
is exactly the rule a decoder would reach for. The 10 that carry one kind are the
rarest sizes, not a band.
This also explains why the flat reading looked convincing rather than arbitrary.
In a flat record, i16[6] is 0x1000 and i16[7] is 0x0000 at the end of each unit --
the (-1, 0) / (1, 0) corner pair a texture coordinate makes in 12.4 fixed point.
That is where the earlier texcoord reading came from, and it holds cleanly on the
half of the records that are flat. It was never wrong about those; it was applied
to all of them.
It does not settle the stride. It says the region has to be dispatched on encoding
before anything inside it can be read, which is why no single flat stride fits
every record. Asserted in tie_vertex_regions_split_between_vif_and_flat.
ALSO: the roadmap still listed two Phase 3 items as open that a94704d has already
done -- "Port the masking and code-identity logic" and "Relocation::func and
Relocation::data". Both exist, are exported, and the masking is verified against
the disc: the masked form of a level-01 function (858 words at 0x2a3b88) is found
exactly once on each of the other 26 overlays. Both now marked done with the
evidence attached, rather than left stale.
I ran clippy --fix over overlay_reloc.rs, saw it rewrite 110 lines of the new
Phase 3 code, reverted it. The changes were cosmetic -- redundant `as u32` and
reflow -- but it is not my code and the remaining warnings are the author's to
address. Its 11 warnings are pre-existing and left alone.
258 tests.
Ported ReRAC's (vendor/rerac/crates/rc-formats/src/level_overlay.rs) mask /
Relocation machinery into gc-formats:
- parse_overlay_sections, read_overlay: section table {dest,size,kind,entry},
ends on entry mismatch; kind==8 = BSS; plus LevelOverlay with .text words,
text_start and function starts.
- mask: hides relocated fields (j/jal targets, lui of main-RAM 0x0010..0x0040,
%lo/move/$gp completions tracked per-register).
- walk/HiState/address_refs/key: PS2 code walk with branch-delay-slot register
state.
- Relocation::new/same_code/copies/func/data: masked-code index, one-copy
lookup, data-address relocations.
A real-disc test asserts that a level-01 engine function's relocated copy is
found exactly once on all 26 other overlays (gate: every level-01 function is
reachable by masked-code identity on all 27).
Co-authored-by: reaper <reaper@users.noreply.github.com>
Tie class headers ordered near < mid < far on most classes. Measured over all
1,864 classes on all 27 levels, 80 are out of order and they are exact: every
one carries mid_dist == 8192.0 and far_dist == 8190.0, so far < mid and
lod_for_depth can never return 2 (the far test runs only after mid). Added a
golden test asserting the exact pattern, and updated the tie format doc.
Co-authored-by: reaper <reaper@users.noreply.github.com>
Task was to name the axis order and scale of the tie vertex positions by
fitting the class header's bounding sphere. The sphere does not exist: at
+0x30 those bytes are zero on every class. Scanning the header for
float-like words found +0x10, +0x14 and +0x18 (the LOD switch distances,
already named) and +0x40 on 1,676 of 1,864 classes, which is not a sphere.
What the run produced instead is a retraction. Dumping one packet's vertex
region raw shows the first 16 bytes are header material, not a vertex --
0x0000df20 is not a coordinate, and the words after it are the vif and
material words. A 16-byte group does recur at +64 and +128 with identical
structure, so the stride is not uniform: some elements are 4 bytes, some 16,
and the run length varies.
Two candidate bases were measured and neither clusters. With packet +
vert_ofs the per-class extents run median 64,976, p90 65,517; with class +
vert_ofs, median 55,213, p90 64,352. Both span the whole i16 range, where a
position triple in a class-local frame would span a fraction of it. The region
bounds are right -- vert_ofs + vert_size + scissor_size is the length -- it is
the element layout inside it that is wrong.
The i16[3] marker does not rescue it: bucketing every fourth element by the
marker's high byte gives markers 0, 0x48, 0xff and 0x01 with thousands of
vertices each, and only 0x48 shows a position signature (i16[1] extent 425
against 65,535 on the other two axes). The same marker 0 bucket spans all
three axes at full range, so the bucket does not select a layout.
The 95,612-vertex count and the 82.5% texcoord fixedness figure were computed
on the 16-byte reading and are now untested against the correct stride, which
is not yet known. The LOD1/LOD2-sharing test stays the strongest evidence for
positionality and is stride-insensitive.
The defect itself was measured and committed in 572cd18, but the format-doc
section describing it was lost last session: the patch that should have added it
silently failed its assertion, and only the test and the roadmap entry landed.
Same shape as three other lost edits this session. The section is restored here.
Two candidate fixes are recorded alongside it, both measured and both dead
ends:
1. The first 0xf6 00 at or after the last known packet's end. The offsets
found are spread (1082, 1132, 1148, 1150, 1182, 1214 ...), not concentrated,
because the 115 records with a non-zero strip_count carry packets of unknown
length that sit after the last known end.
2. The block's data ends at a long zero run. The byte before the first 8+ zero
run is not dominated by 0xf6 (2, 4, 1, 64, 3, 32, 19 ...), and the runs are
mostly 8-31 bytes, so run length does not discriminate the block's end.
Both are recorded because they are the obvious things to try next. What would
unblock it: a survey of which offsets do bound blocks (the 461 that work may
share a property), or the level loader's own block-walking code, which needs the
overlay relocation from Phase 3.
Range measured: every tie LOD block on the disc, 2,177 blocks.
The last packet's end is known two ways. Its length comes from its record's
packet_quadwords, and every other record's length is confirmed by the next
record's pointer -- so only the last packet of each table is unconfirmed.
Comparing its computed end against where parse_tie_block stopped: the last
packet's computed end lands 2 to 16 bytes past the located 0xf6, on all 2,150
blocks measured.
Two readings fit. Either the terminator was located too early -- inside the
last packet's data -- or the last packet's length over-predicts by a small
amount. The first is likelier: a length error on the last record of every table
would have to be a coincidence. Neither is settled here, and the test asserts
the measurement rather than the reading.
Why the existing test missed it: "2,177 of 2,177 blocks end on 0xf6" is
satisfied by a false terminator exactly as well as a true one -- it checks
that the byte found IS a 0xf6, not that it is the block's end. A test that
confirms the shape it walked to will pass on the wrong byte.
The cost: the block extent is short by that much, so payload and padding
under-report and anything computed from them is correspondingly short. The
block starts are unaffected and RAM-confirmed.
Recorded as a limitation with the measurement attached, not fixed. A terminator
packet data cannot imitate is its own task, and guessing one would be the
plausible-sounding alternative this project has been bitten by. A packets()
accessor was tried and removed for the same reason: with the extent short it
silently yields the wrong answer, and an API that is quietly wrong is worse
than no API.
Range measured: every tie LOD block on the disc -- 2,177 blocks, 13,042
consecutive record pairs.
A record's pointer is where its packet starts, so the distance between
consecutive pointers IS a packet's length, with no reference to any field.
Comparing that against the record's own fields:
packet_quadwords = vert_ofs + vert_size + scissor_size
12,927 of 12,927 record pairs whose strip_count is zero
0 of 115 record pairs whose it is not
The partition is exact: every record with a non-zero strip_count is an
exception and no record with a zero one is. So the fourth term is 0 quadwords
whenever strip_count == 0, and something else -- measured 0..19 quadwords, and
NOT equal to strip_count -- whenever it is not. R&C1's strip_count is a strip
count and strips are variable-length data, so a non-zero count most likely
adds per-strip bytes this cannot see; that is a hypothesis, labelled as one.
TieRecord gains packet_quadwords and packet_quadwords_are_known. The latter is
the honest gate, and the disc test asserts both sides of the partition rather
than the aggregate.
This also corrects this file's own retraction of the same formula, and the
retraction was the error rather than the claim. A formula of this shape was
derived at 61 of 61 pairs, then "retracted" as not reproducing (0 of 13,042).
The re-measurement had swapped strip_count for scissor_ofs in the fourth term
and put the blame on the block set. Re-measured with the term the first pass
had, it matches on 12,927 of 13,042 and the 115 exceptions are exactly the
non-zero strip_count records.
Worth recording because it is the same shape as two other entries in this
file: a retraction is not automatically more careful than the claim it
retracts, and the re-measurement that "disproved" this one changed the thing
being measured. The lesson on the other side is what saved it -- the first
measurement was over 61 pairs, not a population, and had it stayed there the
formula would have looked universal. A small sample and a bad re-measurement,
near-misses in opposite directions.
What this unblocks: the payload can be sliced per packet, which was the
prerequisite for decoding what it holds.
Range measured: all 27 levels' overlays.
The overlay carries the game's profiler menu definition -- a run of 8-byte
padded names (normal, camera, cam+chr, volume, bars, render, mobys, ties,
shrubs, cpu/gpu, occl, ...) followed by runs of absolute RAM addresses that
point back at those same names. On every level the four addresses after
normal/camera/cam+chr/volume, each minus its target name's overlay offset, give
the same number: 1,735,792.
That is field_00 - 16, where field_00 is the header word that read 1,735,808 on
all 27 levels with nothing attached to it. The overlay is loaded with its
16-byte header stripped, so overlay offset X sits at RAM X + 1,735,792.
OverlayHeader::ram_address carries it, and the disc test asserts the derivation
holds on all 27 levels.
Confirmed, not only derived: level 0's RAM 0x1acf80 holds
"normal\0camera\0cam+chr\0volume\0", byte-identical to overlay 0x5310, and
0x5310 + 1735792 == 0x1acf80. All four names land where the overlay's own table
says. `gc-trace verify-overlay-base` runs that check, savestate-gated the same
way as the tie and tfrag ones.
The overlay is relocated, not copied: only 37.67% of level 0's is
byte-identical at the base, and the 256-entry sine table -- at overlay 0xb650c
-- is not at 0xb650c + base. So the base locates the overlay's image, but
individual tables inside it move. That is the shape of a loader applying
relocations, and it means the base is necessary but not sufficient for reading
the game's tables.
The value of the base is that any overlay offset can now be carried to a RAM
address, which is the prerequisite for reading the game's live level state --
the thing the tie relocation and tfrag lighting checks had to work around.
Two things get deleted rather than finished. The collision u16 table was probed
four ways -- magnitude distribution, level-independence, run starts, run
lengths -- and every one came back level-dependent: the run-start pattern is
identical on 1 of 19 levels. So the first 1024 bytes of the collision payload
are data, not a fixed table, and nothing was decoded. Recorded as the negative
it is rather than left as four half-findings.
Range measured: every tie class on the disc, 1,864 classes over all 27 levels.
Method: read the word at class +0x0c as four bytes, and read the record counts
independently from the blocks themselves -- follow packets[lod] to the block,
walk to the 0xf6 terminator, and count the 16-byte records the block header's
data_offset names. Two paths to the same number, so the check is not circular.
Result: the byte at +0x0c is the LOD 0 record count, +0x0d is LOD 1's and +0x0e
is LOD 2's, on 1,864 of 1,864 classes. Classes with an absent LOD agree too:
a zero packets[lod] gives a zero count and the byte is zero.
This closes the file's oldest lie about the tie header. The comment in
TieClassHeader::parse read "0x0c: the 64 light-slot normals offset. Read past,
not decoded" -- inherited from R&C1's layout, where the word genuinely is an
offset. On R&C2 it is not an offset at all; it is three counts, and R&C1 carries
the same quadruple at +0x20. Skipping the field was the right action on the
wrong information: the field was unknown only because R&C1's map was applied to
R&C2's bytes.
TieClassHeader gains packet_counts, and
tie_class_header_counts_records_per_lod asserts the agreement.
The fourth byte, +0x0f, is left unnamed. It ranges 1..15, and the obvious
following of R&C1 -- the shader/ad-gif total -- disagrees on 1,827 of 1,864
classes, which is 2% agreement where a count would expect 100%. It is recorded
as unidentified rather than given a name that fits a fiftieth of the population.
data/gc-files.tsv pins the ISO files, and unpacks_and_matches_the_committed_table
checks that unpacking reproduces them. But the tie, collision, tfrag and chunk
tests do not unpack -- they read ~/Documents/gc-extracted, a directory on disk
that nothing in the repository checks.
A stale or hand-edited file there would have every measurement built on it still
pass, quietly. Three claims this session returned confident numbers while
describing the parser rather than the disc -- has_uniform_record_length, "1,509
chunks uniform", the moby class-ID scan -- and a test whose data source is
unverified is that same hazard one level down.
data/gc-extracted.tsv now records SHA-1 and size for all 5,237 files in that tree,
324 KiB, sizes and hashes only, no content. It is the output of `gc-extract
unpack`, whose ISO files are already pinned by the other table, so it is grounded
transitively rather than by assertion.
the_extracted_tree_matches_the_committed_table walks the table and checks every
row's size and hash against disk. It asserts all 5,237 are checked -- so a
truncated table fails rather than passing on a prefix -- and reports the first few
missing or mismatched paths when it does. Current result: 5,237 of 5,237 match.
This does not retroactively validate the three retracted claims. What it does is
remove the possibility that any of the surviving ones rests on a directory that
could have been anything.
246 tests, clippy clean.
The record-length retraction had been applied to the tests and to the section
that discovered the problem, but the same claim was still standing in four other
places -- including the rustdoc for has_uniform_record_length(), which is what a
caller would actually read:
crates/gc-formats/src/chunks.rs:206 "Measured: true for 1,509 of 1,509 chunks"
docs/roadmap.md:448 "uniform record length (1,509 of 1,509)"
docs/formats/core-data-regions.md a whole section titled
"Every chunk has a uniform record length"
All three now say 257 of 1,509, and the method's rustdoc explains WHY the old
number was an artefact -- that resolve sliced at the first offset delta, so every
record came out the same length by construction and the predicate was confirming
the parser back to itself.
That section also contained a withdrawal, and the withdrawal was wrong:
"This withdraws the previous commit's variable-length type. That came from
taking minlen/maxlen over all of a type's records at once, pooling across
chunks; rule 3a's pooled claim, third time this session."
The pooling rule fired correctly and was overruled by a measurement with no
independent content. The per-chunk check that should have caught the pooled
signal was itself reporting the parser's own slicing, so it agreed with the wrong
answer for the same reason the original claim did. Two errors stacked: a real
signal dismissed by a rule that had been right twice already, and a predicate
that could not have told the difference.
This is the third time this session that a test was asserting the parser's
behaviour rather than the disc's, and the third time it was the only thing
standing between a claim and a retraction. has_uniform_record_length is the
clearest instance: a predicate named after a property of the data, returning a
confident answer about the code.
245 tests, clippy clean.
Pinned down the one slot that still cleared 100x after the record-boundary fix,
and it is the artifact this section was written to guard against.
The discriminator the original version lacked: a slot that really carries moby
class IDs must SPAN the ID space, so some of its values land above a level's moby
count. The artifact does the opposite.
type slot enrichment distinct values ABOVE the moby count
0xd76cc453 6 305x 9 0
0xf2ef4275 34 64x 9 341
0xf2ef4275 7 46x 16 412
0xf2ef4275 14 34x 16 432
0xd76cc453 9 32x 28 702
The 305x slot's nine values are 1, 3, 4, 5, 10, 11, 12, 13, 15 -- all under a
level's ~215 moby classes, so every one scores as a hit and 65536/215 = 305x.
Nine small numbers is not a class-ID array. Every slot that actually spans the ID
space scores 10x to 64x, which is chance.
So the original claim's own numbers gave it away and it was not read that way:
"5 to 42 distinct values at 153x to 289x" is exactly the artifact's shape -- a
handful of small values, none above the count. The record-boundary fix was never
what made this claim wrong. It was wrong when written, and the retraction is
complete rather than the partial one I committed two turns ago.
The test now separates the two quantities: slots clearing 100x (currently 1,
expected to remain the artifact) and slots clearing 100x WHILE ALSO having values
above the moby count (currently 0, must stay empty). That second number is the
one that means something, and a slot appearing in it would be the first genuine
evidence of a class-ID array anywhere in the container.
Worth noting the per-SLOT bookkeeping: the count of values above the moby count
is keyed by (type, slot), not by type. A first attempt keyed it by type and the
artifact type passed the check on the strength of its ordinary slots -- 783 of its
own values sat above the count at other slots. The check has to be about the
individual slot, or one good slot launders a bad one.
245 tests, clippy clean.
Searched for the pattern that caused the over-read -- a record length derived from
the first difference between two offsets:
grep -rn "\[1\] - \[0\]" crates/gc-formats/src/*.rs
crates/gc-formats/src/chunks.rs:357 (offsets[1] - offsets[0])
One hit, in the one parser that walks an explicit offset table.
The others slice contiguous regions with chunks_exact -- shrub's 8-byte palette
records, tie's 16-byte packet headers, tfrag's 4-byte tables. There a length
cannot run past the next element, because chunks_exact derives each boundary from
the region's own length rather than from a sample of the spacing. Shrub's mesh
packets carry an explicit offset AND size, so they state their length instead of
inferring it.
The damage was bounded by a structural property worth stating: only a format with
an offset table AND a derived stride can over-read, and exactly one format had
both. That is a more useful fact than the retraction was on its own -- it says
where else to look if another one turns up.
The guard is positive rather than a comment.
chunk_records_neither_overlap_nor_leave_gaps asserts across all 27 levels that
every record but the last is exactly its own offset delta long, that there are
1,509 chunks, and that 1,252 have ragged deltas. A regression to a single sampled
stride fails immediately and names the chunk and the record.
245 tests, clippy clean.
The fix changed record lengths in 1,252 of 1,509 chunks, so every committed claim
computed over record bytes needed re-checking rather than assuming. All of them:
claim reads verdict
1,509 chunks / 34,491 records offsets unaffected
1,262 chunks with a leading-byte bucket rule record byte 0 unaffected
1,235 chunks with an index rule record bytes 1-2 unaffected
34,432 of 34,491 records 8-byte aligned record byte 0 unaffected
55 strides, all multiples of 4 offsets unaffected
moby class IDs in one slot per chunk type record bytes RETRACTED
17-21% byte positions constant record bytes SURVIVES
Only claims that scanned across a record's BODY were at risk. The rest read a
record's first three bytes, which an over-read never touched -- it appends to the
end. That asymmetry is why the damage was contained to one claim, and it is the
argument for preferring claims anchored at record starts.
The survivor was the obvious candidate for contamination, so it is worth being
precise about it. Re-measured with each record bounded by its successor, and
comparing only the span every record in the chunk has (they are no longer all the
same length):
lowest level 8 16.7%
highest 20.0%
Still inside the documented band. Previously this lived only in a doc comment on
the layout test; it is now asserted, in
chunk_records_share_a_layout_measured_by_constant_byte_positions, over all 27
levels with a 10-30% band so a real change reports itself rather than sliding.
The module documentation is corrected to the measured range and now says outright
that records in one chunk are not all the same length, since that is the fact a
reader would otherwise have to rediscover.
244 tests, clippy clean.
I reverted the record-boundary fix last commit on the grounds that "fixed-length
records plus inter-record padding" and "variable-length records" produce the same
offset table. That reasoning was wrong, and cheaply so.
stride is the FIRST offset delta. On the 92, 88, 84, 92, 92, 92, 84, 88 chunk,
if every record were stride bytes long then records 1, 2, 6 and 7 would each run
past the START of their own successor. Records are addressed by an explicit
offset table; they cannot overlap. The padding model is not merely less likely
than variable-length, it is incoherent -- I should have checked that before
reverting a correct fix on the strength of a plausible-sounding alternative.
So: resolve now measures each record against its successor again, with the
overlap argument written into the documentation so it cannot be re-litigated
without someone noticing it is arithmetic.
THE RETRACTION, which is the substantive part:
Three tests broke. Two were straightforward, but the third was not.
* constant_positions indexed records as if they were all the same length. Now
that they are not, it compared past the short ones. It takes the minimum
length instead.
* "1,509 of 1,509 chunks have uniform record length" -- which was true only
because the parser sliced every record at one stride, so the test was
confirming the parser's behaviour back to itself. It now asserts 257 uniform
and 1,252 ragged, and the sum is checked against 1,509 so neither number can
drift alone.
* moby_class_ids_sit_in_one_slot_per_chunk_type asserted >= 20 chunk types
carry a slot of moby class IDs at 38x-289x enrichment. It now finds 2, of
which 1 clears 100x. RETRACTED. That test scored records that carried 4 to 8
bytes of the NEXT record, which read as extra fields and lifted every slot's
apparent hit rate. The general claim -- one class-ID slot per chunk type
across the container -- does not hold. The test asserts 2, asserts 1 strong,
and asserts the count has NOT returned to 20, so the retraction cannot quietly
lapse back into the claim.
That is a committed finding withdrawn on the strength of a parser bug. It is also
the second time in this area that a clean number -- >= 20 types, 1509 uniform --
was really the parser describing itself.
243 tests, clippy clean.
The previous commit fixed on one signature. Measuring all of them shows the
pattern is the norm. Taking delta = offset[i+1] - offset[i] against the chunk's
own first delta, levels 0-2, 2,953 deltas:
0 1,723 -4 549 +4 426
-8 146 +8 32
-12 71 +12 21
-16 34 +16 4
-20 27 +20 1
-24 13
Every deviation is a multiple of 4 and most are exactly +/-4; worst observed is 24
bytes in a 7-record chunk. 1,252 of 1,509 chunks have at least one, across 66 of
74 signatures.
WHY THIS DOES NOT BECOME A FIX
Two models produce this offset table and they are observationally identical:
1. records are genuinely variable-length, padded to 4 bytes, stride = largest
2. records are fixed at stride and the offset table carries inter-record
padding, so the gap to the next record is record + padding
Nothing in the offsets distinguishes them. Both say where records start. Both
produce 92, 88, 84, 92, 92, 92, 84, 88 for the case already discussed.
I wrote the change that slices each record against its successor -- model 1 -- and
then reverted it. Two things came out of writing it:
First, it broke three tests, which is the informative part. Those tests assert
"1,509 of 1,509 chunks have uniform record length" and they passed BECAUSE the
parser forced uniformity by slicing every record at a single stride. They were
asserting the parser's behaviour back at itself, and would have kept passing
forever regardless of what the disc contains.
Second, shipping it would have replaced one unverified model with another and
silently changed the record boundaries of 1,252 chunks on the strength of a
guess. The repo's own rule applies: a heuristic count is not an identification,
and here it is not even a count -- it is two equally good explanations of one
table.
WHAT WOULD SETTLE IT: a field inside the record whose end is independently
observable -- a terminator, a count implying a length, or a known value pinning
its position. Under model 2 the disputed bytes are padding and will be uniform or
zero; under model 1 they are data and will not be. For [75 42 ef] those 8 bytes
read a0 00 27 20 at both the tail of one slice and the head of the next, which is
arithmetic rather than evidence, since it is the same eight bytes seen twice.
The uniform model stays, the ambiguity is recorded in resolve()'s documentation
and here, and the 1,509-of-1,509 assertion stands as a statement about what the
parser does rather than about what the disc contains. 243 tests, clippy clean.
[75 42 ef f2 71 b5] -- the signature that is both level-specific and
variable-length -- makes three chunks per level, 81 total:
stride 76 deltas [76, 76, 76, 76, 76, 76, 76, 76] uniform
stride 92 deltas [92, 88, 84, 92, 92, 92, 84, 88] NOT uniform
stride 104 deltas [104, 104, 104, ...] uniform
Identical on all 27 levels: 27 chunks of each shape, three shapes, no others.
So "this signature has strides {76, 92, 104}" is a misdescription. It has three
chunk shapes, one of which is not fixed-stride at all. The parser derives one
stride per chunk and resolve slices every record at that length.
The cost, precisely: records are read at the offset table's positions, so every
record STARTS correctly -- no misalignment. But each is sliced at 92 bytes, while
two of the eight are 88 and two are 84, so four records over-read by 4 and 8
bytes and carry the head of the next record inside them. Bounds checking holds;
this is a data-correctness bug, not a safety one. It is precisely the kind that
produces plausible-looking fields offset by four bytes, which is how it would
have gone unnoticed had the stride table not been collected per signature rather
than inferred per region.
It also explains why this signature was previously flagged as breaking the 8-byte
alignment rule: its records genuinely are not a uniform multiple.
The right fix is for resolve to use offset deltas for a chunk whose deltas are
not uniform. Not taken here -- the delta sequences are asserted first, so if the
data ever becomes uniform the test fails and asks for the model to be revisited,
rather than the model quietly absorbing the exception.
Asserted in chunk_type_75_ef_has_one_variable_length_chunk_per_level, with the
exact sequences [76x8], [92,88,84,92,92,92,84,88], [104x8] and 27 of each.
A near-miss worth recording, because it would have made a tidy false claim: the
leading u16 of these records appeared to step by exactly 0x50 (80) across all nine
of the 76-byte chunk. Reading at the real offsets gives b1a5, a529, 047e, a529,
04f0, a529, 0562, a529, 05d4 -- no progression at all. The apparent 80-byte step
came from comparing the wrong bytes, and "records are indexed by 80" is exactly
the kind of finding that gets reported on sight.
243 tests, clippy clean.
Comparing every chunk record against the same record index in the other 26
levels, across all 74 signatures:
65 of 74 every record byte-identical in all 27 levels
9 of 74 level-specific
0 ragged no signature appeared with different record counts per level
The nine that vary:
[5d 44 13] [75 42 ef] [7f 44 ee] [82 c4 2f] [96 c3 37]
[a2 44 a5] [c6 c3 ef] [df c3 ea] [f3 c4 b9]
This reframes the container. 65 of 74 chunk types are static tables the game ships
unchanged in every level -- constants and lookups, not level content. The largest
by record count are all in that group: [38 c4 7e] stride 328 with 90 records per
level, [a9 c4 d8] stride 308 with 90, [3d c4 9e] stride 368 with 77.
The consequence is about method, and it is worth being blunt about. For a
level-specific type, comparing levels is the strongest lever available: a field
that moves with the level is identifiable by correlating it with something known.
For the other 65 there is nothing to compare -- one copy of the data, repeated 27
times -- so they must be decoded structurally, from what the bytes are. A future
decoder that reaches for per-level comparison as its first tool will find it
useless on the majority of the container.
[75 42 ef] is both level-specific AND the one signature with three strides. A
type that is both variable-length and level-varying is the most informative one
here, and is where per-type field decoding should start.
How this was found, including the wrong turn:
The first look at [53 c4 6c d7 9a 43] (stride 108) showed 45 of 108 byte positions
varying WITHIN a chunk and 810 records all distinct -- which reads like per-record
data. Counting distinct values per u32 slot across all 810 records contradicted
that: 29 to 30 distinct values across 27 levels, i.e. about one per level.
Both readings were wrong because they measure different things. The 45 positions
vary between records within a level; the u32 counts were low because the same 30
values recur in every level. Comparing record i across levels settled it: all 30
records are byte-identical in all 27 levels, and zero byte positions vary across
levels.
Had the u32 distinct counts been the only measurement, the conclusion would have
been "one value per level" -- the opposite of the truth, describing a per-level
parameter table that does not exist. The within-chunk and across-level views had
to be taken together to get a true one.
Asserted in most_chunk_signatures_are_level_independent. 242 tests, clippy clean.
Walked every chunk on all 27 levels, grouped by six-byte header signature:
1,506 chunks, 34,415 records, 74 distinct signatures
55 distinct strides, 52..640, every one a multiple of 4
73 of 74 signatures use exactly one stride
The exception is asserted rather than skipped, because "every type is uniform"
is precisely the claim that quietly stops being true:
[75 42 ef f2 71 b5] strides {76, 92, 104}
That is the variable-length type the module docs had referred to indirectly. It
is now pinned by signature and to those three strides exactly.
The 4-byte alignment is a result, not an artefact. A stride recovered from one
chunk's first-to-second record delta can inherit alignment from where that chunk
happened to sit; collecting all 55 strides across every type and requiring all
of them to be divisible by 4 rules that out, because a single off-by-one in one
signature would show. Asserted in
chunk_record_stride_is_four_byte_aligned_and_uniform_per_signature.
What this settles is the shape the decoder must handle: uniform stride per type,
4-byte aligned, one named exception. Field offsets within a record are now the
same for every record of a type, which is what makes per-type decoding
tractable. The fields themselves are still unnamed.
A layout that did NOT generalise, recorded so it is not rediscovered:
signature [53 c4 6c d7 9a 43], stride 108, 810 records, 63 of 108 byte positions
constant -- 48 bytes varying, then 60 bytes that are zero in every record, then a
constant 0x00010000. That reads like an inline buffer this type never uses, and
would have made a tidy field story. Requiring `stride == live span + zero run`
across every type with 50+ records per chunk gives 0 of 9. The zero tail belongs
to that one type and means nothing general.
Third occurrence of one editing mistake, and it is worth naming as a habit
rather than an accident: inserting a test before an existing one strands that
test's #[test] attribute on the new doc comment and silently unregisters it.
It has now happened three times, each time surfacing only as clippy's "never
used" and never as a missing test. Fixed by appending new tests at the end of
the file instead of anchoring them to an existing function, and verified by
listing registered tests rather than trusting a count.
241 tests, clippy clean, all five recent tests confirmed registered.
Last commit said what would settle the volume candidate and implied one of those
was available. They are not. Recording the failure rather than moving on quietly.
Level 0's data.bin -- the level's file descriptor -- contains the value 799204
exactly once, at 0xab6360, and 799204 is byte-for-byte the offset of the 20-record
144-byte candidate table. A single 4-byte match in 13.3 MB has a chance
expectation of ~0.003, so it was worth checking rather than assuming.
It does not hold up. Each level's own best 144-byte table, searched in that same
level's own data.bin:
level candidate records refs (chance ~0.0035)
0 1,810,724 32 0
1 1,884,884 386 0
2 2,066,804 224 0
3 2,438,228 365 0
4 2,677,780 114 0
5 1,255,780 14 0
6 2,496,020 115 0
7 2,062,072 311 0
None. The level 0 hit at 0xab6360 also sits among high-entropy neighbours, and
level 0's own BEST table -- a different offset, 32 records, not the 20-record one
-- is not referenced either. So the hit does not generalise and is most likely
coincidence.
This retracts what the previous commit implied. Nothing links a 144-byte float
table to volumes: the table is reached by walking structure rather than by an
explicit pointer, so the discriminator I named does not exist, and the candidate
stands exactly where it did.
Recording the three routes that would actually settle it, all of them
unavailable: the game's own volume code (same dead end as the collision anchor
-- a GP-relative string search returns nothing); ground truth, such as a count of
level 0's trigger volumes from a map; or a correlation with a named class, since
gameplay.bin resolves ids through its dispatch records -- but no class name
containing "volume" exists on the disc, which is exactly why this is unavailable.
No test added, because there is no solid claim here to assert. 240 tests,
unchanged.
Volumes were the only type never located, so this is a search report rather than
a result, and it is recorded as a candidate on purpose.
Ruled out first: no engine-side name exists. No level file contains "Volume"
outside UI text ("Music volume", "Volume de la musique") and Italian item
descriptions. The format cannot be found by label and has to be structural.
gameplay.bin is already-decompressed level data. Its head is a count (336 on
level 0) then 16-byte records from byte 416, carrying (type, pad, offset, id).
The id groups them -- each id has exactly two records pointing into two parallel
offset arrays -- and those offsets resolve into a text pool, the first ones
landing mid-string in dialogue ("it Megacorp Lab"). So that part of the file is
dialogue and class dispatch, not geometry.
The structural hit: a table of 144-byte float records. Level 0 has one at
799204, 20 records of which 19 are populated and one is an all-zero terminator,
with 34-35 of each record's 36 float slots world-plausible and +0x0c holding a
varying world position:
(206.5, 284.1, 116.0) (265.0, 445.3, 131.7) (244.0, 407.2, 128.0)
(219.3, 247.0, 107.5) (185.1, 282.6, 104.9) (260.8, 423.6, 130.9)
(248.2, 198.4, 51.8) (185.8, 282.7, 102.6) (247.0, 211.5, 100.6)
A 144-byte record with world positions and float radii is the shape a volume
trigger region should have, and a matching table is present in every level
examined (0-9), each preceded by an all-zero record, counts 23 to 612.
It is NOT called volumes. Nothing ties this table to volumes. Camera bounds,
spawn points and path nodes would present exactly the same thing: float records,
world coordinates, no label. The discriminating evidence has not been gathered,
and writing "volumes" in the roadmap because the shape fits would be the exact
error this log keeps catching.
Two detector bugs, both mine, both confident before they were caught:
1. The validity test accepted zero as a world-plausible float, so every
zero-filled region scored perfectly and the cross-level check returned runs of
225 to 1,514 records that were obviously padding. Requiring non-zero floats
(>=20 of 36) separated real tables from memory immediately.
2. The 144-byte stride came from four offsets I eyeballed as being 144 apart.
Gap 144 was not in the top eight gaps of the detector that found them -- it
survived because three consecutive observations agreed, not because the
measurement supported it. The corrected detector then found much longer runs,
which is the opposite of what the eyeball version implied.
Same shape as every other correction here: a clean small plausible number read as
conclusive because nothing was checked against a control that could disagree.
To actually settle it, the table needs to be correlated with something known --
if a count matches a level's visible trigger regions, or if an id in the record
resolves through the class table to a named object, that is the discriminator.
Until then volumes remain open and the roadmap says so.
sound_bank's head was parsed but never characterised. Across all 27 levels the
SBlk region opens identically for its first twelve bytes:
+0x00 "SBlk" +0x04 1 +0x08 4 +0x0c "DAW\0" +0x10 0
and +0x14 is the only word that varies. It varies with its low sixteen bits zero
on all 27, leaving 0x0092..0x0103 -- 146 to 259.
That is the shape of a record count, and it tracks the body length at close to
100 bytes per unit. It is not one:
body / field_14 91.2 .. 110.8 across 27 levels
any stride 0..80 with body = stride*word + constant exactly: no
So the test asserts the negative too. If a stride ever makes the relation exact
on all 27, sound_bank_sblk_head_is_constant_apart_from_one_word fails and asks
for the record size to be revisited. A near-miss at 100 bytes/record is exactly
what hardens into a name the next time somebody glances at it; the constants are
also checked individually so a real deviation names which one moved.
The DAW region begins with sixteen zero bytes then block-structured data with
16-byte periodicity and slowly varying nibbles -- consistent with block
compressed audio, not decoded. Recorded as such.
Two mistakes, both mine, both the same shape as earlier ones:
1. The constructor bound `format_tag` with a `let` BEFORE the two words that
precede it in the file, consuming bytes 4..8 and shifting every field after
it. The struct literal listed the names in the right order; the statements did
not. A scratch script reading the same bytes gave the opposite answer, which
is what exposed it. Fixed, with a comment, because "read the lets in file
order" gets undone otherwise.
2. Inserting the new test ahead of an existing one stranded that test's #[test]
on my doc comment -- silently unregistering
tie_vertex_element_is_16_bytes_with_fixed_texcoords. This is the SAME mistake
as two commits ago and I made it again. It again surfaced only as clippy's
"never used" rather than as a missing test. Restored and verified by listing
the registered tests rather than trusting the count.
240 tests, clippy clean, both tie vertex tests confirmed present.
Having established that i16[0..2] are positions, ran the same LOD-overlap method
across all eight slots to see whether it discriminates elsewhere. It does not,
and that is worth recording before someone reads the passing numbers as nine
confirmations:
slot same class control ratio
i16[0] 97.0% 3.0% 32x
i16[1] 96.7% 3.2% 30x
i16[2] 96.4% 6.0% 16x
i16[3] 75.3% 78.7% 1x
i16[4..7] 68-91% 52-79% 1x
Only i16[0..2] carry signal, because they are the only slots with tens of
thousands of distinct values. The rest draw from sets of 226 to 1,759 values, so
any two sets of that size overlap heavily regardless of what the field means. A
passing result from a method that has run out of discriminative power is not a
passing result.
Slot 3 excluded as a skin/joint index. It holds 201 values in 0..=254 and 50
distinct values within a single draw packet, so it is per-vertex, not per-material
-- which invites the obvious reading, since tie objects are skinned. Wrong:
mean pairwise distance, same slot-3 value 14,481
mean pairwise distance, different slot-3 value 13,795
ratio 0.95x
control: same position-x bucket 8,105
Vertices sharing a slot-3 value are marginally FARTHER apart than vertices
differing in it. The control halves the distance, so the measurement works; the
0.95x is on the wrong side of neutral, though by too little to claim much on its
own. Combined with no LOD sharing (75.3% against a 78.7% control), slot 3 is
per-vertex, uncorrelated with the geometry it sits in and with the detail level.
Recorded as not understood. A per-vertex animation phase fits the arithmetic and
is labelled a guess rather than offered as a reading.
Slots 6 and 7 are mostly not per-vertex. Distinct values per draw packet, median
over 1,804 records:
i16[0..2] 13-17 constant for the whole packet 0%
i16[4,5] 6, 4 0%
i16[6] 2 43%
i16[7] 4 34%
They take a single value across an entire packet in well over a third of records,
which matches the 91.8%/88.9% dominance of 4096 and 0. Unnamed.
So the tie element is: position, byte-sized index, texcoords, layout tag, and two
slots characterised but not named. Positions, texcoord placement, element size and
the layout selector are all asserted; the remaining three are not, and the reasons
they are not are now written down rather than left as silence.
On "is Phase 2 nearly done": 40 items closed, 9 open. The nine are the hard
ones -- collision cells blocked on the game's own code, volumes not located at
all, moby_anim ruled out but unexplained, chunk record fields unnamed. Progress
on the checklist is real but the remainder is not the easy tail.
The three slots ahead of the index were "unproven position" for want of a test
that could fail. One is available on the disc alone, with the control built in.
LOD 1 and LOD 2 of a tie class are the same object at two levels of detail, so
anything that locates a vertex in that object's own geometry is shared between
the two blocks; a different class' LOD is an unrelated object. Intersecting the
sets of i16[0..2] triples both ways turns "is this a position" into a
measurement. Levels 0-5, ordinary-layout elements only:
same class, LOD1 vs LOD2 96.1% 107 classes, 107 nonzero
different class, control 0.4% 2,144 pairs
A factor of 240. Per level the same-class figure runs 94.4 / 97.9 / 98.1 / 96.7 /
96.8 for levels 0-4. The control stays under 1.0% except for one level-1 pair at
45.9% -- two classes sharing most of their geometry, which is what duplicate or
variant models should look like. Reported, not excluded, and the assertion is on
the mean rather than the max so it does not have to be.
Asserted in tie_vertex_slots_0_to_2_are_shared_positions, including the ratio
(same > 20 * diff) so a change that collapsed both numbers fails rather than
passing twice, and a nonzero-overlap check so no class can silently stop
agreeing with itself.
What this rules out is the point. An index, a handle or a per-vertex counter
would not be shared at 96% between two independently built LOD meshes, nor sit at
0.4% between two objects. With 10.7k-13.2k distinct values per slot over the
full i16 range, these are positions in the object's own frame. Scale and axis
order stay unnamed -- nothing here says centimetres, and axis order is
unconstrained because nothing here assumes one.
Two things worth recording about the process:
1. The diagnostic line printed "1.0%" for 99.4% because the format was
`{same:.1}%` with no `* 100`. The assertions compared fractions and were
always correct; only the readout was wrong. Same family as everything else in
this log -- the readout disagreeing with the claim.
2. Inserting the new test in front of an existing one stranded that test's
#[test] on my doc comment, which silently unregistered
tie_vertex_element_is_16_bytes_with_fixed_texcoords. It showed up as
"never used" in clippy, not as a missing test. Restored; both tie vertex
tests are registered and 239 tests run.
Element layout now reads:
+0x00 i16[0] position x +0x06 u8 index 0..=254
+0x02 i16[1] position y +0x07 u8 layout tag, 0 in 93.4%
+0x04 i16[2] position z +0x08 i16[4] texcoord u
+0x0a i16[5] texcoord v +0x0c/+0x0e i16[6], i16[7] unnamed
Splitting the 95,612 elements of levels 0-2 by i16[3]'s HIGH byte, which is the
layout selector:
0x00 89,341 93.4%
0x03 5,344 5.6%
144 other selector values 927 1.0%
146 selector values in all. Not a two-way switch, and the second case is not 0x01.
Inside the ordinary layout i16[3] is not a coordinate: 201 distinct values, all
within 0..=254, with 0 most common. That is an index. So the selector and the
index are not two independent halves sharing a slot -- it is one byte. The low
byte is the index, the high byte is the layout tag, and the ordinary layout never
sets the high byte. Asserted.
Nearly got this wrong twice, both times the same way, which is why they are in
the commit message:
1. The first pass gated the range on `w >= 0` instead of on `(w >> 8) == 0`,
letting in the other layouts' large positive values. That reported "564
distinct values in 0..=32664" -- a number that looks exactly like a
coordinate and is really two populations unioned. The selector byte is the
filter. The filter is part of the claim.
2. Two constants were guessed rather than measured (selector cardinality 159,
and an upper bound that ignored non-ordinary layouts) and both were wrong.
The cardinality is 146. Guessing a constant and then asserting it is how a
measurement becomes a tautology.
Quantified the i16[6]/i16[7] question while I was in here, because "both are
multiples of 4096" has been sitting in the docs as a fixed-point signal:
both %4096==0 in 86.5% of elements
expected if the two residues were independent: 84.3%
enrichment 1.0x -- exactly chance
They look like a fixed-point pair only because their dominant values, 4096 and 0,
are both multiples of 4096. Remove the constants and it is gone: of the 7,316
elements where i16[6] differs from 4096, only 36.7% are still multiples, and of
the 9,905 where i16[7] differs from 0, ONE is. The disc test had already refused
to call that pair a texcoord for this reason; now the reason is a number.
Remaining slots in the ordinary layout, recorded: i16[0..2] read as position but
are unproven (10.7k-13.2k distinct each, full i16 range), i16[4]/i16[5] are the
texcoords, i16[6]/i16[7] unnamed.
Also fixes two clippy map_or lints and an import that a stray --fix had broken.
238 tests, clippy clean.
The last three commits changed things the documentation still asserted in the
old direction. Fixed:
roadmap
* 227 tests -> 238.
* The level_overlay bullet contradicted itself: it said "the header words are
not named" in the same breath as "+0x00 load base is 0x1A7C80". Now says
+0x00 IS the load base and is named, +0x0c is excluded as the .lit size by
sign (below file length on 16/27, by up to 8.9%) and remains unnamed, and the
unnamed remainder is stated as such.
* Section names aligned with the Phase 3 list (lvl.vtbl, lvl.camvtbl,
lvl.sndvtbl).
core-data-regions.md
* The claim-assertion audit still said the two PCSX2 claims were "NOT
assertable here -- needs a savestate". They are asserted now, gated on
GC_SSTATE, in the same file. Updated, with the two corrections that fell out
of writing them recorded right there.
* "that string is inside the collision routine" was never established. It is
in the string pool, yes; the reference to it is not found. Reworded to say
so, including the zero-hit result.
research log
* field_00 now carries its name and the .lit/$gp/program context.
* The field_0c falsification and the byte-order retraction are both recorded
here as well, so the log and the format file agree.
Nothing left asserting the withdrawn anchor, and no stale test counts.
Last commit claimed exactly one GP-relative occurrence of the collision assert
string's $lo half, at 0x354cc8, and called it an unambiguous anchor. That was
wrong, and building the disassembler is what proved it.
The search compared word bytes 1..3 against the 16-bit immediate. The PS2's MIPS
is little-endian, so an I-type immediate sits in bits 15..0 -- word bytes 0 and
1. Bytes 1..3 are the opcode, $rs and $rt fields. The match was on those.
The instruction at the phantom hit is 0x24a89948:
addiu $t0, $a1, -0x66b8
whose real immediate is 0x9948 -- ten bytes from the 0x99a8 being searched for.
-0x66b8 merely looks like a plausible $gp offset, which is what made it
convincing. With the byte order corrected the count is ZERO, in 2,751,268 bytes.
So the anchor is not established and the assertion is withdrawn. What survives is
still worth having: the functions are named (Camera_CollPrimTest takes (prim,
grid)), $gp = 0x1AEFF0, .lit loads at 0x1A7C80, and the assert string is a unique
copy at overlay+0xd18.
Probably the code reaching that string does not address it through $gp, or
reaches a copy in another section. The decomp's assert-message-names.tsv gives
0x002db350 for level 0 -- inside .lit, but it disassembles to a function epilogue
(jr $ra / addiu $sp, $sp, 0x70), so those are code locations, not string
addresses, and they are not traced to the assert either.
tools/overlay-anchors now does the search with the correct byte order and
carries a comment on exactly the part that was wrong, because that is the part
that will be re-derived otherwise. It also disassembles around an address:
cargo run -q -p gc-overlay-anchors -- <overlay.bin> --disasm 0x2db350 0 64
The useful general lesson, which the roadmap already says and this is a second
demonstration of: a probe that returns a small, clean, narratively satisfying
number is the most dangerous kind of wrong result. "Exactly one hit in 2.75 MB"
read as conclusive. It was an artifact of comparing the wrong two bytes, and it
survived a commit because it fit the story.
OpenRAC/rac2-gc-decomp is the same disc as ours, verified rather than assumed:
ours 3,828,350,976 9db2e33e...b1ce5
theirs 3,828,350,976 9db2e33e...b1ce5
Vendored read-only under vendor/ (already gitignored) and added to PROVENANCE.
It names two constants this parser had measured but never explained:
* overlay.bin +0x00 = 1,735,808 = 0x1A7C80, the .lit load base. It is constant
across all 27 levels because every level's literal section loads at the same
address. So the first header word is the load base of the section the file IS,
which also establishes that overlay.bin is that section -- it holds both code
and the debug string pool. $gp = 0x1AEFF0 comes with it, and that closes the
roadmap's open "$gp for this game" item.
The program around it is seven sections -- .lit .bss .data vtbl camvtbl sndvtbl
.text -- which is precisely the "seven-section parser and relocation" Phase 3
item for this format. Known target now rather than a guess.
I then tested the obvious follow-up and it is wrong: if +0x00 is the .lit load
base, +0x0c ought to be the .lit size. It is SMALLER than the file on 16 of 27
levels, by up to 8.9% (level11: file 3,151,128 vs 2,870,792). Excluded by sign,
not by a near miss. Still unnamed.
Collision: still blocked, but no longer blocked on tooling. The decomp names
Camera_CollPrimTest(void *prim, void *grid) without decompiling it, so the cell
layout is not handed over. What it does supply is $gp, and that is enough to
start. "Camera_CollPrimTest WARNING! - grid out of bounds! (RAR)" is at
overlay+0xd18, the only copy in the file, so:
string overlay+0xd18 -> RAM 0x1a8998; delta to $gp -0x6658; $lo 0x99a8
MIPS puts the $lo half in bytes 1..3 of a 4-byte instruction. Searching the whole
overlay instruction-aligned gives exactly ONE hit:
0x354cc8 (overlay+0x1ad048)
One occurrence in 2.75 MB of code is unambiguous: that is the instruction
materialising the assert string, and its enclosing function is
Camera_CollPrimTest. That is the route in, and it needs no symbols.
tools/overlay-anchors does this step, dependency-free. I started this with
capstone for the disassembly proper and dropped it: capstone 0.14 does not
re-export Mode/Endian from a public path, and an excluded-from-the-workspace
native dependency is a poor trade for a step that does not need one yet. The
anchor search is pure byte work, so it is pure Rust. No native dependency is
added anywhere, and the tool is an ordinary workspace member.
Surveyed the rest of that project against our open items: no volume structures,
no tie vertex layouts, no SBlk/DAW fields, no wavelet tables. It is a guide to
the code, not a replacement for reading it.
The audit two commits ago listed three claims as "not assertable here -- needs a
savestate". With PCSX2 running they are assertable, so they now are. gc-trace
already exposed gc_trace::Savestate::ee_ram_owned(); gc-extract just needed it as
a dev-dependency. Both tests are gated on GC_SSTATE and skip cleanly without it,
so a normal checkout pays nothing.
Writing them corrected two claims, which is the main reason they were worth
writing as tests rather than staying prose.
1. The collision payload is NOT "byte-for-byte identical". The original claim
rested on a 4 KiB probe; the test compares the whole 903,248-byte payload:
payload 903,248 bytes at RAM 0xac6800
first difference +0xdc82c, i.e. 903,212 bytes in
bytes that differ 6 (0.00%)
903,212 of 903,248 bytes are verbatim and 6 are not, all in the final 36.
"Verbatim" was an overclaim on a probe that never looked at the tail. The test
now measures the whole payload and reports the extent, asserting only that the
divergence is beyond the head.
2. The tfrag relocation base is the TABLE address, not the block base. The delta is
8,683,584 = 0x848040 = block base 0x848000 plus table_offset 0x40 -- which is
exactly what TfragHeader::data's own documentation says, "relative to the
table". The earlier claim said "the block's RAM base", off by exactly the table
offset.
Also: +0x10 and +0x38 do NOT share one constant. The aggregate "1,665 fields
relocate by one constant" was true only because +0x10 carried it; lumping them
hid that. The test asserts +0x10 and REPORTS +0x38's delta rather than assuming.
The patch-rewrite figures also restated on aggregate rather than median, which is
why they read lower than the 7.80%/8.12% medians recorded earlier:
patch bytes rewritten -- cannot morph 1.9% of 334, can morph 2.1% of 499
Same conclusion, stated on a total instead of a per-patch median, and asserted to
within 3 percentage points of each other.
Net: the audit table in docs/formats/core-data-regions.md now has no "prose only"
rows for disc-or-savestate-readable claims.
The i16[3] high-byte finding -- the most recent and most subtle result of this
stretch, and the one I corrected twice -- lived only in prose. Two commits is
exactly how long a claim can sit unasserted before it drifts. It now has a test:
tie_vertex_layout_is_selected_by_i16_3_high_byte
which reproduces the figures exactly and, importantly, asserts both directions:
(4,5) and (6,7) are above 0.85 and 0.80 in the ordinary group, (6,7) is BELOW 0.99
so it cannot silently become the constant I once mistook it for, (0,1) and (2,3)
are under 0.20, and every pair in the non-zero group is under 0.15. A future
change in either layout now fails loudly.
Also audited the rest of this session's claims against the test suite and recorded
the result in the format doc:
tie vertex element is 16 bytes asserted
layout selected by i16[3] HIGH byte asserted (this commit)
blocks self-delimiting on 0xf6 asserted per-block
packets == 0 means no block at that LOD asserted
packets[0] == 256 on 1,864 of 1,864 asserted
chunk record length uniform per chunk asserted
moby class IDs in one body slot per type asserted
collision payload verbatim in RAM NOT assertable here -- needs a savestate
tfrag patch bytes all rewritten by LOD same
morphing is unobservable in that image same
The split is deliberate: disc-readable claims are asserted against the disc,
savestate-dependent ones live in plan/pcsx2-verification.md with their method.
What is not acceptable is a disc-readable claim living only in prose -- which is
how the tie i16[3] finding spent two commits before it had a test, and how several
older claims drifted before the roadmap reconciliation caught them.
Two results, one of them a correction.
1. "(6,7) is just the constant 4096/0 pair, not a texcoord" -- wrong. It looked
like an artifact because a constant is trivially a pair of multiples of 4096.
But partitioned by i16[3]'s high byte it is a genuine fixed-point pair in its
own right:
i16[3] high byte ZERO (n = 89,341) divisor rate by slot pair
pair /1 /4 /16 /256
( 0, 1) 100.0% 11.5% 4.6% 3.3% baseline
( 2, 3) 100.0% 11.0% 4.5% 3.3% baseline
( 4, 5) 100.0% 92.9% 92.0% 91.8% the texcoords
( 6, 7) 100.0% 89.5% 87.3% 86.5% a SECOND fixed-point pair
If (6,7) were a constant it would score 100%, not 89.5%. The constant is the
common case WITHIN it, not the whole of it -- which is exactly the distinction
the constant-artifact lesson is about, and I applied it backwards last time.
So the ordinary element carries TWO fixed-point pairs, (4,5) and (6,7), with
(0,1) and (2,3) at baseline. What (6,7)'s role is remains open -- second UV
set, normal, or a zero-heavy fixed-point pair all fit.
2. The element LAYOUT differs between the three cases. Scoring all four slot
pairs within the non-zero high-byte group, no pair stands out:
i16[3] high byte NON-ZERO (n = 6,271)
pair /1 /4 /16 /256
( 0, 1) 100.0% 4.6% 3.5% 0.7%
( 2, 3) 100.0% 2.0% 0.6% 0.0%
( 4, 5) 100.0% 1.9% 0.3% 0.1%
( 6, 7) 100.0% 7.5% 5.7% 4.5%
Best is 7.5% against a 2-5% baseline. So for those 6,271 vertices the 16 bytes
are not laid out as position/texcoord/normal at all -- a different element
shape, selected by the high byte.
This also explains why the global slot-pair score looked ambiguous to begin with:
pooling the two layouts averages a 92% pair and a 7% pair into one number that
describes neither. Fourth time this session that partitioning produced the answer
rather than merely bounding it.
The previous commit said i16[3] selects the texcoord encoding. That was right but
loose -- it is specifically the HIGH BYTE, and the low byte is a separate thing.
Grouped by the high byte across the same 95,612 vertices on levels 0-2:
i16[3] high byte n % / 16 % / 256
0x00 89,341 92.0% 91.8%
0x03 5,344 0.0% 0.0%
0x04 71 0.0% 0.0%
0x40 58 5.2% 0.0%
0x02 47 2.1% 2.1%
0x19, 0x05, 0x20, ... the remaining 73
A high byte of zero is the ordinary case -- 89,341 of 95,612, 93.4% -- at 92%.
EVERY other high byte scores at or near zero. The split is on the byte, not on the
value: i16[3] of 1 and of 771 differ only in the high byte and behave completely
differently.
Within the all-zero case there is a third distribution: i16[3] == 0 exactly (5,851
vertices) scores 62% at 16 and 61% at 256, against 92% for 1..255.
So three cases, selected by the high byte and then the low:
high 0x00, low 0x00 5,851 a third distribution, 62% at 1/16
high 0x00, low 1..255 83,490 the ordinary case, 92% at 1/4096
high non-zero 6,271 0%, an encoding this reading does not see
The 6,271 matches the figure quoted last commit for "vertices using an encoding
this reading does not see", and the two group counts sum consistently with the
committed test's 95,612 -- so this refines the finding rather than changing it.
A measurement trap worth recording: the all-zero case is a SUBSET of high byte
0x00, so summing the per-high-byte groups double-counts it. An intermediate total
came out at 100,792 against the true 94,941 for those groups; the committed test's
figure is the one to trust.
The texcoords are 12-bit fixed on 82.5% of vertices, not 100%, and the residual
had no explanation. Grouped by the vertex's i16[3] slot across 95,612 vertices on
levels 0-2, the buckets separate completely:
i16[3] n % / 4096 % / 256
negative 333 1.2% 1.2%
== 0 5,851 29.2% 60.7%
1..63 28,884 92.7% 94.3%
64..255 54,606 92.3% 93.8%
256..511 8 0.0% 0.0%
512..1023 5,391 0.0% 0.0%
>= 1024 539 0.2% 0.2%
The residual is fully accounted for. 87.3% of vertices sit in 1..255 at about 92.5%
fixedness, contributing 0.873 * 0.925 = 80.8 points; the 5,851 at i16[3] == 0
contribute 1.8 more. That is the 82.5% measured, to the point.
The remaining ~6% are not 12-bit fixed at all. i16[3] >= 256, and negatives, score
0% at BOTH 4096 and 256 -- 6,271 vertices in an encoding this reading does not
see. And i16[3] == 0 scores 29.2% at 4096 but 60.7% at 256, which is a third
distribution rather than a weaker version of the same one.
What this establishes: i16[3] is a per-vertex packed attribute whose value SELECTS
HOW the texcoord pair is stored, rather than an independent field sitting beside
the texcoords. 1-255 is the ordinary case, 0 is special, 256-and-up and negatives
are a different encoding. What those three cases MEAN is still open -- texture
index, normal index, flag or a combination -- but it is now a field whose role is
known, which is more than "packed word".
Worth noting the shape of the claim this replaces: "the texcoords are 12-bit fixed
at i16[4]/i16[5], 73.6% of the time, unexplained". The field position was right;
the 73.6% was a population average over three encodings. Partitioning by the one
variable that separates them turns a mystery into a lookup -- rule 3a again, and the
third time this session that partitioning a mystery produced the answer rather than
merely bounding it.
Read the whole tfrags block from RAM -- 4,379,200 bytes at 0x848000 -- and compared
every one of the 833 patches against its disc copy, grouped by whether it CAN morph:
patches with NO morphing entries 334 differing: max 10.59%, median 7.80%
patches WITH morphing entries 499 differing: max 10.49%, median 8.12%
patches 100% identical to disc 0/334 and 0/499
The two groups are indistinguishable. Morphing is NOT what changes the bytes, and
this disproofs the claim two commits ago that the 8.6% was "morphed positions with
topology and UVs left verbatim". If it were, the 334 patches with no morphing
entries would be untouched -- and none of them is.
Nor is it a pointer relocation: the differing bytes fall in short structured runs
(257-263, 272-277, 305 on patch 0) whose u32 deltas share no common value.
The explanation consistent with everything measured is that the loader REGENERATES
each patch's VU packet stream for the LOD tier it selected. That happens to every
patch, morphing or not -- and it is why morphing shows no signal.
This also settles the verification route, which the previous commit left as "needs
the camera distance at the savestate". It does not, and no camera distance would
help: this memory image shows the LOD-selected form rather than the morph state,
so there is nothing in it to solve a weight against. Verifying the derivation needs
a captured frame at a known view distance instead.
What is established now, all measured on the full population rather than inferred
from one patch:
- data and cube_ofs are relocated pointers, delta 0x848000, on all 833 records
- every patch's bytes are rewritten, ~8%, by LOD-driven packet regeneration
- morphing is not observable in this memory image
- 833 of 833 patches parse; 499 of 833 have morphing entries; 43,613 vertex-info
entries across the level
Two corrections to the previous commit, both from continuing the RAM comparison.
1. The previous commit said the RAM comparison "names two fields the disc-side
work had left as data and an unnamed word". It does not. +0x10 is
TfragHeader::data and +0x38 is TfragHeader::cube_ofs, both already named in the
struct. What RAM adds is that both are POINTERS -- relocated by adding the
block's RAM base -- rather than plain offsets. That is still worth knowing, but
it is not a naming.
2. The previous commit attributed the 8.6% of rewritten bytes to "morphed
positions with topology and UVs left verbatim". That is wrong, and patch 1 shows
why. Comparing 4 KiB, disc 0xdf60 against RAM 0x855f60:
bytes differing 354 of 4,096 = 8.6%
contiguous runs 120
longest run 8 bytes
extent +0xc1 through the end of the window
Patch 1 has 83 positions and 83 vertex-info entries, of which 13 morph. If
morphing rewrote position fields alone, the differing bytes would be a 13/83 slice
of a 498-byte position array -- a couple of dozen bytes in one contiguous region.
Instead 354 bytes differ, spread over 120 short runs across the whole window.
So the loader is not patching positions in place: it REGENERATES the VU packet
stream for the current morph state. That is what morphing is on this hardware --
VU1 GIF packets, not a CPU-side vertex array.
Consequence for verification, and the reason this is worth committing rather than
just noting: the plan of "map RAM vertices to disc indices and solve for the weight"
does not work against this memory image. The positions are inside regenerated
packets, so there is no stable per-vertex correspondence to read a weight out of.
Verifying the morph weight against the original needs the CAMERA DISTANCE at the
savestate, so the weight can be computed from the derivation and compared with the
rendered result, rather than read back out of RAM.
Population figures gathered on the way: 833 of 833 patches parse, carrying 43,613
vertex-info entries; 499 of 833 patches have at least one morphing entry, and
patch 0 -- the first record -- is base-only with none.
Also fixed a measurement error mid-run: I compared patch 1 at disc 0xdf60 against
RAM 0x855f00, off by 0x60, and got "every byte differs". The corrected base is
0x855f60.
The tfrags block header -- table_offset 0x40, tfrag_count 833, lod_distance 32.0
-- hits once in the level-0 savestate at RAM 0x848000, so disc offset 0 maps to
RAM 0x848000 and the block can be compared position for position.
Result 1: exactly two fields per record are rewritten. Comparing all 833 64-byte
table records field by field:
records compared 833
u32 fields differing 1,665 = 833 x 2, and no others
their offsets +0x10 and +0x38, on every record
relocation delta 8,683,584 = 0x848000, on 833 of them
So +0x10 and +0x38 are block-relative pointers and the loader adds the block's
RAM base to each. Every other byte of all 833 records is verbatim. Six fields
differ by 0xF80000 and 0x1F00000 instead -- three records pointing into a
different buffer, recorded rather than averaged.
This names two fields the disc-side work had left as `data` and an unnamed word.
Result 2: behind the pointers, about 8.6% of bytes are rewritten. Following
record 0's +0x10 to disc 0xd040 / RAM 0x855040 and comparing 16 KiB: 14,977 bytes
identical, 1,407 differ = 8.6%. The differences spread across many field positions
rather than concentrating, which is what morphed POSITIONS look like -- topology,
UVs, counts and command lists stay verbatim while a fraction of the position bytes
move. 8.6% is about half of a 16-byte vertex's six position bytes, consistent with
roughly half the vertices in this patch mid-morph at the saved camera distance.
Not established: that the morphed bytes match resolve_vertex's formula. That needs
mapping RAM vertices to disc vertex indices and solving for the weight, which is
the next step -- but the framing is now known well enough to attempt it, and the
two pointer fields are confirmed rather than inferred.
This also answers the "is morphing real?" question that made the earlier tfrags
probe fail to match RAM: the tfrags block is not verbatim in RAM, because both its
table pointers are relocated and its vertex positions are morphed.
Picked tfrag morph targets from the open list and found the premise was wrong.
There are no morph targets in this format. Morphing is a blend toward the midpoint
of the vertex's two PARENT vertices -- g[parent1_position] and
g[parent2_position] -- which are other vertices in the same tfrag. The engine
implements exactly that in resolve_vertex, and it is covered by
morphing_blends_toward_the_parent_midpoint plus
morph_weight_runs_from_zero_to_one_across_its_band and
collapse_distances_are_the_far_edge_of_each_band. So "morph targets unrecovered"
was never a gap; it named a thing that does not exist in the format.
Strip bookkeeping is likewise already there: TfragStrip carries
vertex_count_and_flag, and tfrag_triangles walks the strip command lists and index
arrays to build triangles. That was the other half of the same roadmap entry.
Coverage, all passing: 36 tfrag tests, 9 texture tests, 23 gc-engine tests. The
real-disc texture test decodes all 1595 level textures, with pixels and mip levels,
so the module doc's "not yet recovered: the texel data the texture table points
into" is also stale.
Two stale claims corrected:
1. crates/gc-formats/src/tfrag.rs said "Not yet recovered: the LOD switch and morph
rules in TfragProc, and the texel data the texture table points into. See
docs/plan/tfrag.md." The rules are recovered and tested, the texels are
decoded, and docs/plan/tfrag.md DOES NOT EXIST -- a dangling reference to a
file that was never written.
2. docs/roadmap.md marked tfrag partial with "Still open: morph targets and the
strip/chain bookkeeping". Neither is open.
The honest replacement for both is what is genuinely not claimed: the LOD switch
distances, morph weight and collapse edge are DERIVED from the data -- tiers are
classified from the four V4_8 parent arrays, the weight runs zero to one across
each tier's band -- but have not been checked against PCSX2 the way the lighting
pass was. That is a verification gap, not a parsing gap, and it needs a frame
captured at a known view distance rather than more disc-side measurement.
unblock
Continuing on the verbatim payload with the grid known.
Cells are not zero-separated. Counting maximal runs of non-zero u16 across the
whole payload:
level 0 363 cells 15,115 runs mean run 25.3 u16
level 1 384 cells 24,647 runs mean run 51.5 u16
level 2 53 cells 31,512 runs mean run 30.3 u16
Two to three orders of magnitude apart, so a zero run is not a cell boundary.
Cells are not uniform blocks either: level 0 averages 2,488 bytes per cell and
level 2 averages 43,684, which cannot both be one fixed-size record.
The run lengths are not records. Their histogram has a long tail of
near-power-of-two values -- 2,140, 1,184, 840, 816, 628, 604, 562 -- which is
what you get when a run is bounded by a single zero byte inside otherwise dense
data, not by a delimiter. With the 18%-zeros finding, the payload reads as dense
data with sparse holes rather than a delimited sequence.
So all three segmentation theories are now excluded on the full population rather
than on a sample: not ascending u32 tables, not zero-run delimiters, not fixed
stride.
What would actually unblock this: overlay.bin IS the game's own code, and the
collision parser is in it. Camera_CollPrimTest WARNING! - grid out of bounds!
(RAR) was located in the debug string pool earlier, and that string is inside the
collision routine. So the format is recoverable -- but not by measuring bytes. It
needs disassembly of that routine, or an R&C2 reference for the collision cell
record.
Stating that plainly is worth more than another scan. The disc-side measurement
space for collision is exhausted: the bytes are available, verbatim, the grid is
known, and three independent segmentation theories have failed on the whole
population. Further probing without the loader's own logic will not converge.
With the payload confirmed verbatim in RAM, the disc bytes can be read directly.
The grid dimensions are the first two u16 of the payload: level 0 is 11 x 33 =
363 cells, level 1 24 x 16 = 384, level 2 1 x 53 = 53. Level 2 being one cell wide
is worth noting -- the grids are not all similar in shape.
Two readings disproved, both by measuring the whole payload instead of a sample.
An ascending u32 offset table. A probe from payload +1024 reads f4ae0000 40af0000
c4b2000020b20000, which is three ascending values before the fourth goes down
(0xb2c4 to 0xb220). Read as a table, that was a three-value impression. Scanning
all three levels for maximal ascending u32 runs of 8 or more entries finds zero,
on every level.
A global alternating-zero stride. The head of level 0 shows an obvious (value, 0)
pattern from about i16[37]:
[32..] [1808, 1872, 2198, 0, 34, 86, 25848, 1, 25824, 1,
-13028, 0, -12284, 0, -12228, 0, -12716, 0, ...]
Measured over the whole payload, both alignments score far below what that sample
suggests: even-slot aligned 18% second-half zero, odd-slot aligned 13%. The
sample again.
What the global numbers do say is that the payload is heavily zero-bearing -- 18%
against a chance rate of about 0.0015% -- which is what a sparse or variable-length
encoding looks like, and consistent with the variable-length reading established
earlier. The 18-against-13 split is a real alignment preference but a weak one.
The cell layout is still undecoded. The useful position now is that the bytes can
be read as they are and that they are sparse and variable-length rather than
fixed-stride, so the next attempt should look for a terminator-delimited walk
through the sparse region rather than for a record size.
This overturns the premise collision work was blocked on.
Searching the level-0 savestate for a probe from collision payload +1024 hits at
0xac6c00; a probe from +64 hits at 0xac6840. The difference is 0x3C0 = 960, which
is exactly 1024 - 64, so the payload sits at a fixed relative address -- RAM
0xac6800 for the disc address 0x27e800 -- and reading it back is byte-for-byte the
disc's own bytes:
RAM 0xac6800 0b 00 21 00 b6 08 ad 08 a4 08 6b 08 38 08 63 07
disc 0x27e800 0b 00 21 00 b6 08 ad 08 a4 08 6b 08 38 08 63 07
Nothing is reshaped. That is the opposite of the tie block, where the loader
relocates a pointer inside each record.
The retraction. The previous note reads: "level 1's 24 x 16 grid dimensions
appear in memory, level 0's 11 x 33 appear nowhere, so disc-side decoding has to
work against the reshaped form." Level 0's dimensions ARE in RAM, at 0xac6800 --
0b 00 and 21 00 are the first two bytes of the block, 11 and 33. The search that
missed them looked for 0b 00 00 00 21 00 00 00, the dimensions as a pair of u32s.
They are stored as two u16s, so that pattern never occurs.
So the negative result was an artifact of the search pattern, not evidence about
the data -- and it was then used as a premise for several rounds of disc-side
work. Both lessons were already written down in AGENTS.md and both were re-learned:
a negative from a byte search bounds the search, not the format; and the control
here is one command -- read the predicted RAM address and compare it to the disc --
which would have shown the verbatim copy immediately.
Also recorded while mapping it: the payload contains ascending u32 offset tables
from +1024 onward, and its first two bytes are the grid dimensions, so level 0's
grid is 11 x 33.
The cell layout itself is still undecoded, but the blocker is gone: disc-side
decoding no longer has to work against a reshaped form.
Scoring each adjacent i16 slot pair of the 16-byte tie vertex for 12-bit
fixedness over 36,501 level-0 vertices locates the fields:
(i16[0], i16[1]) 1.1% not a texcoord
(i16[2], i16[3]) 4.8% not a texcoord
(i16[4], i16[5]) 73.6% <- the texcoords
(i16[6], i16[7]) 76.6% <- the near-constant 4096/0 pair
(6,7) scores HIGHER and is not a texcoord. It is the near-constant 4096, 0 pair:
4096 is 1.0 in 12-bit fixed and 0 is zero, so both are trivially multiples of
4096. That is the constant artifact again -- a fixed pair would score 100%
anywhere. The field is (4,5), with (0,1) and (2,3) ruled out at 1.1% and 4.8%.
The first four slots span the full i16 range with 10,803 to 13,421 distinct values
each, so they are neither texcoords nor small fields.
The vertex base is confirmed to be vert_ofs alone. Adding the shader or control
region makes it worse --
vert_ofs only 73.6%
bfc + vert_ofs 69.8%
bfc + control_count*control_size + vert_ofs 66.7%
-- so the 73.6% residual is not a mislocated base. Worth recording because
vert_ofs (3..7) numerically overlaps the shader region (5*shader_count, 5..20
quadwords), which made a combined base look plausible; measurement says the
simple one is right.
The test asserts (4,5) above 60% and (0,1)/(2,3) below 5%/15%, and names the
(6,7) artifact in its failure message so the higher-scoring pair cannot be
mistaken for the answer later.
With TieRecord identified as RAC1's TiePacketHeader, each record's leading word
points at its packet in the block payload. Following it and stepping vert_size
elements from vert_ofs reaches the vertices: 95,612 of them on levels 0-2.
Element size settled by comparison, not assumed. Requiring the texcoord pair to be
12-bit fixed -- both words multiples of 4096, which is how 0x1000 = 1.0 reads:
element 16 bytes : 78,919 / 95,612 = 82.5%
element 8 bytes : 45,661 / 95,612 = 47.8%
element 12 bytes : 44,834 / 95,612 = 44.8%
Sixteen wins by a factor of 1.7 over the nearest alternative.
Observed layout, 8 x i16: the first three slots span the full i16 range with
10,803-13,421 distinct values each; i16[4] and [5] are the texcoords and are both
multiples of 4096 on 82.5%; i16[6] reads 4096 on 85.8% and i16[7] reads 0 on
84.0%.
Not established: what any field means, and why the texcoord rate is 82.5% rather
than 100%. The residual tracks record shape -- (shaders 2, control 7, size 2)
hits 60%, (shaders 1, control 18, size 5) hits 89% -- so it is not one bad block
but something per-packet this reading does not capture.
Worth recording that this is the OPPOSITE of what the disc alone concluded two
commits ago. Reading the header word at +0x00 as an element count gave 0 of 922
blocks dividing; it is the byte offset to the payload, so the table it bounds has
16-byte records. The number was right and the reading was wrong, which is what
the packet-header identification corrected.
The roadmap's tie entry was also stale -- it still said "a 16-byte record is
withdrawn", which the packet-header work had already reversed. Rewritten from the
commits: both indirections, self-delimiting blocks, the TiePacketHeader
identification with its 15,219/15,219 rule, the 16-byte vertex element, and what
remains open. A test asserts the element size against both alternatives so the
82.5% figure cannot be quietly replaced by an assumption.
The 16-byte tie record is R&C1's TiePacketHeader offset for offset: a data
pointer at +0, then R&C1's byte fields at +4 through +0x10. The identification
is not a resemblance but R&C1's own rule -- bfc_distance == 5 * shader_count --
holding on all 15,219 records of all 2,177 blocks on all 27 levels.
Measured over the same set, and reported rather than asserted where it is not
universal:
bfc_distance == 5 * shader_count 15219 / 15219
vert_ofs <= vert_size all
scissor_size <= scissor_ofs all
control_size <= control_count all
color_ofs == vert_size 12640 / 15219
color_ofs == vert_ofs + vert_size 0 / 15219
color_count, strip_count, strip_vertex_count zero 15089 / 15077 / 15078
R&C1's rule for the colour offset holding on none of them is the interesting
line: the two games use that field differently, so it is named color_ofs for
what this game's majority reads as and the 2,579 exceptions are counted rather
than averaged away. TieRecord parses all twelve fields, invariants_hold asserts
only the universal four, and the disc test counts the rest.
A packet-length formula is withdrawn rather than softened. (vert_ofs +
vert_size + scissor_ofs + scissor_size) * 16 was measured at 61 of 61 pairs and
is not reproduced on the correct blocks -- 0 of 13,042. The block set was the
error, not the arithmetic: the script that produced the 61 read packets[2]
where packets[1] was meant, so it measured 461 blocks instead of the 2,177 the
class tables define, and the formula agreed perfectly with a population that
was not the tie blocks. TieRecord::packet_bytes goes with it, and the
retraction is recorded in the format doc and in the struct.
That misread also explains why the earlier RAM verification saw different
numbers for the two inputs -- same lesson as rule 3, arriving through a wrong
subscript instead of a wrong span.
The relocation rule from the previous commit was established by a script that
lived in a shell history, which means the next person would have had to
rediscover it. It is now `gc-trace verify-tie-records <unpacked-dir> <level>
<class> <lod> <src>`: resolve the class, parse the block, search the RAM image
for the loaded record table, report what it found, and exit 0 only when every
record's pointer has become table + word and every field byte is verbatim.
Swept over every (class, lod) slot of two levels, from PCSX2 savestates:
level 0: 87 slots with a block, 87 verified, 0 mismatched, 61 absent
level 1: 104 slots with a block, 104 verified, 0 mismatched, 78 absent
So the rule holds on every LOD block of two levels the game had loaded, which
is what makes the record layout a finding rather than a reading of one block.
Writing the command also caught an off-by-one in the parser it exercises:
TieBlockHeader::records started its slice after the 16-byte header, but the
header IS the first record -- the table runs from the block start to
data_offset, and a block with data_offset 0x50 holds five records, the first
being the header itself and its pointer naming where the payload begins. Found
because the tool's first run relocated nothing while the same check in a script
had passed; the unit test fixture was wrong in the same way and is fixed.
Two operational notes now in the format doc: the live emulator is not a stable
input (verifying level 1 over PINE matched 104 of 104 while the user was on
that planet, and the same command minutes later found no matching copy because
they had returned to level 0 -- a miss there is not a counterexample, so the
report says "no copy matched" rather than "failed"), and a class index past the
table count is an error rather than a mismatch, so a sweep should stop at the
level's class count rather than a round number.
The tie block's word at +0x00 was read as an element count, and the reading
failed in a way worth recording: the payload length divided by it is a whole
number on 0 of 922 blocks, at any record size and any header up to 512 bytes.
That is the correct measurement of the wrong hypothesis -- the word never
described the payload.
Range measured: every LOD block of level 1 whose word is a non-zero multiple
of 16, plus the two read by hand in full.
Method: locate each block's record table in PCSX2's RAM by its twelve field
bytes, which are the only distinctive 12-byte run in a block, then compare
every record against its disc copy. Each block has several copies in RAM, and
the one that matches is the loaded one -- checking the first hit is what made
the first attempt report 0 of 16.
Result: on 16 of 16 blocks, every record's leading word has become
table_address + word, and the other twelve bytes are byte-identical to the
disc. So a block is [16-byte header][record table][vertex payload], the records
are 16 bytes -- a leading u32 pointer plus twelve bytes of fields -- and the
header's word at +0x00 is the byte offset from the block start to the payload,
that is, the length of the table.
That is what the multiple-of-16 was: the table holds 16-byte records. It is
never zero because there is always at least the first record, and its 42
distinct values are table lengths. Each record's pointer is relative to the
table start, relocated at load to table + word.
TieBlockHeader::data_offset and TieBlockHeader::records carry it, bound by the
offset, with TieRecord for the twelve field bytes, and the disc test asserts
the table is bounded on every block. What the records' fields name and what the
payload holds are still open -- the payload is known to be variable-length,
float-shaped, not LZ77 and not VIF1.
The loader note said every record gains a 02 00 prefix in RAM. It does not:
the word is the record's own leading field, already on the disc at every
record. What the loader changes is the pointer -- it addresses the word, two
bytes earlier than the disc's table -- and that pointer shift is what made the
bytes look inserted.
Measured, in order:
- the two bytes before a payload are identical in core_data and in RAM, record
for record: 02 00 on level 0's 90-record chunk at 0x100b302, 08 00 on the
level's first chunk. Nothing is inserted;
- the RAM table's low 24 bits are the payload's minus 2, consistently, so the
record is [leading u16][two index bytes][payload] and the loaded table
addresses the leading word. That is also why the index bytes read as +2 from
the pointer -- the same +2 the earlier RAM result quotes;
- the word is a field, not a pad: uniform across a chunk's records on 1,439 of
1,509 chunks (None on the 70 that vary, reported rather than averaged),
fixed within all 71 chunk types, and only ever 1, 2, 4, 6, 8 or 10 -- 554
chunks carry 2, 520 carry 4.
Two readings of the value were measured and disproved, so they are recorded as
disproved rather than left as candidates: it is not an entry count (the payload
divided by it leaves 57 distinct fractional "sizes", and 162 chunks do not
divide at all), and it is not the number of distinct leading bytes (agrees on 0
of 1,506 chunks).
Chunk gains `prefix: Option<u16>`, and the disc test asserts the uniformity,
the per-type fixity and the value set. The scratch example that produced the
per-type class-ID table is dropped now that the same scan is a committed test,
which also lands here: moby class IDs sit in one body slot per chunk type, with
the slot varying by type and the distinct >= 4 filter load-bearing -- without it
the top slots are constants that read as 305x because any value below the moby
count counts as a hit.
Four commits of chunk work -- uniform record length, the leading-byte phase
counter, the two-byte record index, and the PCSX2 RAM findings -- were written to
docs/formats/core-data-regions.md and docs/plan/pcsx2-verification.md but never
reached the roadmap, because the scripted replacements kept missing their target
text and I did not check that they had applied.
That is the staleness complained about earlier recurring through a different
route: not a missing edit, but a silently-failed one. The tie, collision and
level_overlay entries had landed; only the chunk ones had not.
Rewritten from the commits rather than from memory, and the numbers are the ones
the tests assert: 1,509 chunks, 34,491 records, uniform length 1,509/1,509,
phase counter 1,262/1,509, index rule 1,235/1,509, RAM prefix 02 00 on 90/90.
This is what RAM was needed for: nothing about the loader's transformation is
recoverable from the disc.
Anchor: a 32-byte probe from inside a chunk record in level 0's core_data, searched
in the level-0 savestate. It hits at 0x18538a8 and the record's first bytes at
0x185387e -- exactly chunk_base + record_offset, no rescaling. So the chunk payload
is copied VERBATIM; unlike collision and the shrub packet tables it is not expanded
or repacked.
Result 1: the offset table is rewritten to absolute EE addresses.
disc: f000057c f00006b0 f00007e4 f0000914
RAM: 0xf185387c 0xf18539b0 0xf1853ae4 0xf1853c14
The 0xf0000000|low24 tag becomes 0xf1850000|low24 -- tag replaced by the chunk's
load base, low 24 bits untouched. Consecutive deltas agree exactly (0x134, 0x134,
0x130), which is why the disc-side stride is approximate rather than constant.
Result 2: every record gains a constant 2-byte prefix 02 00, and the rewritten
table points at THAT rather than at the record data. Walking all 90 records from
the RAM dump: prefix 02 00 on 90 of 90; the disc's index bytes preserved unchanged
at +2/+3 on 90 of 90.
Result 3, the useful one: the byte at +2 takes exactly 0, 16, 32, ... 240 -- the
16 values index_rule predicts for step = 16, phase = 0. The mixed-radix record index
derived from the disc is corroborated on the running game's own copy of the chunk.
That is a disc-derived claim validated against the game, which is the strongest form
of check available here.
On disc a chunk is [16-byte header][10 zero bytes][tagged offset table][records].
In RAM it is [16-byte header][10 zero bytes][absolute-pointer table][records each
prefixed with 02 00]. The 02 00 is unexplained -- possibly a record type or an
alignment the loader needs -- but it is constant, so like the two index bytes it
carries no per-record information.
With the record framing settled (leading two bytes = record index), the body is
bytes 2..len. Four probes, mostly negative, recorded so the next attempt does not
repeat them.
No count field. For every chunk type, every u16 and u32 position in the first 64
body bytes, at element strides 1,2,3,4,6,8,12,16: does the stored value equal
(record_len - offset) / stride on 90%+ of records? No type produced a hit. The
record does not carry an element count near its head.
The body is dense, not float-dominated. For the largest type (2430 records,
326-byte bodies): f32 reads within +/-5000 of origin 74%, uniform-random control
49%, zero-byte fraction 13%. Above baseline but far short of the tie blocks
(53-64% against 16%, with 43% zero bytes).
There is an index-like region near the tail. In the largest type, 12 of 163 u16
slots have a maximum below 0x1000 -- too small for float or half-float, so those
look like indices or small counts -- clustered at slots 111-121 and 130-131.
A sample-size correction. Dumping six records of one chunk showed slots 112-119
zero on every record plus a long run of other constants, which reads as "the
record is mostly a per-chunk template". Measured across the 1,171 chunks with at
least 8 records: 15,422 of 164,390 u16 slots are constant across their chunk,
9.4%, and 0 of 1,171 chunks are entirely constant.
So 9.4%, and no chunk is a template. Six records looked like a majority because
that chunk's tail carries shared constants. Per type the figure ranges from 9.4%
overall to 55.6% on one type, so it is reported per type as well as pooled --
the pooled figure alone would hide that spread, which is the same trap rule 3a
describes.
Continuing from the phase counter. Within a chunk:
i : 0 1 2 3 4 5 6 7 8 9 10 11 ... 15 16 17 ...
byte0: 0 16 32 48 64 80 96 112 128 144 160 176 ... 240 0 16 32
byte1: 0 0 0 0 0 0 0 0 0 0 0 0 ... 0 1 1 1
byte0 is step*(j mod g) and byte1 is (j / g), where g = 256/step and
j = index + phase. The pair is the record's index in mixed radix: byte0 carries
the low digits scaled by step, byte1 the quotient.
Measured over all 27 levels: 1,235 of 1,509 chunks fit this on both bytes. Byte 0
alone fits 1,262, so the 27-chunk difference are chunks whose first byte follows
the rule and whose second does not. On 1,235 chunks a record knows its own
position and the two leading bytes hold no per-record information.
ResolvedChunk::index_rule returns the (step, phase) only when the whole chunk fits
on BOTH bytes, so the 274 that do not stay distinguishable and nothing skips a
leading byte on a rule that does not apply to it.
A number that moved when the implementation changed: a first probe reported
1,181; the committed method reports 1,235. The method is right. The probe
accumulated ok0/ok1 flags across candidate phases with an early-exit that could
skip a phase which would have matched both bytes, and it also disagreed with
bucket_rule on the byte-0 count (1,208 against 1,262), which is what gave it away.
Recorded because both numbers came from the same run and only the discrepancy
exposed the probe as wrong. The lesson is narrow: when a one-off probe and a
committed helper disagree, check the probe first.
has a uniform record length
Follows the alignment result and settles what the leading byte is.
Within one chunk the leading byte steps by a constant and wraps:
offsets: 1420 1748 2072 2392 2716 3044 3372 3700 4028 4352 ...
byte0: 0 16 32 48 64 80 96 112 128 144 ...
Sweeping every (step, phase) over all 27 levels:
fully explained: 1,262 of 1,509 chunks
by step: 8 -> 51 chunks, 16 -> 554, 32 -> 520, 64 -> 137
unexplained: 247 chunks across 11 chunk types
So on 1,262 chunks the leading byte carries no per-record information: it is a
deterministic function of the record's index and the chunk's phase, which is a
position marker rather than data. That is why the earlier "16 discrete values,
uniform counts" pattern appeared at all.
ResolvedChunk::bucket_rule returns the (step, phase) when the whole chunk fits and
None otherwise, so the 247 unexplained chunks stay distinguishable and nothing
skips a leading byte on the strength of a rule that does not apply to it.
Every chunk has a uniform record length: 1,509 of 1,509. Length varies BETWEEN
chunks of a type -- one type runs 76 to 104 bytes -- but never within one. This
withdraws the previous commit's "variable-length type", which came from pooling
minlen/maxlen across all of a type's records at once. That is the pooled-claim
error rule 3a exists to prevent, and it is the third time this session a pooled
measurement produced a claim that partitioning reversed. The type is not
variable-length; the chunk is fixed-length and different chunks differ.
Characterised every chunk type -- 77 of them, 1,509 chunks, 34,491 records, all 27
levels.
Byte 0 of every record is a small aligned index, not record content. It takes 16
discrete values per chunk type (0x00, 0x10, ... 0xf0) with near-uniform counts
WITHIN each type. Uniform weighting is the tell: a running byte offset would
weight the buckets unevenly, because records differ in size.
type_id recs len byte0
0x007d7ec438 2430 328 0x00:162 0x10:162 0x20:162 ... 0xf0:135
0x00429ec43d 2079 368 0x00:162 0x10:135 0x20:135 ... 0xf0:108
0x00d76cc453 810 108 0x00:27 0x08:27 0x10:27 ... 0xe8:27 <- steps by 8
The alignment step is per type, not global. The leading byte is a multiple of 8 on
34,432 of 34,491 records but a multiple of 16 on only 33,349 -- the 1,142-record
difference falls in seven types, four of which step by 8. 59 records in three types
break even the 8-byte rule, and that is where the variable-length type lives:
0x00f2ef4275, records 76 to 104 bytes, no single stride to align to.
What the bucket indexes is NOT named. It is reported as CHUNK_RECORD_ALIGNMENT and
record_is_aligned rather than as a parsed field, because naming it would be a guess
while the invariant is solid. The test pins the exact exception set -- 59 records
across 3 types -- so a change there fails loudly.
A measurement I initially got wrong: the first version of the constant claimed a
16-byte multiple on 34,432 records. 34,432 is the multiple-of-8 count; the
multiple-of-16 count is 33,349. Right number, wrong label, and the test caught it.
That is the second time this session a correct number carried a wrong label, after
the tie four-value retraction. Both times the assertion did, which is the argument
for putting these numbers in tests rather than only in prose.
Still open: what the bucket indexes, and the remaining record fields.
parse_tie_block walks to a 0xf6 terminator that every real block carries, on
2177 of 2177 across all 27 levels, followed by zero padding. This is what makes
the blocks parseable: packets[1] bounds one on only 461 of 1864 classes, so the
terminator is the only general way to find one.
The terminator was worth a control, because 0xf6 could have been byte-frequency
coincidence. It occurs in payloads at 0.83x the uniform rate -- slightly LESS
often than chance -- while the last non-zero byte is 0xf6 on 909 of 922 blocks in
the first subset, against 17 of 922 for the same blocks cut at a random point. So
it is a terminator, not a common byte.
packets == 0 means the class has NO block at that LOD: of 3,728 (class, LOD)
slots, 2177 point at a block and 1551 are exactly zero.
Corrected header, over all 2177:
+0x00 count multiple of 16 on all; 42 distinct; never zero
+0x02 zero on all
+0x04 exactly four values: 0x501 (1069), 0xa02 (720), 0xf03 (350),
0x1404 (38)
+0x06 16 distinct, never zero
+0x08 180 distinct, never zero
+0x0a 61 distinct, never zero
+0x0c 125 distinct, never zero, clustered 0x043f..0x0443
+0x0e 11 distinct, zero on 2162 -- the only non-universal field
+0x0c clusters around 1087-1091, close enough to a block length to be worth
stating, but it is NOT the length: the difference against the true payload length
scatters from -12 to -172.
A correction to the previous commit. It reported 3,728 blocks all terminating and
packed_texture taking 842 distinct values, and on that basis retracted the
four-value claim. Both numbers were wrong the same way: the scan did not skip
packets == 0, so it computed class_base + 0 and walked forward from the CLASS
BODY, finding a terminator there by accident. That contaminated 1551 of the 3728.
With the zeros excluded the slot takes exactly four values on all 2177 blocks.
The four-value claim was right; my retraction of it was wrong; and the retraction
was made on a number from the same contaminated set. A retraction is not
automatically better than the claim it retracts -- it needs its own bounds.
Still open: the payload's records. Established this round: blocks are
self-delimiting, absent LODs are marked by a zero packets word, and the header is
fully characterised. Not established: any record boundary inside the payload,
which stays variable-length.
TieBlockHeader parses the block's own 16 bytes, measured over the 922 blocks
where "the next packets offset ends this block" holds, across all 27 levels. A
real-disc test asserts every figure.
+0x00 count multiple of 16 on 922/922; 16..2096; 72 distinct disc-wide
+0x02 zero on 922/922
+0x04 ONE OF FOUR VALUES DISC-WIDE: 0x501, 0xa02, 0xf03, 0x1404
+0x06 15 distinct, 0x0206..0x0512
+0x08 135 distinct, 0x0b05..0x4ee4
+0x0a 45 distinct
+0x0c 94 distinct, 0x010e..0x0553
+0x0e zero on 914/922
+0x04 is packed rather than a magnitude: four values across the whole disc with
the two bytes moving independently, low 1-4 and high 5/10/15/20 in lockstep. That
it is a texture index paired with a scale is a guess; that it is two fields is
not.
The count is not a record count -- block_size minus the header is divisible by it
on 0 of 922, for any record size and any header up to 512 bytes.
On the payload, a control was needed and changed the answer. Reading blocks as
3 x f32 triples, 53-64% of reads land within +/-5000 of origin, against 16% for
a uniform-random control of identical length. So the content is float-shaped and
genuinely not random.
But the rate is flat across every start offset and every stride tried, 53-65%
throughout. A statistic that cannot separate the right stride from the wrong one
cannot identify a stride -- rule 4's exact case. Flatness plus a count that does
not tile means the records are variable length.
That killed two readings. A VIF1 command walk consumes 0 of 88 blocks exactly,
so it is not a VIF1 CPU-format stream. wad::decompress rejects the block at every
start offset, so it is not WAD LZ77.
Scanning every (start, stride) pair from 0x10 to 0x7f for byte positions with a
small distinct-value vocabulary finds none: every position takes all 256 values
at every alignment. For float payload that is expected, so the scan found nothing
rather than confirming anything -- worth saying, because "no fields found" here is
uninformative, not evidence against fields.
Still open: what the payload holds. Established: variable-length, float-shaped,
non-random, not LZ77, not VIF1. Not established: any record boundary.
The previous commit inferred a 16-byte record from "922 of 922 consecutive block
sizes are a multiple of 16". That inference does not hold, and the claim is
withdrawn rather than softened.
What survives:
- Every one of those 922 blocks opens with a u32 that is positive and a multiple
of 4, on 922 of 922, ranging 48 to 1744 with 17 distinct values across level
0's LOD0 blocks. The block opens with a count, not a size and not a pointer.
- Reading that u32 as a byte size and chaining to the next block lands exactly on
the next block start on 0 of 1383 blocks. Not a size chain.
- Assuming [header][count][count uniform records] and solving for the record size
gives no answer: no record size tiles the block for any header size up to 512
bytes. Not 16, not 12, not 8.
- Reading the block as uniform 16-byte units gives fields spanning their entire
legal range -- u32@0 from -2139029504 to 2139029587, u16@8 and u16@10 both
0..65535, and 8969 distinct values in i16@0 on level 0. Uniform units would not
produce that.
So the 16 is a property of the block size, not of a record. The earlier commit
also claimed a record size "on the classes where it can be measured"; those
classes are still correctly identified, but what they measure is the block size.
The strict-increase disproof from two commits ago stands untouched and is what
should have prevented this: the word at +0 was already known not to behave, so
treating 16-byte units as records contradicted a measurement sitting in the same
file. Divisibility is arithmetic; it is not evidence of a record boundary.
Roadmap updated to say block sizes are established and the record layout is not,
with the three measurements that rule the 16-byte reading out.
The previous commit reported "3580 non-empty LOD blocks compute as divisible by
16" and then said the figure should not be trusted, because it came from the
"next packets offset ends this block" rule. That rule is unsound: packets is
strictly increasing on only 461 of 1864 classes. On the other 1403 the two
offsets are unordered and cannot both be block starts into one ascending region.
Restricting to the 461 classes where the rule does hold, 922 of 922 consecutive
block sizes are a multiple of 16. That is the evidence for a 16-byte record, and
it is now scoped to the subset where it can be checked rather than claimed over
all 1864. The other 1403 are unmeasured, not measured-and-different.
packets[0] is a constant, not an offset: it reads 256 on 1864 of 1864 classes,
while packets[1] and packets[2] vary widely and land at different addresses in
core_data on every class. Read as a block start, 256 points just past the
0x48-byte header -- which is how it was being used, and exactly how a field gets
mistaken for a pointer. It is now TIE_PACKETS0 and nothing depends on it.
Still not decoded: what the 16 bytes hold. i16[0] at +4 takes 8969 distinct
values on level 0 and is not the two-valued field a 12-unit sample suggested; the
word at +0 is not a run offset.
The test pins all four numbers -- packets[0] constant, the 922/922 divisibility,
the 461/1864 ordering, and the 0/1864 monotonic disproof -- and asserts that the
ordering is *not* universal, so a future change there fails loudly instead of
quietly invalidating the record size.
resolve_tie_class walks the chain, which is two steps and both are easy to get
wrong:
1. The tie class table lives in core_index.bin, not core_data.bin. On level 0
the tables chain as moby at 192, 238 x 32 bytes later tie at 7808, 74 x 32
later shrub at 10176, which closes. Reading the same table out of
core_data.bin produces plausible garbage rather than an error.
2. ClassEntry::offset_in_asset_wad addresses core_data.bin, despite the field
name and despite core_data.wad sitting alongside it.
Verified: all 1864 tie classes resolve, o_class agreeing on every one, and all
5592 LOD block offsets land inside core_data.
Disproved, and this is the part worth reading: the word at +0 of a 16-byte unit
inside an LOD block. On the first twelve units of level 0's class 0 it rises
strictly -- 288, 1520, 2848, 3968, 5088, 6096, 7312, 8528, 9792, 10976, 12016,
13072 -- which reads exactly like a linked list, and the accompanying i16s looked
quantised, two distinct values in i16[0] and three in i16[1].
Across the population it is strictly increasing on 0 of 1864 blocks, and i16[0]
takes 8969 distinct values on level 0 alone rather than two. The sample was
twelve units out of 1,724,334. Both the monotonic reading and the quantisation
were artifacts of that sample.
That is the pooled-claim error AGENTS.md rule 3a exists to prevent, committed
almost verbatim: the first twelve units looked clean, and I was one step from
writing "u32[0] is a linked list of runs" into a format spec. The test now
asserts the disproof, so it cannot creep back.
Not established: the block's record size. 3580 non-empty LOD blocks compute as
divisible by 16, but some packets offsets run backwards, so the "next offset
delimits this block" end is wrong for them and that figure should not be trusted.
The tie layout stays partial: indirections and header done, records not.
Findings were going to docs/plan/level-assets.md and docs/formats/*.md while
roadmap.md kept listing work as open that has been done for some time. The worst
case was occlusion appearing twice: open at one line with "2 to 25 sectors, small
enough to read whole", and done further down with 412521 cells over 140939 masks.
The open line is deleted.
Also corrected:
- tfrag was listed flatly open while six tfrag entries beneath it were checked.
Now partial: format, topology, textures, LOD, normals and colour are done;
morph targets and strip/chain bookkeeping are not.
- sky was lumped in with volumes as open. Sky is done -- 149 sub-objects across 27
levels -- so it has its own entry, keeping the word at +0x0c documented as not
a count, since it disagrees with the table on most levels and reads 196617 on
level 2 against 4 sub-objects.
- level_overlay was open. It now has a parser and a 27/27 test, so it is partial
rather than done: header, constant block and level data parse, +0x10..+0x1220 is
identical on every level, and the word at +0x0c falls below the file length on 17
levels and above on 10. The header words are still unnamed and the Phase 3
seven-section parser is untouched.
- sound_bank was flatly open. It is partial: container framing parses and bounds
are verified on 27/27, but the SBlk record fields are not decoded and DAW shows
no stride, consistent with ADPCM.
- tie now records what is actually known, including the two indirection chains
that have not been walked: class table entries point into core_data.wad, and the
header's packets[0..3] resolve against the lump base. Level 0 has 74 tie, 238
moby and 7 shrub classes.
- collision now separates the header, which is decoded, from the cell layout,
which is not -- including the 17-entry table at +5932 against a 75-cell grid and
the disproved 64-byte RAM stride.
- moby_anim records the negative result: per-chunk-type record counts are
level-independent, so nothing in the chunk set is keyed by moby class.
The "Where we are" section now states plainly what has parsers and that what
remains open is naming rather than locating, and names the five types with no
decoded fields. Reading that list is how you would have known to ask.
Phase 2 remains in progress. Volumes and moby_anim have no located structure and
SBlk records, tie vertices and collision cells have located structure without
decoded fields.
Where the pointers go. Following them shows what gameplay.bin indexes. The leading
table is 39 words, all of which land inside the file, and three clusters
dominate: near 150000, near 800000 and near 1050000.
The region near 797808 is position data -- 31 consecutive plausible floats that
group cleanly into xyz triples:
246.948 201.015 73.867 | 88.240 186.000 302.598 | 107.712 12.723 183.125
353.591 126.365 19.962 | 182.068 374.001 122.567 | 116.961 265.001 437.866
and the region near 797456 is direction data -- unit-length vectors:
0.969 0.945 0.733 | -0.120 0.433 0.250 | -0.866 0.000 0.267
So the file's leading words are a table of pointers into float arrays, not a flat
list of small integers. The earlier reading of it as "u32 pointers holding
nothing" was too quick: it is a pointer table into data that is present.
Volumes remain unlocated. The position arrays are the right shape to bound a
volume and nothing yet shows the volume records themselves.
moby_anim was the one post-shrub chunk type with a concrete reason to exist: an
instance table would have to scale with the level's moby class count, so
comparing per-type record counts against that count was the obvious test.
It fails, and the way it fails is itself the result: per-type record counts are
level-independent. The same types report the same counts on every level tried:
type_id level0 level1 level2 level3
0x007d7ec438 90 90 90 90
0x00d3d8c4a9 90 90 90 90
0x00429ec43d 77 77 77 77
0x0053c8c4aa 77 77 77 77
0x007626c4a4 50 50 50 50
0x00fb88c462 47 47 47 47
Across all 27 levels there are 77 distinct chunk types, the largest at 2430
records total -- exactly 90 x 27.
What that rules out: nothing in this chunk set is a per-moby-class instance table,
because those counts would vary by level the way the moby class counts do. So
whatever holds moby animation data is either keyed by something other than the
class count, or it is not among these chunk types.
Also visible: the types pair up. 0x007d7ec438 and 0x00d3d8c4a9 both report 90 on
every level; 0x00429ec43d and 0x0053c8c4aa both report 77. Two chunk types
carrying identical, constant record counts on every level is a signature of two
views of the same fixed table rather than two independent tables that happen to
agree.
Still open: the pairings are described but not explained, and no type has been
named. The Moby class-ID enrichment from earlier stands separately -- 9493 hits
against 246.1 expected is real, and the class arrays are contiguous per type, but
that identifies where classes are referenced, not which chunk is moby_anim.
level_overlay was the least-characterised file in the level set -- no parser, no
located structure. It now has gc_formats::overlay and a real-disc test over all 27
levels:
overlay: 27 levels; field_00 1735808, field_08 1;
field_04 37540..=45308, field_0c 2763912..=3027336
(17 below the file length, 10 above);
+0x10..0x1220 identical on all
The header is four words. +0x0c is not the file length: it was recorded as one on
the strength of the first levels measured, where it sat near the file end, but
across all 27 it falls on both sides -- below on 17, above on 10 -- so it is an
ordinary level-specific word. The test asserts this, being the claim most worth
pinning.
The common prefix is 4 bytes, not the whole header: +0x00 is constant at 1735808
while +0x04 and +0x0c vary. Everything from +0x10 to +0x1220 -- 4624 bytes -- is
byte-identical on all 27 levels.
Still not named: what any of the four header words means, and how the
level-specific tail is structured. The module reports them as fields.
Also records in docs/plan/pcsx2-verification.md that the 64-byte RAM stride from
the previous commit is disproved. A 415 KB dump of level 1 finds the longest such
run at 27 records, not 384, and those are identity matrices -- 27 of them, one per
level. It is a per-level transform table, not the collision grid, and the 20.0
reported there was a coincidence.
Searching the savestates for the collision grid dimensions as a u32 pair -- level
0's 11 x 33 and level 1's 24 x 16 -- puts level 1's at several RAM addresses and
level 0's nowhere. So the grid dimensions are not held in RAM as adjacent u32s,
and the collision payload is reshaped on load, as the shrub packet tables are.
A nearby region of level 1's RAM does show structure. At 0x12e048c and
0x12e04cc, 64 bytes apart:
28 01 00 00 | 92 ff 00 00 | 04 00 a0 41 | ...
296 -110 20.0f
Searching for that shared 12-byte tag finds a consecutive run at a 64-byte
stride in slot 2:
0x12df390
0x12df3d0 +0x40
0x12df410 +0x40
plus scattered singles elsewhere. Slot 1 has the tag at four addresses not on a
64-byte stride, so the two levels lay it out differently.
What this is: a 64-byte record carrying two small signed values and a float
reading exactly 20.0 -- the shape of a per-cell entry with coordinates and a
radius or half-extent, which is what the collision grid would hold.
What this is not: it is not decoded. Three consecutive records is not enough to
establish a stride, the tag matches only a few records rather than a whole grid,
and 20.0 may be coincidence. Level 1's grid is 384 cells; nothing here has been
shown to count to 384.
Method note: savestates are compressed on disk, so a byte scan of the file finds
nothing. All RAM search must go through gc-trace find --state, which decompresses
first -- scanning the file directly reports zero hits for a sequence that
demonstrably exists in memory.
Range measured: the first 40 u32 and the whole file, on levels 0 and 1 --
1872256 and 2006400 bytes.
Result: the file is a flat array of u32, opening with a small count then a run of
ascending offsets, terminating on a zero:
level 0 336 797472 798832 799136 464 150672 150688 314800 479920 ...
... 1847600 1845456 797808 1829056 160 1849088 1249808 1847616 0
level 1 496 799904 800128 800432 624 151152 151168 315680 481456 ...
... 1984496 1982400 800112 1966000 160 1988160 1303456 1984512 0
Both levels carry the same shape. The sentinel 160 appears at u32 index 34 on
both and the zero terminator at index 38 on both, with counts 336 against 496
and 464 against 624 at the head. The leading ~38 words are a structured header
whose layout is level-independent apart from its sizes.
The offsets cluster into groups -- near 800000, 150000, 1050000, 1800000 --
which is what a table of pointers into several buffers looks like.
Not decoded: what the pointers point into, what 160 is, and why the zero
terminator sits at the same index on both levels. No gc_formats module exists for
this yet; it is characterised, not parsed.
On volumes: nothing resembling volumes was found here. gameplay.bin is u32
pointers throughout and holds no float data in its first 64 words read as f32 --
59 of 64 are denormals, i.e. small integers reinterpreted. Volumes remain
unlocated on the disc; nothing in the regions examined so far presents itself as
one.
The previous commit described a "debug string table on 16-byte alignment". It is
not a table. Dumping the 32 bytes before each run shows most are the tail of the
previous string:
+3104 "host0:r:/rc2/shock/wavetable.bin" 32 chars
+3120 "ck/wavetable.bin" its own tail
+3152 "/i5/shock/wavetable1.bin"
+3168 "ble1.bin" its own tail
and the 32-byte prefixes are 327 distinct out of 332 -- essentially all different,
because they are just the preceding string's bytes. There is no per-record header
here. It is one contiguous pool of debug text, and the 16-byte alignment is a
property of how the pool was laid out, which makes continuations land on
boundaries and read as separate entries.
The count of 332 "strings" was therefore inflated by the same effect: it counts
continuations as entries.
The substance is unchanged and still useful -- the pool carries the game's own
debug text, including "Camera_CollPrimTest WARNING! - grid out of bounds! (RAR)",
which is direct evidence for reading the collision payload's two leading u16 as
grid dimensions, and the wavetable.bin paths, which bear on the sound bank
framing. Only the framing was wrong.
The pool is located and counted but not parsed, and there is no gc_formats module
for it. The +0x18/+0x20 names are section names but their structure is not
established, and it is not shown that they index anything.
level_overlay was on the open list with no parser and no located structure. It
now has a header naming two sections and a string table.
Range measured: overlay.bin on levels 0, 1 and 2 -- 2751268, 2763668 and
2928424 bytes.
Result 1: the file opens with section names.
+000: 80 7c 1a 00 | dc 99 00 00 | 01 00 00 00 | 88 3b 2a 00
+010: 00 00 00 00 | 00 00 00 00 | "crap\n" | 00 00 00 00 00
+020: "system misc" 00 00 00 00 00 00 00 00 00
+030: 20 80 ff 40 | 00 60 cf 00 | 64 00 | 46 00 | 19 00 00 00
u32@+0x00 is 1735808 on all three levels, +0x08 is 1, and the bytes at +0x18 and
+0x20 spell "crap" and "system misc".
Result 2: a debug string table on 16-byte alignment. 332 such strings on level
0, 362 on level 1, 351 on level 2, first at +0x20 and last between 1.3 MB and
2.5 MB. They are the game's own debug text:
host0:r:/rc2/shock/wavetable.bin
host0:z:/i5/shock/wavetable1.bin
unloaded sound bank %#010x
Camera_CollPrimTest WARNING! - grid out of bounds! (RAR)
render setup
sky draw
pre effects
pre 2 effects
Camera_CollPrimTest WARNING! - grid out of bounds! (RAR) is direct evidence for
the collision reading. It is the game's own string, naming a grid and a
collision test, and it is why the two leading counts in the collision payload are
worth continuing to treat as grid dimensions despite not sizing the payload.
The wavetable.bin paths also confirm the sound bank is loaded from named files,
which bears on the SBlk/DAW framing.
Not done: the string table is located and counted but not parsed -- the records
preceding each string have not been identified, so there is no gc_formats module
for it yet. The +0x18/+0x20 names are section names but their structure is not
established, and it is not yet shown that they index anything.
+5932 is the table's start, not a point inside it: walking backwards, no
predecessor satisfies "ascending with a delta under 4096". The run is 17 entries,
not 19 -- the earlier 19 counted two entries past the jump.
The deltas are
204 212 216 224 224 228 228 228 224 220 216 200 96 96 100 96
then a jump of 41484 to 190508. The deltas rise and then fall -- a symmetric
profile, not a monotone one. That weakens the prefix-sum reading: a cumulative
offset table has no reason to come back down, whereas cell contents that grow and
shrink as the table walks across a level would.
17 against a 75-cell grid, and 75 - 17 = 58. No arithmetic relating the two has
been found. The earlier guess that 19 related to 3x25 did not survive
re-measurement and is not carried forward.
Still open. 17 entries may not correspond to the grid at all -- the table may
cover something else, or a subset of cells. Resolving it needs either the same
offset found in other levels at a position the leading counts predict, or a
reason for 17. Neither exists yet, and this is where the measurement trail goes
cold on collision.
Range measured: longest strictly-ascending u32 run (delta under 65536) in each
level's collision payload, levels 0 to 5.
Level 4 is the clean case. Grid 3x25, payload 2660608, and at payload +5932 a
run of 19 entries whose values are all inside the payload, starting at 146012:
deltas 204, 212, 216, 224, 224, 228, 228
The deltas grow monotonically, which is the signature of a prefix sum over
records of increasing size -- what a per-cell offset table looks like when cell
contents vary.
The others are not:
level 0 11x33 +21232 17 8193, 24575, 8192, 8192 multiples of 8192,
coincidental on
dense data
level 1 24x16 +41452 29 32771, 57339, 24575 wild, no trend
level 2 1x53 +13752 49 24, 64, 396, 104 in-region but
irregular
level 3 19x31 +667152 18 769, 257, 257, 257 values ~5.2e8, far
outside the payload
level 4 3x25 +5932 19 204, 212, 216, 224 growing, in-region
Only level 4's run has both properties a prefix sum needs: values inside the
payload, and deltas that grow.
What this does not give: level 4's run is 19 entries against 75 grid cells, so
it is not one offset per cell -- either most cells share an offset, are empty,
or the table covers something other than the grid. Nothing measured picks between
those.
Next to check: whether 19 relates to 3x25 by any arithmetic, and whether the
same offset at +5932 exists in other levels at a position given by their leading
counts. Neither has been tried.
Range measured: the first four bytes of the payload on all 19 levels that carry
a non-empty one.
Result: two u16 at payload +0 and +2, both non-zero on all 19, ranging 1..74 and
13..112, products 53 to 3256.
level 0 11 x 33 = 363 payload 903248
level 1 24 x 16 = 384 payload 2800496
level 2 1 x 53 = 53 payload 2315296
level 7 40 x 35 = 1400 payload 4049232
level 9 74 x 44 = 3256 payload 2518352
level 13 2 x 64 = 128 payload 3333520
level 26 27 x 112 = 3024 payload 652176
Reported as two counts, not as a named grid. They have the shape of collision
grid dimensions -- bounded, both non-zero, products in the hundreds to thousands
-- but nothing measured ties the product to the payload size or to any per-cell
stride. Level 2's 1 x 53 giving a 2.3 MB payload and level 26's 27 x 112 giving
652 KB shows the product does not size the payload, so either cells vary wildly
in size or these are not grid dimensions. Naming them width and depth would be a
reading, not a measurement.
After the two counts comes a run of u16 that stays below 8192 for 30 to 58
entries depending on level, always an even count, and is not ascending. That is
where the per-cell table most likely starts, and it is the next thing to
resolve.
Exposed as CollisionHeader::first and ::second. They live in the payload rather
than the header, so CollisionHeader::region fills them in and the bare
CollisionHeader::parse leaves them zero -- documented at the call site so the
difference is not mistaken for an absence.
Range measured: level 0, the 73 tie span bodies (bytes +0x40 onward), 2585856
bytes pooled. Per-position distinct counts, an autocorrelation scan against a
shuffled control, and a GS GIF tag test.
Result 1: the bodies are dense. Every one of the 16 slot positions takes 245 to
256 distinct values, byte 15 zero on 81.3% and byte 7 zero on 78.8%. The best
stride peak is 48 at +0.097, barely above 16 at +0.095 -- no period.
Result 2: no GS GIF framing at 16-byte alignment. Testing whether byte 15's low
three bits fall in 0..4, the GS tag range, gives 97.0%.
That looked like a strong hit until the control was done properly. A uniform
shuffle gives 62.6%, exactly 5/8, what a random byte predicts -- so the naive
comparison reads as a 34-point excess.
The distribution-matched null is 97.0%, identical to the observation. The whole
excess was that byte 15 is 81% zero and (0 & 7) <= 4 is always true. Recomputed
per alignment the null tracks the observation everywhere:
align 0 97.0% align 4 75.7% align 8 96.7%
align 2 75.5% align 12 89.3%
This is the control rule earning its place a second time. A uniform shuffle is
not a matched control when the data is skewed, and AGENTS.md already says so;
the near-miss was reading 62.6% as a baseline without checking what the data's
own distribution predicts. The tie bytes concentrate on 0, 1, 2, 3 and 63 --
suggestive in isolation -- and still carry no framing.
What this rules out: the tie bodies are not GS GIF packet streams at 16-byte
alignment and carry no measurable periodicity. With every slot position taking
~256 distinct values they are dense data -- fat vertices, as the roadmap expects
-- and locating the vertex layout needs a reference for this game's instanced
geometry format, not another stride scan.
Range measured: all 1864 tie spans across 27 levels, read at each class entry's
geometry offset.
Method: read +0x00, +0x0c and the three f32 at +0x10, +0x14, +0x18, and count
distinct values per position.
Result:
u32@+0x00 takes exactly one value across all 1864 spans: 0x00000100, 256
+0x0c takes 83 distinct values, so it is not constant
the three floats take 103 distinct triples built from 42 distinct values,
and those values are round -- 4, 8, 9, 10, 12, 15, 16, 18, 20, 24, 25, 30
with only 10 of the 42 exact powers of two
Two level 0 spans open:
+000 00000100 00005070 000065c0 07050612
+010 41400000 41c80000 42200000 00009460
+000 00000100 00007ca0 0000a040 0605091e
+010 41400000 41c80000 42200000 0000ddf0
This rules out the bounding-box reading. A box would take continuous values;
these come from a 42-value set of round numbers, so they are authored or
quantised quantities -- a grid, a size class, or LOD thresholds -- and nothing
measured separates those.
Correction: these floats were previously read as a bounding-box minimum, from
level 0's first two spans happening to share (12.0, 25.0, 40.0). That triple
holds on 266 of 1864 spans, not all of them, and the value set is quantised.
Not decoded: what the three floats are, what 0x100 at +0x00 counts, and what
the 83 values at +0x0c are.
data.bin is the largest per-level file -- 13.3 MB on level 0, 4.2 to 21.9 MB
across the disc -- and had been characterised only as "4-byte structured".
Range measured: the first 64 u32 of data.bin on levels 0, 1, 2 and 5, compared
against the actual byte lengths of the other extracted level files.
Method: read the first 64 u32 and look for any equal to the size of
core_index.bin, gs_ram.bin, overlay.bin, gameplay.bin, sound_bank.bin or
core_data.bin.
Result: three exact matches, at the same indices on every level.
u32 index 1 overlay.bin 2751268 2763668 2928424 2863676
u32 index 3 core_index.bin 31600 34016 33808 23536
u32 index 5 gs_ram.bin 518656 569344 513024 434176
Each is the exact byte length of the named file and each lands on the same index
on all four levels. Three coincidences at fixed positions is not a coincidence.
So data.bin is a descriptor for the level's other files, not geometry in its own
right. The roadmap's "the large one: 4.2 to 21.9 MB of geometry per level" entry
was wrong and is corrected.
The intervening indices 0, 2 and 4 hold values that are not file sizes and their
meaning is not established; they are plausibly offsets, but no cumulative-sum or
offset+size relation has been checked yet.
What follows the descriptor is still unknown, and the file is large enough that
the descriptor is unlikely to be all of it.
Range measured: all 27 levels that carry collision, reading the 0x40-byte
header each.
Method: read +0x00, +0x04 and +0x08, then check +0x08..+0x40 for zero.
Result:
u32@+0x00 is 0x40 on 27 of 27 -- the header is self-sizing
u32@+0x04 is the payload size, 20 distinct values across the disc
bytes +0x08..+0x40 zero on 22 of 27
Five levels carry a large non-zero word at +0x08:
level 7 1248048 level 12 2541616
level 9 1766832 level 13 3376592
level 14 4060640
So the header is not two fields and 56 bytes of padding: there is a third field
most levels leave at zero and five use. Whether it is a size, an offset or a
second block's length is not established, and nothing measured here separates
those readings. Exposed as CollisionHeader::field_08.
Correction: the collision header was previously described as self-describing
across its full 0x40, which is true for 22 levels and overstates the case for
the other five.
Still not decoded: the payload. It contains ascending u32 runs on levels 0 and
2 -- level 0 at +280 with deltas 12, 12, 12 then a jump, level 2 at +13868 with
deltas 204, 204, 200, 196 -- but irregular deltas fit neither a flat table nor a
simple stride, and level 1's apparent run is a byte-shift false positive.
Nothing about the payload is asserted.
20 of 149 sky sub-objects carry a non-zero word where the other 129 are zero,
so the reserved area is not uniformly padding. Reporting it rather than
asserting it, and clearing three clippy warnings from the unused counter.
Range measured: all 27 levels, each sky region from the core index's sky offset
to the next region.
Method: read the 32-bit offset table at +0x20, bounded by staying inside the
region, and parse each sub-object it names.
Result: 149 sub-objects across 27 levels. Each opens with a count, then a
bounding sphere of four floats at +0x10, then a texture offset at +0x20 and
dimensions at +0x24/+0x26:
+000: 26 00 00 00 | 00 00 00 00 00 00 00 00 00 00 00 00
+010: e0 8d b2 c1 | 9f d6 f6 3d | ef fc fb be | 1c 61 4e 41
+020: 90 99 05 00 | 1d 00 | 1d 00 | 00 00 e8 00 ...
Count 1 to 38, width 3 to 81, radius 1.95 to 14.56, and the radius is positive
on 149 of 149 -- the same sphere shape and the same reasoning as the shrub class
header's.
Correction to an earlier claim: the word at +0x0c is not the sub-object count. It
reads 5 on level 0 where the table holds 6 entries, 3 on level 1 against 4, 7 on
level 3 against 8, and 196617 on level 2 against 4. Across the disc
offsets.len() == count+1 holds on 10 of 27 levels.
The field is reported as SkyHeader::field_0c and nothing depends on it; the
offset table enumerates the sub-objects and is self-limiting. That earlier claim
came from level 0 alone, where 5 against 6 looked like a consistent off-by-one.
It is not an off-by-one -- it is a field that happens to be small on most levels
and is not a count at all.
gc_formats::sky parses the header and the sub-objects. The gated test asserts the
sphere is finite, the radius positive where positive at all, the dimensions
non-zero, and the sub-object list non-empty on every level.
Everything previously reported about five-block sub-lists and fixed-point
payloads was read out of the running game. Checking the same packets on disc
settles what the disc actually stores.
Range measured: level 0, all 29 packets named by the +0x40 tables of the 6
bounded spans, read at their span-relative offsets; 1636 whole 16-byte records
after the 16-byte header.
Neither RAM form exists on disc:
disc RAM
00 00 00 30 per exactly 1 exactly 5
sub-list packet
blocks at 0/32/64 0 of 10 11 of 11
fixed-point byte3 0 of 19 100%
== 0x80
The 29 packets split 10 sub-list / 19 fixed-point by whether they contain
00 00 00 30 at all, and not one of the ten has the block structure, and not one
of the nineteen has the fixed-point marker.
The disc body is a compact 16-byte record stream:
+000: 04 04 00 01 | 00 00 00 00 | 00 00 00 05 | 00 80 0e 6c
+010: 02 00 00 00 | 05 00 00 00 | 32 00 00 00 | 0e 00 00 00
+020: 09 00 00 00 | 00 40 3e 30 | 12 04 00 00 | 05 00 00 00
+030: 09 00 00 00 | 00 40 3e 30 | 12 04 00 00 | 21 00 00 00
Small u32 fields and a repeating 00 40 3e 30 12 04 00 00. Per position over all
1636 records the same two outliers hold: byte 7 takes 4 distinct values and byte
15 takes 2, against 101 to 153 everywhere else.
Byte 7 is per-record, not per-packet: the first record of all 29 packets has
byte 7 = 0x00. The four-value vocabulary varies within a packet body rather than
tagging the packet, which is why grouping packets by it never separated the two
kinds.
Consequence for Phase 2: the payload is encoded on disc and expanded at load, so
a disc parser can bound it and check its signature but cannot read it as either
RAM form. Decoding it means reversing the expansion, not re-describing what the
loader produced. That is the honest boundary of what is known.
The packets found by the +0x40 table have a header the whole disc agrees on.
Range measured: every packet on the disc -- 2056 of them, across the 334
successor-bounded spans -- read at each packet's span-relative offset.
bytes 0-3 04 04 00 01 2056 / 2056
bytes 4-7 zero 2056 / 2056
byte 8 zero 2056 / 2056
byte 11 0x05 2056 / 2056
size % 16 zero 2056 / 2056
Zero exceptions on any of the five.
And the loader fills the rest in. Read back out of the running game the same
records carry byte 8 as a count (0x0b to 0x29 observed), byte 11 as one of four
tag values, and bytes 12-15 as an absolute EE RAM pointer where the disc has a
span-relative offset. Bytes 8-15 are rewritten wholesale on load.
That settles what the disc header is for. It carries no count of its own, and
the count the RAM copy has is the +0x40 table's count arriving by another
route. The disc header is a signature; the RAM header is populated state.
ShrubPacketHeader::parse and Shrub::packet_headers() expose this. parse returns
None unless all the invariants hold, rather than a default, so a packet that
does not match is reported as not-understood instead of half-interpreted.
The gated test asserts all four header invariants and the size alignment on
every packet of every bounded span.
This is the first piece of the mesh packet format represented in gc-formats;
everything known about the packet payloads so far is RAM measurement, since the
disc form of bytes 8-15 is rewritten at load.
The five-block result was measured on level 0 only. Rule 3a asks for the same
measurement per partition, so it was repeated on level 1's savestate before
being written up as a property.
Range measured: savestate slot 2 (level 1), RAM 0x22a000..0x28f52f, 415024
bytes, then six of its pointer-bearing records read to count*16.
The six sampled pointers split 3/3 between the two payload kinds:
00949430 41 00 00 00 30 sub-list
009496c0 13 69 b0 bf 80 fixed-point
00957970 36 00 00 00 30 sub-list
00957bb0 10 58 7e 8d 80 fixed-point
0095b740 12 37 49 41 80 fixed-point
0095d6d0 57 00 00 00 30 sub-list
Taking all six as sub-list gives 3/6 with five blocks. Taking only the three
that actually open 00 00 00 30 gives 3/3, with the first three blocks at 0, 32,
64 in all three. The three fixed-point payloads have zero such blocks, as
expected. The earlier 3/6 was my sampling, not a counterexample -- the six
pointers were never all the same kind of object.
Combined: 11 of 11 sub-list payloads across both levels have exactly five
blocks, the first three at offsets 0, 32, 64.
The fixed-point payloads have an exact signature: byte 3 of every 4-byte group
is 0x80 in 100% of cases -- 52/52, 40/40, 48/48 over the first 512 bytes of the
three level 1 samples, and the same across all nine level 0 payloads. That
separates the two payload kinds without the record tag, which does not select
them.
Correction: level 1's record tags are not level 0's. The same scan finds 18
04 04 00 01 headers against 19, plus 6 02 01 00 01 against 10, and byte 11 is
no longer constant: 0x30 on 10, 0x00 on 5, 0x20 on 2, 0xbc on 1, where level 0
had 0x30 on all 19. "Byte 11 is 0x30, constant" was a level-0 partition result
written up as universal; across both levels it takes four values.
Fixed-point payload entropy over all nine level 0 payloads is 5.51 to 5.89 bits
per byte against 8.0 for uniform, and the mean of the s16 reading varies per
payload from -82 to -907 -- structured, and consistent with vertex positions
relative to a per-payload origin.
Not established: what the five blocks are, and how the fixed-point data decodes
to vertices. Its entropy is far below random, so it is compressed or permuted
rather than stored plainly, and the permutation table seen once in 0x009376f0
remains unexplained.
All eight sub-list payloads from level 0 were read to their exact extent
(count * 16) and compared block by block.
Range measured: savestate slot 1, the packets at 0x009376f0, 0x00939c50,
0x0093cc70, 0x00998880, 0x009c19c0, 0x009cf370, 0x009d2da0 and 0x009f5aa0,
counts 21 to 41, each truncated to exactly count*16 bytes.
Every payload contains exactly five blocks tagged 00 00 00 30, and the first
three are always at offsets 0, 32 and 64:
009376f0 22 0, 32, 64, 184, 304
00939c50 36 0, 32, 64, 268, 400
0093cc70 23 0, 32, 64, 200, 288
00998880 36 0, 32, 64, 272, 404
009c19c0 25 0, 32, 64, 196, 300
009cf370 24 0, 32, 64, 204, 316
009d2da0 41 0, 32, 64, 308, 448
009f5aa0 21 0, 32, 64, 176, 280
Blocks 4 and 5 move with the payload; the first three do not. Eight payloads,
eight different block-4/5 positions, and 0/32/64 in every case.
Per-block fields constant across all eight:
block 0 byte 4 varies 0x78..0xcd
block 1 byte 4 = 0 in 8/8, trailer 00000000 in 8/8
block 2 byte 4 varies independently, bytes 16-20 zero in 8/8
block 3 byte 4 EQUALS BLOCK 0 in 8/8
block 4 byte 4 = 0 in 8/8
Block 2's word is unrelated to block 0's in every case, so the equality is not
every block reading one shared field.
Correction: the "byte permutation table" reported in the previous commit does
not generalise. It appears in 0x009376f0 at offset 212, but a scan across all
eight payloads finds no such run outside that one. It was one payload's block
content, not a format feature. The five-block shape, the fixed 0/32/64 offsets
and the block-0/block-3 equality are what survive eight-way comparison.
Not established: what the five blocks are, why blocks 0 and 3 share a word, and
what moves blocks 4 and 5 -- their offsets are not monotone in count (21 gives
280, 36 gives 400 and 404), so something inside the earlier blocks sets them.
With the extent known from the previous entry (count * 16), a whole sub-list
payload can be read. This is the packet at 0x009376f0, count 22, 352 bytes.
Range measured: savestate slot 1, 0x009376f0..0x0093784f.
The payload is a sequence of blocks, each opening 00 00 00 30, carrying a
repeating word and a short trailer:
00 00 00 30 | cd 00 00 00 cd 00 00 00 cd 00 00 00 cd 00 00 00
01 00 00 05 | 22 c1 01 6e | 00 00 00 00
00 00 00 30 | 00 00 00 45 00 00 00 45 1d 00 00 00 1d 00 00 00 1f
81 01 6d 00 | 00 00 00 00 | 00 00 a4 00
00 00 00 30 | bc c0 03 00 23 17 03 00 e5 dd 00 00 00 00 00 00
02 01 00 01 | c1 80 01 69 | 00 00 00 00
The repeating word differs per block (0xcd, 0x45, 0x03) and the trailer's
second word varies in its high bytes (22 c1 01 6e, 81 01 6d 00, c1 80 01 69).
Every block ends 6e, 69 or 6d at byte 23 -- the same 6e the 02 01 00 01 records
carried at bytes 8-9.
The payload contains a byte permutation table at 0x009377c4:
00 01 02 03 3e 3f 53 54 40 41 42 43 4e 4f 50 51 04 05 06 07 08 ...
Ascending byte values with a handful (3e 3f, 53 54, 40 41 42 43, 4e 4f 50 51)
lifted out of order ahead of the sequential run. Read as a table it maps each
of 0x00..0x39 to a slot -- the shape of a code-length or symbol-ordering table,
and the first thing in this format that is self-evidently a table rather than
data to be interpreted.
Not said: whether it is a Huffman-style code, an index reorder for the vertex
data that follows, or a palette permutation. The sequence continues to 39 then
23 c1 01 6e, after which the next block opens.
Corroboration: the fixed-point payload 0x00937850 begins exactly where this one
ends. The 22-record identity made the boundary exact and the next packet starts
on a payload boundary with nothing in between, so the two payload kinds sit
adjacent in RAM -- consistent with them being two records of the same blob.
The previous entry left one test: compare the count at byte 8 against the
payload extent. Pointer spacing gives the extent, since the packets lie in
ascending order in EE RAM.
Range measured: savestate slot 1, the 17 packets in RAM 0x22a000..0x24037f,
each with its record address, byte-8 count and pointer, sorted by pointer.
Result: eight packets sit exactly count*16 bytes after their predecessor -- a
ratio of 16.00 in every one, no rounding.
0x22a6e8 count 22 ptr 009376f0 gap 352 16.00
0x22a728 count 36 ptr 00939c50 gap 576 16.00
0x22a768 count 23 ptr 0093cc70 gap 368 16.00
0x22a7a8 count 36 ptr 00998880 gap 576 16.00
0x22a7e8 count 25 ptr 009c19c0 gap 400 16.00
0x22a828 count 24 ptr 009cf370 gap 384 16.00
0x22a868 count 41 ptr 009d2da0 gap 656 16.00
0x22a8a8 count 21 ptr 009f5aa0 gap 336 16.00
Eight exact hits at a ratio that must be an integer is not luck. The count at
byte 8 is the record count, and these payloads are 16 bytes per record.
These are exactly the sub-list payloads -- 009376f0, 00939c50, 0093cc70,
00998880 and 009d2da0 were read directly in earlier passes and all open
00 00 00 30 followed by a repeated word.
The other nine have ratios of 168, 419, 559, 31288, 7985, 3458, 764 and 6452
-- no shared factor, far too large to be a record count times anything. Either
their payload is not followed immediately by the next packet in pointer order,
or they are not 16 bytes per record. Ruled out either way: they do not follow
the same rule.
This resolves byte 8, measured two entries ago as "a count, 0x0b to 0x29" and
now shown to size a payload exactly in eight independent cases within one
level. Whether it also sizes the fixed-point payloads is the open half.
Still open: why the fixed-point packets differ, and what the 16 bytes per
sub-list record are -- the repeated word (0xcd, 0xc0, 0xc8, 0x78) differs per
packet, so the records are not identical to each other.
Range measured: savestate slot 1, RAM 0x22a000..0x24037f.
Result: a second four-byte record magic appears at comparable density beside
04 04 00 01 -- 02 01 00 01, ten occurrences against nineteen. The two interleave:
every 02 01 00 01 sits between two 04 04 00 01 records.
0x22a6b8 04 04 00 01 13 00 00 30 70 6a 93 00
0x22a6c8 02 01 00 01 19 c0 4c 6e 00 00 10 00 00 00 00 00
0x22a6e8 04 04 00 01 16 00 00 30 f0 76 93 00
0x22a6f8 04 04 00 01 16 00 00 30 50 78 93 00
0x22a708 02 01 00 01 1e c0 58 6e 00 00 10 00 00 00 00 00
0x22a728 04 04 00 01 24 00 00 30 50 9c 93 00
The 02 01 00 01 records are 16 bytes like the others but carry no pointer --
bytes 12-15 are zero in all ten. Their fields are a count at 4-5 (0x14, 0x19,
0x1e), a constant 0xc0 at 6, a varying byte at 7 (0x30, 0x4c, 0x54, 0x58),
0x6e at 8-9, and 10 00 at 10-11.
The 0x6e at bytes 8-9 matches the ".. c1 01 6e" the sub-list payloads carry at
offset 20-23. That is a second correspondence, and unlike the one retracted in
the previous commit it is a shared value rather than a shared position.
Not settled: this still does not say what discriminates the two payload kinds
among the 04 04 00 01 records, since those share the same magic and the same
0x30 at byte 11 and differ only in the count at byte 8. The 02 01 00 01 records
sit between them rather than replacing them, so they are a companion structure
rather than the missing discriminator.
Next: compare the count at byte 8 against the payload's own length.
The previous commit mapped the disc tag vocabulary onto the three payload
classes. A wider scan says that is wrong.
Range measured: savestate slot 1, RAM 0x22a000..0x24037f -- 91008 bytes, 20
times the span scanned before.
Result: 19 packet headers, and all 19 carry byte 11 = 0x30. There are no 0x6d
or 0x80 tags anywhere in this range. Reading six payloads:
rec 0022a6b8 tag 30 -> 8d 8c 80 80 89 87 7c 80 ... fixed point
rec 0022a6e8 tag 30 -> 00 00 00 30 cd 00 00 00 ... sub-list
rec 0022a6f8 tag 30 -> 92 90 84 80 94 94 86 80 ... fixed point
rec 0022a728 tag 30 -> 00 00 00 30 c0 00 00 00 ... sub-list
rec 0022a738 tag 30 -> 4c 4c 4b 80 4c 4c 4b 80 ... fixed point
rec 0022a768 tag 30 -> 00 00 00 30 78 00 00 00 ... sub-list
Records of identical tag alternate between two payload kinds. The tag does not
select the payload.
What went wrong: the previous entry took the 0x30 and 0x6d that head the
payloads to be the disc-side tag vocabulary, and matched them to payload classes
by that correspondence. The disc vocabulary at record byte 7 (0x00, 0x30, 0x6d,
0x80) and RAM record byte 11 are different fields that merely overlap in two
values. Three payloads each made the coincidence look like a rule.
What is actually established about the RAM header:
0-3 04 04 00 01, constant
4-7 zero, constant
8 a count, 0x0b to 0x29 across the 19 records
9-10 zero, constant
11 0x30, constant on all 19 in this range
12-15 an EE RAM pointer
Byte 8 is the finding that survives: a small count varying per record, and the
one field here doing arithmetic rather than tagging.
Still open: what 0x30 at byte 11 means, what discriminates payload kinds given
the tag does not, and 0x80, which has no sample in 91 KB of RAM.
Dereferenced all nine pointers in the level 0 table that carry the 0x30 tag at
byte 11 and an EE pointer at 12..15, reading 32 bytes each. They fall into
exactly three classes, three payloads each, with no overlap.
A, Q16.15 fixed-point vertex data (3 of 9): every high byte 0x80 or 0x7c,
repeating on an 8-byte boundary, values differing per payload while the
4e 4e 4c 80 tail is shared by two.
00936a70 8d 8c 80 80 89 87 7c 80 ... 4e 4e 4c 80
0093cde0 8a 8a 7f 80 92 90 84 80 ... 4e 4e 4c 80
009cf4f0 6e 6f 68 80 6e 6f 68 80 ... 54 54 51 80
B, a 0x30 sub-list with a repeated word (3 of 9), structurally identical
apart from that word (0xcd, 0xc0, 0xc8), all carrying 01 00 00 05 at offset 16
and a ".. c1 01 6e" at 20.
009376f0 00 00 00 30 cd 00 00 00 cd 00 00 00 ...
00998880 00 00 00 30 c0 00 00 00 c0 00 00 00 ...
009d2da0 00 00 00 30 c8 00 00 00 c8 00 00 00 ...
C, an index/attribute list (3 of 9), both opening 00 c0 05 6d, then a count,
then a run of small ascending u16.
00939700 00 c0 05 6d 3e 00 00 00 01 00 00 00 ...
009c1540 00 c0 05 6d 36 00 00 00 01 00 00 00 ...
So the tag vocabulary measured on disc at record byte 7 -- 0x00, 0x30, 0x6d,
0x80 -- selects the payload:
0x30 class B, the 00 00 00 30 sub-list
0x6d class C, the 00 c0 05 6d attribute list
0x00 class A, the fixed-point vertex data
0x30 heads the class B payloads and 0x6d heads the class C payloads. That was
flagged as a single coincidence two commits back; three payloads each makes it
a rule.
0x80 has no sample among these nine and its meaning stays open.
Correction: the record spacing in this table is not uniformly 0x30. The find
output shows 0x30, 0x10, 0x30 between consecutive packet headers, and only 9
of the 5632 bytes scanned carry both the 0x30 tag and an EE pointer. The
earlier "0x30 spacing" was inferred from a truncated hit list.
The header pointers found in the previous commit are live, so they can be
dereferenced. Two records 0x30 apart in the level 0 table at 0x22a6b8 were
followed.
Range measured: savestate slot 1, reading 0x00936a70 (from the record at
0x22a6b8) and 0x009376f0 (from the record at 0x22a6e8), 64 bytes each.
The two payloads have nothing in common.
0x00936a70 is fixed-point vertex data:
8d 8c 80 80 89 87 7c 80 8d 8c 80 80 89 87 7c 80
4e 4e 4c 80 4e 4e 4c 80 4e 4e 4c 80 4e 4e 4c 80 repeating
Every high byte is 0x80 or 0x7c and the pattern repeats on an 8-byte boundary.
As s16 these are large negatives in a narrow band around -32500, consistent
with Q16.15 fixed point rather than f32, and with a stride of 4 s16 per vertex.
0x009376f0 is a tagged sub-list, not vertex data:
00 00 00 30 cd 00 00 00 cd 00 00 00 cd 00 00 00
cd 00 00 00 01 00 00 05 22 c1 01 6e 00 00 00 00
Structurally different: small integers, a repeating 0xcd, 0x30 tags in the high
byte. Nothing here is position data.
This supports the type reading of the header: the table is a tagged collection
where the tag selects the payload kind, not a uniform array. It also rules out
the alternative that the 16-byte tiling was one uniform record format with
varying fields -- two adjacent records with unrelated payloads is not that.
Still open: the mapping from tag value to payload kind. The 0x30 appearing in
record bytes 8-11 in RAM also heads the second payload, which is suggestive
but is one coincidence so far and is deliberately not written up as a link.
Answering the "what is the tag byte" question needed an anchor. Searching live
RAM from scratch was the wrong way to get one: 04 04 00 01 matches unrelated
data at 0x1ac988 and there is no way to tell a real hit from a coincidental
one. The savestates are the anchor -- slot 1 is level 0, and searching one
gives candidates that can be checked rather than guessed at.
Range measured: savestate slot 1,
~/.config/PCSX2/sstates/SCUS-97268 (38996035).01.p2s, records at 0x22a6b8
onward, read with gc-trace read --state.
Result: 40+ records beginning 04 04 00 01 00 00 00 00 at 0x30 spacing -- the
packet framing identified on disc -- with the second half of each header
rewritten:
disc: 04 04 00 01 | 00 00 00 00 | 00 00 00 05 | 00 80 0e 6c
RAM: 04 04 00 01 | 00 00 00 00 | 13 00 00 30 | 70 6a 93 00
Bytes 0-7 unchanged; bytes 8-15 are not. On disc bytes 12-13 are the constant
00 80 and byte 15 the constant 6c; in RAM bytes 12-15 hold a pointer --
0x00936a70, 0x009376f0, 0x00937850, 0x00939c50 in consecutive records, all
inside EE RAM and ascending across the table as a walk of consecutive objects
would be. Byte 11 becomes 0x30 in every record seen, against a constant 0x05 on
disc, and bytes 8-9 hold a varying small value (0x13, 0x16, 0x24).
So the packet table at +0x40 is what the loader walks: span-relative on disc,
absolute in RAM. This was inferred from one class before; these 40+ records
show it directly.
Consequence for the tag bytes: the vocabularies at record bytes 7 and 15 are
real, but byte 15 sits in a field the loader overwrites with a pointer. Its
disc values are read as data by something rather than stored as a tag.
Not settled: whether byte 7's vocabulary is also rewritten, and whether 0x6d
marks payload the loader expands. That needs these records compared one by one
against their disc spans, which is cheap now that the anchor exists.
Range measured: level 0, the 1636 sixteen-byte records following the header
across all 29 packets. Per-byte distinct counts, not pooled.
Method: count distinct values and zero frequency at each of the 16 positions,
then tabulate the joint distribution of the two low-entropy positions.
Result: fourteen positions are uniformly varied, 101 to 153 distinct values,
zero in 13% to 58% of records. Two are not:
byte 7 4 distinct: 0 (1432), 0x30 (135), 0x6d (40), 0x80 (29)
byte 15 2 distinct: 0 (1618), 0x6d (18)
Both sit in the last byte of an 8-byte half, eight bytes apart, and both take
values from a small closed vocabulary rather than the full byte range. The
joint distribution shows no record where both are non-zero, and 0x6d appears in
either position but never both:
(0,0) 1414 (0,0x6d) 18 (0x30,0) 135 (0x6d,0) 40 (0x80,0) 29
Each 8-byte half carries a tag or subtype byte in its last position, from a
small closed set. The shared 0x6d is what suggests one vocabulary used twice
rather than two unrelated fields.
This answers the question the previous commit left open. There is no per-record
length field: the best candidate offsets hold the value 16 on at most 13% of
records, which is the chance rate for small integers, not a self-describing
record. The records are therefore fixed 16 bytes, and the tiling is not a
coincidence of variable-length records that happen to be multiples of 16.
Still open: what the four and two tag values mean, and what the 14 varied bytes
encode. The tag vocabulary is small enough to enumerate against known PS2
primitive lists, which is the reference to bring in next.
The packet table decode left the packet contents open, with a warning that a
VIF reading had already been retracted once and needed a control rather than a
pattern match. This is that measurement.
Range measured: level 0, the 29 packets of the 6 successor-bounded shrub spans,
26640 bytes. Per-packet; nothing pooled.
Method: autocorrelation within the largest packet against a distribution
matched shuffled control, then per-byte distinct counts across all 29 packets.
Result 1: all 29 packet sizes are exact multiples of 16, and the internal stride
scan peaks at 16 (+0.283) with 32, 48, 64, 80 and 128 all elevated behind it --
the signature of a record array, not a coincidental spacing.
Result 2: record 0 is a 16-byte header. Constant across all 29 packets at bytes
0-3 (04 04 00 01), 4-11 (zero), 12-13 (00 80) and 15 (6c). Bytes 14 and 16-23
vary. The 04 04 00 01 prefix looks like a magic or format version.
Result 3: header byte 14 is echoed in the body. It takes values 06, 07, 0e and
11 across packets, and the same value appears at offset 12 of record 1 in every
packet sampled. A real repeated field, not two small values coinciding.
Not established: no header field matches the record count. The packet holds
1056/16 = 66 records and none of the candidate u32 or u16 header offsets at 8,
10, 12, 14, 16, 18 or 20 equals 66, 65 or 67 on more than half the packets. So
the count is either computed from a size field elsewhere in the span, or the
records are variable length and merely 16-byte aligned.
That distinction is unresolved and matters. Fixed 16-byte records make the
payload a flat table; variable records make the 16-byte alignment a coincidence
of every record being a multiple of 16 long. The stride scan cannot separate
the two, because both peak at 16.
The retracted VIF reading is not revived: the tiling is uniform across all 29
packets with no command-shaped tail, and a VIF packet would show a changing
command byte last rather than the constant 6c at header byte 15.
Next: read the record count from the packet table in the span rather than from
the packet header, which is what would separate the two readings.
The open question from the shrub decode was where the mesh lives: the blob
looked like header, palette and optional billboard, with the 333 classes
without a billboard having nothing to draw. Answering it took the running
game, not the disc.
Diffing a loaded shrub class against its disc bytes in the live PCSX2 (level
1, class 1) shows what the loader rewrites: a table at +0x40 of (offset,
size) pairs, whose offsets are span-relative on disc and absolute in RAM with
the sizes unchanged -- all 21 entries matching offset for offset and size for
size -- plus a relocated palette pointer at +0x2c and a pointer at +0x18,
zero on disc, that asserts the parser enforces.
So the blob is header, packet table, packets, [billboard], palette: the mesh
was here the whole time. Why it was missed is the part worth recording --
data_offset points at the END of the span, so reading the body as
data_offset..span_end accounts for the last few hundred bytes and leaves the
20 KB in front of it unexamined. A correct bound and a body small enough to
look plausible are together enough to hide a region.
Range measured: every shrub span on the disc, 358 over 24 levels, each bounded
by its successor class or the located chunk container.
Method: read count at +0x28, then the entries at +0x40, and require them to
tile -- the first packet at align16(0x40 + count*8), every entry exactly size
past the previous, and the last ending at the palette, or 0x40 short of it on
the 25 spans that carry a billboard.
Result: 358 of 358 tile exactly, holding 2,239 packets. That is what pins the
field meanings: count is the packet count (1..55 on the disc), data_offset is
the palette's offset, and the earlier "the body is not sized by count" note
was true for a reason nobody had understood -- the count sizes the packets.
Shrub::packets() exposes them and returns None rather than a partial list
when a table does not tile. Still open: the packet contents (a VIF reading was
already retracted once, so this needs a control rather than a pattern match),
and the 0x20-byte records the loader points at from +0x18.
PCSX2 was set up and working but existed only in this machine's filesystem
and in people's heads, so every comparison looked like a one-off observation
rather than something re-runnable. The setup section in
docs/plan/pcsx2-verification.md now records where the emulator, the BIOS, the
disc, the savestates and the PINE socket are, how a state is captured, and
which command the disc integration tests need -- and AGENTS.md and README.md
point at it from their orientation sections. One operational detail worth
spelling out: the emulator's directory is lowercase (pcsx2), which is not
what people type.
Savestate slot 2 is the second planet, so the comparison runs on a level with
a different geometry extent, light bank and runtime-set population:
level1: 401/401 headers, 401/401 pointers, bank byte-identical
vertices compared: 26470 (105880 colour bytes) -- equal: 100.00%
The live emulator agrees with the savestate byte for byte. Level 1 is a
different planet, so the two fixes in the previous commit are not level-0
coincidences: the normal's z sign and the unlit-record base colour both hold
across 268,280 colour bytes on two levels.
The harness that compares the port against the game's own memory existed but
had never been compiled: tools/trace did not build (27 errors, all in the
argument plumbing), and the tfrag_light.rs light-bank correction it motivated
was unverified. This finishes it and runs it, and the run finds two real bugs.
Result, on level 0 against three inputs -- the live emulator over PINE and two
PCSX2 savestates from the same level, agreeing byte for byte:
headers matching the disc on invariant bytes: 833/833
data pointers relocated as table + disc offset: 833/833
light bank in RAM at 0x50ab20, byte-identical to the disc's
vertices compared: 40600 (162400 colour bytes)
equal: 162400 (100.00%)
The locator finds the level's tfrags block in RAM by tfrag 0's bounding
sphere and accepts a candidate only when every header matches the disc on the
bytes the game leaves alone and every data pointer is the relocated form, so
"the block was found" is itself a verification rather than an assumption. The
bank is searched in RAM by its exact section bytes, which is what settles its
framing: offset at +0x04, an i32 set count, twelve bytes of pad, then the sets.
The previous reading (offset at +0x00, count at +0x04) was wrong.
Two bugs the comparison found, both of which produced plausible output:
1. vertex_normal_bits negated x and y and left z positive, while
NormalTable::normal negates all three. dA came out -0.707 where the game
had +0.707, so max(dA, dA*wA) took the wrong branch and every vertex lit
near its base colour: dark, but not wrong-dark. 95.88% agreement with the
bug, 100.00% without.
2. A record whose select names a set the bank lacks drew transparent black.
The game writes the record's base colour; 3,130 of level 0's 40,600
vertices do this (select 0x000f, set 15 -- a runtime slot the level's
directional bank does not have). The vertex is unlit, not invisible.
Both are now regression tests in the engine. The comparison prints the pass's
per-vertex terms (light_vertex_terms) for the first mismatches, because a
colour mismatch alone cannot be attributed and the comparison is worthless
without attribution -- that is how the second bug was identified from a
three-line read rather than guessed at.
Docs: docs/plan/pcsx2-verification.md records the harness, the result, the
two bugs and what to compare next; roadmap and level-assets.md stop carrying
lighting magnitudes as an open item. Findings reach tests only as numbers --
no test reads a savestate, a dump or a live emulator.
The shrub body's payload encoding was the open question after the span-bound
work, and it answered itself once the R&C1 reference was in view: R&C1's shrub
class header keeps the field offsets R&C2 shows (o_class at 0x24, a count at
0x28, the data offset at 0x2c), which says which bytes to look at, and the disc
says what they hold.
Range measured: all 358 shrub class spans over the 24 levels that have them,
each bounded by its successor class or the located chunk container. Nothing
outside the spans is read.
Result. The body is a palette of 24 unit-length s16 normals -- s16 x, y, z and
a zero pad word per 8-byte record -- followed by all-zero padding records.
Three independent measurements, each of which fails for any other reading:
- all 8,592 live entries have squared length within 65,535 of 32767^2, and
the bound is tight (the largest observed deviation is exactly 65,535). That
is 32768^2 - 32767^2, which is what a unit vector scaled by 32768 produces
when an axis-aligned normal quantises to -32768 and the positive axis clamps
to 32767; components run the full s16 range and 1,765 entries are
axis-aligned. A random s16 triple is off by about a billion, so one entry in
fifteen thousand would pass by chance: all 8,592 pass.
- the padding records are always at index 24 or later and always all zero,
never a live entry. That is what separates the palette from geometry, and it
is the observation the three earlier stride-fitting attempts were missing.
- 17 classes carry a palette of 24 identical (0, 0, 1) entries. A position or
index table cannot produce that; a normal palette for a flat-facing class
must.
The header decodes as: sphere at +0x00 (centre + radius, radius positive on
all 358 spans, 1.72..41.57), size at +0x10, mode_bits and scratch at +0x14,
a zero word at +0x18, the billboard offset at +0x1c, and scale at +0x20. The
parser asserts the two invariants that hold without exception (scratch and the
reserved word are zero on all 358) plus a positive radius and scale, and
reports mode_bits rather than asserting it: it is 4 on 16 of the 358 spans.
The billboard offset is the strongest self-identification after o_class
because it is confirmed from a second file: it is non-zero on exactly the 25
classes whose core-index entry carries billboard texture fields, pairing
correctly both ways. At the offset sits R&C1's ShrubBillboard shape, register
block and all -- fade_distance, then TEX1_1 (K -233..-168, MMIN 4, address
0x14), TEX0_1 (index 1..4, address 0x06) and MIPTBP1_1 (address 0x34), all
three register addresses checked in the disc test.
Two corrections this produces, both recorded in docs/plan/level-assets.md
because the old readings were confidently wrong rather than merely
unverified:
- "reserved is 0 or a denormal" had the two u16 halves the wrong way round:
the 16 non-zero spans are mode_bits = 4 with scratch = 0. As an f32 the word
is still a denormal, so the PS2 reads zero either way -- but the field is a
flag word, not a magnitude, which is exactly the FPU point worth making.
- "bounds" was tested as a min.xyz/max.xyz box, which holds on only 146 of
358 spans, and the field was left unnamed as a result. It is a bounding
sphere: the fourth float is positive on all 358 spans and the centres
cluster near the class origin. The 146/358 figure was a property of the
wrong question.
Still open: the three floats in the billboard record (they repeat
byte-identically across levels -- 16 distinct triples among 25 records, one on
four classes across levels 1, 4, 12 and 21 -- so they are shared art data
rather than dimensions), size, mode_bits, count, and where the shrub mesh
geometry lives. R&C1 puts the mesh packets in this blob; R&C2's blob holds
only the palette, so the 333 classes without a billboard must draw meshes from
somewhere else.
The shrub region's end was the last unlocated bound in core_data. It was not
in the core index, and unlike moby->tie it had no successor boundary to chain
against, so the last shrub class in every level ran out to the lump's end and
the aggregates over it (the "3.5M records / 28MB of shrub body") were
artifacts of that oversized span.
Range measured: for each level, from the last shrub class's geometry offset to
the constant trailer. Nothing outside that range is read.
Method: scan forward from the last shrub class for the pair (a) the constant
two bytes f6 7c sitting immediately in front of (b) a chunk signature -- a
16-byte header ending 0xff, ten zero bytes, then four or more 0xf0-tagged
offsets. The two halves are each weak alone, and the halves are measured
separately: the marker pair occurs 25 to 103 times in a 5 to 17 MB lump, so
matching it alone picks dozens of wrong offsets; the chunk shape matches
random data often enough to need four tagged offsets before it is believed.
Result: on all 27 levels the marker sits exactly two bytes before the first
chunk header, and the two parts of the signature are never found apart. No
chunk signature occurs inside any last shrub body, on any level. The three
levels without shrub (10, 25, 26) carry the same marker before the container
that closes tie instead, which is what makes the locator a property of the
region rather than a shrub rule.
Consequences:
- All 358 shrub class bodies are bounded and are whole 8-byte records. That
was not assertable while the last class in each level ran to the lump's end,
and it is the check the boundary earns its keep for.
- 9,600 records in total, against the 8,938 previously measurable in
successor-bounded spans; 662 of them sit in the 24 last-class bodies. The
shrub region totals 2,033,776 bytes across the 24 levels that have one.
- The chunk container is 25,775,850 bytes across 27 levels (1,236 chunks,
22,562 records). The container measurements were previously taken over "the
900,000 bytes before the trailer", a window that did not align to its real
extent: on level 0 it is 690,302 bytes, and 27 of 1,123 chunks were
truncated at that invented edge. chunk_containers_resolve_on_every_level now
reads the region from core_layout instead of measuring backwards from the
end of the lump.
core_layout takes core_data now: the container's start is the one bound that
cannot be derived from the index. It gains the chunks region, so every byte of
level-specific core_data belongs to a named region. Where the signature is not
found the end stays None -- the region is left visible rather than clamped to
the tail.
Still open: the marker's meaning (constant across 27 levels, at exactly this
position, unnamed) and the shrub body payload's encoding.
2026-10-11 06:20:48 +02:00
46 changed files with 24879 additions and 406 deletions
@@ -7,6 +7,7 @@ code, and those notices travel with anything derived from it.
| Project | Role | Licence | Code copied into this repo? |
| Project | Role | Licence | Code copied into this repo? |
|---|---|---|---|
|---|---|---|---|
| [OpenRAC/rac2-gc-decomp](https://github.com/OpenRAC/rac2-gc-decomp) | Byte-exact reconstruction research for the same disc (SHA-256 verified identical) -- supplied `$gp`, the `.lit` load base, the seven-section program layout, the `lvl.vtbl`/`lvl.camvtbl`/`lvl.sndvtbl` dispatch record shapes, and the 256-byte `MobyInstance` field map transcribed into `gc_formats::moby` | GPL-3.0-or-later | Not yet; vendored read-only at `vendor/rac2-gc-decomp`. Note: its *function addresses* did not transfer (they land in NOBITS memory); its *struct layouts* are used as layout claims, not verified against disc bytes |
| [ReRAC](https://github.com/re-rac/rerac) | Native Rust/Bevy port of Ratchet & Clank (2002) — the architecture this follows | ISC | Not yet; derived from, attribute when used |
| [ReRAC](https://github.com/re-rac/rerac) | Native Rust/Bevy port of Ratchet & Clank (2002) — the architecture this follows | ISC | Not yet; derived from, attribute when used |
| [OpenRAC / rac2-gc-decomp](https://github.com/OpenRAC/rac2-gc-decomp) | Matching decompilation research for this game | MIT | No — consulted for documentation |
| [OpenRAC / rac2-gc-decomp](https://github.com/OpenRAC/rac2-gc-decomp) | Matching decompilation research for this game | MIT | No — consulted for documentation |
| [rac2-gc-decomp community reference](https://github.com/OpenRAC/rac2-gc-decomp/blob/RAC2/docs/COMMUNITY-ENGINE-REFERENCE.md) | Engine architecture notes, moby struct layout | MIT | Documentation only |
| [rac2-gc-decomp community reference](https://github.com/OpenRAC/rac2-gc-decomp/blob/RAC2/docs/COMMUNITY-ENGINE-REFERENCE.md) | Engine architecture notes, moby struct layout | MIT | Documentation only |
@@ -117,4 +118,23 @@ Rust dependencies keep their own licences; see `Cargo.lock` and each crate's
1. Record it in this file, with its licence and what you took.
1. Record it in this file, with its licence and what you took.
2. Keep the upstream notice intact in any file containing derived code.
2. Keep the upstream notice intact in any file containing derived code.
3. If it would change this project's licensing terms, stop and say so first.
3. If it would change this project's licensing terms, stop and say so first.
## Update 2026-10-15: decomp advanced to 8a174cb
Vendored checkout moved from `aca4f59` to **`8a174cb`**, 18 commits ahead
(`origin/master`). All 18 are `src/game` function-matching work; **no header,
doc, config or `config/levels/functions.tsv` changes**, so the function extents and
level catalogue this project uses are unaffected.
What is new and usable: **14 of the newly matched functions fall inside the
overlay `.text`** (`0x2a3b80`-`0x447900`), so they are decomp addresses verified
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
ReaperOfVeriod
