9f4f4acecbb78b056107918600d6f65225e8fc44
180
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 ReaperOfVeriod
|
9f4f4acecb |
The collision assert string is never referenced: the last route is closed
Followed the route I recorded as remaining -- find how the code addresses the
assert string at 0x001a8998 in .lit. It does not address it.
Level 0's program is 2,751,268 bytes. Searching it for both ways an address can
be materialised:
address low half 0x8998 0 occurrences, word-aligned or not
$gp-relative offset 0x99a8 0 occurrences, word-aligned or not
exact 32-bit 0x001a8998 0 occurrences
Validated by a positive control in the same run: the .lit load base's own low
half, 0x7c80, occurs exactly ONCE, word-aligned. A search that finds that will
find any real reference, so the zeros are real.
So no instruction and no data word in the retail level program references the
collision assert string -- not by lui/addiu, not by a literal pointer, not by a
table entry. The text is present in .lit at overlay+0xd18, and the same text also
appears in the boot binary at file offset 0xA9558, but nothing points at either.
The reading that fits: the assertion is compiled out in retail and the string is
dead data. Debug string pools routinely keep messages whose only user was an
#if-ed-out assert. The decomp's table lists this name because it recovered it
FROM THE STRING, not because it found code calling it -- which is worth noting,
because a name in a symbol table looks like evidence of a caller and is not one.
That closes the last route this file records for collision. Getting the cell
layout from the disc would need a build with assertions enabled, a debug listing,
or an independent description of R&C2's collision cell. None of those is on this
disc, and saying so is more useful than another scan.
Summary of collision, all of it now checked: the payload is verbatim in RAM, it
does not walk as fixed-stride cells, the disc-side segmentation is exhausted, the
string that would have led to the routine is unreferenced, and both of the
decomp's addresses are dead ends.
|
||
|
ReaperOfVeriod
|
66a96111d7 |
Kill the 48-byte loop lead: the function takes no pointer arguments
Finished the task I named last commit rather than leaving it as a lead.
Camera_CollPrimTest is declared Camera_CollPrimTest(void *prim, void *grid) --
two pointer arguments, which in MIPS arrive in $a0 and $a1.
The function's prologue is at 0x002db078, matching the +0x70 teardown at
0x002db354. The two instructions after it:
0x002db078: addiu $sp, $sp, -0x70
0x002db07c: lw $v1, -0x3f4c($v0) <- $v0, not an incoming argument
0x002db080: addiu $a0, $zero, 0x6f <- $a0 set to a constant
$v0 is caller-saved scratch, not an argument register, and $a0 is overwritten
with the constant 111 on the next instruction. No void* reaches this function, so
it is not Camera_CollPrimTest, whatever the assert table labels it.
It reads four consecutive words from boot memory -- 0x1a7348 through 0x1a7354 --
all below the level program base at 0x001A7C80, which is the boot's resident
region and identical in every program. A different data source from the collision
payload, and the 48-byte stride loop belongs to whatever that structure is.
So the lead is dead on evidence rather than on doubt, and both routes the decomp
appeared to offer are now closed:
Camera_CollPrimTest @ 0x001E7A50 lands in NOBITS, all zero
GridOutOfBounds @ 0x002db350 a function with no pointer arguments
Neither address reaches the collision parser. Worth noting how cheap the check was
once the tool worked: three instructions past a prologue is enough to disqualify a
function whose name asserts a signature. The expensive part was not the analysis,
it was the disassembler that stopped after five words and made everything look
empty.
The route that remains is the one already recorded -- find how the code addresses
the assert STRING at 0x001a8998 in .lit, which is not a $gp-relative load and is
not a symbol address in either table. Note that the string and the code are five
megabytes apart in different sections, so whatever reference exists is not a
simple local reference, and that is the thing to look for.
|
||
|
ReaperOfVeriod
|
2f5979e652 |
Update the decomp; read .text, and fix the disassembler that made it unreadable
Vendored rac2-gc-decomp is 13 commits behind, 12,375 lines added, almost all src/.
It names no collision data structure, so the format is still not handed over. Two
things are usable, and one precondition had to be fixed first.
THE PRECONDITION: the disassembler was stopping after five instructions.
tools/overlay-anchors called disasm_all across the whole window, which halts at
the first word capstone will not decode. In code regions that is almost
immediately, so every window came back with a handful of instructions and I had
concluded there was nothing there. It now walks four bytes at a time, prints
.word for anything undecodable, and reports how many failed. That is the whole
reason the earlier attempts produced nothing -- not that the code was absent.
WHAT THE DECOMP'S ASSERT TABLE ACTUALLY GIVES
assert-message-names.tsv maps each assert name to an address. For level 0,
CameraCollPrimTestGridOutOfBounds is 0x002db350. That address is not the assert
call and not the string. It is:
0x002db350: jr $ra
0x002db354: addiu $sp, $sp, 0x70
an epilogue. The address lands on the last group of the function before it, and
0x002db35c is a fresh prologue. It is also in .text (0x002a3b80..0x00447900),
five megabytes from the .lit string at 0x001a8998 -- which explains the "$gp search
returns zero" result from the other side. The string and the code are in different
sections, five megabytes apart.
WHAT IS ACTUALLY THERE
Immediately before the epilogue:
0x002db2dc: lwc1 $f20, 0xc($s0)
0x002db2ec: add.s $f20, $f20, $f21
0x002db2f0: c.olt.s $f0, $f20
0x002db304: jal 0x2db4b8 <- the assert call
0x002db314: lw $v0, ($s1) <- a count
0x002db31c: slt $v0, $s2, $v0
0x002db320: bnez $v0, 0x2db2cc <- loop back
0x002db324: addiu $s0, $s0, 0x30 <- stride 48 bytes
A loop over 48-byte records, reading a float at +12 and accumulating, bounded by a
count, with f22 set to 1.0f just above it -- a minimum-height clamp.
NOT CLAIMED to be the collision grid. It is a strided float loop beside an assert
the decomp labels CameraCollPrimTestGridOutOfBounds. A plausible address near a
plausible assert is a lead, not an identification, and that is the exact gap this
log keeps tripping over. Confirming it means tracing the function's arguments back
to the collision payload and showing the grid's leading u16s -- level 0 is 11 x 33
-- drive the count at ($s1). Not done, and not asserted.
The undecodable words are VU1 microcode (COP2 opcodes), which is expected: the
function interleaves VIF-fed VU1 work with FPU arithmetic, like tfrag's.
Tools build, clippy clean, 260 tests.
|
||
|
ReaperOfVeriod
|
508a03fa4a |
The decomp names both collision functions; their addresses are not ours
rac2-gc-decomp names the two collision entry points:
Camera_CollPrimTest 0x001E7A50 camera-to-world boundary check
MB_CheckCollPill 0x001E8890 pill/capsule check for moby instances
That looked like the collision unblock. The $gp string search has failed twice
now, and a named function at a known address is exactly what it failed to
produce.
They are not usable. Checked against levels 0, 5, 13 and 26 with
parse_overlay_sections:
Camera_CollPrimTest @ 0x1e7a50 -> section 1, kind 8 (NOBITS)
MB_CheckCollPill @ 0x1e8890 -> section 1, kind 8 (NOBITS)
Both land inside the level program's zero-filled section. Every byte is zero and
the following 256 bytes contain no non-zero byte at all -- code cannot be there.
Same on all four levels, so it is not a level with an unusual layout.
So the addresses belong to a different address space than a parsed level overlay.
Most likely the boot executable's own layout, or the v2.00 build; that repository
covers both. SCUS_972.68 is USA v1.01 and our disc matches it exactly, but that
fixes the disc, not which program's address space a symbol address is quoted in.
The disc matching is exactly the kind of check that looks sufficient and is not --
a disc can pin the bytes and still leave you looking in the wrong program.
Worth noting the method that caught it. The temptation was to take the address,
open the overlay at that offset, and start reading instructions. What settled it
was checking which section the address falls in, and its KIND: a NOBITS section is
zero-filled by definition, so an address inside one is a contradiction rather than
a lead. The section table made that a two-line check.
This closes the shortest route to the collision cell layout, for a checkable
reason rather than a vague one. The route that remains is the one already
recorded: find how the code addresses the Camera_CollPrimTest assert string, which
is not by a $gp-relative load.
|
||
|
ReaperOfVeriod
|
39462d2cf9 |
Correct: overlay.bin is the seven-section program, not the .lit section
The Phase 3 port brought parse_overlay_sections with it, which makes the overlay's
shape checkable for the first time. It corrects a claim I committed when
vendoring the decomp.
What I wrote then: overlay.bin +0x00 is 0x1A7C80, that is the .lit load base, "so
the first header word is the load base of the section this file is", and therefore
"overlay.bin IS the level's .lit section".
It is not. overlay.bin is the whole seven-section program. .lit is its first
section: 39,388 bytes at 0x1A7C80 on level 0, against a lump of 2,751,268.
section 0 .lit 0x1A7C80 on all 27 levels
section 1 .bss kind 8 (NOBITS) on all 27, zero non-zero bytes
sections 2-6 .data, lvl.vtbl, lvl.camvtbl, lvl.sndvtbl, .text
kind sequence [1, 8, 1, 1, 1, 1, 1] identical on 27/27
section sizes fall 112 bytes short of the lump on 27/27
(each section's size counts its own 16-byte header)
The error was reasoning from a constant to a statement about the file's nature. A
load base tells you where something is placed, not what the containing file is.
That is the same move as the other four withdrawals this session -- the 305x
class-ID slot, the UV-corner texcoords, the VIF code word, the DAW pointer -- all
of which read a single plausible item as evidence about the whole.
Worth noting what survives. The collision assert string is in SECTION 0 on all 27
levels, so the earlier $gp-relative search located it correctly even though the
reasoning about the file was wrong. The string being in .lit and the file being
.lit are different statements and only the first is true. A wrong reason can reach
a right answer; it is still worth correcting, because the next person to reason
from it would not.
Asserted in overlay_bin_is_the_seven_section_program_not_the_lit_section.
259 tests.
|
||
|
ReaperOfVeriod
|
4ca0308499 |
SBlk body head: the DAW size, stored twice
Picked sound_bank rather than tie, where three discriminators have now come back
negative and a fourth would be guesswork.
The body after the 24-byte head opens with twelve words. Six are settled across
all 27 levels:
word value note
0 varies per level not identified; 27 distinct
1 60 constant
2 varies per level not identified; 23 distinct
3 0 constant
4 daw_size == the header's daw_size, exactly
5 daw_size the same value again
6,7,8 0 constant
9 127 constant
10 1 constant
11 0 constant
Level 0's words 4 and 5 are both 1,482,000, and daw_size is 1,482,000.
THE POINTER THAT WAS NOT ONE
Words 4 and 5 were briefly recorded as pointers into the DAW region. They are not.
The test that promoted them was "does this value fall inside [daw_offset,
daw_offset + daw_size)" -- and the DAW region is ~1.5 MB starting near the front of
a ~1.5 MB extent, so almost any plausible 32-bit word lands inside it. 1,482,000 is
merely just under the top of exactly that range.
A range test is not an identification. What survives is EQUALITY with daw_size,
exact on all 27 levels and asserted, which is a claim that can fail rather than one
that is almost always true.
That is the fourth time this session a plausible reading of a single number has had
to be withdrawn for something falsifiable: the 305x class-ID slot, the UV-corner
texcoords, the VIF code word, and now this. The pattern is consistent enough to be
worth stating as a rule -- a test that passes almost always is measuring the shape
of the data, not a property of it.
Word 0 gets one note: levels 1 and 4 hold 13,042,052 and 13,042,054, two apart,
while their daw_sizes differ by 63,504. Whatever it is, it is not a size.
The remainder of the SBlk body -- the per-sound record table -- is still undecoded,
so sound_bank stays open. A SBlkBodyHead struct carries the twelve words with the
constants asserted in its docs.
259 tests.
|
||
|
ReaperOfVeriod
|
ec433e3341 |
Correct my own "VIF packets" reading: the split is real, the identification is not
Last commit read bit 31 of a tie vertex region's first word as "VIF code word"
and told a two-encoding story on it. The split is real; the identification is
wrong, and I checked it before building anything on top.
Walking those regions as VIF packets, levels 0-2, 914 records:
walk reaches the region end 17
walk dies after exactly ONE packet 1,571
packets before dying: min 0, median 1, max 4
tfrag walks VIF the same way -- VifCode::decode, then num * 4 payload words -- and
reaches the end of its lists every time. So the walk is not the problem; these
regions are not VIF packet streams. Bit 31 is set on whatever this word actually
is, and that is still unknown.
The specific thing that made me overreach: I quoted 0xf5c1a3c5 as "a VIF code
word, cmd 0x75 num 193" and treated that as establishing the format. Decoding one
word as a packet header says nothing about whether the next 193 words are its
payload. Here they are not -- the walk dies immediately after.
This is the third time in this area that one plausible word produced a confident
story: the 305x class-ID slot, the UV-corner texcoords, and now this. All three
shared a shape -- a single readable item at the head of a population, treated as
evidence about the whole. The bit is the only thing the two kinds reliably
differ on, so it stays; the name goes.
The test now asserts BOTH the split and the walk failure, which is the part worth
keeping. If these ever do turn out to be VIF, the test fails and says so, rather
than two findings drifting apart silently -- which is how "half VIF, half flat"
would have sat in the docs indefinitely.
Docs corrected in both roadmap and format file, including the quoted example.
258 tests.
|
||
|
ReaperOfVeriod
|
4ab30e1f4d |
Retract the tie texcoord field: a dominant-value artifact, one level down
Picked the tie flat encoding to close, since |
||
|
ReaperOfVeriod
|
1e90c7de6d |
Verify the Phase 2 gate, and give every blocked item an unblock condition
Starting from the phase's stated exit criteria rather than the checklist, because
"complete" should mean the gate, not the feeling of the list.
THE GATE IS MET. Both halves, checked by inspection:
* Golden tests. Every module in gc-formats/src is referenced by a test or another
module's test path; none is unreferenced. gc-extract/tests/disc.rs carries 50
real-disc tests, and overlay_reloc's mask test resolves a level-01 function on
all 26 other overlays.
* Counts are failures, not warnings. No parser writes warning-style output
outside #[cfg(test)] -- the only eprintln! calls in gc-formats are inside test
modules. Count and length problems surface as typed errors: 42 uses of
ErrorKind::Invalid / InvalidLength, including global_wad's "pair table is not a
whole number of entries", which is precisely a header count disagreeing with
the data.
THE GOAL IS NOT MET, and saying otherwise would be the thing AGENTS.md forbids.
The goal reads "every asset type parses into plain Rust values". Four do not:
collision cells, volumes, moby_anim and the tfrag morph are located or framed but
have no decoded fields. The gate was written to be checkable and it does not test
for that, which is a gap in the gate rather than a reason to declare victory.
So the remaining gap in the phase's own discipline is that the four blocked items
did not say what would unblock them, which AGENTS.md requires and which none of
them did. Closed now, each with the specific unblock condition:
collision cells the game addresses its assert string by something other than a
$gp-relative load -- a %hi/%lo split that does not straddle
zero, a pointer table, or another section. A plain $gp search
returns zero and always will.
volumes ground truth: a count of level 0's trigger volumes from a map,
to match against the 144-byte table's population. No class name
containing "volume" exists on the disc, so the dispatch route is
closed.
moby_anim needs moby class records decoded past the header -- a
dependency, not a measurement.
tfrag morph needs a frame captured at a KNOWN camera distance, so the
expected blend is computable. Requires driving the running game,
not analysing a static savestate.
None of these is an approximation, and none is marked done.
258 tests, 50 in the disc suite, clippy clean on my own code.
|
||
|
ReaperOfVeriod
|
67ce068c7f |
Tie vertex regions are two encodings, not one; mark Phase 3's port done
Read up on the five commits that landed since |
||
 ReaperOfVeriodandreaper
|
a94704de18 |
port Phase 3: overlay masking + code-identity + relocation
Ported ReRAC's (vendor/rerac/crates/rc-formats/src/level_overlay.rs) mask /
Relocation machinery into gc-formats:
- parse_overlay_sections, read_overlay: section table {dest,size,kind,entry},
ends on entry mismatch; kind==8 = BSS; plus LevelOverlay with .text words,
text_start and function starts.
- mask: hides relocated fields (j/jal targets, lui of main-RAM 0x0010..0x0040,
%lo/move/$gp completions tracked per-register).
- walk/HiState/address_refs/key: PS2 code walk with branch-delay-slot register
state.
- Relocation::new/same_code/copies/func/data: masked-code index, one-copy
lookup, data-address relocations.
A real-disc test asserts that a level-01 engine function's relocated copy is
found exactly once on all 26 other overlays (gate: every level-01 function is
reachable by masked-code identity on all 27).
Co-authored-by: reaper <reaper@users.noreply.github.com>
|
||
|
ReaperOfVeriodandreaper
|
8030dbf006 |
close: tie LOD distance ordering exceptions - 80/1864 classes have mid=8192 far=8190
Tie class headers ordered near < mid < far on most classes. Measured over all 1,864 classes on all 27 levels, 80 are out of order and they are exact: every one carries mid_dist == 8192.0 and far_dist == 8190.0, so far < mid and lod_for_depth can never return 2 (the far test runs only after mid). Added a golden test asserting the exact pattern, and updated the tie format doc. Co-authored-by: reaper <reaper@users.noreply.github.com> |
||
|
ReaperOfVeriod
|
e5cd3df40e |
Retract the tie vertex position claim; the stride is not 16
Task was to name the axis order and scale of the tie vertex positions by fitting the class header's bounding sphere. The sphere does not exist: at +0x30 those bytes are zero on every class. Scanning the header for float-like words found +0x10, +0x14 and +0x18 (the LOD switch distances, already named) and +0x40 on 1,676 of 1,864 classes, which is not a sphere. What the run produced instead is a retraction. Dumping one packet's vertex region raw shows the first 16 bytes are header material, not a vertex -- 0x0000df20 is not a coordinate, and the words after it are the vif and material words. A 16-byte group does recur at +64 and +128 with identical structure, so the stride is not uniform: some elements are 4 bytes, some 16, and the run length varies. Two candidate bases were measured and neither clusters. With packet + vert_ofs the per-class extents run median 64,976, p90 65,517; with class + vert_ofs, median 55,213, p90 64,352. Both span the whole i16 range, where a position triple in a class-local frame would span a fraction of it. The region bounds are right -- vert_ofs + vert_size + scissor_size is the length -- it is the element layout inside it that is wrong. The i16[3] marker does not rescue it: bucketing every fourth element by the marker's high byte gives markers 0, 0x48, 0xff and 0x01 with thousands of vertices each, and only 0x48 shows a position signature (i16[1] extent 425 against 65,535 on the other two axes). The same marker 0 bucket spans all three axes at full range, so the bucket does not select a layout. The 95,612-vertex count and the 82.5% texcoord fixedness figure were computed on the 16-byte reading and are now untested against the correct stride, which is not yet known. The LOD1/LOD2-sharing test stays the strongest evidence for positionality and is stride-insensitive. |
||
|
ReaperOfVeriod
|
855aadf523 |
Record the tie terminator defect in the format doc, plus two failed fixes
The defect itself was measured and committed in
|
||
|
ReaperOfVeriod
|
572cd1898d |
Record a defect packet slicing found: the tie block terminator is early
Range measured: every tie LOD block on the disc, 2,177 blocks. The last packet's end is known two ways. Its length comes from its record's packet_quadwords, and every other record's length is confirmed by the next record's pointer -- so only the last packet of each table is unconfirmed. Comparing its computed end against where parse_tie_block stopped: the last packet's computed end lands 2 to 16 bytes past the located 0xf6, on all 2,150 blocks measured. Two readings fit. Either the terminator was located too early -- inside the last packet's data -- or the last packet's length over-predicts by a small amount. The first is likelier: a length error on the last record of every table would have to be a coincidence. Neither is settled here, and the test asserts the measurement rather than the reading. Why the existing test missed it: "2,177 of 2,177 blocks end on 0xf6" is satisfied by a false terminator exactly as well as a true one -- it checks that the byte found IS a 0xf6, not that it is the block's end. A test that confirms the shape it walked to will pass on the wrong byte. The cost: the block extent is short by that much, so payload and padding under-report and anything computed from them is correspondingly short. The block starts are unaffected and RAM-confirmed. Recorded as a limitation with the measurement attached, not fixed. A terminator packet data cannot imitate is its own task, and guessing one would be the plausible-sounding alternative this project has been bitten by. A packets() accessor was tried and removed for the same reason: with the extent short it silently yields the wrong answer, and an API that is quietly wrong is worse than no API. |
||
|
ReaperOfVeriod
|
1cfe744501 |
Settle the tie packet length, and correct the retraction that hid it
Range measured: every tie LOD block on the disc -- 2,177 blocks, 13,042
consecutive record pairs.
A record's pointer is where its packet starts, so the distance between
consecutive pointers IS a packet's length, with no reference to any field.
Comparing that against the record's own fields:
packet_quadwords = vert_ofs + vert_size + scissor_size
12,927 of 12,927 record pairs whose strip_count is zero
0 of 115 record pairs whose it is not
The partition is exact: every record with a non-zero strip_count is an
exception and no record with a zero one is. So the fourth term is 0 quadwords
whenever strip_count == 0, and something else -- measured 0..19 quadwords, and
NOT equal to strip_count -- whenever it is not. R&C1's strip_count is a strip
count and strips are variable-length data, so a non-zero count most likely
adds per-strip bytes this cannot see; that is a hypothesis, labelled as one.
TieRecord gains packet_quadwords and packet_quadwords_are_known. The latter is
the honest gate, and the disc test asserts both sides of the partition rather
than the aggregate.
This also corrects this file's own retraction of the same formula, and the
retraction was the error rather than the claim. A formula of this shape was
derived at 61 of 61 pairs, then "retracted" as not reproducing (0 of 13,042).
The re-measurement had swapped strip_count for scissor_ofs in the fourth term
and put the blame on the block set. Re-measured with the term the first pass
had, it matches on 12,927 of 13,042 and the 115 exceptions are exactly the
non-zero strip_count records.
Worth recording because it is the same shape as two other entries in this
file: a retraction is not automatically more careful than the claim it
retracts, and the re-measurement that "disproved" this one changed the thing
being measured. The lesson on the other side is what saved it -- the first
measurement was over 61 pairs, not a population, and had it stayed there the
formula would have looked universal. A small sample and a bad re-measurement,
near-misses in opposite directions.
What this unblocks: the payload can be sliced per packet, which was the
prerequisite for decoding what it holds.
|
||
|
ReaperOfVeriod
|
4abce518e5 |
Decode the overlay's load base, from its own profiler table
Range measured: all 27 levels' overlays. The overlay carries the game's profiler menu definition -- a run of 8-byte padded names (normal, camera, cam+chr, volume, bars, render, mobys, ties, shrubs, cpu/gpu, occl, ...) followed by runs of absolute RAM addresses that point back at those same names. On every level the four addresses after normal/camera/cam+chr/volume, each minus its target name's overlay offset, give the same number: 1,735,792. That is field_00 - 16, where field_00 is the header word that read 1,735,808 on all 27 levels with nothing attached to it. The overlay is loaded with its 16-byte header stripped, so overlay offset X sits at RAM X + 1,735,792. OverlayHeader::ram_address carries it, and the disc test asserts the derivation holds on all 27 levels. Confirmed, not only derived: level 0's RAM 0x1acf80 holds "normal\0camera\0cam+chr\0volume\0", byte-identical to overlay 0x5310, and 0x5310 + 1735792 == 0x1acf80. All four names land where the overlay's own table says. `gc-trace verify-overlay-base` runs that check, savestate-gated the same way as the tie and tfrag ones. The overlay is relocated, not copied: only 37.67% of level 0's is byte-identical at the base, and the 256-entry sine table -- at overlay 0xb650c -- is not at 0xb650c + base. So the base locates the overlay's image, but individual tables inside it move. That is the shape of a loader applying relocations, and it means the base is necessary but not sufficient for reading the game's tables. The value of the base is that any overlay offset can now be carried to a RAM address, which is the prerequisite for reading the game's live level state -- the thing the tie relocation and tfrag lighting checks had to work around. Two things get deleted rather than finished. The collision u16 table was probed four ways -- magnitude distribution, level-independence, run starts, run lengths -- and every one came back level-dependent: the run-start pattern is identical on 1 of 19 levels. So the first 1024 bytes of the collision payload are data, not a fixed table, and nothing was decoded. Recorded as the negative it is rather than left as four half-findings. |
||
|
ReaperOfVeriod
|
55ea619b4c |
Decode the tie class header's +0x0c word: three LOD record counts
Range measured: every tie class on the disc, 1,864 classes over all 27 levels. Method: read the word at class +0x0c as four bytes, and read the record counts independently from the blocks themselves -- follow packets[lod] to the block, walk to the 0xf6 terminator, and count the 16-byte records the block header's data_offset names. Two paths to the same number, so the check is not circular. Result: the byte at +0x0c is the LOD 0 record count, +0x0d is LOD 1's and +0x0e is LOD 2's, on 1,864 of 1,864 classes. Classes with an absent LOD agree too: a zero packets[lod] gives a zero count and the byte is zero. This closes the file's oldest lie about the tie header. The comment in TieClassHeader::parse read "0x0c: the 64 light-slot normals offset. Read past, not decoded" -- inherited from R&C1's layout, where the word genuinely is an offset. On R&C2 it is not an offset at all; it is three counts, and R&C1 carries the same quadruple at +0x20. Skipping the field was the right action on the wrong information: the field was unknown only because R&C1's map was applied to R&C2's bytes. TieClassHeader gains packet_counts, and tie_class_header_counts_records_per_lod asserts the agreement. The fourth byte, +0x0f, is left unnamed. It ranges 1..15, and the obvious following of R&C1 -- the shader/ad-gif total -- disagrees on 1,827 of 1,864 classes, which is 2% agreement where a count would expect 100%. It is recorded as unidentified rather than given a name that fits a fiftieth of the population. |
||
|
ReaperOfVeriod
|
f1f0b517a5 |
Pin the extracted tree, which every disc test reads and nothing verified
data/gc-files.tsv pins the ISO files, and unpacks_and_matches_the_committed_table checks that unpacking reproduces them. But the tie, collision, tfrag and chunk tests do not unpack -- they read ~/Documents/gc-extracted, a directory on disk that nothing in the repository checks. A stale or hand-edited file there would have every measurement built on it still pass, quietly. Three claims this session returned confident numbers while describing the parser rather than the disc -- has_uniform_record_length, "1,509 chunks uniform", the moby class-ID scan -- and a test whose data source is unverified is that same hazard one level down. data/gc-extracted.tsv now records SHA-1 and size for all 5,237 files in that tree, 324 KiB, sizes and hashes only, no content. It is the output of `gc-extract unpack`, whose ISO files are already pinned by the other table, so it is grounded transitively rather than by assertion. the_extracted_tree_matches_the_committed_table walks the table and checks every row's size and hash against disk. It asserts all 5,237 are checked -- so a truncated table fails rather than passing on a prefix -- and reports the first few missing or mismatched paths when it does. Current result: 5,237 of 5,237 match. This does not retroactively validate the three retracted claims. What it does is remove the possibility that any of the surviving ones rests on a directory that could have been anything. 246 tests, clippy clean. |
||
|
ReaperOfVeriod
|
8ccc3ddda4 |
Retract the uniformity claim everywhere it still stood, including the API docs
The record-length retraction had been applied to the tests and to the section
that discovered the problem, but the same claim was still standing in four other
places -- including the rustdoc for has_uniform_record_length(), which is what a
caller would actually read:
crates/gc-formats/src/chunks.rs:206 "Measured: true for 1,509 of 1,509 chunks"
docs/roadmap.md:448 "uniform record length (1,509 of 1,509)"
docs/formats/core-data-regions.md a whole section titled
"Every chunk has a uniform record length"
All three now say 257 of 1,509, and the method's rustdoc explains WHY the old
number was an artefact -- that resolve sliced at the first offset delta, so every
record came out the same length by construction and the predicate was confirming
the parser back to itself.
That section also contained a withdrawal, and the withdrawal was wrong:
"This withdraws the previous commit's variable-length type. That came from
taking minlen/maxlen over all of a type's records at once, pooling across
chunks; rule 3a's pooled claim, third time this session."
The pooling rule fired correctly and was overruled by a measurement with no
independent content. The per-chunk check that should have caught the pooled
signal was itself reporting the parser's own slicing, so it agreed with the wrong
answer for the same reason the original claim did. Two errors stacked: a real
signal dismissed by a rule that had been right twice already, and a predicate
that could not have told the difference.
This is the third time this session that a test was asserting the parser's
behaviour rather than the disc's, and the third time it was the only thing
standing between a claim and a retraction. has_uniform_record_length is the
clearest instance: a predicate named after a property of the data, returning a
confident answer about the code.
245 tests, clippy clean.
|
||
|
ReaperOfVeriod
|
2d3880b624 |
The class-ID retraction goes further: the claim was wrong before the fix
Pinned down the one slot that still cleared 100x after the record-boundary fix,
and it is the artifact this section was written to guard against.
The discriminator the original version lacked: a slot that really carries moby
class IDs must SPAN the ID space, so some of its values land above a level's moby
count. The artifact does the opposite.
type slot enrichment distinct values ABOVE the moby count
0xd76cc453 6 305x 9 0
0xf2ef4275 34 64x 9 341
0xf2ef4275 7 46x 16 412
0xf2ef4275 14 34x 16 432
0xd76cc453 9 32x 28 702
The 305x slot's nine values are 1, 3, 4, 5, 10, 11, 12, 13, 15 -- all under a
level's ~215 moby classes, so every one scores as a hit and 65536/215 = 305x.
Nine small numbers is not a class-ID array. Every slot that actually spans the ID
space scores 10x to 64x, which is chance.
So the original claim's own numbers gave it away and it was not read that way:
"5 to 42 distinct values at 153x to 289x" is exactly the artifact's shape -- a
handful of small values, none above the count. The record-boundary fix was never
what made this claim wrong. It was wrong when written, and the retraction is
complete rather than the partial one I committed two turns ago.
The test now separates the two quantities: slots clearing 100x (currently 1,
expected to remain the artifact) and slots clearing 100x WHILE ALSO having values
above the moby count (currently 0, must stay empty). That second number is the
one that means something, and a slot appearing in it would be the first genuine
evidence of a class-ID array anywhere in the container.
Worth noting the per-SLOT bookkeeping: the count of values above the moby count
is keyed by (type, slot), not by type. A first attempt keyed it by type and the
artifact type passed the check on the strength of its ordinary slots -- 783 of its
own values sat above the count at other slots. The check has to be about the
individual slot, or one good slot launders a bad one.
245 tests, clippy clean.
|
||
|
ReaperOfVeriod
|
d159195510 |
General guard for the over-read class, and the audit showing it was unique
Searched for the pattern that caused the over-read -- a record length derived from
the first difference between two offsets:
grep -rn "\[1\] - \[0\]" crates/gc-formats/src/*.rs
crates/gc-formats/src/chunks.rs:357 (offsets[1] - offsets[0])
One hit, in the one parser that walks an explicit offset table.
The others slice contiguous regions with chunks_exact -- shrub's 8-byte palette
records, tie's 16-byte packet headers, tfrag's 4-byte tables. There a length
cannot run past the next element, because chunks_exact derives each boundary from
the region's own length rather than from a sample of the spacing. Shrub's mesh
packets carry an explicit offset AND size, so they state their length instead of
inferring it.
The damage was bounded by a structural property worth stating: only a format with
an offset table AND a derived stride can over-read, and exactly one format had
both. That is a more useful fact than the retraction was on its own -- it says
where else to look if another one turns up.
The guard is positive rather than a comment.
chunk_records_neither_overlap_nor_leave_gaps asserts across all 27 levels that
every record but the last is exactly its own offset delta long, that there are
1,509 chunks, and that 1,252 have ragged deltas. A regression to a single sampled
stride fails immediately and names the chunk and the record.
245 tests, clippy clean.
|
||
|
ReaperOfVeriod
|
92aa0fe49f |
Audit what the record-boundary fix contaminated: one retraction, one survivor
The fix changed record lengths in 1,252 of 1,509 chunks, so every committed claim
computed over record bytes needed re-checking rather than assuming. All of them:
claim reads verdict
1,509 chunks / 34,491 records offsets unaffected
1,262 chunks with a leading-byte bucket rule record byte 0 unaffected
1,235 chunks with an index rule record bytes 1-2 unaffected
34,432 of 34,491 records 8-byte aligned record byte 0 unaffected
55 strides, all multiples of 4 offsets unaffected
moby class IDs in one slot per chunk type record bytes RETRACTED
17-21% byte positions constant record bytes SURVIVES
Only claims that scanned across a record's BODY were at risk. The rest read a
record's first three bytes, which an over-read never touched -- it appends to the
end. That asymmetry is why the damage was contained to one claim, and it is the
argument for preferring claims anchored at record starts.
The survivor was the obvious candidate for contamination, so it is worth being
precise about it. Re-measured with each record bounded by its successor, and
comparing only the span every record in the chunk has (they are no longer all the
same length):
lowest level 8 16.7%
highest 20.0%
Still inside the documented band. Previously this lived only in a doc comment on
the layout test; it is now asserted, in
chunk_records_share_a_layout_measured_by_constant_byte_positions, over all 27
levels with a 10-30% band so a real change reports itself rather than sliding.
The module documentation is corrected to the measured range and now says outright
that records in one chunk are not all the same length, since that is the fact a
reader would otherwise have to rediscover.
244 tests, clippy clean.
|
||
|
ReaperOfVeriod
|
4fab4d0359 |
Fix the record boundaries after all, and retract a claim built on the bug
I reverted the record-boundary fix last commit on the grounds that "fixed-length
records plus inter-record padding" and "variable-length records" produce the same
offset table. That reasoning was wrong, and cheaply so.
stride is the FIRST offset delta. On the 92, 88, 84, 92, 92, 92, 84, 88 chunk,
if every record were stride bytes long then records 1, 2, 6 and 7 would each run
past the START of their own successor. Records are addressed by an explicit
offset table; they cannot overlap. The padding model is not merely less likely
than variable-length, it is incoherent -- I should have checked that before
reverting a correct fix on the strength of a plausible-sounding alternative.
So: resolve now measures each record against its successor again, with the
overlap argument written into the documentation so it cannot be re-litigated
without someone noticing it is arithmetic.
THE RETRACTION, which is the substantive part:
Three tests broke. Two were straightforward, but the third was not.
* constant_positions indexed records as if they were all the same length. Now
that they are not, it compared past the short ones. It takes the minimum
length instead.
* "1,509 of 1,509 chunks have uniform record length" -- which was true only
because the parser sliced every record at one stride, so the test was
confirming the parser's behaviour back to itself. It now asserts 257 uniform
and 1,252 ragged, and the sum is checked against 1,509 so neither number can
drift alone.
* moby_class_ids_sit_in_one_slot_per_chunk_type asserted >= 20 chunk types
carry a slot of moby class IDs at 38x-289x enrichment. It now finds 2, of
which 1 clears 100x. RETRACTED. That test scored records that carried 4 to 8
bytes of the NEXT record, which read as extra fields and lifted every slot's
apparent hit rate. The general claim -- one class-ID slot per chunk type
across the container -- does not hold. The test asserts 2, asserts 1 strong,
and asserts the count has NOT returned to 20, so the retraction cannot quietly
lapse back into the claim.
That is a committed finding withdrawn on the strength of a parser bug. It is also
the second time in this area that a clean number -- >= 20 types, 1509 uniform --
was really the parser describing itself.
243 tests, clippy clean.
|
||
|
ReaperOfVeriod
|
5f0396e3d7 |
Measure every chunk's record offsets; the cause is ambiguous, so no fix
The previous commit fixed on one signature. Measuring all of them shows the
pattern is the norm. Taking delta = offset[i+1] - offset[i] against the chunk's
own first delta, levels 0-2, 2,953 deltas:
0 1,723 -4 549 +4 426
-8 146 +8 32
-12 71 +12 21
-16 34 +16 4
-20 27 +20 1
-24 13
Every deviation is a multiple of 4 and most are exactly +/-4; worst observed is 24
bytes in a 7-record chunk. 1,252 of 1,509 chunks have at least one, across 66 of
74 signatures.
WHY THIS DOES NOT BECOME A FIX
Two models produce this offset table and they are observationally identical:
1. records are genuinely variable-length, padded to 4 bytes, stride = largest
2. records are fixed at stride and the offset table carries inter-record
padding, so the gap to the next record is record + padding
Nothing in the offsets distinguishes them. Both say where records start. Both
produce 92, 88, 84, 92, 92, 92, 84, 88 for the case already discussed.
I wrote the change that slices each record against its successor -- model 1 -- and
then reverted it. Two things came out of writing it:
First, it broke three tests, which is the informative part. Those tests assert
"1,509 of 1,509 chunks have uniform record length" and they passed BECAUSE the
parser forced uniformity by slicing every record at a single stride. They were
asserting the parser's behaviour back at itself, and would have kept passing
forever regardless of what the disc contains.
Second, shipping it would have replaced one unverified model with another and
silently changed the record boundaries of 1,252 chunks on the strength of a
guess. The repo's own rule applies: a heuristic count is not an identification,
and here it is not even a count -- it is two equally good explanations of one
table.
WHAT WOULD SETTLE IT: a field inside the record whose end is independently
observable -- a terminator, a count implying a length, or a known value pinning
its position. Under model 2 the disputed bytes are padding and will be uniform or
zero; under model 1 they are data and will not be. For [75 42 ef] those 8 bytes
read a0 00 27 20 at both the tail of one slice and the head of the next, which is
arithmetic rather than evidence, since it is the same eight bytes seen twice.
The uniform model stays, the ambiguity is recorded in resolve()'s documentation
and here, and the 1,509-of-1,509 assertion stands as a statement about what the
parser does rather than about what the disc contains. 243 tests, clippy clean.
|
||
|
ReaperOfVeriod
|
b9e9835639 |
A parser defect: one chunk type is not fixed-stride, and over-reads
[75 42 ef f2 71 b5] -- the signature that is both level-specific and
variable-length -- makes three chunks per level, 81 total:
stride 76 deltas [76, 76, 76, 76, 76, 76, 76, 76] uniform
stride 92 deltas [92, 88, 84, 92, 92, 92, 84, 88] NOT uniform
stride 104 deltas [104, 104, 104, ...] uniform
Identical on all 27 levels: 27 chunks of each shape, three shapes, no others.
So "this signature has strides {76, 92, 104}" is a misdescription. It has three
chunk shapes, one of which is not fixed-stride at all. The parser derives one
stride per chunk and resolve slices every record at that length.
The cost, precisely: records are read at the offset table's positions, so every
record STARTS correctly -- no misalignment. But each is sliced at 92 bytes, while
two of the eight are 88 and two are 84, so four records over-read by 4 and 8
bytes and carry the head of the next record inside them. Bounds checking holds;
this is a data-correctness bug, not a safety one. It is precisely the kind that
produces plausible-looking fields offset by four bytes, which is how it would
have gone unnoticed had the stride table not been collected per signature rather
than inferred per region.
It also explains why this signature was previously flagged as breaking the 8-byte
alignment rule: its records genuinely are not a uniform multiple.
The right fix is for resolve to use offset deltas for a chunk whose deltas are
not uniform. Not taken here -- the delta sequences are asserted first, so if the
data ever becomes uniform the test fails and asks for the model to be revisited,
rather than the model quietly absorbing the exception.
Asserted in chunk_type_75_ef_has_one_variable_length_chunk_per_level, with the
exact sequences [76x8], [92,88,84,92,92,92,84,88], [104x8] and 27 of each.
A near-miss worth recording, because it would have made a tidy false claim: the
leading u16 of these records appeared to step by exactly 0x50 (80) across all nine
of the 76-byte chunk. Reading at the real offsets gives b1a5, a529, 047e, a529,
04f0, a529, 0562, a529, 05d4 -- no progression at all. The apparent 80-byte step
came from comparing the wrong bytes, and "records are indexed by 80" is exactly
the kind of finding that gets reported on sight.
243 tests, clippy clean.
|
||
|
ReaperOfVeriod
|
af62032ddc |
Most of the chunk container is level-independent: 65 of 74 signatures
Comparing every chunk record against the same record index in the other 26
levels, across all 74 signatures:
65 of 74 every record byte-identical in all 27 levels
9 of 74 level-specific
0 ragged no signature appeared with different record counts per level
The nine that vary:
[5d 44 13] [75 42 ef] [7f 44 ee] [82 c4 2f] [96 c3 37]
[a2 44 a5] [c6 c3 ef] [df c3 ea] [f3 c4 b9]
This reframes the container. 65 of 74 chunk types are static tables the game ships
unchanged in every level -- constants and lookups, not level content. The largest
by record count are all in that group: [38 c4 7e] stride 328 with 90 records per
level, [a9 c4 d8] stride 308 with 90, [3d c4 9e] stride 368 with 77.
The consequence is about method, and it is worth being blunt about. For a
level-specific type, comparing levels is the strongest lever available: a field
that moves with the level is identifiable by correlating it with something known.
For the other 65 there is nothing to compare -- one copy of the data, repeated 27
times -- so they must be decoded structurally, from what the bytes are. A future
decoder that reaches for per-level comparison as its first tool will find it
useless on the majority of the container.
[75 42 ef] is both level-specific AND the one signature with three strides. A
type that is both variable-length and level-varying is the most informative one
here, and is where per-type field decoding should start.
How this was found, including the wrong turn:
The first look at [53 c4 6c d7 9a 43] (stride 108) showed 45 of 108 byte positions
varying WITHIN a chunk and 810 records all distinct -- which reads like per-record
data. Counting distinct values per u32 slot across all 810 records contradicted
that: 29 to 30 distinct values across 27 levels, i.e. about one per level.
Both readings were wrong because they measure different things. The 45 positions
vary between records within a level; the u32 counts were low because the same 30
values recur in every level. Comparing record i across levels settled it: all 30
records are byte-identical in all 27 levels, and zero byte positions vary across
levels.
Had the u32 distinct counts been the only measurement, the conclusion would have
been "one value per level" -- the opposite of the truth, describing a per-level
parameter table that does not exist. The within-chunk and across-level views had
to be taken together to get a true one.
Asserted in most_chunk_signatures_are_level_independent. 242 tests, clippy clean.
|
||
|
ReaperOfVeriod
|
59dba3e58d |
Chunk records: the stride table, and the exception pinned by name
Walked every chunk on all 27 levels, grouped by six-byte header signature:
1,506 chunks, 34,415 records, 74 distinct signatures
55 distinct strides, 52..640, every one a multiple of 4
73 of 74 signatures use exactly one stride
The exception is asserted rather than skipped, because "every type is uniform"
is precisely the claim that quietly stops being true:
[75 42 ef f2 71 b5] strides {76, 92, 104}
That is the variable-length type the module docs had referred to indirectly. It
is now pinned by signature and to those three strides exactly.
The 4-byte alignment is a result, not an artefact. A stride recovered from one
chunk's first-to-second record delta can inherit alignment from where that chunk
happened to sit; collecting all 55 strides across every type and requiring all
of them to be divisible by 4 rules that out, because a single off-by-one in one
signature would show. Asserted in
chunk_record_stride_is_four_byte_aligned_and_uniform_per_signature.
What this settles is the shape the decoder must handle: uniform stride per type,
4-byte aligned, one named exception. Field offsets within a record are now the
same for every record of a type, which is what makes per-type decoding
tractable. The fields themselves are still unnamed.
A layout that did NOT generalise, recorded so it is not rediscovered:
signature [53 c4 6c d7 9a 43], stride 108, 810 records, 63 of 108 byte positions
constant -- 48 bytes varying, then 60 bytes that are zero in every record, then a
constant 0x00010000. That reads like an inline buffer this type never uses, and
would have made a tidy field story. Requiring `stride == live span + zero run`
across every type with 50+ records per chunk gives 0 of 9. The zero tail belongs
to that one type and means nothing general.
Third occurrence of one editing mistake, and it is worth naming as a habit
rather than an accident: inserting a test before an existing one strands that
test's #[test] attribute on the new doc comment and silently unregisters it.
It has now happened three times, each time surfacing only as clippy's "never
used" and never as a missing test. Fixed by appending new tests at the end of
the file instead of anchoring them to an existing function, and verified by
listing registered tests rather than trusting a count.
241 tests, clippy clean, all five recent tests confirmed registered.
|
||
|
ReaperOfVeriod
|
040386c0fb |
Volumes: the reference that did not generalise
Last commit said what would settle the volume candidate and implied one of those
was available. They are not. Recording the failure rather than moving on quietly.
Level 0's data.bin -- the level's file descriptor -- contains the value 799204
exactly once, at 0xab6360, and 799204 is byte-for-byte the offset of the 20-record
144-byte candidate table. A single 4-byte match in 13.3 MB has a chance
expectation of ~0.003, so it was worth checking rather than assuming.
It does not hold up. Each level's own best 144-byte table, searched in that same
level's own data.bin:
level candidate records refs (chance ~0.0035)
0 1,810,724 32 0
1 1,884,884 386 0
2 2,066,804 224 0
3 2,438,228 365 0
4 2,677,780 114 0
5 1,255,780 14 0
6 2,496,020 115 0
7 2,062,072 311 0
None. The level 0 hit at 0xab6360 also sits among high-entropy neighbours, and
level 0's own BEST table -- a different offset, 32 records, not the 20-record one
-- is not referenced either. So the hit does not generalise and is most likely
coincidence.
This retracts what the previous commit implied. Nothing links a 144-byte float
table to volumes: the table is reached by walking structure rather than by an
explicit pointer, so the discriminator I named does not exist, and the candidate
stands exactly where it did.
Recording the three routes that would actually settle it, all of them
unavailable: the game's own volume code (same dead end as the collision anchor
-- a GP-relative string search returns nothing); ground truth, such as a count of
level 0's trigger volumes from a map; or a correlation with a named class, since
gameplay.bin resolves ids through its dispatch records -- but no class name
containing "volume" exists on the disc, which is exactly why this is unavailable.
No test added, because there is no solid claim here to assert. 240 tests,
unchanged.
|
||
|
ReaperOfVeriod
|
ca277a0a01 |
Volumes: a candidate table found, explicitly not called volumes
Volumes were the only type never located, so this is a search report rather than
a result, and it is recorded as a candidate on purpose.
Ruled out first: no engine-side name exists. No level file contains "Volume"
outside UI text ("Music volume", "Volume de la musique") and Italian item
descriptions. The format cannot be found by label and has to be structural.
gameplay.bin is already-decompressed level data. Its head is a count (336 on
level 0) then 16-byte records from byte 416, carrying (type, pad, offset, id).
The id groups them -- each id has exactly two records pointing into two parallel
offset arrays -- and those offsets resolve into a text pool, the first ones
landing mid-string in dialogue ("it Megacorp Lab"). So that part of the file is
dialogue and class dispatch, not geometry.
The structural hit: a table of 144-byte float records. Level 0 has one at
799204, 20 records of which 19 are populated and one is an all-zero terminator,
with 34-35 of each record's 36 float slots world-plausible and +0x0c holding a
varying world position:
(206.5, 284.1, 116.0) (265.0, 445.3, 131.7) (244.0, 407.2, 128.0)
(219.3, 247.0, 107.5) (185.1, 282.6, 104.9) (260.8, 423.6, 130.9)
(248.2, 198.4, 51.8) (185.8, 282.7, 102.6) (247.0, 211.5, 100.6)
A 144-byte record with world positions and float radii is the shape a volume
trigger region should have, and a matching table is present in every level
examined (0-9), each preceded by an all-zero record, counts 23 to 612.
It is NOT called volumes. Nothing ties this table to volumes. Camera bounds,
spawn points and path nodes would present exactly the same thing: float records,
world coordinates, no label. The discriminating evidence has not been gathered,
and writing "volumes" in the roadmap because the shape fits would be the exact
error this log keeps catching.
Two detector bugs, both mine, both confident before they were caught:
1. The validity test accepted zero as a world-plausible float, so every
zero-filled region scored perfectly and the cross-level check returned runs of
225 to 1,514 records that were obviously padding. Requiring non-zero floats
(>=20 of 36) separated real tables from memory immediately.
2. The 144-byte stride came from four offsets I eyeballed as being 144 apart.
Gap 144 was not in the top eight gaps of the detector that found them -- it
survived because three consecutive observations agreed, not because the
measurement supported it. The corrected detector then found much longer runs,
which is the opposite of what the eyeball version implied.
Same shape as every other correction here: a clean small plausible number read as
conclusive because nothing was checked against a control that could disagree.
To actually settle it, the table needs to be correlated with something known --
if a count matches a level's visible trigger regions, or if an id in the record
resolves through the class table to a named object, that is the discriminator.
Until then volumes remain open and the roadmap says so.
|
||
|
ReaperOfVeriod
|
28ab7e094b |
SBlk head characterised: four constants and one word that is not a count
sound_bank's head was parsed but never characterised. Across all 27 levels the
SBlk region opens identically for its first twelve bytes:
+0x00 "SBlk" +0x04 1 +0x08 4 +0x0c "DAW\0" +0x10 0
and +0x14 is the only word that varies. It varies with its low sixteen bits zero
on all 27, leaving 0x0092..0x0103 -- 146 to 259.
That is the shape of a record count, and it tracks the body length at close to
100 bytes per unit. It is not one:
body / field_14 91.2 .. 110.8 across 27 levels
any stride 0..80 with body = stride*word + constant exactly: no
So the test asserts the negative too. If a stride ever makes the relation exact
on all 27, sound_bank_sblk_head_is_constant_apart_from_one_word fails and asks
for the record size to be revisited. A near-miss at 100 bytes/record is exactly
what hardens into a name the next time somebody glances at it; the constants are
also checked individually so a real deviation names which one moved.
The DAW region begins with sixteen zero bytes then block-structured data with
16-byte periodicity and slowly varying nibbles -- consistent with block
compressed audio, not decoded. Recorded as such.
Two mistakes, both mine, both the same shape as earlier ones:
1. The constructor bound `format_tag` with a `let` BEFORE the two words that
precede it in the file, consuming bytes 4..8 and shifting every field after
it. The struct literal listed the names in the right order; the statements did
not. A scratch script reading the same bytes gave the opposite answer, which
is what exposed it. Fixed, with a comment, because "read the lets in file
order" gets undone otherwise.
2. Inserting the new test ahead of an existing one stranded that test's #[test]
on my doc comment -- silently unregistering
tie_vertex_element_is_16_bytes_with_fixed_texcoords. This is the SAME mistake
as two commits ago and I made it again. It again surfaced only as clippy's
"never used" rather than as a missing test. Restored and verified by listing
the registered tests rather than trusting the count.
240 tests, clippy clean, both tie vertex tests confirmed present.
|
||
|
ReaperOfVeriod
|
bc5ebe559f |
Tie slots 3, 6, 7: two readings excluded, and why the LOD test stops working
Having established that i16[0..2] are positions, ran the same LOD-overlap method
across all eight slots to see whether it discriminates elsewhere. It does not,
and that is worth recording before someone reads the passing numbers as nine
confirmations:
slot same class control ratio
i16[0] 97.0% 3.0% 32x
i16[1] 96.7% 3.2% 30x
i16[2] 96.4% 6.0% 16x
i16[3] 75.3% 78.7% 1x
i16[4..7] 68-91% 52-79% 1x
Only i16[0..2] carry signal, because they are the only slots with tens of
thousands of distinct values. The rest draw from sets of 226 to 1,759 values, so
any two sets of that size overlap heavily regardless of what the field means. A
passing result from a method that has run out of discriminative power is not a
passing result.
Slot 3 excluded as a skin/joint index. It holds 201 values in 0..=254 and 50
distinct values within a single draw packet, so it is per-vertex, not per-material
-- which invites the obvious reading, since tie objects are skinned. Wrong:
mean pairwise distance, same slot-3 value 14,481
mean pairwise distance, different slot-3 value 13,795
ratio 0.95x
control: same position-x bucket 8,105
Vertices sharing a slot-3 value are marginally FARTHER apart than vertices
differing in it. The control halves the distance, so the measurement works; the
0.95x is on the wrong side of neutral, though by too little to claim much on its
own. Combined with no LOD sharing (75.3% against a 78.7% control), slot 3 is
per-vertex, uncorrelated with the geometry it sits in and with the detail level.
Recorded as not understood. A per-vertex animation phase fits the arithmetic and
is labelled a guess rather than offered as a reading.
Slots 6 and 7 are mostly not per-vertex. Distinct values per draw packet, median
over 1,804 records:
i16[0..2] 13-17 constant for the whole packet 0%
i16[4,5] 6, 4 0%
i16[6] 2 43%
i16[7] 4 34%
They take a single value across an entire packet in well over a third of records,
which matches the 91.8%/88.9% dominance of 4096 and 0. Unnamed.
So the tie element is: position, byte-sized index, texcoords, layout tag, and two
slots characterised but not named. Positions, texcoord placement, element size and
the layout selector are all asserted; the remaining three are not, and the reasons
they are not are now written down rather than left as silence.
On "is Phase 2 nearly done": 40 items closed, 9 open. The nine are the hard
ones -- collision cells blocked on the game's own code, volumes not located at
all, moby_anim ruled out but unexplained, chunk record fields unnamed. Progress
on the checklist is real but the remainder is not the easy tail.
|
||
|
ReaperOfVeriod
|
e86de3c73e |
Tie i16[0..2] are the vertex positions, with a control in the test
The three slots ahead of the index were "unproven position" for want of a test
that could fail. One is available on the disc alone, with the control built in.
LOD 1 and LOD 2 of a tie class are the same object at two levels of detail, so
anything that locates a vertex in that object's own geometry is shared between
the two blocks; a different class' LOD is an unrelated object. Intersecting the
sets of i16[0..2] triples both ways turns "is this a position" into a
measurement. Levels 0-5, ordinary-layout elements only:
same class, LOD1 vs LOD2 96.1% 107 classes, 107 nonzero
different class, control 0.4% 2,144 pairs
A factor of 240. Per level the same-class figure runs 94.4 / 97.9 / 98.1 / 96.7 /
96.8 for levels 0-4. The control stays under 1.0% except for one level-1 pair at
45.9% -- two classes sharing most of their geometry, which is what duplicate or
variant models should look like. Reported, not excluded, and the assertion is on
the mean rather than the max so it does not have to be.
Asserted in tie_vertex_slots_0_to_2_are_shared_positions, including the ratio
(same > 20 * diff) so a change that collapsed both numbers fails rather than
passing twice, and a nonzero-overlap check so no class can silently stop
agreeing with itself.
What this rules out is the point. An index, a handle or a per-vertex counter
would not be shared at 96% between two independently built LOD meshes, nor sit at
0.4% between two objects. With 10.7k-13.2k distinct values per slot over the
full i16 range, these are positions in the object's own frame. Scale and axis
order stay unnamed -- nothing here says centimetres, and axis order is
unconstrained because nothing here assumes one.
Two things worth recording about the process:
1. The diagnostic line printed "1.0%" for 99.4% because the format was
`{same:.1}%` with no `* 100`. The assertions compared fractions and were
always correct; only the readout was wrong. Same family as everything else in
this log -- the readout disagreeing with the claim.
2. Inserting the new test in front of an existing one stranded that test's
#[test] on my doc comment, which silently unregistered
tie_vertex_element_is_16_bytes_with_fixed_texcoords. It showed up as
"never used" in clippy, not as a missing test. Restored; both tie vertex
tests are registered and 239 tests run.
Element layout now reads:
+0x00 i16[0] position x +0x06 u8 index 0..=254
+0x02 i16[1] position y +0x07 u8 layout tag, 0 in 93.4%
+0x04 i16[2] position z +0x08 i16[4] texcoord u
+0x0a i16[5] texcoord v +0x0c/+0x0e i16[6], i16[7] unnamed
|
||
|
ReaperOfVeriod
|
06cf2fd46d |
Tie vertex slot 3 is a byte-sized index, not a coordinate
Splitting the 95,612 elements of levels 0-2 by i16[3]'s HIGH byte, which is the
layout selector:
0x00 89,341 93.4%
0x03 5,344 5.6%
144 other selector values 927 1.0%
146 selector values in all. Not a two-way switch, and the second case is not 0x01.
Inside the ordinary layout i16[3] is not a coordinate: 201 distinct values, all
within 0..=254, with 0 most common. That is an index. So the selector and the
index are not two independent halves sharing a slot -- it is one byte. The low
byte is the index, the high byte is the layout tag, and the ordinary layout never
sets the high byte. Asserted.
Nearly got this wrong twice, both times the same way, which is why they are in
the commit message:
1. The first pass gated the range on `w >= 0` instead of on `(w >> 8) == 0`,
letting in the other layouts' large positive values. That reported "564
distinct values in 0..=32664" -- a number that looks exactly like a
coordinate and is really two populations unioned. The selector byte is the
filter. The filter is part of the claim.
2. Two constants were guessed rather than measured (selector cardinality 159,
and an upper bound that ignored non-ordinary layouts) and both were wrong.
The cardinality is 146. Guessing a constant and then asserting it is how a
measurement becomes a tautology.
Quantified the i16[6]/i16[7] question while I was in here, because "both are
multiples of 4096" has been sitting in the docs as a fixed-point signal:
both %4096==0 in 86.5% of elements
expected if the two residues were independent: 84.3%
enrichment 1.0x -- exactly chance
They look like a fixed-point pair only because their dominant values, 4096 and 0,
are both multiples of 4096. Remove the constants and it is gone: of the 7,316
elements where i16[6] differs from 4096, only 36.7% are still multiples, and of
the 9,905 where i16[7] differs from 0, ONE is. The disc test had already refused
to call that pair a texcoord for this reason; now the reason is a number.
Remaining slots in the ordinary layout, recorded: i16[0..2] read as position but
are unproven (10.7k-13.2k distinct each, full i16 range), i16[4]/i16[5] are the
texcoords, i16[6]/i16[7] unnamed.
Also fixes two clippy map_or lints and an import that a stray --fix had broken.
238 tests, clippy clean.
|
||
|
ReaperOfVeriod
|
1fa1913e38 |
Reconcile the docs after the last three corrections
The last three commits changed things the documentation still asserted in the
old direction. Fixed:
roadmap
* 227 tests -> 238.
* The level_overlay bullet contradicted itself: it said "the header words are
not named" in the same breath as "+0x00 load base is 0x1A7C80". Now says
+0x00 IS the load base and is named, +0x0c is excluded as the .lit size by
sign (below file length on 16/27, by up to 8.9%) and remains unnamed, and the
unnamed remainder is stated as such.
* Section names aligned with the Phase 3 list (lvl.vtbl, lvl.camvtbl,
lvl.sndvtbl).
core-data-regions.md
* The claim-assertion audit still said the two PCSX2 claims were "NOT
assertable here -- needs a savestate". They are asserted now, gated on
GC_SSTATE, in the same file. Updated, with the two corrections that fell out
of writing them recorded right there.
* "that string is inside the collision routine" was never established. It is
in the string pool, yes; the reference to it is not found. Reworded to say
so, including the zero-hit result.
research log
* field_00 now carries its name and the .lit/$gp/program context.
* The field_0c falsification and the byte-order retraction are both recorded
here as well, so the log and the format file agree.
Nothing left asserting the withdrawn anchor, and no stale test counts.
|
||
|
ReaperOfVeriod
|
282e633a7c |
Withdraw the collision anchor: it was a byte-order bug in my own search
Last commit claimed exactly one GP-relative occurrence of the collision assert
string's $lo half, at 0x354cc8, and called it an unambiguous anchor. That was
wrong, and building the disassembler is what proved it.
The search compared word bytes 1..3 against the 16-bit immediate. The PS2's MIPS
is little-endian, so an I-type immediate sits in bits 15..0 -- word bytes 0 and
1. Bytes 1..3 are the opcode, $rs and $rt fields. The match was on those.
The instruction at the phantom hit is 0x24a89948:
addiu $t0, $a1, -0x66b8
whose real immediate is 0x9948 -- ten bytes from the 0x99a8 being searched for.
-0x66b8 merely looks like a plausible $gp offset, which is what made it
convincing. With the byte order corrected the count is ZERO, in 2,751,268 bytes.
So the anchor is not established and the assertion is withdrawn. What survives is
still worth having: the functions are named (Camera_CollPrimTest takes (prim,
grid)), $gp = 0x1AEFF0, .lit loads at 0x1A7C80, and the assert string is a unique
copy at overlay+0xd18.
Probably the code reaching that string does not address it through $gp, or
reaches a copy in another section. The decomp's assert-message-names.tsv gives
0x002db350 for level 0 -- inside .lit, but it disassembles to a function epilogue
(jr $ra / addiu $sp, $sp, 0x70), so those are code locations, not string
addresses, and they are not traced to the assert either.
tools/overlay-anchors now does the search with the correct byte order and
carries a comment on exactly the part that was wrong, because that is the part
that will be re-derived otherwise. It also disassembles around an address:
cargo run -q -p gc-overlay-anchors -- <overlay.bin> --disasm 0x2db350 0 64
The useful general lesson, which the roadmap already says and this is a second
demonstration of: a probe that returns a small, clean, narratively satisfying
number is the most dangerous kind of wrong result. "Exactly one hit in 2.75 MB"
read as conclusive. It was an artifact of comparing the wrong two bytes, and it
survived a commit because it fit the story.
|
||
|
ReaperOfVeriod
|
1f903efe23 |
An external decomp turns two overlay constants from measured into named
OpenRAC/rac2-gc-decomp is the same disc as ours, verified rather than assumed:
ours 3,828,350,976 9db2e33e...b1ce5
theirs 3,828,350,976 9db2e33e...b1ce5
Vendored read-only under vendor/ (already gitignored) and added to PROVENANCE.
It names two constants this parser had measured but never explained:
* overlay.bin +0x00 = 1,735,808 = 0x1A7C80, the .lit load base. It is constant
across all 27 levels because every level's literal section loads at the same
address. So the first header word is the load base of the section the file IS,
which also establishes that overlay.bin is that section -- it holds both code
and the debug string pool. $gp = 0x1AEFF0 comes with it, and that closes the
roadmap's open "$gp for this game" item.
The program around it is seven sections -- .lit .bss .data vtbl camvtbl sndvtbl
.text -- which is precisely the "seven-section parser and relocation" Phase 3
item for this format. Known target now rather than a guess.
I then tested the obvious follow-up and it is wrong: if +0x00 is the .lit load
base, +0x0c ought to be the .lit size. It is SMALLER than the file on 16 of 27
levels, by up to 8.9% (level11: file 3,151,128 vs 2,870,792). Excluded by sign,
not by a near miss. Still unnamed.
Collision: still blocked, but no longer blocked on tooling. The decomp names
Camera_CollPrimTest(void *prim, void *grid) without decompiling it, so the cell
layout is not handed over. What it does supply is $gp, and that is enough to
start. "Camera_CollPrimTest WARNING! - grid out of bounds! (RAR)" is at
overlay+0xd18, the only copy in the file, so:
string overlay+0xd18 -> RAM 0x1a8998; delta to $gp -0x6658; $lo 0x99a8
MIPS puts the $lo half in bytes 1..3 of a 4-byte instruction. Searching the whole
overlay instruction-aligned gives exactly ONE hit:
0x354cc8 (overlay+0x1ad048)
One occurrence in 2.75 MB of code is unambiguous: that is the instruction
materialising the assert string, and its enclosing function is
Camera_CollPrimTest. That is the route in, and it needs no symbols.
tools/overlay-anchors does this step, dependency-free. I started this with
capstone for the disassembly proper and dropped it: capstone 0.14 does not
re-export Mode/Endian from a public path, and an excluded-from-the-workspace
native dependency is a poor trade for a step that does not need one yet. The
anchor search is pure byte work, so it is pure Rust. No native dependency is
added anywhere, and the tool is an ordinary workspace member.
Surveyed the rest of that project against our open items: no volume structures,
no tie vertex layouts, no SBlk/DAW fields, no wavelet tables. It is a guide to
the code, not a replacement for reading it.
|
||
|
ReaperOfVeriod
|
26a0918178 |
Gate the PCSX2 claims as tests; two corrections fall out of writing them
The audit two commits ago listed three claims as "not assertable here -- needs a
savestate". With PCSX2 running they are assertable, so they now are. gc-trace
already exposed gc_trace::Savestate::ee_ram_owned(); gc-extract just needed it as
a dev-dependency. Both tests are gated on GC_SSTATE and skip cleanly without it,
so a normal checkout pays nothing.
Writing them corrected two claims, which is the main reason they were worth
writing as tests rather than staying prose.
1. The collision payload is NOT "byte-for-byte identical". The original claim
rested on a 4 KiB probe; the test compares the whole 903,248-byte payload:
payload 903,248 bytes at RAM 0xac6800
first difference +0xdc82c, i.e. 903,212 bytes in
bytes that differ 6 (0.00%)
903,212 of 903,248 bytes are verbatim and 6 are not, all in the final 36.
"Verbatim" was an overclaim on a probe that never looked at the tail. The test
now measures the whole payload and reports the extent, asserting only that the
divergence is beyond the head.
2. The tfrag relocation base is the TABLE address, not the block base. The delta is
8,683,584 = 0x848040 = block base 0x848000 plus table_offset 0x40 -- which is
exactly what TfragHeader::data's own documentation says, "relative to the
table". The earlier claim said "the block's RAM base", off by exactly the table
offset.
Also: +0x10 and +0x38 do NOT share one constant. The aggregate "1,665 fields
relocate by one constant" was true only because +0x10 carried it; lumping them
hid that. The test asserts +0x10 and REPORTS +0x38's delta rather than assuming.
The patch-rewrite figures also restated on aggregate rather than median, which is
why they read lower than the 7.80%/8.12% medians recorded earlier:
patch bytes rewritten -- cannot morph 1.9% of 334, can morph 2.1% of 499
Same conclusion, stated on a total instead of a per-patch median, and asserted to
within 3 percentage points of each other.
Net: the audit table in docs/formats/core-data-regions.md now has no "prose only"
rows for disc-or-savestate-readable claims.
|
||
|
ReaperOfVeriod
|
9f938f47de |
Assert the tie vertex layout finding in a test, and audit which claims are asserted
The i16[3] high-byte finding -- the most recent and most subtle result of this stretch, and the one I corrected twice -- lived only in prose. Two commits is exactly how long a claim can sit unasserted before it drifts. It now has a test: tie_vertex_layout_is_selected_by_i16_3_high_byte which reproduces the figures exactly and, importantly, asserts both directions: (4,5) and (6,7) are above 0.85 and 0.80 in the ordinary group, (6,7) is BELOW 0.99 so it cannot silently become the constant I once mistook it for, (0,1) and (2,3) are under 0.20, and every pair in the non-zero group is under 0.15. A future change in either layout now fails loudly. Also audited the rest of this session's claims against the test suite and recorded the result in the format doc: tie vertex element is 16 bytes asserted layout selected by i16[3] HIGH byte asserted (this commit) blocks self-delimiting on 0xf6 asserted per-block packets == 0 means no block at that LOD asserted packets[0] == 256 on 1,864 of 1,864 asserted chunk record length uniform per chunk asserted moby class IDs in one body slot per type asserted collision payload verbatim in RAM NOT assertable here -- needs a savestate tfrag patch bytes all rewritten by LOD same morphing is unobservable in that image same The split is deliberate: disc-readable claims are asserted against the disc, savestate-dependent ones live in plan/pcsx2-verification.md with their method. What is not acceptable is a disc-readable claim living only in prose -- which is how the tie i16[3] finding spent two commits before it had a test, and how several older claims drifted before the roadmap reconciliation caught them. |
||
|
ReaperOfVeriod
|
922248a4ec |
Tie: the element layout differs by i16[3] high byte, and (6,7) is a real pair
Two results, one of them a correction.
1. "(6,7) is just the constant 4096/0 pair, not a texcoord" -- wrong. It looked
like an artifact because a constant is trivially a pair of multiples of 4096.
But partitioned by i16[3]'s high byte it is a genuine fixed-point pair in its
own right:
i16[3] high byte ZERO (n = 89,341) divisor rate by slot pair
pair /1 /4 /16 /256
( 0, 1) 100.0% 11.5% 4.6% 3.3% baseline
( 2, 3) 100.0% 11.0% 4.5% 3.3% baseline
( 4, 5) 100.0% 92.9% 92.0% 91.8% the texcoords
( 6, 7) 100.0% 89.5% 87.3% 86.5% a SECOND fixed-point pair
If (6,7) were a constant it would score 100%, not 89.5%. The constant is the
common case WITHIN it, not the whole of it -- which is exactly the distinction
the constant-artifact lesson is about, and I applied it backwards last time.
So the ordinary element carries TWO fixed-point pairs, (4,5) and (6,7), with
(0,1) and (2,3) at baseline. What (6,7)'s role is remains open -- second UV
set, normal, or a zero-heavy fixed-point pair all fit.
2. The element LAYOUT differs between the three cases. Scoring all four slot
pairs within the non-zero high-byte group, no pair stands out:
i16[3] high byte NON-ZERO (n = 6,271)
pair /1 /4 /16 /256
( 0, 1) 100.0% 4.6% 3.5% 0.7%
( 2, 3) 100.0% 2.0% 0.6% 0.0%
( 4, 5) 100.0% 1.9% 0.3% 0.1%
( 6, 7) 100.0% 7.5% 5.7% 4.5%
Best is 7.5% against a 2-5% baseline. So for those 6,271 vertices the 16 bytes
are not laid out as position/texcoord/normal at all -- a different element
shape, selected by the high byte.
This also explains why the global slot-pair score looked ambiguous to begin with:
pooling the two layouts averages a 92% pair and a 7% pair into one number that
describes neither. Fourth time this session that partitioning produced the answer
rather than merely bounding it.
|
||
|
ReaperOfVeriod
|
ed3e85a0f2 |
Refine the tie texcoord finding: it is i16[3]'s HIGH byte that selects
The previous commit said i16[3] selects the texcoord encoding. That was right but loose -- it is specifically the HIGH BYTE, and the low byte is a separate thing. Grouped by the high byte across the same 95,612 vertices on levels 0-2: i16[3] high byte n % / 16 % / 256 0x00 89,341 92.0% 91.8% 0x03 5,344 0.0% 0.0% 0x04 71 0.0% 0.0% 0x40 58 5.2% 0.0% 0x02 47 2.1% 2.1% 0x19, 0x05, 0x20, ... the remaining 73 A high byte of zero is the ordinary case -- 89,341 of 95,612, 93.4% -- at 92%. EVERY other high byte scores at or near zero. The split is on the byte, not on the value: i16[3] of 1 and of 771 differ only in the high byte and behave completely differently. Within the all-zero case there is a third distribution: i16[3] == 0 exactly (5,851 vertices) scores 62% at 16 and 61% at 256, against 92% for 1..255. So three cases, selected by the high byte and then the low: high 0x00, low 0x00 5,851 a third distribution, 62% at 1/16 high 0x00, low 1..255 83,490 the ordinary case, 92% at 1/4096 high non-zero 6,271 0%, an encoding this reading does not see The 6,271 matches the figure quoted last commit for "vertices using an encoding this reading does not see", and the two group counts sum consistently with the committed test's 95,612 -- so this refines the finding rather than changing it. A measurement trap worth recording: the all-zero case is a SUBSET of high byte 0x00, so summing the per-high-byte groups double-counts it. An intermediate total came out at 100,792 against the true 94,941 for those groups; the committed test's figure is the one to trust. |
||
|
ReaperOfVeriod
|
e27c40e738 |
Explain the tie texcoord residual: i16[3] selects the encoding
The texcoords are 12-bit fixed on 82.5% of vertices, not 100%, and the residual had no explanation. Grouped by the vertex's i16[3] slot across 95,612 vertices on levels 0-2, the buckets separate completely: i16[3] n % / 4096 % / 256 negative 333 1.2% 1.2% == 0 5,851 29.2% 60.7% 1..63 28,884 92.7% 94.3% 64..255 54,606 92.3% 93.8% 256..511 8 0.0% 0.0% 512..1023 5,391 0.0% 0.0% >= 1024 539 0.2% 0.2% The residual is fully accounted for. 87.3% of vertices sit in 1..255 at about 92.5% fixedness, contributing 0.873 * 0.925 = 80.8 points; the 5,851 at i16[3] == 0 contribute 1.8 more. That is the 82.5% measured, to the point. The remaining ~6% are not 12-bit fixed at all. i16[3] >= 256, and negatives, score 0% at BOTH 4096 and 256 -- 6,271 vertices in an encoding this reading does not see. And i16[3] == 0 scores 29.2% at 4096 but 60.7% at 256, which is a third distribution rather than a weaker version of the same one. What this establishes: i16[3] is a per-vertex packed attribute whose value SELECTS HOW the texcoord pair is stored, rather than an independent field sitting beside the texcoords. 1-255 is the ordinary case, 0 is special, 256-and-up and negatives are a different encoding. What those three cases MEAN is still open -- texture index, normal index, flag or a combination -- but it is now a field whose role is known, which is more than "packed word". Worth noting the shape of the claim this replaces: "the texcoords are 12-bit fixed at i16[4]/i16[5], 73.6% of the time, unexplained". The field position was right; the 73.6% was a population average over three encodings. Partitioning by the one variable that separates them turns a mystery into a lookup -- rule 3a again, and the third time this session that partitioning a mystery produced the answer rather than merely bounding it. |
||
|
ReaperOfVeriod
|
3ee0b3d0dc |
Disprove that morphing causes the tfrag byte changes: LOD selection does
Read the whole tfrags block from RAM -- 4,379,200 bytes at 0x848000 -- and compared
every one of the 833 patches against its disc copy, grouped by whether it CAN morph:
patches with NO morphing entries 334 differing: max 10.59%, median 7.80%
patches WITH morphing entries 499 differing: max 10.49%, median 8.12%
patches 100% identical to disc 0/334 and 0/499
The two groups are indistinguishable. Morphing is NOT what changes the bytes, and
this disproofs the claim two commits ago that the 8.6% was "morphed positions with
topology and UVs left verbatim". If it were, the 334 patches with no morphing
entries would be untouched -- and none of them is.
Nor is it a pointer relocation: the differing bytes fall in short structured runs
(257-263, 272-277, 305 on patch 0) whose u32 deltas share no common value.
The explanation consistent with everything measured is that the loader REGENERATES
each patch's VU packet stream for the LOD tier it selected. That happens to every
patch, morphing or not -- and it is why morphing shows no signal.
This also settles the verification route, which the previous commit left as "needs
the camera distance at the savestate". It does not, and no camera distance would
help: this memory image shows the LOD-selected form rather than the morph state,
so there is nothing in it to solve a weight against. Verifying the derivation needs
a captured frame at a known view distance instead.
What is established now, all measured on the full population rather than inferred
from one patch:
- data and cube_ofs are relocated pointers, delta 0x848000, on all 833 records
- every patch's bytes are rewritten, ~8%, by LOD-driven packet regeneration
- morphing is not observable in this memory image
- 833 of 833 patches parse; 499 of 833 have morphing entries; 43,613 vertex-info
entries across the level
|
||
|
ReaperOfVeriod
|
aff977e8c5 |
Correct the tfrag pointer-field claim, and show morphing regenerates VU packets
Two corrections to the previous commit, both from continuing the RAM comparison.
1. The previous commit said the RAM comparison "names two fields the disc-side
work had left as data and an unnamed word". It does not. +0x10 is
TfragHeader::data and +0x38 is TfragHeader::cube_ofs, both already named in the
struct. What RAM adds is that both are POINTERS -- relocated by adding the
block's RAM base -- rather than plain offsets. That is still worth knowing, but
it is not a naming.
2. The previous commit attributed the 8.6% of rewritten bytes to "morphed
positions with topology and UVs left verbatim". That is wrong, and patch 1 shows
why. Comparing 4 KiB, disc 0xdf60 against RAM 0x855f60:
bytes differing 354 of 4,096 = 8.6%
contiguous runs 120
longest run 8 bytes
extent +0xc1 through the end of the window
Patch 1 has 83 positions and 83 vertex-info entries, of which 13 morph. If
morphing rewrote position fields alone, the differing bytes would be a 13/83 slice
of a 498-byte position array -- a couple of dozen bytes in one contiguous region.
Instead 354 bytes differ, spread over 120 short runs across the whole window.
So the loader is not patching positions in place: it REGENERATES the VU packet
stream for the current morph state. That is what morphing is on this hardware --
VU1 GIF packets, not a CPU-side vertex array.
Consequence for verification, and the reason this is worth committing rather than
just noting: the plan of "map RAM vertices to disc indices and solve for the weight"
does not work against this memory image. The positions are inside regenerated
packets, so there is no stable per-vertex correspondence to read a weight out of.
Verifying the morph weight against the original needs the CAMERA DISTANCE at the
savestate, so the weight can be computed from the derivation and compared with the
rendered result, rather than read back out of RAM.
Population figures gathered on the way: 833 of 833 patches parse, carrying 43,613
vertex-info entries; 499 of 833 patches have at least one morphing entry, and
patch 0 -- the first record -- is base-only with none.
Also fixed a measurement error mid-run: I compared patch 1 at disc 0xdf60 against
RAM 0x855f00, off by 0x60, and got "every byte differs". The corrected base is
0x855f60.
|
||
|
ReaperOfVeriod
|
93591a79de |
Verify the tfrag record framing against RAM: two pointer fields, morphed vertices
The tfrags block header -- table_offset 0x40, tfrag_count 833, lod_distance 32.0 -- hits once in the level-0 savestate at RAM 0x848000, so disc offset 0 maps to RAM 0x848000 and the block can be compared position for position. Result 1: exactly two fields per record are rewritten. Comparing all 833 64-byte table records field by field: records compared 833 u32 fields differing 1,665 = 833 x 2, and no others their offsets +0x10 and +0x38, on every record relocation delta 8,683,584 = 0x848000, on 833 of them So +0x10 and +0x38 are block-relative pointers and the loader adds the block's RAM base to each. Every other byte of all 833 records is verbatim. Six fields differ by 0xF80000 and 0x1F00000 instead -- three records pointing into a different buffer, recorded rather than averaged. This names two fields the disc-side work had left as `data` and an unnamed word. Result 2: behind the pointers, about 8.6% of bytes are rewritten. Following record 0's +0x10 to disc 0xd040 / RAM 0x855040 and comparing 16 KiB: 14,977 bytes identical, 1,407 differ = 8.6%. The differences spread across many field positions rather than concentrating, which is what morphed POSITIONS look like -- topology, UVs, counts and command lists stay verbatim while a fraction of the position bytes move. 8.6% is about half of a 16-byte vertex's six position bytes, consistent with roughly half the vertices in this patch mid-morph at the saved camera distance. Not established: that the morphed bytes match resolve_vertex's formula. That needs mapping RAM vertices to disc vertex indices and solving for the weight, which is the next step -- but the framing is now known well enough to attempt it, and the two pointer fields are confirmed rather than inferred. This also answers the "is morphing real?" question that made the earlier tfrags probe fail to match RAM: the tfrags block is not verbatim in RAM, because both its table pointers are relocated and its vertex positions are morphed. |
||
|
ReaperOfVeriod
|
6d0c1b6fd1 |
Close out tfrag on the disc side, and fix two stale "not recovered" claims
Picked tfrag morph targets from the open list and found the premise was wrong. There are no morph targets in this format. Morphing is a blend toward the midpoint of the vertex's two PARENT vertices -- g[parent1_position] and g[parent2_position] -- which are other vertices in the same tfrag. The engine implements exactly that in resolve_vertex, and it is covered by morphing_blends_toward_the_parent_midpoint plus morph_weight_runs_from_zero_to_one_across_its_band and collapse_distances_are_the_far_edge_of_each_band. So "morph targets unrecovered" was never a gap; it named a thing that does not exist in the format. Strip bookkeeping is likewise already there: TfragStrip carries vertex_count_and_flag, and tfrag_triangles walks the strip command lists and index arrays to build triangles. That was the other half of the same roadmap entry. Coverage, all passing: 36 tfrag tests, 9 texture tests, 23 gc-engine tests. The real-disc texture test decodes all 1595 level textures, with pixels and mip levels, so the module doc's "not yet recovered: the texel data the texture table points into" is also stale. Two stale claims corrected: 1. crates/gc-formats/src/tfrag.rs said "Not yet recovered: the LOD switch and morph rules in TfragProc, and the texel data the texture table points into. See docs/plan/tfrag.md." The rules are recovered and tested, the texels are decoded, and docs/plan/tfrag.md DOES NOT EXIST -- a dangling reference to a file that was never written. 2. docs/roadmap.md marked tfrag partial with "Still open: morph targets and the strip/chain bookkeeping". Neither is open. The honest replacement for both is what is genuinely not claimed: the LOD switch distances, morph weight and collapse edge are DERIVED from the data -- tiers are classified from the four V4_8 parent arrays, the weight runs zero to one across each tier's band -- but have not been checked against PCSX2 the way the lighting pass was. That is a verification gap, not a parsing gap, and it needs a frame captured at a known view distance rather than more disc-side measurement. |
||
|
ReaperOfVeriod
|
d56b439502 |
Exclude the collision segmentation theories on the full population, and name the
unblock Continuing on the verbatim payload with the grid known. Cells are not zero-separated. Counting maximal runs of non-zero u16 across the whole payload: level 0 363 cells 15,115 runs mean run 25.3 u16 level 1 384 cells 24,647 runs mean run 51.5 u16 level 2 53 cells 31,512 runs mean run 30.3 u16 Two to three orders of magnitude apart, so a zero run is not a cell boundary. Cells are not uniform blocks either: level 0 averages 2,488 bytes per cell and level 2 averages 43,684, which cannot both be one fixed-size record. The run lengths are not records. Their histogram has a long tail of near-power-of-two values -- 2,140, 1,184, 840, 816, 628, 604, 562 -- which is what you get when a run is bounded by a single zero byte inside otherwise dense data, not by a delimiter. With the 18%-zeros finding, the payload reads as dense data with sparse holes rather than a delimited sequence. So all three segmentation theories are now excluded on the full population rather than on a sample: not ascending u32 tables, not zero-run delimiters, not fixed stride. What would actually unblock this: overlay.bin IS the game's own code, and the collision parser is in it. Camera_CollPrimTest WARNING! - grid out of bounds! (RAR) was located in the debug string pool earlier, and that string is inside the collision routine. So the format is recoverable -- but not by measuring bytes. It needs disassembly of that routine, or an R&C2 reference for the collision cell record. Stating that plainly is worth more than another scan. The disc-side measurement space for collision is exhausted: the bytes are available, verbatim, the grid is known, and three independent segmentation theories have failed on the whole population. Further probing without the loader's own logic will not converge. |
||
|
ReaperOfVeriod
|
338af9b7bd |
Fix the collision grid dims, and disprove two sample artifacts in the payload
With the payload confirmed verbatim in RAM, the disc bytes can be read directly.
The grid dimensions are the first two u16 of the payload: level 0 is 11 x 33 =
363 cells, level 1 24 x 16 = 384, level 2 1 x 53 = 53. Level 2 being one cell wide
is worth noting -- the grids are not all similar in shape.
Two readings disproved, both by measuring the whole payload instead of a sample.
An ascending u32 offset table. A probe from payload +1024 reads f4ae0000 40af0000
c4b2000020b20000, which is three ascending values before the fourth goes down
(0xb2c4 to 0xb220). Read as a table, that was a three-value impression. Scanning
all three levels for maximal ascending u32 runs of 8 or more entries finds zero,
on every level.
A global alternating-zero stride. The head of level 0 shows an obvious (value, 0)
pattern from about i16[37]:
[32..] [1808, 1872, 2198, 0, 34, 86, 25848, 1, 25824, 1,
-13028, 0, -12284, 0, -12228, 0, -12716, 0, ...]
Measured over the whole payload, both alignments score far below what that sample
suggests: even-slot aligned 18% second-half zero, odd-slot aligned 13%. The
sample again.
What the global numbers do say is that the payload is heavily zero-bearing -- 18%
against a chance rate of about 0.0015% -- which is what a sparse or variable-length
encoding looks like, and consistent with the variable-length reading established
earlier. The 18-against-13 split is a real alignment preference but a weak one.
The cell layout is still undecoded. The useful position now is that the bytes can
be read as they are and that they are sparse and variable-length rather than
fixed-stride, so the next attempt should look for a terminator-delimited walk
through the sparse region rather than for a record size.
|
||
|
ReaperOfVeriod
|
d5e637930b |
Show the collision payload is verbatim in RAM, and retract the "not in RAM" claim
This overturns the premise collision work was blocked on. Searching the level-0 savestate for a probe from collision payload +1024 hits at 0xac6c00; a probe from +64 hits at 0xac6840. The difference is 0x3C0 = 960, which is exactly 1024 - 64, so the payload sits at a fixed relative address -- RAM 0xac6800 for the disc address 0x27e800 -- and reading it back is byte-for-byte the disc's own bytes: RAM 0xac6800 0b 00 21 00 b6 08 ad 08 a4 08 6b 08 38 08 63 07 disc 0x27e800 0b 00 21 00 b6 08 ad 08 a4 08 6b 08 38 08 63 07 Nothing is reshaped. That is the opposite of the tie block, where the loader relocates a pointer inside each record. The retraction. The previous note reads: "level 1's 24 x 16 grid dimensions appear in memory, level 0's 11 x 33 appear nowhere, so disc-side decoding has to work against the reshaped form." Level 0's dimensions ARE in RAM, at 0xac6800 -- 0b 00 and 21 00 are the first two bytes of the block, 11 and 33. The search that missed them looked for 0b 00 00 00 21 00 00 00, the dimensions as a pair of u32s. They are stored as two u16s, so that pattern never occurs. So the negative result was an artifact of the search pattern, not evidence about the data -- and it was then used as a premise for several rounds of disc-side work. Both lessons were already written down in AGENTS.md and both were re-learned: a negative from a byte search bounds the search, not the format; and the control here is one command -- read the predicted RAM address and compare it to the disc -- which would have shown the verbatim copy immediately. Also recorded while mapping it: the payload contains ascending u32 offset tables from +1024 onward, and its first two bytes are the grid dimensions, so level 0's grid is 11 x 33. The cell layout itself is still undecoded, but the blocker is gone: disc-side decoding no longer has to work against a reshaped form. |
||
|
ReaperOfVeriod
|
180fc27b04 |
Locate the tie texcoord field at i16[4]/i16[5], and rule out the (6,7) artifact
Scoring each adjacent i16 slot pair of the 16-byte tie vertex for 12-bit fixedness over 36,501 level-0 vertices locates the fields: (i16[0], i16[1]) 1.1% not a texcoord (i16[2], i16[3]) 4.8% not a texcoord (i16[4], i16[5]) 73.6% <- the texcoords (i16[6], i16[7]) 76.6% <- the near-constant 4096/0 pair (6,7) scores HIGHER and is not a texcoord. It is the near-constant 4096, 0 pair: 4096 is 1.0 in 12-bit fixed and 0 is zero, so both are trivially multiples of 4096. That is the constant artifact again -- a fixed pair would score 100% anywhere. The field is (4,5), with (0,1) and (2,3) ruled out at 1.1% and 4.8%. The first four slots span the full i16 range with 10,803 to 13,421 distinct values each, so they are neither texcoords nor small fields. The vertex base is confirmed to be vert_ofs alone. Adding the shader or control region makes it worse -- vert_ofs only 73.6% bfc + vert_ofs 69.8% bfc + control_count*control_size + vert_ofs 66.7% -- so the 73.6% residual is not a mislocated base. Worth recording because vert_ofs (3..7) numerically overlaps the shader region (5*shader_count, 5..20 quadwords), which made a combined base look plausible; measurement says the simple one is right. The test asserts (4,5) above 60% and (0,1)/(2,3) below 5%/15%, and names the (6,7) artifact in its failure message so the higher-scoring pair cannot be mistaken for the answer later. |