06531fb66c7cd1c07e62cc1304e864b28ee3ba00
60
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 ReaperOfVeriod
|
06531fb66c |
The 394 does not survive: 0xc0's rate rises with immediate-set size
Ran the size-independent statistic |
||
|
ReaperOfVeriod
|
db1e69c0ab |
Ranking the 394 by size is backwards; record the confound
Ranking the moby_classes candidates largest-first is the wrong first cut. The top entries are 34,316 and 26,432 bytes with 631 and 421 distinct immediates, median candidate 852. A function with 631 distinct immediates is far more likely to contain any given value, 0xc0 included, than one with 80, purely by pigeonhole. Sorting by size therefore selects for the functions most likely to be false positives, and the 34,316-byte outlier is probably a mis-bounded extent -- the decomp corrects three level-00 entries -- rather than a 34 KB loader. The 394 needs a size-independent statistic: whether a function contains 0xc0 more often than its immediate count predicts, not whether it contains it at all. Not measured. Recorded as a negative about the method rather than committed as progress. No test -- there is no assertion here to make. 283 passing with the disc present, clippy clean. |
||
|
ReaperOfVeriod
|
3ea5ab799f |
moby_classes.offset is referenced by 394 functions; its neighbours by 1 to 14
First positive result of the loader hunt, and it exists only because of the
control that two commits ago I said was missing.
Across 2,323 level-0 functions of at least 128 bytes -- extents from the decomp's
functions.tsv -- the immediate 0x00c0, which is moby_classes.offset, appears in
394 functions. Every value adjacent to it appears in 1 to 14.
0xb8:8 0xb9:2 0xba:2 0xbb:2 0xbc:3 0xbd:5 0xbe:5 0xbf:3
0xc1:3 0xc2:3 0xc3:2 0xc4:1 0xc5:2 0xc6:3 0xc7:1 0xc8:14
This is what
|
||
|
ReaperOfVeriod
|
59957a02a2 |
Attribute the 0xc0 immediate sites to functions: 410 candidates
The decomp's functions.tsv makes this cheap -- 3,835 level-0 functions with start
and size, so each immediate-bearing instruction is attributable to its container.
410 of 3,835 functions (10.7%) contain one; sizes 20 to 34,316, median 808.
Ten per cent is what a common small immediate buys, so this confirms
|
||
|
ReaperOfVeriod
|
56b00fb305 |
Fix the immediate scan by filtering on opcode; still no signal
Saying the previous method 'cannot tell' was right but the reason deserved a step
rather than a hand-wave: it is fixable by counting immediates only where they are
immediates, which needs an opcode filter and no disassembler.
All 27 levels: 11,911,320 instructions, 2,985,897 carrying an immediate
(addiu/lui/ori/xori/andi/sltiu), 13,813 distinct immediate values.
moby_classes 0x00c0 17,699
tie_classes 0x1e80 17
And now there is a control the raw version never had -- the program's most
frequent immediates are 0x0001 (200,784), 0x001b (149,259), 0x0010 (137,523),
0xffff (98,312). 0xc0 at 17,699 sits below the top of that small-value cluster
rather than standing out, so it is not a distinctive reference.
Result unchanged from the raw scan and now defensible: no signal. The loader is
not findable by frequency, and the remaining route is structural dataflow --
functions touching both the chunk offset and a class span.
No test: asserting that an offset is unremarkable is not an invariant worth
locking, and the two numbers are already recorded with their denominator.
|
||
|
ReaperOfVeriod
|
27891ce5b4 |
Cross-reference scan by raw immediate: the method cannot tell
Sought the moby class loader by counting raw u16 occurrences of each chunk offset in .text. moby_classes 0xc0 hits 49,230 times, tie_classes 0x1e80 hits 111 -- and that contrast means nothing. Small u16 values are simply common: the most frequent value in .text occurs 1,834,535 times. A two-byte value that is frequent by construction beats a rarer one regardless of what either references, so the comparison has no shared denominator. Rule 3a again, one level below where it has bitten before -- not the measurement being wrong but the comparison being uninterpretable. Recorded as a methodological negative rather than a finding. Doing it properly means counting immediates only where the instruction decoder says they are immediates, which is what tools/overlay-anchors exists for. No test added: a test asserting a meaningless comparison would make the next reader trust it. |
||
|
ReaperOfVeriod
|
743107e864 |
moby_anim: the class table is not the loader
Ruled out the cheap approach: moby_classes (offset 0xc0, count 238 in level 0) holds float-shaped values, and only 120 of the first 4,096 u32s land inside the overlay .text against ~259 expected by chance. Fewer than chance, so the loader is not dispatched from a pointer here. Not concluded what the 238 entries are -- the float reading is suggestive but was taken at one stride on one level. |
||
|
ReaperOfVeriod
|
6d26da3630 |
The control fails my own claim: 0x1c/0x34 select nothing distinctive
|
||
|
ReaperOfVeriod
|
2ff2abbd32 |
Two class-header words are base-relative offsets pointing past the tie packets
Chasing where the AD-GIF is, since |
||
|
ReaperOfVeriod
|
f43a1c2e6e |
bfc_distance is not the control region's length; the AD-GIF is not in the packet
Follows
|
||
|
ReaperOfVeriod
|
9bb04cca7b |
The tie control region is not a VIF or GIF tag stream; and vert_ofs is mis-named
Went after the GS's GIF packets in the tie AD-GIF region. Got a negative result,
a new universal invariant, and a retraction of my own first probe, which reached
the right answer for the wrong reason twice.
THE NEGATIVE RESULT
Tried the leading `5 * shader_count` quadwords of each packet (the
`bfc_distance` field's stated meaning) as VIF tags -- command in bits 0-6, size
in 7-14, mark in 15 -- and as GIF tags -- type in the top byte of word 1, length
in bits 0-2 of word 0. Both uniform. Over 13,042 records: 10,443 distinct values
of word 0, no GIF tag type above a count of 2. That is the control result rule 4
asks for, and it says stop.
THE RETRACTION
The first probe scanned `bfc_distance` quadwords, which for most records
overruns `vert_ofs` and runs into the vertex data. Pooling a control region with
vertex positions -- two populations with different structure -- gives a uniform
answer, so "not a tag stream" came out right from a measurement that was
measuring the wrong bytes. That is exactly the rule 3a failure, and I only caught
it because the hex dump I took next looked structured while the histogram said
random.
Re-measuring quadword 0 alone, on its own population, gives the same uniformity.
The conclusion survives; the reasoning did not, and both are recorded.
THE PARTITION, AND A NEW INVARIANT
The hex dump showed the packet layout directly:
q[0 .. vert_ofs) control 3..8 quadwords, peak at 5
q[vert_ofs .. +vert_size) vertices
q[vert_ofs+vert_size ..] scissor
and `scissor_ofs == vert_ofs + vert_size` holds on **15,219 of 15,219** records
across all 27 levels, including every one whose strip_count is non-zero. That is a
derived quantity stored explicitly, and it holds everywhere -- which is the check
that confirms the three regions really are laid out the way the existing packet
length formula assumed. It is now a hard check in invariants_hold(), which is
something the previous four could not be.
The fixture in tie.rs failed the moment I added it: it had vert_ofs 4 and
vert_size 0x20 with scissor_ofs 0x30, which is 4 + 0x24. The synthetic data was
never constrained by a relation that did not exist yet. Fixed to 0x24.
vert_ofs IS MIS-NAMED
It is not a vertex offset. It is the *length* of the leading control region, and
the vertex data begins at that many quadwords into the packet. The field name
stays because the on-disc order is fixed and everything already written about this
record says vert_ofs, but the doc now states what the value is and gives the
distribution.
STILL UNKNOWN
What the control region contains. Not standard DMA tags; possibly pre-merged GS
register writes in another framing, or per-class state rather than per-primitive
draw data. Recorded as an open question rather than guessed at.
23 + 19 + 58 + 156 + 22 = 278 passing with the disc present (disc binary 35.67s),
clippy clean.
|
||
|
ReaperOfVeriod
|
1fa1913e38 |
Reconcile the docs after the last three corrections
The last three commits changed things the documentation still asserted in the
old direction. Fixed:
roadmap
* 227 tests -> 238.
* The level_overlay bullet contradicted itself: it said "the header words are
not named" in the same breath as "+0x00 load base is 0x1A7C80". Now says
+0x00 IS the load base and is named, +0x0c is excluded as the .lit size by
sign (below file length on 16/27, by up to 8.9%) and remains unnamed, and the
unnamed remainder is stated as such.
* Section names aligned with the Phase 3 list (lvl.vtbl, lvl.camvtbl,
lvl.sndvtbl).
core-data-regions.md
* The claim-assertion audit still said the two PCSX2 claims were "NOT
assertable here -- needs a savestate". They are asserted now, gated on
GC_SSTATE, in the same file. Updated, with the two corrections that fell out
of writing them recorded right there.
* "that string is inside the collision routine" was never established. It is
in the string pool, yes; the reference to it is not found. Reworded to say
so, including the zero-hit result.
research log
* field_00 now carries its name and the .lit/$gp/program context.
* The field_0c falsification and the byte-order retraction are both recorded
here as well, so the log and the format file agree.
Nothing left asserting the withdrawn anchor, and no stale test counts.
|
||
|
ReaperOfVeriod
|
d8b8ccc6d4 |
Follow the gameplay.bin pointers: they address position and direction arrays
Where the pointers go. Following them shows what gameplay.bin indexes. The leading table is 39 words, all of which land inside the file, and three clusters dominate: near 150000, near 800000 and near 1050000. The region near 797808 is position data -- 31 consecutive plausible floats that group cleanly into xyz triples: 246.948 201.015 73.867 | 88.240 186.000 302.598 | 107.712 12.723 183.125 353.591 126.365 19.962 | 182.068 374.001 122.567 | 116.961 265.001 437.866 and the region near 797456 is direction data -- unit-length vectors: 0.969 0.945 0.733 | -0.120 0.433 0.250 | -0.866 0.000 0.267 So the file's leading words are a table of pointers into float arrays, not a flat list of small integers. The earlier reading of it as "u32 pointers holding nothing" was too quick: it is a pointer table into data that is present. Volumes remain unlocated. The position arrays are the right shape to bound a volume and nothing yet shows the volume records themselves. |
||
|
ReaperOfVeriod
|
b17ec17127 |
Rule out the moby_anim instance-table reading of the chunk types
moby_anim was the one post-shrub chunk type with a concrete reason to exist: an instance table would have to scale with the level's moby class count, so comparing per-type record counts against that count was the obvious test. It fails, and the way it fails is itself the result: per-type record counts are level-independent. The same types report the same counts on every level tried: type_id level0 level1 level2 level3 0x007d7ec438 90 90 90 90 0x00d3d8c4a9 90 90 90 90 0x00429ec43d 77 77 77 77 0x0053c8c4aa 77 77 77 77 0x007626c4a4 50 50 50 50 0x00fb88c462 47 47 47 47 Across all 27 levels there are 77 distinct chunk types, the largest at 2430 records total -- exactly 90 x 27. What that rules out: nothing in this chunk set is a per-moby-class instance table, because those counts would vary by level the way the moby class counts do. So whatever holds moby animation data is either keyed by something other than the class count, or it is not among these chunk types. Also visible: the types pair up. 0x007d7ec438 and 0x00d3d8c4a9 both report 90 on every level; 0x00429ec43d and 0x0053c8c4aa both report 77. Two chunk types carrying identical, constant record counts on every level is a signature of two views of the same fixed table rather than two independent tables that happen to agree. Still open: the pairings are described but not explained, and no type has been named. The Moby class-ID enrichment from earlier stands separately -- 9493 hits against 246.1 expected is real, and the class arrays are contiguous per type, but that identifies where classes are referenced, not which chunk is moby_anim. |
||
|
ReaperOfVeriod
|
02ced5ee83 |
Parse the per-level overlay: header, constant block, level data
level_overlay was the least-characterised file in the level set -- no parser, no located structure. It now has gc_formats::overlay and a real-disc test over all 27 levels: overlay: 27 levels; field_00 1735808, field_08 1; field_04 37540..=45308, field_0c 2763912..=3027336 (17 below the file length, 10 above); +0x10..0x1220 identical on all The header is four words. +0x0c is not the file length: it was recorded as one on the strength of the first levels measured, where it sat near the file end, but across all 27 it falls on both sides -- below on 17, above on 10 -- so it is an ordinary level-specific word. The test asserts this, being the claim most worth pinning. The common prefix is 4 bytes, not the whole header: +0x00 is constant at 1735808 while +0x04 and +0x0c vary. Everything from +0x10 to +0x1220 -- 4624 bytes -- is byte-identical on all 27 levels. Still not named: what any of the four header words means, and how the level-specific tail is structured. The module reports them as fields. Also records in docs/plan/pcsx2-verification.md that the 64-byte RAM stride from the previous commit is disproved. A 415 KB dump of level 1 finds the longest such run at 27 records, not 384, and those are identity matrices -- 27 of them, one per level. It is a per-level transform table, not the collision grid, and the 20.0 reported there was a coincidence. |
||
|
ReaperOfVeriod
|
d3e1475f64 |
Characterise gameplay.bin as a flat pointer table, and record volumes as unlocated
Range measured: the first 40 u32 and the whole file, on levels 0 and 1 --
1872256 and 2006400 bytes.
Result: the file is a flat array of u32, opening with a small count then a run of
ascending offsets, terminating on a zero:
level 0 336 797472 798832 799136 464 150672 150688 314800 479920 ...
... 1847600 1845456 797808 1829056 160 1849088 1249808 1847616 0
level 1 496 799904 800128 800432 624 151152 151168 315680 481456 ...
... 1984496 1982400 800112 1966000 160 1988160 1303456 1984512 0
Both levels carry the same shape. The sentinel 160 appears at u32 index 34 on
both and the zero terminator at index 38 on both, with counts 336 against 496
and 464 against 624 at the head. The leading ~38 words are a structured header
whose layout is level-independent apart from its sizes.
The offsets cluster into groups -- near 800000, 150000, 1050000, 1800000 --
which is what a table of pointers into several buffers looks like.
Not decoded: what the pointers point into, what 160 is, and why the zero
terminator sits at the same index on both levels. No gc_formats module exists for
this yet; it is characterised, not parsed.
On volumes: nothing resembling volumes was found here. gameplay.bin is u32
pointers throughout and holds no float data in its first 64 words read as f32 --
59 of 64 are denormals, i.e. small integers reinterpreted. Volumes remain
unlocated on the disc; nothing in the regions examined so far presents itself as
one.
|
||
|
ReaperOfVeriod
|
575d1a5bdd |
Correct the overlay strings from a table to a pool
The previous commit described a "debug string table on 16-byte alignment". It is not a table. Dumping the 32 bytes before each run shows most are the tail of the previous string: +3104 "host0:r:/rc2/shock/wavetable.bin" 32 chars +3120 "ck/wavetable.bin" its own tail +3152 "/i5/shock/wavetable1.bin" +3168 "ble1.bin" its own tail and the 32-byte prefixes are 327 distinct out of 332 -- essentially all different, because they are just the preceding string's bytes. There is no per-record header here. It is one contiguous pool of debug text, and the 16-byte alignment is a property of how the pool was laid out, which makes continuations land on boundaries and read as separate entries. The count of 332 "strings" was therefore inflated by the same effect: it counts continuations as entries. The substance is unchanged and still useful -- the pool carries the game's own debug text, including "Camera_CollPrimTest WARNING! - grid out of bounds! (RAR)", which is direct evidence for reading the collision payload's two leading u16 as grid dimensions, and the wavetable.bin paths, which bear on the sound bank framing. Only the framing was wrong. The pool is located and counted but not parsed, and there is no gc_formats module for it. The +0x18/+0x20 names are section names but their structure is not established, and it is not shown that they index anything. |
||
|
ReaperOfVeriod
|
a5a33b9bc0 |
Locate level_overlay's sections and a 16-aligned debug string table
level_overlay was on the open list with no parser and no located structure. It now has a header naming two sections and a string table. Range measured: overlay.bin on levels 0, 1 and 2 -- 2751268, 2763668 and 2928424 bytes. Result 1: the file opens with section names. +000: 80 7c 1a 00 | dc 99 00 00 | 01 00 00 00 | 88 3b 2a 00 +010: 00 00 00 00 | 00 00 00 00 | "crap\n" | 00 00 00 00 00 +020: "system misc" 00 00 00 00 00 00 00 00 00 +030: 20 80 ff 40 | 00 60 cf 00 | 64 00 | 46 00 | 19 00 00 00 u32@+0x00 is 1735808 on all three levels, +0x08 is 1, and the bytes at +0x18 and +0x20 spell "crap" and "system misc". Result 2: a debug string table on 16-byte alignment. 332 such strings on level 0, 362 on level 1, 351 on level 2, first at +0x20 and last between 1.3 MB and 2.5 MB. They are the game's own debug text: host0:r:/rc2/shock/wavetable.bin host0:z:/i5/shock/wavetable1.bin unloaded sound bank %#010x Camera_CollPrimTest WARNING! - grid out of bounds! (RAR) render setup sky draw pre effects pre 2 effects Camera_CollPrimTest WARNING! - grid out of bounds! (RAR) is direct evidence for the collision reading. It is the game's own string, naming a grid and a collision test, and it is why the two leading counts in the collision payload are worth continuing to treat as grid dimensions despite not sizing the payload. The wavetable.bin paths also confirm the sound bank is loaded from named files, which bears on the SBlk/DAW framing. Not done: the string table is located and counted but not parsed -- the records preceding each string have not been identified, so there is no gc_formats module for it yet. The +0x18/+0x20 names are section names but their structure is not established, and it is not yet shown that they index anything. |
||
|
ReaperOfVeriod
|
e58ab0e815 |
Correct the collision table's extent: 17 entries, deltas rise then fall
+5932 is the table's start, not a point inside it: walking backwards, no predecessor satisfies "ascending with a delta under 4096". The run is 17 entries, not 19 -- the earlier 19 counted two entries past the jump. The deltas are 204 212 216 224 224 228 228 228 224 220 216 200 96 96 100 96 then a jump of 41484 to 190508. The deltas rise and then fall -- a symmetric profile, not a monotone one. That weakens the prefix-sum reading: a cumulative offset table has no reason to come back down, whereas cell contents that grow and shrink as the table walks across a level would. 17 against a 75-cell grid, and 75 - 17 = 58. No arithmetic relating the two has been found. The earlier guess that 19 related to 3x25 did not survive re-measurement and is not carried forward. Still open. 17 entries may not correspond to the grid at all -- the table may cover something else, or a subset of cells. Resolving it needs either the same offset found in other levels at a position the leading counts predict, or a reason for 17. Neither exists yet, and this is where the measurement trail goes cold on collision. |
||
|
ReaperOfVeriod
|
b66e77855c |
Locate a per-cell offset table in collision, on level 4
Range measured: longest strictly-ascending u32 run (delta under 65536) in each
level's collision payload, levels 0 to 5.
Level 4 is the clean case. Grid 3x25, payload 2660608, and at payload +5932 a
run of 19 entries whose values are all inside the payload, starting at 146012:
deltas 204, 212, 216, 224, 224, 228, 228
The deltas grow monotonically, which is the signature of a prefix sum over
records of increasing size -- what a per-cell offset table looks like when cell
contents vary.
The others are not:
level 0 11x33 +21232 17 8193, 24575, 8192, 8192 multiples of 8192,
coincidental on
dense data
level 1 24x16 +41452 29 32771, 57339, 24575 wild, no trend
level 2 1x53 +13752 49 24, 64, 396, 104 in-region but
irregular
level 3 19x31 +667152 18 769, 257, 257, 257 values ~5.2e8, far
outside the payload
level 4 3x25 +5932 19 204, 212, 216, 224 growing, in-region
Only level 4's run has both properties a prefix sum needs: values inside the
payload, and deltas that grow.
What this does not give: level 4's run is 19 entries against 75 grid cells, so
it is not one offset per cell -- either most cells share an offset, are empty,
or the table covers something other than the grid. Nothing measured picks between
those.
Next to check: whether 19 relates to 3x25 by any arithmetic, and whether the
same offset at +5932 exists in other levels at a position given by their leading
counts. Neither has been tried.
|
||
|
ReaperOfVeriod
|
2569cc06ab |
Decode the collision payload's leading two counts
Range measured: the first four bytes of the payload on all 19 levels that carry a non-empty one. Result: two u16 at payload +0 and +2, both non-zero on all 19, ranging 1..74 and 13..112, products 53 to 3256. level 0 11 x 33 = 363 payload 903248 level 1 24 x 16 = 384 payload 2800496 level 2 1 x 53 = 53 payload 2315296 level 7 40 x 35 = 1400 payload 4049232 level 9 74 x 44 = 3256 payload 2518352 level 13 2 x 64 = 128 payload 3333520 level 26 27 x 112 = 3024 payload 652176 Reported as two counts, not as a named grid. They have the shape of collision grid dimensions -- bounded, both non-zero, products in the hundreds to thousands -- but nothing measured ties the product to the payload size or to any per-cell stride. Level 2's 1 x 53 giving a 2.3 MB payload and level 26's 27 x 112 giving 652 KB shows the product does not size the payload, so either cells vary wildly in size or these are not grid dimensions. Naming them width and depth would be a reading, not a measurement. After the two counts comes a run of u16 that stays below 8192 for 30 to 58 entries depending on level, always an even count, and is not ascending. That is where the per-cell table most likely starts, and it is the next thing to resolve. Exposed as CollisionHeader::first and ::second. They live in the payload rather than the header, so CollisionHeader::region fills them in and the bare CollisionHeader::parse leaves them zero -- documented at the call site so the difference is not mistaken for an absence. |
||
|
ReaperOfVeriod
|
72bbd5da38 |
Rule out GS packet framing in tie, with a control that nearly fooled me
Range measured: level 0, the 73 tie span bodies (bytes +0x40 onward), 2585856 bytes pooled. Per-position distinct counts, an autocorrelation scan against a shuffled control, and a GS GIF tag test. Result 1: the bodies are dense. Every one of the 16 slot positions takes 245 to 256 distinct values, byte 15 zero on 81.3% and byte 7 zero on 78.8%. The best stride peak is 48 at +0.097, barely above 16 at +0.095 -- no period. Result 2: no GS GIF framing at 16-byte alignment. Testing whether byte 15's low three bits fall in 0..4, the GS tag range, gives 97.0%. That looked like a strong hit until the control was done properly. A uniform shuffle gives 62.6%, exactly 5/8, what a random byte predicts -- so the naive comparison reads as a 34-point excess. The distribution-matched null is 97.0%, identical to the observation. The whole excess was that byte 15 is 81% zero and (0 & 7) <= 4 is always true. Recomputed per alignment the null tracks the observation everywhere: align 0 97.0% align 4 75.7% align 8 96.7% align 2 75.5% align 12 89.3% This is the control rule earning its place a second time. A uniform shuffle is not a matched control when the data is skewed, and AGENTS.md already says so; the near-miss was reading 62.6% as a baseline without checking what the data's own distribution predicts. The tie bytes concentrate on 0, 1, 2, 3 and 63 -- suggestive in isolation -- and still carry no framing. What this rules out: the tie bodies are not GS GIF packet streams at 16-byte alignment and carry no measurable periodicity. With every slot position taking ~256 distinct values they are dense data -- fat vertices, as the roadmap expects -- and locating the vertex layout needs a reference for this game's instanced geometry format, not another stride scan. |
||
|
ReaperOfVeriod
|
86ff2e6052 |
Measure tie spans: a constant first word and three quantized floats
Range measured: all 1864 tie spans across 27 levels, read at each class entry's
geometry offset.
Method: read +0x00, +0x0c and the three f32 at +0x10, +0x14, +0x18, and count
distinct values per position.
Result:
u32@+0x00 takes exactly one value across all 1864 spans: 0x00000100, 256
+0x0c takes 83 distinct values, so it is not constant
the three floats take 103 distinct triples built from 42 distinct values,
and those values are round -- 4, 8, 9, 10, 12, 15, 16, 18, 20, 24, 25, 30
with only 10 of the 42 exact powers of two
Two level 0 spans open:
+000 00000100 00005070 000065c0 07050612
+010 41400000 41c80000 42200000 00009460
+000 00000100 00007ca0 0000a040 0605091e
+010 41400000 41c80000 42200000 0000ddf0
This rules out the bounding-box reading. A box would take continuous values;
these come from a 42-value set of round numbers, so they are authored or
quantised quantities -- a grid, a size class, or LOD thresholds -- and nothing
measured separates those.
Correction: these floats were previously read as a bounding-box minimum, from
level 0's first two spans happening to share (12.0, 25.0, 40.0). That triple
holds on 266 of 1864 spans, not all of them, and the value set is quantised.
Not decoded: what the three floats are, what 0x100 at +0x00 counts, and what
the 83 values at +0x0c are.
|
||
|
ReaperOfVeriod
|
32a24574be |
Identify data.bin as the level file descriptor, not geometry
data.bin is the largest per-level file -- 13.3 MB on level 0, 4.2 to 21.9 MB across the disc -- and had been characterised only as "4-byte structured". Range measured: the first 64 u32 of data.bin on levels 0, 1, 2 and 5, compared against the actual byte lengths of the other extracted level files. Method: read the first 64 u32 and look for any equal to the size of core_index.bin, gs_ram.bin, overlay.bin, gameplay.bin, sound_bank.bin or core_data.bin. Result: three exact matches, at the same indices on every level. u32 index 1 overlay.bin 2751268 2763668 2928424 2863676 u32 index 3 core_index.bin 31600 34016 33808 23536 u32 index 5 gs_ram.bin 518656 569344 513024 434176 Each is the exact byte length of the named file and each lands on the same index on all four levels. Three coincidences at fixed positions is not a coincidence. So data.bin is a descriptor for the level's other files, not geometry in its own right. The roadmap's "the large one: 4.2 to 21.9 MB of geometry per level" entry was wrong and is corrected. The intervening indices 0, 2 and 4 hold values that are not file sizes and their meaning is not established; they are plausibly offsets, but no cumulative-sum or offset+size relation has been checked yet. What follows the descriptor is still unknown, and the file is large enough that the descriptor is unlikely to be all of it. |
||
|
ReaperOfVeriod
|
5610d173b0 |
Narrow the collision header to two fields plus a third used by five levels
Range measured: all 27 levels that carry collision, reading the 0x40-byte
header each.
Method: read +0x00, +0x04 and +0x08, then check +0x08..+0x40 for zero.
Result:
u32@+0x00 is 0x40 on 27 of 27 -- the header is self-sizing
u32@+0x04 is the payload size, 20 distinct values across the disc
bytes +0x08..+0x40 zero on 22 of 27
Five levels carry a large non-zero word at +0x08:
level 7 1248048 level 12 2541616
level 9 1766832 level 13 3376592
level 14 4060640
So the header is not two fields and 56 bytes of padding: there is a third field
most levels leave at zero and five use. Whether it is a size, an offset or a
second block's length is not established, and nothing measured here separates
those readings. Exposed as CollisionHeader::field_08.
Correction: the collision header was previously described as self-describing
across its full 0x40, which is true for 22 levels and overstates the case for
the other five.
Still not decoded: the payload. It contains ascending u32 runs on levels 0 and
2 -- level 0 at +280 with deltas 12, 12, 12 then a jump, level 2 at +13868 with
deltas 204, 204, 200, 196 -- but irregular deltas fit neither a flat table nor a
simple stride, and level 1's apparent run is a byte-shift false positive.
Nothing about the payload is asserted.
|
||
|
ReaperOfVeriod
|
681e7e62ef |
Parse the sky block: 149 sub-objects with a bounding sphere, and correct the count field
Range measured: all 27 levels, each sky region from the core index's sky offset to the next region. Method: read the 32-bit offset table at +0x20, bounded by staying inside the region, and parse each sub-object it names. Result: 149 sub-objects across 27 levels. Each opens with a count, then a bounding sphere of four floats at +0x10, then a texture offset at +0x20 and dimensions at +0x24/+0x26: +000: 26 00 00 00 | 00 00 00 00 00 00 00 00 00 00 00 00 +010: e0 8d b2 c1 | 9f d6 f6 3d | ef fc fb be | 1c 61 4e 41 +020: 90 99 05 00 | 1d 00 | 1d 00 | 00 00 e8 00 ... Count 1 to 38, width 3 to 81, radius 1.95 to 14.56, and the radius is positive on 149 of 149 -- the same sphere shape and the same reasoning as the shrub class header's. Correction to an earlier claim: the word at +0x0c is not the sub-object count. It reads 5 on level 0 where the table holds 6 entries, 3 on level 1 against 4, 7 on level 3 against 8, and 196617 on level 2 against 4. Across the disc offsets.len() == count+1 holds on 10 of 27 levels. The field is reported as SkyHeader::field_0c and nothing depends on it; the offset table enumerates the sub-objects and is self-limiting. That earlier claim came from level 0 alone, where 5 against 6 looked like a consistent off-by-one. It is not an off-by-one -- it is a field that happens to be small on most levels and is not a count at all. gc_formats::sky parses the header and the sub-objects. The gated test asserts the sphere is finite, the radius positive where positive at all, the dimensions non-zero, and the sub-object list non-empty on every level. |
||
|
ReaperOfVeriod
|
5ade83cf0a |
Parse the shrub packet header, a fixed signature on all 2056 disc packets
The packets found by the +0x40 table have a header the whole disc agrees on. Range measured: every packet on the disc -- 2056 of them, across the 334 successor-bounded spans -- read at each packet's span-relative offset. bytes 0-3 04 04 00 01 2056 / 2056 bytes 4-7 zero 2056 / 2056 byte 8 zero 2056 / 2056 byte 11 0x05 2056 / 2056 size % 16 zero 2056 / 2056 Zero exceptions on any of the five. And the loader fills the rest in. Read back out of the running game the same records carry byte 8 as a count (0x0b to 0x29 observed), byte 11 as one of four tag values, and bytes 12-15 as an absolute EE RAM pointer where the disc has a span-relative offset. Bytes 8-15 are rewritten wholesale on load. That settles what the disc header is for. It carries no count of its own, and the count the RAM copy has is the +0x40 table's count arriving by another route. The disc header is a signature; the RAM header is populated state. ShrubPacketHeader::parse and Shrub::packet_headers() expose this. parse returns None unless all the invariants hold, rather than a default, so a packet that does not match is reported as not-understood instead of half-interpreted. The gated test asserts all four header invariants and the size alignment on every packet of every bounded span. This is the first piece of the mesh packet format represented in gc-formats; everything known about the packet payloads so far is RAM measurement, since the disc form of bytes 8-15 is rewritten at load. |
||
|
ReaperOfVeriod
|
d1aec9f866 |
Show shrub packet records are two tagged 8-byte halves, fixed length
Range measured: level 0, the 1636 sixteen-byte records following the header across all 29 packets. Per-byte distinct counts, not pooled. Method: count distinct values and zero frequency at each of the 16 positions, then tabulate the joint distribution of the two low-entropy positions. Result: fourteen positions are uniformly varied, 101 to 153 distinct values, zero in 13% to 58% of records. Two are not: byte 7 4 distinct: 0 (1432), 0x30 (135), 0x6d (40), 0x80 (29) byte 15 2 distinct: 0 (1618), 0x6d (18) Both sit in the last byte of an 8-byte half, eight bytes apart, and both take values from a small closed vocabulary rather than the full byte range. The joint distribution shows no record where both are non-zero, and 0x6d appears in either position but never both: (0,0) 1414 (0,0x6d) 18 (0x30,0) 135 (0x6d,0) 40 (0x80,0) 29 Each 8-byte half carries a tag or subtype byte in its last position, from a small closed set. The shared 0x6d is what suggests one vocabulary used twice rather than two unrelated fields. This answers the question the previous commit left open. There is no per-record length field: the best candidate offsets hold the value 16 on at most 13% of records, which is the chance rate for small integers, not a self-describing record. The records are therefore fixed 16 bytes, and the tiling is not a coincidence of variable-length records that happen to be multiples of 16. Still open: what the four and two tag values mean, and what the 14 varied bytes encode. The tag vocabulary is small enough to enumerate against known PS2 primitive lists, which is the reference to bring in next. |
||
|
ReaperOfVeriod
|
e6dd60ec3a |
Characterise shrub mesh packets: 16-byte header, 16-byte tiling
The packet table decode left the packet contents open, with a warning that a VIF reading had already been retracted once and needed a control rather than a pattern match. This is that measurement. Range measured: level 0, the 29 packets of the 6 successor-bounded shrub spans, 26640 bytes. Per-packet; nothing pooled. Method: autocorrelation within the largest packet against a distribution matched shuffled control, then per-byte distinct counts across all 29 packets. Result 1: all 29 packet sizes are exact multiples of 16, and the internal stride scan peaks at 16 (+0.283) with 32, 48, 64, 80 and 128 all elevated behind it -- the signature of a record array, not a coincidental spacing. Result 2: record 0 is a 16-byte header. Constant across all 29 packets at bytes 0-3 (04 04 00 01), 4-11 (zero), 12-13 (00 80) and 15 (6c). Bytes 14 and 16-23 vary. The 04 04 00 01 prefix looks like a magic or format version. Result 3: header byte 14 is echoed in the body. It takes values 06, 07, 0e and 11 across packets, and the same value appears at offset 12 of record 1 in every packet sampled. A real repeated field, not two small values coinciding. Not established: no header field matches the record count. The packet holds 1056/16 = 66 records and none of the candidate u32 or u16 header offsets at 8, 10, 12, 14, 16, 18 or 20 equals 66, 65 or 67 on more than half the packets. So the count is either computed from a size field elsewhere in the span, or the records are variable length and merely 16-byte aligned. That distinction is unresolved and matters. Fixed 16-byte records make the payload a flat table; variable records make the 16-byte alignment a coincidence of every record being a multiple of 16 long. The stride scan cannot separate the two, because both peak at 16. The retracted VIF reading is not revived: the tiling is uniform across all 29 packets with no command-shaped tail, and a VIF packet would show a changing command byte last rather than the constant 6c at header byte 15. Next: read the record count from the packet table in the span rather than from the packet header, which is what would separate the two readings. |
||
|
ReaperOfVeriod
|
aa2b14ad66 |
The shrub mesh geometry is in the class blob after all
The open question from the shrub decode was where the mesh lives: the blob looked like header, palette and optional billboard, with the 333 classes without a billboard having nothing to draw. Answering it took the running game, not the disc. Diffing a loaded shrub class against its disc bytes in the live PCSX2 (level 1, class 1) shows what the loader rewrites: a table at +0x40 of (offset, size) pairs, whose offsets are span-relative on disc and absolute in RAM with the sizes unchanged -- all 21 entries matching offset for offset and size for size -- plus a relocated palette pointer at +0x2c and a pointer at +0x18, zero on disc, that asserts the parser enforces. So the blob is header, packet table, packets, [billboard], palette: the mesh was here the whole time. Why it was missed is the part worth recording -- data_offset points at the END of the span, so reading the body as data_offset..span_end accounts for the last few hundred bytes and leaves the 20 KB in front of it unexamined. A correct bound and a body small enough to look plausible are together enough to hide a region. Range measured: every shrub span on the disc, 358 over 24 levels, each bounded by its successor class or the located chunk container. Method: read count at +0x28, then the entries at +0x40, and require them to tile -- the first packet at align16(0x40 + count*8), every entry exactly size past the previous, and the last ending at the palette, or 0x40 short of it on the 25 spans that carry a billboard. Result: 358 of 358 tile exactly, holding 2,239 packets. That is what pins the field meanings: count is the packet count (1..55 on the disc), data_offset is the palette's offset, and the earlier "the body is not sized by count" note was true for a reason nobody had understood -- the count sizes the packets. Shrub::packets() exposes them and returns None rather than a partial list when a table does not tile. Still open: the packet contents (a VIF reading was already retracted once, so this needs a control rather than a pattern match), and the 0x20-byte records the loader points at from +0x18. |
||
|
ReaperOfVeriod
|
69a8398f6e |
Verify the tfrag lighting magnitudes against the running game
The harness that compares the port against the game's own memory existed but had never been compiled: tools/trace did not build (27 errors, all in the argument plumbing), and the tfrag_light.rs light-bank correction it motivated was unverified. This finishes it and runs it, and the run finds two real bugs. Result, on level 0 against three inputs -- the live emulator over PINE and two PCSX2 savestates from the same level, agreeing byte for byte: headers matching the disc on invariant bytes: 833/833 data pointers relocated as table + disc offset: 833/833 light bank in RAM at 0x50ab20, byte-identical to the disc's vertices compared: 40600 (162400 colour bytes) equal: 162400 (100.00%) The locator finds the level's tfrags block in RAM by tfrag 0's bounding sphere and accepts a candidate only when every header matches the disc on the bytes the game leaves alone and every data pointer is the relocated form, so "the block was found" is itself a verification rather than an assumption. The bank is searched in RAM by its exact section bytes, which is what settles its framing: offset at +0x04, an i32 set count, twelve bytes of pad, then the sets. The previous reading (offset at +0x00, count at +0x04) was wrong. Two bugs the comparison found, both of which produced plausible output: 1. vertex_normal_bits negated x and y and left z positive, while NormalTable::normal negates all three. dA came out -0.707 where the game had +0.707, so max(dA, dA*wA) took the wrong branch and every vertex lit near its base colour: dark, but not wrong-dark. 95.88% agreement with the bug, 100.00% without. 2. A record whose select names a set the bank lacks drew transparent black. The game writes the record's base colour; 3,130 of level 0's 40,600 vertices do this (select 0x000f, set 15 -- a runtime slot the level's directional bank does not have). The vertex is unlit, not invisible. Both are now regression tests in the engine. The comparison prints the pass's per-vertex terms (light_vertex_terms) for the first mismatches, because a colour mismatch alone cannot be attributed and the comparison is worthless without attribution -- that is how the second bug was identified from a three-line read rather than guessed at. Docs: docs/plan/pcsx2-verification.md records the harness, the result, the two bugs and what to compare next; roadmap and level-assets.md stop carrying lighting magnitudes as an open item. Findings reach tests only as numbers -- no test reads a savestate, a dump or a live emulator. |
||
|
ReaperOfVeriod
|
2c7555fe16 |
Decode the shrub class blob: a 24-normal palette and a billboard record
The shrub body's payload encoding was the open question after the span-bound work, and it answered itself once the R&C1 reference was in view: R&C1's shrub class header keeps the field offsets R&C2 shows (o_class at 0x24, a count at 0x28, the data offset at 0x2c), which says which bytes to look at, and the disc says what they hold. Range measured: all 358 shrub class spans over the 24 levels that have them, each bounded by its successor class or the located chunk container. Nothing outside the spans is read. Result. The body is a palette of 24 unit-length s16 normals -- s16 x, y, z and a zero pad word per 8-byte record -- followed by all-zero padding records. Three independent measurements, each of which fails for any other reading: - all 8,592 live entries have squared length within 65,535 of 32767^2, and the bound is tight (the largest observed deviation is exactly 65,535). That is 32768^2 - 32767^2, which is what a unit vector scaled by 32768 produces when an axis-aligned normal quantises to -32768 and the positive axis clamps to 32767; components run the full s16 range and 1,765 entries are axis-aligned. A random s16 triple is off by about a billion, so one entry in fifteen thousand would pass by chance: all 8,592 pass. - the padding records are always at index 24 or later and always all zero, never a live entry. That is what separates the palette from geometry, and it is the observation the three earlier stride-fitting attempts were missing. - 17 classes carry a palette of 24 identical (0, 0, 1) entries. A position or index table cannot produce that; a normal palette for a flat-facing class must. The header decodes as: sphere at +0x00 (centre + radius, radius positive on all 358 spans, 1.72..41.57), size at +0x10, mode_bits and scratch at +0x14, a zero word at +0x18, the billboard offset at +0x1c, and scale at +0x20. The parser asserts the two invariants that hold without exception (scratch and the reserved word are zero on all 358) plus a positive radius and scale, and reports mode_bits rather than asserting it: it is 4 on 16 of the 358 spans. The billboard offset is the strongest self-identification after o_class because it is confirmed from a second file: it is non-zero on exactly the 25 classes whose core-index entry carries billboard texture fields, pairing correctly both ways. At the offset sits R&C1's ShrubBillboard shape, register block and all -- fade_distance, then TEX1_1 (K -233..-168, MMIN 4, address 0x14), TEX0_1 (index 1..4, address 0x06) and MIPTBP1_1 (address 0x34), all three register addresses checked in the disc test. Two corrections this produces, both recorded in docs/plan/level-assets.md because the old readings were confidently wrong rather than merely unverified: - "reserved is 0 or a denormal" had the two u16 halves the wrong way round: the 16 non-zero spans are mode_bits = 4 with scratch = 0. As an f32 the word is still a denormal, so the PS2 reads zero either way -- but the field is a flag word, not a magnitude, which is exactly the FPU point worth making. - "bounds" was tested as a min.xyz/max.xyz box, which holds on only 146 of 358 spans, and the field was left unnamed as a result. It is a bounding sphere: the fourth float is positive on all 358 spans and the centres cluster near the class origin. The 146/358 figure was a property of the wrong question. Still open: the three floats in the billboard record (they repeat byte-identically across levels -- 16 distinct triples among 25 records, one on four classes across levels 1, 4, 12 and 21 -- so they are shared art data rather than dimensions), size, mode_bits, count, and where the shrub mesh geometry lives. R&C1 puts the mesh packets in this blob; R&C2's blob holds only the palette, so the 333 classes without a billboard must draw meshes from somewhere else. |
||
|
ReaperOfVeriod
|
ccea53807a |
Resolve the shrub region bound: the chunk container closes it
The shrub region's end was the last unlocated bound in core_data. It was not in the core index, and unlike moby->tie it had no successor boundary to chain against, so the last shrub class in every level ran out to the lump's end and the aggregates over it (the "3.5M records / 28MB of shrub body") were artifacts of that oversized span. Range measured: for each level, from the last shrub class's geometry offset to the constant trailer. Nothing outside that range is read. Method: scan forward from the last shrub class for the pair (a) the constant two bytes f6 7c sitting immediately in front of (b) a chunk signature -- a 16-byte header ending 0xff, ten zero bytes, then four or more 0xf0-tagged offsets. The two halves are each weak alone, and the halves are measured separately: the marker pair occurs 25 to 103 times in a 5 to 17 MB lump, so matching it alone picks dozens of wrong offsets; the chunk shape matches random data often enough to need four tagged offsets before it is believed. Result: on all 27 levels the marker sits exactly two bytes before the first chunk header, and the two parts of the signature are never found apart. No chunk signature occurs inside any last shrub body, on any level. The three levels without shrub (10, 25, 26) carry the same marker before the container that closes tie instead, which is what makes the locator a property of the region rather than a shrub rule. Consequences: - All 358 shrub class bodies are bounded and are whole 8-byte records. That was not assertable while the last class in each level ran to the lump's end, and it is the check the boundary earns its keep for. - 9,600 records in total, against the 8,938 previously measurable in successor-bounded spans; 662 of them sit in the 24 last-class bodies. The shrub region totals 2,033,776 bytes across the 24 levels that have one. - The chunk container is 25,775,850 bytes across 27 levels (1,236 chunks, 22,562 records). The container measurements were previously taken over "the 900,000 bytes before the trailer", a window that did not align to its real extent: on level 0 it is 690,302 bytes, and 27 of 1,123 chunks were truncated at that invented edge. chunk_containers_resolve_on_every_level now reads the region from core_layout instead of measuring backwards from the end of the lump. core_layout takes core_data now: the container's start is the one bound that cannot be derived from the index. It gains the chunks region, so every byte of level-specific core_data belongs to a named region. Where the signature is not found the end stays None -- the region is left visible rather than clamped to the tail. Still open: the marker's meaning (constant across 27 levels, at exactly this position, unnamed) and the shrub body payload's encoding. |
||
|
ReaperOfVeriod
|
aec462df21 |
Record that shrub span bounds are wrong, and what that invalidates
A follow-up measurement contradicted the body size implied by the differential analysis, and the cause is bad span bounds rather than anything in the format. Shrub spans were bounded by the next shrub offset, with the last running to the end of core_data. On level 0 that gives the final shrub class a 925,008-byte span, absorbing most of the lump's tail. The moby-to-tie boundary is clean by comparison: last moby offset 13414592, first tie 13416192, exactly 1600 bytes apart. What that invalidates: - The 3.5M record / 28 MB body figure is an artifact of the oversized final span and should not be quoted. - The "payload looks uniform-random" measurement ran over the same contaminated range, so it says nothing about shrub geometry either way. - The differential result itself still stands -- it was computed on groups, and the groups used had bounded spans. The global aggregates built on it do not. What is needed is the offset ending the shrub region. It is not in the core index, and there is no obvious successor boundary the way moby->tie has one. This is the second confident-sounding number in this format that turned out to rest on a bad bound, the first being the VIF reading. Aggregates across many spans are only as good as the span bounds under them, and bounds are the easy thing to get wrong because nothing errors when you do. |
||
|
ReaperOfVeriod
|
52228c14ed |
Retract the VIF reading of the shrub body
The previous commit concluded the shrub body's 6-byte payload was VIF-packed, on the strength of recurring patterns like 7ea5 and 1ab6. That was pattern matching dressed up as a conclusion, and a control kills it. Decoding every 32-bit word of every shrub body as a VifCode: shrub bodies 24.9% unpack, 128 distinct cmds uniform random 25.1% unpack, 128 distinct cmds sound_bank (definitely not VIF) 19.0% unpack, 122 distinct cmds gs_ram (mixed) 13.6% unpack, 128 distinct cmds moby class 0 span (known VIF) 40.1% unpack, 105 distinct cmds Shrub sits on the random baseline and exercises all 128 possible command bytes, which is what unstructured data looks like. Worth being precise about what this does and does not show. The test is weak -- even the known-VIF region only reaches 40.1%, well short of a clean signal -- so this does not prove shrub is not VIF. It proves the claim was never established, which is the part that was wrong. Spotting three recurring byte values and naming them a format is the same mistake as the three stride guesses it was meant to replace, one level down. A hypothesis needs a discriminating test before it goes in a commit message, and I skipped that step because the finding was welcome. What still stands is the part that rests on differential analysis rather than recognition: 8-byte records, 6 varying and 2 constant, 144 varying bytes per group regardless of count. The payload's encoding is unknown. |
||
|
ReaperOfVeriod
|
572753e7c1 |
Find the shrub body stride by diffing spans instead of fitting it
Stride-fitting the shrub body failed three times, so this attacks it from the other side: group spans by (count, data_offset, tail length) and byte diff every member of each group. Positions that differ across a group are geometry; positions that never differ are structure. The answer is blunt. 144 bytes vary in every group, whether that group has count 1, 2, 3, 5 or 6 and a tail of 192, 208 or 240 bytes. The varying data is not sized by count at all, which is why three stride guesses were fitting the wrong thing -- and why they kept looking nearly plausible. Those 144 bytes fall in runs of 6, every 8 bytes. The body is an array of 8-byte records: 6 varying bytes and 2 constant. The payload is not half-floats -- decoding as half3 gives 245.25 and -168.375, which no shrub has. The patterns (7ea5, 1ab6, 43d9, af80) and their repetition are VIF-packed data, the same quadruple-word packing moby and tfrag use. Filler around it is the PS2 NaN 0x7fff0000 rather than zeros, which is why the body read as noise in the earlier dumps. So the open question is the VIF code assignment, not the stride, and the decoder written for tfrag is the obvious place to start. |
||
|
ReaperOfVeriod
|
f6bbbf8d5f |
Add gc-extract probe to dump structures through the real parser
Reading these formats by unpacking raw offsets out of core_index.bin has cost real time. The index interleaves single-word region offsets with count/offset pairs, so miscounting one shifts every field after it: that produced a page of implausible collision headers on all 27 levels, sent me chasing a tie-breakoff that turned out not to exist, and nearly had me "fix" a stride bug in code that was already right. probe dumps a level's core header and all three class tables through LevelCoreHeader, so a field name printed is the field name the parser uses. It also reports how many entries carry geometry inside core_data, which is what pins the class table strides rather than guessing at them: shrub reads 7/7 sane at 0x30 against 4/7 read at 0x20. Reading a structure by hand in a shell still works; this just stops it being the default, and stops the offset arithmetic being re-derived every time a new region is investigated. |
||
|
ReaperOfVeriod
|
c4ba84232a |
Record that tie does not reuse moby's part table
Checked the obvious hypothesis -- that tie, being the other class-geometry table, would share moby's part-table format -- and it does not. The tiling rule vertex_offset + 16 * vertex_count == next_part holds for 0 of 25,024 records across the 1,864 tie spans, against 43,467/43,467 for moby. Tie's second field also reads as an absolute offset rather than a span-relative one, which is the first sign the record shape differs. Past the verified 0x48-byte header the bytes are f32-shaped rather than the expected VIF marker, but they are not a fixed array either: the run of consecutive plausible floats varies from 1 to 6 across spans (822 spans have 6, 455 have 1, 373 have 2). So there is more header past 0x48 and it is not one array of distances -- the floats after that point are a different population from the LOD distances before it, including negative values. Recorded so the next attempt starts from that rather than re-trying the moby template. |
||
|
ReaperOfVeriod
|
3ca8522be1 |
Record shrub geometry findings and the per-table entry strides
The three class tables do not share an entry size. Measured by how many entries give a geometry offset inside core_data plus a plausible o_class, over all 27 levels: moby stride 0x20 5769 / 5796 tie stride 0x20 1864 / 1864 shrub stride 0x30 356 / 358 Reading shrub at 0x20 still yields 231 sane-looking entries out of 358, which is the hazard: wrong often enough to look right. The committed code already handles this -- ShrubClassEntry is 48 bytes and carries billboard texture fields the other two lack -- so this is measurement, not a fix. Shrub geometry is not moby's part-table format: all 358 spans have a leading word of 0 or one too large to index a table into, so none contain a moby-style part table. Two header fields do self-identify, both exact across all 358 spans: +0x24 equals the class entry's own o_class, and +0x2c is a data offset landing inside the span. +0x10 reads 5.0f on 281 of 358, so it is a value rather than a marker and carries no structural weight. Rest of the shrub header undecoded. |
||
|
ReaperOfVeriod
|
bbb10c678d |
Parse the moby per-class part table
A moby class span does not start with its geometry. The first u32 is the offset of a part table, and the table is 16-byte records of vertex offset, vertex count, an unestablished second count, the next part offset and flags. Parts tile: vertex_offset + 16 * vertex_count == next_part. The stride is never assumed anywhere -- it falls out of that identity, which is what pins the record size and the field boundaries. The identity holds for 43,467 of 43,467 part records across every moby class on all 27 levels, and the parser rejects any span that violates it rather than returning a plausible wrong answer. Two terminations are needed. A zero vertex_offset ends the run, but 172 records on the disc have a zero vertex_offset and a zero next_part while still carrying a vertex count. Stopping only on the first leaves 99.6% instead of 100% and quietly includes a second table as if it were parts. 1,270,802 vertices over 4,560 class spans; 116 spans carry no geometry. Vertices are 16 bytes of VIF-packed half-float quads cycling through four distinct words. Handed back raw -- unpacking them is separate work from finding them, and second_count is recorded but not guessed at. |
||
|
ReaperOfVeriod
|
1f96ef09b9 |
Record core index field order, sky and moby findings
Three things this session that are cheap to rediscover wrongly and expensive to rediscover wrongly twice. The core index interleaves single-word region offsets with count/offset pairs, so gs_ram and the three class tables are two words each while tfrags..collision are one. Treating the pairs as single words shifts every field after them, which is how collision got read as sky and produced a page of implausible collision headers on all 27 levels. Sky is not uniform. Most levels share a header shape, but level 2 carries extra fields and the count word at +0x0a over-reports against the number of real offset slots, so no single fixed header fits. Moby geometry starts with zero words rather than a header and then packed half-float VIF data. Not decoded; recorded so the next attempt starts from the data instead of re-deriving the layout. |
||
|
ReaperOfVeriod
|
79025b22d6 |
Decode the tie class header prefix that RAC1 and RAC2 agree on
Tie is the easy one of the three geometry subsystems -- instanced static props, no skeleton -- so it is the right place to start. RAC1's class header is 0x80 bytes. RAC2 agrees with it over the first 0x48 and diverges after. From 0x48 the RAC1 layout does not describe RAC2 data: lod_info at 0x50 reads back as noise, packet_count and texture_count are zero on every class examined, and the normals offset at 0x0c points 117,788,434 bytes into a 16.9 MB lump. Only the agreed-on prefix is decoded. What makes it trustworthy: * o_class at 0x44 equals the core index's class number for all 1,864 tie classes across all 27 levels. A field landing on the right number that often is not a coincidence, and it pins the header base. * The LOD distances are ordered 0 < near < mid < far for 1,784 of 1,864. The 80 exceptions are real, unexplained, and are reported rather than asserted away. Level 0's first tie class has distances 12.0 / 25.0 / 40.0, a plausible prop LOD band, and packet offsets 256, 20,592 and 26,048. The rest is deliberately not guessed at. A field order that produces plausible output and is wrong is the specific failure this project keeps recording, three times now: the gameplay extent, the tfrag record framing and the class tables were all "reasonable" and all wrong. |
||
|
ReaperOfVeriod
|
4c4a78d402 |
Identify the container the class tables point into: core_data itself
assets_compressed_size and assets_decompressed_size at index offsets 0x88 and 0x8c are 9,738,678 and 16,969,296 on level 0 -- exactly the sizes of core_data before and after decompression. There is no separate asset container. The "asset WAD" is core_data, and offset_in_asset_wad indexes the decompressed form directly. Checked rather than assumed, on all 27 levels: the decompressed size equals core_data's exact length, every class offset is either 0 or inside it, and the moby, tie and shrub spans do not overlap each other. An offset of 0 means the slot has no geometry, not the start of the lump; level 0's ninth moby class is 4909 at offset 0. Two assumptions worth discarding while here -- the tables are not sorted by class number (level 0's run 0, 1, 2, 14 ... 4909) and the offsets within a table are not monotonic. Neither is needed, but both look true from the first few entries. The chunk descriptors after the named extents are not it: they exist on only 3 of 27 levels, because level 0's four named extents tile the container exactly. An asset blob there would be missing almost everywhere. What sits at the offsets is raw GS/VIF packet data rather than a further container -- moby[0] begins 00 00 00 00 3f 16 02 55 6f 6f and tie[0] 00 01 00 00 70 50 00 00, which read as packet headers rather than a magic number. None of the three is decoded yet and this does not start that. Moby is skeletal and is the largest single piece left before the rest of a level can draw. Also: the pipeline now writes each level's chunk descriptors where they exist, and LevelCoreHeader decodes the asset lump's two sizes. |
||
|
ReaperOfVeriod
|
e1105543e4 |
Parse the level core class tables: moby, tie and shrub
These were ruled out for want of an entry size. The size is knowable exactly, because each table ends precisely where the next begins: moby 238 entries at 192, 32 bytes: 192 + 238*32 = 7808 = tie's tie 74 entries at 7808, 32 bytes: 7808 + 74*32 = 10176 = shrub's shrub 7 entries at 10176, 48 bytes: 10176 + 7*48 = 10512 = textures That closes on 27 of 27 levels, which is what makes it evidence rather than a coincidence: any wrong entry size breaks the chain immediately. It is the same "does this hold on every level" discriminator the project already uses for the container pair table. Across the disc: 5,796 moby, 1,864 tie and 358 shrub classes, plus the 1,595 tfrag textures. Class entries are 32 bytes -- asset offset, class number, two unknown words and 16 bytes of texture slots -- and a shrub entry adds 16 bytes of billboard slots for 48. The header now decodes through to the texture tables rather than skipping 0x18..0x30 wholesale, which also recovers the moby, tie and shrub texture ranges. Two more pairs sit between shrub textures and the textures base offset. Two things worth recording. ClassEntry::parse read its 16 texture bytes before its four i32s, because the struct literal that uses them comes after the loop that advances the cursor -- so every entry was parsed backwards. Struct literal field order does not reorder the reads that already happened. And every shrub class on level 0 carries all-zero billboard slots, so the dimensions are optional rather than something worth asserting on. This is the third instance of the same failure shape. The gameplay extent was "an unidentified compression" because it was parsed compressed; the tfrag record framing was "unrecoverable" because the entry size was never known. Both looked like properties of the format and were properties of the question. Also records that point lights are a runtime feature rather than missing work: every one of the 18,102 patches has point_lights == 0xffff and the eight-slot bank is filled by gameplay, so there is nothing for them to light in a static level render. And that PCSX2 is not installed, so the comparison loop cannot be built yet -- which is why lighting magnitudes stay unverified. |
||
|
ReaperOfVeriod
|
b8b4a1a3d3 |
Consolidate: tidy level.rs, correct stale docs, pin hardware models
No behaviour change. This is the pass over the last stretch of work to make sure the accumulated result hangs together rather than only each commit in isolation. Code. lit_color had grown a `let _ = (depth, lod_distances)` and a dead branch from repeated edits. The directional pass genuinely does not depend on distance, so the parameters are gone rather than discarded, with a note that the point lights -- not implemented -- will need them. lit_at folded into lit_color, which had become a wrapper around it. Checked for leftover scaffolding: only two env vars remain, both real (GC_ENGINE_SHOT and GC_ENGINE_DEBUG_BOUNDS); GC_ENGINE_UNLIT, GC_ENGINE_DEBUG_COLOR and GC_ENGINE_ABL were never actually written, because those edits silently failed to apply against rustfmt'd source. AGENTS.md. The crate table did not say that gc-engine owns hardware models. ps2.rs is the Emotion Engine's FPU and the lighting pass runs on it, so the boundary is now stated: gc-formats decodes inputs, gc-engine reproduces how the hardware computes with them. That is the distinction rule 2 depends on, and it was implicit. Docs. Three claims had been overtaken by what we learned and were actively misleading: * docs/formats/level-wad.md still said gameplay "does not decode with the WAD decoder". It does; it is a WAD stream. * docs/plan/level-assets.md carried a whole section, "a compression we have not identified", with a ruled-out list including "bit order", "header length" and "the WAD format at all". All three were ruled out against the wrong input. Replaced, with the failure mode kept: a uniform failure across 27 inputs was read as "the decoder is wrong" when it meant "the data was handed over uncompressed", and the positive control -- decompress and look -- was never run. * The same file still said the tfrag "record framing is not recovered". True, but it was a false lead that has since been superseded by the VIF1 framing. Kept as a recorded wrong turn rather than deleted. Also refreshed "Still open" to name what is actually outstanding: point lights, the GS register set, the guard-band clip test, the unread core_index class tables, and -- stated plainly -- that lighting magnitudes are unverified because there is still nothing to compare against. Verified after the changes: 148 tests pass, clippy and fmt clean, all 5,213 extracted files match the hash table, and level 0 still loads at 26,758 triangles with the same LOD split and fully lit. |
||
|
ReaperOfVeriod
|
65b5d366de |
Implement the PS2 float model and the LightTfrags pass
Two pieces, and the first is the reason the second can be trusted. ps2.rs is the EE's FPU on raw u32 bit patterns. It is not f32: there are no denormals (exponent 0 *is* zero), no NaN and no infinity (exponent 255 is an ordinary value, so MAX reads back in Rust as NaN -- a test asserts exactly that, because it is the sharpest available statement of why this module exists), every result truncates toward zero, the adder discards the small operand's bits more than one below the large one's LSB and returns the large operand unchanged across a 25-bit exponent gap, and rsqrt is sqrt-then-divide with truncation between the steps rather than one rounded operation. The lighting pass accumulates all of these across every vertex of every patch, so this is a visible difference rather than a rounding curiosity. tfrag_light.rs is the pass itself: PEXT5 base colour, the select word picking one light set or blending two by bits 8..15 / 256, blended directions re-normalised where unblended ones are not, and base + A.rgb*max(dA, dA*wA) + B.rgb*max(dB, dB*wB). Lighting indexes positions, not vertex-info entries, since a common entry and a LOD entry can share a position. TfragLight gains a `select` field for the u16 at +6. It was briefly reconstructed with an unsafe pointer cast over the struct, which is exactly the sort of thing that breaks silently the moment a field is reordered; it is now a plainly parsed field. The pass does not run yet. The light bank framing the reference uses -- a u32 offset at +0x04, an i32 count there -- gives offset 0xa0d and a count of 698810368 on RAC2 level 0, so RAC2's gameplay extent is not laid out that way. The renderer now counts the vertices it actually lit and prints it, because falling back silently to the stored colours produces a plausible but wrong picture: lighting: 0 of 80274 vertices shaded by LightTfrags; the rest fall back to the tfrag's stored colours (no usable light bank) That line is the current honest state, and the frame has not changed as a result. Where RAC2 keeps its light bank is the open question. |
||
|
ReaperOfVeriod
|
90f02890f8 |
Decode tfrag normals from the overlay's sine table
Normals are per-vertex, and the tfrag's light records store them as an azimuth and elevation *index* into a 256-entry sine table rather than as angles. This decodes that table and builds normals for the renderer. The reference reads the table from an ELF at BOOT_NORMAL_TABLE_VADDR (0x165500) and notes every level overlay carries a copy. The overlay has no ELF header -- it is a raw loadable image starting with code -- so there is no program header to map a virtual address through, and the offset is not fixed either: level 0 has the table at 0xb650c, level 5 at 0xcca30, level 26 at 0xb6284. So it is located by content. Every table starts with exactly (1.0, 0.0), angle zero, and the scan then requires all 256 entries to match cos(i*pi/128), sin(i*pi/128). Validating the whole curve is what makes the search safe: a lone (1.0, 0.0) pair of floats occurs by accident in a 2.7 MB executable, and a test plants one as a decoy. The values are read from the disc rather than recomputed, since those are the numbers the game indexes. A normal is (cos(az)cos(el), sin(az)cos(el), sin(el)) *negated* -- the stored normals point into the surface -- and then given the same Z-up to Y-up swap as positions, or the lighting comes out sideways. The azimuth and elevation fields are stored signed but index 0..=255, so entry 200 must be read as 200, not as -56; there is a test for that. Also frames and parses the level light bank from the gameplay extent, and the pipeline now writes overlay.bin. The per-vertex lighting evaluation is not written yet, so the renderer stays unlit by default. GC_ENGINE_LIT=1 substitutes a host directional light for inspection -- that is a stand-in, not the game's lighting, and it is deliberately not the default. The hash table is regenerated for the 27 new overlay.bin files: now 5,213 files and 5,762,497,559 bytes. |
||
|
ReaperOfVeriod
|
17c7e8ccc9 |
Draw tfrag per-vertex colour, unlit, and record why the frame is dark
The tfrag RGBA array now modulates the texture. It is indexed by *position*, not by vertex-info: several vertex-info entries can share a position, and the array is sized vert_count, which is a position count. Colour follows the same rule as the vertex itself. Morphing blends the position only -- the game keeps the high-detail colour -- while collapse replaces the vertex outright, colour included. Getting that the other way round would tint distant terrain. The material is unlit, and that is a decision rather than a shortcut. Level 0's colours really are dark (patch 0 runs (25,26,26) to (108,144,141)) and the game's lighting is what brightens them, so what is on screen is the true albedo. Worth recording: the obvious move -- add a directional light, or crank ambient brightness -- does nothing at all. With no normals the PBR path is degenerate, so the brightness change has no effect, and a host light would have hidden the gap rather than shown it. The colour data was always reaching the mesh; what is missing is lighting, not a material flag. Written up in plan/level-assets.md, including what LightTfrags actually needs: the light bank from the level's gameplay extent, the 256-entry normal table at BOOT_NORMAL_TABLE_VADDR (0x165500) which every level overlay carries, and the already-parsed azimuth/elevation light records. |
||
|
ReaperOfVeriod
|
192dcde247 |
Render a level: first drawable frame, and the origin bug it exposed
Adds gc-engine's binary and a level.rs that joins the format layer to the renderer: LOD selection and morphing per patch, then triangles grouped by texture so each texture is one draw rather than one per triangle. The PS2 world is Z-up at 1024 raw units per world unit, so the conversion is (x, y, z) -> (x, z, -y). The sign flip on the third axis turns the PS2's right-handed frame into Bevy's left-handed one rather than negating a determinant, which is what stops the level rendering mirrored. Morphing and collapse are kept distinct. Morphing blends a vertex toward the midpoint of its two parents across its tier's band; collapse replaces the vertex outright, position and UV, once it is past the far edge. Morphing is a blend, collapse is a cut, and conflating them makes distant terrain shimmer. Blending has to happen in raw units before the axis swap: mixing a converted parent with an unconverted child scales the blend by 1024. Level 0 renders as 26,758 textured triangles over 92 textures, correct UVs, with LOD selection live (26 patches at LOD 0, 650 at LOD 2 from the start camera). GC_ENGINE_SHOT writes one frame as a PPM and exits. Rendering immediately exposed a bug no test could. Positions are i16 offsets from a per-patch origin, and the obvious rule -- the last non-index STROW row -- is wrong on RAC2, which also uses STROW for per-vertex light records that come *after* the origin. Taking the last one yields [186, 186, 186, 186] and collapses the whole level onto the world origin. The entire level then fit in a 63-unit box and every test still passed: vert_count matched, tri_count matched, every parent and triangle index resolved, because all of those are relative to the same broken origin. Only a screenshot showed it. The rule that works is the STROW immediately preceding the first V3_16 position unpack; with it, level 0 spans x 86..444, y 46..211, z -477..40, matching the patch bounding spheres. That is worth more than the renderer: a parser can be completely self-consistent and still describe the wrong thing, and where a value is absolute rather than relative it has to be checked against something independent. The frame is flat and dark because there are no normals and no lighting yet. That is the next task -- the per-vertex light records -- not something to paper over with an exposure fudge. |
||
|
ReaperOfVeriod
|
71a3185fa5 |
Resolve tfrag LOD parent linkage: guard absent tiers by their counts
The four V4_8 parent arrays are classified by VU address, but the addresses collide: on level 0 patch 1, unk_indices_2_lod_01_addr == parent_indices_lod_0_addr == 291, and unk_indices_2_lod_0_addr == indices_addr == 294. Matching on address alone cannot separate them, and it fails silently -- the arrays still parse, they just hold the wrong numbers. Two corrections, and the first one is worth recording because it looked like a data problem rather than a code one: Each V4_8 element is 4 bytes and holds four indices, so an array of N entries is count*4 bytes, not count. An earlier analysis compared the unpack's element count against the position counts and concluded that 15,000 patches could not hold enough parents. That was the wrong unit; the data was fine. unk_06 and unk_0a suppress an empty tier. When the extra vertex-info count is zero there is no extra array and its address is simply the next tier's, so guarding those branches on the count lets the address fall through to the array that exists. That one change took resolved LOD-0 entries from 28,739 to 341,306. Result: of 886,841 vertex-info entries across all 27 levels, 516,492 are common and have no link by design, and all 370,349 LOD entries resolve -- 29,043 LOD-01 and 341,306 LOD-0 -- with zero failures. Each resolves to a real first parent, a LOD-01 parent that is always a common entry, and three positions inside the patch. A common entry never claims a parent. Morphing can now be completed: a patch knows which LOD to draw, how far to blend each vertex toward its parents, and where those parents are. |