Verify the tfrag lighting magnitudes against the running game

The harness that compares the port against the game's own memory existed but
had never been compiled: tools/trace did not build (27 errors, all in the
argument plumbing), and the tfrag_light.rs light-bank correction it motivated
was unverified. This finishes it and runs it, and the run finds two real bugs.

Result, on level 0 against three inputs -- the live emulator over PINE and two
PCSX2 savestates from the same level, agreeing byte for byte:

  headers matching the disc on invariant bytes: 833/833
  data pointers relocated as table + disc offset: 833/833
  light bank in RAM at 0x50ab20, byte-identical to the disc's
  vertices compared: 40600 (162400 colour bytes)
  equal: 162400 (100.00%)

The locator finds the level's tfrags block in RAM by tfrag 0's bounding
sphere and accepts a candidate only when every header matches the disc on the
bytes the game leaves alone and every data pointer is the relocated form, so
"the block was found" is itself a verification rather than an assumption. The
bank is searched in RAM by its exact section bytes, which is what settles its
framing: offset at +0x04, an i32 set count, twelve bytes of pad, then the sets.
The previous reading (offset at +0x00, count at +0x04) was wrong.

Two bugs the comparison found, both of which produced plausible output:

1. vertex_normal_bits negated x and y and left z positive, while
   NormalTable::normal negates all three. dA came out -0.707 where the game
   had +0.707, so max(dA, dA*wA) took the wrong branch and every vertex lit
   near its base colour: dark, but not wrong-dark. 95.88% agreement with the
   bug, 100.00% without.
2. A record whose select names a set the bank lacks drew transparent black.
   The game writes the record's base colour; 3,130 of level 0's 40,600
   vertices do this (select 0x000f, set 15 -- a runtime slot the level's
   directional bank does not have). The vertex is unlit, not invisible.

Both are now regression tests in the engine. The comparison prints the pass's
per-vertex terms (light_vertex_terms) for the first mismatches, because a
colour mismatch alone cannot be attributed and the comparison is worthless
without attribution -- that is how the second bug was identified from a
three-line read rather than guessed at.

Docs: docs/plan/pcsx2-verification.md records the harness, the result, the
two bugs and what to compare next; roadmap and level-assets.md stop carrying
lighting magnitudes as an open item. Findings reach tests only as numbers --
no test reads a savestate, a dump or a live emulator.
This commit is contained in:
2026-10-11 19:51:25 +02:00
parent 2c7555fe16
commit 69a8398f6e
15 changed files with 2144 additions and 25 deletions
+9 -3
View File
@@ -714,9 +714,15 @@ the fidelity rule.
resolving texture addresses on the host.
- **The guard-band clip test** that sends patches crossing the guard band to
the clipping program.
- **Lighting magnitudes.** The pass runs and the output is plausible; whether
it matches the game needs comparison against reference frames, which needs
the PCSX2 loop that is still to come. Nothing in this file establishes it.
- ~~**Lighting magnitudes.**~~ **Settled 2026-10-11** by the PCSX2
comparison loop (`docs/plan/pcsx2-verification.md`): level 0's 162,400
colour bytes match the game's EE RAM exactly. Two bugs the comparison found
and fixed, both of which produced output that looked plausible:
`vertex_normal_bits` negated x and y but not z (so `max(dA, dA*wA)` took the
wrong branch and every vertex lit near its base colour -- 95.88% agreement
instead of 100%), and a record whose `select` names a set the bank lacks
drew transparent black where the game writes the record's base colour (3,130
of level 0's 40,600 vertices, `select 0x000f`).
- **tie, moby and shrub geometry.** The class tables parse and the container
they point into is identified (below), but none of the three is decoded yet.
Moby is the largest of the three and is skeletal, with an animation evaluator
+119
View File
@@ -0,0 +1,119 @@
# PCSX2 verification (`gc-trace`)
Status as of **2026-10-11**. The tfrag lighting magnitudes are verified; this
is the file that records how, so the next comparison starts from the
harness instead of from scratch.
Tool: `tools/trace` (`gc-trace`). It never ships, no shipped crate depends on
it, and it reads the game's own memory out of a running emulator, a
savestate, or a raw dump.
## What exists now
```text
gc-trace pine-info # protocol handshake
gc-trace dump-ee <src> [--out FILE] # 32 MiB EE RAM image
gc-trace find <src> --bytes HEX # content search (the locators use it too)
gc-trace read <src> --addr HEX --len N
gc-trace info <src> # savestate metadata
gc-trace compare-tfrag-light <unpacked-dir> [--level N] <src>
```
`<src>` is `--pine [slot]`, `--state FILE` / `--state latest`, or `--ee FILE`.
It may sit anywhere on the command line.
The comparison:
1. **Locates the tfrags block in RAM by content** -- the key is tfrag 0's
bounding sphere, sixteen bytes unique in 32 MiB. A candidate is accepted
only when the block header matches, **every** header matches the disc on
the invariant bytes (everything but the relocated `data` pointer and the
runtime occlusion index), and every `data` pointer is the relocated form
`table + disc offset`. Anything less is a plausible wrong read.
2. **Locates the light bank the same way**, by its exact section bytes.
3. **Runs the pass** over the same disc data with
`gc_engine::tfrag_light::light_tfrag`.
4. **Diffs per channel, per vertex**, bucketed by vertex class, with the first
few mismatching vertices' *terms* (`gc_engine::tfrag_light::
light_vertex_terms`) printed so a mismatch can be attributed.
## Result: level 0 (Veldin), verified 2026-10-11
Inputs: the live emulator over PINE, and two PCSX2 savestates from the same
level. All three agree byte for byte.
```text
level0: tfrags block at 0x848000, table at 0x848040, 833 tfrags
headers matching the disc on invariant bytes: 833/833
data pointers relocated as table + disc offset: 833/833
light bank in RAM at 0x50ab20, byte-identical to the disc's
vertices compared: 40600 (162400 colour bytes)
equal: 162400 (100.00%)
runtime-set: 3130 vertices, 3130 fully equal
single-set: 37470 vertices, 37470 fully equal
```
What this establishes:
- **The tfrag locator and the pointer model are right.** 833 of 833 headers
match on invariant bytes and 833 of 833 `data` pointers are relocated
exactly `table + disc offset`, so the block found in RAM is the level's and
the geometry read out of it is the geometry the game drew.
- **The light bank framing is right.** The section found in RAM is
byte-identical to the disc's: offset at `+0x04`, an `i32` set count, twelve
bytes of pad, then 0x40-byte sets. The previous reading -- offset at
`+0x00`, count at `+0x04` -- was wrong, and this is what settled it.
- **The lighting magnitudes match exactly**: every one of the 162,400 colour
bytes the game wrote equals ours. This was the open item in
`docs/roadmap.md` and `docs/plan/level-assets.md` ("Lighting magnitudes are
unverified: they need comparison against reference frames").
## Two bugs the comparison found
Both produced plausible output and neither was visible in a screenshot.
**1. The normal's z was not negated.** `vertex_normal_bits` negated x and y
and left z positive, while the renderer's `NormalTable::normal` negates all
three. The result: `dA = -0.707` where the game had `+0.707`, so
`max(dA, dA·wA)` picked the wrong branch and every vertex lit near its base
colour -- dark, but not *wrong* dark. Level 0 agreed on 95.88% of colour
bytes with the bug and 100.00% without it.
**2. A record with no light set drew transparent black.** `select 0x000f`
names set 15; the level's directional bank has 5 sets, so the pass had no set
to light with and yielded `[0,0,0,0]`. The game writes the record's **base
colour**: 3,130 of level 0's 40,600 vertices do this. The vertex is unlit, not
invisible.
## Finding the next thing
- **Compare another level.** All three inputs available are level 0; another
level needs a savestate taken in it (F1 in-game, or `gc-trace
pine-savestate SLOT`). The harness takes `--level N` already.
- **Point lights.** The runtime-set cluster is set 15 and above; those slots
are filled by gameplay, so they cannot be verified from a level-load image.
Comparing them needs a frame with point lights in it.
- **Blended sets.** Level 0's vertices are all single-set (`select & 0xff00 ==
0`), so the blend path -- the only path `rsqrt` touches -- is untested
against the game.
- **Distil, then test.** A comparison is not a regression. When a fact from
here becomes load-bearing, distil it into a numbers-only fixture under
`tools/trace/tests/` and test against that; no test may read a savestate, a
dump or a live emulator.
## Running it
```sh
cargo run -q -p gc-trace -- pine-info
cargo run -q -p gc-trace -- compare-tfrag-light ~/Documents/gc-extracted --level 0 --pine
cargo run -q -p gc-trace -- compare-tfrag-light ~/Documents/gc-extracted --level 0 \
--state ~/.config/PCSX2/sstates/<name>.p2s
```
Exit status is 0 when every byte matches, 2 on mismatches, 1 on error, so it
can be run as a check. PINE must be enabled in the emulator (Settings →
Advanced); the socket is `$XDG_RUNTIME_DIR/pcsx2.sock` on Linux.
**Nothing here redistributes game data.** The disc, the BIOS and the
savestates are the user's own; the repository holds the tool and the numbers.