Going-Commando/docs/plan/pcsx2-verification.md

# PCSX2 verification (`gc-trace`)

Status as of **2026-10-11**. The tfrag lighting magnitudes are verified; this
is the file that records how, so the next comparison starts from the
harness instead of from scratch.

Tool: `tools/trace` (`gc-trace`). It never ships, no shipped crate depends on
it, and it reads the game's own memory out of a running emulator, a
savestate, or a raw dump.

## What exists now

```text
gc-trace pine-info                       # protocol handshake
gc-trace dump-ee   <src> [--out FILE]     # 32 MiB EE RAM image
gc-trace find      <src> --bytes HEX      # content search (the locators use it too)
gc-trace read      <src> --addr HEX --len N
gc-trace info      <src>                  # savestate metadata
gc-trace compare-tfrag-light <unpacked-dir> [--level N] <src>
```

`<src>` is `--pine [slot]`, `--state FILE` / `--state latest`, or `--ee FILE`.
It may sit anywhere on the command line.

The comparison:

1. **Locates the tfrags block in RAM by content** -- the key is tfrag 0's
   bounding sphere, sixteen bytes unique in 32 MiB. A candidate is accepted
   only when the block header matches, **every** header matches the disc on
   the invariant bytes (everything but the relocated `data` pointer and the
   runtime occlusion index), and every `data` pointer is the relocated form
   `table + disc offset`. Anything less is a plausible wrong read.
2. **Locates the light bank the same way**, by its exact section bytes.
3. **Runs the pass** over the same disc data with
   `gc_engine::tfrag_light::light_tfrag`.
4. **Diffs per channel, per vertex**, bucketed by vertex class, with the first
   few mismatching vertices' *terms* (`gc_engine::tfrag_light::
   light_vertex_terms`) printed so a mismatch can be attributed.

## Result: levels 0 and 1, verified 2026-10-11

Level 0 (Veldin), three inputs -- the live emulator over PINE and two PCSX2
savestates from that level, all agreeing byte for byte:

```text
level0: tfrags block at 0x848000, table at 0x848040, 833 tfrags
  headers matching the disc on invariant bytes: 833/833
  data pointers relocated as table + disc offset: 833/833
  light bank in RAM at 0x50ab20, byte-identical to the disc's

vertices compared: 40600 (162400 colour bytes)
  equal:              162400 (100.00%)
  runtime-set: 3130 vertices, 3130 fully equal
  single-set: 37470 vertices, 37470 fully equal
```

Level 1, from savestate slot 2 and cross-checked against the live emulator:

```text
level1: tfrags block at 0x84c000, table at 0x84c040, 401 tfrags
  headers matching the disc on invariant bytes: 401/401
  data pointers relocated as table + disc offset: 401/401
  light bank in RAM at 0x50f4a0, byte-identical to the disc's

vertices compared: 26470 (105880 colour bytes)
  equal:              105880 (100.00%)
  runtime-set: 129 vertices, 129 fully equal
  single-set: 26341 vertices, 26341 fully equal
```

Level 1 matters more than one more tally: it is a different planet with a
different geometry extent, a different light bank and a different
runtime-set population, and it agrees on every byte. The two fixes below are
therefore not level-0 coincidences.

What this establishes:

- **The tfrag locator and the pointer model are right.** 833 of 833 headers
  match on invariant bytes and 833 of 833 `data` pointers are relocated
  exactly `table + disc offset`, so the block found in RAM is the level's and
  the geometry read out of it is the geometry the game drew.
- **The light bank framing is right.** The section found in RAM is
  byte-identical to the disc's: offset at `+0x04`, an `i32` set count, twelve
  bytes of pad, then 0x40-byte sets. The previous reading -- offset at
  `+0x00`, count at `+0x04` -- was wrong, and this is what settled it.
- **The lighting magnitudes match exactly**: every one of the 162,400 colour
  bytes the game wrote equals ours. This was the open item in
  `docs/roadmap.md` and `docs/plan/level-assets.md` ("Lighting magnitudes are
  unverified: they need comparison against reference frames").

## Two bugs the comparison found

Both produced plausible output and neither was visible in a screenshot.

**1. The normal's z was not negated.** `vertex_normal_bits` negated x and y
and left z positive, while the renderer's `NormalTable::normal` negates all
three. The result: `dA = -0.707` where the game had `+0.707`, so
`max(dA, dA·wA)` picked the wrong branch and every vertex lit near its base
colour -- dark, but not *wrong* dark. Level 0 agreed on 95.88% of colour
bytes with the bug and 100.00% without it.

**2. A record with no light set drew transparent black.** `select 0x000f`
names set 15; the level's directional bank has 5 sets, so the pass had no set
to light with and yielded `[0,0,0,0]`. The game writes the record's **base
colour**: 3,130 of level 0's 40,600 vertices do this. The vertex is unlit, not
invisible.

## Finding the next thing

- **Compare another level.** All three inputs available are level 0; another
  level needs a savestate taken in it (F1 in-game, or `gc-trace
  pine-savestate SLOT`). The harness takes `--level N` already.
- **Point lights.** The runtime-set cluster is set 15 and above; those slots
  are filled by gameplay, so they cannot be verified from a level-load image.
  Comparing them needs a frame with point lights in it.
- **Blended sets.** Both levels' vertices are all single-set (`select &
  0xff00 == 0`), so the blend path -- the only path `rsqrt` touches -- is
  untested against the game.
- **Distil, then test.** A comparison is not a regression. When a fact from
  here becomes load-bearing, distil it into a numbers-only fixture under
  `tools/trace/tests/` and test against that; no test may read a savestate, a
  dump or a live emulator.

## The test setup

PCSX2 is installed and working on this machine, so every comparison above can
be re-run rather than being a one-off observation.

| What | Where |
|---|---|
| Emulator | `~/Desktop/pcsx2/pcsx2-v2.8.2-linux-appimage-x64-Qt.AppImage` — the directory name is **lowercase** on disk |
| Game disc | `~/Desktop/pcsx2/games/Ratchet & Clank - Going Commando.iso`; verified target `SCUS_972.68` v1.01 NTSC ([`docs/disc-identification.md`](../disc-identification.md)) |
| Emulator config | `~/.config/PCSX2/` (`sstates`, `bios`, `memcards`, `inis`, `logs`) |
| BIOS | extracted into `~/.config/PCSX2/bios/` |
| Savestates | `~/.config/PCSX2/sstates/SCUS-97268 (38996035).NN.p2s`, `NN` = slot |
| Unpacked disc | `~/Documents/gc-extracted/`, all 27 levels |
| PINE socket | `$XDG_RUNTIME_DIR/pcsx2.sock` (`/run/user/1000/pcsx2.sock`), slot 28011 |

The emulator boots the disc with `-fastboot`; a level takes about a minute
from a cold start to reach. To capture a state manually, press F1 in-game (it
lands in the slot's `.p2s`); to capture from a script, `gc-trace
pine-savestate SLOT` asks the emulator to write one.

The disc integration tests need the image, not the emulator:

```sh
GC_DISC="$HOME/Desktop/pcsx2/games/Ratchet & Clank - Going Commando.iso" \
    cargo test --workspace
```

## Running it

```sh
cargo run -q -p gc-trace -- pine-info
cargo run -q -p gc-trace -- compare-tfrag-light ~/Documents/gc-extracted --level 0 --pine
cargo run -q -p gc-trace -- compare-tfrag-light ~/Documents/gc-extracted --level 0 \
    --state ~/.config/PCSX2/sstates/<name>.p2s
```

Exit status is 0 when every byte matches, 2 on mismatches, 1 on error, so it
can be run as a check. PINE must be enabled in the emulator (Settings →
Advanced); the socket is `$XDG_RUNTIME_DIR/pcsx2.sock` on Linux.

**Nothing here redistributes game data.** The disc, the BIOS and the
savestates are the user's own; the repository holds the tool and the numbers.

## Shrub packet headers are rewritten in RAM: the loader relocates them

Answering the "what is the tag byte" question needed an anchor, and searching
RAM from scratch was the wrong way to get one -- it finds coincidental matches
(`04 04 00 01` matches unrelated data at `0x1ac988`) and no way to tell them
apart. The savestates are the anchor: slot `.01` is level 0, and searching one
gives candidates that can be *checked* rather than guessed at.

**Range measured:** savestate slot 1,
`~/.config/PCSX2/sstates/SCUS-97268 (38996035).01.p2s`, records at `0x22a6b8`
onward, read with `gc-trace read --state`.

**Result: 40+ records beginning `04 04 00 01 00 00 00 00` at `0x30` spacing**,
which is the packet framing identified on disc, and the loader has rewritten
the second half of each header:

    disc:  04 04 00 01 | 00 00 00 00 | 00 00 00 05 | 00 80 0e 6c
    RAM:   04 04 00 01 | 00 00 00 00 | 13 00 00 30 | 70 6a 93 00

Bytes 0-7 are unchanged. Bytes 8-11 and 12-15 are not:

- on disc, bytes 12-13 are the constant `00 80` and byte 15 the constant `6c`;
- in RAM, bytes 12-15 hold what is plainly a pointer -- `0x00936a70`,
  `0x009376f0`, `0x00937850`, `0x00939c50` in consecutive records, all inside
  EE RAM, and ascending across the table exactly as a walk of consecutive
  objects would be.

Byte 11 becomes `0x30` in every record seen, against a constant `0x05` on
disc, and bytes 8-9 hold a varying small value (`0x13`, `0x16`, `0x24`).

**What this means.** The packet table at `+0x40` is the thing the loader walks:
it holds span-relative offsets on disc and absolute RAM addresses after
loading, which is what the previous pass found and what these 40+ records show
directly. Any header field read past byte 8 on disc is describing something the
loader replaces.

**Consequence for the tag bytes.** The values at record bytes 7 and 15 measured
on disc are *not* stable identifiers -- at least byte 15 sits in a field the
loader overwrites with a pointer. The four-value and two-value vocabularies are
real (0x30, 0x6d, 0x80 and 0 appear against 1636 records of otherwise-uniform
bytes), but byte 15 being a pointer in RAM means its disc values are being read
as data by something, not stored as a tag.

**Not settled:** whether byte 7's vocabulary is also rewritten, and whether
`0x6d` marks a payload that the loader expands. That needs the same records
compared one by one against their disc spans, which is the next step and is
now cheap because the anchor exists.

## Following the pointers: packet payloads differ by type

The header pointers found above are live, so they can be dereferenced. Two
consecutive records from the level 0 table at `0x22a6b8` were followed.

**Range measured:** savestate slot 1, reading `0x00936a70` (from the record at
`0x22a6b8`) and `0x009376f0` (from the record at `0x22a6e8`), 64 bytes each.

**Result: the two payloads have nothing in common.**

`0x00936a70` -- fixed-point vertex data:

    8d 8c 80 80  89 87 7c 80  8d 8c 80 80  89 87 7c 80
    4e 4e 4c 80  4e 4e 4c 80  4e 4e 4c 80  4e 4e 4c 80   (repeating)

Every high byte is `0x80` or `0x7c`, and the pattern repeats on a 8-byte
boundary. Read as `s16` these are large negatives in a narrow band around
-32500. That is consistent with Q16.15 fixed-point rather than `f32`, and with
a stride of 4 `s16` per vertex.

`0x009376f0` -- a tagged sub-list, not vertex data:

    00 00 00 30  cd 00 00 00  cd 00 00 00  cd 00 00 00
    cd 00 00 00  01 00 00 05  22 c1 01 6e  00 00 00 00

Structurally different: small integers, a repeating `0xcd`, and `0x30` tags in
the high byte. Nothing here is position data.

**What this supports.** Two records 0x30 apart in the same table hold payloads
of entirely different kinds. That is direct evidence for the type reading of
the header: the table is a tagged collection where the tag selects what the
payload is, not a uniform array of one thing.

It also rules out the alternative I could not previously exclude -- that the
16-byte tiling was a single uniform record format and the varying bytes were
all fields of it. Two adjacent records with unrelated payloads is not that.

**Still open.** The mapping from tag value to payload kind. The `0x30` that
appears in record bytes 8-11 in RAM also appears at the head of the second
payload, which is suggestive but is one coincidence so far and should not be
written up as a link until it is seen in more than one record.