Everything previously reported about five-block sub-lists and fixed-point
payloads was read out of the running game. Checking the same packets on disc
settles what the disc actually stores.
Range measured: level 0, all 29 packets named by the +0x40 tables of the 6
bounded spans, read at their span-relative offsets; 1636 whole 16-byte records
after the 16-byte header.
Neither RAM form exists on disc:
disc RAM
00 00 00 30 per exactly 1 exactly 5
sub-list packet
blocks at 0/32/64 0 of 10 11 of 11
fixed-point byte3 0 of 19 100%
== 0x80
The 29 packets split 10 sub-list / 19 fixed-point by whether they contain
00 00 00 30 at all, and not one of the ten has the block structure, and not one
of the nineteen has the fixed-point marker.
The disc body is a compact 16-byte record stream:
+000: 04 04 00 01 | 00 00 00 00 | 00 00 00 05 | 00 80 0e 6c
+010: 02 00 00 00 | 05 00 00 00 | 32 00 00 00 | 0e 00 00 00
+020: 09 00 00 00 | 00 40 3e 30 | 12 04 00 00 | 05 00 00 00
+030: 09 00 00 00 | 00 40 3e 30 | 12 04 00 00 | 21 00 00 00
Small u32 fields and a repeating 00 40 3e 30 12 04 00 00. Per position over all
1636 records the same two outliers hold: byte 7 takes 4 distinct values and byte
15 takes 2, against 101 to 153 everywhere else.
Byte 7 is per-record, not per-packet: the first record of all 29 packets has
byte 7 = 0x00. The four-value vocabulary varies within a packet body rather than
tagging the packet, which is why grouping packets by it never separated the two
kinds.
Consequence for Phase 2: the payload is encoded on disc and expanded at load, so
a disc parser can bound it and check its signature but cannot read it as either
RAM form. Decoding it means reversing the expansion, not re-describing what the
loader produced. That is the honest boundary of what is known.
674 lines
31 KiB
Markdown
674 lines
31 KiB
Markdown
# PCSX2 verification (`gc-trace`)
|
|
|
|
Status as of **2026-10-11**. The tfrag lighting magnitudes are verified; this
|
|
is the file that records how, so the next comparison starts from the
|
|
harness instead of from scratch.
|
|
|
|
Tool: `tools/trace` (`gc-trace`). It never ships, no shipped crate depends on
|
|
it, and it reads the game's own memory out of a running emulator, a
|
|
savestate, or a raw dump.
|
|
|
|
## What exists now
|
|
|
|
```text
|
|
gc-trace pine-info # protocol handshake
|
|
gc-trace dump-ee <src> [--out FILE] # 32 MiB EE RAM image
|
|
gc-trace find <src> --bytes HEX # content search (the locators use it too)
|
|
gc-trace read <src> --addr HEX --len N
|
|
gc-trace info <src> # savestate metadata
|
|
gc-trace compare-tfrag-light <unpacked-dir> [--level N] <src>
|
|
```
|
|
|
|
`<src>` is `--pine [slot]`, `--state FILE` / `--state latest`, or `--ee FILE`.
|
|
It may sit anywhere on the command line.
|
|
|
|
The comparison:
|
|
|
|
1. **Locates the tfrags block in RAM by content** -- the key is tfrag 0's
|
|
bounding sphere, sixteen bytes unique in 32 MiB. A candidate is accepted
|
|
only when the block header matches, **every** header matches the disc on
|
|
the invariant bytes (everything but the relocated `data` pointer and the
|
|
runtime occlusion index), and every `data` pointer is the relocated form
|
|
`table + disc offset`. Anything less is a plausible wrong read.
|
|
2. **Locates the light bank the same way**, by its exact section bytes.
|
|
3. **Runs the pass** over the same disc data with
|
|
`gc_engine::tfrag_light::light_tfrag`.
|
|
4. **Diffs per channel, per vertex**, bucketed by vertex class, with the first
|
|
few mismatching vertices' *terms* (`gc_engine::tfrag_light::
|
|
light_vertex_terms`) printed so a mismatch can be attributed.
|
|
|
|
## Result: levels 0 and 1, verified 2026-10-11
|
|
|
|
Level 0 (Veldin), three inputs -- the live emulator over PINE and two PCSX2
|
|
savestates from that level, all agreeing byte for byte:
|
|
|
|
```text
|
|
level0: tfrags block at 0x848000, table at 0x848040, 833 tfrags
|
|
headers matching the disc on invariant bytes: 833/833
|
|
data pointers relocated as table + disc offset: 833/833
|
|
light bank in RAM at 0x50ab20, byte-identical to the disc's
|
|
|
|
vertices compared: 40600 (162400 colour bytes)
|
|
equal: 162400 (100.00%)
|
|
runtime-set: 3130 vertices, 3130 fully equal
|
|
single-set: 37470 vertices, 37470 fully equal
|
|
```
|
|
|
|
Level 1, from savestate slot 2 and cross-checked against the live emulator:
|
|
|
|
```text
|
|
level1: tfrags block at 0x84c000, table at 0x84c040, 401 tfrags
|
|
headers matching the disc on invariant bytes: 401/401
|
|
data pointers relocated as table + disc offset: 401/401
|
|
light bank in RAM at 0x50f4a0, byte-identical to the disc's
|
|
|
|
vertices compared: 26470 (105880 colour bytes)
|
|
equal: 105880 (100.00%)
|
|
runtime-set: 129 vertices, 129 fully equal
|
|
single-set: 26341 vertices, 26341 fully equal
|
|
```
|
|
|
|
Level 1 matters more than one more tally: it is a different planet with a
|
|
different geometry extent, a different light bank and a different
|
|
runtime-set population, and it agrees on every byte. The two fixes below are
|
|
therefore not level-0 coincidences.
|
|
|
|
What this establishes:
|
|
|
|
- **The tfrag locator and the pointer model are right.** 833 of 833 headers
|
|
match on invariant bytes and 833 of 833 `data` pointers are relocated
|
|
exactly `table + disc offset`, so the block found in RAM is the level's and
|
|
the geometry read out of it is the geometry the game drew.
|
|
- **The light bank framing is right.** The section found in RAM is
|
|
byte-identical to the disc's: offset at `+0x04`, an `i32` set count, twelve
|
|
bytes of pad, then 0x40-byte sets. The previous reading -- offset at
|
|
`+0x00`, count at `+0x04` -- was wrong, and this is what settled it.
|
|
- **The lighting magnitudes match exactly**: every one of the 162,400 colour
|
|
bytes the game wrote equals ours. This was the open item in
|
|
`docs/roadmap.md` and `docs/plan/level-assets.md` ("Lighting magnitudes are
|
|
unverified: they need comparison against reference frames").
|
|
|
|
## Two bugs the comparison found
|
|
|
|
Both produced plausible output and neither was visible in a screenshot.
|
|
|
|
**1. The normal's z was not negated.** `vertex_normal_bits` negated x and y
|
|
and left z positive, while the renderer's `NormalTable::normal` negates all
|
|
three. The result: `dA = -0.707` where the game had `+0.707`, so
|
|
`max(dA, dA·wA)` picked the wrong branch and every vertex lit near its base
|
|
colour -- dark, but not *wrong* dark. Level 0 agreed on 95.88% of colour
|
|
bytes with the bug and 100.00% without it.
|
|
|
|
**2. A record with no light set drew transparent black.** `select 0x000f`
|
|
names set 15; the level's directional bank has 5 sets, so the pass had no set
|
|
to light with and yielded `[0,0,0,0]`. The game writes the record's **base
|
|
colour**: 3,130 of level 0's 40,600 vertices do this. The vertex is unlit, not
|
|
invisible.
|
|
|
|
## Finding the next thing
|
|
|
|
- **Compare another level.** All three inputs available are level 0; another
|
|
level needs a savestate taken in it (F1 in-game, or `gc-trace
|
|
pine-savestate SLOT`). The harness takes `--level N` already.
|
|
- **Point lights.** The runtime-set cluster is set 15 and above; those slots
|
|
are filled by gameplay, so they cannot be verified from a level-load image.
|
|
Comparing them needs a frame with point lights in it.
|
|
- **Blended sets.** Both levels' vertices are all single-set (`select &
|
|
0xff00 == 0`), so the blend path -- the only path `rsqrt` touches -- is
|
|
untested against the game.
|
|
- **Distil, then test.** A comparison is not a regression. When a fact from
|
|
here becomes load-bearing, distil it into a numbers-only fixture under
|
|
`tools/trace/tests/` and test against that; no test may read a savestate, a
|
|
dump or a live emulator.
|
|
|
|
## The test setup
|
|
|
|
PCSX2 is installed and working on this machine, so every comparison above can
|
|
be re-run rather than being a one-off observation.
|
|
|
|
| What | Where |
|
|
|---|---|
|
|
| Emulator | `~/Desktop/pcsx2/pcsx2-v2.8.2-linux-appimage-x64-Qt.AppImage` — the directory name is **lowercase** on disk |
|
|
| Game disc | `~/Desktop/pcsx2/games/Ratchet & Clank - Going Commando.iso`; verified target `SCUS_972.68` v1.01 NTSC ([`docs/disc-identification.md`](../disc-identification.md)) |
|
|
| Emulator config | `~/.config/PCSX2/` (`sstates`, `bios`, `memcards`, `inis`, `logs`) |
|
|
| BIOS | extracted into `~/.config/PCSX2/bios/` |
|
|
| Savestates | `~/.config/PCSX2/sstates/SCUS-97268 (38996035).NN.p2s`, `NN` = slot |
|
|
| Unpacked disc | `~/Documents/gc-extracted/`, all 27 levels |
|
|
| PINE socket | `$XDG_RUNTIME_DIR/pcsx2.sock` (`/run/user/1000/pcsx2.sock`), slot 28011 |
|
|
|
|
The emulator boots the disc with `-fastboot`; a level takes about a minute
|
|
from a cold start to reach. To capture a state manually, press F1 in-game (it
|
|
lands in the slot's `.p2s`); to capture from a script, `gc-trace
|
|
pine-savestate SLOT` asks the emulator to write one.
|
|
|
|
The disc integration tests need the image, not the emulator:
|
|
|
|
```sh
|
|
GC_DISC="$HOME/Desktop/pcsx2/games/Ratchet & Clank - Going Commando.iso" \
|
|
cargo test --workspace
|
|
```
|
|
|
|
## Running it
|
|
|
|
```sh
|
|
cargo run -q -p gc-trace -- pine-info
|
|
cargo run -q -p gc-trace -- compare-tfrag-light ~/Documents/gc-extracted --level 0 --pine
|
|
cargo run -q -p gc-trace -- compare-tfrag-light ~/Documents/gc-extracted --level 0 \
|
|
--state ~/.config/PCSX2/sstates/<name>.p2s
|
|
```
|
|
|
|
Exit status is 0 when every byte matches, 2 on mismatches, 1 on error, so it
|
|
can be run as a check. PINE must be enabled in the emulator (Settings →
|
|
Advanced); the socket is `$XDG_RUNTIME_DIR/pcsx2.sock` on Linux.
|
|
|
|
**Nothing here redistributes game data.** The disc, the BIOS and the
|
|
savestates are the user's own; the repository holds the tool and the numbers.
|
|
|
|
## Shrub packet headers are rewritten in RAM: the loader relocates them
|
|
|
|
Answering the "what is the tag byte" question needed an anchor, and searching
|
|
RAM from scratch was the wrong way to get one -- it finds coincidental matches
|
|
(`04 04 00 01` matches unrelated data at `0x1ac988`) and no way to tell them
|
|
apart. The savestates are the anchor: slot `.01` is level 0, and searching one
|
|
gives candidates that can be *checked* rather than guessed at.
|
|
|
|
**Range measured:** savestate slot 1,
|
|
`~/.config/PCSX2/sstates/SCUS-97268 (38996035).01.p2s`, records at `0x22a6b8`
|
|
onward, read with `gc-trace read --state`.
|
|
|
|
**Result: 40+ records beginning `04 04 00 01 00 00 00 00` at `0x30` spacing**,
|
|
which is the packet framing identified on disc, and the loader has rewritten
|
|
the second half of each header:
|
|
|
|
disc: 04 04 00 01 | 00 00 00 00 | 00 00 00 05 | 00 80 0e 6c
|
|
RAM: 04 04 00 01 | 00 00 00 00 | 13 00 00 30 | 70 6a 93 00
|
|
|
|
Bytes 0-7 are unchanged. Bytes 8-11 and 12-15 are not:
|
|
|
|
- on disc, bytes 12-13 are the constant `00 80` and byte 15 the constant `6c`;
|
|
- in RAM, bytes 12-15 hold what is plainly a pointer -- `0x00936a70`,
|
|
`0x009376f0`, `0x00937850`, `0x00939c50` in consecutive records, all inside
|
|
EE RAM, and ascending across the table exactly as a walk of consecutive
|
|
objects would be.
|
|
|
|
Byte 11 becomes `0x30` in every record seen, against a constant `0x05` on
|
|
disc, and bytes 8-9 hold a varying small value (`0x13`, `0x16`, `0x24`).
|
|
|
|
**What this means.** The packet table at `+0x40` is the thing the loader walks:
|
|
it holds span-relative offsets on disc and absolute RAM addresses after
|
|
loading, which is what the previous pass found and what these 40+ records show
|
|
directly. Any header field read past byte 8 on disc is describing something the
|
|
loader replaces.
|
|
|
|
**Consequence for the tag bytes.** The values at record bytes 7 and 15 measured
|
|
on disc are *not* stable identifiers -- at least byte 15 sits in a field the
|
|
loader overwrites with a pointer. The four-value and two-value vocabularies are
|
|
real (0x30, 0x6d, 0x80 and 0 appear against 1636 records of otherwise-uniform
|
|
bytes), but byte 15 being a pointer in RAM means its disc values are being read
|
|
as data by something, not stored as a tag.
|
|
|
|
**Not settled:** whether byte 7's vocabulary is also rewritten, and whether
|
|
`0x6d` marks a payload that the loader expands. That needs the same records
|
|
compared one by one against their disc spans, which is the next step and is
|
|
now cheap because the anchor exists.
|
|
|
|
## Following the pointers: packet payloads differ by type
|
|
|
|
The header pointers found above are live, so they can be dereferenced. Two
|
|
consecutive records from the level 0 table at `0x22a6b8` were followed.
|
|
|
|
**Range measured:** savestate slot 1, reading `0x00936a70` (from the record at
|
|
`0x22a6b8`) and `0x009376f0` (from the record at `0x22a6e8`), 64 bytes each.
|
|
|
|
**Result: the two payloads have nothing in common.**
|
|
|
|
`0x00936a70` -- fixed-point vertex data:
|
|
|
|
8d 8c 80 80 89 87 7c 80 8d 8c 80 80 89 87 7c 80
|
|
4e 4e 4c 80 4e 4e 4c 80 4e 4e 4c 80 4e 4e 4c 80 (repeating)
|
|
|
|
Every high byte is `0x80` or `0x7c`, and the pattern repeats on a 8-byte
|
|
boundary. Read as `s16` these are large negatives in a narrow band around
|
|
-32500. That is consistent with Q16.15 fixed-point rather than `f32`, and with
|
|
a stride of 4 `s16` per vertex.
|
|
|
|
`0x009376f0` -- a tagged sub-list, not vertex data:
|
|
|
|
00 00 00 30 cd 00 00 00 cd 00 00 00 cd 00 00 00
|
|
cd 00 00 00 01 00 00 05 22 c1 01 6e 00 00 00 00
|
|
|
|
Structurally different: small integers, a repeating `0xcd`, and `0x30` tags in
|
|
the high byte. Nothing here is position data.
|
|
|
|
**What this supports.** Two records 0x30 apart in the same table hold payloads
|
|
of entirely different kinds. That is direct evidence for the type reading of
|
|
the header: the table is a tagged collection where the tag selects what the
|
|
payload is, not a uniform array of one thing.
|
|
|
|
It also rules out the alternative I could not previously exclude -- that the
|
|
16-byte tiling was a single uniform record format and the varying bytes were
|
|
all fields of it. Two adjacent records with unrelated payloads is not that.
|
|
|
|
**Still open.** The mapping from tag value to payload kind. The `0x30` that
|
|
appears in record bytes 8-11 in RAM also appears at the head of the second
|
|
payload, which is suggestive but is one coincidence so far and should not be
|
|
written up as a link until it is seen in more than one record.
|
|
|
|
## The tag values map onto three payload kinds
|
|
|
|
Following the pointers out of the level 0 table, all nine that carry the `0x30`
|
|
tag at byte 11 and an EE pointer at 12..15 were dereferenced and read 32 bytes
|
|
each. They fall into exactly three classes, three payloads each, with no
|
|
overlap.
|
|
|
|
**A -- Q16.15 fixed-point vertex data** (3 of 9)
|
|
|
|
00936a70 8d 8c 80 80 89 87 7c 80 8d 8c 80 80 89 87 7c 80
|
|
4e 4e 4c 80 4e 4e 4c 80 4e 4e 4c 80 4e 4e 4c 80
|
|
0093cde0 8a 8a 7f 80 92 90 84 80 8a 8a 7f 80 92 90 84 80
|
|
4e 4e 4c 80 4e 4e 4c 80 4e 4e 4c 80 4e 4e 4c 80
|
|
009cf4f0 6e 6f 68 80 6e 6f 68 80 6e 6f 68 80 6e 6f 68 80
|
|
54 54 51 80 54 54 51 80 54 54 51 80 54 54 51 80
|
|
|
|
Every high byte is `0x80` or `0x7c`, each payload repeats on an 8-byte
|
|
boundary, and the values differ between payloads while the `4e 4e 4c 80` tail
|
|
is shared by two of them.
|
|
|
|
**B -- a `0x30` sub-list with a repeated count** (3 of 9)
|
|
|
|
009376f0 00 00 00 30 cd 00 00 00 cd 00 00 00 cd 00 00 00
|
|
cd 00 00 00 01 00 00 05 22 c1 01 6e 00 00 00 00
|
|
00998880 00 00 00 30 c0 00 00 00 c0 00 00 00 c0 00 00 00
|
|
c0 00 00 00 01 00 00 05 11 c1 01 6e 00 00 00 00
|
|
009d2da0 00 00 00 30 c8 00 00 00 c8 00 00 00 c8 00 00 00
|
|
c8 00 00 00 01 00 00 05 1d c1 01 6e 00 00 00 00
|
|
|
|
Structurally identical apart from the repeated word: `0xcd`, `0xc0`, `0xc8`.
|
|
All three carry `01 00 00 05` at offset 16 and a `.. c1 01 6e` at 20.
|
|
|
|
**C -- an index/attribute list** (3 of 9)
|
|
|
|
00939700 00 c0 05 6d 3e 00 00 00 01 00 00 00 12 00 00 00
|
|
18 00 c0 00 fe 00 fe 00 ff 00 ff 00 11 01 17 01
|
|
009c1540 00 c0 05 6d 36 00 00 00 01 00 00 00 08 00 00 00
|
|
13 00 93 00 c9 00 c9 00 ca 00 ca 00 d2 00 d5 00
|
|
|
|
Both open `00 c0 05 6d`, then a count, then a run of small ascending `u16`.
|
|
|
|
**The mapping.** The tag vocabulary measured on disc -- `0x00`, `0x30`,
|
|
`0x6d`, `0x80` at record byte 7 -- is not decorative:
|
|
|
|
| tag | payload |
|
|
|---|---|
|
|
| `0x30` | class B, the `00 00 00 30` sub-list |
|
|
| `0x6d` | class C, the `00 c0 05 6d` attribute list |
|
|
| `0x00` | class A, the fixed-point vertex data |
|
|
|
|
`0x30` heads the class B payloads and `0x6d` heads the class C payloads, which
|
|
is the link flagged as a single coincidence two entries back. It is now seen in
|
|
three payloads each, so it is a rule rather than an accident.
|
|
|
|
`0x80` did not appear among these nine. Its meaning is still open -- it is the
|
|
fourth value in the vocabulary and there is no sample of it here.
|
|
|
|
**A correction.** The record spacing in this table is not uniformly `0x30`. The
|
|
`find` output shows `0x30`, `0x10`, `0x30`, `0x30` between consecutive packet
|
|
headers, and only 9 of the 5,632 bytes scanned carry both the `0x30` tag and an
|
|
EE pointer. The earlier "records at 0x30 spacing" was inferred from a truncated
|
|
list of hits and does not hold.
|
|
|
|
## Correction: the record tag does not select the payload
|
|
|
|
The previous entry mapped the disc tag vocabulary onto the three payload
|
|
classes. A wider scan says that is wrong.
|
|
|
|
**Range measured:** savestate slot 1, RAM `0x22a000..0x24037f` -- 91,008 bytes,
|
|
20 times the span scanned before.
|
|
|
|
**Result: 19 packet headers, and all 19 carry byte 11 = `0x30`.** There are no
|
|
`0x6d` or `0x80` tags anywhere in this range. Reading six of their payloads:
|
|
|
|
rec 0022a6b8 tag 30 -> 00936a70 8d 8c 80 80 89 87 7c 80 ... (fixed point)
|
|
rec 0022a6e8 tag 30 -> 009376f0 00 00 00 30 cd 00 00 00 ... (sub-list)
|
|
rec 0022a6f8 tag 30 -> 00937850 92 90 84 80 94 94 86 80 ... (fixed point)
|
|
rec 0022a728 tag 30 -> 00939c50 00 00 00 30 c0 00 00 00 ... (sub-list)
|
|
rec 0022a738 tag 30 -> 00939e90 4c 4c 4b 80 4c 4c 4b 80 ... (fixed point)
|
|
rec 0022a768 tag 30 -> 0093cc70 00 00 00 30 78 00 00 00 ... (sub-list)
|
|
|
|
Records of identical tag alternate between two payload kinds. **The tag does
|
|
not select the payload.**
|
|
|
|
**What went wrong.** The previous entry took the `0x30` and `0x6d` that *head
|
|
the payloads* to be the disc-side tag vocabulary, and matched them to the
|
|
payload classes by that correspondence. The disc vocabulary measured at record
|
|
byte 7 -- `0x00`, `0x30`, `0x6d`, `0x80` -- and the RAM record byte 11 were
|
|
treated as the same field. They are not: the payload magics are a property of
|
|
the payload, and record byte 11 is a different thing that happens to be
|
|
constantly `0x30` here.
|
|
|
|
Three payloads each made the coincidence look like a rule. It was a
|
|
correspondence between two vocabularies that overlap in two values.
|
|
|
|
**What is actually established about the RAM header**, per position:
|
|
|
|
| byte | value |
|
|
|---|---|
|
|
| 0-3 | `04 04 00 01`, constant |
|
|
| 4-7 | zero, constant |
|
|
| 8 | a **count**, `0x0b` to `0x29` across the 19 records |
|
|
| 9-10 | zero, constant |
|
|
| 11 | `0x30`, constant on all 19 in this range |
|
|
| 12-15 | an EE RAM pointer |
|
|
|
|
Byte 8 is the finding that survives. It is a small count that varies per
|
|
record, and it is the one field here that is doing arithmetic rather than
|
|
tagging.
|
|
|
|
**Still open:** what `0x30` at byte 11 means, what discriminates the payload
|
|
kinds given that the record tag does not, and the meaning of `0x80`, which has
|
|
no sample in 91 KB of RAM.
|
|
|
|
## The table interleaves two record magics
|
|
|
|
Scanning the same 91 KB for other four-byte record magics beside `04 04 00 01`
|
|
turns up a second one at the same density: **`02 01 00 01`, 10 occurrences**
|
|
against `04 04 00 01`'s 19.
|
|
|
|
**Range measured:** savestate slot 1, RAM `0x22a000..0x24037f`.
|
|
|
|
**Result: the two magics interleave.** Every `02 01 00 01` sits between two
|
|
`04 04 00 01` records:
|
|
|
|
0x22a6b8 04 04 00 01 13 00 00 30 70 6a 93 00
|
|
0x22a6c8 02 01 00 01 19 c0 4c 6e 00 00 10 00 00 00 00 00
|
|
0x22a6e8 04 04 00 01 16 00 00 30 f0 76 93 00
|
|
0x22a6f8 04 04 00 01 16 00 00 30 50 78 93 00
|
|
0x22a708 02 01 00 01 1e c0 58 6e 00 00 10 00 00 00 00 00
|
|
0x22a728 04 04 00 01 24 00 00 30 50 9c 93 00
|
|
0x22a738 04 04 00 01 16 00 00 30 50 9e 93 00
|
|
0x22a748 02 01 00 01 19 c0 54 6e 00 00 10 00 00 00 00 00
|
|
|
|
The `02 01 00 01` records are 16 bytes like the others but carry **no pointer
|
|
** -- bytes 12-15 are zero in all ten. Their fields are a count at 4-5
|
|
(`0x14`, `0x19`, `0x1e`), a constant `0xc0` at 6, a varying byte at 7
|
|
(`0x30`, `0x4c`, `0x54`, `0x58`), `0x6e` at 8-9, and `10 00` at 10-11.
|
|
|
|
**The `0x6e` link.** These records carry `0x6e` at bytes 8-9, and the sub-list
|
|
payloads found earlier carry `.. c1 01 6e` at offset 20-23. That is a second
|
|
correspondence between this record type and the sub-list payloads, independent
|
|
of the one retracted in the previous entry -- and this one is a shared *value*
|
|
rather than a shared *position*.
|
|
|
|
**What this does not settle.** It still does not say what discriminates the two
|
|
payload kinds among the `04 04 00 01` records, because those all carry the same
|
|
magic, the same `0x30` at byte 11 and differ only in the count at byte 8. The
|
|
`02 01 00 01` records sit between them rather than replacing them, so they are
|
|
probably a companion structure, not the missing discriminator.
|
|
|
|
Still open: what distinguishes a fixed-point payload from a sub-list payload
|
|
when the records carrying them look identical in bytes 0-3 and 11. The most
|
|
likely place to look is the count at byte 8 against the payload's own length.
|
|
|
|
## The count at byte 8 sizes the sub-list payload: exactly 16 bytes per record
|
|
|
|
The previous entry left one test: compare the count at byte 8 against the
|
|
payload's own extent. Pointer spacing gives the extent, since the packets are
|
|
laid out in ascending order in EE RAM.
|
|
|
|
**Range measured:** savestate slot 1, the 17 packets in RAM
|
|
`0x22a000..0x24037f`, each with its record address, byte-8 count and pointer,
|
|
sorted by pointer so the next packet is the next object in memory.
|
|
|
|
**Result: eight packets sit exactly `count * 16` bytes after their predecessor
|
|
-- an integer ratio of 16.00 in every one, with no rounding.**
|
|
|
|
record count ptr gap gap/count
|
|
0x22a6e8 22 009376f0 352 16.00
|
|
0x22a728 36 00939c50 576 16.00
|
|
0x22a768 23 0093cc70 368 16.00
|
|
0x22a7a8 36 00998880 576 16.00
|
|
0x22a7e8 25 009c19c0 400 16.00
|
|
0x22a828 24 009cf370 384 16.00
|
|
0x22a868 41 009d2da0 656 16.00
|
|
0x22a8a8 21 009f5aa0 336 16.00
|
|
|
|
Eight exact hits at a ratio that has to be an integer is not sampling luck. The
|
|
count at byte 8 is the **record count**, and these payloads are 16 bytes per
|
|
record.
|
|
|
|
**And these are exactly the sub-list payloads** -- `0x009376f0`, `0x00939c50`,
|
|
`0x0093cc70`, `0x00998880`, `0x009d2da0` were all read directly in earlier
|
|
passes and all open `00 00 00 30` followed by a repeated word.
|
|
|
|
**What the other nine are.** Their ratios are 168, 419, 559, 31288, 7985,
|
|
3458, 764, 6452 -- no shared factor, and far too large to be a record count
|
|
times anything. So either their payload is not followed immediately by the next
|
|
packet in the pointer order (other data intervenes), or they are not 16 bytes
|
|
per record. Both readings are open; what is ruled out is that they follow the
|
|
same rule.
|
|
|
|
**This resolves byte 8.** It was measured as "a count, 0x0b to 0x29" two entries
|
|
ago and is now shown to size a payload exactly, in eight independent cases,
|
|
within a single level. Whether it also sizes the fixed-point payloads is the
|
|
open half.
|
|
|
|
Still open: why the fixed-point packets do not follow the same rule, and what
|
|
the 16 bytes per sub-list record are -- the repeated word (`0xcd`, `0xc0`,
|
|
`0xc8`, `0x78`) differs per packet, so the records are not identical.
|
|
|
|
## Inside a sub-list payload: tagged blocks and a byte permutation table
|
|
|
|
With the extent known (`count * 16`), a whole sub-list payload can be read.
|
|
This is the packet at `0x009376f0`, count 22, 352 bytes, in full.
|
|
|
|
**Range measured:** savestate slot 1, `0x009376f0`..`0x0093784f`, 352 bytes.
|
|
|
|
**Result: the payload is a sequence of blocks, each opening `00 00 00 30`,**
|
|
carrying a repeating word and then a short trailer:
|
|
|
|
00 00 00 30 | cd 00 00 00 cd 00 00 00 cd 00 00 00 cd 00 00 00
|
|
01 00 00 05 | 22 c1 01 6e | 00 00 00 00
|
|
00 00 00 30 | 00 00 00 45 00 00 00 45 1d 00 00 00 1d 00 00 00 1f
|
|
81 01 6d 00 | 00 00 00 00 | 00 00 a4 00
|
|
00 00 00 30 | bc c0 03 00 23 17 03 00 e5 dd 00 00 00 00 00 00
|
|
02 01 00 01 | c1 80 01 69 | 00 00 00 00
|
|
|
|
The repeating word differs per block -- `0xcd`, `0x45`, `0x03` -- and the
|
|
trailer's second word varies in its high bytes: `22 c1 01 6e`, `81 01 6d 00`,
|
|
`c1 80 01 69`. Every block seen ends `... 6e` or `69` or `6d` at byte 23, which
|
|
is the same `6e` the `02 01 00 01` records carried at bytes 8-9.
|
|
|
|
**The payload contains a byte permutation table.** At `0x009377c4`:
|
|
|
|
00 01 02 03 3e 3f 53 54 40 41 42 43 4e 4f 50 51 04 05 06 07 08 ...
|
|
|
|
This is not a position or vertex list -- it is a run of ascending byte values
|
|
with a handful (`3e 3f`, `53 54`, `40 41 42 43`, `4e 4f 50 51`) lifted out of
|
|
order ahead of the sequential run. Read as a table, it maps each of the 0..0x39
|
|
symbols to a slot. That is the shape of a **code-length or symbol-ordering
|
|
table**, and it is the first thing in this format that is self-evidently a table
|
|
rather than data to be interpreted.
|
|
|
|
**What this does not say.** Whether the table is a Huffman-style code, an index
|
|
reorder for the vertex data that follows, or a palette permutation. The
|
|
following bytes continue the sequence to `39` and then `23 c1 01 6e`, after
|
|
which the next block opens.
|
|
|
|
**Corroboration worth noting.** The fixed-point payload `0x00937850` begins
|
|
immediately after this one ends at `0x00937850` exactly -- the 22-record
|
|
identity from the previous entry made the boundary exact, and the next packet
|
|
starts on a payload boundary with nothing in between. The two payload kinds sit
|
|
adjacent in RAM, which is consistent with them being two records of the same
|
|
blob rather than two unrelated objects.
|
|
|
|
## The sub-list payload has a fixed five-block shape
|
|
|
|
All eight sub-list payloads from level 0 were read to their exact extent
|
|
(`count * 16`) and compared block by block. The structure is the same in every
|
|
one.
|
|
|
|
**Range measured:** savestate slot 1, the eight packets at `0x009376f0`,
|
|
`0x00939c50`, `0x0093cc70`, `0x00998880`, `0x009c19c0`, `0x009cf370`,
|
|
`0x009d2da0` and `0x009f5aa0`, counts 21 to 41, each truncated to exactly
|
|
`count * 16` bytes.
|
|
|
|
**Result: every payload contains exactly five blocks tagged `00 00 00 30`, and
|
|
the first three are always at offsets 0, 32 and 64.**
|
|
|
|
| payload | count | blocks at |
|
|
|---|---|---|
|
|
| `009376f0` | 22 | 0, 32, 64, 184, 304 |
|
|
| `00939c50` | 36 | 0, 32, 64, 268, 400 |
|
|
| `0093cc70` | 23 | 0, 32, 64, 200, 288 |
|
|
| `00998880` | 36 | 0, 32, 64, 272, 404 |
|
|
| `009c19c0` | 25 | 0, 32, 64, 196, 300 |
|
|
| `009cf370` | 24 | 0, 32, 64, 204, 316 |
|
|
| `009d2da0` | 41 | 0, 32, 64, 308, 448 |
|
|
| `009f5aa0` | 21 | 0, 32, 64, 176, 280 |
|
|
|
|
Blocks 4 and 5 move with the payload; the first three do not. Eight payloads,
|
|
eight different block-4/5 positions, and 0, 32, 64 in every case.
|
|
|
|
**Per-block fields, constant across all eight:**
|
|
|
|
| block | byte 4 | other |
|
|
|---|---|---|
|
|
| 0 | varies | `0x78`-`0xcd` |
|
|
| 1 | **0** in 8/8 | trailer `00 00 00 00` in 8/8 |
|
|
| 2 | varies independently | bytes 16-20 `00000000` in 8/8 |
|
|
| 3 | **equals block 0 in 8/8** | |
|
|
| 4 | **0** in 8/8 | |
|
|
|
|
The block-0/block-3 equality is exact in all eight:
|
|
|
|
009376f0 0xcd 0x00 0xbc 0xcd 0x00
|
|
00939c50 0xc0 0x00 0xbf 0xc0 0x00
|
|
0093cc70 0x78 0x00 0x30 0x78 0x00
|
|
00998880 0xc0 0x00 0xc9 0xc0 0x00
|
|
009c19c0 0x93 0x00 0x1f 0x93 0x00
|
|
009cf370 0xb5 0x00 0x0d 0xb5 0x00
|
|
009d2da0 0xc8 0x00 0x65 0xc8 0x00
|
|
009f5aa0 0x9b 0x00 0x25 0x9b 0x00
|
|
|
|
Block 2's word is unrelated to block 0's in every case, so the equality is not
|
|
every block reading a shared field.
|
|
|
|
**Correction to the previous entry.** The "byte permutation table" reported
|
|
there does not generalise. It appears in `0x009376f0` at offset 212, but a
|
|
scan for it across all eight payloads finds no run of more than a few bytes
|
|
outside that one. What looked like a table is better described as one payload's
|
|
block content. The `00 00 00 30` block structure, the fixed 0/32/64 offsets and
|
|
the block-0/block-3 equality are the findings that survive eight-way
|
|
comparison; the permutation table was not among them.
|
|
|
|
**Not established.** What the five blocks are, why blocks 0 and 3 carry the
|
|
same word, and what moves blocks 4 and 5 -- their offsets are not monotone in
|
|
the count (`count` 21 gives 280 and `count` 36 gives 400 and 404), so something
|
|
inside the earlier blocks sets them.
|
|
|
|
## Partitioned across two levels: the sub-list shape is universal, the payload kinds are not mixed
|
|
|
|
The five-block result was measured on level 0 only. Rule 3a asks for the same
|
|
measurement per partition, so it was repeated on level 1's savestate before
|
|
being written up as a property.
|
|
|
|
**Range measured:** savestate slot 2 (level 1), RAM `0x22a000..0x28f52f`,
|
|
415,024 bytes, then six of its pointer-bearing records read to `count * 16`.
|
|
|
|
**The six sampled pointers split 3/3 between the two payload kinds:**
|
|
|
|
| pointer | count | first four bytes | classification |
|
|
|---|---|---|---|
|
|
| `00949430` | 41 | `00 00 00 30` | sub-list |
|
|
| `009496c0` | 13 | `69 b0 bf 80` | fixed-point |
|
|
| `00957970` | 36 | `00 00 00 30` | sub-list |
|
|
| `00957bb0` | 10 | `58 7e 8d 80` | fixed-point |
|
|
| `0095b740` | 12 | `37 49 41 80` | fixed-point |
|
|
| `0095d6d0` | 57 | `00 00 00 30` | sub-list |
|
|
|
|
Taking all six as sub-list gives 3/6 with five blocks. Taking only the three
|
|
that actually open `00 00 00 30` gives **3/3**, and the first three blocks land
|
|
at 0, 32, 64 in all three:
|
|
|
|
00949430 41 blocks at 0, 32, 64, 308, 428
|
|
00957970 36 blocks at 0, 32, 64, 268, 376
|
|
0095d6d0 57 blocks at 0, 32, 64, 400, 572
|
|
|
|
The three fixed-point payloads have **zero** `00 00 00 30` blocks, as
|
|
expected. So the earlier 3/6 was my sampling, not a counterexample: the six
|
|
pointers were never all the same kind of object.
|
|
|
|
**Combined: 11 of 11 sub-list payloads across both levels have exactly five
|
|
blocks, the first three at offsets 0, 32, 64.**
|
|
|
|
**The fixed-point payloads have an exact signature.** Byte 3 of every 4-byte
|
|
group is `0x80` in 100% of cases -- 52/52, 40/40 and 48/48 over the first 512
|
|
bytes of the three level 1 samples, and the same holds across all nine level 0
|
|
payloads. That is a clean one-bit-per-group marker and is what separates the
|
|
two payload kinds without needing the record tag, which the previous entries
|
|
established does not select them.
|
|
|
|
**Level 1's record tags are not level 0's.** Scanning the same 64 KB for
|
|
`04 04 00 01` headers gives 18, against 19 on level 0, plus 6 `02 01 00 01`
|
|
against 10. And byte 11 is no longer constant: `0x30` on 10, `0x00` on 5,
|
|
`0x20` on 2 and `0xbc` on 1. Level 0 had `0x30` on all 19.
|
|
|
|
That corrects an earlier statement. "Byte 11 is `0x30`, constant" was a
|
|
level-0 partition result written up as universal. Across both levels it takes
|
|
four values. The constancy was never a property of the format.
|
|
|
|
**Fixed-point payload entropy**, all nine level 0 payloads read 4,096 bytes:
|
|
5.51 to 5.89 bits per byte, against 8.0 for uniform. Structured rather than
|
|
random, and `mean` of the `s16` reading varies per payload from -82 to -907,
|
|
which is consistent with vertex positions relative to a per-payload origin.
|
|
|
|
**Not established.** What the five blocks are, and how the fixed-point data
|
|
decodes to vertices -- the payload entropy is far below random, so it is
|
|
compressed or permuted rather than stored plainly, and the permutation table
|
|
seen once in `0x009376f0` remains unexplained.
|
|
|
|
## The loader produces the RAM forms: the disc payload is neither of them
|
|
|
|
Everything above about five-block sub-lists and fixed-point payloads was read
|
|
out of the running game. Checking the same packets on disc settles what the
|
|
disc actually stores.
|
|
|
|
**Range measured:** level 0, all 29 packets named by the `+0x40` tables of the 6
|
|
bounded spans, read at their span-relative offsets; 1,636 whole 16-byte records
|
|
after the 16-byte header.
|
|
|
|
**Result: neither RAM form exists on disc.**
|
|
|
|
| | disc | RAM |
|
|
|---|---|---|
|
|
| `00 00 00 30` per sub-list packet | **exactly 1** | exactly 5 |
|
|
| sub-list blocks at 0, 32, 64 | **0 of 10** | 11 of 11 |
|
|
| fixed-point byte 3 == `0x80` | **0 of 19** | 100% |
|
|
|
|
The 29 packets split 10 sub-list / 19 fixed-point by whether they contain
|
|
`00 00 00 30` at all, and not one of the ten has the block structure, and not
|
|
one of the nineteen has the fixed-point marker.
|
|
|
|
**What the disc body actually is.** A compact 16-byte record stream:
|
|
|
|
+000: 04 04 00 01 | 00 00 00 00 | 00 00 00 05 | 00 80 0e 6c
|
|
+010: 02 00 00 00 | 05 00 00 00 | 32 00 00 00 | 0e 00 00 00
|
|
+020: 09 00 00 00 | 00 40 3e 30 | 12 04 00 00 | 05 00 00 00
|
|
+030: 09 00 00 00 | 00 40 3e 30 | 12 04 00 00 | 21 00 00 00
|
|
|
|
Small `u32` fields and a repeating `00 40 3e 30 12 04 00 00`. Per position over
|
|
all 1,636 records the same two outliers hold as on the earlier disc pass: byte 7
|
|
takes 4 distinct values and byte 15 takes 2, against 101 to 153 for every other
|
|
position.
|
|
|
|
**And byte 7 is per-record, not per-packet.** The first record of all 29
|
|
packets has byte 7 = `0x00`. So the four-value vocabulary is a field that varies
|
|
within a packet body, not a tag on the packet -- which is why grouping packets
|
|
by it never separated the two kinds.
|
|
|
|
**Consequence for Phase 2.** The packet payload is encoded on disc and expanded
|
|
at load, so the disc parser can bound it and check its signature but cannot read
|
|
it as either RAM form. Decoding it means reversing the expansion, not
|
|
re-describing what the loader produced. That is a larger job than the framing
|
|
work so far, and it is the honest boundary of what is currently known.
|