ReaperOfVeriod
af62032ddc
Comparing every chunk record against the same record index in the other 26
levels, across all 74 signatures:
65 of 74 every record byte-identical in all 27 levels
9 of 74 level-specific
0 ragged no signature appeared with different record counts per level
The nine that vary:
[5d 44 13] [75 42 ef] [7f 44 ee] [82 c4 2f] [96 c3 37]
[a2 44 a5] [c6 c3 ef] [df c3 ea] [f3 c4 b9]
This reframes the container. 65 of 74 chunk types are static tables the game ships
unchanged in every level -- constants and lookups, not level content. The largest
by record count are all in that group: [38 c4 7e] stride 328 with 90 records per
level, [a9 c4 d8] stride 308 with 90, [3d c4 9e] stride 368 with 77.
The consequence is about method, and it is worth being blunt about. For a
level-specific type, comparing levels is the strongest lever available: a field
that moves with the level is identifiable by correlating it with something known.
For the other 65 there is nothing to compare -- one copy of the data, repeated 27
times -- so they must be decoded structurally, from what the bytes are. A future
decoder that reaches for per-level comparison as its first tool will find it
useless on the majority of the container.
[75 42 ef] is both level-specific AND the one signature with three strides. A
type that is both variable-length and level-varying is the most informative one
here, and is where per-type field decoding should start.
How this was found, including the wrong turn:
The first look at [53 c4 6c d7 9a 43] (stride 108) showed 45 of 108 byte positions
varying WITHIN a chunk and 810 records all distinct -- which reads like per-record
data. Counting distinct values per u32 slot across all 810 records contradicted
that: 29 to 30 distinct values across 27 levels, i.e. about one per level.
Both readings were wrong because they measure different things. The 45 positions
vary between records within a level; the u32 counts were low because the same 30
values recur in every level. Comparing record i across levels settled it: all 30
records are byte-identical in all 27 levels, and zero byte positions vary across
levels.
Had the u32 distinct counts been the only measurement, the conclusion would have
been "one value per level" -- the opposite of the truth, describing a per-level
parameter table that does not exist. The within-chunk and across-level views had
to be taken together to get a true one.
Asserted in most_chunk_signatures_are_level_independent. 242 tests, clippy clean.
604 lines
36 KiB
Markdown
604 lines
36 KiB
Markdown
# Roadmap
|
|
|
|
Status as of **2026-10-11**.
|
|
|
|
A from-scratch native Rust reimplementation of Ratchet & Clank: Going Commando
|
|
(PS2). Each phase ends in a **gate** — something checkable, not a feeling that
|
|
it is going well. Phases are ordered so that the riskiest unknowns are retired
|
|
early, while they are still cheap to retire.
|
|
|
|
## Where we are
|
|
|
|
- [x] Target disc identified and verified (`SCUS_972.68` v1.01, NTSC 60 Hz)
|
|
- [x] Workspace builds (`cargo check --workspace`)
|
|
- [x] Repository hygiene in place (pre-commit guard, `.gitignore`, licence
|
|
record)
|
|
- [x] **Phase 1 complete.** The disc pipeline runs end to end and is verified
|
|
against a committed hash table, every one of the 92 asset containers
|
|
parsing. The pipeline now also writes each level's chunk descriptors
|
|
where they exist (3 of 27 levels).
|
|
- [x] **Phase 2 in progress.** Level containers are parsed through their typed
|
|
header, and the `data` extent is parsed as a container of byte ranges. The
|
|
geometry decompresses: level 0 yields 16,969,296 bytes of level content
|
|
from a 9.7 MB stream.
|
|
- [x] **Parsers with real-disc tests, per asset type.** `level_wad`, `data`,
|
|
`core_index` class tables, `tfrag` (format, topology, textures, LOD,
|
|
normals, colour), `occlusion`, `sky`, `moby`, `collision` framing and
|
|
header, `shrub` (blob, packets, palette, billboard), the post-shrub
|
|
`chunks` container, `sound_bank` framing and `level_overlay`. 238 tests
|
|
green against the real disc. **What is open is naming, not locating** --
|
|
see the Phase 2 list for the four types that still have no decoded
|
|
fields: `tie` vertices, collision cells, `SBlk` records, `volumes` and
|
|
`moby_anim`.
|
|
- [x] **Lighting verified against the running game.** PCSX2 2.8.2 with savestates
|
|
for level 0 and level 1. The tfrag lighting pass is byte-exact on both
|
|
against the game's own RAM. Setup and method in
|
|
`plan/pcsx2-verification.md`.
|
|
- [x] **First drawable frame.** `gc-engine` loads a level, applies LOD selection
|
|
and morphing for the camera, groups triangles by texture and draws them
|
|
under a Bevy fly-camera. Level 0 renders as 26,758 textured triangles
|
|
across 92 textures with correct UVs. Set `GC_ENGINE_SHOT=<path.ppm>` to
|
|
capture a frame and exit.
|
|
- [ ] **The origin bug this exposed.** Positions are i16 offsets from a
|
|
per-patch origin, so taking the wrong STROW collapsed the whole level to
|
|
a +-32 unit blob at the world origin while every parser test still
|
|
passed: the vertex count matched, the triangle count matched, only the
|
|
*place* was wrong. Fixed by taking the STROW immediately preceding the
|
|
first position unpack rather than the last one.
|
|
- [x] **Per-vertex colour.** The tfrag's RGBA array now modulates the texture,
|
|
drawn unlit so the frame shows real albedo rather than a faked lighting
|
|
model. Verified against the stored values (level 0 patch 0 runs
|
|
`(25,26,26)` to `(108,144,141)`), which are genuinely dark.
|
|
- [x] **Tfrag normals.** The 256-entry sine table is decoded from the level
|
|
overlay and normals are built per vertex and handed to the renderer. The
|
|
overlay has **no ELF header**, so the table cannot be reached through a
|
|
program header, and its offset differs per level (level 0 `0xb650c`,
|
|
level 5 `0xcca30`); it is located by matching the whole 256-entry curve.
|
|
Normals are negated as stored and get the same axis swap as positions.
|
|
The pipeline now also writes `overlay.bin`.
|
|
- [x] **The PS2 float model.** `gc-engine::ps2` is the EE's FPU on raw bits: no
|
|
denormals, no NaN or infinity, truncation toward zero, the adder's
|
|
operand alignment, and `rsqrt` as sqrt-then-divide. `MAX` reads back in
|
|
Rust as NaN, which is the sharpest available statement of why this is
|
|
not `f32`.
|
|
- [x] **The `LightTfrags` pass** is written and unit-tested against the PS2
|
|
model: PEXT5 base colour, `select` set blending, re-normalised blended
|
|
directions, and `base + A*max(dA, dA*wA) + B*max(dB, dB*wB)`.
|
|
- [x] **The level light bank parses and level 0 is fully lit.** `gameplay` is a
|
|
WAD stream; the extractor had been writing it compressed, which is why
|
|
the bank framing found garbage. All 80,274 vertices of level 0 now go
|
|
through `LightTfrags`. The renderer still counts and reports how many
|
|
vertices it lit, because a silent fallback to stored colours would look
|
|
plausible and be wrong.
|
|
- [x] **The shrub region is bounded, and with it the whole lump.** Shrub's end
|
|
was the last unlocated region bound: it is the start of the post-class
|
|
chunk container, found by a marker-paired signature in `core_data` (the
|
|
index carries no successor offset). The container closes shrub on the
|
|
24 shrub levels and tie on the three without. All 358 shrub class bodies
|
|
are now bounded and are whole 8-byte records, so shrub measurements are
|
|
trustworthy for the first time -- the "3.5M records" figure from the
|
|
unbounded final span is replaced by a measured 9,600.
|
|
- [x] **Lighting *magnitudes* are verified** by the PCSX2 comparison loop.
|
|
`gc-trace compare-tfrag-light` runs the pass over level 0's disc data and
|
|
diffs it against the game's own lit colours in EE RAM (live over PINE, or
|
|
from a savestate): **all 162,400 colour bytes match**, across 833/833
|
|
headers relocated and the light bank byte-identical. Two bugs it found:
|
|
the normal's z was not negated (dark but plausible, 95.88% agreement),
|
|
and records with no light set drew transparent black instead of the base
|
|
colour. See `docs/plan/pcsx2-verification.md`.
|
|
- [ ] Then the GS register set and the guard-band clip test.
|
|
|
|
The disc being the exact build the reference projects pin is the single
|
|
biggest favourable fact so far: no disc conversion, no regional timing work,
|
|
and no re-derivation of the reference baseline.
|
|
|
|
---
|
|
|
|
## Phase 0 — Foundation ✅
|
|
|
|
**Done.**
|
|
|
|
- Disc identified: `SCUS_972.68` v1.01, USA Black Label, NTSC 60 Hz. Boot ELF
|
|
SHA-1 `bacda860cbd6723142e86db56f197e47e728970f`, entry `0x00131AE8` — the
|
|
entry point independently matches the value recorded by `rac2-gc-decomp`,
|
|
which cross-confirms the identification.
|
|
- `tools/identify-disc.py` reproduces the identification against any disc image
|
|
and distinguishes a truncated dump from a non-PS2 image.
|
|
- Five-crate workspace scaffolding, Bevy 0.19.1 pinned, builds clean.
|
|
- `.githooks/pre-commit` blocks game data. Verified against seven cases:
|
|
`.iso`, renamed disc image, WAD container, ADX container, `extracted/`,
|
|
a 60 MiB file, and legitimate source.
|
|
- `AGENTS.md` and `PROVENANCE.md` record conventions and licences.
|
|
|
|
### Open from this phase
|
|
|
|
- `alsa-lib-devel` is not installed, so Bevy's host audio backend is disabled in
|
|
`Cargo.toml`. Audio is the last phase; revisit then.
|
|
- Bevy feature set is deliberately minimal. Re-add features as needed rather
|
|
than switching `default-features` back on wholesale.
|
|
|
|
---
|
|
|
|
## Phase 1 — Disc pipeline ✅
|
|
|
|
**Done.** Gate met: every extracted file matches the committed hash table.
|
|
|
|
- [x] `iso9660` reader (R&C2 lists assets as real files, unlike R&C1)
|
|
- [x] `G/*.WAD` container header parsing — 92 containers, three table shapes
|
|
- [x] WAD LZ77 decompression (`wad.rs`, four packet classes)
|
|
- [x] Unpack every container, bounds-checking each extent
|
|
- [x] SHA-1 table for all 5,135 files (hashes only, never content)
|
|
- [x] `verify` stage, plus integration tests against a real disc
|
|
- [x] Format specs written: `disc-layout.md`, `global-wad.md`, `wad.md`
|
|
- [x] Work record with the negative results: `plan/disc-pipeline.md`
|
|
|
|
```text
|
|
gc-extract identify disc.iso -> SCUS_972.68 v1.01 NTSC, ELF SHA-1 matches
|
|
gc-extract unpack disc.iso out/ -> 5135 files, 4,500,106,960 bytes, 10.8s
|
|
gc-extract verify out/ table.tsv -> OK: 5135 files match
|
|
```
|
|
|
|
**Not done, deliberately:** the table of contents at sector 1001 is documented
|
|
but not read. Enumerating the ISO filesystem is simpler and sufficient; the ToC
|
|
matters when per-level overlay loading is understood in Phase 3.
|
|
|
|
**Two wrong turns are recorded** in `plan/disc-pipeline.md`, because both would
|
|
be easy to repeat: the container pair table read backwards, and the LZ77 stream
|
|
mistaken for a bit stream. Both produced plausible-looking wrong output before
|
|
failing, which is the failure mode worth guarding against.
|
|
|
|
---
|
|
|
|
## Phase 2 — Format parsers
|
|
|
|
**Goal:** every asset type parses into plain Rust values, tested against real
|
|
disc data.
|
|
|
|
Phase 1 hands over the extents; this phase works out what each one contains.
|
|
|
|
**Done so far:**
|
|
|
|
- [x] `level_wad.rs` — the 0x60-byte header on all 27 level containers, naming
|
|
four extents plus three chunk descriptors. Verified: extents tile each
|
|
container in file order (27 of 27).
|
|
- [x] Two facts the format documentation does not state, both measured here:
|
|
sizes in this header are **sector counts** where the generic `G/` table
|
|
uses byte counts, and **`LEVEL21.WAD` reports level id 30**, so container
|
|
order is not level identity.
|
|
|
|
**In progress:**
|
|
|
|
- [x] `data` — the extent header locates `core_index` and `core_data`; the
|
|
geometry decompresses and is verified on level 0.
|
|
- [x] `core_index` class tables — moby, tie and shrub. They were ruled out
|
|
for want of an entry size, which is knowable exactly: each table ends
|
|
precisely where the next begins, and that chain closes on 27 of 27
|
|
levels. 32 bytes per moby/tie entry, 48 per shrub. Across the disc:
|
|
**5,796 moby, 1,864 tie, 358 shrub classes**. Identifying `tfrag` by its
|
|
own structure is what led here.
|
|
- [x] `occlusion` and `sky` offsets located and their first bytes read.
|
|
- [x] Region layout of `core_data` established: tfrag, occlusion, sky, collision
|
|
and `textures_base` are offsets, strictly increasing on all 27 levels.
|
|
Terrain is `[0, occlusion)` and carries inline texels.
|
|
- [x] Region layout **closed**: the chunk container's marker-paired signature
|
|
ends the last class region, and the constant trailer ends the container.
|
|
Every byte of level-specific `core_data` now belongs to a named region.
|
|
- [x] **tfrag vertex format: recovered.** The tfrags block header gives a table
|
|
offset (`0x40`) and a patch count; each patch is a 0x40-byte header whose
|
|
`data` offset is relative to the table, not to the block. Positions are
|
|
V3_16 payloads inside a VIF1 command list, split across three lists per
|
|
patch (common, LOD 0+1, LOD 0). The whole vector format is encoded in the
|
|
command byte and every element size follows from it. Verified on all 27
|
|
levels: **18,102 patches, 821,439 vertices**, each patch yielding exactly
|
|
its own `vert_count`. See `plan/level-assets.md` and `gc-formats::tfrag`.
|
|
- [x] **tfrag topology: recovered.** Vertex-info (UVs plus LOD parenting), the
|
|
strip command lists and the index arrays, classified by unpack format and
|
|
destination VU address. Strip indices address the vertex-info array
|
|
rather than positions. Every patch's strips expand to exactly its own
|
|
`tri_count`: **18,102 patches, 809,879 triangles**, all indices in range.
|
|
See `plan/level-assets.md` and `gc-formats::tfrag`.
|
|
- [x] **tfrag texture bindings and per-vertex data.** The ad-gifs decode to a
|
|
texture index, wrap modes, minification filter and LOD bias; the level
|
|
index gained a texture table (`level_core.rs`) for those indices to land
|
|
in. Per-vertex RGBA and light/normal records recovered from their fixed
|
|
offsets. All 27 levels: 47,950 ad-gifs, every index in range, 90% of the
|
|
level texture tables referenced.
|
|
- [x] **Level textures decode.** 8-bit indexed texels plus a 256-entry CSM1
|
|
CLUT from `gs_ram`, honouring the PS2 CLUT block swap and the 0x80 alpha
|
|
scale. The pipeline now writes `gs_ram.bin`, which it had been skipping.
|
|
All 1,595 tfrag textures on all 27 levels decode; 96.8% fully opaque and
|
|
96.3% with real detail. Verified visually via the new
|
|
`gc-extract dump-texture`, because a wrong lookup still yields a
|
|
well-formed image of the wrong thing.
|
|
- [x] **tfrag LOD selection.** Switch distances scale as `[6L, 4L, 2L]` from the
|
|
block header's float; six draw modes result from testing the bounding
|
|
sphere against `trunc(D * 1024)`, so a patch straddling a threshold takes
|
|
the coarser mode and distance never moves back to finer detail. Morph
|
|
weights run 0..1 across each tier's band, and collapse replaces an entry
|
|
with its first parent rather than blending it.
|
|
- [x] **tfrag parent indices.** The four V4_8 arrays are classified by VU
|
|
address, but the addresses collide, and each V4_8 element holds four
|
|
indices so an array is `count * 4` bytes rather than `count`. With
|
|
`unk_06`/`unk_0a` suppressing absent tiers, all 370,349 LOD entries
|
|
across the 27 levels resolve: 29,043 LOD-01 and 341,306 LOD-0, zero
|
|
failures. See `plan/level-assets.md` for the two corrections.
|
|
- [x] `gameplay` — **it is a WAD stream**, exactly like `core_data`; the
|
|
extractor was writing it compressed. Decompressed, level 0 is 1,872,256
|
|
bytes and holds 5 directional light sets. The earlier "does not decode
|
|
with the WAD LZ77 decoder, bit order and header length ruled out" note
|
|
was wrong about the cause: the decoder was right and the input was not
|
|
what it looked like. Corrected in `plan/level-assets.md`.
|
|
- [~] `sound_bank` — the `SBlk` head is now characterised: `"SBlk", 1, 4,
|
|
"DAW\0", 0` constant on 27/27, and `+0x14` is the only varying word, with
|
|
its low 16 bits always zero (`0x0092..=0x0103`). It tracks body length at
|
|
~100 bytes/unit and is **not** a count -- ratio 91.2..110.8, and no stride
|
|
makes it exact. The absence of a stride is asserted too. Container framing parsed (`SoundBankHeader::parse`,
|
|
`parse_sound_bank`); `SBlk` and `DAW` bounds verified on 27/27. The
|
|
**record fields inside `SBlk` are not decoded**, and `DAW` shows no
|
|
detectable stride, consistent with ADPCM. See the note under
|
|
`level_overlay` below for what the game's own strings say about loading.
|
|
- [x] `data` — **not geometry.** It is the level's file descriptor: its
|
|
first table carries the exact byte lengths of `overlay.bin`,
|
|
`core_index.bin` and `gs_ram.bin` at fixed `u32` indices 1, 3 and 5 on
|
|
every level checked. What follows the descriptor is still unknown.
|
|
|
|
- [x] `tfrag` — **complete on the disc side.** Vertex format, topology, strips
|
|
(`TfragStrip`), texture bindings, per-vertex colour and normals, LOD
|
|
selection, parent indices, morphing and collapse. 36 format tests, 9 texture
|
|
tests, 23 engine tests. The morphing is a blend toward the **parent
|
|
vertices'** midpoint, not toward separate morph targets — there are no
|
|
morph targets in this format, which is what the old "morph targets
|
|
unrecovered" note wrongly implied.
|
|
- [x] The tfrag **table record has exactly two relocated pointer fields**, at
|
|
`+0x10` and `+0x38`, verified against PCSX2 across all **833** records: the
|
|
loader adds the **table's** RAM address (`0x848040` = block base plus
|
|
`table_offset`, matching `TfragHeader::data`'s own "relative to the table")
|
|
to `+0x10`; every other byte of every record is verbatim, and 1,665 fields
|
|
differ where 1,666 were possible.
|
|
They are the existing `TfragHeader::data` and `::cube_ofs` -- what RAM adds
|
|
is that both are **pointers**, not plain offsets. All **833** patches parse,
|
|
carrying **43,613** vertex-info entries; **499 of 833** have at least one
|
|
morphing entry.
|
|
- [x] Behind those pointers, **every patch's bytes are rewritten** -- and it is
|
|
**not morphing**. Across all 833 patches: the 334 with **no** morphing
|
|
entries differ by a median **7.80%**, the 499 with morphing entries by
|
|
**8.12%**, and **0 of 833** are byte-identical to disc. Morphing does not
|
|
predict which bytes change. Nor is it a relocation: the differences fall in
|
|
short structured runs with no common `u32` addend. The loader regenerates
|
|
each patch's **VU packet stream for the LOD tier it selected**, which
|
|
happens to every patch regardless of morphing.
|
|
- [ ] What tfrag is **not** yet verified is that the morph matches
|
|
`resolve_vertex`'s formula. Reading a weight back out of RAM does **not**
|
|
work -- the positions live inside regenerated VU packets, so there is no
|
|
stable per-vertex correspondence. The LOD switch distances, morph weight and
|
|
collapse edge remain derived from the data (tiers classified from the four
|
|
V4_8 parent arrays, weight running zero to one across each band); verifying
|
|
them needs the **camera distance at the savestate** so the weight can be
|
|
computed and compared against the rendered result.
|
|
- [x] `tie` — **fully chained.** `resolve_tie_class` walks both indirections: the
|
|
class table is in `core_index.bin`, and the entry's `offset_in_asset_wad`
|
|
addresses `core_data.bin` (not `core_data.wad`). All **1,864** tie classes
|
|
resolve with `o_class` agreeing on every one. A `packets` word of **zero
|
|
means no block at that LOD** -- 2,177 real blocks, 1,551 absent slots.
|
|
- [x] Tie **blocks are self-delimiting**: `parse_tie_block` walks to the `0xf6`
|
|
terminator that every real block carries (2,177 of 2,177), followed by zero
|
|
padding. This is what makes blocks parseable at all -- `packets[1]` bounds
|
|
one on only 461 of 1,864 classes.
|
|
- [x] The tie **16-byte record is R&C1's `TiePacketHeader`.** Identified not by
|
|
resemblance but by R&C1's own rule, `bfc_distance == 5 * shader_count`,
|
|
holding on **15,219 of 15,219** records across all 2,177 blocks. A block is
|
|
`[16-byte header][record table][vertex payload]`, and the header word at
|
|
`+0x00` is the byte offset from the block start to the payload. The earlier
|
|
"variable-length, not 16-byte records" reading was wrong: the word is not
|
|
an element count, which is why dividing the payload by it gave 0 of 922.
|
|
- [x] The tie **vertex element is 16 bytes**, measured over 95,612 vertices on
|
|
levels 0-2 and settled by comparison: a 12-bit-fixed texcoord hit rate of
|
|
**82.5%** at 16 bytes against 47.8% at 8 and 44.8% at 12.
|
|
- [x] `i16[0..2]` are the **vertex positions**: LOD1/LOD2 of one class share 96.1%
|
|
of their position triples, while LOD1 of A against LOD2 of B shares 0.4% --
|
|
a factor of 240, asserted including the ratio. Axis order and scale unnamed.
|
|
- [x] `i16[3]` is a **byte-sized index**, not a coordinate: 201 distinct values in
|
|
`0..=254` inside the ordinary layout, whose high byte is the layout tag
|
|
(93.4% of elements are `0x00`, 5.6% are `0x03`, 146 selector values in all).
|
|
Asserted.
|
|
- [x] The tie **texcoord field is located**: `i16[4]`/`i16[5]` of the 16-byte
|
|
element, scoring 73.6% for 12-bit fixedness over 36,501 level-0 vertices,
|
|
against 1.1% for `(0,1)` and 4.8% for `(2,3)`. The `(6,7)` pair scores
|
|
*higher* at 76.6% and is **not** a texcoord -- it is the near-constant
|
|
`4096, 0` pair, which is trivially two multiples of 4,096.
|
|
- [x] `i16[3]` **selects the texcoord encoding**, and it is specifically the
|
|
**high byte**. Grouped across 95,612 vertices on levels 0-2: high byte `0x00`
|
|
covers 89,341 vertices (93.4%) at 92% 12-bit fixed, while **every** other
|
|
high byte scores 0%. The split is on the byte, not the value -- `i16[3]` of 1
|
|
and of 771 differ only in the high byte and behave completely differently.
|
|
Within the all-zero case, `i16[3] == 0` exactly (5,851) is a third
|
|
distribution at 62%. Three cases: `0x00/0x00`, `0x00/1..255`, and non-zero
|
|
high byte.
|
|
- [~] The remaining tie vertex **fields are not named**, and two readings are now
|
|
excluded. `i16[6]`/`i16[7]` are constant across a whole draw packet in
|
|
43%/34% of records, so they are usually per-packet rather than per-vertex.
|
|
`i16[3]` is per-vertex (50 distinct values per packet) but **not a skin or
|
|
joint index**: vertices sharing it are 0.95x *farther* apart than vertices
|
|
differing in it, against a control that halves. It is also unshared across
|
|
LODs. Recorded as not understood rather than guessed.
|
|
`i16` range with 10,803-13,421 distinct values each. `i16[0..2]` are most
|
|
likely position, but that is not established.
|
|
- [~] `shrub` — class table done (48-byte entries, not 32; reading shrub at
|
|
32 still gives 231 sane-looking entries of 358, which is the hazard).
|
|
Geometry header parsed and checked field by field on all 358 spans:
|
|
`+0x24` equals the entry's own `o_class`, `+0x2c` is an in-span data
|
|
offset, `+0x16`/`+0x18` are zero, and `+0x1c` names a billboard on
|
|
exactly the 25 classes whose core-index entry carries billboard texture
|
|
fields. Shrub does **not** use moby's part-table format. The body is a
|
|
**24-entry palette of unit-length s16 normals** (squared length within
|
|
65,535 of 32767² on all 8,592 entries — a random triple would miss by a
|
|
billion) plus zero padding records; the billboard record carries K, MMIN
|
|
and the TEX0/MIPTBP addresses as R&C1's does. What remains open: the
|
|
three floats in the billboard record, `size`, `mode_bits`, `count`, and
|
|
**where the mesh geometry lives** — R&C1 puts the mesh packets in this
|
|
blob and R&C2's blob holds only the palette.
|
|
- [x] **The tie LOD block is a table of 16-byte records.** The header's word at
|
|
`+0x00` is the byte offset from the block start to the vertex payload --
|
|
the record table's length -- not a count, which is why the payload never
|
|
divides by it. Each record is a leading `u32` pointer **relative to the
|
|
table start** plus twelve verbatim field bytes, relocated at load to
|
|
`table + word`.
|
|
- [x] **The record is R&C1's `TiePacketHeader`.** R&C1's own rule
|
|
`bfc_distance == 5 * shader_count` holds on **all 15,219 records of all
|
|
2,177 blocks on all 27 levels**. R&C1's rule for the colour offset holds
|
|
on **none** of them, so the games use that field differently.
|
|
- [x] **The tie relocation verifies on two levels.** `gc-trace
|
|
verify-tie-records` checks it against PCSX2's RAM: **87 of 87** blocks on
|
|
level 0 and **104 of 104** on level 1, zero mismatches, which is what
|
|
makes the record layout a finding rather than a reading of one block.
|
|
- [x] **The shrub mesh geometry is in the class blob.** A packet table at
|
|
`+0x40` names `count` packets between the table and the palette, and
|
|
they tile on **358 of 358 spans** (2,239 packets), which makes `count`
|
|
the packet count. Confirmed against the running game, which relocates
|
|
the table's offsets to absolute addresses and `data_offset` likewise.
|
|
- [x] `moby` — per-class model geometry. A class span opens with a `u32`
|
|
pointing at a part table of 16-byte records; parts tile exactly, with
|
|
`vertex_offset + 16 * vertex_count == next_part` holding for **43,467 of
|
|
43,467** parts across every class on all 27 levels. That identity is what
|
|
makes the record size fall out of the data rather than being assumed.
|
|
1,270,802 vertices over 4,560 class spans; 116 spans carry no geometry.
|
|
Vertices are 16-byte VIF-packed half-float quads, handed back raw.
|
|
- [ ] `moby_anim` — **still open, and the obvious reading is ruled out.** An
|
|
instance table would have to scale with the level's moby class count, but
|
|
**per-chunk-type record counts are level-independent** -- the same types
|
|
report 90, 77, 50 and 47 on every level tried, and the largest type
|
|
totals 2,430 = 90 x 27. Nothing in the chunk set is keyed by moby class.
|
|
The types also pair up (two types both reporting 90 on every level,
|
|
another pair both 77), which looks like two views of one fixed table
|
|
rather than two independent tables.
|
|
- [x] `collision` — container framing and header fields. A 0x40-byte header
|
|
self-describes: it carries its own size and the payload size, and
|
|
`header + payload` stays inside `core_data` on all 27 levels. 41,481,520
|
|
payload bytes across 19 levels; 8 declare zero length. Header bytes `+8`
|
|
reads zero on 22 levels and a large value on 7/9/12/13/14; the payload's
|
|
two leading `u16` are non-zero on all 19 non-empty levels (1-74 and
|
|
13-112) and are reported as `CollisionHeader::first`/`::second`.
|
|
- [ ] The collision **cell layout is not decoded.** The payload does not walk
|
|
as fixed-stride cells. On level 4 a table starts at payload `+5932` and
|
|
runs **17 entries** -- deltas rise then fall -- against a 75-cell grid,
|
|
with no arithmetic relating the two. A 64-byte RAM stride looked promising
|
|
and is **disproved** -- the longest such run is 27 records of identity
|
|
matrices, one per level.
|
|
- [x] **The collision payload is essentially verbatim in RAM, not reshaped.**
|
|
Level 0's sits at RAM `0xac6800` for a disc address of `0x27e800`:
|
|
**903,212 of 903,248 bytes identical**, first difference at `+0xdc82c`, with
|
|
**6** bytes differing in the final 36. Asserted by a savestate-gated test
|
|
that compares the whole payload. This **overturns** the earlier claim that the grid dimensions
|
|
are absent from RAM: level 0's `11 x 33` are the first two bytes of that
|
|
block. The earlier negative came from searching for the dimensions as a
|
|
`u32` pair when they are two `u16`s -- a bad pattern, not a property of
|
|
the data. The payload contains ascending `u32` offset tables from `+1024`
|
|
onward, and disc-side decoding is no longer blocked on reshaping. See
|
|
`plan/pcsx2-verification.md`.
|
|
- [x] `occlusion` — the PVS grid. A z/y/x tree of u16 node offsets over 4x4x4-unit
|
|
cells, populated cells naming 128-byte masks. **412,521 cells over
|
|
140,939 masks** across the 24 levels that have one; the mask array ends
|
|
within 48 bytes of the block boundary on every level. 3 carry no occlusion
|
|
data, and an offset of 0 means *absent* rather than "at the start" — the
|
|
latter parses as a grid whose mask array happens to fit, which is how it
|
|
nearly went wrong.
|
|
- [x] `sky` — 149 sub-objects across all 27 levels, each with a count, a
|
|
four-float bounding sphere and texture dimensions. The offset table at
|
|
`+0x20` is self-limiting: entries run while strictly inside the region.
|
|
**The word at `+0x0c` is not a count** -- it disagrees with the table on
|
|
most levels and reads 196,617 on level 2 against 4 sub-objects.
|
|
- [~] `volumes` — **still not identified, but a candidate exists.** Every level's
|
|
`gameplay.bin` carries a table of **144-byte float records** with world
|
|
positions at `+0x0c`, preceded by an all-zero record (levels 0-9: 23 to 612
|
|
records). That is the shape of a volume trigger region and it is not tied
|
|
to anything else in the file -- but camera bounds, spawn points and path
|
|
nodes would look identical, and there is no engine-side `Volume` string to
|
|
discriminate with. **Recorded as candidate, not volumes.** Two detector
|
|
bugs on the way are written up: a validity test that counted zero as
|
|
world-plausible scored zero-filled memory perfectly, and the 144-byte stride
|
|
rested on four eyeballed offsets that the gap histogram did not support.
|
|
A promising lead also failed: level 0's `data.bin` references one candidate
|
|
offset exactly once (chance ~0.003), but **none** of seven other levels'
|
|
candidates are referenced by their own `data.bin`, so the table is reached
|
|
by walking structure and nothing links it to volumes. Not located. Nothing examined so far presents itself as a
|
|
volume: `core_data`, the chunk region and `gameplay.bin` all yield
|
|
structures that are accounted for by something else.
|
|
- [~] `level_overlay` — header, constant block and level data parsed
|
|
(`gc_formats::overlay`, 27/27). `+0x10..+0x1220` -- 4,624 bytes -- is
|
|
byte-identical on every level; the word at `+0x0c` is **not** the file
|
|
length, falling below it on 17 levels and above on 10. The debug string
|
|
pool it opens with carries `Camera_CollPrimTest WARNING! - grid out of
|
|
bounds! (RAR)`, independent support for reading the collision payload's
|
|
two leading `u16`s as grid dimensions. `+0x00` **is** the load base of
|
|
the section this file is, `0x1A7C80`, constant on all 27 levels because
|
|
every level's `.lit` loads there. `+0x0c` is **not** the `.lit` size -- it
|
|
falls *below* the file length on 16 of 27 levels, by up to 8.9%, so it is
|
|
excluded by sign -- and it is still unnamed. **The remaining header words
|
|
are unnamed and the level-specific tail is not structured yet**; the
|
|
Phase 3 seven-section parser and relocation are untouched, though the
|
|
seven sections are now named (`lvl.vtbl`, `lvl.camvtbl`, `lvl.sndvtbl`).
|
|
- [x] **The post-shrub chunk container.** A 16-byte header, a 10-byte zero gap,
|
|
then offsets tagged `0xF0000000 | low24` resolving to `chunk_start + low`;
|
|
header byte 14 is the record count, equal to `offsets.len()` on 1,123 of
|
|
1,123 chunks across all 27 levels. **77 distinct chunk types, 1,509
|
|
chunks, 34,491 records.** Moby class IDs enrich at 9,493 hits against
|
|
246.1 expected with contiguous per-type arrays -- but the arrays land in a
|
|
different place for each type, so no single pooled scan finds them.
|
|
- [x] **Chunk record framing is closed.** Every chunk has a **uniform record
|
|
length** (1,509 of 1,509) -- length varies *between* chunks of a type, never
|
|
within one, which withdrew an earlier "variable-length type" claim that had
|
|
been pooled across chunks. The record's leading byte is a **per-chunk phase
|
|
counter** (`bucket_rule`, 1,262 of 1,509 chunks) and on **1,235** of them
|
|
*both* leading bytes encode the record's own index in mixed radix
|
|
(`index_rule`), so the leading bytes carry no per-record information.
|
|
- [x] **The loader's transformation, from PCSX2 RAM.** Disc offsets
|
|
`0xf0000000 | low24` are rewritten in RAM to absolute EE pointers
|
|
`0xf1850000 | low24` -- tag replaced by the chunk's load base, low 24 bits
|
|
untouched. The payload itself is copied **verbatim**. The index finding
|
|
is corroborated live: the byte at `+2` takes exactly `0, 16, ... 240`, the
|
|
16 values `index_rule` predicts.
|
|
- [x] **Corrected: a record starts two bytes before the table's offset.** The
|
|
`02 00` read in RAM as a loader-inserted prefix is the record's own
|
|
leading word, already on the disc at every record. The loaded table
|
|
simply points two bytes earlier than the disc's, at that word. It is a
|
|
field: uniform across a chunk's records on **1,439 of 1,509** chunks,
|
|
fixed within **all 71 types**, values `{1, 2, 4, 6, 8, 10}`. Disproved
|
|
readings: an entry count (57 distinct fractional "sizes") and a
|
|
distinct-leading-byte count (0 of 1,506 agree).
|
|
- [~] The chunk **record fields are not decoded**, and no type is named -- but the
|
|
stride table is now measured: **74 signatures, 55 distinct strides, 52..640,
|
|
every stride a multiple of 4**, and 73 of 74 signatures use a single stride.
|
|
The exception is pinned: `[75 42 ef f2 71 b5]` at strides `{76, 92, 104}`.
|
|
Asserted including the exception, so "every type is uniform" cannot rot.
|
|
A per-type layout found for one type (48 bytes of data then 60 permanently
|
|
zero) does **not** generalise -- 0 of 9 types fit that shape.
|
|
**65 of the 74 signatures are byte-identical in all 27 levels** (9 are
|
|
level-specific, 0 ragged), so most of the container is static game tables
|
|
rather than level content -- asserted. Consequence for method: the 65
|
|
cannot be decoded by comparing levels, because there is nothing to compare.
|
|
`[75 42 ef]` is both level-specific *and* the three-stride type, so it is
|
|
where per-type decoding should start. The body
|
|
is dense (74% of `f32` reads plausible against a 49% random control) with
|
|
an index-like tail -- 12 of 163 `u16` slots in the largest type stay below
|
|
`0x1000` -- but **no count field** exists in the first 64 body bytes at any
|
|
width or stride tried. Only **9.4%** of `u16` slots are per-chunk constant,
|
|
so records are not templates.
|
|
- [x] **Moby class arrays are contiguous per type, and the pooled scan was
|
|
wrong.** See rule 3a in `AGENTS.md`: the original claim of a `u16` array at
|
|
slots 129-140 was the largest-stride type dominating an aggregate. Per
|
|
type, every one of the 26 places its array somewhere else.
|
|
|
|
**Gate:** every parser has a golden test against bytes from the real disc, and
|
|
reports mismatched header counts as failures rather than warnings.
|
|
|
|
**Discipline:** bound every read. Typed errors, no panics, no reads past the
|
|
buffer.
|
|
|
|
---
|
|
|
|
## Phase 3 — Overlay system
|
|
|
|
**Goal:** make 27 level overlays tractable.
|
|
|
|
Every overlay carries the same seven sections (`.lit`, `.bss`, `.data`,
|
|
`lvl.vtbl`, `lvl.camvtbl`, `lvl.sndvtbl`, `.text`) and the same engine code,
|
|
linked at different addresses. ReRAC solves this by masking relocations — `j`/
|
|
`jal` targets, `lui` immediates of main-RAM addresses, the `%lo` that completes
|
|
them, and `$gp`-relative operands — then matching the masked words to find a
|
|
reference function's copy on another overlay.
|
|
|
|
- [ ] Port the masking and code-identity logic
|
|
- [ ] `Relocation::func` and `Relocation::data`
|
|
- [x] `$gp` for this game (`0x001AEFF0`, against R&C1's `0x166c00`) -- confirmed
|
|
by `OpenRAC/rac2-gc-decomp` `docs/LEVELS.md` for all 27 levels
|
|
- [ ] Extend the level table from 19 to 27
|
|
|
|
**Gate:** every level-01 function's relocated copy is found on all 27 overlays;
|
|
cross-checked against an independent cluster hash.
|
|
|
|
**Why this is high-value:** it turns "27 overlays of unknown code" into a table
|
|
lookup, and it is the main thing inherited from ReRAC.
|
|
|
|
---
|
|
|
|
## Phase 4 — Engine core
|
|
|
|
**Goal:** draw a frame.
|
|
|
|
- [ ] Graphics Synthesizer in wgpu, with the GS's own rules: its
|
|
`round(log2(z/n) + K)` mip selection, blend and alpha-test behaviour,
|
|
alpha `0x80` meaning 1.0, CLAMP wrap
|
|
- [ ] VU0/VU1 skinning and lighting on the PS2 float model
|
|
- [ ] LOD/morph/cull per `TfragProc` and `TieProc`
|
|
- [ ] Occlusion culling, fog, shadows
|
|
- [ ] GIF/VIF packet stream interpretation
|
|
|
|
**Gate:** CPU and GPU paths agree on every vertex of a reference level — the
|
|
same identity check ReRAC runs.
|
|
|
|
**Risk:** VU1 microcode is **binary** in the game's `.vutext` section, so it is
|
|
reverse engineered rather than parsed. This is the hardest single item in the
|
|
project and the main threat to the timeline. Note that ReRAC's own moby VU1
|
|
analysis trips their disassembler on two opcodes.
|
|
|
|
---
|
|
|
|
## Phase 5 — Veldin vertical slice
|
|
|
|
**Goal:** one planet, playable start to finish.
|
|
|
|
- [ ] Ratchet's controller on a **spherical** planet
|
|
- [ ] Chargeboots: hold/release timing, chained jumps, dash
|
|
- [ ] Wrench and the camera
|
|
- [ ] Collision and the HUD
|
|
|
|
**Gate:** Veldin is completable and visually matches PCSX2 frame-for-frame under
|
|
replayed input.
|
|
|
|
**This is the go/no-go for the project.** If Veldin works, the other 26 levels
|
|
are content. If it does not, the reason is discovered while it is still cheap.
|
|
Spherical gravity is the thing most likely to break, because it invalidates
|
|
R&C1's movement and camera assumptions wholesale.
|
|
|
|
---
|
|
|
|
## Phase 6 — Remaining overlays and the shell
|
|
|
|
- [ ] Per-level moby class dispatch via `lvl.vtbl`
|
|
- [ ] Remaining 26 levels
|
|
- [ ] Planets, travel, the Gadgetron vendor, RPG upgrades
|
|
- [ ] Weapon modification bitmasks (lock-on, shock, acid, …)
|
|
- [ ] Minigames and challenges
|
|
- [ ] The 25-state memory-card FSM and its UI
|
|
- [ ] IFF chunk saves, including import of a R&C1 save to unlock legacy weapons
|
|
|
|
---
|
|
|
|
## Phase 7 — Audio and video
|
|
|
|
- [ ] 989snd v2 audio decode and playback (not CRI ADX — a different codec)
|
|
- [ ] SPU2 mixing, reverb zones, level sounds
|
|
- [ ] IPU/MPEG-2 video for the title screen and cutscenes
|
|
|
|
---
|
|
|
|
## Phase 8 — Fidelity pass
|
|
|
|
Differential testing against PCSX2 across every level:
|
|
|
|
- [ ] Recorded-input frame comparison, per planet
|
|
- [ ] Geometry, fog, blend, alpha, texture sampling
|
|
- [ ] Audio sync and 60 Hz timing
|
|
- [ ] Behavioural comparison: traversal routes, boss timing, RNG
|
|
|
|
---
|
|
|
|
## Principles
|
|
|
|
- **A gate is pass/fail, not a percentage.** "Looks fine" is not progress.
|
|
- **Negative results are written down.** A ruled-out approach saves the next
|
|
attempt as much as a working one.
|
|
- **The float model is not negotiable.** Approximate is not the same as
|
|
faithful, and the difference is visible in how the game feels.
|
|
- **Fidelity over tidiness.** Where the original does something that looks like
|
|
a bug, reproduce it and say why in a comment. See `AGENTS.md`. |