ReaperOfVeriod
65b5d366de
Two pieces, and the first is the reason the second can be trusted. ps2.rs is the EE's FPU on raw u32 bit patterns. It is not f32: there are no denormals (exponent 0 *is* zero), no NaN and no infinity (exponent 255 is an ordinary value, so MAX reads back in Rust as NaN -- a test asserts exactly that, because it is the sharpest available statement of why this module exists), every result truncates toward zero, the adder discards the small operand's bits more than one below the large one's LSB and returns the large operand unchanged across a 25-bit exponent gap, and rsqrt is sqrt-then-divide with truncation between the steps rather than one rounded operation. The lighting pass accumulates all of these across every vertex of every patch, so this is a visible difference rather than a rounding curiosity. tfrag_light.rs is the pass itself: PEXT5 base colour, the select word picking one light set or blending two by bits 8..15 / 256, blended directions re-normalised where unblended ones are not, and base + A.rgb*max(dA, dA*wA) + B.rgb*max(dB, dB*wB). Lighting indexes positions, not vertex-info entries, since a common entry and a LOD entry can share a position. TfragLight gains a `select` field for the u16 at +6. It was briefly reconstructed with an unsafe pointer cast over the struct, which is exactly the sort of thing that breaks silently the moment a field is reordered; it is now a plainly parsed field. The pass does not run yet. The light bank framing the reference uses -- a u32 offset at +0x04, an i32 count there -- gives offset 0xa0d and a count of 698810368 on RAC2 level 0, so RAC2's gameplay extent is not laid out that way. The renderer now counts the vertices it actually lit and prints it, because falling back silently to the stored colours produces a plausible but wrong picture: lighting: 0 of 80274 vertices shaded by LightTfrags; the rest fall back to the tfrag's stored colours (no usable light bank) That line is the current honest state, and the frame has not changed as a result. Where RAC2 keeps its light bank is the open question.
321 lines
16 KiB
Markdown
321 lines
16 KiB
Markdown
# Roadmap
|
|
|
|
Status as of **2026-10-09**.
|
|
|
|
A from-scratch native Rust reimplementation of Ratchet & Clank: Going Commando
|
|
(PS2). Each phase ends in a **gate** — something checkable, not a feeling that
|
|
it is going well. Phases are ordered so that the riskiest unknowns are retired
|
|
early, while they are still cheap to retire.
|
|
|
|
## Where we are
|
|
|
|
- [x] Target disc identified and verified (`SCUS_972.68` v1.01, NTSC 60 Hz)
|
|
- [x] Workspace builds (`cargo check --workspace`)
|
|
- [x] Repository hygiene in place (pre-commit guard, `.gitignore`, licence
|
|
record)
|
|
- [x] **Phase 1 complete.** The disc pipeline runs end to end and is verified
|
|
against a committed hash table: 5,213 files, 5,762,497,559 bytes, every
|
|
one of the 92 asset containers parsing.
|
|
- [x] **Phase 2 in progress.** Level containers are parsed through their typed
|
|
header, and the `data` extent is parsed as a container of byte ranges. The
|
|
geometry decompresses: level 0 yields 16,969,296 bytes of level content
|
|
from a 9.7 MB stream.
|
|
- [x] **First drawable frame.** `gc-engine` loads a level, applies LOD selection
|
|
and morphing for the camera, groups triangles by texture and draws them
|
|
under a Bevy fly-camera. Level 0 renders as 26,758 textured triangles
|
|
across 92 textures with correct UVs. Set `GC_ENGINE_SHOT=<path.ppm>` to
|
|
capture a frame and exit.
|
|
- [ ] **The origin bug this exposed.** Positions are i16 offsets from a
|
|
per-patch origin, so taking the wrong STROW collapsed the whole level to
|
|
a +-32 unit blob at the world origin while every parser test still
|
|
passed: the vertex count matched, the triangle count matched, only the
|
|
*place* was wrong. Fixed by taking the STROW immediately preceding the
|
|
first position unpack rather than the last one.
|
|
- [x] **Per-vertex colour.** The tfrag's RGBA array now modulates the texture,
|
|
drawn unlit so the frame shows real albedo rather than a faked lighting
|
|
model. Verified against the stored values (level 0 patch 0 runs
|
|
`(25,26,26)` to `(108,144,141)`), which are genuinely dark.
|
|
- [x] **Tfrag normals.** The 256-entry sine table is decoded from the level
|
|
overlay and normals are built per vertex and handed to the renderer. The
|
|
overlay has **no ELF header**, so the table cannot be reached through a
|
|
program header, and its offset differs per level (level 0 `0xb650c`,
|
|
level 5 `0xcca30`); it is located by matching the whole 256-entry curve.
|
|
Normals are negated as stored and get the same axis swap as positions.
|
|
The pipeline now also writes `overlay.bin`.
|
|
- [x] **The PS2 float model.** `gc-engine::ps2` is the EE's FPU on raw bits: no
|
|
denormals, no NaN or infinity, truncation toward zero, the adder's
|
|
operand alignment, and `rsqrt` as sqrt-then-divide. `MAX` reads back in
|
|
Rust as NaN, which is the sharpest available statement of why this is
|
|
not `f32`.
|
|
- [x] **The `LightTfrags` pass** is written and unit-tested against the PS2
|
|
model: PEXT5 base colour, `select` set blending, re-normalised blended
|
|
directions, and `base + A*max(dA, dA*wA) + B*max(dB, dB*wB)`.
|
|
- [ ] **The level light bank does not parse on RAC2.** The reference's framing
|
|
(u32 offset at +0x04, i32 count there) yields offset `0xa0d` and a
|
|
nonsense count on level 0. The renderer now reports how many vertices it
|
|
actually lit, which is currently 0 of 80,274 -- so the frame falls back
|
|
to the stored colours and says so rather than looking plausibly lit.
|
|
Detail in `plan/level-assets.md`.
|
|
- [ ] Then the GS register set and the guard-band clip test.
|
|
|
|
The disc being the exact build the reference projects pin is the single
|
|
biggest favourable fact so far: no disc conversion, no regional timing work,
|
|
and no re-derivation of the reference baseline.
|
|
|
|
---
|
|
|
|
## Phase 0 — Foundation ✅
|
|
|
|
**Done.**
|
|
|
|
- Disc identified: `SCUS_972.68` v1.01, USA Black Label, NTSC 60 Hz. Boot ELF
|
|
SHA-1 `bacda860cbd6723142e86db56f197e47e728970f`, entry `0x00131AE8` — the
|
|
entry point independently matches the value recorded by `rac2-gc-decomp`,
|
|
which cross-confirms the identification.
|
|
- `tools/identify-disc.py` reproduces the identification against any disc image
|
|
and distinguishes a truncated dump from a non-PS2 image.
|
|
- Five-crate workspace scaffolding, Bevy 0.19.1 pinned, builds clean.
|
|
- `.githooks/pre-commit` blocks game data. Verified against seven cases:
|
|
`.iso`, renamed disc image, WAD container, ADX container, `extracted/`,
|
|
a 60 MiB file, and legitimate source.
|
|
- `AGENTS.md` and `PROVENANCE.md` record conventions and licences.
|
|
|
|
### Open from this phase
|
|
|
|
- `alsa-lib-devel` is not installed, so Bevy's host audio backend is disabled in
|
|
`Cargo.toml`. Audio is the last phase; revisit then.
|
|
- Bevy feature set is deliberately minimal. Re-add features as needed rather
|
|
than switching `default-features` back on wholesale.
|
|
|
|
---
|
|
|
|
## Phase 1 — Disc pipeline ✅
|
|
|
|
**Done.** Gate met: every extracted file matches the committed hash table.
|
|
|
|
- [x] `iso9660` reader (R&C2 lists assets as real files, unlike R&C1)
|
|
- [x] `G/*.WAD` container header parsing — 92 containers, three table shapes
|
|
- [x] WAD LZ77 decompression (`wad.rs`, four packet classes)
|
|
- [x] Unpack every container, bounds-checking each extent
|
|
- [x] SHA-1 table for all 5,135 files (hashes only, never content)
|
|
- [x] `verify` stage, plus integration tests against a real disc
|
|
- [x] Format specs written: `disc-layout.md`, `global-wad.md`, `wad.md`
|
|
- [x] Work record with the negative results: `plan/disc-pipeline.md`
|
|
|
|
```text
|
|
gc-extract identify disc.iso -> SCUS_972.68 v1.01 NTSC, ELF SHA-1 matches
|
|
gc-extract unpack disc.iso out/ -> 5135 files, 4,500,106,960 bytes, 10.8s
|
|
gc-extract verify out/ table.tsv -> OK: 5135 files match
|
|
```
|
|
|
|
**Not done, deliberately:** the table of contents at sector 1001 is documented
|
|
but not read. Enumerating the ISO filesystem is simpler and sufficient; the ToC
|
|
matters when per-level overlay loading is understood in Phase 3.
|
|
|
|
**Two wrong turns are recorded** in `plan/disc-pipeline.md`, because both would
|
|
be easy to repeat: the container pair table read backwards, and the LZ77 stream
|
|
mistaken for a bit stream. Both produced plausible-looking wrong output before
|
|
failing, which is the failure mode worth guarding against.
|
|
|
|
---
|
|
|
|
## Phase 2 — Format parsers
|
|
|
|
**Goal:** every asset type parses into plain Rust values, tested against real
|
|
disc data.
|
|
|
|
Phase 1 hands over the extents; this phase works out what each one contains.
|
|
|
|
**Done so far:**
|
|
|
|
- [x] `level_wad.rs` — the 0x60-byte header on all 27 level containers, naming
|
|
four extents plus three chunk descriptors. Verified: extents tile each
|
|
container in file order (27 of 27).
|
|
- [x] Two facts the format documentation does not state, both measured here:
|
|
sizes in this header are **sector counts** where the generic `G/` table
|
|
uses byte counts, and **`LEVEL21.WAD` reports level id 30**, so container
|
|
order is not level identity.
|
|
|
|
**In progress:**
|
|
|
|
- [x] `data` — the extent header locates `core_index` and `core_data`; the
|
|
geometry decompresses and is verified on level 0.
|
|
- [ ] `core_index` class tables — the header base is confirmed (the compressed
|
|
and decompressed sizes of `core_data` sit at the same offsets in all 27
|
|
levels), and every offset lands inside `core_data`. But the class-table
|
|
fields do not decode under any field order or stride tried. Ruled out and
|
|
written up in `plan/level-assets.md`. Next: identify `tfrag` by its own
|
|
structure instead of through the header.
|
|
- [x] `occlusion` and `sky` offsets located and their first bytes read.
|
|
- [x] Region layout of `core_data` established: tfrag, occlusion, sky, collision
|
|
and `textures_base` are offsets, strictly increasing on all 27 levels.
|
|
Terrain is `[0, occlusion)` and carries inline texels.
|
|
- [x] **tfrag vertex format: recovered.** The tfrags block header gives a table
|
|
offset (`0x40`) and a patch count; each patch is a 0x40-byte header whose
|
|
`data` offset is relative to the table, not to the block. Positions are
|
|
V3_16 payloads inside a VIF1 command list, split across three lists per
|
|
patch (common, LOD 0+1, LOD 0). The whole vector format is encoded in the
|
|
command byte and every element size follows from it. Verified on all 27
|
|
levels: **18,102 patches, 821,439 vertices**, each patch yielding exactly
|
|
its own `vert_count`. See `plan/level-assets.md` and `gc-formats::tfrag`.
|
|
- [x] **tfrag topology: recovered.** Vertex-info (UVs plus LOD parenting), the
|
|
strip command lists and the index arrays, classified by unpack format and
|
|
destination VU address. Strip indices address the vertex-info array
|
|
rather than positions. Every patch's strips expand to exactly its own
|
|
`tri_count`: **18,102 patches, 809,879 triangles**, all indices in range.
|
|
See `plan/level-assets.md` and `gc-formats::tfrag`.
|
|
- [x] **tfrag texture bindings and per-vertex data.** The ad-gifs decode to a
|
|
texture index, wrap modes, minification filter and LOD bias; the level
|
|
index gained a texture table (`level_core.rs`) for those indices to land
|
|
in. Per-vertex RGBA and light/normal records recovered from their fixed
|
|
offsets. All 27 levels: 47,950 ad-gifs, every index in range, 90% of the
|
|
level texture tables referenced.
|
|
- [x] **Level textures decode.** 8-bit indexed texels plus a 256-entry CSM1
|
|
CLUT from `gs_ram`, honouring the PS2 CLUT block swap and the 0x80 alpha
|
|
scale. The pipeline now writes `gs_ram.bin`, which it had been skipping.
|
|
All 1,595 tfrag textures on all 27 levels decode; 96.8% fully opaque and
|
|
96.3% with real detail. Verified visually via the new
|
|
`gc-extract dump-texture`, because a wrong lookup still yields a
|
|
well-formed image of the wrong thing.
|
|
- [x] **tfrag LOD selection.** Switch distances scale as `[6L, 4L, 2L]` from the
|
|
block header's float; six draw modes result from testing the bounding
|
|
sphere against `trunc(D * 1024)`, so a patch straddling a threshold takes
|
|
the coarser mode and distance never moves back to finer detail. Morph
|
|
weights run 0..1 across each tier's band, and collapse replaces an entry
|
|
with its first parent rather than blending it.
|
|
- [x] **tfrag parent indices.** The four V4_8 arrays are classified by VU
|
|
address, but the addresses collide, and each V4_8 element holds four
|
|
indices so an array is `count * 4` bytes rather than `count`. With
|
|
`unk_06`/`unk_0a` suppressing absent tiers, all 370,349 LOD entries
|
|
across the 27 levels resolve: 29,043 LOD-01 and 341,306 LOD-0, zero
|
|
failures. See `plan/level-assets.md` for the two corrections.
|
|
- [ ] `gameplay` — begins `WAD` + a per-level byte, and does *not* decode with
|
|
the WAD LZ77 decoder. All 27 levels fail identically, so the input is
|
|
wrong rather than the decoder. Ruled out: bit order, header length, the
|
|
LZ77 framing. Hypotheses queued in `plan/level-assets.md`.
|
|
- [ ] `occlusion` — 2 to 25 sectors, small enough to read whole.
|
|
- [ ] `sound_bank` — `SBlk` and `DAW` signatures found in all 27.
|
|
- [ ] `data` — the large one: 4.2 to 21.9 MB of geometry per level.
|
|
|
|
- [ ] `tfrag` — terrain: LOD chains, strips, morph targets
|
|
- [ ] `tie` — instanced environment geometry, fat vertices
|
|
- [ ] `shrub` — foliage and decorative meshes
|
|
- [ ] `moby` — dynamic actors; `moby_anim` — skeletal animation
|
|
- [ ] `collision` — cells, packed vertices, faces
|
|
- [ ] `occlusion` — the PVS grid
|
|
- [ ] `sky`, `volumes`, `textures` (CSM1 CLUT + PS2 alpha)
|
|
- [ ] `level_overlay` — the seven-section parser and relocation from Phase 3
|
|
|
|
**Gate:** every parser has a golden test against bytes from the real disc, and
|
|
reports mismatched header counts as failures rather than warnings.
|
|
|
|
**Discipline:** bound every read. Typed errors, no panics, no reads past the
|
|
buffer.
|
|
|
|
---
|
|
|
|
## Phase 3 — Overlay system
|
|
|
|
**Goal:** make 27 level overlays tractable.
|
|
|
|
Every overlay carries the same seven sections (`.lit`, `.bss`, `.data`,
|
|
`lvl.vtbl`, `lvl.camvtbl`, `lvl.sndvtbl`, `.text`) and the same engine code,
|
|
linked at different addresses. ReRAC solves this by masking relocations — `j`/
|
|
`jal` targets, `lui` immediates of main-RAM addresses, the `%lo` that completes
|
|
them, and `$gp`-relative operands — then matching the masked words to find a
|
|
reference function's copy on another overlay.
|
|
|
|
- [ ] Port the masking and code-identity logic
|
|
- [ ] `Relocation::func` and `Relocation::data`
|
|
- [ ] `$gp` for this game (`0x001AEFF0`, against R&C1's `0x166c00`)
|
|
- [ ] Extend the level table from 19 to 27
|
|
|
|
**Gate:** every level-01 function's relocated copy is found on all 27 overlays;
|
|
cross-checked against an independent cluster hash.
|
|
|
|
**Why this is high-value:** it turns "27 overlays of unknown code" into a table
|
|
lookup, and it is the main thing inherited from ReRAC.
|
|
|
|
---
|
|
|
|
## Phase 4 — Engine core
|
|
|
|
**Goal:** draw a frame.
|
|
|
|
- [ ] Graphics Synthesizer in wgpu, with the GS's own rules: its
|
|
`round(log2(z/n) + K)` mip selection, blend and alpha-test behaviour,
|
|
alpha `0x80` meaning 1.0, CLAMP wrap
|
|
- [ ] VU0/VU1 skinning and lighting on the PS2 float model
|
|
- [ ] LOD/morph/cull per `TfragProc` and `TieProc`
|
|
- [ ] Occlusion culling, fog, shadows
|
|
- [ ] GIF/VIF packet stream interpretation
|
|
|
|
**Gate:** CPU and GPU paths agree on every vertex of a reference level — the
|
|
same identity check ReRAC runs.
|
|
|
|
**Risk:** VU1 microcode is **binary** in the game's `.vutext` section, so it is
|
|
reverse engineered rather than parsed. This is the hardest single item in the
|
|
project and the main threat to the timeline. Note that ReRAC's own moby VU1
|
|
analysis trips their disassembler on two opcodes.
|
|
|
|
---
|
|
|
|
## Phase 5 — Veldin vertical slice
|
|
|
|
**Goal:** one planet, playable start to finish.
|
|
|
|
- [ ] Ratchet's controller on a **spherical** planet
|
|
- [ ] Chargeboots: hold/release timing, chained jumps, dash
|
|
- [ ] Wrench and the camera
|
|
- [ ] Collision and the HUD
|
|
|
|
**Gate:** Veldin is completable and visually matches PCSX2 frame-for-frame under
|
|
replayed input.
|
|
|
|
**This is the go/no-go for the project.** If Veldin works, the other 26 levels
|
|
are content. If it does not, the reason is discovered while it is still cheap.
|
|
Spherical gravity is the thing most likely to break, because it invalidates
|
|
R&C1's movement and camera assumptions wholesale.
|
|
|
|
---
|
|
|
|
## Phase 6 — Remaining overlays and the shell
|
|
|
|
- [ ] Per-level moby class dispatch via `lvl.vtbl`
|
|
- [ ] Remaining 26 levels
|
|
- [ ] Planets, travel, the Gadgetron vendor, RPG upgrades
|
|
- [ ] Weapon modification bitmasks (lock-on, shock, acid, …)
|
|
- [ ] Minigames and challenges
|
|
- [ ] The 25-state memory-card FSM and its UI
|
|
- [ ] IFF chunk saves, including import of a R&C1 save to unlock legacy weapons
|
|
|
|
---
|
|
|
|
## Phase 7 — Audio and video
|
|
|
|
- [ ] 989snd v2 audio decode and playback (not CRI ADX — a different codec)
|
|
- [ ] SPU2 mixing, reverb zones, level sounds
|
|
- [ ] IPU/MPEG-2 video for the title screen and cutscenes
|
|
|
|
---
|
|
|
|
## Phase 8 — Fidelity pass
|
|
|
|
Differential testing against PCSX2 across every level:
|
|
|
|
- [ ] Recorded-input frame comparison, per planet
|
|
- [ ] Geometry, fog, blend, alpha, texture sampling
|
|
- [ ] Audio sync and 60 Hz timing
|
|
- [ ] Behavioural comparison: traversal routes, boss timing, RNG
|
|
|
|
---
|
|
|
|
## Principles
|
|
|
|
- **A gate is pass/fail, not a percentage.** "Looks fine" is not progress.
|
|
- **Negative results are written down.** A ruled-out approach saves the next
|
|
attempt as much as a working one.
|
|
- **The float model is not negotiable.** Approximate is not the same as
|
|
faithful, and the difference is visible in how the game feels.
|
|
- **Fidelity over tidiness.** Where the original does something that looks like
|
|
a bug, reproduce it and say why in a comment. See `AGENTS.md`. |