Going-Commando/docs/roadmap.md

Roadmap

Status as of 2026-10-11.

A from-scratch native Rust reimplementation of Ratchet & Clank: Going Commando (PS2). Each phase ends in a gate — something checkable, not a feeling that it is going well. Phases are ordered so that the riskiest unknowns are retired early, while they are still cheap to retire.

Where we are

• Target disc identified and verified (SCUS_972.68 v1.01, NTSC 60 Hz)
• Workspace builds (cargo check --workspace)
• Repository hygiene in place (pre-commit guard, .gitignore, licence record)
• Phase 1 complete. The disc pipeline runs end to end and is verified against a committed hash table, every one of the 92 asset containers parsing. The pipeline now also writes each level's chunk descriptors where they exist (3 of 27 levels).
• Phase 2 in progress. Level containers are parsed through their typed header, and the data extent is parsed as a container of byte ranges. The geometry decompresses: level 0 yields 16,969,296 bytes of level content from a 9.7 MB stream.
• First drawable frame. gc-engine loads a level, applies LOD selection and morphing for the camera, groups triangles by texture and draws them under a Bevy fly-camera. Level 0 renders as 26,758 textured triangles across 92 textures with correct UVs. Set GC_ENGINE_SHOT=<path.ppm> to capture a frame and exit.
• The origin bug this exposed. Positions are i16 offsets from a per-patch origin, so taking the wrong STROW collapsed the whole level to a +-32 unit blob at the world origin while every parser test still passed: the vertex count matched, the triangle count matched, only the place was wrong. Fixed by taking the STROW immediately preceding the first position unpack rather than the last one.
• Per-vertex colour. The tfrag's RGBA array now modulates the texture, drawn unlit so the frame shows real albedo rather than a faked lighting model. Verified against the stored values (level 0 patch 0 runs (25,26,26) to (108,144,141)), which are genuinely dark.
• Tfrag normals. The 256-entry sine table is decoded from the level overlay and normals are built per vertex and handed to the renderer. The overlay has no ELF header, so the table cannot be reached through a program header, and its offset differs per level (level 0 0xb650c, level 5 0xcca30); it is located by matching the whole 256-entry curve. Normals are negated as stored and get the same axis swap as positions. The pipeline now also writes overlay.bin.
• The PS2 float model. gc-engine::ps2 is the EE's FPU on raw bits: no denormals, no NaN or infinity, truncation toward zero, the adder's operand alignment, and rsqrt as sqrt-then-divide. MAX reads back in Rust as NaN, which is the sharpest available statement of why this is not f32.
• The LightTfrags pass is written and unit-tested against the PS2 model: PEXT5 base colour, select set blending, re-normalised blended directions, and base + A*max(dA, dA*wA) + B*max(dB, dB*wB).
• The level light bank parses and level 0 is fully lit. gameplay is a WAD stream; the extractor had been writing it compressed, which is why the bank framing found garbage. All 80,274 vertices of level 0 now go through LightTfrags. The renderer still counts and reports how many vertices it lit, because a silent fallback to stored colours would look plausible and be wrong.
• The shrub region is bounded, and with it the whole lump. Shrub's end was the last unlocated region bound: it is the start of the post-class chunk container, found by a marker-paired signature in core_data (the index carries no successor offset). The container closes shrub on the 24 shrub levels and tie on the three without. All 358 shrub class bodies are now bounded and are whole 8-byte records, so shrub measurements are trustworthy for the first time -- the "3.5M records" figure from the unbounded final span is replaced by a measured 9,600.
• Lighting magnitudes are verified by the PCSX2 comparison loop. gc-trace compare-tfrag-light runs the pass over level 0's disc data and diffs it against the game's own lit colours in EE RAM (live over PINE, or from a savestate): all 162,400 colour bytes match, across 833/833 headers relocated and the light bank byte-identical. Two bugs it found: the normal's z was not negated (dark but plausible, 95.88% agreement), and records with no light set drew transparent black instead of the base colour. See docs/plan/pcsx2-verification.md.
• Then the GS register set and the guard-band clip test.

The disc being the exact build the reference projects pin is the single biggest favourable fact so far: no disc conversion, no regional timing work, and no re-derivation of the reference baseline.

---

Phase 0 — Foundation ✅

Done.

• Disc identified: SCUS_972.68 v1.01, USA Black Label, NTSC 60 Hz. Boot ELF SHA-1 bacda860cbd6723142e86db56f197e47e728970f, entry 0x00131AE8 — the entry point independently matches the value recorded by rac2-gc-decomp, which cross-confirms the identification.
• tools/identify-disc.py reproduces the identification against any disc image and distinguishes a truncated dump from a non-PS2 image.
• Five-crate workspace scaffolding, Bevy 0.19.1 pinned, builds clean.
• .githooks/pre-commit blocks game data. Verified against seven cases: .iso, renamed disc image, WAD container, ADX container, extracted/, a 60 MiB file, and legitimate source.
• AGENTS.md and PROVENANCE.md record conventions and licences.

Open from this phase

• alsa-lib-devel is not installed, so Bevy's host audio backend is disabled in Cargo.toml. Audio is the last phase; revisit then.
• Bevy feature set is deliberately minimal. Re-add features as needed rather than switching default-features back on wholesale.

---

Phase 1 — Disc pipeline ✅

Done. Gate met: every extracted file matches the committed hash table.

• iso9660 reader (R&C2 lists assets as real files, unlike R&C1)
• G/*.WAD container header parsing — 92 containers, three table shapes
• WAD LZ77 decompression (wad.rs, four packet classes)
• Unpack every container, bounds-checking each extent
• SHA-1 table for all 5,135 files (hashes only, never content)
• verify stage, plus integration tests against a real disc
• Format specs written: disc-layout.md, global-wad.md, wad.md
• Work record with the negative results: plan/disc-pipeline.md

gc-extract identify disc.iso        -> SCUS_972.68 v1.01 NTSC, ELF SHA-1 matches
gc-extract unpack   disc.iso out/   -> 5135 files, 4,500,106,960 bytes, 10.8s
gc-extract verify   out/ table.tsv -> OK: 5135 files match

Not done, deliberately: the table of contents at sector 1001 is documented but not read. Enumerating the ISO filesystem is simpler and sufficient; the ToC matters when per-level overlay loading is understood in Phase 3.

Two wrong turns are recorded in plan/disc-pipeline.md, because both would be easy to repeat: the container pair table read backwards, and the LZ77 stream mistaken for a bit stream. Both produced plausible-looking wrong output before failing, which is the failure mode worth guarding against.

---

Phase 2 — Format parsers

Goal: every asset type parses into plain Rust values, tested against real disc data.

Phase 1 hands over the extents; this phase works out what each one contains.

Done so far:

• level_wad.rs — the 0x60-byte header on all 27 level containers, naming four extents plus three chunk descriptors. Verified: extents tile each container in file order (27 of 27).
• Two facts the format documentation does not state, both measured here: sizes in this header are sector counts where the generic G/ table uses byte counts, and LEVEL21.WAD reports level id 30, so container order is not level identity.

In progress:

• data — the extent header locates core_index and core_data; the geometry decompresses and is verified on level 0.

• core_index class tables — moby, tie and shrub. They were ruled out for want of an entry size, which is knowable exactly: each table ends precisely where the next begins, and that chain closes on 27 of 27 levels. 32 bytes per moby/tie entry, 48 per shrub. Across the disc: 5,796 moby, 1,864 tie, 358 shrub classes. Identifying tfrag by its own structure is what led here.

• occlusion and sky offsets located and their first bytes read.

• Region layout of core_data established: tfrag, occlusion, sky, collision and textures_base are offsets, strictly increasing on all 27 levels. Terrain is [0, occlusion) and carries inline texels.

• Region layout closed: the chunk container's marker-paired signature ends the last class region, and the constant trailer ends the container. Every byte of level-specific core_data now belongs to a named region.

• tfrag vertex format: recovered. The tfrags block header gives a table offset (0x40) and a patch count; each patch is a 0x40-byte header whose data offset is relative to the table, not to the block. Positions are V3_16 payloads inside a VIF1 command list, split across three lists per patch (common, LOD 0+1, LOD 0). The whole vector format is encoded in the command byte and every element size follows from it. Verified on all 27 levels: 18,102 patches, 821,439 vertices, each patch yielding exactly its own vert_count. See plan/level-assets.md and gc-formats::tfrag.

• tfrag topology: recovered. Vertex-info (UVs plus LOD parenting), the strip command lists and the index arrays, classified by unpack format and destination VU address. Strip indices address the vertex-info array rather than positions. Every patch's strips expand to exactly its own tri_count: 18,102 patches, 809,879 triangles, all indices in range. See plan/level-assets.md and gc-formats::tfrag.

• tfrag texture bindings and per-vertex data. The ad-gifs decode to a texture index, wrap modes, minification filter and LOD bias; the level index gained a texture table (level_core.rs) for those indices to land in. Per-vertex RGBA and light/normal records recovered from their fixed offsets. All 27 levels: 47,950 ad-gifs, every index in range, 90% of the level texture tables referenced.

• Level textures decode. 8-bit indexed texels plus a 256-entry CSM1 CLUT from gs_ram, honouring the PS2 CLUT block swap and the 0x80 alpha scale. The pipeline now writes gs_ram.bin, which it had been skipping. All 1,595 tfrag textures on all 27 levels decode; 96.8% fully opaque and 96.3% with real detail. Verified visually via the new gc-extract dump-texture, because a wrong lookup still yields a well-formed image of the wrong thing.

• tfrag LOD selection. Switch distances scale as [6L, 4L, 2L] from the block header's float; six draw modes result from testing the bounding sphere against trunc(D * 1024), so a patch straddling a threshold takes the coarser mode and distance never moves back to finer detail. Morph weights run 0..1 across each tier's band, and collapse replaces an entry with its first parent rather than blending it.

• tfrag parent indices. The four V4_8 arrays are classified by VU address, but the addresses collide, and each V4_8 element holds four indices so an array is count * 4 bytes rather than count. With unk_06/unk_0a suppressing absent tiers, all 370,349 LOD entries across the 27 levels resolve: 29,043 LOD-01 and 341,306 LOD-0, zero failures. See plan/level-assets.md for the two corrections.

• gameplay — it is a WAD stream, exactly like core_data; the extractor was writing it compressed. Decompressed, level 0 is 1,872,256 bytes and holds 5 directional light sets. The earlier "does not decode with the WAD LZ77 decoder, bit order and header length ruled out" note was wrong about the cause: the decoder was right and the input was not what it looked like. Corrected in plan/level-assets.md.

• occlusion — 2 to 25 sectors, small enough to read whole.

• sound_bank — SBlk and DAW signatures found in all 27.

• data — the large one: 4.2 to 21.9 MB of geometry per level.

• tfrag — terrain: LOD chains, strips, morph targets

• tie — instanced environment geometry, fat vertices

• [~] shrub — class table done (48-byte entries, not 32; reading shrub at 32 still gives 231 sane-looking entries of 358, which is the hazard). Geometry header parsed and checked field by field on all 358 spans: +0x24 equals the entry's own o_class, +0x2c is an in-span data offset, +0x16/+0x18 are zero, and +0x1c names a billboard on exactly the 25 classes whose core-index entry carries billboard texture fields. Shrub does not use moby's part-table format. The body is a 24-entry palette of unit-length s16 normals (squared length within 65,535 of 32767² on all 8,592 entries — a random triple would miss by a billion) plus zero padding records; the billboard record carries K, MMIN and the TEX0/MIPTBP addresses as R&C1's does. What remains open: the three floats in the billboard record, size, mode_bits, count, and where the mesh geometry lives — R&C1 puts the mesh packets in this blob and R&C2's blob holds only the palette.

• moby — per-class model geometry. A class span opens with a u32 pointing at a part table of 16-byte records; parts tile exactly, with vertex_offset + 16 * vertex_count == next_part holding for 43,467 of 43,467 parts across every class on all 27 levels. That identity is what makes the record size fall out of the data rather than being assumed. 1,270,802 vertices over 4,560 class spans; 116 spans carry no geometry. Vertices are 16-byte VIF-packed half-float quads, handed back raw.

• moby_anim — skeletal animation, still open

• collision — container framing. A 0x40-byte header self-describes: it carries its own size and the payload size, and header + payload stays inside core_data on all 27 levels. 41,481,520 payload bytes across 19 levels; 8 levels declare zero length. The cell layout is not decoded -- the payload does not walk as fixed-stride cells, and variable-length faces stop partway into the first cell rather than tiling it.

• occlusion — the PVS grid. A z/y/x tree of u16 node offsets over 4x4x4-unit cells, populated cells naming 128-byte masks. 412,521 cells over 140,939 masks across the 24 levels that have one; the mask array ends within 48 bytes of the block boundary on every level. 3 carry no occlusion data, and an offset of 0 means absent rather than "at the start" — the latter parses as a grid whose mask array happens to fit, which is how it nearly went wrong.

• sky, volumes; textures are done (CSM1 CLUT + PS2 alpha)

• level_overlay — the seven-section parser and relocation from Phase 3

Gate: every parser has a golden test against bytes from the real disc, and reports mismatched header counts as failures rather than warnings.

Discipline: bound every read. Typed errors, no panics, no reads past the buffer.

---

Phase 3 — Overlay system

Goal: make 27 level overlays tractable.

Every overlay carries the same seven sections (.lit, .bss, .data, lvl.vtbl, lvl.camvtbl, lvl.sndvtbl, .text) and the same engine code, linked at different addresses. ReRAC solves this by masking relocations — j/ jal targets, lui immediates of main-RAM addresses, the %lo that completes them, and $gp-relative operands — then matching the masked words to find a reference function's copy on another overlay.

• Port the masking and code-identity logic
• Relocation::func and Relocation::data
• $gp for this game (0x001AEFF0, against R&C1's 0x166c00)
• Extend the level table from 19 to 27

Gate: every level-01 function's relocated copy is found on all 27 overlays; cross-checked against an independent cluster hash.

Why this is high-value: it turns "27 overlays of unknown code" into a table lookup, and it is the main thing inherited from ReRAC.

---

Phase 4 — Engine core

Goal: draw a frame.

• Graphics Synthesizer in wgpu, with the GS's own rules: its round(log2(z/n) + K) mip selection, blend and alpha-test behaviour, alpha 0x80 meaning 1.0, CLAMP wrap
• VU0/VU1 skinning and lighting on the PS2 float model
• LOD/morph/cull per TfragProc and TieProc
• Occlusion culling, fog, shadows
• GIF/VIF packet stream interpretation

Gate: CPU and GPU paths agree on every vertex of a reference level — the same identity check ReRAC runs.

Risk: VU1 microcode is binary in the game's .vutext section, so it is reverse engineered rather than parsed. This is the hardest single item in the project and the main threat to the timeline. Note that ReRAC's own moby VU1 analysis trips their disassembler on two opcodes.

---

Phase 5 — Veldin vertical slice

Goal: one planet, playable start to finish.

• Ratchet's controller on a spherical planet
• Chargeboots: hold/release timing, chained jumps, dash
• Wrench and the camera
• Collision and the HUD

Gate: Veldin is completable and visually matches PCSX2 frame-for-frame under replayed input.

This is the go/no-go for the project. If Veldin works, the other 26 levels are content. If it does not, the reason is discovered while it is still cheap. Spherical gravity is the thing most likely to break, because it invalidates R&C1's movement and camera assumptions wholesale.

---

Phase 6 — Remaining overlays and the shell

• Per-level moby class dispatch via lvl.vtbl
• Remaining 26 levels
• Planets, travel, the Gadgetron vendor, RPG upgrades
• Weapon modification bitmasks (lock-on, shock, acid, …)
• Minigames and challenges
• The 25-state memory-card FSM and its UI
• IFF chunk saves, including import of a R&C1 save to unlock legacy weapons

---

Phase 7 — Audio and video

• 989snd v2 audio decode and playback (not CRI ADX — a different codec)
• SPU2 mixing, reverb zones, level sounds
• IPU/MPEG-2 video for the title screen and cutscenes

---

Phase 8 — Fidelity pass

Differential testing against PCSX2 across every level:

• Recorded-input frame comparison, per planet
• Geometry, fog, blend, alpha, texture sampling
• Audio sync and 60 Hz timing
• Behavioural comparison: traversal routes, boss timing, RNG

---

Principles

• A gate is pass/fail, not a percentage. "Looks fine" is not progress.
• Negative results are written down. A ruled-out approach saves the next attempt as much as a working one.
• The float model is not negotiable. Approximate is not the same as faithful, and the difference is visible in how the game feels.
• Fidelity over tidiness. Where the original does something that looks like a bug, reproduce it and say why in a comment. See AGENTS.md.