ReaperOfVeriod
a9e43e39c1
The first typed structure read from real data. Every one of the 27 level containers begins with the same 0x60-byte header, which turns the generic pair table into named assets: data, sound_bank, gameplay and occlusion, plus three chunk descriptors. Two things about this header are worth stating because both are silent failures rather than errors. Sizes here are sector counts, not byte counts. The generic G/ pair table on the same disc uses byte counts, so both readings type-check and the wrong one is off by a factor of 2048. The tiling check settles it: read as sectors, the four extents tile every one of the 27 containers exactly; read as bytes they overlap each other. Level id is not container order. LEVEL21.WAD reports id 30, so a level was cut from the disc and its number left a gap. Anything keying off file order -- a save slot, the planet select screen -- would mislabel that planet. There is also a second trap in the same area: the ISO filesystem sorts names alphabetically, so LEVEL10 precedes LEVEL2, and any tool needing order has to sort numerically. The header's field at 0x04 is zero on all 27 levels. It is not the container size despite looking like one, so the code reads it as a runtime field and takes the container length from the ISO entry instead. Measured across all 27 levels, three invariants now hold in tests: extents tile in file order; extent coverage plus chunks reaches at least the end of the container (21 levels land exactly on its length, 6 leave 134 to 689 sectors of trailing data); and exactly one level's id differs from its container index. The coverage assertion is >= rather than == on purpose -- 21 levels end one sector past the file end because the header occupies sector 0, and asserting equality would truncate a level's tail. gameplay is left unparsed and that is recorded rather than hidden. It begins "WAD" plus a per-level byte and does not decode with the WAD LZ77 decoder: all 27 levels produce plausible output for the first packet and then fail on a rule the format's own encoder never breaks, which means the input is wrong rather than the decoder. docs/plan/level-assets.md records what is ruled out and what to try next. 66 unit tests, plus 7 integration tests against the real disc. The unpack hash table is unchanged, so this adds parsing without altering the pipeline.
233 lines
9.3 KiB
Markdown
233 lines
9.3 KiB
Markdown
# Roadmap
|
|
|
|
Status as of **2026-10-09**.
|
|
|
|
A from-scratch native Rust reimplementation of Ratchet & Clank: Going Commando
|
|
(PS2). Each phase ends in a **gate** — something checkable, not a feeling that
|
|
it is going well. Phases are ordered so that the riskiest unknowns are retired
|
|
early, while they are still cheap to retire.
|
|
|
|
## Where we are
|
|
|
|
- [x] Target disc identified and verified (`SCUS_972.68` v1.01, NTSC 60 Hz)
|
|
- [x] Workspace builds (`cargo check --workspace`)
|
|
- [x] Repository hygiene in place (pre-commit guard, `.gitignore`, licence
|
|
record)
|
|
- [x] **Phase 1 complete.** The disc pipeline runs end to end and is verified
|
|
against a committed hash table: 5,135 files, 4,500,106,960 bytes, every
|
|
one of the 92 asset containers parsing.
|
|
- [x] **Phase 2 started.** The level container header is parsed and typed, which
|
|
names the `data`, `sound_bank`, `gameplay` and `occlusion` extents inside
|
|
each of the 27 level containers.
|
|
- [ ] Phase 2 next: the `gameplay` extent, then `data`.
|
|
|
|
The disc being the exact build the reference projects pin is the single
|
|
biggest favourable fact so far: no disc conversion, no regional timing work,
|
|
and no re-derivation of the reference baseline.
|
|
|
|
---
|
|
|
|
## Phase 0 — Foundation ✅
|
|
|
|
**Done.**
|
|
|
|
- Disc identified: `SCUS_972.68` v1.01, USA Black Label, NTSC 60 Hz. Boot ELF
|
|
SHA-1 `bacda860cbd6723142e86db56f197e47e728970f`, entry `0x00131AE8` — the
|
|
entry point independently matches the value recorded by `rac2-gc-decomp`,
|
|
which cross-confirms the identification.
|
|
- `tools/identify-disc.py` reproduces the identification against any disc image
|
|
and distinguishes a truncated dump from a non-PS2 image.
|
|
- Five-crate workspace scaffolding, Bevy 0.19.1 pinned, builds clean.
|
|
- `.githooks/pre-commit` blocks game data. Verified against seven cases:
|
|
`.iso`, renamed disc image, WAD container, ADX container, `extracted/`,
|
|
a 60 MiB file, and legitimate source.
|
|
- `AGENTS.md` and `PROVENANCE.md` record conventions and licences.
|
|
|
|
### Open from this phase
|
|
|
|
- `alsa-lib-devel` is not installed, so Bevy's host audio backend is disabled in
|
|
`Cargo.toml`. Audio is the last phase; revisit then.
|
|
- Bevy feature set is deliberately minimal. Re-add features as needed rather
|
|
than switching `default-features` back on wholesale.
|
|
|
|
---
|
|
|
|
## Phase 1 — Disc pipeline ✅
|
|
|
|
**Done.** Gate met: every extracted file matches the committed hash table.
|
|
|
|
- [x] `iso9660` reader (R&C2 lists assets as real files, unlike R&C1)
|
|
- [x] `G/*.WAD` container header parsing — 92 containers, three table shapes
|
|
- [x] WAD LZ77 decompression (`wad.rs`, four packet classes)
|
|
- [x] Unpack every container, bounds-checking each extent
|
|
- [x] SHA-1 table for all 5,135 files (hashes only, never content)
|
|
- [x] `verify` stage, plus integration tests against a real disc
|
|
- [x] Format specs written: `disc-layout.md`, `global-wad.md`, `wad.md`
|
|
- [x] Work record with the negative results: `plan/disc-pipeline.md`
|
|
|
|
```text
|
|
gc-extract identify disc.iso -> SCUS_972.68 v1.01 NTSC, ELF SHA-1 matches
|
|
gc-extract unpack disc.iso out/ -> 5135 files, 4,500,106,960 bytes, 10.8s
|
|
gc-extract verify out/ table.tsv -> OK: 5135 files match
|
|
```
|
|
|
|
**Not done, deliberately:** the table of contents at sector 1001 is documented
|
|
but not read. Enumerating the ISO filesystem is simpler and sufficient; the ToC
|
|
matters when per-level overlay loading is understood in Phase 3.
|
|
|
|
**Two wrong turns are recorded** in `plan/disc-pipeline.md`, because both would
|
|
be easy to repeat: the container pair table read backwards, and the LZ77 stream
|
|
mistaken for a bit stream. Both produced plausible-looking wrong output before
|
|
failing, which is the failure mode worth guarding against.
|
|
|
|
---
|
|
|
|
## Phase 2 — Format parsers
|
|
|
|
**Goal:** every asset type parses into plain Rust values, tested against real
|
|
disc data.
|
|
|
|
Phase 1 hands over the extents; this phase works out what each one contains.
|
|
|
|
**Done so far:**
|
|
|
|
- [x] `level_wad.rs` — the 0x60-byte header on all 27 level containers, naming
|
|
four extents plus three chunk descriptors. Verified: extents tile each
|
|
container in file order (27 of 27).
|
|
- [x] Two facts the format documentation does not state, both measured here:
|
|
sizes in this header are **sector counts** where the generic `G/` table
|
|
uses byte counts, and **`LEVEL21.WAD` reports level id 30**, so container
|
|
order is not level identity.
|
|
|
|
**In progress:**
|
|
|
|
- [ ] `gameplay` — begins `WAD` + a per-level byte, and does *not* decode with
|
|
the WAD LZ77 decoder. All 27 levels fail identically, so the input is
|
|
wrong rather than the decoder. Ruled out: bit order, header length, the
|
|
LZ77 framing. Hypotheses queued in `plan/level-assets.md`.
|
|
- [ ] `occlusion` — 2 to 25 sectors, small enough to read whole.
|
|
- [ ] `sound_bank` — `SBlk` and `DAW` signatures found in all 27.
|
|
- [ ] `data` — the large one: 4.2 to 21.9 MB of geometry per level.
|
|
|
|
- [ ] `tfrag` — terrain: LOD chains, strips, morph targets
|
|
- [ ] `tie` — instanced environment geometry, fat vertices
|
|
- [ ] `shrub` — foliage and decorative meshes
|
|
- [ ] `moby` — dynamic actors; `moby_anim` — skeletal animation
|
|
- [ ] `collision` — cells, packed vertices, faces
|
|
- [ ] `occlusion` — the PVS grid
|
|
- [ ] `sky`, `volumes`, `textures` (CSM1 CLUT + PS2 alpha)
|
|
- [ ] `level_overlay` — the seven-section parser and relocation from Phase 3
|
|
|
|
**Gate:** every parser has a golden test against bytes from the real disc, and
|
|
reports mismatched header counts as failures rather than warnings.
|
|
|
|
**Discipline:** bound every read. Typed errors, no panics, no reads past the
|
|
buffer.
|
|
|
|
---
|
|
|
|
## Phase 3 — Overlay system
|
|
|
|
**Goal:** make 27 level overlays tractable.
|
|
|
|
Every overlay carries the same seven sections (`.lit`, `.bss`, `.data`,
|
|
`lvl.vtbl`, `lvl.camvtbl`, `lvl.sndvtbl`, `.text`) and the same engine code,
|
|
linked at different addresses. ReRAC solves this by masking relocations — `j`/
|
|
`jal` targets, `lui` immediates of main-RAM addresses, the `%lo` that completes
|
|
them, and `$gp`-relative operands — then matching the masked words to find a
|
|
reference function's copy on another overlay.
|
|
|
|
- [ ] Port the masking and code-identity logic
|
|
- [ ] `Relocation::func` and `Relocation::data`
|
|
- [ ] `$gp` for this game (`0x001AEFF0`, against R&C1's `0x166c00`)
|
|
- [ ] Extend the level table from 19 to 27
|
|
|
|
**Gate:** every level-01 function's relocated copy is found on all 27 overlays;
|
|
cross-checked against an independent cluster hash.
|
|
|
|
**Why this is high-value:** it turns "27 overlays of unknown code" into a table
|
|
lookup, and it is the main thing inherited from ReRAC.
|
|
|
|
---
|
|
|
|
## Phase 4 — Engine core
|
|
|
|
**Goal:** draw a frame.
|
|
|
|
- [ ] Graphics Synthesizer in wgpu, with the GS's own rules: its
|
|
`round(log2(z/n) + K)` mip selection, blend and alpha-test behaviour,
|
|
alpha `0x80` meaning 1.0, CLAMP wrap
|
|
- [ ] VU0/VU1 skinning and lighting on the PS2 float model
|
|
- [ ] LOD/morph/cull per `TfragProc` and `TieProc`
|
|
- [ ] Occlusion culling, fog, shadows
|
|
- [ ] GIF/VIF packet stream interpretation
|
|
|
|
**Gate:** CPU and GPU paths agree on every vertex of a reference level — the
|
|
same identity check ReRAC runs.
|
|
|
|
**Risk:** VU1 microcode is **binary** in the game's `.vutext` section, so it is
|
|
reverse engineered rather than parsed. This is the hardest single item in the
|
|
project and the main threat to the timeline. Note that ReRAC's own moby VU1
|
|
analysis trips their disassembler on two opcodes.
|
|
|
|
---
|
|
|
|
## Phase 5 — Veldin vertical slice
|
|
|
|
**Goal:** one planet, playable start to finish.
|
|
|
|
- [ ] Ratchet's controller on a **spherical** planet
|
|
- [ ] Chargeboots: hold/release timing, chained jumps, dash
|
|
- [ ] Wrench and the camera
|
|
- [ ] Collision and the HUD
|
|
|
|
**Gate:** Veldin is completable and visually matches PCSX2 frame-for-frame under
|
|
replayed input.
|
|
|
|
**This is the go/no-go for the project.** If Veldin works, the other 26 levels
|
|
are content. If it does not, the reason is discovered while it is still cheap.
|
|
Spherical gravity is the thing most likely to break, because it invalidates
|
|
R&C1's movement and camera assumptions wholesale.
|
|
|
|
---
|
|
|
|
## Phase 6 — Remaining overlays and the shell
|
|
|
|
- [ ] Per-level moby class dispatch via `lvl.vtbl`
|
|
- [ ] Remaining 26 levels
|
|
- [ ] Planets, travel, the Gadgetron vendor, RPG upgrades
|
|
- [ ] Weapon modification bitmasks (lock-on, shock, acid, …)
|
|
- [ ] Minigames and challenges
|
|
- [ ] The 25-state memory-card FSM and its UI
|
|
- [ ] IFF chunk saves, including import of a R&C1 save to unlock legacy weapons
|
|
|
|
---
|
|
|
|
## Phase 7 — Audio and video
|
|
|
|
- [ ] 989snd v2 audio decode and playback (not CRI ADX — a different codec)
|
|
- [ ] SPU2 mixing, reverb zones, level sounds
|
|
- [ ] IPU/MPEG-2 video for the title screen and cutscenes
|
|
|
|
---
|
|
|
|
## Phase 8 — Fidelity pass
|
|
|
|
Differential testing against PCSX2 across every level:
|
|
|
|
- [ ] Recorded-input frame comparison, per planet
|
|
- [ ] Geometry, fog, blend, alpha, texture sampling
|
|
- [ ] Audio sync and 60 Hz timing
|
|
- [ ] Behavioural comparison: traversal routes, boss timing, RNG
|
|
|
|
---
|
|
|
|
## Principles
|
|
|
|
- **A gate is pass/fail, not a percentage.** "Looks fine" is not progress.
|
|
- **Negative results are written down.** A ruled-out approach saves the next
|
|
attempt as much as a working one.
|
|
- **The float model is not negotiable.** Approximate is not the same as
|
|
faithful, and the difference is visible in how the game feels.
|
|
- **Fidelity over tidiness.** Where the original does something that looks like
|
|
a bug, reproduce it and say why in a comment. See `AGENTS.md`. |