ReaperOfVeriod/Going-Commando
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Consolidate: tidy level.rs, correct stale docs, pin hardware models

Browse Source 
No behaviour change. This is the pass over the last stretch of work to
make sure the accumulated result hangs together rather than only each
commit in isolation.

Code. lit_color had grown a `let _ = (depth, lod_distances)` and a dead
branch from repeated edits. The directional pass genuinely does not
depend on distance, so the parameters are gone rather than discarded,
with a note that the point lights -- not implemented -- will need them.
lit_at folded into lit_color, which had become a wrapper around it.
Checked for leftover scaffolding: only two env vars remain, both real
(GC_ENGINE_SHOT and GC_ENGINE_DEBUG_BOUNDS); GC_ENGINE_UNLIT,
GC_ENGINE_DEBUG_COLOR and GC_ENGINE_ABL were never actually written,
because those edits silently failed to apply against rustfmt'd source.

AGENTS.md. The crate table did not say that gc-engine owns hardware
models. ps2.rs is the Emotion Engine's FPU and the lighting pass runs
on it, so the boundary is now stated: gc-formats decodes inputs,
gc-engine reproduces how the hardware computes with them. That is the
distinction rule 2 depends on, and it was implicit.

Docs. Three claims had been overtaken by what we learned and were
actively misleading:

* docs/formats/level-wad.md still said gameplay "does not decode with
  the WAD decoder". It does; it is a WAD stream.
* docs/plan/level-assets.md carried a whole section, "a compression we
  have not identified", with a ruled-out list including "bit order",
  "header length" and "the WAD format at all". All three were ruled out
  against the wrong input. Replaced, with the failure mode kept: a
  uniform failure across 27 inputs was read as "the decoder is wrong"
  when it meant "the data was handed over uncompressed", and the
  positive control -- decompress and look -- was never run.
* The same file still said the tfrag "record framing is not recovered".
  True, but it was a false lead that has since been superseded by the
  VIF1 framing. Kept as a recorded wrong turn rather than deleted.

Also refreshed "Still open" to name what is actually outstanding:
point lights, the GS register set, the guard-band clip test, the
unread core_index class tables, and -- stated plainly -- that lighting
magnitudes are unverified because there is still nothing to compare
against.

Verified after the changes: 148 tests pass, clippy and fmt clean, all
5,213 extracted files match the hash table, and level 0 still loads at
26,758 triangles with the same LOD split and fully lit.
This commit is contained in:
ReaperOfVeriod
2026-10-10 03:37:03 +02:00
parent bcf0d1f0cc
commit b8b4a1a3d3
5 changed files with 96 additions and 97 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+8 -1
View File
@@ -77,13 +77,20 @@ Read these before working in an unfamiliar area:
| `gc-extract` | Disc pipeline: identify, unpack, verify. |
| `gc-game` | Game logic: hero, weapons, classes, progression, saves. |
| `gc-video` | IPU/MPEG-2 decode. |
| `gc-engine` | Bevy app: GS, VU0/VU1, renderer, frame loop. |
| `gc-engine` | Bevy app: GS, VU0/VU1, renderer, frame loop. Also owns the **hardware models** — `ps2.rs` is the Emotion Engine's FPU, and the tfrag lighting pass runs on it. |
The dependency direction is `gc-formats` ← `gc-extract`; `gc-formats` ←
`gc-game` ← `gc-engine`; `gc-video` ← `gc-game` ← `gc-engine`. `gc-formats`
depends on nothing of ours. Do not invert these — a format parser that can
reach into the engine cannot be tested in isolation.
A hardware model belongs in `gc-engine`, not `gc-formats`. `gc-formats`
decodes *inputs* — including the data a hardware model consumes, such as the
overlay's sine table — but arithmetic that reproduces how the PS2 computes
something is an engine concern. `tfrag_light.rs` (formats) holds the normal
table and the light bank framing; `tfrag_light.rs` (engine) holds the pass
that uses them.
## Conventions
- Rust 2021 edition, workspace inheritance for all shared config. Do not
crates/gc-engine/src/level.rs
+24 -25
View File
@@ -211,52 +211,51 @@ pub fn vertex_normal(g: &TfragGeometry, vinfo: usize, table: Option<&NormalTable
[n[0], n[2], -n[1]]
}
/// A vertex's colour after the lighting pass.
/// A vertex's colour after the lighting pass, and whether it was lit at all.
///
/// Falls back to the tfrag's stored colour when there is no light bank or no
/// normal table. That is a real difference from the game, which has both, so
/// the renderer reports how many patches were actually lit rather than
/// quietly showing an unlit level as if it were a lit one.
/// Lighting runs over *positions*, not vertex-info entries: a common entry and
/// a LOD entry can share a position, and the light record is indexed by
/// position, so every entry resolves through the same lookup.
///
/// Takes no distance. The directional pass does not depend on it; the point
/// lights, which are not implemented yet, will.
///
/// Falls back to the tfrag's stored colour when there is no usable light bank.
/// That is a real difference from the game, so the caller reports how many
/// vertices were actually lit rather than quietly showing stored colours as
/// though they were lit.
fn lit_color(
g: &TfragGeometry,
vinfo: usize,
stored: [f32; 4],
lighting: Option<(&NormalTable, &LightBank)>,
depth: f32,
lod_distances: [f32; 3],
) -> ([f32; 4], bool) {
let _ = (depth, lod_distances);
// Lighting runs over *positions*, not vertex-info entries: a common entry
// and a LOD entry can share a position, and the light record is indexed by
// position. So every entry resolves through the same lookup.
let Some((table, bank)) = lighting else {
let Some((table, bank)) = lighting.filter(|(_, b)| !b.sets.is_empty()) else {
return (stored, false);
};
if bank.sets.is_empty() {
return (stored, false);
}
(lit_at(g, vinfo, table, bank), true)
}
fn lit_at(g: &TfragGeometry, vinfo: usize, table: &NormalTable, bank: &LightBank) -> [f32; 4] {
let i = g.position_index(vinfo);
let Some(light) = g.lights.get(i) else {
return [1.0, 1.0, 1.0, 1.0];
return ([1.0, 1.0, 1.0, 1.0], true);
};
let n = crate::tfrag_light::vertex_normal_bits(
table,
light.azimuth_index(),
light.elevation_index(),
);
let Some(c) = crate::tfrag_light::light_vertex(bank, n, light) else {
return [1.0, 1.0, 1.0, 1.0];
};
match crate::tfrag_light::light_vertex(bank, n, light) {
Some(c) => (
[
c[0] as f32 / 255.0,
c[1] as f32 / 255.0,
c[2] as f32 / 255.0,
1.0,
]
],
true,
),
// The light set this vertex names does not exist. Count it as unlit
// rather than inventing a colour.
None => (stored, false),
}
}
/// The stored colour of a vertex's position, or opaque white if absent.
@@ -441,7 +440,7 @@ pub fn build_level(src: &LevelSource<'_>, camera: [f32; 3]) -> anyhow::Result<Le
let r = resolve_vertex(&g, v, depth, lod_distances, collapse);
batch.positions.push(r.position);
batch.uvs.push(r.uv);
let (colour, was_lit) = lit_color(&g, v, r.color, lighting, depth, lod_distances);
let (colour, was_lit) = lit_color(&g, v, r.color, lighting);
if was_lit {
stats.lit_vertices += 1;
}
docs/formats/level-wad.md
+9 -7
View File
@@ -117,13 +117,15 @@ not yet identified.
## Not yet parsed
- **`data`** — the geometry. This is the next target and where the visible
level comes from: tfrag terrain, tie instances, shrub foliage, moby classes,
sky shells and textures.
- **`gameplay`** — begins `WAD` + a per-level id byte + a count, and behaves
like an LZ77 stream but does **not** decode with the WAD decoder (all 27
levels fail with "adjacent literal packets", which the format forbids). It is
a different compression, or a different framing. Recorded in
- **`data`** — the geometry. Parsed through
[`level-data.md`](level-data.md); `core_index`, `gs_ram` and `core_data` are
read, and `core_data` decompresses.
- **`gameplay`** — a WAD LZ77 stream, exactly like `core_data`, and it
decompresses. The magic is `WADy\r\n`: byte 3 is `0x79` on level 0 rather
than `0x0d`, which reads as a per-level variation but is just part of the
32-bit little-endian stream length. Decompressed it holds the level's light
bank — 5 directional light sets on level 0. The earlier reading of "not a
WAD" came from parsing the *compressed* bytes; see
[`../plan/level-assets.md`](../plan/level-assets.md).
- **`sound_bank`** — begins with `SBlk` at `+24` and `DAW` at `+36`; both
signatures are present in all 27 levels.
docs/plan/level-assets.md
+46 -57
View File
@@ -49,57 +49,25 @@ Two facts worth carrying forward:
table where they are byte counts. Both are on the same disc.
- **`LEVEL21.WAD` reports level id 30.** Container order is not level identity.
## `gameplay`: a compression we have not identified
## `gameplay`: solved, and the old answer was wrong
This is the one genuinely puzzling result so far, and it is worth recording in
full because the next attempt will otherwise repeat the work.
**It is a WAD LZ77 stream.** It decompresses with the decoder in `wad.rs` like
`core_data` does, and the decompressed extent holds the level's light bank.
The extent begins:
This section used to be the longest open question here, and it was wrong. The
reasoning was sound and the conclusion was not: byte 3 of the magic is `0x79`
on level 0 rather than the `0x0d` the WAD docs suggest, which reads as a
per-level variation; but byte 3 is part of the 32-bit little-endian stream
length, and `0x0a0d79` is simply level 0's compressed length. An earlier
attempt decompressed "starting after an 8-byte header" and read the resulting
garbage as evidence of a different format.
```text
offset 0 57 41 44 "WAD"
offset 3 79 0d 0a 0d <- varies per level
offset 4+ <payload>
```
The failure mode is the one worth keeping: a uniform failure on all 27 inputs
was taken to mean the decoder was wrong, when it meant the *data* was being
handed over uncompressed. A positive control -- decompress it and look at the
result -- was never run, and it would have answered the question in one line.
The variation is the tell. For the LZ77 WAD container the fourth byte is part of
a 32-bit little-endian length; here byte 3 differs per level (0x79, 0x3a, 0xd8,
0xa9, ...) while byte 4 is a small count (13, 88, 13, 105, 163, ...) and bytes 6
and 7 are always zero.
**Attempt: treat it as the known WAD LZ77 stream.** Decoded all 27 levels with
the decoder in `wad.rs`, starting after an 8-byte header. Every one produces
plausible output for the first packet — `0000 0000 0000 0750 0100 0020 2b0c...`
looks like real level data — and then fails on all 27 with "adjacent literal
packets", which the format's encoder never emits.
A uniform failure on all 27 inputs means the input is wrong, not the decoder.
Ruled out:
- **Bit order.** The WAD container is a flag-byte packet stream, not a bit
stream, so no ordering applies. Both orderings fail identically.
- **Header length.** Tried 8 bytes. A different skip would not fix a failure
that occurs uniformly in the first few packets.
- **The WAD format at all.** The first packet (flag `0x00`) decodes cleanly as a
literal run of 18 bytes, which is *why* it looks plausible, but the run
immediately after breaks the format's own rule. Whatever this is, it is not
WAD LZ77.
Next to try, in order of cheapness:
1. Whether the payload is uncompressed and the header is a table. `(len - 8)`
is 196..497 across the levels and divides by neither 8 nor a plausible stride,
which argues against a flat pair table — but a *variable-length* table
(counts of counts) fits that range.
2. Whether byte 4 is an entry count and bytes 5.. are records of a fixed but
non-obvious width.
3. Whether the LZ77 variant differs only in the "adjacent literals" rule.
None of these is expensive to test; each is a few lines against all 27 levels at
once. The right move is to write a throwaway script that tries many hypotheses
across all 27 and reports which produce self-consistent results — the approach
that worked for the container pair table, where "does this hold on every level"
was the discriminator that the sector/byte ambiguity failed.
See "The light bank was there the whole time" below.
## `sound_bank`: signatures found, structure unknown
@@ -170,11 +138,20 @@ Earlier claims in this file that have since been **disproven**:
Tested: every stride from 2 to 16 bytes, every starting offset, three and four
components. Best result 5.0% of points on-sphere, which is chance.
So the record framing is **not recovered**. What is established is the region
layout and that terrain is vertex-soup packed alongside inline texels — which
means the vertex format must be identified from a longer run than a single
record, most likely by locating a run of vertices whose magnitudes are bounded
by the planet radius and whose Y values vary smoothly.
So the record framing is **not recovered** — but that framing was a false lead,
and the terrain has since been parsed in full. This section records the wrong
turn and is kept only so it is not repeated.
The region layout below is right and terrain really is packed alongside inline
texels, but the "64-byte record" idea was the wrong way in. Tfrags are not a
flat vertex table: they are VIF1 command lists whose unpacks are classified by
format *and* destination VU address, and the vertices come out of a V3_16
unpack spread across three lists per patch. See "tfrag terrain" below for the
framing that works and the four wrong turns along the way.
The lesson is the recurring one in this file: a hypothesis that produces
plausible-looking output is not thereby recovered, and a "ruled out" entry is
only as good as the control it was tested against.
## Where the class tables are not
@@ -625,10 +602,22 @@ the fidelity rule.
### Still open
The lighting above, the GS register values the level load builds from the
texture entry (TBP0/TBP1 and so on), and the guard-band clip test that sends
patches crossing the guard band to the clipping program.
- **Point lights.** The directional pass runs; `point_light_vertex` and its
`r^2 - |p - L|^2` cull test are not written, and they are the one part of
the lighting that *is* distance-dependent.
- **The GS register set** the level load builds from the texture entry
(TBP0/TBP1 and so on), for a renderer that emulates the GS rather than
resolving texture addresses on the host.
- **The guard-band clip test** that sends patches crossing the guard band to
the clipping program.
- **Lighting magnitudes.** The pass runs and the output is plausible; whether
it matches the game needs comparison against reference frames, which needs
the PCSX2 loop that is still to come. Nothing in this file establishes it.
- **The `core_index` class tables**, still unread after every field order and
stride tried. tfrag, texture and lighting data came from identifying each
structure on its own terms instead.
A tfrag can now be turned into positioned, UV-mapped, textured triangles,
A tfrag can now be turned into positioned, UV-mapped, textured, lit triangles,
told which LOD to draw, and morphed toward its parents at the right rate.
Everything needed for a first frame is parsed and verified.
Everything needed for a first frame is parsed, and the frame is verified by
screenshot rather than by assertion.
docs/roadmap.md
+4 -2
View File
@@ -146,8 +146,10 @@ Phase 1 hands over the extents; this phase works out what each one contains.
and decompressed sizes of `core_data` sit at the same offsets in all 27
levels), and every offset lands inside `core_data`. But the class-table
fields do not decode under any field order or stride tried. Ruled out and
written up in `plan/level-assets.md`. Next: identify `tfrag` by its own
structure instead of through the header.
written up in `plan/level-assets.md`. Identifying `tfrag` by its own
structure instead, which is what the note suggested, worked: tfrag,
texture and lighting data are all parsed. The class tables themselves
remain unread.
- [x] `occlusion` and `sky` offsets located and their first bytes read.
- [x] Region layout of `core_data` established: tfrag, occlusion, sky, collision
and `textures_base` are offsets, strictly increasing on all 27 levels.