Consolidate: tidy level.rs, correct stale docs, pin hardware models

No behaviour change. This is the pass over the last stretch of work to
make sure the accumulated result hangs together rather than only each
commit in isolation.

Code. lit_color had grown a `let _ = (depth, lod_distances)` and a dead
branch from repeated edits. The directional pass genuinely does not
depend on distance, so the parameters are gone rather than discarded,
with a note that the point lights -- not implemented -- will need them.
lit_at folded into lit_color, which had become a wrapper around it.
Checked for leftover scaffolding: only two env vars remain, both real
(GC_ENGINE_SHOT and GC_ENGINE_DEBUG_BOUNDS); GC_ENGINE_UNLIT,
GC_ENGINE_DEBUG_COLOR and GC_ENGINE_ABL were never actually written,
because those edits silently failed to apply against rustfmt'd source.

AGENTS.md. The crate table did not say that gc-engine owns hardware
models. ps2.rs is the Emotion Engine's FPU and the lighting pass runs
on it, so the boundary is now stated: gc-formats decodes inputs,
gc-engine reproduces how the hardware computes with them. That is the
distinction rule 2 depends on, and it was implicit.

Docs. Three claims had been overtaken by what we learned and were
actively misleading:

* docs/formats/level-wad.md still said gameplay "does not decode with
  the WAD decoder". It does; it is a WAD stream.
* docs/plan/level-assets.md carried a whole section, "a compression we
  have not identified", with a ruled-out list including "bit order",
  "header length" and "the WAD format at all". All three were ruled out
  against the wrong input. Replaced, with the failure mode kept: a
  uniform failure across 27 inputs was read as "the decoder is wrong"
  when it meant "the data was handed over uncompressed", and the
  positive control -- decompress and look -- was never run.
* The same file still said the tfrag "record framing is not recovered".
  True, but it was a false lead that has since been superseded by the
  VIF1 framing. Kept as a recorded wrong turn rather than deleted.

Also refreshed "Still open" to name what is actually outstanding:
point lights, the GS register set, the guard-band clip test, the
unread core_index class tables, and -- stated plainly -- that lighting
magnitudes are unverified because there is still nothing to compare
against.

Verified after the changes: 148 tests pass, clippy and fmt clean, all
5,213 extracted files match the hash table, and level 0 still loads at
26,758 triangles with the same LOD split and fully lit.
This commit is contained in:
2026-10-10 03:37:03 +02:00
parent bcf0d1f0cc
commit b8b4a1a3d3
5 changed files with 96 additions and 97 deletions
+46 -57
View File
@@ -49,57 +49,25 @@ Two facts worth carrying forward:
table where they are byte counts. Both are on the same disc.
- **`LEVEL21.WAD` reports level id 30.** Container order is not level identity.
## `gameplay`: a compression we have not identified
## `gameplay`: solved, and the old answer was wrong
This is the one genuinely puzzling result so far, and it is worth recording in
full because the next attempt will otherwise repeat the work.
**It is a WAD LZ77 stream.** It decompresses with the decoder in `wad.rs` like
`core_data` does, and the decompressed extent holds the level's light bank.
The extent begins:
This section used to be the longest open question here, and it was wrong. The
reasoning was sound and the conclusion was not: byte 3 of the magic is `0x79`
on level 0 rather than the `0x0d` the WAD docs suggest, which reads as a
per-level variation; but byte 3 is part of the 32-bit little-endian stream
length, and `0x0a0d79` is simply level 0's compressed length. An earlier
attempt decompressed "starting after an 8-byte header" and read the resulting
garbage as evidence of a different format.
```text
offset 0 57 41 44 "WAD"
offset 3 79 0d 0a 0d <- varies per level
offset 4+ <payload>
```
The failure mode is the one worth keeping: a uniform failure on all 27 inputs
was taken to mean the decoder was wrong, when it meant the *data* was being
handed over uncompressed. A positive control -- decompress it and look at the
result -- was never run, and it would have answered the question in one line.
The variation is the tell. For the LZ77 WAD container the fourth byte is part of
a 32-bit little-endian length; here byte 3 differs per level (0x79, 0x3a, 0xd8,
0xa9, ...) while byte 4 is a small count (13, 88, 13, 105, 163, ...) and bytes 6
and 7 are always zero.
**Attempt: treat it as the known WAD LZ77 stream.** Decoded all 27 levels with
the decoder in `wad.rs`, starting after an 8-byte header. Every one produces
plausible output for the first packet — `0000 0000 0000 0750 0100 0020 2b0c...`
looks like real level data — and then fails on all 27 with "adjacent literal
packets", which the format's encoder never emits.
A uniform failure on all 27 inputs means the input is wrong, not the decoder.
Ruled out:
- **Bit order.** The WAD container is a flag-byte packet stream, not a bit
stream, so no ordering applies. Both orderings fail identically.
- **Header length.** Tried 8 bytes. A different skip would not fix a failure
that occurs uniformly in the first few packets.
- **The WAD format at all.** The first packet (flag `0x00`) decodes cleanly as a
literal run of 18 bytes, which is *why* it looks plausible, but the run
immediately after breaks the format's own rule. Whatever this is, it is not
WAD LZ77.
Next to try, in order of cheapness:
1. Whether the payload is uncompressed and the header is a table. `(len - 8)`
is 196..497 across the levels and divides by neither 8 nor a plausible stride,
which argues against a flat pair table — but a *variable-length* table
(counts of counts) fits that range.
2. Whether byte 4 is an entry count and bytes 5.. are records of a fixed but
non-obvious width.
3. Whether the LZ77 variant differs only in the "adjacent literals" rule.
None of these is expensive to test; each is a few lines against all 27 levels at
once. The right move is to write a throwaway script that tries many hypotheses
across all 27 and reports which produce self-consistent results — the approach
that worked for the container pair table, where "does this hold on every level"
was the discriminator that the sector/byte ambiguity failed.
See "The light bank was there the whole time" below.
## `sound_bank`: signatures found, structure unknown
@@ -170,11 +138,20 @@ Earlier claims in this file that have since been **disproven**:
Tested: every stride from 2 to 16 bytes, every starting offset, three and four
components. Best result 5.0% of points on-sphere, which is chance.
So the record framing is **not recovered**. What is established is the region
layout and that terrain is vertex-soup packed alongside inline texels — which
means the vertex format must be identified from a longer run than a single
record, most likely by locating a run of vertices whose magnitudes are bounded
by the planet radius and whose Y values vary smoothly.
So the record framing is **not recovered** — but that framing was a false lead,
and the terrain has since been parsed in full. This section records the wrong
turn and is kept only so it is not repeated.
The region layout below is right and terrain really is packed alongside inline
texels, but the "64-byte record" idea was the wrong way in. Tfrags are not a
flat vertex table: they are VIF1 command lists whose unpacks are classified by
format *and* destination VU address, and the vertices come out of a V3_16
unpack spread across three lists per patch. See "tfrag terrain" below for the
framing that works and the four wrong turns along the way.
The lesson is the recurring one in this file: a hypothesis that produces
plausible-looking output is not thereby recovered, and a "ruled out" entry is
only as good as the control it was tested against.
## Where the class tables are not
@@ -625,10 +602,22 @@ the fidelity rule.
### Still open
The lighting above, the GS register values the level load builds from the
texture entry (TBP0/TBP1 and so on), and the guard-band clip test that sends
patches crossing the guard band to the clipping program.
- **Point lights.** The directional pass runs; `point_light_vertex` and its
`r^2 - |p - L|^2` cull test are not written, and they are the one part of
the lighting that *is* distance-dependent.
- **The GS register set** the level load builds from the texture entry
(TBP0/TBP1 and so on), for a renderer that emulates the GS rather than
resolving texture addresses on the host.
- **The guard-band clip test** that sends patches crossing the guard band to
the clipping program.
- **Lighting magnitudes.** The pass runs and the output is plausible; whether
it matches the game needs comparison against reference frames, which needs
the PCSX2 loop that is still to come. Nothing in this file establishes it.
- **The `core_index` class tables**, still unread after every field order and
stride tried. tfrag, texture and lighting data came from identifying each
structure on its own terms instead.
A tfrag can now be turned into positioned, UV-mapped, textured triangles,
A tfrag can now be turned into positioned, UV-mapped, textured, lit triangles,
told which LOD to draw, and morphed toward its parents at the right rate.
Everything needed for a first frame is parsed and verified.
Everything needed for a first frame is parsed, and the frame is verified by
screenshot rather than by assertion.