Two pieces, and the first is the reason the second can be trusted. ps2.rs is the EE's FPU on raw u32 bit patterns. It is not f32: there are no denormals (exponent 0 *is* zero), no NaN and no infinity (exponent 255 is an ordinary value, so MAX reads back in Rust as NaN -- a test asserts exactly that, because it is the sharpest available statement of why this module exists), every result truncates toward zero, the adder discards the small operand's bits more than one below the large one's LSB and returns the large operand unchanged across a 25-bit exponent gap, and rsqrt is sqrt-then-divide with truncation between the steps rather than one rounded operation. The lighting pass accumulates all of these across every vertex of every patch, so this is a visible difference rather than a rounding curiosity. tfrag_light.rs is the pass itself: PEXT5 base colour, the select word picking one light set or blending two by bits 8..15 / 256, blended directions re-normalised where unblended ones are not, and base + A.rgb*max(dA, dA*wA) + B.rgb*max(dB, dB*wB). Lighting indexes positions, not vertex-info entries, since a common entry and a LOD entry can share a position. TfragLight gains a `select` field for the u16 at +6. It was briefly reconstructed with an unsafe pointer cast over the struct, which is exactly the sort of thing that breaks silently the moment a field is reordered; it is now a plainly parsed field. The pass does not run yet. The light bank framing the reference uses -- a u32 offset at +0x04, an i32 count there -- gives offset 0xa0d and a count of 698810368 on RAC2 level 0, so RAC2's gameplay extent is not laid out that way. The renderer now counts the vertices it actually lit and prints it, because falling back silently to the stored colours produces a plausible but wrong picture: lighting: 0 of 80274 vertices shaded by LightTfrags; the rest fall back to the tfrag's stored colours (no usable light bank) That line is the current honest state, and the frame has not changed as a result. Where RAC2 keeps its light bank is the open question.
635 lines
30 KiB
Markdown
635 lines
30 KiB
Markdown
# Level assets
|
||
|
||
Phase 2 work record. What has been established about the four extents inside
|
||
each `LEVEL<n>.WAD`, and what remains.
|
||
|
||
## A bug found and fixed
|
||
|
||
Reading the `data` extent's contents exposed a defect in the Phase 1 extractor.
|
||
|
||
Level extents are declared in **sector counts**; the generic `G/` pair table
|
||
uses **byte counts**. The extractor routed level containers through the generic
|
||
path, so `LEVEL0.WAD`'s `data` extent — 6,484 sectors — was taken as 6,484
|
||
*bytes*. Level 0 yielded a 6,484-byte "geometry" file instead of 13,279,232
|
||
bytes. Across 27 levels that silently discarded roughly 300 MB of level content
|
||
while every test still passed.
|
||
|
||
The tell was arithmetic, not a crash: the extent's own header declared an
|
||
uncompressed total of 13,277,920 bytes, and the ratio against the 6,484-byte
|
||
file was 2047.7× — one sector, to within rounding. A ratio that lands on 2048
|
||
is a unit error.
|
||
|
||
Fixed by extracting level containers through the typed `LevelHeader` instead of
|
||
the generic table. Level 0 now yields:
|
||
|
||
```text
|
||
data.bin 13,279,232 the level container's data extent
|
||
core_data.wad 9,738,678 the geometry, compressed
|
||
core_data.bin 16,969,296 the geometry, decompressed
|
||
core_index.bin 31,600 the geometry directory
|
||
gameplay.bin 659,456
|
||
sound_bank.bin 1,503,232
|
||
```
|
||
|
||
**Lesson:** a unit error is invisible to a type system and to a passing test. The
|
||
check that catches it is arithmetic against an independent number the file
|
||
itself declares. Any extent whose contents describe a larger structure should
|
||
have that structure's own total compared against the extent.
|
||
|
||
## Done: the container header
|
||
|
||
`crates/gc-formats/src/level_wad.rs` parses the 0x60-byte header on all 27
|
||
levels and names the four extents. See
|
||
[`../formats/level-wad.md`](../formats/level-wad.md) for the layout and the
|
||
measured invariants.
|
||
|
||
Two facts worth carrying forward:
|
||
|
||
- **Sizes in this header are sector counts**, unlike the generic `G/*.WAD` pair
|
||
table where they are byte counts. Both are on the same disc.
|
||
- **`LEVEL21.WAD` reports level id 30.** Container order is not level identity.
|
||
|
||
## `gameplay`: a compression we have not identified
|
||
|
||
This is the one genuinely puzzling result so far, and it is worth recording in
|
||
full because the next attempt will otherwise repeat the work.
|
||
|
||
The extent begins:
|
||
|
||
```text
|
||
offset 0 57 41 44 "WAD"
|
||
offset 3 79 0d 0a 0d <- varies per level
|
||
offset 4+ <payload>
|
||
```
|
||
|
||
The variation is the tell. For the LZ77 WAD container the fourth byte is part of
|
||
a 32-bit little-endian length; here byte 3 differs per level (0x79, 0x3a, 0xd8,
|
||
0xa9, ...) while byte 4 is a small count (13, 88, 13, 105, 163, ...) and bytes 6
|
||
and 7 are always zero.
|
||
|
||
**Attempt: treat it as the known WAD LZ77 stream.** Decoded all 27 levels with
|
||
the decoder in `wad.rs`, starting after an 8-byte header. Every one produces
|
||
plausible output for the first packet — `0000 0000 0000 0750 0100 0020 2b0c...`
|
||
looks like real level data — and then fails on all 27 with "adjacent literal
|
||
packets", which the format's encoder never emits.
|
||
|
||
A uniform failure on all 27 inputs means the input is wrong, not the decoder.
|
||
Ruled out:
|
||
|
||
- **Bit order.** The WAD container is a flag-byte packet stream, not a bit
|
||
stream, so no ordering applies. Both orderings fail identically.
|
||
- **Header length.** Tried 8 bytes. A different skip would not fix a failure
|
||
that occurs uniformly in the first few packets.
|
||
- **The WAD format at all.** The first packet (flag `0x00`) decodes cleanly as a
|
||
literal run of 18 bytes, which is *why* it looks plausible, but the run
|
||
immediately after breaks the format's own rule. Whatever this is, it is not
|
||
WAD LZ77.
|
||
|
||
Next to try, in order of cheapness:
|
||
|
||
1. Whether the payload is uncompressed and the header is a table. `(len - 8)`
|
||
is 196..497 across the levels and divides by neither 8 nor a plausible stride,
|
||
which argues against a flat pair table — but a *variable-length* table
|
||
(counts of counts) fits that range.
|
||
2. Whether byte 4 is an entry count and bytes 5.. are records of a fixed but
|
||
non-obvious width.
|
||
3. Whether the LZ77 variant differs only in the "adjacent literals" rule.
|
||
|
||
None of these is expensive to test; each is a few lines against all 27 levels at
|
||
once. The right move is to write a throwaway script that tries many hypotheses
|
||
across all 27 and reports which produce self-consistent results — the approach
|
||
that worked for the container pair table, where "does this hold on every level"
|
||
was the discriminator that the sector/byte ambiguity failed.
|
||
|
||
## `sound_bank`: signatures found, structure unknown
|
||
|
||
```text
|
||
+0x18 "SBlk" present in all 27 levels
|
||
+0x24 "DAW" present in all 27 levels
|
||
```
|
||
|
||
Sizes 458..833 bytes. Too small to be the level's audio; likely an index into
|
||
the global `AUDIO.WAD`.
|
||
|
||
## `occlusion`: header located, contents a u16 array
|
||
|
||
The header's `occlusion` offset (0x0c) is trustworthy — every offset in it lands
|
||
inside `core_data`. Level 0's occlusion block starts at 2,117,888 and reads as a
|
||
`u16` array:
|
||
|
||
```text
|
||
5152 0 12 23 14 19 24 29 32 35 38 41 44 47 50 53 56 59 65 82 96 100 113 126 143 164 183 ...
|
||
```
|
||
|
||
An ascending run of 65 values with mostly small differences, then it restarts.
|
||
That is consistent with per-cell visibility records rather than geometry. The
|
||
meaning of the pairs — and the 0 that ends each group — is not yet established.
|
||
|
||
## `sky`: plausible, unparsed
|
||
|
||
Header offset 0x10 = 2,192,576. Opens `0 0 0 0 0 0 6 0 0 0 0 0 5 0 0 0 40 0 0 0 c0 0 0 0 40 ...`
|
||
— small values including 64 and 192, which look like sky-shell counts and
|
||
dimensions rather than offsets.
|
||
|
||
## `tfrag` region: `[0, occlusion)`
|
||
|
||
The header's fields at 0x0c, 0x10, 0x14 and 0x60 are **offsets**, and they are
|
||
strictly increasing on all 27 levels. The differences between them give region
|
||
sizes that are plausible on every level:
|
||
|
||
| Region | Range | Level 0 size |
|
||
|---|---|---|
|
||
| tfrag | `[0, occlusion)` | 2.12 MB |
|
||
| occlusion | `[occlusion, sky)` | 0.07 MB |
|
||
| sky | `[sky, collision)` | 0.42 MB |
|
||
| collision | `[collision, textures_base)` | 0.90 MB |
|
||
|
||
Ruled out: reading 0x0c/0x10/0x14 as *sizes* laid end to end. The cumulative
|
||
total does not reach `textures_base` on any level, and the byte at the start of
|
||
each supposed region is not a plausible structure. The offsets model is the one
|
||
that holds everywhere.
|
||
|
||
### What is in the terrain region
|
||
|
||
**Textures are stored inline with the geometry**, not in a separate bank. A scan
|
||
for the PS2 texel pattern `18 18 18 80` (grey texels with the 0x80 alpha that
|
||
means "opaque" in the GS's convention) finds 6,739 occurrences spread across 138
|
||
clusters, the first at `0xd9f8` — inside the tfrag region. So terrain carries its
|
||
own texels, and `textures_base` names something else.
|
||
|
||
Earlier claims in this file that have since been **disproven**:
|
||
|
||
- ~~"The terrain is 382 records of 64 bytes."~~ The 0x48 byte at offset +3 is
|
||
simply the most common byte in the region (945 of 2000 sampled slots); it is
|
||
not a record marker. Over 33,091 slots at that position only 4,127 hold 0x48.
|
||
- ~~"Byte +3 is constant at 0x48 across 300 records."~~ True for the first 382
|
||
records and false beyond, which is how the false pattern was found.
|
||
- ~~"Coordinates are fixed-point s16 with a scale of 8."~~ The scale-8 reading
|
||
is supported by one coincidence (17,496 / 8 = 2,187, the planet radius) but no
|
||
grouping of the record into triples yields points on a sphere of that radius.
|
||
Tested: every stride from 2 to 16 bytes, every starting offset, three and four
|
||
components. Best result 5.0% of points on-sphere, which is chance.
|
||
|
||
So the record framing is **not recovered**. What is established is the region
|
||
layout and that terrain is vertex-soup packed alongside inline texels — which
|
||
means the vertex format must be identified from a longer run than a single
|
||
record, most likely by locating a run of vertices whose magnitudes are bounded
|
||
by the planet radius and whose Y values vary smoothly.
|
||
|
||
## Where the class tables are not
|
||
|
||
The header's `moby_classes` and friends do not decode. Read as
|
||
`(offset, count)` they give plausible numbers, but the entries at those offsets
|
||
are not valid records. Ruled out:
|
||
|
||
- **Field order.** Both `(offset, count)` and `(count, offset)` produce garbage.
|
||
- **Stride.** Tried 8, 16 and 32 bytes per entry.
|
||
- **Which file the offsets index.** Tried `core_data` and `core_index`.
|
||
- **Header base.** Not the issue: the compressed size (0x88) and decompressed
|
||
size (0x8c) of `core_data` appear at the *same offsets* in all 27 levels'
|
||
`core_index`, which pins the header base at 0 with certainty.
|
||
|
||
One observation worth keeping: every second value in the class-table fields is
|
||
exactly divisible by 16, and they increase monotonically across the nine tables.
|
||
That is consistent with `count * 16` sizes, but the corresponding offsets
|
||
cluster at the very start of the file and do not yield valid records either.
|
||
|
||
The discriminator used to reject candidates is worth stating, because weaker
|
||
tests produce false positives: a real class table's `offset_in_asset_wad`
|
||
values increase. A search for "32-byte records with increasing offsets" returns
|
||
several hundred candidates across the file, and the top one turned out to be a
|
||
texture table whose `o_class` field merely happened to equal `offset + 16`.
|
||
Scoring on plausibility alone is not enough; the entries have to actually chain.
|
||
|
||
Next: rather than guess the header further, identify `tfrag` by its own
|
||
structure. Terrain is the largest subsystem and the one that decides whether
|
||
anything draws, so it is the right next target even though it is the harder one.
|
||
|
||
*(Done — see "tfrag terrain" below. Identifying tfrag by its own structure
|
||
worked where the class-table guess did not.)*
|
||
|
||
## `data`: the real target, untouched
|
||
|
||
6,484 sectors on level 0, down to 2,102 on level 24 — 4.2 to 21.9 MB. This is
|
||
the level's geometry and the largest thing in the container. Next target.
|
||
|
||
What is known from the community reference and from Insomniac's engine family:
|
||
it splits into four geometry subsystems — `tfrag` (terrain fragments),
|
||
`tie` (instanced environment structures), `shrub` (foliage) and `moby` (dynamic
|
||
actors) — plus sky shells, collision data and textures. ReRAC's documentation of
|
||
the equivalent structures in the first game is the closest reference available,
|
||
though R&C2's spherical worlds change how they are used.
|
||
|
||
## Order of work
|
||
|
||
1. `gameplay` — smallest, fully bounded, and the format is identified enough to
|
||
make progress. A win here validates the approach before the large effort.
|
||
2. `occlusion` — tiny.
|
||
3. `sound_bank` — small.
|
||
4. `data` — the large one, and the one that decides whether anything renders.
|
||
|
||
Doing the small extents first is not just convenience: each one is a contained
|
||
reverse-engineering problem that either resolves or tells us something, whereas
|
||
`data` will keep producing ambiguous readings until we have something to check
|
||
them against.
|
||
|
||
## Method note
|
||
|
||
Every format question this phase has been settled the same way: write the
|
||
hypothesis, test it against **all 27 levels at once**, and require an invariant
|
||
that holds everywhere. The sector/byte ambiguity in Phase 1 survived because it
|
||
happened to be correct on the one container checked by hand. The level header
|
||
was fixed in a single pass because the tiling check failed immediately on
|
||
levels 1, 2, 4, 7, 8, 11, 19 and 20.
|
||
## tfrag terrain: block structure and vertex positions
|
||
|
||
Settled. `gc-formats::tfrag` parses the block, and a gated test decodes every
|
||
patch on all 27 levels.
|
||
|
||
### Block and patch framing
|
||
|
||
The tfrags block opens with a 12-byte header: `table_offset` (always `0x40`),
|
||
`tfrag_count`, and an `f32` LOD distance (`32.0`). The patch table follows at
|
||
`table_offset`, `tfrag_count` records of `0x40` bytes each. On level 0 that is
|
||
833 patches.
|
||
|
||
A patch header's `data` offset is **relative to `table_offset`**, not to the
|
||
start of the block. Getting this wrong happens to work for the first patch and
|
||
silently produces garbage afterwards.
|
||
|
||
### Positions live in three lists, not one
|
||
|
||
`vert_count` is the *total* across all three position tiers. Within a patch's
|
||
data block the VIF lists do not run in ascending offset order — the LOD-2 list
|
||
sits *before* the common list — and the LOD-0-only list has no offset of its
|
||
own, starting instead at `shared_ofs + lod_1_size * 16`. The ranges, in transfer
|
||
order, are:
|
||
|
||
| List | Range (from the patch data block) | Holds |
|
||
|---|---|---|
|
||
| common | `shared_ofs .. lod_1_ofs` | VU header, ad-gifs, vertex info, origin, positions |
|
||
| LOD 2 | `lod_2_ofs .. shared_ofs` | strips, indices |
|
||
| LOD 1 | `lod_1_ofs .. lod_0_ofs` | strips, indices |
|
||
| LOD 0+1 | `lod_0_ofs .. shared_ofs + lod_1_size*16` | parents, vertex info, positions |
|
||
| LOD 0 | `shared_ofs + lod_1_size*16 .. rgba_ofs` | positions, strips, indices, parents |
|
||
|
||
Scanning only the common list and expecting `num == vert_count` finds positions
|
||
in roughly a quarter of patches. The other three quarters keep most of their
|
||
positions in the LOD lists. That was the first wrong turn and it looked
|
||
plausible: every patch it did match matched *exactly*.
|
||
|
||
### The VIF1 code word
|
||
|
||
Two rules had to be right before any of this worked, and both were wrong on the
|
||
first attempt.
|
||
|
||
**The vector format lives in the command byte, not the immediate.** Bits 2-3 of
|
||
the command are `vn` (component count minus one) and bits 0-1 are `vl` (0 =
|
||
32-bit, 1 = 16-bit, 2 = 8-bit, 3 = 5-bit), so
|
||
|
||
element_size = ((32 >> vl) * (vn + 1)) / 8
|
||
|
||
Terrain positions are `vn=2, vl=1` — command `0x69` — at 6 bytes each. Reading
|
||
the format out of the immediate's low nibble, which is the more commonly quoted
|
||
layout, yields 12 or 16 bytes and quietly misaligns everything after it.
|
||
|
||
**Unpack payloads pad to 4 bytes, and several non-unpack commands carry inline
|
||
data.** An unpack advances `(count * element_size + 3) & !3`. `STROW`/`STCOL`
|
||
carry 16 bytes, `STMASK` 4, `MPG` `num * 8`, `DIRECT`/`DIRECTHL` `imm * 16`;
|
||
every other non-unpack is a bare 4-byte code word. Ignoring the first four
|
||
desynchronises the walk, and a desynchronised walk still finds code words that
|
||
look like positions — it just finds the wrong number of them.
|
||
|
||
The 4-byte padding is easy to misread. With 75 elements the payload is 450
|
||
bytes and the following word lands at +452, where `0x01000404` (STCYCL) reads
|
||
perfectly plausibly. The word at +450 is `0x04040000` (ITOP), which is the real
|
||
one. Taking the plausible-looking reading is how this was briefly wrong.
|
||
|
||
### Verification
|
||
|
||
Requiring every patch to yield exactly its own `vert_count` is the invariant
|
||
that caught both errors, and it is the check the parser is built around: a
|
||
mismatch means the walk desynchronised, so `tfrag_positions` returns `None`
|
||
rather than partial geometry. Across all 27 levels: 18,102 patches,
|
||
821,439 vertices, zero mismatches.
|
||
|
||
### Topology: strips, indices and triangles
|
||
|
||
Also settled. `parse_tfrag` walks each list and classifies every unpack by
|
||
format *and* destination VU address, using the VU header at the head of the
|
||
common list (5 V4_16 vectors, unsigned, to address 0). That table is what
|
||
separates the strips from the indices from the parent arrays — they are all
|
||
V4_8, and only the address tells them apart.
|
||
|
||
| `(vn, vl)` | Meaning |
|
||
|---|---|
|
||
| (3, 1), unsigned, addr 0 | the VU header itself |
|
||
| (3, 0) | ad-gifs, 5 quadwords each |
|
||
| (2, 1) | positions, V3_16 |
|
||
| (3, 1) | vertex info: UV, LOD parent, own position |
|
||
| (3, 2) | strips (signed) and indices (unsigned) |
|
||
|
||
**Strip indices address the vertex-info array, not the position array.** Several
|
||
vertex-info entries can share a position, so a strip index resolves through
|
||
`vertex_info[i].vertex / 2` to reach a position.
|
||
|
||
A strip record is four bytes: a vertex count, a packet flag and an ad-gif
|
||
offset. A negative count is biased by −128, and *the first record of a list is
|
||
always biased* even when its count is positive — a real patch opens with −116 or
|
||
−124, never a small positive number. A zero count ends the list. A run of `n`
|
||
vertices is a GS triangle strip of `n − 2` triangles with alternating winding.
|
||
|
||
**Bank the LODs by index, not by arrival.** The lists are walked in transfer
|
||
order but the strip and index arrays have to be stored by *which LOD they
|
||
belong to*. Appending them as they are found fills the arrays LOD-2-first,
|
||
which looks fine until the triangle counts come out low.
|
||
|
||
### Verification
|
||
|
||
`tri_count` is stored in the patch header and cannot be derived from the lists
|
||
without walking them, so reproducing it is a strong check. All 27 levels:
|
||
**18,102 patches, 809,879 triangles, every patch matching its header**, with
|
||
every triangle index inside that LOD's vertex-info range and every ad-gif index
|
||
inside the ad-gif array.
|
||
|
||
Getting there took four wrong turns, each of which failed quietly: reading the
|
||
element size from the immediate, 2-byte payload padding, ignoring non-unpack
|
||
payloads, and banking strips by arrival order. Every one produced parseable,
|
||
plausible geometry; only the header's own triangle count caught them.
|
||
|
||
### Texture bindings, colours and lights
|
||
|
||
Also settled. Adds `level_core.rs` for the index header and the texture table,
|
||
which is the join between tfrags and texel data.
|
||
|
||
A tfrag's **ad-gifs** are five A+D quadwords (0x50 bytes) holding packed
|
||
fields rather than GS registers; the level load rewrites them in place before
|
||
drawing. On disc `TEX0_1.data_lo` indexes the level's texture table,
|
||
`TEX1_1.data_lo` is the LOD bias K and `data_hi` the minification filter, and
|
||
`CLAMP_1` holds the two wrap modes. The texture table is a count/offset pair at
|
||
index offset 0x30, entries of 16 bytes.
|
||
|
||
The table checks out on its own terms: on level 0 it is 93 entries at 0x2900,
|
||
and 0x2900 + 93 x 16 lands exactly on the next range at 0x38.
|
||
|
||
Three details worth writing down:
|
||
|
||
* **K is a signed 12-bit value in 1/16 units**, sign-extended by shifting the
|
||
field up 4 and back down. Retail values are -137..-89, which is -8.56..-5.56
|
||
mip levels. Reading the field as unsigned gives 1911 instead of -137.
|
||
* **`data_hi` of TEX1_1 is MMIN, and 4 is `LinearMipmapNearest`**, not
|
||
`LinearMipmapLinear` as the bit layout alone suggests.
|
||
* **RGBA and the light/normal records are not in the VIF lists.** They sit at
|
||
fixed offsets in the patch's data block: `rgba_size * 4` four-byte colours at
|
||
`rgba_ofs`, and `vert_count` eight-byte records at `light_ofs + 0x10`. The
|
||
origin also lives there — if no STROW row supplied one, the quadword at
|
||
`light_ofs` is the origin.
|
||
|
||
### Verification
|
||
|
||
Every ad-gif must index a texture that exists, or the geometry and the texture
|
||
data have been misread independently of each other. Across all 27 levels:
|
||
**47,950 ad-gifs over 18,102 patches, all indices in range**, referencing
|
||
1429 of the 1595 level texture entries (90%). Each patch's light array is
|
||
exactly `vert_count` records and its RGBA array exactly `rgba_size * 4`.
|
||
|
||
### Texels: 8-bit indexed images and CSM1 palettes
|
||
|
||
Settled. Adds `texture.rs` and makes the pipeline write `gs_ram.bin`.
|
||
|
||
A level texture is not pixels, it is **palette indices**: `width * height`
|
||
bytes at `core_data[textures_base + data_offset]`. The colour comes from a
|
||
256-entry CLUT (1024 bytes) in GS RAM at `palette * 0x100`. `gs_ram` is a byte
|
||
range in the level's data extent header at offset 0x10 — 518,656 bytes on level
|
||
0 — and the extractor was not writing it out, so nothing could be decoded
|
||
before this.
|
||
|
||
Two PS2 conventions have to be honoured, and neither is a design choice:
|
||
|
||
* **CSM1 CLUT order.** Within each 32-entry group the two middle 8-entry
|
||
blocks are swapped: bits 3 and 4 of the index are exchanged. The remap is an
|
||
involution.
|
||
* **Alpha runs 0..=0x80, not 0..=255.** `0x80` is fully opaque and anything
|
||
above it clamps.
|
||
|
||
The mip chain splits across two buffers. Levels 0 and 1 are in the texture
|
||
block, with level 1 immediately after level 0 at `data_offset + w*h`; levels 2
|
||
and 3 are resident in GS RAM at `mipmap * 0x100` and `pad * 0x100`. Tfrag
|
||
textures are square, so every level halves evenly.
|
||
|
||
### Verification
|
||
|
||
Numbers alone do not settle this — every step is a table lookup, so a wrong
|
||
CLUT remap still yields a well-formed image of the wrong thing. Two layers:
|
||
|
||
* **Shape.** All 1,595 tfrag textures across all 27 levels decode; 96.8% are
|
||
fully opaque and 96.3% use more than eight distinct colours. A flat,
|
||
transparent or single-colour result would point at a CLUT or offset mistake.
|
||
* **Eyes.** `gc-extract dump-texture` writes any tfrag texture, or any of its
|
||
mip levels, as a binary PPM. Decoded level 0 texture 0 is a clean bevelled
|
||
metal panel with rivets and specular highlights; texture 74 is a vent grille
|
||
whose punched windows come back fully transparent. Both confirm the CLUT
|
||
remap, the alpha scale and the pixel base at once, which no assertion in the
|
||
suite does.
|
||
|
||
That tool is worth keeping: it is the cheapest way to tell a decode that is
|
||
merely well-formed from one that is right.
|
||
|
||
### LOD selection
|
||
|
||
Mostly settled. The block header's float at +0x08 gives `L`, and the switch
|
||
distances are `[D0, D1, D2] = [6L, 4L, 2L]` world units; on every level `L` is
|
||
32.0, so `[192, 128, 64]`. The distance test compares the patch's bounding
|
||
sphere, truncated to integers, against `trunc(D * 1024)`. Six modes result —
|
||
LOD 2, LOD 1 collapse, LOD 1 morph, LOD 0 collapse, LOD 0 morph, LOD 0 — and
|
||
the *sphere* is what is tested, not the centre, so a patch straddling a
|
||
threshold takes the coarser mode. That is what stops geometry popping as the
|
||
camera moves, and it is worth testing for directly: sweeping the depth must
|
||
never move back to a finer mode.
|
||
|
||
A morph weight runs 0 to 1 across its tier's band: LOD-01 between D1 and D0,
|
||
LOD-0 between D2 and D1. Past the far edge of the band an entry is not morphed
|
||
but *replaced* by its first parent — a collapse, not a blend.
|
||
|
||
### Parent indices: settled, but only after two corrections
|
||
|
||
The four V4_8 arrays (`parent_indices_lod01`, `unk_indices_2_lod01`,
|
||
`parent_indices_lod0`, `unk_indices_2_lod0`) are classified by VU address,
|
||
and **the addresses are not unique**. On level 0 patch 1,
|
||
`unk_indices_2_lod_01_addr == parent_indices_lod_0_addr == 291`, and
|
||
`unk_indices_2_lod_0_addr == indices_addr == 294`. So address matching alone
|
||
cannot separate them, and getting it wrong is silent: the arrays still parse,
|
||
they just hold the wrong numbers.
|
||
|
||
Two things were needed to fix it, and the first attempt at getting it wrong is
|
||
worth recording because it looked like a data problem rather than a code one.
|
||
|
||
**Each V4_8 element is 4 bytes and holds four indices**, so an array of N
|
||
entries is `count * 4` bytes, not `count`. An early analysis compared the
|
||
unpack's element count against the position counts and concluded that 15,000
|
||
patches could not possibly hold enough parents. That was the wrong unit; the
|
||
data was fine.
|
||
|
||
**`unk_06` and `unk_0a` suppress an empty tier.** When the extra vertex-info
|
||
count is zero there is no extra array, and its VU address is simply the next
|
||
tier's. Guarding those two branches on `unk_06 != 0` / `unk_0a != 0` lets the
|
||
address fall through to the array that actually exists. This one change took
|
||
resolved LOD-0 entries from 28,739 to 341,306.
|
||
|
||
### Verification
|
||
|
||
Every vertex-info entry across all 27 levels is now accounted for: of 886,841
|
||
entries, 516,492 are common and have no link by design, and **all 370,349 LOD
|
||
entries resolve** — 29,043 LOD-01 and 341,306 LOD-0 — with zero failures. Each
|
||
resolves to a first parent that is real vertex info, a LOD-01 parent that is
|
||
always a common entry, and three positions inside the patch. A common entry
|
||
never claims a parent.
|
||
|
||
Ruled out along the way: address-only classification (addresses collide), and
|
||
one parent array entry per unpack element (wrong unit).
|
||
|
||
### The origin, and why no parser test could catch it
|
||
|
||
Positions are `i16`, so a single patch spans at most +-32 world units around
|
||
its own origin. The origin comes from an STROW row, and the obvious rule --
|
||
the last non-index STROW in the list -- is **wrong on RAC2**. RAC2 also uses
|
||
STROW for the per-vertex light records, and those rows come *after* the origin,
|
||
so taking the last one yields `[186, 186, 186, 186]` and every patch collapses
|
||
onto the world origin.
|
||
|
||
Nothing in the test suite noticed, and that is the point worth recording:
|
||
|
||
* `vert_count` still matched, because it counts positions, not places.
|
||
* `tri_count` still matched, because the topology never changed.
|
||
* Every parent, morph target and triangle index still resolved, because those
|
||
are all relative to the same broken origin.
|
||
|
||
The entire level fit in a 63 x 62 x 62 unit box centred on the origin, and all
|
||
of it was self-consistent. Only rendering exposed it, and only because the
|
||
screenshot put the camera where the bounding spheres said the terrain was.
|
||
|
||
The rule that works: **the origin is the STROW immediately preceding the first
|
||
V3_16 position unpack** in the common list. With that, level 0's vertices span
|
||
x 86..444, y 46..211, z -477..40, which matches the patch bounding spheres.
|
||
|
||
Lesson for the rest of the project: a parser can be entirely self-consistent
|
||
and still describe the wrong thing. Where a value is absolute rather than
|
||
relative, check it against an independent source -- here, the bounding spheres.
|
||
|
||
### Why the first frame is dark
|
||
|
||
The renderer draws unlit: texture times the tfrag's own per-vertex colour.
|
||
That product is genuinely dark -- level 0 patch 0's colours are `(25, 26, 26)`
|
||
and `(92, 121, 119)` in 0..255 -- and the game's lighting is what brightens it.
|
||
|
||
Trying a host directional light or cranking ambient brightness was the wrong
|
||
instinct and it showed: with no normals the PBR path is degenerate, so
|
||
brightness changes did nothing at all. Faking it would also have hidden the
|
||
gap. The colour attribute is picked up from the mesh automatically, so it is
|
||
the *colour* data that is missing lighting, not a flag that was left off.
|
||
|
||
The real model is `LightTfrags`. The PS2 float model and the pass itself are
|
||
now written (`gc-engine::ps2`, `gc-engine::tfrag_light`); what is missing is the
|
||
light bank. The three inputs were:
|
||
|
||
* the **light records** -- already parsed: azimuth and elevation as *indices*
|
||
into the overlay's sine table, plus the base colour;
|
||
* the **normal table** -- now decoded (`gc_formats::tfrag_light`), and normals
|
||
are built and fed to the renderer;
|
||
* the **light bank** from the level's `gameplay` extent -- **not working**,
|
||
see below.
|
||
|
||
### Why the frame is still dark: the light bank does not parse on RAC2
|
||
|
||
The reference frames the bank as "a u32 offset at +0x04, then an `i32` count
|
||
there, then the sets". On RAC2 level 0 that offset is `0xa0d`, and the `i32`
|
||
there is `698810368` -- not a count, and not a plausible one. `gameplay.bin`
|
||
begins with `0x794e...`, not with a header of the shape the reference expects,
|
||
so RAC2's gameplay extent is not laid out this way, or is compressed before
|
||
this point.
|
||
|
||
The pass therefore never runs. Rather than let the renderer quietly draw the
|
||
stored colours and look plausible, it counts what it lit and says so:
|
||
|
||
```text
|
||
lighting: 0 of 80274 vertices shaded by LightTfrags; the rest fall back to the
|
||
tfrag's stored colours (no usable light bank)
|
||
```
|
||
|
||
That line is the honest state, and it is why the frame has not changed. What
|
||
`gameplay.bin` actually is, and where RAC2 keeps its light bank, is the open
|
||
question — `0x794e...` looks like it may be a compressed or encrypted
|
||
container, in which case the bank is somewhere else entirely.
|
||
|
||
### The PS2 float model
|
||
|
||
`gc-engine::ps2` is the EE's FPU on raw `u32` bit patterns, and it is not
|
||
`f32`:
|
||
|
||
* **no denormals** -- exponent 0 *is* zero, whatever the mantissa
|
||
* **no NaN, no infinity** -- exponent 255 is an ordinary value, which means
|
||
`MAX` reads back in Rust as NaN. A test asserts exactly that, because it is
|
||
the clearest single statement of why this module exists.
|
||
* **every result truncates toward zero**, and `1/3` comes out below `1/3`
|
||
* **the adder aligns operands**: the smaller operand loses the bits more than
|
||
one below the larger's LSB, and a 25-bit exponent gap returns the larger
|
||
operand unchanged
|
||
* **`rsqrt` is sqrt-then-divide**, with truncation between, not one rounded step
|
||
|
||
`mul`, `add`, `div`, `sqrt`, `itof12` and `max`/`min` all differ from `f32` in
|
||
at least one case a lighting pass will hit. The pass accumulates them across
|
||
every vertex of every patch, so this is a visible difference rather than a
|
||
rounding curiosity.
|
||
|
||
### The lighting pass itself
|
||
|
||
Per vertex: `base + A.rgb * max(dA, dA*wA) + B.rgb * max(dB, dB*wB)`, where
|
||
`base` is the record's 5:5:5:1 colour expanded by PEXT5, `dA`/`dB` are
|
||
three-component dot products against the light directions, and `wA`/`wB` come
|
||
out of the colour w lanes. `select` picks one light set or blends two by
|
||
`bits 8..15 / 256`, and a blended direction is re-normalised where an unblended
|
||
one is not.
|
||
|
||
One field had to be added to `TfragLight` to carry this: `select`, the `u16` at
|
||
+6 of the record. It was briefly reconstructed with an `unsafe` pointer cast
|
||
over the struct -- which is exactly the kind of thing that breaks silently the
|
||
moment a field is reordered, and is now a plain parsed field.
|
||
|
||
### The normal table, and why it is found by searching
|
||
|
||
The reference reads the table from an ELF at `BOOT_NORMAL_TABLE_VADDR`
|
||
(`0x165500`) and notes every level overlay carries a copy. **The overlay has no
|
||
ELF header** -- it is a raw loadable image starting with code -- so there is no
|
||
program header to map a virtual address through, and the offset is not fixed
|
||
either: level 0 has the table at `0xb650c`, level 5 at `0xcca30`, level 26 at
|
||
`0xb6284`.
|
||
|
||
So it is located by content. Every table starts with exactly `(1.0, 0.0)` --
|
||
angle zero -- and the scan then requires all 256 entries to match
|
||
`cos(i·pi/128), sin(i·pi/128)`. Validating the whole curve is what makes the
|
||
search safe: a lone `(1.0, 0.0)` pair of floats occurs by accident in a 2.7 MB
|
||
executable, and a test plants one as a decoy. The values read from the disc are
|
||
used as-is rather than recomputed, since those are the numbers the game indexes.
|
||
|
||
A normal is `(cos(az)cos(el), sin(az)cos(el), sin(el))` **negated** -- the
|
||
stored normals point into the surface -- then given the same Z-up to Y-up swap
|
||
as positions, or the lighting comes out sideways. The azimuth and elevation
|
||
fields are stored signed but index 0..=255, so the byte is what matters, not its
|
||
sign: entry 200 is read as 200, not as -56.
|
||
|
||
Given the table, a normal is `(cos(az)cos(el), sin(az)cos(el), sin(el))`,
|
||
negated, exactly as `decode_normal` builds it. The lighting pass itself is
|
||
`base + A.rgb * max(dA, dA*wA) + B.rgb * max(dB, dB*wB)` per vertex, and it
|
||
must be reproduced in the PS2's float model rather than in Rust `f32`, per
|
||
the fidelity rule.
|
||
|
||
### Still open
|
||
|
||
The lighting above, the GS register values the level load builds from the
|
||
texture entry (TBP0/TBP1 and so on), and the guard-band clip test that sends
|
||
patches crossing the guard band to the clipping program.
|
||
|
||
A tfrag can now be turned into positioned, UV-mapped, textured triangles,
|
||
told which LOD to draw, and morphed toward its parents at the right rate.
|
||
Everything needed for a first frame is parsed and verified.
|