The anchor 20655c7 said had to be verified before following. It is.
docs/AUG6-RETAIL-ANCHORS.md recovers UpdateCamera_6 and InitCamera_24 as class/role
identifiers in the two dispatch tables, so role 6 of lvl.camvtbl is the camera
update function. The decomp's addresses are in the same space as ours -- its
func_L00_002A3B80 is this overlay's .text dest exactly -- so the claim is checkable
rather than taken on faith.
Three checks, because PROVENANCE.md records that the decomp's function addresses
have failed once here:
1. Role 6 of level 0's lvl.camvtbl names handler 0x004284a8.
2. overlay_bytes_at reads real code there: 0x3c014094, a lui.
3. tools/overlay-anchors decodes the first six words with zero failures:
lui $at, 0x4094
ori $at, $at, 0x7ae1 ; $at = 0x40947ae1
mtc1 $at, $f1 ; a float into an FP register
lw $v0, 0x2290($s4)
lui $at, 0x3f00
mtc1 $at, $f3 ; 0x3f000000 = 0.5f
Two mtc1 into FP registers and a struct field at +0x2290: a function doing
floating-point work on a camera struct. Three agreeing checks are a different kind
of evidence from any one of them, which is the lesson of the moby_anim chase.
Caveat recorded: level 0's lvl.camvtbl is 0xb4 bytes, 13 records and 13 distinct
ids, but several ids read as implausible large values (4,350,904 and similar). The
record layout is not uniformly id/handler, so role 6 being clean may be luck. Worth
remembering when the other roles are read.
Next, unblocked: read all of 0x4284a8 and find the store that writes 0x18aa70.
That confirms m4 by locating the write site rather than inferring it from input
response, and the frustum should be whatever the same function writes nearby.
295 passing, clippy clean.
3028 lines
147 KiB
Markdown
3028 lines
147 KiB
Markdown
# Level assets
|
||
|
||
Phase 2 work record. What has been established about the four extents inside
|
||
each `LEVEL<n>.WAD`, and what remains.
|
||
|
||
## A bug found and fixed
|
||
|
||
Reading the `data` extent's contents exposed a defect in the Phase 1 extractor.
|
||
|
||
Level extents are declared in **sector counts**; the generic `G/` pair table
|
||
uses **byte counts**. The extractor routed level containers through the generic
|
||
path, so `LEVEL0.WAD`'s `data` extent — 6,484 sectors — was taken as 6,484
|
||
*bytes*. Level 0 yielded a 6,484-byte "geometry" file instead of 13,279,232
|
||
bytes. Across 27 levels that silently discarded roughly 300 MB of level content
|
||
while every test still passed.
|
||
|
||
The tell was arithmetic, not a crash: the extent's own header declared an
|
||
uncompressed total of 13,277,920 bytes, and the ratio against the 6,484-byte
|
||
file was 2047.7× — one sector, to within rounding. A ratio that lands on 2048
|
||
is a unit error.
|
||
|
||
Fixed by extracting level containers through the typed `LevelHeader` instead of
|
||
the generic table. Level 0 now yields:
|
||
|
||
```text
|
||
data.bin 13,279,232 the level container's data extent
|
||
core_data.wad 9,738,678 the geometry, compressed
|
||
core_data.bin 16,969,296 the geometry, decompressed
|
||
core_index.bin 31,600 the geometry directory
|
||
gameplay.bin 659,456
|
||
sound_bank.bin 1,503,232
|
||
```
|
||
|
||
**Lesson:** a unit error is invisible to a type system and to a passing test. The
|
||
check that catches it is arithmetic against an independent number the file
|
||
itself declares. Any extent whose contents describe a larger structure should
|
||
have that structure's own total compared against the extent.
|
||
|
||
## Done: the container header
|
||
|
||
`crates/gc-formats/src/level_wad.rs` parses the 0x60-byte header on all 27
|
||
levels and names the four extents. See
|
||
[`../formats/level-wad.md`](../formats/level-wad.md) for the layout and the
|
||
measured invariants.
|
||
|
||
Two facts worth carrying forward:
|
||
|
||
- **Sizes in this header are sector counts**, unlike the generic `G/*.WAD` pair
|
||
table where they are byte counts. Both are on the same disc.
|
||
- **`LEVEL21.WAD` reports level id 30.** Container order is not level identity.
|
||
|
||
## `gameplay`: solved, and the old answer was wrong
|
||
|
||
**It is a WAD LZ77 stream.** It decompresses with the decoder in `wad.rs` like
|
||
`core_data` does, and the decompressed extent holds the level's light bank.
|
||
|
||
This section used to be the longest open question here, and it was wrong. The
|
||
reasoning was sound and the conclusion was not: byte 3 of the magic is `0x79`
|
||
on level 0 rather than the `0x0d` the WAD docs suggest, which reads as a
|
||
per-level variation; but byte 3 is part of the 32-bit little-endian stream
|
||
length, and `0x0a0d79` is simply level 0's compressed length. An earlier
|
||
attempt decompressed "starting after an 8-byte header" and read the resulting
|
||
garbage as evidence of a different format.
|
||
|
||
The failure mode is the one worth keeping: a uniform failure on all 27 inputs
|
||
was taken to mean the decoder was wrong, when it meant the *data* was being
|
||
handed over uncompressed. A positive control -- decompress it and look at the
|
||
result -- was never run, and it would have answered the question in one line.
|
||
|
||
See "The light bank was there the whole time" below.
|
||
|
||
## `sound_bank`: signatures found, structure unknown
|
||
|
||
```text
|
||
+0x18 "SBlk" present in all 27 levels
|
||
+0x24 "DAW" present in all 27 levels
|
||
```
|
||
|
||
Sizes 458..833 bytes. Too small to be the level's audio; likely an index into
|
||
the global `AUDIO.WAD`.
|
||
|
||
## `occlusion`: header located, contents a u16 array
|
||
|
||
The header's `occlusion` offset (0x0c) is trustworthy — every offset in it lands
|
||
inside `core_data`. Level 0's occlusion block starts at 2,117,888 and reads as a
|
||
`u16` array:
|
||
|
||
```text
|
||
5152 0 12 23 14 19 24 29 32 35 38 41 44 47 50 53 56 59 65 82 96 100 113 126 143 164 183 ...
|
||
```
|
||
|
||
An ascending run of 65 values with mostly small differences, then it restarts.
|
||
That is consistent with per-cell visibility records rather than geometry. The
|
||
meaning of the pairs — and the 0 that ends each group — is not yet established.
|
||
|
||
## `sky`: plausible, unparsed
|
||
|
||
Header offset 0x10 = 2,192,576. Opens `0 0 0 0 0 0 6 0 0 0 0 0 5 0 0 0 40 0 0 0 c0 0 0 0 40 ...`
|
||
— small values including 64 and 192, which look like sky-shell counts and
|
||
dimensions rather than offsets.
|
||
|
||
## `tfrag` region: `[0, occlusion)`
|
||
|
||
The header's fields at 0x0c, 0x10, 0x14 and 0x60 are **offsets**, and they are
|
||
strictly increasing on all 27 levels. The differences between them give region
|
||
sizes that are plausible on every level:
|
||
|
||
| Region | Range | Level 0 size |
|
||
|---|---|---|
|
||
| tfrag | `[0, occlusion)` | 2.12 MB |
|
||
| occlusion | `[occlusion, sky)` | 0.07 MB |
|
||
| sky | `[sky, collision)` | 0.42 MB |
|
||
| collision | `[collision, textures_base)` | 0.90 MB |
|
||
|
||
Ruled out: reading 0x0c/0x10/0x14 as *sizes* laid end to end. The cumulative
|
||
total does not reach `textures_base` on any level, and the byte at the start of
|
||
each supposed region is not a plausible structure. The offsets model is the one
|
||
that holds everywhere.
|
||
|
||
### What is in the terrain region
|
||
|
||
**Textures are stored inline with the geometry**, not in a separate bank. A scan
|
||
for the PS2 texel pattern `18 18 18 80` (grey texels with the 0x80 alpha that
|
||
means "opaque" in the GS's convention) finds 6,739 occurrences spread across 138
|
||
clusters, the first at `0xd9f8` — inside the tfrag region. So terrain carries its
|
||
own texels, and `textures_base` names something else.
|
||
|
||
Earlier claims in this file that have since been **disproven**:
|
||
|
||
- ~~"The terrain is 382 records of 64 bytes."~~ The 0x48 byte at offset +3 is
|
||
simply the most common byte in the region (945 of 2000 sampled slots); it is
|
||
not a record marker. Over 33,091 slots at that position only 4,127 hold 0x48.
|
||
- ~~"Byte +3 is constant at 0x48 across 300 records."~~ True for the first 382
|
||
records and false beyond, which is how the false pattern was found.
|
||
- ~~"Coordinates are fixed-point s16 with a scale of 8."~~ The scale-8 reading
|
||
is supported by one coincidence (17,496 / 8 = 2,187, the planet radius) but no
|
||
grouping of the record into triples yields points on a sphere of that radius.
|
||
Tested: every stride from 2 to 16 bytes, every starting offset, three and four
|
||
components. Best result 5.0% of points on-sphere, which is chance.
|
||
|
||
So the record framing is **not recovered** — but that framing was a false lead,
|
||
and the terrain has since been parsed in full. This section records the wrong
|
||
turn and is kept only so it is not repeated.
|
||
|
||
The region layout below is right and terrain really is packed alongside inline
|
||
texels, but the "64-byte record" idea was the wrong way in. Tfrags are not a
|
||
flat vertex table: they are VIF1 command lists whose unpacks are classified by
|
||
format *and* destination VU address, and the vertices come out of a V3_16
|
||
unpack spread across three lists per patch. See "tfrag terrain" below for the
|
||
framing that works and the four wrong turns along the way.
|
||
|
||
The lesson is the recurring one in this file: a hypothesis that produces
|
||
plausible-looking output is not thereby recovered, and a "ruled out" entry is
|
||
only as good as the control it was tested against.
|
||
|
||
## Where the class tables are not
|
||
|
||
The header's `moby_classes` and friends do not decode. Read as
|
||
`(offset, count)` they give plausible numbers, but the entries at those offsets
|
||
are not valid records. Ruled out:
|
||
|
||
- **Field order.** Both `(offset, count)` and `(count, offset)` produce garbage.
|
||
- **Stride.** Tried 8, 16 and 32 bytes per entry.
|
||
- **Which file the offsets index.** Tried `core_data` and `core_index`.
|
||
- **Header base.** Not the issue: the compressed size (0x88) and decompressed
|
||
size (0x8c) of `core_data` appear at the *same offsets* in all 27 levels'
|
||
`core_index`, which pins the header base at 0 with certainty.
|
||
|
||
One observation worth keeping: every second value in the class-table fields is
|
||
exactly divisible by 16, and they increase monotonically across the nine tables.
|
||
That is consistent with `count * 16` sizes, but the corresponding offsets
|
||
cluster at the very start of the file and do not yield valid records either.
|
||
|
||
The discriminator used to reject candidates is worth stating, because weaker
|
||
tests produce false positives: a real class table's `offset_in_asset_wad`
|
||
values increase. A search for "32-byte records with increasing offsets" returns
|
||
several hundred candidates across the file, and the top one turned out to be a
|
||
texture table whose `o_class` field merely happened to equal `offset + 16`.
|
||
Scoring on plausibility alone is not enough; the entries have to actually chain.
|
||
|
||
Next: rather than guess the header further, identify `tfrag` by its own
|
||
structure. Terrain is the largest subsystem and the one that decides whether
|
||
anything draws, so it is the right next target even though it is the harder one.
|
||
|
||
*(Done — see "tfrag terrain" below. Identifying tfrag by its own structure
|
||
worked where the class-table guess did not.)*
|
||
|
||
## `data`: the real target, untouched
|
||
|
||
6,484 sectors on level 0, down to 2,102 on level 24 — 4.2 to 21.9 MB. This is
|
||
the level's geometry and the largest thing in the container. Next target.
|
||
|
||
What is known from the community reference and from Insomniac's engine family:
|
||
it splits into four geometry subsystems — `tfrag` (terrain fragments),
|
||
`tie` (instanced environment structures), `shrub` (foliage) and `moby` (dynamic
|
||
actors) — plus sky shells, collision data and textures. ReRAC's documentation of
|
||
the equivalent structures in the first game is the closest reference available,
|
||
though R&C2's spherical worlds change how they are used.
|
||
|
||
## Order of work
|
||
|
||
1. `gameplay` — smallest, fully bounded, and the format is identified enough to
|
||
make progress. A win here validates the approach before the large effort.
|
||
2. `occlusion` — tiny.
|
||
3. `sound_bank` — small.
|
||
4. `data` — the large one, and the one that decides whether anything renders.
|
||
|
||
Doing the small extents first is not just convenience: each one is a contained
|
||
reverse-engineering problem that either resolves or tells us something, whereas
|
||
`data` will keep producing ambiguous readings until we have something to check
|
||
them against.
|
||
|
||
## Method note
|
||
|
||
Every format question this phase has been settled the same way: write the
|
||
hypothesis, test it against **all 27 levels at once**, and require an invariant
|
||
that holds everywhere. The sector/byte ambiguity in Phase 1 survived because it
|
||
happened to be correct on the one container checked by hand. The level header
|
||
was fixed in a single pass because the tiling check failed immediately on
|
||
levels 1, 2, 4, 7, 8, 11, 19 and 20.
|
||
## tfrag terrain: block structure and vertex positions
|
||
|
||
Settled. `gc-formats::tfrag` parses the block, and a gated test decodes every
|
||
patch on all 27 levels.
|
||
|
||
### Block and patch framing
|
||
|
||
The tfrags block opens with a 12-byte header: `table_offset` (always `0x40`),
|
||
`tfrag_count`, and an `f32` LOD distance (`32.0`). The patch table follows at
|
||
`table_offset`, `tfrag_count` records of `0x40` bytes each. On level 0 that is
|
||
833 patches.
|
||
|
||
A patch header's `data` offset is **relative to `table_offset`**, not to the
|
||
start of the block. Getting this wrong happens to work for the first patch and
|
||
silently produces garbage afterwards.
|
||
|
||
### Positions live in three lists, not one
|
||
|
||
`vert_count` is the *total* across all three position tiers. Within a patch's
|
||
data block the VIF lists do not run in ascending offset order — the LOD-2 list
|
||
sits *before* the common list — and the LOD-0-only list has no offset of its
|
||
own, starting instead at `shared_ofs + lod_1_size * 16`. The ranges, in transfer
|
||
order, are:
|
||
|
||
| List | Range (from the patch data block) | Holds |
|
||
|---|---|---|
|
||
| common | `shared_ofs .. lod_1_ofs` | VU header, ad-gifs, vertex info, origin, positions |
|
||
| LOD 2 | `lod_2_ofs .. shared_ofs` | strips, indices |
|
||
| LOD 1 | `lod_1_ofs .. lod_0_ofs` | strips, indices |
|
||
| LOD 0+1 | `lod_0_ofs .. shared_ofs + lod_1_size*16` | parents, vertex info, positions |
|
||
| LOD 0 | `shared_ofs + lod_1_size*16 .. rgba_ofs` | positions, strips, indices, parents |
|
||
|
||
Scanning only the common list and expecting `num == vert_count` finds positions
|
||
in roughly a quarter of patches. The other three quarters keep most of their
|
||
positions in the LOD lists. That was the first wrong turn and it looked
|
||
plausible: every patch it did match matched *exactly*.
|
||
|
||
### The VIF1 code word
|
||
|
||
Two rules had to be right before any of this worked, and both were wrong on the
|
||
first attempt.
|
||
|
||
**The vector format lives in the command byte, not the immediate.** Bits 2-3 of
|
||
the command are `vn` (component count minus one) and bits 0-1 are `vl` (0 =
|
||
32-bit, 1 = 16-bit, 2 = 8-bit, 3 = 5-bit), so
|
||
|
||
element_size = ((32 >> vl) * (vn + 1)) / 8
|
||
|
||
Terrain positions are `vn=2, vl=1` — command `0x69` — at 6 bytes each. Reading
|
||
the format out of the immediate's low nibble, which is the more commonly quoted
|
||
layout, yields 12 or 16 bytes and quietly misaligns everything after it.
|
||
|
||
**Unpack payloads pad to 4 bytes, and several non-unpack commands carry inline
|
||
data.** An unpack advances `(count * element_size + 3) & !3`. `STROW`/`STCOL`
|
||
carry 16 bytes, `STMASK` 4, `MPG` `num * 8`, `DIRECT`/`DIRECTHL` `imm * 16`;
|
||
every other non-unpack is a bare 4-byte code word. Ignoring the first four
|
||
desynchronises the walk, and a desynchronised walk still finds code words that
|
||
look like positions — it just finds the wrong number of them.
|
||
|
||
The 4-byte padding is easy to misread. With 75 elements the payload is 450
|
||
bytes and the following word lands at +452, where `0x01000404` (STCYCL) reads
|
||
perfectly plausibly. The word at +450 is `0x04040000` (ITOP), which is the real
|
||
one. Taking the plausible-looking reading is how this was briefly wrong.
|
||
|
||
### Verification
|
||
|
||
Requiring every patch to yield exactly its own `vert_count` is the invariant
|
||
that caught both errors, and it is the check the parser is built around: a
|
||
mismatch means the walk desynchronised, so `tfrag_positions` returns `None`
|
||
rather than partial geometry. Across all 27 levels: 18,102 patches,
|
||
821,439 vertices, zero mismatches.
|
||
|
||
### Topology: strips, indices and triangles
|
||
|
||
Also settled. `parse_tfrag` walks each list and classifies every unpack by
|
||
format *and* destination VU address, using the VU header at the head of the
|
||
common list (5 V4_16 vectors, unsigned, to address 0). That table is what
|
||
separates the strips from the indices from the parent arrays — they are all
|
||
V4_8, and only the address tells them apart.
|
||
|
||
| `(vn, vl)` | Meaning |
|
||
|---|---|
|
||
| (3, 1), unsigned, addr 0 | the VU header itself |
|
||
| (3, 0) | ad-gifs, 5 quadwords each |
|
||
| (2, 1) | positions, V3_16 |
|
||
| (3, 1) | vertex info: UV, LOD parent, own position |
|
||
| (3, 2) | strips (signed) and indices (unsigned) |
|
||
|
||
**Strip indices address the vertex-info array, not the position array.** Several
|
||
vertex-info entries can share a position, so a strip index resolves through
|
||
`vertex_info[i].vertex / 2` to reach a position.
|
||
|
||
A strip record is four bytes: a vertex count, a packet flag and an ad-gif
|
||
offset. A negative count is biased by −128, and *the first record of a list is
|
||
always biased* even when its count is positive — a real patch opens with −116 or
|
||
−124, never a small positive number. A zero count ends the list. A run of `n`
|
||
vertices is a GS triangle strip of `n − 2` triangles with alternating winding.
|
||
|
||
**Bank the LODs by index, not by arrival.** The lists are walked in transfer
|
||
order but the strip and index arrays have to be stored by *which LOD they
|
||
belong to*. Appending them as they are found fills the arrays LOD-2-first,
|
||
which looks fine until the triangle counts come out low.
|
||
|
||
### Verification
|
||
|
||
`tri_count` is stored in the patch header and cannot be derived from the lists
|
||
without walking them, so reproducing it is a strong check. All 27 levels:
|
||
**18,102 patches, 809,879 triangles, every patch matching its header**, with
|
||
every triangle index inside that LOD's vertex-info range and every ad-gif index
|
||
inside the ad-gif array.
|
||
|
||
Getting there took four wrong turns, each of which failed quietly: reading the
|
||
element size from the immediate, 2-byte payload padding, ignoring non-unpack
|
||
payloads, and banking strips by arrival order. Every one produced parseable,
|
||
plausible geometry; only the header's own triangle count caught them.
|
||
|
||
### Texture bindings, colours and lights
|
||
|
||
Also settled. Adds `level_core.rs` for the index header and the texture table,
|
||
which is the join between tfrags and texel data.
|
||
|
||
A tfrag's **ad-gifs** are five A+D quadwords (0x50 bytes) holding packed
|
||
fields rather than GS registers; the level load rewrites them in place before
|
||
drawing. On disc `TEX0_1.data_lo` indexes the level's texture table,
|
||
`TEX1_1.data_lo` is the LOD bias K and `data_hi` the minification filter, and
|
||
`CLAMP_1` holds the two wrap modes. The texture table is a count/offset pair at
|
||
index offset 0x30, entries of 16 bytes.
|
||
|
||
The table checks out on its own terms: on level 0 it is 93 entries at 0x2900,
|
||
and 0x2900 + 93 x 16 lands exactly on the next range at 0x38.
|
||
|
||
Three details worth writing down:
|
||
|
||
* **K is a signed 12-bit value in 1/16 units**, sign-extended by shifting the
|
||
field up 4 and back down. Retail values are -137..-89, which is -8.56..-5.56
|
||
mip levels. Reading the field as unsigned gives 1911 instead of -137.
|
||
* **`data_hi` of TEX1_1 is MMIN, and 4 is `LinearMipmapNearest`**, not
|
||
`LinearMipmapLinear` as the bit layout alone suggests.
|
||
* **RGBA and the light/normal records are not in the VIF lists.** They sit at
|
||
fixed offsets in the patch's data block: `rgba_size * 4` four-byte colours at
|
||
`rgba_ofs`, and `vert_count` eight-byte records at `light_ofs + 0x10`. The
|
||
origin also lives there — if no STROW row supplied one, the quadword at
|
||
`light_ofs` is the origin.
|
||
|
||
### Verification
|
||
|
||
Every ad-gif must index a texture that exists, or the geometry and the texture
|
||
data have been misread independently of each other. Across all 27 levels:
|
||
**47,950 ad-gifs over 18,102 patches, all indices in range**, referencing
|
||
1429 of the 1595 level texture entries (90%). Each patch's light array is
|
||
exactly `vert_count` records and its RGBA array exactly `rgba_size * 4`.
|
||
|
||
### Texels: 8-bit indexed images and CSM1 palettes
|
||
|
||
Settled. Adds `texture.rs` and makes the pipeline write `gs_ram.bin`.
|
||
|
||
A level texture is not pixels, it is **palette indices**: `width * height`
|
||
bytes at `core_data[textures_base + data_offset]`. The colour comes from a
|
||
256-entry CLUT (1024 bytes) in GS RAM at `palette * 0x100`. `gs_ram` is a byte
|
||
range in the level's data extent header at offset 0x10 — 518,656 bytes on level
|
||
0 — and the extractor was not writing it out, so nothing could be decoded
|
||
before this.
|
||
|
||
Two PS2 conventions have to be honoured, and neither is a design choice:
|
||
|
||
* **CSM1 CLUT order.** Within each 32-entry group the two middle 8-entry
|
||
blocks are swapped: bits 3 and 4 of the index are exchanged. The remap is an
|
||
involution.
|
||
* **Alpha runs 0..=0x80, not 0..=255.** `0x80` is fully opaque and anything
|
||
above it clamps.
|
||
|
||
The mip chain splits across two buffers. Levels 0 and 1 are in the texture
|
||
block, with level 1 immediately after level 0 at `data_offset + w*h`; levels 2
|
||
and 3 are resident in GS RAM at `mipmap * 0x100` and `pad * 0x100`. Tfrag
|
||
textures are square, so every level halves evenly.
|
||
|
||
### Verification
|
||
|
||
Numbers alone do not settle this — every step is a table lookup, so a wrong
|
||
CLUT remap still yields a well-formed image of the wrong thing. Two layers:
|
||
|
||
* **Shape.** All 1,595 tfrag textures across all 27 levels decode; 96.8% are
|
||
fully opaque and 96.3% use more than eight distinct colours. A flat,
|
||
transparent or single-colour result would point at a CLUT or offset mistake.
|
||
* **Eyes.** `gc-extract dump-texture` writes any tfrag texture, or any of its
|
||
mip levels, as a binary PPM. Decoded level 0 texture 0 is a clean bevelled
|
||
metal panel with rivets and specular highlights; texture 74 is a vent grille
|
||
whose punched windows come back fully transparent. Both confirm the CLUT
|
||
remap, the alpha scale and the pixel base at once, which no assertion in the
|
||
suite does.
|
||
|
||
That tool is worth keeping: it is the cheapest way to tell a decode that is
|
||
merely well-formed from one that is right.
|
||
|
||
### LOD selection
|
||
|
||
Mostly settled. The block header's float at +0x08 gives `L`, and the switch
|
||
distances are `[D0, D1, D2] = [6L, 4L, 2L]` world units; on every level `L` is
|
||
32.0, so `[192, 128, 64]`. The distance test compares the patch's bounding
|
||
sphere, truncated to integers, against `trunc(D * 1024)`. Six modes result —
|
||
LOD 2, LOD 1 collapse, LOD 1 morph, LOD 0 collapse, LOD 0 morph, LOD 0 — and
|
||
the *sphere* is what is tested, not the centre, so a patch straddling a
|
||
threshold takes the coarser mode. That is what stops geometry popping as the
|
||
camera moves, and it is worth testing for directly: sweeping the depth must
|
||
never move back to a finer mode.
|
||
|
||
A morph weight runs 0 to 1 across its tier's band: LOD-01 between D1 and D0,
|
||
LOD-0 between D2 and D1. Past the far edge of the band an entry is not morphed
|
||
but *replaced* by its first parent — a collapse, not a blend.
|
||
|
||
### Parent indices: settled, but only after two corrections
|
||
|
||
The four V4_8 arrays (`parent_indices_lod01`, `unk_indices_2_lod01`,
|
||
`parent_indices_lod0`, `unk_indices_2_lod0`) are classified by VU address,
|
||
and **the addresses are not unique**. On level 0 patch 1,
|
||
`unk_indices_2_lod_01_addr == parent_indices_lod_0_addr == 291`, and
|
||
`unk_indices_2_lod_0_addr == indices_addr == 294`. So address matching alone
|
||
cannot separate them, and getting it wrong is silent: the arrays still parse,
|
||
they just hold the wrong numbers.
|
||
|
||
Two things were needed to fix it, and the first attempt at getting it wrong is
|
||
worth recording because it looked like a data problem rather than a code one.
|
||
|
||
**Each V4_8 element is 4 bytes and holds four indices**, so an array of N
|
||
entries is `count * 4` bytes, not `count`. An early analysis compared the
|
||
unpack's element count against the position counts and concluded that 15,000
|
||
patches could not possibly hold enough parents. That was the wrong unit; the
|
||
data was fine.
|
||
|
||
**`unk_06` and `unk_0a` suppress an empty tier.** When the extra vertex-info
|
||
count is zero there is no extra array, and its VU address is simply the next
|
||
tier's. Guarding those two branches on `unk_06 != 0` / `unk_0a != 0` lets the
|
||
address fall through to the array that actually exists. This one change took
|
||
resolved LOD-0 entries from 28,739 to 341,306.
|
||
|
||
### Verification
|
||
|
||
Every vertex-info entry across all 27 levels is now accounted for: of 886,841
|
||
entries, 516,492 are common and have no link by design, and **all 370,349 LOD
|
||
entries resolve** — 29,043 LOD-01 and 341,306 LOD-0 — with zero failures. Each
|
||
resolves to a first parent that is real vertex info, a LOD-01 parent that is
|
||
always a common entry, and three positions inside the patch. A common entry
|
||
never claims a parent.
|
||
|
||
Ruled out along the way: address-only classification (addresses collide), and
|
||
one parent array entry per unpack element (wrong unit).
|
||
|
||
### The origin, and why no parser test could catch it
|
||
|
||
Positions are `i16`, so a single patch spans at most +-32 world units around
|
||
its own origin. The origin comes from an STROW row, and the obvious rule --
|
||
the last non-index STROW in the list -- is **wrong on RAC2**. RAC2 also uses
|
||
STROW for the per-vertex light records, and those rows come *after* the origin,
|
||
so taking the last one yields `[186, 186, 186, 186]` and every patch collapses
|
||
onto the world origin.
|
||
|
||
Nothing in the test suite noticed, and that is the point worth recording:
|
||
|
||
* `vert_count` still matched, because it counts positions, not places.
|
||
* `tri_count` still matched, because the topology never changed.
|
||
* Every parent, morph target and triangle index still resolved, because those
|
||
are all relative to the same broken origin.
|
||
|
||
The entire level fit in a 63 x 62 x 62 unit box centred on the origin, and all
|
||
of it was self-consistent. Only rendering exposed it, and only because the
|
||
screenshot put the camera where the bounding spheres said the terrain was.
|
||
|
||
The rule that works: **the origin is the STROW immediately preceding the first
|
||
V3_16 position unpack** in the common list. With that, level 0's vertices span
|
||
x 86..444, y 46..211, z -477..40, which matches the patch bounding spheres.
|
||
|
||
Lesson for the rest of the project: a parser can be entirely self-consistent
|
||
and still describe the wrong thing. Where a value is absolute rather than
|
||
relative, check it against an independent source -- here, the bounding spheres.
|
||
|
||
### Why the first frame is dark
|
||
|
||
The renderer draws unlit: texture times the tfrag's own per-vertex colour.
|
||
That product is genuinely dark -- level 0 patch 0's colours are `(25, 26, 26)`
|
||
and `(92, 121, 119)` in 0..255 -- and the game's lighting is what brightens it.
|
||
|
||
Trying a host directional light or cranking ambient brightness was the wrong
|
||
instinct and it showed: with no normals the PBR path is degenerate, so
|
||
brightness changes did nothing at all. Faking it would also have hidden the
|
||
gap. The colour attribute is picked up from the mesh automatically, so it is
|
||
the *colour* data that is missing lighting, not a flag that was left off.
|
||
|
||
The real model is `LightTfrags`. The PS2 float model and the pass itself are
|
||
now written (`gc-engine::ps2`, `gc-engine::tfrag_light`); what is missing is the
|
||
light bank. The three inputs were:
|
||
|
||
* the **light records** -- already parsed: azimuth and elevation as *indices*
|
||
into the overlay's sine table, plus the base colour;
|
||
* the **normal table** -- now decoded (`gc_formats::tfrag_light`), and normals
|
||
are built and fed to the renderer;
|
||
* the **light bank** from the level's `gameplay` extent -- **not working**,
|
||
see below.
|
||
|
||
### Point lights are a runtime feature, not a level-load one
|
||
|
||
Every patch on every level has `point_lights == 0xffff` (none) and
|
||
`dir_lights_one == -1`, checked across all 18,102 patches. The eight-slot point
|
||
light bank is filled by gameplay at run time, not read from the disc.
|
||
|
||
So `point_light_vertex` is not missing work for a static level render -- it
|
||
would light nothing. The directional pass *is* the complete lighting for a
|
||
level-load frame, and the point-light path belongs with the hero, explosions
|
||
and the other things that spawn light.
|
||
|
||
`dir_lights_one == -1` on every patch also means the per-vertex `select` word
|
||
is what selects a light set, never the header field -- which is what the
|
||
implementation uses.
|
||
|
||
### The class tables were not unreadable, only unsized
|
||
|
||
`core_index` holds moby, tie and shrub class tables at 0x18, 0x20 and 0x28,
|
||
followed by the texture tables. They had been ruled out for want of an entry
|
||
size, and every field order and stride tried.
|
||
|
||
The sizes are knowable exactly, because **each table ends precisely where the
|
||
next begins**:
|
||
|
||
- moby 238 entries at 192, 32 bytes each: 192 + 238x32 = 7808 = tie's offset
|
||
- tie 74 at 7808, 32 bytes: 7808 + 74x32 = 10176 = shrub's offset
|
||
- shrub 7 at 10176, 48 bytes: 10176 + 7x48 = 10512 = the tfrag texture table
|
||
- tfrag textures 93 at 10512, 16 bytes: 10512 + 93x16 = 12000 = moby textures
|
||
|
||
That closes on **27 of 27 levels**, which is what makes it evidence rather
|
||
than a coincidence: any wrong entry size would break the chain immediately.
|
||
|
||
The class entries are 32 bytes -- `{asset offset, class number, two unknown
|
||
words, 16 bytes of texture slots}` -- and a shrub entry adds 16 bytes of
|
||
billboard slots for 48. Across all 27 levels: **5,796 moby, 1,864 tie and 358
|
||
shrub classes**, and 1,595 tfrag textures.
|
||
|
||
Every shrub class on level 0 carries all-zero billboard slots, so the billboard
|
||
dimensions are optional rather than something to assert on.
|
||
|
||
The failure here is the third instance of the same shape. The gameplay extent
|
||
was "an unidentified compression" because it was being parsed compressed; the
|
||
tfrag record framing was "unrecoverable" because the entry size was never
|
||
known. Both looked like properties of the format and were properties of the
|
||
question.
|
||
|
||
### Where the class tables point
|
||
|
||
`assets_compressed_size` and `assets_decompressed_size` at index offsets 0x88
|
||
and 0x8c are 9,738,678 and 16,969,296 on level 0 -- exactly the sizes of
|
||
`core_data` before and after decompression. There is no separate asset
|
||
container: the "asset WAD" *is* `core_data`, and `offset_in_asset_wad` indexes
|
||
the decompressed form directly.
|
||
|
||
That is checkable rather than merely plausible, and a test checks it on all 27
|
||
levels:
|
||
|
||
* `assets_decompressed_size` equals `core_data`'s exact length;
|
||
* every class offset is either 0 or inside `core_data`;
|
||
* the moby, tie and shrub spans do not overlap each other.
|
||
|
||
An offset of **0 means the slot has no geometry**, not the start of the lump.
|
||
On level 0 the moby table's ninth entry is class 4909 at offset 0.
|
||
|
||
Two assumptions worth discarding: the tables are **not** sorted by class number
|
||
(level 0's moby classes run 0, 1, 2, 14, ... 4909), and the offsets within a
|
||
table are **not** monotonic. Neither is needed, but both look true from the
|
||
first few entries.
|
||
|
||
The chunk descriptors after the named extents are not it. They exist on only
|
||
3 of 27 levels -- level 0's four named extents tile the container exactly --
|
||
so an asset blob there would be missing almost everywhere.
|
||
|
||
At the offsets sit raw GS/VIF packet data, not a further container: moby[0]
|
||
begins `00 00 00 00 3f 16 02 55 6f 6f`, tie[0] `00 01 00 00 70 50 00 00`,
|
||
which read as packet headers rather than a magic number.
|
||
|
||
### Tie: the class header only partly matches
|
||
|
||
Tie is the easy one of the three -- instanced static geometry, no skeleton -- so
|
||
it is the right place to start.
|
||
|
||
RAC1's class header is 0x80 bytes. RAC2 agrees with it over the first `0x48`:
|
||
the per-LOD packet offsets, the three LOD distances and `o_class` all check out.
|
||
From `0x48` on it does not. `lod_info` at `0x50` reads back as noise,
|
||
`packet_count` and `texture_count` are zero on every class examined, and the
|
||
`normals` offset at `0x0c` points 117,788,434 bytes into a 16.9 MB lump.
|
||
|
||
So only the agreed-on prefix is decoded. The evidence for trusting it:
|
||
|
||
- **`o_class` at `0x44` equals the core index's class number on all 1,864 tie
|
||
classes across all 27 levels.** A field landing on the right number that many
|
||
times is not a coincidence, and it is what pins the header base.
|
||
- The LOD distances are ordered `0 < near < mid < far` for **1,784 of 1,864**.
|
||
The 80 exceptions are real, unexplained, and are reported rather than
|
||
asserted away.
|
||
|
||
Level 0's first tie class has distances 12.0 / 25.0 / 40.0, which is a
|
||
plausible prop LOD band, and packet offsets 256, 20,592 and 26,048.
|
||
|
||
The remainder is not guessed at. A field order that produces plausible output
|
||
and is wrong is the specific failure this file keeps recording -- the gameplay
|
||
extent, the tfrag record framing and the class tables have all been it.
|
||
|
||
### Why the frame is still dark: the light bank does not parse on RAC2
|
||
|
||
The reference frames the bank as "a u32 offset at +0x04, then an `i32` count
|
||
there, then the sets". On RAC2 level 0 that offset is `0xa0d`, and the `i32`
|
||
there is `698810368` -- not a count, and not a plausible one. `gameplay.bin`
|
||
begins with `0x794e...`, not with a header of the shape the reference expects,
|
||
so RAC2's gameplay extent is not laid out this way, or is compressed before
|
||
this point.
|
||
|
||
The pass therefore never runs. Rather than let the renderer quietly draw the
|
||
stored colours and look plausible, it counts what it lit and says so:
|
||
|
||
```text
|
||
lighting: 0 of 80274 vertices shaded by LightTfrags; the rest fall back to the
|
||
tfrag's stored colours (no usable light bank)
|
||
```
|
||
|
||
That line is the honest state, and it is why the frame has not changed. What
|
||
`gameplay.bin` actually is, and where RAC2 keeps its light bank, is the open
|
||
question — `0x794e...` looks like it may be a compressed or encrypted
|
||
container, in which case the bank is somewhere else entirely.
|
||
|
||
### The PS2 float model
|
||
|
||
`gc-engine::ps2` is the EE's FPU on raw `u32` bit patterns, and it is not
|
||
`f32`:
|
||
|
||
* **no denormals** -- exponent 0 *is* zero, whatever the mantissa
|
||
* **no NaN, no infinity** -- exponent 255 is an ordinary value, which means
|
||
`MAX` reads back in Rust as NaN. A test asserts exactly that, because it is
|
||
the clearest single statement of why this module exists.
|
||
* **every result truncates toward zero**, and `1/3` comes out below `1/3`
|
||
* **the adder aligns operands**: the smaller operand loses the bits more than
|
||
one below the larger's LSB, and a 25-bit exponent gap returns the larger
|
||
operand unchanged
|
||
* **`rsqrt` is sqrt-then-divide**, with truncation between, not one rounded step
|
||
|
||
`mul`, `add`, `div`, `sqrt`, `itof12` and `max`/`min` all differ from `f32` in
|
||
at least one case a lighting pass will hit. The pass accumulates them across
|
||
every vertex of every patch, so this is a visible difference rather than a
|
||
rounding curiosity.
|
||
|
||
### The lighting pass itself
|
||
|
||
Per vertex: `base + A.rgb * max(dA, dA*wA) + B.rgb * max(dB, dB*wB)`, where
|
||
`base` is the record's 5:5:5:1 colour expanded by PEXT5, `dA`/`dB` are
|
||
three-component dot products against the light directions, and `wA`/`wB` come
|
||
out of the colour w lanes. `select` picks one light set or blends two by
|
||
`bits 8..15 / 256`, and a blended direction is re-normalised where an unblended
|
||
one is not.
|
||
|
||
One field had to be added to `TfragLight` to carry this: `select`, the `u16` at
|
||
+6 of the record. It was briefly reconstructed with an `unsafe` pointer cast
|
||
over the struct -- which is exactly the kind of thing that breaks silently the
|
||
moment a field is reordered, and is now a plain parsed field.
|
||
|
||
### The normal table, and why it is found by searching
|
||
|
||
The reference reads the table from an ELF at `BOOT_NORMAL_TABLE_VADDR`
|
||
(`0x165500`) and notes every level overlay carries a copy. **The overlay has no
|
||
ELF header** -- it is a raw loadable image starting with code -- so there is no
|
||
program header to map a virtual address through, and the offset is not fixed
|
||
either: level 0 has the table at `0xb650c`, level 5 at `0xcca30`, level 26 at
|
||
`0xb6284`.
|
||
|
||
So it is located by content. Every table starts with exactly `(1.0, 0.0)` --
|
||
angle zero -- and the scan then requires all 256 entries to match
|
||
`cos(i·pi/128), sin(i·pi/128)`. Validating the whole curve is what makes the
|
||
search safe: a lone `(1.0, 0.0)` pair of floats occurs by accident in a 2.7 MB
|
||
executable, and a test plants one as a decoy. The values read from the disc are
|
||
used as-is rather than recomputed, since those are the numbers the game indexes.
|
||
|
||
A normal is `(cos(az)cos(el), sin(az)cos(el), sin(el))` **negated** -- the
|
||
stored normals point into the surface -- then given the same Z-up to Y-up swap
|
||
as positions, or the lighting comes out sideways. The azimuth and elevation
|
||
fields are stored signed but index 0..=255, so the byte is what matters, not its
|
||
sign: entry 200 is read as 200, not as -56.
|
||
|
||
Given the table, a normal is `(cos(az)cos(el), sin(az)cos(el), sin(el))`,
|
||
negated, exactly as `decode_normal` builds it. The lighting pass itself is
|
||
`base + A.rgb * max(dA, dA*wA) + B.rgb * max(dB, dB*wB)` per vertex, and it
|
||
must be reproduced in the PS2's float model rather than in Rust `f32`, per
|
||
the fidelity rule.
|
||
|
||
### Still open
|
||
|
||
- **Point lights.** The directional pass runs; `point_light_vertex` and its
|
||
`r^2 - |p - L|^2` cull test are not written, and they are the one part of
|
||
the lighting that *is* distance-dependent.
|
||
- ~~**The GS register set**~~ **Partly settled 2026-10-14.**
|
||
`gc_formats::gs::tex0_of` decodes TEX0 (TBP0, TBW, TW, TH, PSM) from a texture
|
||
entry, verified over all 1,595 textures on all 27 levels: every dimension is a
|
||
power of two, so the exponent fields are exact, and TBP0 is **region-relative**
|
||
because 19 of 27 levels have a `textures_base` only 64-byte aligned.
|
||
`texture_slot_bytes` is `w*h + (w/2)*(h/2)` -- level 0 plus one mip -- on
|
||
1,560 of 1,564 ascending gaps, with four bounded exceptions. Still open: the
|
||
**TEXCLUT** register (the CLUT's CBP/CSA/CSM), MIPTBP1/2 wiring, and the
|
||
A+D values the ad-gifs carry relative to this base.
|
||
- **The guard-band clip test** that sends patches crossing the guard band to
|
||
the clipping program. The geometry side is now in place -- the 8-corner clip
|
||
cube at header `+0x38` is parsed and tested (`core-data-regions.md`, the
|
||
clip-cube section). What is outstanding is the **camera matrix** and the
|
||
view frustum as a clip volume, both per-frame game state, so the test itself
|
||
needs a savestate with the camera pinned. `draw_mode` is unchanged and already
|
||
matches the reference's mode table.
|
||
- ~~**Lighting magnitudes.**~~ **Settled 2026-10-11** by the PCSX2
|
||
comparison loop (`docs/plan/pcsx2-verification.md`): level 0's 162,400
|
||
colour bytes match the game's EE RAM exactly. Two bugs the comparison found
|
||
and fixed, both of which produced output that looked plausible:
|
||
`vertex_normal_bits` negated x and y but not z (so `max(dA, dA*wA)` took the
|
||
wrong branch and every vertex lit near its base colour -- 95.88% agreement
|
||
instead of 100%), and a record whose `select` names a set the bank lacks
|
||
drew transparent black where the game writes the record's base colour (3,130
|
||
of level 0's 40,600 vertices, `select 0x000f`).
|
||
- **tie, moby and shrub geometry.** The class tables parse and the container
|
||
they point into is identified (below), but none of the three is decoded yet.
|
||
Moby is the largest of the three and is skeletal, with an animation evaluator
|
||
in the microcode; it is the single biggest piece left before the rest of a
|
||
level can draw.
|
||
- **Point lights** are spawned by gameplay at runtime and are absent from the
|
||
disc -- every one of the 18,102 patches has `point_lights == 0xffff` -- so
|
||
there is nothing for them to light until the game runs.
|
||
|
||
A tfrag can now be turned into positioned, UV-mapped, textured, lit triangles,
|
||
told which LOD to draw, and morphed toward its parents at the right rate.
|
||
Everything needed for a first frame is parsed, and the frame is verified by
|
||
screenshot rather than by assertion.
|
||
|
||
## Core index field order (2026-10-11)
|
||
|
||
Repeatedly re-deriving these by hand wasted real time this session, so the
|
||
order is written out. It matches `LevelCoreHeader::parse` in
|
||
`crates/gc-formats/src/level_core.rs`:
|
||
|
||
| offset | field |
|
||
|--------|-------|
|
||
| 0x00 | gs_ram count |
|
||
| 0x04 | gs_ram offset |
|
||
| 0x08 | tfrags |
|
||
| 0x0c | occlusion |
|
||
| 0x10 | sky |
|
||
| 0x14 | collision |
|
||
| 0x18 | moby_classes count |
|
||
| 0x1c | moby_classes offset |
|
||
| 0x20 | tie_classes count |
|
||
| 0x24 | tie_classes offset |
|
||
| 0x28 | shrub_classes count |
|
||
| 0x2c | shrub_classes offset |
|
||
|
||
Note `gs_ram` and the three class tables are **count/offset pairs**, so every
|
||
one of them is two words. The region offsets (`tfrags` through `collision`) are
|
||
single words. Counting the class tables as one word each shifts every field
|
||
after them, which is how `collision` got read as `sky` and produced a page of
|
||
implausible collision headers on all 27 levels.
|
||
|
||
## Sky (2026-10-11)
|
||
|
||
Region is `[sky, collision)` and is well defined, but the header is **not**
|
||
yet decoded and the layout is not uniform.
|
||
|
||
Reading `u16` at the head, most levels give:
|
||
|
||
```
|
||
+0x00 0
|
||
+0x02 0
|
||
+0x04 4 or 6 varies per level
|
||
+0x06 0
|
||
+0x08 0 level 2 is the exception
|
||
+0x0a 3..6 looks like a layer count
|
||
+0x0c 0
|
||
+0x0e 64 (0x40)
|
||
+0x10 128 or 192 varies
|
||
+0x12 64 (0x40)
|
||
```
|
||
|
||
Level 2 breaks the pattern: `+0x08` is 0x0109 and the three words after the
|
||
apparent count are shifted and different (`80 00 00 00 00 01 00 00 40 00`).
|
||
So a single fixed header does not fit every level, and the count at `+0x0a`
|
||
cannot be trusted to be one.
|
||
|
||
A table of offsets follows at `+0x20` for the layers. The offsets ascend and
|
||
sit in the last 8-15% of the region, but the count of *non-zero* offsets is
|
||
sometimes less than the declared count (level 26 declares 6, has 4 real
|
||
offsets and 2 zero slots). Unresolved.
|
||
|
||
## Moby geometry (2026-10-11)
|
||
|
||
Class entries are 0x20 bytes in `core_index.bin`, fields
|
||
`geom_offset, o_class, unknown_8, unknown_c, textures[16]`; all 238 level-0
|
||
moby classes carry `textures == ffff..ff`, i.e. no texture slots.
|
||
|
||
`geom_offset` points into `core_data`, and the spans tile without overlap.
|
||
|
||
The span does not start with the geometry. Its first `u32` is the offset of a
|
||
part table, and the table is 16-byte records:
|
||
|
||
```
|
||
+0x00 u32 vertex_offset from the start of the class
|
||
+0x04 u16 vertex_count
|
||
+0x06 u16 second_count meaning not established
|
||
+0x08 u32 next_part
|
||
+0x0c u32 flags
|
||
```
|
||
|
||
Parts tile: `vertex_offset + 16 * vertex_count == next_part`. The stride is not
|
||
assumed -- it falls out of that identity, which is what pins the record size
|
||
and the field boundaries. It holds for 43,467 of 43,467 part records across
|
||
every moby class on all 27 levels.
|
||
|
||
Two terminations are needed, not one. A zero `vertex_offset` ends the run, but
|
||
172 records across the disc have a zero `vertex_offset` *and* a zero
|
||
`next_part` while still carrying a vertex count; stopping only on the second
|
||
is what leaves 99.6% instead of 100%. A zero `next_part` means the record
|
||
starts a different table rather than a part.
|
||
|
||
Totals: 1,270,802 vertices over 4,560 class spans, 116 of which carry no
|
||
geometry.
|
||
|
||
Vertices are 16 bytes of VIF-packed half-float quads, cycling through four
|
||
distinct 32-bit words. Not unpacked -- handed back raw.
|
||
|
||
## Shrub geometry (2026-10-11)
|
||
|
||
Class entry strides differ between the three tables. Measured by how many
|
||
entries yield a geometry offset inside `core_data` and a plausible `o_class`,
|
||
across all 27 levels:
|
||
|
||
| table | stride | sane entries |
|
||
|-------|--------|--------------|
|
||
| moby | 0x20 | 5769 / 5796 |
|
||
| tie | 0x20 | 1864 / 1864 |
|
||
| shrub | 0x30 | 356 / 358 |
|
||
|
||
Reading shrub at 0x20 still yields 231 sane-looking entries out of 358, which
|
||
is the problem: it is wrong often enough to look right. The stride is already
|
||
handled -- `ShrubClassEntry` is 48 bytes, carrying billboard texture fields the
|
||
other two do not have.
|
||
|
||
Shrub geometry is **not** the moby part-table format. All 358 spans have a
|
||
leading word of 0 or one too large to index a table into, so none of them
|
||
contain a moby-style part table.
|
||
|
||
The header does self-identify in two places, both exact across all 358 spans:
|
||
|
||
- `span + 0x24` is a `u32` equal to the class entry's own `o_class`. 358/358.
|
||
- `span + 0x2c` is a `u32` data offset that lands inside the span. 358/358.
|
||
|
||
`span + 0x10` reads 5.0f on 281 of 358 spans, so it is a value rather than a
|
||
constant marker and carries no structural weight. The header is otherwise
|
||
undecoded: the first 0x20 bytes are packed data rather than floats, despite
|
||
`+0x10` landing exactly on 5.0.
|
||
|
||
## Tie geometry (2026-10-11)
|
||
|
||
Tie does **not** reuse moby's part-table format. The rule
|
||
`vertex_offset + 16 * vertex_count == next_part` holds for **0 of 25,024**
|
||
records tried across the 1,864 tie class spans, against 43,467/43,467 for
|
||
moby. Tie's second field also reads as an absolute offset rather than a
|
||
span-relative one, which is the first sign the record shape differs.
|
||
|
||
At `+0x48`, where the verified header currently ends, the bytes are f32-shaped
|
||
(10.0, 40.0, 50.0, 4.0, -100.0, -25.0, 5.0, 3.125), not a VIF marker as
|
||
expected. They are not a fixed-size array either: the run of consecutive
|
||
plausible floats varies from 1 to 6 across spans, with 822 spans having 6,
|
||
455 having 1 and 373 having 2. So there is more header past `0x48`, and it
|
||
is not a single array of distances.
|
||
|
||
### Probe tool
|
||
|
||
`gc-extract probe <unpacked-dir> <level> [max-rows]` dumps the core header and
|
||
all three class tables through `LevelCoreHeader`, which is the point. The
|
||
table above and the shrub offset below were both established by hand first and
|
||
one of them nearly sent me to fix a bug that was not there. The tool also
|
||
reports how many entries carry geometry inside `core_data`, which is how the
|
||
shrub stride was pinned: 7/7 at 0x30 against 4/7 read at 0x20.
|
||
|
||
Note the LOD distances already decoded sit *before* this point (12.0, 25.0,
|
||
40.0 at `+0x10`), and the floats after `0x48` are a different population --
|
||
negative values, and a repeated leading 40.0.
|
||
|
||
Unresolved. The packet geometry remains the open question, unchanged.
|
||
|
||
## Shrub body (2026-10-11, later)
|
||
|
||
Stride-fitting the body failed three times, so this came from a different
|
||
angle: group spans by `(count, data_offset, tail_length)` and byte-diff every
|
||
member of each group. Byte positions that differ across a group are geometry;
|
||
positions that never differ are structure.
|
||
|
||
The result is blunt. **144 bytes vary in every group**, whether the group has
|
||
count 1, 2, 3, 5 or 6 and a tail of 192, 208 or 240 bytes. So the varying data
|
||
is not sized by `count` at all, and the three strides that failed were fitting
|
||
the wrong thing.
|
||
|
||
Those 144 bytes sit in runs of 6, every 8 bytes: `(0..5), (8..13), (16..21),
|
||
...`. The body is an array of **8-byte records, 6 varying and 2 constant**.
|
||
|
||
The payload is not half-floats. Decoding the 6 bytes as a `half3` gives 245.25
|
||
and -168.375, which no shrub has.
|
||
|
||
**Correction, later the same day.** This section originally concluded the
|
||
payload was VIF-packed, on the strength of recurring patterns like `7ea5` and
|
||
`1ab6`. That conclusion does not survive a control. Decoding every 32-bit word
|
||
of every shrub body as a `VifCode`:
|
||
|
||
| region | words decoding as UNPACK | distinct cmd bytes |
|
||
|---|---|---|
|
||
| shrub bodies | 24.9% | 128 (all of them) |
|
||
| uniform random | 25.1% | 128 (all of them) |
|
||
| sound_bank (definitely not VIF) | 19.0% | 122 |
|
||
| gs_ram (mixed) | 13.6% | 128 |
|
||
| moby class 0 span (known VIF) | 40.1% | 105 |
|
||
|
||
Shrub sits on the random baseline, and all 128 possible command bytes occur,
|
||
which is what unstructured data looks like. The VIF reading was pattern
|
||
matching on a handful of recurring byte values, and it was wrong.
|
||
|
||
Note the test is weak in general: even the known-VIF region only reads 40.1%.
|
||
So this does not prove shrub is *not* VIF -- it proves the claim was never
|
||
established. Treating three recurring bytes as a format is the same mistake as
|
||
the three stride guesses, just at a different level.
|
||
|
||
What survives from this section is the differential finding, which rests on
|
||
grouping and diffing rather than on recognising byte patterns: 8-byte records,
|
||
6 varying and 2 constant, 144 varying bytes per group regardless of `count`.
|
||
The payload's encoding is still unknown.
|
||
|
||
### Shrub span bounds are wrong (later still)
|
||
|
||
A follow-up measurement contradicts the body size implied above, and the
|
||
reason is a bug in how spans are bounded rather than anything in the format.
|
||
|
||
Shrub class spans were bounded by the *next shrub offset*, with the last one
|
||
running to the end of `core_data`. On level 0 that makes the final shrub span
|
||
**925,008 bytes** -- a single shrub class absorbing most of the tail of the
|
||
lump. The moby-to-tie boundary is clean by comparison: last moby offset
|
||
13414592, first tie offset 13416192, a gap of exactly 1600 bytes.
|
||
|
||
Consequences, stated plainly:
|
||
|
||
- The "3.5 million records / 28 MB of shrub body" figure is an artifact of
|
||
that oversized final span and should not be quoted.
|
||
- The payload appearing uniform-random across the full u16 range is measured
|
||
over that same contaminated range, so it is not evidence about shrub
|
||
geometry either way.
|
||
- The differential result (8-byte records, 144 varying bytes) was computed on
|
||
*groups*, and the specific groups used had bounded spans, so that
|
||
measurement stands. The global statistics built on top of it do not.
|
||
|
||
What is needed: the offset that ends the shrub region. It is not in the core
|
||
index, and unlike moby->tie there is no obvious successor boundary. Until
|
||
that is found, shrub span sizes are unreliable and anything summed over all
|
||
358 spans is suspect.
|
||
|
||
This is the second time in this format that a confident aggregate hid a bad
|
||
bound -- the first was the VIF reading. The pattern is worth noting: aggregates
|
||
across many spans are only as trustworthy as the span bounds underneath them,
|
||
and those are the easy thing to get wrong because nothing errors.
|
||
|
||
## Shrub span bounds: resolved (2026-10-11, later still)
|
||
|
||
The offset that ends the shrub region is the **start of the chunk container**,
|
||
and it is found by signature rather than read from the core index, which has no
|
||
successor offset for shrub.
|
||
|
||
**Range measured:** for each level, from the last shrub class's geometry offset
|
||
to the constant trailer. Nothing outside that range is read.
|
||
|
||
**Method:** scan forward from the last shrub class for two bytes `f6 7c`
|
||
sitting immediately in front of a chunk signature (16-byte header ending `0xff`,
|
||
ten zero bytes, four or more `0xf0`-tagged offsets). Each half of that
|
||
signature is weak alone -- the marker pair occurs 25 to 103 times per lump --
|
||
and together they are exact.
|
||
|
||
**Result: on all 27 levels the marker sits exactly two bytes before the first
|
||
chunk header.** The two parts of the signature are never found apart, and no
|
||
chunk signature occurs inside any last shrub body. The three levels without
|
||
shrub (10, 25, 26) carry the same marker before the container that closes
|
||
*tie*, so the locator is a property of the region, not a shrub rule.
|
||
`gc_formats::chunks::chunk_region_start` implements it and
|
||
`regions::core_layout` closes shrub with it.
|
||
|
||
Consequences, all measured over the new bounds:
|
||
|
||
- **All 358 shrub class bodies are bounded and well formed** -- every one is
|
||
whole 8-byte records. That was not assertable while the last class in each
|
||
level ran out to the lump's end, and it is the check the boundary earns.
|
||
- **9,600 records in total**, against the 8,938 previously measurable in
|
||
successor-bounded spans; 662 of them sit in the 24 last-class bodies, 24 to
|
||
30 per level. The retracted "3.5 million records / 28 MB of shrub body"
|
||
figure stays retracted: the correct total for the shrub region is
|
||
**2,033,776 bytes** across the 24 levels that have one.
|
||
- The chunk container is **25,775,850 bytes across 27 levels** (1,236 chunks,
|
||
22,562 records), not an ~690-900 KB afterthought per level. The earlier
|
||
container measurements were taken over "the 900,000 bytes before the
|
||
trailer", a window that did not align to the container's real extent: on
|
||
level 0 it is 690,302 bytes, and 27 of 1,123 chunks were truncated at that
|
||
invented edge. The chunk test now reads the region from `core_layout`.
|
||
|
||
What is still not known: the marker's meaning (constant across 27 levels, at
|
||
exactly this position, unnamed). The shrub body payload's encoding is the next
|
||
section.
|
||
|
||
## The shrub class blob: palette and billboard (2026-10-11, after the bound)
|
||
|
||
With every span bounded by the chunk container, the shrub class blob reads
|
||
cleanly against the R&C1 format (`vendor/rerac/docs/formats/shrub_sky_rac1.md`,
|
||
derived from the Wrench sources). R&C1's shrub class header keeps the same
|
||
field offsets R&C2 shows -- `o_class` at 0x24, a count at 0x28, the data offset
|
||
at 0x2c -- so the reference says which bytes to look at and the disc says what
|
||
they hold.
|
||
|
||
**Range measured:** all 358 shrub class spans over the 24 levels that have
|
||
them, each bounded by its successor class or the located chunk container.
|
||
Nothing outside the spans is read.
|
||
|
||
### The body is a 24-entry normal palette
|
||
|
||
Two things made this identifiable where three stride-fitting attempts had
|
||
failed. First the shape: every body is 8-byte records whose last two bytes are
|
||
zero, and R&C1's palette is documented as 24 records of exactly that shape
|
||
(`s16` x, y, z, pad). Second the count: bodies hold 24, 26, 28 or 30 records
|
||
with the extra ones **always all zero and always at index 24 or later** --
|
||
never a live entry. So the body is 24 normals plus zero padding, and the
|
||
padding is what distinguishes it from geometry.
|
||
|
||
The palette is real normals, and the test says how firmly:
|
||
|
||
- all 8,592 live entries have squared length within **65,535** of `32767^2`,
|
||
and the bound is tight -- the largest observed deviation is exactly 65,535.
|
||
That number is not fitted: it is `32768^2 - 32767^2`, which is what a unit
|
||
vector scaled by 32768 produces when an axis-aligned normal quantises to
|
||
`-32768` while the positive axis clamps to `32767`. Components run the full
|
||
`s16` range (-32768..32767) and 1,765 of the 8,592 entries are axis-aligned,
|
||
which is what a coarse direction codebook looks like.
|
||
- a random `s16` triple has squared length spread about a *billion* either
|
||
side of `32767^2`, so one entry in fifteen thousand would pass by chance.
|
||
All 8,592 pass.
|
||
- the clincher: 17 classes carry a palette of 24 identical `(0, 0, 1)`
|
||
entries. A position or index table cannot produce that; a normal palette for
|
||
a flat-facing class must.
|
||
|
||
This is also where shrub lighting comes from -- R&C1 lights each instance
|
||
against its class's 24-direction palette. Unlike the tfrag's per-vertex light
|
||
records, which store azimuth/elevation *indices* into the overlay's sine table,
|
||
the shrub palette stores the directions outright, so the overlay table is not
|
||
needed for shrubs.
|
||
|
||
### The header, byte by byte
|
||
|
||
**Range measured:** the 0x30 bytes of every span.
|
||
|
||
| offset | field | measured |
|
||
|---|---|---|
|
||
| 0x00 | f32[4] `sphere` | centre (xyz) + radius (w). Finite on all 358, **radius positive on all 358** (1.72..41.57), centres near the class's own origin |
|
||
| 0x10 | f32 `size` | 3.0..20.0, positive on all 358 |
|
||
| 0x14 | u16 `mode_bits` | **0 on 342 spans, 4 on 16** |
|
||
| 0x16 | u16 `scratch` | **0 on all 358** (asserted by the parser) |
|
||
| 0x18 | u32 `reserved` | **0 on all 358** (asserted by the parser) |
|
||
| 0x1c | u32 `billboard_offset` | 0 on 333 spans, in-span on 25 |
|
||
| 0x20 | f32 `scale` | 0.0044..1.8, positive on all 358 |
|
||
| 0x24 | u32 `o_class` | equals the class entry's own, 358/358 |
|
||
| 0x28 | u32 `count` | 1..55, meaning not established |
|
||
| 0x2c | u32 `data_offset` | inside the span, 358/358 |
|
||
|
||
Two corrections this produces, both recorded because the old readings were
|
||
confidently wrong rather than merely unverified:
|
||
|
||
- **"reserved is 0 or a denormal" was the wrong shape.** The 16 spans with a
|
||
non-zero value there have `mode_bits = 4` and `scratch = 0`, not the other
|
||
way round. As an `f32` the word at `+0x14` is still a denormal, so the PS2
|
||
reads zero either way -- but the field is a flag word, not a magnitude, and
|
||
the demo is the point: the EE's FPU has no denormals, so `04000000` and zero
|
||
are the same value to the hardware.
|
||
- **"bounds" is a bounding sphere, not a min/max box.** The old reading tested
|
||
`min.xyz/max.xyz` ordering, found it on only 146 of 358 spans, and stopped
|
||
naming the field. It is centre + radius, as in R&C1: the fourth float is
|
||
positive on all 358 spans and the centres cluster near the class origin
|
||
within +-24, which is where a class-local sphere belongs. The 146/358 figure
|
||
was a property of the wrong question.
|
||
|
||
### The billboard record
|
||
|
||
The billboard offset at `+0x1c` is the strongest self-identification in the
|
||
header after `o_class`, because it is **confirmed from a second file**: it is
|
||
non-zero on exactly the 25 classes whose core-index entry carries billboard
|
||
texture fields, in both directions -- 25 records and 25 entries, pairing
|
||
correctly. R&C1 states the same correspondence ("a class has a billboard record
|
||
exactly when its core-index billboard texture has width != 0").
|
||
|
||
At the offset sits R&C1's `ShrubBillboard` shape, register block and all:
|
||
|
||
| offset | field | measured |
|
||
|---|---|---|
|
||
| 0x00 | f32 `fade_distance` | 5..1000; 512 reads as 0 under R&C1's `trunc(fade) & 0xff`, i.e. billboard-only |
|
||
| 0x04 | f32[3] | **not established** -- see below |
|
||
| 0x10 | TEX1_1: K, MMIN, address | K -233..-168, MMIN 4 on all 25, address 0x14 |
|
||
| 0x20 | TEX0_1: data_lo, address | index 1..4, address 0x06 |
|
||
| 0x30 | MIPTBP1_1: address | 0x34, data words zero (filled at level load) |
|
||
|
||
The three floats at `+0x04..+0x10` are the part that does *not* carry over.
|
||
R&C1's are `width`, `height` and `z_ofs` in class units; R&C2's are an order of
|
||
magnitude too large and too varied to be dimensions of a 32..128 px billboard,
|
||
and they **repeat byte-identically across levels**: 16 distinct triples among
|
||
the 25 records, with one appearing on four classes across levels 1, 4, 12 and
|
||
21. Dimensions would not do that; shared art assets would. They are handed back
|
||
unnamed.
|
||
|
||
Two further facts about them, measured: the whole record repeats with them --
|
||
three classes on different levels carry identical `fade`, floats, K and scale
|
||
-- and the class entries they belong to carry identical `sphere` too. So the
|
||
blobs themselves are shared between levels, which is the same behaviour the
|
||
chunk container showed with its byte-identical tables.
|
||
|
||
### The packets: the mesh geometry is in the blob (2026-10-11, later)
|
||
|
||
The open question above -- where the mesh geometry lives -- answered itself
|
||
once the running game was consulted. The running PCSX2 loads shrub classes
|
||
into RAM, and diffing a loaded blob against its disc bytes shows what the
|
||
loader rewrites:
|
||
|
||
- **`+0x40` holds a table of `count` entries of `(offset, size)`**, both
|
||
relative to the span start on disc, rewritten to absolute addresses in RAM
|
||
with the sizes unchanged. Verified on level 1 class 1: all 21 entries match
|
||
the disc offset for offset and the size for size after adding the blob's
|
||
RAM base.
|
||
- **`+0x2c` (the palette offset) is relocated** the same way, to an absolute
|
||
pointer.
|
||
- **`+0x18` -- zero on disc, asserted by the parser -- is written with a
|
||
pointer at load**, to a table of 0x20-byte records that look instance-shaped
|
||
(a position, a size, a count, and a `0xffff` light slot). Not established;
|
||
recorded because it is where the next question starts.
|
||
|
||
So the blob is header, packet table, packets, [billboard], palette -- the same
|
||
shape as R&C1's, and the mesh was here the whole time. The reason it was
|
||
missed is worth writing down: **`data_offset` points at the *end* of the
|
||
span**, so reading the body as `data_offset..span_end` accounts for the last
|
||
few hundred bytes and leaves the 20 KB in front of it unexamined. A bound that
|
||
is correct and a body that is small enough to look plausible are together
|
||
enough to hide a region.
|
||
|
||
**Range measured:** every shrub class span on the disc -- 358 spans over 24
|
||
levels, each bounded by its successor class or the located chunk container.
|
||
|
||
**Method:** for each span, read `count` from `+0x28`, then the entries at
|
||
`+0x40`, and require them to tile: the first packet starts at
|
||
`align16(0x40 + count * 8)` -- the table, rounded up to 16 -- every entry is
|
||
exactly `size` past the previous one, and the last ends at the palette, or
|
||
exactly 0x40 short of it on the spans that carry a billboard record in
|
||
between.
|
||
|
||
**Result: 358 of 358 tile exactly, holding 2,239 packets.** No span's
|
||
deviations: the first-packet offsets, the sizes and the ends all agree with
|
||
the rule. That is what pins the field meanings -- `count` is the **packet
|
||
count** (1..55 on the disc), and `data_offset` is the palette's offset -- and
|
||
it is why the earlier "the body is not sized by count" observation was true
|
||
for a reason that had not been understood: the body is not sized by the count
|
||
because the count sizes the packets.
|
||
|
||
`gc_formats::shrub::Shrub::packets()` exposes them, and returns `None` rather
|
||
than a partial list when a table does not tile.
|
||
|
||
### Still open
|
||
|
||
- The three billboard floats, `size`, and `mode_bits` (4 on 16 spans).
|
||
- The packet contents. They are VIF command lists in R&C1; the R&C2 packets
|
||
have not been decoded, and the VIF reading of a shrub body was already
|
||
retracted once, so this needs a control, not a pattern match.
|
||
- The 0x20-byte records the loader points at from `+0x18.
|
||
|
||
## Shrub mesh packets: a 16-byte header and 16-byte records
|
||
|
||
The packet table decode (commit aa2b14a) left the packet *contents* open, with
|
||
a warning that a VIF reading had already been retracted once and so needed a
|
||
control rather than a pattern match. This is that measurement.
|
||
|
||
**Range measured:** level 0, the 29 packets of the 6 successor-bounded shrub
|
||
spans, 26,640 bytes. Per-class and per-packet; nothing pooled.
|
||
|
||
**Method:** autocorrelation within the largest packet against a
|
||
distribution-matched shuffled control, then per-byte distinct counts across all
|
||
29 packets.
|
||
|
||
**Result 1 -- the packets tile at 16 bytes.** All 29 packet sizes are exact
|
||
multiples of 16, and the internal stride scan peaks at 16 (+0.283) with 32,
|
||
48, 64, 80 and 128 all elevated behind it -- the signature of a record array
|
||
rather than a single coincidental spacing.
|
||
|
||
**Result 2 -- record 0 is a 16-byte header.** Across all 29 packets these bytes
|
||
are constant: 0-3 `04 04 00 01`, 4-11 all zero, 12-13 `00 80`, 15 `6c`. Bytes
|
||
14 and 16-23 vary. The `04 04 00 01` prefix is constant and looks like a magic
|
||
or format version.
|
||
|
||
**Result 3 -- header byte 14 is echoed in the body.** Header byte 14 takes
|
||
values 06, 07, 0e and 11 across packets, and in every packet the same value
|
||
appears at offset 12 of record 1. That correspondence holds on all 29 packets
|
||
sampled, so it is a real repeated field rather than a coincidence of two
|
||
values happening to be small.
|
||
|
||
**Not established.** No header field matches the record count: the packet holds
|
||
1056/16 = 66 records, and none of the candidate u32 or u16 header offsets at 8,
|
||
10, 12, 14, 16, 18 or 20 equals 66, 65 or 67 on more than half the packets. So
|
||
the record count is either computed from a size field elsewhere in the span, or
|
||
the records are variable length and merely 16-byte aligned.
|
||
|
||
That distinction matters and is unresolved: if records are fixed 16 bytes the
|
||
payload is a flat table; if they are variable, the 16-byte alignment is a
|
||
coincidence of every record happening to be a multiple of 16 long. The stride
|
||
scan cannot separate the two, because both produce a peak at 16.
|
||
|
||
**Follow-up: the records are two 8-byte halves, each with a tag byte.**
|
||
|
||
**Range measured:** level 0, the 1,636 sixteen-byte records following the
|
||
header across all 29 packets. Per-byte distinct counts, not pooled.
|
||
|
||
**Method:** count distinct values and zero frequency at each of the 16
|
||
positions; then tabulate the joint distribution of the two low-entropy
|
||
positions.
|
||
|
||
**Result.** Fourteen of the sixteen positions are uniformly varied -- 101 to
|
||
153 distinct values, zero in 13% to 58% of records. Two are not:
|
||
|
||
| byte | distinct | values (count) |
|
||
|---|---|---|
|
||
| 7 | **4** | 0 (1432), 0x30 (135), 0x6d (40), 0x80 (29) |
|
||
| 15 | **2** | 0 (1618), 0x6d (18) |
|
||
|
||
Both sit in the last byte of an 8-byte half, eight bytes apart, and both take
|
||
values from a small vocabulary rather than the full byte range. The joint
|
||
distribution is sparse and shows no case where both are non-zero except
|
||
`0x6d` appearing in either position but never both:
|
||
|
||
(0,0) 1414 (0,0x6d) 18 (0x30,0) 135 (0x6d,0) 40 (0x80,0) 29
|
||
|
||
**Reading.** Each 8-byte half carries a tag or subtype byte in its last
|
||
position, drawn from a small closed set. The shared value `0x6d` in both
|
||
positions is what suggests one vocabulary used twice rather than two unrelated
|
||
byte fields.
|
||
|
||
**What this settles about variable length.** There is **no per-record length
|
||
field**: the best candidate offsets hold the value 16 on at most 13% of
|
||
records, which is the rate small integers appear by chance, not a self-
|
||
describing record. So the records are fixed 16 bytes, and the unresolved
|
||
question from the previous entry is answered -- the tiling is not a coincidence
|
||
of records that happen to be multiples of 16 long.
|
||
|
||
**Still open:** what the four and two tag values mean, and what the 14 varied
|
||
bytes encode. The tag vocabulary is small enough to enumerate against known
|
||
PS2 primitive lists, which is the obvious next reference to bring in.
|
||
|
||
**The retracted VIF reading is not being revived.** Nothing here identifies a
|
||
VIF stream: the 16-byte tiling is uniform across all 29 packets with no
|
||
command-shaped tail, and a VIF packet would show a changing command byte in its
|
||
last position rather than the constant `6c` at header byte 15.
|
||
|
||
## Shrub packet headers: a fixed disc signature, rewritten on load
|
||
|
||
The packets found by the `+0x40` table have a header the whole disc agrees on.
|
||
|
||
**Range measured:** every packet on the disc -- **2,056 of them**, across the
|
||
334 successor-bounded spans -- read at each packet's span-relative offset.
|
||
|
||
**Result:**
|
||
|
||
| bytes | value | packets |
|
||
|---|---|---|
|
||
| 0-3 | `04 04 00 01` | **2056 / 2056** |
|
||
| 4-7 | zero | **2056 / 2056** |
|
||
| 8 | zero | **2056 / 2056** |
|
||
| 11 | `0x05` | **2056 / 2056** |
|
||
|
||
Every packet's size is also a multiple of 16 (2056 / 2056). Zero exceptions on
|
||
any of the four.
|
||
|
||
**And the loader fills the rest in.** Read back out of the running game, the
|
||
same records carry byte 8 as a **count** (0x0b to 0x29 observed), byte 11 as
|
||
one of four tag values, and bytes 12-15 as an absolute EE RAM pointer where the
|
||
disc has a span-relative offset.
|
||
|
||
So **bytes 8-15 are rewritten wholesale on load.** What that means for reading
|
||
the disc: the header carries no count of its own, and the one the RAM copy has
|
||
is the `+0x40` table's `count` arriving by another route. The disc header is a
|
||
signature; the RAM header is populated state.
|
||
|
||
`ShrubPacketHeader::parse` and `Shrub::packet_headers()` expose this, and the
|
||
gated test asserts all four invariants plus the size alignment on every packet
|
||
of every bounded span.
|
||
|
||
## Sky: the sub-object table, and a count field that is not a count
|
||
|
||
**Range measured:** all 27 levels, each sky region from the core index's `sky`
|
||
offset to the next region.
|
||
|
||
**Method:** read the 32-bit offset table at `+0x20`, bounded by staying inside
|
||
the region, and parse each sub-object it names.
|
||
|
||
**Result: 149 sub-objects across 27 levels.** Each opens with a count, then a
|
||
bounding sphere of four floats at `+0x10`, then a texture offset at `+0x20` and
|
||
dimensions at `+0x24`/`+0x26`:
|
||
|
||
+000: 26 00 00 00 | 00 00 00 00 00 00 00 00 00 00 00 00
|
||
+010: e0 8d b2 c1 | 9f d6 f6 3d | ef fc fb be | 1c 61 4e 41
|
||
+020: 90 99 05 00 | 1d 00 | 1d 00 | 00 00 e8 00 ...
|
||
|
||
Count 1 to 38, width 3 to 81, radius 1.95 to 14.56, and the radius is positive
|
||
on **149 of 149** -- the same sphere shape and the same reasoning as the shrub
|
||
class header's.
|
||
|
||
**Correction to an earlier claim.** The word at `+0x0c` is **not** the
|
||
sub-object count. It reads 5 on level 0 where the table holds 6 entries, 3 on
|
||
level 1 against 4, 7 on level 3 against 8, and 196,617 on level 2 against 4.
|
||
Across the disc `offsets.len() == count + 1` holds on **10 of 27** levels. The
|
||
field is reported as `SkyHeader::field_0c` and nothing depends on it; the
|
||
offset table enumerates the sub-objects and is self-limiting.
|
||
|
||
That earlier claim came from level 0 alone, where 5 against 6 looked like a
|
||
consistent off-by-one. It is not an off-by-one -- it is a field that happens to
|
||
be small on most levels and is not a count at all.
|
||
|
||
`gc_formats::sky` parses the header and the sub-objects, and the gated test
|
||
asserts the sphere is finite, the radius positive where it is positive at all,
|
||
the dimensions non-zero, and the sub-object list non-empty on every level.
|
||
|
||
## The collision header is two fields, plus a third on five levels
|
||
|
||
**Range measured:** all 27 levels that carry collision, reading the 0x40-byte
|
||
header each.
|
||
|
||
**Method:** read `+0x00`, `+0x04` and `+0x08`, then check `+0x08..+0x40` for
|
||
zero.
|
||
|
||
**Result:**
|
||
|
||
- `u32@+0x00` is `0x40` on **27 of 27** -- the header is self-sizing.
|
||
- `u32@+0x04` is the payload size, 20 distinct values across the disc.
|
||
- **Bytes `+0x08..+0x40` are zero on 22 of 27**, but five levels carry a large
|
||
non-zero word at `+0x08`:
|
||
|
||
| level | `+0x08` |
|
||
|---|---|
|
||
| 7 | 1,248,048 |
|
||
| 9 | 1,766,832 |
|
||
| 12 | 2,541,616 |
|
||
| 13 | 3,376,592 |
|
||
| 14 | 4,060,640 |
|
||
|
||
So the header is not simply "two fields and 56 bytes of padding" -- there is a
|
||
third field that most levels leave at zero and five use. Whether it is a size,
|
||
an offset or a second block's length is not established; nothing measured here
|
||
separates those readings. It is exposed as `CollisionHeader::field_08`.
|
||
|
||
**Correction to the framing description.** The collision header was previously
|
||
described as self-describing across its full 0x40, which is true for 22 levels
|
||
and overstates the case for the other five.
|
||
|
||
**Still not decoded:** the payload. It contains ascending `u32` runs on levels
|
||
0 and 2 -- level 0 at `+280` with deltas 12, 12, 12 then a jump, level 2 at
|
||
`+13868` with deltas 204, 204, 200, 196 -- but the irregular deltas fit neither
|
||
a flat table nor a simple stride, and level 1's apparent run is a byte-shift
|
||
false positive. Nothing about the payload is asserted.
|
||
|
||
## data.bin is the level's file descriptor
|
||
|
||
`data.bin` is the largest per-level file -- 13.3 MB on level 0, 4.2 to 21.9 MB
|
||
across the disc -- and had been characterised only as "4-byte structured".
|
||
|
||
**Range measured:** the first 64 `u32` of `data.bin` on levels 0, 1, 2 and 5,
|
||
compared against the actual byte lengths of the other extracted level files.
|
||
|
||
**Method:** read the first 64 `u32`, and look for any that equals the size of
|
||
`core_index.bin`, `gs_ram.bin`, `overlay.bin`, `gameplay.bin`, `sound_bank.bin`
|
||
or `core_data.bin`.
|
||
|
||
**Result: three exact matches, at the same indices on every level.**
|
||
|
||
| u32 index | field | level 0 | level 1 | level 2 | level 5 |
|
||
|---|---|---|---|---|---|
|
||
| 1 | `overlay.bin` size | 2,751,268 | 2,763,668 | 2,928,424 | 2,863,676 |
|
||
| 3 | `core_index.bin` size | 31,600 | 34,016 | 33,808 | 23,536 |
|
||
| 5 | `gs_ram.bin` size | 518,656 | 569,344 | 513,024 | 434,176 |
|
||
|
||
Each is the *exact* byte length of the named file, and each lands on the same
|
||
index on all four levels. Three coincidences at fixed positions is not a
|
||
coincidence.
|
||
|
||
**So `data.bin` is a descriptor for the level's other files**, not geometry in
|
||
its own right -- which is what the roadmap's "the large one: 4.2 to 21.9 MB of
|
||
geometry per level" entry assumed. The intervening indices (0, 2, 4) hold
|
||
values that are not file sizes, and their meaning is not established; they are
|
||
plausibly offsets, but no cumulative-sum or offset+size relation has been
|
||
checked yet.
|
||
|
||
**This retires a wrong roadmap entry and opens a question.** `data.bin` should
|
||
not be listed as unparsed geometry. What it contains *after* the descriptor is
|
||
still unknown, and the file is large enough that the descriptor is unlikely to
|
||
be all of it.
|
||
|
||
## Tie spans: a constant first word, and three quantized floats
|
||
|
||
**Range measured:** all **1,864 tie spans across 27 levels**, read at each
|
||
class entry's geometry offset.
|
||
|
||
**Method:** read `+0x00`, `+0x0c` and the three `f32` at `+0x10`, `+0x14`,
|
||
`+0x18`, and count distinct values per position.
|
||
|
||
**Result:**
|
||
|
||
- `u32@+0x00` takes **exactly one value across all 1,864 spans** -- `0x00000100`,
|
||
256. A constant first word across the whole disc.
|
||
- `+0x0c` takes **83 distinct values**, so it is not constant.
|
||
- The three floats take **103 distinct triples** built from **42 distinct
|
||
values**, and those values are round: 4, 8, 9, 10, 12, 15, 16, 18, 20, 24,
|
||
25, 30 ... Only 10 of the 42 are exact powers of two.
|
||
|
||
Two spans on level 0 open:
|
||
|
||
+000 00000100 00005070 000065c0 07050612
|
||
+010 41400000 41c80000 42200000 00009460
|
||
|
||
and the same second, different first-two:
|
||
|
||
+000 00000100 00007ca0 0000a040 0605091e
|
||
+010 41400000 41c80000 42200000 0000ddf0
|
||
|
||
**What this rules out.** The three floats are not a general bounding box: a box
|
||
would take continuous values, and these come from a 42-value set of round
|
||
numbers. They are authored or quantised quantities -- a grid, a size class, or a
|
||
set of LOD thresholds -- and nothing measured here separates those.
|
||
|
||
**Correction to an earlier reading.** These floats were previously read as a
|
||
bounding-box minimum, from level 0's first two spans happening to share
|
||
(12.0, 25.0, 40.0). That triple holds on 266 of 1,864 spans, not all of them,
|
||
and the value set is quantised. The bounding-box reading does not survive.
|
||
|
||
**Not decoded:** what the three floats are, what `0x100` at `+0x00` counts,
|
||
and what the 83 values at `+0x0c` are.
|
||
|
||
## Tie geometry: no GS packet framing, and a control that nearly fooled me
|
||
|
||
**Range measured:** level 0, the 73 tie span bodies (bytes `+0x40` onward),
|
||
**2,585,856 bytes** pooled. Per-position distinct counts, an autocorrelation
|
||
scan against a shuffled control, and a GS GIF tag test.
|
||
|
||
**Result 1 -- the bodies are dense.** Every one of the 16 slot positions takes
|
||
245 to 256 distinct values, with byte 15 zero on 81.3% and byte 7 zero on 78.8%.
|
||
The best stride peak is 48 at +0.097, barely above 16 at +0.095 -- no period.
|
||
|
||
**Result 2 -- no GS GIF framing at 16-byte alignment.** Testing whether byte 15's
|
||
low three bits fall in 0..4, the GS tag range, gives **97.0%**.
|
||
|
||
That looked like a strong hit until the control was done properly. A uniform
|
||
shuffle gives **62.6%** -- which is exactly 5/8, what a random byte predicts --
|
||
so the naive comparison reads as a 34-point excess.
|
||
|
||
**The distribution-matched null is 97.0%, identical to the observation.** The
|
||
whole "excess" was that byte 15 is 81% zero, and `(0 & 7) <= 4` is always true.
|
||
Recomputed per alignment, the null tracks the observation everywhere:
|
||
|
||
| align | matched null |
|
||
|---|---|
|
||
| 0 | 97.0% |
|
||
| 2 | 75.5% |
|
||
| 4 | 75.7% |
|
||
| 8 | 96.7% |
|
||
| 12 | 89.3% |
|
||
|
||
**This is the control rule earning its place a second time.** A uniform shuffle
|
||
is not a matched control when the data is skewed, and `AGENTS.md` already says
|
||
so; the near-miss here was reading the 62.6% as a baseline without checking
|
||
what the data's own distribution predicts. The tie bytes are concentrated on
|
||
0, 1, 2, 3 and 63 -- suggestive in isolation -- and still carry no framing.
|
||
|
||
**What this rules out.** The tie bodies are not GS GIF packet streams at 16-byte
|
||
alignment, and they carry no measurable periodicity. With every slot position
|
||
taking ~256 distinct values they are dense data -- fat vertices, as the roadmap
|
||
expects -- and locating the vertex layout inside them needs a reference for this
|
||
game's instanced-geometry format, not another stride scan.
|
||
|
||
## The collision payload opens with two counts
|
||
|
||
**Range measured:** the first four bytes of the payload on all **19** levels
|
||
that carry a non-empty one.
|
||
|
||
**Result:** two `u16` at payload `+0` and `+2`, both non-zero on all 19, ranging
|
||
1..74 and 13..112, products 53 to 3,256.
|
||
|
||
| level | first | second | product | payload bytes |
|
||
|---|---|---|---|---|
|
||
| 0 | 11 | 33 | 363 | 903,248 |
|
||
| 1 | 24 | 16 | 384 | 2,800,496 |
|
||
| 2 | 1 | 53 | 53 | 2,315,296 |
|
||
| 7 | 40 | 35 | 1,400 | 4,049,232 |
|
||
| 9 | 74 | 44 | 3,256 | 2,518,352 |
|
||
| 13 | 2 | 64 | 128 | 3,333,520 |
|
||
| 26 | 27 | 112 | 3,024 | 652,176 |
|
||
|
||
**Reported as two counts, not as a grid.** They have the shape of collision
|
||
grid dimensions -- bounded, both non-zero, products in the hundreds to
|
||
thousands -- but nothing measured ties the product to the payload size or to any
|
||
per-cell stride. Level 2's `1 x 53` giving a 2.3 MB payload and level 26's
|
||
`27 x 112` giving 652 KB shows the product does not size the payload, which
|
||
means either cells vary wildly in size or these are not grid dimensions. Naming
|
||
them width and depth would be a reading, not a measurement.
|
||
|
||
**After the two counts** comes a run of `u16` that stays below 8,192 for 30 to
|
||
58 entries depending on level, always an even count, and is not ascending. That
|
||
is where the per-cell table is likely to start, and it is the next thing to
|
||
resolve.
|
||
|
||
Exposed as `CollisionHeader::first` and `::second`.
|
||
|
||
## Collision: a per-cell offset table, located on level 4
|
||
|
||
**Range measured:** longest strictly-ascending `u32` run (delta under 65,536) in
|
||
each level's collision payload, levels 0 to 5.
|
||
|
||
**Level 4 is the clean case.** Grid `3 x 25`, payload 2,660,608 bytes, and at
|
||
payload `+5932` a run of 19 entries whose values are all inside the payload,
|
||
starting at 146,012:
|
||
|
||
deltas: 204, 212, 216, 224, 224, 228, 228
|
||
|
||
**The deltas grow monotonically.** That is the signature of a prefix sum over
|
||
records of increasing size, which is what a per-cell offset table looks like when
|
||
cell contents vary.
|
||
|
||
**The others are not.**
|
||
|
||
| level | grid | run at | entries | first deltas | verdict |
|
||
|---|---|---|---|---|---|
|
||
| 0 | 11x33 | +21232 | 17 | 8193, 24575, 8192, 8192 | multiples of 8,192 — coincidental on dense data |
|
||
| 1 | 24x16 | +41452 | 29 | 32771, 57339, 24575 | wild, no trend |
|
||
| 2 | 1x53 | +13752 | 49 | 24, 64, 396, 104 | in-region but irregular |
|
||
| 3 | 19x31 | +667152 | 18 | 769, 257, 257, 257 | values ~5.2e8, far outside the payload |
|
||
| 4 | **3x25** | **+5932** | **19** | **204, 212, 216, 224** | **growing, in-region** |
|
||
|
||
Only level 4's run has both properties a prefix sum needs: values inside the
|
||
payload, and deltas that grow.
|
||
|
||
**What this does and does not give.** Level 4's run is 19 entries against 75
|
||
grid cells, so it is not one offset per cell -- either most cells share an
|
||
offset, are empty, or the table covers something other than the grid. Nothing
|
||
measured here picks between those.
|
||
|
||
**Followed up.** `+5932` is the table's *start*, not a point inside it: walking
|
||
backwards, no predecessor satisfies "ascending with a delta under 4,096". The
|
||
run is **17 entries**, not 19 -- the earlier 19 counted two entries past the
|
||
jump.
|
||
|
||
The deltas are
|
||
|
||
204 212 216 224 224 228 228 228 224 220 216 200 96 96 100 96
|
||
|
||
then a jump of 41,484 to 190,508. So the deltas **rise and then fall** -- a
|
||
symmetric profile, not a monotone one. That weakens the prefix-sum reading
|
||
somewhat: a cumulative offset table has no reason to come back down, whereas
|
||
cell contents that grow and shrink as the table walks across a level would.
|
||
|
||
**17 against a 75-cell grid, and 75 - 17 = 58.** No arithmetic relating the two
|
||
has been found. The earlier guess that 19 related to `3 x 25` did not survive
|
||
re-measurement and should not be carried forward.
|
||
|
||
**Still open.** 17 entries may not correspond to the grid at all -- the table may
|
||
cover something else, or cover a subset of cells. Resolving it needs either the
|
||
same offset found in other levels at a position the leading counts predict, or a
|
||
reason for 17. Neither exists yet, and this is the point at which the
|
||
measurement trail goes cold on collision.
|
||
|
||
## overlay.bin: named sections and a 16-aligned debug string table
|
||
|
||
`level_overlay` was on the open list with no parser and no located structure. It
|
||
now has one: a header naming two sections, and a string table.
|
||
|
||
**Range measured:** `overlay.bin` on levels 0, 1 and 2 -- 2,751,268,
|
||
2,763,668 and 2,928,424 bytes.
|
||
|
||
**Result 1 -- the file opens with section names.**
|
||
|
||
+000: 80 7c 1a 00 | dc 99 00 00 | 01 00 00 00 | 88 3b 2a 00
|
||
+010: 00 00 00 00 | 00 00 00 00 | "crap\n" | 00 00 00 00 00
|
||
+020: "system misc" 00 00 00 00 00 00 00 00 00
|
||
+030: 20 80 ff 40 | 00 60 cf 00 | 64 00 | 46 00 | 19 00 00 00
|
||
|
||
`u32@+0x00` is **1,735,808 on all three levels**, `+0x08` is `1`, and the bytes
|
||
at `+0x18` and `+0x20` spell **"crap"** and **"system misc"**.
|
||
|
||
**Result 2 -- a debug string pool, 16-byte aligned.** 332 printable runs on
|
||
level 0, 362 on level 1, 351 on level 2, the first at `+0x20` and the last
|
||
between 1.3 MB and 2.5 MB. They are the game's own debug text:
|
||
|
||
host0:r:/rc2/shock/wavetable.bin
|
||
host0:z:/i5/shock/wavetable1.bin
|
||
unloaded sound bank %#010x
|
||
Camera_CollPrimTest WARNING! - grid out of bounds! (RAR)
|
||
render setup
|
||
sky draw
|
||
pre effects
|
||
pre 2 effects
|
||
|
||
**`Camera_CollPrimTest WARNING! - grid out of bounds! (RAR)` is direct evidence
|
||
for the collision reading.** It is the game's own string, naming a grid and a
|
||
collision test, and it is why the two leading counts in the collision payload
|
||
are worth continuing to treat as grid dimensions despite not sizing the payload.
|
||
|
||
The `wavetable.bin` paths also confirm the sound bank is loaded from named
|
||
files, which bears on the `SBlk`/`DAW` framing.
|
||
|
||
**Correction to the framing above.** This was first written up as a *string
|
||
table* of records. It is not. Dumping the 32 bytes before each run shows most of
|
||
them are the tail of the previous string:
|
||
|
||
+3104 "host0:r:/rc2/shock/wavetable.bin" (32 chars)
|
||
+3120 "ck/wavetable.bin" -- its own tail
|
||
+3152 "/i5/shock/wavetable1.bin"
|
||
+3168 "ble1.bin" -- its own tail
|
||
|
||
and the 32-byte prefixes are **327 distinct out of 332**, i.e. essentially all
|
||
different, because they are just the preceding string's bytes. There is no
|
||
per-record header here. It is one contiguous pool of debug text, and the
|
||
16-byte alignment is a property of how the pool was laid out, which makes
|
||
continuations land on boundaries and read as separate entries.
|
||
|
||
**Not done.** The pool is located and counted but not parsed, and there is no
|
||
`gc_formats` module for it. The `+0x18`/`+0x20` names are section names but
|
||
their structure is not established, and it is not yet shown that they index
|
||
anything.
|
||
|
||
## gameplay.bin is a flat pointer table
|
||
|
||
**Range measured:** the first 40 `u32` and the whole file, on levels 0 and 1 --
|
||
1,872,256 and 2,006,400 bytes.
|
||
|
||
**Result:** the file is a flat array of `u32`, opening with a small count and
|
||
then a run of ascending offsets, and **terminating on a zero**:
|
||
|
||
level 0: 336 797472 798832 799136 464 150672 150688 314800 479920 ...
|
||
... 1847600 1845456 797808 1829056 160 1849088 1249808 1847616 0
|
||
|
||
level 1: 496 799904 800128 800432 624 151152 151168 315680 481456 ...
|
||
... 1984496 1982400 800112 1966000 160 1988160 1303456 1984512 0
|
||
|
||
**Both levels carry the same shape.** The same sentinel value, `160`, appears at
|
||
`u32` index 34 on both, and the zero terminator at index 38 on both, with counts
|
||
336 against 496 and 464 against 624 at the head. So the leading ~38 words are a
|
||
structured header whose layout is level-independent apart from its sizes.
|
||
|
||
The offsets cluster into groups -- near 800,000, near 150,000, near 1,050,000,
|
||
near 1,800,000 -- which is what a table of pointers into several buffers would
|
||
look like.
|
||
|
||
**Not decoded.** What the pointers point into, what `160` is, and why the zero
|
||
terminator sits at the same index on both levels. No `gc_formats` module exists
|
||
for this yet; it is characterised, not parsed.
|
||
|
||
**Where the pointers go.** Following them shows what `gameplay.bin` indexes. The
|
||
leading table is 39 words, all of which land inside the file, and three clusters
|
||
dominate: near 150,000, near 800,000 and near 1,050,000.
|
||
|
||
The region near 797,808 is **position data** -- 31 consecutive plausible floats
|
||
that group cleanly into xyz triples:
|
||
|
||
246.948 201.015 73.867 | 88.240 186.000 302.598 | 107.712 12.723 183.125
|
||
353.591 126.365 19.962 | 182.068 374.001 122.567 | 116.961 265.001 437.866
|
||
|
||
and the region near 797,456 is **direction data** -- unit-length vectors:
|
||
|
||
0.969 0.945 0.733 | -0.120 0.433 0.250 | -0.866 0.000 0.267
|
||
|
||
So the file's leading words are a table of pointers into float arrays, not a
|
||
flat list of small integers. The earlier reading of it as "u32 pointers holding
|
||
nothing" was too quick: it is a pointer table *into data that is present*.
|
||
|
||
**On volumes.** Nothing resembling volumes was found here. `gameplay.bin` is
|
||
u32 pointers throughout and holds no float data in its first 64 words read as
|
||
`f32` (59 of 64 are denormals, i.e. small integers reinterpreted). Volumes remain
|
||
unlocated on the disc -- nothing in the regions examined so far presents itself
|
||
as one.
|
||
|
||
### level_overlay: parsed
|
||
|
||
`level_overlay` now has a module. `gc_formats::overlay` parses a 16-byte header,
|
||
splits out the constant block and the level-specific tail, and a real-disc test
|
||
covers all 27 levels:
|
||
|
||
overlay: 27 levels; field_00 1735808 = 0x1A7C80, field_08 1;
|
||
field_04 37540..=45308, field_0c 2763912..=3027336
|
||
(17 below the file length, 10 above);
|
||
+0x10..0x1220 identical on all
|
||
|
||
Later, from OpenRAC/rac2-gc-decomp (same disc, SHA-256 verified identical):
|
||
|
||
field_00 IS the .lit load base -- 0x1A7C80, constant on all 27 levels
|
||
because every level's literal section loads at the same address.
|
||
overlay.bin IS that section: it holds code and the string pool.
|
||
$gp 0x1A7C80 + section offset is computed against $gp = 0x1AEFF0.
|
||
program seven sections: .lit .bss .data lvl.vtbl lvl.camvtbl
|
||
lvl.sndvtbl .text
|
||
|
||
field_0c tested as the .lit size and FAILED: it is smaller than the file on 16 of
|
||
27 levels, by up to 8.9% (level11 file 3,151,128 vs field_0c 2,870,792). Excluded
|
||
by sign, not by a near miss. Still unnamed.
|
||
|
||
The collision assert string was then used as an anchor and the first attempt was
|
||
WRONG -- a byte-order bug, recorded in core-data-regions.md. The PS2's MIPS is
|
||
little-endian, so an I-type immediate is word bytes 0 and 1; searching bytes 1..3
|
||
matches opcode/$rs instead and manufactured a single convincing "hit". Corrected,
|
||
the count is zero. The anchor is NOT established.
|
||
|
||
**`+0x0c` is not the file length.** It was recorded as one on the strength of the
|
||
first levels measured, where it sat close to the file end. Across all 27 it falls
|
||
on **both sides** -- below on 17, above on 10 -- so it is an ordinary
|
||
level-specific word. The test asserts exactly this, since it is the claim most
|
||
worth pinning.
|
||
|
||
**The common prefix is 4 bytes, not the whole header.** `+0x00` is constant at
|
||
1,735,808, but `+0x04` and `+0x0c` vary, so the prefix stops at `+0x04`.
|
||
Everything from `+0x10` to `+0x1220` -- **4,624 bytes** -- is byte-identical on
|
||
all 27 levels.
|
||
|
||
What is still not named: what any of the four header words *means*, and how the
|
||
level-specific tail is structured. The module reports them as fields.
|
||
|
||
### moby_anim: the instance-table hypothesis is out
|
||
|
||
`moby_anim` was the one post-shrub chunk type with a concrete reason to exist --
|
||
an instance table would have to scale with the level's moby class count, so
|
||
comparing per-type record counts against that count was the obvious test.
|
||
|
||
**It fails, and the way it fails is itself the result: per-type record counts are
|
||
level-independent.** The same types report the same counts on every level tried:
|
||
|
||
type_id level0 level1 level2 level3
|
||
0x007d7ec438 90 90 90 90
|
||
0x00d3d8c4a9 90 90 90 90
|
||
0x00429ec43d 77 77 77 77
|
||
0x0053c8c4aa 77 77 77 77
|
||
0x007626c4a4 50 50 50 50
|
||
0x00fb88c462 47 47 47 47
|
||
|
||
Across all 27 levels there are 77 distinct chunk types, the largest at 2,430
|
||
records total -- which is exactly 90 x 27.
|
||
|
||
**What that rules out.** Nothing in this chunk set is a per-moby-class instance
|
||
table, because those counts would vary by level the way the moby class counts do.
|
||
So whatever holds moby animation data is either keyed by something other than the
|
||
class count, or it is not among these chunk types.
|
||
|
||
**Also visible: the types pair up.** 0x007d7ec438 and 0x00d3d8c4a9 both report 90
|
||
on every level; 0x00429ec43d and 0x0053c8c4aa both report 77. Two chunk types
|
||
carrying identical, constant record counts on every level is a signature of two
|
||
views of the same fixed table -- one indexed one way and one another -- rather
|
||
than two independent tables that happen to agree.
|
||
|
||
**Still open.** The pairings are described but not explained, and no type has been
|
||
named. The Moby class-ID enrichment from earlier stands separately: 9,493 hits
|
||
against 246.1 expected is real, and the class arrays are contiguous per type, but
|
||
that identifies *where classes are referenced*, not which chunk is `moby_anim`.
|
||
|
||
|
||
## The tie control region is not a tag stream (negative result)
|
||
|
||
Went looking for the GS's GIF packets, which the tie packet header calls AD-GIF
|
||
(`shader_count` at `+0x04`, and `bfc_distance == 5 * shader_count` at `+0x05`).
|
||
Tried the first `5 * shader_count` quadwords of each packet as VIF tags (command
|
||
in bits 0-6, size in 7-14, mark in 15) and as GIF tags (type in the top byte of
|
||
word 1, length in bits 0-2 of word 0).
|
||
|
||
**Both came out uniform.** Over 13,042 records, 10,443 distinct values of word 0
|
||
and no GIF tag type above a count of 2. By rule 4 that is the control result:
|
||
the region is not a tag stream, and no further effort should go into decoding it
|
||
as one.
|
||
|
||
The first version of that probe was wrong and worth recording, because it reached
|
||
the *opposite* conclusion twice. It scanned `bfc_distance` quadwords, which for
|
||
most records overruns `vert_ofs` and runs into the vertex data. Pooling a
|
||
control region with vertex positions -- two populations with different
|
||
structure -- produced the uniform answer, and "not a tag stream" was the right
|
||
conclusion for the wrong reason. Re-measuring quadword 0 alone on its own gives
|
||
the same uniformity, so the conclusion survives; the reasoning did not.
|
||
|
||
What the probe did establish is the **packet partition**:
|
||
|
||
q[0 .. vert_ofs) control 3..8 quadwords (peak 5)
|
||
q[vert_ofs .. +vert_size) vertices
|
||
q[vert_ofs+vert_size ..] scissor
|
||
|
||
and a new **universal invariant**: `scissor_ofs == vert_ofs + vert_size` on
|
||
**15,219 of 15,219** records across all 27 levels, including every record whose
|
||
`strip_count` is non-zero. That is a derived quantity stored explicitly, and it
|
||
holds everywhere -- which is the check that confirms the three regions really are
|
||
laid out as the packet-length formula assumed.
|
||
|
||
It also found `vert_ofs` mis-named. It is not a vertex offset; it is the *length*
|
||
of the leading control region. The vertex data begins at that many quadwords into
|
||
the packet. The field name is left alone because the on-disc order is fixed and
|
||
everything already written about this record says `vert_ofs`, but the doc now
|
||
says what the value is.
|
||
|
||
**What the control region is:** per-record, not shared. 12,294 distinct blobs
|
||
across 13,042 records, mixed within 2,410 of 3,549 classes, and different between
|
||
LOD 0 and LOD 1 on all 500 classes that have both. It varies per primitive.
|
||
|
||
Partitioned by its own length it is *not* uniform in character -- the pooled
|
||
count hid this:
|
||
|
||
vert_ofs 48 bytes (3 qw) 5 distinct blobs
|
||
vert_ofs 64 bytes (4 qw) 3,175
|
||
vert_ofs 80 bytes (5 qw) 4,309
|
||
vert_ofs 96 bytes (6 qw) 3,281
|
||
vert_ofs 112 bytes (7 qw) 1,523
|
||
vert_ofs 128 bytes (8 qw) 1
|
||
|
||
The shortest and longest are effectively constant -- five blobs and one -- and
|
||
everything in between is per-record unique. That is a shape, not a noise floor.
|
||
|
||
## bfc_distance is not the control region's length
|
||
|
||
The other half of the answer. `bfc_distance == 5 * shader_count` holds on every
|
||
record and has always been documented as "quadwords of ad-gif data" -- which, read
|
||
as a length within the packet, would make it the control region's size.
|
||
|
||
It cannot be. `bfc_distance` is 5..20 quadwords; `vert_ofs`, the control region,
|
||
is 3..8.
|
||
|
||
Pooled, the two overlap on 9,315 of 15,219 records (61%) -- a number describing
|
||
nothing. Partitioned by `shader_count`, as rule 3a requires, it is exact:
|
||
|
||
shader_count 1: 2,772 of 8,676 exceed (32%)
|
||
shader_count 2: 4,649 of 4,649 exceed (100%)
|
||
shader_count 3: 1,729 of 1,729 exceed (100%)
|
||
shader_count 4: 165 of 165 exceed (100%)
|
||
|
||
For two or more shaders it exceeds on every record, without exception. For one
|
||
shader it exceeds exactly when `vert_ofs < 5`, which is the 32%.
|
||
|
||
**So the AD-GIF is not stored in the tie packet.** The packet is control,
|
||
vertices, scissor. `bfc_distance` is a per-record quantity relating to the shader
|
||
count whose meaning is not established, and the "quadwords of ad-gif data" reading
|
||
of it is wrong. The field doc said so in terms before this measurement; it does
|
||
not now.
|
||
|
||
|
||
## Where the AD-GIF is not: class-header offsets 0x1c and 0x34
|
||
|
||
Two words in the class header's undecoded tail are offsets, and they are the
|
||
best remaining lead on the AD-GIF.
|
||
|
||
`base + 0x1c` and `base + 0x34` land inside `core_data` on **1,864 of 1,864
|
||
classes**, so both are relative to the class base rather than absolute, and both
|
||
are non-zero on every class. And both land **past** that class's own tie packet
|
||
blocks on every class -- they address per-class data stored somewhere else in
|
||
core_data, not the tie geometry.
|
||
|
||
What is there is structured, not random. Class 0 at 0x1c:
|
||
|
||
54 5d 5f 0c 44 44 44 01 03 11 19 00 1c 00 60 0a 30 08 60 0a
|
||
ff 00 ff ff 20 0c 50 02 44 38 2b 50
|
||
|
||
and class 3 at the same field:
|
||
|
||
20 05 c5 00 00 80 01 00 26 05 cb 00 21 88 01 00 2c 05 c0 00
|
||
00 80 01 00 32 05 c0 00 00 80 01 00
|
||
|
||
Class 3 is a fixed-stride table with `80 01 00` recurring. Class 5's 0x34 field
|
||
has `38 38 38` and `ff 00 ff ff`. Class 2's 0x1c field is a monotone byte table:
|
||
`26 27 0e 0f 10 11 12 13 24 25 15 1e 17 1f 14 15 16 17 20 21 22 23`.
|
||
|
||
These read like compiled shader programs with small-integer parameters, or as
|
||
lookup tables. **They are not identified**, and the resemblance to anything
|
||
specific is exactly the kind of finding rule 4 exists to refuse -- none of these
|
||
three fragments has been shown to beat a control, and they were picked because
|
||
they looked like something.
|
||
|
||
## The control, and a retraction
|
||
|
||
The last paragraph of the previous section implied the two offsets select
|
||
*distinctive* content. They do not.
|
||
|
||
Measured adjacent-equal-byte rate over 64 bytes, 1,864 classes:
|
||
|
||
0x1c 0.12688
|
||
0x34 0.12106
|
||
control 0.14109 (the same offset + 0x2000)
|
||
random 0.00391
|
||
|
||
All three are ~30x above the random baseline, so the region really is structured
|
||
data and not noise -- that part stands. But **0x1c and 0x34 are no more
|
||
structured than an arbitrary offset in the same blob**, and 0x34 is slightly
|
||
*less*. So the hex fragments in the previous section were three hand-picked
|
||
examples from a structured region, and "they look like shader programs with
|
||
small-integer parameters" said something about this part of core_data, not about
|
||
what these fields address.
|
||
|
||
**Retracted:** the suggestion that the offset targets are identifiable content.
|
||
Still established, and independent of all of the above: the offsets are
|
||
base-relative, non-zero, in range on 1,864 of 1,864 classes, and always land past
|
||
the class's own tie packet blocks. What they address remains unknown, and the
|
||
earlier finding that the packet's control region is not a tag stream still stands
|
||
-- it was measured by uniformity within the packet, not by comparison to
|
||
somewhere else in core_data.
|
||
|
||
|
||
## moby_anim: the class table is not the loader (negative result)
|
||
|
||
`moby_anim` needs the moby class loader. The obvious place to look for a loader
|
||
is the moby class table, since a loader is usually dispatched from one. Checked
|
||
that first and it is ruled out.
|
||
|
||
`moby_classes` is an `ArrayRange`: offset `0xc0`, count 238, in level 0. The
|
||
words there are not pointers. Read as floats they are
|
||
|
||
0x4846a9e3 0x47f40000 0x46d0b0de -> ~50,986 61,000 26,671
|
||
|
||
which is position/scale-shaped data. And scanning every u32 in the first
|
||
`0x4000` bytes for values landing inside the overlay's `.text`
|
||
(`0x2a3b80..0x447900`) gives **120 hits out of 4,096**, against roughly **259**
|
||
expected if the values were unrelated to the region.
|
||
|
||
So the table holds *fewer* .text-range values than chance, not more. Whatever the
|
||
moby class loader is, it is not dispatched from a pointer in this table.
|
||
|
||
That rules out the cheap approach. Finding the loader means cross-referencing
|
||
from somewhere else -- most likely a constructor that reads the class table and
|
||
returns runtime state, which is in `.text` and has to be found by scanning for
|
||
code that touches both `moby_classes.offset` and the class spans. That is a
|
||
larger search than a bounded probe, and it is the unblock condition already
|
||
recorded for `moby_anim`.
|
||
|
||
Not concluded: what the 238 entries *are*. The float reading is suggestive, not
|
||
established -- it was read at one stride, on one level, and rule 3a applies.
|
||
|
||
### Cross-reference scan by raw immediate: does not discriminate
|
||
|
||
The natural next probe is to search `.text` for immediates matching each chunk's
|
||
offset. Run over all 27 levels (0x2d70260 bytes, 23.8M u16 values):
|
||
|
||
moby_classes 0x00c0 49,230 occurrences
|
||
tie_classes 0x1e80 111 occurrences
|
||
|
||
**This comparison is meaningless and must not be quoted as evidence.** The two
|
||
offsets differ in magnitude, and small values are simply common: the single most
|
||
frequent u16 in `.text` occurs 1,834,535 times, so a two-byte value that is
|
||
common by construction will beat a rarer one no matter what it references. The
|
||
whole scan is confounded by value frequency, which is rule 3a again, one level
|
||
below where it has bitten before -- not the measurement being wrong but the
|
||
*comparison* having no shared denominator.
|
||
|
||
Doing it properly means counting immediates only where they are immediates.
|
||
|
||
**Done, and it needs no disassembler** -- filter on opcode and take bits 15..0.
|
||
Counting only instructions that carry an immediate (`addiu`, `lui`, `ori`,
|
||
`xori`, `andi`, `sltiu`) across all 27 levels: 11,911,320 instructions,
|
||
2,985,897 with an immediate, 13,813 distinct immediate values.
|
||
|
||
moby_classes 0x00c0 17,699
|
||
tie_classes 0x1e80 17
|
||
|
||
Now the comparison has a denominator, and it has a control that the previous
|
||
version lacked: the most frequent immediates in the whole program are
|
||
|
||
0x0001 200,784 0x0010 137,523 0x0004 57,248
|
||
0x001b 149,259 0xffff 98,312 0x0002 64,450
|
||
|
||
So 0xc0 at 17,699 is *below* the top of the small-value cluster rather than
|
||
distinctive. The previous commit's "this method cannot tell" was right, and the
|
||
reason was worth one more step rather than a hand-wave: the method was fixable by
|
||
filtering on opcode, which costs nothing and needs no capstone.
|
||
|
||
**Result: still no signal.** Nothing in the immediate stream points at
|
||
`moby_classes` more than at the other small offsets the program uses. The loader
|
||
is not findable by frequency.
|
||
|
||
### Mapping the sites to functions: 410 candidates, which is not a narrowing
|
||
|
||
The next step the decomp makes cheap. `functions.tsv` gives 3,835 level-0
|
||
functions with start and size, so each `0xc0` immediate-bearing instruction can be
|
||
attributed to its containing function.
|
||
|
||
**410 of 3,835 functions (10.7%) contain one.** Sizes run 20 bytes (extent
|
||
tail, effectively a mis-bound) to 34,316, median 808.
|
||
|
||
Ten per cent is what a common small immediate buys, so this confirms the previous
|
||
commit rather than narrowing anything -- 410 candidates is not a shortlist. What
|
||
it does establish is that the attribution machinery works and that function
|
||
extents from the decomp are usable for exactly this.
|
||
|
||
The loader hunt now has a concrete next step rather than a vague one: of those
|
||
410, find the ones that *also* touch a class span -- a tie or moby class base --
|
||
and those are the constructors. That is dataflow rather than counting, and it is
|
||
where this should resume.
|
||
|
||
Note the one caution this measurement earns: a 20-byte "function" containing the
|
||
immediate is not a real candidate, it is a function whose extent ends just after
|
||
some other function's instruction. Sizes need a floor before the set is used.
|
||
|
||
|
||
## moby_classes.offset is a load-bearing constant
|
||
|
||
First positive result of the loader hunt, and it turns entirely on the control.
|
||
|
||
Across 2,323 level-0 functions of at least 128 bytes (extents from the decomp's
|
||
`functions.tsv`), the immediate `0x00c0` -- `moby_classes.offset` -- appears in
|
||
|
||
**394 functions**
|
||
|
||
and every value immediately adjacent to it appears in 1 to 14:
|
||
|
||
0xb8:8 0xb9:2 0xba:2 0xbb:2 0xbc:3 0xbd:5 0xbe:5 0xbf:3
|
||
0xc1:3 0xc2:3 0xc3:2 0xc4:1 0xc5:2 0xc6:3 0xc7:1 0xc8:14
|
||
|
||
That is the control 27891ce said was missing. `0xc0` is small and small
|
||
immediates are common, so its raw frequency proved nothing -- 17,699 occurrences
|
||
across the program, with `0x0001` appearing 200,784 times. The neighbours have
|
||
the *same magnitude* and none of the frequency, which is what makes 394 mean
|
||
something instead of being an artifact of the value being short.
|
||
|
||
Against the five other level-0 chunk offsets used as a second control, all of
|
||
which are larger and so would be biased in the *other* direction:
|
||
|
||
shrub_classes 0x27c0 1 tfrag_textures 0x2910 2
|
||
moby_textures 0x2ee0 2 tie_textures 0x3960 0
|
||
shrub_textures 0x3e20 2
|
||
|
||
So the chunk offsets the loader would use are referenced about once each, and
|
||
`moby_classes` is referenced 394 times. The program uses this constant heavily.
|
||
|
||
**What this does not say.** An immediate appearing in a function is not proof the
|
||
chunk is *read* there -- it is proof the constant is used, which is the claim
|
||
being asserted and all the test asserts. It does not say which of the 394 is the
|
||
loader, or that the loader is among them.
|
||
|
||
**Also measured, and negative:** zero functions contain both the `moby_classes`
|
||
and `tie_classes` offsets, so there is no single function that indexes both chunk
|
||
tables by immediate. The moby loader, if it exists in this form, does not reach
|
||
the tie table the same way.
|
||
|
||
### Ranking by size does not narrow, and biases toward false positives
|
||
|
||
The obvious next move was to rank the 394 largest-first and start with the
|
||
biggest. Ranked, over 2,323 functions, median candidate size 852:
|
||
|
||
0x2ba7d8 34,316 bytes 631 distinct immediates
|
||
0x2cb3a8 26,432 421
|
||
0x300618 11,348 155
|
||
0x3ae900 8,440 208
|
||
... median 852
|
||
|
||
**This ranking is probably backwards.** A function with 631 distinct immediates
|
||
is far more likely to contain any given value -- including `0xc0` -- than one with
|
||
80, purely by pigeonhole. Sorting by size therefore selects *for* the very
|
||
functions most likely to be false positives, and the 34,316-byte outlier is
|
||
probably a mis-bounded extent (the decomp corrects three level-00 entries, per
|
||
`docs/LEVELS.md`) rather than a 34 KB loader.
|
||
|
||
So the 394 needs a size-independent statistic before it is a shortlist. The right
|
||
measure is whether a function contains `0xc0` **more often than its immediate
|
||
count predicts** -- expected count from the neighbourhood rate, observed count
|
||
inside that function -- rather than whether it contains it at all. That has not
|
||
been measured.
|
||
|
||
Recorded rather than claimed as progress: ranking by size is the wrong first cut,
|
||
and the reason is a confound that would have made the result look better than it
|
||
was.
|
||
|
||
### The size-independent statistic: the 394 does not survive
|
||
|
||
Run it, and the confound db1e69c predicted is confirmed. Bucketing every level-0
|
||
function by how many distinct immediates it carries, and taking the rate at which
|
||
each bucket contains `0xc0`:
|
||
|
||
distinct immediates with 0xc0 / total rate
|
||
0- 15 72 / 1,082 6.7%
|
||
16- 31 119 / 713 16.7%
|
||
32- 47 104 / 291 35.7%
|
||
48- 63 39 / 119 32.8%
|
||
64- 79 31 / 59 52.5%
|
||
80- 95 13 / 25 52.0%
|
||
112- 127 3 / 7 42.9%
|
||
624+ 1 / 1 100%
|
||
|
||
The rate rises monotonically with set size, from 6.7% to over 50%. That is
|
||
pigeonhole: a function with hundreds of distinct immediates has more chances to
|
||
contain any particular value. **The 394 is mostly a list of large functions**, and
|
||
"394 of 2,323" was never a shortlist. db1e69c was right and 3ea5ab7's framing was
|
||
wrong.
|
||
|
||
This does **not** retract the neighbour result. `0xc0` being enriched over
|
||
0xb8-0xc8 and `0xc0` being enriched in bigger functions are two different claims
|
||
and both can hold. What it retracts is the inference that the 394 are candidates
|
||
for anything.
|
||
|
||
**The control that would settle it:** `0xc0`'s rate against the best of its
|
||
sixteen neighbours, **within the same set-size bucket**.
|
||
|
||
bucket n 0xc0 best neighbour ratio
|
||
0-15 1,082 72 0xb8:1 72.0x
|
||
16-31 713 119 0xc8:4 29.8x
|
||
32-47 291 104 0xb8:3 34.7x
|
||
48-63 119 39 0xb8:2 19.5x
|
||
64-79 59 31 0xc8:2 15.5x
|
||
80-95 25 13 0xb8:1 13.0x
|
||
|
||
**Enriched in every bucket, 13x to 72x, and the ratio does not decay as sets grow.**
|
||
That is the opposite of a pigeonhole artifact -- a volume effect would shrink the
|
||
ratio in the big buckets, and it rises instead.
|
||
|
||
The load-bearing row is the smallest. **72 of 1,082 functions carrying fewer than
|
||
16 distinct immediates contain `0xc0`, where the best neighbour appears once.** A
|
||
set that small cannot produce that by count alone. The constant is genuinely
|
||
special, and the fact is size-independent.
|
||
|
||
**What this revives.** The 300 candidates with 48 or fewer distinct immediates
|
||
are no longer noise -- at matched size, neighbours hit 0.4% to 1.4% in those
|
||
buckets where `0xc0` hits 35%. A function using fewer than 16 constants, one of
|
||
which is `0xc0`, is a function that uses this offset *deliberately*. Those are the
|
||
shortlist, and `moby_classes_offset_enrichment_survives_set_size_matching` is the
|
||
test that will catch them being thrown away again.
|
||
|
||
Still true and still worth saying: presence of an immediate is not proof the chunk
|
||
is read there. It is proof the constant is used, which is all either test asserts.
|
||
|
||
|
||
### The 72 concentrated candidates, and an ambiguity that has to be stated
|
||
|
||
Sorted by fewest distinct immediates, the 72 candidates under 16 immediates look
|
||
like this (first 14, immediate sets truncated):
|
||
|
||
0x2dc3e0 220b 6 imm 0x10 0x20 0x30 0x90 0xc0 0xff70
|
||
0x310748 240b 6 imm 0x10 0x40 0xa0 0xc0 0x4480 0xff60
|
||
0x31ed88 292b 6 imm 0x40 0x70 0x80 0xc0 0x3f80 0xff40
|
||
0x3a8060 148b 6 imm 0x40 0xc0 0xd0 0xe0 0x3f80 0xffc0
|
||
0x2e8de0 696b 8 imm 0x01 0x0f 0xc0 0x0fdc 0x3fc9 0xf000 0xff40 0xffff
|
||
0x3114e0 160b 8 imm 0x01 0x10 0x30 0x40 0x80 0xc0 0x3a80 0xff80
|
||
|
||
Two observations, and the second is a problem.
|
||
|
||
**First: none of these contains another chunk offset.** Across all 394 candidates,
|
||
exactly one contains any other level-0 chunk offset, and none of the 72 does.
|
||
|
||
**Second, and this is why the first is not the finding it looks like:** the other
|
||
chunk offsets are `0x1e80`, `0x27c0`, `0x2910`, `0x2ee0`, `0x3960`, `0x3e20` --
|
||
all above `0xff`, and globally rare (0 to 2 functions each, measured in
|
||
`3ea5ab7`). Their absence here is exactly what rarity predicts and carries no
|
||
information. "Only 1 of 394 references another chunk" sounds like evidence of
|
||
moby-specificity and is not; it restates that those offsets are rare everywhere.
|
||
|
||
**The real ambiguity:** these immediate sets are full of values that look like
|
||
`lui` high halves. `0x3f80` is the high half of `1.0f`. `0xff70`, `0xff40`,
|
||
`0xffc0`, `0xff80` are negative high halves. `0xc0 << 16` is `0x00c00000`, a small
|
||
float. So `0xc0` in these functions may be a **float constant high half**, not a
|
||
chunk offset at all -- and the neighbour control would not catch that, because
|
||
`0xb8` and `0xc8` are equally plausible float high halves and they are rare for
|
||
their own reasons.
|
||
|
||
What survives: `0xc0` is used far more than adjacent constants of the same
|
||
magnitude, at every set size. What does **not** survive is the reading of it as a
|
||
chunk offset. That now has two live explanations -- a moby chunk reference, or a
|
||
float constant -- and only disassembly separates them.
|
||
|
||
`0x3a8060` at 148 bytes with six immediates is the cheapest thing to disassemble
|
||
and would settle it: if the `lui` at that site is paired with an `ori` it is a
|
||
float, and if it feeds an `addiu` computing `base + 0xc0` it is the chunk.
|
||
|
||
### Disassembled: it is an address, not a float
|
||
|
||
Settled. At `0x3a8060`:
|
||
|
||
263200c0 addiu r17, r18, 0x00c0 r17 = r18 + 0xc0
|
||
263000d0 addiu r17, r16, 0x00d0 r17 = r16 + 0xd0
|
||
262400e0 addiu r17, r4, 0x00e0 r17 = r4 + 0xe0
|
||
3c013f80 lui r0, r1, 0x3f80 the actual float constant: 1.0f
|
||
|
||
So `0xc0` is an **address offset**. The float constant is elsewhere in the same
|
||
function and is `0x3f80`, exactly where it should be.
|
||
|
||
And the three offsets are `0xc0`, `0xd0`, `0xe0` -- **16 bytes apart**. The
|
||
function is walking a table of fixed-stride structures, indexing the field at
|
||
offset `0xc0`. That is what a class-table lookup looks like, and it is consistent
|
||
with `moby_classes` starting at `0xc0` in core_data: the chunk's field at its own
|
||
offset, or a struct whose first accessed member sits there.
|
||
|
||
**What this settles:** `0xc0` is an address computation, not a float constant. The
|
||
ambiguity from 05d9bf6 is closed for this function, and by extension the reading
|
||
of `0xc0` as `moby_classes.offset` survives.
|
||
|
||
**What it does not settle:** that `0x3a8060` is the moby class *loader*. It is a
|
||
function that indexes a 16-byte-stride table at offset `0xc0`. That is consistent
|
||
with many things -- a moby class walk, a different structure that also lives at
|
||
`0xc0`, or ordinary field access on an object passed in `r18`. Identifying which
|
||
needs the register provenance: where `r18` comes from, and whether it is
|
||
`core_data + moby_classes.offset`.
|
||
|
||
The next step is therefore narrow and concrete: trace `r18` back to its
|
||
definition. If it is loaded as `core_data + 0xc0`, that is the chunk. The
|
||
preceding instruction at this function's entry sets up a 64-byte frame and saves
|
||
`r16`-`r20`, so the walk is over arguments rather than a global -- which is itself
|
||
informative, because a *loader* would more likely take the chunk as an argument
|
||
from its caller than re-derive it.
|
||
|
||
### Call sites: one caller, and it does not compute base + 0xc0
|
||
|
||
Traced as `cee17c9` proposed. `0x3a8060` has **exactly one call site**, at
|
||
`0x3a7d88`:
|
||
|
||
0x3a7d70 lui r1, 0xbf80
|
||
0x3a7d7c jal ...
|
||
0x3a7d84 addiu r18, r2, 0x202d <-- r18 = r2 + 0x202d
|
||
0x3a7d88 jal 0x3a8060
|
||
|
||
No `0xc0` anywhere in the eight instructions before the call, and `r18` is built
|
||
from `r2 + 0x202d` -- a different offset entirely. So the `0xc0` inside the callee
|
||
is a **field offset applied to a pointer the caller already built**, not a chunk
|
||
offset the caller resolved.
|
||
|
||
Two readings remain, neither settled:
|
||
|
||
- `r18` is `core_data`, making `r18 + 0xc0` equal `&moby_classes`. This fits,
|
||
because level 0's `moby_classes.offset` genuinely is `0xc0`.
|
||
- `0xc0` is a member offset in an unrelated structure and the match is
|
||
coincidence.
|
||
|
||
Settling it needs the provenance of `r2` at the call site. That is real dataflow,
|
||
not counting, and it is where this stops. Six commits of immediate arithmetic
|
||
could not answer it, and the one disassembly that could has now shown the answer
|
||
is not as simple as "the caller loads the chunk".
|
||
|
||
**The lesson, stated plainly so it is not relearned:** a constant's frequency, and
|
||
even its use, say nothing about what it means. Six rounds of counting produced one
|
||
robust fact -- `0xc0` is used far more than adjacent constants -- and not one
|
||
identified function. A single disassembly settled in twelve instructions what six
|
||
counting passes could not, and immediately complicated the answer.
|
||
|
||
### Provenance: r2 is never written, and the enclosing function has no callers
|
||
|
||
The trace `eca1a16` asked for. The call site `0x3a7d88` sits inside the function
|
||
at `0x3a7c30`, 580 bytes, 145 instructions.
|
||
|
||
**`r2` is never written in that function.** Scanning every instruction for one
|
||
that stores into `r2` (`sw` or `sb` with `rs == 2`) finds none across all 580
|
||
bytes. So `r2` at the `jal 0x3a8060` is not defined anywhere in the enclosing
|
||
function -- it arrives from outside.
|
||
|
||
**And `0x3a7c30` has no `jal` callers.** Scanning the whole of level-0 `.text`
|
||
for `jal` instructions targeting it finds zero. It is also **not** a `lvl.vtbl`
|
||
handler: level 0's vtbl has 233 records and 185 distinct handler addresses, and
|
||
neither `0x3a7c30` nor `0x3a8060` is among them.
|
||
|
||
That rules out the tidy story -- "the vtbl dispatcher calls it with the class
|
||
pointer in `v0`" -- which was the obvious guess given `r2` arrives unexplained.
|
||
The remaining possibilities are an indirect call through a pointer (`jalr`), a
|
||
dispatch table in data rather than in the vtbl, or code that is not called at all.
|
||
|
||
Not distinguished. The immediate question for anyone continuing is whether
|
||
`0x3a7c30` is a target of any `jalr` in the level, and if so from where.
|
||
|
||
**This is the end of what counting can do here.** The chain is now:
|
||
`? -> 0x3a7c30 (no static caller) -> 0x3a8060 -> addiu r17, r18, 0xc0`, with `r2`
|
||
supplied by whatever reaches `0x3a7c30`. Every remaining question is about
|
||
control flow and register provenance, not about counts.
|
||
|
||
### Correction to 349b498: there is a caller, and I had a decode bug
|
||
|
||
`349b498` reported "0x3a7c30 has no `jal` callers anywhere in level-0 `.text`".
|
||
**That was wrong, and the bug was mine.** The scan built the branch target as
|
||
|
||
((instr & 0x03ff_ffff) << 2) | 0x0c00_0000
|
||
|
||
without masking to the upper four bits. J-type targets take their top nibble from
|
||
the opcode field, which is zero here, so the correct form is just
|
||
`(instr & 0x03ffffff) << 2`. The un-masked constant added `0x0c000000` to every
|
||
target, so the comparison never matched anything.
|
||
|
||
Corrected scan: **`0x3a7c30` has exactly one `jal` caller, at `0x3a3d84`.** No
|
||
`j` tail-calls, and the function is still not a `lvl.vtbl` handler and still not a
|
||
pointer in any level-0 data file.
|
||
|
||
### r2 is a return value, not a caller argument
|
||
|
||
Tracing backwards from `0x3a3d84` inside the calling function:
|
||
|
||
0x3a3ce4 lw r3, 0x1500(r2) r2 already live, used as a base
|
||
0x3a3d70 jal 0x2ee178
|
||
0x3a3d84 jal 0x3a7c30 r2 is v0, i.e. the *result* of the call above
|
||
|
||
No instruction between the two writes `r2`. It is simply the return value of the
|
||
function at `0x2ee178`, passed straight through as the first argument to
|
||
`0x3a7c30`.
|
||
|
||
So the chain is
|
||
|
||
0x2ee178 --returns a pointer--> 0x3a7c30 --passes it in r2-->
|
||
0x3a8060 --addiu r17, r18, 0xc0--> 16-byte-stride walk
|
||
|
||
**The next question is now one function:** does `0x2ee178` return
|
||
`core_data + moby_classes.offset`? It is a small, self-contained function and it is
|
||
the only thing between the program and the constant. If it returns a pointer built
|
||
with `addiu r2, base, 0xc0`, the reading is confirmed and the moby class loader
|
||
has been located. If it returns something else, the `0xc0` in `0x3a8060` is an
|
||
unrelated struct member.
|
||
|
||
**Recorded as the lesson it is:** a negative result from a hand-rolled decoder is
|
||
worth nothing until the decoder is checked against a case where the answer is
|
||
already known. `jal 0x3a8060` decoded correctly under a different constant in the
|
||
same probe and I did not notice, because one probe reporting a plausible number is
|
||
as convincing as one reporting zero.
|
||
|
||
### Second decode error, and why the hand-rolled decoder is now untrusted
|
||
|
||
Following `e1482ab`, I disassembled the function that supplies `r2`. Two errors,
|
||
both mine:
|
||
|
||
**First: transposed digits.** I read the `jal` target as `0x2ee178`. It is
|
||
`0x2ee138` -- `0x0bb84e << 2` is 3,072,312. I disassembled the wrong address and
|
||
got obviously nonsensical output, which is the only reason it was caught.
|
||
|
||
**Second, and the one that matters: at the correct address the bytes do not
|
||
decode.**
|
||
|
||
0x2ee138 d8a10000 opcode 0x36
|
||
0x2ee13c d8c20000 opcode 0x36
|
||
0x2ee140 4bc2086c opcode 0x12 (COP2)
|
||
0x2ee144 00000000 nop
|
||
0x2ee148 03e00008 jr $ra
|
||
|
||
`0x36` is not a valid MIPS major opcode for user-space code. The only coherent
|
||
readings are that this is data rather than code, that the function extent from the
|
||
decomp is mis-bounded, or that the `jal` target is still wrong.
|
||
|
||
**I am stopping here rather than continuing.** Two decode bugs in the same
|
||
investigation -- an un-masked branch target that reported a false negative, and a
|
||
transposed hex address that produced plausible-looking garbage -- are enough to
|
||
make anything else produced by this same hand-rolled decoder untrustworthy. The
|
||
next step is not another immediate scan or another hand-decoded disassembly. It is
|
||
to use `tools/overlay-anchors`, which links capstone and has a proper MIPS
|
||
decoder, and to verify it against a known-correct function before trusting
|
||
anything it says.
|
||
|
||
The chain as far as it is established, with nothing inferred beyond it:
|
||
|
||
0x3a3d70 jal 0x2ee138 target decoded, callee not yet identified
|
||
0x3a3d84 jal 0x3a7c30 r2 = v0, the return value of the above
|
||
0x3a7c30 580 bytes, never writes r2
|
||
0x3a7d88 jal 0x3a8060 passes r2 through
|
||
0x3a8060 addiu r17, r18, 0xc0 with 0xd0 and 0xe0; 16-byte-stride walk
|
||
|
||
Everything above that is verified; what `0x2ee138` returns is **unknown**, and it
|
||
is the single remaining link between the program and `moby_classes.offset`.
|
||
|
||
### d97e7df's conclusion was wrong: my section offset was wrong, not the bytes
|
||
|
||
Ran `tools/overlay-anchors --disasm` on `0x2ee138` as `d97e7df` said to. It
|
||
reports "6 of 12 words did not decode" -- but compare the *words*:
|
||
|
||
tool: 0x2ee138 0x4bc00083
|
||
my read: 0x2ee138 0xd8a10000
|
||
|
||
Different bytes at the same address. **My offset into the section is wrong**, so
|
||
the "opcode 0x36 is not a valid MIPS opcode" finding was about bytes that were
|
||
never at that address. That invalidates the conclusion in `d97e7df` as well as the
|
||
"second decode error" framing in it -- the decoder was never the problem.
|
||
|
||
The tool's words are `0x4bc00083` and friends, which are **COP2** (opcode 0x12),
|
||
i.e. VU1 vector-unit instructions, which capstone will not render under a plain
|
||
MIPS32 configuration. So capstone's "did not decode" is expected for VU code and
|
||
carries no information either.
|
||
|
||
Four separate bugs in this one investigation now, all mine:
|
||
|
||
1. un-masked J-type branch target -> false negative on callers
|
||
2. transposed hex address -> disassembled the wrong place
|
||
3. concluded "invalid opcode" from bytes that were not at that address
|
||
4. the section-offset mapping itself is wrong
|
||
|
||
**The one lesson, and it is the same one four times over:** every one of these was
|
||
a self-consistent probe reporting a clean number. The discipline that would have
|
||
caught all four is the one `d97e7df` recommended and I then skipped -- verify the
|
||
tool against a known-correct case *before* using it to generate a finding. The
|
||
known-correct case was sitting in the decomp's own extents, and I used my own
|
||
arithmetic instead.
|
||
|
||
**Nothing about `0x2ee138` is now known.** Not that it is VU code, not that it is
|
||
data, not that it is a function. The address `0x2ee138` is the correctly decoded
|
||
`jal` target from `0x3a3d70`; what is at that address is unestablished.
|
||
|
||
The correct next step is to fix the address-to-overlay mapping in `gc-formats`
|
||
(`OverlaySection` gives `dest` and `data`, and the mapping between them is what
|
||
has been wrong all along), verify it against a section whose contents are
|
||
independently known, and only then re-read `0x2ee138`. Until that mapping is
|
||
verified, **no address in this investigation that I computed by hand is
|
||
trustworthy** -- including the `0x3a8060` walk, which happened to decode but may
|
||
have been read from the wrong bytes.
|
||
|
||
### Root cause verified: the per-section indexing used throughout was wrong
|
||
|
||
The check `f2d7e7e` asked for, finally performed.
|
||
|
||
**Reference case:** `tools/overlay-anchors` disassembles `0x2ee138` as
|
||
`0x4bc00083`. Reading the flat overlay at `0x2ee138 - LIT_BASE` gives the bytes
|
||
`83 00 c0 4b`, which is little-endian `0x4bc00083`. **Match.** The correct mapping
|
||
is
|
||
|
||
off = ram_address - LIT_BASE, indexed into the whole overlay.bin
|
||
|
||
with `LIT_BASE = 0x1A7C80`.
|
||
|
||
**What I used instead**, in every probe in this investigation:
|
||
|
||
off = addr - secs[6].dest, indexed into secs[6].data
|
||
|
||
and `OverlaySection::data` **includes the 16-byte section header** ("the 16-byte
|
||
header included, since it is the first `size` bytes of the section in the table",
|
||
`overlay_reloc.rs`). So every read was displaced, and the addresses I printed
|
||
against the bytes they actually came from were wrong.
|
||
|
||
**Scope of the invalidation, stated as precisely as I can:**
|
||
|
||
- `cee17c9` -- the `addiu r17, r18, 0xc0` disassembly, the 16-byte-stride walk,
|
||
the "it is an address not a float" conclusion. **Built on wrong bytes.**
|
||
- `3ea5ab7`, `a2f8c92` -- the immediate-frequency and neighbour-enrichment
|
||
numbers. These scanned `[off, off+size)` per function, so each window was
|
||
displaced by a constant. A uniform displacement preserves most of a function's
|
||
body and loses the first few instructions, so the counts are *approximately*
|
||
right and the ratios are probably robust -- but they are not measured on the
|
||
bytes they claim to measure, and cannot be quoted as exact.
|
||
- `e1482ab`, `d97e7df`, `f2d7e7e` -- the `jal` decoding itself was verified
|
||
correct, so the branch *targets* stand; it is what lives at them that was read
|
||
wrongly.
|
||
|
||
**The one result that survives intact** is the one that never went through the
|
||
section index at all: the chunk-offset measurements in the tie and overlay work,
|
||
which read `core_data` and the overlay flat.
|
||
|
||
**The fix, for whoever continues:** never index `secs[n].data` by
|
||
`addr - secs[n].dest`. Either use `overlay[addr - LIT_BASE]`, or add a helper that
|
||
subtracts the 16-byte header from `data` and states that it does. The second is
|
||
better -- it puts the header in one place instead of making every call site
|
||
remember.
|
||
|
||
And then: **verify it**, the way the check above verifies, against a word the
|
||
decomp already pins. I had that verification available from the first probe and
|
||
skipped it sixteen times.
|
||
|
||
### Redone on correct bytes: the disassembly chain is gone
|
||
|
||
Re-read through `overlay_bytes_at` (the verified flat mapping from `cd16485`).
|
||
|
||
**`0x2ee138` is VU1 microcode**, not a function:
|
||
|
||
0x2ee138 4bc00083 COP2
|
||
0x2ee13c 4bc1086a COP2
|
||
0x2ee140 4b01083d COP2
|
||
0x2ee144 4b01104a COP2
|
||
0x2ee148 48240800 COP2
|
||
0x2ee14c 03e00008 jr $ra
|
||
|
||
Five COP2 vector-unit instructions and a return stub. Nothing you `jal` into. So
|
||
the `jal 0x2ee138` read from `0x3a3d70` cannot be right either -- and that
|
||
instruction was itself read from displaced bytes, so it must be re-read before
|
||
anything is concluded about it.
|
||
|
||
**`0x3a8060` is not what `cee17c9` said.** On correct bytes:
|
||
|
||
0x3a8060 7c820000 (opcode 0x1f -- not a MIPS user opcode)
|
||
0x3a8064 0080182d SPECIAL0
|
||
0x3a8068 3c020019 lui
|
||
0x3a8070 8c47c0b0 lw r2, 0xc0b0(r7)
|
||
0x3a807c 0c0b792c jal
|
||
|
||
**There is no `addiu r17, r18, 0xc0`, no 16-byte-stride walk, and no function
|
||
prologue.** `cee17c9`'s central finding does not exist. The "it is an address, not
|
||
a float" conclusion was about bytes that were never at that address.
|
||
|
||
**State of the investigation, honestly:**
|
||
|
||
| result | status |
|
||
|---|---|
|
||
| `overlay_bytes_at` mapping | **verified** (`cd16485`) |
|
||
| `0x2ee138` is VU1 microcode | verified just now, on correct bytes |
|
||
| `0x3a8060` walk, 16-byte stride, address-not-float | **retracted** |
|
||
| immediate frequency / neighbour enrichment | **not re-measured**; uniformly displaced, ratios plausibly robust, exact counts not |
|
||
| the `jal` chain from `e1482ab` | **unverified** -- built from displaced bytes, and one target is now known to be VU code |
|
||
|
||
**What is left of `moby_anim`:** nothing. Every disassembly-derived result is
|
||
gone, the one frequency result needs re-measuring from scratch on correct bytes,
|
||
and no loader has been located.
|
||
|
||
The honest next step is to redo the frequency measurement with `overlay_bytes_at`
|
||
and the decomp extents, and to accept that it may well produce nothing -- the
|
||
`0xc0` enrichment was a property of displaced windows, and there is no reason to
|
||
expect it to survive.
|
||
|
||
|
||
### The frequency result survives; the disassembly does not
|
||
|
||
Redone through `overlay_bytes_at`:
|
||
|
||
functions kept 2,313
|
||
moby_classes 0x00c0 391 (displaced: 394)
|
||
best neighbour (0x00c8) 15 (displaced: 4)
|
||
|
||
bucket n 0xc0 best neighbour ratio (displaced ratio)
|
||
0-15 1,033 73 2 36.5x 72.0x
|
||
16-31 759 118 4 29.5x 29.8x
|
||
32-47 283 89 2 44.5x 34.7x
|
||
48-63 118 41 3 13.7x 19.5x
|
||
64-79 55 32 2 16.0x 15.5x
|
||
80-95 37 22 2 11.0x 13.0x
|
||
|
||
**It survives, and the reason is worth stating because it separates the two
|
||
results.** The displacement was uniform -- every window shifted by the same amount
|
||
-- so a function's *contents* barely changed and a frequency count over a window is
|
||
robust to it. A disassembly is not: it depends on the exact alignment of the bytes,
|
||
which is precisely what was wrong. Hence 394 -> 391 with the same shape, while the
|
||
`0x3a8060` walk evaporated entirely.
|
||
|
||
So the correct summary of this investigation is not "it was all wrong". It is:
|
||
|
||
- **the frequency result is real**, and is now measured on verified bytes with a
|
||
test that will fail if the enrichment goes;
|
||
- **the disassembly result was never real**, and the whole `jal` chain built on it
|
||
is retracted;
|
||
- **no loader has been located**, and 0x2ee138 is VU microcode.
|
||
|
||
The lesson is about which measurements survive a mistake and which do not, not
|
||
about avoiding mistakes. A count over a window tolerates a constant displacement; a
|
||
decode does not. Sixteen probes leaned on the kind of measurement that does not,
|
||
and reported every one with the same confidence as the kind that does.
|
||
|
||
**Next:** with the mapping verified and the enrichment confirmed, the ~391 functions
|
||
are a legitimate shortlist -- but the disassembly that would consume them must go
|
||
through `tools/overlay-anchors`, whose output has now been checked against a word
|
||
the build pins.
|
||
|
||
### The shortlist points at VU microcode, not CPU code
|
||
|
||
391 candidates on correct bytes, 73 with fewer than 16 immediates. The densest is
|
||
`0x2dca20` -- 164 bytes, **three** distinct immediates, one of which is `0xc0`.
|
||
|
||
Disassembled through `tools/overlay-anchors`:
|
||
|
||
0x2dca20: bnel $a1, $fp, 0x2dca38
|
||
0x2dca24: lbu $a2, ($s7)
|
||
0x2dca28: xori.b $w0, $w0, 0xa2 <-- VU
|
||
0x2dca2c: ext $v0, $s0, 0, 1 <-- VU
|
||
0x2dca30: b 0x2dcac8
|
||
0x2dca38: sll $a1, $a1, 4
|
||
0x2dca40: .word 0x0200202d
|
||
0x2dca48: addu $a2, $v0, $a2 <-- 3-operand: MIPS-III, not MIPS32
|
||
|
||
`xori.b` and `ext` are VU1 mnemonics and the three-operand `addu` is not MIPS32 at
|
||
all. **Capstone is decoding COP2.** So `0x2dca20` is VU microcode, not a CPU
|
||
function.
|
||
|
||
That is the same answer `0x2ee138` gave. Two of the densest candidates in the
|
||
"moby_classes offset" shortlist are vector-unit code.
|
||
|
||
**This calls the frequency result into question again, and more seriously than
|
||
the displacement did.** The enrichment measured 391 CPU functions carrying
|
||
immediate `0x00c0` at 11x-45x over neighbours, and that measurement filtered on
|
||
opcodes 0x09/0x0d/0x0f/0x0c/0x05/0x06 -- which **excludes COP2**. So VU code
|
||
cannot explain those counts directly. But the *function extents* come from the
|
||
decomp's CPU analysis, and if any extent spans a VU region then the window being
|
||
scanned is mixed, and the immediate set attributed to that "function" is not a
|
||
function's.
|
||
|
||
**Honest status:** the enrichment is real as a measurement of those windows. What
|
||
those windows are is now doubtful, because the decomp's extents have not been
|
||
verified to land on CPU code, and two independent spot-checks of the densest
|
||
candidates landed in VU microcode.
|
||
|
||
**What this needs before it can be used:** verify that a known CPU function's decomp
|
||
extent actually contains CPU instructions -- same shape as the `cd16485` check.
|
||
Decode a handful of extents and require them to begin with a plausible prologue
|
||
(`addiu $sp, $sp, -N` or `lui $gp`). Any extent that does not is a mis-bound or
|
||
VU-overlapping window, and the 391 has to be recomputed over the survivors.
|
||
|
||
That check is cheap, it is the same check that would have caught this six commits
|
||
ago, and it is the honest next step. It is also, at this point, the fourth time a
|
||
"real shortlist" has turned out to need one more verification first.
|
||
|
||
### Extent check: inconclusive, and I am not going to read a result into it
|
||
|
||
Tried to answer `91b0995`'s question -- do the decomp's function extents land on
|
||
CPU code? -- by checking whether each begins with a prologue. Counting only
|
||
`addiu $sp, $sp, -N` and `lui $gp`:
|
||
|
||
total 3,835 | cpu prologue 29 | VU-first 66 | nop-first 243 | other 3,492
|
||
|
||
**29 of 3,835 looks damning and is not.** The "other" bucket contains textbook
|
||
prologues that my two patterns miss:
|
||
|
||
0x2a4ef0 ffb00000 ffb10008 sw $s0,0($sp); sw $s1,8($sp)
|
||
0x2a48f0 3c013f80 44816000 lui $1,0x3f80; bc1t <-- valid CPU, FP branch
|
||
0x2a4928 0240282d 24060002 slt $v1,$s0,$s1; sltu $a2,$zero
|
||
|
||
PS2 leaf functions routinely start by saving registers with no frame adjustment,
|
||
and functions whose extent starts after the `addiu $sp` look the same. Two patterns
|
||
cannot classify 3,835 extents.
|
||
|
||
**A second thing I cannot explain, recorded rather than guessed:** `ffb00000` has
|
||
major opcode `0x3f`, which is unused in MIPS32 -- yet `ffb00000 ffb10008` is
|
||
unmistakably a register-save prologue, and `ffbf0020` for `sw $ra, 0x20($sp)`
|
||
appears throughout this code. Either the EE encodes `sw` outside MIPS32, or my
|
||
opcode field is being read wrong somewhere. **I have not resolved this**, and it
|
||
undercuts every opcode-based check in this investigation, including the ones that
|
||
produced the 391 and the neighbour ratios.
|
||
|
||
**So the position after twenty-one commits on `moby_anim`:**
|
||
|
||
- `overlay_bytes_at` mapping: verified
|
||
- `0x2ee138`, `0x2dca20`: VU microcode, verified
|
||
- the 391 candidates: real as a measurement of the windows scanned; what those
|
||
windows are remains unestablished, because the window boundaries come from the
|
||
decomp and those have not been shown to delimit CPU code
|
||
- **the opcode field itself is not understood well enough for these checks to be
|
||
trusted**, which invalidates the last few in particular
|
||
|
||
**What this needs:** a single authoritative disassembly of one extent, compared
|
||
line by line against `tools/overlay-anchors`, to establish what the opcode field
|
||
means for this code. Everything else is downstream of that and I have been
|
||
reasoning upstream of it.
|
||
|
||
That is the same mistake one level down from the section-indexing bug: I used a
|
||
decoding I had never checked to produce results, and then measured the results
|
||
with the same unverified decoding. The check existed as a tool from the first
|
||
commit.
|
||
|
||
## Camera position in EE RAM: the differential measurement
|
||
|
||
The guard-band clip test needs the camera, and the doc says so explicitly -- "both
|
||
are per-frame game state, so this needs the camera at a savestate". Two savestates
|
||
give a differential: **anything that moved between them is per-frame state**, and
|
||
anything static is not. That is the measurement, and it needs no knowledge of where
|
||
the camera lives.
|
||
|
||
Frames `.01` and `.04`, both 33,554,432 bytes. Of every 16-byte-aligned triple of
|
||
finite floats with magnitude 0.001..100,000, **2,005 differ** between the two.
|
||
|
||
Two shapes in the output, and they are clearly different things:
|
||
|
||
0x152c70 [247.000, 194.000, 49.890] -> [300.170, 227.480, 128.000]
|
||
|
||
0x188660 [251.864, 195.641, 51.895] -> [298.396, 224.011, 128.948]
|
||
0x188670 [251.864, 195.641, 51.895] -> [298.396, 224.011, 128.948]
|
||
0x188680 [251.864, 195.641, 51.895] -> [298.396, 224.011, 128.948]
|
||
0x188690 [251.864, 195.641, 51.895] -> [298.396, 224.011, 128.948]
|
||
|
||
The `0x1886xx` block repeats the same triple four times at 16-byte stride -- that
|
||
is a per-object array, not a camera. And `0x18aa70` holds
|
||
`[0.033, -0.212, 0.977] -> [0.506, 0.644, 0.574]`, a near-unit vector that reads as
|
||
a direction rather than a position.
|
||
|
||
**`0x152c70` is the interesting one** and it is only a candidate:
|
||
|
||
[247.000, 194.000, 49.890] -> [300.170, 227.480, 128.000]
|
||
|
||
Small positive coordinates in the range Aranos occupies, moving between frames, at
|
||
an address that is a singleton rather than an array element.
|
||
|
||
**Not claimed as the camera position.** 2,005 candidates is not a shortlist, and
|
||
this stretch has produced enough confident-looking numbers built on unverified
|
||
premises that the bar for saying "this is X" is an independent check, not a
|
||
plausible shape.
|
||
|
||
**What would confirm it**, in order of cheapness:
|
||
|
||
1. The same address should move *smoothly* between adjacent frames. Take `.01`,
|
||
`.02`, `.03`, `.04` and check the series is monotone-ish rather than jumping --
|
||
a camera between consecutive frames moves a little, not 50 units.
|
||
2. It should be a **singleton**: the camera is one thing, not an array. `0x152c70`
|
||
is already in a sparse region, but that needs confirming rather than assuming.
|
||
3. Perturb the camera in an emulator and watch that address alone change.
|
||
|
||
Point 3 is the decisive one and it is cheap -- it is the same technique the
|
||
lighting verification used. Points 1 and 2 need no emulator at all and run off the
|
||
savestates already on disk.
|
||
|
||
### Confirmation step 1 rejects 0x152c70, and the savestates are not consecutive
|
||
|
||
Ran the cheapest of the three checks on the `a991088` candidate:
|
||
|
||
S 0x152c70 across 4 frames:
|
||
[247.000, 194.000, 49.890] step 0.000
|
||
[293.960, 411.810, 115.000] step 232.133
|
||
[506.306, 311.614, 116.656] step 234.804
|
||
[300.170, 227.480, 128.000] step 222.933
|
||
|
||
**Rejected.** A camera between consecutive frames moves a little; this moves ~230
|
||
units per step, and the series is not monotone in any axis. `0x152c70` is
|
||
per-frame state, but it is not a camera position.
|
||
|
||
That is the check working. `a991088` recorded the prediction in advance -- "a
|
||
camera between consecutive frames moves a little, not 50 units" -- and the
|
||
measurement says otherwise, so the candidate is out rather than reinterpreted.
|
||
|
||
**The more useful consequence: these savestates are not consecutive frames.** They
|
||
are four separate captures from different points, so "moves smoothly between
|
||
frames" cannot be used to filter the 2,005 candidates at all until consecutive
|
||
frames exist. That is a gap in the method, not a gap in the data.
|
||
|
||
**What this changes about the plan.** The three confirmations from `a991088` were
|
||
ordered by cheapness, and the two cheapest both need something the savestates do
|
||
not provide: consecutive frames for smoothness, and an actual camera perturbation
|
||
for the third. Both require a PCSX2 PINE session -- capture four consecutive
|
||
frames, then move the camera and re-capture. The `docs/plan/pcsx2-verification.md`
|
||
setup exists for exactly this, and the tfrag-lighting verification used it.
|
||
|
||
So the camera hunt moves from "offline, savestates on disk" to "needs a session",
|
||
which is the same category as the tfrag morph. Worth knowing now rather than after
|
||
several more offline probes: **the differential measurement narrowed 2,005
|
||
candidates to nothing, and it did that in one step, because the confirmation test
|
||
was written before the measurement was interpreted.**
|
||
|
||
|
||
## The guard-band clip test, implemented and unit-tested
|
||
|
||
`1aa94ee` moved the camera hunt behind a PCSX2 session. The camera is only part of
|
||
this item. The test itself is a pure function over a camera and a frustum, and it
|
||
can be written, reviewed and tested now -- so that when the camera is located it
|
||
drops straight in rather than needing writing under time pressure.
|
||
|
||
New `gc_formats::guard_band`, implementing `docs/plan/tfrag_rac1.md` §3b.2 as
|
||
attributed: steps 1-7, returning `Cull`, `Normal`, or `ClipAtLod0`. Seven unit
|
||
tests cover directly-ahead, behind-camera, beyond-far, straddling the side planes,
|
||
all-corners-outside-one-plane, the integer sqrt, and a rotation.
|
||
|
||
**The geometry side it consumes is already verified.** `TfragGeometry::bsphere`,
|
||
`clip_corners` (8 corners, median 1.000x the bounding radius from their centroid)
|
||
and `draw_mode` (matching the reference table) all exist and are tested. So this
|
||
is not speculative plumbing: the inputs are parsed, and the only missing piece is
|
||
the per-frame camera.
|
||
|
||
**What the first version got wrong, and it was the interesting part.** The
|
||
reference culls when *one* plane has every corner outside it. I first wrote "every
|
||
corner is outside some plane" -- which culls patches that merely straddle
|
||
*different* planes. The unit test caught it immediately: a patch whose corners
|
||
were beyond the far plane in one axis and the side plane in another came back
|
||
`Cull` when it should have been `ClipAtLod0`. Fixed by tracking a per-corner 4-bit
|
||
mask of which planes it is outside, and testing each plane across all corners.
|
||
|
||
That is the whole guard band's purpose: a patch outside the frustum but inside the
|
||
guard band must still be drawn, by the clipping program, so that objects partly
|
||
off-screen do not pop. Collapsing those two conditions into one silently turns
|
||
"4x guard band" into "immediate cull", and the symptom would be objects
|
||
disappearing at the screen edge rather than any error.
|
||
|
||
**Not verified against the game.** Steps 1-4 are implemented from the reference and
|
||
have not been compared to a real frame, for the reason above. The unit tests check
|
||
the algorithm implements the documented rule; they do not check the rule is what
|
||
the game does. The `tan` scale in particular -- 1024 for 1.0 -- is my reading of
|
||
the fixed-point convention and is **not** established, and the clip-cube scale of
|
||
64 is likewise R&C1's documented value, unverified. Both are noted in the module
|
||
doc so they are not mistaken for measured facts.
|
||
|
||
## The camera region is at 0x18aa70 -- found live, verified by orthonormality
|
||
|
||
With PCSX2 on Aranos, the savestate problem went away: **PINE reads EE RAM live**,
|
||
so two dumps seconds apart are consecutive frames by construction, which is exactly
|
||
what the offline savestates could not provide.
|
||
|
||
Two live dumps 1.5s apart. Of every 16-byte-aligned triple of finite plausible
|
||
floats, **1,654 differ**; 1,501 are singletons and 153 repeat a neighbouring
|
||
triple (those are the per-object arrays). Most of the singletons are mob positions
|
||
moving 4-5 units. Two shapes stood out:
|
||
|
||
0x18aaa0 [247.299, 193.133, 50.453] -> ... moved 0.26 over 2s
|
||
0x18aa70..0x18ab2c unit vectors that reoriented
|
||
|
||
The second region is **nine 4x4 matrices at 64-byte stride** (this note first said
|
||
three; see the correction below):
|
||
|
||
0x18aa70 0.0338 -0.1983 0.9796 0.0000
|
||
-0.0275 -0.9799 -0.1974 0.0000
|
||
0.9990 -0.0203 -0.0386 0.0000
|
||
247.2990 193.1330 50.4526 1.0000
|
||
0x18aab0 (same shape, t = 247.239, 193.102, 50.497)
|
||
0x18aaf0 (same shape, t = 246.940, 193.155, 50.151)
|
||
|
||
**The control that makes this an identification:** all three upper 3x3 blocks are
|
||
orthonormal to within 1e-6 -- row norms 0.999995..0.999998, pairwise dot products
|
||
1e-6 and below. Three consecutive float triples in a random 32 MiB region being
|
||
unit length *and* mutually perpendicular is not a coincidence anyone gets twice.
|
||
The residual is f32 rounding, which is the expected result on an FPU that is not
|
||
IEEE-754 (AGENTS rule 2).
|
||
|
||
Each fourth row is `(x, y, z, 1.0)` -- affine, and the translation is the camera
|
||
position, shared across all three to within 0.4 units. Three matrices with one
|
||
position is what a view / modelview / lighting family looks like.
|
||
|
||
**Camera position on Aranos: (247.3, 193.1, 50.5).** Test asserts orthonormality,
|
||
`w == 1`, and that the translation moves smoothly but not wildly -- the second half
|
||
is the check `1aa94ee` needed and the savestates could not supply.
|
||
|
||
**What this unblocks.** `guard_band` step 1 is `R * (bsphere - camera * 1024)`.
|
||
Both operands are now available: `R` is rows 0-2 of `0x18aa70`, and the camera is
|
||
its fourth row. The clip test can be run against a real frame for the first time.
|
||
|
||
**Still to establish**, and deliberately not assumed: which of the three matrices is
|
||
the one step 1 wants, whether the translation is world or camera space (the
|
||
guard-band algorithm multiplies by 1024, which implies the disc's own 12.4 fixed
|
||
point, so the float translation may need converting first), and the frustum
|
||
planes, which the guard band needs and which are not in this structure.
|
||
|
||
### Correction: it is nine matrices, not three
|
||
|
||
`dcd4dd6` reported "three 4x4 matrices at 64-byte stride" and read them as a
|
||
view / modelview / lighting family. **There are nine.** They are contiguous from
|
||
`0x18aa70` and the structure breaks at `0x18acb0`, where the row norms are
|
||
`0, 0, 0.97` and `w = 0`.
|
||
|
||
m0 247.299 193.133 50.451 m3 247.122 193.125 50.140 m6 247.211 193.074 50.517
|
||
m1 247.239 193.102 50.496 m4 247.030 193.092 50.799 m7 247.041 193.091 50.457
|
||
m2 246.940 193.157 50.150 m5 247.033 193.062 50.633 m8 247.033 193.062 50.633
|
||
|
||
All nine are orthonormal to 1e-6, all nine are affine, all nine carry a
|
||
translation inside a 0.4-unit box, and all nine translate smoothly between live
|
||
frames (0.003-0.008 in 1.5s). Nine different rotations about one position is not a
|
||
view/projection/lighting family -- it is a **table indexed by orientation**, and
|
||
the index is what is unknown.
|
||
|
||
So `dcd4dd6` was right that the region holds the camera's position and wrong that
|
||
three was the extent. It stopped at three because three happened to read as a
|
||
sensible story, which is the failure this project already documents in rule 4 and
|
||
that I have now committed twice in this investigation.
|
||
|
||
**What survives, unchanged:** the translation is the camera position, it is
|
||
(247.3, 193.1, 50.5), and it moves smoothly between consecutive live frames while
|
||
the rotation is changing much faster. That is still the identification, and it is
|
||
still verified by orthonormality plus the smooth-motion control.
|
||
|
||
**What is now open, and is larger than it looked:** which entry of the nine is the
|
||
camera the guard-band test wants. Not "pick m0" -- nine orientations about one point
|
||
is a table with an index, and the index is what step 1's `R` has to be. There is a
|
||
plausible reading that these are the nine legal camera orientations on a spherical
|
||
planet, which would be a genuinely interesting find and is **not** established by
|
||
anything measured here.
|
||
|
||
Two cheap checks that would settle it, both needing only the live emulator:
|
||
move the camera and see whether all nine translations move together or only one
|
||
(the former says "one camera, nine cached frames"; the latter says "nine things");
|
||
and change the camera's pitch to see whether the nine rotations reindex.
|
||
|
||
### m4 is the live camera orientation -- 20x outlier against 8 controls
|
||
|
||
`d4e81ba` left one question with a two-way answer: is the camera's rotation one of
|
||
the nine, and if so which. Answered by rotating the camera in the live emulator and
|
||
watching which matrix responds.
|
||
|
||
Ten captures at ~0.6s intervals while the camera was being turned. Angle through
|
||
which each matrix's first row rotated, measured from capture 1:
|
||
|
||
idx 1->4 1->7 1->10
|
||
m0 0.37 0.27 0.53
|
||
m1 1.31 1.07 1.34
|
||
m2 3.89 1.38 1.43
|
||
m3 2.25 1.18 1.40
|
||
m4 30.60 18.49 37.21 <-- the camera
|
||
m5 1.70 1.45 1.82
|
||
m6 1.24 0.98 1.29
|
||
m7 1.88 1.34 1.73
|
||
m8 1.70 1.45 1.82
|
||
|
||
**m4 at `0x18ab70`, and the eight others are the control.** They sit under 4
|
||
degrees across the whole sweep while m4 goes to 30-37 -- a 20x outlier -- and its
|
||
motion is non-monotonic, tracking the input being turned one way then back.
|
||
|
||
The translations told the complementary half: **all nine move together**, by the
|
||
same few thousandths. So the nine share one camera *position* and carry nine
|
||
different *orientations*, of which one is live and follows the look direction.
|
||
|
||
**This is what `guard_band` step 1 needs**, and both operands are now located:
|
||
|
||
R rows 0-2 of the matrix at 0x18ab70
|
||
camera translation, its fourth row: (247.1, 193.4, 50.4) on Aranos
|
||
|
||
**What this does not establish.** Why there are nine. They are not nine cached
|
||
frames of one camera -- if they were, all nine would have swung by 30 degrees
|
||
together. They are nine distinct orientations about a shared point with one of them
|
||
selected, and a table selected by an index fits. The nine-legal-orientations-on-a-
|
||
spherical-planet reading remains **plausible and unproven**; what is proven is
|
||
which entry is live. No reindexing was observed over the sweep, so if the index
|
||
does move, it does not on camera rotation alone.
|
||
|
||
### Frustum search: bounded, not found
|
||
|
||
`e7eec01` located `R` and the camera. The remaining operand is the frustum --
|
||
`near`, `far`, `tan`. Searched with the same shape of filter that found the
|
||
camera: a structural invariant first, a differential second.
|
||
|
||
**First attempt, discarded.** Scanning for a perspective-matrix signature (a
|
||
nonzero `col3` in row 2 with zero elsewhere) returned **11,846 hits**. That is
|
||
noise, and the filter was worthless. Recording it because the number looks like a
|
||
result and is not.
|
||
|
||
**Second attempt, the right filter.** Unit-length 3-vectors at 16-byte alignment --
|
||
the same invariant that identified the camera's matrices. 9,514 of them, of which
|
||
188 form clusters of four or more within 256 bytes:
|
||
|
||
0x1793f0 .. 0x1796d0 36 vectors, 736 bytes
|
||
0x189e20 .. 0x189e80 6 vectors, 96 bytes
|
||
0x18a550 .. 0x18a580 4 vectors, 48 bytes
|
||
0x18aa70 .. 0x18ac90 27 vectors, 544 bytes <- the nine camera matrices
|
||
0x18b260 .. 0x18b640 49 vectors, 992 bytes
|
||
|
||
**The six-vector cluster at `0x189e20` is a frustum's shape** -- six planes is the
|
||
right count and 96 bytes is the right size -- and it is not one. It reads:
|
||
|
||
0x189e20 [-0.4106, 0.9118, 0.0001] |x|=1.0000
|
||
0x189e30 [-0.9118, -0.4106, 0.0001] |x|=1.0000
|
||
0x189e40 [ 0.0001, -0.0001, 1.0000] |x|=1.0000
|
||
0x189e50 [ 0.0000, 0.0000, 0.0000 ] w=1.000
|
||
... repeats at 0x30 stride ...
|
||
|
||
Three unit vectors and a zero affine row: an orthonormal basis with no
|
||
translation, which is an orientation record, not a plane set. A translation sits
|
||
at `0x189ea0` = (246.857, 193.205, 49.891) -- near the camera position, so this
|
||
region is a table of positioned things.
|
||
|
||
**So the frustum is still not located, and I am not going to pick the six-vector
|
||
cluster because six is the number a frustum has.** That would be rule 4's failure
|
||
with an exact number attached to make it feel earned, which is the specific thing
|
||
this project keeps warning about and the thing I have already committed twice in
|
||
this investigation.
|
||
|
||
**What the search did establish:** the camera-adjacent structured regions in EE
|
||
RAM are the nine matrices at `0x18aa70`, orientation records at `0x189e20`, and
|
||
larger clusters at `0x1793f0` and `0x18b260`. The frustum, if it is stored as
|
||
planes, is not among them. If it is stored as `near`/`far`/`tan` scalars -- which
|
||
is what `guard_band`'s signature suggests, since it takes `Frustum { near, far, tan }`
|
||
rather than six planes -- then the search should be for a stable float pair with a
|
||
plausible ratio, not for normals at all. That is the next search, and it is cheap.
|
||
|
||
### Stop scanning RAM: the camera function is the anchor
|
||
|
||
Two more RAM searches for the frustum, both unproductive, and then the thing that
|
||
should have been looked up first.
|
||
|
||
**Non-orthonormal 4x4 matrices within 4 KB of the camera**, on the reasoning that
|
||
the projection must live nearby: 79 candidates, and every one of them is a sliding
|
||
window over data rather than a matrix -- `m23` and `m32` landing on the translation
|
||
and unrelated neighbours. The filter is not discriminating and the region does not
|
||
contain an obvious projection.
|
||
|
||
**The decomp names the camera.** `include/engine.h` and
|
||
`docs/AUG6-RETAIL-ANCHORS.md` reference **`UpdateCamera_6`**, **`InitCamera_24`**
|
||
and `Camera_CollPrimTest`, and `docs/COMMUNITY-ENGINE-REFERENCE.md` places
|
||
`lvl.camvtbl` -- the camera vtable -- at `0x0026E900`. The camera dispatch table is
|
||
already characterised: **217 records, 14 distinct ids** on level 0, handler
|
||
addresses in the overlay's `.text`.
|
||
|
||
That is the right way in. The camera writes its matrices at `0x18aa70`; the
|
||
function that does so is reachable from `camvtbl`, and the frustum will be whatever
|
||
that same function writes nearby. **Reading the camera function finds both the
|
||
write site that confirms `m4` and the frustum constants beside it**, without
|
||
guessing at float patterns.
|
||
|
||
**Caveat that applies before starting**, and is the reason this was not done first:
|
||
the decomp's function addresses have already failed once in this project --
|
||
`PROVENANCE.md` records that the decomp's addresses do not resolve for the moby
|
||
class loader, and that struct layouts transcribed from the decomp were usable while
|
||
addresses were not. So `UpdateCamera_6`'s address has to be **verified against the
|
||
overlay** before following it, not trusted from the name.
|
||
|
||
**This is where the frustum hunt should resume**, and it is a different kind of
|
||
task from the three searches above: follow a known call site rather than pattern-
|
||
match a 32 MiB heap.
|
||
|
||
### UpdateCamera_6 verified at 0x4284a8
|
||
|
||
The anchor `20655c7` said must be verified before following. It is.
|
||
|
||
`docs/AUG6-RETAIL-ANCHORS.md` recovers `UpdateCamera_6` and `InitCamera_24` as
|
||
**class/role identifiers in the two dispatch tables**, so role 6 of `lvl.camvtbl`
|
||
*is* the camera update function. And the decomp's addresses are in the same space as
|
||
ours -- its `func_L00_002A3B80` is this overlay's `.text` dest exactly -- so the
|
||
claim is checkable rather than something to take on faith.
|
||
|
||
Three independent checks, because `PROVENANCE.md` records that the decomp's function
|
||
addresses have failed once here:
|
||
|
||
1. Role 6 of level 0's `lvl.camvtbl` names handler **`0x004284a8`**.
|
||
2. `overlay_bytes_at` reads real code there: `0x3c014094`, a `lui`.
|
||
3. `tools/overlay-anchors` decodes the first six words with **zero** failures:
|
||
|
||
0x4284a8 lui $at, 0x4094
|
||
0x4284ac ori $at, $at, 0x7ae1 ; $at = 0x40947ae1
|
||
0x4284b0 mtc1 $at, $f1 ; a float into an FP register
|
||
0x4284b4 lw $v0, 0x2290($s4)
|
||
0x4284b8 lui $at, 0x3f00
|
||
0x4284bc mtc1 $at, $f3 ; 0x3f000000 = 0.5f
|
||
|
||
Two `mtc1` into FP registers, and a struct field at `+0x2290`. That is a function
|
||
doing floating-point work on a camera struct -- which is what a camera update
|
||
should look like, and not what a random address gives. Three checks agreeing is a
|
||
different kind of evidence from any one of them.
|
||
|
||
**Level 0's `lvl.camvtbl` is 0xb4 bytes: 13 records, 13 distinct ids.** Note that
|
||
several ids read as implausible large values (4,350,904 and similar), so the
|
||
table's record layout is **not** uniformly id/handler and role 6 being clean may be
|
||
lucky. Worth keeping in mind when the other roles are read.
|
||
|
||
**Next, now unblocked:** read the whole of `0x4284a8` and find the store that
|
||
writes `0x18aa70`. That confirms `m4` by locating the write site rather than by
|
||
inference from input response, and the frustum constants should be whatever the
|
||
same function writes nearby.
|