ReaperOfVeriod
27891ce5b4
Sought the moby class loader by counting raw u16 occurrences of each chunk offset in .text. moby_classes 0xc0 hits 49,230 times, tie_classes 0x1e80 hits 111 -- and that contrast means nothing. Small u16 values are simply common: the most frequent value in .text occurs 1,834,535 times. A two-byte value that is frequent by construction beats a rarer one regardless of what either references, so the comparison has no shared denominator. Rule 3a again, one level below where it has bitten before -- not the measurement being wrong but the comparison being uninterpretable. Recorded as a methodological negative rather than a finding. Doing it properly means counting immediates only where the instruction decoder says they are immediates, which is what tools/overlay-anchors exists for. No test added: a test asserting a meaningless comparison would make the next reader trust it.
1963 lines
93 KiB
Markdown
1963 lines
93 KiB
Markdown
# Level assets
|
||
|
||
Phase 2 work record. What has been established about the four extents inside
|
||
each `LEVEL<n>.WAD`, and what remains.
|
||
|
||
## A bug found and fixed
|
||
|
||
Reading the `data` extent's contents exposed a defect in the Phase 1 extractor.
|
||
|
||
Level extents are declared in **sector counts**; the generic `G/` pair table
|
||
uses **byte counts**. The extractor routed level containers through the generic
|
||
path, so `LEVEL0.WAD`'s `data` extent — 6,484 sectors — was taken as 6,484
|
||
*bytes*. Level 0 yielded a 6,484-byte "geometry" file instead of 13,279,232
|
||
bytes. Across 27 levels that silently discarded roughly 300 MB of level content
|
||
while every test still passed.
|
||
|
||
The tell was arithmetic, not a crash: the extent's own header declared an
|
||
uncompressed total of 13,277,920 bytes, and the ratio against the 6,484-byte
|
||
file was 2047.7× — one sector, to within rounding. A ratio that lands on 2048
|
||
is a unit error.
|
||
|
||
Fixed by extracting level containers through the typed `LevelHeader` instead of
|
||
the generic table. Level 0 now yields:
|
||
|
||
```text
|
||
data.bin 13,279,232 the level container's data extent
|
||
core_data.wad 9,738,678 the geometry, compressed
|
||
core_data.bin 16,969,296 the geometry, decompressed
|
||
core_index.bin 31,600 the geometry directory
|
||
gameplay.bin 659,456
|
||
sound_bank.bin 1,503,232
|
||
```
|
||
|
||
**Lesson:** a unit error is invisible to a type system and to a passing test. The
|
||
check that catches it is arithmetic against an independent number the file
|
||
itself declares. Any extent whose contents describe a larger structure should
|
||
have that structure's own total compared against the extent.
|
||
|
||
## Done: the container header
|
||
|
||
`crates/gc-formats/src/level_wad.rs` parses the 0x60-byte header on all 27
|
||
levels and names the four extents. See
|
||
[`../formats/level-wad.md`](../formats/level-wad.md) for the layout and the
|
||
measured invariants.
|
||
|
||
Two facts worth carrying forward:
|
||
|
||
- **Sizes in this header are sector counts**, unlike the generic `G/*.WAD` pair
|
||
table where they are byte counts. Both are on the same disc.
|
||
- **`LEVEL21.WAD` reports level id 30.** Container order is not level identity.
|
||
|
||
## `gameplay`: solved, and the old answer was wrong
|
||
|
||
**It is a WAD LZ77 stream.** It decompresses with the decoder in `wad.rs` like
|
||
`core_data` does, and the decompressed extent holds the level's light bank.
|
||
|
||
This section used to be the longest open question here, and it was wrong. The
|
||
reasoning was sound and the conclusion was not: byte 3 of the magic is `0x79`
|
||
on level 0 rather than the `0x0d` the WAD docs suggest, which reads as a
|
||
per-level variation; but byte 3 is part of the 32-bit little-endian stream
|
||
length, and `0x0a0d79` is simply level 0's compressed length. An earlier
|
||
attempt decompressed "starting after an 8-byte header" and read the resulting
|
||
garbage as evidence of a different format.
|
||
|
||
The failure mode is the one worth keeping: a uniform failure on all 27 inputs
|
||
was taken to mean the decoder was wrong, when it meant the *data* was being
|
||
handed over uncompressed. A positive control -- decompress it and look at the
|
||
result -- was never run, and it would have answered the question in one line.
|
||
|
||
See "The light bank was there the whole time" below.
|
||
|
||
## `sound_bank`: signatures found, structure unknown
|
||
|
||
```text
|
||
+0x18 "SBlk" present in all 27 levels
|
||
+0x24 "DAW" present in all 27 levels
|
||
```
|
||
|
||
Sizes 458..833 bytes. Too small to be the level's audio; likely an index into
|
||
the global `AUDIO.WAD`.
|
||
|
||
## `occlusion`: header located, contents a u16 array
|
||
|
||
The header's `occlusion` offset (0x0c) is trustworthy — every offset in it lands
|
||
inside `core_data`. Level 0's occlusion block starts at 2,117,888 and reads as a
|
||
`u16` array:
|
||
|
||
```text
|
||
5152 0 12 23 14 19 24 29 32 35 38 41 44 47 50 53 56 59 65 82 96 100 113 126 143 164 183 ...
|
||
```
|
||
|
||
An ascending run of 65 values with mostly small differences, then it restarts.
|
||
That is consistent with per-cell visibility records rather than geometry. The
|
||
meaning of the pairs — and the 0 that ends each group — is not yet established.
|
||
|
||
## `sky`: plausible, unparsed
|
||
|
||
Header offset 0x10 = 2,192,576. Opens `0 0 0 0 0 0 6 0 0 0 0 0 5 0 0 0 40 0 0 0 c0 0 0 0 40 ...`
|
||
— small values including 64 and 192, which look like sky-shell counts and
|
||
dimensions rather than offsets.
|
||
|
||
## `tfrag` region: `[0, occlusion)`
|
||
|
||
The header's fields at 0x0c, 0x10, 0x14 and 0x60 are **offsets**, and they are
|
||
strictly increasing on all 27 levels. The differences between them give region
|
||
sizes that are plausible on every level:
|
||
|
||
| Region | Range | Level 0 size |
|
||
|---|---|---|
|
||
| tfrag | `[0, occlusion)` | 2.12 MB |
|
||
| occlusion | `[occlusion, sky)` | 0.07 MB |
|
||
| sky | `[sky, collision)` | 0.42 MB |
|
||
| collision | `[collision, textures_base)` | 0.90 MB |
|
||
|
||
Ruled out: reading 0x0c/0x10/0x14 as *sizes* laid end to end. The cumulative
|
||
total does not reach `textures_base` on any level, and the byte at the start of
|
||
each supposed region is not a plausible structure. The offsets model is the one
|
||
that holds everywhere.
|
||
|
||
### What is in the terrain region
|
||
|
||
**Textures are stored inline with the geometry**, not in a separate bank. A scan
|
||
for the PS2 texel pattern `18 18 18 80` (grey texels with the 0x80 alpha that
|
||
means "opaque" in the GS's convention) finds 6,739 occurrences spread across 138
|
||
clusters, the first at `0xd9f8` — inside the tfrag region. So terrain carries its
|
||
own texels, and `textures_base` names something else.
|
||
|
||
Earlier claims in this file that have since been **disproven**:
|
||
|
||
- ~~"The terrain is 382 records of 64 bytes."~~ The 0x48 byte at offset +3 is
|
||
simply the most common byte in the region (945 of 2000 sampled slots); it is
|
||
not a record marker. Over 33,091 slots at that position only 4,127 hold 0x48.
|
||
- ~~"Byte +3 is constant at 0x48 across 300 records."~~ True for the first 382
|
||
records and false beyond, which is how the false pattern was found.
|
||
- ~~"Coordinates are fixed-point s16 with a scale of 8."~~ The scale-8 reading
|
||
is supported by one coincidence (17,496 / 8 = 2,187, the planet radius) but no
|
||
grouping of the record into triples yields points on a sphere of that radius.
|
||
Tested: every stride from 2 to 16 bytes, every starting offset, three and four
|
||
components. Best result 5.0% of points on-sphere, which is chance.
|
||
|
||
So the record framing is **not recovered** — but that framing was a false lead,
|
||
and the terrain has since been parsed in full. This section records the wrong
|
||
turn and is kept only so it is not repeated.
|
||
|
||
The region layout below is right and terrain really is packed alongside inline
|
||
texels, but the "64-byte record" idea was the wrong way in. Tfrags are not a
|
||
flat vertex table: they are VIF1 command lists whose unpacks are classified by
|
||
format *and* destination VU address, and the vertices come out of a V3_16
|
||
unpack spread across three lists per patch. See "tfrag terrain" below for the
|
||
framing that works and the four wrong turns along the way.
|
||
|
||
The lesson is the recurring one in this file: a hypothesis that produces
|
||
plausible-looking output is not thereby recovered, and a "ruled out" entry is
|
||
only as good as the control it was tested against.
|
||
|
||
## Where the class tables are not
|
||
|
||
The header's `moby_classes` and friends do not decode. Read as
|
||
`(offset, count)` they give plausible numbers, but the entries at those offsets
|
||
are not valid records. Ruled out:
|
||
|
||
- **Field order.** Both `(offset, count)` and `(count, offset)` produce garbage.
|
||
- **Stride.** Tried 8, 16 and 32 bytes per entry.
|
||
- **Which file the offsets index.** Tried `core_data` and `core_index`.
|
||
- **Header base.** Not the issue: the compressed size (0x88) and decompressed
|
||
size (0x8c) of `core_data` appear at the *same offsets* in all 27 levels'
|
||
`core_index`, which pins the header base at 0 with certainty.
|
||
|
||
One observation worth keeping: every second value in the class-table fields is
|
||
exactly divisible by 16, and they increase monotonically across the nine tables.
|
||
That is consistent with `count * 16` sizes, but the corresponding offsets
|
||
cluster at the very start of the file and do not yield valid records either.
|
||
|
||
The discriminator used to reject candidates is worth stating, because weaker
|
||
tests produce false positives: a real class table's `offset_in_asset_wad`
|
||
values increase. A search for "32-byte records with increasing offsets" returns
|
||
several hundred candidates across the file, and the top one turned out to be a
|
||
texture table whose `o_class` field merely happened to equal `offset + 16`.
|
||
Scoring on plausibility alone is not enough; the entries have to actually chain.
|
||
|
||
Next: rather than guess the header further, identify `tfrag` by its own
|
||
structure. Terrain is the largest subsystem and the one that decides whether
|
||
anything draws, so it is the right next target even though it is the harder one.
|
||
|
||
*(Done — see "tfrag terrain" below. Identifying tfrag by its own structure
|
||
worked where the class-table guess did not.)*
|
||
|
||
## `data`: the real target, untouched
|
||
|
||
6,484 sectors on level 0, down to 2,102 on level 24 — 4.2 to 21.9 MB. This is
|
||
the level's geometry and the largest thing in the container. Next target.
|
||
|
||
What is known from the community reference and from Insomniac's engine family:
|
||
it splits into four geometry subsystems — `tfrag` (terrain fragments),
|
||
`tie` (instanced environment structures), `shrub` (foliage) and `moby` (dynamic
|
||
actors) — plus sky shells, collision data and textures. ReRAC's documentation of
|
||
the equivalent structures in the first game is the closest reference available,
|
||
though R&C2's spherical worlds change how they are used.
|
||
|
||
## Order of work
|
||
|
||
1. `gameplay` — smallest, fully bounded, and the format is identified enough to
|
||
make progress. A win here validates the approach before the large effort.
|
||
2. `occlusion` — tiny.
|
||
3. `sound_bank` — small.
|
||
4. `data` — the large one, and the one that decides whether anything renders.
|
||
|
||
Doing the small extents first is not just convenience: each one is a contained
|
||
reverse-engineering problem that either resolves or tells us something, whereas
|
||
`data` will keep producing ambiguous readings until we have something to check
|
||
them against.
|
||
|
||
## Method note
|
||
|
||
Every format question this phase has been settled the same way: write the
|
||
hypothesis, test it against **all 27 levels at once**, and require an invariant
|
||
that holds everywhere. The sector/byte ambiguity in Phase 1 survived because it
|
||
happened to be correct on the one container checked by hand. The level header
|
||
was fixed in a single pass because the tiling check failed immediately on
|
||
levels 1, 2, 4, 7, 8, 11, 19 and 20.
|
||
## tfrag terrain: block structure and vertex positions
|
||
|
||
Settled. `gc-formats::tfrag` parses the block, and a gated test decodes every
|
||
patch on all 27 levels.
|
||
|
||
### Block and patch framing
|
||
|
||
The tfrags block opens with a 12-byte header: `table_offset` (always `0x40`),
|
||
`tfrag_count`, and an `f32` LOD distance (`32.0`). The patch table follows at
|
||
`table_offset`, `tfrag_count` records of `0x40` bytes each. On level 0 that is
|
||
833 patches.
|
||
|
||
A patch header's `data` offset is **relative to `table_offset`**, not to the
|
||
start of the block. Getting this wrong happens to work for the first patch and
|
||
silently produces garbage afterwards.
|
||
|
||
### Positions live in three lists, not one
|
||
|
||
`vert_count` is the *total* across all three position tiers. Within a patch's
|
||
data block the VIF lists do not run in ascending offset order — the LOD-2 list
|
||
sits *before* the common list — and the LOD-0-only list has no offset of its
|
||
own, starting instead at `shared_ofs + lod_1_size * 16`. The ranges, in transfer
|
||
order, are:
|
||
|
||
| List | Range (from the patch data block) | Holds |
|
||
|---|---|---|
|
||
| common | `shared_ofs .. lod_1_ofs` | VU header, ad-gifs, vertex info, origin, positions |
|
||
| LOD 2 | `lod_2_ofs .. shared_ofs` | strips, indices |
|
||
| LOD 1 | `lod_1_ofs .. lod_0_ofs` | strips, indices |
|
||
| LOD 0+1 | `lod_0_ofs .. shared_ofs + lod_1_size*16` | parents, vertex info, positions |
|
||
| LOD 0 | `shared_ofs + lod_1_size*16 .. rgba_ofs` | positions, strips, indices, parents |
|
||
|
||
Scanning only the common list and expecting `num == vert_count` finds positions
|
||
in roughly a quarter of patches. The other three quarters keep most of their
|
||
positions in the LOD lists. That was the first wrong turn and it looked
|
||
plausible: every patch it did match matched *exactly*.
|
||
|
||
### The VIF1 code word
|
||
|
||
Two rules had to be right before any of this worked, and both were wrong on the
|
||
first attempt.
|
||
|
||
**The vector format lives in the command byte, not the immediate.** Bits 2-3 of
|
||
the command are `vn` (component count minus one) and bits 0-1 are `vl` (0 =
|
||
32-bit, 1 = 16-bit, 2 = 8-bit, 3 = 5-bit), so
|
||
|
||
element_size = ((32 >> vl) * (vn + 1)) / 8
|
||
|
||
Terrain positions are `vn=2, vl=1` — command `0x69` — at 6 bytes each. Reading
|
||
the format out of the immediate's low nibble, which is the more commonly quoted
|
||
layout, yields 12 or 16 bytes and quietly misaligns everything after it.
|
||
|
||
**Unpack payloads pad to 4 bytes, and several non-unpack commands carry inline
|
||
data.** An unpack advances `(count * element_size + 3) & !3`. `STROW`/`STCOL`
|
||
carry 16 bytes, `STMASK` 4, `MPG` `num * 8`, `DIRECT`/`DIRECTHL` `imm * 16`;
|
||
every other non-unpack is a bare 4-byte code word. Ignoring the first four
|
||
desynchronises the walk, and a desynchronised walk still finds code words that
|
||
look like positions — it just finds the wrong number of them.
|
||
|
||
The 4-byte padding is easy to misread. With 75 elements the payload is 450
|
||
bytes and the following word lands at +452, where `0x01000404` (STCYCL) reads
|
||
perfectly plausibly. The word at +450 is `0x04040000` (ITOP), which is the real
|
||
one. Taking the plausible-looking reading is how this was briefly wrong.
|
||
|
||
### Verification
|
||
|
||
Requiring every patch to yield exactly its own `vert_count` is the invariant
|
||
that caught both errors, and it is the check the parser is built around: a
|
||
mismatch means the walk desynchronised, so `tfrag_positions` returns `None`
|
||
rather than partial geometry. Across all 27 levels: 18,102 patches,
|
||
821,439 vertices, zero mismatches.
|
||
|
||
### Topology: strips, indices and triangles
|
||
|
||
Also settled. `parse_tfrag` walks each list and classifies every unpack by
|
||
format *and* destination VU address, using the VU header at the head of the
|
||
common list (5 V4_16 vectors, unsigned, to address 0). That table is what
|
||
separates the strips from the indices from the parent arrays — they are all
|
||
V4_8, and only the address tells them apart.
|
||
|
||
| `(vn, vl)` | Meaning |
|
||
|---|---|
|
||
| (3, 1), unsigned, addr 0 | the VU header itself |
|
||
| (3, 0) | ad-gifs, 5 quadwords each |
|
||
| (2, 1) | positions, V3_16 |
|
||
| (3, 1) | vertex info: UV, LOD parent, own position |
|
||
| (3, 2) | strips (signed) and indices (unsigned) |
|
||
|
||
**Strip indices address the vertex-info array, not the position array.** Several
|
||
vertex-info entries can share a position, so a strip index resolves through
|
||
`vertex_info[i].vertex / 2` to reach a position.
|
||
|
||
A strip record is four bytes: a vertex count, a packet flag and an ad-gif
|
||
offset. A negative count is biased by −128, and *the first record of a list is
|
||
always biased* even when its count is positive — a real patch opens with −116 or
|
||
−124, never a small positive number. A zero count ends the list. A run of `n`
|
||
vertices is a GS triangle strip of `n − 2` triangles with alternating winding.
|
||
|
||
**Bank the LODs by index, not by arrival.** The lists are walked in transfer
|
||
order but the strip and index arrays have to be stored by *which LOD they
|
||
belong to*. Appending them as they are found fills the arrays LOD-2-first,
|
||
which looks fine until the triangle counts come out low.
|
||
|
||
### Verification
|
||
|
||
`tri_count` is stored in the patch header and cannot be derived from the lists
|
||
without walking them, so reproducing it is a strong check. All 27 levels:
|
||
**18,102 patches, 809,879 triangles, every patch matching its header**, with
|
||
every triangle index inside that LOD's vertex-info range and every ad-gif index
|
||
inside the ad-gif array.
|
||
|
||
Getting there took four wrong turns, each of which failed quietly: reading the
|
||
element size from the immediate, 2-byte payload padding, ignoring non-unpack
|
||
payloads, and banking strips by arrival order. Every one produced parseable,
|
||
plausible geometry; only the header's own triangle count caught them.
|
||
|
||
### Texture bindings, colours and lights
|
||
|
||
Also settled. Adds `level_core.rs` for the index header and the texture table,
|
||
which is the join between tfrags and texel data.
|
||
|
||
A tfrag's **ad-gifs** are five A+D quadwords (0x50 bytes) holding packed
|
||
fields rather than GS registers; the level load rewrites them in place before
|
||
drawing. On disc `TEX0_1.data_lo` indexes the level's texture table,
|
||
`TEX1_1.data_lo` is the LOD bias K and `data_hi` the minification filter, and
|
||
`CLAMP_1` holds the two wrap modes. The texture table is a count/offset pair at
|
||
index offset 0x30, entries of 16 bytes.
|
||
|
||
The table checks out on its own terms: on level 0 it is 93 entries at 0x2900,
|
||
and 0x2900 + 93 x 16 lands exactly on the next range at 0x38.
|
||
|
||
Three details worth writing down:
|
||
|
||
* **K is a signed 12-bit value in 1/16 units**, sign-extended by shifting the
|
||
field up 4 and back down. Retail values are -137..-89, which is -8.56..-5.56
|
||
mip levels. Reading the field as unsigned gives 1911 instead of -137.
|
||
* **`data_hi` of TEX1_1 is MMIN, and 4 is `LinearMipmapNearest`**, not
|
||
`LinearMipmapLinear` as the bit layout alone suggests.
|
||
* **RGBA and the light/normal records are not in the VIF lists.** They sit at
|
||
fixed offsets in the patch's data block: `rgba_size * 4` four-byte colours at
|
||
`rgba_ofs`, and `vert_count` eight-byte records at `light_ofs + 0x10`. The
|
||
origin also lives there — if no STROW row supplied one, the quadword at
|
||
`light_ofs` is the origin.
|
||
|
||
### Verification
|
||
|
||
Every ad-gif must index a texture that exists, or the geometry and the texture
|
||
data have been misread independently of each other. Across all 27 levels:
|
||
**47,950 ad-gifs over 18,102 patches, all indices in range**, referencing
|
||
1429 of the 1595 level texture entries (90%). Each patch's light array is
|
||
exactly `vert_count` records and its RGBA array exactly `rgba_size * 4`.
|
||
|
||
### Texels: 8-bit indexed images and CSM1 palettes
|
||
|
||
Settled. Adds `texture.rs` and makes the pipeline write `gs_ram.bin`.
|
||
|
||
A level texture is not pixels, it is **palette indices**: `width * height`
|
||
bytes at `core_data[textures_base + data_offset]`. The colour comes from a
|
||
256-entry CLUT (1024 bytes) in GS RAM at `palette * 0x100`. `gs_ram` is a byte
|
||
range in the level's data extent header at offset 0x10 — 518,656 bytes on level
|
||
0 — and the extractor was not writing it out, so nothing could be decoded
|
||
before this.
|
||
|
||
Two PS2 conventions have to be honoured, and neither is a design choice:
|
||
|
||
* **CSM1 CLUT order.** Within each 32-entry group the two middle 8-entry
|
||
blocks are swapped: bits 3 and 4 of the index are exchanged. The remap is an
|
||
involution.
|
||
* **Alpha runs 0..=0x80, not 0..=255.** `0x80` is fully opaque and anything
|
||
above it clamps.
|
||
|
||
The mip chain splits across two buffers. Levels 0 and 1 are in the texture
|
||
block, with level 1 immediately after level 0 at `data_offset + w*h`; levels 2
|
||
and 3 are resident in GS RAM at `mipmap * 0x100` and `pad * 0x100`. Tfrag
|
||
textures are square, so every level halves evenly.
|
||
|
||
### Verification
|
||
|
||
Numbers alone do not settle this — every step is a table lookup, so a wrong
|
||
CLUT remap still yields a well-formed image of the wrong thing. Two layers:
|
||
|
||
* **Shape.** All 1,595 tfrag textures across all 27 levels decode; 96.8% are
|
||
fully opaque and 96.3% use more than eight distinct colours. A flat,
|
||
transparent or single-colour result would point at a CLUT or offset mistake.
|
||
* **Eyes.** `gc-extract dump-texture` writes any tfrag texture, or any of its
|
||
mip levels, as a binary PPM. Decoded level 0 texture 0 is a clean bevelled
|
||
metal panel with rivets and specular highlights; texture 74 is a vent grille
|
||
whose punched windows come back fully transparent. Both confirm the CLUT
|
||
remap, the alpha scale and the pixel base at once, which no assertion in the
|
||
suite does.
|
||
|
||
That tool is worth keeping: it is the cheapest way to tell a decode that is
|
||
merely well-formed from one that is right.
|
||
|
||
### LOD selection
|
||
|
||
Mostly settled. The block header's float at +0x08 gives `L`, and the switch
|
||
distances are `[D0, D1, D2] = [6L, 4L, 2L]` world units; on every level `L` is
|
||
32.0, so `[192, 128, 64]`. The distance test compares the patch's bounding
|
||
sphere, truncated to integers, against `trunc(D * 1024)`. Six modes result —
|
||
LOD 2, LOD 1 collapse, LOD 1 morph, LOD 0 collapse, LOD 0 morph, LOD 0 — and
|
||
the *sphere* is what is tested, not the centre, so a patch straddling a
|
||
threshold takes the coarser mode. That is what stops geometry popping as the
|
||
camera moves, and it is worth testing for directly: sweeping the depth must
|
||
never move back to a finer mode.
|
||
|
||
A morph weight runs 0 to 1 across its tier's band: LOD-01 between D1 and D0,
|
||
LOD-0 between D2 and D1. Past the far edge of the band an entry is not morphed
|
||
but *replaced* by its first parent — a collapse, not a blend.
|
||
|
||
### Parent indices: settled, but only after two corrections
|
||
|
||
The four V4_8 arrays (`parent_indices_lod01`, `unk_indices_2_lod01`,
|
||
`parent_indices_lod0`, `unk_indices_2_lod0`) are classified by VU address,
|
||
and **the addresses are not unique**. On level 0 patch 1,
|
||
`unk_indices_2_lod_01_addr == parent_indices_lod_0_addr == 291`, and
|
||
`unk_indices_2_lod_0_addr == indices_addr == 294`. So address matching alone
|
||
cannot separate them, and getting it wrong is silent: the arrays still parse,
|
||
they just hold the wrong numbers.
|
||
|
||
Two things were needed to fix it, and the first attempt at getting it wrong is
|
||
worth recording because it looked like a data problem rather than a code one.
|
||
|
||
**Each V4_8 element is 4 bytes and holds four indices**, so an array of N
|
||
entries is `count * 4` bytes, not `count`. An early analysis compared the
|
||
unpack's element count against the position counts and concluded that 15,000
|
||
patches could not possibly hold enough parents. That was the wrong unit; the
|
||
data was fine.
|
||
|
||
**`unk_06` and `unk_0a` suppress an empty tier.** When the extra vertex-info
|
||
count is zero there is no extra array, and its VU address is simply the next
|
||
tier's. Guarding those two branches on `unk_06 != 0` / `unk_0a != 0` lets the
|
||
address fall through to the array that actually exists. This one change took
|
||
resolved LOD-0 entries from 28,739 to 341,306.
|
||
|
||
### Verification
|
||
|
||
Every vertex-info entry across all 27 levels is now accounted for: of 886,841
|
||
entries, 516,492 are common and have no link by design, and **all 370,349 LOD
|
||
entries resolve** — 29,043 LOD-01 and 341,306 LOD-0 — with zero failures. Each
|
||
resolves to a first parent that is real vertex info, a LOD-01 parent that is
|
||
always a common entry, and three positions inside the patch. A common entry
|
||
never claims a parent.
|
||
|
||
Ruled out along the way: address-only classification (addresses collide), and
|
||
one parent array entry per unpack element (wrong unit).
|
||
|
||
### The origin, and why no parser test could catch it
|
||
|
||
Positions are `i16`, so a single patch spans at most +-32 world units around
|
||
its own origin. The origin comes from an STROW row, and the obvious rule --
|
||
the last non-index STROW in the list -- is **wrong on RAC2**. RAC2 also uses
|
||
STROW for the per-vertex light records, and those rows come *after* the origin,
|
||
so taking the last one yields `[186, 186, 186, 186]` and every patch collapses
|
||
onto the world origin.
|
||
|
||
Nothing in the test suite noticed, and that is the point worth recording:
|
||
|
||
* `vert_count` still matched, because it counts positions, not places.
|
||
* `tri_count` still matched, because the topology never changed.
|
||
* Every parent, morph target and triangle index still resolved, because those
|
||
are all relative to the same broken origin.
|
||
|
||
The entire level fit in a 63 x 62 x 62 unit box centred on the origin, and all
|
||
of it was self-consistent. Only rendering exposed it, and only because the
|
||
screenshot put the camera where the bounding spheres said the terrain was.
|
||
|
||
The rule that works: **the origin is the STROW immediately preceding the first
|
||
V3_16 position unpack** in the common list. With that, level 0's vertices span
|
||
x 86..444, y 46..211, z -477..40, which matches the patch bounding spheres.
|
||
|
||
Lesson for the rest of the project: a parser can be entirely self-consistent
|
||
and still describe the wrong thing. Where a value is absolute rather than
|
||
relative, check it against an independent source -- here, the bounding spheres.
|
||
|
||
### Why the first frame is dark
|
||
|
||
The renderer draws unlit: texture times the tfrag's own per-vertex colour.
|
||
That product is genuinely dark -- level 0 patch 0's colours are `(25, 26, 26)`
|
||
and `(92, 121, 119)` in 0..255 -- and the game's lighting is what brightens it.
|
||
|
||
Trying a host directional light or cranking ambient brightness was the wrong
|
||
instinct and it showed: with no normals the PBR path is degenerate, so
|
||
brightness changes did nothing at all. Faking it would also have hidden the
|
||
gap. The colour attribute is picked up from the mesh automatically, so it is
|
||
the *colour* data that is missing lighting, not a flag that was left off.
|
||
|
||
The real model is `LightTfrags`. The PS2 float model and the pass itself are
|
||
now written (`gc-engine::ps2`, `gc-engine::tfrag_light`); what is missing is the
|
||
light bank. The three inputs were:
|
||
|
||
* the **light records** -- already parsed: azimuth and elevation as *indices*
|
||
into the overlay's sine table, plus the base colour;
|
||
* the **normal table** -- now decoded (`gc_formats::tfrag_light`), and normals
|
||
are built and fed to the renderer;
|
||
* the **light bank** from the level's `gameplay` extent -- **not working**,
|
||
see below.
|
||
|
||
### Point lights are a runtime feature, not a level-load one
|
||
|
||
Every patch on every level has `point_lights == 0xffff` (none) and
|
||
`dir_lights_one == -1`, checked across all 18,102 patches. The eight-slot point
|
||
light bank is filled by gameplay at run time, not read from the disc.
|
||
|
||
So `point_light_vertex` is not missing work for a static level render -- it
|
||
would light nothing. The directional pass *is* the complete lighting for a
|
||
level-load frame, and the point-light path belongs with the hero, explosions
|
||
and the other things that spawn light.
|
||
|
||
`dir_lights_one == -1` on every patch also means the per-vertex `select` word
|
||
is what selects a light set, never the header field -- which is what the
|
||
implementation uses.
|
||
|
||
### The class tables were not unreadable, only unsized
|
||
|
||
`core_index` holds moby, tie and shrub class tables at 0x18, 0x20 and 0x28,
|
||
followed by the texture tables. They had been ruled out for want of an entry
|
||
size, and every field order and stride tried.
|
||
|
||
The sizes are knowable exactly, because **each table ends precisely where the
|
||
next begins**:
|
||
|
||
- moby 238 entries at 192, 32 bytes each: 192 + 238x32 = 7808 = tie's offset
|
||
- tie 74 at 7808, 32 bytes: 7808 + 74x32 = 10176 = shrub's offset
|
||
- shrub 7 at 10176, 48 bytes: 10176 + 7x48 = 10512 = the tfrag texture table
|
||
- tfrag textures 93 at 10512, 16 bytes: 10512 + 93x16 = 12000 = moby textures
|
||
|
||
That closes on **27 of 27 levels**, which is what makes it evidence rather
|
||
than a coincidence: any wrong entry size would break the chain immediately.
|
||
|
||
The class entries are 32 bytes -- `{asset offset, class number, two unknown
|
||
words, 16 bytes of texture slots}` -- and a shrub entry adds 16 bytes of
|
||
billboard slots for 48. Across all 27 levels: **5,796 moby, 1,864 tie and 358
|
||
shrub classes**, and 1,595 tfrag textures.
|
||
|
||
Every shrub class on level 0 carries all-zero billboard slots, so the billboard
|
||
dimensions are optional rather than something to assert on.
|
||
|
||
The failure here is the third instance of the same shape. The gameplay extent
|
||
was "an unidentified compression" because it was being parsed compressed; the
|
||
tfrag record framing was "unrecoverable" because the entry size was never
|
||
known. Both looked like properties of the format and were properties of the
|
||
question.
|
||
|
||
### Where the class tables point
|
||
|
||
`assets_compressed_size` and `assets_decompressed_size` at index offsets 0x88
|
||
and 0x8c are 9,738,678 and 16,969,296 on level 0 -- exactly the sizes of
|
||
`core_data` before and after decompression. There is no separate asset
|
||
container: the "asset WAD" *is* `core_data`, and `offset_in_asset_wad` indexes
|
||
the decompressed form directly.
|
||
|
||
That is checkable rather than merely plausible, and a test checks it on all 27
|
||
levels:
|
||
|
||
* `assets_decompressed_size` equals `core_data`'s exact length;
|
||
* every class offset is either 0 or inside `core_data`;
|
||
* the moby, tie and shrub spans do not overlap each other.
|
||
|
||
An offset of **0 means the slot has no geometry**, not the start of the lump.
|
||
On level 0 the moby table's ninth entry is class 4909 at offset 0.
|
||
|
||
Two assumptions worth discarding: the tables are **not** sorted by class number
|
||
(level 0's moby classes run 0, 1, 2, 14, ... 4909), and the offsets within a
|
||
table are **not** monotonic. Neither is needed, but both look true from the
|
||
first few entries.
|
||
|
||
The chunk descriptors after the named extents are not it. They exist on only
|
||
3 of 27 levels -- level 0's four named extents tile the container exactly --
|
||
so an asset blob there would be missing almost everywhere.
|
||
|
||
At the offsets sit raw GS/VIF packet data, not a further container: moby[0]
|
||
begins `00 00 00 00 3f 16 02 55 6f 6f`, tie[0] `00 01 00 00 70 50 00 00`,
|
||
which read as packet headers rather than a magic number.
|
||
|
||
### Tie: the class header only partly matches
|
||
|
||
Tie is the easy one of the three -- instanced static geometry, no skeleton -- so
|
||
it is the right place to start.
|
||
|
||
RAC1's class header is 0x80 bytes. RAC2 agrees with it over the first `0x48`:
|
||
the per-LOD packet offsets, the three LOD distances and `o_class` all check out.
|
||
From `0x48` on it does not. `lod_info` at `0x50` reads back as noise,
|
||
`packet_count` and `texture_count` are zero on every class examined, and the
|
||
`normals` offset at `0x0c` points 117,788,434 bytes into a 16.9 MB lump.
|
||
|
||
So only the agreed-on prefix is decoded. The evidence for trusting it:
|
||
|
||
- **`o_class` at `0x44` equals the core index's class number on all 1,864 tie
|
||
classes across all 27 levels.** A field landing on the right number that many
|
||
times is not a coincidence, and it is what pins the header base.
|
||
- The LOD distances are ordered `0 < near < mid < far` for **1,784 of 1,864**.
|
||
The 80 exceptions are real, unexplained, and are reported rather than
|
||
asserted away.
|
||
|
||
Level 0's first tie class has distances 12.0 / 25.0 / 40.0, which is a
|
||
plausible prop LOD band, and packet offsets 256, 20,592 and 26,048.
|
||
|
||
The remainder is not guessed at. A field order that produces plausible output
|
||
and is wrong is the specific failure this file keeps recording -- the gameplay
|
||
extent, the tfrag record framing and the class tables have all been it.
|
||
|
||
### Why the frame is still dark: the light bank does not parse on RAC2
|
||
|
||
The reference frames the bank as "a u32 offset at +0x04, then an `i32` count
|
||
there, then the sets". On RAC2 level 0 that offset is `0xa0d`, and the `i32`
|
||
there is `698810368` -- not a count, and not a plausible one. `gameplay.bin`
|
||
begins with `0x794e...`, not with a header of the shape the reference expects,
|
||
so RAC2's gameplay extent is not laid out this way, or is compressed before
|
||
this point.
|
||
|
||
The pass therefore never runs. Rather than let the renderer quietly draw the
|
||
stored colours and look plausible, it counts what it lit and says so:
|
||
|
||
```text
|
||
lighting: 0 of 80274 vertices shaded by LightTfrags; the rest fall back to the
|
||
tfrag's stored colours (no usable light bank)
|
||
```
|
||
|
||
That line is the honest state, and it is why the frame has not changed. What
|
||
`gameplay.bin` actually is, and where RAC2 keeps its light bank, is the open
|
||
question — `0x794e...` looks like it may be a compressed or encrypted
|
||
container, in which case the bank is somewhere else entirely.
|
||
|
||
### The PS2 float model
|
||
|
||
`gc-engine::ps2` is the EE's FPU on raw `u32` bit patterns, and it is not
|
||
`f32`:
|
||
|
||
* **no denormals** -- exponent 0 *is* zero, whatever the mantissa
|
||
* **no NaN, no infinity** -- exponent 255 is an ordinary value, which means
|
||
`MAX` reads back in Rust as NaN. A test asserts exactly that, because it is
|
||
the clearest single statement of why this module exists.
|
||
* **every result truncates toward zero**, and `1/3` comes out below `1/3`
|
||
* **the adder aligns operands**: the smaller operand loses the bits more than
|
||
one below the larger's LSB, and a 25-bit exponent gap returns the larger
|
||
operand unchanged
|
||
* **`rsqrt` is sqrt-then-divide**, with truncation between, not one rounded step
|
||
|
||
`mul`, `add`, `div`, `sqrt`, `itof12` and `max`/`min` all differ from `f32` in
|
||
at least one case a lighting pass will hit. The pass accumulates them across
|
||
every vertex of every patch, so this is a visible difference rather than a
|
||
rounding curiosity.
|
||
|
||
### The lighting pass itself
|
||
|
||
Per vertex: `base + A.rgb * max(dA, dA*wA) + B.rgb * max(dB, dB*wB)`, where
|
||
`base` is the record's 5:5:5:1 colour expanded by PEXT5, `dA`/`dB` are
|
||
three-component dot products against the light directions, and `wA`/`wB` come
|
||
out of the colour w lanes. `select` picks one light set or blends two by
|
||
`bits 8..15 / 256`, and a blended direction is re-normalised where an unblended
|
||
one is not.
|
||
|
||
One field had to be added to `TfragLight` to carry this: `select`, the `u16` at
|
||
+6 of the record. It was briefly reconstructed with an `unsafe` pointer cast
|
||
over the struct -- which is exactly the kind of thing that breaks silently the
|
||
moment a field is reordered, and is now a plain parsed field.
|
||
|
||
### The normal table, and why it is found by searching
|
||
|
||
The reference reads the table from an ELF at `BOOT_NORMAL_TABLE_VADDR`
|
||
(`0x165500`) and notes every level overlay carries a copy. **The overlay has no
|
||
ELF header** -- it is a raw loadable image starting with code -- so there is no
|
||
program header to map a virtual address through, and the offset is not fixed
|
||
either: level 0 has the table at `0xb650c`, level 5 at `0xcca30`, level 26 at
|
||
`0xb6284`.
|
||
|
||
So it is located by content. Every table starts with exactly `(1.0, 0.0)` --
|
||
angle zero -- and the scan then requires all 256 entries to match
|
||
`cos(i·pi/128), sin(i·pi/128)`. Validating the whole curve is what makes the
|
||
search safe: a lone `(1.0, 0.0)` pair of floats occurs by accident in a 2.7 MB
|
||
executable, and a test plants one as a decoy. The values read from the disc are
|
||
used as-is rather than recomputed, since those are the numbers the game indexes.
|
||
|
||
A normal is `(cos(az)cos(el), sin(az)cos(el), sin(el))` **negated** -- the
|
||
stored normals point into the surface -- then given the same Z-up to Y-up swap
|
||
as positions, or the lighting comes out sideways. The azimuth and elevation
|
||
fields are stored signed but index 0..=255, so the byte is what matters, not its
|
||
sign: entry 200 is read as 200, not as -56.
|
||
|
||
Given the table, a normal is `(cos(az)cos(el), sin(az)cos(el), sin(el))`,
|
||
negated, exactly as `decode_normal` builds it. The lighting pass itself is
|
||
`base + A.rgb * max(dA, dA*wA) + B.rgb * max(dB, dB*wB)` per vertex, and it
|
||
must be reproduced in the PS2's float model rather than in Rust `f32`, per
|
||
the fidelity rule.
|
||
|
||
### Still open
|
||
|
||
- **Point lights.** The directional pass runs; `point_light_vertex` and its
|
||
`r^2 - |p - L|^2` cull test are not written, and they are the one part of
|
||
the lighting that *is* distance-dependent.
|
||
- **The GS register set** the level load builds from the texture entry
|
||
(TBP0/TBP1 and so on), for a renderer that emulates the GS rather than
|
||
resolving texture addresses on the host.
|
||
- **The guard-band clip test** that sends patches crossing the guard band to
|
||
the clipping program.
|
||
- ~~**Lighting magnitudes.**~~ **Settled 2026-10-11** by the PCSX2
|
||
comparison loop (`docs/plan/pcsx2-verification.md`): level 0's 162,400
|
||
colour bytes match the game's EE RAM exactly. Two bugs the comparison found
|
||
and fixed, both of which produced output that looked plausible:
|
||
`vertex_normal_bits` negated x and y but not z (so `max(dA, dA*wA)` took the
|
||
wrong branch and every vertex lit near its base colour -- 95.88% agreement
|
||
instead of 100%), and a record whose `select` names a set the bank lacks
|
||
drew transparent black where the game writes the record's base colour (3,130
|
||
of level 0's 40,600 vertices, `select 0x000f`).
|
||
- **tie, moby and shrub geometry.** The class tables parse and the container
|
||
they point into is identified (below), but none of the three is decoded yet.
|
||
Moby is the largest of the three and is skeletal, with an animation evaluator
|
||
in the microcode; it is the single biggest piece left before the rest of a
|
||
level can draw.
|
||
- **Point lights** are spawned by gameplay at runtime and are absent from the
|
||
disc -- every one of the 18,102 patches has `point_lights == 0xffff` -- so
|
||
there is nothing for them to light until the game runs.
|
||
|
||
A tfrag can now be turned into positioned, UV-mapped, textured, lit triangles,
|
||
told which LOD to draw, and morphed toward its parents at the right rate.
|
||
Everything needed for a first frame is parsed, and the frame is verified by
|
||
screenshot rather than by assertion.
|
||
|
||
## Core index field order (2026-10-11)
|
||
|
||
Repeatedly re-deriving these by hand wasted real time this session, so the
|
||
order is written out. It matches `LevelCoreHeader::parse` in
|
||
`crates/gc-formats/src/level_core.rs`:
|
||
|
||
| offset | field |
|
||
|--------|-------|
|
||
| 0x00 | gs_ram count |
|
||
| 0x04 | gs_ram offset |
|
||
| 0x08 | tfrags |
|
||
| 0x0c | occlusion |
|
||
| 0x10 | sky |
|
||
| 0x14 | collision |
|
||
| 0x18 | moby_classes count |
|
||
| 0x1c | moby_classes offset |
|
||
| 0x20 | tie_classes count |
|
||
| 0x24 | tie_classes offset |
|
||
| 0x28 | shrub_classes count |
|
||
| 0x2c | shrub_classes offset |
|
||
|
||
Note `gs_ram` and the three class tables are **count/offset pairs**, so every
|
||
one of them is two words. The region offsets (`tfrags` through `collision`) are
|
||
single words. Counting the class tables as one word each shifts every field
|
||
after them, which is how `collision` got read as `sky` and produced a page of
|
||
implausible collision headers on all 27 levels.
|
||
|
||
## Sky (2026-10-11)
|
||
|
||
Region is `[sky, collision)` and is well defined, but the header is **not**
|
||
yet decoded and the layout is not uniform.
|
||
|
||
Reading `u16` at the head, most levels give:
|
||
|
||
```
|
||
+0x00 0
|
||
+0x02 0
|
||
+0x04 4 or 6 varies per level
|
||
+0x06 0
|
||
+0x08 0 level 2 is the exception
|
||
+0x0a 3..6 looks like a layer count
|
||
+0x0c 0
|
||
+0x0e 64 (0x40)
|
||
+0x10 128 or 192 varies
|
||
+0x12 64 (0x40)
|
||
```
|
||
|
||
Level 2 breaks the pattern: `+0x08` is 0x0109 and the three words after the
|
||
apparent count are shifted and different (`80 00 00 00 00 01 00 00 40 00`).
|
||
So a single fixed header does not fit every level, and the count at `+0x0a`
|
||
cannot be trusted to be one.
|
||
|
||
A table of offsets follows at `+0x20` for the layers. The offsets ascend and
|
||
sit in the last 8-15% of the region, but the count of *non-zero* offsets is
|
||
sometimes less than the declared count (level 26 declares 6, has 4 real
|
||
offsets and 2 zero slots). Unresolved.
|
||
|
||
## Moby geometry (2026-10-11)
|
||
|
||
Class entries are 0x20 bytes in `core_index.bin`, fields
|
||
`geom_offset, o_class, unknown_8, unknown_c, textures[16]`; all 238 level-0
|
||
moby classes carry `textures == ffff..ff`, i.e. no texture slots.
|
||
|
||
`geom_offset` points into `core_data`, and the spans tile without overlap.
|
||
|
||
The span does not start with the geometry. Its first `u32` is the offset of a
|
||
part table, and the table is 16-byte records:
|
||
|
||
```
|
||
+0x00 u32 vertex_offset from the start of the class
|
||
+0x04 u16 vertex_count
|
||
+0x06 u16 second_count meaning not established
|
||
+0x08 u32 next_part
|
||
+0x0c u32 flags
|
||
```
|
||
|
||
Parts tile: `vertex_offset + 16 * vertex_count == next_part`. The stride is not
|
||
assumed -- it falls out of that identity, which is what pins the record size
|
||
and the field boundaries. It holds for 43,467 of 43,467 part records across
|
||
every moby class on all 27 levels.
|
||
|
||
Two terminations are needed, not one. A zero `vertex_offset` ends the run, but
|
||
172 records across the disc have a zero `vertex_offset` *and* a zero
|
||
`next_part` while still carrying a vertex count; stopping only on the second
|
||
is what leaves 99.6% instead of 100%. A zero `next_part` means the record
|
||
starts a different table rather than a part.
|
||
|
||
Totals: 1,270,802 vertices over 4,560 class spans, 116 of which carry no
|
||
geometry.
|
||
|
||
Vertices are 16 bytes of VIF-packed half-float quads, cycling through four
|
||
distinct 32-bit words. Not unpacked -- handed back raw.
|
||
|
||
## Shrub geometry (2026-10-11)
|
||
|
||
Class entry strides differ between the three tables. Measured by how many
|
||
entries yield a geometry offset inside `core_data` and a plausible `o_class`,
|
||
across all 27 levels:
|
||
|
||
| table | stride | sane entries |
|
||
|-------|--------|--------------|
|
||
| moby | 0x20 | 5769 / 5796 |
|
||
| tie | 0x20 | 1864 / 1864 |
|
||
| shrub | 0x30 | 356 / 358 |
|
||
|
||
Reading shrub at 0x20 still yields 231 sane-looking entries out of 358, which
|
||
is the problem: it is wrong often enough to look right. The stride is already
|
||
handled -- `ShrubClassEntry` is 48 bytes, carrying billboard texture fields the
|
||
other two do not have.
|
||
|
||
Shrub geometry is **not** the moby part-table format. All 358 spans have a
|
||
leading word of 0 or one too large to index a table into, so none of them
|
||
contain a moby-style part table.
|
||
|
||
The header does self-identify in two places, both exact across all 358 spans:
|
||
|
||
- `span + 0x24` is a `u32` equal to the class entry's own `o_class`. 358/358.
|
||
- `span + 0x2c` is a `u32` data offset that lands inside the span. 358/358.
|
||
|
||
`span + 0x10` reads 5.0f on 281 of 358 spans, so it is a value rather than a
|
||
constant marker and carries no structural weight. The header is otherwise
|
||
undecoded: the first 0x20 bytes are packed data rather than floats, despite
|
||
`+0x10` landing exactly on 5.0.
|
||
|
||
## Tie geometry (2026-10-11)
|
||
|
||
Tie does **not** reuse moby's part-table format. The rule
|
||
`vertex_offset + 16 * vertex_count == next_part` holds for **0 of 25,024**
|
||
records tried across the 1,864 tie class spans, against 43,467/43,467 for
|
||
moby. Tie's second field also reads as an absolute offset rather than a
|
||
span-relative one, which is the first sign the record shape differs.
|
||
|
||
At `+0x48`, where the verified header currently ends, the bytes are f32-shaped
|
||
(10.0, 40.0, 50.0, 4.0, -100.0, -25.0, 5.0, 3.125), not a VIF marker as
|
||
expected. They are not a fixed-size array either: the run of consecutive
|
||
plausible floats varies from 1 to 6 across spans, with 822 spans having 6,
|
||
455 having 1 and 373 having 2. So there is more header past `0x48`, and it
|
||
is not a single array of distances.
|
||
|
||
### Probe tool
|
||
|
||
`gc-extract probe <unpacked-dir> <level> [max-rows]` dumps the core header and
|
||
all three class tables through `LevelCoreHeader`, which is the point. The
|
||
table above and the shrub offset below were both established by hand first and
|
||
one of them nearly sent me to fix a bug that was not there. The tool also
|
||
reports how many entries carry geometry inside `core_data`, which is how the
|
||
shrub stride was pinned: 7/7 at 0x30 against 4/7 read at 0x20.
|
||
|
||
Note the LOD distances already decoded sit *before* this point (12.0, 25.0,
|
||
40.0 at `+0x10`), and the floats after `0x48` are a different population --
|
||
negative values, and a repeated leading 40.0.
|
||
|
||
Unresolved. The packet geometry remains the open question, unchanged.
|
||
|
||
## Shrub body (2026-10-11, later)
|
||
|
||
Stride-fitting the body failed three times, so this came from a different
|
||
angle: group spans by `(count, data_offset, tail_length)` and byte-diff every
|
||
member of each group. Byte positions that differ across a group are geometry;
|
||
positions that never differ are structure.
|
||
|
||
The result is blunt. **144 bytes vary in every group**, whether the group has
|
||
count 1, 2, 3, 5 or 6 and a tail of 192, 208 or 240 bytes. So the varying data
|
||
is not sized by `count` at all, and the three strides that failed were fitting
|
||
the wrong thing.
|
||
|
||
Those 144 bytes sit in runs of 6, every 8 bytes: `(0..5), (8..13), (16..21),
|
||
...`. The body is an array of **8-byte records, 6 varying and 2 constant**.
|
||
|
||
The payload is not half-floats. Decoding the 6 bytes as a `half3` gives 245.25
|
||
and -168.375, which no shrub has.
|
||
|
||
**Correction, later the same day.** This section originally concluded the
|
||
payload was VIF-packed, on the strength of recurring patterns like `7ea5` and
|
||
`1ab6`. That conclusion does not survive a control. Decoding every 32-bit word
|
||
of every shrub body as a `VifCode`:
|
||
|
||
| region | words decoding as UNPACK | distinct cmd bytes |
|
||
|---|---|---|
|
||
| shrub bodies | 24.9% | 128 (all of them) |
|
||
| uniform random | 25.1% | 128 (all of them) |
|
||
| sound_bank (definitely not VIF) | 19.0% | 122 |
|
||
| gs_ram (mixed) | 13.6% | 128 |
|
||
| moby class 0 span (known VIF) | 40.1% | 105 |
|
||
|
||
Shrub sits on the random baseline, and all 128 possible command bytes occur,
|
||
which is what unstructured data looks like. The VIF reading was pattern
|
||
matching on a handful of recurring byte values, and it was wrong.
|
||
|
||
Note the test is weak in general: even the known-VIF region only reads 40.1%.
|
||
So this does not prove shrub is *not* VIF -- it proves the claim was never
|
||
established. Treating three recurring bytes as a format is the same mistake as
|
||
the three stride guesses, just at a different level.
|
||
|
||
What survives from this section is the differential finding, which rests on
|
||
grouping and diffing rather than on recognising byte patterns: 8-byte records,
|
||
6 varying and 2 constant, 144 varying bytes per group regardless of `count`.
|
||
The payload's encoding is still unknown.
|
||
|
||
### Shrub span bounds are wrong (later still)
|
||
|
||
A follow-up measurement contradicts the body size implied above, and the
|
||
reason is a bug in how spans are bounded rather than anything in the format.
|
||
|
||
Shrub class spans were bounded by the *next shrub offset*, with the last one
|
||
running to the end of `core_data`. On level 0 that makes the final shrub span
|
||
**925,008 bytes** -- a single shrub class absorbing most of the tail of the
|
||
lump. The moby-to-tie boundary is clean by comparison: last moby offset
|
||
13414592, first tie offset 13416192, a gap of exactly 1600 bytes.
|
||
|
||
Consequences, stated plainly:
|
||
|
||
- The "3.5 million records / 28 MB of shrub body" figure is an artifact of
|
||
that oversized final span and should not be quoted.
|
||
- The payload appearing uniform-random across the full u16 range is measured
|
||
over that same contaminated range, so it is not evidence about shrub
|
||
geometry either way.
|
||
- The differential result (8-byte records, 144 varying bytes) was computed on
|
||
*groups*, and the specific groups used had bounded spans, so that
|
||
measurement stands. The global statistics built on top of it do not.
|
||
|
||
What is needed: the offset that ends the shrub region. It is not in the core
|
||
index, and unlike moby->tie there is no obvious successor boundary. Until
|
||
that is found, shrub span sizes are unreliable and anything summed over all
|
||
358 spans is suspect.
|
||
|
||
This is the second time in this format that a confident aggregate hid a bad
|
||
bound -- the first was the VIF reading. The pattern is worth noting: aggregates
|
||
across many spans are only as trustworthy as the span bounds underneath them,
|
||
and those are the easy thing to get wrong because nothing errors.
|
||
|
||
## Shrub span bounds: resolved (2026-10-11, later still)
|
||
|
||
The offset that ends the shrub region is the **start of the chunk container**,
|
||
and it is found by signature rather than read from the core index, which has no
|
||
successor offset for shrub.
|
||
|
||
**Range measured:** for each level, from the last shrub class's geometry offset
|
||
to the constant trailer. Nothing outside that range is read.
|
||
|
||
**Method:** scan forward from the last shrub class for two bytes `f6 7c`
|
||
sitting immediately in front of a chunk signature (16-byte header ending `0xff`,
|
||
ten zero bytes, four or more `0xf0`-tagged offsets). Each half of that
|
||
signature is weak alone -- the marker pair occurs 25 to 103 times per lump --
|
||
and together they are exact.
|
||
|
||
**Result: on all 27 levels the marker sits exactly two bytes before the first
|
||
chunk header.** The two parts of the signature are never found apart, and no
|
||
chunk signature occurs inside any last shrub body. The three levels without
|
||
shrub (10, 25, 26) carry the same marker before the container that closes
|
||
*tie*, so the locator is a property of the region, not a shrub rule.
|
||
`gc_formats::chunks::chunk_region_start` implements it and
|
||
`regions::core_layout` closes shrub with it.
|
||
|
||
Consequences, all measured over the new bounds:
|
||
|
||
- **All 358 shrub class bodies are bounded and well formed** -- every one is
|
||
whole 8-byte records. That was not assertable while the last class in each
|
||
level ran out to the lump's end, and it is the check the boundary earns.
|
||
- **9,600 records in total**, against the 8,938 previously measurable in
|
||
successor-bounded spans; 662 of them sit in the 24 last-class bodies, 24 to
|
||
30 per level. The retracted "3.5 million records / 28 MB of shrub body"
|
||
figure stays retracted: the correct total for the shrub region is
|
||
**2,033,776 bytes** across the 24 levels that have one.
|
||
- The chunk container is **25,775,850 bytes across 27 levels** (1,236 chunks,
|
||
22,562 records), not an ~690-900 KB afterthought per level. The earlier
|
||
container measurements were taken over "the 900,000 bytes before the
|
||
trailer", a window that did not align to the container's real extent: on
|
||
level 0 it is 690,302 bytes, and 27 of 1,123 chunks were truncated at that
|
||
invented edge. The chunk test now reads the region from `core_layout`.
|
||
|
||
What is still not known: the marker's meaning (constant across 27 levels, at
|
||
exactly this position, unnamed). The shrub body payload's encoding is the next
|
||
section.
|
||
|
||
## The shrub class blob: palette and billboard (2026-10-11, after the bound)
|
||
|
||
With every span bounded by the chunk container, the shrub class blob reads
|
||
cleanly against the R&C1 format (`vendor/rerac/docs/formats/shrub_sky_rac1.md`,
|
||
derived from the Wrench sources). R&C1's shrub class header keeps the same
|
||
field offsets R&C2 shows -- `o_class` at 0x24, a count at 0x28, the data offset
|
||
at 0x2c -- so the reference says which bytes to look at and the disc says what
|
||
they hold.
|
||
|
||
**Range measured:** all 358 shrub class spans over the 24 levels that have
|
||
them, each bounded by its successor class or the located chunk container.
|
||
Nothing outside the spans is read.
|
||
|
||
### The body is a 24-entry normal palette
|
||
|
||
Two things made this identifiable where three stride-fitting attempts had
|
||
failed. First the shape: every body is 8-byte records whose last two bytes are
|
||
zero, and R&C1's palette is documented as 24 records of exactly that shape
|
||
(`s16` x, y, z, pad). Second the count: bodies hold 24, 26, 28 or 30 records
|
||
with the extra ones **always all zero and always at index 24 or later** --
|
||
never a live entry. So the body is 24 normals plus zero padding, and the
|
||
padding is what distinguishes it from geometry.
|
||
|
||
The palette is real normals, and the test says how firmly:
|
||
|
||
- all 8,592 live entries have squared length within **65,535** of `32767^2`,
|
||
and the bound is tight -- the largest observed deviation is exactly 65,535.
|
||
That number is not fitted: it is `32768^2 - 32767^2`, which is what a unit
|
||
vector scaled by 32768 produces when an axis-aligned normal quantises to
|
||
`-32768` while the positive axis clamps to `32767`. Components run the full
|
||
`s16` range (-32768..32767) and 1,765 of the 8,592 entries are axis-aligned,
|
||
which is what a coarse direction codebook looks like.
|
||
- a random `s16` triple has squared length spread about a *billion* either
|
||
side of `32767^2`, so one entry in fifteen thousand would pass by chance.
|
||
All 8,592 pass.
|
||
- the clincher: 17 classes carry a palette of 24 identical `(0, 0, 1)`
|
||
entries. A position or index table cannot produce that; a normal palette for
|
||
a flat-facing class must.
|
||
|
||
This is also where shrub lighting comes from -- R&C1 lights each instance
|
||
against its class's 24-direction palette. Unlike the tfrag's per-vertex light
|
||
records, which store azimuth/elevation *indices* into the overlay's sine table,
|
||
the shrub palette stores the directions outright, so the overlay table is not
|
||
needed for shrubs.
|
||
|
||
### The header, byte by byte
|
||
|
||
**Range measured:** the 0x30 bytes of every span.
|
||
|
||
| offset | field | measured |
|
||
|---|---|---|
|
||
| 0x00 | f32[4] `sphere` | centre (xyz) + radius (w). Finite on all 358, **radius positive on all 358** (1.72..41.57), centres near the class's own origin |
|
||
| 0x10 | f32 `size` | 3.0..20.0, positive on all 358 |
|
||
| 0x14 | u16 `mode_bits` | **0 on 342 spans, 4 on 16** |
|
||
| 0x16 | u16 `scratch` | **0 on all 358** (asserted by the parser) |
|
||
| 0x18 | u32 `reserved` | **0 on all 358** (asserted by the parser) |
|
||
| 0x1c | u32 `billboard_offset` | 0 on 333 spans, in-span on 25 |
|
||
| 0x20 | f32 `scale` | 0.0044..1.8, positive on all 358 |
|
||
| 0x24 | u32 `o_class` | equals the class entry's own, 358/358 |
|
||
| 0x28 | u32 `count` | 1..55, meaning not established |
|
||
| 0x2c | u32 `data_offset` | inside the span, 358/358 |
|
||
|
||
Two corrections this produces, both recorded because the old readings were
|
||
confidently wrong rather than merely unverified:
|
||
|
||
- **"reserved is 0 or a denormal" was the wrong shape.** The 16 spans with a
|
||
non-zero value there have `mode_bits = 4` and `scratch = 0`, not the other
|
||
way round. As an `f32` the word at `+0x14` is still a denormal, so the PS2
|
||
reads zero either way -- but the field is a flag word, not a magnitude, and
|
||
the demo is the point: the EE's FPU has no denormals, so `04000000` and zero
|
||
are the same value to the hardware.
|
||
- **"bounds" is a bounding sphere, not a min/max box.** The old reading tested
|
||
`min.xyz/max.xyz` ordering, found it on only 146 of 358 spans, and stopped
|
||
naming the field. It is centre + radius, as in R&C1: the fourth float is
|
||
positive on all 358 spans and the centres cluster near the class origin
|
||
within +-24, which is where a class-local sphere belongs. The 146/358 figure
|
||
was a property of the wrong question.
|
||
|
||
### The billboard record
|
||
|
||
The billboard offset at `+0x1c` is the strongest self-identification in the
|
||
header after `o_class`, because it is **confirmed from a second file**: it is
|
||
non-zero on exactly the 25 classes whose core-index entry carries billboard
|
||
texture fields, in both directions -- 25 records and 25 entries, pairing
|
||
correctly. R&C1 states the same correspondence ("a class has a billboard record
|
||
exactly when its core-index billboard texture has width != 0").
|
||
|
||
At the offset sits R&C1's `ShrubBillboard` shape, register block and all:
|
||
|
||
| offset | field | measured |
|
||
|---|---|---|
|
||
| 0x00 | f32 `fade_distance` | 5..1000; 512 reads as 0 under R&C1's `trunc(fade) & 0xff`, i.e. billboard-only |
|
||
| 0x04 | f32[3] | **not established** -- see below |
|
||
| 0x10 | TEX1_1: K, MMIN, address | K -233..-168, MMIN 4 on all 25, address 0x14 |
|
||
| 0x20 | TEX0_1: data_lo, address | index 1..4, address 0x06 |
|
||
| 0x30 | MIPTBP1_1: address | 0x34, data words zero (filled at level load) |
|
||
|
||
The three floats at `+0x04..+0x10` are the part that does *not* carry over.
|
||
R&C1's are `width`, `height` and `z_ofs` in class units; R&C2's are an order of
|
||
magnitude too large and too varied to be dimensions of a 32..128 px billboard,
|
||
and they **repeat byte-identically across levels**: 16 distinct triples among
|
||
the 25 records, with one appearing on four classes across levels 1, 4, 12 and
|
||
21. Dimensions would not do that; shared art assets would. They are handed back
|
||
unnamed.
|
||
|
||
Two further facts about them, measured: the whole record repeats with them --
|
||
three classes on different levels carry identical `fade`, floats, K and scale
|
||
-- and the class entries they belong to carry identical `sphere` too. So the
|
||
blobs themselves are shared between levels, which is the same behaviour the
|
||
chunk container showed with its byte-identical tables.
|
||
|
||
### The packets: the mesh geometry is in the blob (2026-10-11, later)
|
||
|
||
The open question above -- where the mesh geometry lives -- answered itself
|
||
once the running game was consulted. The running PCSX2 loads shrub classes
|
||
into RAM, and diffing a loaded blob against its disc bytes shows what the
|
||
loader rewrites:
|
||
|
||
- **`+0x40` holds a table of `count` entries of `(offset, size)`**, both
|
||
relative to the span start on disc, rewritten to absolute addresses in RAM
|
||
with the sizes unchanged. Verified on level 1 class 1: all 21 entries match
|
||
the disc offset for offset and the size for size after adding the blob's
|
||
RAM base.
|
||
- **`+0x2c` (the palette offset) is relocated** the same way, to an absolute
|
||
pointer.
|
||
- **`+0x18` -- zero on disc, asserted by the parser -- is written with a
|
||
pointer at load**, to a table of 0x20-byte records that look instance-shaped
|
||
(a position, a size, a count, and a `0xffff` light slot). Not established;
|
||
recorded because it is where the next question starts.
|
||
|
||
So the blob is header, packet table, packets, [billboard], palette -- the same
|
||
shape as R&C1's, and the mesh was here the whole time. The reason it was
|
||
missed is worth writing down: **`data_offset` points at the *end* of the
|
||
span**, so reading the body as `data_offset..span_end` accounts for the last
|
||
few hundred bytes and leaves the 20 KB in front of it unexamined. A bound that
|
||
is correct and a body that is small enough to look plausible are together
|
||
enough to hide a region.
|
||
|
||
**Range measured:** every shrub class span on the disc -- 358 spans over 24
|
||
levels, each bounded by its successor class or the located chunk container.
|
||
|
||
**Method:** for each span, read `count` from `+0x28`, then the entries at
|
||
`+0x40`, and require them to tile: the first packet starts at
|
||
`align16(0x40 + count * 8)` -- the table, rounded up to 16 -- every entry is
|
||
exactly `size` past the previous one, and the last ends at the palette, or
|
||
exactly 0x40 short of it on the spans that carry a billboard record in
|
||
between.
|
||
|
||
**Result: 358 of 358 tile exactly, holding 2,239 packets.** No span's
|
||
deviations: the first-packet offsets, the sizes and the ends all agree with
|
||
the rule. That is what pins the field meanings -- `count` is the **packet
|
||
count** (1..55 on the disc), and `data_offset` is the palette's offset -- and
|
||
it is why the earlier "the body is not sized by count" observation was true
|
||
for a reason that had not been understood: the body is not sized by the count
|
||
because the count sizes the packets.
|
||
|
||
`gc_formats::shrub::Shrub::packets()` exposes them, and returns `None` rather
|
||
than a partial list when a table does not tile.
|
||
|
||
### Still open
|
||
|
||
- The three billboard floats, `size`, and `mode_bits` (4 on 16 spans).
|
||
- The packet contents. They are VIF command lists in R&C1; the R&C2 packets
|
||
have not been decoded, and the VIF reading of a shrub body was already
|
||
retracted once, so this needs a control, not a pattern match.
|
||
- The 0x20-byte records the loader points at from `+0x18.
|
||
|
||
## Shrub mesh packets: a 16-byte header and 16-byte records
|
||
|
||
The packet table decode (commit aa2b14a) left the packet *contents* open, with
|
||
a warning that a VIF reading had already been retracted once and so needed a
|
||
control rather than a pattern match. This is that measurement.
|
||
|
||
**Range measured:** level 0, the 29 packets of the 6 successor-bounded shrub
|
||
spans, 26,640 bytes. Per-class and per-packet; nothing pooled.
|
||
|
||
**Method:** autocorrelation within the largest packet against a
|
||
distribution-matched shuffled control, then per-byte distinct counts across all
|
||
29 packets.
|
||
|
||
**Result 1 -- the packets tile at 16 bytes.** All 29 packet sizes are exact
|
||
multiples of 16, and the internal stride scan peaks at 16 (+0.283) with 32,
|
||
48, 64, 80 and 128 all elevated behind it -- the signature of a record array
|
||
rather than a single coincidental spacing.
|
||
|
||
**Result 2 -- record 0 is a 16-byte header.** Across all 29 packets these bytes
|
||
are constant: 0-3 `04 04 00 01`, 4-11 all zero, 12-13 `00 80`, 15 `6c`. Bytes
|
||
14 and 16-23 vary. The `04 04 00 01` prefix is constant and looks like a magic
|
||
or format version.
|
||
|
||
**Result 3 -- header byte 14 is echoed in the body.** Header byte 14 takes
|
||
values 06, 07, 0e and 11 across packets, and in every packet the same value
|
||
appears at offset 12 of record 1. That correspondence holds on all 29 packets
|
||
sampled, so it is a real repeated field rather than a coincidence of two
|
||
values happening to be small.
|
||
|
||
**Not established.** No header field matches the record count: the packet holds
|
||
1056/16 = 66 records, and none of the candidate u32 or u16 header offsets at 8,
|
||
10, 12, 14, 16, 18 or 20 equals 66, 65 or 67 on more than half the packets. So
|
||
the record count is either computed from a size field elsewhere in the span, or
|
||
the records are variable length and merely 16-byte aligned.
|
||
|
||
That distinction matters and is unresolved: if records are fixed 16 bytes the
|
||
payload is a flat table; if they are variable, the 16-byte alignment is a
|
||
coincidence of every record happening to be a multiple of 16 long. The stride
|
||
scan cannot separate the two, because both produce a peak at 16.
|
||
|
||
**Follow-up: the records are two 8-byte halves, each with a tag byte.**
|
||
|
||
**Range measured:** level 0, the 1,636 sixteen-byte records following the
|
||
header across all 29 packets. Per-byte distinct counts, not pooled.
|
||
|
||
**Method:** count distinct values and zero frequency at each of the 16
|
||
positions; then tabulate the joint distribution of the two low-entropy
|
||
positions.
|
||
|
||
**Result.** Fourteen of the sixteen positions are uniformly varied -- 101 to
|
||
153 distinct values, zero in 13% to 58% of records. Two are not:
|
||
|
||
| byte | distinct | values (count) |
|
||
|---|---|---|
|
||
| 7 | **4** | 0 (1432), 0x30 (135), 0x6d (40), 0x80 (29) |
|
||
| 15 | **2** | 0 (1618), 0x6d (18) |
|
||
|
||
Both sit in the last byte of an 8-byte half, eight bytes apart, and both take
|
||
values from a small vocabulary rather than the full byte range. The joint
|
||
distribution is sparse and shows no case where both are non-zero except
|
||
`0x6d` appearing in either position but never both:
|
||
|
||
(0,0) 1414 (0,0x6d) 18 (0x30,0) 135 (0x6d,0) 40 (0x80,0) 29
|
||
|
||
**Reading.** Each 8-byte half carries a tag or subtype byte in its last
|
||
position, drawn from a small closed set. The shared value `0x6d` in both
|
||
positions is what suggests one vocabulary used twice rather than two unrelated
|
||
byte fields.
|
||
|
||
**What this settles about variable length.** There is **no per-record length
|
||
field**: the best candidate offsets hold the value 16 on at most 13% of
|
||
records, which is the rate small integers appear by chance, not a self-
|
||
describing record. So the records are fixed 16 bytes, and the unresolved
|
||
question from the previous entry is answered -- the tiling is not a coincidence
|
||
of records that happen to be multiples of 16 long.
|
||
|
||
**Still open:** what the four and two tag values mean, and what the 14 varied
|
||
bytes encode. The tag vocabulary is small enough to enumerate against known
|
||
PS2 primitive lists, which is the obvious next reference to bring in.
|
||
|
||
**The retracted VIF reading is not being revived.** Nothing here identifies a
|
||
VIF stream: the 16-byte tiling is uniform across all 29 packets with no
|
||
command-shaped tail, and a VIF packet would show a changing command byte in its
|
||
last position rather than the constant `6c` at header byte 15.
|
||
|
||
## Shrub packet headers: a fixed disc signature, rewritten on load
|
||
|
||
The packets found by the `+0x40` table have a header the whole disc agrees on.
|
||
|
||
**Range measured:** every packet on the disc -- **2,056 of them**, across the
|
||
334 successor-bounded spans -- read at each packet's span-relative offset.
|
||
|
||
**Result:**
|
||
|
||
| bytes | value | packets |
|
||
|---|---|---|
|
||
| 0-3 | `04 04 00 01` | **2056 / 2056** |
|
||
| 4-7 | zero | **2056 / 2056** |
|
||
| 8 | zero | **2056 / 2056** |
|
||
| 11 | `0x05` | **2056 / 2056** |
|
||
|
||
Every packet's size is also a multiple of 16 (2056 / 2056). Zero exceptions on
|
||
any of the four.
|
||
|
||
**And the loader fills the rest in.** Read back out of the running game, the
|
||
same records carry byte 8 as a **count** (0x0b to 0x29 observed), byte 11 as
|
||
one of four tag values, and bytes 12-15 as an absolute EE RAM pointer where the
|
||
disc has a span-relative offset.
|
||
|
||
So **bytes 8-15 are rewritten wholesale on load.** What that means for reading
|
||
the disc: the header carries no count of its own, and the one the RAM copy has
|
||
is the `+0x40` table's `count` arriving by another route. The disc header is a
|
||
signature; the RAM header is populated state.
|
||
|
||
`ShrubPacketHeader::parse` and `Shrub::packet_headers()` expose this, and the
|
||
gated test asserts all four invariants plus the size alignment on every packet
|
||
of every bounded span.
|
||
|
||
## Sky: the sub-object table, and a count field that is not a count
|
||
|
||
**Range measured:** all 27 levels, each sky region from the core index's `sky`
|
||
offset to the next region.
|
||
|
||
**Method:** read the 32-bit offset table at `+0x20`, bounded by staying inside
|
||
the region, and parse each sub-object it names.
|
||
|
||
**Result: 149 sub-objects across 27 levels.** Each opens with a count, then a
|
||
bounding sphere of four floats at `+0x10`, then a texture offset at `+0x20` and
|
||
dimensions at `+0x24`/`+0x26`:
|
||
|
||
+000: 26 00 00 00 | 00 00 00 00 00 00 00 00 00 00 00 00
|
||
+010: e0 8d b2 c1 | 9f d6 f6 3d | ef fc fb be | 1c 61 4e 41
|
||
+020: 90 99 05 00 | 1d 00 | 1d 00 | 00 00 e8 00 ...
|
||
|
||
Count 1 to 38, width 3 to 81, radius 1.95 to 14.56, and the radius is positive
|
||
on **149 of 149** -- the same sphere shape and the same reasoning as the shrub
|
||
class header's.
|
||
|
||
**Correction to an earlier claim.** The word at `+0x0c` is **not** the
|
||
sub-object count. It reads 5 on level 0 where the table holds 6 entries, 3 on
|
||
level 1 against 4, 7 on level 3 against 8, and 196,617 on level 2 against 4.
|
||
Across the disc `offsets.len() == count + 1` holds on **10 of 27** levels. The
|
||
field is reported as `SkyHeader::field_0c` and nothing depends on it; the
|
||
offset table enumerates the sub-objects and is self-limiting.
|
||
|
||
That earlier claim came from level 0 alone, where 5 against 6 looked like a
|
||
consistent off-by-one. It is not an off-by-one -- it is a field that happens to
|
||
be small on most levels and is not a count at all.
|
||
|
||
`gc_formats::sky` parses the header and the sub-objects, and the gated test
|
||
asserts the sphere is finite, the radius positive where it is positive at all,
|
||
the dimensions non-zero, and the sub-object list non-empty on every level.
|
||
|
||
## The collision header is two fields, plus a third on five levels
|
||
|
||
**Range measured:** all 27 levels that carry collision, reading the 0x40-byte
|
||
header each.
|
||
|
||
**Method:** read `+0x00`, `+0x04` and `+0x08`, then check `+0x08..+0x40` for
|
||
zero.
|
||
|
||
**Result:**
|
||
|
||
- `u32@+0x00` is `0x40` on **27 of 27** -- the header is self-sizing.
|
||
- `u32@+0x04` is the payload size, 20 distinct values across the disc.
|
||
- **Bytes `+0x08..+0x40` are zero on 22 of 27**, but five levels carry a large
|
||
non-zero word at `+0x08`:
|
||
|
||
| level | `+0x08` |
|
||
|---|---|
|
||
| 7 | 1,248,048 |
|
||
| 9 | 1,766,832 |
|
||
| 12 | 2,541,616 |
|
||
| 13 | 3,376,592 |
|
||
| 14 | 4,060,640 |
|
||
|
||
So the header is not simply "two fields and 56 bytes of padding" -- there is a
|
||
third field that most levels leave at zero and five use. Whether it is a size,
|
||
an offset or a second block's length is not established; nothing measured here
|
||
separates those readings. It is exposed as `CollisionHeader::field_08`.
|
||
|
||
**Correction to the framing description.** The collision header was previously
|
||
described as self-describing across its full 0x40, which is true for 22 levels
|
||
and overstates the case for the other five.
|
||
|
||
**Still not decoded:** the payload. It contains ascending `u32` runs on levels
|
||
0 and 2 -- level 0 at `+280` with deltas 12, 12, 12 then a jump, level 2 at
|
||
`+13868` with deltas 204, 204, 200, 196 -- but the irregular deltas fit neither
|
||
a flat table nor a simple stride, and level 1's apparent run is a byte-shift
|
||
false positive. Nothing about the payload is asserted.
|
||
|
||
## data.bin is the level's file descriptor
|
||
|
||
`data.bin` is the largest per-level file -- 13.3 MB on level 0, 4.2 to 21.9 MB
|
||
across the disc -- and had been characterised only as "4-byte structured".
|
||
|
||
**Range measured:** the first 64 `u32` of `data.bin` on levels 0, 1, 2 and 5,
|
||
compared against the actual byte lengths of the other extracted level files.
|
||
|
||
**Method:** read the first 64 `u32`, and look for any that equals the size of
|
||
`core_index.bin`, `gs_ram.bin`, `overlay.bin`, `gameplay.bin`, `sound_bank.bin`
|
||
or `core_data.bin`.
|
||
|
||
**Result: three exact matches, at the same indices on every level.**
|
||
|
||
| u32 index | field | level 0 | level 1 | level 2 | level 5 |
|
||
|---|---|---|---|---|---|
|
||
| 1 | `overlay.bin` size | 2,751,268 | 2,763,668 | 2,928,424 | 2,863,676 |
|
||
| 3 | `core_index.bin` size | 31,600 | 34,016 | 33,808 | 23,536 |
|
||
| 5 | `gs_ram.bin` size | 518,656 | 569,344 | 513,024 | 434,176 |
|
||
|
||
Each is the *exact* byte length of the named file, and each lands on the same
|
||
index on all four levels. Three coincidences at fixed positions is not a
|
||
coincidence.
|
||
|
||
**So `data.bin` is a descriptor for the level's other files**, not geometry in
|
||
its own right -- which is what the roadmap's "the large one: 4.2 to 21.9 MB of
|
||
geometry per level" entry assumed. The intervening indices (0, 2, 4) hold
|
||
values that are not file sizes, and their meaning is not established; they are
|
||
plausibly offsets, but no cumulative-sum or offset+size relation has been
|
||
checked yet.
|
||
|
||
**This retires a wrong roadmap entry and opens a question.** `data.bin` should
|
||
not be listed as unparsed geometry. What it contains *after* the descriptor is
|
||
still unknown, and the file is large enough that the descriptor is unlikely to
|
||
be all of it.
|
||
|
||
## Tie spans: a constant first word, and three quantized floats
|
||
|
||
**Range measured:** all **1,864 tie spans across 27 levels**, read at each
|
||
class entry's geometry offset.
|
||
|
||
**Method:** read `+0x00`, `+0x0c` and the three `f32` at `+0x10`, `+0x14`,
|
||
`+0x18`, and count distinct values per position.
|
||
|
||
**Result:**
|
||
|
||
- `u32@+0x00` takes **exactly one value across all 1,864 spans** -- `0x00000100`,
|
||
256. A constant first word across the whole disc.
|
||
- `+0x0c` takes **83 distinct values**, so it is not constant.
|
||
- The three floats take **103 distinct triples** built from **42 distinct
|
||
values**, and those values are round: 4, 8, 9, 10, 12, 15, 16, 18, 20, 24,
|
||
25, 30 ... Only 10 of the 42 are exact powers of two.
|
||
|
||
Two spans on level 0 open:
|
||
|
||
+000 00000100 00005070 000065c0 07050612
|
||
+010 41400000 41c80000 42200000 00009460
|
||
|
||
and the same second, different first-two:
|
||
|
||
+000 00000100 00007ca0 0000a040 0605091e
|
||
+010 41400000 41c80000 42200000 0000ddf0
|
||
|
||
**What this rules out.** The three floats are not a general bounding box: a box
|
||
would take continuous values, and these come from a 42-value set of round
|
||
numbers. They are authored or quantised quantities -- a grid, a size class, or a
|
||
set of LOD thresholds -- and nothing measured here separates those.
|
||
|
||
**Correction to an earlier reading.** These floats were previously read as a
|
||
bounding-box minimum, from level 0's first two spans happening to share
|
||
(12.0, 25.0, 40.0). That triple holds on 266 of 1,864 spans, not all of them,
|
||
and the value set is quantised. The bounding-box reading does not survive.
|
||
|
||
**Not decoded:** what the three floats are, what `0x100` at `+0x00` counts,
|
||
and what the 83 values at `+0x0c` are.
|
||
|
||
## Tie geometry: no GS packet framing, and a control that nearly fooled me
|
||
|
||
**Range measured:** level 0, the 73 tie span bodies (bytes `+0x40` onward),
|
||
**2,585,856 bytes** pooled. Per-position distinct counts, an autocorrelation
|
||
scan against a shuffled control, and a GS GIF tag test.
|
||
|
||
**Result 1 -- the bodies are dense.** Every one of the 16 slot positions takes
|
||
245 to 256 distinct values, with byte 15 zero on 81.3% and byte 7 zero on 78.8%.
|
||
The best stride peak is 48 at +0.097, barely above 16 at +0.095 -- no period.
|
||
|
||
**Result 2 -- no GS GIF framing at 16-byte alignment.** Testing whether byte 15's
|
||
low three bits fall in 0..4, the GS tag range, gives **97.0%**.
|
||
|
||
That looked like a strong hit until the control was done properly. A uniform
|
||
shuffle gives **62.6%** -- which is exactly 5/8, what a random byte predicts --
|
||
so the naive comparison reads as a 34-point excess.
|
||
|
||
**The distribution-matched null is 97.0%, identical to the observation.** The
|
||
whole "excess" was that byte 15 is 81% zero, and `(0 & 7) <= 4` is always true.
|
||
Recomputed per alignment, the null tracks the observation everywhere:
|
||
|
||
| align | matched null |
|
||
|---|---|
|
||
| 0 | 97.0% |
|
||
| 2 | 75.5% |
|
||
| 4 | 75.7% |
|
||
| 8 | 96.7% |
|
||
| 12 | 89.3% |
|
||
|
||
**This is the control rule earning its place a second time.** A uniform shuffle
|
||
is not a matched control when the data is skewed, and `AGENTS.md` already says
|
||
so; the near-miss here was reading the 62.6% as a baseline without checking
|
||
what the data's own distribution predicts. The tie bytes are concentrated on
|
||
0, 1, 2, 3 and 63 -- suggestive in isolation -- and still carry no framing.
|
||
|
||
**What this rules out.** The tie bodies are not GS GIF packet streams at 16-byte
|
||
alignment, and they carry no measurable periodicity. With every slot position
|
||
taking ~256 distinct values they are dense data -- fat vertices, as the roadmap
|
||
expects -- and locating the vertex layout inside them needs a reference for this
|
||
game's instanced-geometry format, not another stride scan.
|
||
|
||
## The collision payload opens with two counts
|
||
|
||
**Range measured:** the first four bytes of the payload on all **19** levels
|
||
that carry a non-empty one.
|
||
|
||
**Result:** two `u16` at payload `+0` and `+2`, both non-zero on all 19, ranging
|
||
1..74 and 13..112, products 53 to 3,256.
|
||
|
||
| level | first | second | product | payload bytes |
|
||
|---|---|---|---|---|
|
||
| 0 | 11 | 33 | 363 | 903,248 |
|
||
| 1 | 24 | 16 | 384 | 2,800,496 |
|
||
| 2 | 1 | 53 | 53 | 2,315,296 |
|
||
| 7 | 40 | 35 | 1,400 | 4,049,232 |
|
||
| 9 | 74 | 44 | 3,256 | 2,518,352 |
|
||
| 13 | 2 | 64 | 128 | 3,333,520 |
|
||
| 26 | 27 | 112 | 3,024 | 652,176 |
|
||
|
||
**Reported as two counts, not as a grid.** They have the shape of collision
|
||
grid dimensions -- bounded, both non-zero, products in the hundreds to
|
||
thousands -- but nothing measured ties the product to the payload size or to any
|
||
per-cell stride. Level 2's `1 x 53` giving a 2.3 MB payload and level 26's
|
||
`27 x 112` giving 652 KB shows the product does not size the payload, which
|
||
means either cells vary wildly in size or these are not grid dimensions. Naming
|
||
them width and depth would be a reading, not a measurement.
|
||
|
||
**After the two counts** comes a run of `u16` that stays below 8,192 for 30 to
|
||
58 entries depending on level, always an even count, and is not ascending. That
|
||
is where the per-cell table is likely to start, and it is the next thing to
|
||
resolve.
|
||
|
||
Exposed as `CollisionHeader::first` and `::second`.
|
||
|
||
## Collision: a per-cell offset table, located on level 4
|
||
|
||
**Range measured:** longest strictly-ascending `u32` run (delta under 65,536) in
|
||
each level's collision payload, levels 0 to 5.
|
||
|
||
**Level 4 is the clean case.** Grid `3 x 25`, payload 2,660,608 bytes, and at
|
||
payload `+5932` a run of 19 entries whose values are all inside the payload,
|
||
starting at 146,012:
|
||
|
||
deltas: 204, 212, 216, 224, 224, 228, 228
|
||
|
||
**The deltas grow monotonically.** That is the signature of a prefix sum over
|
||
records of increasing size, which is what a per-cell offset table looks like when
|
||
cell contents vary.
|
||
|
||
**The others are not.**
|
||
|
||
| level | grid | run at | entries | first deltas | verdict |
|
||
|---|---|---|---|---|---|
|
||
| 0 | 11x33 | +21232 | 17 | 8193, 24575, 8192, 8192 | multiples of 8,192 — coincidental on dense data |
|
||
| 1 | 24x16 | +41452 | 29 | 32771, 57339, 24575 | wild, no trend |
|
||
| 2 | 1x53 | +13752 | 49 | 24, 64, 396, 104 | in-region but irregular |
|
||
| 3 | 19x31 | +667152 | 18 | 769, 257, 257, 257 | values ~5.2e8, far outside the payload |
|
||
| 4 | **3x25** | **+5932** | **19** | **204, 212, 216, 224** | **growing, in-region** |
|
||
|
||
Only level 4's run has both properties a prefix sum needs: values inside the
|
||
payload, and deltas that grow.
|
||
|
||
**What this does and does not give.** Level 4's run is 19 entries against 75
|
||
grid cells, so it is not one offset per cell -- either most cells share an
|
||
offset, are empty, or the table covers something other than the grid. Nothing
|
||
measured here picks between those.
|
||
|
||
**Followed up.** `+5932` is the table's *start*, not a point inside it: walking
|
||
backwards, no predecessor satisfies "ascending with a delta under 4,096". The
|
||
run is **17 entries**, not 19 -- the earlier 19 counted two entries past the
|
||
jump.
|
||
|
||
The deltas are
|
||
|
||
204 212 216 224 224 228 228 228 224 220 216 200 96 96 100 96
|
||
|
||
then a jump of 41,484 to 190,508. So the deltas **rise and then fall** -- a
|
||
symmetric profile, not a monotone one. That weakens the prefix-sum reading
|
||
somewhat: a cumulative offset table has no reason to come back down, whereas
|
||
cell contents that grow and shrink as the table walks across a level would.
|
||
|
||
**17 against a 75-cell grid, and 75 - 17 = 58.** No arithmetic relating the two
|
||
has been found. The earlier guess that 19 related to `3 x 25` did not survive
|
||
re-measurement and should not be carried forward.
|
||
|
||
**Still open.** 17 entries may not correspond to the grid at all -- the table may
|
||
cover something else, or cover a subset of cells. Resolving it needs either the
|
||
same offset found in other levels at a position the leading counts predict, or a
|
||
reason for 17. Neither exists yet, and this is the point at which the
|
||
measurement trail goes cold on collision.
|
||
|
||
## overlay.bin: named sections and a 16-aligned debug string table
|
||
|
||
`level_overlay` was on the open list with no parser and no located structure. It
|
||
now has one: a header naming two sections, and a string table.
|
||
|
||
**Range measured:** `overlay.bin` on levels 0, 1 and 2 -- 2,751,268,
|
||
2,763,668 and 2,928,424 bytes.
|
||
|
||
**Result 1 -- the file opens with section names.**
|
||
|
||
+000: 80 7c 1a 00 | dc 99 00 00 | 01 00 00 00 | 88 3b 2a 00
|
||
+010: 00 00 00 00 | 00 00 00 00 | "crap\n" | 00 00 00 00 00
|
||
+020: "system misc" 00 00 00 00 00 00 00 00 00
|
||
+030: 20 80 ff 40 | 00 60 cf 00 | 64 00 | 46 00 | 19 00 00 00
|
||
|
||
`u32@+0x00` is **1,735,808 on all three levels**, `+0x08` is `1`, and the bytes
|
||
at `+0x18` and `+0x20` spell **"crap"** and **"system misc"**.
|
||
|
||
**Result 2 -- a debug string pool, 16-byte aligned.** 332 printable runs on
|
||
level 0, 362 on level 1, 351 on level 2, the first at `+0x20` and the last
|
||
between 1.3 MB and 2.5 MB. They are the game's own debug text:
|
||
|
||
host0:r:/rc2/shock/wavetable.bin
|
||
host0:z:/i5/shock/wavetable1.bin
|
||
unloaded sound bank %#010x
|
||
Camera_CollPrimTest WARNING! - grid out of bounds! (RAR)
|
||
render setup
|
||
sky draw
|
||
pre effects
|
||
pre 2 effects
|
||
|
||
**`Camera_CollPrimTest WARNING! - grid out of bounds! (RAR)` is direct evidence
|
||
for the collision reading.** It is the game's own string, naming a grid and a
|
||
collision test, and it is why the two leading counts in the collision payload
|
||
are worth continuing to treat as grid dimensions despite not sizing the payload.
|
||
|
||
The `wavetable.bin` paths also confirm the sound bank is loaded from named
|
||
files, which bears on the `SBlk`/`DAW` framing.
|
||
|
||
**Correction to the framing above.** This was first written up as a *string
|
||
table* of records. It is not. Dumping the 32 bytes before each run shows most of
|
||
them are the tail of the previous string:
|
||
|
||
+3104 "host0:r:/rc2/shock/wavetable.bin" (32 chars)
|
||
+3120 "ck/wavetable.bin" -- its own tail
|
||
+3152 "/i5/shock/wavetable1.bin"
|
||
+3168 "ble1.bin" -- its own tail
|
||
|
||
and the 32-byte prefixes are **327 distinct out of 332**, i.e. essentially all
|
||
different, because they are just the preceding string's bytes. There is no
|
||
per-record header here. It is one contiguous pool of debug text, and the
|
||
16-byte alignment is a property of how the pool was laid out, which makes
|
||
continuations land on boundaries and read as separate entries.
|
||
|
||
**Not done.** The pool is located and counted but not parsed, and there is no
|
||
`gc_formats` module for it. The `+0x18`/`+0x20` names are section names but
|
||
their structure is not established, and it is not yet shown that they index
|
||
anything.
|
||
|
||
## gameplay.bin is a flat pointer table
|
||
|
||
**Range measured:** the first 40 `u32` and the whole file, on levels 0 and 1 --
|
||
1,872,256 and 2,006,400 bytes.
|
||
|
||
**Result:** the file is a flat array of `u32`, opening with a small count and
|
||
then a run of ascending offsets, and **terminating on a zero**:
|
||
|
||
level 0: 336 797472 798832 799136 464 150672 150688 314800 479920 ...
|
||
... 1847600 1845456 797808 1829056 160 1849088 1249808 1847616 0
|
||
|
||
level 1: 496 799904 800128 800432 624 151152 151168 315680 481456 ...
|
||
... 1984496 1982400 800112 1966000 160 1988160 1303456 1984512 0
|
||
|
||
**Both levels carry the same shape.** The same sentinel value, `160`, appears at
|
||
`u32` index 34 on both, and the zero terminator at index 38 on both, with counts
|
||
336 against 496 and 464 against 624 at the head. So the leading ~38 words are a
|
||
structured header whose layout is level-independent apart from its sizes.
|
||
|
||
The offsets cluster into groups -- near 800,000, near 150,000, near 1,050,000,
|
||
near 1,800,000 -- which is what a table of pointers into several buffers would
|
||
look like.
|
||
|
||
**Not decoded.** What the pointers point into, what `160` is, and why the zero
|
||
terminator sits at the same index on both levels. No `gc_formats` module exists
|
||
for this yet; it is characterised, not parsed.
|
||
|
||
**Where the pointers go.** Following them shows what `gameplay.bin` indexes. The
|
||
leading table is 39 words, all of which land inside the file, and three clusters
|
||
dominate: near 150,000, near 800,000 and near 1,050,000.
|
||
|
||
The region near 797,808 is **position data** -- 31 consecutive plausible floats
|
||
that group cleanly into xyz triples:
|
||
|
||
246.948 201.015 73.867 | 88.240 186.000 302.598 | 107.712 12.723 183.125
|
||
353.591 126.365 19.962 | 182.068 374.001 122.567 | 116.961 265.001 437.866
|
||
|
||
and the region near 797,456 is **direction data** -- unit-length vectors:
|
||
|
||
0.969 0.945 0.733 | -0.120 0.433 0.250 | -0.866 0.000 0.267
|
||
|
||
So the file's leading words are a table of pointers into float arrays, not a
|
||
flat list of small integers. The earlier reading of it as "u32 pointers holding
|
||
nothing" was too quick: it is a pointer table *into data that is present*.
|
||
|
||
**On volumes.** Nothing resembling volumes was found here. `gameplay.bin` is
|
||
u32 pointers throughout and holds no float data in its first 64 words read as
|
||
`f32` (59 of 64 are denormals, i.e. small integers reinterpreted). Volumes remain
|
||
unlocated on the disc -- nothing in the regions examined so far presents itself
|
||
as one.
|
||
|
||
### level_overlay: parsed
|
||
|
||
`level_overlay` now has a module. `gc_formats::overlay` parses a 16-byte header,
|
||
splits out the constant block and the level-specific tail, and a real-disc test
|
||
covers all 27 levels:
|
||
|
||
overlay: 27 levels; field_00 1735808 = 0x1A7C80, field_08 1;
|
||
field_04 37540..=45308, field_0c 2763912..=3027336
|
||
(17 below the file length, 10 above);
|
||
+0x10..0x1220 identical on all
|
||
|
||
Later, from OpenRAC/rac2-gc-decomp (same disc, SHA-256 verified identical):
|
||
|
||
field_00 IS the .lit load base -- 0x1A7C80, constant on all 27 levels
|
||
because every level's literal section loads at the same address.
|
||
overlay.bin IS that section: it holds code and the string pool.
|
||
$gp 0x1A7C80 + section offset is computed against $gp = 0x1AEFF0.
|
||
program seven sections: .lit .bss .data lvl.vtbl lvl.camvtbl
|
||
lvl.sndvtbl .text
|
||
|
||
field_0c tested as the .lit size and FAILED: it is smaller than the file on 16 of
|
||
27 levels, by up to 8.9% (level11 file 3,151,128 vs field_0c 2,870,792). Excluded
|
||
by sign, not by a near miss. Still unnamed.
|
||
|
||
The collision assert string was then used as an anchor and the first attempt was
|
||
WRONG -- a byte-order bug, recorded in core-data-regions.md. The PS2's MIPS is
|
||
little-endian, so an I-type immediate is word bytes 0 and 1; searching bytes 1..3
|
||
matches opcode/$rs instead and manufactured a single convincing "hit". Corrected,
|
||
the count is zero. The anchor is NOT established.
|
||
|
||
**`+0x0c` is not the file length.** It was recorded as one on the strength of the
|
||
first levels measured, where it sat close to the file end. Across all 27 it falls
|
||
on **both sides** -- below on 17, above on 10 -- so it is an ordinary
|
||
level-specific word. The test asserts exactly this, since it is the claim most
|
||
worth pinning.
|
||
|
||
**The common prefix is 4 bytes, not the whole header.** `+0x00` is constant at
|
||
1,735,808, but `+0x04` and `+0x0c` vary, so the prefix stops at `+0x04`.
|
||
Everything from `+0x10` to `+0x1220` -- **4,624 bytes** -- is byte-identical on
|
||
all 27 levels.
|
||
|
||
What is still not named: what any of the four header words *means*, and how the
|
||
level-specific tail is structured. The module reports them as fields.
|
||
|
||
### moby_anim: the instance-table hypothesis is out
|
||
|
||
`moby_anim` was the one post-shrub chunk type with a concrete reason to exist --
|
||
an instance table would have to scale with the level's moby class count, so
|
||
comparing per-type record counts against that count was the obvious test.
|
||
|
||
**It fails, and the way it fails is itself the result: per-type record counts are
|
||
level-independent.** The same types report the same counts on every level tried:
|
||
|
||
type_id level0 level1 level2 level3
|
||
0x007d7ec438 90 90 90 90
|
||
0x00d3d8c4a9 90 90 90 90
|
||
0x00429ec43d 77 77 77 77
|
||
0x0053c8c4aa 77 77 77 77
|
||
0x007626c4a4 50 50 50 50
|
||
0x00fb88c462 47 47 47 47
|
||
|
||
Across all 27 levels there are 77 distinct chunk types, the largest at 2,430
|
||
records total -- which is exactly 90 x 27.
|
||
|
||
**What that rules out.** Nothing in this chunk set is a per-moby-class instance
|
||
table, because those counts would vary by level the way the moby class counts do.
|
||
So whatever holds moby animation data is either keyed by something other than the
|
||
class count, or it is not among these chunk types.
|
||
|
||
**Also visible: the types pair up.** 0x007d7ec438 and 0x00d3d8c4a9 both report 90
|
||
on every level; 0x00429ec43d and 0x0053c8c4aa both report 77. Two chunk types
|
||
carrying identical, constant record counts on every level is a signature of two
|
||
views of the same fixed table -- one indexed one way and one another -- rather
|
||
than two independent tables that happen to agree.
|
||
|
||
**Still open.** The pairings are described but not explained, and no type has been
|
||
named. The Moby class-ID enrichment from earlier stands separately: 9,493 hits
|
||
against 246.1 expected is real, and the class arrays are contiguous per type, but
|
||
that identifies *where classes are referenced*, not which chunk is `moby_anim`.
|
||
|
||
|
||
## The tie control region is not a tag stream (negative result)
|
||
|
||
Went looking for the GS's GIF packets, which the tie packet header calls AD-GIF
|
||
(`shader_count` at `+0x04`, and `bfc_distance == 5 * shader_count` at `+0x05`).
|
||
Tried the first `5 * shader_count` quadwords of each packet as VIF tags (command
|
||
in bits 0-6, size in 7-14, mark in 15) and as GIF tags (type in the top byte of
|
||
word 1, length in bits 0-2 of word 0).
|
||
|
||
**Both came out uniform.** Over 13,042 records, 10,443 distinct values of word 0
|
||
and no GIF tag type above a count of 2. By rule 4 that is the control result:
|
||
the region is not a tag stream, and no further effort should go into decoding it
|
||
as one.
|
||
|
||
The first version of that probe was wrong and worth recording, because it reached
|
||
the *opposite* conclusion twice. It scanned `bfc_distance` quadwords, which for
|
||
most records overruns `vert_ofs` and runs into the vertex data. Pooling a
|
||
control region with vertex positions -- two populations with different
|
||
structure -- produced the uniform answer, and "not a tag stream" was the right
|
||
conclusion for the wrong reason. Re-measuring quadword 0 alone on its own gives
|
||
the same uniformity, so the conclusion survives; the reasoning did not.
|
||
|
||
What the probe did establish is the **packet partition**:
|
||
|
||
q[0 .. vert_ofs) control 3..8 quadwords (peak 5)
|
||
q[vert_ofs .. +vert_size) vertices
|
||
q[vert_ofs+vert_size ..] scissor
|
||
|
||
and a new **universal invariant**: `scissor_ofs == vert_ofs + vert_size` on
|
||
**15,219 of 15,219** records across all 27 levels, including every record whose
|
||
`strip_count` is non-zero. That is a derived quantity stored explicitly, and it
|
||
holds everywhere -- which is the check that confirms the three regions really are
|
||
laid out as the packet-length formula assumed.
|
||
|
||
It also found `vert_ofs` mis-named. It is not a vertex offset; it is the *length*
|
||
of the leading control region. The vertex data begins at that many quadwords into
|
||
the packet. The field name is left alone because the on-disc order is fixed and
|
||
everything already written about this record says `vert_ofs`, but the doc now
|
||
says what the value is.
|
||
|
||
**What the control region is:** per-record, not shared. 12,294 distinct blobs
|
||
across 13,042 records, mixed within 2,410 of 3,549 classes, and different between
|
||
LOD 0 and LOD 1 on all 500 classes that have both. It varies per primitive.
|
||
|
||
Partitioned by its own length it is *not* uniform in character -- the pooled
|
||
count hid this:
|
||
|
||
vert_ofs 48 bytes (3 qw) 5 distinct blobs
|
||
vert_ofs 64 bytes (4 qw) 3,175
|
||
vert_ofs 80 bytes (5 qw) 4,309
|
||
vert_ofs 96 bytes (6 qw) 3,281
|
||
vert_ofs 112 bytes (7 qw) 1,523
|
||
vert_ofs 128 bytes (8 qw) 1
|
||
|
||
The shortest and longest are effectively constant -- five blobs and one -- and
|
||
everything in between is per-record unique. That is a shape, not a noise floor.
|
||
|
||
## bfc_distance is not the control region's length
|
||
|
||
The other half of the answer. `bfc_distance == 5 * shader_count` holds on every
|
||
record and has always been documented as "quadwords of ad-gif data" -- which, read
|
||
as a length within the packet, would make it the control region's size.
|
||
|
||
It cannot be. `bfc_distance` is 5..20 quadwords; `vert_ofs`, the control region,
|
||
is 3..8.
|
||
|
||
Pooled, the two overlap on 9,315 of 15,219 records (61%) -- a number describing
|
||
nothing. Partitioned by `shader_count`, as rule 3a requires, it is exact:
|
||
|
||
shader_count 1: 2,772 of 8,676 exceed (32%)
|
||
shader_count 2: 4,649 of 4,649 exceed (100%)
|
||
shader_count 3: 1,729 of 1,729 exceed (100%)
|
||
shader_count 4: 165 of 165 exceed (100%)
|
||
|
||
For two or more shaders it exceeds on every record, without exception. For one
|
||
shader it exceeds exactly when `vert_ofs < 5`, which is the 32%.
|
||
|
||
**So the AD-GIF is not stored in the tie packet.** The packet is control,
|
||
vertices, scissor. `bfc_distance` is a per-record quantity relating to the shader
|
||
count whose meaning is not established, and the "quadwords of ad-gif data" reading
|
||
of it is wrong. The field doc said so in terms before this measurement; it does
|
||
not now.
|
||
|
||
|
||
## Where the AD-GIF is not: class-header offsets 0x1c and 0x34
|
||
|
||
Two words in the class header's undecoded tail are offsets, and they are the
|
||
best remaining lead on the AD-GIF.
|
||
|
||
`base + 0x1c` and `base + 0x34` land inside `core_data` on **1,864 of 1,864
|
||
classes**, so both are relative to the class base rather than absolute, and both
|
||
are non-zero on every class. And both land **past** that class's own tie packet
|
||
blocks on every class -- they address per-class data stored somewhere else in
|
||
core_data, not the tie geometry.
|
||
|
||
What is there is structured, not random. Class 0 at 0x1c:
|
||
|
||
54 5d 5f 0c 44 44 44 01 03 11 19 00 1c 00 60 0a 30 08 60 0a
|
||
ff 00 ff ff 20 0c 50 02 44 38 2b 50
|
||
|
||
and class 3 at the same field:
|
||
|
||
20 05 c5 00 00 80 01 00 26 05 cb 00 21 88 01 00 2c 05 c0 00
|
||
00 80 01 00 32 05 c0 00 00 80 01 00
|
||
|
||
Class 3 is a fixed-stride table with `80 01 00` recurring. Class 5's 0x34 field
|
||
has `38 38 38` and `ff 00 ff ff`. Class 2's 0x1c field is a monotone byte table:
|
||
`26 27 0e 0f 10 11 12 13 24 25 15 1e 17 1f 14 15 16 17 20 21 22 23`.
|
||
|
||
These read like compiled shader programs with small-integer parameters, or as
|
||
lookup tables. **They are not identified**, and the resemblance to anything
|
||
specific is exactly the kind of finding rule 4 exists to refuse -- none of these
|
||
three fragments has been shown to beat a control, and they were picked because
|
||
they looked like something.
|
||
|
||
## The control, and a retraction
|
||
|
||
The last paragraph of the previous section implied the two offsets select
|
||
*distinctive* content. They do not.
|
||
|
||
Measured adjacent-equal-byte rate over 64 bytes, 1,864 classes:
|
||
|
||
0x1c 0.12688
|
||
0x34 0.12106
|
||
control 0.14109 (the same offset + 0x2000)
|
||
random 0.00391
|
||
|
||
All three are ~30x above the random baseline, so the region really is structured
|
||
data and not noise -- that part stands. But **0x1c and 0x34 are no more
|
||
structured than an arbitrary offset in the same blob**, and 0x34 is slightly
|
||
*less*. So the hex fragments in the previous section were three hand-picked
|
||
examples from a structured region, and "they look like shader programs with
|
||
small-integer parameters" said something about this part of core_data, not about
|
||
what these fields address.
|
||
|
||
**Retracted:** the suggestion that the offset targets are identifiable content.
|
||
Still established, and independent of all of the above: the offsets are
|
||
base-relative, non-zero, in range on 1,864 of 1,864 classes, and always land past
|
||
the class's own tie packet blocks. What they address remains unknown, and the
|
||
earlier finding that the packet's control region is not a tag stream still stands
|
||
-- it was measured by uniformity within the packet, not by comparison to
|
||
somewhere else in core_data.
|
||
|
||
|
||
## moby_anim: the class table is not the loader (negative result)
|
||
|
||
`moby_anim` needs the moby class loader. The obvious place to look for a loader
|
||
is the moby class table, since a loader is usually dispatched from one. Checked
|
||
that first and it is ruled out.
|
||
|
||
`moby_classes` is an `ArrayRange`: offset `0xc0`, count 238, in level 0. The
|
||
words there are not pointers. Read as floats they are
|
||
|
||
0x4846a9e3 0x47f40000 0x46d0b0de -> ~50,986 61,000 26,671
|
||
|
||
which is position/scale-shaped data. And scanning every u32 in the first
|
||
`0x4000` bytes for values landing inside the overlay's `.text`
|
||
(`0x2a3b80..0x447900`) gives **120 hits out of 4,096**, against roughly **259**
|
||
expected if the values were unrelated to the region.
|
||
|
||
So the table holds *fewer* .text-range values than chance, not more. Whatever the
|
||
moby class loader is, it is not dispatched from a pointer in this table.
|
||
|
||
That rules out the cheap approach. Finding the loader means cross-referencing
|
||
from somewhere else -- most likely a constructor that reads the class table and
|
||
returns runtime state, which is in `.text` and has to be found by scanning for
|
||
code that touches both `moby_classes.offset` and the class spans. That is a
|
||
larger search than a bounded probe, and it is the unblock condition already
|
||
recorded for `moby_anim`.
|
||
|
||
Not concluded: what the 238 entries *are*. The float reading is suggestive, not
|
||
established -- it was read at one stride, on one level, and rule 3a applies.
|
||
|
||
### Cross-reference scan by raw immediate: does not discriminate
|
||
|
||
The natural next probe is to search `.text` for immediates matching each chunk's
|
||
offset. Run over all 27 levels (0x2d70260 bytes, 23.8M u16 values):
|
||
|
||
moby_classes 0x00c0 49,230 occurrences
|
||
tie_classes 0x1e80 111 occurrences
|
||
|
||
**This comparison is meaningless and must not be quoted as evidence.** The two
|
||
offsets differ in magnitude, and small values are simply common: the single most
|
||
frequent u16 in `.text` occurs 1,834,535 times, so a two-byte value that is
|
||
common by construction will beat a rarer one no matter what it references. The
|
||
whole scan is confounded by value frequency, which is rule 3a again, one level
|
||
below where it has bitten before -- not the measurement being wrong but the
|
||
*comparison* having no shared denominator.
|
||
|
||
Doing it properly means decoding instructions and counting immediates only where
|
||
they are immediates, which is what `tools/overlay-anchors` is for. Until that
|
||
exists the answer is not "moby_classes is referenced more" or "less", it is
|
||
"this method cannot tell".
|