Files
Going-Commando/docs/formats/core-data-regions.md
T
ReaperOfVeriod 6e5486de88 Docs audit: nine stale class-ID counts left behind by the terminator fix
Checked the docs for claims this session retracted. Found a real one: the vtbl
terminator fix (cc52060) corrected the distinct class-id count from 1,163 to 1,162,
and nine places in the format spec and two in the roadmap still said 1,163.

    docs/formats/core-data-regions.md   9 occurrences
    docs/roadmap.md                     2 occurrences

The reason they survived is the same shape as the vtbl bug itself. The test was
fixed and the new number asserted, but the prose kept the old one, and the prose is
what a reader takes away. A commit that corrects a measurement and does not correct
the document it was measured in has only half-landed.

Checked the numerators before touching them rather than blanket-replacing: 1,135 of
the ids are at or above level 0's moby_classes.count, and 1,133 are at or above the
largest per-level table. Both were already correct -- only the denominators were
wrong -- so a find-and-replace across the number would have corrupted two correct
figures. The tests confirm:

    lvl.vtbl: 1162 distinct class ids from 5822 records; max 0x13ba (5050);
    moby_classes.count is 238, and 1135 of 1162 ids (97.7%) are at or above it

Left one deliberate occurrence of "1,163" in the spec, in a note explaining that the
figure was 1,163 until the terminator fix and why, so the correction is traceable
rather than looking like a typo.

293 passing with the disc and savestate present, clippy clean. Documentation only.
2026-10-15 03:15:02 +02:00

3773 lines
179 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# core_data region bounds
How a level's `core_data` lump divides up, and how each region is bounded.
## The rule
A region's end is the **next region's start**, never `core_data.len()`. The
last region in a layout is the exception, and when its successor is unknown the
end is `None` rather than a guess.
This exists because region bounds were being re-derived per format during
probing, usually by running from one known offset to the next with the last
span running to the end of the lump. On level 0 that made the final shrub class
a 925,008-byte span absorbing most of the lump's tail. Every aggregate computed
over it -- record counts, value distributions, "this data looks random" -- was
silently wrong, and nothing errored.
`gc_formats::regions::core_layout` computes these once. Callers use it rather
than inventing their own.
## Layout
| region | start | end |
|------------|----------------|----------------------|
| `tfrags` | 0 | `occlusion` |
| `occlusion`| `occlusion` | `sky` |
| `sky` | `sky` | `collision` |
| `collision`| `collision` | first moby class |
| `moby` | first moby | first tie class |
| `tie` | first tie | first shrub class, |
| | | else chunk container |
| `shrub` | first shrub | chunk container |
| `chunks` | chunk container| lump length − 0x38A10|
Class regions start at the lowest non-zero geometry offset in their class
table. A zero entry means *absent*, not position zero -- the same trap the
occlusion and collision headers each fell into separately.
Not every level has every region: level 10 has no occlusion and no shrub.
## The constant trailer
Comparing all 27 levels byte-for-byte from the end: the last **231,952 bytes
(0x38A10)** of every level's `core_data` are identical. Level-specific content
therefore ends at `len - 0x38A10`, and the final region is bounded there rather
than at `core_data.len()`.
This is what closes the chunk container. The regions it separates from level
content are measured on the disc: **2,033,776 bytes of shrub region across the
24 levels that have one, and 25,775,850 bytes of chunk container** across all
27. The disc test asserts the bound on every level.
What the trailer *is* is not established. It may be fixed data, or padding
from a fixed-size decompression buffer. Either way it is constant, which is
all that is needed to bound the tail.
## Found: the chunk container closes the class region
The offset ending the shrub region is not in the core index, and the class
table that precedes the container carries no successor offset for it -- unlike
moby and tie, whose boundaries fall out of the tables. Texture texels do not
bound it either: they sit before moby, not after shrub. So the boundary is
located in `core_data` itself.
**Range measured:** for each level, from the last shrub class's geometry
offset to the constant trailer. Nothing outside that range is read.
**Method:** scan forward from the last shrub class for the pair (a) the
constant two bytes `f6 7c`, sitting immediately in front of (b) a chunk
signature -- a 16-byte header ending `0xff`, ten zero bytes, then four or more
`0xf0`-tagged offsets. `gc_formats::chunks::chunk_region_start`.
Each half of that signature is weak on its own, which is why the halves are
measured separately:
- the pair `f6 7c` occurs **25 to 103 times** in a 5 to 17 MB lump, so
matching it alone picks dozens of wrong offsets;
- the chunk shape matches random data often enough that `parse_chunk` demands
four tagged offsets before a signature is believed.
**Result: on all 27 levels the pair sits exactly two bytes before the first
chunk header.** The two are never found apart. This is the boundary, and
`core_layout` closes shrub with it -- `shrub`'s end is the container's start,
and the container's own end is the trailer, so every byte of level-specific
`core_data` now belongs to a named region.
Two controls, both asserted by the disc test `chunk_container_closes_the_last_class_region`:
- **No signature inside the body.** Scanning the last shrub body for a bare
chunk signature finds none, on any level -- so the boundary search was not
matching something it should have skipped, and the marker's job is
corroboration rather than load-bearing disambiguation.
- **The three levels without shrub** (10, 25, 26) carry the same marker two
bytes before the container. There the container closes *tie* instead, which
is what makes the locator a property of the region rather than a shrub
rule.
On level 0:
| what | range | bytes |
|---|---|---|
| last shrub class data | 16046800 .. 16047040 | 240 |
| marker `f6 7c` | 16047040 .. 16047042 | 2 |
| **chunk container** | 16047042 .. 16737344 | **690,302** |
| constant trailer | 16737344 .. 16969296 | 231,952 |
The marker's meaning is not established. It is constant across all 27 levels
and it appears at exactly this position, but whether it is a terminator, an end
count or a tag is unknown, so it is not named.
**Correction this produces.** The chunk-container measurements recorded below
were taken over "the 900,000 bytes before the constant trailer" -- a window
wide enough to contain the container on every level but not aligned to its
true extent. Chunks sitting at the window's edge were truncated (27 of 1,123),
and the region's size was never correctly stated. Both are now bounded
properly: on level 0 the container is 690,302 bytes, not ~900,000, and the
25,775,850 bytes of chunk data across 27 levels are the measured total rather
than a windowed sample. The `chunk_containers_resolve_on_every_level` test now
reads the region from `core_layout` instead of measuring backwards from the
end of the lump.
## Correction: it is mostly texture data, not an unnamed region
Adding the region table to `gc-extract probe` made the gap visible
immediately, which is the point of printing it.
On level 0:
```
collision 2615232 .. 8832192 6216960 bytes
```
but collision's own header declares a payload of 903,248 bytes. So the range
between the collision header and the first moby class is **6.2 MB**, and
almost none of it is collision.
The first conclusion drawn from this -- that ~6.2 MB of unidentified data sat
between collision and moby, possibly `moby_anim` -- **was wrong**, and it was
wrong because of the same failure that produced the other two bad numbers this
session: quoting a figure before bounding it.
Checking the bytes rather than just the sizes: the range is dominated by 8-bit
indexed texture data. Byte-value concentration (a texture block uses a small
palette; code and collision data does not) puts 18 of 21 sampled 4 KB blocks
under 200 distinct byte values, against 249 for tfrags and 209 for collision.
So the real layout is:
| region | start | end |
|------------|----------|------------------|
| collision | 2615232 | 3518544 |
| textures | 3518848 | ~8832192 |
| moby | 8832192 | first tie class |
The texture table's `data_offset` values only reach 5326208, which is why the
gap looked real: mip chains extend well past the last entry's offset. A region
sized from a table of first-level offsets understates the data it points at.
The "3 of 21 blocks not texture-like" figure in the previous revision was an
artifact of sampling every 256 KB. Re-scanned at 4 KB granularity the whole
range is texture data: ~100 blocks stand out, and looking at them they are
smooth gradient ramps -- sky and fog texture, exactly what gradient texels look
like. A noisy gradient has a high distinct byte count without being code.
So the range between collision and moby is **entirely texture data**, and there
is no separate `moby_anim` region there. Combined with the correction above,
two claims made about this range have now been wrong: the "6.2 MB of unknown
data" and the "3 unexplained blocks".
Where `moby_anim` actually lives is therefore still open, but it is not here.
The remaining candidate is the unresolved tail after the last shrub class,
which on level 0 is ~925 KB of real, non-zero data that nothing yet accounts
for.
## Where shrub data ends, and the region after it
**Range measured:** for each level, the last shrub class's `data_offset` field
(`class + 0x2c`) through to the start of the constant trailer. Nothing outside
that range is read.
**Method:** the last shrub class's body is an array of 8-byte records whose
final two bytes are `00 00`. Scanning forward in 8-byte steps for the first
record whose trailing pair differs locates the body's end. This is a search
for a specific stated pattern, not a statistic, so it either finds the
boundary or reports that it did not.
**Result:** on **24 of 24** levels the boundary lands exactly on an 8-byte
record boundary. Body lengths are 192, 208, 224 or 240 bytes -- the same sizes
seen when shrub classes were grouped and diffed earlier.
On level 0:
| what | range | bytes |
|---|---|---|
| last shrub class data | 16046800 .. 16047040 | 240 |
| **next region** | 16047040 .. 16737344 | **690,304** |
| constant trailer | 16737344 .. 16969296 | 231,952 |
So the 690,304 bytes between the last shrub class and the trailer are a
distinct region, not shrub data. What they contain is not established.
Two things are worth noting about how this was found, given the rule in
`AGENTS.md`: the earlier NaN-filler search found nothing at the start of this
range and was about to conclude the body had zero length, which would have been
wrong. The last shrub class is a real class with real data, and the NaN filler
only appears in degenerate ones. Checking the actual byte pattern rather than
concluding from a failed heuristic is what caught it.
### First structure in the region after shrub
**Range measured:** level 0, `16047040 .. 16737344` (690,304 bytes), starting
at the first byte after the last shrub class's 240-byte body. Nothing outside
that range is read.
**Method:** reading the bytes at `+0x20` as `u32` shows ten consecutive
entries whose low halves increase by a constant **0x94 (148)**, followed by a
zero. Read as u32 the high half is `0xf000` for the first nine and `0x0a40`
for the tenth, so the table is ten entries long and ends at `+0x48`
(absolute `16047112`).
**Result, stated plainly:** a ten-entry table on a 148-byte stride. What the
entries mean, what 148 refers to, and why there are ten, are all unestablished.
Ten does not match any count already known for this level -- 238 moby
classes, 74 tie, 7 shrub, 93 tfrag textures -- so this is not yet tied to any
structure elsewhere in the lump.
### Follow-up: the 148 stride is a coincidence
Treating the ten entries' low halves as offsets from the region start lands on
blocks exactly 148 bytes apart, the first nine of which begin `08 00` followed
by a 16-bit id stepping by 0x40. That is a convincing-looking structure and it
does not survive being extended: walking the same 148-byte stride across the
rest of the region, **only 14 of 4657 blocks** start with `08 00`. The first
nine agree and then the pattern breaks.
So the ten entries are not a pointer table. The constant stride held for
eight consecutive values, which is far more than chance should produce and far
less than a structure needs.
Recorded because the lead was committed as a lead and this is where it goes.
Nothing else about the region has been identified: it is real, non-zero data
of unknown type, bounded, and unopened beyond these ~150 bytes.
## The region after shrub has a repeating structure at stride 368
**Range measured:** level 0, `16047040 .. 16737344`, sampled as the first
65,536 bytes. Cross-checked on every level that has the region, each sampled
the same way from its own region start.
**Method:** autocorrelation. For each candidate stride, count how often a
4-byte sequence at offset *i* repeats at *i + stride*, as a fraction of
comparable positions. A random control of the same size is measured
identically and reads flat at 0.0000 for every stride, so the statistic
discriminates.
**Result:**
| stride | region match rate |
|---|---|
| 368 | 0.1110 |
| 340 | 0.0935 |
| 337–339 | 0.026–0.029 (background) |
| random control | 0.0000 |
Stride **368** is the strongest peak. On level 0, 340 is a secondary peak
well clear of the background; the two differ by 28, which is unexplained.
**Cross-check:** the peak stride was recomputed independently for every level
that has the region, and **all 24 peak at exactly 368**. That is the part
that makes this worth recording -- a single level's peak could be a property
of that level's data, but 24 of 24 agreeing is a format invariant.
What a 368-byte record *is* remains unestablished. This is the same position
the shrub body was in before differential analysis identified its records,
and the same method is the obvious next step.
### Differential analysis at stride 368: no constant substructure
**Range measured:** level 0, `16047040 .. 16737344`. 1,875 whole 368-byte
records from the region start; variation checked across the first 60.
**Method:** the same differential analysis that identified the shrub body --
compare records byte by byte and record which positions differ.
**Result:** **all 368 positions vary.** There is no constant field, no padding
and no header inside a record. Unlike the shrub body, where differential
analysis immediately separated a varying payload from a constant two-byte
tail, this format has nothing constant to separate.
That does not contradict the autocorrelation peak. Checking which byte
positions carry the 368-stride matches over 20,000 positions: 168 exact 4-byte
repeats, distributed evenly at 0.84% across all four byte offsets. So whole
4-byte sequences recur at exactly 368 while no individual position is constant
-- consistent with records containing entries that repeat at a fixed relative
index, rather than with a fixed-layout record.
**So the 368 stride stands and the records remain unidentified.** The shrub
method failed here not because it was the wrong tool but because it needs
something constant to latch onto, and there is nothing.
## Sky opens with a 256-byte-row image
**Range measured:** level 0, `2192576 .. 2615232` (the sky region, 422,656
bytes). Periodicity start and end located by sliding an 8 KB window in 256-byte
steps through the region and measuring the 256-stride match rate.
**Method:** autocorrelation as above, then a periodicity scan. A random
control of the same size reads flat at 0.0000 for every stride tested.
**Result:**
- Peak stride in the first 64 KB is **256** at a 0.772 match rate. Strides 255,
257 and 4 score nearly as high, which is what a true period looks like
smeared by sampling.
- Across levels the peak is 256 on 21, 512 on 5 and 128 on 1. 512 and 128 are
a harmonic and a sub-multiple, so the fundamental period is **256 bytes**.
**Correction to the control.** The original comparison used uniform random
bytes, which read flat at 0.0000 -- and made the 0.772 look like an enormous
signal. It is not: 91.8% of the region's bytes are the same value (`0x07`), so
a stride-256 comparison matches trivially wherever both positions are `0x07`.
The right control is the same bytes shuffled, which preserves the value
distribution and destroys only the ordering. That control reads **0.512** at
stride 256, not 0.0000. The true excess is 0.772 − 0.512 = **+0.26**, a real
signal and a far smaller one than the original framing suggested.
**And the extent is wrong too.** Re-measured against the matched control, the
excess over the value distribution is:
| offset | excess |
|---|---|
| +0 .. +49152 | +0.38 .. +0.45 |
| +53248 | +0.26 |
| +57344 | +0.07 |
| +61440 | −0.13 |
So the 256-periodic data ends between `+53248` and `+57344` — somewhere in
rows 205 to 224 — **not** at `+60160` as first reported. The original figure
came from a raw threshold with no distribution control, and 91.8% of one byte
value drags that threshold wherever the dominant run ends.
The 256-byte period itself stands, cross-checked on 21 of 27 levels with the
remaining levels showing its harmonic and sub-multiple. The image dimensions
do not: an earlier "256x236" is wrong and should not be used.
So the sky region is not one thing. Its first ~60 KB is an image; the rest is
whatever the layer offsets address. Both halves still need work, and the
image is the tractable one.
## Re-verified: the 368 stride, against a matched control
The sky autocorrelation was found to be inflated by a control that did not
match the data, so the 368 result was re-checked the same way.
**Range measured:** each level's post-shrub region start, first 65,536 bytes.
**Method:** as before, but the control is the same bytes shuffled rather than
uniform random bytes. Shuffling preserves the value distribution and destroys
only the ordering, which is exactly what the control needs to do.
**Result:**
- Level 0's region is evenly distributed: 256 distinct byte values, most
common only 15.6%. Uniform random was therefore the *appropriate* control
here, and the shuffled control independently confirms it, reading **0.0000**
at strides 340, 368 and 4.
- Level 0 peak remains 368 at 0.1110, against 0.0000 for the matched control.
- Recomputed across all levels with matched controls: **24 of 24 peak at
exactly 368, every one with excess above 0.05.**
So the 368-byte period stands, now with a control appropriate to its data.
The contrast with sky is the point: the same statistic on the same kind of
question gave a real answer in one region and an inflated one in the other,
and the only thing that told them apart was whether the control matched the
byte distribution.
## Sky sub-objects: structure is consistent, stride is not
**Range measured:** every level's sky region, from the count field at `+0x0c`
through the sub-object offsets at `+0x20`. Each sub-object measured over its
first 16,384 bytes.
**Method:** as above, peak-stride autocorrelation per sub-object with a
matched control (the same bytes shuffled, seeded per level and index).
**Field note:** the sub-object count is a `u32` at **`+0x0c`**, not `+0x0a`.
Reading `+0x0a` gives 327,680 on level 0, which is not a count.
**Result, level 0:** five sub-objects at offsets 365760, 388880, 396432,
406784 and 414032, of lengths 23120, 7552, 10352, 7248 and 1536. Each has
internal structure well clear of its matched control.
**But the strides do not repeat across levels.** Recomputing per level:
| sub-object | peak strides seen across levels |
|---|---|
| 0 | 4 (x6), 8 (x13), 32, 288 |
| 1 | 16 (x3), 32 (x6), 288 |
| 2 | 8, 16 (x5), 256, 288 |
| 3 | 8, 16 (x6), 216 |
| 4 | 8, 16 (x3), 512 (x4) |
So the earlier per-index stride table, read off level 0 alone, was
over-fitted to one level. What is consistent is weaker and more honest: every
sub-object sits above its matched control, and most peak at a small stride of
4, 8, 16 or 32, which suggests 16- or 32-bit records whose stride varies with
content rather than being fixed.
**Level 2's header remains genuinely different**: its count field at `+0x0c`
reads 196,617 and is unusable, while every other level yields a plausible
count. Whatever makes it different is still unexplained.
### Sky sub-object 0: sparse, with duplicated entries
**Range measured:** level 0, sky region `+365760 .. +388880`, 23,120 bytes,
read as 2,890 records of 8.
**Method:** record composition, plus the adjacent-pair and zero-density
statistics below. The matched-control autocorrelation puts the peak at stride
8 (0.217 real, 0.000 shuffled), so 8 is the unit to describe.
**What it is made of:**
- **1,185 distinct `u32` values** across 5,780 words.
- **Mean 6.99 of 8 bytes are live** -- the previous revision of this file said
"zero", which is inverted and wrong. The live-byte histogram is 0:15, 2:1,
4:31, 5:238, 6:461, 7:1030, 8:1114, so most records are nearly full rather
than sparse.
- **549 of 2,890 adjacent pairs (19%) are exact duplicates of each other.** The
most common 8-byte record, appearing 283 times, is `d7d7d780 d7d7d780` -- a
4-byte value stored twice. Others follow the same shape: `00051980 00051980`,
`00082580 00082580`.
**Byte 7 is anomalous.** It takes only **14 distinct values** where bytes 0-6
take 142-204. Its common values are `0x00` (1378), `0xff` (806) and `0x80`
(592) -- the high bytes of signed 16-bit values, which is consistent with
byte 7 being an MSB of something rather than ordinary data. Unexplained.
**Not decoded.** A signed-16-bit reading was tried and does not obviously
help: the values span -32743..32633, close to the full range, with no
per-position structure (533, 371, 401 and 187 distinct at positions 0-3).
Recorded as a characterisation, with its range, so the next attempt starts
from the composition rather than re-deriving it.
### Sky sub-object 0 is a slowly-varying sequence, not independent records
**Range measured:** level 0, sky region `+365760 .. +388880`, 23,120 bytes.
**Method:** Pearson correlation between the byte sequence and itself at
increasing lag, against a shuffled control. This measures whether consecutive
records are *related*, which the position-wise distinct-value counts could
not show.
**Result:**
| lag (bytes) | correlation |
|---|---|
| 1 | +0.050 |
| 4 | +0.560 |
| **8** | **+0.746** |
| 16 | +0.666 |
| 32 | +0.602 |
| 64 | +0.432 |
| 128 | +0.140 |
| 256 | −0.044 |
| **512** | **+0.304** |
| 1024 | +0.247 |
| shuffled control (lag 1) | +0.003 |
The profile has a peak at lag 8, decays to a zero crossing near 256, and then
a second, smaller peak at 512 before decaying again.
This is what settles the earlier "no constant substructure" reading. The
records do not share fixed fields, but they are *not independent*: consecutive
8-byte records are strongly related, and that relation repeats at 512 bytes,
i.e. every 64 records.
Slowly varying values, related within a record, repeating every 64 records, is
the shape of a keyframe or lookup table -- which is what one would expect to
find in sky data, where a colour or fog parameter is sampled rather than
computed per vertex.
**Still not decoded.** Nothing here names the values or the sampling. But the
structure is now characterised well enough that a decode attempt has a shape
to aim at rather than a field to hunt for.
### Sky sub-objects share one shape, across every level
**Range measured:** sub-objects of the sky region on levels 0, 1, 3, 7, 13
and 26 -- 25 sub-objects in total, each measured over its whole length.
**Method:** as above, correlation at lag against the byte sequence.
**Result:**
| quantity | range over 25 sub-objects |
|---|---|
| lag 8 | +0.53 .. +0.79 |
| lag 64 | +0.16 .. +0.63 |
| lag 512 | -0.13 .. +0.42 (not universal) |
| lag 1024 | -0.05 .. +0.37 (not universal) |
**Lag 8 and lag 64 are positive on every sub-object of every level tested.**
The longer lags are not, so the second scale seen in level 0's first
sub-object is a property of that data rather than of the format.
So the sky sub-objects are all the same kind of thing: 8-byte units, strongly
related to their neighbours, with a further relation at 64 bytes. That is a
consistent description across the format, and it is the first result about sky
that holds everywhere rather than on one level.
**Not decoded.** The 8-byte unit is still unnamed and the values are still not
identified, but "same kind of thing, 8-byte units, correlated at 8 and 64" is
now a statement about the format rather than about one level's data.
## Sound bank SBlk has a 40-byte period (27/27 levels)
**Range measured:** each level's `SBlk` region -- the descriptor table between
`sblk_offset` and `daw_offset`, from the sound bank header rather than from
`core_data`. 17-20 KB per level.
**Method:** peak-stride autocorrelation with a matched control (the same bytes
shuffled, same seed, every level).
**Result:** the peak is at **exactly 40 bytes on all 27 levels**, every one with
excess above 0.2. The peak is also sharply isolated on level 0 -- 0.454 excess
at stride 40 against 0.207 to 0.278 for strides 36 to 44 -- so it is a period
rather than a broad tendency.
That is the first structural result about the sound bank beyond the container
framing already committed.
**Record count:** 487, 443, 428, 494 and 447 whole 40-byte records on levels 0,
1, 2, 13 and 26. The region is not an exact multiple of 40 on any level
(19484 leaves 4 bytes over), so 40 is a period rather than a record size with
no remainder.
**Unexplained:** which byte positions are constant varies by level -- level 0
has 7, 11, 31 and 35 constant; level 1 has 15, 19, 23, 35 and 39; level 2 has
7, 23, 27 and 31. A fixed 40-byte layout would have the same constant
positions everywhere, so either the records have a variable tail or 40 is a
period in something else that the records sit inside.
Also measured, for the record:
- `DAW`, the sample region, shows **no** stride above threshold against its
control. Consistent with ADPCM nibble-packed audio, and consistent with the
container framing being the only part worth decoding.
- `gameplay.bin` peaks at 16 bytes (excess 0.267 at strides 16, 32 and 64).
- `gs_ram.bin` has almost no stride structure (0.043 at 32) but a lag-8
correlation of +0.503, so it is smooth rather than tabular.
## Tie class spans: 16-byte unit, 48-byte period (dominant, not universal)
**Range measured:** every tie class span on levels 0, 3, 6 and every third
level after it -- 386 spans of 16 KB or more, bounded by consecutive class
offsets.
**Method:** peak-stride autocorrelation with a matched control, over the first
32,768 bytes of each span.
**Result:**
| peak stride | spans |
|---|---|
| 48 | 285 |
| 16 | 58 |
| 64 | 29 |
| 96 | 11 |
| 24 | 2 |
| 32 | 1 |
**48 bytes dominates but is not universal**, and the weaker peak is 58 spans
against 285 -- so unlike the sound bank's 40, this is a majority rather than a
constant.
Every observed peak is a multiple of 16 (16, 24, 32, 48, 64, 96), which
suggests a **16-byte base unit** with a 48-byte period arising from how those
units group, rather than 48 being the record itself.
This is weaker evidence than the sound bank's 40-byte result, which was
sharply isolated and identical on all 27 levels. Here the peak is weaker
(0.09-0.22 excess against sound bank's 0.45) and it moves between levels and
spans. Recorded at the confidence it deserves.
## Two independent measurements agree on moby's 16-byte vertex
The moby part table established `vertex_offset + 16 * vertex_count ==
next_part` on 43,467 of 43,467 parts, which fixes the vertex stride at 16.
Autocorrelation over the spans reaches the same number without using that
result at all.
**Range measured:** moby class spans of 16 KB or more on level 0 -- 60 spans,
each bounded by consecutive class offsets, measured over the first 32,768
bytes.
**Method:** peak-stride autocorrelation against a matched control, searching
strides 4 to 64.
**Result:**
| peak stride | spans |
|---|---|
| **16** | **43** |
| 8 | 8 |
| 48 | 4 |
| 32 | 3 |
| 24 | 2 |
16 dominates, and it is the same 16 the part table produces from a completely
different argument -- a tiling identity over part records rather than a
periodicity measurement over span contents.
Two unrelated methods agreeing is worth more than either alone, because the
autocorrelation would be vulnerable to exactly the kinds of artifact this
project has already been bitten by: a dominant byte value would inflate it,
and a period that happened to divide the window would look the same as a real
one. Neither applies here, and the tiling identity cannot be inflated at all.
This is the strongest form of evidence produced in this project: a structural
invariant confirmed by two methods that do not share a failure mode.
## Sound bank SBlk: a 12-byte marker that does not explain the 40-byte period
**Range measured:** level 0 `SBlk`, 24..19508 (19,484 bytes).
**Method:** positions of the `0x7f`/`0x47f` marker words, and their spacing.
**Result:** those markers sit at **12-byte spacing on 144 of roughly 180
occurrences**, with 24-byte and larger gaps accounting for the rest. So there
is a 12-byte structure in the data.
**But that contradicts the autocorrelation.** If records were 12 bytes, stride
12 should be the peak. It is not:
| stride | excess over shuffled |
|---|---|
| 12 | +0.237 |
| 20 | +0.175 |
| 24 | +0.237 |
| **40** | **+0.454** |
| 48 | +0.239 |
40 is not a multiple of 12, so the two measurements disagree rather than one
explaining the other. Recorded as an open discrepancy rather than resolved in
favour of either: the marker spacing says 12, the period says 40, and 40 being
twice as strong is the more surprising of the two.
### Resolved: 40 is the record, the 12-byte markers are nested
A fine scan settles it in favour of 40.
| stride | excess |
|---|---|
| **40** | **+0.454** |
| 39 | +0.266 |
| 41 | +0.215 |
| 38 | +0.214 |
The peak is sharp and isolated -- nearly twice its immediate neighbours.
And the multiples confirm it: 80 scores +0.376, 120 +0.394, 160 +0.340,
while the multiples of 12 (24, 48, 96, 144) all sit at 0.21-0.24.
So the record is **40 bytes**, and the `0x7f`/`0x47f` markers at 12-byte
spacing are a nested structure *inside* those records rather than the record
boundary. Their spacing is irregular -- 12, 24 and larger -- which is exactly
what one would expect of a field that is present in some records and absent in
others.
The earlier "they disagree" framing was wrong in one respect: the marker
spacing never claimed to be the record size. Read as a claim about markers
inside records rather than about the record itself, the two measurements were
never in conflict.
## SBlk record layout at 40 bytes
**Range measured:** level 0 `SBlk`, 24..19508, aligned at +0 — 487 whole
records.
**Method:** for each of 40 candidate alignments, record the number of byte
positions that are near-constant across records, and keep the best.
**Result:** alignment +0 is best, with 10 near-constant positions. Four are
exactly constant and all zero: positions **7, 11, 31 and 35**.
Distinct values per byte position across the 487 records runs from 1 to 155,
with position 29 the most variable by a wide margin (155 against 21-48
elsewhere) and positions 2, 10, 27, 34 and 39 low (2 to 8).
The first record carries the `SBlk` magic and the `DAW` tag rather than
descriptor data, so it is a header rather than an entry — which is worth
accounting for before reading any field at a fixed offset.
**Not decoded.** The layout is established and the constant positions are
known; the fields are not named, and which of the 40 bytes are an index, a
length and an offset is still open.
## SBlk: the field values do not read as record fields
**Range measured:** level 0 `SBlk`, 24..19508.
**Method:** read at 4-byte alignment and rank the most common `u32` values.
**Result:** the common values are `0x0` (2261), `0x1` (369), `0x7f` (168),
`0x5a000000` (125), `0x1eff7f7f` (125), `0x1dfcc` (124), `0x19fc0` (93),
`0x7c34240d` (91), `0x7c3821c8` (91).
**This does not look like a table of records with typed fields.** A 40-byte
record layout whose fields are an index, a length and an offset would not
produce `0x1eff7f7f` and `0x7c34240d` a hundred times each, nor would those
values survive a byte shift.
So either the 40-byte period is not a record boundary, or the region is an
encoded stream -- something like varint or Huffman-coded data -- that happens
to have a 40-byte statistical period.
The `0x7f` values are suggestive: in a varint encoding a byte of `0x7f` is the
last value representable in seven bits, so it appears as a frequent
saturated byte. That is consistent with a coded stream and inconsistent with
a fixed-layout record.
**Left open.** The 40-byte period is solid (27/27 levels, sharply isolated, its
multiples confirm it), and the alignment with four constant zero positions is
solid. What sits in those bytes is not a record layout, and the two findings
are in tension in a way this project has not otherwise seen: a confirmed
period that is not a record boundary.
## Correction: the 368 stride is real, but confined to one ~15 KB window
The claim earlier in this file that the post-shrub region has a 368-byte
period *throughout* is wrong. Re-measured over the whole region, the
autocorrelation curve is flat and near noise:
| stride | excess over matched shuffled control |
|---|---|
| 8 | +0.027 |
| 12 | +0.023 |
| 148 | +0.009 |
| 340 | +0.018 |
| **368** | **+0.018** |
| 632 | +0.020 |
Nothing peaks at 368. The figure of 0.1110 recorded earlier does not
reproduce under any sampling convention tried (step 1, 2 or 4 through the
window; all give ~0.018).
**Where the structure actually is.** Scanning 8 KB windows at 1 KB steps for
a 368-excess above 0.15 finds exactly one substantial region:
region+41984 .. region+57344 = 15,360 bytes (abs 16089024..16104384)
plus a 4 KB patch at region+650240. Everything between and around is flat.
**Within that window the 368 stride is unambiguous:**
| stride | excess |
|---|---|
| **368** | **+0.315** |
| 736 | +0.284 |
| 364 | +0.024 |
| 372 | +0.021 |
| 732 | +0.027 |
The peak is sharp, 736 confirms it as a multiple, and the neighbours are at
noise.
**Record shape.** 41 whole 368-byte records. Mean distinct values per
position is 16 of 41, and **nine positions are exactly constant and zero**:
329 through 336, and 340. Twenty-five positions have 3 or fewer distinct
values, clustered in 325-349 -- a tail region rather than scattered padding.
So the corrected statement is: a 15,360-byte window of the post-shrub region
is a table of 41 records at stride 368, each with a roughly 43-byte tail
whose fields are mostly zero. The surrounding 675 KB is not that table.
**What went wrong before.** The original measurement was taken over a window
where 368 held and then reported as a property of the region. The 15 KB
window holds 41/1841 of the region's records, and averaging that over the
whole region flattens the peak to the noise floor. The re-verification pass
checked the stride but not whether it applied uniformly, so it confirmed the
number without testing the claim it was attached to.
## The post-shrub region is a chunk container (explains the 368 stride)
**Range measured:** level 0 post-shrub, 16047040..16737344 (690,304 bytes).
**Method:** scan for the block shape `16-byte header ending 0xff` followed by
10 zero bytes followed by a run of `u32` whose top nibble is `0xF`.
**Result: 28 chunk headers on level 0**, present on all 27 levels. Each chunk
is:
+0x00 16 bytes, last byte 0xff (opaque; differs per chunk)
+0x10 10 zero bytes
+0x1a N tagged offsets, top nibble 0xF
and within a chunk the offsets step by a **chunk-specific stride** — 148,
340, 368, 288, 320, 236, 272, 384, 116, 160, 250, 580, 576 all occur on
level 0. The stride is near-constant within a chunk (e.g. 148,148,148,148...
then 340,340,336,340...) but differs between chunks.
**This resolves two earlier results.**
The "368-byte period" is one chunk's stride, not a property of the region.
That is why the region-wide autocorrelation is flat: the region holds chunks
with ~15 different strides averaging out to noise. The 15,360-byte window where
368 scored +0.315 is the third chunk, at region+30066, whose stride is 368.
The "ten-entry table" I earlier disconfirmed is real — chunks carry 5 to 30
tagged offsets, with 10 the most common on level 0. It was disconfirmed because
I tested for a fixed stride of 148 across a whole region; the table exists,
but 148 is one chunk's stride among many.
**Two corrections this produces.** Both earlier claims were half-right: real
structure, wrong scope. I reported a local measurement as a regional one in
both cases, which is the same mistake twice.
**Not decoded.** What the 16-byte header is, what the `0xF` nibble tags, where
the chunk data lives, and what the chunk *means* are all open. The container
framing is established; its contents are not.
## Chunk records resolve, and the layout is consistent across levels
**Range measured:** each level's post-shrub region, taken as the 900,000 bytes
before the constant trailer.
**Method:** for each chunk, read the tagged offsets as `0xF0000000 | low24`,
take the stride as the first delta, load one record per offset at
`chunk_start + low`, and count byte positions that are constant across that
chunk's records.
**Result: 1,096 chunks and roughly 13,000 records resolve on all 27 levels.**
Every level's chunks show 16.7-20.0% of byte positions constant across their own
records — a consistent field layout rather than noise.
Level 0 in detail: 27 chunks resolve, 466 records, 20.8% constant. The
stride-368 chunk (77 records) has 26 constant positions out of 368; the
stride-340 chunk (77 records) has 25 out of 340. Constant positions cluster
near the record end — 267-276 and following for stride 368, 258-268 for stride
340 — which is a footer or tail, not scattered padding.
**The tag is uniform.** Every tagged offset seen has top byte `0xF0`, i.e.
`0xF0000000 | low24`, on all levels.
**What this settles.** The container framing from the previous entry is not a
coincidence of byte patterns: the offsets genuinely address records, and those
records share a field layout. The chunk-relative base is confirmed because
resolving at `chunk_start + low` yields records with ~20% constant positions,
which does not happen by chance.
**Still not decoded.** The 16-byte header (opaque, per-chunk), the meaning of
the `0xF0` tag, which fields are index/length/offset, and whether a chunk is
moby_anim or something else. A parser can extract chunks and records
bounds-checked today; naming fields needs a domain argument this project has
not yet found.
## The chunk header: byte 14 is the record count
**Range measured:** all 27 levels, 1,123 chunks, over the 900,000 bytes
before the constant trailer.
**Method:** compare header byte 14 against the number of offsets actually
resolved for that chunk.
**Result: byte 14 equals the record count on 1,123 of 1,123 chunks, zero
mismatches.** This is the first genuinely *decoded* field in this container
rather than a measured correlation.
The parser now truncates the offset table to header byte 14, so a stray
tagged word appearing in a chunk's record data cannot silently extend the
table -- which would otherwise let the count disagree with its own header.
**Two more header bytes are near-constant and look like ASCII:**
- byte 13: only two values across 1,123 chunks -- `0x45` (1,076) and `0x46`
(47).
- byte 9: three values -- `0x44` (26), `0x45` (1,084), `0x46` (13).
`0x44`, `0x45` and `0x46` are ASCII `D`, `E` and `F`. That three-letter tag
appearing at two different byte positions with different distributions is
suggestive of a type or class field, but it is not established: the remaining
14 header bytes are high-entropy and do not read as ASCII at any shift tried,
so the ASCII in bytes 9 and 13 could be incidental.
**Still open:** the 14 opaque header bytes, whether bytes 9 and 13 are type
tags, which record fields are index/length/offset, and whether a chunk is
moby_anim. The `D`/`E`/`F` values are suggestive enough to be worth testing
against a known type table, but that has not been done.
## The header is a type ID, and the type determines the stride
**Range measured:** all 27 levels, 1,123 chunks.
**Method:** group chunks by their 16-byte header; for each distinct header,
collect the set of strides observed across every level it appears on.
**Result: only 64 distinct headers across 1,123 chunks.** The header is a
type identifier, not a per-chunk descriptor -- the same 16 bytes recur on
every level.
**63 of the 64 headers imply exactly one stride.** So header to record layout
is very nearly a function: the type determines the record size.
**Resolved: the single exception is three types sharing a header.**
Header `7542eff271b522a79e45348cca4509ff` was reported with three strides, 76,
92 and 104. It is not one type with a variable stride, and not a misparse.
It appears **three times per level**, always with record count 9, always in the
same order and always at the same offsets within the region. On level 0 and on
every other level checked:
| offset | stride | count | first offset | last offset |
|---|---|---|---|---|
| 863842 | 92 | 9 | 928 | 1640 |
| 867554 | 76 | 9 | 912 | 1520 |
| 872866 | 104 | 9 | 1000 | 1832 |
The offsets are 3,712 apart each time and the strides differ, so these are
three separate chunks that happen to share 16 header bytes.
**Consequence for the type ID.** 64 distinct headers for 66 distinct
record layouts, not 64 for 64. The header is *nearly* a type ID but does not
uniquely determine the layout: whatever distinguishes these three is not in the
16 bytes the header captures. Candidates not excluded: a distinguishing byte
sits outside the 16 captured, or the layout varies within a type.
This does not change the earlier conclusion that 63 of 64 headers imply one
stride -- that remains 63 of 64 -- but the mapping is not a bijection and
should not be used as one.
**The header does not encode the stride.** All 90 two-byte windows of
`header[0..14]` were tested: none maps to a unique stride across every group.
So there is no simple two-byte field to read the record size from, and the
stride must come from the offset table, as the parser already does, by
differencing the first two tagged offsets.
That is the right implementation anyway, and this makes it more than a
convenience: given three chunks sharing one header with three different
strides, any dispatch that assumed "header implies size" would be wrong for
1,123 chunks' worth of type coverage on at least one layout. Reading the
stride from the offsets is the only interpretation consistent with all the
measurements.
**The ASCII reading is confirmed as a type tag.** Grouping by header bytes 9
and 13:
| tag | chunks | distinct strides |
|---|---|---|
| `DE` | 26 | 2 (184, 344) |
| `EE` | 1050 | 46 |
| `EF` | 34 | 2 (268, 640) |
| `FF` | 13 | 1 (520) |
The rare tags are tightly coupled to their layouts -- `FF` is a single stride
across all 13 of its chunks -- while `EE` is heterogeneous with 46. That
pattern is what a type tag should look like, and it settles the earlier
suggestion that the D/E/F bytes were incidental: they are not.
**What this is good for.** A chunk can now be dispatched on its header to a
record layout, and 63 of 64 headers determine that layout's size without
inspecting any record. That is a usable parse entry point, and it is the first
point in this container where a field is doing real work rather than merely
correlating.
**Still open:** what the 64 types mean, which record fields are
index/length/offset, and whether any type is moby_anim.
## RETRACTED: chunk records referencing moby class IDs at 38x chance
This section previously claimed that chunk records reference moby class IDs, with
one body slot per chunk type, at 38x-289x enrichment over chance, and that >= 20
chunk types showed it.
**That was a measurement of the parser, not of the disc.**
`resolve` sliced every record at the chunk's *first* offset delta. On 1,252 of
1,509 chunks the later deltas are smaller -- word-aligned multiples of 4, most
exactly +/-4 -- so those slices ran past the record's real end and carried **4 to
8 bytes of the following record**. A record with the head of the next one glued
to its tail scores as though it carried extra fields, and a slot scan across
those bytes lifts apparent hit rates without anything being true.
With record boundaries measured against each record's successor, and ragged
chunks skipped, what survives is:
chunk types with a varying class-ID-like slot 2 (was >= 20)
of those, ALSO spanning the ID space 0
So the general claim -- one class-ID slot per chunk type, across the container --
does not hold and is withdrawn.
### And the flaw was there from the start
The discriminator the original version lacked is simple: **a slot that really
carries moby class IDs must span the ID space**, so some of its values land above
a level's moby count. The artifact does the opposite.
type slot enrichment distinct values ABOVE the moby count
0xd76cc453 6 305x 9 0
0xf2ef4275 34 64x 9 341
0xf2ef4275 7 46x 16 412
0xf2ef4275 14 34x 16 432
0xd76cc453 9 32x 28 702
The 305x slot's nine values are `1, 3, 4, 5, 10, 11, 12, 13, 15` -- all under a
level's ~215 moby classes, so every one scores as a hit and 65536/215 = 305x.
Every slot that actually spans the ID space scores 10x to 64x, which is chance.
So the original claim's own numbers gave it away and it was not read that way:
**"5 to 42 distinct values at 153x to 289x"** is exactly the artifact's shape --
a handful of small values, none above the count. The record-boundary fix was
never what made this claim wrong. It was wrong when written, and the retraction
is complete rather than partial.
The test now separates the two: it counts slots clearing 100x (currently 1) and
slots that clear 100x *while also* having values above the moby count (currently
0). The second must stay empty; the first is expected to be the artifact. The
count above the moby count is tracked per **slot**, not per type, because a type
with one artifact slot and several ordinary slots must not have the ordinary
ones vouch for it.
The `distinct >= 4` filter this section introduced is still load-bearing and is
unaffected: without it the top slots are constants that read as 305x, because any
value below the moby count counts as a hit.
## Shrub body records on level 0: six varying bytes, two constant
**Range measured:** level 0, shrub class table 7 entries at core index offset
10176. Six classes are bounded by a successor; the seventh runs to the end of
`core_data` and is excluded, because its span contains the whole post-shrub
region and is not wholly records.
**Method:** bound each span by the next class's geometry offset, take the body
from `data_offset` to the span end, and split into 8-byte records.
**Result: 164 records across 6 bounded bodies**, body lengths 192 (twice), 208
(twice), 224, 240.
Per-byte, over those 164 records:
| byte | distinct | zero |
|---|---|---|
| 0 | 74 | 60 |
| 1 | 58 | 65 |
| 2 | 76 | 43 |
| 3 | 62 | 48 |
| 4 | 72 | 35 |
| 5 | 59 | 38 |
| **6** | **1** | **164** |
| **7** | **1** | **164** |
Bytes 6 and 7 are zero in every record. Bytes 0 to 5 take 58 to 76 distinct
values of a possible 164, with only 21-39% zero -- that is, uniformly varied
rather than sparse.
Read as `u32` the fields are not plausible: `u32[0]` ranges 0 to 4,294,871,316
and no field is monotone across the body. So the six bytes are not two packed
32-bit values.
**What this settles.** The six-varying-plus-two-constant shape holds on level 0
as a per-level partition rather than only in aggregate across the disc's
8,938 records, which is what rule 3a asks for. Two trailing zero bytes are
padding or a flag pair, not data.
**What it does not.** Resolved by a follow-up measurement: the six bytes are
not an `f32` pair either, and not narrow `u16`s.
- `f32 @0`: 158 of 164 finite, but the range runs to about 1e35. Not positions
or normals. The 6 non-finite values (3.7%) are far above the 0.05% that
random bytes would produce, so the field is *structured* but is not an `f32`.
- `f32 @4`: all 164 finite, range 0 to 9.18e-5. This looks promising until you
notice bytes 6 and 7 are always zero, which pins this value to a coarse grid
and explains the tiny range. It is not independent evidence.
- `u16 @0`, `@2`, `@4`: distinct counts 85, 90 and 80, each spanning essentially
the whole 0..65534 range. These are not small counts or indices.
So excluded are: two packed `u32`, an `f32` pair, and narrow `u16` fields. The
six bytes remain uninterpreted.
**Process note.** The first attempt at this used `ClassEntry` and a
hand-computed `o_class` offset instead of `ShrubClassEntry`, and every
`ShrubHeader::region` call failed. The parser was not wrong; the caller was.
The working call shape was already in the disc test.
## tie: both indirections, and one reading disproved
**The chain, which is two steps and easy to get wrong.** The tie class **table**
lives in the level's `core_index.bin`, not `core_data.bin`. On level 0 the three
tables chain as moby at `192`, 238 x 32 bytes later tie at `7808`, 74 x 32 later
shrub at `10176`, which closes. Reading the same table out of `core_data.bin`
produces plausible garbage rather than an error -- `offset_in_asset_wad` comes
back as 1,217,208,842, which is outside every buffer, so the mistake announces
itself if you look. Then `ClassEntry::offset_in_asset_wad` addresses
**`core_data.bin`**, despite its name and despite `core_data.wad` sitting
alongside it.
**Verified:** all **1,864** tie classes resolve, `o_class` agreeing on every one,
and all **5,592** LOD block offsets land inside `core_data`.
**Disproved:** the word at `+0` of a 16-byte unit inside an LOD block. On the
first twelve units of level 0's class 0 it rises strictly -- 288, 1520, 2848,
3968, 5088, 6096, 7312, 8528, 9792, 10976, 12016, 13072 -- which reads exactly
like a linked list, and the accompanying `i16`s looked quantised (two distinct
values in `i16[0]`, three in `i16[1]`).
Across the whole population it is strictly increasing on **0 of 1,864** blocks,
and `i16[0]` takes **8,969** distinct values on level 0 alone rather than two.
The sample was twelve units out of 1,724,334. Both the monotonic reading and the
quantisation were artifacts of that sample, and the disproof is asserted in
`crates/gc-extract/tests/disc.rs` so it cannot creep back.
**Record size: RETRACTED. Block size is a multiple of 16; the records are not a
uniform 16 bytes.** The earlier
"divisible by 16" figure was computed from the "next `packets` offset ends this
block" rule, which is unsound because **`packets` is strictly increasing on only
461 of 1,864 classes**. On the other 1,403 the two offsets are unordered, so they
cannot both be block starts into one ascending region.
Restricting to the 461 classes where the rule does hold: **922 of 922**
consecutive block sizes are a multiple of 16. That was taken as evidence for a
16-byte record. **It is not**, and the record-size claim is withdrawn.
What actually survives, and what killed it:
- Every one of those **922 blocks opens with a `u32` that is positive and a
multiple of 4** (922/922), ranging 48 to 1,744 with 17 distinct values across
level 0's LOD0 blocks. So the block opens with a count, not a size and not a
pointer.
- Reading that `u32` as a byte size and chaining to the next block lands exactly
on the next block start on **0 of 1,383** blocks. It is not a size chain.
- Assuming `[header][count][count uniform records]` and solving for the record
size: **no record size tiles the block for any header size up to 512 bytes.**
Not 16, not 12, not 8.
- Reading the block as uniform 16-byte units gives fields spanning their entire
legal range -- `u32@0` from -2,139,029,504 to 2,139,029,587, `u16@8` and
`u16@10` both 0..65,535, and 8,969 distinct values in `i16@0` alone on level 0.
Uniform units would not produce that.
So the 16 is a property of the **block size**, not of a record. The previous
commit's "record size is 16 bytes" was an inference from a divisibility fact that
does not carry it, and it is withdrawn rather than softened.
**`packets[0]` is a constant, not an offset.** It reads **256 on 1,864 of 1,864**
classes, while `packets[1]` and `packets[2]` vary widely and land at different
addresses in `core_data` on every class. Read as a block start, 256 points just
past the 0x48-byte header -- which is exactly how it was being used, and exactly
the kind of field that gets mistaken for a pointer. It is now
`TIE_PACKETS0` and nothing depends on it.
**Still not decoded:** what the 16 bytes hold. `i16[0]` at `+4` takes 8,969
distinct values on level 0 and is not the two-valued field a 12-unit sample
suggested; the word at `+0` is not a run offset.
## tie: the LOD block header, decoded
Over the **922** blocks where "the next `packets` offset ends this block" holds,
the block's own 16-byte header is measured. `gc_formats::tie::TieBlockHeader`
parses it and a real-disc test asserts every line below.
+0x00 u16 count multiple of 16 on 922/922; range 16..2096;
72 distinct values disc-wide
+0x02 u16 zero on 922/922
+0x04 u16 ONE OF FOUR VALUES DISC-WIDE: 0x501, 0xa02, 0xf03,
0x1404 -- low byte 1..4, high byte 5/10/15/20
+0x06 u16 15 distinct, 0x0206..0x0512
+0x08 u16 135 distinct, 0x0b05..0x4ee4
+0x0a u16 45 distinct
+0x0c u16 94 distinct, 0x010e..0x0553
+0x0e u16 zero on 914/922
**`+0x04` is packed, not a magnitude.** Four values across the whole disc, with
the two bytes moving independently (low 1-4, high 5/10/15/20 in lockstep), is two
small fields sharing a word. That it is a texture index paired with a scale is a
guess; that it is two fields is not.
**The count is not a record count.** `block_size - header` is divisible by it on
**0 of 922** blocks, for *any* record size and *any* header up to 512 bytes.
Combined with block sizes always being multiples of 16, that rules out a uniform
stride without a header.
## tie: the block is a table of 16-byte records (from RAM)
**Range measured:** every tie LOD block of level 1 whose leading word is a
non-zero multiple of 16 -- 16 blocks -- plus the two read by hand in full.
**Method:** locate each block's record table in PCSX2's RAM by its twelve field
bytes (the records are the only part of a block with a distinctive 12-byte
run), then compare every record against its disc copy. Each block has several
copies in RAM; the one that matches is the loaded copy.
**Result: on 16 of 16 blocks, every record's leading word has become
`table_address + word`, and the other twelve bytes are byte-identical to the
disc.** So:
- A block is `[16-byte header][record table][vertex payload]`, and the records
are **16 bytes**: a leading `u32` pointer and twelve bytes of fields.
- The header's word at `+0x00` is the **byte offset from the block start to
the vertex payload** -- that is, the length of the record table. This is what
the "count" reading was groping at: it is a multiple of 16 on all 2,177 blocks
because the table holds 16-byte records, it is never zero because there is
always at least the header-sized first record, and its 42 distinct values are
table lengths, not counts of anything in the payload.
- Every record's leading word is a pointer **relative to the start of the record
table**, relocated at load to `table + word`. The twelve field bytes are
copied verbatim.
This also explains the dead end that ran before it. The payload length divided
by the word at `+0x00` is a whole number on **0 of 922** blocks, at any record
size and any header up to 512 bytes -- which is the correct measurement of the
wrong hypothesis: the word never described the payload, it described where the
payload starts. A count of what follows a pointer is not the pointer.
`TieBlockHeader::data_offset` and `TieBlockHeader::records` carry it, bound by
the offset, and the disc test asserts the table is bounded on every block.
### The records are R&C1's `TiePacketHeader` (2026-10-11)
With the record table located and the fields isolated, the reference settles
what they are. **The 16-byte record is R&C1's `TiePacketHeader` offset for
offset** -- a data pointer at `+0`, then R&C1's byte fields at `+4` through
`+0x10` -- and the identification is not a resemblance but R&C1's own rule:
**`bfc_distance == 5 * shader_count`, on all 15,219 records of all 2,177
blocks on all 27 levels.**
Measured over the same set:
| relation | holds |
|---|---|
| `bfc_distance == 5 * shader_count` | **15,219 / 15,219** |
| `vert_ofs <= vert_size` | all |
| `scissor_size <= scissor_ofs` | all |
| `control_size <= control_count` | all |
| `color_ofs == vert_size` | 12,640 / 15,219 |
| `color_ofs == vert_ofs + vert_size` (R&C1's rule) | **0 / 15,219** |
| `color_count`, `strip_count`, `strip_vertex_count` zero | 15,089 / 15,077 / 15,078 |
`TieRecord` parses all twelve fields; `invariants_hold` asserts only the
universal four, and the disc test counts the rest. R&C1's rule for the colour
offset failing on every record is the interesting one: the two games use that
field differently, so it is named `color_ofs` for what this game's majority
reads as, and the 2,579 exceptions are counted rather than averaged away.
**A packet-length formula is retracted.** `(vert_ofs + vert_size +
scissor_ofs + scissor_size) * 16` was measured at 61 of 61 pairs and is not
reproduced on the correct blocks -- 0 of 13,042. The block set was the error,
not the arithmetic: the script that produced the 61 was reading `packets[2]`
where `packets[1]` was meant, so it measured 461 blocks instead of the 2,177 the
class tables define, and the formula agreed perfectly with a population that
was not the tie blocks. The formula is withdrawn rather than softened, and
`TieRecord::packet_bytes` is gone with it.
### It is now a command, and it verifies two levels
The check above was a script; it is now `gc-trace verify-tie-records
<unpacked-dir> <level> <class> <lod> <src>`, which resolves the class, parses
the block, searches the RAM image for the loaded record table and reports what
it found. Exit status is 0 when every record relocated with verbatim fields.
Run over every (class, lod) slot of two levels, from PCSX2 savestates:
| level | slots with a block | verified | mismatched | absent |
|---|---|---|---|---|
| 0 | 87 | **87** | 0 | 61 |
| 1 | 104 | **104** | 0 | 78 |
So the relocation rule holds on **every** LOD block of two levels that the game
had loaded. Two operational notes worth keeping:
- **The live emulator is not a stable input.** Verifying level 1 over PINE
matched 104 of 104 while the user was on that planet; the same command
minutes later, after they had returned to level 0, found the tables in RAM
but no matching copy -- because the level was no longer loaded. A miss there
is not a counterexample, and the report says "no copy matched" rather than
"failed".
- **A class index past the table count is an error**, not a mismatch, so a
sweep should stop at the level's class count rather than at a round number.
What the records' fields name and what the payload holds are still open.
## The tie packet's length, from its record's fields (2026-10-11)
**Range measured:** every tie LOD block on the disc -- 2,177 blocks, 13,042
consecutive record pairs.
**Method:** a record's pointer is where its packet starts, so the distance
between consecutive pointers **is** a packet's length, with no reference to any
field. Comparing that against the record's own fields gives the length rule.
**Result: `packet_quadwords = vert_ofs + vert_size + scissor_size`, on 12,927
of 12,927 record pairs whose `strip_count` is zero -- and on 0 of 115 whose it
is not.** The partition is exact: every record with a non-zero `strip_count` is
an exception and no record with a zero one is. The fourth term is 0 quadwords
whenever `strip_count == 0`, and something else (measured 0..19 quadwords, and
*not* equal to `strip_count`) whenever it is not. R&C1's `strip_count` is a
strip count and strips are variable-length data, so a non-zero count most
likely adds per-strip bytes this cannot see -- a hypothesis, labelled as one.
`TieRecord::packet_quadwords` and `TieRecord::packet_quadwords_are_known` carry
it; the latter is the honest gate, and the disc test asserts both sides of the
partition.
**This settles a dispute inside this file, and the retraction was the error.**
A formula of this shape was derived at 61 of 61 pairs, then "retracted" as not
reproducing -- re-measured at 0 of 13,042. The retraction was wrong: the
re-measurement had swapped `strip_count` for `scissor_ofs` in the fourth term,
and the blame went on the block set. Re-measured with the term the first pass
had, it matches on 12,927 of 13,042, and the 115 exceptions are precisely the
non-zero `strip_count` records.
The lesson is worth keeping because it is the same shape as two others here: a
retraction is not automatically more careful than the claim it retracts, and the
re-measurement that "disproved" this one changed the thing it was measuring.
The lesson on the other side is the one that saved it: the first measurement
was over 61 pairs, which is not a population -- and had it stayed at 61 the
formula would have looked universal. Both the small sample and the bad
re-measurement were near-misses in opposite directions.
What this unblocks: the payload can now be sliced per packet, which was the
prerequisite for decoding what it holds.
### What the payload is, and what retired the coordinate reading
The payload is **not random**, and that took a control to establish. Reading each
block as 3 x `f32` triples, 53-64% of reads land within +/-5000 of origin; a
uniform-random control of identical length gives **16%**. So the content really
is float-shaped.
But the rate is **flat across every start offset and every stride tried**
(0x10..0x40 start, 8/12/16/20/24 stride: 53-65% throughout). A statistic that
cannot distinguish the right stride from the wrong one cannot identify a stride,
which is rule 4's exact case. The flatness is the informative part: combined
with the count not tiling, it says the records are **variable length**.
A VIF1 command walk was implemented and tested against the blocks: it consumes
**0 of 88** exactly. So the payload is variable-length and not a VIF1 CPU-format
stream. Not WAD LZ77 either -- `wad::decompress` rejects the block at every
start offset tried.
Scanning every (start, stride) pair from 0x10 to 0x7f for byte positions with a
small distinct-value vocabulary -- the signature of integer index fields -- finds
**none**: every byte position takes all 256 values at every alignment. For float
payload that is expected, and it is why the field scan found nothing rather than
confirming anything.
**Still open: what the payload holds.** Established is that it is variable-length,
float-shaped, non-random, not LZ77, and not VIF1. Not established is any record
boundary.
## tie: the LOD block is self-delimiting, and `packets == 0` means absent
**The terminator.** Every tie LOD block ends with the byte `0xf6` followed by
zeros, on **2,177 of 2,177** real blocks across all 27 levels. That was worth a
control, because `0xf6` could have been a coincidence of byte frequency: it
occurs in payloads at **0.83x** the uniform rate, i.e. slightly *less* often
than chance, while the last non-zero byte is `0xf6` on 909 of 922 blocks in the
first subset tested against **17 of 922** for the same blocks cut at a random
point. So it is a terminator, not a common byte.
This is what makes the blocks parseable at all. `packets[1]` bounds a block on
only 461 of 1,864 classes, so the terminator is the only general way to find one.
**`packets == 0` means no block at that LOD.** Of the 3,728 (class, LOD) slots,
**2,177** point at a block and **1,551** are exactly zero. The zero cases are not
degenerate blocks -- they are absent ones.
### A measurement I got wrong, twice, in opposite directions
The first pass over this header reported **3,728 blocks, all terminating** and
`packed_texture` taking **842** distinct values, contradicting an earlier
four-value result. I retracted the four-value claim on that basis.
Both numbers were wrong, and they were wrong the same way. The scan did not skip
`packets == 0`; it computed `class_base + 0` and walked forward from the **class
body**, finding a terminator there by accident. That contaminated 1,551 of the
3,728.
With the zeros excluded, the slot takes **exactly four** values on all 2,177
blocks -- 0x501 on 1,069, 0xa02 on 720, 0xf03 on 350, 0x1404 on 38 -- and all
2,177 terminate.
So: the four-value claim was right, my retraction of it was wrong, and the
retraction was made on a number from the same contaminated set. A retraction is
not automatically better than the claim it retracts; it needs its own bounds.
## The located `0xf6` terminator is inside the last packet (2026-10-11)
**Range measured:** every tie LOD block on the disc, 2,177 blocks.
**Method:** the last packet's end is known two ways. Its length comes from its
record's `packet_quadwords`, and every *other* record's length is confirmed by
the next record's pointer -- so only the last packet of each table is
unconfirmed. Comparing its computed end against where `parse_tie_block` stopped:
**the last packet's computed end lands 2 to 16 bytes past the located `0xf6`,
on all 2,150 blocks measured.** The error is not a multiple of 16, so it is not
a length error: a packet ending exactly at the terminator would give a
quadword-aligned difference. The `0xf6 00` pair therefore sits *inside* the last
packet's data, 2 to 16 bytes before its end.
Two readings fit: either the terminator was located too early, or the last
packet's length over-predicts by an amount that isn't a multiple of 16. The
first is likelier, and the test asserts the measurement rather than the
reading. What still holds either way: the lengths are right for every record
but the last (12,927 of 12,927 record pairs), and the block *starts* are
RAM-confirmed. The block *extent* is what is short.
**Why the existing test missed it.** `2,177 of 2,177 blocks end on 0xf6` is
satisfied by a false terminator exactly as well as a true one -- it checks that
the byte it walked to *is* a `0xf6`, not that it is the block's end. A test that
confirms the shape it walked to will pass on the wrong byte.
The cost: `payload` and `padding` under-report, so anything computed from them
is correspondingly short.
### Two candidate fixes, both measured and neither clean (2026-10-11)
**Range measured:** every tie LOD block, 2,169 blocks with a known last packet.
**Candidate 1: the first `0xf6 00` at or after the last known packet's end.**
Fails. The offsets found are spread (1082, 1132, 1148, 1150, 1182, 1214 ...),
not concentrated, because the 115 records with a non-zero `strip_count` carry
packets whose length is unknown and therefore sit after the last known end.
The scan lands inside those.
**Candidate 2: the block's data ends at a long zero run.** Fails. Walking
forward from the last known packet's end to the first run of 8+ zero bytes, the
byte preceding the run is not dominated by `0xf6` (2, 4, 1, 64, 3, 32, 19 ...
across 2,169 blocks) and the runs themselves are mostly 8-31 bytes. The zero
padding after a block is short and irregular, so run length does not
discriminate the block's end.
Recorded as negative results because both are the obvious thing to try next
and both are dead ends. What would unblock it: a survey of which offsets *do*
bound blocks (the 461 that work may share a property), or the level loader's
own block-walking code, which needs the overlay relocation from Phase 3.
### Corrected header, over all 2,177 blocks
+0x00 count multiple of 16 on all; 42 distinct; never zero
+0x02 zero on all
+0x04 exactly four values: 0x501, 0xa02, 0xf03, 0x1404
+0x06 16 distinct, never zero
+0x08 180 distinct, never zero
+0x0a 61 distinct, never zero
+0x0c 125 distinct, never zero, clustered 0x043f..0x0443
+0x0e 11 distinct, zero on 2,162 -- the only non-universal field
`+0x0c` clusters around 1,087 to 1,091, close enough to a block length to be
worth stating, but it is **not** the length: the difference against the true
payload length scatters from -12 to -172.
## chunk records: the leading byte is a bucket index, not data
Characterised across **all 77 chunk types**, 1,509 chunks and 34,491 records on
all 27 levels.
**Byte 0 of every record is a small aligned index.** It takes 16 discrete values
per chunk type -- 0x00, 0x10, ... 0xf0 -- with near-uniform counts *within* each
type. Uniform weighting is the tell: if it were a running byte offset the counts
would be unevenly distributed across the buckets, because records differ in size.
```
type_id recs len byte0
0x007d7ec438 2430 328 0x00:162 0x10:162 0x20:162 ... 0xf0:135
0x00d3d8c4a9 2430 308 0x00:162 0x10:162 0x20:162 ... 0xf0:135
0x00429ec43d 2079 368 0x00:162 0x10:135 0x20:135 ... 0xf0:108
0x00d76cc453 810 108 0x00:27 0x08:27 0x10:27 ... 0xe8:27 <- steps by 8
```
**The alignment step is per type, not global.** The leading byte is a multiple
of 8 on **34,432 of 34,491** records, but a multiple of 16 on only **33,349** --
the 1,142-record difference falls in seven types, four of which step by 8.
59 records in three types break even the 8-byte rule. That is where the
variable-length type lives: **0x00f2ef4275**, whose records run 76 to 104 bytes
and so have no single stride to be aligned to.
**Not named.** What the bucket indexes is not established. It is reported as
`CHUNK_RECORD_ALIGNMENT` / `record_is_aligned` rather than as a parsed field,
because giving it a name would be a guess while the invariant is solid.
### A measurement I initially got wrong
The first version of the constant claimed a 16-byte multiple on 34,432 records.
34,432 is the **multiple-of-8** count; the multiple-of-16 count is 33,349. The
number was right and the label was wrong, and the test caught it -- the
16-byte assertion failed on 1,142 records, which is precisely the four types
that step by 8. Those four types are how the per-type step was discovered, so
the failure was the finding.
Worth stating plainly: this is the second time this session a correct number
carried a wrong label, after the tie four-value retraction. Both times the test
was the thing that caught it, which is the argument for asserting the number in
a test at all rather than only in prose.
## The chunk record's leading byte is a per-chunk phase counter
Follow-up to the alignment result, and it settles what the leading byte is.
Within one chunk the leading byte steps by a constant and wraps:
offsets: 1420 1748 2072 2392 2716 3044 3372 3700 4028 4352 ...
byte0: 0 16 32 48 64 80 96 112 128 144 ...
i.e. `byte0 == step * ((index + phase) mod (256 / step))`. Sweeping every
`(step, phase)` over all 27 levels:
fully explained: 1,262 of 1,509 chunks
by step: 8 -> 51 chunks, 16 -> 554, 32 -> 520, 64 -> 137
unexplained: 247 chunks across 11 chunk types
**So on 1,262 chunks the leading byte carries no per-record information.** It is
a deterministic function of the record's index and the chunk's phase. That makes
it a position marker, not data -- which is why the earlier "16 discrete values
with uniform counts" pattern showed up.
`ResolvedChunk::bucket_rule` returns the `(step, phase)` when the whole chunk
fits and `None` otherwise, so the 247 unexplained chunks are distinguishable
from the 1,262 explained ones and nothing skips a leading byte on the strength of
a rule that does not apply.
### RETRACTED: "every chunk has a uniform record length"
**Claimed: 1,509 of 1,509 chunks. True: 257 of 1,509.**
Length varies between chunks of the same type -- one type runs 76 to 104 bytes --
and on 1,252 chunks it varies *within* one as well.
**Both halves of that old entry were wrong, and both were mine.**
"Never within one chunk" was `resolve` slicing every record at the chunk's first
offset delta. Every record then came out the same length by construction, so the
predicate was confirming the parser back to itself.
And this section's *withdrawal* of "variable-length type" was wrong too. It read
the pooled observation as a partitioning artifact -- `minlen`/`maxlen` taken across
all of a type's records at once. Rule 3a is right about pooling in general, and it
was applied to a real signal here: the type genuinely has variable-length records,
and the per-chunk check that would have caught it was itself reporting the
parser's own slicing.
Two errors stacked. The pooling rule fired correctly and was overruled by a
measurement that had no independent content.
## The first TWO bytes encode the record's own index
Continuing from the phase counter. Within a chunk:
i : 0 1 2 3 4 5 6 7 8 9 10 11 ... 15 16 17 ...
byte0: 0 16 32 48 64 80 96 112 128 144 160 176 ... 240 0 16 32
byte1: 0 0 0 0 0 0 0 0 0 0 0 0 ... 0 1 1 1
`byte0` is `step * (j mod g)` and **`byte1` is `(j / g)`**, where `g = 256/step`
and `j = index + phase`. So the pair is the record's index in mixed radix:
`byte0` carries the low digits scaled by `step`, `byte1` the quotient.
**Measured over all 27 levels: 1,235 of 1,509 chunks** fit this on both bytes.
Byte 0 alone fits 1,262, so the 27-chunk difference are chunks whose first byte
follows the rule and whose second does not.
**So on 1,235 chunks a record knows its own position**, and the two leading bytes
hold no per-record information. `ResolvedChunk::index_rule` returns the
`(step, phase)` only when the *whole* chunk fits on both bytes, so the 274 that
do not stay distinguishable.
### A number that moved when the implementation changed
A first probe reported **1,181**; the committed method reports **1,235**. The
method is right. The probe accumulated `ok0`/`ok1` flags across candidate phases
with an early-exit that could skip a phase which would have matched both bytes,
and it also disagreed with `bucket_rule` on the byte-0 count (1,208 against
1,262), which is what gave it away.
Recorded because the two numbers came from the same run and only the discrepancy
exposed the probe as wrong. The lesson is narrow and worth stating: when a
one-off probe and a committed helper disagree, check the probe first.
## Chunk record bodies: what was ruled out
With the framing settled (leading two bytes = record index), the body is bytes
`2..len`. Four probes on it, mostly negative.
**No count field.** For every chunk type, every `u16` and `u32` position in the
first 64 body bytes, at element strides 1, 2, 3, 4, 6, 8, 12 and 16: does the
stored value equal `(record_len - offset) / stride` on 90% or more of records?
**No type produced a hit.** The record does not carry an element count near its
head.
**The body is dense, not float-dominated.** For the largest type (2,430 records,
326-byte bodies):
f32 reads within +/-5000 of origin : 74%
uniform-random control : 49%
zero-byte fraction : 13%
Above baseline but far short of what the tie blocks look like (53-64% against
16%, with 43% zero bytes). The chunk bodies are denser and less float-shaped than
tie geometry.
**There is an index-like region near the tail.** In the largest type, **12 of 163**
`u16` slots have a maximum below `0x1000` -- too small for float or half-float
data, so those slots look like indices or small counts. They cluster at slots
111-121 and 130-131.
### A sample-size correction
Dumping six records of one chunk showed slots 112-119 **zero on every record** and
a long run of other constant slots, which read as "the record is mostly a
per-chunk template". Measured properly across the 1,171 chunks with at least 8
records:
u16 slots constant across every record in their chunk : 15,422 of 164,390 (9.4%)
chunks whose records are ENTIRELY constant : 0 of 1,171
So 9.4%, and no chunk is a template. Six records looked like a majority because
that chunk's tail happens to carry shared constants -- the per-type figures range
from 9.4% overall to 55.6% on one type. A pooled figure over all types would have
hidden that spread again, so it is reported per type as well.
## What the loader does to a chunk (from RAM)
Verified against PCSX2 level-0 RAM; method and full dump in
`plan/pcsx2-verification.md`.
The chunk payload is copied **verbatim** -- a probe from inside a record hits at
`chunk_base + record_offset` with no rescaling. Two things change around it:
1. **The offset table is rewritten to absolute EE addresses.** Disc offsets
`0xf000057c, 0xf00006b0, 0xf00007e4, 0xf0000914` become
`0xf185387c, 0xf18539b0, 0xf1853ae4, 0xf1853c14` in RAM. The
`0xf0000000 | low24` tag becomes `0xf1850000 | low24` -- tag replaced by the
load base, low 24 bits untouched.
2. **Every record gains a constant 2-byte prefix `02 00`,** and the rewritten
table points at *that* rather than at the record data. Measured on all
**90 of 90** records: prefix `02 00` on every one, and the disc's index bytes
preserved unchanged at `+2`/`+3` on every one.
The third result is the useful one: the byte at `+2` takes exactly
`0, 16, 32, ... 240` -- the 16 values `index_rule` predicts for `step = 16,
phase = 0`. The mixed-radix record index, derived from the disc, is corroborated
on the running game's own copy.
### Correction: the `02 00` is on the disc, and a record starts two bytes earlier
What the loader does to the *pointer* is as described above. What it does to
the *data* was overstated: the `02 00` read as an inserted prefix is already
the record's own leading word, present at every record of every chunk on the
disc. Three measurements, in order:
- **The 2 bytes before the payload are on disc.** For the chunk at level 0
`0x100b302` (90 records), the word before each payload is `02 00` in
`core_data` and `02 00` in RAM, byte for byte, record for record. The first
chunk of the same level carries `08 00` instead. Nothing was inserted.
- **So the RAM table's pointer is simply two bytes earlier.** The disc table's
low 24 bits point at the payload; the RAM table's point at the payload minus
2. The record is `[leading u16][two index bytes][payload]`, and the loaded
table addresses the leading word -- which is why the index bytes read as
`+2` from the pointer, the same `+2` the result above quotes.
- **The leading word is a field, not a prefix.** It is the same at every
record of a chunk on **1,439 of 1,509 chunks** (`None` on the 70 whose
records disagree, which the parser reports rather than averaging), it is
fixed within **every one of the 71 chunk types**, and its values are only
`1, 2, 4, 6, 8, 10` -- 554 chunks carry 2, 520 carry 4. `Chunk::prefix`
carries it.
What the value counts is not established, and two readings were measured and
disproved rather than left as guesses:
- **Not an entry count.** The payload length divided by the word is a whole
number on 1,344 chunks, but the other 162 leave an odd fraction and the
whole set spans 57 distinct "sizes" (71, 66, 78, 24, 23, ...). A real entry
size is a small clean number, not a set like that.
- **Not the number of distinct leading bytes.** That agrees on 0 of 1,506
chunks.
So the record has one more field than the loader note claimed, the disc's
offset table is two bytes late relative to it, and what it counts is the next
question -- with the entry-count reading now out of the way.
## The tie vertex element is 16 bytes
With `TieRecord` identified as R&C1's `TiePacketHeader`, each record's leading
word points at its packet in the block payload. Following it and stepping
`vert_size` elements from `vert_ofs` reaches the vertices.
**Range measured: 95,612 vertices** on levels 0-2.
**Element size, settled by comparison rather than assumed.** Requiring the
texcoord pair to be 12-bit fixed -- both words multiples of 4,096, which is how
0x1000 = 1.0 reads -- gives:
element 16 bytes : 78,919 / 95,612 = 82.5%
element 8 bytes : 45,661 / 95,612 = 47.8%
element 12 bytes : 44,834 / 95,612 = 44.8%
Sixteen wins by a factor of 1.7 over the nearest alternative, so it is the
element size.
**Observed layout**, 8 x `i16`. Scoring each adjacent slot pair for 12-bit
fixedness over 36,501 level-0 vertices locates the fields:
(i16[0], i16[1]) 1.1% not a texcoord
(i16[2], i16[3]) 4.8% not a texcoord
(i16[4], i16[5]) 73.6% <- SCORED as the texcoords; see the retraction below
(i16[6], i16[7]) 76.6% <- the near-constant 4096/0 pair
**`(6,7)` scores higher and is not a texcoord.** It is the near-constant
`4096, 0` pair -- `4096` is 1.0 in 12-bit fixed and `0` is zero, so *both* are
trivially multiples of 4,096. That is the constant artifact again: a fixed pair
would score 100% anywhere. The field is `(4,5)` at 73.6%, with `(0,1)` and `(2,3)`
ruled out at 1.1% and 4.8%.
The first four slots span the full `i16` range with 10,803 to 13,421 distinct
values each, so they are neither texcoords nor small fields.
**The vertex base is `vert_ofs` alone.** Adding the shader or control region to
it makes things *worse*, so the 73.6% residual is not a mislocated base:
vert_ofs only 73.6%
bfc + vert_ofs 69.8%
bfc + control_count*control_size + vert_ofs 66.7%
**Not established:** what any of the eight fields *means*, and why the texcoord
rate is 82.5% rather than 100%. The residual varies with record shape -- records
with `(shaders 2, control 7, control_size 2)` hit 60%, `(shaders 1, control 18,
control_size 5)` hit 89% -- so it is not one bad block but something that varies
per packet, possibly a second texcoord encoding for records that need it.
Recorded because the 16-byte answer is the *opposite* of what the earlier
commit concluded from the disc alone. Reading the header word at `+0x00` as an
element count gave 0 of 922 blocks dividing; it is the byte offset to the payload,
so the table it bounds has 16-byte records. The number was right and the reading
was wrong, which is the error the packet-header identification corrected.
## Collision payload: what the verbatim mapping settles, and what it does not
With the payload confirmed verbatim in RAM, the disc bytes can be read directly.
The grid dimensions are the **first two `u16`** of the payload -- level 0 reads
**11 x 33 = 363 cells**, level 1 **24 x 16 = 384**, level 2 **1 x 53 = 53**.
Level 2 being one cell wide is worth noting; the grids are not all similar in
shape.
**Disproved this round -- an ascending `u32` offset table.** A probe from payload
`+1024` reads `f4ae0000 40af0000 c4b2000020b20000 ...`, which is three ascending
values before the fourth goes *down* (`0xb2c4` to `0xb220`). Read as a table that
was a three-value impression. Scanning all three levels for maximal ascending
`u32` runs of 8 or more entries finds **zero**, on every level.
**Disproved this round -- a global alternating-zero stride.** The head of level 0
shows an obvious `(value, 0)` pattern from about `i16[37]` onward:
[32..] [1808, 1872, 2198, 0, 34, 86, 25848, 1, 25824, 1,
-13028, 0, -12284, 0, -12228, 0, -12716, 0, ...]
Measured over the whole payload rather than the first 96 values, both alignments
score far below what that sample suggests:
even-slot aligned: second half zero on 18% of pairs
odd-slot aligned : second half zero on 13% of pairs
The sample again. What the global numbers *do* say is that the payload is heavily
zero-bearing -- 18% against a chance rate of about 0.0015% -- which is what a
sparse or variable-length encoding looks like, and is consistent with the
variable-length reading established earlier. The 18-against-13 split is a real
alignment preference but a weak one.
**Still not decoded.** The cell layout. The useful position now is that the bytes
can be read as they are, and that they are sparse and variable-length rather than
fixed-stride -- so the next attempt should look for a terminator-delimited walk
through the sparse region, not for a record size.
## Collision: the segmentation is not recoverable from zero structure
Continuing on the verbatim payload, with the grid known.
**Cells are not zero-separated.** Counting maximal runs of non-zero `u16` across
the whole payload:
level 0 363 cells 15,115 runs mean run 25.3 u16
level 1 384 cells 24,647 runs mean run 51.5 u16
level 2 53 cells 31,512 runs mean run 30.3 u16
Two to three orders of magnitude apart, so a zero run is not a cell boundary.
Cells are not uniform blocks either: level 0 averages 2,488 bytes per cell and
level 2 averages 43,684, which cannot both be one fixed-size record.
**The run lengths are not records.** Their histogram has a long tail of
near-power-of-two values -- 2,140, 1,184, 840, 816, 628, 604, 562 -- which is
what you get when a run is bounded by a single zero *byte* inside otherwise dense
data, not by a delimiter. Combined with the 18%-zeros finding, the payload reads
as dense data with sparse holes rather than as a delimited sequence.
**So: not ascending `u32` tables, not zero-run delimiters, not fixed stride.**
All three segmentation theories are now excluded on the full population rather
than on a sample.
### What would actually unblock this
`overlay.bin` **is the game's own code** -- it is the level's `.lit` section,
loaded at `0x1A7C80` -- and the collision debug string
`Camera_CollPrimTest WARNING! - grid out of bounds! (RAR)` is in its string pool.
So the parser is reachable. What is *not* established is the reference to it: a
`$gp`-relative search for the string returns **zero** instruction-aligned hits,
so the code addresses that string by some other means. The format is therefore
recoverable, but not by measuring bytes: it needs disassembly of that routine or
a reference for R&C2's collision cell record.
Stating this plainly is worth more than another scan. The disc-side measurement
space for collision is now genuinely exhausted -- the bytes are available, they
are verbatim, the grid is known, and three independent segmentation theories have
failed on the whole population. Further probing without the loader's own logic is
not going to converge.
## Tie class header: `+0x0c` is the three LOD record counts (2026-10-11)
**Range measured:** every tie class on the disc -- 1,864 classes, all 27 levels.
**Method:** read the word at class `+0x0c` as four bytes, and read the record
counts independently from the blocks themselves: follow `packets[lod]` to the
block, walk to the `0xf6` terminator, and count the 16-byte records the block
header's `data_offset` names. Two paths to the same number, so the check is not
circular.
**Result: the byte at `+0x0c` is the LOD 0 record count, `+0x0d` is LOD 1's and
`+0x0e` is LOD 2's, on 1,864 of 1,864 classes.** On every level the three
counts agree, including the classes where a LOD is absent -- a zero
`packets[lod]` gives a zero count, matching the byte.
This closes the file's oldest lie about the tie header: the comment in
`TieClassHeader::parse` said "0x0c: the 64 light-slot normals offset. Read past,
not decoded", inherited from R&C1's layout where the word *is* an offset. Here
it is not an offset at all -- it is three counts -- and R&C1 carries the same
quadruple at `+0x20`. The earlier note was the right action on the wrong
information: skipping an unknown field is correct, and the field was only
unknown because R&C1's map was applied to R&C2's bytes.
`TieClassHeader::packet_counts` carries the three, and
`tie_class_header_counts_records_per_lod` asserts the agreement.
**The fourth byte, `+0x0f`, is unidentified.** It ranges 1..15 across the disc.
The obvious reading, following R&C1's `packet_count[3]` being `texture_count`,
is a material count, and it is **not** the shader total: the sum of
`shader_count` over a class's records disagrees on 1,827 of 1,864 classes.
Left unnamed rather than guessed at -- a count that matches on 2% of the
population is not a count.
## Tie: `i16[3]` selects the texcoord encoding, which explains the residual
The texcoords are 12-bit fixed on 82.5% of vertices, not 100%. Grouped by the
vertex's `i16[3]` slot across **95,612 vertices on levels 0-2**, the buckets
separate completely:
i16[3] n % / 4096 % / 256
negative 333 1.2% 1.2%
== 0 5,851 29.2% 60.7%
1..63 28,884 92.7% 94.3%
64..255 54,606 92.3% 93.8%
256..511 8 0.0% 0.0%
512..1023 5,391 0.0% 0.0%
>= 1024 539 0.2% 0.2%
**The residual is fully accounted for.** 87.3% of vertices sit in `1..255` at
about 92.5% fixedness, which contributes 0.873 x 0.925 = 80.8 points; the 5,851
at `i16[3] == 0` contribute 1.8 more. That is the 82.5% measured, to the point.
The remaining ~6% are **not 12-bit fixed at all**:
- `i16[3] >= 256`, and negatives, score **0%** at both 4,096 and 256 -- 6,271
vertices using an encoding this reading does not see;
- `i16[3] == 0` scores 29.2% at 4,096 but **60.7% at 256**, which is a third
distribution rather than a weaker version of the same one.
**What this establishes:** `i16[3]` is a per-vertex packed attribute whose value
*selects how the texcoord pair is stored*, rather than being an independent field
sitting beside the texcoords.
**Refined: it is the HIGH BYTE that selects.** Grouped by the high byte across the
same 95,612 vertices on levels 0-2:
i16[3] high byte n % / 16 % / 256
0x00 89,341 92.0% 91.8%
0x03 5,344 0.0% 0.0%
0x04 71 0.0% 0.0%
0x40 58 5.2% 0.0%
0x02 47 2.1% 2.1%
0x19, 0x05, 0x20, ... the remaining 73
A high byte of **zero** is the ordinary case -- 89,341 of 95,612, or **93.4%** --
and it scores 92%. **Every** other high byte scores at or near zero. The split is
on the byte, not on the value: `i16[3]` values 1 and 771 differ only in the high
byte and behave completely differently.
Within the all-zero case there is a third distribution: `i16[3] == 0` exactly
(5,851 vertices) scores 62% at 16 and 61% at 256, against 92% for `1..255`.
**So three cases, selected by the high byte and then the low:**
high 0x00, low 0x00 5,851 a third distribution, 62% at 1/16
high 0x00, low 1..255 83,490 the ordinary case, 92% at 1/4096
high non-zero 6,271 0%, an encoding this reading does not see
### The element layout differs between those cases
Scoring all four slot pairs within the non-zero high-byte group, **no pair
stands out** -- the best is `(6,7)` at 7.5% for a divisor of 4, against a 2-5%
baseline:
i16[3] high byte NON-ZERO (n = 6,271)
pair /1 /4 /16 /256
( 0, 1) 100.0% 4.6% 3.5% 0.7%
( 2, 3) 100.0% 2.0% 0.6% 0.0%
( 4, 5) 100.0% 1.9% 0.3% 0.1%
( 6, 7) 100.0% 7.5% 5.7% 4.5%
So for those 6,271 vertices the 16 bytes are **not laid out as
position/texcoord/normal** at all -- none of the four pairs carries fixed-point
coordinates. Whatever they are, it is a different element shape, selected by the
high byte of `i16[3]`.
This also settles why the global score looked ambiguous: pooling the two layouts
averages a 92% pair and a 7% pair into one number that describes neither.
**Not established:** what any of those three cases means. Whether `i16[3]` is a
texture index, a normal index, a flag or a combination is still open -- but it is
now a field whose *role* is known, which is more than "packed word".
Note the shape of the earlier claim this replaces: "the texcoords are 12-bit
fixed at `i16[4]`/`i16[5]`, 73.6% of the time, unexplained". The field position
was right; the 73.6% was a population average over three encodings, which is
exactly the pooled-number problem rule 3a describes -- partitioning by the one
variable that separates them turns a mystery into a lookup.
### Which of these findings are asserted
Not every claim here has a test behind it, and the recent ones are the ones most
likely to be lost. Audit against `crates/gc-extract/tests/disc.rs`:
claim asserted in
------------------------------------------- -----------------------------------
tie vertex element is 16 bytes tie_vertex_element_is_16_bytes_
with_fixed_texcoords
layout selected by i16[3] HIGH byte tie_vertex_layout_is_selected_by_
i16_3_high_byte
blocks self-delimiting on 0xf6 tie_classes_resolve_through_both_
indirections (per-block assert)
packets == 0 means no block at that LOD same test
packets[0] == 256 on 1,864 of 1,864 same test
chunk record length uniform per chunk tie-classes test + chunk_records_
start_16_byte_aligned
moby class IDs in one body slot per type RETRACTED -- see above; the test
now asserts 2 types, 1 strong
collision payload essentially verbatim collision_payload_is_verbatim_
in_ram (GC_SSTATE-gated)
tfrag pointers relocate, patches rewrite tfrag_table_pointers_relocate_
and_every_patch_is_rewritten
(GC_SSTATE-gated)
The last two used to be marked "not assertable here -- needs a savestate". They
are asserted now, in the same file, gated on `GC_SSTATE` so a checkout without
PCSX2 skips them. Writing them as tests is what corrected two claims: the
collision payload is 903,212 of 903,248 bytes identical rather than wholly
verbatim, and the tfrag relocation base is the *table* address `0x848040`, not
the block base.
The split is deliberate: anything readable from the disc is asserted against the
disc; anything that needs a running PCSX2 is asserted against a savestate when
one is available, and documented in the plan file with its method when it is not.
What is *not* acceptable is a claim living only in prose -- which is how the tie
`i16[3]` finding spent two commits before it had a test, and how the collision
"verbatim" claim survived long enough to be wrong.
## An external decompilation project is the same disc, and names the overlay
`OpenRAC/rac2-gc-decomp` ("Going Commando USA v1.01: byte-exact reconstruction
tools and native-port research. No game assets.") is byte-for-byte the disc this
parser was built against:
ours 3,828,350,976 bytes
9db2e33e276133cc283647fa3279b37911955e123d6199d10065547eaa9b1ce5
theirs 3,828,350,976 bytes
9db2e33e276133cc283647fa3279b37911955e123d6199d10065547eaa9b1ce5
So its findings are usable directly rather than as an analogy. Two of them name
things this file had measured but not explained:
| what | value | source | status here |
| --- | --- | --- | --- |
| `.lit` load base | `0x1A7C80`, same on all 27 levels | `docs/LEVELS.md` | **newly named** |
| `$gp` | `0x1AEFF0`, same on all 27 levels | `docs/LEVELS.md` | newly available |
The measured constant `overlay.bin +0x00 = 1,735,808` **is** `0x1A7C80`. The
reason it is identical across all 27 levels is now known rather than merely
observed: every level's literal section loads at the same address, because
loading a level copies its program over the boot's memory from there. So the
first header word is not a size and not a checksum — it is the **load base of
the section this file is**.
That also settles what the file is. `overlay.bin` is the level's `.lit` section,
which is why it holds both code and the debug string pool.
The program around it has seven sections, per `docs/LEVELS.md`:
.lit .bss .data vtbl camvtbl sndvtbl .text
which is exactly the "seven-section parser and relocation" already listed as a
Phase 3 item for this format. It is now a known target rather than a guess.
### `+0x0c` is not the `.lit` size
The obvious next guess -- now that `+0x00` is known to be the `.lit` load base,
`+0x0c` might be the `.lit` size -- is wrong, and measurably so:
| level | file size | `+0x0c` | excess |
| --- | ---: | ---: | ---: |
| level0 | 2,751,268 | 2,767,752 | +16,484 |
| level11 | 3,151,128 | 2,870,792 | **-280,336** |
| level2 | 2,928,424 | 2,780,808 | **-147,616** |
| level24 | 2,615,776 | 2,763,912 | +148,136 |
`+0x0c` is **smaller than the file** on 16 of 27 levels, by up to 8.9%. It
cannot be a size of the section it sits at the front of. It is still unnamed. The point here is only that the size hypothesis is excluded
by sign, not by a near miss.
### Collision: the code that asserts is now locatable
The collision blocker was "needs the game's own code". The decomp names the
functions (`Camera_CollPrimTest(void *prim, void *grid)`, `MB_CheckCollPill`)
but does **not** decompile them, so the cell layout is still not handed over.
What it does supply is `$gp`, and that is enough to start.
`Camera_CollPrimTest WARNING! - grid out of bounds! (RAR)` sits at
`overlay.bin+0xd18`, the only copy in the file, so:
string overlay+0xd18 -> RAM 0x1a8998
$gp 0x1aeff0
delta -0x6658 ($lo half 0x99a8)
MIPS materialises a GP-relative address as a `%hi`/`%lo` pair, and the `$lo` half
is a 16-bit immediate in the instruction stream. So the anchor search is well
defined -- and it returns **nothing**:
$lo half 0x99a8 (low byte 0xa8, high byte 0x99)
occurrences 0, instruction-aligned, in 2,751,268 bytes
**An earlier version of this file claimed exactly one hit, at `0x354cc8`, and
called it an unambiguous anchor. That was wrong, and the mistake was worth
recording.** The search compared word bytes 1..3 against the immediate. The PS2's
MIPS is little-endian, so an I-type immediate is in bits 15..0, which is word
bytes **0** and **1** -- bytes 1..3 are the opcode, `$rs` and `$rt` fields. The
match was on those, not on an address.
The instruction at the phantom hit is `0x24a89948`:
addiu $t0, $a1, -0x66b8
whose real immediate is `0x9948`, ten bytes away from the `0x99a8` being
searched for, and whose `-0x66b8` merely looks like a plausible `$gp` offset.
With the byte order corrected, the count is zero.
So the collision anchor is **not established**, and the assertion that it was has
been withdrawn. What survives is weaker and still useful:
* `Camera_CollPrimTest` and `MB_CheckCollPill` are named, with
`Camera_CollPrimTest(void *prim, void *grid)`.
* `$gp = 0x1AEFF0`, and `.lit` loads at `0x1A7C80`.
* The assert string is at `overlay+0xd18`, a unique copy.
The likely reason the search comes back empty is that the code reaching that
string does not go through `$gp` -- or reaches a copy in another section. The
decomp's `assert-message-names.tsv` gives `0x002db350` for level 0, which is
inside `.lit` but disassembles to a function epilogue (`jr $ra` /
`addiu $sp, $sp, 0x70`), so those are code locations, not string addresses, and
they are not yet traced to the assert either.
`tools/overlay-anchors` does the anchor search and is dependency-free. Its
correctly-ordered result for the collision string is zero, and it now carries a
comment explaining the byte order, since that is precisely the part that was
wrong:
cargo run -q -p gc-overlay-anchors -- \
/home/reaper/Documents/gc-extracted/level0/overlay.bin \
"Camera_CollPrimTest WARNING!"
Next step, if collision is still worth it: find how the assert string is actually
addressed -- a `%hi`/`%lo` split that does not straddle zero, a pointer table, or
simply the right section -- before assuming a `$gp` relative load.
### What this project did *not* give up
Surveyed against the open items: no volume structures, no `TieTex`/tie vertex
layouts, no `SBlk`/`DAW` record fields, no wavelet tables. Collision has the
API but not the data. It is a guide to the code, not a replacement for reading
it.
## The tie vertex's selector byte, and what slot 3 is not
`i16[3]` is called the layout selector because its **high** byte partitions the
elements. Splitting the 95,612 elements of levels 0-2 by that byte:
| high byte | elements | share |
| --- | ---: | ---: |
| `0x00` | 89,341 | 93.4% |
| `0x03` | 5,344 | 5.6% |
| the other 144 values | 927 | 1.0% |
146 distinct selector values in all -- not a two-way switch but a long tail
behind one dominant case, and the second case is not `0x01`.
Inside the ordinary layout, `i16[3]` is **not a coordinate**:
distinct values 201
range 0 ..= 254
201 distinct values inside a byte's range, with 0 the most common, is an index.
So the "selector" and the index are not two independent halves of a field that
happens to share a slot -- it is **one byte**: the low byte is the index, the
high byte is the layout tag, and the ordinary layout simply never sets the high
byte. Asserted in `tie_vertex_element_is_16_bytes_with_fixed_texcoords`.
This was nearly measured wrong. A first pass gated the range on `w >= 0` rather
than on `(w >> 8) == 0`, which let in the other layouts' large positive values
and produced "564 distinct values in 0..=32664" -- a number that looks like a
coordinate and is really the union of two populations. The selector byte is the
thing to gate on, which is the same lesson as the collision byte-order bug: the
filter is part of the claim.
The remaining slots, all in the ordinary layout:
| slot | distinct | range | reads as |
| --- | ---: | --- | --- |
| `i16[0..2]` | 10.7k-13.2k each | full `i16` | position, unproven |
| `i16[3]` | 201 | `0..=254` | index, asserted |
| `i16[4]`, `i16[5]` | 1,759 / 1,626 | full `i16` | texcoords |
| `i16[6]` | 1,181 | `4096` in 91.8% | unnamed |
| `i16[7]` | 226 | `0` in 88.9% | unnamed |
`i16[6]`/`i16[7]` are **not** a fixed-point pair, and the "both are multiples of
4096" score is an artifact worth naming: 86.5% of elements have both divisible by
4096, against 84.3% expected if the two residues were independent -- an
enrichment of **1.0x**, i.e. exactly chance. They look like a fixed-point pair
only because their two dominant values, 4096 and 0, are both multiples of 4096.
Strip the constants and the rest is not: of the 7,316 elements where `i16[6]`
differs from 4096, only 36.7% are still multiples, and of the 9,905 where
`i16[7]` differs from 0, **one** is. The disc test already refused to call it a
texcoord for this reason; that caution is now quantified.
## `i16[0..2]` are the vertex positions
The three slots ahead of the index were "unproven position" for want of a test
that could fail. There is one available on the disc alone, with a built-in
control.
**LOD 1 and LOD 2 of a tie class are the same object at two levels of detail.**
Anything that locates a vertex in that object's own geometry is therefore shared
between the two blocks, while a *different* class' LOD is an unrelated object.
So: intersect the sets of `i16[0..2]` triples, same class versus different
class, and if positions are what is in those slots the first number should be
large and the second should be at zero.
Levels 0-5, ordinary-layout elements only:
same class, LOD1 vs LOD2 96.1% over 107 classes, 107 of them nonzero
different class, control 0.4% over 2,144 pairs
A factor of 240. Per level the same-class figure runs 94.4% (level 0), 97.9%
(1), 98.1% (2), 96.7% (3), 96.8% (4), and the control never exceeds 1.0% except
for one level-1 pair at 45.9% -- two classes in that level sharing most of their
geometry, which is what duplicate or variant models should look like, and is
reported rather than excluded.
Asserted in `tie_vertex_slots_0_to_2_are_shared_positions`, including the
separation ratio (`same > 20 * diff`) so a change that collapsed both numbers
would fail rather than pass twice.
What this rules out is the important part. An index, a handle, or any per-vertex
counter would not be shared at 96% between two independently built LOD meshes, and
would not sit at 0.4% between two objects. Combined with 10.7k-13.2k distinct
values per slot across the full `i16` range, `i16[0..2]` are a position in the
object's own frame. The *scale* and the axis order remain unnamed -- nothing here
says whether these are centimetres, and the axis order is unconstrained because
nothing here assumes one.
The element now reads:
+0x00 i16[0] position x (unproven axis order and scale)
+0x02 i16[1] position y
+0x04 i16[2] position z
+0x06 u8 index, 0..=254
+0x07 u8 layout tag, 0 in 93.4% of elements
+0x08 i16[4] texcoord u
+0x0a i16[5] texcoord v
+0x0c i16[6] unnamed
+0x0e i16[7] unnamed
One caveat worth keeping visible: a test that prints a fraction with `{:.1}%`
and forgets the `* 100` reports "1.0%" for 99.4%. That happened here, in the
diagnostic line only -- the assertions compare fractions and were always right --
but it is the same family of error as everything else in this log: the readout
disagreeing with the claim.
## Tie slots 3, 6 and 7: what they are not
Having established that `i16[0..2]` are positions, the remaining slots were
attacked with two more measurements rather than by inspection.
### Does slot 3 index a skeleton?
`i16[3]` holds 201 distinct values in `0..=254`, and **50 distinct values within a
single draw packet** (median, 1,804 records). A material or texture-set index
would be constant per packet, so it is per vertex -- which invites the obvious
reading: tie objects are skinned, and slot 3 is a joint index.
> **CORRECTED.** An earlier version of this concluded "slot 3 is not a skin
> index". The measurement does not support that, and the inference from it was
> wrong. The correction is the reasoning, not the numbers.
mean pairwise distance, same slot-3 value 14,481
mean pairwise distance, different slot-3 value 13,795
ratio different/same 0.95x
control: same position-x bucket 8,105
The numbers are fine: vertices sharing a slot-3 value are about as far apart as
vertices differing in it, and the control shows the measurement works -- bucket
by position-x and pairs collapse to 8,105, half the global distance.
What does not follow is "so it is not a joint index". The premise was that
vertices sharing a joint sit together. That is only true for a *tightly
localised* joint. It is false in the cases that are actually common:
* a rigid tie object -- a leaf, a banner, a piece of foliage -- puts **every**
vertex on joint 0, so "vertices sharing a joint" is every vertex, and the mean
distance is the mesh's own diameter;
* a root or hip bone's influence region can span most of a mesh;
* a bone's weights fall off gradually, so membership is not a spatial partition.
A 0.95x ratio is what a rigid object bound to one joint **predicts**. Absence of
clustering is therefore fully consistent with the joint reading, and the test
never had the power to separate them.
That makes the hypothesis stronger rather than weaker, for a separate reason:
`MobyInstance` carries `jointCnt` at `0x57` and `jointCache` at `0x58`, so this
engine has joints and per-moby joint state. A per-vertex field in `0..=254` is
exactly the shape of a joint index.
What is still true, and is all that was ever established: slot 3 is **per vertex**
(50 distinct values per packet), in `0..=254`, and **not shared across LODs**
(75.3% same-class against a 78.7% control). A joint index shared across a class's
two LODs would be expected to share, which is the one thing arguing against it --
though two LODs may also bind a rigid mesh identically without that being
informative.
So: **not understood**, with the joint reading now the leading candidate rather
than a disproved one.
### Slots 6 and 7 are mostly not per-vertex
Distinct values per draw packet, median over 1,804 records:
| slot | median | constant for the whole packet |
| --- | ---: | ---: |
| `i16[0..2]` | 13-17 | 0% |
| `i16[4]`, `i16[5]` | 6, 4 | 0% |
| `i16[6]` | 2 | **43%** |
| `i16[7]` | 4 | **34%** |
`i16[6]` and `i16[7]` take a single value across the entire packet in well over a
third of records, and two or three values in the median. Whatever they are, they
are usually a property of the packet rather than of the vertex -- consistent with
the 91.8%/88.9% dominance of 4096 and 0 respectively. They are left unnamed.
### A note on why the per-slot LOD test does not generalise
Run across all eight slots, the LOD1/LOD2 overlap test discriminates only on
`i16[0..2]`:
| slot | same class | control | ratio |
| --- | ---: | ---: | ---: |
| `i16[0]` | 97.0% | 3.0% | 32x |
| `i16[1]` | 96.7% | 3.2% | 30x |
| `i16[2]` | 96.4% | 6.0% | 16x |
| `i16[3]` | 75.3% | 78.7% | 1x |
| `i16[4..7]` | 68-91% | 52-79% | 1x |
The other slots come from small value sets -- 226 to 1,759 distinct values across
a level -- so two arbitrary sets of that size overlap heavily whether or not the
field means anything. The test's power came from `i16[0..2]` being the only slots
with tens of thousands of distinct values. **A passing result from a method that
has run out of discriminative power is not a passing result**, and the 1x rows are
recorded so the table is not later mistaken for nine confirmations.
## The `SBlk` head: four constants, one unidentified word
The sound extent's 24-byte head was parsed but not characterised. Across all 27
levels the `SBlk` region opens identically for its first twelve bytes:
+0x00 "SBlk"
+0x04 1
+0x08 4
+0x0c "DAW\0"
+0x10 0
+0x14 varies
`+0x14` is the only part that differs between levels, and it differs with its
**low sixteen bits zero on all 27**, leaving `0x0092..=0x0103` -- 146 to 259.
That is the shape of a record count, and it tracks the `SBlk` body length at
close to 100 bytes per unit. It is not one:
body / field_14 91.2 .. 110.8 across the 27 levels
any stride 0..80 making body = stride * word + constant exactly: no
A near-miss at 100 bytes/record is exactly the kind of thing that hardens into a
name the next time somebody glances at it, so the test asserts the negative
alongside the positive: if a stride ever makes the relation exact on all 27, the
test fails and asks for the record size to be revisited.
Asserted in `sound_bank_sblk_head_is_constant_apart_from_one_word`, which checks
the four constants individually so a real deviation names which moved, the zero
low half, the value range, and the absence of a fixed stride.
The `DAW` region begins with sixteen zero bytes and then block-structured data
with 16-byte periodicity and slowly varying nibbles, consistent with block
compressed audio. It is not decoded.
Two notes on how this was measured. Reading the level 0 head directly from a
scratch script gave a field order that the parser disagreed with, because the
constructor bound `format_tag` with a `let` *before* the two words that precede
it in the file -- consuming bytes 4..8 and shifting everything after. The struct
literal named the fields in the right order; the statements did not. It is fixed
with a comment saying so, since "read the `let`s in file order" is the kind of
thing that gets undone.
## Volumes: a candidate table, and why it is not yet called one
Volumes were the only asset type never located at all, so this is a search
report rather than a result.
**Ruled out first: there is no engine-side name.** No level file contains the
string `Volume` outside UI text ("Music volume", "Volume de la musique") and
Italian item descriptions. So the format cannot be identified by a label and
must be found structurally.
`gameplay.bin` on disc is the already-decompressed level gameplay data. Its head
is a count (`336` on level 0) and then 16-byte records from byte `416`. Those
records carry `(type, pad, offset, id)`; the `id` field groups them, each id
having exactly two records pointing into two parallel offset arrays. The offsets
they point at resolve into a text pool -- the first ones land mid-string in
dialogue ("it Megacorp Lab"), so this is the dialogue and class-dispatch side of
the file, not geometry.
The structural hit is a table of **144-byte float records**. Level 0 has one at
`799204`:
20 records of 144 bytes, 19 populated and one all-zero terminator
34-35 of every record's 36 float slots are world-plausible
+0x0c holds a world position, varying per record:
[0] (206.5, 284.1, 116.0) [6] (244.0, 407.2, 128.0)
[1] (219.3, 247.0, 107.5) [7] (265.3, 443.7, 13548.6)
[2] (248.2, 198.4, 51.8) [8] (185.1, 282.6, 104.9)
[3] (247.2, 203.6, 82.4) [9] (185.8, 282.7, 102.6)
[4] (155.7, 372.7, 121.1) [10] (260.8, 423.6, 130.9)
[5] (265.0, 445.3, 131.7) [11] (247.0, 211.5, 100.6)
A 144-byte record with world-space positions and float radii is exactly the shape
a volume trigger region should have, and a matching table exists in every level
examined (levels 0-9), each preceded by an all-zero record, with counts from 23
to 612.
**It is not called volumes here.** Nothing yet ties this table to volumes
specifically. It could equally be camera bounds, spawn points, or path nodes --
all of which would present 144-byte float records with world positions and no
label. The distinguishing evidence has not been gathered, and the honest state
is *candidate*.
### Two detector bugs worth recording
Both were mine, and both produced confident nonsense before being caught.
1. The first validity test accepted zero as a world-plausible float. Every
zero-filled region then scores perfectly, and the cross-level check returned
runs of 225 to 1,514 records -- obviously padding. Requiring **non-zero**
floats (>=20 of 36) immediately separated real tables from memory.
2. The stride claim rested on four eyeballed offsets 144 apart. Gap 144 was not
in the top eight gaps of the detector that found them; it survived because
three consecutive observations agreed, not because the measurement supported
it. The corrected detector then found longer runs, which is the opposite of
what the eyeball version claimed.
The same shape as every other correction in this log: a clean, small, plausible
number read as conclusive because it was not checked against a control that
could have disagreed.
### The reference that did not generalise
Level 0's `data.bin` -- the level's file descriptor -- contains the value
`799204` exactly once, at `0xab6360`, and `799204` is byte-for-byte the offset
of the 20-record candidate table above. A single 4-byte match in 13.3 MB has a
chance expectation of about 0.003, which is suggestive enough to be worth
checking properly rather than assuming.
It does not hold up. Taking each level's own best 144-byte table and searching
that level's own `data.bin`:
level candidate offset records references found (chance ~0.0035)
0 1,810,724 32 0
1 1,884,884 386 0
2 2,066,804 224 0
3 2,438,228 365 0
4 2,677,780 114 0
5 1,255,780 14 0
6 2,496,020 115 0
7 2,062,072 311 0
Not one of the eight is referenced. The level 0 match at `0xab6360` sits in a
neighbourhood of high-entropy bytes, and level 0's own *best* table -- a
different offset, 32 records, not the 20-record one -- is not referenced either.
So the single hit does not generalise and is most likely coincidence.
The consequence is worth stating plainly because it retracts what the previous
section implied: **nothing links a 144-byte float table to volumes.** The table
is reached by walking structure rather than by an explicit pointer, so the
approach that would have settled it does not work, and the candidate stands
exactly where it did.
What would settle it, and none of these are cheap:
* The game's own volume code, which has the same byte-order problem as the
collision anchor did -- a GP-relative string search returning nothing.
* Ground truth: a count of level 0's trigger volumes from a map or a guide,
to match against a table's population.
* A correlation with a *named* class, since `gameplay.bin` resolves ids
through its dispatch records. If an id in the candidate resolved to a class
whose name is a volume trigger, that would be decisive. No class name
containing "volume" exists on the disc, which is why this is not available.
## The overlay's load base, from its own profiler table (2026-10-11)
**Range measured:** all 27 levels' overlays.
**Method:** the overlay carries the game's **profiler menu definition** -- a run
of 8-byte-padded names (`normal`, `camera`, `cam+chr`, `volume`, `bars`,
`render`, `mobys`, `ties`, `shrubs`, `cpu/gpu`, `occl`, ...) followed by runs of
absolute RAM addresses that point **back at those same names**. On every level
the four addresses after `normal`/`camera`/`cam+chr`/`volume`, each minus its
target name's overlay offset, give the same number: **1,735,792**.
That number is `field_00 - 16`, where `field_00` is the header word that
previously read 1,735,808 on all 27 levels with no meaning attached. So the
overlay is loaded with its 16-byte header stripped: overlay offset `X` sits at
RAM `X + 1,735,792`. `OverlayHeader::ram_address` carries it.
**Confirmed, not only derived.** The same bases were checked in PCSX2's RAM:
level 0's `0x1acf80` holds `"normal\0camera\0cam+chr\0volume\0"`, byte-identical
to overlay `0x5310`, and `0x5310 + 1735792 == 0x1acf80`. All four names land
where the overlay's own table says. `gc-trace verify-overlay-base` runs the
check.
**The overlay is relocated, not copied.** Only **37.67%** of level 0's overlay
is byte-identical at the predicted base, and the 256-entry sine table -- at
overlay `0xb650c` -- is *not* at `0xb650c + base`. So the base locates the
overlay's image, but individual tables inside it move. That is the shape of a
loader applying relocations, and it means the base is necessary but not
sufficient for reading the game's tables.
The value of the base is that any overlay offset can now be carried to a RAM
address, which is the prerequisite for reading the game's live level state --
the thing the tie relocation and the tfrag lighting checks had to work around.
Note the profiler table's own vocabulary, since it is the first named structure
in the overlay: `volume` is one of the game's profiling categories, so volumes
are a real subsystem. It does not locate the data.
## Chunk records: what the stride table settles, and one signature it does not
Walked every chunk on all 27 levels and grouped by its six-byte header signature:
1,506 chunks, 34,415 records, 74 distinct signatures
55 distinct strides, from 52 to 640
every one of those 55 strides is a multiple of 4
73 of 74 signatures use exactly ONE stride
**The single exception**, asserted rather than skipped so that "every type is
uniform" cannot quietly stop being true:
signature [75, 42, ef, f2, 71, b5] strides {76, 92, 104}
This is the variable-length type the module documentation already referred to
indirectly. It is now named by its signature and pinned to those three strides
exactly. Everything else is a fixed-stride array, which is the structural fact
that makes per-type field decoding tractable at all: a record's field offsets are
the same for every record of its type.
The 4-byte alignment is a real result rather than an artefact of how the stride
was inferred. A stride recovered from one chunk's first-to-second record delta
could inherit alignment from where that chunk happened to sit; collecting all 55
distinct strides across every type and requiring all of them to be divisible by
4 rules that out, because an off-by-one in one signature would show up.
### A record layout that did not generalise
One type looked like a clean find. Signature `[53, c4, 6c, d7, 9a, 43]`, stride
108, 810 records, 63 of 108 byte positions constant:
+0x00 .. +0x30 48 bytes varying per record
+0x30 .. +0x6c 60 bytes that are ZERO in every record
+0x68 0x00010000, constant
A fixed data block followed by a permanently empty region reads like an inline
buffer that this type never uses, and that would have been a tidy field story.
It does not generalise. Requiring a contiguous always-zero run and checking
`stride == live span + zero run` across every type with 50+ records per chunk
gave **0 of 9**. The zero tail belongs to that one type and means nothing
general. Recorded because "48 bytes then 60 dead bytes" is specific enough to
look like a layout and would otherwise be rediscovered and over-read.
Neither of these decodes any record's *fields*. What is now settled is the
shape the decoder has to handle: uniform stride per type, 4-byte aligned, with
one named exception carrying three strides.
## Tie: the "vertex positions" claim does not survive a stride test (2026-10-11)
**Range measured:** level 0, one class, one 58-quadword packet region.
The claim from earlier reads, `i16[0..2]` of a 16-byte element are the vertex
positions, and 95,612 vertices were "measured" on that reading. A raw dump of
one packet kills it:
0000df20 00900000 075006c0 094000c0
1f736c63 00544848 00281506 0d40001b
0d400a90 ffff00ff 00010ef0 80004253
4846a9e3 48a44001 47f40000 46d0b0de
0000ee50 00c00000 08a007e0 099000f0 <- group repeats at +64
0f807e72 01484848 00321208 0cf00017
0cf00ab0 ffff00ff 00020e60 80002e46
4858389a 48a44001 47f40000 46d7bbe3
**The first 16 bytes are not a vertex at all** -- they are the packet's own
header material: `0x0000df20` is not a coordinate and the following words are
the vif/material words. The 16-byte group at +0 recurs at +64 and +128 with
identical structure (a small i16, then 0, 0, then an index) and its three
instances carry the values -8416, -4528, -784. So the stride is **not** 16
throughout: some elements are 4 bytes, some 16, and the run length varies.
**The period-4 marker does not generalise either.** Grouping every fourth
16-byte element by the high byte of `i16[3]` gives marker 0 (n 9,204), 0x48
(n 4,140), 0xff (n 4,912) and 0x01 (n 1,137), and only 0x48 shows a
plausible position signature -- `i16[1]` extent 425 against `i16[0]` and
`i16[2]` at 65,535, which is the height axis of a flat object. But the same
marker 0 bucket has all three slots spanning the full `i16` range, so the
bucket does not select a layout.
**Two candidate vertex bases were measured and neither clusters.** With
`packet + vert_ofs` the per-class extents run median 64,976 and p90 65,517;
with `class + vert_ofs` they run median 55,213 and p90 64,352. Both span the
whole `i16` range, whereas a real position triple in a level-class local frame
must span a fraction of it. `vert_ofs + vert_size + scissor_size` is the
region length, so the region bounds are right -- what is wrong is the element
layout inside it.
**Consequence for earlier claims, stated plainly.** The 95,612-vertex
measurement and the 82.5% texcoord fixedness figure were both computed on the
16-byte reading and are not superseded by this -- they are untested against the
correct stride, which is not yet known. The LOD1/LOD2-sharing test
(96.1% same class against 0.4% different) remains the strongest evidence that
the first slots are positional, and it is insensitive to stride.
## The GS texture register set, and why TBP0 is region-relative
**Range measured:** all 27 levels, **1,595 level textures**.
`gc_formats::gs` gains the TEX0 register a GS emulator needs, decoded from a
texture entry: `tbp0`, `tbw`, `tw`, `th`, `psm`, plus `texture_slot_bytes` and
`clut_base`. Three measurements back the decode.
**Every dimension is a power of two.** 1,595 textures, **zero** exceptions, in
four classes: 128x128 (1,249), 64x64 (303), 256x256 (22), 32x32 (21). The GS
addresses textures by exponent, so TW and TH are `log2` of the dimensions and a
texture that was not a power of two could not be addressed at all. That the
whole population is one is the measurand for the decode.
**TBP0 is region-relative, not data-relative.** 19 of the 27 levels have a
`textures_base` that is only 64-byte aligned -- mod 256 reads 0, 64, 128 or 192
-- while every entry's own `data_offset` is 256-aligned on all 1,595. So a base
pointer computed from `core_data` would not be a whole number of 256-byte blocks
on those levels. The loader uploads the texture region to a GS-aligned base and
the pointer counts from there.
**The slot is level 0 plus one mip level.** The stride between consecutive
entries is `w*h + (w/2)*(h/2)` on 1,560 of the 1,564 ascending gaps: 32x32 gives
1,280, 64x64 gives 5,120, 128x128 gives 20,480, 256x256 gives 81,920. The
game stores the base image and one mip, at one byte per texel. `ty` reads **3 on
all 1,595**, so it is not the stored level count -- naming what it is instead of
calling it a mip count is the point.
Four gaps break the rule, and four more run backwards. They are reported as a
bounded exception rather than folded into an average: the four ascending
outliers are 712,960 to 1,064,960 bytes, two orders of magnitude past any slot
the game allocates, so they are boundaries the slot rule does not describe
rather than slots that came out different. The four reverse gaps mean the table
is not sorted by `data_offset`, which is itself worth knowing -- a consumer that
walks the region by table order would visit it out of sequence.
The CLUT address is the established `palette * 0x100`, restated as
`clut_base` because a GS emulator asks for it as TEXCLUT's CBP.
## The tfrag clip cube is at `+0x38`, and `+0x2e` is the mini-sphere array## The tfrag clip cube is at `+0x38`, and `+0x2e` is the mini-sphere array
**Range measured:** level 0, all 833 tfrag headers.
The parser had a field named `cube_ofs` at header `+0x2e`. R&C1 documents that
slot as `msphere_ofs` and puts the 8-corner clip cube at `+0x38`, and the disc
agrees with R&C1 on both:
* reading `+0x2e` as an array of `msphere_count` (`+0x2c`) `Vec4f` entries
gives **833 of 833** well-formed arrays -- four finite floats per entry, every
radius non-negative. `+0x2c` takes 4, 3, 5, 6 and 16 among others;
* reading it as eight `s16` clip corners gave data that was not corner-shaped,
which is how the mistake was visible in the first place;
* the real cube, at `+0x38`, is **non-zero on 833 of 833** and its eight points
sit inside the patch's own block on all of them.
So `TfragHeader::cube_ofs` at `+0x2e` is now `msphere_ofs`, the count it pairs
with is parsed, and `clip_cube_ofs` at `+0x38` is new.
### What the cube's corners are
Eight points, each read as four `s16` at 8-byte stride and scaled by 64 -- the
scale is R&C1's documented one and is **not** verified against the game. What
is measured:
corners, all with 8 distinct x values 833 of 833
|centroid - bsphere centre| / r median 0.147, max 0.595
corner distance from centroid / r median 1.000
The median is exactly the radius, so the eight points lie on the bounding
sphere. They are **not** an axis-aligned box -- every tfrag has eight distinct x
values, so the box, if it is one, is oriented. Testing whether the eight points
form a cube needs the corner *ordering*, which is not recoverable from the data:
assuming opposite pairs are `(0,7) (1,6) (2,5) (3,4)` gives a body-diagonal
equals `2r` match on only 99 of 833, so either that assumption is wrong or the
array is not a cube. Recorded as unknown rather than guessed at.
### The guard-band clip test, and what it now needs
The algorithm is documented in the vendored R&C1 reference (`tfrag_rac1.md`
§3b.2), attributed to it rather than verified here:
1. `d = R * (bsphere.xyz - camera * 1024)` in camera space;
2. `far = int(d.z + r)`, `near = int(d.z - r)`, truncated;
3. cull when `near > 512000` or `far <= n`, or on the side planes
`tan*z - |x| +- r*sqrt(1 + tan^2)`;
4. "wholly inside" is both side margins non-negative, `far < f`, `near >= n`;
5. otherwise re-test the 8 corners from `clip_cube_ofs`, against the frustum
unscaled and against a volume with xy scaled by `(0.25, 0.25)` -- the 4x
guard band;
6. all corners outside one unscaled plane culls; any corner outside the guard
volume sends the tfrag to the clipping program at LOD 0 with no morphing;
7. otherwise `draw_mode` decides.
`draw_mode` itself already matches the reference's table in this repository
(`gc_formats::tfrag::TfragGeometry::draw_mode`). What step 1 needs and what we
still do not have is the **camera matrix**, and what step 5 needs is the view
frustum planes as a clip volume. Both are per-frame game state, so this needs
the camera at a savestate -- the same blocker the morph verification had before
the PINE session made it measurable. The difference now is that the geometry
side is parsed and tested, so only the camera side is outstanding.
## The tfrag morph is not a rewrite of the position arrays (2026-10-14)## The tfrag morph is not a rewrite of the position arrays (2026-10-14)
**Range measured:** level 0, all 833 tfrag patches, live PINE RAM against the
disc.
The morph was recorded as unverifiable because "reading a weight back out of
RAM does not work -- the positions live inside regenerated VU packets, so there
is no stable per-vertex correspondence". That reading is right about the
correspondence and wrong about the cause, and the cause is now measurable.
**The game rewrites bytes inside the tfrag blocks, and none of them are in a
position.**
patches 833
positions 40,600 (disc == ram)
positions identical 40,600
block bytes differing 166,465
inside a position unpack 0
inside another unpack 49,484
in a VIF command word 116,981
The position ranges total 490,224 bytes and not one of them differs. The two
measurements agree independently: walking the VIF lists and comparing parsed
positions gives 40,600 identical and zero changed, and a raw per-byte diff of
each patch's own extent classifies every differing byte.
So the blend is not precomputed into the geometry. It happens at draw time, in
the VU stream -- which is the standard PS2 arrangement and is why no weight was
ever readable from RAM: there was nothing there to read. The rewrites are
mostly packet framing (70% command words) plus some other unpack payloads
(30%), which is consistent with the loader rebuilding the VIF packet for the
tier it selected while leaving the vertex data alone.
**Extents.** Each patch's extent is the gap to the next patch's `data` pointer,
which is exact and needs no guess about which header field marks a patch's end.
The block is located in RAM by the first table record matched under the
invariant mask, the same way `gc-trace` locates it.
Asserted by `tfrag_positions_are_verbatim_and_the_rewrites_are_elsewhere`,
gated on `GC_SSTATE`. The test also requires the rewrites to be non-trivial
(more than 100,000 bytes) and to be mostly command words, so a RAM image that
happened to be a verbatim copy of the disc -- which would make the zero look
good for the wrong reason -- fails instead.
## Most of the chunk container is level-independent
Comparing every chunk record against the same record index in the other 26
levels, across all 74 signatures:
65 of 74 signatures: every record byte-identical in all 27 levels
9 of 74 signatures: level-specific
0 ragged -- no signature appeared with different record counts per level
The nine that vary:
[5d 44 13] [75 42 ef] [7f 44 ee] [82 c4 2f] [96 c3 37]
[a2 44 a5] [c6 c3 ef] [df c3 ea] [f3 c4 b9]
This reframes the whole container. **65 of 74 chunk types are static tables the
game ships unchanged in every level** -- constants and lookups, not level
content. The largest by record count are all in that group: `[38 c4 7e]` at
stride 328 with 90 records per level, `[a9 c4 d8]` at 308 with 90, `[3d c4 9e]`
at 368 with 77. Each contributes the same bytes to all 27 levels.
The practical consequence is about method. For a level-specific type, comparing
levels is a powerful lever: a field that changes with the level is identifiable
by correlating it with something known. For the other 65 there is nothing to
compare -- there is one copy of the data, repeated 27 times -- so they have to be
decoded structurally, from what the bytes *are*, not from how they vary. Any
future decoder that reaches for per-level comparison as its first tool will find
it useless on the majority of the container.
`[75 42 ef]` appears in the level-specific list, and it is also the one signature
with three strides. A type that is both variable-length and level-varying is the
most informative one in the container, and is where per-type field decoding should
start.
### How this was found, and the near-miss before it
The first look at one signature (`[53 c4 6c d7 9a 43]`, stride 108) showed 45 of
108 byte positions varying *within* a chunk, and 810 records all distinct --
which reads like per-record data. Counting distinct values per `u32` slot across
all 810 records contradicted that immediately: 29 to 30 distinct values across 27
levels, i.e. roughly one per level.
Both readings were wrong, and the resolution is that they measure different
things. The 45 positions vary *between records within a level*. The u32 distinct
counts were low because the same 30 values recur in every level. Comparing record
*i* across levels settled it: **all 30 records are byte-identical in all 27
levels**, and zero byte positions vary across levels.
Had the u32 distinct counts been the only measurement taken, the conclusion would
have been "one value per level" -- which is the opposite of the truth, and would
have described a per-level parameter table that does not exist.
## A parser defect: one chunk type is not fixed-stride
`[75 42 ef f2 71 b5]` -- the signature that is both level-specific and
variable-length -- produces **three chunks per level**, 81 in total. Their record
offsets step like this:
stride 76 deltas [76, 76, 76, 76, 76, 76, 76, 76] uniform
stride 92 deltas [92, 88, 84, 92, 92, 92, 84, 88] NOT uniform
stride 104 deltas [104, 104, 104, 104, 104, 104, ...] uniform
Identical on all 27 levels -- 27 chunks of each shape, three shapes, no others.
So the description "this signature has strides {76, 92, 104}" is a
**misdescription**. It has three chunk shapes, one of which is not fixed-stride
at all. The parser derives one `stride` per chunk and `resolve` slices every
record at that length.
What that costs, precisely:
* Records are read at the **offset table's** positions, so every record *starts*
correctly. There is no misalignment.
* But each is sliced at **92** bytes. Two of the eight are 88 bytes and two are
84, so **four records over-read by 4 and 8 bytes** and carry the head of the
next record inside them.
Bounds checking holds -- the slices are inside the region -- so this is a
correctness bug in the data, not a safety one. It is exactly the kind that
produces plausible-looking fields that are offset by four bytes, which is how it
would have gone unnoticed had the stride table not been collected per signature.
It also explains why this signature was flagged earlier as breaking the 8-byte
alignment rule: its records genuinely are not a uniform multiple.
**This is a defect in the parser, not in the data**, and the right fix is for
`resolve` to use the offset deltas rather than a single stride for a chunk whose
deltas are not uniform. Recorded here first, because the assertion in
`chunk_type_75_ef_has_one_variable_length_chunk_per_level` pins the delta
sequences -- if the data ever does become uniform, the test fails and asks for
the model to be revisited rather than the other way round.
### A near-miss that nearly hid this
The leading `u16` of these records appeared to step by exactly `0x50` (80) for
all nine records of the 76-byte chunk -- a tidy, meaningful-looking pattern that
would have been easy to report as "records are indexed by 80". Reading the
records at their real offsets shows the lead values are `b1a5, a529, 047e,
a529, 04f0, a529, 0562, a529, 05d4` -- no arithmetic progression at all. The
apparent 80-byte step came from comparing the wrong bytes.
## 1,252 chunks have non-uniform record offsets, and the cause is genuinely ambiguous
The previous section fixed on one signature. Measuring every chunk instead shows
the pattern is the norm, not the exception. Taking `delta = offset[i+1] -
offset[i]` and comparing it to the chunk's own first delta:
delta - first delta, levels 0-2, 2,953 deltas
0 1,723 -4 549 +4 426
-8 146 +8 32
-12 71 +12 21
-16 34 +16 4
-20 27 +20 1
-24 13
**Every deviation is a multiple of 4, and most are exactly +/-4.** Worst observed
deviation is 24 bytes, in a 7-record chunk. `1,252 of 1,509` chunks have at least
one such deviation, across 66 of the 74 signatures.
### Why this does not become a fix
There are two models that produce this offset table, and they are
**observationally identical**:
1. Records are genuinely variable-length, padded to a 4-byte boundary, with
`stride` recording the largest.
2. Records are fixed-length at `stride`, and the offset table carries
**inter-record padding** -- so the gap to the next record is the record plus
some padding.
Nothing in the offset table distinguishes them. Both say where records start and
both produce `92, 88, 84, 92, 92, 92, 84, 88` for the case above.
A change to `resolve` to slice each record against its successor implements
model 1. It was written, and it broke three tests -- which is worth noting,
because those three tests asserted "1,509 chunks have uniform record length" and
passed **because the parser forced uniformity by slicing at a single stride**.
They were asserting the parser's behaviour back at itself.
It was reverted. Slicing to the successor is only correct under model 1, and the
evidence does not establish model 1. Shipping it would have replaced one
unverified model with another, and would have silently changed 1,252 chunks'
record boundaries on the strength of a guess.
**What would settle it:** a field inside the record whose end is independently
observable -- a terminator, a count that implies a length, or a field whose
value is known and whose position therefore pins the end. Under model 2 those
bytes are padding and will be uniform or zero; under model 1 they are data and
will not be. For `[75 42 ef]` the 8 bytes at issue begin `a0 00 27 20` in both the
tail of one slice and the head of the next -- which is arithmetic, not evidence,
since the two views are of the same bytes.
The uniform model stays in place, the ambiguity is recorded here and in
[`resolve`]'s docs, and the existing 1,509-of-1,509 assertion stands as a
statement about what the parser does, not about what the disc contains.
## Audit: which claims were contaminated by the over-read?
The record-boundary fix changed the length of records in 1,252 of 1,509 chunks,
so every committed claim computed over *record bytes* had to be re-checked. All of
them:
| claim | reads | verdict |
| --- | --- | --- |
| 1,509 chunks, 34,491 records | offsets, not records | unaffected |
| 1,262 chunks with a leading-byte bucket rule | record byte 0 | unaffected |
| 1,235 chunks with an index rule | record bytes 1-2 | unaffected |
| 34,432 of 34,491 records 8-byte aligned | record byte 0 | unaffected |
| 55 distinct strides, all multiples of 4 | offsets | unaffected |
| moby class IDs in one slot per chunk type | record bytes | **retracted** |
| 17-21% of byte positions constant | record bytes | **survives** |
Only the claims that scanned *across* a record's body were at risk, and of those
one was retracted (the class-ID slot) and one survived (the constant-byte
statistic). The rest read the first three bytes of a record, which the over-read
never touched -- an over-read appends to the end, so a claim about a record's
*start* was never in danger. That asymmetry is why the damage was contained, and
it is the reason to prefer claims anchored at record starts.
The surviving one is worth stating precisely because it was the obvious candidate
for contamination. Re-measured with each record bounded by its successor, and
comparing only the span every record in the chunk has:
per level, % of byte positions constant within a chunk
lowest level 8 16.7%
highest 20.0%
Still inside the documented band, now asserted rather than only described, in
`chunk_records_share_a_layout_measured_by_constant_byte_positions`.
## The over-read bug was unique to one format, and why
Searched for the pattern that caused it -- a record length derived from the
*first* difference between two offsets:
grep -rn "\[1\] - \[0\]" crates/gc-formats/src/*.rs
crates/gc-formats/src/chunks.rs:357 (offsets[1] - offsets[0])
One hit, in the one parser that walks an explicit offset table.
The other parsers slice contiguous regions with `chunks_exact` -- shrub's 8-byte
palette records, tie's 16-byte packet headers, tfrag's 4-byte tables. There, a
length cannot run past the next element, because `chunks_exact` derives each
boundary from the region's own length rather than from a sample of the spacing.
Shrub's mesh packets are named by an explicit offset *and size* table, so they
carry their length rather than inferring it.
So the damage was bounded by a structural property: **only a format with an
offset table and a derived stride can over-read**, and exactly one format had
both.
The general guard is now positive rather than a comment.
`chunk_records_neither_overlap_nor_leave_gaps` asserts, across all 27 levels,
that every record except the last is exactly its own offset delta long, and that
1,509 chunks contain 1,252 with ragged deltas. A regression to a single sampled
stride fails immediately and says which chunk and which record.
## CORRECTED: the tie vertex region splits in two, but not into "VIF and flat"
This section previously read bit 31 of the region's first word as "this is a VIF
code word", and told a two-encoding story on it -- half VIF packets, half flat.
The split is real. **The identification is not.**
The stride retraction found the vertex region is not a uniform array -- some runs
4 bytes, some 16, run length varying. The split that accounts for it is visible
in the region's first word: a VIF code word has bit 31 set.
first word has bit 31 set 2,007
first word has bit 31 clear 2,102
(levels 0-5, every record with vert_size > 0)
The two kinds look like this:
flat, class 0 rec 0, 1024 bytes VIF, class 1 rec 0
+0 0x27667942 0x0062fbf1 +0 0xf5c1a3c5 CODE cmd 0x75 num 193
+8 0x0000f000 0x00001000 +4 0x00aef9c2
+16 0x67264af1 0x005f0585 +8 0x10001000
+24 0x1000e000 0x00001000 +12 0x00001000
... 16-byte units repeating, third word
walking down f000, e000, f000,
d000, d000, c000, c000; last
word constant at 0x00001000
So roughly half of all tie records begin with a word that has bit 31 set and
half do not. The split is overwhelmingly *not* a function of `vert_size` -- 62 of
the 72 values carry both kinds -- so "short objects are one thing, long ones
another" is false, which is the rule a decoder would otherwise reach for.
### They are not VIF streams
Bit 31 set is exactly what a VIF code word has, and that was the whole basis for
calling them VIF. Walking them as VIF packets says otherwise. Levels 0-2, 914
records:
walk reaches the region end 17
walk dies after exactly ONE packet 1,571
packets before dying: min 0, median 1, max 4
`tfrag` walks VIF the same way -- `VifCode::decode`, then `num * 4` payload words
-- and reaches the end of its lists every time. The walk is not the problem.
These regions are not VIF packet streams, so bit 31 is set on whatever this word
actually is, which remains unknown.
The original example quoted `0xf5c1a3c5` as a code word with cmd `0x75` num 193.
Decoding one word as a packet header says nothing about whether the next 193
words are its payload, and here they are not.
Both findings are now asserted in the same test, the split and the walk failure,
so that if these ever do turn out to be VIF the test fails and says so rather
than the two quietly drifting apart.
This does not settle the stride. It says the region needs to be dispatched on the
encoding before anything inside it can be read, and it explains why the flat
reading fit some records well -- `i16[6]` being `0x1000` and `i16[7]` `0x0000` at
the end of each 16-byte unit is the (-1, 0) / (1, 0) corner pair that a texture
coordinate produces in 12.4 fixed point, which is where the earlier texcoord
reading came from and why it looked convincing on the records that are flat.
Asserted in `tie_vertex_regions_split_between_vif_and_flat`.
## RETRACTED: the tie texcoord field is at `i16[4]`/`i16[5]`
The 73.6% figure above, and the claim that slot pair is the texcoord, do not
survive. Two separate problems.
**It was measured on a reading that has since been retracted.** The 73.6% was
computed across all records at a 16-byte element stride. `e5cd3df` established
that the stride is not 16 -- half the records are VIF packets and half are flat --
so the numbers describe neither population cleanly.
**And on the flat records, the fixed-point reading fails.** The question that
separates a 12.4 fixed-point texcoord from anything else is how many of a slot's
distinct values are multiples of 4096 at all. For a texcoord it is all of them.
Measured on the **flat** records only, levels 0-2, 930 records and 47,909
vertices:
slot distinct of which are multiples of 4096
i16[4] 1,488 16
i16[5] 1,261 16
i16[6] 966 4
i16[7] 489 5
One percent. The dominant *pairs* do look like UV corners -- `(0,0)` 5,773,
`(0,4096)` 5,500, `(4096,4096)` 5,059, `(4096,0)` 4,521 -- and that is exactly
why the claim was persuasive. But it is a statement about the most frequent
values, not about the field, and 1,472 of the 1,488 values in `i16[4]` are not
multiples of 4096 at all.
This is the same failure shape as the moby class-ID artifact, one level down: a
small set of dominant values that happen to satisfy the test, scored as though
the whole population did. The 73.6% is the arithmetic being right about the
popular values and the conclusion being wrong about the field.
So: on a flat record, `i16[4]`/`i16[5]` are **not identified**, and the texcoord
slot of a tie vertex is unknown. What survives from the original section is the
boundary work -- `vert_ofs + vert_size + scissor_size` is the region length -- and
the LOD1/LOD2 sharing result, which is stride-insensitive.
## The `SBlk` body head: the DAW size, twice
The body after the 24-byte head opens with twelve words, and six of them are
settled across all 27 levels:
word value note
0 varies per level not identified; 27 distinct
1 60 constant
2 varies per level not identified; 23 distinct
3 0 constant
4 daw_size == the header's daw_size, exactly
5 daw_size the same value again
6 0 constant
7 0 constant
8 0 constant
9 127 constant
10 1 constant
11 0 constant
Level 0's words 4 and 5 are both 1,482,000, and `daw_size` is 1,482,000.
### The pointer that was not one
Words 4 and 5 were briefly recorded as pointers into the DAW region. They are not.
The test that promoted them was "does this value fall inside
`[daw_offset, daw_offset + daw_size)`", and the DAW region is ~1.5 MB starting
near the front of a ~1.5 MB extent -- so almost any plausible 32-bit word lands in
it, and 1,482,000 is just under the top of exactly that range.
**A range test is not an identification.** The claim that survives is *equality
with `daw_size`*, which is exact on all 27 levels and is asserted. That is the
fourth time in this session that a plausible reading of one number has had to be
withdrawn in favour of a claim that can fail.
Word 0 is worth one note: levels 1 and 4 hold 13,042,052 and 13,042,054, two
apart, while their `daw_size`s differ by 63,504. Whatever it is, it is not a
size.
Asserted in `sound_bank_sblk_body_head_carries_the_daw_size_twice`. The remainder
of the body -- a per-sound record table -- is still undecoded.
## The decomp names the collision functions, but their addresses are not ours
`OpenRAC/rac2-gc-decomp` names both collision entry points:
Camera_CollPrimTest 0x001E7A50 camera-to-world boundary check
MB_CheckCollPill 0x001E8890 pill/capsule check for moby instances
These looked like the collision unblock: a named function at a known address is
exactly what the `$gp` string search failed to produce. They are not usable.
Checked against four levels -- 0, 5, 13, 26 -- with `parse_overlay_sections`:
Camera_CollPrimTest @ 0x1e7a50 -> section 1, kind 8 (NOBITS)
MB_CheckCollPill @ 0x1e8890 -> section 1, kind 8 (NOBITS)
Both land inside the level program's zero-filled section. Every byte there is
zero, and the following 256 bytes contain no non-zero byte at all. Code cannot be
there, and both addresses are in the same place on all four levels.
So those addresses belong to a different address space than a parsed level
overlay -- most likely the boot executable's own layout, or the v2.00 build, both
of which that repository also covers. `SCUS_972.68` is USA v1.01 and our disc
matches it exactly, but that fixes the *disc*, not which program's address space a
symbol address is quoted in.
This closes the shortest route to the collision cell layout that the decomp
appeared to offer, and it closes it for a checkable reason rather than a vague
one. The route that remains is the one already recorded: find how the code
addresses the `Camera_CollPrimTest` assert string, which is not by a `$gp`-relative
load.
## A route into `.text`, and one thing the decomp's assert table does not mean
The decomp gained 13 commits (12,375 lines, almost all `src/`). Nothing in it
names a collision data structure, so the format is not handed over. Two things
are usable.
### `assert-message-names.tsv` addresses are code, in `.text`
The table maps each assert's name to an address, 25 overlays each. For level 0,
`CameraCollPrimTestGridOutOfBounds` is `0x002db350`.
That address is **not** the assert call and not the string. It is:
0x002db350: jr $ra
0x002db354: addiu $sp, $sp, 0x70
0x002db358: nop
an **epilogue** -- the address lands on the last instruction group of the
function before it. The `0x70` is the frame being torn down; `0x002db35c`
immediately after is a *new* prologue (`addiu $sp, $sp, -0x80`).
So the table's addresses are function-ish locations in `.text` (level 0's
`0x002db350` is well past `.lit`, which ends at `0x001b165c`, and inside
`0x002a3b80..0x00447900`). They are not string references, which is why the
earlier "$gp-relative search returns zero" result is now explained from the other
side: the string is in `.lit` at `0x001a8998` and the code is in `.text` five
megabytes away.
### What is actually in that function
Immediately before the epilogue:
0x002db2dc: lwc1 $f20, 0xc($s0)
0x002db2ec: add.s $f20, $f20, $f21
0x002db2f0: c.olt.s $f0, $f20
0x002db304: jal 0x2db4b8 <- the assert call
0x002db308: mov.s $f12, $f21
0x002db30c: bnez $v0, 0x2db32c
0x002db314: lw $v0, ($s1) <- a count
0x002db318: addiu $s2, $s2, 1
0x002db31c: slt $v0, $s2, $v0
0x002db320: bnez $v0, 0x2db2cc <- loop back
0x002db324: addiu $s0, $s0, 0x30 <- stride 48 bytes
A loop over records of **48 bytes**, reading a float at `+12` and accumulating,
guarded by a count at `($s1)`. `lui $at, 0x3f80` / `mtc1 $at, $f22` sets `f22` to
1.0f, which is a minimum-height clamp.
### It is not the collision grid, and the signature says so
This is checked rather than assumed. `Camera_CollPrimTest` is declared
`Camera_CollPrimTest(void *prim, void *grid)` -- two pointer arguments, which in
MIPS arrive in `$a0` and `$a1`.
The function's prologue is `0x002db078: addiu $sp, $sp, -0x70`, matching the
`+0x70` teardown at `0x002db354`. The two instructions after it are:
0x002db078: addiu $sp, $sp, -0x70
0x002db07c: lw $v1, -0x3f4c($v0) <- $v0, not an incoming argument
0x002db080: addiu $a0, $zero, 0x6f <- $a0 set to a constant
`$v0` is caller-saved scratch, not an argument register, and `$a0` is
immediately overwritten with the constant 111. **No `void *` reaches this
function**, so it is not `Camera_CollPrimTest` whatever the assert table labels.
It reads four consecutive words from boot memory instead -- `0x1a7348`,
`0x1a734c`, `0x1a7350`, `0x1a7354` -- all below the level program base at
`0x001A7C80`, which is the boot's resident region and identical in every
program. That is a different data source from the collision payload, and the
48-byte stride loop belongs to whatever that structure is.
So this lead is dead, and it is dead on evidence rather than on doubt. Both routes
the decomp appeared to offer are now closed:
| route | status |
| --- | --- |
| `Camera_CollPrimTest` at `0x001E7A50` | lands in NOBITS, all zero |
| `CameraCollPrimTestGridOutOfBounds` at `0x002db350` | a function with no pointer arguments |
Neither address reaches the collision parser.
### And the last route is closed too: the string is never referenced
The route that remained was to find how the code addresses the assert **string**
at `0x001a8998` in `.lit`. It does not address it. Level 0's program is
2,751,268 bytes; searching it for both ways an address can be materialised:
address low half 0x8998 0 occurrences, word-aligned or not
$gp-relative offset 0x99a8 0 occurrences, word-aligned or not
exact 32-bit 0x001a8998 0 occurrences
The method is validated by a positive control in the same run: the `.lit` load
base's own low half, `0x7c80`, occurs exactly **once**, word-aligned. A search
that finds that will find any real reference.
So **no instruction and no data word in the retail level program references the
collision assert string's address.** Not by `lui`/`addiu`, not by a literal
pointer, not by a table entry. The string text is present in `.lit` at
`overlay+0xd18`, and the same text also appears in the boot binary at file offset
`0xA9558`, but nothing points at either.
The reading that fits: **the assert is compiled out in retail and the string is
dead data.** Debug string pools routinely keep messages whose only user was a
`#if`-ed-out assertion. The decomp's own table lists this name because it recovered
it from the string, not because it found code calling it.
That is the last route recorded in this file, and it is closed. Reaching the
collision cell layout from the disc would require a different kind of evidence --
a build with assertions enabled, a debug listing, or an independent description
of R&C2's collision cell -- none of which is on this disc.
Which is the honest summary of collision after all of it: the payload is verbatim
in RAM, it does not walk as fixed-stride cells, the disc-side segmentation is
exhausted, the string that would have led to the routine is unreferenced, and both
of the decomp's addresses are dead ends.
### The disassembler needed fixing first
`tools/overlay-anchors` called `disasm_all` over the whole window, which stops at
the first word capstone will not decode. In code regions that is almost
immediately, so every window came back five instructions long -- which is why
earlier attempts at this produced nothing. It now walks four bytes at a time,
prints `.word` for anything undecodable, and reports the failure count.
The undecodable words here are VU1 microcode (`0x0260202d` and friends are COP2
opcodes), which is expected and is itself informative: this function interleaves
VIF-fed VU1 work with FPU arithmetic, like `tfrag`'s.
## `lvl.vtbl` gives the real class ids, and the old bound was wrong
`lvl.vtbl` is **section 3** of the level program -- one of the seven sections the
Phase 3 port made visible. Its records are three words:
[0] oClass the moby class identifier
[1] handler address of UpdateMoby_<oClass> in this level's .text
[2] auxiliary the same value for every record of a level, 26 of 27
(level 1 has exactly one record pointing elsewhere, 0x2a5d94)
Walking all 27 levels:
5,878 records
1,162 distinct class ids
range 0 .. 0x13BA (5,050)
median 38 new ids contributed per level
The decomp's `docs/MOBY-DISPATCH-TABLES.md` describes exactly this layout, and
independently notes that 5,598 of 5,788 handlers are preceded by a `jr ra` within
three words -- so the addresses are function entries, not arbitrary words.
### Why this retires the earlier bound
`LevelCoreHeader::moby_classes.count` is **238** on level 0. **1,135 of the 1,162
class ids -- 97.6% -- are at or above that number.**
So "value < moby_classes.count" was never a test for "this is a class id". It was
a test for "this is a small number". That is the mechanism behind the chunk
class-ID artifact:
slot values 1, 3, 4, 5, 10, 11, 12, 13, 15
every one is below 238
so every one scored as a hit
65536 / 238 = 275x ... and the largest such score was 305x
The bound was wrong *before* the artifact was noticed, and the artifact was a
consequence of it. Retracting the claim for "small dominant values" was correct
but incomplete -- the reason those values looked significant is that the test
selected for smallness.
### Re-scored against the real id set
With the game's own 1,162 ids as the validity test, no chunk slot separates:
best slot 0.375 (type f2ef4275 slot 34)
chance rate 0.018 (1,162 ids in 65,536 values)
a dozen slots 0.25 .. 0.38, with no gap between them
Twenty times chance is not nothing, but a dozen slots packed into the same band
with nothing above them is what a broad property looks like, not a field. The
positive finding stands and is now usable: **the class-id space is known, and it
is not the class count.** Asserted in
`moby_class_ids_come_from_lvl_vtbl_not_from_the_class_count`.
## `oClass` is a global identifier, and not an index into the class table
Two axes that are easy to conflate, and are not the same thing:
lvl.vtbl oClass 1,162 distinct across 27 levels, largest 5,050
(was 1,163 until the terminator fix -- see
"The vtbl terminator" below; the loop read past the
end of the table)
core_index moby_classes.count 136 .. 259 per level
**1,133 of the 1,162 identifiers are at or above the largest per-level class
table.** So `oClass` cannot index that table -- nearly every identifier would fall
off the end. They are different axes: the vtbl names a *type* of moby, the class
table lists the *models* a level instantiates.
The identifier space is shared rather than renumbered per level: 1,162 distinct
values come out of 5,878 records, and level 0 contributes 234 while levels 1-8
contribute 72, 69, 77, 54, 41, 68 and 28 -- each level roughly its own share, not
one level dominating. An earlier note here said "most of the game's classes appear
in the first level"; the data says 20%, and the assertion is written to fail if
that ever changes.
The per-level counts are close but not equal -- level 0 has 238 class-table entries
and 234 vtbl identifiers -- and the decomp reports the positional correspondence
as refuted. Nothing here depends on that.
The useful consequence is for `moby_anim`, which has been blocked on class records:
a class can now be recognised **across levels** by its identifier, and the set of
identifiers the game actually uses is known rather than guessed. Asserted in
`vtbl_class_ids_are_global_and_do_not_index_the_class_table`.
## All three dispatch tables now decode
The level program carries three dispatch tables in its own sections. With the
Phase 3 section parser they are all readable, and each divides exactly by its
record width:
section file width record
vtbl lvl.vtbl 3 oClass, UpdateMoby_<oClass>, auxiliary
camvtbl lvl.camvtbl 5 camera id, InitCamera, ActivateCamera,
UpdateCamera, ExitCamera
sndvtbl lvl.sndvtbl 2 sound id, UpdateSound_<id>
camvtbl 217 records, 14 distinct ids, max 0x1B
sndvtbl 32 records, 6 distinct ids, max 0x07
The load-bearing check is the **zero remainder**: a wrong record width would leave
a partial record on most levels rather than none, and it leaves none on any of the
27 for either table. Section sizes across the levels are 40..52 words for the
camera tables and 0..8 for the sound tables.
Level 0's camera records, so the shape is visible rather than inferred:
[0] 0x0 0x4263b0 0x4233f8 0x4263b8 0x426478
[1] 0x4 0x426c78 0x427008 0x4266e8 0x4270c0
[2] 0x5 0x427ad8 0x427628 0x427ae0 0x427b38
and its sound records:
[0] 0x0 0x42ac18
[1] 0x1 0x42ad98
[2] 0x3 0x42b038
Both identifier spaces are **global and shared**: 217 camera records carry only 14
distinct ids and 32 sound records only 6, so levels re-use ids rather than
renumbering -- the same sharing as the moby table's 1,162 distinct out of 5,822.
This is what makes the tables usable rather than merely present. A per-level
dispatch entry is now addressable by a game-wide id, and the handler addresses are
function entries in that level's `.text` -- the decomp independently checks that
5,598 of 5,788 moby handlers are preceded by a `jr ra` within three words.
The camera and sound handler addresses fall inside section 6 of the same overlay,
which the working disassembler can read directly.
Asserted in `camera_and_sound_dispatch_tables_decode_at_their_documented_widths`.
## The handler chain works, with two caveats and one discrepancy
With the dispatch tables decoded and the disassembler fixed, a moby handler can
actually be read. Level 0's class 2, handler `0x00380a88`:
0x00380a84: jal 0x2ee0d0
0x00380a88: .word 0x0240202d <- VU1 microcode, not a decoding failure
0x00380a8c: lwc1 $f12, 0x40($s1)
0x00380a98: addiu $a0, $s0, 0x20
0x00380aa0: swc1 $f0, 0x18($s4)
0x00380ab0: swc1 $f0, 0x14($s4)
Floats read out of one structure and written back into another, with VU1 work
alongside -- which is what a transform update should look like.
### Two caveats before reading one
**VU1 microcode is interleaved with CPU code.** `0x0240202d` is a COP2 opcode, not
an undecodable word. A disassembly window will run `.word` lines that are
instructions in a different processor, and a tool that counts them as failures is
counting the wrong thing.
**Handlers are not all independent functions.** Level 0's class 2 is at
`0x380a88` and class `0x0E` at `0x380a90` -- **eight bytes apart** -- and 72 of the
5,788 records sit within 16 bytes of the one before them. So consecutive classes
share one routine with per-class entry points inside it, and reading "a handler"
gives a fragment, not a whole function.
### And a discrepancy with the reference
`docs/MOBY-DISPATCH-TABLES.md` reports 5,598 of 5,788 moby handlers (96.7%)
preceded by a `jr ra` within three words. Measured here, the same way, over the
same 5,788 records: **5,488 (94.8%)**.
Same population, same test, 110 handlers different. Both figures confirm the
premise, so nothing turns on it, and the residual is recorded rather than explained
away. The reference measured on 2026-10-02 against its own pinned overlay set, so
a difference in method or version is as plausible as a difference in the data, and
there is no basis here for saying which. The test asserts the premise (>90%) and
deliberately not either exact figure.
Asserted in `moby_handlers_are_overwhelmingly_function_entries`.
## How many behaviours the dispatch tables actually contain
Records and behaviours are not the same number, and the difference is where one
routine serves several classes. Over all 27 levels:
table records distinct classes distinct handlers handlers serving >1
vtbl 5,822 1,162 4,737 400
camvtbl 217 14 216 1
sndvtbl 32 6 32 0
So sharing is real but modest, and it is almost entirely in the moby table:
5,822 records collapse to 4,737 addresses, the median handler serves one class and
the most-shared serves 34. The sound table is exactly one handler per record, and
the camera table is one short of that.
That matches the earlier observation that 72 moby records sit within 16 bytes of
their predecessor: 5,822 - 4,737 is 1,085 records sharing an address with
another, and the tightest of those are adjacent in the table.
The practical figure is **4,737**. A class's behaviour is not one of 5,822
routines, and looking for "the animator's function" by assuming one class means one
function would search the wrong number of places.
Asserted in `dispatch_tables_contain_far_fewer_behaviours_than_records`, which
also fails if a table ever reports more behaviours than records -- an
impossibility worth pinning because it is the shape the counts would take if the
record widths were wrong.
## The moby instance, 256 bytes, named
The decomp carries an authentic `MobyInstance` layout of 0x100 bytes with every
field named, and it is now transcribed into `gc_formats::moby::MobyInstance`
with a 67-entry offset map. Two fields matter for the open items:
0x40 animSeq animation sequence pointer
0x44 animSeqT animation time parameter
0x48 animSpeed playback speed
0x50 animLayers animation layer list
0x57 jointCnt joint count -- non-zero means the moby is skinned
0x58 jointCache joint matrix cache
0xBC oClass object class id -- the dispatch key
`oClass` closes a loop that was open for several commits. It is the field the
runtime uses to reach the `lvl.vtbl` handler tables, so an instance and its
routine are now linked by a named offset rather than by an assumption about which
axis the two tables share.
`jointCnt` and `jointCache` also settle something from earlier in this log by
contradiction rather than confirmation. A tie vertex's `i16[3]` was tested as a
skin/joint index, which the measurement **did not settle** -- see the correction
above. That earlier result was over-read, and the joint reading is now the
leading candidate rather than a disproved one. Either way this is the *moby*
struct and a moby is a different object from a tie, so the two are not evidence
about each other.
What this does **not** do is find `moby_anim`. It names the instance-side state --
a sequence pointer, a layer list, a joint cache -- which means the asset to locate
is whatever those point at, and we still do not know where that is. The item's
unblock condition is now sharper: it needs the data those pointers reference, not
the struct.
### Provenance and what the test can check
The layout is transcribed, not derived here, and it is not verified against disc
data because it cannot be: moby instances are runtime objects and nothing of that
shape is in the container. What the test asserts is what can be asserted without
the bytes -- that the offsets ascend, do not overlap, and land the last field
exactly on 0x100, and that `o_class` sits at 0xBC immediately before `triggers`.
Worth keeping distinct from the collision finding: that project's **function
addresses** did not transfer, landing in NOBITS memory. A struct's field order is a
different kind of claim from an address in a program's layout, so the failure of
one does not impeach the other -- but the distinction is real, and `PROVENANCE.md`
now says which is which.
## Two tie tests still measure a layout that was retracted
The stride retraction (`e5cd3df`) disowned the 16-byte element reading and the
95,612-vertex population that came with it. Two committed tests still compute
exactly those figures and still pass:
tie_vertex_element_is_16_bytes_with_fixed_texcoords
tie_vertex_layout_is_selected_by_i16_3_high_byte
They pass because the arithmetic is self-consistent about a **withdrawn**
population -- the same shape as the chunk uniformity claim, one level up. The
parser slices at 16 bytes, the test reads the slices, the numbers agree, and
nothing in the loop can notice that the stride is wrong.
Both are kept, and marked. The reason is specific: each is the record of what the
16-byte reading *produces*, so the retraction is visibly about the stride rather
than about the arithmetic. Deleting or renaming them would erase the thing the
retraction is a retraction *of*, and there would be no way to check later that the
numbers moved because the layout changed rather than because someone edited a
constant.
The marking says so at the top of both:
> Do not quote the 82.5% figure as a property of tie vertices. It is a property
> of one reading of the vertex region, on a population the retraction disowned.
### Two denominators, both correct, 92.9% apart
The layout test reports `(4,5)` as 12-bit fixed on **92.9% of vertices**. A
different measurement, taken on flat records only, found that only **16 of 1,488**
distinct `i16[4]` values are multiples of 4096.
Both are right. Most vertices use the handful of UV-corner values -- `(0,0)`,
`(0,4096)`, `(4096,4096)`, `(4096,0)` -- which are all multiples of 4096, so the
per-vertex rate is high. The other 1,472 values are not, so the per-distinct-value
rate is 1%.
That gap is the artifact, and naming both denominators is the most useful thing
this section can do about it. A percentage over vertices and a percentage over
distinct values answer different questions, and only one of them was ever asked
about.
## The tie vertex stride, searched properly: 64 where one exists, none for 62%
The 16-byte reading was retracted without a replacement. Searching for one: for
every tie vertex region, try strides 4, 8, ... up to `len / 16` and keep any where
some byte offset is constant across every unit.
regions examined 1,844
no stride found at all 1,139 (61.7%)
stride 64 491
stride 56 120
stride 72 49
stride 48 12
stride 16 3
**64 bytes, not 16, wherever a stride is detectable** -- which is now read the
other way round. The 64 bytes are **four 16-byte elements with four different
layouts**, cycling `+00 +10 +20 +30`. So 16 *is* the element size, and 64 is the
period of the layout. The corrected measurement is below.
Those regions are genuinely variable, or the element is larger than the region
divided by sixteen units, which is the same statement.
### The control that changed the answer
The first run of this search reported **1,561** regions at stride 64. It was
wrong, and wrong in a way worth recording: a large stride divides a region into
fewer units, and fewer units makes an accidental constant more likely. Length is
fixed across strides, so the widest candidate always has the fewest units and the
best chance of constancy by luck alone.
Requiring every candidate stride to cover the **same** minimum of 16 units, and
preferring the **smaller** stride when hit counts tie, drops it to 491 and
promotes 56 and 72 into view -- which were invisible before.
That is rule 3a's sampling failure in its purest form: an aggregate over
candidates with different denominators, where the largest denominator wins for a
reason that has nothing to do with the data. The first number was not merely
imprecise, it was reporting the artefact.
### What this does not settle
64 is the stride where one is detectable, which is not the same as the stride.
For the 1,139 regions with no constant byte under any tested stride, the element
is still unknown, and the tie vertex layout remains open. What is settled is that
**no single layout describes every element** -- and the retraction that followed
was itself half wrong, because it read "16 is not the stride" where the data
says "16 is the element and the layout cycles".
## Corrected: the 64-byte period is four layouts, and it beats a shuffle control
The retraction in `e5cd3df` said the 16-byte element reading does not survive a
stride test. It survives one; the stride just was not being looked for properly.
**Range measured:** every tie vertex region on all 27 levels, **15,219 regions**.
A byte offset is *constant at stride `s`* when every unit of length `s` agrees
there. Requiring at least three units, a stride that divides the region exactly,
and at least eight constant bytes:
regions with a signature 7,314 of 15,219
same test on shuffled bytes 731 of 15,219
stride 64 6,463 of the 7,314
**The signature is ten times the control**, so it is a property of the data and
not of the test. Dumping the regions that carry it shows four *different*
16-byte layouts repeating at `+00 +10 +20 +30`: one is a small i16 followed by
two zeroes, one carries `255, -1` mid-element, and the other two carry neither.
So **the element is 16 bytes and the layout cycles with period four**.
This is a partial retraction of a retraction, recorded as such. What `e5cd3df`
correctly established is that the four layouts are not the same, and that its
extents spanned the whole `i16` range because it pooled four layouts into one
set -- rule 3a's failure arriving through a different door. What it wrongly
concluded is that 16 is not the stride.
### What still does not identify the position
Four tests were run against the four layouts and **none separates them**:
| test | group 0 | group 1 | group 2 | group 3 |
|---|---|---|---|---|
| LOD1/LOD2 triples shared, same class | 16.1% | 16.2% | 16.2% | 16.3% |
| words as `f32` in `1e3..1e7` | 23% | 23% | 23% | 23% |
| words as `i32` in `1e6..1.6e6` | 0% | 2% | 6% | 0% |
| `f32` finiteness | 99% | 77% | 99% | 99% |
The `f32` and `i32` rates are what uniform random gives -- those two rows carry
their own control -- and the LOD1/LOD2 sharing is the same in every group to
within a fifth of a point. Four measurements agreeing this closely say the four
groups are one population for these statistics, not that any of them is a
position. The position remains unnamed, and the `i16[0..2]` claim stays
retracted; it is just retracted for a different reason than the one recorded at
the time.
`tie_vertex_region_has_a_stride_64_period` asserts the three bounds above, plus
the shuffle margin, so a claim that the period is not 64 fails rather than
sailing through.
## Attempting the moby_anim unblock: the class header, and where it stalls
`MobyInstance` names `animSeq`, `animLayers`, `jointCnt` and `jointCache` as
runtime pointers. Those pointers are useless on the disc -- they are RAM
addresses. What is *on* the disc is whatever they point at, reached from the moby
class blob, so that blob's header is where the unblock has to happen.
Level 0, 187 moby classes. The span opens with the part-table offset, which is
`0x80` on 69 classes, `0x90` on 34, and ranges from 0 to 229,184 across the rest --
so a class carries a header of roughly `0x40` bytes before its parts, and that
header is unexplored.
Per byte over `+0x00..+0x40`, how many of the 187 classes are non-zero:
+0x00 177 the part-table offset (0, 128, 144, ...)
+0x07 185 8 distinct values, 1..3
+0x26 181 100 distinct -- varies per class
+0x27 187 8 distinct
+0x2d 187 8 distinct
+0x2e 12
+0x30 170 120 distinct
+0x3c 184 120 distinct
+0x3e 187 110 distinct
+0x3f 187 5 distinct
Six or seven fields there are constant across every class, and about a dozen
vary. None is yet identifiable: nothing has the shape of "pointer to a separate
region" -- the varying fields hold values in the tens of thousands, which is the
same order as the span offsets themselves, so they may be intra-span pointers or
may be sizes.
**This is where it stalls, and the stall is honest.** The header is dense with
unexplained fields and the obvious move -- find the one that points outside the
span -- does not resolve, because every varying field is small enough to be an
intra-span offset. Distinguishing them needs either the engine's class loader
(readable now, in the same overlay's `.text`) or a class whose animation is known
from outside, and neither has been done.
### And the discriminator fails for a sharper reason than "they are small"
Two tests, both cheap:
**No header `u32` equals the span's own table offset** on any of the 187 classes.
So the varying fields are not self-relative pointers to the part table.
**The largest value in any header `u32` is 3,332,678,933.** Some of these fields
are not offsets at all -- at `+0x0c` the value is 16,711,748 across classes, which
is packed data or an identifier, not a displacement.
So the header's varying fields are a **mixture**: some hold in-span offsets
(`+0x10..+0x1c` run 192 to 10,800, comfortably inside the spans) and some hold
packed values far too large to be offsets. There is no single "the field that
points to the animation" to find, because not all of them point anywhere.
That is a real narrowing even though it is a negative: **if per-class animation
data exists on the disc, it is inside the class span**, not at a separate pointer
target. The search therefore stops looking for a pointer and starts looking for a
region *within* the span that the geometry parser does not already account for --
which is a bounded question with the parser as its control.
So `moby_anim` is still **not** unblocked, but the question is now bounded rather
than open-ended, and the next step is to subtract the parsed geometry from the
span and look at what remains.
## Phase 3 gate, second half: our masking against an independent clustering
The Phase 3 gate asks for the relocated-copy result to be "cross-checked against
an independent cluster hash". `OpenRAC/rac2-gc-decomp`'s
`config/levels/functions.tsv` is exactly that: 7,765 functions carrying a content
**fingerprint** and the list of levels each appears on. Of those, **3,452 appear on
all 27**, falling into **2,984 distinct fingerprints**.
Cross-checking against our own masking, over the level-0 copy of each of those:
our masked groups 3,389
reference fingerprint groups 2,984
pairs the reference merges, we split 3,231
pairs we merge, the reference splits 0
**Zero.** That is the number that matters. Whenever our masking says two functions
are the same code, the independent fingerprint says so too -- our method never
merges what theirs separates. It is *strictly finer*: 3,389 groups against 2,984,
splitting 3,231 pairs the reference keeps together.
So the two clusterings are compatible, with ours the more discriminating of the
two. The likely reason ours splits: the reference fingerprint evidently ignores
some of what our mask preserves, so functions it considers identical differ in the
words our mask keeps. That is a difference in coarseness, not a contradiction.
### A comparison that was invalid before it was valid
The first run of this cross-check reported near-unanimous disagreement -- 1,900
pairs differing out of 3,392 -- which looked like a real conflict. It was a
category error in the harness: `key()` masks the **first four words**, while the
reference fingerprints the **whole function**. Comparing a four-word summary
against a whole-body hash is not a comparison of two clusterings.
With `mask()` over the entire body instead, the numbers above are what the two
methods actually say. Recording it because the invalid version would have read
as a significant finding, and the correction is not visible in the numbers alone.
The gate's second half is therefore met, with the caveat that "agrees" means
"never contradicts", and our result is the finer of the two partitions.
## The vtbl terminator, and why three counts were all defensible
Three committed tests counted `lvl.vtbl` records and disagreed: 5,878 records /
1,162 ids, 5,822 / 1,162, and 5,788. All three were measuring something, and none
of the tests was simply wrong.
**5,822 records / 1,162 ids is the table.** The record array terminates on
`id == 0xffffffff` in the *first* word of a 3-word record. Two tests were skipping
that word with `continue` and walking on into whatever follows the table, which
picked up 56 records past the end and one extra class id. Breaking at the
terminator is the correct reading of a dispatched table, and both tests now do.
**5,788 is the records with a real handler.** Of the 5,822 records, 34 carry the
terminator in the *handler* word with a valid id — slots that exist in the table
but have no geometry. A third test breaks on the handler word rather than the id
word, and so stops 34 short. Both stopping points are correct for what that test
is measuring.
**5,878 was the bug.** It was the first number above, reading past the table.
Two of those bad counts were *pinned* -- `assert_eq!(classes.len(), 1163)` in two
separate tests -- so the walk-past-the-terminator behaviour had been deliberately
recorded at some point, not merely overlooked. Both now assert 1,162.
Every count involved is now asserted exactly rather than as a bound. The general
point is the one worth keeping: **a test that prints a measurement and asserts a
threshold cannot detect drift it is looser than.** `records > 5_000` was satisfied
by 5,878 and by 5,822 alike, so the wrong loop sat there indefinitely and the only
reason it surfaced was that a different test counted the same table differently.
The disagreement was not a decomp mismatch and not a hardware question. It was two
sentences in a test saying `if c != MAX { ... }` and `if c == MAX { break }`, which
are the same condition with opposite answers to whether the table ends there.