Followed the chain through -- dispatch table to handler address to disassembly --
and read level 0's class 2 handler at 0x00380a88:
0x00380a84: jal 0x2ee0d0
0x00380a88: .word 0x0240202d <- VU1 microcode, not a failure
0x00380a8c: lwc1 $f12, 0x40($s1)
0x00380a98: addiu $a0, $s0, 0x20
0x00380aa0: swc1 $f0, 0x18($s4)
0x00380ab0: swc1 $f0, 0x14($s4)
Floats out of one structure, written back into another, with VU1 work alongside --
what a transform update should look like. The chain from an asset observation to the
game's own code is now complete and usable.
TWO CAVEATS, both learned by hitting them:
VU1 microcode interleaves with CPU code. 0x0240202d is a COP2 opcode, not an
undecodable word. A tool that counts .word lines as failures is counting the wrong
thing -- a trap I walked into earlier today with the region walkers, now recorded
so it is not repeated.
Handlers are not all independent functions. Level 0's class 2 is at 0x380a88 and
class 0x0E at 0x380a90 -- EIGHT BYTES apart -- and 72 of the 5,788 records sit
within 16 bytes of the one before them. Consecutive classes share one routine with
per-class entry points inside it, so reading "a handler" gives a fragment. That
matters downstream: a class's behaviour is not one function, and assuming
otherwise would misattribute shared code.
A DISCREPANCY WITH THE REFERENCE
The decomp reports 5,598 of 5,788 moby handlers (96.7%) preceded by a jr ra within
three words. Measured here, the same way, over the same 5,788 records: 5,488
(94.8%). Same population, same test, 110 handlers different.
Both figures support the premise, so nothing turns on it, and I am recording it
rather than picking the number that suits. The reference measured on 2026-10-02
against its own pinned overlay set, so a method or version difference is as
plausible as a data difference and there is no basis here for saying which. The
test asserts the premise (>90%) and deliberately NOT either exact figure.
Asserted in moby_handlers_are_overwhelmingly_function_entries.
264 tests.
3243 lines
152 KiB
Markdown
3243 lines
152 KiB
Markdown
# core_data region bounds
|
||
|
||
How a level's `core_data` lump divides up, and how each region is bounded.
|
||
|
||
## The rule
|
||
|
||
A region's end is the **next region's start**, never `core_data.len()`. The
|
||
last region in a layout is the exception, and when its successor is unknown the
|
||
end is `None` rather than a guess.
|
||
|
||
This exists because region bounds were being re-derived per format during
|
||
probing, usually by running from one known offset to the next with the last
|
||
span running to the end of the lump. On level 0 that made the final shrub class
|
||
a 925,008-byte span absorbing most of the lump's tail. Every aggregate computed
|
||
over it -- record counts, value distributions, "this data looks random" -- was
|
||
silently wrong, and nothing errored.
|
||
|
||
`gc_formats::regions::core_layout` computes these once. Callers use it rather
|
||
than inventing their own.
|
||
|
||
## Layout
|
||
|
||
| region | start | end |
|
||
|------------|----------------|----------------------|
|
||
| `tfrags` | 0 | `occlusion` |
|
||
| `occlusion`| `occlusion` | `sky` |
|
||
| `sky` | `sky` | `collision` |
|
||
| `collision`| `collision` | first moby class |
|
||
| `moby` | first moby | first tie class |
|
||
| `tie` | first tie | first shrub class, |
|
||
| | | else chunk container |
|
||
| `shrub` | first shrub | chunk container |
|
||
| `chunks` | chunk container| lump length − 0x38A10|
|
||
|
||
Class regions start at the lowest non-zero geometry offset in their class
|
||
table. A zero entry means *absent*, not position zero -- the same trap the
|
||
occlusion and collision headers each fell into separately.
|
||
|
||
Not every level has every region: level 10 has no occlusion and no shrub.
|
||
|
||
## The constant trailer
|
||
|
||
Comparing all 27 levels byte-for-byte from the end: the last **231,952 bytes
|
||
(0x38A10)** of every level's `core_data` are identical. Level-specific content
|
||
therefore ends at `len - 0x38A10`, and the final region is bounded there rather
|
||
than at `core_data.len()`.
|
||
|
||
This is what closes the chunk container. The regions it separates from level
|
||
content are measured on the disc: **2,033,776 bytes of shrub region across the
|
||
24 levels that have one, and 25,775,850 bytes of chunk container** across all
|
||
27. The disc test asserts the bound on every level.
|
||
|
||
What the trailer *is* is not established. It may be fixed data, or padding
|
||
from a fixed-size decompression buffer. Either way it is constant, which is
|
||
all that is needed to bound the tail.
|
||
|
||
## Found: the chunk container closes the class region
|
||
|
||
The offset ending the shrub region is not in the core index, and the class
|
||
table that precedes the container carries no successor offset for it -- unlike
|
||
moby and tie, whose boundaries fall out of the tables. Texture texels do not
|
||
bound it either: they sit before moby, not after shrub. So the boundary is
|
||
located in `core_data` itself.
|
||
|
||
**Range measured:** for each level, from the last shrub class's geometry
|
||
offset to the constant trailer. Nothing outside that range is read.
|
||
|
||
**Method:** scan forward from the last shrub class for the pair (a) the
|
||
constant two bytes `f6 7c`, sitting immediately in front of (b) a chunk
|
||
signature -- a 16-byte header ending `0xff`, ten zero bytes, then four or more
|
||
`0xf0`-tagged offsets. `gc_formats::chunks::chunk_region_start`.
|
||
|
||
Each half of that signature is weak on its own, which is why the halves are
|
||
measured separately:
|
||
|
||
- the pair `f6 7c` occurs **25 to 103 times** in a 5 to 17 MB lump, so
|
||
matching it alone picks dozens of wrong offsets;
|
||
- the chunk shape matches random data often enough that `parse_chunk` demands
|
||
four tagged offsets before a signature is believed.
|
||
|
||
**Result: on all 27 levels the pair sits exactly two bytes before the first
|
||
chunk header.** The two are never found apart. This is the boundary, and
|
||
`core_layout` closes shrub with it -- `shrub`'s end is the container's start,
|
||
and the container's own end is the trailer, so every byte of level-specific
|
||
`core_data` now belongs to a named region.
|
||
|
||
Two controls, both asserted by the disc test `chunk_container_closes_the_last_class_region`:
|
||
|
||
- **No signature inside the body.** Scanning the last shrub body for a bare
|
||
chunk signature finds none, on any level -- so the boundary search was not
|
||
matching something it should have skipped, and the marker's job is
|
||
corroboration rather than load-bearing disambiguation.
|
||
- **The three levels without shrub** (10, 25, 26) carry the same marker two
|
||
bytes before the container. There the container closes *tie* instead, which
|
||
is what makes the locator a property of the region rather than a shrub
|
||
rule.
|
||
|
||
On level 0:
|
||
|
||
| what | range | bytes |
|
||
|---|---|---|
|
||
| last shrub class data | 16046800 .. 16047040 | 240 |
|
||
| marker `f6 7c` | 16047040 .. 16047042 | 2 |
|
||
| **chunk container** | 16047042 .. 16737344 | **690,302** |
|
||
| constant trailer | 16737344 .. 16969296 | 231,952 |
|
||
|
||
The marker's meaning is not established. It is constant across all 27 levels
|
||
and it appears at exactly this position, but whether it is a terminator, an end
|
||
count or a tag is unknown, so it is not named.
|
||
|
||
**Correction this produces.** The chunk-container measurements recorded below
|
||
were taken over "the 900,000 bytes before the constant trailer" -- a window
|
||
wide enough to contain the container on every level but not aligned to its
|
||
true extent. Chunks sitting at the window's edge were truncated (27 of 1,123),
|
||
and the region's size was never correctly stated. Both are now bounded
|
||
properly: on level 0 the container is 690,302 bytes, not ~900,000, and the
|
||
25,775,850 bytes of chunk data across 27 levels are the measured total rather
|
||
than a windowed sample. The `chunk_containers_resolve_on_every_level` test now
|
||
reads the region from `core_layout` instead of measuring backwards from the
|
||
end of the lump.
|
||
|
||
## Correction: it is mostly texture data, not an unnamed region
|
||
|
||
Adding the region table to `gc-extract probe` made the gap visible
|
||
immediately, which is the point of printing it.
|
||
|
||
On level 0:
|
||
|
||
```
|
||
collision 2615232 .. 8832192 6216960 bytes
|
||
```
|
||
|
||
but collision's own header declares a payload of 903,248 bytes. So the range
|
||
between the collision header and the first moby class is **6.2 MB**, and
|
||
almost none of it is collision.
|
||
|
||
The first conclusion drawn from this -- that ~6.2 MB of unidentified data sat
|
||
between collision and moby, possibly `moby_anim` -- **was wrong**, and it was
|
||
wrong because of the same failure that produced the other two bad numbers this
|
||
session: quoting a figure before bounding it.
|
||
|
||
Checking the bytes rather than just the sizes: the range is dominated by 8-bit
|
||
indexed texture data. Byte-value concentration (a texture block uses a small
|
||
palette; code and collision data does not) puts 18 of 21 sampled 4 KB blocks
|
||
under 200 distinct byte values, against 249 for tfrags and 209 for collision.
|
||
|
||
So the real layout is:
|
||
|
||
| region | start | end |
|
||
|------------|----------|------------------|
|
||
| collision | 2615232 | 3518544 |
|
||
| textures | 3518848 | ~8832192 |
|
||
| moby | 8832192 | first tie class |
|
||
|
||
The texture table's `data_offset` values only reach 5326208, which is why the
|
||
gap looked real: mip chains extend well past the last entry's offset. A region
|
||
sized from a table of first-level offsets understates the data it points at.
|
||
|
||
The "3 of 21 blocks not texture-like" figure in the previous revision was an
|
||
artifact of sampling every 256 KB. Re-scanned at 4 KB granularity the whole
|
||
range is texture data: ~100 blocks stand out, and looking at them they are
|
||
smooth gradient ramps -- sky and fog texture, exactly what gradient texels look
|
||
like. A noisy gradient has a high distinct byte count without being code.
|
||
|
||
So the range between collision and moby is **entirely texture data**, and there
|
||
is no separate `moby_anim` region there. Combined with the correction above,
|
||
two claims made about this range have now been wrong: the "6.2 MB of unknown
|
||
data" and the "3 unexplained blocks".
|
||
|
||
Where `moby_anim` actually lives is therefore still open, but it is not here.
|
||
The remaining candidate is the unresolved tail after the last shrub class,
|
||
which on level 0 is ~925 KB of real, non-zero data that nothing yet accounts
|
||
for.
|
||
|
||
## Where shrub data ends, and the region after it
|
||
|
||
**Range measured:** for each level, the last shrub class's `data_offset` field
|
||
(`class + 0x2c`) through to the start of the constant trailer. Nothing outside
|
||
that range is read.
|
||
|
||
**Method:** the last shrub class's body is an array of 8-byte records whose
|
||
final two bytes are `00 00`. Scanning forward in 8-byte steps for the first
|
||
record whose trailing pair differs locates the body's end. This is a search
|
||
for a specific stated pattern, not a statistic, so it either finds the
|
||
boundary or reports that it did not.
|
||
|
||
**Result:** on **24 of 24** levels the boundary lands exactly on an 8-byte
|
||
record boundary. Body lengths are 192, 208, 224 or 240 bytes -- the same sizes
|
||
seen when shrub classes were grouped and diffed earlier.
|
||
|
||
On level 0:
|
||
|
||
| what | range | bytes |
|
||
|---|---|---|
|
||
| last shrub class data | 16046800 .. 16047040 | 240 |
|
||
| **next region** | 16047040 .. 16737344 | **690,304** |
|
||
| constant trailer | 16737344 .. 16969296 | 231,952 |
|
||
|
||
So the 690,304 bytes between the last shrub class and the trailer are a
|
||
distinct region, not shrub data. What they contain is not established.
|
||
|
||
Two things are worth noting about how this was found, given the rule in
|
||
`AGENTS.md`: the earlier NaN-filler search found nothing at the start of this
|
||
range and was about to conclude the body had zero length, which would have been
|
||
wrong. The last shrub class is a real class with real data, and the NaN filler
|
||
only appears in degenerate ones. Checking the actual byte pattern rather than
|
||
concluding from a failed heuristic is what caught it.
|
||
|
||
### First structure in the region after shrub
|
||
|
||
**Range measured:** level 0, `16047040 .. 16737344` (690,304 bytes), starting
|
||
at the first byte after the last shrub class's 240-byte body. Nothing outside
|
||
that range is read.
|
||
|
||
**Method:** reading the bytes at `+0x20` as `u32` shows ten consecutive
|
||
entries whose low halves increase by a constant **0x94 (148)**, followed by a
|
||
zero. Read as u32 the high half is `0xf000` for the first nine and `0x0a40`
|
||
for the tenth, so the table is ten entries long and ends at `+0x48`
|
||
(absolute `16047112`).
|
||
|
||
**Result, stated plainly:** a ten-entry table on a 148-byte stride. What the
|
||
entries mean, what 148 refers to, and why there are ten, are all unestablished.
|
||
Ten does not match any count already known for this level -- 238 moby
|
||
classes, 74 tie, 7 shrub, 93 tfrag textures -- so this is not yet tied to any
|
||
structure elsewhere in the lump.
|
||
|
||
### Follow-up: the 148 stride is a coincidence
|
||
|
||
Treating the ten entries' low halves as offsets from the region start lands on
|
||
blocks exactly 148 bytes apart, the first nine of which begin `08 00` followed
|
||
by a 16-bit id stepping by 0x40. That is a convincing-looking structure and it
|
||
does not survive being extended: walking the same 148-byte stride across the
|
||
rest of the region, **only 14 of 4657 blocks** start with `08 00`. The first
|
||
nine agree and then the pattern breaks.
|
||
|
||
So the ten entries are not a pointer table. The constant stride held for
|
||
eight consecutive values, which is far more than chance should produce and far
|
||
less than a structure needs.
|
||
|
||
Recorded because the lead was committed as a lead and this is where it goes.
|
||
Nothing else about the region has been identified: it is real, non-zero data
|
||
of unknown type, bounded, and unopened beyond these ~150 bytes.
|
||
|
||
## The region after shrub has a repeating structure at stride 368
|
||
|
||
**Range measured:** level 0, `16047040 .. 16737344`, sampled as the first
|
||
65,536 bytes. Cross-checked on every level that has the region, each sampled
|
||
the same way from its own region start.
|
||
|
||
**Method:** autocorrelation. For each candidate stride, count how often a
|
||
4-byte sequence at offset *i* repeats at *i + stride*, as a fraction of
|
||
comparable positions. A random control of the same size is measured
|
||
identically and reads flat at 0.0000 for every stride, so the statistic
|
||
discriminates.
|
||
|
||
**Result:**
|
||
|
||
| stride | region match rate |
|
||
|---|---|
|
||
| 368 | 0.1110 |
|
||
| 340 | 0.0935 |
|
||
| 337–339 | 0.026–0.029 (background) |
|
||
| random control | 0.0000 |
|
||
|
||
Stride **368** is the strongest peak. On level 0, 340 is a secondary peak
|
||
well clear of the background; the two differ by 28, which is unexplained.
|
||
|
||
**Cross-check:** the peak stride was recomputed independently for every level
|
||
that has the region, and **all 24 peak at exactly 368**. That is the part
|
||
that makes this worth recording -- a single level's peak could be a property
|
||
of that level's data, but 24 of 24 agreeing is a format invariant.
|
||
|
||
What a 368-byte record *is* remains unestablished. This is the same position
|
||
the shrub body was in before differential analysis identified its records,
|
||
and the same method is the obvious next step.
|
||
|
||
### Differential analysis at stride 368: no constant substructure
|
||
|
||
**Range measured:** level 0, `16047040 .. 16737344`. 1,875 whole 368-byte
|
||
records from the region start; variation checked across the first 60.
|
||
|
||
**Method:** the same differential analysis that identified the shrub body --
|
||
compare records byte by byte and record which positions differ.
|
||
|
||
**Result:** **all 368 positions vary.** There is no constant field, no padding
|
||
and no header inside a record. Unlike the shrub body, where differential
|
||
analysis immediately separated a varying payload from a constant two-byte
|
||
tail, this format has nothing constant to separate.
|
||
|
||
That does not contradict the autocorrelation peak. Checking which byte
|
||
positions carry the 368-stride matches over 20,000 positions: 168 exact 4-byte
|
||
repeats, distributed evenly at 0.84% across all four byte offsets. So whole
|
||
4-byte sequences recur at exactly 368 while no individual position is constant
|
||
-- consistent with records containing entries that repeat at a fixed relative
|
||
index, rather than with a fixed-layout record.
|
||
|
||
**So the 368 stride stands and the records remain unidentified.** The shrub
|
||
method failed here not because it was the wrong tool but because it needs
|
||
something constant to latch onto, and there is nothing.
|
||
|
||
## Sky opens with a 256-byte-row image
|
||
|
||
**Range measured:** level 0, `2192576 .. 2615232` (the sky region, 422,656
|
||
bytes). Periodicity start and end located by sliding an 8 KB window in 256-byte
|
||
steps through the region and measuring the 256-stride match rate.
|
||
|
||
**Method:** autocorrelation as above, then a periodicity scan. A random
|
||
control of the same size reads flat at 0.0000 for every stride tested.
|
||
|
||
**Result:**
|
||
|
||
- Peak stride in the first 64 KB is **256** at a 0.772 match rate. Strides 255,
|
||
257 and 4 score nearly as high, which is what a true period looks like
|
||
smeared by sampling.
|
||
- Across levels the peak is 256 on 21, 512 on 5 and 128 on 1. 512 and 128 are
|
||
a harmonic and a sub-multiple, so the fundamental period is **256 bytes**.
|
||
|
||
**Correction to the control.** The original comparison used uniform random
|
||
bytes, which read flat at 0.0000 -- and made the 0.772 look like an enormous
|
||
signal. It is not: 91.8% of the region's bytes are the same value (`0x07`), so
|
||
a stride-256 comparison matches trivially wherever both positions are `0x07`.
|
||
|
||
The right control is the same bytes shuffled, which preserves the value
|
||
distribution and destroys only the ordering. That control reads **0.512** at
|
||
stride 256, not 0.0000. The true excess is 0.772 − 0.512 = **+0.26**, a real
|
||
signal and a far smaller one than the original framing suggested.
|
||
|
||
**And the extent is wrong too.** Re-measured against the matched control, the
|
||
excess over the value distribution is:
|
||
|
||
| offset | excess |
|
||
|---|---|
|
||
| +0 .. +49152 | +0.38 .. +0.45 |
|
||
| +53248 | +0.26 |
|
||
| +57344 | +0.07 |
|
||
| +61440 | −0.13 |
|
||
|
||
So the 256-periodic data ends between `+53248` and `+57344` — somewhere in
|
||
rows 205 to 224 — **not** at `+60160` as first reported. The original figure
|
||
came from a raw threshold with no distribution control, and 91.8% of one byte
|
||
value drags that threshold wherever the dominant run ends.
|
||
|
||
The 256-byte period itself stands, cross-checked on 21 of 27 levels with the
|
||
remaining levels showing its harmonic and sub-multiple. The image dimensions
|
||
do not: an earlier "256x236" is wrong and should not be used.
|
||
|
||
So the sky region is not one thing. Its first ~60 KB is an image; the rest is
|
||
whatever the layer offsets address. Both halves still need work, and the
|
||
image is the tractable one.
|
||
|
||
## Re-verified: the 368 stride, against a matched control
|
||
|
||
The sky autocorrelation was found to be inflated by a control that did not
|
||
match the data, so the 368 result was re-checked the same way.
|
||
|
||
**Range measured:** each level's post-shrub region start, first 65,536 bytes.
|
||
|
||
**Method:** as before, but the control is the same bytes shuffled rather than
|
||
uniform random bytes. Shuffling preserves the value distribution and destroys
|
||
only the ordering, which is exactly what the control needs to do.
|
||
|
||
**Result:**
|
||
|
||
- Level 0's region is evenly distributed: 256 distinct byte values, most
|
||
common only 15.6%. Uniform random was therefore the *appropriate* control
|
||
here, and the shuffled control independently confirms it, reading **0.0000**
|
||
at strides 340, 368 and 4.
|
||
- Level 0 peak remains 368 at 0.1110, against 0.0000 for the matched control.
|
||
- Recomputed across all levels with matched controls: **24 of 24 peak at
|
||
exactly 368, every one with excess above 0.05.**
|
||
|
||
So the 368-byte period stands, now with a control appropriate to its data.
|
||
The contrast with sky is the point: the same statistic on the same kind of
|
||
question gave a real answer in one region and an inflated one in the other,
|
||
and the only thing that told them apart was whether the control matched the
|
||
byte distribution.
|
||
|
||
## Sky sub-objects: structure is consistent, stride is not
|
||
|
||
**Range measured:** every level's sky region, from the count field at `+0x0c`
|
||
through the sub-object offsets at `+0x20`. Each sub-object measured over its
|
||
first 16,384 bytes.
|
||
|
||
**Method:** as above, peak-stride autocorrelation per sub-object with a
|
||
matched control (the same bytes shuffled, seeded per level and index).
|
||
|
||
**Field note:** the sub-object count is a `u32` at **`+0x0c`**, not `+0x0a`.
|
||
Reading `+0x0a` gives 327,680 on level 0, which is not a count.
|
||
|
||
**Result, level 0:** five sub-objects at offsets 365760, 388880, 396432,
|
||
406784 and 414032, of lengths 23120, 7552, 10352, 7248 and 1536. Each has
|
||
internal structure well clear of its matched control.
|
||
|
||
**But the strides do not repeat across levels.** Recomputing per level:
|
||
|
||
| sub-object | peak strides seen across levels |
|
||
|---|---|
|
||
| 0 | 4 (x6), 8 (x13), 32, 288 |
|
||
| 1 | 16 (x3), 32 (x6), 288 |
|
||
| 2 | 8, 16 (x5), 256, 288 |
|
||
| 3 | 8, 16 (x6), 216 |
|
||
| 4 | 8, 16 (x3), 512 (x4) |
|
||
|
||
So the earlier per-index stride table, read off level 0 alone, was
|
||
over-fitted to one level. What is consistent is weaker and more honest: every
|
||
sub-object sits above its matched control, and most peak at a small stride of
|
||
4, 8, 16 or 32, which suggests 16- or 32-bit records whose stride varies with
|
||
content rather than being fixed.
|
||
|
||
**Level 2's header remains genuinely different**: its count field at `+0x0c`
|
||
reads 196,617 and is unusable, while every other level yields a plausible
|
||
count. Whatever makes it different is still unexplained.
|
||
|
||
### Sky sub-object 0: sparse, with duplicated entries
|
||
|
||
**Range measured:** level 0, sky region `+365760 .. +388880`, 23,120 bytes,
|
||
read as 2,890 records of 8.
|
||
|
||
**Method:** record composition, plus the adjacent-pair and zero-density
|
||
statistics below. The matched-control autocorrelation puts the peak at stride
|
||
8 (0.217 real, 0.000 shuffled), so 8 is the unit to describe.
|
||
|
||
**What it is made of:**
|
||
|
||
- **1,185 distinct `u32` values** across 5,780 words.
|
||
- **Mean 6.99 of 8 bytes are live** -- the previous revision of this file said
|
||
"zero", which is inverted and wrong. The live-byte histogram is 0:15, 2:1,
|
||
4:31, 5:238, 6:461, 7:1030, 8:1114, so most records are nearly full rather
|
||
than sparse.
|
||
- **549 of 2,890 adjacent pairs (19%) are exact duplicates of each other.** The
|
||
most common 8-byte record, appearing 283 times, is `d7d7d780 d7d7d780` -- a
|
||
4-byte value stored twice. Others follow the same shape: `00051980 00051980`,
|
||
`00082580 00082580`.
|
||
|
||
**Byte 7 is anomalous.** It takes only **14 distinct values** where bytes 0-6
|
||
take 142-204. Its common values are `0x00` (1378), `0xff` (806) and `0x80`
|
||
(592) -- the high bytes of signed 16-bit values, which is consistent with
|
||
byte 7 being an MSB of something rather than ordinary data. Unexplained.
|
||
|
||
**Not decoded.** A signed-16-bit reading was tried and does not obviously
|
||
help: the values span -32743..32633, close to the full range, with no
|
||
per-position structure (533, 371, 401 and 187 distinct at positions 0-3).
|
||
Recorded as a characterisation, with its range, so the next attempt starts
|
||
from the composition rather than re-deriving it.
|
||
|
||
### Sky sub-object 0 is a slowly-varying sequence, not independent records
|
||
|
||
**Range measured:** level 0, sky region `+365760 .. +388880`, 23,120 bytes.
|
||
|
||
**Method:** Pearson correlation between the byte sequence and itself at
|
||
increasing lag, against a shuffled control. This measures whether consecutive
|
||
records are *related*, which the position-wise distinct-value counts could
|
||
not show.
|
||
|
||
**Result:**
|
||
|
||
| lag (bytes) | correlation |
|
||
|---|---|
|
||
| 1 | +0.050 |
|
||
| 4 | +0.560 |
|
||
| **8** | **+0.746** |
|
||
| 16 | +0.666 |
|
||
| 32 | +0.602 |
|
||
| 64 | +0.432 |
|
||
| 128 | +0.140 |
|
||
| 256 | −0.044 |
|
||
| **512** | **+0.304** |
|
||
| 1024 | +0.247 |
|
||
| shuffled control (lag 1) | +0.003 |
|
||
|
||
The profile has a peak at lag 8, decays to a zero crossing near 256, and then
|
||
a second, smaller peak at 512 before decaying again.
|
||
|
||
This is what settles the earlier "no constant substructure" reading. The
|
||
records do not share fixed fields, but they are *not independent*: consecutive
|
||
8-byte records are strongly related, and that relation repeats at 512 bytes,
|
||
i.e. every 64 records.
|
||
|
||
Slowly varying values, related within a record, repeating every 64 records, is
|
||
the shape of a keyframe or lookup table -- which is what one would expect to
|
||
find in sky data, where a colour or fog parameter is sampled rather than
|
||
computed per vertex.
|
||
|
||
**Still not decoded.** Nothing here names the values or the sampling. But the
|
||
structure is now characterised well enough that a decode attempt has a shape
|
||
to aim at rather than a field to hunt for.
|
||
|
||
### Sky sub-objects share one shape, across every level
|
||
|
||
**Range measured:** sub-objects of the sky region on levels 0, 1, 3, 7, 13
|
||
and 26 -- 25 sub-objects in total, each measured over its whole length.
|
||
|
||
**Method:** as above, correlation at lag against the byte sequence.
|
||
|
||
**Result:**
|
||
|
||
| quantity | range over 25 sub-objects |
|
||
|---|---|
|
||
| lag 8 | +0.53 .. +0.79 |
|
||
| lag 64 | +0.16 .. +0.63 |
|
||
| lag 512 | -0.13 .. +0.42 (not universal) |
|
||
| lag 1024 | -0.05 .. +0.37 (not universal) |
|
||
|
||
**Lag 8 and lag 64 are positive on every sub-object of every level tested.**
|
||
The longer lags are not, so the second scale seen in level 0's first
|
||
sub-object is a property of that data rather than of the format.
|
||
|
||
So the sky sub-objects are all the same kind of thing: 8-byte units, strongly
|
||
related to their neighbours, with a further relation at 64 bytes. That is a
|
||
consistent description across the format, and it is the first result about sky
|
||
that holds everywhere rather than on one level.
|
||
|
||
**Not decoded.** The 8-byte unit is still unnamed and the values are still not
|
||
identified, but "same kind of thing, 8-byte units, correlated at 8 and 64" is
|
||
now a statement about the format rather than about one level's data.
|
||
|
||
## Sound bank SBlk has a 40-byte period (27/27 levels)
|
||
|
||
**Range measured:** each level's `SBlk` region -- the descriptor table between
|
||
`sblk_offset` and `daw_offset`, from the sound bank header rather than from
|
||
`core_data`. 17-20 KB per level.
|
||
|
||
**Method:** peak-stride autocorrelation with a matched control (the same bytes
|
||
shuffled, same seed, every level).
|
||
|
||
**Result:** the peak is at **exactly 40 bytes on all 27 levels**, every one with
|
||
excess above 0.2. The peak is also sharply isolated on level 0 -- 0.454 excess
|
||
at stride 40 against 0.207 to 0.278 for strides 36 to 44 -- so it is a period
|
||
rather than a broad tendency.
|
||
|
||
That is the first structural result about the sound bank beyond the container
|
||
framing already committed.
|
||
|
||
**Record count:** 487, 443, 428, 494 and 447 whole 40-byte records on levels 0,
|
||
1, 2, 13 and 26. The region is not an exact multiple of 40 on any level
|
||
(19484 leaves 4 bytes over), so 40 is a period rather than a record size with
|
||
no remainder.
|
||
|
||
**Unexplained:** which byte positions are constant varies by level -- level 0
|
||
has 7, 11, 31 and 35 constant; level 1 has 15, 19, 23, 35 and 39; level 2 has
|
||
7, 23, 27 and 31. A fixed 40-byte layout would have the same constant
|
||
positions everywhere, so either the records have a variable tail or 40 is a
|
||
period in something else that the records sit inside.
|
||
|
||
Also measured, for the record:
|
||
|
||
- `DAW`, the sample region, shows **no** stride above threshold against its
|
||
control. Consistent with ADPCM nibble-packed audio, and consistent with the
|
||
container framing being the only part worth decoding.
|
||
- `gameplay.bin` peaks at 16 bytes (excess 0.267 at strides 16, 32 and 64).
|
||
- `gs_ram.bin` has almost no stride structure (0.043 at 32) but a lag-8
|
||
correlation of +0.503, so it is smooth rather than tabular.
|
||
|
||
## Tie class spans: 16-byte unit, 48-byte period (dominant, not universal)
|
||
|
||
**Range measured:** every tie class span on levels 0, 3, 6 and every third
|
||
level after it -- 386 spans of 16 KB or more, bounded by consecutive class
|
||
offsets.
|
||
|
||
**Method:** peak-stride autocorrelation with a matched control, over the first
|
||
32,768 bytes of each span.
|
||
|
||
**Result:**
|
||
|
||
| peak stride | spans |
|
||
|---|---|
|
||
| 48 | 285 |
|
||
| 16 | 58 |
|
||
| 64 | 29 |
|
||
| 96 | 11 |
|
||
| 24 | 2 |
|
||
| 32 | 1 |
|
||
|
||
**48 bytes dominates but is not universal**, and the weaker peak is 58 spans
|
||
against 285 -- so unlike the sound bank's 40, this is a majority rather than a
|
||
constant.
|
||
|
||
Every observed peak is a multiple of 16 (16, 24, 32, 48, 64, 96), which
|
||
suggests a **16-byte base unit** with a 48-byte period arising from how those
|
||
units group, rather than 48 being the record itself.
|
||
|
||
This is weaker evidence than the sound bank's 40-byte result, which was
|
||
sharply isolated and identical on all 27 levels. Here the peak is weaker
|
||
(0.09-0.22 excess against sound bank's 0.45) and it moves between levels and
|
||
spans. Recorded at the confidence it deserves.
|
||
|
||
## Two independent measurements agree on moby's 16-byte vertex
|
||
|
||
The moby part table established `vertex_offset + 16 * vertex_count ==
|
||
next_part` on 43,467 of 43,467 parts, which fixes the vertex stride at 16.
|
||
Autocorrelation over the spans reaches the same number without using that
|
||
result at all.
|
||
|
||
**Range measured:** moby class spans of 16 KB or more on level 0 -- 60 spans,
|
||
each bounded by consecutive class offsets, measured over the first 32,768
|
||
bytes.
|
||
|
||
**Method:** peak-stride autocorrelation against a matched control, searching
|
||
strides 4 to 64.
|
||
|
||
**Result:**
|
||
|
||
| peak stride | spans |
|
||
|---|---|
|
||
| **16** | **43** |
|
||
| 8 | 8 |
|
||
| 48 | 4 |
|
||
| 32 | 3 |
|
||
| 24 | 2 |
|
||
|
||
16 dominates, and it is the same 16 the part table produces from a completely
|
||
different argument -- a tiling identity over part records rather than a
|
||
periodicity measurement over span contents.
|
||
|
||
Two unrelated methods agreeing is worth more than either alone, because the
|
||
autocorrelation would be vulnerable to exactly the kinds of artifact this
|
||
project has already been bitten by: a dominant byte value would inflate it,
|
||
and a period that happened to divide the window would look the same as a real
|
||
one. Neither applies here, and the tiling identity cannot be inflated at all.
|
||
|
||
This is the strongest form of evidence produced in this project: a structural
|
||
invariant confirmed by two methods that do not share a failure mode.
|
||
|
||
## Sound bank SBlk: a 12-byte marker that does not explain the 40-byte period
|
||
|
||
**Range measured:** level 0 `SBlk`, 24..19508 (19,484 bytes).
|
||
|
||
**Method:** positions of the `0x7f`/`0x47f` marker words, and their spacing.
|
||
|
||
**Result:** those markers sit at **12-byte spacing on 144 of roughly 180
|
||
occurrences**, with 24-byte and larger gaps accounting for the rest. So there
|
||
is a 12-byte structure in the data.
|
||
|
||
**But that contradicts the autocorrelation.** If records were 12 bytes, stride
|
||
12 should be the peak. It is not:
|
||
|
||
| stride | excess over shuffled |
|
||
|---|---|
|
||
| 12 | +0.237 |
|
||
| 20 | +0.175 |
|
||
| 24 | +0.237 |
|
||
| **40** | **+0.454** |
|
||
| 48 | +0.239 |
|
||
|
||
40 is not a multiple of 12, so the two measurements disagree rather than one
|
||
explaining the other. Recorded as an open discrepancy rather than resolved in
|
||
favour of either: the marker spacing says 12, the period says 40, and 40 being
|
||
twice as strong is the more surprising of the two.
|
||
|
||
### Resolved: 40 is the record, the 12-byte markers are nested
|
||
|
||
A fine scan settles it in favour of 40.
|
||
|
||
| stride | excess |
|
||
|---|---|
|
||
| **40** | **+0.454** |
|
||
| 39 | +0.266 |
|
||
| 41 | +0.215 |
|
||
| 38 | +0.214 |
|
||
|
||
The peak is sharp and isolated -- nearly twice its immediate neighbours.
|
||
And the multiples confirm it: 80 scores +0.376, 120 +0.394, 160 +0.340,
|
||
while the multiples of 12 (24, 48, 96, 144) all sit at 0.21-0.24.
|
||
|
||
So the record is **40 bytes**, and the `0x7f`/`0x47f` markers at 12-byte
|
||
spacing are a nested structure *inside* those records rather than the record
|
||
boundary. Their spacing is irregular -- 12, 24 and larger -- which is exactly
|
||
what one would expect of a field that is present in some records and absent in
|
||
others.
|
||
|
||
The earlier "they disagree" framing was wrong in one respect: the marker
|
||
spacing never claimed to be the record size. Read as a claim about markers
|
||
inside records rather than about the record itself, the two measurements were
|
||
never in conflict.
|
||
|
||
## SBlk record layout at 40 bytes
|
||
|
||
**Range measured:** level 0 `SBlk`, 24..19508, aligned at +0 — 487 whole
|
||
records.
|
||
|
||
**Method:** for each of 40 candidate alignments, record the number of byte
|
||
positions that are near-constant across records, and keep the best.
|
||
|
||
**Result:** alignment +0 is best, with 10 near-constant positions. Four are
|
||
exactly constant and all zero: positions **7, 11, 31 and 35**.
|
||
|
||
Distinct values per byte position across the 487 records runs from 1 to 155,
|
||
with position 29 the most variable by a wide margin (155 against 21-48
|
||
elsewhere) and positions 2, 10, 27, 34 and 39 low (2 to 8).
|
||
|
||
The first record carries the `SBlk` magic and the `DAW` tag rather than
|
||
descriptor data, so it is a header rather than an entry — which is worth
|
||
accounting for before reading any field at a fixed offset.
|
||
|
||
**Not decoded.** The layout is established and the constant positions are
|
||
known; the fields are not named, and which of the 40 bytes are an index, a
|
||
length and an offset is still open.
|
||
|
||
## SBlk: the field values do not read as record fields
|
||
|
||
**Range measured:** level 0 `SBlk`, 24..19508.
|
||
|
||
**Method:** read at 4-byte alignment and rank the most common `u32` values.
|
||
|
||
**Result:** the common values are `0x0` (2261), `0x1` (369), `0x7f` (168),
|
||
`0x5a000000` (125), `0x1eff7f7f` (125), `0x1dfcc` (124), `0x19fc0` (93),
|
||
`0x7c34240d` (91), `0x7c3821c8` (91).
|
||
|
||
**This does not look like a table of records with typed fields.** A 40-byte
|
||
record layout whose fields are an index, a length and an offset would not
|
||
produce `0x1eff7f7f` and `0x7c34240d` a hundred times each, nor would those
|
||
values survive a byte shift.
|
||
|
||
So either the 40-byte period is not a record boundary, or the region is an
|
||
encoded stream -- something like varint or Huffman-coded data -- that happens
|
||
to have a 40-byte statistical period.
|
||
|
||
The `0x7f` values are suggestive: in a varint encoding a byte of `0x7f` is the
|
||
last value representable in seven bits, so it appears as a frequent
|
||
saturated byte. That is consistent with a coded stream and inconsistent with
|
||
a fixed-layout record.
|
||
|
||
**Left open.** The 40-byte period is solid (27/27 levels, sharply isolated, its
|
||
multiples confirm it), and the alignment with four constant zero positions is
|
||
solid. What sits in those bytes is not a record layout, and the two findings
|
||
are in tension in a way this project has not otherwise seen: a confirmed
|
||
period that is not a record boundary.
|
||
|
||
## Correction: the 368 stride is real, but confined to one ~15 KB window
|
||
|
||
The claim earlier in this file that the post-shrub region has a 368-byte
|
||
period *throughout* is wrong. Re-measured over the whole region, the
|
||
autocorrelation curve is flat and near noise:
|
||
|
||
| stride | excess over matched shuffled control |
|
||
|---|---|
|
||
| 8 | +0.027 |
|
||
| 12 | +0.023 |
|
||
| 148 | +0.009 |
|
||
| 340 | +0.018 |
|
||
| **368** | **+0.018** |
|
||
| 632 | +0.020 |
|
||
|
||
Nothing peaks at 368. The figure of 0.1110 recorded earlier does not
|
||
reproduce under any sampling convention tried (step 1, 2 or 4 through the
|
||
window; all give ~0.018).
|
||
|
||
**Where the structure actually is.** Scanning 8 KB windows at 1 KB steps for
|
||
a 368-excess above 0.15 finds exactly one substantial region:
|
||
|
||
region+41984 .. region+57344 = 15,360 bytes (abs 16089024..16104384)
|
||
|
||
plus a 4 KB patch at region+650240. Everything between and around is flat.
|
||
|
||
**Within that window the 368 stride is unambiguous:**
|
||
|
||
| stride | excess |
|
||
|---|---|
|
||
| **368** | **+0.315** |
|
||
| 736 | +0.284 |
|
||
| 364 | +0.024 |
|
||
| 372 | +0.021 |
|
||
| 732 | +0.027 |
|
||
|
||
The peak is sharp, 736 confirms it as a multiple, and the neighbours are at
|
||
noise.
|
||
|
||
**Record shape.** 41 whole 368-byte records. Mean distinct values per
|
||
position is 16 of 41, and **nine positions are exactly constant and zero**:
|
||
329 through 336, and 340. Twenty-five positions have 3 or fewer distinct
|
||
values, clustered in 325-349 -- a tail region rather than scattered padding.
|
||
|
||
So the corrected statement is: a 15,360-byte window of the post-shrub region
|
||
is a table of 41 records at stride 368, each with a roughly 43-byte tail
|
||
whose fields are mostly zero. The surrounding 675 KB is not that table.
|
||
|
||
**What went wrong before.** The original measurement was taken over a window
|
||
where 368 held and then reported as a property of the region. The 15 KB
|
||
window holds 41/1841 of the region's records, and averaging that over the
|
||
whole region flattens the peak to the noise floor. The re-verification pass
|
||
checked the stride but not whether it applied uniformly, so it confirmed the
|
||
number without testing the claim it was attached to.
|
||
|
||
## The post-shrub region is a chunk container (explains the 368 stride)
|
||
|
||
**Range measured:** level 0 post-shrub, 16047040..16737344 (690,304 bytes).
|
||
|
||
**Method:** scan for the block shape `16-byte header ending 0xff` followed by
|
||
10 zero bytes followed by a run of `u32` whose top nibble is `0xF`.
|
||
|
||
**Result: 28 chunk headers on level 0**, present on all 27 levels. Each chunk
|
||
is:
|
||
|
||
+0x00 16 bytes, last byte 0xff (opaque; differs per chunk)
|
||
+0x10 10 zero bytes
|
||
+0x1a N tagged offsets, top nibble 0xF
|
||
|
||
and within a chunk the offsets step by a **chunk-specific stride** — 148,
|
||
340, 368, 288, 320, 236, 272, 384, 116, 160, 250, 580, 576 all occur on
|
||
level 0. The stride is near-constant within a chunk (e.g. 148,148,148,148...
|
||
then 340,340,336,340...) but differs between chunks.
|
||
|
||
**This resolves two earlier results.**
|
||
|
||
The "368-byte period" is one chunk's stride, not a property of the region.
|
||
That is why the region-wide autocorrelation is flat: the region holds chunks
|
||
with ~15 different strides averaging out to noise. The 15,360-byte window where
|
||
368 scored +0.315 is the third chunk, at region+30066, whose stride is 368.
|
||
|
||
The "ten-entry table" I earlier disconfirmed is real — chunks carry 5 to 30
|
||
tagged offsets, with 10 the most common on level 0. It was disconfirmed because
|
||
I tested for a fixed stride of 148 across a whole region; the table exists,
|
||
but 148 is one chunk's stride among many.
|
||
|
||
**Two corrections this produces.** Both earlier claims were half-right: real
|
||
structure, wrong scope. I reported a local measurement as a regional one in
|
||
both cases, which is the same mistake twice.
|
||
|
||
**Not decoded.** What the 16-byte header is, what the `0xF` nibble tags, where
|
||
the chunk data lives, and what the chunk *means* are all open. The container
|
||
framing is established; its contents are not.
|
||
|
||
## Chunk records resolve, and the layout is consistent across levels
|
||
|
||
**Range measured:** each level's post-shrub region, taken as the 900,000 bytes
|
||
before the constant trailer.
|
||
|
||
**Method:** for each chunk, read the tagged offsets as `0xF0000000 | low24`,
|
||
take the stride as the first delta, load one record per offset at
|
||
`chunk_start + low`, and count byte positions that are constant across that
|
||
chunk's records.
|
||
|
||
**Result: 1,096 chunks and roughly 13,000 records resolve on all 27 levels.**
|
||
Every level's chunks show 16.7-20.0% of byte positions constant across their own
|
||
records — a consistent field layout rather than noise.
|
||
|
||
Level 0 in detail: 27 chunks resolve, 466 records, 20.8% constant. The
|
||
stride-368 chunk (77 records) has 26 constant positions out of 368; the
|
||
stride-340 chunk (77 records) has 25 out of 340. Constant positions cluster
|
||
near the record end — 267-276 and following for stride 368, 258-268 for stride
|
||
340 — which is a footer or tail, not scattered padding.
|
||
|
||
**The tag is uniform.** Every tagged offset seen has top byte `0xF0`, i.e.
|
||
`0xF0000000 | low24`, on all levels.
|
||
|
||
**What this settles.** The container framing from the previous entry is not a
|
||
coincidence of byte patterns: the offsets genuinely address records, and those
|
||
records share a field layout. The chunk-relative base is confirmed because
|
||
resolving at `chunk_start + low` yields records with ~20% constant positions,
|
||
which does not happen by chance.
|
||
|
||
**Still not decoded.** The 16-byte header (opaque, per-chunk), the meaning of
|
||
the `0xF0` tag, which fields are index/length/offset, and whether a chunk is
|
||
moby_anim or something else. A parser can extract chunks and records
|
||
bounds-checked today; naming fields needs a domain argument this project has
|
||
not yet found.
|
||
|
||
## The chunk header: byte 14 is the record count
|
||
|
||
**Range measured:** all 27 levels, 1,123 chunks, over the 900,000 bytes
|
||
before the constant trailer.
|
||
|
||
**Method:** compare header byte 14 against the number of offsets actually
|
||
resolved for that chunk.
|
||
|
||
**Result: byte 14 equals the record count on 1,123 of 1,123 chunks, zero
|
||
mismatches.** This is the first genuinely *decoded* field in this container
|
||
rather than a measured correlation.
|
||
|
||
The parser now truncates the offset table to header byte 14, so a stray
|
||
tagged word appearing in a chunk's record data cannot silently extend the
|
||
table -- which would otherwise let the count disagree with its own header.
|
||
|
||
**Two more header bytes are near-constant and look like ASCII:**
|
||
|
||
- byte 13: only two values across 1,123 chunks -- `0x45` (1,076) and `0x46`
|
||
(47).
|
||
- byte 9: three values -- `0x44` (26), `0x45` (1,084), `0x46` (13).
|
||
|
||
`0x44`, `0x45` and `0x46` are ASCII `D`, `E` and `F`. That three-letter tag
|
||
appearing at two different byte positions with different distributions is
|
||
suggestive of a type or class field, but it is not established: the remaining
|
||
14 header bytes are high-entropy and do not read as ASCII at any shift tried,
|
||
so the ASCII in bytes 9 and 13 could be incidental.
|
||
|
||
**Still open:** the 14 opaque header bytes, whether bytes 9 and 13 are type
|
||
tags, which record fields are index/length/offset, and whether a chunk is
|
||
moby_anim. The `D`/`E`/`F` values are suggestive enough to be worth testing
|
||
against a known type table, but that has not been done.
|
||
|
||
## The header is a type ID, and the type determines the stride
|
||
|
||
**Range measured:** all 27 levels, 1,123 chunks.
|
||
|
||
**Method:** group chunks by their 16-byte header; for each distinct header,
|
||
collect the set of strides observed across every level it appears on.
|
||
|
||
**Result: only 64 distinct headers across 1,123 chunks.** The header is a
|
||
type identifier, not a per-chunk descriptor -- the same 16 bytes recur on
|
||
every level.
|
||
|
||
**63 of the 64 headers imply exactly one stride.** So header to record layout
|
||
is very nearly a function: the type determines the record size.
|
||
|
||
**Resolved: the single exception is three types sharing a header.**
|
||
|
||
Header `7542eff271b522a79e45348cca4509ff` was reported with three strides, 76,
|
||
92 and 104. It is not one type with a variable stride, and not a misparse.
|
||
|
||
It appears **three times per level**, always with record count 9, always in the
|
||
same order and always at the same offsets within the region. On level 0 and on
|
||
every other level checked:
|
||
|
||
| offset | stride | count | first offset | last offset |
|
||
|---|---|---|---|---|
|
||
| 863842 | 92 | 9 | 928 | 1640 |
|
||
| 867554 | 76 | 9 | 912 | 1520 |
|
||
| 872866 | 104 | 9 | 1000 | 1832 |
|
||
|
||
The offsets are 3,712 apart each time and the strides differ, so these are
|
||
three separate chunks that happen to share 16 header bytes.
|
||
|
||
**Consequence for the type ID.** 64 distinct headers for 66 distinct
|
||
record layouts, not 64 for 64. The header is *nearly* a type ID but does not
|
||
uniquely determine the layout: whatever distinguishes these three is not in the
|
||
16 bytes the header captures. Candidates not excluded: a distinguishing byte
|
||
sits outside the 16 captured, or the layout varies within a type.
|
||
|
||
This does not change the earlier conclusion that 63 of 64 headers imply one
|
||
stride -- that remains 63 of 64 -- but the mapping is not a bijection and
|
||
should not be used as one.
|
||
|
||
**The header does not encode the stride.** All 90 two-byte windows of
|
||
`header[0..14]` were tested: none maps to a unique stride across every group.
|
||
So there is no simple two-byte field to read the record size from, and the
|
||
stride must come from the offset table, as the parser already does, by
|
||
differencing the first two tagged offsets.
|
||
|
||
That is the right implementation anyway, and this makes it more than a
|
||
convenience: given three chunks sharing one header with three different
|
||
strides, any dispatch that assumed "header implies size" would be wrong for
|
||
1,123 chunks' worth of type coverage on at least one layout. Reading the
|
||
stride from the offsets is the only interpretation consistent with all the
|
||
measurements.
|
||
|
||
**The ASCII reading is confirmed as a type tag.** Grouping by header bytes 9
|
||
and 13:
|
||
|
||
| tag | chunks | distinct strides |
|
||
|---|---|---|
|
||
| `DE` | 26 | 2 (184, 344) |
|
||
| `EE` | 1050 | 46 |
|
||
| `EF` | 34 | 2 (268, 640) |
|
||
| `FF` | 13 | 1 (520) |
|
||
|
||
The rare tags are tightly coupled to their layouts -- `FF` is a single stride
|
||
across all 13 of its chunks -- while `EE` is heterogeneous with 46. That
|
||
pattern is what a type tag should look like, and it settles the earlier
|
||
suggestion that the D/E/F bytes were incidental: they are not.
|
||
|
||
**What this is good for.** A chunk can now be dispatched on its header to a
|
||
record layout, and 63 of 64 headers determine that layout's size without
|
||
inspecting any record. That is a usable parse entry point, and it is the first
|
||
point in this container where a field is doing real work rather than merely
|
||
correlating.
|
||
|
||
**Still open:** what the 64 types mean, which record fields are
|
||
index/length/offset, and whether any type is moby_anim.
|
||
|
||
## RETRACTED: chunk records referencing moby class IDs at 38x chance
|
||
|
||
This section previously claimed that chunk records reference moby class IDs, with
|
||
one body slot per chunk type, at 38x-289x enrichment over chance, and that >= 20
|
||
chunk types showed it.
|
||
|
||
**That was a measurement of the parser, not of the disc.**
|
||
|
||
`resolve` sliced every record at the chunk's *first* offset delta. On 1,252 of
|
||
1,509 chunks the later deltas are smaller -- word-aligned multiples of 4, most
|
||
exactly +/-4 -- so those slices ran past the record's real end and carried **4 to
|
||
8 bytes of the following record**. A record with the head of the next one glued
|
||
to its tail scores as though it carried extra fields, and a slot scan across
|
||
those bytes lifts apparent hit rates without anything being true.
|
||
|
||
With record boundaries measured against each record's successor, and ragged
|
||
chunks skipped, what survives is:
|
||
|
||
chunk types with a varying class-ID-like slot 2 (was >= 20)
|
||
of those, ALSO spanning the ID space 0
|
||
|
||
So the general claim -- one class-ID slot per chunk type, across the container --
|
||
does not hold and is withdrawn.
|
||
|
||
### And the flaw was there from the start
|
||
|
||
The discriminator the original version lacked is simple: **a slot that really
|
||
carries moby class IDs must span the ID space**, so some of its values land above
|
||
a level's moby count. The artifact does the opposite.
|
||
|
||
type slot enrichment distinct values ABOVE the moby count
|
||
0xd76cc453 6 305x 9 0
|
||
0xf2ef4275 34 64x 9 341
|
||
0xf2ef4275 7 46x 16 412
|
||
0xf2ef4275 14 34x 16 432
|
||
0xd76cc453 9 32x 28 702
|
||
|
||
The 305x slot's nine values are `1, 3, 4, 5, 10, 11, 12, 13, 15` -- all under a
|
||
level's ~215 moby classes, so every one scores as a hit and 65536/215 = 305x.
|
||
Every slot that actually spans the ID space scores 10x to 64x, which is chance.
|
||
|
||
So the original claim's own numbers gave it away and it was not read that way:
|
||
**"5 to 42 distinct values at 153x to 289x"** is exactly the artifact's shape --
|
||
a handful of small values, none above the count. The record-boundary fix was
|
||
never what made this claim wrong. It was wrong when written, and the retraction
|
||
is complete rather than partial.
|
||
|
||
The test now separates the two: it counts slots clearing 100x (currently 1) and
|
||
slots that clear 100x *while also* having values above the moby count (currently
|
||
0). The second must stay empty; the first is expected to be the artifact. The
|
||
count above the moby count is tracked per **slot**, not per type, because a type
|
||
with one artifact slot and several ordinary slots must not have the ordinary
|
||
ones vouch for it.
|
||
|
||
The `distinct >= 4` filter this section introduced is still load-bearing and is
|
||
unaffected: without it the top slots are constants that read as 305x, because any
|
||
value below the moby count counts as a hit.
|
||
|
||
|
||
## Shrub body records on level 0: six varying bytes, two constant
|
||
|
||
**Range measured:** level 0, shrub class table 7 entries at core index offset
|
||
10176. Six classes are bounded by a successor; the seventh runs to the end of
|
||
`core_data` and is excluded, because its span contains the whole post-shrub
|
||
region and is not wholly records.
|
||
|
||
**Method:** bound each span by the next class's geometry offset, take the body
|
||
from `data_offset` to the span end, and split into 8-byte records.
|
||
|
||
**Result: 164 records across 6 bounded bodies**, body lengths 192 (twice), 208
|
||
(twice), 224, 240.
|
||
|
||
Per-byte, over those 164 records:
|
||
|
||
| byte | distinct | zero |
|
||
|---|---|---|
|
||
| 0 | 74 | 60 |
|
||
| 1 | 58 | 65 |
|
||
| 2 | 76 | 43 |
|
||
| 3 | 62 | 48 |
|
||
| 4 | 72 | 35 |
|
||
| 5 | 59 | 38 |
|
||
| **6** | **1** | **164** |
|
||
| **7** | **1** | **164** |
|
||
|
||
Bytes 6 and 7 are zero in every record. Bytes 0 to 5 take 58 to 76 distinct
|
||
values of a possible 164, with only 21-39% zero -- that is, uniformly varied
|
||
rather than sparse.
|
||
|
||
Read as `u32` the fields are not plausible: `u32[0]` ranges 0 to 4,294,871,316
|
||
and no field is monotone across the body. So the six bytes are not two packed
|
||
32-bit values.
|
||
|
||
**What this settles.** The six-varying-plus-two-constant shape holds on level 0
|
||
as a per-level partition rather than only in aggregate across the disc's
|
||
8,938 records, which is what rule 3a asks for. Two trailing zero bytes are
|
||
padding or a flag pair, not data.
|
||
|
||
**What it does not.** Resolved by a follow-up measurement: the six bytes are
|
||
not an `f32` pair either, and not narrow `u16`s.
|
||
|
||
- `f32 @0`: 158 of 164 finite, but the range runs to about 1e35. Not positions
|
||
or normals. The 6 non-finite values (3.7%) are far above the 0.05% that
|
||
random bytes would produce, so the field is *structured* but is not an `f32`.
|
||
- `f32 @4`: all 164 finite, range 0 to 9.18e-5. This looks promising until you
|
||
notice bytes 6 and 7 are always zero, which pins this value to a coarse grid
|
||
and explains the tiny range. It is not independent evidence.
|
||
- `u16 @0`, `@2`, `@4`: distinct counts 85, 90 and 80, each spanning essentially
|
||
the whole 0..65534 range. These are not small counts or indices.
|
||
|
||
So excluded are: two packed `u32`, an `f32` pair, and narrow `u16` fields. The
|
||
six bytes remain uninterpreted.
|
||
|
||
**Process note.** The first attempt at this used `ClassEntry` and a
|
||
hand-computed `o_class` offset instead of `ShrubClassEntry`, and every
|
||
`ShrubHeader::region` call failed. The parser was not wrong; the caller was.
|
||
The working call shape was already in the disc test.
|
||
|
||
## tie: both indirections, and one reading disproved
|
||
|
||
**The chain, which is two steps and easy to get wrong.** The tie class **table**
|
||
lives in the level's `core_index.bin`, not `core_data.bin`. On level 0 the three
|
||
tables chain as moby at `192`, 238 x 32 bytes later tie at `7808`, 74 x 32 later
|
||
shrub at `10176`, which closes. Reading the same table out of `core_data.bin`
|
||
produces plausible garbage rather than an error -- `offset_in_asset_wad` comes
|
||
back as 1,217,208,842, which is outside every buffer, so the mistake announces
|
||
itself if you look. Then `ClassEntry::offset_in_asset_wad` addresses
|
||
**`core_data.bin`**, despite its name and despite `core_data.wad` sitting
|
||
alongside it.
|
||
|
||
**Verified:** all **1,864** tie classes resolve, `o_class` agreeing on every one,
|
||
and all **5,592** LOD block offsets land inside `core_data`.
|
||
|
||
**Disproved:** the word at `+0` of a 16-byte unit inside an LOD block. On the
|
||
first twelve units of level 0's class 0 it rises strictly -- 288, 1520, 2848,
|
||
3968, 5088, 6096, 7312, 8528, 9792, 10976, 12016, 13072 -- which reads exactly
|
||
like a linked list, and the accompanying `i16`s looked quantised (two distinct
|
||
values in `i16[0]`, three in `i16[1]`).
|
||
|
||
Across the whole population it is strictly increasing on **0 of 1,864** blocks,
|
||
and `i16[0]` takes **8,969** distinct values on level 0 alone rather than two.
|
||
The sample was twelve units out of 1,724,334. Both the monotonic reading and the
|
||
quantisation were artifacts of that sample, and the disproof is asserted in
|
||
`crates/gc-extract/tests/disc.rs` so it cannot creep back.
|
||
|
||
**Record size: RETRACTED. Block size is a multiple of 16; the records are not a
|
||
uniform 16 bytes.** The earlier
|
||
"divisible by 16" figure was computed from the "next `packets` offset ends this
|
||
block" rule, which is unsound because **`packets` is strictly increasing on only
|
||
461 of 1,864 classes**. On the other 1,403 the two offsets are unordered, so they
|
||
cannot both be block starts into one ascending region.
|
||
|
||
Restricting to the 461 classes where the rule does hold: **922 of 922**
|
||
consecutive block sizes are a multiple of 16. That was taken as evidence for a
|
||
16-byte record. **It is not**, and the record-size claim is withdrawn.
|
||
|
||
What actually survives, and what killed it:
|
||
|
||
- Every one of those **922 blocks opens with a `u32` that is positive and a
|
||
multiple of 4** (922/922), ranging 48 to 1,744 with 17 distinct values across
|
||
level 0's LOD0 blocks. So the block opens with a count, not a size and not a
|
||
pointer.
|
||
- Reading that `u32` as a byte size and chaining to the next block lands exactly
|
||
on the next block start on **0 of 1,383** blocks. It is not a size chain.
|
||
- Assuming `[header][count][count uniform records]` and solving for the record
|
||
size: **no record size tiles the block for any header size up to 512 bytes.**
|
||
Not 16, not 12, not 8.
|
||
- Reading the block as uniform 16-byte units gives fields spanning their entire
|
||
legal range -- `u32@0` from -2,139,029,504 to 2,139,029,587, `u16@8` and
|
||
`u16@10` both 0..65,535, and 8,969 distinct values in `i16@0` alone on level 0.
|
||
Uniform units would not produce that.
|
||
|
||
So the 16 is a property of the **block size**, not of a record. The previous
|
||
commit's "record size is 16 bytes" was an inference from a divisibility fact that
|
||
does not carry it, and it is withdrawn rather than softened.
|
||
|
||
**`packets[0]` is a constant, not an offset.** It reads **256 on 1,864 of 1,864**
|
||
classes, while `packets[1]` and `packets[2]` vary widely and land at different
|
||
addresses in `core_data` on every class. Read as a block start, 256 points just
|
||
past the 0x48-byte header -- which is exactly how it was being used, and exactly
|
||
the kind of field that gets mistaken for a pointer. It is now
|
||
`TIE_PACKETS0` and nothing depends on it.
|
||
|
||
**Still not decoded:** what the 16 bytes hold. `i16[0]` at `+4` takes 8,969
|
||
distinct values on level 0 and is not the two-valued field a 12-unit sample
|
||
suggested; the word at `+0` is not a run offset.
|
||
|
||
## tie: the LOD block header, decoded
|
||
|
||
Over the **922** blocks where "the next `packets` offset ends this block" holds,
|
||
the block's own 16-byte header is measured. `gc_formats::tie::TieBlockHeader`
|
||
parses it and a real-disc test asserts every line below.
|
||
|
||
+0x00 u16 count multiple of 16 on 922/922; range 16..2096;
|
||
72 distinct values disc-wide
|
||
+0x02 u16 zero on 922/922
|
||
+0x04 u16 ONE OF FOUR VALUES DISC-WIDE: 0x501, 0xa02, 0xf03,
|
||
0x1404 -- low byte 1..4, high byte 5/10/15/20
|
||
+0x06 u16 15 distinct, 0x0206..0x0512
|
||
+0x08 u16 135 distinct, 0x0b05..0x4ee4
|
||
+0x0a u16 45 distinct
|
||
+0x0c u16 94 distinct, 0x010e..0x0553
|
||
+0x0e u16 zero on 914/922
|
||
|
||
**`+0x04` is packed, not a magnitude.** Four values across the whole disc, with
|
||
the two bytes moving independently (low 1-4, high 5/10/15/20 in lockstep), is two
|
||
small fields sharing a word. That it is a texture index paired with a scale is a
|
||
guess; that it is two fields is not.
|
||
|
||
**The count is not a record count.** `block_size - header` is divisible by it on
|
||
**0 of 922** blocks, for *any* record size and *any* header up to 512 bytes.
|
||
Combined with block sizes always being multiples of 16, that rules out a uniform
|
||
stride without a header.
|
||
|
||
## tie: the block is a table of 16-byte records (from RAM)
|
||
|
||
**Range measured:** every tie LOD block of level 1 whose leading word is a
|
||
non-zero multiple of 16 -- 16 blocks -- plus the two read by hand in full.
|
||
|
||
**Method:** locate each block's record table in PCSX2's RAM by its twelve field
|
||
bytes (the records are the only part of a block with a distinctive 12-byte
|
||
run), then compare every record against its disc copy. Each block has several
|
||
copies in RAM; the one that matches is the loaded copy.
|
||
|
||
**Result: on 16 of 16 blocks, every record's leading word has become
|
||
`table_address + word`, and the other twelve bytes are byte-identical to the
|
||
disc.** So:
|
||
|
||
- A block is `[16-byte header][record table][vertex payload]`, and the records
|
||
are **16 bytes**: a leading `u32` pointer and twelve bytes of fields.
|
||
- The header's word at `+0x00` is the **byte offset from the block start to
|
||
the vertex payload** -- that is, the length of the record table. This is what
|
||
the "count" reading was groping at: it is a multiple of 16 on all 2,177 blocks
|
||
because the table holds 16-byte records, it is never zero because there is
|
||
always at least the header-sized first record, and its 42 distinct values are
|
||
table lengths, not counts of anything in the payload.
|
||
- Every record's leading word is a pointer **relative to the start of the record
|
||
table**, relocated at load to `table + word`. The twelve field bytes are
|
||
copied verbatim.
|
||
|
||
This also explains the dead end that ran before it. The payload length divided
|
||
by the word at `+0x00` is a whole number on **0 of 922** blocks, at any record
|
||
size and any header up to 512 bytes -- which is the correct measurement of the
|
||
wrong hypothesis: the word never described the payload, it described where the
|
||
payload starts. A count of what follows a pointer is not the pointer.
|
||
|
||
`TieBlockHeader::data_offset` and `TieBlockHeader::records` carry it, bound by
|
||
the offset, and the disc test asserts the table is bounded on every block.
|
||
|
||
### The records are R&C1's `TiePacketHeader` (2026-10-11)
|
||
|
||
With the record table located and the fields isolated, the reference settles
|
||
what they are. **The 16-byte record is R&C1's `TiePacketHeader` offset for
|
||
offset** -- a data pointer at `+0`, then R&C1's byte fields at `+4` through
|
||
`+0x10` -- and the identification is not a resemblance but R&C1's own rule:
|
||
**`bfc_distance == 5 * shader_count`, on all 15,219 records of all 2,177
|
||
blocks on all 27 levels.**
|
||
|
||
Measured over the same set:
|
||
|
||
| relation | holds |
|
||
|---|---|
|
||
| `bfc_distance == 5 * shader_count` | **15,219 / 15,219** |
|
||
| `vert_ofs <= vert_size` | all |
|
||
| `scissor_size <= scissor_ofs` | all |
|
||
| `control_size <= control_count` | all |
|
||
| `color_ofs == vert_size` | 12,640 / 15,219 |
|
||
| `color_ofs == vert_ofs + vert_size` (R&C1's rule) | **0 / 15,219** |
|
||
| `color_count`, `strip_count`, `strip_vertex_count` zero | 15,089 / 15,077 / 15,078 |
|
||
|
||
`TieRecord` parses all twelve fields; `invariants_hold` asserts only the
|
||
universal four, and the disc test counts the rest. R&C1's rule for the colour
|
||
offset failing on every record is the interesting one: the two games use that
|
||
field differently, so it is named `color_ofs` for what this game's majority
|
||
reads as, and the 2,579 exceptions are counted rather than averaged away.
|
||
|
||
**A packet-length formula is retracted.** `(vert_ofs + vert_size +
|
||
scissor_ofs + scissor_size) * 16` was measured at 61 of 61 pairs and is not
|
||
reproduced on the correct blocks -- 0 of 13,042. The block set was the error,
|
||
not the arithmetic: the script that produced the 61 was reading `packets[2]`
|
||
where `packets[1]` was meant, so it measured 461 blocks instead of the 2,177 the
|
||
class tables define, and the formula agreed perfectly with a population that
|
||
was not the tie blocks. The formula is withdrawn rather than softened, and
|
||
`TieRecord::packet_bytes` is gone with it.
|
||
|
||
### It is now a command, and it verifies two levels
|
||
|
||
The check above was a script; it is now `gc-trace verify-tie-records
|
||
<unpacked-dir> <level> <class> <lod> <src>`, which resolves the class, parses
|
||
the block, searches the RAM image for the loaded record table and reports what
|
||
it found. Exit status is 0 when every record relocated with verbatim fields.
|
||
|
||
Run over every (class, lod) slot of two levels, from PCSX2 savestates:
|
||
|
||
| level | slots with a block | verified | mismatched | absent |
|
||
|---|---|---|---|---|
|
||
| 0 | 87 | **87** | 0 | 61 |
|
||
| 1 | 104 | **104** | 0 | 78 |
|
||
|
||
So the relocation rule holds on **every** LOD block of two levels that the game
|
||
had loaded. Two operational notes worth keeping:
|
||
|
||
- **The live emulator is not a stable input.** Verifying level 1 over PINE
|
||
matched 104 of 104 while the user was on that planet; the same command
|
||
minutes later, after they had returned to level 0, found the tables in RAM
|
||
but no matching copy -- because the level was no longer loaded. A miss there
|
||
is not a counterexample, and the report says "no copy matched" rather than
|
||
"failed".
|
||
- **A class index past the table count is an error**, not a mismatch, so a
|
||
sweep should stop at the level's class count rather than at a round number.
|
||
|
||
What the records' fields name and what the payload holds are still open.
|
||
|
||
## The tie packet's length, from its record's fields (2026-10-11)
|
||
|
||
**Range measured:** every tie LOD block on the disc -- 2,177 blocks, 13,042
|
||
consecutive record pairs.
|
||
|
||
**Method:** a record's pointer is where its packet starts, so the distance
|
||
between consecutive pointers **is** a packet's length, with no reference to any
|
||
field. Comparing that against the record's own fields gives the length rule.
|
||
|
||
**Result: `packet_quadwords = vert_ofs + vert_size + scissor_size`, on 12,927
|
||
of 12,927 record pairs whose `strip_count` is zero -- and on 0 of 115 whose it
|
||
is not.** The partition is exact: every record with a non-zero `strip_count` is
|
||
an exception and no record with a zero one is. The fourth term is 0 quadwords
|
||
whenever `strip_count == 0`, and something else (measured 0..19 quadwords, and
|
||
*not* equal to `strip_count`) whenever it is not. R&C1's `strip_count` is a
|
||
strip count and strips are variable-length data, so a non-zero count most
|
||
likely adds per-strip bytes this cannot see -- a hypothesis, labelled as one.
|
||
|
||
`TieRecord::packet_quadwords` and `TieRecord::packet_quadwords_are_known` carry
|
||
it; the latter is the honest gate, and the disc test asserts both sides of the
|
||
partition.
|
||
|
||
**This settles a dispute inside this file, and the retraction was the error.**
|
||
A formula of this shape was derived at 61 of 61 pairs, then "retracted" as not
|
||
reproducing -- re-measured at 0 of 13,042. The retraction was wrong: the
|
||
re-measurement had swapped `strip_count` for `scissor_ofs` in the fourth term,
|
||
and the blame went on the block set. Re-measured with the term the first pass
|
||
had, it matches on 12,927 of 13,042, and the 115 exceptions are precisely the
|
||
non-zero `strip_count` records.
|
||
|
||
The lesson is worth keeping because it is the same shape as two others here: a
|
||
retraction is not automatically more careful than the claim it retracts, and the
|
||
re-measurement that "disproved" this one changed the thing it was measuring.
|
||
The lesson on the other side is the one that saved it: the first measurement
|
||
was over 61 pairs, which is not a population -- and had it stayed at 61 the
|
||
formula would have looked universal. Both the small sample and the bad
|
||
re-measurement were near-misses in opposite directions.
|
||
|
||
What this unblocks: the payload can now be sliced per packet, which was the
|
||
prerequisite for decoding what it holds.
|
||
|
||
### What the payload is, and what retired the coordinate reading
|
||
|
||
The payload is **not random**, and that took a control to establish. Reading each
|
||
block as 3 x `f32` triples, 53-64% of reads land within +/-5000 of origin; a
|
||
uniform-random control of identical length gives **16%**. So the content really
|
||
is float-shaped.
|
||
|
||
But the rate is **flat across every start offset and every stride tried**
|
||
(0x10..0x40 start, 8/12/16/20/24 stride: 53-65% throughout). A statistic that
|
||
cannot distinguish the right stride from the wrong one cannot identify a stride,
|
||
which is rule 4's exact case. The flatness is the informative part: combined
|
||
with the count not tiling, it says the records are **variable length**.
|
||
|
||
A VIF1 command walk was implemented and tested against the blocks: it consumes
|
||
**0 of 88** exactly. So the payload is variable-length and not a VIF1 CPU-format
|
||
stream. Not WAD LZ77 either -- `wad::decompress` rejects the block at every
|
||
start offset tried.
|
||
|
||
Scanning every (start, stride) pair from 0x10 to 0x7f for byte positions with a
|
||
small distinct-value vocabulary -- the signature of integer index fields -- finds
|
||
**none**: every byte position takes all 256 values at every alignment. For float
|
||
payload that is expected, and it is why the field scan found nothing rather than
|
||
confirming anything.
|
||
|
||
**Still open: what the payload holds.** Established is that it is variable-length,
|
||
float-shaped, non-random, not LZ77, and not VIF1. Not established is any record
|
||
boundary.
|
||
|
||
## tie: the LOD block is self-delimiting, and `packets == 0` means absent
|
||
|
||
**The terminator.** Every tie LOD block ends with the byte `0xf6` followed by
|
||
zeros, on **2,177 of 2,177** real blocks across all 27 levels. That was worth a
|
||
control, because `0xf6` could have been a coincidence of byte frequency: it
|
||
occurs in payloads at **0.83x** the uniform rate, i.e. slightly *less* often
|
||
than chance, while the last non-zero byte is `0xf6` on 909 of 922 blocks in the
|
||
first subset tested against **17 of 922** for the same blocks cut at a random
|
||
point. So it is a terminator, not a common byte.
|
||
|
||
This is what makes the blocks parseable at all. `packets[1]` bounds a block on
|
||
only 461 of 1,864 classes, so the terminator is the only general way to find one.
|
||
|
||
**`packets == 0` means no block at that LOD.** Of the 3,728 (class, LOD) slots,
|
||
**2,177** point at a block and **1,551** are exactly zero. The zero cases are not
|
||
degenerate blocks -- they are absent ones.
|
||
|
||
### A measurement I got wrong, twice, in opposite directions
|
||
|
||
The first pass over this header reported **3,728 blocks, all terminating** and
|
||
`packed_texture` taking **842** distinct values, contradicting an earlier
|
||
four-value result. I retracted the four-value claim on that basis.
|
||
|
||
Both numbers were wrong, and they were wrong the same way. The scan did not skip
|
||
`packets == 0`; it computed `class_base + 0` and walked forward from the **class
|
||
body**, finding a terminator there by accident. That contaminated 1,551 of the
|
||
3,728.
|
||
|
||
With the zeros excluded, the slot takes **exactly four** values on all 2,177
|
||
blocks -- 0x501 on 1,069, 0xa02 on 720, 0xf03 on 350, 0x1404 on 38 -- and all
|
||
2,177 terminate.
|
||
|
||
So: the four-value claim was right, my retraction of it was wrong, and the
|
||
retraction was made on a number from the same contaminated set. A retraction is
|
||
not automatically better than the claim it retracts; it needs its own bounds.
|
||
|
||
## The located `0xf6` terminator is inside the last packet (2026-10-11)
|
||
|
||
**Range measured:** every tie LOD block on the disc, 2,177 blocks.
|
||
|
||
**Method:** the last packet's end is known two ways. Its length comes from its
|
||
record's `packet_quadwords`, and every *other* record's length is confirmed by
|
||
the next record's pointer -- so only the last packet of each table is
|
||
unconfirmed. Comparing its computed end against where `parse_tie_block` stopped:
|
||
|
||
**the last packet's computed end lands 2 to 16 bytes past the located `0xf6`,
|
||
on all 2,150 blocks measured.** The error is not a multiple of 16, so it is not
|
||
a length error: a packet ending exactly at the terminator would give a
|
||
quadword-aligned difference. The `0xf6 00` pair therefore sits *inside* the last
|
||
packet's data, 2 to 16 bytes before its end.
|
||
|
||
Two readings fit: either the terminator was located too early, or the last
|
||
packet's length over-predicts by an amount that isn't a multiple of 16. The
|
||
first is likelier, and the test asserts the measurement rather than the
|
||
reading. What still holds either way: the lengths are right for every record
|
||
but the last (12,927 of 12,927 record pairs), and the block *starts* are
|
||
RAM-confirmed. The block *extent* is what is short.
|
||
|
||
**Why the existing test missed it.** `2,177 of 2,177 blocks end on 0xf6` is
|
||
satisfied by a false terminator exactly as well as a true one -- it checks that
|
||
the byte it walked to *is* a `0xf6`, not that it is the block's end. A test that
|
||
confirms the shape it walked to will pass on the wrong byte.
|
||
|
||
The cost: `payload` and `padding` under-report, so anything computed from them
|
||
is correspondingly short.
|
||
|
||
### Two candidate fixes, both measured and neither clean (2026-10-11)
|
||
|
||
**Range measured:** every tie LOD block, 2,169 blocks with a known last packet.
|
||
|
||
**Candidate 1: the first `0xf6 00` at or after the last known packet's end.**
|
||
Fails. The offsets found are spread (1082, 1132, 1148, 1150, 1182, 1214 ...),
|
||
not concentrated, because the 115 records with a non-zero `strip_count` carry
|
||
packets whose length is unknown and therefore sit after the last known end.
|
||
The scan lands inside those.
|
||
|
||
**Candidate 2: the block's data ends at a long zero run.** Fails. Walking
|
||
forward from the last known packet's end to the first run of 8+ zero bytes, the
|
||
byte preceding the run is not dominated by `0xf6` (2, 4, 1, 64, 3, 32, 19 ...
|
||
across 2,169 blocks) and the runs themselves are mostly 8-31 bytes. The zero
|
||
padding after a block is short and irregular, so run length does not
|
||
discriminate the block's end.
|
||
|
||
Recorded as negative results because both are the obvious thing to try next
|
||
and both are dead ends. What would unblock it: a survey of which offsets *do*
|
||
bound blocks (the 461 that work may share a property), or the level loader's
|
||
own block-walking code, which needs the overlay relocation from Phase 3.
|
||
|
||
### Corrected header, over all 2,177 blocks
|
||
|
||
+0x00 count multiple of 16 on all; 42 distinct; never zero
|
||
+0x02 zero on all
|
||
+0x04 exactly four values: 0x501, 0xa02, 0xf03, 0x1404
|
||
+0x06 16 distinct, never zero
|
||
+0x08 180 distinct, never zero
|
||
+0x0a 61 distinct, never zero
|
||
+0x0c 125 distinct, never zero, clustered 0x043f..0x0443
|
||
+0x0e 11 distinct, zero on 2,162 -- the only non-universal field
|
||
|
||
`+0x0c` clusters around 1,087 to 1,091, close enough to a block length to be
|
||
worth stating, but it is **not** the length: the difference against the true
|
||
payload length scatters from -12 to -172.
|
||
|
||
## chunk records: the leading byte is a bucket index, not data
|
||
|
||
Characterised across **all 77 chunk types**, 1,509 chunks and 34,491 records on
|
||
all 27 levels.
|
||
|
||
**Byte 0 of every record is a small aligned index.** It takes 16 discrete values
|
||
per chunk type -- 0x00, 0x10, ... 0xf0 -- with near-uniform counts *within* each
|
||
type. Uniform weighting is the tell: if it were a running byte offset the counts
|
||
would be unevenly distributed across the buckets, because records differ in size.
|
||
|
||
```
|
||
type_id recs len byte0
|
||
0x007d7ec438 2430 328 0x00:162 0x10:162 0x20:162 ... 0xf0:135
|
||
0x00d3d8c4a9 2430 308 0x00:162 0x10:162 0x20:162 ... 0xf0:135
|
||
0x00429ec43d 2079 368 0x00:162 0x10:135 0x20:135 ... 0xf0:108
|
||
0x00d76cc453 810 108 0x00:27 0x08:27 0x10:27 ... 0xe8:27 <- steps by 8
|
||
```
|
||
|
||
**The alignment step is per type, not global.** The leading byte is a multiple
|
||
of 8 on **34,432 of 34,491** records, but a multiple of 16 on only **33,349** --
|
||
the 1,142-record difference falls in seven types, four of which step by 8.
|
||
|
||
59 records in three types break even the 8-byte rule. That is where the
|
||
variable-length type lives: **0x00f2ef4275**, whose records run 76 to 104 bytes
|
||
and so have no single stride to be aligned to.
|
||
|
||
**Not named.** What the bucket indexes is not established. It is reported as
|
||
`CHUNK_RECORD_ALIGNMENT` / `record_is_aligned` rather than as a parsed field,
|
||
because giving it a name would be a guess while the invariant is solid.
|
||
|
||
### A measurement I initially got wrong
|
||
|
||
The first version of the constant claimed a 16-byte multiple on 34,432 records.
|
||
34,432 is the **multiple-of-8** count; the multiple-of-16 count is 33,349. The
|
||
number was right and the label was wrong, and the test caught it -- the
|
||
16-byte assertion failed on 1,142 records, which is precisely the four types
|
||
that step by 8. Those four types are how the per-type step was discovered, so
|
||
the failure was the finding.
|
||
|
||
Worth stating plainly: this is the second time this session a correct number
|
||
carried a wrong label, after the tie four-value retraction. Both times the test
|
||
was the thing that caught it, which is the argument for asserting the number in
|
||
a test at all rather than only in prose.
|
||
|
||
## The chunk record's leading byte is a per-chunk phase counter
|
||
|
||
Follow-up to the alignment result, and it settles what the leading byte is.
|
||
|
||
Within one chunk the leading byte steps by a constant and wraps:
|
||
|
||
offsets: 1420 1748 2072 2392 2716 3044 3372 3700 4028 4352 ...
|
||
byte0: 0 16 32 48 64 80 96 112 128 144 ...
|
||
|
||
i.e. `byte0 == step * ((index + phase) mod (256 / step))`. Sweeping every
|
||
`(step, phase)` over all 27 levels:
|
||
|
||
fully explained: 1,262 of 1,509 chunks
|
||
by step: 8 -> 51 chunks, 16 -> 554, 32 -> 520, 64 -> 137
|
||
unexplained: 247 chunks across 11 chunk types
|
||
|
||
**So on 1,262 chunks the leading byte carries no per-record information.** It is
|
||
a deterministic function of the record's index and the chunk's phase. That makes
|
||
it a position marker, not data -- which is why the earlier "16 discrete values
|
||
with uniform counts" pattern showed up.
|
||
|
||
`ResolvedChunk::bucket_rule` returns the `(step, phase)` when the whole chunk
|
||
fits and `None` otherwise, so the 247 unexplained chunks are distinguishable
|
||
from the 1,262 explained ones and nothing skips a leading byte on the strength of
|
||
a rule that does not apply.
|
||
|
||
### RETRACTED: "every chunk has a uniform record length"
|
||
|
||
**Claimed: 1,509 of 1,509 chunks. True: 257 of 1,509.**
|
||
|
||
Length varies between chunks of the same type -- one type runs 76 to 104 bytes --
|
||
and on 1,252 chunks it varies *within* one as well.
|
||
|
||
**Both halves of that old entry were wrong, and both were mine.**
|
||
|
||
"Never within one chunk" was `resolve` slicing every record at the chunk's first
|
||
offset delta. Every record then came out the same length by construction, so the
|
||
predicate was confirming the parser back to itself.
|
||
|
||
And this section's *withdrawal* of "variable-length type" was wrong too. It read
|
||
the pooled observation as a partitioning artifact -- `minlen`/`maxlen` taken across
|
||
all of a type's records at once. Rule 3a is right about pooling in general, and it
|
||
was applied to a real signal here: the type genuinely has variable-length records,
|
||
and the per-chunk check that would have caught it was itself reporting the
|
||
parser's own slicing.
|
||
|
||
Two errors stacked. The pooling rule fired correctly and was overruled by a
|
||
measurement that had no independent content.
|
||
|
||
## The first TWO bytes encode the record's own index
|
||
|
||
Continuing from the phase counter. Within a chunk:
|
||
|
||
i : 0 1 2 3 4 5 6 7 8 9 10 11 ... 15 16 17 ...
|
||
byte0: 0 16 32 48 64 80 96 112 128 144 160 176 ... 240 0 16 32
|
||
byte1: 0 0 0 0 0 0 0 0 0 0 0 0 ... 0 1 1 1
|
||
|
||
`byte0` is `step * (j mod g)` and **`byte1` is `(j / g)`**, where `g = 256/step`
|
||
and `j = index + phase`. So the pair is the record's index in mixed radix:
|
||
`byte0` carries the low digits scaled by `step`, `byte1` the quotient.
|
||
|
||
**Measured over all 27 levels: 1,235 of 1,509 chunks** fit this on both bytes.
|
||
Byte 0 alone fits 1,262, so the 27-chunk difference are chunks whose first byte
|
||
follows the rule and whose second does not.
|
||
|
||
**So on 1,235 chunks a record knows its own position**, and the two leading bytes
|
||
hold no per-record information. `ResolvedChunk::index_rule` returns the
|
||
`(step, phase)` only when the *whole* chunk fits on both bytes, so the 274 that
|
||
do not stay distinguishable.
|
||
|
||
### A number that moved when the implementation changed
|
||
|
||
A first probe reported **1,181**; the committed method reports **1,235**. The
|
||
method is right. The probe accumulated `ok0`/`ok1` flags across candidate phases
|
||
with an early-exit that could skip a phase which would have matched both bytes,
|
||
and it also disagreed with `bucket_rule` on the byte-0 count (1,208 against
|
||
1,262), which is what gave it away.
|
||
|
||
Recorded because the two numbers came from the same run and only the discrepancy
|
||
exposed the probe as wrong. The lesson is narrow and worth stating: when a
|
||
one-off probe and a committed helper disagree, check the probe first.
|
||
|
||
## Chunk record bodies: what was ruled out
|
||
|
||
With the framing settled (leading two bytes = record index), the body is bytes
|
||
`2..len`. Four probes on it, mostly negative.
|
||
|
||
**No count field.** For every chunk type, every `u16` and `u32` position in the
|
||
first 64 body bytes, at element strides 1, 2, 3, 4, 6, 8, 12 and 16: does the
|
||
stored value equal `(record_len - offset) / stride` on 90% or more of records?
|
||
**No type produced a hit.** The record does not carry an element count near its
|
||
head.
|
||
|
||
**The body is dense, not float-dominated.** For the largest type (2,430 records,
|
||
326-byte bodies):
|
||
|
||
f32 reads within +/-5000 of origin : 74%
|
||
uniform-random control : 49%
|
||
zero-byte fraction : 13%
|
||
|
||
Above baseline but far short of what the tie blocks look like (53-64% against
|
||
16%, with 43% zero bytes). The chunk bodies are denser and less float-shaped than
|
||
tie geometry.
|
||
|
||
**There is an index-like region near the tail.** In the largest type, **12 of 163**
|
||
`u16` slots have a maximum below `0x1000` -- too small for float or half-float
|
||
data, so those slots look like indices or small counts. They cluster at slots
|
||
111-121 and 130-131.
|
||
|
||
### A sample-size correction
|
||
|
||
Dumping six records of one chunk showed slots 112-119 **zero on every record** and
|
||
a long run of other constant slots, which read as "the record is mostly a
|
||
per-chunk template". Measured properly across the 1,171 chunks with at least 8
|
||
records:
|
||
|
||
u16 slots constant across every record in their chunk : 15,422 of 164,390 (9.4%)
|
||
chunks whose records are ENTIRELY constant : 0 of 1,171
|
||
|
||
So 9.4%, and no chunk is a template. Six records looked like a majority because
|
||
that chunk's tail happens to carry shared constants -- the per-type figures range
|
||
from 9.4% overall to 55.6% on one type. A pooled figure over all types would have
|
||
hidden that spread again, so it is reported per type as well.
|
||
|
||
## What the loader does to a chunk (from RAM)
|
||
|
||
Verified against PCSX2 level-0 RAM; method and full dump in
|
||
`plan/pcsx2-verification.md`.
|
||
|
||
The chunk payload is copied **verbatim** -- a probe from inside a record hits at
|
||
`chunk_base + record_offset` with no rescaling. Two things change around it:
|
||
|
||
1. **The offset table is rewritten to absolute EE addresses.** Disc offsets
|
||
`0xf000057c, 0xf00006b0, 0xf00007e4, 0xf0000914` become
|
||
`0xf185387c, 0xf18539b0, 0xf1853ae4, 0xf1853c14` in RAM. The
|
||
`0xf0000000 | low24` tag becomes `0xf1850000 | low24` -- tag replaced by the
|
||
load base, low 24 bits untouched.
|
||
|
||
2. **Every record gains a constant 2-byte prefix `02 00`,** and the rewritten
|
||
table points at *that* rather than at the record data. Measured on all
|
||
**90 of 90** records: prefix `02 00` on every one, and the disc's index bytes
|
||
preserved unchanged at `+2`/`+3` on every one.
|
||
|
||
The third result is the useful one: the byte at `+2` takes exactly
|
||
`0, 16, 32, ... 240` -- the 16 values `index_rule` predicts for `step = 16,
|
||
phase = 0`. The mixed-radix record index, derived from the disc, is corroborated
|
||
on the running game's own copy.
|
||
|
||
### Correction: the `02 00` is on the disc, and a record starts two bytes earlier
|
||
|
||
What the loader does to the *pointer* is as described above. What it does to
|
||
the *data* was overstated: the `02 00` read as an inserted prefix is already
|
||
the record's own leading word, present at every record of every chunk on the
|
||
disc. Three measurements, in order:
|
||
|
||
- **The 2 bytes before the payload are on disc.** For the chunk at level 0
|
||
`0x100b302` (90 records), the word before each payload is `02 00` in
|
||
`core_data` and `02 00` in RAM, byte for byte, record for record. The first
|
||
chunk of the same level carries `08 00` instead. Nothing was inserted.
|
||
- **So the RAM table's pointer is simply two bytes earlier.** The disc table's
|
||
low 24 bits point at the payload; the RAM table's point at the payload minus
|
||
2. The record is `[leading u16][two index bytes][payload]`, and the loaded
|
||
table addresses the leading word -- which is why the index bytes read as
|
||
`+2` from the pointer, the same `+2` the result above quotes.
|
||
- **The leading word is a field, not a prefix.** It is the same at every
|
||
record of a chunk on **1,439 of 1,509 chunks** (`None` on the 70 whose
|
||
records disagree, which the parser reports rather than averaging), it is
|
||
fixed within **every one of the 71 chunk types**, and its values are only
|
||
`1, 2, 4, 6, 8, 10` -- 554 chunks carry 2, 520 carry 4. `Chunk::prefix`
|
||
carries it.
|
||
|
||
What the value counts is not established, and two readings were measured and
|
||
disproved rather than left as guesses:
|
||
|
||
- **Not an entry count.** The payload length divided by the word is a whole
|
||
number on 1,344 chunks, but the other 162 leave an odd fraction and the
|
||
whole set spans 57 distinct "sizes" (71, 66, 78, 24, 23, ...). A real entry
|
||
size is a small clean number, not a set like that.
|
||
- **Not the number of distinct leading bytes.** That agrees on 0 of 1,506
|
||
chunks.
|
||
|
||
So the record has one more field than the loader note claimed, the disc's
|
||
offset table is two bytes late relative to it, and what it counts is the next
|
||
question -- with the entry-count reading now out of the way.
|
||
|
||
## The tie vertex element is 16 bytes
|
||
|
||
With `TieRecord` identified as R&C1's `TiePacketHeader`, each record's leading
|
||
word points at its packet in the block payload. Following it and stepping
|
||
`vert_size` elements from `vert_ofs` reaches the vertices.
|
||
|
||
**Range measured: 95,612 vertices** on levels 0-2.
|
||
|
||
**Element size, settled by comparison rather than assumed.** Requiring the
|
||
texcoord pair to be 12-bit fixed -- both words multiples of 4,096, which is how
|
||
0x1000 = 1.0 reads -- gives:
|
||
|
||
element 16 bytes : 78,919 / 95,612 = 82.5%
|
||
element 8 bytes : 45,661 / 95,612 = 47.8%
|
||
element 12 bytes : 44,834 / 95,612 = 44.8%
|
||
|
||
Sixteen wins by a factor of 1.7 over the nearest alternative, so it is the
|
||
element size.
|
||
|
||
**Observed layout**, 8 x `i16`. Scoring each adjacent slot pair for 12-bit
|
||
fixedness over 36,501 level-0 vertices locates the fields:
|
||
|
||
(i16[0], i16[1]) 1.1% not a texcoord
|
||
(i16[2], i16[3]) 4.8% not a texcoord
|
||
(i16[4], i16[5]) 73.6% <- SCORED as the texcoords; see the retraction below
|
||
(i16[6], i16[7]) 76.6% <- the near-constant 4096/0 pair
|
||
|
||
**`(6,7)` scores higher and is not a texcoord.** It is the near-constant
|
||
`4096, 0` pair -- `4096` is 1.0 in 12-bit fixed and `0` is zero, so *both* are
|
||
trivially multiples of 4,096. That is the constant artifact again: a fixed pair
|
||
would score 100% anywhere. The field is `(4,5)` at 73.6%, with `(0,1)` and `(2,3)`
|
||
ruled out at 1.1% and 4.8%.
|
||
|
||
The first four slots span the full `i16` range with 10,803 to 13,421 distinct
|
||
values each, so they are neither texcoords nor small fields.
|
||
|
||
**The vertex base is `vert_ofs` alone.** Adding the shader or control region to
|
||
it makes things *worse*, so the 73.6% residual is not a mislocated base:
|
||
|
||
vert_ofs only 73.6%
|
||
bfc + vert_ofs 69.8%
|
||
bfc + control_count*control_size + vert_ofs 66.7%
|
||
|
||
**Not established:** what any of the eight fields *means*, and why the texcoord
|
||
rate is 82.5% rather than 100%. The residual varies with record shape -- records
|
||
with `(shaders 2, control 7, control_size 2)` hit 60%, `(shaders 1, control 18,
|
||
control_size 5)` hit 89% -- so it is not one bad block but something that varies
|
||
per packet, possibly a second texcoord encoding for records that need it.
|
||
|
||
Recorded because the 16-byte answer is the *opposite* of what the earlier
|
||
commit concluded from the disc alone. Reading the header word at `+0x00` as an
|
||
element count gave 0 of 922 blocks dividing; it is the byte offset to the payload,
|
||
so the table it bounds has 16-byte records. The number was right and the reading
|
||
was wrong, which is the error the packet-header identification corrected.
|
||
|
||
## Collision payload: what the verbatim mapping settles, and what it does not
|
||
|
||
With the payload confirmed verbatim in RAM, the disc bytes can be read directly.
|
||
The grid dimensions are the **first two `u16`** of the payload -- level 0 reads
|
||
**11 x 33 = 363 cells**, level 1 **24 x 16 = 384**, level 2 **1 x 53 = 53**.
|
||
Level 2 being one cell wide is worth noting; the grids are not all similar in
|
||
shape.
|
||
|
||
**Disproved this round -- an ascending `u32` offset table.** A probe from payload
|
||
`+1024` reads `f4ae0000 40af0000 c4b2000020b20000 ...`, which is three ascending
|
||
values before the fourth goes *down* (`0xb2c4` to `0xb220`). Read as a table that
|
||
was a three-value impression. Scanning all three levels for maximal ascending
|
||
`u32` runs of 8 or more entries finds **zero**, on every level.
|
||
|
||
**Disproved this round -- a global alternating-zero stride.** The head of level 0
|
||
shows an obvious `(value, 0)` pattern from about `i16[37]` onward:
|
||
|
||
[32..] [1808, 1872, 2198, 0, 34, 86, 25848, 1, 25824, 1,
|
||
-13028, 0, -12284, 0, -12228, 0, -12716, 0, ...]
|
||
|
||
Measured over the whole payload rather than the first 96 values, both alignments
|
||
score far below what that sample suggests:
|
||
|
||
even-slot aligned: second half zero on 18% of pairs
|
||
odd-slot aligned : second half zero on 13% of pairs
|
||
|
||
The sample again. What the global numbers *do* say is that the payload is heavily
|
||
zero-bearing -- 18% against a chance rate of about 0.0015% -- which is what a
|
||
sparse or variable-length encoding looks like, and is consistent with the
|
||
variable-length reading established earlier. The 18-against-13 split is a real
|
||
alignment preference but a weak one.
|
||
|
||
**Still not decoded.** The cell layout. The useful position now is that the bytes
|
||
can be read as they are, and that they are sparse and variable-length rather than
|
||
fixed-stride -- so the next attempt should look for a terminator-delimited walk
|
||
through the sparse region, not for a record size.
|
||
|
||
## Collision: the segmentation is not recoverable from zero structure
|
||
|
||
Continuing on the verbatim payload, with the grid known.
|
||
|
||
**Cells are not zero-separated.** Counting maximal runs of non-zero `u16` across
|
||
the whole payload:
|
||
|
||
level 0 363 cells 15,115 runs mean run 25.3 u16
|
||
level 1 384 cells 24,647 runs mean run 51.5 u16
|
||
level 2 53 cells 31,512 runs mean run 30.3 u16
|
||
|
||
Two to three orders of magnitude apart, so a zero run is not a cell boundary.
|
||
Cells are not uniform blocks either: level 0 averages 2,488 bytes per cell and
|
||
level 2 averages 43,684, which cannot both be one fixed-size record.
|
||
|
||
**The run lengths are not records.** Their histogram has a long tail of
|
||
near-power-of-two values -- 2,140, 1,184, 840, 816, 628, 604, 562 -- which is
|
||
what you get when a run is bounded by a single zero *byte* inside otherwise dense
|
||
data, not by a delimiter. Combined with the 18%-zeros finding, the payload reads
|
||
as dense data with sparse holes rather than as a delimited sequence.
|
||
|
||
**So: not ascending `u32` tables, not zero-run delimiters, not fixed stride.**
|
||
All three segmentation theories are now excluded on the full population rather
|
||
than on a sample.
|
||
|
||
### What would actually unblock this
|
||
|
||
`overlay.bin` **is the game's own code** -- it is the level's `.lit` section,
|
||
loaded at `0x1A7C80` -- and the collision debug string
|
||
`Camera_CollPrimTest WARNING! - grid out of bounds! (RAR)` is in its string pool.
|
||
So the parser is reachable. What is *not* established is the reference to it: a
|
||
`$gp`-relative search for the string returns **zero** instruction-aligned hits,
|
||
so the code addresses that string by some other means. The format is therefore
|
||
recoverable, but not by measuring bytes: it needs disassembly of that routine or
|
||
a reference for R&C2's collision cell record.
|
||
|
||
Stating this plainly is worth more than another scan. The disc-side measurement
|
||
space for collision is now genuinely exhausted -- the bytes are available, they
|
||
are verbatim, the grid is known, and three independent segmentation theories have
|
||
failed on the whole population. Further probing without the loader's own logic is
|
||
not going to converge.
|
||
|
||
## Tie class header: `+0x0c` is the three LOD record counts (2026-10-11)
|
||
|
||
**Range measured:** every tie class on the disc -- 1,864 classes, all 27 levels.
|
||
|
||
**Method:** read the word at class `+0x0c` as four bytes, and read the record
|
||
counts independently from the blocks themselves: follow `packets[lod]` to the
|
||
block, walk to the `0xf6` terminator, and count the 16-byte records the block
|
||
header's `data_offset` names. Two paths to the same number, so the check is not
|
||
circular.
|
||
|
||
**Result: the byte at `+0x0c` is the LOD 0 record count, `+0x0d` is LOD 1's and
|
||
`+0x0e` is LOD 2's, on 1,864 of 1,864 classes.** On every level the three
|
||
counts agree, including the classes where a LOD is absent -- a zero
|
||
`packets[lod]` gives a zero count, matching the byte.
|
||
|
||
This closes the file's oldest lie about the tie header: the comment in
|
||
`TieClassHeader::parse` said "0x0c: the 64 light-slot normals offset. Read past,
|
||
not decoded", inherited from R&C1's layout where the word *is* an offset. Here
|
||
it is not an offset at all -- it is three counts -- and R&C1 carries the same
|
||
quadruple at `+0x20`. The earlier note was the right action on the wrong
|
||
information: skipping an unknown field is correct, and the field was only
|
||
unknown because R&C1's map was applied to R&C2's bytes.
|
||
|
||
`TieClassHeader::packet_counts` carries the three, and
|
||
`tie_class_header_counts_records_per_lod` asserts the agreement.
|
||
|
||
**The fourth byte, `+0x0f`, is unidentified.** It ranges 1..15 across the disc.
|
||
The obvious reading, following R&C1's `packet_count[3]` being `texture_count`,
|
||
is a material count, and it is **not** the shader total: the sum of
|
||
`shader_count` over a class's records disagrees on 1,827 of 1,864 classes.
|
||
Left unnamed rather than guessed at -- a count that matches on 2% of the
|
||
population is not a count.
|
||
|
||
## Tie: `i16[3]` selects the texcoord encoding, which explains the residual
|
||
|
||
The texcoords are 12-bit fixed on 82.5% of vertices, not 100%. Grouped by the
|
||
vertex's `i16[3]` slot across **95,612 vertices on levels 0-2**, the buckets
|
||
separate completely:
|
||
|
||
i16[3] n % / 4096 % / 256
|
||
negative 333 1.2% 1.2%
|
||
== 0 5,851 29.2% 60.7%
|
||
1..63 28,884 92.7% 94.3%
|
||
64..255 54,606 92.3% 93.8%
|
||
256..511 8 0.0% 0.0%
|
||
512..1023 5,391 0.0% 0.0%
|
||
>= 1024 539 0.2% 0.2%
|
||
|
||
**The residual is fully accounted for.** 87.3% of vertices sit in `1..255` at
|
||
about 92.5% fixedness, which contributes 0.873 x 0.925 = 80.8 points; the 5,851
|
||
at `i16[3] == 0` contribute 1.8 more. That is the 82.5% measured, to the point.
|
||
|
||
The remaining ~6% are **not 12-bit fixed at all**:
|
||
|
||
- `i16[3] >= 256`, and negatives, score **0%** at both 4,096 and 256 -- 6,271
|
||
vertices using an encoding this reading does not see;
|
||
- `i16[3] == 0` scores 29.2% at 4,096 but **60.7% at 256**, which is a third
|
||
distribution rather than a weaker version of the same one.
|
||
|
||
**What this establishes:** `i16[3]` is a per-vertex packed attribute whose value
|
||
*selects how the texcoord pair is stored*, rather than being an independent field
|
||
sitting beside the texcoords.
|
||
|
||
**Refined: it is the HIGH BYTE that selects.** Grouped by the high byte across the
|
||
same 95,612 vertices on levels 0-2:
|
||
|
||
i16[3] high byte n % / 16 % / 256
|
||
0x00 89,341 92.0% 91.8%
|
||
0x03 5,344 0.0% 0.0%
|
||
0x04 71 0.0% 0.0%
|
||
0x40 58 5.2% 0.0%
|
||
0x02 47 2.1% 2.1%
|
||
0x19, 0x05, 0x20, ... the remaining 73
|
||
|
||
A high byte of **zero** is the ordinary case -- 89,341 of 95,612, or **93.4%** --
|
||
and it scores 92%. **Every** other high byte scores at or near zero. The split is
|
||
on the byte, not on the value: `i16[3]` values 1 and 771 differ only in the high
|
||
byte and behave completely differently.
|
||
|
||
Within the all-zero case there is a third distribution: `i16[3] == 0` exactly
|
||
(5,851 vertices) scores 62% at 16 and 61% at 256, against 92% for `1..255`.
|
||
|
||
**So three cases, selected by the high byte and then the low:**
|
||
|
||
high 0x00, low 0x00 5,851 a third distribution, 62% at 1/16
|
||
high 0x00, low 1..255 83,490 the ordinary case, 92% at 1/4096
|
||
high non-zero 6,271 0%, an encoding this reading does not see
|
||
|
||
### The element layout differs between those cases
|
||
|
||
Scoring all four slot pairs within the non-zero high-byte group, **no pair
|
||
stands out** -- the best is `(6,7)` at 7.5% for a divisor of 4, against a 2-5%
|
||
baseline:
|
||
|
||
i16[3] high byte NON-ZERO (n = 6,271)
|
||
pair /1 /4 /16 /256
|
||
( 0, 1) 100.0% 4.6% 3.5% 0.7%
|
||
( 2, 3) 100.0% 2.0% 0.6% 0.0%
|
||
( 4, 5) 100.0% 1.9% 0.3% 0.1%
|
||
( 6, 7) 100.0% 7.5% 5.7% 4.5%
|
||
|
||
So for those 6,271 vertices the 16 bytes are **not laid out as
|
||
position/texcoord/normal** at all -- none of the four pairs carries fixed-point
|
||
coordinates. Whatever they are, it is a different element shape, selected by the
|
||
high byte of `i16[3]`.
|
||
|
||
This also settles why the global score looked ambiguous: pooling the two layouts
|
||
averages a 92% pair and a 7% pair into one number that describes neither.
|
||
|
||
**Not established:** what any of those three cases means. Whether `i16[3]` is a
|
||
texture index, a normal index, a flag or a combination is still open -- but it is
|
||
now a field whose *role* is known, which is more than "packed word".
|
||
|
||
Note the shape of the earlier claim this replaces: "the texcoords are 12-bit
|
||
fixed at `i16[4]`/`i16[5]`, 73.6% of the time, unexplained". The field position
|
||
was right; the 73.6% was a population average over three encodings, which is
|
||
exactly the pooled-number problem rule 3a describes -- partitioning by the one
|
||
variable that separates them turns a mystery into a lookup.
|
||
|
||
### Which of these findings are asserted
|
||
|
||
Not every claim here has a test behind it, and the recent ones are the ones most
|
||
likely to be lost. Audit against `crates/gc-extract/tests/disc.rs`:
|
||
|
||
claim asserted in
|
||
------------------------------------------- -----------------------------------
|
||
tie vertex element is 16 bytes tie_vertex_element_is_16_bytes_
|
||
with_fixed_texcoords
|
||
layout selected by i16[3] HIGH byte tie_vertex_layout_is_selected_by_
|
||
i16_3_high_byte
|
||
blocks self-delimiting on 0xf6 tie_classes_resolve_through_both_
|
||
indirections (per-block assert)
|
||
packets == 0 means no block at that LOD same test
|
||
packets[0] == 256 on 1,864 of 1,864 same test
|
||
chunk record length uniform per chunk tie-classes test + chunk_records_
|
||
start_16_byte_aligned
|
||
moby class IDs in one body slot per type RETRACTED -- see above; the test
|
||
now asserts 2 types, 1 strong
|
||
collision payload essentially verbatim collision_payload_is_verbatim_
|
||
in_ram (GC_SSTATE-gated)
|
||
tfrag pointers relocate, patches rewrite tfrag_table_pointers_relocate_
|
||
and_every_patch_is_rewritten
|
||
(GC_SSTATE-gated)
|
||
|
||
The last two used to be marked "not assertable here -- needs a savestate". They
|
||
are asserted now, in the same file, gated on `GC_SSTATE` so a checkout without
|
||
PCSX2 skips them. Writing them as tests is what corrected two claims: the
|
||
collision payload is 903,212 of 903,248 bytes identical rather than wholly
|
||
verbatim, and the tfrag relocation base is the *table* address `0x848040`, not
|
||
the block base.
|
||
|
||
The split is deliberate: anything readable from the disc is asserted against the
|
||
disc; anything that needs a running PCSX2 is asserted against a savestate when
|
||
one is available, and documented in the plan file with its method when it is not.
|
||
What is *not* acceptable is a claim living only in prose -- which is how the tie
|
||
`i16[3]` finding spent two commits before it had a test, and how the collision
|
||
"verbatim" claim survived long enough to be wrong.
|
||
|
||
## An external decompilation project is the same disc, and names the overlay
|
||
|
||
`OpenRAC/rac2-gc-decomp` ("Going Commando USA v1.01: byte-exact reconstruction
|
||
tools and native-port research. No game assets.") is byte-for-byte the disc this
|
||
parser was built against:
|
||
|
||
ours 3,828,350,976 bytes
|
||
9db2e33e276133cc283647fa3279b37911955e123d6199d10065547eaa9b1ce5
|
||
theirs 3,828,350,976 bytes
|
||
9db2e33e276133cc283647fa3279b37911955e123d6199d10065547eaa9b1ce5
|
||
|
||
So its findings are usable directly rather than as an analogy. Two of them name
|
||
things this file had measured but not explained:
|
||
|
||
| what | value | source | status here |
|
||
| --- | --- | --- | --- |
|
||
| `.lit` load base | `0x1A7C80`, same on all 27 levels | `docs/LEVELS.md` | **newly named** |
|
||
| `$gp` | `0x1AEFF0`, same on all 27 levels | `docs/LEVELS.md` | newly available |
|
||
|
||
The measured constant `overlay.bin +0x00 = 1,735,808` **is** `0x1A7C80`. The
|
||
reason it is identical across all 27 levels is now known rather than merely
|
||
observed: every level's literal section loads at the same address, because
|
||
loading a level copies its program over the boot's memory from there. So the
|
||
first header word is not a size and not a checksum — it is the **load base of
|
||
the section this file is**.
|
||
|
||
That also settles what the file is. `overlay.bin` is the level's `.lit` section,
|
||
which is why it holds both code and the debug string pool.
|
||
|
||
The program around it has seven sections, per `docs/LEVELS.md`:
|
||
|
||
.lit .bss .data vtbl camvtbl sndvtbl .text
|
||
|
||
which is exactly the "seven-section parser and relocation" already listed as a
|
||
Phase 3 item for this format. It is now a known target rather than a guess.
|
||
|
||
### `+0x0c` is not the `.lit` size
|
||
|
||
The obvious next guess -- now that `+0x00` is known to be the `.lit` load base,
|
||
`+0x0c` might be the `.lit` size -- is wrong, and measurably so:
|
||
|
||
| level | file size | `+0x0c` | excess |
|
||
| --- | ---: | ---: | ---: |
|
||
| level0 | 2,751,268 | 2,767,752 | +16,484 |
|
||
| level11 | 3,151,128 | 2,870,792 | **-280,336** |
|
||
| level2 | 2,928,424 | 2,780,808 | **-147,616** |
|
||
| level24 | 2,615,776 | 2,763,912 | +148,136 |
|
||
|
||
`+0x0c` is **smaller than the file** on 16 of 27 levels, by up to 8.9%. It
|
||
cannot be a size of the section it sits at the front of. It is still unnamed. The point here is only that the size hypothesis is excluded
|
||
by sign, not by a near miss.
|
||
|
||
### Collision: the code that asserts is now locatable
|
||
|
||
The collision blocker was "needs the game's own code". The decomp names the
|
||
functions (`Camera_CollPrimTest(void *prim, void *grid)`, `MB_CheckCollPill`)
|
||
but does **not** decompile them, so the cell layout is still not handed over.
|
||
What it does supply is `$gp`, and that is enough to start.
|
||
|
||
`Camera_CollPrimTest WARNING! - grid out of bounds! (RAR)` sits at
|
||
`overlay.bin+0xd18`, the only copy in the file, so:
|
||
|
||
string overlay+0xd18 -> RAM 0x1a8998
|
||
$gp 0x1aeff0
|
||
delta -0x6658 ($lo half 0x99a8)
|
||
|
||
MIPS materialises a GP-relative address as a `%hi`/`%lo` pair, and the `$lo` half
|
||
is a 16-bit immediate in the instruction stream. So the anchor search is well
|
||
defined -- and it returns **nothing**:
|
||
|
||
$lo half 0x99a8 (low byte 0xa8, high byte 0x99)
|
||
occurrences 0, instruction-aligned, in 2,751,268 bytes
|
||
|
||
**An earlier version of this file claimed exactly one hit, at `0x354cc8`, and
|
||
called it an unambiguous anchor. That was wrong, and the mistake was worth
|
||
recording.** The search compared word bytes 1..3 against the immediate. The PS2's
|
||
MIPS is little-endian, so an I-type immediate is in bits 15..0, which is word
|
||
bytes **0** and **1** -- bytes 1..3 are the opcode, `$rs` and `$rt` fields. The
|
||
match was on those, not on an address.
|
||
|
||
The instruction at the phantom hit is `0x24a89948`:
|
||
|
||
addiu $t0, $a1, -0x66b8
|
||
|
||
whose real immediate is `0x9948`, ten bytes away from the `0x99a8` being
|
||
searched for, and whose `-0x66b8` merely looks like a plausible `$gp` offset.
|
||
With the byte order corrected, the count is zero.
|
||
|
||
So the collision anchor is **not established**, and the assertion that it was has
|
||
been withdrawn. What survives is weaker and still useful:
|
||
|
||
* `Camera_CollPrimTest` and `MB_CheckCollPill` are named, with
|
||
`Camera_CollPrimTest(void *prim, void *grid)`.
|
||
* `$gp = 0x1AEFF0`, and `.lit` loads at `0x1A7C80`.
|
||
* The assert string is at `overlay+0xd18`, a unique copy.
|
||
|
||
The likely reason the search comes back empty is that the code reaching that
|
||
string does not go through `$gp` -- or reaches a copy in another section. The
|
||
decomp's `assert-message-names.tsv` gives `0x002db350` for level 0, which is
|
||
inside `.lit` but disassembles to a function epilogue (`jr $ra` /
|
||
`addiu $sp, $sp, 0x70`), so those are code locations, not string addresses, and
|
||
they are not yet traced to the assert either.
|
||
|
||
`tools/overlay-anchors` does the anchor search and is dependency-free. Its
|
||
correctly-ordered result for the collision string is zero, and it now carries a
|
||
comment explaining the byte order, since that is precisely the part that was
|
||
wrong:
|
||
|
||
cargo run -q -p gc-overlay-anchors -- \
|
||
/home/reaper/Documents/gc-extracted/level0/overlay.bin \
|
||
"Camera_CollPrimTest WARNING!"
|
||
|
||
Next step, if collision is still worth it: find how the assert string is actually
|
||
addressed -- a `%hi`/`%lo` split that does not straddle zero, a pointer table, or
|
||
simply the right section -- before assuming a `$gp` relative load.
|
||
|
||
### What this project did *not* give up
|
||
|
||
Surveyed against the open items: no volume structures, no `TieTex`/tie vertex
|
||
layouts, no `SBlk`/`DAW` record fields, no wavelet tables. Collision has the
|
||
API but not the data. It is a guide to the code, not a replacement for reading
|
||
it.
|
||
|
||
## The tie vertex's selector byte, and what slot 3 is not
|
||
|
||
`i16[3]` is called the layout selector because its **high** byte partitions the
|
||
elements. Splitting the 95,612 elements of levels 0-2 by that byte:
|
||
|
||
| high byte | elements | share |
|
||
| --- | ---: | ---: |
|
||
| `0x00` | 89,341 | 93.4% |
|
||
| `0x03` | 5,344 | 5.6% |
|
||
| the other 144 values | 927 | 1.0% |
|
||
|
||
146 distinct selector values in all -- not a two-way switch but a long tail
|
||
behind one dominant case, and the second case is not `0x01`.
|
||
|
||
Inside the ordinary layout, `i16[3]` is **not a coordinate**:
|
||
|
||
distinct values 201
|
||
range 0 ..= 254
|
||
|
||
201 distinct values inside a byte's range, with 0 the most common, is an index.
|
||
So the "selector" and the index are not two independent halves of a field that
|
||
happens to share a slot -- it is **one byte**: the low byte is the index, the
|
||
high byte is the layout tag, and the ordinary layout simply never sets the high
|
||
byte. Asserted in `tie_vertex_element_is_16_bytes_with_fixed_texcoords`.
|
||
|
||
This was nearly measured wrong. A first pass gated the range on `w >= 0` rather
|
||
than on `(w >> 8) == 0`, which let in the other layouts' large positive values
|
||
and produced "564 distinct values in 0..=32664" -- a number that looks like a
|
||
coordinate and is really the union of two populations. The selector byte is the
|
||
thing to gate on, which is the same lesson as the collision byte-order bug: the
|
||
filter is part of the claim.
|
||
|
||
The remaining slots, all in the ordinary layout:
|
||
|
||
| slot | distinct | range | reads as |
|
||
| --- | ---: | --- | --- |
|
||
| `i16[0..2]` | 10.7k-13.2k each | full `i16` | position, unproven |
|
||
| `i16[3]` | 201 | `0..=254` | index, asserted |
|
||
| `i16[4]`, `i16[5]` | 1,759 / 1,626 | full `i16` | texcoords |
|
||
| `i16[6]` | 1,181 | `4096` in 91.8% | unnamed |
|
||
| `i16[7]` | 226 | `0` in 88.9% | unnamed |
|
||
|
||
`i16[6]`/`i16[7]` are **not** a fixed-point pair, and the "both are multiples of
|
||
4096" score is an artifact worth naming: 86.5% of elements have both divisible by
|
||
4096, against 84.3% expected if the two residues were independent -- an
|
||
enrichment of **1.0x**, i.e. exactly chance. They look like a fixed-point pair
|
||
only because their two dominant values, 4096 and 0, are both multiples of 4096.
|
||
Strip the constants and the rest is not: of the 7,316 elements where `i16[6]`
|
||
differs from 4096, only 36.7% are still multiples, and of the 9,905 where
|
||
`i16[7]` differs from 0, **one** is. The disc test already refused to call it a
|
||
texcoord for this reason; that caution is now quantified.
|
||
|
||
## `i16[0..2]` are the vertex positions
|
||
|
||
The three slots ahead of the index were "unproven position" for want of a test
|
||
that could fail. There is one available on the disc alone, with a built-in
|
||
control.
|
||
|
||
**LOD 1 and LOD 2 of a tie class are the same object at two levels of detail.**
|
||
Anything that locates a vertex in that object's own geometry is therefore shared
|
||
between the two blocks, while a *different* class' LOD is an unrelated object.
|
||
So: intersect the sets of `i16[0..2]` triples, same class versus different
|
||
class, and if positions are what is in those slots the first number should be
|
||
large and the second should be at zero.
|
||
|
||
Levels 0-5, ordinary-layout elements only:
|
||
|
||
same class, LOD1 vs LOD2 96.1% over 107 classes, 107 of them nonzero
|
||
different class, control 0.4% over 2,144 pairs
|
||
|
||
A factor of 240. Per level the same-class figure runs 94.4% (level 0), 97.9%
|
||
(1), 98.1% (2), 96.7% (3), 96.8% (4), and the control never exceeds 1.0% except
|
||
for one level-1 pair at 45.9% -- two classes in that level sharing most of their
|
||
geometry, which is what duplicate or variant models should look like, and is
|
||
reported rather than excluded.
|
||
|
||
Asserted in `tie_vertex_slots_0_to_2_are_shared_positions`, including the
|
||
separation ratio (`same > 20 * diff`) so a change that collapsed both numbers
|
||
would fail rather than pass twice.
|
||
|
||
What this rules out is the important part. An index, a handle, or any per-vertex
|
||
counter would not be shared at 96% between two independently built LOD meshes, and
|
||
would not sit at 0.4% between two objects. Combined with 10.7k-13.2k distinct
|
||
values per slot across the full `i16` range, `i16[0..2]` are a position in the
|
||
object's own frame. The *scale* and the axis order remain unnamed -- nothing here
|
||
says whether these are centimetres, and the axis order is unconstrained because
|
||
nothing here assumes one.
|
||
|
||
The element now reads:
|
||
|
||
+0x00 i16[0] position x (unproven axis order and scale)
|
||
+0x02 i16[1] position y
|
||
+0x04 i16[2] position z
|
||
+0x06 u8 index, 0..=254
|
||
+0x07 u8 layout tag, 0 in 93.4% of elements
|
||
+0x08 i16[4] texcoord u
|
||
+0x0a i16[5] texcoord v
|
||
+0x0c i16[6] unnamed
|
||
+0x0e i16[7] unnamed
|
||
|
||
One caveat worth keeping visible: a test that prints a fraction with `{:.1}%`
|
||
and forgets the `* 100` reports "1.0%" for 99.4%. That happened here, in the
|
||
diagnostic line only -- the assertions compare fractions and were always right --
|
||
but it is the same family of error as everything else in this log: the readout
|
||
disagreeing with the claim.
|
||
|
||
## Tie slots 3, 6 and 7: what they are not
|
||
|
||
Having established that `i16[0..2]` are positions, the remaining slots were
|
||
attacked with two more measurements rather than by inspection.
|
||
|
||
### Does slot 3 index a skeleton?
|
||
|
||
`i16[3]` holds 201 distinct values in `0..=254`, and **50 distinct values within a
|
||
single draw packet** (median, 1,804 records). A material or texture-set index
|
||
would be constant per packet, so it is per vertex -- which invites the obvious
|
||
reading: tie objects are skinned, and slot 3 is a joint index.
|
||
|
||
That reading is wrong. If it named a joint, vertices sharing it would sit
|
||
together. They do not:
|
||
|
||
mean pairwise distance, same slot-3 value 14,481
|
||
mean pairwise distance, different slot-3 value 13,795
|
||
ratio different/same 0.95x
|
||
control: same position-x bucket 8,105
|
||
|
||
Vertices sharing a slot-3 value are if anything marginally *farther* apart than
|
||
vertices differing in it -- 0.95x, the wrong side of neutral, though not far
|
||
enough to claim much on its own. The control shows the measurement works:
|
||
bucket by position-x and pairs collapse to 8,105, half the global distance.
|
||
|
||
So slot 3 is not a skin index. It is also not shared across LODs: 75.3%
|
||
same-class against a 78.7% control, i.e. no signal at all. Per-vertex, in
|
||
`0..=254`, correlated with neither the geometry it sits in nor the detail level.
|
||
**It is not understood**, and no reading is offered here. A per-vertex phase
|
||
would fit the arithmetic, but that is a guess and is labelled as one.
|
||
|
||
### Slots 6 and 7 are mostly not per-vertex
|
||
|
||
Distinct values per draw packet, median over 1,804 records:
|
||
|
||
| slot | median | constant for the whole packet |
|
||
| --- | ---: | ---: |
|
||
| `i16[0..2]` | 13-17 | 0% |
|
||
| `i16[4]`, `i16[5]` | 6, 4 | 0% |
|
||
| `i16[6]` | 2 | **43%** |
|
||
| `i16[7]` | 4 | **34%** |
|
||
|
||
`i16[6]` and `i16[7]` take a single value across the entire packet in well over a
|
||
third of records, and two or three values in the median. Whatever they are, they
|
||
are usually a property of the packet rather than of the vertex -- consistent with
|
||
the 91.8%/88.9% dominance of 4096 and 0 respectively. They are left unnamed.
|
||
|
||
### A note on why the per-slot LOD test does not generalise
|
||
|
||
Run across all eight slots, the LOD1/LOD2 overlap test discriminates only on
|
||
`i16[0..2]`:
|
||
|
||
| slot | same class | control | ratio |
|
||
| --- | ---: | ---: | ---: |
|
||
| `i16[0]` | 97.0% | 3.0% | 32x |
|
||
| `i16[1]` | 96.7% | 3.2% | 30x |
|
||
| `i16[2]` | 96.4% | 6.0% | 16x |
|
||
| `i16[3]` | 75.3% | 78.7% | 1x |
|
||
| `i16[4..7]` | 68-91% | 52-79% | 1x |
|
||
|
||
The other slots come from small value sets -- 226 to 1,759 distinct values across
|
||
a level -- so two arbitrary sets of that size overlap heavily whether or not the
|
||
field means anything. The test's power came from `i16[0..2]` being the only slots
|
||
with tens of thousands of distinct values. **A passing result from a method that
|
||
has run out of discriminative power is not a passing result**, and the 1x rows are
|
||
recorded so the table is not later mistaken for nine confirmations.
|
||
|
||
## The `SBlk` head: four constants, one unidentified word
|
||
|
||
The sound extent's 24-byte head was parsed but not characterised. Across all 27
|
||
levels the `SBlk` region opens identically for its first twelve bytes:
|
||
|
||
+0x00 "SBlk"
|
||
+0x04 1
|
||
+0x08 4
|
||
+0x0c "DAW\0"
|
||
+0x10 0
|
||
+0x14 varies
|
||
|
||
`+0x14` is the only part that differs between levels, and it differs with its
|
||
**low sixteen bits zero on all 27**, leaving `0x0092..=0x0103` -- 146 to 259.
|
||
|
||
That is the shape of a record count, and it tracks the `SBlk` body length at
|
||
close to 100 bytes per unit. It is not one:
|
||
|
||
body / field_14 91.2 .. 110.8 across the 27 levels
|
||
any stride 0..80 making body = stride * word + constant exactly: no
|
||
|
||
A near-miss at 100 bytes/record is exactly the kind of thing that hardens into a
|
||
name the next time somebody glances at it, so the test asserts the negative
|
||
alongside the positive: if a stride ever makes the relation exact on all 27, the
|
||
test fails and asks for the record size to be revisited.
|
||
|
||
Asserted in `sound_bank_sblk_head_is_constant_apart_from_one_word`, which checks
|
||
the four constants individually so a real deviation names which moved, the zero
|
||
low half, the value range, and the absence of a fixed stride.
|
||
|
||
The `DAW` region begins with sixteen zero bytes and then block-structured data
|
||
with 16-byte periodicity and slowly varying nibbles, consistent with block
|
||
compressed audio. It is not decoded.
|
||
|
||
Two notes on how this was measured. Reading the level 0 head directly from a
|
||
scratch script gave a field order that the parser disagreed with, because the
|
||
constructor bound `format_tag` with a `let` *before* the two words that precede
|
||
it in the file -- consuming bytes 4..8 and shifting everything after. The struct
|
||
literal named the fields in the right order; the statements did not. It is fixed
|
||
with a comment saying so, since "read the `let`s in file order" is the kind of
|
||
thing that gets undone.
|
||
|
||
## Volumes: a candidate table, and why it is not yet called one
|
||
|
||
Volumes were the only asset type never located at all, so this is a search
|
||
report rather than a result.
|
||
|
||
**Ruled out first: there is no engine-side name.** No level file contains the
|
||
string `Volume` outside UI text ("Music volume", "Volume de la musique") and
|
||
Italian item descriptions. So the format cannot be identified by a label and
|
||
must be found structurally.
|
||
|
||
`gameplay.bin` on disc is the already-decompressed level gameplay data. Its head
|
||
is a count (`336` on level 0) and then 16-byte records from byte `416`. Those
|
||
records carry `(type, pad, offset, id)`; the `id` field groups them, each id
|
||
having exactly two records pointing into two parallel offset arrays. The offsets
|
||
they point at resolve into a text pool -- the first ones land mid-string in
|
||
dialogue ("it Megacorp Lab"), so this is the dialogue and class-dispatch side of
|
||
the file, not geometry.
|
||
|
||
The structural hit is a table of **144-byte float records**. Level 0 has one at
|
||
`799204`:
|
||
|
||
20 records of 144 bytes, 19 populated and one all-zero terminator
|
||
34-35 of every record's 36 float slots are world-plausible
|
||
+0x0c holds a world position, varying per record:
|
||
|
||
[0] (206.5, 284.1, 116.0) [6] (244.0, 407.2, 128.0)
|
||
[1] (219.3, 247.0, 107.5) [7] (265.3, 443.7, 13548.6)
|
||
[2] (248.2, 198.4, 51.8) [8] (185.1, 282.6, 104.9)
|
||
[3] (247.2, 203.6, 82.4) [9] (185.8, 282.7, 102.6)
|
||
[4] (155.7, 372.7, 121.1) [10] (260.8, 423.6, 130.9)
|
||
[5] (265.0, 445.3, 131.7) [11] (247.0, 211.5, 100.6)
|
||
|
||
A 144-byte record with world-space positions and float radii is exactly the shape
|
||
a volume trigger region should have, and a matching table exists in every level
|
||
examined (levels 0-9), each preceded by an all-zero record, with counts from 23
|
||
to 612.
|
||
|
||
**It is not called volumes here.** Nothing yet ties this table to volumes
|
||
specifically. It could equally be camera bounds, spawn points, or path nodes --
|
||
all of which would present 144-byte float records with world positions and no
|
||
label. The distinguishing evidence has not been gathered, and the honest state
|
||
is *candidate*.
|
||
|
||
### Two detector bugs worth recording
|
||
|
||
Both were mine, and both produced confident nonsense before being caught.
|
||
|
||
1. The first validity test accepted zero as a world-plausible float. Every
|
||
zero-filled region then scores perfectly, and the cross-level check returned
|
||
runs of 225 to 1,514 records -- obviously padding. Requiring **non-zero**
|
||
floats (>=20 of 36) immediately separated real tables from memory.
|
||
2. The stride claim rested on four eyeballed offsets 144 apart. Gap 144 was not
|
||
in the top eight gaps of the detector that found them; it survived because
|
||
three consecutive observations agreed, not because the measurement supported
|
||
it. The corrected detector then found longer runs, which is the opposite of
|
||
what the eyeball version claimed.
|
||
|
||
The same shape as every other correction in this log: a clean, small, plausible
|
||
number read as conclusive because it was not checked against a control that
|
||
could have disagreed.
|
||
|
||
### The reference that did not generalise
|
||
|
||
Level 0's `data.bin` -- the level's file descriptor -- contains the value
|
||
`799204` exactly once, at `0xab6360`, and `799204` is byte-for-byte the offset
|
||
of the 20-record candidate table above. A single 4-byte match in 13.3 MB has a
|
||
chance expectation of about 0.003, which is suggestive enough to be worth
|
||
checking properly rather than assuming.
|
||
|
||
It does not hold up. Taking each level's own best 144-byte table and searching
|
||
that level's own `data.bin`:
|
||
|
||
level candidate offset records references found (chance ~0.0035)
|
||
0 1,810,724 32 0
|
||
1 1,884,884 386 0
|
||
2 2,066,804 224 0
|
||
3 2,438,228 365 0
|
||
4 2,677,780 114 0
|
||
5 1,255,780 14 0
|
||
6 2,496,020 115 0
|
||
7 2,062,072 311 0
|
||
|
||
Not one of the eight is referenced. The level 0 match at `0xab6360` sits in a
|
||
neighbourhood of high-entropy bytes, and level 0's own *best* table -- a
|
||
different offset, 32 records, not the 20-record one -- is not referenced either.
|
||
So the single hit does not generalise and is most likely coincidence.
|
||
|
||
The consequence is worth stating plainly because it retracts what the previous
|
||
section implied: **nothing links a 144-byte float table to volumes.** The table
|
||
is reached by walking structure rather than by an explicit pointer, so the
|
||
approach that would have settled it does not work, and the candidate stands
|
||
exactly where it did.
|
||
|
||
What would settle it, and none of these are cheap:
|
||
|
||
* The game's own volume code, which has the same byte-order problem as the
|
||
collision anchor did -- a GP-relative string search returning nothing.
|
||
* Ground truth: a count of level 0's trigger volumes from a map or a guide,
|
||
to match against a table's population.
|
||
* A correlation with a *named* class, since `gameplay.bin` resolves ids
|
||
through its dispatch records. If an id in the candidate resolved to a class
|
||
whose name is a volume trigger, that would be decisive. No class name
|
||
containing "volume" exists on the disc, which is why this is not available.
|
||
|
||
## The overlay's load base, from its own profiler table (2026-10-11)
|
||
|
||
**Range measured:** all 27 levels' overlays.
|
||
|
||
**Method:** the overlay carries the game's **profiler menu definition** -- a run
|
||
of 8-byte-padded names (`normal`, `camera`, `cam+chr`, `volume`, `bars`,
|
||
`render`, `mobys`, `ties`, `shrubs`, `cpu/gpu`, `occl`, ...) followed by runs of
|
||
absolute RAM addresses that point **back at those same names**. On every level
|
||
the four addresses after `normal`/`camera`/`cam+chr`/`volume`, each minus its
|
||
target name's overlay offset, give the same number: **1,735,792**.
|
||
|
||
That number is `field_00 - 16`, where `field_00` is the header word that
|
||
previously read 1,735,808 on all 27 levels with no meaning attached. So the
|
||
overlay is loaded with its 16-byte header stripped: overlay offset `X` sits at
|
||
RAM `X + 1,735,792`. `OverlayHeader::ram_address` carries it.
|
||
|
||
**Confirmed, not only derived.** The same bases were checked in PCSX2's RAM:
|
||
level 0's `0x1acf80` holds `"normal\0camera\0cam+chr\0volume\0"`, byte-identical
|
||
to overlay `0x5310`, and `0x5310 + 1735792 == 0x1acf80`. All four names land
|
||
where the overlay's own table says. `gc-trace verify-overlay-base` runs the
|
||
check.
|
||
|
||
**The overlay is relocated, not copied.** Only **37.67%** of level 0's overlay
|
||
is byte-identical at the predicted base, and the 256-entry sine table -- at
|
||
overlay `0xb650c` -- is *not* at `0xb650c + base`. So the base locates the
|
||
overlay's image, but individual tables inside it move. That is the shape of a
|
||
loader applying relocations, and it means the base is necessary but not
|
||
sufficient for reading the game's tables.
|
||
|
||
The value of the base is that any overlay offset can now be carried to a RAM
|
||
address, which is the prerequisite for reading the game's live level state --
|
||
the thing the tie relocation and the tfrag lighting checks had to work around.
|
||
|
||
Note the profiler table's own vocabulary, since it is the first named structure
|
||
in the overlay: `volume` is one of the game's profiling categories, so volumes
|
||
are a real subsystem. It does not locate the data.
|
||
|
||
## Chunk records: what the stride table settles, and one signature it does not
|
||
|
||
Walked every chunk on all 27 levels and grouped by its six-byte header signature:
|
||
|
||
1,506 chunks, 34,415 records, 74 distinct signatures
|
||
55 distinct strides, from 52 to 640
|
||
every one of those 55 strides is a multiple of 4
|
||
73 of 74 signatures use exactly ONE stride
|
||
|
||
**The single exception**, asserted rather than skipped so that "every type is
|
||
uniform" cannot quietly stop being true:
|
||
|
||
signature [75, 42, ef, f2, 71, b5] strides {76, 92, 104}
|
||
|
||
This is the variable-length type the module documentation already referred to
|
||
indirectly. It is now named by its signature and pinned to those three strides
|
||
exactly. Everything else is a fixed-stride array, which is the structural fact
|
||
that makes per-type field decoding tractable at all: a record's field offsets are
|
||
the same for every record of its type.
|
||
|
||
The 4-byte alignment is a real result rather than an artefact of how the stride
|
||
was inferred. A stride recovered from one chunk's first-to-second record delta
|
||
could inherit alignment from where that chunk happened to sit; collecting all 55
|
||
distinct strides across every type and requiring all of them to be divisible by
|
||
4 rules that out, because an off-by-one in one signature would show up.
|
||
|
||
### A record layout that did not generalise
|
||
|
||
One type looked like a clean find. Signature `[53, c4, 6c, d7, 9a, 43]`, stride
|
||
108, 810 records, 63 of 108 byte positions constant:
|
||
|
||
+0x00 .. +0x30 48 bytes varying per record
|
||
+0x30 .. +0x6c 60 bytes that are ZERO in every record
|
||
+0x68 0x00010000, constant
|
||
|
||
A fixed data block followed by a permanently empty region reads like an inline
|
||
buffer that this type never uses, and that would have been a tidy field story.
|
||
|
||
It does not generalise. Requiring a contiguous always-zero run and checking
|
||
`stride == live span + zero run` across every type with 50+ records per chunk
|
||
gave **0 of 9**. The zero tail belongs to that one type and means nothing
|
||
general. Recorded because "48 bytes then 60 dead bytes" is specific enough to
|
||
look like a layout and would otherwise be rediscovered and over-read.
|
||
|
||
Neither of these decodes any record's *fields*. What is now settled is the
|
||
shape the decoder has to handle: uniform stride per type, 4-byte aligned, with
|
||
one named exception carrying three strides.
|
||
|
||
## Tie: the "vertex positions" claim does not survive a stride test (2026-10-11)
|
||
|
||
**Range measured:** level 0, one class, one 58-quadword packet region.
|
||
|
||
The claim from earlier reads, `i16[0..2]` of a 16-byte element are the vertex
|
||
positions, and 95,612 vertices were "measured" on that reading. A raw dump of
|
||
one packet kills it:
|
||
|
||
0000df20 00900000 075006c0 094000c0
|
||
1f736c63 00544848 00281506 0d40001b
|
||
0d400a90 ffff00ff 00010ef0 80004253
|
||
4846a9e3 48a44001 47f40000 46d0b0de
|
||
0000ee50 00c00000 08a007e0 099000f0 <- group repeats at +64
|
||
0f807e72 01484848 00321208 0cf00017
|
||
0cf00ab0 ffff00ff 00020e60 80002e46
|
||
4858389a 48a44001 47f40000 46d7bbe3
|
||
|
||
**The first 16 bytes are not a vertex at all** -- they are the packet's own
|
||
header material: `0x0000df20` is not a coordinate and the following words are
|
||
the vif/material words. The 16-byte group at +0 recurs at +64 and +128 with
|
||
identical structure (a small i16, then 0, 0, then an index) and its three
|
||
instances carry the values -8416, -4528, -784. So the stride is **not** 16
|
||
throughout: some elements are 4 bytes, some 16, and the run length varies.
|
||
|
||
**The period-4 marker does not generalise either.** Grouping every fourth
|
||
16-byte element by the high byte of `i16[3]` gives marker 0 (n 9,204), 0x48
|
||
(n 4,140), 0xff (n 4,912) and 0x01 (n 1,137), and only 0x48 shows a
|
||
plausible position signature -- `i16[1]` extent 425 against `i16[0]` and
|
||
`i16[2]` at 65,535, which is the height axis of a flat object. But the same
|
||
marker 0 bucket has all three slots spanning the full `i16` range, so the
|
||
bucket does not select a layout.
|
||
|
||
**Two candidate vertex bases were measured and neither clusters.** With
|
||
`packet + vert_ofs` the per-class extents run median 64,976 and p90 65,517;
|
||
with `class + vert_ofs` they run median 55,213 and p90 64,352. Both span the
|
||
whole `i16` range, whereas a real position triple in a level-class local frame
|
||
must span a fraction of it. `vert_ofs + vert_size + scissor_size` is the
|
||
region length, so the region bounds are right -- what is wrong is the element
|
||
layout inside it.
|
||
|
||
**Consequence for earlier claims, stated plainly.** The 95,612-vertex
|
||
measurement and the 82.5% texcoord fixedness figure were both computed on the
|
||
16-byte reading and are not superseded by this -- they are untested against the
|
||
correct stride, which is not yet known. The LOD1/LOD2-sharing test
|
||
(96.1% same class against 0.4% different) remains the strongest evidence that
|
||
the first slots are positional, and it is insensitive to stride.
|
||
|
||
## Most of the chunk container is level-independent
|
||
|
||
Comparing every chunk record against the same record index in the other 26
|
||
levels, across all 74 signatures:
|
||
|
||
65 of 74 signatures: every record byte-identical in all 27 levels
|
||
9 of 74 signatures: level-specific
|
||
0 ragged -- no signature appeared with different record counts per level
|
||
|
||
The nine that vary:
|
||
|
||
[5d 44 13] [75 42 ef] [7f 44 ee] [82 c4 2f] [96 c3 37]
|
||
[a2 44 a5] [c6 c3 ef] [df c3 ea] [f3 c4 b9]
|
||
|
||
This reframes the whole container. **65 of 74 chunk types are static tables the
|
||
game ships unchanged in every level** -- constants and lookups, not level
|
||
content. The largest by record count are all in that group: `[38 c4 7e]` at
|
||
stride 328 with 90 records per level, `[a9 c4 d8]` at 308 with 90, `[3d c4 9e]`
|
||
at 368 with 77. Each contributes the same bytes to all 27 levels.
|
||
|
||
The practical consequence is about method. For a level-specific type, comparing
|
||
levels is a powerful lever: a field that changes with the level is identifiable
|
||
by correlating it with something known. For the other 65 there is nothing to
|
||
compare -- there is one copy of the data, repeated 27 times -- so they have to be
|
||
decoded structurally, from what the bytes *are*, not from how they vary. Any
|
||
future decoder that reaches for per-level comparison as its first tool will find
|
||
it useless on the majority of the container.
|
||
|
||
`[75 42 ef]` appears in the level-specific list, and it is also the one signature
|
||
with three strides. A type that is both variable-length and level-varying is the
|
||
most informative one in the container, and is where per-type field decoding should
|
||
start.
|
||
|
||
### How this was found, and the near-miss before it
|
||
|
||
The first look at one signature (`[53 c4 6c d7 9a 43]`, stride 108) showed 45 of
|
||
108 byte positions varying *within* a chunk, and 810 records all distinct --
|
||
which reads like per-record data. Counting distinct values per `u32` slot across
|
||
all 810 records contradicted that immediately: 29 to 30 distinct values across 27
|
||
levels, i.e. roughly one per level.
|
||
|
||
Both readings were wrong, and the resolution is that they measure different
|
||
things. The 45 positions vary *between records within a level*. The u32 distinct
|
||
counts were low because the same 30 values recur in every level. Comparing record
|
||
*i* across levels settled it: **all 30 records are byte-identical in all 27
|
||
levels**, and zero byte positions vary across levels.
|
||
|
||
Had the u32 distinct counts been the only measurement taken, the conclusion would
|
||
have been "one value per level" -- which is the opposite of the truth, and would
|
||
have described a per-level parameter table that does not exist.
|
||
|
||
## A parser defect: one chunk type is not fixed-stride
|
||
|
||
`[75 42 ef f2 71 b5]` -- the signature that is both level-specific and
|
||
variable-length -- produces **three chunks per level**, 81 in total. Their record
|
||
offsets step like this:
|
||
|
||
stride 76 deltas [76, 76, 76, 76, 76, 76, 76, 76] uniform
|
||
stride 92 deltas [92, 88, 84, 92, 92, 92, 84, 88] NOT uniform
|
||
stride 104 deltas [104, 104, 104, 104, 104, 104, ...] uniform
|
||
|
||
Identical on all 27 levels -- 27 chunks of each shape, three shapes, no others.
|
||
|
||
So the description "this signature has strides {76, 92, 104}" is a
|
||
**misdescription**. It has three chunk shapes, one of which is not fixed-stride
|
||
at all. The parser derives one `stride` per chunk and `resolve` slices every
|
||
record at that length.
|
||
|
||
What that costs, precisely:
|
||
|
||
* Records are read at the **offset table's** positions, so every record *starts*
|
||
correctly. There is no misalignment.
|
||
* But each is sliced at **92** bytes. Two of the eight are 88 bytes and two are
|
||
84, so **four records over-read by 4 and 8 bytes** and carry the head of the
|
||
next record inside them.
|
||
|
||
Bounds checking holds -- the slices are inside the region -- so this is a
|
||
correctness bug in the data, not a safety one. It is exactly the kind that
|
||
produces plausible-looking fields that are offset by four bytes, which is how it
|
||
would have gone unnoticed had the stride table not been collected per signature.
|
||
|
||
It also explains why this signature was flagged earlier as breaking the 8-byte
|
||
alignment rule: its records genuinely are not a uniform multiple.
|
||
|
||
**This is a defect in the parser, not in the data**, and the right fix is for
|
||
`resolve` to use the offset deltas rather than a single stride for a chunk whose
|
||
deltas are not uniform. Recorded here first, because the assertion in
|
||
`chunk_type_75_ef_has_one_variable_length_chunk_per_level` pins the delta
|
||
sequences -- if the data ever does become uniform, the test fails and asks for
|
||
the model to be revisited rather than the other way round.
|
||
|
||
### A near-miss that nearly hid this
|
||
|
||
The leading `u16` of these records appeared to step by exactly `0x50` (80) for
|
||
all nine records of the 76-byte chunk -- a tidy, meaningful-looking pattern that
|
||
would have been easy to report as "records are indexed by 80". Reading the
|
||
records at their real offsets shows the lead values are `b1a5, a529, 047e,
|
||
a529, 04f0, a529, 0562, a529, 05d4` -- no arithmetic progression at all. The
|
||
apparent 80-byte step came from comparing the wrong bytes.
|
||
|
||
## 1,252 chunks have non-uniform record offsets, and the cause is genuinely ambiguous
|
||
|
||
The previous section fixed on one signature. Measuring every chunk instead shows
|
||
the pattern is the norm, not the exception. Taking `delta = offset[i+1] -
|
||
offset[i]` and comparing it to the chunk's own first delta:
|
||
|
||
delta - first delta, levels 0-2, 2,953 deltas
|
||
|
||
0 1,723 -4 549 +4 426
|
||
-8 146 +8 32
|
||
-12 71 +12 21
|
||
-16 34 +16 4
|
||
-20 27 +20 1
|
||
-24 13
|
||
|
||
**Every deviation is a multiple of 4, and most are exactly +/-4.** Worst observed
|
||
deviation is 24 bytes, in a 7-record chunk. `1,252 of 1,509` chunks have at least
|
||
one such deviation, across 66 of the 74 signatures.
|
||
|
||
### Why this does not become a fix
|
||
|
||
There are two models that produce this offset table, and they are
|
||
**observationally identical**:
|
||
|
||
1. Records are genuinely variable-length, padded to a 4-byte boundary, with
|
||
`stride` recording the largest.
|
||
2. Records are fixed-length at `stride`, and the offset table carries
|
||
**inter-record padding** -- so the gap to the next record is the record plus
|
||
some padding.
|
||
|
||
Nothing in the offset table distinguishes them. Both say where records start and
|
||
both produce `92, 88, 84, 92, 92, 92, 84, 88` for the case above.
|
||
|
||
A change to `resolve` to slice each record against its successor implements
|
||
model 1. It was written, and it broke three tests -- which is worth noting,
|
||
because those three tests asserted "1,509 chunks have uniform record length" and
|
||
passed **because the parser forced uniformity by slicing at a single stride**.
|
||
They were asserting the parser's behaviour back at itself.
|
||
|
||
It was reverted. Slicing to the successor is only correct under model 1, and the
|
||
evidence does not establish model 1. Shipping it would have replaced one
|
||
unverified model with another, and would have silently changed 1,252 chunks'
|
||
record boundaries on the strength of a guess.
|
||
|
||
**What would settle it:** a field inside the record whose end is independently
|
||
observable -- a terminator, a count that implies a length, or a field whose
|
||
value is known and whose position therefore pins the end. Under model 2 those
|
||
bytes are padding and will be uniform or zero; under model 1 they are data and
|
||
will not be. For `[75 42 ef]` the 8 bytes at issue begin `a0 00 27 20` in both the
|
||
tail of one slice and the head of the next -- which is arithmetic, not evidence,
|
||
since the two views are of the same bytes.
|
||
|
||
The uniform model stays in place, the ambiguity is recorded here and in
|
||
[`resolve`]'s docs, and the existing 1,509-of-1,509 assertion stands as a
|
||
statement about what the parser does, not about what the disc contains.
|
||
|
||
## Audit: which claims were contaminated by the over-read?
|
||
|
||
The record-boundary fix changed the length of records in 1,252 of 1,509 chunks,
|
||
so every committed claim computed over *record bytes* had to be re-checked. All of
|
||
them:
|
||
|
||
| claim | reads | verdict |
|
||
| --- | --- | --- |
|
||
| 1,509 chunks, 34,491 records | offsets, not records | unaffected |
|
||
| 1,262 chunks with a leading-byte bucket rule | record byte 0 | unaffected |
|
||
| 1,235 chunks with an index rule | record bytes 1-2 | unaffected |
|
||
| 34,432 of 34,491 records 8-byte aligned | record byte 0 | unaffected |
|
||
| 55 distinct strides, all multiples of 4 | offsets | unaffected |
|
||
| moby class IDs in one slot per chunk type | record bytes | **retracted** |
|
||
| 17-21% of byte positions constant | record bytes | **survives** |
|
||
|
||
Only the claims that scanned *across* a record's body were at risk, and of those
|
||
one was retracted (the class-ID slot) and one survived (the constant-byte
|
||
statistic). The rest read the first three bytes of a record, which the over-read
|
||
never touched -- an over-read appends to the end, so a claim about a record's
|
||
*start* was never in danger. That asymmetry is why the damage was contained, and
|
||
it is the reason to prefer claims anchored at record starts.
|
||
|
||
The surviving one is worth stating precisely because it was the obvious candidate
|
||
for contamination. Re-measured with each record bounded by its successor, and
|
||
comparing only the span every record in the chunk has:
|
||
|
||
per level, % of byte positions constant within a chunk
|
||
lowest level 8 16.7%
|
||
highest 20.0%
|
||
|
||
Still inside the documented band, now asserted rather than only described, in
|
||
`chunk_records_share_a_layout_measured_by_constant_byte_positions`.
|
||
|
||
## The over-read bug was unique to one format, and why
|
||
|
||
Searched for the pattern that caused it -- a record length derived from the
|
||
*first* difference between two offsets:
|
||
|
||
grep -rn "\[1\] - \[0\]" crates/gc-formats/src/*.rs
|
||
crates/gc-formats/src/chunks.rs:357 (offsets[1] - offsets[0])
|
||
|
||
One hit, in the one parser that walks an explicit offset table.
|
||
|
||
The other parsers slice contiguous regions with `chunks_exact` -- shrub's 8-byte
|
||
palette records, tie's 16-byte packet headers, tfrag's 4-byte tables. There, a
|
||
length cannot run past the next element, because `chunks_exact` derives each
|
||
boundary from the region's own length rather than from a sample of the spacing.
|
||
Shrub's mesh packets are named by an explicit offset *and size* table, so they
|
||
carry their length rather than inferring it.
|
||
|
||
So the damage was bounded by a structural property: **only a format with an
|
||
offset table and a derived stride can over-read**, and exactly one format had
|
||
both.
|
||
|
||
The general guard is now positive rather than a comment.
|
||
`chunk_records_neither_overlap_nor_leave_gaps` asserts, across all 27 levels,
|
||
that every record except the last is exactly its own offset delta long, and that
|
||
1,509 chunks contain 1,252 with ragged deltas. A regression to a single sampled
|
||
stride fails immediately and says which chunk and which record.
|
||
|
||
## CORRECTED: the tie vertex region splits in two, but not into "VIF and flat"
|
||
|
||
This section previously read bit 31 of the region's first word as "this is a VIF
|
||
code word", and told a two-encoding story on it -- half VIF packets, half flat.
|
||
The split is real. **The identification is not.**
|
||
|
||
The stride retraction found the vertex region is not a uniform array -- some runs
|
||
4 bytes, some 16, run length varying. The split that accounts for it is visible
|
||
in the region's first word: a VIF code word has bit 31 set.
|
||
|
||
first word has bit 31 set 2,007
|
||
first word has bit 31 clear 2,102
|
||
(levels 0-5, every record with vert_size > 0)
|
||
|
||
The two kinds look like this:
|
||
|
||
flat, class 0 rec 0, 1024 bytes VIF, class 1 rec 0
|
||
+0 0x27667942 0x0062fbf1 +0 0xf5c1a3c5 CODE cmd 0x75 num 193
|
||
+8 0x0000f000 0x00001000 +4 0x00aef9c2
|
||
+16 0x67264af1 0x005f0585 +8 0x10001000
|
||
+24 0x1000e000 0x00001000 +12 0x00001000
|
||
... 16-byte units repeating, third word
|
||
walking down f000, e000, f000,
|
||
d000, d000, c000, c000; last
|
||
word constant at 0x00001000
|
||
|
||
So roughly half of all tie records begin with a word that has bit 31 set and
|
||
half do not. The split is overwhelmingly *not* a function of `vert_size` -- 62 of
|
||
the 72 values carry both kinds -- so "short objects are one thing, long ones
|
||
another" is false, which is the rule a decoder would otherwise reach for.
|
||
|
||
### They are not VIF streams
|
||
|
||
Bit 31 set is exactly what a VIF code word has, and that was the whole basis for
|
||
calling them VIF. Walking them as VIF packets says otherwise. Levels 0-2, 914
|
||
records:
|
||
|
||
walk reaches the region end 17
|
||
walk dies after exactly ONE packet 1,571
|
||
packets before dying: min 0, median 1, max 4
|
||
|
||
`tfrag` walks VIF the same way -- `VifCode::decode`, then `num * 4` payload words
|
||
-- and reaches the end of its lists every time. The walk is not the problem.
|
||
These regions are not VIF packet streams, so bit 31 is set on whatever this word
|
||
actually is, which remains unknown.
|
||
|
||
The original example quoted `0xf5c1a3c5` as a code word with cmd `0x75` num 193.
|
||
Decoding one word as a packet header says nothing about whether the next 193
|
||
words are its payload, and here they are not.
|
||
|
||
Both findings are now asserted in the same test, the split and the walk failure,
|
||
so that if these ever do turn out to be VIF the test fails and says so rather
|
||
than the two quietly drifting apart.
|
||
|
||
This does not settle the stride. It says the region needs to be dispatched on the
|
||
encoding before anything inside it can be read, and it explains why the flat
|
||
reading fit some records well -- `i16[6]` being `0x1000` and `i16[7]` `0x0000` at
|
||
the end of each 16-byte unit is the (-1, 0) / (1, 0) corner pair that a texture
|
||
coordinate produces in 12.4 fixed point, which is where the earlier texcoord
|
||
reading came from and why it looked convincing on the records that are flat.
|
||
|
||
Asserted in `tie_vertex_regions_split_between_vif_and_flat`.
|
||
|
||
|
||
## RETRACTED: the tie texcoord field is at `i16[4]`/`i16[5]`
|
||
|
||
The 73.6% figure above, and the claim that slot pair is the texcoord, do not
|
||
survive. Two separate problems.
|
||
|
||
**It was measured on a reading that has since been retracted.** The 73.6% was
|
||
computed across all records at a 16-byte element stride. `e5cd3df` established
|
||
that the stride is not 16 -- half the records are VIF packets and half are flat --
|
||
so the numbers describe neither population cleanly.
|
||
|
||
**And on the flat records, the fixed-point reading fails.** The question that
|
||
separates a 12.4 fixed-point texcoord from anything else is how many of a slot's
|
||
distinct values are multiples of 4096 at all. For a texcoord it is all of them.
|
||
Measured on the **flat** records only, levels 0-2, 930 records and 47,909
|
||
vertices:
|
||
|
||
slot distinct of which are multiples of 4096
|
||
i16[4] 1,488 16
|
||
i16[5] 1,261 16
|
||
i16[6] 966 4
|
||
i16[7] 489 5
|
||
|
||
One percent. The dominant *pairs* do look like UV corners -- `(0,0)` 5,773,
|
||
`(0,4096)` 5,500, `(4096,4096)` 5,059, `(4096,0)` 4,521 -- and that is exactly
|
||
why the claim was persuasive. But it is a statement about the most frequent
|
||
values, not about the field, and 1,472 of the 1,488 values in `i16[4]` are not
|
||
multiples of 4096 at all.
|
||
|
||
This is the same failure shape as the moby class-ID artifact, one level down: a
|
||
small set of dominant values that happen to satisfy the test, scored as though
|
||
the whole population did. The 73.6% is the arithmetic being right about the
|
||
popular values and the conclusion being wrong about the field.
|
||
|
||
So: on a flat record, `i16[4]`/`i16[5]` are **not identified**, and the texcoord
|
||
slot of a tie vertex is unknown. What survives from the original section is the
|
||
boundary work -- `vert_ofs + vert_size + scissor_size` is the region length -- and
|
||
the LOD1/LOD2 sharing result, which is stride-insensitive.
|
||
|
||
|
||
## The `SBlk` body head: the DAW size, twice
|
||
|
||
The body after the 24-byte head opens with twelve words, and six of them are
|
||
settled across all 27 levels:
|
||
|
||
word value note
|
||
0 varies per level not identified; 27 distinct
|
||
1 60 constant
|
||
2 varies per level not identified; 23 distinct
|
||
3 0 constant
|
||
4 daw_size == the header's daw_size, exactly
|
||
5 daw_size the same value again
|
||
6 0 constant
|
||
7 0 constant
|
||
8 0 constant
|
||
9 127 constant
|
||
10 1 constant
|
||
11 0 constant
|
||
|
||
Level 0's words 4 and 5 are both 1,482,000, and `daw_size` is 1,482,000.
|
||
|
||
### The pointer that was not one
|
||
|
||
Words 4 and 5 were briefly recorded as pointers into the DAW region. They are not.
|
||
The test that promoted them was "does this value fall inside
|
||
`[daw_offset, daw_offset + daw_size)`", and the DAW region is ~1.5 MB starting
|
||
near the front of a ~1.5 MB extent -- so almost any plausible 32-bit word lands in
|
||
it, and 1,482,000 is just under the top of exactly that range.
|
||
|
||
**A range test is not an identification.** The claim that survives is *equality
|
||
with `daw_size`*, which is exact on all 27 levels and is asserted. That is the
|
||
fourth time in this session that a plausible reading of one number has had to be
|
||
withdrawn in favour of a claim that can fail.
|
||
|
||
Word 0 is worth one note: levels 1 and 4 hold 13,042,052 and 13,042,054, two
|
||
apart, while their `daw_size`s differ by 63,504. Whatever it is, it is not a
|
||
size.
|
||
|
||
Asserted in `sound_bank_sblk_body_head_carries_the_daw_size_twice`. The remainder
|
||
of the body -- a per-sound record table -- is still undecoded.
|
||
|
||
## The decomp names the collision functions, but their addresses are not ours
|
||
|
||
`OpenRAC/rac2-gc-decomp` names both collision entry points:
|
||
|
||
Camera_CollPrimTest 0x001E7A50 camera-to-world boundary check
|
||
MB_CheckCollPill 0x001E8890 pill/capsule check for moby instances
|
||
|
||
These looked like the collision unblock: a named function at a known address is
|
||
exactly what the `$gp` string search failed to produce. They are not usable.
|
||
Checked against four levels -- 0, 5, 13, 26 -- with `parse_overlay_sections`:
|
||
|
||
Camera_CollPrimTest @ 0x1e7a50 -> section 1, kind 8 (NOBITS)
|
||
MB_CheckCollPill @ 0x1e8890 -> section 1, kind 8 (NOBITS)
|
||
|
||
Both land inside the level program's zero-filled section. Every byte there is
|
||
zero, and the following 256 bytes contain no non-zero byte at all. Code cannot be
|
||
there, and both addresses are in the same place on all four levels.
|
||
|
||
So those addresses belong to a different address space than a parsed level
|
||
overlay -- most likely the boot executable's own layout, or the v2.00 build, both
|
||
of which that repository also covers. `SCUS_972.68` is USA v1.01 and our disc
|
||
matches it exactly, but that fixes the *disc*, not which program's address space a
|
||
symbol address is quoted in.
|
||
|
||
This closes the shortest route to the collision cell layout that the decomp
|
||
appeared to offer, and it closes it for a checkable reason rather than a vague
|
||
one. The route that remains is the one already recorded: find how the code
|
||
addresses the `Camera_CollPrimTest` assert string, which is not by a `$gp`-relative
|
||
load.
|
||
|
||
## A route into `.text`, and one thing the decomp's assert table does not mean
|
||
|
||
The decomp gained 13 commits (12,375 lines, almost all `src/`). Nothing in it
|
||
names a collision data structure, so the format is not handed over. Two things
|
||
are usable.
|
||
|
||
### `assert-message-names.tsv` addresses are code, in `.text`
|
||
|
||
The table maps each assert's name to an address, 25 overlays each. For level 0,
|
||
`CameraCollPrimTestGridOutOfBounds` is `0x002db350`.
|
||
|
||
That address is **not** the assert call and not the string. It is:
|
||
|
||
0x002db350: jr $ra
|
||
0x002db354: addiu $sp, $sp, 0x70
|
||
0x002db358: nop
|
||
|
||
an **epilogue** -- the address lands on the last instruction group of the
|
||
function before it. The `0x70` is the frame being torn down; `0x002db35c`
|
||
immediately after is a *new* prologue (`addiu $sp, $sp, -0x80`).
|
||
|
||
So the table's addresses are function-ish locations in `.text` (level 0's
|
||
`0x002db350` is well past `.lit`, which ends at `0x001b165c`, and inside
|
||
`0x002a3b80..0x00447900`). They are not string references, which is why the
|
||
earlier "$gp-relative search returns zero" result is now explained from the other
|
||
side: the string is in `.lit` at `0x001a8998` and the code is in `.text` five
|
||
megabytes away.
|
||
|
||
### What is actually in that function
|
||
|
||
Immediately before the epilogue:
|
||
|
||
0x002db2dc: lwc1 $f20, 0xc($s0)
|
||
0x002db2ec: add.s $f20, $f20, $f21
|
||
0x002db2f0: c.olt.s $f0, $f20
|
||
0x002db304: jal 0x2db4b8 <- the assert call
|
||
0x002db308: mov.s $f12, $f21
|
||
0x002db30c: bnez $v0, 0x2db32c
|
||
0x002db314: lw $v0, ($s1) <- a count
|
||
0x002db318: addiu $s2, $s2, 1
|
||
0x002db31c: slt $v0, $s2, $v0
|
||
0x002db320: bnez $v0, 0x2db2cc <- loop back
|
||
0x002db324: addiu $s0, $s0, 0x30 <- stride 48 bytes
|
||
|
||
A loop over records of **48 bytes**, reading a float at `+12` and accumulating,
|
||
guarded by a count at `($s1)`. `lui $at, 0x3f80` / `mtc1 $at, $f22` sets `f22` to
|
||
1.0f, which is a minimum-height clamp.
|
||
|
||
### It is not the collision grid, and the signature says so
|
||
|
||
This is checked rather than assumed. `Camera_CollPrimTest` is declared
|
||
`Camera_CollPrimTest(void *prim, void *grid)` -- two pointer arguments, which in
|
||
MIPS arrive in `$a0` and `$a1`.
|
||
|
||
The function's prologue is `0x002db078: addiu $sp, $sp, -0x70`, matching the
|
||
`+0x70` teardown at `0x002db354`. The two instructions after it are:
|
||
|
||
0x002db078: addiu $sp, $sp, -0x70
|
||
0x002db07c: lw $v1, -0x3f4c($v0) <- $v0, not an incoming argument
|
||
0x002db080: addiu $a0, $zero, 0x6f <- $a0 set to a constant
|
||
|
||
`$v0` is caller-saved scratch, not an argument register, and `$a0` is
|
||
immediately overwritten with the constant 111. **No `void *` reaches this
|
||
function**, so it is not `Camera_CollPrimTest` whatever the assert table labels.
|
||
|
||
It reads four consecutive words from boot memory instead -- `0x1a7348`,
|
||
`0x1a734c`, `0x1a7350`, `0x1a7354` -- all below the level program base at
|
||
`0x001A7C80`, which is the boot's resident region and identical in every
|
||
program. That is a different data source from the collision payload, and the
|
||
48-byte stride loop belongs to whatever that structure is.
|
||
|
||
So this lead is dead, and it is dead on evidence rather than on doubt. Both routes
|
||
the decomp appeared to offer are now closed:
|
||
|
||
| route | status |
|
||
| --- | --- |
|
||
| `Camera_CollPrimTest` at `0x001E7A50` | lands in NOBITS, all zero |
|
||
| `CameraCollPrimTestGridOutOfBounds` at `0x002db350` | a function with no pointer arguments |
|
||
|
||
Neither address reaches the collision parser.
|
||
|
||
### And the last route is closed too: the string is never referenced
|
||
|
||
The route that remained was to find how the code addresses the assert **string**
|
||
at `0x001a8998` in `.lit`. It does not address it. Level 0's program is
|
||
2,751,268 bytes; searching it for both ways an address can be materialised:
|
||
|
||
address low half 0x8998 0 occurrences, word-aligned or not
|
||
$gp-relative offset 0x99a8 0 occurrences, word-aligned or not
|
||
exact 32-bit 0x001a8998 0 occurrences
|
||
|
||
The method is validated by a positive control in the same run: the `.lit` load
|
||
base's own low half, `0x7c80`, occurs exactly **once**, word-aligned. A search
|
||
that finds that will find any real reference.
|
||
|
||
So **no instruction and no data word in the retail level program references the
|
||
collision assert string's address.** Not by `lui`/`addiu`, not by a literal
|
||
pointer, not by a table entry. The string text is present in `.lit` at
|
||
`overlay+0xd18`, and the same text also appears in the boot binary at file offset
|
||
`0xA9558`, but nothing points at either.
|
||
|
||
The reading that fits: **the assert is compiled out in retail and the string is
|
||
dead data.** Debug string pools routinely keep messages whose only user was a
|
||
`#if`-ed-out assertion. The decomp's own table lists this name because it recovered
|
||
it from the string, not because it found code calling it.
|
||
|
||
That is the last route recorded in this file, and it is closed. Reaching the
|
||
collision cell layout from the disc would require a different kind of evidence --
|
||
a build with assertions enabled, a debug listing, or an independent description
|
||
of R&C2's collision cell -- none of which is on this disc.
|
||
|
||
Which is the honest summary of collision after all of it: the payload is verbatim
|
||
in RAM, it does not walk as fixed-stride cells, the disc-side segmentation is
|
||
exhausted, the string that would have led to the routine is unreferenced, and both
|
||
of the decomp's addresses are dead ends.
|
||
|
||
### The disassembler needed fixing first
|
||
|
||
`tools/overlay-anchors` called `disasm_all` over the whole window, which stops at
|
||
the first word capstone will not decode. In code regions that is almost
|
||
immediately, so every window came back five instructions long -- which is why
|
||
earlier attempts at this produced nothing. It now walks four bytes at a time,
|
||
prints `.word` for anything undecodable, and reports the failure count.
|
||
|
||
The undecodable words here are VU1 microcode (`0x0260202d` and friends are COP2
|
||
opcodes), which is expected and is itself informative: this function interleaves
|
||
VIF-fed VU1 work with FPU arithmetic, like `tfrag`'s.
|
||
|
||
|
||
## `lvl.vtbl` gives the real class ids, and the old bound was wrong
|
||
|
||
`lvl.vtbl` is **section 3** of the level program -- one of the seven sections the
|
||
Phase 3 port made visible. Its records are three words:
|
||
|
||
[0] oClass the moby class identifier
|
||
[1] handler address of UpdateMoby_<oClass> in this level's .text
|
||
[2] auxiliary the same value for every record of a level, 26 of 27
|
||
(level 1 has exactly one record pointing elsewhere, 0x2a5d94)
|
||
|
||
Walking all 27 levels:
|
||
|
||
5,878 records
|
||
1,163 distinct class ids
|
||
range 0 .. 0x13BA (5,050)
|
||
median 38 new ids contributed per level
|
||
|
||
The decomp's `docs/MOBY-DISPATCH-TABLES.md` describes exactly this layout, and
|
||
independently notes that 5,598 of 5,788 handlers are preceded by a `jr ra` within
|
||
three words -- so the addresses are function entries, not arbitrary words.
|
||
|
||
### Why this retires the earlier bound
|
||
|
||
`LevelCoreHeader::moby_classes.count` is **238** on level 0. **1,135 of the 1,163
|
||
class ids -- 97.6% -- are at or above that number.**
|
||
|
||
So "value < moby_classes.count" was never a test for "this is a class id". It was
|
||
a test for "this is a small number". That is the mechanism behind the chunk
|
||
class-ID artifact:
|
||
|
||
slot values 1, 3, 4, 5, 10, 11, 12, 13, 15
|
||
every one is below 238
|
||
so every one scored as a hit
|
||
65536 / 238 = 275x ... and the largest such score was 305x
|
||
|
||
The bound was wrong *before* the artifact was noticed, and the artifact was a
|
||
consequence of it. Retracting the claim for "small dominant values" was correct
|
||
but incomplete -- the reason those values looked significant is that the test
|
||
selected for smallness.
|
||
|
||
### Re-scored against the real id set
|
||
|
||
With the game's own 1,163 ids as the validity test, no chunk slot separates:
|
||
|
||
best slot 0.375 (type f2ef4275 slot 34)
|
||
chance rate 0.018 (1,163 ids in 65,536 values)
|
||
a dozen slots 0.25 .. 0.38, with no gap between them
|
||
|
||
Twenty times chance is not nothing, but a dozen slots packed into the same band
|
||
with nothing above them is what a broad property looks like, not a field. The
|
||
positive finding stands and is now usable: **the class-id space is known, and it
|
||
is not the class count.** Asserted in
|
||
`moby_class_ids_come_from_lvl_vtbl_not_from_the_class_count`.
|
||
|
||
|
||
## `oClass` is a global identifier, and not an index into the class table
|
||
|
||
Two axes that are easy to conflate, and are not the same thing:
|
||
|
||
lvl.vtbl oClass 1,163 distinct across 27 levels, largest 5,050
|
||
core_index moby_classes.count 136 .. 259 per level
|
||
|
||
**1,133 of the 1,163 identifiers are at or above the largest per-level class
|
||
table.** So `oClass` cannot index that table -- nearly every identifier would fall
|
||
off the end. They are different axes: the vtbl names a *type* of moby, the class
|
||
table lists the *models* a level instantiates.
|
||
|
||
The identifier space is shared rather than renumbered per level: 1,163 distinct
|
||
values come out of 5,878 records, and level 0 contributes 234 while levels 1-8
|
||
contribute 72, 69, 77, 54, 41, 68 and 28 -- each level roughly its own share, not
|
||
one level dominating. An earlier note here said "most of the game's classes appear
|
||
in the first level"; the data says 20%, and the assertion is written to fail if
|
||
that ever changes.
|
||
|
||
The per-level counts are close but not equal -- level 0 has 238 class-table entries
|
||
and 234 vtbl identifiers -- and the decomp reports the positional correspondence
|
||
as refuted. Nothing here depends on that.
|
||
|
||
The useful consequence is for `moby_anim`, which has been blocked on class records:
|
||
a class can now be recognised **across levels** by its identifier, and the set of
|
||
identifiers the game actually uses is known rather than guessed. Asserted in
|
||
`vtbl_class_ids_are_global_and_do_not_index_the_class_table`.
|
||
|
||
|
||
## All three dispatch tables now decode
|
||
|
||
The level program carries three dispatch tables in its own sections. With the
|
||
Phase 3 section parser they are all readable, and each divides exactly by its
|
||
record width:
|
||
|
||
section file width record
|
||
vtbl lvl.vtbl 3 oClass, UpdateMoby_<oClass>, auxiliary
|
||
camvtbl lvl.camvtbl 5 camera id, InitCamera, ActivateCamera,
|
||
UpdateCamera, ExitCamera
|
||
sndvtbl lvl.sndvtbl 2 sound id, UpdateSound_<id>
|
||
|
||
camvtbl 217 records, 14 distinct ids, max 0x1B
|
||
sndvtbl 32 records, 6 distinct ids, max 0x07
|
||
|
||
The load-bearing check is the **zero remainder**: a wrong record width would leave
|
||
a partial record on most levels rather than none, and it leaves none on any of the
|
||
27 for either table. Section sizes across the levels are 40..52 words for the
|
||
camera tables and 0..8 for the sound tables.
|
||
|
||
Level 0's camera records, so the shape is visible rather than inferred:
|
||
|
||
[0] 0x0 0x4263b0 0x4233f8 0x4263b8 0x426478
|
||
[1] 0x4 0x426c78 0x427008 0x4266e8 0x4270c0
|
||
[2] 0x5 0x427ad8 0x427628 0x427ae0 0x427b38
|
||
|
||
and its sound records:
|
||
|
||
[0] 0x0 0x42ac18
|
||
[1] 0x1 0x42ad98
|
||
[2] 0x3 0x42b038
|
||
|
||
Both identifier spaces are **global and shared**: 217 camera records carry only 14
|
||
distinct ids and 32 sound records only 6, so levels re-use ids rather than
|
||
renumbering -- the same sharing as the moby table's 1,163 distinct out of 5,822.
|
||
|
||
This is what makes the tables usable rather than merely present. A per-level
|
||
dispatch entry is now addressable by a game-wide id, and the handler addresses are
|
||
function entries in that level's `.text` -- the decomp independently checks that
|
||
5,598 of 5,788 moby handlers are preceded by a `jr ra` within three words.
|
||
|
||
The camera and sound handler addresses fall inside section 6 of the same overlay,
|
||
which the working disassembler can read directly.
|
||
|
||
Asserted in `camera_and_sound_dispatch_tables_decode_at_their_documented_widths`.
|
||
|
||
|
||
## The handler chain works, with two caveats and one discrepancy
|
||
|
||
With the dispatch tables decoded and the disassembler fixed, a moby handler can
|
||
actually be read. Level 0's class 2, handler `0x00380a88`:
|
||
|
||
0x00380a84: jal 0x2ee0d0
|
||
0x00380a88: .word 0x0240202d <- VU1 microcode, not a decoding failure
|
||
0x00380a8c: lwc1 $f12, 0x40($s1)
|
||
0x00380a98: addiu $a0, $s0, 0x20
|
||
0x00380aa0: swc1 $f0, 0x18($s4)
|
||
0x00380ab0: swc1 $f0, 0x14($s4)
|
||
|
||
Floats read out of one structure and written back into another, with VU1 work
|
||
alongside -- which is what a transform update should look like.
|
||
|
||
### Two caveats before reading one
|
||
|
||
**VU1 microcode is interleaved with CPU code.** `0x0240202d` is a COP2 opcode, not
|
||
an undecodable word. A disassembly window will run `.word` lines that are
|
||
instructions in a different processor, and a tool that counts them as failures is
|
||
counting the wrong thing.
|
||
|
||
**Handlers are not all independent functions.** Level 0's class 2 is at
|
||
`0x380a88` and class `0x0E` at `0x380a90` -- **eight bytes apart** -- and 72 of the
|
||
5,788 records sit within 16 bytes of the one before them. So consecutive classes
|
||
share one routine with per-class entry points inside it, and reading "a handler"
|
||
gives a fragment, not a whole function.
|
||
|
||
### And a discrepancy with the reference
|
||
|
||
`docs/MOBY-DISPATCH-TABLES.md` reports 5,598 of 5,788 moby handlers (96.7%)
|
||
preceded by a `jr ra` within three words. Measured here, the same way, over the
|
||
same 5,788 records: **5,488 (94.8%)**.
|
||
|
||
Same population, same test, 110 handlers different. Both figures confirm the
|
||
premise, so nothing turns on it, and the residual is recorded rather than explained
|
||
away. The reference measured on 2026-10-02 against its own pinned overlay set, so
|
||
a difference in method or version is as plausible as a difference in the data, and
|
||
there is no basis here for saying which. The test asserts the premise (>90%) and
|
||
deliberately not either exact figure.
|
||
|
||
Asserted in `moby_handlers_are_overwhelmingly_function_entries`.
|