Last commit claimed exactly one GP-relative occurrence of the collision assert
string's $lo half, at 0x354cc8, and called it an unambiguous anchor. That was
wrong, and building the disassembler is what proved it.
The search compared word bytes 1..3 against the 16-bit immediate. The PS2's MIPS
is little-endian, so an I-type immediate sits in bits 15..0 -- word bytes 0 and
1. Bytes 1..3 are the opcode, $rs and $rt fields. The match was on those.
The instruction at the phantom hit is 0x24a89948:
addiu $t0, $a1, -0x66b8
whose real immediate is 0x9948 -- ten bytes from the 0x99a8 being searched for.
-0x66b8 merely looks like a plausible $gp offset, which is what made it
convincing. With the byte order corrected the count is ZERO, in 2,751,268 bytes.
So the anchor is not established and the assertion is withdrawn. What survives is
still worth having: the functions are named (Camera_CollPrimTest takes (prim,
grid)), $gp = 0x1AEFF0, .lit loads at 0x1A7C80, and the assert string is a unique
copy at overlay+0xd18.
Probably the code reaching that string does not address it through $gp, or
reaches a copy in another section. The decomp's assert-message-names.tsv gives
0x002db350 for level 0 -- inside .lit, but it disassembles to a function epilogue
(jr $ra / addiu $sp, $sp, 0x70), so those are code locations, not string
addresses, and they are not traced to the assert either.
tools/overlay-anchors now does the search with the correct byte order and
carries a comment on exactly the part that was wrong, because that is the part
that will be re-derived otherwise. It also disassembles around an address:
cargo run -q -p gc-overlay-anchors -- <overlay.bin> --disasm 0x2db350 0 64
The useful general lesson, which the roadmap already says and this is a second
demonstration of: a probe that returns a small, clean, narratively satisfying
number is the most dangerous kind of wrong result. "Exactly one hit in 2.75 MB"
read as conclusive. It was an artifact of comparing the wrong two bytes, and it
survived a commit because it fit the story.
1961 lines
90 KiB
Markdown
1961 lines
90 KiB
Markdown
# core_data region bounds
|
||
|
||
How a level's `core_data` lump divides up, and how each region is bounded.
|
||
|
||
## The rule
|
||
|
||
A region's end is the **next region's start**, never `core_data.len()`. The
|
||
last region in a layout is the exception, and when its successor is unknown the
|
||
end is `None` rather than a guess.
|
||
|
||
This exists because region bounds were being re-derived per format during
|
||
probing, usually by running from one known offset to the next with the last
|
||
span running to the end of the lump. On level 0 that made the final shrub class
|
||
a 925,008-byte span absorbing most of the lump's tail. Every aggregate computed
|
||
over it -- record counts, value distributions, "this data looks random" -- was
|
||
silently wrong, and nothing errored.
|
||
|
||
`gc_formats::regions::core_layout` computes these once. Callers use it rather
|
||
than inventing their own.
|
||
|
||
## Layout
|
||
|
||
| region | start | end |
|
||
|------------|----------------|----------------------|
|
||
| `tfrags` | 0 | `occlusion` |
|
||
| `occlusion`| `occlusion` | `sky` |
|
||
| `sky` | `sky` | `collision` |
|
||
| `collision`| `collision` | first moby class |
|
||
| `moby` | first moby | first tie class |
|
||
| `tie` | first tie | first shrub class, |
|
||
| | | else chunk container |
|
||
| `shrub` | first shrub | chunk container |
|
||
| `chunks` | chunk container| lump length − 0x38A10|
|
||
|
||
Class regions start at the lowest non-zero geometry offset in their class
|
||
table. A zero entry means *absent*, not position zero -- the same trap the
|
||
occlusion and collision headers each fell into separately.
|
||
|
||
Not every level has every region: level 10 has no occlusion and no shrub.
|
||
|
||
## The constant trailer
|
||
|
||
Comparing all 27 levels byte-for-byte from the end: the last **231,952 bytes
|
||
(0x38A10)** of every level's `core_data` are identical. Level-specific content
|
||
therefore ends at `len - 0x38A10`, and the final region is bounded there rather
|
||
than at `core_data.len()`.
|
||
|
||
This is what closes the chunk container. The regions it separates from level
|
||
content are measured on the disc: **2,033,776 bytes of shrub region across the
|
||
24 levels that have one, and 25,775,850 bytes of chunk container** across all
|
||
27. The disc test asserts the bound on every level.
|
||
|
||
What the trailer *is* is not established. It may be fixed data, or padding
|
||
from a fixed-size decompression buffer. Either way it is constant, which is
|
||
all that is needed to bound the tail.
|
||
|
||
## Found: the chunk container closes the class region
|
||
|
||
The offset ending the shrub region is not in the core index, and the class
|
||
table that precedes the container carries no successor offset for it -- unlike
|
||
moby and tie, whose boundaries fall out of the tables. Texture texels do not
|
||
bound it either: they sit before moby, not after shrub. So the boundary is
|
||
located in `core_data` itself.
|
||
|
||
**Range measured:** for each level, from the last shrub class's geometry
|
||
offset to the constant trailer. Nothing outside that range is read.
|
||
|
||
**Method:** scan forward from the last shrub class for the pair (a) the
|
||
constant two bytes `f6 7c`, sitting immediately in front of (b) a chunk
|
||
signature -- a 16-byte header ending `0xff`, ten zero bytes, then four or more
|
||
`0xf0`-tagged offsets. `gc_formats::chunks::chunk_region_start`.
|
||
|
||
Each half of that signature is weak on its own, which is why the halves are
|
||
measured separately:
|
||
|
||
- the pair `f6 7c` occurs **25 to 103 times** in a 5 to 17 MB lump, so
|
||
matching it alone picks dozens of wrong offsets;
|
||
- the chunk shape matches random data often enough that `parse_chunk` demands
|
||
four tagged offsets before a signature is believed.
|
||
|
||
**Result: on all 27 levels the pair sits exactly two bytes before the first
|
||
chunk header.** The two are never found apart. This is the boundary, and
|
||
`core_layout` closes shrub with it -- `shrub`'s end is the container's start,
|
||
and the container's own end is the trailer, so every byte of level-specific
|
||
`core_data` now belongs to a named region.
|
||
|
||
Two controls, both asserted by the disc test `chunk_container_closes_the_last_class_region`:
|
||
|
||
- **No signature inside the body.** Scanning the last shrub body for a bare
|
||
chunk signature finds none, on any level -- so the boundary search was not
|
||
matching something it should have skipped, and the marker's job is
|
||
corroboration rather than load-bearing disambiguation.
|
||
- **The three levels without shrub** (10, 25, 26) carry the same marker two
|
||
bytes before the container. There the container closes *tie* instead, which
|
||
is what makes the locator a property of the region rather than a shrub
|
||
rule.
|
||
|
||
On level 0:
|
||
|
||
| what | range | bytes |
|
||
|---|---|---|
|
||
| last shrub class data | 16046800 .. 16047040 | 240 |
|
||
| marker `f6 7c` | 16047040 .. 16047042 | 2 |
|
||
| **chunk container** | 16047042 .. 16737344 | **690,302** |
|
||
| constant trailer | 16737344 .. 16969296 | 231,952 |
|
||
|
||
The marker's meaning is not established. It is constant across all 27 levels
|
||
and it appears at exactly this position, but whether it is a terminator, an end
|
||
count or a tag is unknown, so it is not named.
|
||
|
||
**Correction this produces.** The chunk-container measurements recorded below
|
||
were taken over "the 900,000 bytes before the constant trailer" -- a window
|
||
wide enough to contain the container on every level but not aligned to its
|
||
true extent. Chunks sitting at the window's edge were truncated (27 of 1,123),
|
||
and the region's size was never correctly stated. Both are now bounded
|
||
properly: on level 0 the container is 690,302 bytes, not ~900,000, and the
|
||
25,775,850 bytes of chunk data across 27 levels are the measured total rather
|
||
than a windowed sample. The `chunk_containers_resolve_on_every_level` test now
|
||
reads the region from `core_layout` instead of measuring backwards from the
|
||
end of the lump.
|
||
|
||
## Correction: it is mostly texture data, not an unnamed region
|
||
|
||
Adding the region table to `gc-extract probe` made the gap visible
|
||
immediately, which is the point of printing it.
|
||
|
||
On level 0:
|
||
|
||
```
|
||
collision 2615232 .. 8832192 6216960 bytes
|
||
```
|
||
|
||
but collision's own header declares a payload of 903,248 bytes. So the range
|
||
between the collision header and the first moby class is **6.2 MB**, and
|
||
almost none of it is collision.
|
||
|
||
The first conclusion drawn from this -- that ~6.2 MB of unidentified data sat
|
||
between collision and moby, possibly `moby_anim` -- **was wrong**, and it was
|
||
wrong because of the same failure that produced the other two bad numbers this
|
||
session: quoting a figure before bounding it.
|
||
|
||
Checking the bytes rather than just the sizes: the range is dominated by 8-bit
|
||
indexed texture data. Byte-value concentration (a texture block uses a small
|
||
palette; code and collision data does not) puts 18 of 21 sampled 4 KB blocks
|
||
under 200 distinct byte values, against 249 for tfrags and 209 for collision.
|
||
|
||
So the real layout is:
|
||
|
||
| region | start | end |
|
||
|------------|----------|------------------|
|
||
| collision | 2615232 | 3518544 |
|
||
| textures | 3518848 | ~8832192 |
|
||
| moby | 8832192 | first tie class |
|
||
|
||
The texture table's `data_offset` values only reach 5326208, which is why the
|
||
gap looked real: mip chains extend well past the last entry's offset. A region
|
||
sized from a table of first-level offsets understates the data it points at.
|
||
|
||
The "3 of 21 blocks not texture-like" figure in the previous revision was an
|
||
artifact of sampling every 256 KB. Re-scanned at 4 KB granularity the whole
|
||
range is texture data: ~100 blocks stand out, and looking at them they are
|
||
smooth gradient ramps -- sky and fog texture, exactly what gradient texels look
|
||
like. A noisy gradient has a high distinct byte count without being code.
|
||
|
||
So the range between collision and moby is **entirely texture data**, and there
|
||
is no separate `moby_anim` region there. Combined with the correction above,
|
||
two claims made about this range have now been wrong: the "6.2 MB of unknown
|
||
data" and the "3 unexplained blocks".
|
||
|
||
Where `moby_anim` actually lives is therefore still open, but it is not here.
|
||
The remaining candidate is the unresolved tail after the last shrub class,
|
||
which on level 0 is ~925 KB of real, non-zero data that nothing yet accounts
|
||
for.
|
||
|
||
## Where shrub data ends, and the region after it
|
||
|
||
**Range measured:** for each level, the last shrub class's `data_offset` field
|
||
(`class + 0x2c`) through to the start of the constant trailer. Nothing outside
|
||
that range is read.
|
||
|
||
**Method:** the last shrub class's body is an array of 8-byte records whose
|
||
final two bytes are `00 00`. Scanning forward in 8-byte steps for the first
|
||
record whose trailing pair differs locates the body's end. This is a search
|
||
for a specific stated pattern, not a statistic, so it either finds the
|
||
boundary or reports that it did not.
|
||
|
||
**Result:** on **24 of 24** levels the boundary lands exactly on an 8-byte
|
||
record boundary. Body lengths are 192, 208, 224 or 240 bytes -- the same sizes
|
||
seen when shrub classes were grouped and diffed earlier.
|
||
|
||
On level 0:
|
||
|
||
| what | range | bytes |
|
||
|---|---|---|
|
||
| last shrub class data | 16046800 .. 16047040 | 240 |
|
||
| **next region** | 16047040 .. 16737344 | **690,304** |
|
||
| constant trailer | 16737344 .. 16969296 | 231,952 |
|
||
|
||
So the 690,304 bytes between the last shrub class and the trailer are a
|
||
distinct region, not shrub data. What they contain is not established.
|
||
|
||
Two things are worth noting about how this was found, given the rule in
|
||
`AGENTS.md`: the earlier NaN-filler search found nothing at the start of this
|
||
range and was about to conclude the body had zero length, which would have been
|
||
wrong. The last shrub class is a real class with real data, and the NaN filler
|
||
only appears in degenerate ones. Checking the actual byte pattern rather than
|
||
concluding from a failed heuristic is what caught it.
|
||
|
||
### First structure in the region after shrub
|
||
|
||
**Range measured:** level 0, `16047040 .. 16737344` (690,304 bytes), starting
|
||
at the first byte after the last shrub class's 240-byte body. Nothing outside
|
||
that range is read.
|
||
|
||
**Method:** reading the bytes at `+0x20` as `u32` shows ten consecutive
|
||
entries whose low halves increase by a constant **0x94 (148)**, followed by a
|
||
zero. Read as u32 the high half is `0xf000` for the first nine and `0x0a40`
|
||
for the tenth, so the table is ten entries long and ends at `+0x48`
|
||
(absolute `16047112`).
|
||
|
||
**Result, stated plainly:** a ten-entry table on a 148-byte stride. What the
|
||
entries mean, what 148 refers to, and why there are ten, are all unestablished.
|
||
Ten does not match any count already known for this level -- 238 moby
|
||
classes, 74 tie, 7 shrub, 93 tfrag textures -- so this is not yet tied to any
|
||
structure elsewhere in the lump.
|
||
|
||
### Follow-up: the 148 stride is a coincidence
|
||
|
||
Treating the ten entries' low halves as offsets from the region start lands on
|
||
blocks exactly 148 bytes apart, the first nine of which begin `08 00` followed
|
||
by a 16-bit id stepping by 0x40. That is a convincing-looking structure and it
|
||
does not survive being extended: walking the same 148-byte stride across the
|
||
rest of the region, **only 14 of 4657 blocks** start with `08 00`. The first
|
||
nine agree and then the pattern breaks.
|
||
|
||
So the ten entries are not a pointer table. The constant stride held for
|
||
eight consecutive values, which is far more than chance should produce and far
|
||
less than a structure needs.
|
||
|
||
Recorded because the lead was committed as a lead and this is where it goes.
|
||
Nothing else about the region has been identified: it is real, non-zero data
|
||
of unknown type, bounded, and unopened beyond these ~150 bytes.
|
||
|
||
## The region after shrub has a repeating structure at stride 368
|
||
|
||
**Range measured:** level 0, `16047040 .. 16737344`, sampled as the first
|
||
65,536 bytes. Cross-checked on every level that has the region, each sampled
|
||
the same way from its own region start.
|
||
|
||
**Method:** autocorrelation. For each candidate stride, count how often a
|
||
4-byte sequence at offset *i* repeats at *i + stride*, as a fraction of
|
||
comparable positions. A random control of the same size is measured
|
||
identically and reads flat at 0.0000 for every stride, so the statistic
|
||
discriminates.
|
||
|
||
**Result:**
|
||
|
||
| stride | region match rate |
|
||
|---|---|
|
||
| 368 | 0.1110 |
|
||
| 340 | 0.0935 |
|
||
| 337–339 | 0.026–0.029 (background) |
|
||
| random control | 0.0000 |
|
||
|
||
Stride **368** is the strongest peak. On level 0, 340 is a secondary peak
|
||
well clear of the background; the two differ by 28, which is unexplained.
|
||
|
||
**Cross-check:** the peak stride was recomputed independently for every level
|
||
that has the region, and **all 24 peak at exactly 368**. That is the part
|
||
that makes this worth recording -- a single level's peak could be a property
|
||
of that level's data, but 24 of 24 agreeing is a format invariant.
|
||
|
||
What a 368-byte record *is* remains unestablished. This is the same position
|
||
the shrub body was in before differential analysis identified its records,
|
||
and the same method is the obvious next step.
|
||
|
||
### Differential analysis at stride 368: no constant substructure
|
||
|
||
**Range measured:** level 0, `16047040 .. 16737344`. 1,875 whole 368-byte
|
||
records from the region start; variation checked across the first 60.
|
||
|
||
**Method:** the same differential analysis that identified the shrub body --
|
||
compare records byte by byte and record which positions differ.
|
||
|
||
**Result:** **all 368 positions vary.** There is no constant field, no padding
|
||
and no header inside a record. Unlike the shrub body, where differential
|
||
analysis immediately separated a varying payload from a constant two-byte
|
||
tail, this format has nothing constant to separate.
|
||
|
||
That does not contradict the autocorrelation peak. Checking which byte
|
||
positions carry the 368-stride matches over 20,000 positions: 168 exact 4-byte
|
||
repeats, distributed evenly at 0.84% across all four byte offsets. So whole
|
||
4-byte sequences recur at exactly 368 while no individual position is constant
|
||
-- consistent with records containing entries that repeat at a fixed relative
|
||
index, rather than with a fixed-layout record.
|
||
|
||
**So the 368 stride stands and the records remain unidentified.** The shrub
|
||
method failed here not because it was the wrong tool but because it needs
|
||
something constant to latch onto, and there is nothing.
|
||
|
||
## Sky opens with a 256-byte-row image
|
||
|
||
**Range measured:** level 0, `2192576 .. 2615232` (the sky region, 422,656
|
||
bytes). Periodicity start and end located by sliding an 8 KB window in 256-byte
|
||
steps through the region and measuring the 256-stride match rate.
|
||
|
||
**Method:** autocorrelation as above, then a periodicity scan. A random
|
||
control of the same size reads flat at 0.0000 for every stride tested.
|
||
|
||
**Result:**
|
||
|
||
- Peak stride in the first 64 KB is **256** at a 0.772 match rate. Strides 255,
|
||
257 and 4 score nearly as high, which is what a true period looks like
|
||
smeared by sampling.
|
||
- Across levels the peak is 256 on 21, 512 on 5 and 128 on 1. 512 and 128 are
|
||
a harmonic and a sub-multiple, so the fundamental period is **256 bytes**.
|
||
|
||
**Correction to the control.** The original comparison used uniform random
|
||
bytes, which read flat at 0.0000 -- and made the 0.772 look like an enormous
|
||
signal. It is not: 91.8% of the region's bytes are the same value (`0x07`), so
|
||
a stride-256 comparison matches trivially wherever both positions are `0x07`.
|
||
|
||
The right control is the same bytes shuffled, which preserves the value
|
||
distribution and destroys only the ordering. That control reads **0.512** at
|
||
stride 256, not 0.0000. The true excess is 0.772 − 0.512 = **+0.26**, a real
|
||
signal and a far smaller one than the original framing suggested.
|
||
|
||
**And the extent is wrong too.** Re-measured against the matched control, the
|
||
excess over the value distribution is:
|
||
|
||
| offset | excess |
|
||
|---|---|
|
||
| +0 .. +49152 | +0.38 .. +0.45 |
|
||
| +53248 | +0.26 |
|
||
| +57344 | +0.07 |
|
||
| +61440 | −0.13 |
|
||
|
||
So the 256-periodic data ends between `+53248` and `+57344` — somewhere in
|
||
rows 205 to 224 — **not** at `+60160` as first reported. The original figure
|
||
came from a raw threshold with no distribution control, and 91.8% of one byte
|
||
value drags that threshold wherever the dominant run ends.
|
||
|
||
The 256-byte period itself stands, cross-checked on 21 of 27 levels with the
|
||
remaining levels showing its harmonic and sub-multiple. The image dimensions
|
||
do not: an earlier "256x236" is wrong and should not be used.
|
||
|
||
So the sky region is not one thing. Its first ~60 KB is an image; the rest is
|
||
whatever the layer offsets address. Both halves still need work, and the
|
||
image is the tractable one.
|
||
|
||
## Re-verified: the 368 stride, against a matched control
|
||
|
||
The sky autocorrelation was found to be inflated by a control that did not
|
||
match the data, so the 368 result was re-checked the same way.
|
||
|
||
**Range measured:** each level's post-shrub region start, first 65,536 bytes.
|
||
|
||
**Method:** as before, but the control is the same bytes shuffled rather than
|
||
uniform random bytes. Shuffling preserves the value distribution and destroys
|
||
only the ordering, which is exactly what the control needs to do.
|
||
|
||
**Result:**
|
||
|
||
- Level 0's region is evenly distributed: 256 distinct byte values, most
|
||
common only 15.6%. Uniform random was therefore the *appropriate* control
|
||
here, and the shuffled control independently confirms it, reading **0.0000**
|
||
at strides 340, 368 and 4.
|
||
- Level 0 peak remains 368 at 0.1110, against 0.0000 for the matched control.
|
||
- Recomputed across all levels with matched controls: **24 of 24 peak at
|
||
exactly 368, every one with excess above 0.05.**
|
||
|
||
So the 368-byte period stands, now with a control appropriate to its data.
|
||
The contrast with sky is the point: the same statistic on the same kind of
|
||
question gave a real answer in one region and an inflated one in the other,
|
||
and the only thing that told them apart was whether the control matched the
|
||
byte distribution.
|
||
|
||
## Sky sub-objects: structure is consistent, stride is not
|
||
|
||
**Range measured:** every level's sky region, from the count field at `+0x0c`
|
||
through the sub-object offsets at `+0x20`. Each sub-object measured over its
|
||
first 16,384 bytes.
|
||
|
||
**Method:** as above, peak-stride autocorrelation per sub-object with a
|
||
matched control (the same bytes shuffled, seeded per level and index).
|
||
|
||
**Field note:** the sub-object count is a `u32` at **`+0x0c`**, not `+0x0a`.
|
||
Reading `+0x0a` gives 327,680 on level 0, which is not a count.
|
||
|
||
**Result, level 0:** five sub-objects at offsets 365760, 388880, 396432,
|
||
406784 and 414032, of lengths 23120, 7552, 10352, 7248 and 1536. Each has
|
||
internal structure well clear of its matched control.
|
||
|
||
**But the strides do not repeat across levels.** Recomputing per level:
|
||
|
||
| sub-object | peak strides seen across levels |
|
||
|---|---|
|
||
| 0 | 4 (x6), 8 (x13), 32, 288 |
|
||
| 1 | 16 (x3), 32 (x6), 288 |
|
||
| 2 | 8, 16 (x5), 256, 288 |
|
||
| 3 | 8, 16 (x6), 216 |
|
||
| 4 | 8, 16 (x3), 512 (x4) |
|
||
|
||
So the earlier per-index stride table, read off level 0 alone, was
|
||
over-fitted to one level. What is consistent is weaker and more honest: every
|
||
sub-object sits above its matched control, and most peak at a small stride of
|
||
4, 8, 16 or 32, which suggests 16- or 32-bit records whose stride varies with
|
||
content rather than being fixed.
|
||
|
||
**Level 2's header remains genuinely different**: its count field at `+0x0c`
|
||
reads 196,617 and is unusable, while every other level yields a plausible
|
||
count. Whatever makes it different is still unexplained.
|
||
|
||
### Sky sub-object 0: sparse, with duplicated entries
|
||
|
||
**Range measured:** level 0, sky region `+365760 .. +388880`, 23,120 bytes,
|
||
read as 2,890 records of 8.
|
||
|
||
**Method:** record composition, plus the adjacent-pair and zero-density
|
||
statistics below. The matched-control autocorrelation puts the peak at stride
|
||
8 (0.217 real, 0.000 shuffled), so 8 is the unit to describe.
|
||
|
||
**What it is made of:**
|
||
|
||
- **1,185 distinct `u32` values** across 5,780 words.
|
||
- **Mean 6.99 of 8 bytes are live** -- the previous revision of this file said
|
||
"zero", which is inverted and wrong. The live-byte histogram is 0:15, 2:1,
|
||
4:31, 5:238, 6:461, 7:1030, 8:1114, so most records are nearly full rather
|
||
than sparse.
|
||
- **549 of 2,890 adjacent pairs (19%) are exact duplicates of each other.** The
|
||
most common 8-byte record, appearing 283 times, is `d7d7d780 d7d7d780` -- a
|
||
4-byte value stored twice. Others follow the same shape: `00051980 00051980`,
|
||
`00082580 00082580`.
|
||
|
||
**Byte 7 is anomalous.** It takes only **14 distinct values** where bytes 0-6
|
||
take 142-204. Its common values are `0x00` (1378), `0xff` (806) and `0x80`
|
||
(592) -- the high bytes of signed 16-bit values, which is consistent with
|
||
byte 7 being an MSB of something rather than ordinary data. Unexplained.
|
||
|
||
**Not decoded.** A signed-16-bit reading was tried and does not obviously
|
||
help: the values span -32743..32633, close to the full range, with no
|
||
per-position structure (533, 371, 401 and 187 distinct at positions 0-3).
|
||
Recorded as a characterisation, with its range, so the next attempt starts
|
||
from the composition rather than re-deriving it.
|
||
|
||
### Sky sub-object 0 is a slowly-varying sequence, not independent records
|
||
|
||
**Range measured:** level 0, sky region `+365760 .. +388880`, 23,120 bytes.
|
||
|
||
**Method:** Pearson correlation between the byte sequence and itself at
|
||
increasing lag, against a shuffled control. This measures whether consecutive
|
||
records are *related*, which the position-wise distinct-value counts could
|
||
not show.
|
||
|
||
**Result:**
|
||
|
||
| lag (bytes) | correlation |
|
||
|---|---|
|
||
| 1 | +0.050 |
|
||
| 4 | +0.560 |
|
||
| **8** | **+0.746** |
|
||
| 16 | +0.666 |
|
||
| 32 | +0.602 |
|
||
| 64 | +0.432 |
|
||
| 128 | +0.140 |
|
||
| 256 | −0.044 |
|
||
| **512** | **+0.304** |
|
||
| 1024 | +0.247 |
|
||
| shuffled control (lag 1) | +0.003 |
|
||
|
||
The profile has a peak at lag 8, decays to a zero crossing near 256, and then
|
||
a second, smaller peak at 512 before decaying again.
|
||
|
||
This is what settles the earlier "no constant substructure" reading. The
|
||
records do not share fixed fields, but they are *not independent*: consecutive
|
||
8-byte records are strongly related, and that relation repeats at 512 bytes,
|
||
i.e. every 64 records.
|
||
|
||
Slowly varying values, related within a record, repeating every 64 records, is
|
||
the shape of a keyframe or lookup table -- which is what one would expect to
|
||
find in sky data, where a colour or fog parameter is sampled rather than
|
||
computed per vertex.
|
||
|
||
**Still not decoded.** Nothing here names the values or the sampling. But the
|
||
structure is now characterised well enough that a decode attempt has a shape
|
||
to aim at rather than a field to hunt for.
|
||
|
||
### Sky sub-objects share one shape, across every level
|
||
|
||
**Range measured:** sub-objects of the sky region on levels 0, 1, 3, 7, 13
|
||
and 26 -- 25 sub-objects in total, each measured over its whole length.
|
||
|
||
**Method:** as above, correlation at lag against the byte sequence.
|
||
|
||
**Result:**
|
||
|
||
| quantity | range over 25 sub-objects |
|
||
|---|---|
|
||
| lag 8 | +0.53 .. +0.79 |
|
||
| lag 64 | +0.16 .. +0.63 |
|
||
| lag 512 | -0.13 .. +0.42 (not universal) |
|
||
| lag 1024 | -0.05 .. +0.37 (not universal) |
|
||
|
||
**Lag 8 and lag 64 are positive on every sub-object of every level tested.**
|
||
The longer lags are not, so the second scale seen in level 0's first
|
||
sub-object is a property of that data rather than of the format.
|
||
|
||
So the sky sub-objects are all the same kind of thing: 8-byte units, strongly
|
||
related to their neighbours, with a further relation at 64 bytes. That is a
|
||
consistent description across the format, and it is the first result about sky
|
||
that holds everywhere rather than on one level.
|
||
|
||
**Not decoded.** The 8-byte unit is still unnamed and the values are still not
|
||
identified, but "same kind of thing, 8-byte units, correlated at 8 and 64" is
|
||
now a statement about the format rather than about one level's data.
|
||
|
||
## Sound bank SBlk has a 40-byte period (27/27 levels)
|
||
|
||
**Range measured:** each level's `SBlk` region -- the descriptor table between
|
||
`sblk_offset` and `daw_offset`, from the sound bank header rather than from
|
||
`core_data`. 17-20 KB per level.
|
||
|
||
**Method:** peak-stride autocorrelation with a matched control (the same bytes
|
||
shuffled, same seed, every level).
|
||
|
||
**Result:** the peak is at **exactly 40 bytes on all 27 levels**, every one with
|
||
excess above 0.2. The peak is also sharply isolated on level 0 -- 0.454 excess
|
||
at stride 40 against 0.207 to 0.278 for strides 36 to 44 -- so it is a period
|
||
rather than a broad tendency.
|
||
|
||
That is the first structural result about the sound bank beyond the container
|
||
framing already committed.
|
||
|
||
**Record count:** 487, 443, 428, 494 and 447 whole 40-byte records on levels 0,
|
||
1, 2, 13 and 26. The region is not an exact multiple of 40 on any level
|
||
(19484 leaves 4 bytes over), so 40 is a period rather than a record size with
|
||
no remainder.
|
||
|
||
**Unexplained:** which byte positions are constant varies by level -- level 0
|
||
has 7, 11, 31 and 35 constant; level 1 has 15, 19, 23, 35 and 39; level 2 has
|
||
7, 23, 27 and 31. A fixed 40-byte layout would have the same constant
|
||
positions everywhere, so either the records have a variable tail or 40 is a
|
||
period in something else that the records sit inside.
|
||
|
||
Also measured, for the record:
|
||
|
||
- `DAW`, the sample region, shows **no** stride above threshold against its
|
||
control. Consistent with ADPCM nibble-packed audio, and consistent with the
|
||
container framing being the only part worth decoding.
|
||
- `gameplay.bin` peaks at 16 bytes (excess 0.267 at strides 16, 32 and 64).
|
||
- `gs_ram.bin` has almost no stride structure (0.043 at 32) but a lag-8
|
||
correlation of +0.503, so it is smooth rather than tabular.
|
||
|
||
## Tie class spans: 16-byte unit, 48-byte period (dominant, not universal)
|
||
|
||
**Range measured:** every tie class span on levels 0, 3, 6 and every third
|
||
level after it -- 386 spans of 16 KB or more, bounded by consecutive class
|
||
offsets.
|
||
|
||
**Method:** peak-stride autocorrelation with a matched control, over the first
|
||
32,768 bytes of each span.
|
||
|
||
**Result:**
|
||
|
||
| peak stride | spans |
|
||
|---|---|
|
||
| 48 | 285 |
|
||
| 16 | 58 |
|
||
| 64 | 29 |
|
||
| 96 | 11 |
|
||
| 24 | 2 |
|
||
| 32 | 1 |
|
||
|
||
**48 bytes dominates but is not universal**, and the weaker peak is 58 spans
|
||
against 285 -- so unlike the sound bank's 40, this is a majority rather than a
|
||
constant.
|
||
|
||
Every observed peak is a multiple of 16 (16, 24, 32, 48, 64, 96), which
|
||
suggests a **16-byte base unit** with a 48-byte period arising from how those
|
||
units group, rather than 48 being the record itself.
|
||
|
||
This is weaker evidence than the sound bank's 40-byte result, which was
|
||
sharply isolated and identical on all 27 levels. Here the peak is weaker
|
||
(0.09-0.22 excess against sound bank's 0.45) and it moves between levels and
|
||
spans. Recorded at the confidence it deserves.
|
||
|
||
## Two independent measurements agree on moby's 16-byte vertex
|
||
|
||
The moby part table established `vertex_offset + 16 * vertex_count ==
|
||
next_part` on 43,467 of 43,467 parts, which fixes the vertex stride at 16.
|
||
Autocorrelation over the spans reaches the same number without using that
|
||
result at all.
|
||
|
||
**Range measured:** moby class spans of 16 KB or more on level 0 -- 60 spans,
|
||
each bounded by consecutive class offsets, measured over the first 32,768
|
||
bytes.
|
||
|
||
**Method:** peak-stride autocorrelation against a matched control, searching
|
||
strides 4 to 64.
|
||
|
||
**Result:**
|
||
|
||
| peak stride | spans |
|
||
|---|---|
|
||
| **16** | **43** |
|
||
| 8 | 8 |
|
||
| 48 | 4 |
|
||
| 32 | 3 |
|
||
| 24 | 2 |
|
||
|
||
16 dominates, and it is the same 16 the part table produces from a completely
|
||
different argument -- a tiling identity over part records rather than a
|
||
periodicity measurement over span contents.
|
||
|
||
Two unrelated methods agreeing is worth more than either alone, because the
|
||
autocorrelation would be vulnerable to exactly the kinds of artifact this
|
||
project has already been bitten by: a dominant byte value would inflate it,
|
||
and a period that happened to divide the window would look the same as a real
|
||
one. Neither applies here, and the tiling identity cannot be inflated at all.
|
||
|
||
This is the strongest form of evidence produced in this project: a structural
|
||
invariant confirmed by two methods that do not share a failure mode.
|
||
|
||
## Sound bank SBlk: a 12-byte marker that does not explain the 40-byte period
|
||
|
||
**Range measured:** level 0 `SBlk`, 24..19508 (19,484 bytes).
|
||
|
||
**Method:** positions of the `0x7f`/`0x47f` marker words, and their spacing.
|
||
|
||
**Result:** those markers sit at **12-byte spacing on 144 of roughly 180
|
||
occurrences**, with 24-byte and larger gaps accounting for the rest. So there
|
||
is a 12-byte structure in the data.
|
||
|
||
**But that contradicts the autocorrelation.** If records were 12 bytes, stride
|
||
12 should be the peak. It is not:
|
||
|
||
| stride | excess over shuffled |
|
||
|---|---|
|
||
| 12 | +0.237 |
|
||
| 20 | +0.175 |
|
||
| 24 | +0.237 |
|
||
| **40** | **+0.454** |
|
||
| 48 | +0.239 |
|
||
|
||
40 is not a multiple of 12, so the two measurements disagree rather than one
|
||
explaining the other. Recorded as an open discrepancy rather than resolved in
|
||
favour of either: the marker spacing says 12, the period says 40, and 40 being
|
||
twice as strong is the more surprising of the two.
|
||
|
||
### Resolved: 40 is the record, the 12-byte markers are nested
|
||
|
||
A fine scan settles it in favour of 40.
|
||
|
||
| stride | excess |
|
||
|---|---|
|
||
| **40** | **+0.454** |
|
||
| 39 | +0.266 |
|
||
| 41 | +0.215 |
|
||
| 38 | +0.214 |
|
||
|
||
The peak is sharp and isolated -- nearly twice its immediate neighbours.
|
||
And the multiples confirm it: 80 scores +0.376, 120 +0.394, 160 +0.340,
|
||
while the multiples of 12 (24, 48, 96, 144) all sit at 0.21-0.24.
|
||
|
||
So the record is **40 bytes**, and the `0x7f`/`0x47f` markers at 12-byte
|
||
spacing are a nested structure *inside* those records rather than the record
|
||
boundary. Their spacing is irregular -- 12, 24 and larger -- which is exactly
|
||
what one would expect of a field that is present in some records and absent in
|
||
others.
|
||
|
||
The earlier "they disagree" framing was wrong in one respect: the marker
|
||
spacing never claimed to be the record size. Read as a claim about markers
|
||
inside records rather than about the record itself, the two measurements were
|
||
never in conflict.
|
||
|
||
## SBlk record layout at 40 bytes
|
||
|
||
**Range measured:** level 0 `SBlk`, 24..19508, aligned at +0 — 487 whole
|
||
records.
|
||
|
||
**Method:** for each of 40 candidate alignments, record the number of byte
|
||
positions that are near-constant across records, and keep the best.
|
||
|
||
**Result:** alignment +0 is best, with 10 near-constant positions. Four are
|
||
exactly constant and all zero: positions **7, 11, 31 and 35**.
|
||
|
||
Distinct values per byte position across the 487 records runs from 1 to 155,
|
||
with position 29 the most variable by a wide margin (155 against 21-48
|
||
elsewhere) and positions 2, 10, 27, 34 and 39 low (2 to 8).
|
||
|
||
The first record carries the `SBlk` magic and the `DAW` tag rather than
|
||
descriptor data, so it is a header rather than an entry — which is worth
|
||
accounting for before reading any field at a fixed offset.
|
||
|
||
**Not decoded.** The layout is established and the constant positions are
|
||
known; the fields are not named, and which of the 40 bytes are an index, a
|
||
length and an offset is still open.
|
||
|
||
## SBlk: the field values do not read as record fields
|
||
|
||
**Range measured:** level 0 `SBlk`, 24..19508.
|
||
|
||
**Method:** read at 4-byte alignment and rank the most common `u32` values.
|
||
|
||
**Result:** the common values are `0x0` (2261), `0x1` (369), `0x7f` (168),
|
||
`0x5a000000` (125), `0x1eff7f7f` (125), `0x1dfcc` (124), `0x19fc0` (93),
|
||
`0x7c34240d` (91), `0x7c3821c8` (91).
|
||
|
||
**This does not look like a table of records with typed fields.** A 40-byte
|
||
record layout whose fields are an index, a length and an offset would not
|
||
produce `0x1eff7f7f` and `0x7c34240d` a hundred times each, nor would those
|
||
values survive a byte shift.
|
||
|
||
So either the 40-byte period is not a record boundary, or the region is an
|
||
encoded stream -- something like varint or Huffman-coded data -- that happens
|
||
to have a 40-byte statistical period.
|
||
|
||
The `0x7f` values are suggestive: in a varint encoding a byte of `0x7f` is the
|
||
last value representable in seven bits, so it appears as a frequent
|
||
saturated byte. That is consistent with a coded stream and inconsistent with
|
||
a fixed-layout record.
|
||
|
||
**Left open.** The 40-byte period is solid (27/27 levels, sharply isolated, its
|
||
multiples confirm it), and the alignment with four constant zero positions is
|
||
solid. What sits in those bytes is not a record layout, and the two findings
|
||
are in tension in a way this project has not otherwise seen: a confirmed
|
||
period that is not a record boundary.
|
||
|
||
## Correction: the 368 stride is real, but confined to one ~15 KB window
|
||
|
||
The claim earlier in this file that the post-shrub region has a 368-byte
|
||
period *throughout* is wrong. Re-measured over the whole region, the
|
||
autocorrelation curve is flat and near noise:
|
||
|
||
| stride | excess over matched shuffled control |
|
||
|---|---|
|
||
| 8 | +0.027 |
|
||
| 12 | +0.023 |
|
||
| 148 | +0.009 |
|
||
| 340 | +0.018 |
|
||
| **368** | **+0.018** |
|
||
| 632 | +0.020 |
|
||
|
||
Nothing peaks at 368. The figure of 0.1110 recorded earlier does not
|
||
reproduce under any sampling convention tried (step 1, 2 or 4 through the
|
||
window; all give ~0.018).
|
||
|
||
**Where the structure actually is.** Scanning 8 KB windows at 1 KB steps for
|
||
a 368-excess above 0.15 finds exactly one substantial region:
|
||
|
||
region+41984 .. region+57344 = 15,360 bytes (abs 16089024..16104384)
|
||
|
||
plus a 4 KB patch at region+650240. Everything between and around is flat.
|
||
|
||
**Within that window the 368 stride is unambiguous:**
|
||
|
||
| stride | excess |
|
||
|---|---|
|
||
| **368** | **+0.315** |
|
||
| 736 | +0.284 |
|
||
| 364 | +0.024 |
|
||
| 372 | +0.021 |
|
||
| 732 | +0.027 |
|
||
|
||
The peak is sharp, 736 confirms it as a multiple, and the neighbours are at
|
||
noise.
|
||
|
||
**Record shape.** 41 whole 368-byte records. Mean distinct values per
|
||
position is 16 of 41, and **nine positions are exactly constant and zero**:
|
||
329 through 336, and 340. Twenty-five positions have 3 or fewer distinct
|
||
values, clustered in 325-349 -- a tail region rather than scattered padding.
|
||
|
||
So the corrected statement is: a 15,360-byte window of the post-shrub region
|
||
is a table of 41 records at stride 368, each with a roughly 43-byte tail
|
||
whose fields are mostly zero. The surrounding 675 KB is not that table.
|
||
|
||
**What went wrong before.** The original measurement was taken over a window
|
||
where 368 held and then reported as a property of the region. The 15 KB
|
||
window holds 41/1841 of the region's records, and averaging that over the
|
||
whole region flattens the peak to the noise floor. The re-verification pass
|
||
checked the stride but not whether it applied uniformly, so it confirmed the
|
||
number without testing the claim it was attached to.
|
||
|
||
## The post-shrub region is a chunk container (explains the 368 stride)
|
||
|
||
**Range measured:** level 0 post-shrub, 16047040..16737344 (690,304 bytes).
|
||
|
||
**Method:** scan for the block shape `16-byte header ending 0xff` followed by
|
||
10 zero bytes followed by a run of `u32` whose top nibble is `0xF`.
|
||
|
||
**Result: 28 chunk headers on level 0**, present on all 27 levels. Each chunk
|
||
is:
|
||
|
||
+0x00 16 bytes, last byte 0xff (opaque; differs per chunk)
|
||
+0x10 10 zero bytes
|
||
+0x1a N tagged offsets, top nibble 0xF
|
||
|
||
and within a chunk the offsets step by a **chunk-specific stride** — 148,
|
||
340, 368, 288, 320, 236, 272, 384, 116, 160, 250, 580, 576 all occur on
|
||
level 0. The stride is near-constant within a chunk (e.g. 148,148,148,148...
|
||
then 340,340,336,340...) but differs between chunks.
|
||
|
||
**This resolves two earlier results.**
|
||
|
||
The "368-byte period" is one chunk's stride, not a property of the region.
|
||
That is why the region-wide autocorrelation is flat: the region holds chunks
|
||
with ~15 different strides averaging out to noise. The 15,360-byte window where
|
||
368 scored +0.315 is the third chunk, at region+30066, whose stride is 368.
|
||
|
||
The "ten-entry table" I earlier disconfirmed is real — chunks carry 5 to 30
|
||
tagged offsets, with 10 the most common on level 0. It was disconfirmed because
|
||
I tested for a fixed stride of 148 across a whole region; the table exists,
|
||
but 148 is one chunk's stride among many.
|
||
|
||
**Two corrections this produces.** Both earlier claims were half-right: real
|
||
structure, wrong scope. I reported a local measurement as a regional one in
|
||
both cases, which is the same mistake twice.
|
||
|
||
**Not decoded.** What the 16-byte header is, what the `0xF` nibble tags, where
|
||
the chunk data lives, and what the chunk *means* are all open. The container
|
||
framing is established; its contents are not.
|
||
|
||
## Chunk records resolve, and the layout is consistent across levels
|
||
|
||
**Range measured:** each level's post-shrub region, taken as the 900,000 bytes
|
||
before the constant trailer.
|
||
|
||
**Method:** for each chunk, read the tagged offsets as `0xF0000000 | low24`,
|
||
take the stride as the first delta, load one record per offset at
|
||
`chunk_start + low`, and count byte positions that are constant across that
|
||
chunk's records.
|
||
|
||
**Result: 1,096 chunks and roughly 13,000 records resolve on all 27 levels.**
|
||
Every level's chunks show 17-21% of byte positions constant across their own
|
||
records — a consistent field layout rather than noise.
|
||
|
||
Level 0 in detail: 27 chunks resolve, 466 records, 20.8% constant. The
|
||
stride-368 chunk (77 records) has 26 constant positions out of 368; the
|
||
stride-340 chunk (77 records) has 25 out of 340. Constant positions cluster
|
||
near the record end — 267-276 and following for stride 368, 258-268 for stride
|
||
340 — which is a footer or tail, not scattered padding.
|
||
|
||
**The tag is uniform.** Every tagged offset seen has top byte `0xF0`, i.e.
|
||
`0xF0000000 | low24`, on all levels.
|
||
|
||
**What this settles.** The container framing from the previous entry is not a
|
||
coincidence of byte patterns: the offsets genuinely address records, and those
|
||
records share a field layout. The chunk-relative base is confirmed because
|
||
resolving at `chunk_start + low` yields records with ~20% constant positions,
|
||
which does not happen by chance.
|
||
|
||
**Still not decoded.** The 16-byte header (opaque, per-chunk), the meaning of
|
||
the `0xF0` tag, which fields are index/length/offset, and whether a chunk is
|
||
moby_anim or something else. A parser can extract chunks and records
|
||
bounds-checked today; naming fields needs a domain argument this project has
|
||
not yet found.
|
||
|
||
## The chunk header: byte 14 is the record count
|
||
|
||
**Range measured:** all 27 levels, 1,123 chunks, over the 900,000 bytes
|
||
before the constant trailer.
|
||
|
||
**Method:** compare header byte 14 against the number of offsets actually
|
||
resolved for that chunk.
|
||
|
||
**Result: byte 14 equals the record count on 1,123 of 1,123 chunks, zero
|
||
mismatches.** This is the first genuinely *decoded* field in this container
|
||
rather than a measured correlation.
|
||
|
||
The parser now truncates the offset table to header byte 14, so a stray
|
||
tagged word appearing in a chunk's record data cannot silently extend the
|
||
table -- which would otherwise let the count disagree with its own header.
|
||
|
||
**Two more header bytes are near-constant and look like ASCII:**
|
||
|
||
- byte 13: only two values across 1,123 chunks -- `0x45` (1,076) and `0x46`
|
||
(47).
|
||
- byte 9: three values -- `0x44` (26), `0x45` (1,084), `0x46` (13).
|
||
|
||
`0x44`, `0x45` and `0x46` are ASCII `D`, `E` and `F`. That three-letter tag
|
||
appearing at two different byte positions with different distributions is
|
||
suggestive of a type or class field, but it is not established: the remaining
|
||
14 header bytes are high-entropy and do not read as ASCII at any shift tried,
|
||
so the ASCII in bytes 9 and 13 could be incidental.
|
||
|
||
**Still open:** the 14 opaque header bytes, whether bytes 9 and 13 are type
|
||
tags, which record fields are index/length/offset, and whether a chunk is
|
||
moby_anim. The `D`/`E`/`F` values are suggestive enough to be worth testing
|
||
against a known type table, but that has not been done.
|
||
|
||
## The header is a type ID, and the type determines the stride
|
||
|
||
**Range measured:** all 27 levels, 1,123 chunks.
|
||
|
||
**Method:** group chunks by their 16-byte header; for each distinct header,
|
||
collect the set of strides observed across every level it appears on.
|
||
|
||
**Result: only 64 distinct headers across 1,123 chunks.** The header is a
|
||
type identifier, not a per-chunk descriptor -- the same 16 bytes recur on
|
||
every level.
|
||
|
||
**63 of the 64 headers imply exactly one stride.** So header to record layout
|
||
is very nearly a function: the type determines the record size.
|
||
|
||
**Resolved: the single exception is three types sharing a header.**
|
||
|
||
Header `7542eff271b522a79e45348cca4509ff` was reported with three strides, 76,
|
||
92 and 104. It is not one type with a variable stride, and not a misparse.
|
||
|
||
It appears **three times per level**, always with record count 9, always in the
|
||
same order and always at the same offsets within the region. On level 0 and on
|
||
every other level checked:
|
||
|
||
| offset | stride | count | first offset | last offset |
|
||
|---|---|---|---|---|
|
||
| 863842 | 92 | 9 | 928 | 1640 |
|
||
| 867554 | 76 | 9 | 912 | 1520 |
|
||
| 872866 | 104 | 9 | 1000 | 1832 |
|
||
|
||
The offsets are 3,712 apart each time and the strides differ, so these are
|
||
three separate chunks that happen to share 16 header bytes.
|
||
|
||
**Consequence for the type ID.** 64 distinct headers for 66 distinct
|
||
record layouts, not 64 for 64. The header is *nearly* a type ID but does not
|
||
uniquely determine the layout: whatever distinguishes these three is not in the
|
||
16 bytes the header captures. Candidates not excluded: a distinguishing byte
|
||
sits outside the 16 captured, or the layout varies within a type.
|
||
|
||
This does not change the earlier conclusion that 63 of 64 headers imply one
|
||
stride -- that remains 63 of 64 -- but the mapping is not a bijection and
|
||
should not be used as one.
|
||
|
||
**The header does not encode the stride.** All 90 two-byte windows of
|
||
`header[0..14]` were tested: none maps to a unique stride across every group.
|
||
So there is no simple two-byte field to read the record size from, and the
|
||
stride must come from the offset table, as the parser already does, by
|
||
differencing the first two tagged offsets.
|
||
|
||
That is the right implementation anyway, and this makes it more than a
|
||
convenience: given three chunks sharing one header with three different
|
||
strides, any dispatch that assumed "header implies size" would be wrong for
|
||
1,123 chunks' worth of type coverage on at least one layout. Reading the
|
||
stride from the offsets is the only interpretation consistent with all the
|
||
measurements.
|
||
|
||
**The ASCII reading is confirmed as a type tag.** Grouping by header bytes 9
|
||
and 13:
|
||
|
||
| tag | chunks | distinct strides |
|
||
|---|---|---|
|
||
| `DE` | 26 | 2 (184, 344) |
|
||
| `EE` | 1050 | 46 |
|
||
| `EF` | 34 | 2 (268, 640) |
|
||
| `FF` | 13 | 1 (520) |
|
||
|
||
The rare tags are tightly coupled to their layouts -- `FF` is a single stride
|
||
across all 13 of its chunks -- while `EE` is heterogeneous with 46. That
|
||
pattern is what a type tag should look like, and it settles the earlier
|
||
suggestion that the D/E/F bytes were incidental: they are not.
|
||
|
||
**What this is good for.** A chunk can now be dispatched on its header to a
|
||
record layout, and 63 of 64 headers determine that layout's size without
|
||
inspecting any record. That is a usable parse entry point, and it is the first
|
||
point in this container where a field is doing real work rather than merely
|
||
correlating.
|
||
|
||
**Still open:** what the 64 types mean, which record fields are
|
||
index/length/offset, and whether any type is moby_anim.
|
||
|
||
## Chunk records reference moby class IDs at 38x chance
|
||
|
||
**Range measured:** level 0, the 900,000 bytes before the trailer; 28 chunks,
|
||
475 records, 67,780 `u16` slots. Reference set: the 238 distinct `o_class`
|
||
values from level 0's moby class table, read through `LevelCoreHeader`.
|
||
|
||
**Method:** scan every `u16` in every record; count how many are in the moby
|
||
class set. Compare against the rate expected if hits were chance, at
|
||
238/65536 per slot.
|
||
|
||
**Result: 9,493 hits against 246.1 expected -- 38.6x excess.**
|
||
|
||
The hits are not spread. They concentrate in a contiguous run of `u16` slots:
|
||
|
||
| slot | hits | | slot | hits |
|
||
|---|---|---|---|---|
|
||
| 138 | 188 | | 134 | 173 |
|
||
| 135 | 182 | | 136 | 168 |
|
||
| 137 | 180 | | 129 | 139 |
|
||
| | | | 130 | 131 |
|
||
|
||
Slots 129 through 140 -- twelve consecutive `u16`, that is bytes 258 through
|
||
280 of the record -- carry 105 to 188 hits each, and nothing else in the record
|
||
comes close except a separate cluster at slot 38.
|
||
|
||
**What this looks like.** A contiguous twelve-value array in which most entries
|
||
are moby class identifiers is what a model or instance definition listing the
|
||
mobies it references would look like. Every chunk contributes in proportion to
|
||
its record count: the stride-340 chunk with 77 records has 1,451 hits, the
|
||
stride-368 chunk 1,384, the stride-148 chunk with 10 records has 254.
|
||
|
||
**What this does not show.** Enrichment is not identification. A 38x excess
|
||
over chance is strong evidence that these values are moby class IDs rather
|
||
than incidental small integers, but it does not establish that the containing
|
||
chunk is `moby_anim` -- an instance table, a model list and an animation table
|
||
would all reference class IDs. The format is still unnamed, and so is the
|
||
meaning of slots 129-140.
|
||
|
||
**Correction to the above: the slot offsets were a pooling artifact.**
|
||
|
||
The entry above reported slots 129 through 140. Repeating the scan per chunk
|
||
type shows that is wrong: **every chunk type places its class array at a
|
||
different offset.**
|
||
|
||
| type (header prefix) | top hit slots |
|
||
|---|---|
|
||
| `aac4` | 129-134 |
|
||
| `82c4` | 104-109 |
|
||
| `4ec4` | 106-111 |
|
||
| `b8c4` | 120-125 |
|
||
| `9943` | 82-87 |
|
||
| `79c4` | 111-115 |
|
||
| `3245` | 24-29 |
|
||
|
||
Slots 129-140 were simply the type with the largest stride, dominating a scan
|
||
that pooled all types together. The previous entry flagged exactly this as
|
||
unverified, and the check confirmed the flag.
|
||
|
||
**What survives the correction.** In every one of the 26 types the top hits
|
||
form a **contiguous run of consecutive `u16` slots**. That is the real
|
||
invariant, and it is per-type: each chunk type has its own class array at its
|
||
own offset, of its own length.
|
||
|
||
So the corrected statement is: chunk records contain a contiguous array of
|
||
moby class IDs, and where that array sits depends on the chunk's type header.
|
||
This is consistent with each type being a distinct record layout, which is
|
||
what the header-implies-one-stride result already indicated -- but the class
|
||
array is a per-type field, not a common one at a fixed offset.
|
||
|
||
The moby reference itself is unaffected: 38.6x excess measured across all
|
||
records stands, since it does not depend on which slots were counted.
|
||
|
||
## Shrub body records on level 0: six varying bytes, two constant
|
||
|
||
**Range measured:** level 0, shrub class table 7 entries at core index offset
|
||
10176. Six classes are bounded by a successor; the seventh runs to the end of
|
||
`core_data` and is excluded, because its span contains the whole post-shrub
|
||
region and is not wholly records.
|
||
|
||
**Method:** bound each span by the next class's geometry offset, take the body
|
||
from `data_offset` to the span end, and split into 8-byte records.
|
||
|
||
**Result: 164 records across 6 bounded bodies**, body lengths 192 (twice), 208
|
||
(twice), 224, 240.
|
||
|
||
Per-byte, over those 164 records:
|
||
|
||
| byte | distinct | zero |
|
||
|---|---|---|
|
||
| 0 | 74 | 60 |
|
||
| 1 | 58 | 65 |
|
||
| 2 | 76 | 43 |
|
||
| 3 | 62 | 48 |
|
||
| 4 | 72 | 35 |
|
||
| 5 | 59 | 38 |
|
||
| **6** | **1** | **164** |
|
||
| **7** | **1** | **164** |
|
||
|
||
Bytes 6 and 7 are zero in every record. Bytes 0 to 5 take 58 to 76 distinct
|
||
values of a possible 164, with only 21-39% zero -- that is, uniformly varied
|
||
rather than sparse.
|
||
|
||
Read as `u32` the fields are not plausible: `u32[0]` ranges 0 to 4,294,871,316
|
||
and no field is monotone across the body. So the six bytes are not two packed
|
||
32-bit values.
|
||
|
||
**What this settles.** The six-varying-plus-two-constant shape holds on level 0
|
||
as a per-level partition rather than only in aggregate across the disc's
|
||
8,938 records, which is what rule 3a asks for. Two trailing zero bytes are
|
||
padding or a flag pair, not data.
|
||
|
||
**What it does not.** Resolved by a follow-up measurement: the six bytes are
|
||
not an `f32` pair either, and not narrow `u16`s.
|
||
|
||
- `f32 @0`: 158 of 164 finite, but the range runs to about 1e35. Not positions
|
||
or normals. The 6 non-finite values (3.7%) are far above the 0.05% that
|
||
random bytes would produce, so the field is *structured* but is not an `f32`.
|
||
- `f32 @4`: all 164 finite, range 0 to 9.18e-5. This looks promising until you
|
||
notice bytes 6 and 7 are always zero, which pins this value to a coarse grid
|
||
and explains the tiny range. It is not independent evidence.
|
||
- `u16 @0`, `@2`, `@4`: distinct counts 85, 90 and 80, each spanning essentially
|
||
the whole 0..65534 range. These are not small counts or indices.
|
||
|
||
So excluded are: two packed `u32`, an `f32` pair, and narrow `u16` fields. The
|
||
six bytes remain uninterpreted.
|
||
|
||
**Process note.** The first attempt at this used `ClassEntry` and a
|
||
hand-computed `o_class` offset instead of `ShrubClassEntry`, and every
|
||
`ShrubHeader::region` call failed. The parser was not wrong; the caller was.
|
||
The working call shape was already in the disc test.
|
||
|
||
## tie: both indirections, and one reading disproved
|
||
|
||
**The chain, which is two steps and easy to get wrong.** The tie class **table**
|
||
lives in the level's `core_index.bin`, not `core_data.bin`. On level 0 the three
|
||
tables chain as moby at `192`, 238 x 32 bytes later tie at `7808`, 74 x 32 later
|
||
shrub at `10176`, which closes. Reading the same table out of `core_data.bin`
|
||
produces plausible garbage rather than an error -- `offset_in_asset_wad` comes
|
||
back as 1,217,208,842, which is outside every buffer, so the mistake announces
|
||
itself if you look. Then `ClassEntry::offset_in_asset_wad` addresses
|
||
**`core_data.bin`**, despite its name and despite `core_data.wad` sitting
|
||
alongside it.
|
||
|
||
**Verified:** all **1,864** tie classes resolve, `o_class` agreeing on every one,
|
||
and all **5,592** LOD block offsets land inside `core_data`.
|
||
|
||
**Disproved:** the word at `+0` of a 16-byte unit inside an LOD block. On the
|
||
first twelve units of level 0's class 0 it rises strictly -- 288, 1520, 2848,
|
||
3968, 5088, 6096, 7312, 8528, 9792, 10976, 12016, 13072 -- which reads exactly
|
||
like a linked list, and the accompanying `i16`s looked quantised (two distinct
|
||
values in `i16[0]`, three in `i16[1]`).
|
||
|
||
Across the whole population it is strictly increasing on **0 of 1,864** blocks,
|
||
and `i16[0]` takes **8,969** distinct values on level 0 alone rather than two.
|
||
The sample was twelve units out of 1,724,334. Both the monotonic reading and the
|
||
quantisation were artifacts of that sample, and the disproof is asserted in
|
||
`crates/gc-extract/tests/disc.rs` so it cannot creep back.
|
||
|
||
**Record size: RETRACTED. Block size is a multiple of 16; the records are not a
|
||
uniform 16 bytes.** The earlier
|
||
"divisible by 16" figure was computed from the "next `packets` offset ends this
|
||
block" rule, which is unsound because **`packets` is strictly increasing on only
|
||
461 of 1,864 classes**. On the other 1,403 the two offsets are unordered, so they
|
||
cannot both be block starts into one ascending region.
|
||
|
||
Restricting to the 461 classes where the rule does hold: **922 of 922**
|
||
consecutive block sizes are a multiple of 16. That was taken as evidence for a
|
||
16-byte record. **It is not**, and the record-size claim is withdrawn.
|
||
|
||
What actually survives, and what killed it:
|
||
|
||
- Every one of those **922 blocks opens with a `u32` that is positive and a
|
||
multiple of 4** (922/922), ranging 48 to 1,744 with 17 distinct values across
|
||
level 0's LOD0 blocks. So the block opens with a count, not a size and not a
|
||
pointer.
|
||
- Reading that `u32` as a byte size and chaining to the next block lands exactly
|
||
on the next block start on **0 of 1,383** blocks. It is not a size chain.
|
||
- Assuming `[header][count][count uniform records]` and solving for the record
|
||
size: **no record size tiles the block for any header size up to 512 bytes.**
|
||
Not 16, not 12, not 8.
|
||
- Reading the block as uniform 16-byte units gives fields spanning their entire
|
||
legal range -- `u32@0` from -2,139,029,504 to 2,139,029,587, `u16@8` and
|
||
`u16@10` both 0..65,535, and 8,969 distinct values in `i16@0` alone on level 0.
|
||
Uniform units would not produce that.
|
||
|
||
So the 16 is a property of the **block size**, not of a record. The previous
|
||
commit's "record size is 16 bytes" was an inference from a divisibility fact that
|
||
does not carry it, and it is withdrawn rather than softened.
|
||
|
||
**`packets[0]` is a constant, not an offset.** It reads **256 on 1,864 of 1,864**
|
||
classes, while `packets[1]` and `packets[2]` vary widely and land at different
|
||
addresses in `core_data` on every class. Read as a block start, 256 points just
|
||
past the 0x48-byte header -- which is exactly how it was being used, and exactly
|
||
the kind of field that gets mistaken for a pointer. It is now
|
||
`TIE_PACKETS0` and nothing depends on it.
|
||
|
||
**Still not decoded:** what the 16 bytes hold. `i16[0]` at `+4` takes 8,969
|
||
distinct values on level 0 and is not the two-valued field a 12-unit sample
|
||
suggested; the word at `+0` is not a run offset.
|
||
|
||
## tie: the LOD block header, decoded
|
||
|
||
Over the **922** blocks where "the next `packets` offset ends this block" holds,
|
||
the block's own 16-byte header is measured. `gc_formats::tie::TieBlockHeader`
|
||
parses it and a real-disc test asserts every line below.
|
||
|
||
+0x00 u16 count multiple of 16 on 922/922; range 16..2096;
|
||
72 distinct values disc-wide
|
||
+0x02 u16 zero on 922/922
|
||
+0x04 u16 ONE OF FOUR VALUES DISC-WIDE: 0x501, 0xa02, 0xf03,
|
||
0x1404 -- low byte 1..4, high byte 5/10/15/20
|
||
+0x06 u16 15 distinct, 0x0206..0x0512
|
||
+0x08 u16 135 distinct, 0x0b05..0x4ee4
|
||
+0x0a u16 45 distinct
|
||
+0x0c u16 94 distinct, 0x010e..0x0553
|
||
+0x0e u16 zero on 914/922
|
||
|
||
**`+0x04` is packed, not a magnitude.** Four values across the whole disc, with
|
||
the two bytes moving independently (low 1-4, high 5/10/15/20 in lockstep), is two
|
||
small fields sharing a word. That it is a texture index paired with a scale is a
|
||
guess; that it is two fields is not.
|
||
|
||
**The count is not a record count.** `block_size - header` is divisible by it on
|
||
**0 of 922** blocks, for *any* record size and *any* header up to 512 bytes.
|
||
Combined with block sizes always being multiples of 16, that rules out a uniform
|
||
stride without a header.
|
||
|
||
## tie: the block is a table of 16-byte records (from RAM)
|
||
|
||
**Range measured:** every tie LOD block of level 1 whose leading word is a
|
||
non-zero multiple of 16 -- 16 blocks -- plus the two read by hand in full.
|
||
|
||
**Method:** locate each block's record table in PCSX2's RAM by its twelve field
|
||
bytes (the records are the only part of a block with a distinctive 12-byte
|
||
run), then compare every record against its disc copy. Each block has several
|
||
copies in RAM; the one that matches is the loaded copy.
|
||
|
||
**Result: on 16 of 16 blocks, every record's leading word has become
|
||
`table_address + word`, and the other twelve bytes are byte-identical to the
|
||
disc.** So:
|
||
|
||
- A block is `[16-byte header][record table][vertex payload]`, and the records
|
||
are **16 bytes**: a leading `u32` pointer and twelve bytes of fields.
|
||
- The header's word at `+0x00` is the **byte offset from the block start to
|
||
the vertex payload** -- that is, the length of the record table. This is what
|
||
the "count" reading was groping at: it is a multiple of 16 on all 2,177 blocks
|
||
because the table holds 16-byte records, it is never zero because there is
|
||
always at least the header-sized first record, and its 42 distinct values are
|
||
table lengths, not counts of anything in the payload.
|
||
- Every record's leading word is a pointer **relative to the start of the record
|
||
table**, relocated at load to `table + word`. The twelve field bytes are
|
||
copied verbatim.
|
||
|
||
This also explains the dead end that ran before it. The payload length divided
|
||
by the word at `+0x00` is a whole number on **0 of 922** blocks, at any record
|
||
size and any header up to 512 bytes -- which is the correct measurement of the
|
||
wrong hypothesis: the word never described the payload, it described where the
|
||
payload starts. A count of what follows a pointer is not the pointer.
|
||
|
||
`TieBlockHeader::data_offset` and `TieBlockHeader::records` carry it, bound by
|
||
the offset, and the disc test asserts the table is bounded on every block.
|
||
|
||
### The records are R&C1's `TiePacketHeader` (2026-10-11)
|
||
|
||
With the record table located and the fields isolated, the reference settles
|
||
what they are. **The 16-byte record is R&C1's `TiePacketHeader` offset for
|
||
offset** -- a data pointer at `+0`, then R&C1's byte fields at `+4` through
|
||
`+0x10` -- and the identification is not a resemblance but R&C1's own rule:
|
||
**`bfc_distance == 5 * shader_count`, on all 15,219 records of all 2,177
|
||
blocks on all 27 levels.**
|
||
|
||
Measured over the same set:
|
||
|
||
| relation | holds |
|
||
|---|---|
|
||
| `bfc_distance == 5 * shader_count` | **15,219 / 15,219** |
|
||
| `vert_ofs <= vert_size` | all |
|
||
| `scissor_size <= scissor_ofs` | all |
|
||
| `control_size <= control_count` | all |
|
||
| `color_ofs == vert_size` | 12,640 / 15,219 |
|
||
| `color_ofs == vert_ofs + vert_size` (R&C1's rule) | **0 / 15,219** |
|
||
| `color_count`, `strip_count`, `strip_vertex_count` zero | 15,089 / 15,077 / 15,078 |
|
||
|
||
`TieRecord` parses all twelve fields; `invariants_hold` asserts only the
|
||
universal four, and the disc test counts the rest. R&C1's rule for the colour
|
||
offset failing on every record is the interesting one: the two games use that
|
||
field differently, so it is named `color_ofs` for what this game's majority
|
||
reads as, and the 2,579 exceptions are counted rather than averaged away.
|
||
|
||
**A packet-length formula is retracted.** `(vert_ofs + vert_size +
|
||
scissor_ofs + scissor_size) * 16` was measured at 61 of 61 pairs and is not
|
||
reproduced on the correct blocks -- 0 of 13,042. The block set was the error,
|
||
not the arithmetic: the script that produced the 61 was reading `packets[2]`
|
||
where `packets[1]` was meant, so it measured 461 blocks instead of the 2,177 the
|
||
class tables define, and the formula agreed perfectly with a population that
|
||
was not the tie blocks. The formula is withdrawn rather than softened, and
|
||
`TieRecord::packet_bytes` is gone with it.
|
||
|
||
### It is now a command, and it verifies two levels
|
||
|
||
The check above was a script; it is now `gc-trace verify-tie-records
|
||
<unpacked-dir> <level> <class> <lod> <src>`, which resolves the class, parses
|
||
the block, searches the RAM image for the loaded record table and reports what
|
||
it found. Exit status is 0 when every record relocated with verbatim fields.
|
||
|
||
Run over every (class, lod) slot of two levels, from PCSX2 savestates:
|
||
|
||
| level | slots with a block | verified | mismatched | absent |
|
||
|---|---|---|---|---|
|
||
| 0 | 87 | **87** | 0 | 61 |
|
||
| 1 | 104 | **104** | 0 | 78 |
|
||
|
||
So the relocation rule holds on **every** LOD block of two levels that the game
|
||
had loaded. Two operational notes worth keeping:
|
||
|
||
- **The live emulator is not a stable input.** Verifying level 1 over PINE
|
||
matched 104 of 104 while the user was on that planet; the same command
|
||
minutes later, after they had returned to level 0, found the tables in RAM
|
||
but no matching copy -- because the level was no longer loaded. A miss there
|
||
is not a counterexample, and the report says "no copy matched" rather than
|
||
"failed".
|
||
- **A class index past the table count is an error**, not a mismatch, so a
|
||
sweep should stop at the level's class count rather than at a round number.
|
||
|
||
What the records' fields name and what the payload holds are still open.
|
||
|
||
### What the payload is, and what retired the coordinate reading
|
||
|
||
The payload is **not random**, and that took a control to establish. Reading each
|
||
block as 3 x `f32` triples, 53-64% of reads land within +/-5000 of origin; a
|
||
uniform-random control of identical length gives **16%**. So the content really
|
||
is float-shaped.
|
||
|
||
But the rate is **flat across every start offset and every stride tried**
|
||
(0x10..0x40 start, 8/12/16/20/24 stride: 53-65% throughout). A statistic that
|
||
cannot distinguish the right stride from the wrong one cannot identify a stride,
|
||
which is rule 4's exact case. The flatness is the informative part: combined
|
||
with the count not tiling, it says the records are **variable length**.
|
||
|
||
A VIF1 command walk was implemented and tested against the blocks: it consumes
|
||
**0 of 88** exactly. So the payload is variable-length and not a VIF1 CPU-format
|
||
stream. Not WAD LZ77 either -- `wad::decompress` rejects the block at every
|
||
start offset tried.
|
||
|
||
Scanning every (start, stride) pair from 0x10 to 0x7f for byte positions with a
|
||
small distinct-value vocabulary -- the signature of integer index fields -- finds
|
||
**none**: every byte position takes all 256 values at every alignment. For float
|
||
payload that is expected, and it is why the field scan found nothing rather than
|
||
confirming anything.
|
||
|
||
**Still open: what the payload holds.** Established is that it is variable-length,
|
||
float-shaped, non-random, not LZ77, and not VIF1. Not established is any record
|
||
boundary.
|
||
|
||
## tie: the LOD block is self-delimiting, and `packets == 0` means absent
|
||
|
||
**The terminator.** Every tie LOD block ends with the byte `0xf6` followed by
|
||
zeros, on **2,177 of 2,177** real blocks across all 27 levels. That was worth a
|
||
control, because `0xf6` could have been a coincidence of byte frequency: it
|
||
occurs in payloads at **0.83x** the uniform rate, i.e. slightly *less* often
|
||
than chance, while the last non-zero byte is `0xf6` on 909 of 922 blocks in the
|
||
first subset tested against **17 of 922** for the same blocks cut at a random
|
||
point. So it is a terminator, not a common byte.
|
||
|
||
This is what makes the blocks parseable at all. `packets[1]` bounds a block on
|
||
only 461 of 1,864 classes, so the terminator is the only general way to find one.
|
||
|
||
**`packets == 0` means no block at that LOD.** Of the 3,728 (class, LOD) slots,
|
||
**2,177** point at a block and **1,551** are exactly zero. The zero cases are not
|
||
degenerate blocks -- they are absent ones.
|
||
|
||
### A measurement I got wrong, twice, in opposite directions
|
||
|
||
The first pass over this header reported **3,728 blocks, all terminating** and
|
||
`packed_texture` taking **842** distinct values, contradicting an earlier
|
||
four-value result. I retracted the four-value claim on that basis.
|
||
|
||
Both numbers were wrong, and they were wrong the same way. The scan did not skip
|
||
`packets == 0`; it computed `class_base + 0` and walked forward from the **class
|
||
body**, finding a terminator there by accident. That contaminated 1,551 of the
|
||
3,728.
|
||
|
||
With the zeros excluded, the slot takes **exactly four** values on all 2,177
|
||
blocks -- 0x501 on 1,069, 0xa02 on 720, 0xf03 on 350, 0x1404 on 38 -- and all
|
||
2,177 terminate.
|
||
|
||
So: the four-value claim was right, my retraction of it was wrong, and the
|
||
retraction was made on a number from the same contaminated set. A retraction is
|
||
not automatically better than the claim it retracts; it needs its own bounds.
|
||
|
||
### Corrected header, over all 2,177 blocks
|
||
|
||
+0x00 count multiple of 16 on all; 42 distinct; never zero
|
||
+0x02 zero on all
|
||
+0x04 exactly four values: 0x501, 0xa02, 0xf03, 0x1404
|
||
+0x06 16 distinct, never zero
|
||
+0x08 180 distinct, never zero
|
||
+0x0a 61 distinct, never zero
|
||
+0x0c 125 distinct, never zero, clustered 0x043f..0x0443
|
||
+0x0e 11 distinct, zero on 2,162 -- the only non-universal field
|
||
|
||
`+0x0c` clusters around 1,087 to 1,091, close enough to a block length to be
|
||
worth stating, but it is **not** the length: the difference against the true
|
||
payload length scatters from -12 to -172.
|
||
|
||
## chunk records: the leading byte is a bucket index, not data
|
||
|
||
Characterised across **all 77 chunk types**, 1,509 chunks and 34,491 records on
|
||
all 27 levels.
|
||
|
||
**Byte 0 of every record is a small aligned index.** It takes 16 discrete values
|
||
per chunk type -- 0x00, 0x10, ... 0xf0 -- with near-uniform counts *within* each
|
||
type. Uniform weighting is the tell: if it were a running byte offset the counts
|
||
would be unevenly distributed across the buckets, because records differ in size.
|
||
|
||
```
|
||
type_id recs len byte0
|
||
0x007d7ec438 2430 328 0x00:162 0x10:162 0x20:162 ... 0xf0:135
|
||
0x00d3d8c4a9 2430 308 0x00:162 0x10:162 0x20:162 ... 0xf0:135
|
||
0x00429ec43d 2079 368 0x00:162 0x10:135 0x20:135 ... 0xf0:108
|
||
0x00d76cc453 810 108 0x00:27 0x08:27 0x10:27 ... 0xe8:27 <- steps by 8
|
||
```
|
||
|
||
**The alignment step is per type, not global.** The leading byte is a multiple
|
||
of 8 on **34,432 of 34,491** records, but a multiple of 16 on only **33,349** --
|
||
the 1,142-record difference falls in seven types, four of which step by 8.
|
||
|
||
59 records in three types break even the 8-byte rule. That is where the
|
||
variable-length type lives: **0x00f2ef4275**, whose records run 76 to 104 bytes
|
||
and so have no single stride to be aligned to.
|
||
|
||
**Not named.** What the bucket indexes is not established. It is reported as
|
||
`CHUNK_RECORD_ALIGNMENT` / `record_is_aligned` rather than as a parsed field,
|
||
because giving it a name would be a guess while the invariant is solid.
|
||
|
||
### A measurement I initially got wrong
|
||
|
||
The first version of the constant claimed a 16-byte multiple on 34,432 records.
|
||
34,432 is the **multiple-of-8** count; the multiple-of-16 count is 33,349. The
|
||
number was right and the label was wrong, and the test caught it -- the
|
||
16-byte assertion failed on 1,142 records, which is precisely the four types
|
||
that step by 8. Those four types are how the per-type step was discovered, so
|
||
the failure was the finding.
|
||
|
||
Worth stating plainly: this is the second time this session a correct number
|
||
carried a wrong label, after the tie four-value retraction. Both times the test
|
||
was the thing that caught it, which is the argument for asserting the number in
|
||
a test at all rather than only in prose.
|
||
|
||
## The chunk record's leading byte is a per-chunk phase counter
|
||
|
||
Follow-up to the alignment result, and it settles what the leading byte is.
|
||
|
||
Within one chunk the leading byte steps by a constant and wraps:
|
||
|
||
offsets: 1420 1748 2072 2392 2716 3044 3372 3700 4028 4352 ...
|
||
byte0: 0 16 32 48 64 80 96 112 128 144 ...
|
||
|
||
i.e. `byte0 == step * ((index + phase) mod (256 / step))`. Sweeping every
|
||
`(step, phase)` over all 27 levels:
|
||
|
||
fully explained: 1,262 of 1,509 chunks
|
||
by step: 8 -> 51 chunks, 16 -> 554, 32 -> 520, 64 -> 137
|
||
unexplained: 247 chunks across 11 chunk types
|
||
|
||
**So on 1,262 chunks the leading byte carries no per-record information.** It is
|
||
a deterministic function of the record's index and the chunk's phase. That makes
|
||
it a position marker, not data -- which is why the earlier "16 discrete values
|
||
with uniform counts" pattern showed up.
|
||
|
||
`ResolvedChunk::bucket_rule` returns the `(step, phase)` when the whole chunk
|
||
fits and `None` otherwise, so the 247 unexplained chunks are distinguishable
|
||
from the 1,262 explained ones and nothing skips a leading byte on the strength of
|
||
a rule that does not apply.
|
||
|
||
### Every chunk has a uniform record length
|
||
|
||
**1,509 of 1,509 chunks.** Length varies between chunks of the same type -- one
|
||
type runs 76 to 104 bytes -- but never within one chunk.
|
||
|
||
This **withdraws the previous commit's "variable-length type"**. That came from
|
||
taking `minlen`/`maxlen` over all of a type's records at once, pooling across
|
||
chunks; rule 3a's pooled claim, third time this session. Within any one chunk the
|
||
length is fixed.
|
||
|
||
## The first TWO bytes encode the record's own index
|
||
|
||
Continuing from the phase counter. Within a chunk:
|
||
|
||
i : 0 1 2 3 4 5 6 7 8 9 10 11 ... 15 16 17 ...
|
||
byte0: 0 16 32 48 64 80 96 112 128 144 160 176 ... 240 0 16 32
|
||
byte1: 0 0 0 0 0 0 0 0 0 0 0 0 ... 0 1 1 1
|
||
|
||
`byte0` is `step * (j mod g)` and **`byte1` is `(j / g)`**, where `g = 256/step`
|
||
and `j = index + phase`. So the pair is the record's index in mixed radix:
|
||
`byte0` carries the low digits scaled by `step`, `byte1` the quotient.
|
||
|
||
**Measured over all 27 levels: 1,235 of 1,509 chunks** fit this on both bytes.
|
||
Byte 0 alone fits 1,262, so the 27-chunk difference are chunks whose first byte
|
||
follows the rule and whose second does not.
|
||
|
||
**So on 1,235 chunks a record knows its own position**, and the two leading bytes
|
||
hold no per-record information. `ResolvedChunk::index_rule` returns the
|
||
`(step, phase)` only when the *whole* chunk fits on both bytes, so the 274 that
|
||
do not stay distinguishable.
|
||
|
||
### A number that moved when the implementation changed
|
||
|
||
A first probe reported **1,181**; the committed method reports **1,235**. The
|
||
method is right. The probe accumulated `ok0`/`ok1` flags across candidate phases
|
||
with an early-exit that could skip a phase which would have matched both bytes,
|
||
and it also disagreed with `bucket_rule` on the byte-0 count (1,208 against
|
||
1,262), which is what gave it away.
|
||
|
||
Recorded because the two numbers came from the same run and only the discrepancy
|
||
exposed the probe as wrong. The lesson is narrow and worth stating: when a
|
||
one-off probe and a committed helper disagree, check the probe first.
|
||
|
||
## Chunk record bodies: what was ruled out
|
||
|
||
With the framing settled (leading two bytes = record index), the body is bytes
|
||
`2..len`. Four probes on it, mostly negative.
|
||
|
||
**No count field.** For every chunk type, every `u16` and `u32` position in the
|
||
first 64 body bytes, at element strides 1, 2, 3, 4, 6, 8, 12 and 16: does the
|
||
stored value equal `(record_len - offset) / stride` on 90% or more of records?
|
||
**No type produced a hit.** The record does not carry an element count near its
|
||
head.
|
||
|
||
**The body is dense, not float-dominated.** For the largest type (2,430 records,
|
||
326-byte bodies):
|
||
|
||
f32 reads within +/-5000 of origin : 74%
|
||
uniform-random control : 49%
|
||
zero-byte fraction : 13%
|
||
|
||
Above baseline but far short of what the tie blocks look like (53-64% against
|
||
16%, with 43% zero bytes). The chunk bodies are denser and less float-shaped than
|
||
tie geometry.
|
||
|
||
**There is an index-like region near the tail.** In the largest type, **12 of 163**
|
||
`u16` slots have a maximum below `0x1000` -- too small for float or half-float
|
||
data, so those slots look like indices or small counts. They cluster at slots
|
||
111-121 and 130-131.
|
||
|
||
### A sample-size correction
|
||
|
||
Dumping six records of one chunk showed slots 112-119 **zero on every record** and
|
||
a long run of other constant slots, which read as "the record is mostly a
|
||
per-chunk template". Measured properly across the 1,171 chunks with at least 8
|
||
records:
|
||
|
||
u16 slots constant across every record in their chunk : 15,422 of 164,390 (9.4%)
|
||
chunks whose records are ENTIRELY constant : 0 of 1,171
|
||
|
||
So 9.4%, and no chunk is a template. Six records looked like a majority because
|
||
that chunk's tail happens to carry shared constants -- the per-type figures range
|
||
from 9.4% overall to 55.6% on one type. A pooled figure over all types would have
|
||
hidden that spread again, so it is reported per type as well.
|
||
|
||
## What the loader does to a chunk (from RAM)
|
||
|
||
Verified against PCSX2 level-0 RAM; method and full dump in
|
||
`plan/pcsx2-verification.md`.
|
||
|
||
The chunk payload is copied **verbatim** -- a probe from inside a record hits at
|
||
`chunk_base + record_offset` with no rescaling. Two things change around it:
|
||
|
||
1. **The offset table is rewritten to absolute EE addresses.** Disc offsets
|
||
`0xf000057c, 0xf00006b0, 0xf00007e4, 0xf0000914` become
|
||
`0xf185387c, 0xf18539b0, 0xf1853ae4, 0xf1853c14` in RAM. The
|
||
`0xf0000000 | low24` tag becomes `0xf1850000 | low24` -- tag replaced by the
|
||
load base, low 24 bits untouched.
|
||
|
||
2. **Every record gains a constant 2-byte prefix `02 00`,** and the rewritten
|
||
table points at *that* rather than at the record data. Measured on all
|
||
**90 of 90** records: prefix `02 00` on every one, and the disc's index bytes
|
||
preserved unchanged at `+2`/`+3` on every one.
|
||
|
||
The third result is the useful one: the byte at `+2` takes exactly
|
||
`0, 16, 32, ... 240` -- the 16 values `index_rule` predicts for `step = 16,
|
||
phase = 0`. The mixed-radix record index, derived from the disc, is corroborated
|
||
on the running game's own copy.
|
||
|
||
### Correction: the `02 00` is on the disc, and a record starts two bytes earlier
|
||
|
||
What the loader does to the *pointer* is as described above. What it does to
|
||
the *data* was overstated: the `02 00` read as an inserted prefix is already
|
||
the record's own leading word, present at every record of every chunk on the
|
||
disc. Three measurements, in order:
|
||
|
||
- **The 2 bytes before the payload are on disc.** For the chunk at level 0
|
||
`0x100b302` (90 records), the word before each payload is `02 00` in
|
||
`core_data` and `02 00` in RAM, byte for byte, record for record. The first
|
||
chunk of the same level carries `08 00` instead. Nothing was inserted.
|
||
- **So the RAM table's pointer is simply two bytes earlier.** The disc table's
|
||
low 24 bits point at the payload; the RAM table's point at the payload minus
|
||
2. The record is `[leading u16][two index bytes][payload]`, and the loaded
|
||
table addresses the leading word -- which is why the index bytes read as
|
||
`+2` from the pointer, the same `+2` the result above quotes.
|
||
- **The leading word is a field, not a prefix.** It is the same at every
|
||
record of a chunk on **1,439 of 1,509 chunks** (`None` on the 70 whose
|
||
records disagree, which the parser reports rather than averaging), it is
|
||
fixed within **every one of the 71 chunk types**, and its values are only
|
||
`1, 2, 4, 6, 8, 10` -- 554 chunks carry 2, 520 carry 4. `Chunk::prefix`
|
||
carries it.
|
||
|
||
What the value counts is not established, and two readings were measured and
|
||
disproved rather than left as guesses:
|
||
|
||
- **Not an entry count.** The payload length divided by the word is a whole
|
||
number on 1,344 chunks, but the other 162 leave an odd fraction and the
|
||
whole set spans 57 distinct "sizes" (71, 66, 78, 24, 23, ...). A real entry
|
||
size is a small clean number, not a set like that.
|
||
- **Not the number of distinct leading bytes.** That agrees on 0 of 1,506
|
||
chunks.
|
||
|
||
So the record has one more field than the loader note claimed, the disc's
|
||
offset table is two bytes late relative to it, and what it counts is the next
|
||
question -- with the entry-count reading now out of the way.
|
||
|
||
## The tie vertex element is 16 bytes
|
||
|
||
With `TieRecord` identified as R&C1's `TiePacketHeader`, each record's leading
|
||
word points at its packet in the block payload. Following it and stepping
|
||
`vert_size` elements from `vert_ofs` reaches the vertices.
|
||
|
||
**Range measured: 95,612 vertices** on levels 0-2.
|
||
|
||
**Element size, settled by comparison rather than assumed.** Requiring the
|
||
texcoord pair to be 12-bit fixed -- both words multiples of 4,096, which is how
|
||
0x1000 = 1.0 reads -- gives:
|
||
|
||
element 16 bytes : 78,919 / 95,612 = 82.5%
|
||
element 8 bytes : 45,661 / 95,612 = 47.8%
|
||
element 12 bytes : 44,834 / 95,612 = 44.8%
|
||
|
||
Sixteen wins by a factor of 1.7 over the nearest alternative, so it is the
|
||
element size.
|
||
|
||
**Observed layout**, 8 x `i16`. Scoring each adjacent slot pair for 12-bit
|
||
fixedness over 36,501 level-0 vertices locates the fields:
|
||
|
||
(i16[0], i16[1]) 1.1% not a texcoord
|
||
(i16[2], i16[3]) 4.8% not a texcoord
|
||
(i16[4], i16[5]) 73.6% <- the texcoords
|
||
(i16[6], i16[7]) 76.6% <- the near-constant 4096/0 pair
|
||
|
||
**`(6,7)` scores higher and is not a texcoord.** It is the near-constant
|
||
`4096, 0` pair -- `4096` is 1.0 in 12-bit fixed and `0` is zero, so *both* are
|
||
trivially multiples of 4,096. That is the constant artifact again: a fixed pair
|
||
would score 100% anywhere. The field is `(4,5)` at 73.6%, with `(0,1)` and `(2,3)`
|
||
ruled out at 1.1% and 4.8%.
|
||
|
||
The first four slots span the full `i16` range with 10,803 to 13,421 distinct
|
||
values each, so they are neither texcoords nor small fields.
|
||
|
||
**The vertex base is `vert_ofs` alone.** Adding the shader or control region to
|
||
it makes things *worse*, so the 73.6% residual is not a mislocated base:
|
||
|
||
vert_ofs only 73.6%
|
||
bfc + vert_ofs 69.8%
|
||
bfc + control_count*control_size + vert_ofs 66.7%
|
||
|
||
**Not established:** what any of the eight fields *means*, and why the texcoord
|
||
rate is 82.5% rather than 100%. The residual varies with record shape -- records
|
||
with `(shaders 2, control 7, control_size 2)` hit 60%, `(shaders 1, control 18,
|
||
control_size 5)` hit 89% -- so it is not one bad block but something that varies
|
||
per packet, possibly a second texcoord encoding for records that need it.
|
||
|
||
Recorded because the 16-byte answer is the *opposite* of what the earlier
|
||
commit concluded from the disc alone. Reading the header word at `+0x00` as an
|
||
element count gave 0 of 922 blocks dividing; it is the byte offset to the payload,
|
||
so the table it bounds has 16-byte records. The number was right and the reading
|
||
was wrong, which is the error the packet-header identification corrected.
|
||
|
||
## Collision payload: what the verbatim mapping settles, and what it does not
|
||
|
||
With the payload confirmed verbatim in RAM, the disc bytes can be read directly.
|
||
The grid dimensions are the **first two `u16`** of the payload -- level 0 reads
|
||
**11 x 33 = 363 cells**, level 1 **24 x 16 = 384**, level 2 **1 x 53 = 53**.
|
||
Level 2 being one cell wide is worth noting; the grids are not all similar in
|
||
shape.
|
||
|
||
**Disproved this round -- an ascending `u32` offset table.** A probe from payload
|
||
`+1024` reads `f4ae0000 40af0000 c4b2000020b20000 ...`, which is three ascending
|
||
values before the fourth goes *down* (`0xb2c4` to `0xb220`). Read as a table that
|
||
was a three-value impression. Scanning all three levels for maximal ascending
|
||
`u32` runs of 8 or more entries finds **zero**, on every level.
|
||
|
||
**Disproved this round -- a global alternating-zero stride.** The head of level 0
|
||
shows an obvious `(value, 0)` pattern from about `i16[37]` onward:
|
||
|
||
[32..] [1808, 1872, 2198, 0, 34, 86, 25848, 1, 25824, 1,
|
||
-13028, 0, -12284, 0, -12228, 0, -12716, 0, ...]
|
||
|
||
Measured over the whole payload rather than the first 96 values, both alignments
|
||
score far below what that sample suggests:
|
||
|
||
even-slot aligned: second half zero on 18% of pairs
|
||
odd-slot aligned : second half zero on 13% of pairs
|
||
|
||
The sample again. What the global numbers *do* say is that the payload is heavily
|
||
zero-bearing -- 18% against a chance rate of about 0.0015% -- which is what a
|
||
sparse or variable-length encoding looks like, and is consistent with the
|
||
variable-length reading established earlier. The 18-against-13 split is a real
|
||
alignment preference but a weak one.
|
||
|
||
**Still not decoded.** The cell layout. The useful position now is that the bytes
|
||
can be read as they are, and that they are sparse and variable-length rather than
|
||
fixed-stride -- so the next attempt should look for a terminator-delimited walk
|
||
through the sparse region, not for a record size.
|
||
|
||
## Collision: the segmentation is not recoverable from zero structure
|
||
|
||
Continuing on the verbatim payload, with the grid known.
|
||
|
||
**Cells are not zero-separated.** Counting maximal runs of non-zero `u16` across
|
||
the whole payload:
|
||
|
||
level 0 363 cells 15,115 runs mean run 25.3 u16
|
||
level 1 384 cells 24,647 runs mean run 51.5 u16
|
||
level 2 53 cells 31,512 runs mean run 30.3 u16
|
||
|
||
Two to three orders of magnitude apart, so a zero run is not a cell boundary.
|
||
Cells are not uniform blocks either: level 0 averages 2,488 bytes per cell and
|
||
level 2 averages 43,684, which cannot both be one fixed-size record.
|
||
|
||
**The run lengths are not records.** Their histogram has a long tail of
|
||
near-power-of-two values -- 2,140, 1,184, 840, 816, 628, 604, 562 -- which is
|
||
what you get when a run is bounded by a single zero *byte* inside otherwise dense
|
||
data, not by a delimiter. Combined with the 18%-zeros finding, the payload reads
|
||
as dense data with sparse holes rather than as a delimited sequence.
|
||
|
||
**So: not ascending `u32` tables, not zero-run delimiters, not fixed stride.**
|
||
All three segmentation theories are now excluded on the full population rather
|
||
than on a sample.
|
||
|
||
### What would actually unblock this
|
||
|
||
`overlay.bin` **is the game's own code**, and the collision parser is in it --
|
||
`Camera_CollPrimTest WARNING! - grid out of bounds! (RAR)` was located in the
|
||
debug string pool earlier, and that string is inside the collision routine. The
|
||
format is therefore recoverable, but not by measuring bytes: it needs either
|
||
disassembly of that routine or a reference for R&C2's collision cell record.
|
||
|
||
Stating this plainly is worth more than another scan. The disc-side measurement
|
||
space for collision is now genuinely exhausted -- the bytes are available, they
|
||
are verbatim, the grid is known, and three independent segmentation theories have
|
||
failed on the whole population. Further probing without the loader's own logic is
|
||
not going to converge.
|
||
|
||
## Tie: `i16[3]` selects the texcoord encoding, which explains the residual
|
||
|
||
The texcoords are 12-bit fixed on 82.5% of vertices, not 100%. Grouped by the
|
||
vertex's `i16[3]` slot across **95,612 vertices on levels 0-2**, the buckets
|
||
separate completely:
|
||
|
||
i16[3] n % / 4096 % / 256
|
||
negative 333 1.2% 1.2%
|
||
== 0 5,851 29.2% 60.7%
|
||
1..63 28,884 92.7% 94.3%
|
||
64..255 54,606 92.3% 93.8%
|
||
256..511 8 0.0% 0.0%
|
||
512..1023 5,391 0.0% 0.0%
|
||
>= 1024 539 0.2% 0.2%
|
||
|
||
**The residual is fully accounted for.** 87.3% of vertices sit in `1..255` at
|
||
about 92.5% fixedness, which contributes 0.873 x 0.925 = 80.8 points; the 5,851
|
||
at `i16[3] == 0` contribute 1.8 more. That is the 82.5% measured, to the point.
|
||
|
||
The remaining ~6% are **not 12-bit fixed at all**:
|
||
|
||
- `i16[3] >= 256`, and negatives, score **0%** at both 4,096 and 256 -- 6,271
|
||
vertices using an encoding this reading does not see;
|
||
- `i16[3] == 0` scores 29.2% at 4,096 but **60.7% at 256**, which is a third
|
||
distribution rather than a weaker version of the same one.
|
||
|
||
**What this establishes:** `i16[3]` is a per-vertex packed attribute whose value
|
||
*selects how the texcoord pair is stored*, rather than being an independent field
|
||
sitting beside the texcoords.
|
||
|
||
**Refined: it is the HIGH BYTE that selects.** Grouped by the high byte across the
|
||
same 95,612 vertices on levels 0-2:
|
||
|
||
i16[3] high byte n % / 16 % / 256
|
||
0x00 89,341 92.0% 91.8%
|
||
0x03 5,344 0.0% 0.0%
|
||
0x04 71 0.0% 0.0%
|
||
0x40 58 5.2% 0.0%
|
||
0x02 47 2.1% 2.1%
|
||
0x19, 0x05, 0x20, ... the remaining 73
|
||
|
||
A high byte of **zero** is the ordinary case -- 89,341 of 95,612, or **93.4%** --
|
||
and it scores 92%. **Every** other high byte scores at or near zero. The split is
|
||
on the byte, not on the value: `i16[3]` values 1 and 771 differ only in the high
|
||
byte and behave completely differently.
|
||
|
||
Within the all-zero case there is a third distribution: `i16[3] == 0` exactly
|
||
(5,851 vertices) scores 62% at 16 and 61% at 256, against 92% for `1..255`.
|
||
|
||
**So three cases, selected by the high byte and then the low:**
|
||
|
||
high 0x00, low 0x00 5,851 a third distribution, 62% at 1/16
|
||
high 0x00, low 1..255 83,490 the ordinary case, 92% at 1/4096
|
||
high non-zero 6,271 0%, an encoding this reading does not see
|
||
|
||
### The element layout differs between those cases
|
||
|
||
Scoring all four slot pairs within the non-zero high-byte group, **no pair
|
||
stands out** -- the best is `(6,7)` at 7.5% for a divisor of 4, against a 2-5%
|
||
baseline:
|
||
|
||
i16[3] high byte NON-ZERO (n = 6,271)
|
||
pair /1 /4 /16 /256
|
||
( 0, 1) 100.0% 4.6% 3.5% 0.7%
|
||
( 2, 3) 100.0% 2.0% 0.6% 0.0%
|
||
( 4, 5) 100.0% 1.9% 0.3% 0.1%
|
||
( 6, 7) 100.0% 7.5% 5.7% 4.5%
|
||
|
||
So for those 6,271 vertices the 16 bytes are **not laid out as
|
||
position/texcoord/normal** at all -- none of the four pairs carries fixed-point
|
||
coordinates. Whatever they are, it is a different element shape, selected by the
|
||
high byte of `i16[3]`.
|
||
|
||
This also settles why the global score looked ambiguous: pooling the two layouts
|
||
averages a 92% pair and a 7% pair into one number that describes neither.
|
||
|
||
**Not established:** what any of those three cases means. Whether `i16[3]` is a
|
||
texture index, a normal index, a flag or a combination is still open -- but it is
|
||
now a field whose *role* is known, which is more than "packed word".
|
||
|
||
Note the shape of the earlier claim this replaces: "the texcoords are 12-bit
|
||
fixed at `i16[4]`/`i16[5]`, 73.6% of the time, unexplained". The field position
|
||
was right; the 73.6% was a population average over three encodings, which is
|
||
exactly the pooled-number problem rule 3a describes -- partitioning by the one
|
||
variable that separates them turns a mystery into a lookup.
|
||
|
||
### Which of these findings are asserted
|
||
|
||
Not every claim here has a test behind it, and the recent ones are the ones most
|
||
likely to be lost. Audit against `crates/gc-extract/tests/disc.rs`:
|
||
|
||
claim asserted in
|
||
------------------------------------------- -----------------------------------
|
||
tie vertex element is 16 bytes tie_vertex_element_is_16_bytes_
|
||
with_fixed_texcoords
|
||
layout selected by i16[3] HIGH byte tie_vertex_layout_is_selected_by_
|
||
i16_3_high_byte
|
||
blocks self-delimiting on 0xf6 tie_classes_resolve_through_both_
|
||
indirections (per-block assert)
|
||
packets == 0 means no block at that LOD same test
|
||
packets[0] == 256 on 1,864 of 1,864 same test
|
||
chunk record length uniform per chunk tie-classes test + chunk_records_
|
||
start_16_byte_aligned
|
||
moby class IDs in one body slot per type moby_class_ids_sit_in_one_slot_
|
||
per_chunk_type
|
||
collision payload verbatim in RAM NOT assertable here -- needs a
|
||
savestate, so it lives in
|
||
plan/pcsx2-verification.md
|
||
tfrag patch bytes all rewritten by LOD same -- savestate-dependent
|
||
morphing is unobservable in that image same -- savestate-dependent
|
||
|
||
The split is deliberate: anything readable from the disc is asserted against the
|
||
disc, and anything that needs a running PCSX2 is documented in the plan file with
|
||
its method. What is *not* acceptable is a disc-readable claim living only in
|
||
prose, which is how the tie `i16[3]` finding spent two commits before it had a
|
||
test.
|
||
|
||
## An external decompilation project is the same disc, and names the overlay
|
||
|
||
`OpenRAC/rac2-gc-decomp` ("Going Commando USA v1.01: byte-exact reconstruction
|
||
tools and native-port research. No game assets.") is byte-for-byte the disc this
|
||
parser was built against:
|
||
|
||
ours 3,828,350,976 bytes
|
||
9db2e33e276133cc283647fa3279b37911955e123d6199d10065547eaa9b1ce5
|
||
theirs 3,828,350,976 bytes
|
||
9db2e33e276133cc283647fa3279b37911955e123d6199d10065547eaa9b1ce5
|
||
|
||
So its findings are usable directly rather than as an analogy. Two of them name
|
||
things this file had measured but not explained:
|
||
|
||
| what | value | source | status here |
|
||
| --- | --- | --- | --- |
|
||
| `.lit` load base | `0x1A7C80`, same on all 27 levels | `docs/LEVELS.md` | **newly named** |
|
||
| `$gp` | `0x1AEFF0`, same on all 27 levels | `docs/LEVELS.md` | newly available |
|
||
|
||
The measured constant `overlay.bin +0x00 = 1,735,808` **is** `0x1A7C80`. The
|
||
reason it is identical across all 27 levels is now known rather than merely
|
||
observed: every level's literal section loads at the same address, because
|
||
loading a level copies its program over the boot's memory from there. So the
|
||
first header word is not a size and not a checksum — it is the **load base of
|
||
the section this file is**.
|
||
|
||
That also settles what the file is. `overlay.bin` is the level's `.lit` section,
|
||
which is why it holds both code and the debug string pool.
|
||
|
||
The program around it has seven sections, per `docs/LEVELS.md`:
|
||
|
||
.lit .bss .data vtbl camvtbl sndvtbl .text
|
||
|
||
which is exactly the "seven-section parser and relocation" already listed as a
|
||
Phase 3 item for this format. It is now a known target rather than a guess.
|
||
|
||
### `+0x0c` is not the `.lit` size
|
||
|
||
The obvious next guess -- now that `+0x00` is known to be the `.lit` load base,
|
||
`+0x0c` might be the `.lit` size -- is wrong, and measurably so:
|
||
|
||
| level | file size | `+0x0c` | excess |
|
||
| --- | ---: | ---: | ---: |
|
||
| level0 | 2,751,268 | 2,767,752 | +16,484 |
|
||
| level11 | 3,151,128 | 2,870,792 | **-280,336** |
|
||
| level2 | 2,928,424 | 2,780,808 | **-147,616** |
|
||
| level24 | 2,615,776 | 2,763,912 | +148,136 |
|
||
|
||
`+0x0c` is **smaller than the file** on 16 of 27 levels, by up to 8.9%. It
|
||
cannot be a size of the section it sits at the front of. It is still unnamed. The point here is only that the size hypothesis is excluded
|
||
by sign, not by a near miss.
|
||
|
||
### Collision: the code that asserts is now locatable
|
||
|
||
The collision blocker was "needs the game's own code". The decomp names the
|
||
functions (`Camera_CollPrimTest(void *prim, void *grid)`, `MB_CheckCollPill`)
|
||
but does **not** decompile them, so the cell layout is still not handed over.
|
||
What it does supply is `$gp`, and that is enough to start.
|
||
|
||
`Camera_CollPrimTest WARNING! - grid out of bounds! (RAR)` sits at
|
||
`overlay.bin+0xd18`, the only copy in the file, so:
|
||
|
||
string overlay+0xd18 -> RAM 0x1a8998
|
||
$gp 0x1aeff0
|
||
delta -0x6658 ($lo half 0x99a8)
|
||
|
||
MIPS materialises a GP-relative address as a `%hi`/`%lo` pair, and the `$lo` half
|
||
is a 16-bit immediate in the instruction stream. So the anchor search is well
|
||
defined -- and it returns **nothing**:
|
||
|
||
$lo half 0x99a8 (low byte 0xa8, high byte 0x99)
|
||
occurrences 0, instruction-aligned, in 2,751,268 bytes
|
||
|
||
**An earlier version of this file claimed exactly one hit, at `0x354cc8`, and
|
||
called it an unambiguous anchor. That was wrong, and the mistake was worth
|
||
recording.** The search compared word bytes 1..3 against the immediate. The PS2's
|
||
MIPS is little-endian, so an I-type immediate is in bits 15..0, which is word
|
||
bytes **0** and **1** -- bytes 1..3 are the opcode, `$rs` and `$rt` fields. The
|
||
match was on those, not on an address.
|
||
|
||
The instruction at the phantom hit is `0x24a89948`:
|
||
|
||
addiu $t0, $a1, -0x66b8
|
||
|
||
whose real immediate is `0x9948`, ten bytes away from the `0x99a8` being
|
||
searched for, and whose `-0x66b8` merely looks like a plausible `$gp` offset.
|
||
With the byte order corrected, the count is zero.
|
||
|
||
So the collision anchor is **not established**, and the assertion that it was has
|
||
been withdrawn. What survives is weaker and still useful:
|
||
|
||
* `Camera_CollPrimTest` and `MB_CheckCollPill` are named, with
|
||
`Camera_CollPrimTest(void *prim, void *grid)`.
|
||
* `$gp = 0x1AEFF0`, and `.lit` loads at `0x1A7C80`.
|
||
* The assert string is at `overlay+0xd18`, a unique copy.
|
||
|
||
The likely reason the search comes back empty is that the code reaching that
|
||
string does not go through `$gp` -- or reaches a copy in another section. The
|
||
decomp's `assert-message-names.tsv` gives `0x002db350` for level 0, which is
|
||
inside `.lit` but disassembles to a function epilogue (`jr $ra` /
|
||
`addiu $sp, $sp, 0x70`), so those are code locations, not string addresses, and
|
||
they are not yet traced to the assert either.
|
||
|
||
`tools/overlay-anchors` does the anchor search and is dependency-free. Its
|
||
correctly-ordered result for the collision string is zero, and it now carries a
|
||
comment explaining the byte order, since that is precisely the part that was
|
||
wrong:
|
||
|
||
cargo run -q -p gc-overlay-anchors -- \
|
||
/home/reaper/Documents/gc-extracted/level0/overlay.bin \
|
||
"Camera_CollPrimTest WARNING!"
|
||
|
||
Next step, if collision is still worth it: find how the assert string is actually
|
||
addressed -- a `%hi`/`%lo` split that does not straddle zero, a pointer table, or
|
||
simply the right section -- before assuming a `$gp` relative load.
|
||
|
||
### What this project did *not* give up
|
||
|
||
Surveyed against the open items: no volume structures, no `TieTex`/tie vertex
|
||
layouts, no `SBlk`/`DAW` record fields, no wavelet tables. Collision has the
|
||
API but not the data. It is a guide to the code, not a replacement for reading
|
||
it.
|