ReaperOfVeriod
8d57a09b76
Comparing all 27 levels byte-for-byte from the end: the last 231,952 bytes (0x38A10) of every level's core_data are identical. Level-specific content therefore ends at len - 0x38A10, which is a real bound for the final region and is not core_data.len(). This closes shrub. Across the 24 levels that have a shrub region the bound gives 24,442,928 bytes total, and the disc test asserts on every level that the region ends strictly inside the lump rather than at its end. What the trailer is remains unestablished -- fixed data, or padding from a fixed-size decompression buffer -- but it is constant, which is all a bound needs. The data between the last shrub class and the trailer is high entropy: 183 to 256 distinct byte values per 4 KB block, mean 254, where texture data sits under 200. It is real content of unknown type and is the most likely home for moby_anim, now that the range between collision and moby has been ruled out as texture. A lump shorter than the trailer now yields end: None rather than a backwards bound, covered by a test.
119 lines
5.1 KiB
Markdown
119 lines
5.1 KiB
Markdown
# core_data region bounds
|
||
|
||
How a level's `core_data` lump divides up, and how each region is bounded.
|
||
|
||
## The rule
|
||
|
||
A region's end is the **next region's start**, never `core_data.len()`. The
|
||
last region in a layout is the exception, and when its successor is unknown the
|
||
end is `None` rather than a guess.
|
||
|
||
This exists because region bounds were being re-derived per format during
|
||
probing, usually by running from one known offset to the next with the last
|
||
span running to the end of the lump. On level 0 that made the final shrub class
|
||
a 925,008-byte span absorbing most of the lump's tail. Every aggregate computed
|
||
over it -- record counts, value distributions, "this data looks random" -- was
|
||
silently wrong, and nothing errored.
|
||
|
||
`gc_formats::regions::core_layout` computes these once. Callers use it rather
|
||
than inventing their own.
|
||
|
||
## Layout
|
||
|
||
| region | start | end |
|
||
|------------|----------------|----------------------|
|
||
| `tfrags` | 0 | `occlusion` |
|
||
| `occlusion`| `occlusion` | `sky` |
|
||
| `sky` | `sky` | `collision` |
|
||
| `collision`| `collision` | first moby class |
|
||
| `moby` | first moby | first tie class |
|
||
| `tie` | first tie | first shrub class |
|
||
| `shrub` | first shrub | lump length − 0x38A10 |
|
||
|
||
Class regions start at the lowest non-zero geometry offset in their class
|
||
table. A zero entry means *absent*, not position zero -- the same trap the
|
||
occlusion and collision headers each fell into separately.
|
||
|
||
Not every level has every region: level 10 has no occlusion and no shrub.
|
||
|
||
## The constant trailer
|
||
|
||
Comparing all 27 levels byte-for-byte from the end: the last **231,952 bytes
|
||
(0x38A10)** of every level's `core_data` are identical. Level-specific content
|
||
therefore ends at `len - 0x38A10`, and the final region is bounded there rather
|
||
than at `core_data.len()`.
|
||
|
||
This is what closes shrub. Across the 24 levels that have a shrub region it
|
||
gives **24,442,928 bytes** total, and it is the same bound the disc test
|
||
asserts on every level.
|
||
|
||
What the trailer *is* is not established. It may be fixed data, or padding
|
||
from a fixed-size decompression buffer. Either way it is constant, which is
|
||
all that is needed to bound the tail.
|
||
|
||
## Open: what was in the region before it
|
||
|
||
The offset ending the shrub region is not in the core index.
|
||
|
||
Texture texels do **not** bound it. They sit before moby, not after shrub: on
|
||
level 0 the texel range is 3539328..5326208 while the last shrub class starts
|
||
at 16044288, leaving ~925 KB of real, non-zero data after the last shrub class
|
||
that nothing yet accounts for.
|
||
|
||
`core_layout` reports `end: None` for shrub on the 24 levels that have one, and
|
||
the disc test asserts it, so the gap stays visible rather than being papered
|
||
over by clamping to the end of the lump.
|
||
|
||
## Correction: it is mostly texture data, not an unnamed region
|
||
|
||
Adding the region table to `gc-extract probe` made the gap visible
|
||
immediately, which is the point of printing it.
|
||
|
||
On level 0:
|
||
|
||
```
|
||
collision 2615232 .. 8832192 6216960 bytes
|
||
```
|
||
|
||
but collision's own header declares a payload of 903,248 bytes. So the range
|
||
between the collision header and the first moby class is **6.2 MB**, and
|
||
almost none of it is collision.
|
||
|
||
The first conclusion drawn from this -- that ~6.2 MB of unidentified data sat
|
||
between collision and moby, possibly `moby_anim` -- **was wrong**, and it was
|
||
wrong because of the same failure that produced the other two bad numbers this
|
||
session: quoting a figure before bounding it.
|
||
|
||
Checking the bytes rather than just the sizes: the range is dominated by 8-bit
|
||
indexed texture data. Byte-value concentration (a texture block uses a small
|
||
palette; code and collision data does not) puts 18 of 21 sampled 4 KB blocks
|
||
under 200 distinct byte values, against 249 for tfrags and 209 for collision.
|
||
|
||
So the real layout is:
|
||
|
||
| region | start | end |
|
||
|------------|----------|------------------|
|
||
| collision | 2615232 | 3518544 |
|
||
| textures | 3518848 | ~8832192 |
|
||
| moby | 8832192 | first tie class |
|
||
|
||
The texture table's `data_offset` values only reach 5326208, which is why the
|
||
gap looked real: mip chains extend well past the last entry's offset. A region
|
||
sized from a table of first-level offsets understates the data it points at.
|
||
|
||
The "3 of 21 blocks not texture-like" figure in the previous revision was an
|
||
artifact of sampling every 256 KB. Re-scanned at 4 KB granularity the whole
|
||
range is texture data: ~100 blocks stand out, and looking at them they are
|
||
smooth gradient ramps -- sky and fog texture, exactly what gradient texels look
|
||
like. A noisy gradient has a high distinct byte count without being code.
|
||
|
||
So the range between collision and moby is **entirely texture data**, and there
|
||
is no separate `moby_anim` region there. Combined with the correction above,
|
||
two claims made about this range have now been wrong: the "6.2 MB of unknown
|
||
data" and the "3 unexplained blocks".
|
||
|
||
Where `moby_anim` actually lives is therefore still open, but it is not here.
|
||
The remaining candidate is the unresolved tail after the last shrub class,
|
||
which on level 0 is ~925 KB of real, non-zero data that nothing yet accounts
|
||
for.
|