ReaperOfVeriod
8882d8fc26
The "3 of 21 blocks not texture-like" figure from the last commit was an artifact of sampling every 256 KB and missing everything between. Re-scanned at 4 KB granularity there are about 100 such blocks, and looking at them rather than only counting them, they are smooth gradient ramps -- sky and fog texture. A noisy gradient has a high distinct byte count without being code, which is exactly the mistake the byte-count heuristic invites. So collision..moby is entirely texture data. There is no separate moby_anim region there, and two claims about this range have now been wrong: the "6.2 MB of unknown data" and the "3 unexplained blocks". Both came from sizes and counts taken without looking at what they described. moby_anim is therefore still unlocated, but it is not here. The remaining candidate is the unresolved tail after the last shrub class -- ~925 KB of real, non-zero data on level 0 that nothing accounts for. Recording the two corrections together rather than one at a time, because the mechanism is the same in both: a heuristic count is not an identification.
104 lines
4.5 KiB
Markdown
104 lines
4.5 KiB
Markdown
# core_data region bounds
|
|
|
|
How a level's `core_data` lump divides up, and how each region is bounded.
|
|
|
|
## The rule
|
|
|
|
A region's end is the **next region's start**, never `core_data.len()`. The
|
|
last region in a layout is the exception, and when its successor is unknown the
|
|
end is `None` rather than a guess.
|
|
|
|
This exists because region bounds were being re-derived per format during
|
|
probing, usually by running from one known offset to the next with the last
|
|
span running to the end of the lump. On level 0 that made the final shrub class
|
|
a 925,008-byte span absorbing most of the lump's tail. Every aggregate computed
|
|
over it -- record counts, value distributions, "this data looks random" -- was
|
|
silently wrong, and nothing errored.
|
|
|
|
`gc_formats::regions::core_layout` computes these once. Callers use it rather
|
|
than inventing their own.
|
|
|
|
## Layout
|
|
|
|
| region | start | end |
|
|
|------------|----------------|----------------------|
|
|
| `tfrags` | 0 | `occlusion` |
|
|
| `occlusion`| `occlusion` | `sky` |
|
|
| `sky` | `sky` | `collision` |
|
|
| `collision`| `collision` | first moby class |
|
|
| `moby` | first moby | first tie class |
|
|
| `tie` | first tie | first shrub class |
|
|
| `shrub` | first shrub | **unknown** |
|
|
|
|
Class regions start at the lowest non-zero geometry offset in their class
|
|
table. A zero entry means *absent*, not position zero -- the same trap the
|
|
occlusion and collision headers each fell into separately.
|
|
|
|
Not every level has every region: level 10 has no occlusion and no shrub.
|
|
|
|
## Open: where shrub ends
|
|
|
|
The offset ending the shrub region is not in the core index.
|
|
|
|
Texture texels do **not** bound it. They sit before moby, not after shrub: on
|
|
level 0 the texel range is 3539328..5326208 while the last shrub class starts
|
|
at 16044288, leaving ~925 KB of real, non-zero data after the last shrub class
|
|
that nothing yet accounts for.
|
|
|
|
`core_layout` reports `end: None` for shrub on the 24 levels that have one, and
|
|
the disc test asserts it, so the gap stays visible rather than being papered
|
|
over by clamping to the end of the lump.
|
|
|
|
## Correction: it is mostly texture data, not an unnamed region
|
|
|
|
Adding the region table to `gc-extract probe` made the gap visible
|
|
immediately, which is the point of printing it.
|
|
|
|
On level 0:
|
|
|
|
```
|
|
collision 2615232 .. 8832192 6216960 bytes
|
|
```
|
|
|
|
but collision's own header declares a payload of 903,248 bytes. So the range
|
|
between the collision header and the first moby class is **6.2 MB**, and
|
|
almost none of it is collision.
|
|
|
|
The first conclusion drawn from this -- that ~6.2 MB of unidentified data sat
|
|
between collision and moby, possibly `moby_anim` -- **was wrong**, and it was
|
|
wrong because of the same failure that produced the other two bad numbers this
|
|
session: quoting a figure before bounding it.
|
|
|
|
Checking the bytes rather than just the sizes: the range is dominated by 8-bit
|
|
indexed texture data. Byte-value concentration (a texture block uses a small
|
|
palette; code and collision data does not) puts 18 of 21 sampled 4 KB blocks
|
|
under 200 distinct byte values, against 249 for tfrags and 209 for collision.
|
|
|
|
So the real layout is:
|
|
|
|
| region | start | end |
|
|
|------------|----------|------------------|
|
|
| collision | 2615232 | 3518544 |
|
|
| textures | 3518848 | ~8832192 |
|
|
| moby | 8832192 | first tie class |
|
|
|
|
The texture table's `data_offset` values only reach 5326208, which is why the
|
|
gap looked real: mip chains extend well past the last entry's offset. A region
|
|
sized from a table of first-level offsets understates the data it points at.
|
|
|
|
The "3 of 21 blocks not texture-like" figure in the previous revision was an
|
|
artifact of sampling every 256 KB. Re-scanned at 4 KB granularity the whole
|
|
range is texture data: ~100 blocks stand out, and looking at them they are
|
|
smooth gradient ramps -- sky and fog texture, exactly what gradient texels look
|
|
like. A noisy gradient has a high distinct byte count without being code.
|
|
|
|
So the range between collision and moby is **entirely texture data**, and there
|
|
is no separate `moby_anim` region there. Combined with the correction above,
|
|
two claims made about this range have now been wrong: the "6.2 MB of unknown
|
|
data" and the "3 unexplained blocks".
|
|
|
|
Where `moby_anim` actually lives is therefore still open, but it is not here.
|
|
The remaining candidate is the unresolved tail after the last shrub class,
|
|
which on level 0 is ~925 KB of real, non-zero data that nothing yet accounts
|
|
for.
|