ReaperOfVeriod
db414b3d5f
The previous commit claimed ~6.2 MB of unidentified data sat between the collision header and the first moby class, and speculated it might be moby_anim. That was wrong, and it came from quoting a size before bounding the thing it measured. Looking at the bytes rather than only the offsets: the range is dominated by 8-bit indexed texture data. Byte-value concentration distinguishes them cleanly -- a texture block draws on a small palette, code and collision data does not. 18 of 21 sampled 4 KB blocks fall under 200 distinct byte values, against 249 for tfrags and 209 for collision. The real layout on level 0: collision 2615232 .. 3518544 (header plus its declared 903,248 payload) textures 3518848 .. ~8832192 moby 8832192 .. The texture table's data_offset values stop at 5326208, which is what made the gap look genuine. Mip chains extend well past the last entry's offset, so sizing a region from a table of first-level offsets understates the data it points at. Three of the 21 blocks are not texture-like, so something is mixed in that is still unidentified. It is not 6.2 MB of moby_anim. This is the third number this session that looked load-bearing and turned out to rest on an unbounded measurement. The pattern is consistent enough to be worth stating plainly: sizes derived from tables or offsets are not evidence until the bytes underneath have been looked at.
91 lines
3.8 KiB
Markdown
91 lines
3.8 KiB
Markdown
# core_data region bounds
|
|
|
|
How a level's `core_data` lump divides up, and how each region is bounded.
|
|
|
|
## The rule
|
|
|
|
A region's end is the **next region's start**, never `core_data.len()`. The
|
|
last region in a layout is the exception, and when its successor is unknown the
|
|
end is `None` rather than a guess.
|
|
|
|
This exists because region bounds were being re-derived per format during
|
|
probing, usually by running from one known offset to the next with the last
|
|
span running to the end of the lump. On level 0 that made the final shrub class
|
|
a 925,008-byte span absorbing most of the lump's tail. Every aggregate computed
|
|
over it -- record counts, value distributions, "this data looks random" -- was
|
|
silently wrong, and nothing errored.
|
|
|
|
`gc_formats::regions::core_layout` computes these once. Callers use it rather
|
|
than inventing their own.
|
|
|
|
## Layout
|
|
|
|
| region | start | end |
|
|
|------------|----------------|----------------------|
|
|
| `tfrags` | 0 | `occlusion` |
|
|
| `occlusion`| `occlusion` | `sky` |
|
|
| `sky` | `sky` | `collision` |
|
|
| `collision`| `collision` | first moby class |
|
|
| `moby` | first moby | first tie class |
|
|
| `tie` | first tie | first shrub class |
|
|
| `shrub` | first shrub | **unknown** |
|
|
|
|
Class regions start at the lowest non-zero geometry offset in their class
|
|
table. A zero entry means *absent*, not position zero -- the same trap the
|
|
occlusion and collision headers each fell into separately.
|
|
|
|
Not every level has every region: level 10 has no occlusion and no shrub.
|
|
|
|
## Open: where shrub ends
|
|
|
|
The offset ending the shrub region is not in the core index.
|
|
|
|
Texture texels do **not** bound it. They sit before moby, not after shrub: on
|
|
level 0 the texel range is 3539328..5326208 while the last shrub class starts
|
|
at 16044288, leaving ~925 KB of real, non-zero data after the last shrub class
|
|
that nothing yet accounts for.
|
|
|
|
`core_layout` reports `end: None` for shrub on the 24 levels that have one, and
|
|
the disc test asserts it, so the gap stays visible rather than being papered
|
|
over by clamping to the end of the lump.
|
|
|
|
## Correction: it is mostly texture data, not an unnamed region
|
|
|
|
Adding the region table to `gc-extract probe` made the gap visible
|
|
immediately, which is the point of printing it.
|
|
|
|
On level 0:
|
|
|
|
```
|
|
collision 2615232 .. 8832192 6216960 bytes
|
|
```
|
|
|
|
but collision's own header declares a payload of 903,248 bytes. So the range
|
|
between the collision header and the first moby class is **6.2 MB**, and
|
|
almost none of it is collision.
|
|
|
|
The first conclusion drawn from this -- that ~6.2 MB of unidentified data sat
|
|
between collision and moby, possibly `moby_anim` -- **was wrong**, and it was
|
|
wrong because of the same failure that produced the other two bad numbers this
|
|
session: quoting a figure before bounding it.
|
|
|
|
Checking the bytes rather than just the sizes: the range is dominated by 8-bit
|
|
indexed texture data. Byte-value concentration (a texture block uses a small
|
|
palette; code and collision data does not) puts 18 of 21 sampled 4 KB blocks
|
|
under 200 distinct byte values, against 249 for tfrags and 209 for collision.
|
|
|
|
So the real layout is:
|
|
|
|
| region | start | end |
|
|
|------------|----------|------------------|
|
|
| collision | 2615232 | 3518544 |
|
|
| textures | 3518848 | ~8832192 |
|
|
| moby | 8832192 | first tie class |
|
|
|
|
The texture table's `data_offset` values only reach 5326208, which is why the
|
|
gap looked real: mip chains extend well past the last entry's offset. A region
|
|
sized from a table of first-level offsets understates the data it points at.
|
|
|
|
Three of the 21 sampled blocks are not texture-like, so something is mixed in
|
|
and is not yet identified, but it is not 6.2 MB of `moby_anim`.
|