ReaperOfVeriod
db414b3d5f
The previous commit claimed ~6.2 MB of unidentified data sat between the collision header and the first moby class, and speculated it might be moby_anim. That was wrong, and it came from quoting a size before bounding the thing it measured. Looking at the bytes rather than only the offsets: the range is dominated by 8-bit indexed texture data. Byte-value concentration distinguishes them cleanly -- a texture block draws on a small palette, code and collision data does not. 18 of 21 sampled 4 KB blocks fall under 200 distinct byte values, against 249 for tfrags and 209 for collision. The real layout on level 0: collision 2615232 .. 3518544 (header plus its declared 903,248 payload) textures 3518848 .. ~8832192 moby 8832192 .. The texture table's data_offset values stop at 5326208, which is what made the gap look genuine. Mip chains extend well past the last entry's offset, so sizing a region from a table of first-level offsets understates the data it points at. Three of the 21 blocks are not texture-like, so something is mixed in that is still unidentified. It is not 6.2 MB of moby_anim. This is the third number this session that looked load-bearing and turned out to rest on an unbounded measurement. The pattern is consistent enough to be worth stating plainly: sizes derived from tables or offsets are not evidence until the bytes underneath have been looked at.
3.8 KiB
core_data region bounds
How a level's core_data lump divides up, and how each region is bounded.
The rule
A region's end is the next region's start, never core_data.len(). The
last region in a layout is the exception, and when its successor is unknown the
end is None rather than a guess.
This exists because region bounds were being re-derived per format during probing, usually by running from one known offset to the next with the last span running to the end of the lump. On level 0 that made the final shrub class a 925,008-byte span absorbing most of the lump's tail. Every aggregate computed over it -- record counts, value distributions, "this data looks random" -- was silently wrong, and nothing errored.
gc_formats::regions::core_layout computes these once. Callers use it rather
than inventing their own.
Layout
| region | start | end |
|---|---|---|
tfrags |
0 | occlusion |
occlusion |
occlusion |
sky |
sky |
sky |
collision |
collision |
collision |
first moby class |
moby |
first moby | first tie class |
tie |
first tie | first shrub class |
shrub |
first shrub | unknown |
Class regions start at the lowest non-zero geometry offset in their class table. A zero entry means absent, not position zero -- the same trap the occlusion and collision headers each fell into separately.
Not every level has every region: level 10 has no occlusion and no shrub.
Open: where shrub ends
The offset ending the shrub region is not in the core index.
Texture texels do not bound it. They sit before moby, not after shrub: on level 0 the texel range is 3539328..5326208 while the last shrub class starts at 16044288, leaving ~925 KB of real, non-zero data after the last shrub class that nothing yet accounts for.
core_layout reports end: None for shrub on the 24 levels that have one, and
the disc test asserts it, so the gap stays visible rather than being papered
over by clamping to the end of the lump.
Correction: it is mostly texture data, not an unnamed region
Adding the region table to gc-extract probe made the gap visible
immediately, which is the point of printing it.
On level 0:
collision 2615232 .. 8832192 6216960 bytes
but collision's own header declares a payload of 903,248 bytes. So the range between the collision header and the first moby class is 6.2 MB, and almost none of it is collision.
The first conclusion drawn from this -- that ~6.2 MB of unidentified data sat
between collision and moby, possibly moby_anim -- was wrong, and it was
wrong because of the same failure that produced the other two bad numbers this
session: quoting a figure before bounding it.
Checking the bytes rather than just the sizes: the range is dominated by 8-bit indexed texture data. Byte-value concentration (a texture block uses a small palette; code and collision data does not) puts 18 of 21 sampled 4 KB blocks under 200 distinct byte values, against 249 for tfrags and 209 for collision.
So the real layout is:
| region | start | end |
|---|---|---|
| collision | 2615232 | 3518544 |
| textures | 3518848 | ~8832192 |
| moby | 8832192 | first tie class |
The texture table's data_offset values only reach 5326208, which is why the
gap looked real: mip chains extend well past the last entry's offset. A region
sized from a table of first-level offsets understates the data it points at.
Three of the 21 sampled blocks are not texture-like, so something is mixed in
and is not yet identified, but it is not 6.2 MB of moby_anim.