ReaperOfVeriod
8882d8fc26
The "3 of 21 blocks not texture-like" figure from the last commit was an artifact of sampling every 256 KB and missing everything between. Re-scanned at 4 KB granularity there are about 100 such blocks, and looking at them rather than only counting them, they are smooth gradient ramps -- sky and fog texture. A noisy gradient has a high distinct byte count without being code, which is exactly the mistake the byte-count heuristic invites. So collision..moby is entirely texture data. There is no separate moby_anim region there, and two claims about this range have now been wrong: the "6.2 MB of unknown data" and the "3 unexplained blocks". Both came from sizes and counts taken without looking at what they described. moby_anim is therefore still unlocated, but it is not here. The remaining candidate is the unresolved tail after the last shrub class -- ~925 KB of real, non-zero data on level 0 that nothing accounts for. Recording the two corrections together rather than one at a time, because the mechanism is the same in both: a heuristic count is not an identification.
4.5 KiB
core_data region bounds
How a level's core_data lump divides up, and how each region is bounded.
The rule
A region's end is the next region's start, never core_data.len(). The
last region in a layout is the exception, and when its successor is unknown the
end is None rather than a guess.
This exists because region bounds were being re-derived per format during probing, usually by running from one known offset to the next with the last span running to the end of the lump. On level 0 that made the final shrub class a 925,008-byte span absorbing most of the lump's tail. Every aggregate computed over it -- record counts, value distributions, "this data looks random" -- was silently wrong, and nothing errored.
gc_formats::regions::core_layout computes these once. Callers use it rather
than inventing their own.
Layout
| region | start | end |
|---|---|---|
tfrags |
0 | occlusion |
occlusion |
occlusion |
sky |
sky |
sky |
collision |
collision |
collision |
first moby class |
moby |
first moby | first tie class |
tie |
first tie | first shrub class |
shrub |
first shrub | unknown |
Class regions start at the lowest non-zero geometry offset in their class table. A zero entry means absent, not position zero -- the same trap the occlusion and collision headers each fell into separately.
Not every level has every region: level 10 has no occlusion and no shrub.
Open: where shrub ends
The offset ending the shrub region is not in the core index.
Texture texels do not bound it. They sit before moby, not after shrub: on level 0 the texel range is 3539328..5326208 while the last shrub class starts at 16044288, leaving ~925 KB of real, non-zero data after the last shrub class that nothing yet accounts for.
core_layout reports end: None for shrub on the 24 levels that have one, and
the disc test asserts it, so the gap stays visible rather than being papered
over by clamping to the end of the lump.
Correction: it is mostly texture data, not an unnamed region
Adding the region table to gc-extract probe made the gap visible
immediately, which is the point of printing it.
On level 0:
collision 2615232 .. 8832192 6216960 bytes
but collision's own header declares a payload of 903,248 bytes. So the range between the collision header and the first moby class is 6.2 MB, and almost none of it is collision.
The first conclusion drawn from this -- that ~6.2 MB of unidentified data sat
between collision and moby, possibly moby_anim -- was wrong, and it was
wrong because of the same failure that produced the other two bad numbers this
session: quoting a figure before bounding it.
Checking the bytes rather than just the sizes: the range is dominated by 8-bit indexed texture data. Byte-value concentration (a texture block uses a small palette; code and collision data does not) puts 18 of 21 sampled 4 KB blocks under 200 distinct byte values, against 249 for tfrags and 209 for collision.
So the real layout is:
| region | start | end |
|---|---|---|
| collision | 2615232 | 3518544 |
| textures | 3518848 | ~8832192 |
| moby | 8832192 | first tie class |
The texture table's data_offset values only reach 5326208, which is why the
gap looked real: mip chains extend well past the last entry's offset. A region
sized from a table of first-level offsets understates the data it points at.
The "3 of 21 blocks not texture-like" figure in the previous revision was an artifact of sampling every 256 KB. Re-scanned at 4 KB granularity the whole range is texture data: ~100 blocks stand out, and looking at them they are smooth gradient ramps -- sky and fog texture, exactly what gradient texels look like. A noisy gradient has a high distinct byte count without being code.
So the range between collision and moby is entirely texture data, and there
is no separate moby_anim region there. Combined with the correction above,
two claims made about this range have now been wrong: the "6.2 MB of unknown
data" and the "3 unexplained blocks".
Where moby_anim actually lives is therefore still open, but it is not here.
The remaining candidate is the unresolved tail after the last shrub class,
which on level 0 is ~925 KB of real, non-zero data that nothing yet accounts
for.