Going-Commando/docs/formats/core-data-regions.md

core_data region bounds

How a level's core_data lump divides up, and how each region is bounded.

The rule

A region's end is the next region's start, never core_data.len(). The last region in a layout is the exception, and when its successor is unknown the end is None rather than a guess.

This exists because region bounds were being re-derived per format during probing, usually by running from one known offset to the next with the last span running to the end of the lump. On level 0 that made the final shrub class a 925,008-byte span absorbing most of the lump's tail. Every aggregate computed over it -- record counts, value distributions, "this data looks random" -- was silently wrong, and nothing errored.

gc_formats::regions::core_layout computes these once. Callers use it rather than inventing their own.

Layout

region	start	end
tfrags	0	occlusion
occlusion	occlusion	sky
sky	sky	collision
collision	collision	first moby class
moby	first moby	first tie class
tie	first tie	first shrub class
shrub	first shrub	lump length − 0x38A10

Class regions start at the lowest non-zero geometry offset in their class table. A zero entry means absent, not position zero -- the same trap the occlusion and collision headers each fell into separately.

Not every level has every region: level 10 has no occlusion and no shrub.

The constant trailer

Comparing all 27 levels byte-for-byte from the end: the last 231,952 bytes (0x38A10) of every level's core_data are identical. Level-specific content therefore ends at len - 0x38A10, and the final region is bounded there rather than at core_data.len().

This is what closes shrub. Across the 24 levels that have a shrub region it gives 24,442,928 bytes total, and it is the same bound the disc test asserts on every level.

What the trailer is is not established. It may be fixed data, or padding from a fixed-size decompression buffer. Either way it is constant, which is all that is needed to bound the tail.

Open: what was in the region before it

The offset ending the shrub region is not in the core index.

Texture texels do not bound it. They sit before moby, not after shrub: on level 0 the texel range is 3539328..5326208 while the last shrub class starts at 16044288, leaving ~925 KB of real, non-zero data after the last shrub class that nothing yet accounts for.

core_layout reports end: None for shrub on the 24 levels that have one, and the disc test asserts it, so the gap stays visible rather than being papered over by clamping to the end of the lump.

Correction: it is mostly texture data, not an unnamed region

Adding the region table to gc-extract probe made the gap visible immediately, which is the point of printing it.

On level 0:

collision       2615232 .. 8832192       6216960 bytes

but collision's own header declares a payload of 903,248 bytes. So the range between the collision header and the first moby class is 6.2 MB, and almost none of it is collision.

The first conclusion drawn from this -- that ~6.2 MB of unidentified data sat between collision and moby, possibly moby_anim -- was wrong, and it was wrong because of the same failure that produced the other two bad numbers this session: quoting a figure before bounding it.

Checking the bytes rather than just the sizes: the range is dominated by 8-bit indexed texture data. Byte-value concentration (a texture block uses a small palette; code and collision data does not) puts 18 of 21 sampled 4 KB blocks under 200 distinct byte values, against 249 for tfrags and 209 for collision.

So the real layout is:

region	start	end
collision	2615232	3518544
textures	3518848	~8832192
moby	8832192	first tie class

The texture table's data_offset values only reach 5326208, which is why the gap looked real: mip chains extend well past the last entry's offset. A region sized from a table of first-level offsets understates the data it points at.

The "3 of 21 blocks not texture-like" figure in the previous revision was an artifact of sampling every 256 KB. Re-scanned at 4 KB granularity the whole range is texture data: ~100 blocks stand out, and looking at them they are smooth gradient ramps -- sky and fog texture, exactly what gradient texels look like. A noisy gradient has a high distinct byte count without being code.

So the range between collision and moby is entirely texture data, and there is no separate moby_anim region there. Combined with the correction above, two claims made about this range have now been wrong: the "6.2 MB of unknown data" and the "3 unexplained blocks".

Where moby_anim actually lives is therefore still open, but it is not here. The remaining candidate is the unresolved tail after the last shrub class, which on level 0 is ~925 KB of real, non-zero data that nothing yet accounts for.

Where shrub data ends, and the region after it

Range measured: for each level, the last shrub class's data_offset field (class + 0x2c) through to the start of the constant trailer. Nothing outside that range is read.

Method: the last shrub class's body is an array of 8-byte records whose final two bytes are 00 00. Scanning forward in 8-byte steps for the first record whose trailing pair differs locates the body's end. This is a search for a specific stated pattern, not a statistic, so it either finds the boundary or reports that it did not.

Result: on 24 of 24 levels the boundary lands exactly on an 8-byte record boundary. Body lengths are 192, 208, 224 or 240 bytes -- the same sizes seen when shrub classes were grouped and diffed earlier.

On level 0:

what	range	bytes
last shrub class data	16046800 .. 16047040	240
next region	16047040 .. 16737344	690,304
constant trailer	16737344 .. 16969296	231,952

So the 690,304 bytes between the last shrub class and the trailer are a distinct region, not shrub data. What they contain is not established.

Two things are worth noting about how this was found, given the rule in AGENTS.md: the earlier NaN-filler search found nothing at the start of this range and was about to conclude the body had zero length, which would have been wrong. The last shrub class is a real class with real data, and the NaN filler only appears in degenerate ones. Checking the actual byte pattern rather than concluding from a failed heuristic is what caught it.

First structure in the region after shrub

Range measured: level 0, 16047040 .. 16737344 (690,304 bytes), starting at the first byte after the last shrub class's 240-byte body. Nothing outside that range is read.

Method: reading the bytes at +0x20 as u32 shows ten consecutive entries whose low halves increase by a constant 0x94 (148), followed by a zero. Read as u32 the high half is 0xf000 for the first nine and 0x0a40 for the tenth, so the table is ten entries long and ends at +0x48 (absolute 16047112).

Result, stated plainly: a ten-entry table on a 148-byte stride. What the entries mean, what 148 refers to, and why there are ten, are all unestablished. Ten does not match any count already known for this level -- 238 moby classes, 74 tie, 7 shrub, 93 tfrag textures -- so this is not yet tied to any structure elsewhere in the lump.

Follow-up: the 148 stride is a coincidence

Treating the ten entries' low halves as offsets from the region start lands on blocks exactly 148 bytes apart, the first nine of which begin 08 00 followed by a 16-bit id stepping by 0x40. That is a convincing-looking structure and it does not survive being extended: walking the same 148-byte stride across the rest of the region, only 14 of 4657 blocks start with 08 00. The first nine agree and then the pattern breaks.

So the ten entries are not a pointer table. The constant stride held for eight consecutive values, which is far more than chance should produce and far less than a structure needs.

Recorded because the lead was committed as a lead and this is where it goes. Nothing else about the region has been identified: it is real, non-zero data of unknown type, bounded, and unopened beyond these ~150 bytes.

The region after shrub has a repeating structure at stride 368

Range measured: level 0, 16047040 .. 16737344, sampled as the first 65,536 bytes. Cross-checked on every level that has the region, each sampled the same way from its own region start.

Method: autocorrelation. For each candidate stride, count how often a 4-byte sequence at offset i repeats at i + stride, as a fraction of comparable positions. A random control of the same size is measured identically and reads flat at 0.0000 for every stride, so the statistic discriminates.

Result:

stride	region match rate
368	0.1110
340	0.0935
337–339	0.026–0.029 (background)
random control	0.0000

Stride 368 is the strongest peak. On level 0, 340 is a secondary peak well clear of the background; the two differ by 28, which is unexplained.

Cross-check: the peak stride was recomputed independently for every level that has the region, and all 24 peak at exactly 368. That is the part that makes this worth recording -- a single level's peak could be a property of that level's data, but 24 of 24 agreeing is a format invariant.

What a 368-byte record is remains unestablished. This is the same position the shrub body was in before differential analysis identified its records, and the same method is the obvious next step.