# core_data region bounds

How a level's `core_data` lump divides up, and how each region is bounded.

## The rule

A region's end is the **next region's start**, never `core_data.len()`. The
last region in a layout is the exception, and when its successor is unknown the
end is `None` rather than a guess.

This exists because region bounds were being re-derived per format during
probing, usually by running from one known offset to the next with the last
span running to the end of the lump. On level 0 that made the final shrub class
a 925,008-byte span absorbing most of the lump's tail. Every aggregate computed
over it -- record counts, value distributions, "this data looks random" -- was
silently wrong, and nothing errored.

`gc_formats::regions::core_layout` computes these once. Callers use it rather
than inventing their own.

## Layout

| region     | start          | end                  |
|------------|----------------|----------------------|
| `tfrags`   | 0              | `occlusion`          |
| `occlusion`| `occlusion`    | `sky`                |
| `sky`      | `sky`          | `collision`          |
| `collision`| `collision`    | first moby class     |
| `moby`     | first moby     | first tie class      |
| `tie`      | first tie      | first shrub class    |
| `shrub`    | first shrub    | lump length − 0x38A10 |

Class regions start at the lowest non-zero geometry offset in their class
table. A zero entry means *absent*, not position zero -- the same trap the
occlusion and collision headers each fell into separately.

Not every level has every region: level 10 has no occlusion and no shrub.

## The constant trailer

Comparing all 27 levels byte-for-byte from the end: the last **231,952 bytes
(0x38A10)** of every level's `core_data` are identical. Level-specific content
therefore ends at `len - 0x38A10`, and the final region is bounded there rather
than at `core_data.len()`.

This is what closes shrub. Across the 24 levels that have a shrub region it
gives **24,442,928 bytes** total, and it is the same bound the disc test
asserts on every level.

What the trailer *is* is not established. It may be fixed data, or padding
from a fixed-size decompression buffer. Either way it is constant, which is
all that is needed to bound the tail.

## Open: what was in the region before it

The offset ending the shrub region is not in the core index.

Texture texels do **not** bound it. They sit before moby, not after shrub: on
level 0 the texel range is 3539328..5326208 while the last shrub class starts
at 16044288, leaving ~925 KB of real, non-zero data after the last shrub class
that nothing yet accounts for.

`core_layout` reports `end: None` for shrub on the 24 levels that have one, and
the disc test asserts it, so the gap stays visible rather than being papered
over by clamping to the end of the lump.

## Correction: it is mostly texture data, not an unnamed region

Adding the region table to `gc-extract probe` made the gap visible
immediately, which is the point of printing it.

On level 0:

```
collision       2615232 .. 8832192       6216960 bytes
```

but collision's own header declares a payload of 903,248 bytes. So the range
between the collision header and the first moby class is **6.2 MB**, and
almost none of it is collision.

The first conclusion drawn from this -- that ~6.2 MB of unidentified data sat
between collision and moby, possibly `moby_anim` -- **was wrong**, and it was
wrong because of the same failure that produced the other two bad numbers this
session: quoting a figure before bounding it.

Checking the bytes rather than just the sizes: the range is dominated by 8-bit
indexed texture data. Byte-value concentration (a texture block uses a small
palette; code and collision data does not) puts 18 of 21 sampled 4 KB blocks
under 200 distinct byte values, against 249 for tfrags and 209 for collision.

So the real layout is:

| region     | start    | end              |
|------------|----------|------------------|
| collision  | 2615232  | 3518544          |
| textures   | 3518848  | ~8832192         |
| moby       | 8832192  | first tie class  |

The texture table's `data_offset` values only reach 5326208, which is why the
gap looked real: mip chains extend well past the last entry's offset. A region
sized from a table of first-level offsets understates the data it points at.

The "3 of 21 blocks not texture-like" figure in the previous revision was an
artifact of sampling every 256 KB. Re-scanned at 4 KB granularity the whole
range is texture data: ~100 blocks stand out, and looking at them they are
smooth gradient ramps -- sky and fog texture, exactly what gradient texels look
like. A noisy gradient has a high distinct byte count without being code.

So the range between collision and moby is **entirely texture data**, and there
is no separate `moby_anim` region there. Combined with the correction above,
two claims made about this range have now been wrong: the "6.2 MB of unknown
data" and the "3 unexplained blocks".

Where `moby_anim` actually lives is therefore still open, but it is not here.
The remaining candidate is the unresolved tail after the last shrub class,
which on level 0 is ~925 KB of real, non-zero data that nothing yet accounts
for.

## Where shrub data ends, and the region after it

**Range measured:** for each level, the last shrub class's `data_offset` field
(`class + 0x2c`) through to the start of the constant trailer. Nothing outside
that range is read.

**Method:** the last shrub class's body is an array of 8-byte records whose
final two bytes are `00 00`. Scanning forward in 8-byte steps for the first
record whose trailing pair differs locates the body's end. This is a search
for a specific stated pattern, not a statistic, so it either finds the
boundary or reports that it did not.

**Result:** on **24 of 24** levels the boundary lands exactly on an 8-byte
record boundary. Body lengths are 192, 208, 224 or 240 bytes -- the same sizes
seen when shrub classes were grouped and diffed earlier.

On level 0:

| what | range | bytes |
|---|---|---|
| last shrub class data | 16046800 .. 16047040 | 240 |
| **next region** | 16047040 .. 16737344 | **690,304** |
| constant trailer | 16737344 .. 16969296 | 231,952 |

So the 690,304 bytes between the last shrub class and the trailer are a
distinct region, not shrub data. What they contain is not established.

Two things are worth noting about how this was found, given the rule in
`AGENTS.md`: the earlier NaN-filler search found nothing at the start of this
range and was about to conclude the body had zero length, which would have been
wrong. The last shrub class is a real class with real data, and the NaN filler
only appears in degenerate ones. Checking the actual byte pattern rather than
concluding from a failed heuristic is what caught it.

### First structure in the region after shrub

**Range measured:** level 0, `16047040 .. 16737344` (690,304 bytes), starting
at the first byte after the last shrub class's 240-byte body. Nothing outside
that range is read.

**Method:** reading the bytes at `+0x20` as `u32` shows ten consecutive
entries whose low halves increase by a constant **0x94 (148)**, followed by a
zero. Read as u32 the high half is `0xf000` for the first nine and `0x0a40`
for the tenth, so the table is ten entries long and ends at `+0x48`
(absolute `16047112`).

**Result, stated plainly:** a ten-entry table on a 148-byte stride. What the
entries mean, what 148 refers to, and why there are ten, are all unestablished.
Ten does not match any count already known for this level -- 238 moby
classes, 74 tie, 7 shrub, 93 tfrag textures -- so this is not yet tied to any
structure elsewhere in the lump.

### Follow-up: the 148 stride is a coincidence

Treating the ten entries' low halves as offsets from the region start lands on
blocks exactly 148 bytes apart, the first nine of which begin `08 00` followed
by a 16-bit id stepping by 0x40. That is a convincing-looking structure and it
does not survive being extended: walking the same 148-byte stride across the
rest of the region, **only 14 of 4657 blocks** start with `08 00`. The first
nine agree and then the pattern breaks.

So the ten entries are not a pointer table. The constant stride held for
eight consecutive values, which is far more than chance should produce and far
less than a structure needs.

Recorded because the lead was committed as a lead and this is where it goes.
Nothing else about the region has been identified: it is real, non-zero data
of unknown type, bounded, and unopened beyond these ~150 bytes.

## The region after shrub has a repeating structure at stride 368

**Range measured:** level 0, `16047040 .. 16737344`, sampled as the first
65,536 bytes. Cross-checked on every level that has the region, each sampled
the same way from its own region start.

**Method:** autocorrelation. For each candidate stride, count how often a
4-byte sequence at offset *i* repeats at *i + stride*, as a fraction of
comparable positions. A random control of the same size is measured
identically and reads flat at 0.0000 for every stride, so the statistic
discriminates.

**Result:**

| stride | region match rate |
|---|---|
| 368 | 0.1110 |
| 340 | 0.0935 |
| 337–339 | 0.026–0.029 (background) |
| random control | 0.0000 |

Stride **368** is the strongest peak. On level 0, 340 is a secondary peak
well clear of the background; the two differ by 28, which is unexplained.

**Cross-check:** the peak stride was recomputed independently for every level
that has the region, and **all 24 peak at exactly 368**. That is the part
that makes this worth recording -- a single level's peak could be a property
of that level's data, but 24 of 24 agreeing is a format invariant.

What a 368-byte record *is* remains unestablished. This is the same position
the shrub body was in before differential analysis identified its records,
and the same method is the obvious next step.

### Differential analysis at stride 368: no constant substructure

**Range measured:** level 0, `16047040 .. 16737344`. 1,875 whole 368-byte
records from the region start; variation checked across the first 60.

**Method:** the same differential analysis that identified the shrub body --
compare records byte by byte and record which positions differ.

**Result:** **all 368 positions vary.** There is no constant field, no padding
and no header inside a record. Unlike the shrub body, where differential
analysis immediately separated a varying payload from a constant two-byte
tail, this format has nothing constant to separate.

That does not contradict the autocorrelation peak. Checking which byte
positions carry the 368-stride matches over 20,000 positions: 168 exact 4-byte
repeats, distributed evenly at 0.84% across all four byte offsets. So whole
4-byte sequences recur at exactly 368 while no individual position is constant
-- consistent with records containing entries that repeat at a fixed relative
index, rather than with a fixed-layout record.

**So the 368 stride stands and the records remain unidentified.** The shrub
method failed here not because it was the wrong tool but because it needs
something constant to latch onto, and there is nothing.

## Sky opens with a 256-byte-row image

**Range measured:** level 0, `2192576 .. 2615232` (the sky region, 422,656
bytes). Periodicity start and end located by sliding an 8 KB window in 256-byte
steps through the region and measuring the 256-stride match rate.

**Method:** autocorrelation as above, then a periodicity scan. A random
control of the same size reads flat at 0.0000 for every stride tested.

**Result:**

- Peak stride in the first 64 KB is **256** at a 0.772 match rate. Strides 255,
  257 and 4 score nearly as high, which is what a true period looks like
  smeared by sampling.
- Across levels the peak is 256 on 21, 512 on 5 and 128 on 1. 512 and 128 are
  a harmonic and a sub-multiple, so the fundamental period is **256 bytes**.

**Correction to the control.** The original comparison used uniform random
bytes, which read flat at 0.0000 -- and made the 0.772 look like an enormous
signal. It is not: 91.8% of the region's bytes are the same value (`0x07`), so
a stride-256 comparison matches trivially wherever both positions are `0x07`.

The right control is the same bytes shuffled, which preserves the value
distribution and destroys only the ordering. That control reads **0.512** at
stride 256, not 0.0000. The true excess is 0.772 − 0.512 = **+0.26**, a real
signal and a far smaller one than the original framing suggested.

**And the extent is wrong too.** Re-measured against the matched control, the
excess over the value distribution is:

| offset | excess |
|---|---|
| +0 .. +49152 | +0.38 .. +0.45 |
| +53248 | +0.26 |
| +57344 | +0.07 |
| +61440 | −0.13 |

So the 256-periodic data ends between `+53248` and `+57344` — somewhere in
rows 205 to 224 — **not** at `+60160` as first reported. The original figure
came from a raw threshold with no distribution control, and 91.8% of one byte
value drags that threshold wherever the dominant run ends.

The 256-byte period itself stands, cross-checked on 21 of 27 levels with the
remaining levels showing its harmonic and sub-multiple. The image dimensions
do not: an earlier "256x236" is wrong and should not be used.

So the sky region is not one thing. Its first ~60 KB is an image; the rest is
whatever the layer offsets address. Both halves still need work, and the
image is the tractable one.

## Re-verified: the 368 stride, against a matched control

The sky autocorrelation was found to be inflated by a control that did not
match the data, so the 368 result was re-checked the same way.

**Range measured:** each level's post-shrub region start, first 65,536 bytes.

**Method:** as before, but the control is the same bytes shuffled rather than
uniform random bytes. Shuffling preserves the value distribution and destroys
only the ordering, which is exactly what the control needs to do.

**Result:**

- Level 0's region is evenly distributed: 256 distinct byte values, most
  common only 15.6%. Uniform random was therefore the *appropriate* control
  here, and the shuffled control independently confirms it, reading **0.0000**
  at strides 340, 368 and 4.
- Level 0 peak remains 368 at 0.1110, against 0.0000 for the matched control.
- Recomputed across all levels with matched controls: **24 of 24 peak at
  exactly 368, every one with excess above 0.05.**

So the 368-byte period stands, now with a control appropriate to its data.
The contrast with sky is the point: the same statistic on the same kind of
question gave a real answer in one region and an inflated one in the other,
and the only thing that told them apart was whether the control matched the
byte distribution.

## Sky sub-objects: structure is consistent, stride is not

**Range measured:** every level's sky region, from the count field at `+0x0c`
through the sub-object offsets at `+0x20`. Each sub-object measured over its
first 16,384 bytes.

**Method:** as above, peak-stride autocorrelation per sub-object with a
matched control (the same bytes shuffled, seeded per level and index).

**Field note:** the sub-object count is a `u32` at **`+0x0c`**, not `+0x0a`.
Reading `+0x0a` gives 327,680 on level 0, which is not a count.

**Result, level 0:** five sub-objects at offsets 365760, 388880, 396432,
406784 and 414032, of lengths 23120, 7552, 10352, 7248 and 1536. Each has
internal structure well clear of its matched control.

**But the strides do not repeat across levels.** Recomputing per level:

| sub-object | peak strides seen across levels |
|---|---|
| 0 | 4 (x6), 8 (x13), 32, 288 |
| 1 | 16 (x3), 32 (x6), 288 |
| 2 | 8, 16 (x5), 256, 288 |
| 3 | 8, 16 (x6), 216 |
| 4 | 8, 16 (x3), 512 (x4) |

So the earlier per-index stride table, read off level 0 alone, was
over-fitted to one level. What is consistent is weaker and more honest: every
sub-object sits above its matched control, and most peak at a small stride of
4, 8, 16 or 32, which suggests 16- or 32-bit records whose stride varies with
content rather than being fixed.

**Level 2's header remains genuinely different**: its count field at `+0x0c`
reads 196,617 and is unusable, while every other level yields a plausible
count. Whatever makes it different is still unexplained.

### Sky sub-object 0: sparse, with duplicated entries

**Range measured:** level 0, sky region `+365760 .. +388880`, 23,120 bytes,
read as 2,890 records of 8.

**Method:** record composition, plus the adjacent-pair and zero-density
statistics below. The matched-control autocorrelation puts the peak at stride
8 (0.217 real, 0.000 shuffled), so 8 is the unit to describe.

**What it is made of:**

- **1,185 distinct `u32` values** across 5,780 words.
- **Mean 6.99 of 8 bytes are live** -- the previous revision of this file said
  "zero", which is inverted and wrong. The live-byte histogram is 0:15, 2:1,
  4:31, 5:238, 6:461, 7:1030, 8:1114, so most records are nearly full rather
  than sparse.
- **549 of 2,890 adjacent pairs (19%) are exact duplicates of each other.** The
  most common 8-byte record, appearing 283 times, is `d7d7d780 d7d7d780` -- a
  4-byte value stored twice. Others follow the same shape: `00051980 00051980`,
  `00082580 00082580`.

**Byte 7 is anomalous.** It takes only **14 distinct values** where bytes 0-6
take 142-204. Its common values are `0x00` (1378), `0xff` (806) and `0x80`
(592) -- the high bytes of signed 16-bit values, which is consistent with
byte 7 being an MSB of something rather than ordinary data. Unexplained.

**Not decoded.** A signed-16-bit reading was tried and does not obviously
help: the values span -32743..32633, close to the full range, with no
per-position structure (533, 371, 401 and 187 distinct at positions 0-3).
Recorded as a characterisation, with its range, so the next attempt starts
from the composition rather than re-deriving it.

### Sky sub-object 0 is a slowly-varying sequence, not independent records

**Range measured:** level 0, sky region `+365760 .. +388880`, 23,120 bytes.

**Method:** Pearson correlation between the byte sequence and itself at
increasing lag, against a shuffled control. This measures whether consecutive
records are *related*, which the position-wise distinct-value counts could
not show.

**Result:**

| lag (bytes) | correlation |
|---|---|
| 1 | +0.050 |
| 4 | +0.560 |
| **8** | **+0.746** |
| 16 | +0.666 |
| 32 | +0.602 |
| 64 | +0.432 |
| 128 | +0.140 |
| 256 | −0.044 |
| **512** | **+0.304** |
| 1024 | +0.247 |
| shuffled control (lag 1) | +0.003 |

The profile has a peak at lag 8, decays to a zero crossing near 256, and then
a second, smaller peak at 512 before decaying again.

This is what settles the earlier "no constant substructure" reading. The
records do not share fixed fields, but they are *not independent*: consecutive
8-byte records are strongly related, and that relation repeats at 512 bytes,
i.e. every 64 records.

Slowly varying values, related within a record, repeating every 64 records, is
the shape of a keyframe or lookup table -- which is what one would expect to
find in sky data, where a colour or fog parameter is sampled rather than
computed per vertex.

**Still not decoded.** Nothing here names the values or the sampling. But the
structure is now characterised well enough that a decode attempt has a shape
to aim at rather than a field to hunt for.

### Sky sub-objects share one shape, across every level

**Range measured:** sub-objects of the sky region on levels 0, 1, 3, 7, 13
and 26 -- 25 sub-objects in total, each measured over its whole length.

**Method:** as above, correlation at lag against the byte sequence.

**Result:**

| quantity | range over 25 sub-objects |
|---|---|
| lag 8 | +0.53 .. +0.79 |
| lag 64 | +0.16 .. +0.63 |
| lag 512 | -0.13 .. +0.42 (not universal) |
| lag 1024 | -0.05 .. +0.37 (not universal) |

**Lag 8 and lag 64 are positive on every sub-object of every level tested.**
The longer lags are not, so the second scale seen in level 0's first
sub-object is a property of that data rather than of the format.

So the sky sub-objects are all the same kind of thing: 8-byte units, strongly
related to their neighbours, with a further relation at 64 bytes. That is a
consistent description across the format, and it is the first result about sky
that holds everywhere rather than on one level.

**Not decoded.** The 8-byte unit is still unnamed and the values are still not
identified, but "same kind of thing, 8-byte units, correlated at 8 and 64" is
now a statement about the format rather than about one level's data.

## Sound bank SBlk has a 40-byte period (27/27 levels)

**Range measured:** each level's `SBlk` region -- the descriptor table between
`sblk_offset` and `daw_offset`, from the sound bank header rather than from
`core_data`. 17-20 KB per level.

**Method:** peak-stride autocorrelation with a matched control (the same bytes
shuffled, same seed, every level).

**Result:** the peak is at **exactly 40 bytes on all 27 levels**, every one with
excess above 0.2. The peak is also sharply isolated on level 0 -- 0.454 excess
at stride 40 against 0.207 to 0.278 for strides 36 to 44 -- so it is a period
rather than a broad tendency.

That is the first structural result about the sound bank beyond the container
framing already committed.

**Record count:** 487, 443, 428, 494 and 447 whole 40-byte records on levels 0,
1, 2, 13 and 26. The region is not an exact multiple of 40 on any level
(19484 leaves 4 bytes over), so 40 is a period rather than a record size with
no remainder.

**Unexplained:** which byte positions are constant varies by level -- level 0
has 7, 11, 31 and 35 constant; level 1 has 15, 19, 23, 35 and 39; level 2 has
7, 23, 27 and 31. A fixed 40-byte layout would have the same constant
positions everywhere, so either the records have a variable tail or 40 is a
period in something else that the records sit inside.

Also measured, for the record:

- `DAW`, the sample region, shows **no** stride above threshold against its
  control. Consistent with ADPCM nibble-packed audio, and consistent with the
  container framing being the only part worth decoding.
- `gameplay.bin` peaks at 16 bytes (excess 0.267 at strides 16, 32 and 64).
- `gs_ram.bin` has almost no stride structure (0.043 at 32) but a lag-8
  correlation of +0.503, so it is smooth rather than tabular.

## Tie class spans: 16-byte unit, 48-byte period (dominant, not universal)

**Range measured:** every tie class span on levels 0, 3, 6 and every third
level after it -- 386 spans of 16 KB or more, bounded by consecutive class
offsets.

**Method:** peak-stride autocorrelation with a matched control, over the first
32,768 bytes of each span.

**Result:**

| peak stride | spans |
|---|---|
| 48 | 285 |
| 16 | 58 |
| 64 | 29 |
| 96 | 11 |
| 24 | 2 |
| 32 | 1 |

**48 bytes dominates but is not universal**, and the weaker peak is 58 spans
against 285 -- so unlike the sound bank's 40, this is a majority rather than a
constant.

Every observed peak is a multiple of 16 (16, 24, 32, 48, 64, 96), which
suggests a **16-byte base unit** with a 48-byte period arising from how those
units group, rather than 48 being the record itself.

This is weaker evidence than the sound bank's 40-byte result, which was
sharply isolated and identical on all 27 levels. Here the peak is weaker
(0.09-0.22 excess against sound bank's 0.45) and it moves between levels and
spans. Recorded at the confidence it deserves.

## Two independent measurements agree on moby's 16-byte vertex

The moby part table established `vertex_offset + 16 * vertex_count ==
next_part` on 43,467 of 43,467 parts, which fixes the vertex stride at 16.
Autocorrelation over the spans reaches the same number without using that
result at all.

**Range measured:** moby class spans of 16 KB or more on level 0 -- 60 spans,
each bounded by consecutive class offsets, measured over the first 32,768
bytes.

**Method:** peak-stride autocorrelation against a matched control, searching
strides 4 to 64.

**Result:**

| peak stride | spans |
|---|---|
| **16** | **43** |
| 8 | 8 |
| 48 | 4 |
| 32 | 3 |
| 24 | 2 |

16 dominates, and it is the same 16 the part table produces from a completely
different argument -- a tiling identity over part records rather than a
periodicity measurement over span contents.

Two unrelated methods agreeing is worth more than either alone, because the
autocorrelation would be vulnerable to exactly the kinds of artifact this
project has already been bitten by: a dominant byte value would inflate it,
and a period that happened to divide the window would look the same as a real
one. Neither applies here, and the tiling identity cannot be inflated at all.

This is the strongest form of evidence produced in this project: a structural
invariant confirmed by two methods that do not share a failure mode.

## Sound bank SBlk: a 12-byte marker that does not explain the 40-byte period

**Range measured:** level 0 `SBlk`, 24..19508 (19,484 bytes).

**Method:** positions of the `0x7f`/`0x47f` marker words, and their spacing.

**Result:** those markers sit at **12-byte spacing on 144 of roughly 180
occurrences**, with 24-byte and larger gaps accounting for the rest. So there
is a 12-byte structure in the data.

**But that contradicts the autocorrelation.** If records were 12 bytes, stride
12 should be the peak. It is not:

| stride | excess over shuffled |
|---|---|
| 12 | +0.237 |
| 20 | +0.175 |
| 24 | +0.237 |
| **40** | **+0.454** |
| 48 | +0.239 |

40 is not a multiple of 12, so the two measurements disagree rather than one
explaining the other. Recorded as an open discrepancy rather than resolved in
favour of either: the marker spacing says 12, the period says 40, and 40 being
twice as strong is the more surprising of the two.

### Resolved: 40 is the record, the 12-byte markers are nested

A fine scan settles it in favour of 40.

| stride | excess |
|---|---|
| **40** | **+0.454** |
| 39 | +0.266 |
| 41 | +0.215 |
| 38 | +0.214 |

The peak is sharp and isolated -- nearly twice its immediate neighbours.
And the multiples confirm it: 80 scores +0.376, 120 +0.394, 160 +0.340,
while the multiples of 12 (24, 48, 96, 144) all sit at 0.21-0.24.

So the record is **40 bytes**, and the `0x7f`/`0x47f` markers at 12-byte
spacing are a nested structure *inside* those records rather than the record
boundary. Their spacing is irregular -- 12, 24 and larger -- which is exactly
what one would expect of a field that is present in some records and absent in
others.

The earlier "they disagree" framing was wrong in one respect: the marker
spacing never claimed to be the record size. Read as a claim about markers
inside records rather than about the record itself, the two measurements were
never in conflict.