ReaperOfVeriod
98c0d1ab3c
The sky autocorrelation turned out to be inflated by a control that did not match the data, so the 368 result was re-checked the same way rather than being left on the evidence I now know to be weak. Range measured: each level's post-shrub region start, first 65536 bytes. Method: as before, but the control is the same bytes shuffled rather than uniform random. Shuffling preserves the value distribution and destroys only the ordering, which is what a control for this statistic needs to do. Level 0's region is evenly distributed -- 256 distinct byte values, most common only 15.6% -- so uniform random was the appropriate control here, and the shuffled control independently confirms it, reading 0.0000 at strides 340, 368 and 4 against the real data's 0.1110 at 368. Recomputed across all levels with matched controls: 24 of 24 peak at exactly 368, every one with excess above 0.05. So the 368 period stands, now with a control appropriate to its data. The contrast is the useful part: the same statistic, asked the same question of two regions, gave a real answer in one and an inflated one in the other, and the only thing that separated them was whether the control matched the byte distribution.
322 lines
14 KiB
Markdown
322 lines
14 KiB
Markdown
# core_data region bounds
|
||
|
||
How a level's `core_data` lump divides up, and how each region is bounded.
|
||
|
||
## The rule
|
||
|
||
A region's end is the **next region's start**, never `core_data.len()`. The
|
||
last region in a layout is the exception, and when its successor is unknown the
|
||
end is `None` rather than a guess.
|
||
|
||
This exists because region bounds were being re-derived per format during
|
||
probing, usually by running from one known offset to the next with the last
|
||
span running to the end of the lump. On level 0 that made the final shrub class
|
||
a 925,008-byte span absorbing most of the lump's tail. Every aggregate computed
|
||
over it -- record counts, value distributions, "this data looks random" -- was
|
||
silently wrong, and nothing errored.
|
||
|
||
`gc_formats::regions::core_layout` computes these once. Callers use it rather
|
||
than inventing their own.
|
||
|
||
## Layout
|
||
|
||
| region | start | end |
|
||
|------------|----------------|----------------------|
|
||
| `tfrags` | 0 | `occlusion` |
|
||
| `occlusion`| `occlusion` | `sky` |
|
||
| `sky` | `sky` | `collision` |
|
||
| `collision`| `collision` | first moby class |
|
||
| `moby` | first moby | first tie class |
|
||
| `tie` | first tie | first shrub class |
|
||
| `shrub` | first shrub | lump length − 0x38A10 |
|
||
|
||
Class regions start at the lowest non-zero geometry offset in their class
|
||
table. A zero entry means *absent*, not position zero -- the same trap the
|
||
occlusion and collision headers each fell into separately.
|
||
|
||
Not every level has every region: level 10 has no occlusion and no shrub.
|
||
|
||
## The constant trailer
|
||
|
||
Comparing all 27 levels byte-for-byte from the end: the last **231,952 bytes
|
||
(0x38A10)** of every level's `core_data` are identical. Level-specific content
|
||
therefore ends at `len - 0x38A10`, and the final region is bounded there rather
|
||
than at `core_data.len()`.
|
||
|
||
This is what closes shrub. Across the 24 levels that have a shrub region it
|
||
gives **24,442,928 bytes** total, and it is the same bound the disc test
|
||
asserts on every level.
|
||
|
||
What the trailer *is* is not established. It may be fixed data, or padding
|
||
from a fixed-size decompression buffer. Either way it is constant, which is
|
||
all that is needed to bound the tail.
|
||
|
||
## Open: what was in the region before it
|
||
|
||
The offset ending the shrub region is not in the core index.
|
||
|
||
Texture texels do **not** bound it. They sit before moby, not after shrub: on
|
||
level 0 the texel range is 3539328..5326208 while the last shrub class starts
|
||
at 16044288, leaving ~925 KB of real, non-zero data after the last shrub class
|
||
that nothing yet accounts for.
|
||
|
||
`core_layout` reports `end: None` for shrub on the 24 levels that have one, and
|
||
the disc test asserts it, so the gap stays visible rather than being papered
|
||
over by clamping to the end of the lump.
|
||
|
||
## Correction: it is mostly texture data, not an unnamed region
|
||
|
||
Adding the region table to `gc-extract probe` made the gap visible
|
||
immediately, which is the point of printing it.
|
||
|
||
On level 0:
|
||
|
||
```
|
||
collision 2615232 .. 8832192 6216960 bytes
|
||
```
|
||
|
||
but collision's own header declares a payload of 903,248 bytes. So the range
|
||
between the collision header and the first moby class is **6.2 MB**, and
|
||
almost none of it is collision.
|
||
|
||
The first conclusion drawn from this -- that ~6.2 MB of unidentified data sat
|
||
between collision and moby, possibly `moby_anim` -- **was wrong**, and it was
|
||
wrong because of the same failure that produced the other two bad numbers this
|
||
session: quoting a figure before bounding it.
|
||
|
||
Checking the bytes rather than just the sizes: the range is dominated by 8-bit
|
||
indexed texture data. Byte-value concentration (a texture block uses a small
|
||
palette; code and collision data does not) puts 18 of 21 sampled 4 KB blocks
|
||
under 200 distinct byte values, against 249 for tfrags and 209 for collision.
|
||
|
||
So the real layout is:
|
||
|
||
| region | start | end |
|
||
|------------|----------|------------------|
|
||
| collision | 2615232 | 3518544 |
|
||
| textures | 3518848 | ~8832192 |
|
||
| moby | 8832192 | first tie class |
|
||
|
||
The texture table's `data_offset` values only reach 5326208, which is why the
|
||
gap looked real: mip chains extend well past the last entry's offset. A region
|
||
sized from a table of first-level offsets understates the data it points at.
|
||
|
||
The "3 of 21 blocks not texture-like" figure in the previous revision was an
|
||
artifact of sampling every 256 KB. Re-scanned at 4 KB granularity the whole
|
||
range is texture data: ~100 blocks stand out, and looking at them they are
|
||
smooth gradient ramps -- sky and fog texture, exactly what gradient texels look
|
||
like. A noisy gradient has a high distinct byte count without being code.
|
||
|
||
So the range between collision and moby is **entirely texture data**, and there
|
||
is no separate `moby_anim` region there. Combined with the correction above,
|
||
two claims made about this range have now been wrong: the "6.2 MB of unknown
|
||
data" and the "3 unexplained blocks".
|
||
|
||
Where `moby_anim` actually lives is therefore still open, but it is not here.
|
||
The remaining candidate is the unresolved tail after the last shrub class,
|
||
which on level 0 is ~925 KB of real, non-zero data that nothing yet accounts
|
||
for.
|
||
|
||
## Where shrub data ends, and the region after it
|
||
|
||
**Range measured:** for each level, the last shrub class's `data_offset` field
|
||
(`class + 0x2c`) through to the start of the constant trailer. Nothing outside
|
||
that range is read.
|
||
|
||
**Method:** the last shrub class's body is an array of 8-byte records whose
|
||
final two bytes are `00 00`. Scanning forward in 8-byte steps for the first
|
||
record whose trailing pair differs locates the body's end. This is a search
|
||
for a specific stated pattern, not a statistic, so it either finds the
|
||
boundary or reports that it did not.
|
||
|
||
**Result:** on **24 of 24** levels the boundary lands exactly on an 8-byte
|
||
record boundary. Body lengths are 192, 208, 224 or 240 bytes -- the same sizes
|
||
seen when shrub classes were grouped and diffed earlier.
|
||
|
||
On level 0:
|
||
|
||
| what | range | bytes |
|
||
|---|---|---|
|
||
| last shrub class data | 16046800 .. 16047040 | 240 |
|
||
| **next region** | 16047040 .. 16737344 | **690,304** |
|
||
| constant trailer | 16737344 .. 16969296 | 231,952 |
|
||
|
||
So the 690,304 bytes between the last shrub class and the trailer are a
|
||
distinct region, not shrub data. What they contain is not established.
|
||
|
||
Two things are worth noting about how this was found, given the rule in
|
||
`AGENTS.md`: the earlier NaN-filler search found nothing at the start of this
|
||
range and was about to conclude the body had zero length, which would have been
|
||
wrong. The last shrub class is a real class with real data, and the NaN filler
|
||
only appears in degenerate ones. Checking the actual byte pattern rather than
|
||
concluding from a failed heuristic is what caught it.
|
||
|
||
### First structure in the region after shrub
|
||
|
||
**Range measured:** level 0, `16047040 .. 16737344` (690,304 bytes), starting
|
||
at the first byte after the last shrub class's 240-byte body. Nothing outside
|
||
that range is read.
|
||
|
||
**Method:** reading the bytes at `+0x20` as `u32` shows ten consecutive
|
||
entries whose low halves increase by a constant **0x94 (148)**, followed by a
|
||
zero. Read as u32 the high half is `0xf000` for the first nine and `0x0a40`
|
||
for the tenth, so the table is ten entries long and ends at `+0x48`
|
||
(absolute `16047112`).
|
||
|
||
**Result, stated plainly:** a ten-entry table on a 148-byte stride. What the
|
||
entries mean, what 148 refers to, and why there are ten, are all unestablished.
|
||
Ten does not match any count already known for this level -- 238 moby
|
||
classes, 74 tie, 7 shrub, 93 tfrag textures -- so this is not yet tied to any
|
||
structure elsewhere in the lump.
|
||
|
||
### Follow-up: the 148 stride is a coincidence
|
||
|
||
Treating the ten entries' low halves as offsets from the region start lands on
|
||
blocks exactly 148 bytes apart, the first nine of which begin `08 00` followed
|
||
by a 16-bit id stepping by 0x40. That is a convincing-looking structure and it
|
||
does not survive being extended: walking the same 148-byte stride across the
|
||
rest of the region, **only 14 of 4657 blocks** start with `08 00`. The first
|
||
nine agree and then the pattern breaks.
|
||
|
||
So the ten entries are not a pointer table. The constant stride held for
|
||
eight consecutive values, which is far more than chance should produce and far
|
||
less than a structure needs.
|
||
|
||
Recorded because the lead was committed as a lead and this is where it goes.
|
||
Nothing else about the region has been identified: it is real, non-zero data
|
||
of unknown type, bounded, and unopened beyond these ~150 bytes.
|
||
|
||
## The region after shrub has a repeating structure at stride 368
|
||
|
||
**Range measured:** level 0, `16047040 .. 16737344`, sampled as the first
|
||
65,536 bytes. Cross-checked on every level that has the region, each sampled
|
||
the same way from its own region start.
|
||
|
||
**Method:** autocorrelation. For each candidate stride, count how often a
|
||
4-byte sequence at offset *i* repeats at *i + stride*, as a fraction of
|
||
comparable positions. A random control of the same size is measured
|
||
identically and reads flat at 0.0000 for every stride, so the statistic
|
||
discriminates.
|
||
|
||
**Result:**
|
||
|
||
| stride | region match rate |
|
||
|---|---|
|
||
| 368 | 0.1110 |
|
||
| 340 | 0.0935 |
|
||
| 337–339 | 0.026–0.029 (background) |
|
||
| random control | 0.0000 |
|
||
|
||
Stride **368** is the strongest peak. On level 0, 340 is a secondary peak
|
||
well clear of the background; the two differ by 28, which is unexplained.
|
||
|
||
**Cross-check:** the peak stride was recomputed independently for every level
|
||
that has the region, and **all 24 peak at exactly 368**. That is the part
|
||
that makes this worth recording -- a single level's peak could be a property
|
||
of that level's data, but 24 of 24 agreeing is a format invariant.
|
||
|
||
What a 368-byte record *is* remains unestablished. This is the same position
|
||
the shrub body was in before differential analysis identified its records,
|
||
and the same method is the obvious next step.
|
||
|
||
### Differential analysis at stride 368: no constant substructure
|
||
|
||
**Range measured:** level 0, `16047040 .. 16737344`. 1,875 whole 368-byte
|
||
records from the region start; variation checked across the first 60.
|
||
|
||
**Method:** the same differential analysis that identified the shrub body --
|
||
compare records byte by byte and record which positions differ.
|
||
|
||
**Result:** **all 368 positions vary.** There is no constant field, no padding
|
||
and no header inside a record. Unlike the shrub body, where differential
|
||
analysis immediately separated a varying payload from a constant two-byte
|
||
tail, this format has nothing constant to separate.
|
||
|
||
That does not contradict the autocorrelation peak. Checking which byte
|
||
positions carry the 368-stride matches over 20,000 positions: 168 exact 4-byte
|
||
repeats, distributed evenly at 0.84% across all four byte offsets. So whole
|
||
4-byte sequences recur at exactly 368 while no individual position is constant
|
||
-- consistent with records containing entries that repeat at a fixed relative
|
||
index, rather than with a fixed-layout record.
|
||
|
||
**So the 368 stride stands and the records remain unidentified.** The shrub
|
||
method failed here not because it was the wrong tool but because it needs
|
||
something constant to latch onto, and there is nothing.
|
||
|
||
## Sky opens with a 256-byte-row image
|
||
|
||
**Range measured:** level 0, `2192576 .. 2615232` (the sky region, 422,656
|
||
bytes). Periodicity start and end located by sliding an 8 KB window in 256-byte
|
||
steps through the region and measuring the 256-stride match rate.
|
||
|
||
**Method:** autocorrelation as above, then a periodicity scan. A random
|
||
control of the same size reads flat at 0.0000 for every stride tested.
|
||
|
||
**Result:**
|
||
|
||
- Peak stride in the first 64 KB is **256** at a 0.772 match rate. Strides 255,
|
||
257 and 4 score nearly as high, which is what a true period looks like
|
||
smeared by sampling.
|
||
- Across levels the peak is 256 on 21, 512 on 5 and 128 on 1. 512 and 128 are
|
||
a harmonic and a sub-multiple, so the fundamental period is **256 bytes**.
|
||
|
||
**Correction to the control.** The original comparison used uniform random
|
||
bytes, which read flat at 0.0000 -- and made the 0.772 look like an enormous
|
||
signal. It is not: 91.8% of the region's bytes are the same value (`0x07`), so
|
||
a stride-256 comparison matches trivially wherever both positions are `0x07`.
|
||
|
||
The right control is the same bytes shuffled, which preserves the value
|
||
distribution and destroys only the ordering. That control reads **0.512** at
|
||
stride 256, not 0.0000. The true excess is 0.772 − 0.512 = **+0.26**, a real
|
||
signal and a far smaller one than the original framing suggested.
|
||
|
||
**And the extent is wrong too.** Re-measured against the matched control, the
|
||
excess over the value distribution is:
|
||
|
||
| offset | excess |
|
||
|---|---|
|
||
| +0 .. +49152 | +0.38 .. +0.45 |
|
||
| +53248 | +0.26 |
|
||
| +57344 | +0.07 |
|
||
| +61440 | −0.13 |
|
||
|
||
So the 256-periodic data ends between `+53248` and `+57344` — somewhere in
|
||
rows 205 to 224 — **not** at `+60160` as first reported. The original figure
|
||
came from a raw threshold with no distribution control, and 91.8% of one byte
|
||
value drags that threshold wherever the dominant run ends.
|
||
|
||
The 256-byte period itself stands, cross-checked on 21 of 27 levels with the
|
||
remaining levels showing its harmonic and sub-multiple. The image dimensions
|
||
do not: an earlier "256x236" is wrong and should not be used.
|
||
|
||
So the sky region is not one thing. Its first ~60 KB is an image; the rest is
|
||
whatever the layer offsets address. Both halves still need work, and the
|
||
image is the tractable one.
|
||
|
||
## Re-verified: the 368 stride, against a matched control
|
||
|
||
The sky autocorrelation was found to be inflated by a control that did not
|
||
match the data, so the 368 result was re-checked the same way.
|
||
|
||
**Range measured:** each level's post-shrub region start, first 65,536 bytes.
|
||
|
||
**Method:** as before, but the control is the same bytes shuffled rather than
|
||
uniform random bytes. Shuffling preserves the value distribution and destroys
|
||
only the ordering, which is exactly what the control needs to do.
|
||
|
||
**Result:**
|
||
|
||
- Level 0's region is evenly distributed: 256 distinct byte values, most
|
||
common only 15.6%. Uniform random was therefore the *appropriate* control
|
||
here, and the shuffled control independently confirms it, reading **0.0000**
|
||
at strides 340, 368 and 4.
|
||
- Level 0 peak remains 368 at 0.1110, against 0.0000 for the matched control.
|
||
- Recomputed across all levels with matched controls: **24 of 24 peak at
|
||
exactly 368, every one with excess above 0.05.**
|
||
|
||
So the 368-byte period stands, now with a control appropriate to its data.
|
||
The contrast with sky is the point: the same statistic on the same kind of
|
||
question gave a real answer in one region and an inflated one in the other,
|
||
and the only thing that told them apart was whether the control matched the
|
||
byte distribution.
|