Correct the terrain findings: region layout confirmed, record framing disproven

Two claims in the previous commit were wrong and are retracted here rather than
left standing for the next person to build on.

"382 records of 64 bytes" and "byte +3 is 0x48 in every record" were both
artefacts. The 0x48 at that offset is simply the most common byte in the region --
945 of 2000 sampled slots -- not a marker. Over 33,091 slots at that position
only 4,127 hold 0x48. The pattern holds for the first 382 records and then
breaks, which is exactly how a coincidence of a dense byte value masquerades as
structure.

"The coordinates are fixed-point s16 with a scale of 8" is also not supported.
It rested on one coincidence: 17,496 / 8 = 2,187, the planet radius. No grouping
of the record into coordinate triples produces points on a sphere of that radius.
Tested every stride from 2 to 16 bytes at every starting offset, with three and
four components; the best result is 5.0% of points on-sphere, which is chance.

What is now established, and holds on all 27 levels:

  core_data region layout, as strictly increasing offsets
    tfrag       [0, occlusion)
    occlusion   [occlusion, sky)
    sky         [sky, collision)
    collision   [collision, textures_base)

Ruled out the competing reading where 0x0c/0x10/0x14 are sizes laid end to end:
the cumulative total misses textures_base on every level and the leading bytes
of each supposed region are not plausible structures.

Also new: terrain carries its texels inline. A scan for the PS2 texel pattern
18 18 18 80 finds 6,739 occurrences in 138 clusters starting at 0xd9f8, well
inside the tfrag region, so textures_base names something other than terrain
textures.

The vertex format remains unrecovered and the record says so. The next attempt
should look for a longer run of plausible vertices rather than re-deriving a
framing from one record.
This commit is contained in:
2026-10-09 16:37:19 +02:00
parent 84d94337f7
commit 3ec548efe8
2 changed files with 42 additions and 72 deletions
+36 -72
View File
@@ -131,86 +131,50 @@ Header offset 0x10 = 2,192,576. Opens `0 0 0 0 0 0 6 0 0 0 0 0 5 0 0 0 40 0 0 0
— small values including 64 and 192, which look like sky-shell counts and
dimensions rather than offsets.
## `tfrag` located: terrain is `[0, occlusion)`
## `tfrag` region: `[0, occlusion)`
The header's field at 0x08 is **zero on all 27 levels**, and 0x0c, 0x10, 0x14,
0x60, 0x64 and 0x68 are strictly increasing on every level. Reading 0x08 as the
terrain offset and 0x0c as its end gives:
The header's fields at 0x0c, 0x10, 0x14 and 0x60 are **offsets**, and they are
strictly increasing on all 27 levels. The differences between them give region
sizes that are plausible on every level:
```text
tfrag [0, occlusion) level 0: 2.12 MB of terrain
occlusion [occl, sky)
sky [sky, collision)
collision [collision, textures_base)
```
So the terrain is the first region of `core_data`, and `tfrag = 0` is a real
value, not a missing field.
### What the terrain region contains
The opening bytes are a small header, after which the data is a repeating
**64-byte record** starting at 0x40:
```text
0x40 f5 ff 6d 48 01 00 77 48 cb a3 db 47 58 44 50 46 40 d0 00 ...
0x80 51 89 6b 48 9c 15 87 48 c2 57 b5 47 9a 24 fa 46 20 df 00 ...
0xc0 e3 a9 46 48 01 40 a4 48 00 00 f4 47 de b0 d0 46 50 ee 00 ...
```
What is established about these records:
- Byte columns 3, 7, 11 and 15 are **`0x48` in every record**. Those are the
high bytes of four `s16` values, i.e. a fixed Y coordinate of 0x4848-ish on a
planet whose radius is about 2,187 units.
- Columns 16 and 17 increase steadily across records (0x40, 0x50, 0x60 … and
0xd0, 0xdf, 0xee …), consistent with a grid walk.
- Columns 48–51 hold a repeating `ffff00ff` pattern, which is characteristic of
GS packet headers rather than vertex data.
- Bytes at 0x4c–0x4f repeat `ff 00 ff ff` on **every** record.
### The record layout, as far as it is established
```text
382 records of 64 bytes, from 0x40 to 0x5fc0
```
That count is exact: 383 breaks the pattern at 0x5fc0. Per column, over 300
records:
| Column | Value | Notes |
| Region | Range | Level 0 size |
|---|---|---|
| +3 | `0x48` | 300/300 records. A marker, not data: the high byte of the `s16` at +2 |
| +19, +20, +21, +23, +43, +45 | `0x00` | 300/300 |
| +52..55 | `ff 00 ff ff` | 300/300. A fixed signature |
| +2, +6, +10, +14, +18 … | varying | the data |
| tfrag | `[0, occlusion)` | 2.12 MB |
| occlusion | `[occlusion, sky)` | 0.07 MB |
| sky | `[sky, collision)` | 0.42 MB |
| collision | `[collision, textures_base)` | 0.90 MB |
**Coordinates are fixed-point `s16` with a scale of 8.** Established directly:
the value 17,496 appears in the first record and `17496 / 8 = 2187`, which is
the radius of a Ratchet & Clank planet to the unit. Values that are exactly
divisible by 8 occur in long runs elsewhere in the region too, which is what a
fixed-point coordinate array looks like.
Ruled out: reading 0x0c/0x10/0x14 as *sizes* laid end to end. The cumulative
total does not reach `textures_base` on any level, and the byte at the start of
each supposed region is not a plausible structure. The offsets model is the one
that holds everywhere.
**Not yet determined:** which column is which of x, y, z, and whether the 64
bytes are one vertex, two, or a quad header plus data. A VIF packet
interpretation of the first tag (`0xfff5`, command `0x1f`, 53 words) runs off
its data after 216 bytes and does not chain, so the `ff 00 ff ff` signature is
more likely a record delimiter than a GS packet.
### What is in the terrain region
The next test is mechanical: take one 64-byte record, try every grouping into
3-component vertices, and keep the one whose resulting points all lie within a
sphere of radius ~2,187. That is a decidable question and does not need more
speculation.
**Textures are stored inline with the geometry**, not in a separate bank. A scan
for the PS2 texel pattern `18 18 18 80` (grey texels with the 0x80 alpha that
means "opaque" in the GS's convention) finds 6,739 occurrences spread across 138
clusters, the first at `0xd9f8` — inside the tfrag region. So terrain carries its
own texels, and `textures_base` names something else.
Ruled out along the way:
Earlier claims in this file that have since been **disproven**:
- **Uncompressed `f32` vertex data.** The longest run of plausible coordinate
floats in the whole 16.9 MB is a run of zeros in the collision block. Terrain
vertices are packed.
- **`GS packet tags` at every position.** Byte frequencies in 0xE0..0xEF are
high but uniform, which is what compressed or packed data looks like, not a
discrete packet stream.
- **A uniform 48-byte record.** The pattern repeats at 64 bytes; 48 does not.
- ~~"The terrain is 382 records of 64 bytes."~~ The 0x48 byte at offset +3 is
simply the most common byte in the region (945 of 2000 sampled slots); it is
not a record marker. Over 33,091 slots at that position only 4,127 hold 0x48.
- ~~"Byte +3 is constant at 0x48 across 300 records."~~ True for the first 382
records and false beyond, which is how the false pattern was found.
- ~~"Coordinates are fixed-point s16 with a scale of 8."~~ The scale-8 reading
is supported by one coincidence (17,496 / 8 = 2,187, the planet radius) but no
grouping of the record into triples yields points on a sphere of that radius.
Tested: every stride from 2 to 16 bytes, every starting offset, three and four
components. Best result 5.0% of points on-sphere, which is chance.
So the record framing is **not recovered**. What is established is the region
layout and that terrain is vertex-soup packed alongside inline texels — which
means the vertex format must be identified from a longer run than a single
record, most likely by locating a run of vertices whose magnitudes are bounded
by the planet radius and whose Y values vary smoothly.
## Where the class tables are not
+6
View File
@@ -111,6 +111,12 @@ Phase 1 hands over the extents; this phase works out what each one contains.
written up in `plan/level-assets.md`. Next: identify `tfrag` by its own
structure instead of through the header.
- [x] `occlusion` and `sky` offsets located and their first bytes read.
- [x] Region layout of `core_data` established: tfrag, occlusion, sky, collision
and `textures_base` are offsets, strictly increasing on all 27 levels.
Terrain is `[0, occlusion)` and carries inline texels.
- [ ] **tfrag vertex format: not recovered.** The 64-byte record framing and the
scale-8 fixed-point reading were both tested and disproven; see
`plan/level-assets.md` for what was ruled out.
- [ ] `gameplay` — begins `WAD` + a per-level byte, and does *not* decode with
the WAD LZ77 decoder. All 27 levels fail identically, so the input is
wrong rather than the decoder. Ruled out: bit order, header length, the