Correct the terrain findings: region layout confirmed, record framing disproven
Two claims in the previous commit were wrong and are retracted here rather than
left standing for the next person to build on.
"382 records of 64 bytes" and "byte +3 is 0x48 in every record" were both
artefacts. The 0x48 at that offset is simply the most common byte in the region --
945 of 2000 sampled slots -- not a marker. Over 33,091 slots at that position
only 4,127 hold 0x48. The pattern holds for the first 382 records and then
breaks, which is exactly how a coincidence of a dense byte value masquerades as
structure.
"The coordinates are fixed-point s16 with a scale of 8" is also not supported.
It rested on one coincidence: 17,496 / 8 = 2,187, the planet radius. No grouping
of the record into coordinate triples produces points on a sphere of that radius.
Tested every stride from 2 to 16 bytes at every starting offset, with three and
four components; the best result is 5.0% of points on-sphere, which is chance.
What is now established, and holds on all 27 levels:
core_data region layout, as strictly increasing offsets
tfrag [0, occlusion)
occlusion [occlusion, sky)
sky [sky, collision)
collision [collision, textures_base)
Ruled out the competing reading where 0x0c/0x10/0x14 are sizes laid end to end:
the cumulative total misses textures_base on every level and the leading bytes
of each supposed region are not plausible structures.
Also new: terrain carries its texels inline. A scan for the PS2 texel pattern
18 18 18 80 finds 6,739 occurrences in 138 clusters starting at 0xd9f8, well
inside the tfrag region, so textures_base names something other than terrain
textures.
The vertex format remains unrecovered and the record says so. The next attempt
should look for a longer run of plausible vertices rather than re-deriving a
framing from one record.
This commit is contained in:
+36
-72
@@ -131,86 +131,50 @@ Header offset 0x10 = 2,192,576. Opens `0 0 0 0 0 0 6 0 0 0 0 0 5 0 0 0 40 0 0 0
|
||||
— small values including 64 and 192, which look like sky-shell counts and
|
||||
dimensions rather than offsets.
|
||||
|
||||
## `tfrag` located: terrain is `[0, occlusion)`
|
||||
## `tfrag` region: `[0, occlusion)`
|
||||
|
||||
The header's field at 0x08 is **zero on all 27 levels**, and 0x0c, 0x10, 0x14,
|
||||
0x60, 0x64 and 0x68 are strictly increasing on every level. Reading 0x08 as the
|
||||
terrain offset and 0x0c as its end gives:
|
||||
The header's fields at 0x0c, 0x10, 0x14 and 0x60 are **offsets**, and they are
|
||||
strictly increasing on all 27 levels. The differences between them give region
|
||||
sizes that are plausible on every level:
|
||||
|
||||
```text
|
||||
tfrag [0, occlusion) level 0: 2.12 MB of terrain
|
||||
occlusion [occl, sky)
|
||||
sky [sky, collision)
|
||||
collision [collision, textures_base)
|
||||
```
|
||||
|
||||
So the terrain is the first region of `core_data`, and `tfrag = 0` is a real
|
||||
value, not a missing field.
|
||||
|
||||
### What the terrain region contains
|
||||
|
||||
The opening bytes are a small header, after which the data is a repeating
|
||||
**64-byte record** starting at 0x40:
|
||||
|
||||
```text
|
||||
0x40 f5 ff 6d 48 01 00 77 48 cb a3 db 47 58 44 50 46 40 d0 00 ...
|
||||
0x80 51 89 6b 48 9c 15 87 48 c2 57 b5 47 9a 24 fa 46 20 df 00 ...
|
||||
0xc0 e3 a9 46 48 01 40 a4 48 00 00 f4 47 de b0 d0 46 50 ee 00 ...
|
||||
```
|
||||
|
||||
What is established about these records:
|
||||
|
||||
- Byte columns 3, 7, 11 and 15 are **`0x48` in every record**. Those are the
|
||||
high bytes of four `s16` values, i.e. a fixed Y coordinate of 0x4848-ish on a
|
||||
planet whose radius is about 2,187 units.
|
||||
- Columns 16 and 17 increase steadily across records (0x40, 0x50, 0x60 … and
|
||||
0xd0, 0xdf, 0xee …), consistent with a grid walk.
|
||||
- Columns 48–51 hold a repeating `ffff00ff` pattern, which is characteristic of
|
||||
GS packet headers rather than vertex data.
|
||||
- Bytes at 0x4c–0x4f repeat `ff 00 ff ff` on **every** record.
|
||||
|
||||
### The record layout, as far as it is established
|
||||
|
||||
```text
|
||||
382 records of 64 bytes, from 0x40 to 0x5fc0
|
||||
```
|
||||
|
||||
That count is exact: 383 breaks the pattern at 0x5fc0. Per column, over 300
|
||||
records:
|
||||
|
||||
| Column | Value | Notes |
|
||||
| Region | Range | Level 0 size |
|
||||
|---|---|---|
|
||||
| +3 | `0x48` | 300/300 records. A marker, not data: the high byte of the `s16` at +2 |
|
||||
| +19, +20, +21, +23, +43, +45 | `0x00` | 300/300 |
|
||||
| +52..55 | `ff 00 ff ff` | 300/300. A fixed signature |
|
||||
| +2, +6, +10, +14, +18 … | varying | the data |
|
||||
| tfrag | `[0, occlusion)` | 2.12 MB |
|
||||
| occlusion | `[occlusion, sky)` | 0.07 MB |
|
||||
| sky | `[sky, collision)` | 0.42 MB |
|
||||
| collision | `[collision, textures_base)` | 0.90 MB |
|
||||
|
||||
**Coordinates are fixed-point `s16` with a scale of 8.** Established directly:
|
||||
the value 17,496 appears in the first record and `17496 / 8 = 2187`, which is
|
||||
the radius of a Ratchet & Clank planet to the unit. Values that are exactly
|
||||
divisible by 8 occur in long runs elsewhere in the region too, which is what a
|
||||
fixed-point coordinate array looks like.
|
||||
Ruled out: reading 0x0c/0x10/0x14 as *sizes* laid end to end. The cumulative
|
||||
total does not reach `textures_base` on any level, and the byte at the start of
|
||||
each supposed region is not a plausible structure. The offsets model is the one
|
||||
that holds everywhere.
|
||||
|
||||
**Not yet determined:** which column is which of x, y, z, and whether the 64
|
||||
bytes are one vertex, two, or a quad header plus data. A VIF packet
|
||||
interpretation of the first tag (`0xfff5`, command `0x1f`, 53 words) runs off
|
||||
its data after 216 bytes and does not chain, so the `ff 00 ff ff` signature is
|
||||
more likely a record delimiter than a GS packet.
|
||||
### What is in the terrain region
|
||||
|
||||
The next test is mechanical: take one 64-byte record, try every grouping into
|
||||
3-component vertices, and keep the one whose resulting points all lie within a
|
||||
sphere of radius ~2,187. That is a decidable question and does not need more
|
||||
speculation.
|
||||
**Textures are stored inline with the geometry**, not in a separate bank. A scan
|
||||
for the PS2 texel pattern `18 18 18 80` (grey texels with the 0x80 alpha that
|
||||
means "opaque" in the GS's convention) finds 6,739 occurrences spread across 138
|
||||
clusters, the first at `0xd9f8` — inside the tfrag region. So terrain carries its
|
||||
own texels, and `textures_base` names something else.
|
||||
|
||||
Ruled out along the way:
|
||||
Earlier claims in this file that have since been **disproven**:
|
||||
|
||||
- **Uncompressed `f32` vertex data.** The longest run of plausible coordinate
|
||||
floats in the whole 16.9 MB is a run of zeros in the collision block. Terrain
|
||||
vertices are packed.
|
||||
- **`GS packet tags` at every position.** Byte frequencies in 0xE0..0xEF are
|
||||
high but uniform, which is what compressed or packed data looks like, not a
|
||||
discrete packet stream.
|
||||
- **A uniform 48-byte record.** The pattern repeats at 64 bytes; 48 does not.
|
||||
- ~~"The terrain is 382 records of 64 bytes."~~ The 0x48 byte at offset +3 is
|
||||
simply the most common byte in the region (945 of 2000 sampled slots); it is
|
||||
not a record marker. Over 33,091 slots at that position only 4,127 hold 0x48.
|
||||
- ~~"Byte +3 is constant at 0x48 across 300 records."~~ True for the first 382
|
||||
records and false beyond, which is how the false pattern was found.
|
||||
- ~~"Coordinates are fixed-point s16 with a scale of 8."~~ The scale-8 reading
|
||||
is supported by one coincidence (17,496 / 8 = 2,187, the planet radius) but no
|
||||
grouping of the record into triples yields points on a sphere of that radius.
|
||||
Tested: every stride from 2 to 16 bytes, every starting offset, three and four
|
||||
components. Best result 5.0% of points on-sphere, which is chance.
|
||||
|
||||
So the record framing is **not recovered**. What is established is the region
|
||||
layout and that terrain is vertex-soup packed alongside inline texels — which
|
||||
means the vertex format must be identified from a longer run than a single
|
||||
record, most likely by locating a run of vertices whose magnitudes are bounded
|
||||
by the planet radius and whose Y values vary smoothly.
|
||||
|
||||
## Where the class tables are not
|
||||
|
||||
|
||||
@@ -111,6 +111,12 @@ Phase 1 hands over the extents; this phase works out what each one contains.
|
||||
written up in `plan/level-assets.md`. Next: identify `tfrag` by its own
|
||||
structure instead of through the header.
|
||||
- [x] `occlusion` and `sky` offsets located and their first bytes read.
|
||||
- [x] Region layout of `core_data` established: tfrag, occlusion, sky, collision
|
||||
and `textures_base` are offsets, strictly increasing on all 27 levels.
|
||||
Terrain is `[0, occlusion)` and carries inline texels.
|
||||
- [ ] **tfrag vertex format: not recovered.** The 64-byte record framing and the
|
||||
scale-8 fixed-point reading were both tested and disproven; see
|
||||
`plan/level-assets.md` for what was ruled out.
|
||||
- [ ] `gameplay` — begins `WAD` + a per-level byte, and does *not* decode with
|
||||
the WAD LZ77 decoder. All 27 levels fail identically, so the input is
|
||||
wrong rather than the decoder. Ruled out: bit order, header length, the
|
||||
|
||||
Reference in New Issue
Block a user