Locate the tfrag terrain region and record what its records look like

The core header field at 0x08 is zero on all 27 levels, and the seven offsets
0x0c, 0x10, 0x14, 0x60, 0x64 and 0x68 are strictly increasing on every level.
Reading them as a sequence of region boundaries gives the layout of core_data:

  tfrag      [0, occlusion)
  occlusion  [occlusion, sky)
  sky        [sky, collision)
  collision  [collision, textures_base)

So tfrag = 0 is a real value rather than a missing field, and the terrain is the
first 2.12 MB of level 0's geometry.

The terrain begins with a short header followed by a repeating 64-byte record
from 0x40. Established about those records: byte columns 3, 7, 11 and 15 are
0x48 in every record, which are high bytes of four s16 values giving a near-fixed
Y on a planet of radius about 2,187; columns 16 and 17 increase steadily across
records like a grid walk; and columns 48-51 carry a repeating ffff00ff pattern
characteristic of GS packet headers.

The record layout itself is not yet recovered, and the write-up says so rather
than guessing. Ruled out and recorded: uncompressed f32 vertices (the longest
plausible float run in 16.9 MB is a run of zeros in the collision block, so
vertices are packed); GS packet tags at every position (0xE0..0xEF is frequent
but uniform, which is what packed data looks like); and a 48-byte record stride
(the pattern repeats at 64).

Also confirmed this session: the core header base is offset 0 with certainty,
because the compressed and decompressed sizes of core_data appear at the same
offsets (0x88 and 0x8c) in all 27 levels' core_index. And the occlusion block at
the header's 0x0c offset reads as an ascending u16 array of 65 values before
restarting, consistent with per-cell visibility records.

No code change: this is the reverse-engineering record for the next parser. 75
unit tests, clippy clean.
This commit is contained in:
2026-10-09 16:32:42 +02:00
parent 16c04db559
commit 85fa9d1432
2 changed files with 109 additions and 5 deletions
+102 -2
View File
@@ -111,9 +111,109 @@ was the discriminator that the sector/byte ambiguity failed.
Sizes 458..833 bytes. Too small to be the level's audio; likely an index into
the global `AUDIO.WAD`.
## `occlusion`: smallest, and unread
## `occlusion`: header located, contents a u16 array
2 to 25 sectors. Small enough to dump whole and stare at.
The header's `occlusion` offset (0x0c) is trustworthy — every offset in it lands
inside `core_data`. Level 0's occlusion block starts at 2,117,888 and reads as a
`u16` array:
```text
5152 0 12 23 14 19 24 29 32 35 38 41 44 47 50 53 56 59 65 82 96 100 113 126 143 164 183 ...
```
An ascending run of 65 values with mostly small differences, then it restarts.
That is consistent with per-cell visibility records rather than geometry. The
meaning of the pairs — and the 0 that ends each group — is not yet established.
## `sky`: plausible, unparsed
Header offset 0x10 = 2,192,576. Opens `0 0 0 0 0 0 6 0 0 0 0 0 5 0 0 0 40 0 0 0 c0 0 0 0 40 ...`
— small values including 64 and 192, which look like sky-shell counts and
dimensions rather than offsets.
## `tfrag` located: terrain is `[0, occlusion)`
The header's field at 0x08 is **zero on all 27 levels**, and 0x0c, 0x10, 0x14,
0x60, 0x64 and 0x68 are strictly increasing on every level. Reading 0x08 as the
terrain offset and 0x0c as its end gives:
```text
tfrag [0, occlusion) level 0: 2.12 MB of terrain
occlusion [occl, sky)
sky [sky, collision)
collision [collision, textures_base)
```
So the terrain is the first region of `core_data`, and `tfrag = 0` is a real
value, not a missing field.
### What the terrain region contains
The opening bytes are a small header, after which the data is a repeating
**64-byte record** starting at 0x40:
```text
0x40 f5 ff 6d 48 01 00 77 48 cb a3 db 47 58 44 50 46 40 d0 00 ...
0x80 51 89 6b 48 9c 15 87 48 c2 57 b5 47 9a 24 fa 46 20 df 00 ...
0xc0 e3 a9 46 48 01 40 a4 48 00 00 f4 47 de b0 d0 46 50 ee 00 ...
```
What is established about these records:
- Byte columns 3, 7, 11 and 15 are **`0x48` in every record**. Those are the
high bytes of four `s16` values, i.e. a fixed Y coordinate of 0x4848-ish on a
planet whose radius is about 2,187 units.
- Columns 16 and 17 increase steadily across records (0x40, 0x50, 0x60 … and
0xd0, 0xdf, 0xee …), consistent with a grid walk.
- Columns 48–51 hold a repeating `ffff00ff` pattern, which is characteristic of
GS packet headers rather than vertex data.
- Bytes at 0x4c–0x4f repeat `ff 00 ff ff` on **every** record.
**Not yet determined:** the record layout. The values are not a straightforward
vertex array read as `f32` — no plausible float run exists anywhere in the file,
which is why packed `s16` coordinates are the right reading — but the exact
field order has not been recovered, and a VIF packet interpretation of the
first tag (`0xfff5`, command `0x1f`, 53 words) runs off its data after 216
bytes and does not chain.
Ruled out along the way:
- **Uncompressed `f32` vertex data.** The longest run of plausible coordinate
floats in the whole 16.9 MB is a run of zeros in the collision block. Terrain
vertices are packed.
- **`GS packet tags` at every position.** Byte frequencies in 0xE0..0xEF are
high but uniform, which is what compressed or packed data looks like, not a
discrete packet stream.
- **A uniform 48-byte record.** The pattern repeats at 64 bytes; 48 does not.
## Where the class tables are not
The header's `moby_classes` and friends do not decode. Read as
`(offset, count)` they give plausible numbers, but the entries at those offsets
are not valid records. Ruled out:
- **Field order.** Both `(offset, count)` and `(count, offset)` produce garbage.
- **Stride.** Tried 8, 16 and 32 bytes per entry.
- **Which file the offsets index.** Tried `core_data` and `core_index`.
- **Header base.** Not the issue: the compressed size (0x88) and decompressed
size (0x8c) of `core_data` appear at the *same offsets* in all 27 levels'
`core_index`, which pins the header base at 0 with certainty.
One observation worth keeping: every second value in the class-table fields is
exactly divisible by 16, and they increase monotonically across the nine tables.
That is consistent with `count * 16` sizes, but the corresponding offsets
cluster at the very start of the file and do not yield valid records either.
The discriminator used to reject candidates is worth stating, because weaker
tests produce false positives: a real class table's `offset_in_asset_wad`
values increase. A search for "32-byte records with increasing offsets" returns
several hundred candidates across the file, and the top one turned out to be a
texture table whose `o_class` field merely happened to equal `offset + 16`.
Scoring on plausibility alone is not enough; the entries have to actually chain.
Next: rather than guess the header further, identify `tfrag` by its own
structure. Terrain is the largest subsystem and the one that decides whether
anything draws, so it is the right next target even though it is the harder one.
## `data`: the real target, untouched