Locate the tfrag terrain region and record what its records look like
The core header field at 0x08 is zero on all 27 levels, and the seven offsets 0x0c, 0x10, 0x14, 0x60, 0x64 and 0x68 are strictly increasing on every level. Reading them as a sequence of region boundaries gives the layout of core_data: tfrag [0, occlusion) occlusion [occlusion, sky) sky [sky, collision) collision [collision, textures_base) So tfrag = 0 is a real value rather than a missing field, and the terrain is the first 2.12 MB of level 0's geometry. The terrain begins with a short header followed by a repeating 64-byte record from 0x40. Established about those records: byte columns 3, 7, 11 and 15 are 0x48 in every record, which are high bytes of four s16 values giving a near-fixed Y on a planet of radius about 2,187; columns 16 and 17 increase steadily across records like a grid walk; and columns 48-51 carry a repeating ffff00ff pattern characteristic of GS packet headers. The record layout itself is not yet recovered, and the write-up says so rather than guessing. Ruled out and recorded: uncompressed f32 vertices (the longest plausible float run in 16.9 MB is a run of zeros in the collision block, so vertices are packed); GS packet tags at every position (0xE0..0xEF is frequent but uniform, which is what packed data looks like); and a 48-byte record stride (the pattern repeats at 64). Also confirmed this session: the core header base is offset 0 with certainty, because the compressed and decompressed sizes of core_data appear at the same offsets (0x88 and 0x8c) in all 27 levels' core_index. And the occlusion block at the header's 0x0c offset reads as an ascending u16 array of 65 values before restarting, consistent with per-cell visibility records. No code change: this is the reverse-engineering record for the next parser. 75 unit tests, clippy clean.
This commit is contained in:
+102
-2
@@ -111,9 +111,109 @@ was the discriminator that the sector/byte ambiguity failed.
|
||||
Sizes 458..833 bytes. Too small to be the level's audio; likely an index into
|
||||
the global `AUDIO.WAD`.
|
||||
|
||||
## `occlusion`: smallest, and unread
|
||||
## `occlusion`: header located, contents a u16 array
|
||||
|
||||
2 to 25 sectors. Small enough to dump whole and stare at.
|
||||
The header's `occlusion` offset (0x0c) is trustworthy — every offset in it lands
|
||||
inside `core_data`. Level 0's occlusion block starts at 2,117,888 and reads as a
|
||||
`u16` array:
|
||||
|
||||
```text
|
||||
5152 0 12 23 14 19 24 29 32 35 38 41 44 47 50 53 56 59 65 82 96 100 113 126 143 164 183 ...
|
||||
```
|
||||
|
||||
An ascending run of 65 values with mostly small differences, then it restarts.
|
||||
That is consistent with per-cell visibility records rather than geometry. The
|
||||
meaning of the pairs — and the 0 that ends each group — is not yet established.
|
||||
|
||||
## `sky`: plausible, unparsed
|
||||
|
||||
Header offset 0x10 = 2,192,576. Opens `0 0 0 0 0 0 6 0 0 0 0 0 5 0 0 0 40 0 0 0 c0 0 0 0 40 ...`
|
||||
— small values including 64 and 192, which look like sky-shell counts and
|
||||
dimensions rather than offsets.
|
||||
|
||||
## `tfrag` located: terrain is `[0, occlusion)`
|
||||
|
||||
The header's field at 0x08 is **zero on all 27 levels**, and 0x0c, 0x10, 0x14,
|
||||
0x60, 0x64 and 0x68 are strictly increasing on every level. Reading 0x08 as the
|
||||
terrain offset and 0x0c as its end gives:
|
||||
|
||||
```text
|
||||
tfrag [0, occlusion) level 0: 2.12 MB of terrain
|
||||
occlusion [occl, sky)
|
||||
sky [sky, collision)
|
||||
collision [collision, textures_base)
|
||||
```
|
||||
|
||||
So the terrain is the first region of `core_data`, and `tfrag = 0` is a real
|
||||
value, not a missing field.
|
||||
|
||||
### What the terrain region contains
|
||||
|
||||
The opening bytes are a small header, after which the data is a repeating
|
||||
**64-byte record** starting at 0x40:
|
||||
|
||||
```text
|
||||
0x40 f5 ff 6d 48 01 00 77 48 cb a3 db 47 58 44 50 46 40 d0 00 ...
|
||||
0x80 51 89 6b 48 9c 15 87 48 c2 57 b5 47 9a 24 fa 46 20 df 00 ...
|
||||
0xc0 e3 a9 46 48 01 40 a4 48 00 00 f4 47 de b0 d0 46 50 ee 00 ...
|
||||
```
|
||||
|
||||
What is established about these records:
|
||||
|
||||
- Byte columns 3, 7, 11 and 15 are **`0x48` in every record**. Those are the
|
||||
high bytes of four `s16` values, i.e. a fixed Y coordinate of 0x4848-ish on a
|
||||
planet whose radius is about 2,187 units.
|
||||
- Columns 16 and 17 increase steadily across records (0x40, 0x50, 0x60 … and
|
||||
0xd0, 0xdf, 0xee …), consistent with a grid walk.
|
||||
- Columns 48–51 hold a repeating `ffff00ff` pattern, which is characteristic of
|
||||
GS packet headers rather than vertex data.
|
||||
- Bytes at 0x4c–0x4f repeat `ff 00 ff ff` on **every** record.
|
||||
|
||||
**Not yet determined:** the record layout. The values are not a straightforward
|
||||
vertex array read as `f32` — no plausible float run exists anywhere in the file,
|
||||
which is why packed `s16` coordinates are the right reading — but the exact
|
||||
field order has not been recovered, and a VIF packet interpretation of the
|
||||
first tag (`0xfff5`, command `0x1f`, 53 words) runs off its data after 216
|
||||
bytes and does not chain.
|
||||
|
||||
Ruled out along the way:
|
||||
|
||||
- **Uncompressed `f32` vertex data.** The longest run of plausible coordinate
|
||||
floats in the whole 16.9 MB is a run of zeros in the collision block. Terrain
|
||||
vertices are packed.
|
||||
- **`GS packet tags` at every position.** Byte frequencies in 0xE0..0xEF are
|
||||
high but uniform, which is what compressed or packed data looks like, not a
|
||||
discrete packet stream.
|
||||
- **A uniform 48-byte record.** The pattern repeats at 64 bytes; 48 does not.
|
||||
|
||||
## Where the class tables are not
|
||||
|
||||
The header's `moby_classes` and friends do not decode. Read as
|
||||
`(offset, count)` they give plausible numbers, but the entries at those offsets
|
||||
are not valid records. Ruled out:
|
||||
|
||||
- **Field order.** Both `(offset, count)` and `(count, offset)` produce garbage.
|
||||
- **Stride.** Tried 8, 16 and 32 bytes per entry.
|
||||
- **Which file the offsets index.** Tried `core_data` and `core_index`.
|
||||
- **Header base.** Not the issue: the compressed size (0x88) and decompressed
|
||||
size (0x8c) of `core_data` appear at the *same offsets* in all 27 levels'
|
||||
`core_index`, which pins the header base at 0 with certainty.
|
||||
|
||||
One observation worth keeping: every second value in the class-table fields is
|
||||
exactly divisible by 16, and they increase monotonically across the nine tables.
|
||||
That is consistent with `count * 16` sizes, but the corresponding offsets
|
||||
cluster at the very start of the file and do not yield valid records either.
|
||||
|
||||
The discriminator used to reject candidates is worth stating, because weaker
|
||||
tests produce false positives: a real class table's `offset_in_asset_wad`
|
||||
values increase. A search for "32-byte records with increasing offsets" returns
|
||||
several hundred candidates across the file, and the top one turned out to be a
|
||||
texture table whose `o_class` field merely happened to equal `offset + 16`.
|
||||
Scoring on plausibility alone is not enough; the entries have to actually chain.
|
||||
|
||||
Next: rather than guess the header further, identify `tfrag` by its own
|
||||
structure. Terrain is the largest subsystem and the one that decides whether
|
||||
anything draws, so it is the right next target even though it is the harder one.
|
||||
|
||||
## `data`: the real target, untouched
|
||||
|
||||
|
||||
+7
-3
@@ -104,9 +104,13 @@ Phase 1 hands over the extents; this phase works out what each one contains.
|
||||
|
||||
- [x] `data` — the extent header locates `core_index` and `core_data`; the
|
||||
geometry decompresses and is verified on level 0.
|
||||
- [ ] `core_index` class tables — offsets all land inside `core_data`, which
|
||||
confirms the header, but the `ArrayRange` fields do not decode to a usable
|
||||
class table under the documented layout. Next thing to resolve.
|
||||
- [ ] `core_index` class tables — the header base is confirmed (the compressed
|
||||
and decompressed sizes of `core_data` sit at the same offsets in all 27
|
||||
levels), and every offset lands inside `core_data`. But the class-table
|
||||
fields do not decode under any field order or stride tried. Ruled out and
|
||||
written up in `plan/level-assets.md`. Next: identify `tfrag` by its own
|
||||
structure instead of through the header.
|
||||
- [x] `occlusion` and `sky` offsets located and their first bytes read.
|
||||
- [ ] `gameplay` — begins `WAD` + a per-level byte, and does *not* decode with
|
||||
the WAD LZ77 decoder. All 27 levels fail identically, so the input is
|
||||
wrong rather than the decoder. Ruled out: bit order, header length, the
|
||||
|
||||
Reference in New Issue
Block a user