diff --git a/docs/plan/level-assets.md b/docs/plan/level-assets.md index 9898901..52e16e5 100644 --- a/docs/plan/level-assets.md +++ b/docs/plan/level-assets.md @@ -111,9 +111,109 @@ was the discriminator that the sector/byte ambiguity failed. Sizes 458..833 bytes. Too small to be the level's audio; likely an index into the global `AUDIO.WAD`. -## `occlusion`: smallest, and unread +## `occlusion`: header located, contents a u16 array -2 to 25 sectors. Small enough to dump whole and stare at. +The header's `occlusion` offset (0x0c) is trustworthy — every offset in it lands +inside `core_data`. Level 0's occlusion block starts at 2,117,888 and reads as a +`u16` array: + +```text +5152 0 12 23 14 19 24 29 32 35 38 41 44 47 50 53 56 59 65 82 96 100 113 126 143 164 183 ... +``` + +An ascending run of 65 values with mostly small differences, then it restarts. +That is consistent with per-cell visibility records rather than geometry. The +meaning of the pairs — and the 0 that ends each group — is not yet established. + +## `sky`: plausible, unparsed + +Header offset 0x10 = 2,192,576. Opens `0 0 0 0 0 0 6 0 0 0 0 0 5 0 0 0 40 0 0 0 c0 0 0 0 40 ...` +— small values including 64 and 192, which look like sky-shell counts and +dimensions rather than offsets. + +## `tfrag` located: terrain is `[0, occlusion)` + +The header's field at 0x08 is **zero on all 27 levels**, and 0x0c, 0x10, 0x14, +0x60, 0x64 and 0x68 are strictly increasing on every level. Reading 0x08 as the +terrain offset and 0x0c as its end gives: + +```text +tfrag [0, occlusion) level 0: 2.12 MB of terrain +occlusion [occl, sky) +sky [sky, collision) +collision [collision, textures_base) +``` + +So the terrain is the first region of `core_data`, and `tfrag = 0` is a real +value, not a missing field. + +### What the terrain region contains + +The opening bytes are a small header, after which the data is a repeating +**64-byte record** starting at 0x40: + +```text +0x40 f5 ff 6d 48 01 00 77 48 cb a3 db 47 58 44 50 46 40 d0 00 ... +0x80 51 89 6b 48 9c 15 87 48 c2 57 b5 47 9a 24 fa 46 20 df 00 ... +0xc0 e3 a9 46 48 01 40 a4 48 00 00 f4 47 de b0 d0 46 50 ee 00 ... +``` + +What is established about these records: + +- Byte columns 3, 7, 11 and 15 are **`0x48` in every record**. Those are the + high bytes of four `s16` values, i.e. a fixed Y coordinate of 0x4848-ish on a + planet whose radius is about 2,187 units. +- Columns 16 and 17 increase steadily across records (0x40, 0x50, 0x60 … and + 0xd0, 0xdf, 0xee …), consistent with a grid walk. +- Columns 48–51 hold a repeating `ffff00ff` pattern, which is characteristic of + GS packet headers rather than vertex data. +- Bytes at 0x4c–0x4f repeat `ff 00 ff ff` on **every** record. + +**Not yet determined:** the record layout. The values are not a straightforward +vertex array read as `f32` — no plausible float run exists anywhere in the file, +which is why packed `s16` coordinates are the right reading — but the exact +field order has not been recovered, and a VIF packet interpretation of the +first tag (`0xfff5`, command `0x1f`, 53 words) runs off its data after 216 +bytes and does not chain. + +Ruled out along the way: + +- **Uncompressed `f32` vertex data.** The longest run of plausible coordinate + floats in the whole 16.9 MB is a run of zeros in the collision block. Terrain + vertices are packed. +- **`GS packet tags` at every position.** Byte frequencies in 0xE0..0xEF are + high but uniform, which is what compressed or packed data looks like, not a + discrete packet stream. +- **A uniform 48-byte record.** The pattern repeats at 64 bytes; 48 does not. + +## Where the class tables are not + +The header's `moby_classes` and friends do not decode. Read as +`(offset, count)` they give plausible numbers, but the entries at those offsets +are not valid records. Ruled out: + +- **Field order.** Both `(offset, count)` and `(count, offset)` produce garbage. +- **Stride.** Tried 8, 16 and 32 bytes per entry. +- **Which file the offsets index.** Tried `core_data` and `core_index`. +- **Header base.** Not the issue: the compressed size (0x88) and decompressed + size (0x8c) of `core_data` appear at the *same offsets* in all 27 levels' + `core_index`, which pins the header base at 0 with certainty. + +One observation worth keeping: every second value in the class-table fields is +exactly divisible by 16, and they increase monotonically across the nine tables. +That is consistent with `count * 16` sizes, but the corresponding offsets +cluster at the very start of the file and do not yield valid records either. + +The discriminator used to reject candidates is worth stating, because weaker +tests produce false positives: a real class table's `offset_in_asset_wad` +values increase. A search for "32-byte records with increasing offsets" returns +several hundred candidates across the file, and the top one turned out to be a +texture table whose `o_class` field merely happened to equal `offset + 16`. +Scoring on plausibility alone is not enough; the entries have to actually chain. + +Next: rather than guess the header further, identify `tfrag` by its own +structure. Terrain is the largest subsystem and the one that decides whether +anything draws, so it is the right next target even though it is the harder one. ## `data`: the real target, untouched diff --git a/docs/roadmap.md b/docs/roadmap.md index 10a1656..53f3a03 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -104,9 +104,13 @@ Phase 1 hands over the extents; this phase works out what each one contains. - [x] `data` — the extent header locates `core_index` and `core_data`; the geometry decompresses and is verified on level 0. -- [ ] `core_index` class tables — offsets all land inside `core_data`, which - confirms the header, but the `ArrayRange` fields do not decode to a usable - class table under the documented layout. Next thing to resolve. +- [ ] `core_index` class tables — the header base is confirmed (the compressed + and decompressed sizes of `core_data` sit at the same offsets in all 27 + levels), and every offset lands inside `core_data`. But the class-table + fields do not decode under any field order or stride tried. Ruled out and + written up in `plan/level-assets.md`. Next: identify `tfrag` by its own + structure instead of through the header. +- [x] `occlusion` and `sky` offsets located and their first bytes read. - [ ] `gameplay` — begins `WAD` + a per-level byte, and does *not* decode with the WAD LZ77 decoder. All 27 levels fail identically, so the input is wrong rather than the decoder. Ruled out: bit order, header length, the