Going-Commando/docs/plan/level-assets.md

# Level assets

Phase 2 work record. What has been established about the four extents inside
each `LEVEL<n>.WAD`, and what remains.

## A bug found and fixed

Reading the `data` extent's contents exposed a defect in the Phase 1 extractor.

Level extents are declared in **sector counts**; the generic `G/` pair table
uses **byte counts**. The extractor routed level containers through the generic
path, so `LEVEL0.WAD`'s `data` extent — 6,484 sectors — was taken as 6,484
*bytes*. Level 0 yielded a 6,484-byte "geometry" file instead of 13,279,232
bytes. Across 27 levels that silently discarded roughly 300 MB of level content
while every test still passed.

The tell was arithmetic, not a crash: the extent's own header declared an
uncompressed total of 13,277,920 bytes, and the ratio against the 6,484-byte
file was 2047.7× — one sector, to within rounding. A ratio that lands on 2048
is a unit error.

Fixed by extracting level containers through the typed `LevelHeader` instead of
the generic table. Level 0 now yields:

```text
data.bin         13,279,232   the level container's data extent
core_data.wad     9,738,678   the geometry, compressed
core_data.bin    16,969,296   the geometry, decompressed
core_index.bin       31,600   the geometry directory
gameplay.bin       659,456
sound_bank.bin   1,503,232
```

**Lesson:** a unit error is invisible to a type system and to a passing test. The
check that catches it is arithmetic against an independent number the file
itself declares. Any extent whose contents describe a larger structure should
have that structure's own total compared against the extent.

## Done: the container header

`crates/gc-formats/src/level_wad.rs` parses the 0x60-byte header on all 27
levels and names the four extents. See
[`../formats/level-wad.md`](../formats/level-wad.md) for the layout and the
measured invariants.

Two facts worth carrying forward:

- **Sizes in this header are sector counts**, unlike the generic `G/*.WAD` pair
  table where they are byte counts. Both are on the same disc.
- **`LEVEL21.WAD` reports level id 30.** Container order is not level identity.

## `gameplay`: a compression we have not identified

This is the one genuinely puzzling result so far, and it is worth recording in
full because the next attempt will otherwise repeat the work.

The extent begins:

```text
offset 0  57 41 44   "WAD"
offset 3  79 0d 0a 0d   <- varies per level
offset 4+ <payload>
```

The variation is the tell. For the LZ77 WAD container the fourth byte is part of
a 32-bit little-endian length; here byte 3 differs per level (0x79, 0x3a, 0xd8,
0xa9, ...) while byte 4 is a small count (13, 88, 13, 105, 163, ...) and bytes 6
and 7 are always zero.

**Attempt: treat it as the known WAD LZ77 stream.** Decoded all 27 levels with
the decoder in `wad.rs`, starting after an 8-byte header. Every one produces
plausible output for the first packet — `0000 0000 0000 0750 0100 0020 2b0c...`
looks like real level data — and then fails on all 27 with "adjacent literal
packets", which the format's encoder never emits.

A uniform failure on all 27 inputs means the input is wrong, not the decoder.
Ruled out:

- **Bit order.** The WAD container is a flag-byte packet stream, not a bit
  stream, so no ordering applies. Both orderings fail identically.
- **Header length.** Tried 8 bytes. A different skip would not fix a failure
  that occurs uniformly in the first few packets.
- **The WAD format at all.** The first packet (flag `0x00`) decodes cleanly as a
  literal run of 18 bytes, which is *why* it looks plausible, but the run
  immediately after breaks the format's own rule. Whatever this is, it is not
  WAD LZ77.

Next to try, in order of cheapness:

1. Whether the payload is uncompressed and the header is a table. `(len - 8)`
   is 196..497 across the levels and divides by neither 8 nor a plausible stride,
   which argues against a flat pair table — but a *variable-length* table
   (counts of counts) fits that range.
2. Whether byte 4 is an entry count and bytes 5.. are records of a fixed but
   non-obvious width.
3. Whether the LZ77 variant differs only in the "adjacent literals" rule.

None of these is expensive to test; each is a few lines against all 27 levels at
once. The right move is to write a throwaway script that tries many hypotheses
across all 27 and reports which produce self-consistent results — the approach
that worked for the container pair table, where "does this hold on every level"
was the discriminator that the sector/byte ambiguity failed.

## `sound_bank`: signatures found, structure unknown

```text
+0x18  "SBlk"   present in all 27 levels
+0x24  "DAW"    present in all 27 levels
```

Sizes 458..833 bytes. Too small to be the level's audio; likely an index into
the global `AUDIO.WAD`.

## `occlusion`: header located, contents a u16 array

The header's `occlusion` offset (0x0c) is trustworthy — every offset in it lands
inside `core_data`. Level 0's occlusion block starts at 2,117,888 and reads as a
`u16` array:

```text
5152 0 12 23 14 19 24 29 32 35 38 41 44 47 50 53 56 59 65 82 96 100 113 126 143 164 183 ...
```

An ascending run of 65 values with mostly small differences, then it restarts.
That is consistent with per-cell visibility records rather than geometry. The
meaning of the pairs — and the 0 that ends each group — is not yet established.

## `sky`: plausible, unparsed

Header offset 0x10 = 2,192,576. Opens `0 0 0 0 0 0 6 0 0 0 0 0 5 0 0 0 40 0 0 0 c0 0 0 0 40 ...`
— small values including 64 and 192, which look like sky-shell counts and
dimensions rather than offsets.

## `tfrag` located: terrain is `[0, occlusion)`

The header's field at 0x08 is **zero on all 27 levels**, and 0x0c, 0x10, 0x14,
0x60, 0x64 and 0x68 are strictly increasing on every level. Reading 0x08 as the
terrain offset and 0x0c as its end gives:

```text
tfrag      [0, occlusion)      level 0: 2.12 MB of terrain
occlusion  [occl, sky)
sky        [sky, collision)
collision  [collision, textures_base)
```

So the terrain is the first region of `core_data`, and `tfrag = 0` is a real
value, not a missing field.

### What the terrain region contains

The opening bytes are a small header, after which the data is a repeating
**64-byte record** starting at 0x40:

```text
0x40  f5 ff 6d 48 01 00 77 48 cb a3 db 47 58 44 50 46 40 d0 00 ...
0x80  51 89 6b 48 9c 15 87 48 c2 57 b5 47 9a 24 fa 46 20 df 00 ...
0xc0  e3 a9 46 48 01 40 a4 48 00 00 f4 47 de b0 d0 46 50 ee 00 ...
```

What is established about these records:

- Byte columns 3, 7, 11 and 15 are **`0x48` in every record**. Those are the
  high bytes of four `s16` values, i.e. a fixed Y coordinate of 0x4848-ish on a
  planet whose radius is about 2,187 units.
- Columns 16 and 17 increase steadily across records (0x40, 0x50, 0x60 … and
  0xd0, 0xdf, 0xee …), consistent with a grid walk.
- Columns 48–51 hold a repeating `ffff00ff` pattern, which is characteristic of
  GS packet headers rather than vertex data.
- Bytes at 0x4c–0x4f repeat `ff 00 ff ff` on **every** record.

### The record layout, as far as it is established

```text
382 records of 64 bytes, from 0x40 to 0x5fc0
```

That count is exact: 383 breaks the pattern at 0x5fc0. Per column, over 300
records:

| Column | Value | Notes |
|---|---|---|
| +3 | `0x48` | 300/300 records. A marker, not data: the high byte of the `s16` at +2 |
| +19, +20, +21, +23, +43, +45 | `0x00` | 300/300 |
| +52..55 | `ff 00 ff ff` | 300/300. A fixed signature |
| +2, +6, +10, +14, +18 … | varying | the data |

**Coordinates are fixed-point `s16` with a scale of 8.** Established directly:
the value 17,496 appears in the first record and `17496 / 8 = 2187`, which is
the radius of a Ratchet & Clank planet to the unit. Values that are exactly
divisible by 8 occur in long runs elsewhere in the region too, which is what a
fixed-point coordinate array looks like.

**Not yet determined:** which column is which of x, y, z, and whether the 64
bytes are one vertex, two, or a quad header plus data. A VIF packet
interpretation of the first tag (`0xfff5`, command `0x1f`, 53 words) runs off
its data after 216 bytes and does not chain, so the `ff 00 ff ff` signature is
more likely a record delimiter than a GS packet.

The next test is mechanical: take one 64-byte record, try every grouping into
3-component vertices, and keep the one whose resulting points all lie within a
sphere of radius ~2,187. That is a decidable question and does not need more
speculation.

Ruled out along the way:

- **Uncompressed `f32` vertex data.** The longest run of plausible coordinate
  floats in the whole 16.9 MB is a run of zeros in the collision block. Terrain
  vertices are packed.
- **`GS packet tags` at every position.** Byte frequencies in 0xE0..0xEF are
  high but uniform, which is what compressed or packed data looks like, not a
  discrete packet stream.
- **A uniform 48-byte record.** The pattern repeats at 64 bytes; 48 does not.

## Where the class tables are not

The header's `moby_classes` and friends do not decode. Read as
`(offset, count)` they give plausible numbers, but the entries at those offsets
are not valid records. Ruled out:

- **Field order.** Both `(offset, count)` and `(count, offset)` produce garbage.
- **Stride.** Tried 8, 16 and 32 bytes per entry.
- **Which file the offsets index.** Tried `core_data` and `core_index`.
- **Header base.** Not the issue: the compressed size (0x88) and decompressed
  size (0x8c) of `core_data` appear at the *same offsets* in all 27 levels'
  `core_index`, which pins the header base at 0 with certainty.

One observation worth keeping: every second value in the class-table fields is
exactly divisible by 16, and they increase monotonically across the nine tables.
That is consistent with `count * 16` sizes, but the corresponding offsets
cluster at the very start of the file and do not yield valid records either.

The discriminator used to reject candidates is worth stating, because weaker
tests produce false positives: a real class table's `offset_in_asset_wad`
values increase. A search for "32-byte records with increasing offsets" returns
several hundred candidates across the file, and the top one turned out to be a
texture table whose `o_class` field merely happened to equal `offset + 16`.
Scoring on plausibility alone is not enough; the entries have to actually chain.

Next: rather than guess the header further, identify `tfrag` by its own
structure. Terrain is the largest subsystem and the one that decides whether
anything draws, so it is the right next target even though it is the harder one.

## `data`: the real target, untouched

6,484 sectors on level 0, down to 2,102 on level 24 — 4.2 to 21.9 MB. This is
the level's geometry and the largest thing in the container. Next target.

What is known from the community reference and from Insomniac's engine family:
it splits into four geometry subsystems — `tfrag` (terrain fragments),
`tie` (instanced environment structures), `shrub` (foliage) and `moby` (dynamic
actors) — plus sky shells, collision data and textures. ReRAC's documentation of
the equivalent structures in the first game is the closest reference available,
though R&C2's spherical worlds change how they are used.

## Order of work

1. `gameplay` — smallest, fully bounded, and the format is identified enough to
   make progress. A win here validates the approach before the large effort.
2. `occlusion` — tiny.
3. `sound_bank` — small.
4. `data` — the large one, and the one that decides whether anything renders.

Doing the small extents first is not just convenience: each one is a contained
reverse-engineering problem that either resolves or tells us something, whereas
`data` will keep producing ambiguous readings until we have something to check
them against.

## Method note

Every format question this phase has been settled the same way: write the
hypothesis, test it against **all 27 levels at once**, and require an invariant
that holds everywhere. The sector/byte ambiguity in Phase 1 survived because it
happened to be correct on the one container checked by hand. The level header
was fixed in a single pass because the tiling check failed immediately on
levels 1, 2, 4, 7, 8, 11, 19 and 20.