ReaperOfVeriod
3ec548efe8
Two claims in the previous commit were wrong and are retracted here rather than
left standing for the next person to build on.
"382 records of 64 bytes" and "byte +3 is 0x48 in every record" were both
artefacts. The 0x48 at that offset is simply the most common byte in the region --
945 of 2000 sampled slots -- not a marker. Over 33,091 slots at that position
only 4,127 hold 0x48. The pattern holds for the first 382 records and then
breaks, which is exactly how a coincidence of a dense byte value masquerades as
structure.
"The coordinates are fixed-point s16 with a scale of 8" is also not supported.
It rested on one coincidence: 17,496 / 8 = 2,187, the planet radius. No grouping
of the record into coordinate triples produces points on a sphere of that radius.
Tested every stride from 2 to 16 bytes at every starting offset, with three and
four components; the best result is 5.0% of points on-sphere, which is chance.
What is now established, and holds on all 27 levels:
core_data region layout, as strictly increasing offsets
tfrag [0, occlusion)
occlusion [occlusion, sky)
sky [sky, collision)
collision [collision, textures_base)
Ruled out the competing reading where 0x0c/0x10/0x14 are sizes laid end to end:
the cumulative total misses textures_base on every level and the leading bytes
of each supposed region are not plausible structures.
Also new: terrain carries its texels inline. A scan for the PS2 texel pattern
18 18 18 80 finds 6,739 occurrences in 138 clusters starting at 0xd9f8, well
inside the tfrag region, so textures_base names something other than terrain
textures.
The vertex format remains unrecovered and the record says so. The next attempt
should look for a longer run of plausible vertices rather than re-deriving a
framing from one record.
240 lines
11 KiB
Markdown
240 lines
11 KiB
Markdown
# Level assets
|
||
|
||
Phase 2 work record. What has been established about the four extents inside
|
||
each `LEVEL<n>.WAD`, and what remains.
|
||
|
||
## A bug found and fixed
|
||
|
||
Reading the `data` extent's contents exposed a defect in the Phase 1 extractor.
|
||
|
||
Level extents are declared in **sector counts**; the generic `G/` pair table
|
||
uses **byte counts**. The extractor routed level containers through the generic
|
||
path, so `LEVEL0.WAD`'s `data` extent — 6,484 sectors — was taken as 6,484
|
||
*bytes*. Level 0 yielded a 6,484-byte "geometry" file instead of 13,279,232
|
||
bytes. Across 27 levels that silently discarded roughly 300 MB of level content
|
||
while every test still passed.
|
||
|
||
The tell was arithmetic, not a crash: the extent's own header declared an
|
||
uncompressed total of 13,277,920 bytes, and the ratio against the 6,484-byte
|
||
file was 2047.7× — one sector, to within rounding. A ratio that lands on 2048
|
||
is a unit error.
|
||
|
||
Fixed by extracting level containers through the typed `LevelHeader` instead of
|
||
the generic table. Level 0 now yields:
|
||
|
||
```text
|
||
data.bin 13,279,232 the level container's data extent
|
||
core_data.wad 9,738,678 the geometry, compressed
|
||
core_data.bin 16,969,296 the geometry, decompressed
|
||
core_index.bin 31,600 the geometry directory
|
||
gameplay.bin 659,456
|
||
sound_bank.bin 1,503,232
|
||
```
|
||
|
||
**Lesson:** a unit error is invisible to a type system and to a passing test. The
|
||
check that catches it is arithmetic against an independent number the file
|
||
itself declares. Any extent whose contents describe a larger structure should
|
||
have that structure's own total compared against the extent.
|
||
|
||
## Done: the container header
|
||
|
||
`crates/gc-formats/src/level_wad.rs` parses the 0x60-byte header on all 27
|
||
levels and names the four extents. See
|
||
[`../formats/level-wad.md`](../formats/level-wad.md) for the layout and the
|
||
measured invariants.
|
||
|
||
Two facts worth carrying forward:
|
||
|
||
- **Sizes in this header are sector counts**, unlike the generic `G/*.WAD` pair
|
||
table where they are byte counts. Both are on the same disc.
|
||
- **`LEVEL21.WAD` reports level id 30.** Container order is not level identity.
|
||
|
||
## `gameplay`: a compression we have not identified
|
||
|
||
This is the one genuinely puzzling result so far, and it is worth recording in
|
||
full because the next attempt will otherwise repeat the work.
|
||
|
||
The extent begins:
|
||
|
||
```text
|
||
offset 0 57 41 44 "WAD"
|
||
offset 3 79 0d 0a 0d <- varies per level
|
||
offset 4+ <payload>
|
||
```
|
||
|
||
The variation is the tell. For the LZ77 WAD container the fourth byte is part of
|
||
a 32-bit little-endian length; here byte 3 differs per level (0x79, 0x3a, 0xd8,
|
||
0xa9, ...) while byte 4 is a small count (13, 88, 13, 105, 163, ...) and bytes 6
|
||
and 7 are always zero.
|
||
|
||
**Attempt: treat it as the known WAD LZ77 stream.** Decoded all 27 levels with
|
||
the decoder in `wad.rs`, starting after an 8-byte header. Every one produces
|
||
plausible output for the first packet — `0000 0000 0000 0750 0100 0020 2b0c...`
|
||
looks like real level data — and then fails on all 27 with "adjacent literal
|
||
packets", which the format's encoder never emits.
|
||
|
||
A uniform failure on all 27 inputs means the input is wrong, not the decoder.
|
||
Ruled out:
|
||
|
||
- **Bit order.** The WAD container is a flag-byte packet stream, not a bit
|
||
stream, so no ordering applies. Both orderings fail identically.
|
||
- **Header length.** Tried 8 bytes. A different skip would not fix a failure
|
||
that occurs uniformly in the first few packets.
|
||
- **The WAD format at all.** The first packet (flag `0x00`) decodes cleanly as a
|
||
literal run of 18 bytes, which is *why* it looks plausible, but the run
|
||
immediately after breaks the format's own rule. Whatever this is, it is not
|
||
WAD LZ77.
|
||
|
||
Next to try, in order of cheapness:
|
||
|
||
1. Whether the payload is uncompressed and the header is a table. `(len - 8)`
|
||
is 196..497 across the levels and divides by neither 8 nor a plausible stride,
|
||
which argues against a flat pair table — but a *variable-length* table
|
||
(counts of counts) fits that range.
|
||
2. Whether byte 4 is an entry count and bytes 5.. are records of a fixed but
|
||
non-obvious width.
|
||
3. Whether the LZ77 variant differs only in the "adjacent literals" rule.
|
||
|
||
None of these is expensive to test; each is a few lines against all 27 levels at
|
||
once. The right move is to write a throwaway script that tries many hypotheses
|
||
across all 27 and reports which produce self-consistent results — the approach
|
||
that worked for the container pair table, where "does this hold on every level"
|
||
was the discriminator that the sector/byte ambiguity failed.
|
||
|
||
## `sound_bank`: signatures found, structure unknown
|
||
|
||
```text
|
||
+0x18 "SBlk" present in all 27 levels
|
||
+0x24 "DAW" present in all 27 levels
|
||
```
|
||
|
||
Sizes 458..833 bytes. Too small to be the level's audio; likely an index into
|
||
the global `AUDIO.WAD`.
|
||
|
||
## `occlusion`: header located, contents a u16 array
|
||
|
||
The header's `occlusion` offset (0x0c) is trustworthy — every offset in it lands
|
||
inside `core_data`. Level 0's occlusion block starts at 2,117,888 and reads as a
|
||
`u16` array:
|
||
|
||
```text
|
||
5152 0 12 23 14 19 24 29 32 35 38 41 44 47 50 53 56 59 65 82 96 100 113 126 143 164 183 ...
|
||
```
|
||
|
||
An ascending run of 65 values with mostly small differences, then it restarts.
|
||
That is consistent with per-cell visibility records rather than geometry. The
|
||
meaning of the pairs — and the 0 that ends each group — is not yet established.
|
||
|
||
## `sky`: plausible, unparsed
|
||
|
||
Header offset 0x10 = 2,192,576. Opens `0 0 0 0 0 0 6 0 0 0 0 0 5 0 0 0 40 0 0 0 c0 0 0 0 40 ...`
|
||
— small values including 64 and 192, which look like sky-shell counts and
|
||
dimensions rather than offsets.
|
||
|
||
## `tfrag` region: `[0, occlusion)`
|
||
|
||
The header's fields at 0x0c, 0x10, 0x14 and 0x60 are **offsets**, and they are
|
||
strictly increasing on all 27 levels. The differences between them give region
|
||
sizes that are plausible on every level:
|
||
|
||
| Region | Range | Level 0 size |
|
||
|---|---|---|
|
||
| tfrag | `[0, occlusion)` | 2.12 MB |
|
||
| occlusion | `[occlusion, sky)` | 0.07 MB |
|
||
| sky | `[sky, collision)` | 0.42 MB |
|
||
| collision | `[collision, textures_base)` | 0.90 MB |
|
||
|
||
Ruled out: reading 0x0c/0x10/0x14 as *sizes* laid end to end. The cumulative
|
||
total does not reach `textures_base` on any level, and the byte at the start of
|
||
each supposed region is not a plausible structure. The offsets model is the one
|
||
that holds everywhere.
|
||
|
||
### What is in the terrain region
|
||
|
||
**Textures are stored inline with the geometry**, not in a separate bank. A scan
|
||
for the PS2 texel pattern `18 18 18 80` (grey texels with the 0x80 alpha that
|
||
means "opaque" in the GS's convention) finds 6,739 occurrences spread across 138
|
||
clusters, the first at `0xd9f8` — inside the tfrag region. So terrain carries its
|
||
own texels, and `textures_base` names something else.
|
||
|
||
Earlier claims in this file that have since been **disproven**:
|
||
|
||
- ~~"The terrain is 382 records of 64 bytes."~~ The 0x48 byte at offset +3 is
|
||
simply the most common byte in the region (945 of 2000 sampled slots); it is
|
||
not a record marker. Over 33,091 slots at that position only 4,127 hold 0x48.
|
||
- ~~"Byte +3 is constant at 0x48 across 300 records."~~ True for the first 382
|
||
records and false beyond, which is how the false pattern was found.
|
||
- ~~"Coordinates are fixed-point s16 with a scale of 8."~~ The scale-8 reading
|
||
is supported by one coincidence (17,496 / 8 = 2,187, the planet radius) but no
|
||
grouping of the record into triples yields points on a sphere of that radius.
|
||
Tested: every stride from 2 to 16 bytes, every starting offset, three and four
|
||
components. Best result 5.0% of points on-sphere, which is chance.
|
||
|
||
So the record framing is **not recovered**. What is established is the region
|
||
layout and that terrain is vertex-soup packed alongside inline texels — which
|
||
means the vertex format must be identified from a longer run than a single
|
||
record, most likely by locating a run of vertices whose magnitudes are bounded
|
||
by the planet radius and whose Y values vary smoothly.
|
||
|
||
## Where the class tables are not
|
||
|
||
The header's `moby_classes` and friends do not decode. Read as
|
||
`(offset, count)` they give plausible numbers, but the entries at those offsets
|
||
are not valid records. Ruled out:
|
||
|
||
- **Field order.** Both `(offset, count)` and `(count, offset)` produce garbage.
|
||
- **Stride.** Tried 8, 16 and 32 bytes per entry.
|
||
- **Which file the offsets index.** Tried `core_data` and `core_index`.
|
||
- **Header base.** Not the issue: the compressed size (0x88) and decompressed
|
||
size (0x8c) of `core_data` appear at the *same offsets* in all 27 levels'
|
||
`core_index`, which pins the header base at 0 with certainty.
|
||
|
||
One observation worth keeping: every second value in the class-table fields is
|
||
exactly divisible by 16, and they increase monotonically across the nine tables.
|
||
That is consistent with `count * 16` sizes, but the corresponding offsets
|
||
cluster at the very start of the file and do not yield valid records either.
|
||
|
||
The discriminator used to reject candidates is worth stating, because weaker
|
||
tests produce false positives: a real class table's `offset_in_asset_wad`
|
||
values increase. A search for "32-byte records with increasing offsets" returns
|
||
several hundred candidates across the file, and the top one turned out to be a
|
||
texture table whose `o_class` field merely happened to equal `offset + 16`.
|
||
Scoring on plausibility alone is not enough; the entries have to actually chain.
|
||
|
||
Next: rather than guess the header further, identify `tfrag` by its own
|
||
structure. Terrain is the largest subsystem and the one that decides whether
|
||
anything draws, so it is the right next target even though it is the harder one.
|
||
|
||
## `data`: the real target, untouched
|
||
|
||
6,484 sectors on level 0, down to 2,102 on level 24 — 4.2 to 21.9 MB. This is
|
||
the level's geometry and the largest thing in the container. Next target.
|
||
|
||
What is known from the community reference and from Insomniac's engine family:
|
||
it splits into four geometry subsystems — `tfrag` (terrain fragments),
|
||
`tie` (instanced environment structures), `shrub` (foliage) and `moby` (dynamic
|
||
actors) — plus sky shells, collision data and textures. ReRAC's documentation of
|
||
the equivalent structures in the first game is the closest reference available,
|
||
though R&C2's spherical worlds change how they are used.
|
||
|
||
## Order of work
|
||
|
||
1. `gameplay` — smallest, fully bounded, and the format is identified enough to
|
||
make progress. A win here validates the approach before the large effort.
|
||
2. `occlusion` — tiny.
|
||
3. `sound_bank` — small.
|
||
4. `data` — the large one, and the one that decides whether anything renders.
|
||
|
||
Doing the small extents first is not just convenience: each one is a contained
|
||
reverse-engineering problem that either resolves or tells us something, whereas
|
||
`data` will keep producing ambiguous readings until we have something to check
|
||
them against.
|
||
|
||
## Method note
|
||
|
||
Every format question this phase has been settled the same way: write the
|
||
hypothesis, test it against **all 27 levels at once**, and require an invariant
|
||
that holds everywhere. The sector/byte ambiguity in Phase 1 survived because it
|
||
happened to be correct on the one container checked by hand. The level header
|
||
was fixed in a single pass because the tiling check failed immediately on
|
||
levels 1, 2, 4, 7, 8, 11, 19 and 20. |