Fix level extents read as bytes, and extract the level geometry
A defect in the Phase 1 extractor, found by reading the data extent's contents: level extents are declared in sector counts, the generic G/ pair table uses byte counts, and level containers were being routed through the generic path. So LEVEL0.WAD's data extent -- 6,484 sectors -- was taken as 6,484 bytes. Every test still passed, because a truncated file is still a file. The extent's own header gave it away: it declares an uncompressed total of 13,277,920 bytes, and 13,277,920 / 6,484 is 2047.7. A ratio landing on 2048 is a unit error, not a compression ratio. Across 27 levels this had been discarding roughly 300 MB of level content. Level containers now go through the typed LevelHeader instead. Level 0 yields: data.bin 13,279,232 the data extent core_data.wad 9,738,678 the geometry, compressed core_data.bin 16,969,296 the geometry, decompressed core_index.bin 31,600 the geometry directory gameplay.bin 659,456 sound_bank.bin 1,503,232 core_data is a WAD LZ77 stream -- the format already implemented for the game -- and it decompresses cleanly. The first real level content this project has recovered. New: LevelDataHeader parses the data extent's own header, a table of byte ranges naming the level overlay executable, the geometry directory, the GS RAM image, the HUD banks and the geometry. Note this is a third meaning for a "size" field on the same disc, after sector counts in the level header and byte counts in the generic table; all three are documented together in level-data.md so the next parser does not rediscover this the hard way. The geometry directory's offsets were checked against the geometry rather than taken on trust: all twelve land inside the 16.9 MB of decompressed core_data, which is what confirms the header is being read correctly. The class tables themselves do not yet decode under the documented layout, and that is recorded as open work in plan/level-assets.md rather than papered over. 75 unit tests, 9 integration tests against the real disc. Hash table regenerated at 5,159 files and 5,671,915,799 bytes -- hashes only, no content. Extraction now writes to ~/Documents/gc-extracted rather than /tmp, which is 16 GB and did not have room for a 5.3 GB output.
This commit is contained in:
@@ -0,0 +1,108 @@
|
||||
# The level data extent
|
||||
|
||||
Target: `SCUS_972.68` v1.01. Implementation: `crates/gc-formats/src/level_data.rs`.
|
||||
|
||||
The `data` extent of each `LEVEL<n>.WAD` is the level's own container. It holds
|
||||
the level's executable, the geometry directory, the Graphics Synthesizer RAM
|
||||
image, the HUD banks and the geometry itself, each at a byte offset declared by
|
||||
a table at the start of the extent.
|
||||
|
||||
## Header
|
||||
|
||||
```text
|
||||
offset size field
|
||||
0x00 8 overlay the level's executable
|
||||
0x08 8 core_index the geometry directory
|
||||
0x10 8 gs_ram Graphics Synthesizer RAM image
|
||||
0x18 8 hud_header
|
||||
0x20 40 hud_banks[5]
|
||||
0x48 8 core_data the compressed geometry
|
||||
0x50 8 transition_textures
|
||||
```
|
||||
|
||||
Every field is `{offset, size}` in **bytes** relative to the start of the
|
||||
extent. The table is 0x58 bytes and the first extent starts at 0x80, leaving 40
|
||||
bytes unused.
|
||||
|
||||
Level 0, measured:
|
||||
|
||||
| Extent | Offset | Size |
|
||||
|---|---:|---:|
|
||||
| overlay | 128 | 2,751,268 |
|
||||
| core_index | 2,751,424 | 31,600 |
|
||||
| gs_ram | 2,783,040 | 518,656 |
|
||||
| hud_header | 3,301,696 | 4,060 |
|
||||
| hud_banks[0..4] | 3,305,792 … | 44,935 / 138,675 / 16 / 16 / 34,286 |
|
||||
| core_data | 3,523,904 | 9,738,678 |
|
||||
| transition_textures | 13,262,592 | 15,328 |
|
||||
|
||||
Extents are ordered and non-overlapping but not gap-free: 28 bytes separate the
|
||||
end of the overlay from the start of `core_index`. The test asserts ordering
|
||||
rather than tiling for that reason.
|
||||
|
||||
## A third convention for sizes
|
||||
|
||||
Three different meanings for a "size" field appear on this disc. All three are
|
||||
byte offsets; the size unit differs:
|
||||
|
||||
| Where | Size unit |
|
||||
|---|---|
|
||||
| `LEVEL<n>.WAD` named extents | **sectors** |
|
||||
| `G/*.WAD` generic pair table | **bytes** |
|
||||
| `data` extent header | **bytes** |
|
||||
|
||||
The first of those was a real bug: the extractor read level extents through the
|
||||
generic path, taking a 6,484-sector extent as 6,484 bytes and losing 13 MB of
|
||||
every level. See `../plan/level-assets.md`.
|
||||
|
||||
## `core_data`: the geometry
|
||||
|
||||
The largest extent, and a WAD LZ77 stream — the format in
|
||||
[`wad.md`](wad.md), with a 16-byte header whose declared size equals the whole
|
||||
extent, i.e. the stream spans it exactly.
|
||||
|
||||
Level 0 decompresses from 9,738,678 to **16,969,296 bytes**, a 1.74× ratio. It
|
||||
contains the four geometry subsystems (`tfrag` terrain, `tie` environment
|
||||
instances, `shrub` foliage, `moby` actors), sky shells, collision and
|
||||
particles.
|
||||
|
||||
## `core_index`: the geometry directory
|
||||
|
||||
A 31,600-byte structure at the head of which sits a header of offsets into
|
||||
`core_data`. On level 0:
|
||||
|
||||
| Field | Value | Lands inside core_data |
|
||||
|---|---:|---|
|
||||
| gs_ram | 404, count 18,704 | yes |
|
||||
| occlusion | 2,117,888 | yes |
|
||||
| sky | 2,192,576 | yes |
|
||||
| collision | 2,615,232 | yes |
|
||||
| moby_classes | 238 | yes |
|
||||
| tie_classes | 74 | yes |
|
||||
| shrub_classes | 7 | yes |
|
||||
| tfrag_textures | 93 | yes |
|
||||
| moby_textures | 168 | yes |
|
||||
| textures_base | 3,518,848 | yes |
|
||||
| part_bank | 8,141,696 | yes |
|
||||
| fx_bank | 8,332,160 | yes |
|
||||
|
||||
Every offset falls inside the 16.9 MB `core_data`, which is the evidence that
|
||||
the header is being read correctly.
|
||||
|
||||
`sun` and `collision` at their offsets begin with plausible structure — the
|
||||
collision block opens `40 00 00 00 50 c8 0d 00`, a plausible count followed by a
|
||||
size.
|
||||
|
||||
**Not yet resolved:** the `ArrayRange` fields (`moby_classes` and friends) do
|
||||
not decode to a usable class table under the documented layout. Offsets and
|
||||
counts read as plausible numbers but the entries they point at are not valid
|
||||
`{offset, o_class}` records. The GC core header layout differs from the
|
||||
documented R&C1 one in a way not yet determined. Recorded in
|
||||
`../plan/level-assets.md`; this is the next thing to resolve.
|
||||
|
||||
## Provenance
|
||||
|
||||
The structure matches `GcUyaLevelDataHeader` in the Wrench project
|
||||
(<https://github.com/chaoticgd/wrench>, GPLv3). **No Wrench code is copied** —
|
||||
see `PROVENANCE.md`. Everything here is written from the specification and
|
||||
verified against all 27 levels on this disc.
|
||||
@@ -3,6 +3,39 @@
|
||||
Phase 2 work record. What has been established about the four extents inside
|
||||
each `LEVEL<n>.WAD`, and what remains.
|
||||
|
||||
## A bug found and fixed
|
||||
|
||||
Reading the `data` extent's contents exposed a defect in the Phase 1 extractor.
|
||||
|
||||
Level extents are declared in **sector counts**; the generic `G/` pair table
|
||||
uses **byte counts**. The extractor routed level containers through the generic
|
||||
path, so `LEVEL0.WAD`'s `data` extent — 6,484 sectors — was taken as 6,484
|
||||
*bytes*. Level 0 yielded a 6,484-byte "geometry" file instead of 13,279,232
|
||||
bytes. Across 27 levels that silently discarded roughly 300 MB of level content
|
||||
while every test still passed.
|
||||
|
||||
The tell was arithmetic, not a crash: the extent's own header declared an
|
||||
uncompressed total of 13,277,920 bytes, and the ratio against the 6,484-byte
|
||||
file was 2047.7× — one sector, to within rounding. A ratio that lands on 2048
|
||||
is a unit error.
|
||||
|
||||
Fixed by extracting level containers through the typed `LevelHeader` instead of
|
||||
the generic table. Level 0 now yields:
|
||||
|
||||
```text
|
||||
data.bin 13,279,232 the level container's data extent
|
||||
core_data.wad 9,738,678 the geometry, compressed
|
||||
core_data.bin 16,969,296 the geometry, decompressed
|
||||
core_index.bin 31,600 the geometry directory
|
||||
gameplay.bin 659,456
|
||||
sound_bank.bin 1,503,232
|
||||
```
|
||||
|
||||
**Lesson:** a unit error is invisible to a type system and to a passing test. The
|
||||
check that catches it is arithmetic against an independent number the file
|
||||
itself declares. Any extent whose contents describe a larger structure should
|
||||
have that structure's own total compared against the extent.
|
||||
|
||||
## Done: the container header
|
||||
|
||||
`crates/gc-formats/src/level_wad.rs` parses the 0x60-byte header on all 27
|
||||
|
||||
+10
-4
@@ -16,10 +16,11 @@ early, while they are still cheap to retire.
|
||||
- [x] **Phase 1 complete.** The disc pipeline runs end to end and is verified
|
||||
against a committed hash table: 5,135 files, 4,500,106,960 bytes, every
|
||||
one of the 92 asset containers parsing.
|
||||
- [x] **Phase 2 started.** The level container header is parsed and typed, which
|
||||
names the `data`, `sound_bank`, `gameplay` and `occlusion` extents inside
|
||||
each of the 27 level containers.
|
||||
- [ ] Phase 2 next: the `gameplay` extent, then `data`.
|
||||
- [x] **Phase 2 in progress.** Level containers are parsed through their typed
|
||||
header, and the `data` extent is parsed as a container of byte ranges. The
|
||||
geometry decompresses: level 0 yields 16,969,296 bytes of level content
|
||||
from a 9.7 MB stream.
|
||||
- [ ] Next: the geometry directory's class tables, then `tfrag` terrain.
|
||||
|
||||
The disc being the exact build the reference projects pin is the single
|
||||
biggest favourable fact so far: no disc conversion, no regional timing work,
|
||||
@@ -101,6 +102,11 @@ Phase 1 hands over the extents; this phase works out what each one contains.
|
||||
|
||||
**In progress:**
|
||||
|
||||
- [x] `data` — the extent header locates `core_index` and `core_data`; the
|
||||
geometry decompresses and is verified on level 0.
|
||||
- [ ] `core_index` class tables — offsets all land inside `core_data`, which
|
||||
confirms the header, but the `ArrayRange` fields do not decode to a usable
|
||||
class table under the documented layout. Next thing to resolve.
|
||||
- [ ] `gameplay` — begins `WAD` + a per-level byte, and does *not* decode with
|
||||
the WAD LZ77 decoder. All 27 levels fail identically, so the input is
|
||||
wrong rather than the decoder. Ruled out: bit order, header length, the
|
||||
|
||||
Reference in New Issue
Block a user