Fix level extents read as bytes, and extract the level geometry

A defect in the Phase 1 extractor, found by reading the data extent's contents: level extents are declared in sector counts, the generic G/ pair table uses byte counts, and level containers were being routed through the generic path. So LEVEL0.WAD's data extent -- 6,484 sectors -- was taken as 6,484 bytes. Every test still passed, because a truncated file is still a file. The extent's own header gave it away: it declares an uncompressed total of 13,277,920 bytes, and 13,277,920 / 6,484 is 2047.7. A ratio landing on 2048 is a unit error, not a compression ratio. Across 27 levels this had been discarding roughly 300 MB of level content. Level containers now go through the typed LevelHeader instead. Level 0 yields: data.bin 13,279,232 the data extent core_data.wad 9,738,678 the geometry, compressed core_data.bin 16,969,296 the geometry, decompressed core_index.bin 31,600 the geometry directory gameplay.bin 659,456 sound_bank.bin 1,503,232 core_data is a WAD LZ77 stream -- the format already implemented for the game -- and it decompresses cleanly. The first real level content this project has recovered. New: LevelDataHeader parses the data extent's own header, a table of byte ranges naming the level overlay executable, the geometry directory, the GS RAM image, the HUD banks and the geometry. Note this is a third meaning for a "size" field on the same disc, after sector counts in the level header and byte counts in the generic table; all three are documented together in level-data.md so the next parser does not rediscover this the hard way. The geometry directory's offsets were checked against the geometry rather than taken on trust: all twelve land inside the 16.9 MB of decompressed core_data, which is what confirms the header is being read correctly. The class tables themselves do not yet decode under the documented layout, and that is recorded as open work in plan/level-assets.md rather than papered over. 75 unit tests, 9 integration tests against the real disc. Hash table regenerated at 5,159 files and 5,671,915,799 bytes -- hashes only, no content. Extraction now writes to ~/Documents/gc-extracted rather than /tmp, which is 16 GB and did not have room for a 5.3 GB output.
2026-10-09 16:28:26 +02:00
parent a9e43e39c1
commit 16c04db559
9 changed files with 868 additions and 147 deletions
@@ -306,3 +306,82 @@ fn level_extent_sizes_are_sector_counts() {
    // Read as bytes, 734 would be under four sectors and the tiling would break.
    assert_eq!(h.sound_bank.byte_len(), 734 * 2048);
 }
+
+/// The level data extent is a container of byte ranges. Its header locates the
+/// geometry index and the compressed geometry, and every range must lie inside
+/// the extent on all 27 levels.
+#[test]
+fn level_data_extents_are_in_range() {
+    let path = require_disc!();
+    let iso = gc_formats::iso9660::IsoImage::open(&path).expect("ISO should open");
+
+    let mut checked = 0;
+    for entry in iso.files.iter().filter(|e| e.name.starts_with("G/LEVEL")) {
+        let raw =
+            gc_formats::iso9660::read_at(&path, entry.lba, entry.size).expect("read container");
+        let lh = gc_formats::level_wad::LevelHeader::parse(&raw).expect("level header");
+        let data = lh.data;
+        let Some(offset) = data.byte_offset() else {
+            continue;
+        };
+        let len = data.byte_len() as usize;
+        let extent = &raw[offset as usize..offset as usize + len];
+
+        let dh = gc_formats::level_data::LevelDataHeader::parse(extent)
+            .unwrap_or_else(|e| panic!("{}: data extent header: {e}", entry.name));
+
+        for (part, r) in dh.present_parts() {
+            assert!(
+                r.end() <= extent.len() as u64,
+                "{}: {part:?} ends at {} but the extent is {} bytes",
+                entry.name,
+                r.end(),
+                extent.len()
+            );
+        }
+
+        // The geometry is the big one and must be substantial.
+        assert!(
+            dh.core_data.size > 1_000_000,
+            "{}: core_data is only {} bytes",
+            entry.name,
+            dh.core_data.size
+        );
+        checked += 1;
+    }
+    assert_eq!(checked, 27, "expected 27 level containers");
+}
+
+/// The geometry must decompress. This is the first test that touches real level
+/// content rather than headers.
+#[test]
+fn level_geometry_decompresses() {
+    let path = require_disc!();
+    let iso = gc_formats::iso9660::IsoImage::open(&path).expect("ISO should open");
+    let entry = iso.find("G/LEVEL0.WAD").expect("level 0");
+
+    let raw = gc_formats::iso9660::read_at(&path, entry.lba, entry.size).expect("read container");
+    let lh = gc_formats::level_wad::LevelHeader::parse(&raw).expect("level header");
+    let data = lh.data;
+    let extent = &raw[data.byte_offset().unwrap() as usize
+        ..(data.byte_offset().unwrap() + data.byte_len()) as usize];
+
+    let dh = gc_formats::level_data::LevelDataHeader::parse(extent).expect("data header");
+    let core = dh
+        .core_data
+        .slice(extent, extent.len())
+        .expect("core_data slice");
+
+    assert!(
+        gc_formats::wad::is_wad(core),
+        "core_data should start with the WAD magic"
+    );
+
+    let decoded = gc_formats::wad::decompress(core).expect("core_data should decompress");
+    assert!(
+        decoded.len() > core.len(),
+        "decompressed geometry ({}) should exceed the compressed size ({})",
+        decoded.len(),
+        core.len()
+    );
+}