From 4c4a78d402e80ebd081472a38229610524703520 Mon Sep 17 00:00:00 2001 From: ReaperOfVeriod Date: Sun, 11 Oct 2026 02:03:31 +0200 Subject: [PATCH] Identify the container the class tables point into: core_data itself assets_compressed_size and assets_decompressed_size at index offsets 0x88 and 0x8c are 9,738,678 and 16,969,296 on level 0 -- exactly the sizes of core_data before and after decompression. There is no separate asset container. The "asset WAD" is core_data, and offset_in_asset_wad indexes the decompressed form directly. Checked rather than assumed, on all 27 levels: the decompressed size equals core_data's exact length, every class offset is either 0 or inside it, and the moby, tie and shrub spans do not overlap each other. An offset of 0 means the slot has no geometry, not the start of the lump; level 0's ninth moby class is 4909 at offset 0. Two assumptions worth discarding while here -- the tables are not sorted by class number (level 0's run 0, 1, 2, 14 ... 4909) and the offsets within a table are not monotonic. Neither is needed, but both look true from the first few entries. The chunk descriptors after the named extents are not it: they exist on only 3 of 27 levels, because level 0's four named extents tile the container exactly. An asset blob there would be missing almost everywhere. What sits at the offsets is raw GS/VIF packet data rather than a further container -- moby[0] begins 00 00 00 00 3f 16 02 55 6f 6f and tie[0] 00 01 00 00 70 50 00 00, which read as packet headers rather than a magic number. None of the three is decoded yet and this does not start that. Moby is skeletal and is the largest single piece left before the rest of a level can draw. Also: the pipeline now writes each level's chunk descriptors where they exist, and LevelCoreHeader decodes the asset lump's two sizes. --- crates/gc-extract/data/gc-files.tsv | 24 +++++++ crates/gc-extract/src/unpack.rs | 20 ++++++ crates/gc-extract/tests/disc.rs | 107 ++++++++++++++++++++++++++++ crates/gc-formats/src/level_core.rs | 24 ++++++- docs/plan/level-assets.md | 39 +++++++++- docs/roadmap.md | 5 +- 6 files changed, 213 insertions(+), 6 deletions(-) diff --git a/crates/gc-extract/data/gc-files.tsv b/crates/gc-extract/data/gc-files.tsv index 76354bd..03566f6 100644 --- a/crates/gc-extract/data/gc-files.tsv +++ b/crates/gc-extract/data/gc-files.tsv @@ -2496,6 +2496,9 @@ e1baca5f8178da95e7dc496305a36f984be542aa 15 hud/0780.bin acef5e387c8051229eecf70a64fd6daa077b34b3 518656 level0/gs_ram.bin d7045d6c1256a9ffa241acca76e3761c34da3341 2751268 level0/overlay.bin 7ec9566aeae513fe272bc97899a4dbe15a2a74ff 1503232 level0/sound_bank.bin +6d98a8d7b65d6c743c23c27863c9c7dbf3fc4493 3110912 level1/chunk0.bin +e01bcf80d072da3298d854fb095f13cd6834fca6 237568 level1/chunk1.bin +62b48a3295f5d0cb8094a63ec77c2f514b4c5077 321536 level1/chunk3.bin 6b1a358c6eb6039d0df51fce061b58c274e8c99c 19023392 level1/core_data.bin 96b33cef1ac3c9bab5283ccf11bd86b0aa20ebcb 12165629 level1/core_data.wad 7cae5c7c2258f391ef72a8503183e36747214fbd 34016 level1/core_index.bin @@ -2512,6 +2515,9 @@ fe9e5169820bc9272aa5fcfbbabfc74e7b12d1ce 12498944 level10/data.bin 4036bc105fa429b97a725ad40bd4051168c886ad 367872 level10/gs_ram.bin 76fa348a8764cbe97c8caf5a892105007c2bf4a2 2870708 level10/overlay.bin d3e48237b03649578f6962e36f1bc197c3a8af9f 1306624 level10/sound_bank.bin +74ef643e332327bc142155c5570be38ecfe18d10 1988608 level11/chunk0.bin +62eab219f89f2c1374e29dd302e8f86f69882e6e 1521664 level11/chunk1.bin +ab08381b884f6585818780791a27fd8f44366e76 704512 level11/chunk3.bin da253d8b4dc4f8fc482880c54cd832c53bfd4aa0 20717584 level11/core_data.bin a0173c7577180d32e9bfed67464eecf3bc12c033 11414905 level11/core_data.wad 0e0d5289451a3bffa376992d0b6a288421a7a55d 36000 level11/core_index.bin @@ -2577,6 +2583,11 @@ d0f931677cb6589a564937c3d15c13344b923969 15501312 level18/data.bin 7f6a823c4b47fdfb58e29237b680d03e946329db 571136 level18/gs_ram.bin e5370c97b9a2ccdb00678e57f9e18042eab6735d 2937576 level18/overlay.bin b26483cc29ecb8b6d2c71919ff7cd56b09522bc1 1492992 level18/sound_bank.bin +11dbeeb1847ff09621d132adb9ab0a149fc65a5a 3252224 level19/chunk0.bin +0e05525617f49895b26da1521b8d2ceb9f0c52a2 2742272 level19/chunk1.bin +f9110d5cb18a0fbd45a760dc4d926a87db099b7c 1017856 level19/chunk2.bin +c739bdb3ea348b169b63114f2b13149e647070e0 92160 level19/chunk3.bin +77cbf1d1847c0846c13a23983e664f9546feddf8 92160 level19/chunk4.bin 0918d824b72b892a24f46108bbc60f6e875236f6 20233504 level19/core_data.bin b79df8e074db7d3c75bc2b3b9c2a4de61e5dd9cd 11797811 level19/core_data.wad b0ea377a39a616dbd2b15224a9996925c7b28f4d 31536 level19/core_index.bin @@ -2586,6 +2597,9 @@ d5061a551eb354ddf7952c4bfd62767c6e074c3f 497920 level19/gs_ram.bin 877ca7fb8e80d810d6c37fb993795aa10b5b1033 47104 level19/occlusion.bin 428ea51a157b35cdf831f2a6908647d8ffe6a91b 2821856 level19/overlay.bin 8298f799c394542e912e41ff3321ee55c70e0c8f 1570816 level19/sound_bank.bin +e6c6de0fafb808093d6e3f2282e86b3fc623d7cb 2486272 level2/chunk0.bin +84bb6b19112861026a24d84cbf83946e2bf31892 321536 level2/chunk1.bin +6cb42e4727ce5a55cf6e5a9147a32149a2d15891 438272 level2/chunk3.bin 6d56189ad74f04d366116b736ab12888154de833 20209408 level2/core_data.bin f0eb5c033e84092748cf3457818b0d4c85da7a0a 11372886 level2/core_data.wad 73411362c1cfa8faec733705d93e953f1381ef30 33808 level2/core_index.bin @@ -2595,6 +2609,11 @@ f0eb5c033e84092748cf3457818b0d4c85da7a0a 11372886 level2/core_data.wad 01c7d4c7b0ecb44a91694ffde796ff941057191a 28672 level2/occlusion.bin d7182afdf4a986c9c781a9658373281bfc37df22 2928424 level2/overlay.bin bce56b3b899d72c5d89f612bd70447f066c10757 1118208 level2/sound_bank.bin +015dfaf782ade5bdd3815a66b4c381a93ec7cc6d 1462272 level20/chunk0.bin +93481eb3afd7edfe679c878a74d5ec627196cf86 1878016 level20/chunk1.bin +1867fb80ce6d3b592c687e712e9782cd5a43dd6f 565248 level20/chunk2.bin +b81d361a7c7136d92990d720f6f22fbbee9b0d89 342016 level20/chunk3.bin +1baaf377e209e353b2fbb80d408bbc52a67f6b5f 342016 level20/chunk4.bin daf162b58756e2661f3ec891c8736a1611b4dc52 19637472 level20/core_data.bin 7e7203ac3a2dbf578546573a34fa6dce4c336d35 10449039 level20/core_data.wad b12e8599d211addc85cbe138e58157ade380a575 37360 level20/core_index.bin @@ -2658,6 +2677,9 @@ eb8366046b5643cc54d42d800c06bf66ccd4ef36 15157248 level3/data.bin ed1f77cd35e2572adc51f4f43de030f31b8c6ed9 675840 level3/gs_ram.bin 6fb6a7e7a64afd2f1e897f23640866f28154aa1d 2820776 level3/overlay.bin d986825dfc2bed713eb9699504536346658aaa3f 1705984 level3/sound_bank.bin +2778532b95fe6a545004f5a922977cac656519e7 3436544 level4/chunk0.bin +59aead3118f5be41ab3ab545e2476e7fed6842f1 2426880 level4/chunk1.bin +482ac30a9958e484708f9b82e2784a62a0cf753e 303104 level4/chunk3.bin f8c0aa3778b5bcb1995e516045b28ca6ac35502f 21107088 level4/core_data.bin ec75f324670188af581c4fceaa4cb285353f95b8 12171908 level4/core_data.wad cc4997acbdc0a6ac85d427227c04e6af6e34cd46 38080 level4/core_index.bin @@ -2683,6 +2705,7 @@ d5874f71309fb750d37a08977653a4fe1b48672a 2617984 level6/gameplay.bin c404dfa08cab9603e5deeeed9778e4d29b755d88 604160 level6/gs_ram.bin 4d981fd750196894f96f0a24437bc76210a1663a 3086372 level6/overlay.bin 82fcce964396eb508768933d092a557bc1b05c89 1669120 level6/sound_bank.bin +ef8e0a3a46dcb8f29595332ff9545b654130b194 3280896 level7/chunk0.bin 6692290d78706a2b92cba541264ddea9a764e2f2 18209392 level7/core_data.bin f1cae801f1050c57258f93a7035148b3094ece03 10780010 level7/core_data.wad 79e3957d149ce3b00f713eb1757fec89d7db86c4 32736 level7/core_index.bin @@ -2692,6 +2715,7 @@ cc78412e868725caa8e9736653749e4b7eeacba4 2147328 level7/gameplay.bin e02d1b0d2339d72ed5d07abfe105f41e2288f59f 16384 level7/occlusion.bin 82f26617c99f502324fb556006f8f859fda1b5f1 2733840 level7/overlay.bin 8e07c803b709100023c3b44c8842eec9b8f49898 1556480 level7/sound_bank.bin +78e0c64ccb11e29c9bdbfb9c9283a754867d6b5c 868352 level8/chunk0.bin 264a306c3a83b9137d9753b282e9c773cd386713 21086224 level8/core_data.bin 0a96094edd63dc4dd940894b007f85b21874cfeb 9700660 level8/core_data.wad 8d1b6c3a05e9b2140c056df3e538ed537b9d3a33 30624 level8/core_index.bin diff --git a/crates/gc-extract/src/unpack.rs b/crates/gc-extract/src/unpack.rs index 080035e..c6ffc1e 100644 --- a/crates/gc-extract/src/unpack.rs +++ b/crates/gc-extract/src/unpack.rs @@ -137,6 +137,26 @@ pub fn unpack(iso_path: &Path, out_dir: &Path) -> Result, String> } parts += 1; } + // Chunk descriptors follow the named extents. One of them + // is the level's asset WAD, which the moby/tie/shrub class + // tables point into by byte offset. + for (i, range) in header.present_chunks() { + let Some(offset) = range.byte_offset() else { + continue; + }; + let len = range.byte_len() as usize; + if offset + len as u64 > raw.len() as u64 { + continue; + } + write_file( + out_dir, + &format!("{stem}/chunk{i}.bin"), + &raw[offset as usize..offset as usize + len], + &mut written, + ); + parts += 1; + } + // The `data` extent is itself a container of byte ranges. // Split out the geometry directory and the geometry, so the // latter is usable without re-deriving the layout later. diff --git a/crates/gc-extract/tests/disc.rs b/crates/gc-extract/tests/disc.rs index c998065..3f27af6 100644 --- a/crates/gc-extract/tests/disc.rs +++ b/crates/gc-extract/tests/disc.rs @@ -1092,3 +1092,110 @@ fn level_core_class_tables_tile_the_index() { {textures} tfrag textures across {levels} levels" ); } + +/// The moby, tie and shrub class tables point into `core_data`, and they do so +/// without overlapping. +/// +/// `assets_decompressed_size` is the size of `core_data` itself, so there is +/// no separate asset container: the class entries index the decompressed asset +/// lump directly. If that is right, every offset must land inside the lump, the +/// three tables must occupy disjoint spans, and each must be internally sorted +/// so the geometry is laid out in class order. +#[test] +fn class_geometry_offsets_tile_core_data_without_overlap() { + require_disc!(); + gc_formats::iso9660::IsoImage::open(std::path::Path::new( + &std::env::var("GC_DISC").expect("GC_DISC is set by require_disc!"), + )) + .expect("ISO should open"); + let data = std::env::var_os("GC_EXTRACTED") + .map(PathBuf::from) + .unwrap_or_else(|| { + let home = std::env::var("HOME").unwrap_or_else(|_| ".".to_string()); + PathBuf::from(home).join("Documents/gc-extracted") + }); + if !data.exists() { + eprintln!("skipped: unpack the disc first"); + return; + } + + use gc_formats::level_core as lc; + let mut levels = 0usize; + + for n in 0..27 { + let dir = data.join(format!("level{n}")); + let (Ok(index), Ok(core)) = ( + std::fs::read(dir.join("core_index.bin")), + std::fs::read(dir.join("core_data.bin")), + ) else { + continue; + }; + let h = lc::LevelCoreHeader::parse(&index).expect("core header"); + + // The claim under test: the "asset lump" is core_data itself. + assert_eq!( + h.assets_decompressed_size as usize, + core.len(), + "level{n}: assets_decompressed_size is not core_data's length" + ); + + let span = |r: lc::ArrayRange, size: usize| -> Vec { + lc::class_table(&index, r, size, lc::ClassEntry::parse) + .unwrap_or_default() + .iter() + .map(|c| c.offset_in_asset_wad as usize) + .collect() + }; + // Shrub entries are read as ShrubClassEntry so the offsets line up. + let shrub: Vec = lc::class_table( + &index, + h.shrub_classes, + lc::SHRUB_CLASS_SIZE, + lc::ShrubClassEntry::parse, + ) + .unwrap_or_default() + .iter() + .map(|s| s.base.offset_in_asset_wad as usize) + .collect(); + let moby = span(h.moby_classes, lc::MOBY_CLASS_SIZE); + let tie = span(h.tie_classes, lc::TIE_CLASS_SIZE); + + // An offset of 0 marks a slot with no geometry rather than the start of + // the lump, so those are excluded from the span checks. Entries are not + // sorted by class number and the offsets are not monotonic, so nothing + // here assumes an order. + let moby_l: Vec = moby.iter().copied().filter(|&o| o != 0).collect(); + let tie_l: Vec = tie.iter().copied().filter(|&o| o != 0).collect(); + let shrub_l: Vec = shrub.iter().copied().filter(|&o| o != 0).collect(); + + for (name, v) in [("moby", &moby), ("tie", &tie), ("shrub", &shrub)] { + for &o in v { + assert!( + o < core.len(), + "level{n}: {name} offset {o} past core data ({})", + core.len() + ); + } + } + // The three spans must not overlap each other. + for (an, a) in [("moby", &moby_l), ("tie", &tie_l), ("shrub", &shrub_l)] { + for (bn, b) in [("moby", &moby_l), ("tie", &tie_l), ("shrub", &shrub_l)] { + // A level may have no geometry of one kind at all. + if an >= bn || a.is_empty() || b.is_empty() { + continue; + } + let (lo, hi) = (a[0], *a.last().unwrap()); + for &o in b { + assert!( + o >= hi || o < lo, + "level{n}: {an} [{lo},{hi}] overlaps {bn} at {o}" + ); + } + } + } + levels += 1; + } + + assert_eq!(levels, 27, "expected all 27 levels, got {levels}"); + eprintln!("class geometry offsets verified inside core_data on {levels} levels"); +} diff --git a/crates/gc-formats/src/level_core.rs b/crates/gc-formats/src/level_core.rs index d89f363..e0417fd 100644 --- a/crates/gc-formats/src/level_core.rs +++ b/crates/gc-formats/src/level_core.rs @@ -237,6 +237,14 @@ pub struct LevelCoreHeader { pub shrub_textures: ArrayRange, /// Byte offset of texture pixel data within `core_data`. pub textures_base: i32, + /// Compressed size of `core_data`'s WAD stream, and its decompressed size. + /// + /// These describe the whole asset lump, not a separate container: on level + /// 0 they are 9,738,678 and 16,969,296, exactly the sizes of `core_data` + /// before and after decompression. The moby, tie and shrub class tables + /// name geometry by byte offset into the *decompressed* form. + pub assets_compressed_size: i32, + pub assets_decompressed_size: i32, } impl LevelCoreHeader { @@ -262,6 +270,13 @@ impl LevelCoreHeader { // 0x50..0x60: particle and effect texture tables, then the banks. b.skip(0x60 - 0x50, "particle texture tables")?; let textures_base = b.i32("textures base")?; + // 0x60..0x80: particle/effect banks and defs, sound remap, then the + // ratchet sequence table and the scene view size. + b.skip(0x80 - 0x64, "particle banks")?; + // 0x80: gadget table, then the asset lump's two sizes. + b.skip(0x88 - 0x80, "gadget table")?; + let assets_compressed_size = b.i32("assets compressed size")?; + let assets_decompressed_size = b.i32("assets decompressed size")?; Ok(LevelCoreHeader { gs_ram, tfrags, @@ -276,6 +291,8 @@ impl LevelCoreHeader { tie_textures, shrub_textures, textures_base, + assets_compressed_size, + assets_decompressed_size, }) } } @@ -286,7 +303,7 @@ mod tests { /// A header as measured on level 0. fn header() -> Vec { - let mut v = vec![0u8; 0x64]; + let mut v = vec![0u8; 0x90]; v[0x00..0x04].copy_from_slice(&404i32.to_le_bytes()); v[0x04..0x08].copy_from_slice(&18_704i32.to_le_bytes()); v[0x08..0x0c].copy_from_slice(&0i32.to_le_bytes()); @@ -296,6 +313,8 @@ mod tests { v[0x30..0x34].copy_from_slice(&93i32.to_le_bytes()); v[0x34..0x38].copy_from_slice(&10_512i32.to_le_bytes()); v[0x60..0x64].copy_from_slice(&3_518_848i32.to_le_bytes()); + v[0x88..0x8c].copy_from_slice(&9_738_678i32.to_le_bytes()); + v[0x8c..0x90].copy_from_slice(&16_969_296i32.to_le_bytes()); v } @@ -321,11 +340,14 @@ mod tests { } ); assert_eq!(h.textures_base, 3_518_848); + assert_eq!(h.assets_compressed_size, 9_738_678); + assert_eq!(h.assets_decompressed_size, 16_969_296); } #[test] fn a_short_header_is_an_error() { assert!(LevelCoreHeader::parse(&[0u8; 0x20]).is_err()); + assert!(LevelCoreHeader::parse(&[0u8; 0x8c]).is_err()); } #[test] diff --git a/docs/plan/level-assets.md b/docs/plan/level-assets.md index bb79362..5370137 100644 --- a/docs/plan/level-assets.md +++ b/docs/plan/level-assets.md @@ -562,6 +562,37 @@ tfrag record framing was "unrecoverable" because the entry size was never known. Both looked like properties of the format and were properties of the question. +### Where the class tables point + +`assets_compressed_size` and `assets_decompressed_size` at index offsets 0x88 +and 0x8c are 9,738,678 and 16,969,296 on level 0 -- exactly the sizes of +`core_data` before and after decompression. There is no separate asset +container: the "asset WAD" *is* `core_data`, and `offset_in_asset_wad` indexes +the decompressed form directly. + +That is checkable rather than merely plausible, and a test checks it on all 27 +levels: + +* `assets_decompressed_size` equals `core_data`'s exact length; +* every class offset is either 0 or inside `core_data`; +* the moby, tie and shrub spans do not overlap each other. + +An offset of **0 means the slot has no geometry**, not the start of the lump. +On level 0 the moby table's ninth entry is class 4909 at offset 0. + +Two assumptions worth discarding: the tables are **not** sorted by class number +(level 0's moby classes run 0, 1, 2, 14, ... 4909), and the offsets within a +table are **not** monotonic. Neither is needed, but both look true from the +first few entries. + +The chunk descriptors after the named extents are not it. They exist on only +3 of 27 levels -- level 0's four named extents tile the container exactly -- +so an asset blob there would be missing almost everywhere. + +At the offsets sit raw GS/VIF packet data, not a further container: moby[0] +begins `00 00 00 00 3f 16 02 55 6f 6f`, tie[0] `00 01 00 00 70 50 00 00`, +which read as packet headers rather than a magic number. + ### Why the frame is still dark: the light bank does not parse on RAC2 The reference frames the bank as "a u32 offset at +0x04, then an `i32` count @@ -659,9 +690,11 @@ the fidelity rule. - **Lighting magnitudes.** The pass runs and the output is plausible; whether it matches the game needs comparison against reference frames, which needs the PCSX2 loop that is still to come. Nothing in this file establishes it. -- **tie, moby and shrub geometry.** The class tables now parse (below), but - nothing reads the geometry they name yet. `offset_in_asset_wad` points into - the level's asset WAD, which is a separate container. +- **tie, moby and shrub geometry.** The class tables parse and the container + they point into is identified (below), but none of the three is decoded yet. + Moby is the largest of the three and is skeletal, with an animation evaluator + in the microcode; it is the single biggest piece left before the rest of a + level can draw. - **Point lights** are spawned by gameplay at runtime and are absent from the disc -- every one of the 18,102 patches has `point_lights == 0xffff` -- so there is nothing for them to light until the game runs. diff --git a/docs/roadmap.md b/docs/roadmap.md index da1a056..bdd4b91 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -14,8 +14,9 @@ early, while they are still cheap to retire. - [x] Repository hygiene in place (pre-commit guard, `.gitignore`, licence record) - [x] **Phase 1 complete.** The disc pipeline runs end to end and is verified - against a committed hash table: 5,213 files, 5,795,847,447 bytes, every - one of the 92 asset containers parsing. + against a committed hash table, every one of the 92 asset containers + parsing. The pipeline now also writes each level's chunk descriptors + where they exist (3 of 27 levels). - [x] **Phase 2 in progress.** Level containers are parsed through their typed header, and the `data` extent is parsed as a container of byte ranges. The geometry decompresses: level 0 yields 16,969,296 bytes of level content