199ce33dd814da936da01e9457b9e37ed1e65fc7
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 ReaperOfVeriod
|
8b98d388f7 |
Show region bounds in probe, which exposed an unnamed 6.2 MB region
Printing the region table next to the header fields makes the gap visible without having to remember to consult core_layout, and it paid for itself immediately. On level 0 the collision region comes out 6216960 bytes, from 2615232 to the first moby class at 8832192. Collision's own header declares a payload of 903,248. So roughly 6.2 MB sits between the collision header and the first moby class, and almost none of it is collision. The successor rule is correct -- a region does end where the next one begins -- but "the next" is not yet known there. collision..moby is really collision + <something unnamed> .. moby. The name is wrong, not the bound. Candidates for the space, rough order of likelihood: moby and tie model animation data, which is what moby_anim would need, then volumes, then per-class runtime structures. Tfrag texture texels are already placed and sit earlier in the lump. Worth resolving before starting moby_anim, since that format is the most likely tenant. Until then, any probe reading between the collision header and the first moby class is reading something the region table does not describe. |
||
|
ReaperOfVeriod
|
f6bbbf8d5f |
Add gc-extract probe to dump structures through the real parser
Reading these formats by unpacking raw offsets out of core_index.bin has cost real time. The index interleaves single-word region offsets with count/offset pairs, so miscounting one shifts every field after it: that produced a page of implausible collision headers on all 27 levels, sent me chasing a tie-breakoff that turned out not to exist, and nearly had me "fix" a stride bug in code that was already right. probe dumps a level's core header and all three class tables through LevelCoreHeader, so a field name printed is the field name the parser uses. It also reports how many entries carry geometry inside core_data, which is what pins the class table strides rather than guessing at them: shrub reads 7/7 sane at 0x30 against 4/7 read at 0x20. Reading a structure by hand in a shell still works; this just stops it being the default, and stops the offset arithmetic being re-derived every time a new region is investigated. |
||
|
ReaperOfVeriod
|
29fdf89f53 |
Decode level textures: 8-bit indices through a CSM1 palette
Adds texture.rs and makes the pipeline write gs_ram.bin, which it had been skipping. Nothing could be decoded before this: the CLUTs live there. A level texture is not pixels, it is palette indices -- width*height bytes at core_data[textures_base + data_offset], with colour coming from a 256-entry CLUT at gs_ram[palette * 0x100]. gs_ram is a byte range in the level's data extent header at offset 0x10, 518,656 bytes on level 0. Two PS2 conventions have to be honoured and neither is a design choice: within each 32-entry CLUT group the two middle 8-entry blocks are swapped (bits 3 and 4 of the index are exchanged, an involution), and alpha runs 0..=0x80 rather than 0..=255 with 0x80 fully opaque. The mip chain splits across two buffers: levels 0 and 1 are in the texture block with level 1 immediately after level 0, and levels 2 and 3 are resident in gs_ram at mipmap*0x100 and pad*0x100. Tfrag textures are square so every level halves evenly. Verification is two-layered, because every step here is a table lookup and a wrong CLUT remap still produces a well-formed image of the wrong thing. All 1,595 tfrag textures across all 27 levels decode; 96.8% are fully opaque and 96.3% use more than eight colours. On top of that, `gc-extract dump-texture` writes any texture or mip level as a PPM. Level 0 texture 0 comes back as a clean bevelled metal panel with rivets and specular highlights, and texture 74 as a vent grille whose punched windows are fully transparent. That tool is worth keeping: it is the cheapest way to tell a decode that is merely well-formed from one that is right. The hash table is regenerated for the 27 new gs_ram.bin files, now 5,186 files and 5,685,541,143 bytes. |
||
|
ReaperOfVeriod
|
16c04db559 |
Fix level extents read as bytes, and extract the level geometry
A defect in the Phase 1 extractor, found by reading the data extent's contents: level extents are declared in sector counts, the generic G/ pair table uses byte counts, and level containers were being routed through the generic path. So LEVEL0.WAD's data extent -- 6,484 sectors -- was taken as 6,484 bytes. Every test still passed, because a truncated file is still a file. The extent's own header gave it away: it declares an uncompressed total of 13,277,920 bytes, and 13,277,920 / 6,484 is 2047.7. A ratio landing on 2048 is a unit error, not a compression ratio. Across 27 levels this had been discarding roughly 300 MB of level content. Level containers now go through the typed LevelHeader instead. Level 0 yields: data.bin 13,279,232 the data extent core_data.wad 9,738,678 the geometry, compressed core_data.bin 16,969,296 the geometry, decompressed core_index.bin 31,600 the geometry directory gameplay.bin 659,456 sound_bank.bin 1,503,232 core_data is a WAD LZ77 stream -- the format already implemented for the game -- and it decompresses cleanly. The first real level content this project has recovered. New: LevelDataHeader parses the data extent's own header, a table of byte ranges naming the level overlay executable, the geometry directory, the GS RAM image, the HUD banks and the geometry. Note this is a third meaning for a "size" field on the same disc, after sector counts in the level header and byte counts in the generic table; all three are documented together in level-data.md so the next parser does not rediscover this the hard way. The geometry directory's offsets were checked against the geometry rather than taken on trust: all twelve land inside the 16.9 MB of decompressed core_data, which is what confirms the header is being read correctly. The class tables themselves do not yet decode under the documented layout, and that is recorded as open work in plan/level-assets.md rather than papered over. 75 unit tests, 9 integration tests against the real disc. Hash table regenerated at 5,159 files and 5,671,915,799 bytes -- hashes only, no content. Extraction now writes to ~/Documents/gc-extracted rather than /tmp, which is 16 GB and did not have room for a 5.3 GB output. |
||
|
ReaperOfVeriod
|
027373e65b |
Add the disc pipeline: identify, unpack and verify
Phase 1 of the roadmap. The disc is now turned into verified asset files, and
the result is checked against a committed hash table rather than against
whatever happens to be on disk.
gc-extract identify disc.iso SCUS_972.68 v1.01 NTSC, ELF SHA-1 matches
gc-extract unpack disc.iso out/ 5135 files, 4,500,106,960 bytes, 10.8s
gc-extract verify out/ table.tsv OK: 5135 files match
All 92 G/*.WAD containers parse. MPEG1 and MPEG2 are raw video and pass
through whole; the other 90 are indexed containers whose extents are extracted
individually.
Formats recovered and written up in docs/formats/:
- iso9660.rs. R&C2 lists its assets in the filesystem, unlike R&C1, so a
directory walk is the primary asset index. Three details that all fail
quietly rather than loudly: directory records store every 32-bit field twice
(little-endian then big-endian), the CD001 signature is at sector offset 1
rather than 0x8001, and the directory flag at record offset 25 is
authoritative where the identifier's trailing slash is not.
- global_wad.rs. Container headers hold a table of extents. All 92 indexed
containers use {offset in sectors, size in bytes}. Measured, not assumed: the
sector-count reading places every entry of all 27 LEVEL<n> and all 27
AUDIO<n> containers out of range, while the byte reading places all of them
correctly. SCENE<n> is the one exception, a flat sector list whose extents
are sized by the next value.
- wad.rs. Insomniac's LZ77 stream: four flag-selected packet classes, with 0-3
trailing literal bytes per packet carried in unused low bits. This is not a
bit stream, which is why it resists the usual LZ77 reading.
- buf.rs. A bounds-checked reader, so untrusted 2003 disc data produces typed
errors rather than panics.
gc-extract carries the three stages and a SHA-1 implementation with FIPS 180-4
test vectors, so identification works without pulling in a crypto dependency.
Verification: 54 unit tests, plus four integration tests that run only when
GC_DISC names a disc. Those four check identification against the recorded
values, cross-check the Rust identifier against tools/identify-disc.py, parse
all 92 containers, and confirm a full unpack matches the committed table. The
committed table is 316 KB of sizes and SHA-1s for all 5,135 files: hashes only,
no game content, so it is safe to version and it turns "matches retail" into
something a test can assert.
docs/plan/disc-pipeline.md records two wrong turns, because both are easy to
repeat and both failed by producing plausible wrong output rather than an
error: the container table was read backwards, and the LZ77 stream was modelled
as a bit stream. The first survived because it happened to be correct on the
one container used for manual checking.
|