51e81b497e5f3efaed4ae971aae311f9e7a693a5
22
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 ReaperOfVeriod
|
51e81b497e |
Show the sky sub-object shape holds across levels
Range measured: sub-objects of the sky region on levels 0, 1, 3, 7, 13 and 26 -- 25 sub-objects, each over its whole length. Method: correlation at lag against the byte sequence, as before. lag 8 +0.53 .. +0.79 positive on all 25 lag 64 +0.16 .. +0.63 positive on all 25 lag 512 -0.13 .. +0.42 not universal lag 1024 -0.05 .. +0.37 not universal Lag 8 and lag 64 hold on every sub-object of every level tested. The longer lags do not, so the second scale seen in level 0's first sub-object is a property of that data rather than of the format. The sky sub-objects are all the same kind of thing: 8-byte units strongly related to their neighbours, with a further relation at 64 bytes. That is the first result about sky that holds everywhere rather than on one level. The 8-byte unit is still unnamed and the values still unidentified. |
||
|
ReaperOfVeriod
|
b5edbd1810 |
Show sky sub-object 0 is a slowly-varying sequence at two scales
Range measured: level 0, sky region +365760..+388880, 23120 bytes. Method: Pearson correlation between the byte sequence and itself at increasing lag, against a shuffled control. This asks whether consecutive records are related, which the position-wise distinct-value counts could not show. lag 1 +0.050 lag 256 -0.044 lag 4 +0.560 lag 512 +0.304 lag 8 +0.746 lag 1024 +0.247 lag 16 +0.666 lag 2048 +0.079 lag 32 +0.602 shuffled +0.003 lag 64 +0.432 lag 128 +0.140 The profile peaks at lag 8, decays to a zero crossing near 256, then rises again to a second smaller peak at 512 before decaying. So records are related within themselves and repeat every 64 records. This settles the earlier "no constant substructure" reading. The records share no fixed fields, but they are not independent either, and that was the thing the distinct-value counts were failing to capture. Slowly varying values, related within a record, repeating every 64 records, is the shape of a keyframe or lookup table -- which is what sky data should be, where a colour or fog parameter is sampled rather than computed per vertex. Still not decoded: nothing names the values or the sampling. But the shape is now known well enough for a decode attempt to aim at, rather than to hunt. |
||
|
ReaperOfVeriod
|
0ba790df1f |
Correct the sky sub-object 0 live-byte figure; record the byte 7 anomaly
The previous commit said "a mean of 6.99 of every 8 bytes are zero". That is inverted. 6.99 is the mean count of live bytes, and the records are nearly full rather than sparse. The live-byte histogram is 0:15, 2:1, 4:31, 5:238, 6:461, 7:1030, 8:1114. That error also invalidated the reasoning built on it in this session's next message, where the high zero-count was used to explain why the differential analysis found no constant positions. There was no zero-density to explain. The differential result stands -- all 8 positions vary -- but for the plain reason that the records are dense, not because of any artifact. New finding from re-examining it: byte 7 takes only 14 distinct values where bytes 0-6 take 142-204. Its common values are 0x00 (1378), 0xff (806) and 0x80 (592), which are the high bytes of signed 16-bit values. That is consistent with byte 7 being an MSB rather than ordinary data. Unexplained. Also tried and recorded as not helping: a signed 16-bit reading spans -32743..32633, near the full range, with no per-position structure (533, 371, 401 and 187 distinct at positions 0-3). |
||
|
ReaperOfVeriod
|
3855169d48 |
Characterise sky sub-object 0: sparse, with duplicated entries
Range measured: level 0, sky region +365760..+388880, 23120 bytes, read as 2890 records of 8. The matched-control autocorrelation puts the peak at stride 8 (0.217 real, 0.000 shuffled), so 8 is the unit to describe. Composition: 1185 distinct u32 values across 5780 words; a mean of 6.99 of every 8 bytes are zero, so most records are sparse; and 549 of 2890 adjacent pairs, 19%, are exact duplicates of each other. The most common 8-byte record appears 283 times and is d7d7d780 d7d7d780 -- a 4-byte value stored twice. Others follow the same shape: 00051980 00051980, 00082580 00082580. Not decoded. Sparsity with repeated entries is what a keyframe or index table looks like, but nothing here establishes that, and the duplicated-4-byte shape may just as easily be two adjacent records sharing a value. Recorded as a characterisation with its range rather than a finding, because that is what it is. The useful part is that the next attempt does not have to re-derive the composition to know whether a decode is plausible. |
||
|
ReaperOfVeriod
|
a82e4a711f |
Measure sky sub-objects across levels; the stride table was over-fitted
Range measured: every level's sky region, from the count field at +0x0c through the sub-object offsets at +0x20, each sub-object over its first 16384 bytes. Method: peak-stride autocorrelation per sub-object with a matched control -- the same bytes shuffled, seeded per level and index. Field note: the sub-object count is a u32 at +0x0c, not +0x0a. Reading +0x0a gives 327,680 on level 0, which is not a count. A brief probe used the wrong offset and read as though it were a count of 5; it got the right offsets by accident because the offset table starts at +0x20 either way. Level 0 has five sub-objects at 365760, 388880, 396432, 406784 and 414032, of lengths 23120, 7552, 10352, 7248 and 1536, each with structure well clear of its control. But the strides do not repeat across levels: obj 0: 4 (x6), 8 (x13), 32, 288 obj 1: 16 (x3), 32 (x6), 288 obj 2: 8, 16 (x5), 256, 288 obj 3: 8, 16 (x6), 216 obj 4: 8, 16 (x3), 512 (x4) So the per-index stride table read off level 0 alone was over-fitted to one level, and is withdrawn. What is consistent is weaker and more honest: every sub-object sits above its matched control, and most peak at 4, 8, 16 or 32, suggesting 16- or 32-bit records whose stride varies with content. Level 2's header is still genuinely different: its count field reads 196,617 and is unusable while every other level yields a plausible count. |
||
|
ReaperOfVeriod
|
98c0d1ab3c |
Re-verify the 368 stride with a control matched to its data
The sky autocorrelation turned out to be inflated by a control that did not match the data, so the 368 result was re-checked the same way rather than being left on the evidence I now know to be weak. Range measured: each level's post-shrub region start, first 65536 bytes. Method: as before, but the control is the same bytes shuffled rather than uniform random. Shuffling preserves the value distribution and destroys only the ordering, which is what a control for this statistic needs to do. Level 0's region is evenly distributed -- 256 distinct byte values, most common only 15.6% -- so uniform random was the appropriate control here, and the shuffled control independently confirms it, reading 0.0000 at strides 340, 368 and 4 against the real data's 0.1110 at 368. Recomputed across all levels with matched controls: 24 of 24 peak at exactly 368, every one with excess above 0.05. So the 368 period stands, now with a control appropriate to its data. The contrast is the useful part: the same statistic, asked the same question of two regions, gave a real answer in one and an inflated one in the other, and the only thing that separated them was whether the control matched the byte distribution. |
||
|
ReaperOfVeriod
|
462d0eda5a |
Correct the sky control and the image extent
The 256-byte period in the sky region stands, but the evidence I gave for it was badly chosen and one figure in it was wrong. The control was uniform random bytes, which read flat at 0.0000 and made the 0.772 match rate look like an enormous signal. It is not: 91.8% of the region's bytes are the same value, 0x07, so a stride-256 comparison matches trivially wherever both positions are 0x07. The correct control is the same bytes shuffled, which keeps the value distribution and destroys only the ordering. It reads 0.512 at stride 256, not 0.0000. The real excess is +0.26 -- a genuine signal, and much smaller than reported. The extent was wrong for the same reason. Measured against the matched control, the excess over the value distribution is: +0 .. +49152 +0.38 .. +0.45 +53248 +0.26 +57344 +0.07 +61440 -0.13 so the periodic data ends between +53248 and +57344, rows 205 to 224, not at +60160. The earlier number came from a raw threshold with no distribution control, and a byte value covering 91.8% of the data drags such a threshold to wherever the dominant run happens to stop. So: the period is real and cross-checked, the image dimensions are not. Any "256x236" from the previous commit should be discarded. The general lesson is the one AGENTS.md asks for and this missed -- a control has to match the thing being controlled for. Uniform random was the wrong reference for data with a dominant byte value, and it made a weak result look like a strong one. |
||
|
ReaperOfVeriod
|
10f797dcdc |
Find a 256-byte-row image at the start of the sky region
Range measured: level 0, 2192576..2615232, the 422,656-byte sky region. Periodicity start and end located by sliding an 8 KB window in 256-byte steps through the region and measuring the 256-stride match rate. Method: autocorrelation, then a periodicity scan. A random control of the same size reads flat at 0.0000 for every stride tested. The peak stride in the first 64 KB is 256 at a 0.772 match rate. Strides 255, 257 and 4 score nearly as high, which is what a genuine period looks like when sampled rather than assumed -- a useful contrast with the 148-byte "stride" that held for eight values and then broke. Across levels the peak is 256 on 21, 512 on 5 and 128 on 1. 512 and 128 are a harmonic and a sub-multiple of 256, so the fundamental period is 256 bytes. Within level 0 the periodicity holds from +0 to about +60160 and then stops: 236 rows of 256 bytes, i.e. a 256-wide by 236-tall image at one byte per pixel. The remaining ~362,240 bytes -- 86% of the region -- are not 256-periodic and stay unidentified. That is presumably what the layer offset table at +0x20 points into. The sky region is therefore not one thing. Its first ~60 KB is an image and is the tractable half; the rest is still open, as is the non-uniform header where level 2 differs from every other level. |
||
|
ReaperOfVeriod
|
d397a1ee34 |
Differential analysis finds no constant substructure at stride 368
Range measured: level 0, 16047040..16737344. 1875 whole 368-byte records from the region start, variation checked across the first 60. Method: the differential analysis that identified the shrub body -- compare records byte by byte, record which positions differ. Result: all 368 positions vary. No constant field, no padding, no header inside a record. The shrub body separated immediately into a varying payload and a constant two-byte tail; this format has nothing constant to separate. That is consistent with the autocorrelation peak rather than in tension with it. Checking which positions carry the 368-stride matches over 20000 positions gives 168 exact 4-byte repeats, distributed evenly at 0.84% across all four byte offsets. Whole 4-byte sequences recur at exactly 368 while no individual position is constant -- consistent with records containing entries that repeat at a fixed relative index, not with a fixed-layout record. So the 368 stride stands and the records stay unidentified. Worth recording that the shrub method failed here for a specific and predictable reason: it needs something constant to latch onto. Applying a method that worked elsewhere is not the same as it applying here, and saying so is cheaper than the next person assuming the technique transfers. |
||
|
ReaperOfVeriod
|
21791e8f5f |
Find a repeating structure at stride 368 in the region after shrub
Range measured: level 0, 16047040..16737344, sampled as the first 65536 bytes. Cross-checked on every level that has the region, each sampled the same way from its own region start. Method: autocorrelation. For each candidate stride, count how often a 4-byte sequence at offset i repeats at i + stride, as a fraction of comparable positions. A random control of the same size measured identically reads flat at 0.0000 for every stride, so the statistic does discriminate. stride 368 0.1110 stride 340 0.0935 strides 337-339 0.026-0.029 background random control 0.0000 Stride 368 is the strongest peak. On level 0, 340 is a secondary peak well clear of the background; the two differ by 28, unexplained. The cross-check is the part that matters: the peak stride was recomputed independently for every level with the region, and all 24 peak at exactly 368. One level's peak could be a property of that level's data; 24 of 24 agreeing is a format invariant. What a 368-byte record is remains unestablished. This is the same position the shrub body was in before differential analysis identified its records, and the same method is the obvious next step -- apply it to the 368 stride rather than continuing to look for headers. Chosen deliberately over the header search: autocorrelation found this in one pass with a control, where four header guesses found nothing. The shrub body only yielded to differential analysis too, and pattern-matching is what produced this session's four retractions. |
||
|
ReaperOfVeriod
|
690092dfb6 |
Disconfirm the ten-entry table lead from the region after shrub
Range measured: level 0, 16047040..16737344. Treating the ten entries' low halves as offsets from the region start lands on blocks exactly 148 bytes apart, the first nine of which begin 08 00 followed by a 16-bit id stepping by 0x40. That reads as a pointer table and it is a coincidence. Walking the same 148-byte stride across the rest of the region, only 14 of 4657 blocks start with 08 00. The first nine agree, then the pattern breaks completely -- id deltas go from a steady 0x40 to 0x801, 0xf510 and negative values. Eight consecutive values at a constant stride is much more than chance should produce and much less than a structure needs. The lead committed an entry ago is therefore disconfirmed, and the region remains unidentified: real, non-zero data, bounded, and unopened beyond its first ~150 bytes. The open question is unchanged and is where the next attempt should start: the 690,304 bytes on level 0 between the last shrub class and the constant trailer. |
||
|
ReaperOfVeriod
|
ed7e568231 |
Find a ten-entry table on a 148-byte stride after shrub
Range measured: level 0, 16047040..16737344, the 690,304 bytes between the last shrub class's body and the constant trailer. Nothing outside that range is read. Method: reading the bytes at +0x20 as u32 shows ten consecutive entries whose low halves increase by a constant 0x94 (148), followed by a zero. The high half is 0xf000 for the first nine and 0x0a40 for the tenth, so the table is ten entries and ends at +0x48, absolute 16047112. What the entries mean, what 148 refers to, and why there are ten are all unestablished. Ten matches no count already known for this level -- 238 moby classes, 74 tie, 7 shrub, 93 tfrag textures -- so it is not yet tied to anything else in the lump. Recording it as a lead rather than a finding. The stride is a real measurement on a bounded range, but a ten-entry table with an unexplained stride is exactly the shape that looks like a structure while being something else, and the rule in AGENTS.md is that a pattern gets looked at before it gets named. |
||
|
ReaperOfVeriod
|
352a4e5ad7 |
Locate the region after shrub, 690 KB on level 0
Range measured: each level's last shrub class data_offset field (class + 0x2c) through to the start of the constant trailer. Nothing outside that range is read. Method: the last shrub class's body is an array of 8-byte records whose final two bytes are 00 00. Scanning forward in 8-byte steps for the first record whose trailing pair differs locates the body's end. That is a search for a stated pattern rather than a statistic, so it either finds the boundary or says it did not. Result: on 24 of 24 levels the boundary lands exactly on an 8-byte record boundary, with body lengths of 192, 208, 224 or 240 bytes -- the same sizes seen when shrub classes were grouped and diffed. Level 0: last shrub class data 16046800 .. 16047040 240 bytes next region 16047040 .. 16737344 690,304 bytes constant trailer 16737344 .. 16969296 231,952 bytes So 690,304 bytes between the last shrub class and the trailer are a distinct region rather than shrub data. What is in them is not established. They are the leading candidate for moby_anim, now that collision..moby has been ruled out as texture. Worth recording how a wrong conclusion nearly happened here. An earlier probe searched for PS2 NaN filler at the start of this range, found none, and was about to report the body as having zero length. That would have been wrong: the last shrub class is a real class with real data, and NaN filler only appears in degenerate ones. Checking the actual byte pattern instead of concluding from a failed heuristic is what caught it. |
||
|
ReaperOfVeriod
|
8d57a09b76 |
Bound the last region by core_data's constant trailer
Comparing all 27 levels byte-for-byte from the end: the last 231,952 bytes (0x38A10) of every level's core_data are identical. Level-specific content therefore ends at len - 0x38A10, which is a real bound for the final region and is not core_data.len(). This closes shrub. Across the 24 levels that have a shrub region the bound gives 24,442,928 bytes total, and the disc test asserts on every level that the region ends strictly inside the lump rather than at its end. What the trailer is remains unestablished -- fixed data, or padding from a fixed-size decompression buffer -- but it is constant, which is all a bound needs. The data between the last shrub class and the trailer is high entropy: 183 to 256 distinct byte values per 4 KB block, mean 254, where texture data sits under 200. It is real content of unknown type and is the most likely home for moby_anim, now that the range between collision and moby has been ruled out as texture. A lump shorter than the trailer now yields end: None rather than a backwards bound, covered by a test. |
||
|
ReaperOfVeriod
|
8882d8fc26 |
Re-scan at finer granularity: collision..moby is entirely texture
The "3 of 21 blocks not texture-like" figure from the last commit was an artifact of sampling every 256 KB and missing everything between. Re-scanned at 4 KB granularity there are about 100 such blocks, and looking at them rather than only counting them, they are smooth gradient ramps -- sky and fog texture. A noisy gradient has a high distinct byte count without being code, which is exactly the mistake the byte-count heuristic invites. So collision..moby is entirely texture data. There is no separate moby_anim region there, and two claims about this range have now been wrong: the "6.2 MB of unknown data" and the "3 unexplained blocks". Both came from sizes and counts taken without looking at what they described. moby_anim is therefore still unlocated, but it is not here. The remaining candidate is the unresolved tail after the last shrub class -- ~925 KB of real, non-zero data on level 0 that nothing accounts for. Recording the two corrections together rather than one at a time, because the mechanism is the same in both: a heuristic count is not an identification. |
||
|
ReaperOfVeriod
|
db414b3d5f |
Correct: the gap between collision and moby is texture data
The previous commit claimed ~6.2 MB of unidentified data sat between the collision header and the first moby class, and speculated it might be moby_anim. That was wrong, and it came from quoting a size before bounding the thing it measured. Looking at the bytes rather than only the offsets: the range is dominated by 8-bit indexed texture data. Byte-value concentration distinguishes them cleanly -- a texture block draws on a small palette, code and collision data does not. 18 of 21 sampled 4 KB blocks fall under 200 distinct byte values, against 249 for tfrags and 209 for collision. The real layout on level 0: collision 2615232 .. 3518544 (header plus its declared 903,248 payload) textures 3518848 .. ~8832192 moby 8832192 .. The texture table's data_offset values stop at 5326208, which is what made the gap look genuine. Mip chains extend well past the last entry's offset, so sizing a region from a table of first-level offsets understates the data it points at. Three of the 21 blocks are not texture-like, so something is mixed in that is still unidentified. It is not 6.2 MB of moby_anim. This is the third number this session that looked load-bearing and turned out to rest on an unbounded measurement. The pattern is consistent enough to be worth stating plainly: sizes derived from tables or offsets are not evidence until the bytes underneath have been looked at. |
||
|
ReaperOfVeriod
|
8b98d388f7 |
Show region bounds in probe, which exposed an unnamed 6.2 MB region
Printing the region table next to the header fields makes the gap visible without having to remember to consult core_layout, and it paid for itself immediately. On level 0 the collision region comes out 6216960 bytes, from 2615232 to the first moby class at 8832192. Collision's own header declares a payload of 903,248. So roughly 6.2 MB sits between the collision header and the first moby class, and almost none of it is collision. The successor rule is correct -- a region does end where the next one begins -- but "the next" is not yet known there. collision..moby is really collision + <something unnamed> .. moby. The name is wrong, not the bound. Candidates for the space, rough order of likelihood: moby and tie model animation data, which is what moby_anim would need, then volumes, then per-class runtime structures. Tfrag texture texels are already placed and sit earlier in the lump. Worth resolving before starting moby_anim, since that format is the most likely tenant. Until then, any probe reading between the collision header and the first moby class is reading something the region table does not describe. |
||
|
ReaperOfVeriod
|
cb38b8c279 |
Compute core_data region bounds once, from successor boundaries
Region spans were being re-derived per format while probing, usually running from one known offset to the next with the last span running to core_data.len(). On level 0 that gave the final shrub class a 925,008-byte span absorbing most of the lump's tail, and every aggregate computed over it -- record counts, value distributions, "this looks random" -- was silently wrong. Nothing errored, which is what made it survive two commits and a correction. core_layout computes the layout once: each region ends where the next one begins, class regions start at the lowest non-zero geometry offset in their table, and a region whose successor is unknown gets end: None rather than a clamped guess. Zero class entries mean absent, not position zero -- the same trap occlusion and collision each fell into separately. The disc test asserts what the module is actually for: that the unknown is visible. Shrub's end is unresolved on the 24 levels that have a shrub region, and that is asserted rather than hidden. It also found that not every level has every region -- level 10 has neither occlusion nor shrub -- so the test checks ordering against the canonical sequence instead of demanding an identical set. Still open: where the shrub region ends. Texture texels cannot bound it; they sit before moby, not after shrub. See docs/formats/core-data-regions.md. |
||
|
ReaperOfVeriod
|
b8b4a1a3d3 |
Consolidate: tidy level.rs, correct stale docs, pin hardware models
No behaviour change. This is the pass over the last stretch of work to make sure the accumulated result hangs together rather than only each commit in isolation. Code. lit_color had grown a `let _ = (depth, lod_distances)` and a dead branch from repeated edits. The directional pass genuinely does not depend on distance, so the parameters are gone rather than discarded, with a note that the point lights -- not implemented -- will need them. lit_at folded into lit_color, which had become a wrapper around it. Checked for leftover scaffolding: only two env vars remain, both real (GC_ENGINE_SHOT and GC_ENGINE_DEBUG_BOUNDS); GC_ENGINE_UNLIT, GC_ENGINE_DEBUG_COLOR and GC_ENGINE_ABL were never actually written, because those edits silently failed to apply against rustfmt'd source. AGENTS.md. The crate table did not say that gc-engine owns hardware models. ps2.rs is the Emotion Engine's FPU and the lighting pass runs on it, so the boundary is now stated: gc-formats decodes inputs, gc-engine reproduces how the hardware computes with them. That is the distinction rule 2 depends on, and it was implicit. Docs. Three claims had been overtaken by what we learned and were actively misleading: * docs/formats/level-wad.md still said gameplay "does not decode with the WAD decoder". It does; it is a WAD stream. * docs/plan/level-assets.md carried a whole section, "a compression we have not identified", with a ruled-out list including "bit order", "header length" and "the WAD format at all". All three were ruled out against the wrong input. Replaced, with the failure mode kept: a uniform failure across 27 inputs was read as "the decoder is wrong" when it meant "the data was handed over uncompressed", and the positive control -- decompress and look -- was never run. * The same file still said the tfrag "record framing is not recovered". True, but it was a false lead that has since been superseded by the VIF1 framing. Kept as a recorded wrong turn rather than deleted. Also refreshed "Still open" to name what is actually outstanding: point lights, the GS register set, the guard-band clip test, the unread core_index class tables, and -- stated plainly -- that lighting magnitudes are unverified because there is still nothing to compare against. Verified after the changes: 148 tests pass, clippy and fmt clean, all 5,213 extracted files match the hash table, and level 0 still loads at 26,758 triangles with the same LOD split and fully lit. |
||
|
ReaperOfVeriod
|
16c04db559 |
Fix level extents read as bytes, and extract the level geometry
A defect in the Phase 1 extractor, found by reading the data extent's contents: level extents are declared in sector counts, the generic G/ pair table uses byte counts, and level containers were being routed through the generic path. So LEVEL0.WAD's data extent -- 6,484 sectors -- was taken as 6,484 bytes. Every test still passed, because a truncated file is still a file. The extent's own header gave it away: it declares an uncompressed total of 13,277,920 bytes, and 13,277,920 / 6,484 is 2047.7. A ratio landing on 2048 is a unit error, not a compression ratio. Across 27 levels this had been discarding roughly 300 MB of level content. Level containers now go through the typed LevelHeader instead. Level 0 yields: data.bin 13,279,232 the data extent core_data.wad 9,738,678 the geometry, compressed core_data.bin 16,969,296 the geometry, decompressed core_index.bin 31,600 the geometry directory gameplay.bin 659,456 sound_bank.bin 1,503,232 core_data is a WAD LZ77 stream -- the format already implemented for the game -- and it decompresses cleanly. The first real level content this project has recovered. New: LevelDataHeader parses the data extent's own header, a table of byte ranges naming the level overlay executable, the geometry directory, the GS RAM image, the HUD banks and the geometry. Note this is a third meaning for a "size" field on the same disc, after sector counts in the level header and byte counts in the generic table; all three are documented together in level-data.md so the next parser does not rediscover this the hard way. The geometry directory's offsets were checked against the geometry rather than taken on trust: all twelve land inside the 16.9 MB of decompressed core_data, which is what confirms the header is being read correctly. The class tables themselves do not yet decode under the documented layout, and that is recorded as open work in plan/level-assets.md rather than papered over. 75 unit tests, 9 integration tests against the real disc. Hash table regenerated at 5,159 files and 5,671,915,799 bytes -- hashes only, no content. Extraction now writes to ~/Documents/gc-extracted rather than /tmp, which is 16 GB and did not have room for a 5.3 GB output. |
||
|
ReaperOfVeriod
|
a9e43e39c1 |
Parse the level container header, naming each level's four extents
The first typed structure read from real data. Every one of the 27 level containers begins with the same 0x60-byte header, which turns the generic pair table into named assets: data, sound_bank, gameplay and occlusion, plus three chunk descriptors. Two things about this header are worth stating because both are silent failures rather than errors. Sizes here are sector counts, not byte counts. The generic G/ pair table on the same disc uses byte counts, so both readings type-check and the wrong one is off by a factor of 2048. The tiling check settles it: read as sectors, the four extents tile every one of the 27 containers exactly; read as bytes they overlap each other. Level id is not container order. LEVEL21.WAD reports id 30, so a level was cut from the disc and its number left a gap. Anything keying off file order -- a save slot, the planet select screen -- would mislabel that planet. There is also a second trap in the same area: the ISO filesystem sorts names alphabetically, so LEVEL10 precedes LEVEL2, and any tool needing order has to sort numerically. The header's field at 0x04 is zero on all 27 levels. It is not the container size despite looking like one, so the code reads it as a runtime field and takes the container length from the ISO entry instead. Measured across all 27 levels, three invariants now hold in tests: extents tile in file order; extent coverage plus chunks reaches at least the end of the container (21 levels land exactly on its length, 6 leave 134 to 689 sectors of trailing data); and exactly one level's id differs from its container index. The coverage assertion is >= rather than == on purpose -- 21 levels end one sector past the file end because the header occupies sector 0, and asserting equality would truncate a level's tail. gameplay is left unparsed and that is recorded rather than hidden. It begins "WAD" plus a per-level byte and does not decode with the WAD LZ77 decoder: all 27 levels produce plausible output for the first packet and then fail on a rule the format's own encoder never breaks, which means the input is wrong rather than the decoder. docs/plan/level-assets.md records what is ruled out and what to try next. 66 unit tests, plus 7 integration tests against the real disc. The unpack hash table is unchanged, so this adds parsing without altering the pipeline. |
||
|
ReaperOfVeriod
|
027373e65b |
Add the disc pipeline: identify, unpack and verify
Phase 1 of the roadmap. The disc is now turned into verified asset files, and
the result is checked against a committed hash table rather than against
whatever happens to be on disk.
gc-extract identify disc.iso SCUS_972.68 v1.01 NTSC, ELF SHA-1 matches
gc-extract unpack disc.iso out/ 5135 files, 4,500,106,960 bytes, 10.8s
gc-extract verify out/ table.tsv OK: 5135 files match
All 92 G/*.WAD containers parse. MPEG1 and MPEG2 are raw video and pass
through whole; the other 90 are indexed containers whose extents are extracted
individually.
Formats recovered and written up in docs/formats/:
- iso9660.rs. R&C2 lists its assets in the filesystem, unlike R&C1, so a
directory walk is the primary asset index. Three details that all fail
quietly rather than loudly: directory records store every 32-bit field twice
(little-endian then big-endian), the CD001 signature is at sector offset 1
rather than 0x8001, and the directory flag at record offset 25 is
authoritative where the identifier's trailing slash is not.
- global_wad.rs. Container headers hold a table of extents. All 92 indexed
containers use {offset in sectors, size in bytes}. Measured, not assumed: the
sector-count reading places every entry of all 27 LEVEL<n> and all 27
AUDIO<n> containers out of range, while the byte reading places all of them
correctly. SCENE<n> is the one exception, a flat sector list whose extents
are sized by the next value.
- wad.rs. Insomniac's LZ77 stream: four flag-selected packet classes, with 0-3
trailing literal bytes per packet carried in unused low bits. This is not a
bit stream, which is why it resists the usual LZ77 reading.
- buf.rs. A bounds-checked reader, so untrusted 2003 disc data produces typed
errors rather than panics.
gc-extract carries the three stages and a SHA-1 implementation with FIPS 180-4
test vectors, so identification works without pulling in a crypto dependency.
Verification: 54 unit tests, plus four integration tests that run only when
GC_DISC names a disc. Those four check identification against the recorded
values, cross-check the Rust identifier against tools/identify-disc.py, parse
all 92 containers, and confirm a full unpack matches the committed table. The
committed table is 316 KB of sizes and SHA-1s for all 5,135 files: hashes only,
no game content, so it is safe to version and it turns "matches retail" into
something a test can assert.
docs/plan/disc-pipeline.md records two wrong turns, because both are easy to
repeat and both failed by producing plausible wrong output rather than an
error: the container table was read backwards, and the LZ77 stream was modelled
as a bit stream. The first survived because it happened to be correct on the
one container used for manual checking.
|