1e90c7de6dda43f8d9080edba4544848a6ce884b
68
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 ReaperOfVeriod
|
b8b4a1a3d3 |
Consolidate: tidy level.rs, correct stale docs, pin hardware models
No behaviour change. This is the pass over the last stretch of work to make sure the accumulated result hangs together rather than only each commit in isolation. Code. lit_color had grown a `let _ = (depth, lod_distances)` and a dead branch from repeated edits. The directional pass genuinely does not depend on distance, so the parameters are gone rather than discarded, with a note that the point lights -- not implemented -- will need them. lit_at folded into lit_color, which had become a wrapper around it. Checked for leftover scaffolding: only two env vars remain, both real (GC_ENGINE_SHOT and GC_ENGINE_DEBUG_BOUNDS); GC_ENGINE_UNLIT, GC_ENGINE_DEBUG_COLOR and GC_ENGINE_ABL were never actually written, because those edits silently failed to apply against rustfmt'd source. AGENTS.md. The crate table did not say that gc-engine owns hardware models. ps2.rs is the Emotion Engine's FPU and the lighting pass runs on it, so the boundary is now stated: gc-formats decodes inputs, gc-engine reproduces how the hardware computes with them. That is the distinction rule 2 depends on, and it was implicit. Docs. Three claims had been overtaken by what we learned and were actively misleading: * docs/formats/level-wad.md still said gameplay "does not decode with the WAD decoder". It does; it is a WAD stream. * docs/plan/level-assets.md carried a whole section, "a compression we have not identified", with a ruled-out list including "bit order", "header length" and "the WAD format at all". All three were ruled out against the wrong input. Replaced, with the failure mode kept: a uniform failure across 27 inputs was read as "the decoder is wrong" when it meant "the data was handed over uncompressed", and the positive control -- decompress and look -- was never run. * The same file still said the tfrag "record framing is not recovered". True, but it was a false lead that has since been superseded by the VIF1 framing. Kept as a recorded wrong turn rather than deleted. Also refreshed "Still open" to name what is actually outstanding: point lights, the GS register set, the guard-band clip test, the unread core_index class tables, and -- stated plainly -- that lighting magnitudes are unverified because there is still nothing to compare against. Verified after the changes: 148 tests pass, clippy and fmt clean, all 5,213 extracted files match the hash table, and level 0 still loads at 26,758 triangles with the same LOD split and fully lit. |
||
|
ReaperOfVeriod
|
bcf0d1f0cc |
Decompress the gameplay extent: the level light bank now parses
gameplay is a WAD stream, exactly like core_data. The extractor was writing it compressed, so the light bank framing read the WAD header's stream length as a light count: level 0's offset at +0x04 was 0xa0d and the "count" there was 698810368. The file says so in its first six bytes: 57 41 44 79 0d 0a -- "WADy". Decompressed, level 0's gameplay extent is 1,872,256 bytes, the offset at +0x04 is 0xc2b20, and the count there is 5. Five directional light sets. All 80,274 vertices of level 0 now go through LightTfrags, and the renderer's fallback line no longer appears. The frame's colour distribution goes from 295 to 737 distinct sampled colours and takes on the level's warm cast. This corrects a recorded negative result that was simply wrong. docs/roadmap.md carried "gameplay does not decode with the WAD LZ77 decoder, all 27 levels fail identically, so the input is wrong rather than the decoder", with bit order and header length also ruled out. That was true of the data being parsed and wrong about the cause: the decoder was never the problem, it was being handed a file that had not been through it. Same lesson as the origin bug, from the other direction. A recorded failure without a positive control is easy to record and easy to leave; the control here is trivial and should have been run first -- decompress and look. The renderer keeps counting and reporting how many vertices it lit, because a silent fallback to stored colours looks plausible and is wrong. Lighting magnitudes remain unverified: that needs comparison against reference frames via the PCSX2 loop, which is still to come. |
||
|
ReaperOfVeriod
|
65b5d366de |
Implement the PS2 float model and the LightTfrags pass
Two pieces, and the first is the reason the second can be trusted. ps2.rs is the EE's FPU on raw u32 bit patterns. It is not f32: there are no denormals (exponent 0 *is* zero), no NaN and no infinity (exponent 255 is an ordinary value, so MAX reads back in Rust as NaN -- a test asserts exactly that, because it is the sharpest available statement of why this module exists), every result truncates toward zero, the adder discards the small operand's bits more than one below the large one's LSB and returns the large operand unchanged across a 25-bit exponent gap, and rsqrt is sqrt-then-divide with truncation between the steps rather than one rounded operation. The lighting pass accumulates all of these across every vertex of every patch, so this is a visible difference rather than a rounding curiosity. tfrag_light.rs is the pass itself: PEXT5 base colour, the select word picking one light set or blending two by bits 8..15 / 256, blended directions re-normalised where unblended ones are not, and base + A.rgb*max(dA, dA*wA) + B.rgb*max(dB, dB*wB). Lighting indexes positions, not vertex-info entries, since a common entry and a LOD entry can share a position. TfragLight gains a `select` field for the u16 at +6. It was briefly reconstructed with an unsafe pointer cast over the struct, which is exactly the sort of thing that breaks silently the moment a field is reordered; it is now a plainly parsed field. The pass does not run yet. The light bank framing the reference uses -- a u32 offset at +0x04, an i32 count there -- gives offset 0xa0d and a count of 698810368 on RAC2 level 0, so RAC2's gameplay extent is not laid out that way. The renderer now counts the vertices it actually lit and prints it, because falling back silently to the stored colours produces a plausible but wrong picture: lighting: 0 of 80274 vertices shaded by LightTfrags; the rest fall back to the tfrag's stored colours (no usable light bank) That line is the current honest state, and the frame has not changed as a result. Where RAC2 keeps its light bank is the open question. |
||
|
ReaperOfVeriod
|
90f02890f8 |
Decode tfrag normals from the overlay's sine table
Normals are per-vertex, and the tfrag's light records store them as an azimuth and elevation *index* into a 256-entry sine table rather than as angles. This decodes that table and builds normals for the renderer. The reference reads the table from an ELF at BOOT_NORMAL_TABLE_VADDR (0x165500) and notes every level overlay carries a copy. The overlay has no ELF header -- it is a raw loadable image starting with code -- so there is no program header to map a virtual address through, and the offset is not fixed either: level 0 has the table at 0xb650c, level 5 at 0xcca30, level 26 at 0xb6284. So it is located by content. Every table starts with exactly (1.0, 0.0), angle zero, and the scan then requires all 256 entries to match cos(i*pi/128), sin(i*pi/128). Validating the whole curve is what makes the search safe: a lone (1.0, 0.0) pair of floats occurs by accident in a 2.7 MB executable, and a test plants one as a decoy. The values are read from the disc rather than recomputed, since those are the numbers the game indexes. A normal is (cos(az)cos(el), sin(az)cos(el), sin(el)) *negated* -- the stored normals point into the surface -- and then given the same Z-up to Y-up swap as positions, or the lighting comes out sideways. The azimuth and elevation fields are stored signed but index 0..=255, so entry 200 must be read as 200, not as -56; there is a test for that. Also frames and parses the level light bank from the gameplay extent, and the pipeline now writes overlay.bin. The per-vertex lighting evaluation is not written yet, so the renderer stays unlit by default. GC_ENGINE_LIT=1 substitutes a host directional light for inspection -- that is a stand-in, not the game's lighting, and it is deliberately not the default. The hash table is regenerated for the 27 new overlay.bin files: now 5,213 files and 5,762,497,559 bytes. |
||
|
ReaperOfVeriod
|
17c7e8ccc9 |
Draw tfrag per-vertex colour, unlit, and record why the frame is dark
The tfrag RGBA array now modulates the texture. It is indexed by *position*, not by vertex-info: several vertex-info entries can share a position, and the array is sized vert_count, which is a position count. Colour follows the same rule as the vertex itself. Morphing blends the position only -- the game keeps the high-detail colour -- while collapse replaces the vertex outright, colour included. Getting that the other way round would tint distant terrain. The material is unlit, and that is a decision rather than a shortcut. Level 0's colours really are dark (patch 0 runs (25,26,26) to (108,144,141)) and the game's lighting is what brightens them, so what is on screen is the true albedo. Worth recording: the obvious move -- add a directional light, or crank ambient brightness -- does nothing at all. With no normals the PBR path is degenerate, so the brightness change has no effect, and a host light would have hidden the gap rather than shown it. The colour data was always reaching the mesh; what is missing is lighting, not a material flag. Written up in plan/level-assets.md, including what LightTfrags actually needs: the light bank from the level's gameplay extent, the 256-entry normal table at BOOT_NORMAL_TABLE_VADDR (0x165500) which every level overlay carries, and the already-parsed azimuth/elevation light records. |
||
|
ReaperOfVeriod
|
192dcde247 |
Render a level: first drawable frame, and the origin bug it exposed
Adds gc-engine's binary and a level.rs that joins the format layer to the renderer: LOD selection and morphing per patch, then triangles grouped by texture so each texture is one draw rather than one per triangle. The PS2 world is Z-up at 1024 raw units per world unit, so the conversion is (x, y, z) -> (x, z, -y). The sign flip on the third axis turns the PS2's right-handed frame into Bevy's left-handed one rather than negating a determinant, which is what stops the level rendering mirrored. Morphing and collapse are kept distinct. Morphing blends a vertex toward the midpoint of its two parents across its tier's band; collapse replaces the vertex outright, position and UV, once it is past the far edge. Morphing is a blend, collapse is a cut, and conflating them makes distant terrain shimmer. Blending has to happen in raw units before the axis swap: mixing a converted parent with an unconverted child scales the blend by 1024. Level 0 renders as 26,758 textured triangles over 92 textures, correct UVs, with LOD selection live (26 patches at LOD 0, 650 at LOD 2 from the start camera). GC_ENGINE_SHOT writes one frame as a PPM and exits. Rendering immediately exposed a bug no test could. Positions are i16 offsets from a per-patch origin, and the obvious rule -- the last non-index STROW row -- is wrong on RAC2, which also uses STROW for per-vertex light records that come *after* the origin. Taking the last one yields [186, 186, 186, 186] and collapses the whole level onto the world origin. The entire level then fit in a 63-unit box and every test still passed: vert_count matched, tri_count matched, every parent and triangle index resolved, because all of those are relative to the same broken origin. Only a screenshot showed it. The rule that works is the STROW immediately preceding the first V3_16 position unpack; with it, level 0 spans x 86..444, y 46..211, z -477..40, matching the patch bounding spheres. That is worth more than the renderer: a parser can be completely self-consistent and still describe the wrong thing, and where a value is absolute rather than relative it has to be checked against something independent. The frame is flat and dark because there are no normals and no lighting yet. That is the next task -- the per-vertex light records -- not something to paper over with an exposure fudge. |
||
|
ReaperOfVeriod
|
71a3185fa5 |
Resolve tfrag LOD parent linkage: guard absent tiers by their counts
The four V4_8 parent arrays are classified by VU address, but the addresses collide: on level 0 patch 1, unk_indices_2_lod_01_addr == parent_indices_lod_0_addr == 291, and unk_indices_2_lod_0_addr == indices_addr == 294. Matching on address alone cannot separate them, and it fails silently -- the arrays still parse, they just hold the wrong numbers. Two corrections, and the first one is worth recording because it looked like a data problem rather than a code one: Each V4_8 element is 4 bytes and holds four indices, so an array of N entries is count*4 bytes, not count. An earlier analysis compared the unpack's element count against the position counts and concluded that 15,000 patches could not hold enough parents. That was the wrong unit; the data was fine. unk_06 and unk_0a suppress an empty tier. When the extra vertex-info count is zero there is no extra array and its address is simply the next tier's, so guarding those branches on the count lets the address fall through to the array that exists. That one change took resolved LOD-0 entries from 28,739 to 341,306. Result: of 886,841 vertex-info entries across all 27 levels, 516,492 are common and have no link by design, and all 370,349 LOD entries resolve -- 29,043 LOD-01 and 341,306 LOD-0 -- with zero failures. Each resolves to a real first parent, a LOD-01 parent that is always a common entry, and three positions inside the patch. A common entry never claims a parent. Morphing can now be completed: a patch knows which LOD to draw, how far to blend each vertex toward its parents, and where those parents are. |
||
|
ReaperOfVeriod
|
2a5307a157 |
Add tfrag LOD selection, morph weights and parent linkage
The block header's float at +0x08 gives L and the switch distances are [D0, D1, D2] = [6L, 4L, 2L] world units -- 192/128/64 on every level measured. The distance test uses the patch's bounding sphere truncated to integers against trunc(D * 1024), not the sphere's centre, so a patch straddling a threshold takes the coarser mode. Six modes result: LOD 2, LOD 1 collapse, LOD 1 morph, LOD 0 collapse, LOD 0 morph, LOD 0. A morph weight runs 0 to 1 across its tier's band. Past the far edge an entry is not blended but replaced by its first parent -- a collapse. The tested property is that distance never moves back to a finer mode, swept across the whole range, because that is what keeps terrain from popping as the camera moves. The four V4_8 parent arrays are classified by VU address and an unrecognised address is now an error, which is a real check: all 18,102 patches still parse. Every recorded parent resolves to real vertex info and real positions, and no common entry claims a parent. That is as far as it goes, and the limit is a finding rather than a rounding error. The addresses are not unique: on level 0 patch 1, unk_indices_2_lod_01_addr == parent_indices_lod_0_addr == 291, with unk_06 and unk_0a both zero, so nothing but unpacking order separates them. Across all 27 levels 57,782 of 374,349 LOD entries resolve, and 316,548 unresolved LOD-0 entries against 28,739 resolved is far too lopsided to be genuine -- either the arrays are misassigned where addresses collide, or RAC2's positions_lod_0_count means something other than RAC1's. Only 3,274 of 18,102 patches carry enough V4_8 bytes to cover positions_lod_01_count + positions_lod_0_count + extras. The vertex-info tiers themselves are confirmed: the addresses at +9 and +11 match the observed V4_16 unpacks and the three tier counts sum to vert_count on every patch, so the header layout is right and it is specifically the parent arrays that are open. Ruled out so far: address-only classification (collides) and one-parent-per-position (fails on 15,000 patches). Not yet tried: ordering by VU address instead of by declaration, and using unk_06/unk_0a to suppress an empty tier before falling through. Written up in plan/level-assets.md. |
||
|
ReaperOfVeriod
|
29fdf89f53 |
Decode level textures: 8-bit indices through a CSM1 palette
Adds texture.rs and makes the pipeline write gs_ram.bin, which it had been skipping. Nothing could be decoded before this: the CLUTs live there. A level texture is not pixels, it is palette indices -- width*height bytes at core_data[textures_base + data_offset], with colour coming from a 256-entry CLUT at gs_ram[palette * 0x100]. gs_ram is a byte range in the level's data extent header at offset 0x10, 518,656 bytes on level 0. Two PS2 conventions have to be honoured and neither is a design choice: within each 32-entry CLUT group the two middle 8-entry blocks are swapped (bits 3 and 4 of the index are exchanged, an involution), and alpha runs 0..=0x80 rather than 0..=255 with 0x80 fully opaque. The mip chain splits across two buffers: levels 0 and 1 are in the texture block with level 1 immediately after level 0, and levels 2 and 3 are resident in gs_ram at mipmap*0x100 and pad*0x100. Tfrag textures are square so every level halves evenly. Verification is two-layered, because every step here is a table lookup and a wrong CLUT remap still produces a well-formed image of the wrong thing. All 1,595 tfrag textures across all 27 levels decode; 96.8% are fully opaque and 96.3% use more than eight colours. On top of that, `gc-extract dump-texture` writes any texture or mip level as a PPM. Level 0 texture 0 comes back as a clean bevelled metal panel with rivets and specular highlights, and texture 74 as a vent grille whose punched windows are fully transparent. That tool is worth keeping: it is the cheapest way to tell a decode that is merely well-formed from one that is right. The hash table is regenerated for the 27 new gs_ram.bin files, now 5,186 files and 5,685,541,143 bytes. |
||
|
ReaperOfVeriod
|
48d9506cf1 |
Decode tfrag texture bindings, vertex colours and light records
Adds level_core.rs (the uncompressed core index header and texture table) and extends tfrag.rs with the data a draw call needs beyond geometry. An ad-gif is five A+D quadwords holding packed fields rather than GS registers; the level load rewrites them in place before drawing. On disc TEX0_1.data_lo indexes the level's texture table, TEX1_1.data_lo is the LOD bias K and data_hi the minification filter, and CLAMP_1 holds both wrap modes. Three things are easy to get wrong here: * K is a signed 12-bit value in 1/16 units, sign-extended by shifting the field up 4 and back down. Retail values are -137..-89 mip-scaled; reading it unsigned gives 1911 instead of -137. * TEX1_1.data_hi is MMIN, and 4 decodes as LinearMipmapNearest, not LinearMipmapLinear as the raw bit layout suggests. * RGBA and the light/normal records are not in the VIF lists at all. They sit at fixed offsets in the patch data block: rgba_size*4 colours at rgba_ofs, vert_count light records at light_ofs+0x10. The tfrag origin lives there too -- if no STROW row supplied one, the quadword at light_ofs is it. The texture table checks out on its own terms: 93 entries at index offset 0x2900 on level 0, and 0x2900 + 93*16 lands exactly on the next range. Verification: an ad-gif index outside the level's table would mean the geometry and the textures were misread independently, so every one is range-checked against real entries. All 27 levels: 47,950 ad-gifs over 18,102 patches, all in range, referencing 1429 of 1595 level texture entries (90%). Each patch's light array is exactly vert_count records and its RGBA array exactly rgba_size*4. Still open: LOD selection and morphing, and the texel data itself. |
||
|
ReaperOfVeriod
|
9459f15329 |
Parse tfrag topology: vertex info, strips, indices and triangles
Turns tfrag positions into drawable geometry. Every unpack in a patch's VIF lists is now classified by format *and* destination VU address, using the VU header at the head of the common list. That table is what tells the strips from the indices from the parent arrays: they are all V4_8 and only the address distinguishes them. Strip indices address the vertex-info array, not the position array. Several vertex-info entries can share a position, so an index resolves through vertex_info[i].vertex / 2 to reach a position. Vertex-info also carries UVs and the LOD parent link. A strip run of n vertices is a GS triangle strip of n - 2 triangles with alternating winding. The vertex count is biased by -128 when negative, and the first record of a list is always biased even when positive -- real patches open with -116 or -124, never a small positive number. tri_count lives in the patch header and cannot be derived from the lists without walking them, so reproducing it is a strong check: all 27 levels, 18,102 patches, 809,879 triangles, every patch matching its header, every triangle index inside its LOD's vertex-info range. Four wrong turns got there, each of which failed quietly rather than loudly: reading the element size from the immediate instead of the command, 2-byte payload padding instead of 4, ignoring the inline payload of STROW/STCOL/STMASK/MPG/DIRECT, and banking strips by arrival order instead of by LOD. Every one produced parseable, plausible geometry. Only the header's own triangle count caught them. Still open: ad-gif texture bindings, per-vertex RGBA and light records, and the LOD switch and morph rules. |
||
|
ReaperOfVeriod
|
36641b4cf8 |
Parse tfrag terrain blocks and decode all vertex positions
Adds gc-formats::tfrag and a gated real-disc test that decodes every terrain patch on all 27 levels. Two rules decide whether this works at all, and both were wrong on the first attempt: The vector format is encoded in the VIF command byte, not the immediate. Bits 2-3 are vn (component count minus one) and bits 0-1 are vl, giving element_size = ((32 >> vl) * (vn + 1)) / 8. Terrain positions are vn=2, vl=1 (command 0x69) at 6 bytes. Reading the format from the immediate's low nibble -- the more commonly quoted layout -- yields 12 or 16 bytes and misaligns everything after it. Unpack payloads pad to 4 bytes, and several non-unpack commands carry inline data: STROW/STCOL 16, STMASK 4, MPG num*8, DIRECT/DIRECTHL imm*16. Ignoring those desynchronises the walk, and a desynchronised walk still finds code words that look like positions. Positions are also not all in one list. vert_count is the total across three tiers, the LOD-2 list precedes the common one in offset order, and the LOD-0-only list starts at shared_ofs + lod_1_size*16 rather than at an offset of its own. Scanning only the common list matches about a quarter of patches, each exactly -- which is what made it convincing. The invariant is that every patch yields exactly its own vert_count. It caught both errors, so tfrag_positions returns None on a mismatch rather than partial geometry. All 27 levels: 18,102 patches, 821,439 vertices, zero mismatches. Still open: strips, triangle indices and the RGBA vertex-info arrays. Positions alone are not drawable. |
||
|
ReaperOfVeriod
|
3ec548efe8 |
Correct the terrain findings: region layout confirmed, record framing disproven
Two claims in the previous commit were wrong and are retracted here rather than
left standing for the next person to build on.
"382 records of 64 bytes" and "byte +3 is 0x48 in every record" were both
artefacts. The 0x48 at that offset is simply the most common byte in the region --
945 of 2000 sampled slots -- not a marker. Over 33,091 slots at that position
only 4,127 hold 0x48. The pattern holds for the first 382 records and then
breaks, which is exactly how a coincidence of a dense byte value masquerades as
structure.
"The coordinates are fixed-point s16 with a scale of 8" is also not supported.
It rested on one coincidence: 17,496 / 8 = 2,187, the planet radius. No grouping
of the record into coordinate triples produces points on a sphere of that radius.
Tested every stride from 2 to 16 bytes at every starting offset, with three and
four components; the best result is 5.0% of points on-sphere, which is chance.
What is now established, and holds on all 27 levels:
core_data region layout, as strictly increasing offsets
tfrag [0, occlusion)
occlusion [occlusion, sky)
sky [sky, collision)
collision [collision, textures_base)
Ruled out the competing reading where 0x0c/0x10/0x14 are sizes laid end to end:
the cumulative total misses textures_base on every level and the leading bytes
of each supposed region are not plausible structures.
Also new: terrain carries its texels inline. A scan for the PS2 texel pattern
18 18 18 80 finds 6,739 occurrences in 138 clusters starting at 0xd9f8, well
inside the tfrag region, so textures_base names something other than terrain
textures.
The vertex format remains unrecovered and the record says so. The next attempt
should look for a longer run of plausible vertices rather than re-deriving a
framing from one record.
|
||
|
ReaperOfVeriod
|
85fa9d1432 |
Locate the tfrag terrain region and record what its records look like
The core header field at 0x08 is zero on all 27 levels, and the seven offsets 0x0c, 0x10, 0x14, 0x60, 0x64 and 0x68 are strictly increasing on every level. Reading them as a sequence of region boundaries gives the layout of core_data: tfrag [0, occlusion) occlusion [occlusion, sky) sky [sky, collision) collision [collision, textures_base) So tfrag = 0 is a real value rather than a missing field, and the terrain is the first 2.12 MB of level 0's geometry. The terrain begins with a short header followed by a repeating 64-byte record from 0x40. Established about those records: byte columns 3, 7, 11 and 15 are 0x48 in every record, which are high bytes of four s16 values giving a near-fixed Y on a planet of radius about 2,187; columns 16 and 17 increase steadily across records like a grid walk; and columns 48-51 carry a repeating ffff00ff pattern characteristic of GS packet headers. The record layout itself is not yet recovered, and the write-up says so rather than guessing. Ruled out and recorded: uncompressed f32 vertices (the longest plausible float run in 16.9 MB is a run of zeros in the collision block, so vertices are packed); GS packet tags at every position (0xE0..0xEF is frequent but uniform, which is what packed data looks like); and a 48-byte record stride (the pattern repeats at 64). Also confirmed this session: the core header base is offset 0 with certainty, because the compressed and decompressed sizes of core_data appear at the same offsets (0x88 and 0x8c) in all 27 levels' core_index. And the occlusion block at the header's 0x0c offset reads as an ascending u16 array of 65 values before restarting, consistent with per-cell visibility records. No code change: this is the reverse-engineering record for the next parser. 75 unit tests, clippy clean. |
||
|
ReaperOfVeriod
|
16c04db559 |
Fix level extents read as bytes, and extract the level geometry
A defect in the Phase 1 extractor, found by reading the data extent's contents: level extents are declared in sector counts, the generic G/ pair table uses byte counts, and level containers were being routed through the generic path. So LEVEL0.WAD's data extent -- 6,484 sectors -- was taken as 6,484 bytes. Every test still passed, because a truncated file is still a file. The extent's own header gave it away: it declares an uncompressed total of 13,277,920 bytes, and 13,277,920 / 6,484 is 2047.7. A ratio landing on 2048 is a unit error, not a compression ratio. Across 27 levels this had been discarding roughly 300 MB of level content. Level containers now go through the typed LevelHeader instead. Level 0 yields: data.bin 13,279,232 the data extent core_data.wad 9,738,678 the geometry, compressed core_data.bin 16,969,296 the geometry, decompressed core_index.bin 31,600 the geometry directory gameplay.bin 659,456 sound_bank.bin 1,503,232 core_data is a WAD LZ77 stream -- the format already implemented for the game -- and it decompresses cleanly. The first real level content this project has recovered. New: LevelDataHeader parses the data extent's own header, a table of byte ranges naming the level overlay executable, the geometry directory, the GS RAM image, the HUD banks and the geometry. Note this is a third meaning for a "size" field on the same disc, after sector counts in the level header and byte counts in the generic table; all three are documented together in level-data.md so the next parser does not rediscover this the hard way. The geometry directory's offsets were checked against the geometry rather than taken on trust: all twelve land inside the 16.9 MB of decompressed core_data, which is what confirms the header is being read correctly. The class tables themselves do not yet decode under the documented layout, and that is recorded as open work in plan/level-assets.md rather than papered over. 75 unit tests, 9 integration tests against the real disc. Hash table regenerated at 5,159 files and 5,671,915,799 bytes -- hashes only, no content. Extraction now writes to ~/Documents/gc-extracted rather than /tmp, which is 16 GB and did not have room for a 5.3 GB output. |
||
|
ReaperOfVeriod
|
a9e43e39c1 |
Parse the level container header, naming each level's four extents
The first typed structure read from real data. Every one of the 27 level containers begins with the same 0x60-byte header, which turns the generic pair table into named assets: data, sound_bank, gameplay and occlusion, plus three chunk descriptors. Two things about this header are worth stating because both are silent failures rather than errors. Sizes here are sector counts, not byte counts. The generic G/ pair table on the same disc uses byte counts, so both readings type-check and the wrong one is off by a factor of 2048. The tiling check settles it: read as sectors, the four extents tile every one of the 27 containers exactly; read as bytes they overlap each other. Level id is not container order. LEVEL21.WAD reports id 30, so a level was cut from the disc and its number left a gap. Anything keying off file order -- a save slot, the planet select screen -- would mislabel that planet. There is also a second trap in the same area: the ISO filesystem sorts names alphabetically, so LEVEL10 precedes LEVEL2, and any tool needing order has to sort numerically. The header's field at 0x04 is zero on all 27 levels. It is not the container size despite looking like one, so the code reads it as a runtime field and takes the container length from the ISO entry instead. Measured across all 27 levels, three invariants now hold in tests: extents tile in file order; extent coverage plus chunks reaches at least the end of the container (21 levels land exactly on its length, 6 leave 134 to 689 sectors of trailing data); and exactly one level's id differs from its container index. The coverage assertion is >= rather than == on purpose -- 21 levels end one sector past the file end because the header occupies sector 0, and asserting equality would truncate a level's tail. gameplay is left unparsed and that is recorded rather than hidden. It begins "WAD" plus a per-level byte and does not decode with the WAD LZ77 decoder: all 27 levels produce plausible output for the first packet and then fail on a rule the format's own encoder never breaks, which means the input is wrong rather than the decoder. docs/plan/level-assets.md records what is ruled out and what to try next. 66 unit tests, plus 7 integration tests against the real disc. The unpack hash table is unchanged, so this adds parsing without altering the pipeline. |
||
|
ReaperOfVeriod
|
027373e65b |
Add the disc pipeline: identify, unpack and verify
Phase 1 of the roadmap. The disc is now turned into verified asset files, and
the result is checked against a committed hash table rather than against
whatever happens to be on disk.
gc-extract identify disc.iso SCUS_972.68 v1.01 NTSC, ELF SHA-1 matches
gc-extract unpack disc.iso out/ 5135 files, 4,500,106,960 bytes, 10.8s
gc-extract verify out/ table.tsv OK: 5135 files match
All 92 G/*.WAD containers parse. MPEG1 and MPEG2 are raw video and pass
through whole; the other 90 are indexed containers whose extents are extracted
individually.
Formats recovered and written up in docs/formats/:
- iso9660.rs. R&C2 lists its assets in the filesystem, unlike R&C1, so a
directory walk is the primary asset index. Three details that all fail
quietly rather than loudly: directory records store every 32-bit field twice
(little-endian then big-endian), the CD001 signature is at sector offset 1
rather than 0x8001, and the directory flag at record offset 25 is
authoritative where the identifier's trailing slash is not.
- global_wad.rs. Container headers hold a table of extents. All 92 indexed
containers use {offset in sectors, size in bytes}. Measured, not assumed: the
sector-count reading places every entry of all 27 LEVEL<n> and all 27
AUDIO<n> containers out of range, while the byte reading places all of them
correctly. SCENE<n> is the one exception, a flat sector list whose extents
are sized by the next value.
- wad.rs. Insomniac's LZ77 stream: four flag-selected packet classes, with 0-3
trailing literal bytes per packet carried in unused low bits. This is not a
bit stream, which is why it resists the usual LZ77 reading.
- buf.rs. A bounds-checked reader, so untrusted 2003 disc data produces typed
errors rather than panics.
gc-extract carries the three stages and a SHA-1 implementation with FIPS 180-4
test vectors, so identification works without pulling in a crypto dependency.
Verification: 54 unit tests, plus four integration tests that run only when
GC_DISC names a disc. Those four check identification against the recorded
values, cross-check the Rust identifier against tools/identify-disc.py, parse
all 92 containers, and confirm a full unpack matches the committed table. The
committed table is 316 KB of sizes and SHA-1s for all 5,135 files: hashes only,
no game content, so it is safe to version and it turns "matches retail" into
something a test can assert.
docs/plan/disc-pipeline.md records two wrong turns, because both are easy to
repeat and both failed by producing plausible wrong output rather than an
error: the container table was read backwards, and the LZ77 stream was modelled
as a bit stream. The first survived because it happened to be correct on the
one container used for manual checking.
|
||
|
ReaperOfVeriod
|
a02f79a6d1 |
Initialise native Rust port of Ratchet & Clank: Going Commando
Sets up the foundation for reimplementing the PS2 game in Rust rather than emulating it. Nothing is implemented yet; this is the scaffold, the verified target and the working agreements that keep a long project coherent. Target disc, verified against a real image: SCUS_972.68 v1.01, USA Black Label, NTSC 60 Hz boot ELF 2,618,684 bytes, SHA-1 bacda860...d098a77, entry 0x00131AE8 The entry point independently matches the value recorded by the rac2-gc-decomp community reference, which cross-confirms the identification. This is the exact build the upstream research pins; Greatest Hits v2.00 is a different binary and is not the target. NTSC matters beyond label compatibility: PAL discs run 50 Hz, which would make chargeboots and other speed-sensitive movement run at 5/6 speed. Verifying the disc before writing movement code avoids discovering that in Phase 5. Contents: - Five-crate workspace (formats, extract, game, video, engine) on Bevy 0.19.1, matching the version ReRAC uses so its Graphics Synthesizer and VU work stays portable. `cargo check --workspace` is clean. - tools/identify-disc.py, which reproduces the identification against any disc image and distinguishes a truncated dump from a non-PS2 image. - .githooks/pre-commit, refusing game data from entering git history: disc image extensions, asset directories, container signatures at offset 0, and files over 50 MiB. Verified against seven cases including a disc image renamed to look like source. Now that this pushes to a remote, this matters more than it would have locally. - AGENTS.md, the operating agreements: no game data ever, reproduce the PS2 float model rather than approximating it, bound every read, verify against the real disc. - PROVENANCE.md, recording that this follows ReRAC (ISC) and learns from rac2-gc-decomp (MIT). Wrench is used as a written specification only and no GPLv3 code is copied, since that would relicense everything derived from it. RAC2Decomp is explicitly not used: its provenance is disputed in the upstream community reference, and it targets the wrong region. - docs/roadmap.md, phased so the riskiest unknowns are retired early. The two that matter: overlay relocation transfers from ReRAC and makes 27 level overlays tractable, while VU1 microcode is binary and is the hardest single item. Phase 5 is a deliberately scoped Veldin vertical slice as the go/no-go. |