186a700f329c2b766f4076281543ba4c75c7cd80
94
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 ReaperOfVeriod
|
186a700f32 |
Establish the sub-list payload as a fixed five-block shape
All eight sub-list payloads from level 0 were read to their exact extent (count * 16) and compared block by block. Range measured: savestate slot 1, the packets at 0x009376f0, 0x00939c50, 0x0093cc70, 0x00998880, 0x009c19c0, 0x009cf370, 0x009d2da0 and 0x009f5aa0, counts 21 to 41, each truncated to exactly count*16 bytes. Every payload contains exactly five blocks tagged 00 00 00 30, and the first three are always at offsets 0, 32 and 64: 009376f0 22 0, 32, 64, 184, 304 00939c50 36 0, 32, 64, 268, 400 0093cc70 23 0, 32, 64, 200, 288 00998880 36 0, 32, 64, 272, 404 009c19c0 25 0, 32, 64, 196, 300 009cf370 24 0, 32, 64, 204, 316 009d2da0 41 0, 32, 64, 308, 448 009f5aa0 21 0, 32, 64, 176, 280 Blocks 4 and 5 move with the payload; the first three do not. Eight payloads, eight different block-4/5 positions, and 0/32/64 in every case. Per-block fields constant across all eight: block 0 byte 4 varies 0x78..0xcd block 1 byte 4 = 0 in 8/8, trailer 00000000 in 8/8 block 2 byte 4 varies independently, bytes 16-20 zero in 8/8 block 3 byte 4 EQUALS BLOCK 0 in 8/8 block 4 byte 4 = 0 in 8/8 Block 2's word is unrelated to block 0's in every case, so the equality is not every block reading one shared field. Correction: the "byte permutation table" reported in the previous commit does not generalise. It appears in 0x009376f0 at offset 212, but a scan across all eight payloads finds no such run outside that one. It was one payload's block content, not a format feature. The five-block shape, the fixed 0/32/64 offsets and the block-0/block-3 equality are what survive eight-way comparison. Not established: what the five blocks are, why blocks 0 and 3 share a word, and what moves blocks 4 and 5 -- their offsets are not monotone in count (21 gives 280, 36 gives 400 and 404), so something inside the earlier blocks sets them. |
||
|
ReaperOfVeriod
|
3fa64b7003 |
Read a full sub-list payload: tagged blocks and a byte permutation table
With the extent known from the previous entry (count * 16), a whole sub-list
payload can be read. This is the packet at 0x009376f0, count 22, 352 bytes.
Range measured: savestate slot 1, 0x009376f0..0x0093784f.
The payload is a sequence of blocks, each opening 00 00 00 30, carrying a
repeating word and a short trailer:
00 00 00 30 | cd 00 00 00 cd 00 00 00 cd 00 00 00 cd 00 00 00
01 00 00 05 | 22 c1 01 6e | 00 00 00 00
00 00 00 30 | 00 00 00 45 00 00 00 45 1d 00 00 00 1d 00 00 00 1f
81 01 6d 00 | 00 00 00 00 | 00 00 a4 00
00 00 00 30 | bc c0 03 00 23 17 03 00 e5 dd 00 00 00 00 00 00
02 01 00 01 | c1 80 01 69 | 00 00 00 00
The repeating word differs per block (0xcd, 0x45, 0x03) and the trailer's
second word varies in its high bytes (22 c1 01 6e, 81 01 6d 00, c1 80 01 69).
Every block ends 6e, 69 or 6d at byte 23 -- the same 6e the 02 01 00 01 records
carried at bytes 8-9.
The payload contains a byte permutation table at 0x009377c4:
00 01 02 03 3e 3f 53 54 40 41 42 43 4e 4f 50 51 04 05 06 07 08 ...
Ascending byte values with a handful (3e 3f, 53 54, 40 41 42 43, 4e 4f 50 51)
lifted out of order ahead of the sequential run. Read as a table it maps each
of 0x00..0x39 to a slot -- the shape of a code-length or symbol-ordering table,
and the first thing in this format that is self-evidently a table rather than
data to be interpreted.
Not said: whether it is a Huffman-style code, an index reorder for the vertex
data that follows, or a palette permutation. The sequence continues to 39 then
23 c1 01 6e, after which the next block opens.
Corroboration: the fixed-point payload 0x00937850 begins exactly where this one
ends. The 22-record identity made the boundary exact and the next packet starts
on a payload boundary with nothing in between, so the two payload kinds sit
adjacent in RAM -- consistent with them being two records of the same blob.
|
||
|
ReaperOfVeriod
|
f1cf6cd164 |
Show the byte-8 count sizes sub-list payloads at exactly 16 bytes each
The previous entry left one test: compare the count at byte 8 against the payload extent. Pointer spacing gives the extent, since the packets lie in ascending order in EE RAM. Range measured: savestate slot 1, the 17 packets in RAM 0x22a000..0x24037f, each with its record address, byte-8 count and pointer, sorted by pointer. Result: eight packets sit exactly count*16 bytes after their predecessor -- a ratio of 16.00 in every one, no rounding. 0x22a6e8 count 22 ptr 009376f0 gap 352 16.00 0x22a728 count 36 ptr 00939c50 gap 576 16.00 0x22a768 count 23 ptr 0093cc70 gap 368 16.00 0x22a7a8 count 36 ptr 00998880 gap 576 16.00 0x22a7e8 count 25 ptr 009c19c0 gap 400 16.00 0x22a828 count 24 ptr 009cf370 gap 384 16.00 0x22a868 count 41 ptr 009d2da0 gap 656 16.00 0x22a8a8 count 21 ptr 009f5aa0 gap 336 16.00 Eight exact hits at a ratio that must be an integer is not luck. The count at byte 8 is the record count, and these payloads are 16 bytes per record. These are exactly the sub-list payloads -- 009376f0, 00939c50, 0093cc70, 00998880 and 009d2da0 were read directly in earlier passes and all open 00 00 00 30 followed by a repeated word. The other nine have ratios of 168, 419, 559, 31288, 7985, 3458, 764 and 6452 -- no shared factor, far too large to be a record count times anything. Either their payload is not followed immediately by the next packet in pointer order, or they are not 16 bytes per record. Ruled out either way: they do not follow the same rule. This resolves byte 8, measured two entries ago as "a count, 0x0b to 0x29" and now shown to size a payload exactly in eight independent cases within one level. Whether it also sizes the fixed-point payloads is the open half. Still open: why the fixed-point packets differ, and what the 16 bytes per sub-list record are -- the repeated word (0xcd, 0xc0, 0xc8, 0x78) differs per packet, so the records are not identical to each other. |
||
|
ReaperOfVeriod
|
db76319600 |
Find a second packet record magic interleaved with the first
Range measured: savestate slot 1, RAM 0x22a000..0x24037f. Result: a second four-byte record magic appears at comparable density beside 04 04 00 01 -- 02 01 00 01, ten occurrences against nineteen. The two interleave: every 02 01 00 01 sits between two 04 04 00 01 records. 0x22a6b8 04 04 00 01 13 00 00 30 70 6a 93 00 0x22a6c8 02 01 00 01 19 c0 4c 6e 00 00 10 00 00 00 00 00 0x22a6e8 04 04 00 01 16 00 00 30 f0 76 93 00 0x22a6f8 04 04 00 01 16 00 00 30 50 78 93 00 0x22a708 02 01 00 01 1e c0 58 6e 00 00 10 00 00 00 00 00 0x22a728 04 04 00 01 24 00 00 30 50 9c 93 00 The 02 01 00 01 records are 16 bytes like the others but carry no pointer -- bytes 12-15 are zero in all ten. Their fields are a count at 4-5 (0x14, 0x19, 0x1e), a constant 0xc0 at 6, a varying byte at 7 (0x30, 0x4c, 0x54, 0x58), 0x6e at 8-9, and 10 00 at 10-11. The 0x6e at bytes 8-9 matches the ".. c1 01 6e" the sub-list payloads carry at offset 20-23. That is a second correspondence, and unlike the one retracted in the previous commit it is a shared value rather than a shared position. Not settled: this still does not say what discriminates the two payload kinds among the 04 04 00 01 records, since those share the same magic and the same 0x30 at byte 11 and differ only in the count at byte 8. The 02 01 00 01 records sit between them rather than replacing them, so they are a companion structure rather than the missing discriminator. Next: compare the count at byte 8 against the payload's own length. |
||
|
ReaperOfVeriod
|
2d7de9f13e |
Correct the packet tag mapping: the tag does not select the payload
The previous commit mapped the disc tag vocabulary onto the three payload classes. A wider scan says that is wrong. Range measured: savestate slot 1, RAM 0x22a000..0x24037f -- 91008 bytes, 20 times the span scanned before. Result: 19 packet headers, and all 19 carry byte 11 = 0x30. There are no 0x6d or 0x80 tags anywhere in this range. Reading six payloads: rec 0022a6b8 tag 30 -> 8d 8c 80 80 89 87 7c 80 ... fixed point rec 0022a6e8 tag 30 -> 00 00 00 30 cd 00 00 00 ... sub-list rec 0022a6f8 tag 30 -> 92 90 84 80 94 94 86 80 ... fixed point rec 0022a728 tag 30 -> 00 00 00 30 c0 00 00 00 ... sub-list rec 0022a738 tag 30 -> 4c 4c 4b 80 4c 4c 4b 80 ... fixed point rec 0022a768 tag 30 -> 00 00 00 30 78 00 00 00 ... sub-list Records of identical tag alternate between two payload kinds. The tag does not select the payload. What went wrong: the previous entry took the 0x30 and 0x6d that head the payloads to be the disc-side tag vocabulary, and matched them to payload classes by that correspondence. The disc vocabulary at record byte 7 (0x00, 0x30, 0x6d, 0x80) and RAM record byte 11 are different fields that merely overlap in two values. Three payloads each made the coincidence look like a rule. What is actually established about the RAM header: 0-3 04 04 00 01, constant 4-7 zero, constant 8 a count, 0x0b to 0x29 across the 19 records 9-10 zero, constant 11 0x30, constant on all 19 in this range 12-15 an EE RAM pointer Byte 8 is the finding that survives: a small count varying per record, and the one field here doing arithmetic rather than tagging. Still open: what 0x30 at byte 11 means, what discriminates payload kinds given the tag does not, and 0x80, which has no sample in 91 KB of RAM. |
||
|
ReaperOfVeriod
|
0b1793d6af |
Map shrub packet tags onto three payload kinds
Dereferenced all nine pointers in the level 0 table that carry the 0x30 tag at byte 11 and an EE pointer at 12..15, reading 32 bytes each. They fall into exactly three classes, three payloads each, with no overlap. A, Q16.15 fixed-point vertex data (3 of 9): every high byte 0x80 or 0x7c, repeating on an 8-byte boundary, values differing per payload while the 4e 4e 4c 80 tail is shared by two. 00936a70 8d 8c 80 80 89 87 7c 80 ... 4e 4e 4c 80 0093cde0 8a 8a 7f 80 92 90 84 80 ... 4e 4e 4c 80 009cf4f0 6e 6f 68 80 6e 6f 68 80 ... 54 54 51 80 B, a 0x30 sub-list with a repeated word (3 of 9), structurally identical apart from that word (0xcd, 0xc0, 0xc8), all carrying 01 00 00 05 at offset 16 and a ".. c1 01 6e" at 20. 009376f0 00 00 00 30 cd 00 00 00 cd 00 00 00 ... 00998880 00 00 00 30 c0 00 00 00 c0 00 00 00 ... 009d2da0 00 00 00 30 c8 00 00 00 c8 00 00 00 ... C, an index/attribute list (3 of 9), both opening 00 c0 05 6d, then a count, then a run of small ascending u16. 00939700 00 c0 05 6d 3e 00 00 00 01 00 00 00 ... 009c1540 00 c0 05 6d 36 00 00 00 01 00 00 00 ... So the tag vocabulary measured on disc at record byte 7 -- 0x00, 0x30, 0x6d, 0x80 -- selects the payload: 0x30 class B, the 00 00 00 30 sub-list 0x6d class C, the 00 c0 05 6d attribute list 0x00 class A, the fixed-point vertex data 0x30 heads the class B payloads and 0x6d heads the class C payloads. That was flagged as a single coincidence two commits back; three payloads each makes it a rule. 0x80 has no sample among these nine and its meaning stays open. Correction: the record spacing in this table is not uniformly 0x30. The find output shows 0x30, 0x10, 0x30 between consecutive packet headers, and only 9 of the 5632 bytes scanned carry both the 0x30 tag and an EE pointer. The earlier "0x30 spacing" was inferred from a truncated hit list. |
||
|
ReaperOfVeriod
|
ad70e4c4be |
Follow shrub packet pointers: adjacent records hold unrelated payloads
The header pointers found in the previous commit are live, so they can be dereferenced. Two records 0x30 apart in the level 0 table at 0x22a6b8 were followed. Range measured: savestate slot 1, reading 0x00936a70 (from the record at 0x22a6b8) and 0x009376f0 (from the record at 0x22a6e8), 64 bytes each. The two payloads have nothing in common. 0x00936a70 is fixed-point vertex data: 8d 8c 80 80 89 87 7c 80 8d 8c 80 80 89 87 7c 80 4e 4e 4c 80 4e 4e 4c 80 4e 4e 4c 80 4e 4e 4c 80 repeating Every high byte is 0x80 or 0x7c and the pattern repeats on an 8-byte boundary. As s16 these are large negatives in a narrow band around -32500, consistent with Q16.15 fixed point rather than f32, and with a stride of 4 s16 per vertex. 0x009376f0 is a tagged sub-list, not vertex data: 00 00 00 30 cd 00 00 00 cd 00 00 00 cd 00 00 00 cd 00 00 00 01 00 00 05 22 c1 01 6e 00 00 00 00 Structurally different: small integers, a repeating 0xcd, 0x30 tags in the high byte. Nothing here is position data. This supports the type reading of the header: the table is a tagged collection where the tag selects the payload kind, not a uniform array. It also rules out the alternative that the 16-byte tiling was one uniform record format with varying fields -- two adjacent records with unrelated payloads is not that. Still open: the mapping from tag value to payload kind. The 0x30 appearing in record bytes 8-11 in RAM also heads the second payload, which is suggestive but is one coincidence so far and is deliberately not written up as a link. |
||
|
ReaperOfVeriod
|
3854e52019 |
Show the loader rewrites shrub packet headers into RAM pointers
Answering the "what is the tag byte" question needed an anchor. Searching live RAM from scratch was the wrong way to get one: 04 04 00 01 matches unrelated data at 0x1ac988 and there is no way to tell a real hit from a coincidental one. The savestates are the anchor -- slot 1 is level 0, and searching one gives candidates that can be checked rather than guessed at. Range measured: savestate slot 1, ~/.config/PCSX2/sstates/SCUS-97268 (38996035).01.p2s, records at 0x22a6b8 onward, read with gc-trace read --state. Result: 40+ records beginning 04 04 00 01 00 00 00 00 at 0x30 spacing -- the packet framing identified on disc -- with the second half of each header rewritten: disc: 04 04 00 01 | 00 00 00 00 | 00 00 00 05 | 00 80 0e 6c RAM: 04 04 00 01 | 00 00 00 00 | 13 00 00 30 | 70 6a 93 00 Bytes 0-7 unchanged; bytes 8-15 are not. On disc bytes 12-13 are the constant 00 80 and byte 15 the constant 6c; in RAM bytes 12-15 hold a pointer -- 0x00936a70, 0x009376f0, 0x00937850, 0x00939c50 in consecutive records, all inside EE RAM and ascending across the table as a walk of consecutive objects would be. Byte 11 becomes 0x30 in every record seen, against a constant 0x05 on disc, and bytes 8-9 hold a varying small value (0x13, 0x16, 0x24). So the packet table at +0x40 is what the loader walks: span-relative on disc, absolute in RAM. This was inferred from one class before; these 40+ records show it directly. Consequence for the tag bytes: the vocabularies at record bytes 7 and 15 are real, but byte 15 sits in a field the loader overwrites with a pointer. Its disc values are read as data by something rather than stored as a tag. Not settled: whether byte 7's vocabulary is also rewritten, and whether 0x6d marks payload the loader expands. That needs these records compared one by one against their disc spans, which is cheap now that the anchor exists. |
||
|
ReaperOfVeriod
|
d1aec9f866 |
Show shrub packet records are two tagged 8-byte halves, fixed length
Range measured: level 0, the 1636 sixteen-byte records following the header across all 29 packets. Per-byte distinct counts, not pooled. Method: count distinct values and zero frequency at each of the 16 positions, then tabulate the joint distribution of the two low-entropy positions. Result: fourteen positions are uniformly varied, 101 to 153 distinct values, zero in 13% to 58% of records. Two are not: byte 7 4 distinct: 0 (1432), 0x30 (135), 0x6d (40), 0x80 (29) byte 15 2 distinct: 0 (1618), 0x6d (18) Both sit in the last byte of an 8-byte half, eight bytes apart, and both take values from a small closed vocabulary rather than the full byte range. The joint distribution shows no record where both are non-zero, and 0x6d appears in either position but never both: (0,0) 1414 (0,0x6d) 18 (0x30,0) 135 (0x6d,0) 40 (0x80,0) 29 Each 8-byte half carries a tag or subtype byte in its last position, from a small closed set. The shared 0x6d is what suggests one vocabulary used twice rather than two unrelated fields. This answers the question the previous commit left open. There is no per-record length field: the best candidate offsets hold the value 16 on at most 13% of records, which is the chance rate for small integers, not a self-describing record. The records are therefore fixed 16 bytes, and the tiling is not a coincidence of variable-length records that happen to be multiples of 16. Still open: what the four and two tag values mean, and what the 14 varied bytes encode. The tag vocabulary is small enough to enumerate against known PS2 primitive lists, which is the reference to bring in next. |
||
|
ReaperOfVeriod
|
e6dd60ec3a |
Characterise shrub mesh packets: 16-byte header, 16-byte tiling
The packet table decode left the packet contents open, with a warning that a VIF reading had already been retracted once and needed a control rather than a pattern match. This is that measurement. Range measured: level 0, the 29 packets of the 6 successor-bounded shrub spans, 26640 bytes. Per-packet; nothing pooled. Method: autocorrelation within the largest packet against a distribution matched shuffled control, then per-byte distinct counts across all 29 packets. Result 1: all 29 packet sizes are exact multiples of 16, and the internal stride scan peaks at 16 (+0.283) with 32, 48, 64, 80 and 128 all elevated behind it -- the signature of a record array, not a coincidental spacing. Result 2: record 0 is a 16-byte header. Constant across all 29 packets at bytes 0-3 (04 04 00 01), 4-11 (zero), 12-13 (00 80) and 15 (6c). Bytes 14 and 16-23 vary. The 04 04 00 01 prefix looks like a magic or format version. Result 3: header byte 14 is echoed in the body. It takes values 06, 07, 0e and 11 across packets, and the same value appears at offset 12 of record 1 in every packet sampled. A real repeated field, not two small values coinciding. Not established: no header field matches the record count. The packet holds 1056/16 = 66 records and none of the candidate u32 or u16 header offsets at 8, 10, 12, 14, 16, 18 or 20 equals 66, 65 or 67 on more than half the packets. So the count is either computed from a size field elsewhere in the span, or the records are variable length and merely 16-byte aligned. That distinction is unresolved and matters. Fixed 16-byte records make the payload a flat table; variable records make the 16-byte alignment a coincidence of every record being a multiple of 16 long. The stride scan cannot separate the two, because both peak at 16. The retracted VIF reading is not revived: the tiling is uniform across all 29 packets with no command-shaped tail, and a VIF packet would show a changing command byte last rather than the constant 6c at header byte 15. Next: read the record count from the packet table in the span rather than from the packet header, which is what would separate the two readings. |
||
|
ReaperOfVeriod
|
aa2b14ad66 |
The shrub mesh geometry is in the class blob after all
The open question from the shrub decode was where the mesh lives: the blob looked like header, palette and optional billboard, with the 333 classes without a billboard having nothing to draw. Answering it took the running game, not the disc. Diffing a loaded shrub class against its disc bytes in the live PCSX2 (level 1, class 1) shows what the loader rewrites: a table at +0x40 of (offset, size) pairs, whose offsets are span-relative on disc and absolute in RAM with the sizes unchanged -- all 21 entries matching offset for offset and size for size -- plus a relocated palette pointer at +0x2c and a pointer at +0x18, zero on disc, that asserts the parser enforces. So the blob is header, packet table, packets, [billboard], palette: the mesh was here the whole time. Why it was missed is the part worth recording -- data_offset points at the END of the span, so reading the body as data_offset..span_end accounts for the last few hundred bytes and leaves the 20 KB in front of it unexamined. A correct bound and a body small enough to look plausible are together enough to hide a region. Range measured: every shrub span on the disc, 358 over 24 levels, each bounded by its successor class or the located chunk container. Method: read count at +0x28, then the entries at +0x40, and require them to tile -- the first packet at align16(0x40 + count*8), every entry exactly size past the previous, and the last ending at the palette, or 0x40 short of it on the 25 spans that carry a billboard. Result: 358 of 358 tile exactly, holding 2,239 packets. That is what pins the field meanings: count is the packet count (1..55 on the disc), data_offset is the palette's offset, and the earlier "the body is not sized by count" note was true for a reason nobody had understood -- the count sizes the packets. Shrub::packets() exposes them and returns None rather than a partial list when a table does not tile. Still open: the packet contents (a VIF reading was already retracted once, so this needs a control rather than a pattern match), and the 0x20-byte records the loader points at from +0x18. |
||
|
ReaperOfVeriod
|
c83f5104d2 |
Document the PCSX2 test setup
PCSX2 was set up and working but existed only in this machine's filesystem and in people's heads, so every comparison looked like a one-off observation rather than something re-runnable. The setup section in docs/plan/pcsx2-verification.md now records where the emulator, the BIOS, the disc, the savestates and the PINE socket are, how a state is captured, and which command the disc integration tests need -- and AGENTS.md and README.md point at it from their orientation sections. One operational detail worth spelling out: the emulator's directory is lowercase (pcsx2), which is not what people type. |
||
|
ReaperOfVeriod
|
bbdb51e2f9 |
Verify level 1's tfrag lighting against the game
Savestate slot 2 is the second planet, so the comparison runs on a level with a different geometry extent, light bank and runtime-set population: level1: 401/401 headers, 401/401 pointers, bank byte-identical vertices compared: 26470 (105880 colour bytes) -- equal: 100.00% The live emulator agrees with the savestate byte for byte. Level 1 is a different planet, so the two fixes in the previous commit are not level-0 coincidences: the normal's z sign and the unlit-record base colour both hold across 268,280 colour bytes on two levels. |
||
|
ReaperOfVeriod
|
69a8398f6e |
Verify the tfrag lighting magnitudes against the running game
The harness that compares the port against the game's own memory existed but had never been compiled: tools/trace did not build (27 errors, all in the argument plumbing), and the tfrag_light.rs light-bank correction it motivated was unverified. This finishes it and runs it, and the run finds two real bugs. Result, on level 0 against three inputs -- the live emulator over PINE and two PCSX2 savestates from the same level, agreeing byte for byte: headers matching the disc on invariant bytes: 833/833 data pointers relocated as table + disc offset: 833/833 light bank in RAM at 0x50ab20, byte-identical to the disc's vertices compared: 40600 (162400 colour bytes) equal: 162400 (100.00%) The locator finds the level's tfrags block in RAM by tfrag 0's bounding sphere and accepts a candidate only when every header matches the disc on the bytes the game leaves alone and every data pointer is the relocated form, so "the block was found" is itself a verification rather than an assumption. The bank is searched in RAM by its exact section bytes, which is what settles its framing: offset at +0x04, an i32 set count, twelve bytes of pad, then the sets. The previous reading (offset at +0x00, count at +0x04) was wrong. Two bugs the comparison found, both of which produced plausible output: 1. vertex_normal_bits negated x and y and left z positive, while NormalTable::normal negates all three. dA came out -0.707 where the game had +0.707, so max(dA, dA*wA) took the wrong branch and every vertex lit near its base colour: dark, but not wrong-dark. 95.88% agreement with the bug, 100.00% without. 2. A record whose select names a set the bank lacks drew transparent black. The game writes the record's base colour; 3,130 of level 0's 40,600 vertices do this (select 0x000f, set 15 -- a runtime slot the level's directional bank does not have). The vertex is unlit, not invisible. Both are now regression tests in the engine. The comparison prints the pass's per-vertex terms (light_vertex_terms) for the first mismatches, because a colour mismatch alone cannot be attributed and the comparison is worthless without attribution -- that is how the second bug was identified from a three-line read rather than guessed at. Docs: docs/plan/pcsx2-verification.md records the harness, the result, the two bugs and what to compare next; roadmap and level-assets.md stop carrying lighting magnitudes as an open item. Findings reach tests only as numbers -- no test reads a savestate, a dump or a live emulator. |
||
|
ReaperOfVeriod
|
2c7555fe16 |
Decode the shrub class blob: a 24-normal palette and a billboard record
The shrub body's payload encoding was the open question after the span-bound work, and it answered itself once the R&C1 reference was in view: R&C1's shrub class header keeps the field offsets R&C2 shows (o_class at 0x24, a count at 0x28, the data offset at 0x2c), which says which bytes to look at, and the disc says what they hold. Range measured: all 358 shrub class spans over the 24 levels that have them, each bounded by its successor class or the located chunk container. Nothing outside the spans is read. Result. The body is a palette of 24 unit-length s16 normals -- s16 x, y, z and a zero pad word per 8-byte record -- followed by all-zero padding records. Three independent measurements, each of which fails for any other reading: - all 8,592 live entries have squared length within 65,535 of 32767^2, and the bound is tight (the largest observed deviation is exactly 65,535). That is 32768^2 - 32767^2, which is what a unit vector scaled by 32768 produces when an axis-aligned normal quantises to -32768 and the positive axis clamps to 32767; components run the full s16 range and 1,765 entries are axis-aligned. A random s16 triple is off by about a billion, so one entry in fifteen thousand would pass by chance: all 8,592 pass. - the padding records are always at index 24 or later and always all zero, never a live entry. That is what separates the palette from geometry, and it is the observation the three earlier stride-fitting attempts were missing. - 17 classes carry a palette of 24 identical (0, 0, 1) entries. A position or index table cannot produce that; a normal palette for a flat-facing class must. The header decodes as: sphere at +0x00 (centre + radius, radius positive on all 358 spans, 1.72..41.57), size at +0x10, mode_bits and scratch at +0x14, a zero word at +0x18, the billboard offset at +0x1c, and scale at +0x20. The parser asserts the two invariants that hold without exception (scratch and the reserved word are zero on all 358) plus a positive radius and scale, and reports mode_bits rather than asserting it: it is 4 on 16 of the 358 spans. The billboard offset is the strongest self-identification after o_class because it is confirmed from a second file: it is non-zero on exactly the 25 classes whose core-index entry carries billboard texture fields, pairing correctly both ways. At the offset sits R&C1's ShrubBillboard shape, register block and all -- fade_distance, then TEX1_1 (K -233..-168, MMIN 4, address 0x14), TEX0_1 (index 1..4, address 0x06) and MIPTBP1_1 (address 0x34), all three register addresses checked in the disc test. Two corrections this produces, both recorded in docs/plan/level-assets.md because the old readings were confidently wrong rather than merely unverified: - "reserved is 0 or a denormal" had the two u16 halves the wrong way round: the 16 non-zero spans are mode_bits = 4 with scratch = 0. As an f32 the word is still a denormal, so the PS2 reads zero either way -- but the field is a flag word, not a magnitude, which is exactly the FPU point worth making. - "bounds" was tested as a min.xyz/max.xyz box, which holds on only 146 of 358 spans, and the field was left unnamed as a result. It is a bounding sphere: the fourth float is positive on all 358 spans and the centres cluster near the class origin. The 146/358 figure was a property of the wrong question. Still open: the three floats in the billboard record (they repeat byte-identically across levels -- 16 distinct triples among 25 records, one on four classes across levels 1, 4, 12 and 21 -- so they are shared art data rather than dimensions), size, mode_bits, count, and where the shrub mesh geometry lives. R&C1 puts the mesh packets in this blob; R&C2's blob holds only the palette, so the 333 classes without a billboard must draw meshes from somewhere else. |
||
|
ReaperOfVeriod
|
ccea53807a |
Resolve the shrub region bound: the chunk container closes it
The shrub region's end was the last unlocated bound in core_data. It was not in the core index, and unlike moby->tie it had no successor boundary to chain against, so the last shrub class in every level ran out to the lump's end and the aggregates over it (the "3.5M records / 28MB of shrub body") were artifacts of that oversized span. Range measured: for each level, from the last shrub class's geometry offset to the constant trailer. Nothing outside that range is read. Method: scan forward from the last shrub class for the pair (a) the constant two bytes f6 7c sitting immediately in front of (b) a chunk signature -- a 16-byte header ending 0xff, ten zero bytes, then four or more 0xf0-tagged offsets. The two halves are each weak alone, and the halves are measured separately: the marker pair occurs 25 to 103 times in a 5 to 17 MB lump, so matching it alone picks dozens of wrong offsets; the chunk shape matches random data often enough to need four tagged offsets before it is believed. Result: on all 27 levels the marker sits exactly two bytes before the first chunk header, and the two parts of the signature are never found apart. No chunk signature occurs inside any last shrub body, on any level. The three levels without shrub (10, 25, 26) carry the same marker before the container that closes tie instead, which is what makes the locator a property of the region rather than a shrub rule. Consequences: - All 358 shrub class bodies are bounded and are whole 8-byte records. That was not assertable while the last class in each level ran to the lump's end, and it is the check the boundary earns its keep for. - 9,600 records in total, against the 8,938 previously measurable in successor-bounded spans; 662 of them sit in the 24 last-class bodies. The shrub region totals 2,033,776 bytes across the 24 levels that have one. - The chunk container is 25,775,850 bytes across 27 levels (1,236 chunks, 22,562 records). The container measurements were previously taken over "the 900,000 bytes before the trailer", a window that did not align to its real extent: on level 0 it is 690,302 bytes, and 27 of 1,123 chunks were truncated at that invented edge. chunk_containers_resolve_on_every_level now reads the region from core_layout instead of measuring backwards from the end of the lump. core_layout takes core_data now: the container's start is the one bound that cannot be derived from the index. It gains the chunks region, so every byte of level-specific core_data belongs to a named region. Where the signature is not found the end stays None -- the region is left visible rather than clamped to the tail. Still open: the marker's meaning (constant across 27 levels, at exactly this position, unnamed) and the shrub body payload's encoding. |
||
|
ReaperOfVeriod
|
4d1e277a15 |
Exclude f32 and u16 readings of the shrub body record
Range measured: level 0 shrub bodies, 164 records of 8 bytes, following the
previous commit.
The six varying bytes are not an f32 pair:
f32 @0 158/164 finite, range up to ~1e35. Not positions or normals. The 6
non-finite values (3.7%) are far above the 0.05% random bytes would
give, so the field is structured but is not an f32.
f32 @4 164/164 finite, range 0 to 9.18e-5. Looks promising until you notice
bytes 6 and 7 are always zero, which pins it to a coarse grid and
explains the tiny range. Not independent evidence.
u16 @0 85 distinct, range 0..65534
u16 @2 90 distinct, range 0..65534
u16 @4 80 distinct, range 0..65512
The u16 fields are not small counts or indices; they span essentially the whole
range.
Excluded: two packed u32, an f32 pair, and narrow u16 fields. The six bytes
remain uninterpreted.
Worth noting the f32 @4 case as a trap: a plausible-looking finite non-negative
float range that was entirely an artifact of the two constant zero bytes
pinning it. Checking the constant bytes before believing a float reading is now
the order to work in.
|
||
|
ReaperOfVeriod
|
7c0df0f893 |
Characterise shrub body records on level 0 as a per-level partition
Range measured: level 0, shrub class table, 7 entries at core index offset 10176. Six classes are bounded by a successor; the seventh runs to the end of core_data and is excluded because its span contains the whole post-shrub region. Method: bound each span by the next class geometry offset, take the body from data_offset to the span end, split into 8-byte records. Result: 164 records across 6 bounded bodies, lengths 192 twice, 208 twice, 224, 240. Per byte over those 164 records: byte 0: 74 distinct, 60 zero byte 4: 72 distinct, 35 zero byte 1: 58 distinct, 65 zero byte 5: 59 distinct, 38 zero byte 2: 76 distinct, 43 zero byte 6: 1 distinct, 164 zero byte 3: 62 distinct, 48 zero byte 7: 1 distinct, 164 zero Bytes 6 and 7 are zero in every record; bytes 0 to 5 take 58 to 76 distinct values with only 21 to 39% zero, so uniformly varied rather than sparse. Read as u32 the fields are not plausible: u32[0] spans 0 to 4294871316 and no field is monotone across the body. The six bytes are not two packed 32-bit values. This settles the six-varying-plus-two-constant shape on level 0 as a per-level partition, rather than only in aggregate over the disc's 8938 records, which is what rule 3a asks for. Two trailing zero bytes are padding or a flag pair. Unresolved: whether the six bytes are six u8, an f32 plus two u16, or something else. The u32 reading is excluded but no positive reading is established. Process note: the first attempt used ClassEntry and a hand-computed o_class instead of ShrubClassEntry, and every ShrubHeader::region call failed. The parser was fine; the caller was not. |
||
|
ReaperOfVeriod
|
efe4c06be6 |
Show the chunk header does not encode the record stride
All 90 two-byte windows of header[0..14] were tested: none maps to a unique stride across every group. There is no simple two-byte field to read the record size from. The stride must therefore come from the offset table, by differencing the first two tagged offsets, which is what the parser already does. This validates that choice as more than convenience. Given three chunks sharing one header with three different strides, a dispatch that assumed "header implies size" would be wrong for the whole type-ID coverage on at least one layout. Reading the stride from the offsets is the only interpretation consistent with every measurement so far. Negative result recorded deliberately: it rules out a whole family of hypotheses (stride encoded in the header) rather than narrowing one. |
||
|
ReaperOfVeriod
|
1d84562e88 |
Resolve the colliding header: three types share 16 bytes
Header 7542eff271b522a79e45348cca4509ff was reported with three strides, 76, 92 and 104. It is not one type with a variable stride, and not a misparse. It appears three times per level, always with record count 9, always in the same order at the same offsets within the region: offset stride count first last 863842 92 9 928 1640 867554 76 9 912 1520 872866 104 9 1000 1832 The offsets are 3712 apart each time and the strides differ, so these are three separate chunks sharing 16 header bytes. Consequence for the type ID: 64 distinct headers for 66 distinct record layouts, not 64 for 64. The header is nearly a type ID but does not uniquely determine the layout. Not excluded: a distinguishing byte sits outside the 16 captured, or the layout varies within a type. This does not change "63 of 64 headers imply one stride", which remains true, but the mapping is not a bijection and should not be used as one. Worth checking before the parser is extended to dispatch on header alone. |
||
|
ReaperOfVeriod
|
4c4175de7f |
Add a rule requiring measurements to be partitioned before reporting
Rule 3 required bounds and a method for every number, but not that the measurement be partitioned. Three errors in one stretch were all of that kind, and all three passed the control check rule 3 asks for -- because a matched control bounds significance while saying nothing about scope: - a 368-byte period attributed to a whole region, when it was one chunk's stride among ~15. Averaging the strides put the peak at the noise floor, which read as "no structure" rather than "wrong scope". - a ten-entry table disconfirmed, when it existed at a fixed stride among several. - a moby class array reported at u16 slots 129-140, which was the largest-stride chunk type dominating a pooled scan. Per type, all 26 place their array elsewhere. In each case the aggregate was arithmetically correct and the conclusion was wrong. The re-run that caught all three was identical: group by the thing that varies, then look. Rule 3a now requires stating the partition before the number, reporting per partition or saying explicitly that a claim is pooled and unverified, treating a structure found in one partition as unproven until every partition agrees, and retracting rather than softening a result that changes under partitioning. Renumbered the following rules. |
||
|
ReaperOfVeriod
|
b13901a7e7 |
Correct the moby class array offset: it is per chunk type
The previous commit reported moby class IDs at u16 slots 129 through 140. That was a pooling artifact. Repeating the scan per chunk type shows every type places its class array at a different offset: aac4 129-134 82c4 104-109 4ec4 106-111 b8c4 120-125 9943 82-87 79c4 111-115 3245 24-29 and so on across all 26 types Slots 129-140 were simply the type with the largest stride dominating a scan that pooled every type together. The previous commit flagged this as unverified; the check confirmed the flag. What survives is stronger than what it replaced. In every one of the 26 types the top hits form a contiguous run of consecutive u16 slots. The real invariant is therefore per-type: each chunk type has its own class array, at its own offset, of its own length. That fits the header-implies-one-stride result -- each type is a distinct record layout -- but the class array is a per-type field, not a common one at a fixed offset. The moby reference itself is unaffected. The 38.6x excess was measured across all records and does not depend on which slots were counted. |
||
|
ReaperOfVeriod
|
ddfbc6eb33 |
Find moby class references in chunk records at 38x chance
Range measured: level 0, the 900000 bytes before the trailer; 28 chunks, 475 records, 67780 u16 slots. Reference set: the 238 distinct o_class values from level 0's moby class table, read through LevelCoreHeader. Method: scan every u16 in every record, count membership in the moby class set, and compare against the chance rate of 238/65536 per slot. Result: 9493 hits against 246.1 expected, a 38.6x excess. The hits are concentrated, not spread. Slots 129 through 140 -- twelve consecutive u16, bytes 258 to 280 of the record -- carry 105 to 188 hits each. Nothing else comes close except a separate cluster at slot 38. Every chunk contributes in proportion to its record count. A contiguous twelve-value array that is mostly moby class identifiers is what a model or instance definition listing referenced mobies would look like. Enrichment is not identification. A 38x excess over chance is strong evidence these are moby class IDs rather than incidental small integers, but it does not establish that the chunk is moby_anim: an instance table, a model list and an animation table would all reference class IDs. The format and the meaning of slots 129-140 are both still unnamed. Also unverified: whether the twelve slots are always the same fields. The offsets come from the largest records, and this scan pooled all chunk types together, so a type with a different stride could place them elsewhere. Process note: the first version of this test extracted the moby class table by hand and got 67 entries instead of 238, which made every chunk look like a hit and would have supported any conclusion. Reading it through LevelCoreHeader was what made the measurement meaningful. |
||
|
ReaperOfVeriod
|
379c12f8ec |
Identify the chunk header as a type ID that determines the record stride
Range measured: all 27 levels, 1123 chunks. Method: group chunks by their 16-byte header; for each distinct header, collect the set of strides seen across every level it appears on. Result: only 64 distinct headers across 1123 chunks. The header is a type identifier, not a per-chunk descriptor -- the same 16 bytes recur on every level. And 63 of those 64 imply exactly one stride, so header to record layout is very nearly a function. The exception is header 7542eff271b522a79e45348cca4509ff, seen with strides 76, 92 and 104. Either three types share a header or one is a misparse. This also settles the previous entry's open question about D/E/F being incidental. Grouping by header bytes 9 and 13: DE 26 chunks, 2 strides (184, 344) EE 1050 chunks, 46 strides EF 34 chunks, 2 strides (268, 640) FF 13 chunks, 1 stride (520) Rare tags are tightly coupled to their layouts -- FF is a single stride across all 13 chunks -- while EE is heterogeneous. That is what a type tag looks like. Consequence: a chunk can be dispatched on its header to a record layout, and 63 of 64 headers give that layout's size without inspecting a record. First point in this container where a field does real work rather than correlating. Still open: what the 64 types mean, which record fields are index/length/offset, and whether any type is moby_anim. |
||
|
ReaperOfVeriod
|
10188416e2 |
Decode the chunk count field: header byte 14 is the record count
Range measured: all 27 levels, 1123 chunks, over the 900000 bytes before the constant trailer. Method: compare header byte 14 against the number of offsets actually resolved for that chunk. Result: byte 14 equals the record count on 1123 of 1123 chunks, zero mismatches. This is the first genuinely decoded field in this container rather than a measured correlation. The parser now truncates the offset table to header byte 14, so a stray tagged word in a chunk's record data cannot silently extend the table and put the count at odds with its own header. Two more header bytes are near-constant and read as ASCII: byte 13 takes only 0x45 (1076 chunks) and 0x46 (47); byte 9 takes 0x44 (26), 0x45 (1084) and 0x46 (13). Those are D, E and F -- a three-letter tag at two byte positions with different distributions. Recorded as suggestive, not established: the other 14 header bytes are high-entropy and read as ASCII at no shift tried, so the D/E/F may be incidental. Worth testing against a known type table. Still open: the 14 opaque header bytes, whether bytes 9 and 13 are type tags, which record fields are index/length/offset, and whether a chunk is moby_anim. |
||
|
ReaperOfVeriod
|
681333535a |
Add a bounds-checked parser for the post-shrub chunk container
gc_formats::chunks parses the container that fills the region after the last shrub class body and before the constant trailer: +0x00 16 bytes, last byte 0xff (opaque, differs per chunk) +0x10 10 zero bytes +0x1a N tagged offsets, top byte 0xf0 Each tagged offset is 0xF0000000 | low24, and a record is read at chunk_start + low. Resolving that way gives records sharing a field layout, which is what distinguishes the container from a coincidental byte pattern. Fields are not named. The 16-byte header is opaque, the 0xf0 tag is unexplained, and it is not established that a chunk is moby_anim -- a 49 x 368-byte table inside this region is byte-identical on all 27 levels, which is evidence against that identification. What the parser provides is chunk and record extraction, which is what the analysis established. The gated test measures, over all 27 levels: 1123 chunks, 19546 records, 50 distinct strides, 12.4% of byte positions constant across records (per-chunk range 2.8% to 75%). Chance gives a constant position probability of 1/256^n. Three things the test got wrong first, recorded because they are the traps: - A per-chunk threshold of 20% constant positions. Individual chunks run from 2.8% to 75%; the aggregate is the invariant, not the chunk. Asserting 20% per chunk was asserting a property the format does not have. - Requiring every offset to resolve. 27 chunks truncate at the window edge, because the 900000-byte window does not align to chunk boundaries. Counted now rather than asserted away. - read_u32 in the chunk signature check was big-endian while Buf::u32 is little-endian. The tag lives in the high byte of a little-endian word, so the mismatch made every level report zero chunks while parse_chunk -- which goes through Buf -- was correct. Worth remembering that the two paths can disagree on a format this young. Tests: 190 -> 191. |
||
|
ReaperOfVeriod
|
8b0a2545a3 |
Confirm chunk offsets address real records on all 27 levels
Range measured: each level's post-shrub region, 900000 bytes before the constant trailer. Method: per chunk, read tagged offsets as 0xF0000000 | low24, take stride as the first delta, load one record per offset at chunk_start + low, and count byte positions constant across that chunk's records. Result: 1096 chunks and roughly 13000 records resolve on all 27 levels. Every level shows 17-21% of byte positions constant across its own records. Level 0: 27 chunks, 466 records, 20.8% constant. The stride-368 chunk (77 records) has 26 constant positions of 368; the stride-340 chunk has 25 of 340. Constant positions cluster at the record end -- 267-276 for stride 368, 258-268 for stride 340 -- so a footer, not scattered padding. The tag is uniform: every tagged offset has top byte 0xF0 on all levels. This settles the previous entry's framing. Resolving at chunk_start + low yields records with ~20% constant positions, which does not happen by chance, so the offsets genuinely address records and those records share a layout. Still not decoded: the 16-byte header, the 0xF0 tag, which fields are index/length/offset, and whether a chunk is moby_anim. A parser can extract chunks and records bounds-checked today; naming the fields needs a domain argument this project has not found. |
||
|
ReaperOfVeriod
|
1d6cddf3bb |
Identify the post-shrub region as a chunk container
Range measured: level 0 post-shrub, 16047040..16737344, 690304 bytes. Method: scan for a 16-byte header ending 0xff, then 10 zero bytes, then a run of u32 whose top nibble is 0xF. Result: 28 chunk headers on level 0, present on all 27 levels. Each chunk is +0x00 16 bytes, last byte 0xff (opaque, differs per chunk) +0x10 10 zero bytes +0x1a N tagged offsets, top nibble 0xF Within a chunk the offsets step by a chunk-specific stride. On level 0: 148, 340, 368, 288, 320, 236, 272, 384, 116, 160, 250, 580, 576. Near-constant within a chunk, different between chunks. This resolves two earlier results, both of which were half-right. The 368-byte period is one chunk's stride, not a region property. That is why the region-wide autocorrelation is flat: ~15 strides averaging to noise. The 15360-byte window where 368 scored +0.315 is the third chunk at region+30066, whose stride is 368. The "ten-entry table" earlier disconfirmed is real. Chunks carry 5 to 30 tagged offsets, 10 most common on level 0. It was disconfirmed because I tested for a fixed 148 stride across a whole region; the table exists, but 148 is one chunk's stride among many. Both errors were the same one: reporting a local measurement as a regional property. Twice now. The control methodology caught neither, because a matched control bounds significance but says nothing about scope. Not decoded: the 16-byte header, the 0xF tag, where chunk data lives, and what any chunk means. Framing established, contents not. |
||
|
ReaperOfVeriod
|
c60340b1de |
Correct the post-shrub 368 stride: real, but in one 15 KB window
The claim that the post-shrub region has a 368-byte period throughout is wrong. Re-measured over the whole region the curve is flat and near noise: stride 8 +0.027 stride 12 +0.023 stride 148 +0.009 stride 340 +0.018 stride 368 +0.018 stride 632 +0.020 Nothing peaks at 368, and the 0.1110 recorded earlier does not reproduce under any sampling convention tried. Where the structure is: scanning 8 KB windows at 1 KB steps for a 368-excess above 0.15 finds one substantial region, region+41984..+57344, 15360 bytes, plus a 4 KB patch at region+650240. Within that window 368 is unambiguous -- +0.315 against +0.284 at 736, with 364, 372 and 732 all at noise. 41 whole records, mean 16 distinct values per position, and nine positions exactly constant and zero: 329-336 and 340. Twenty-five positions have 3 or fewer distinct values, clustered in 325-349, which is a tail rather than scattered padding. Corrected statement: a 15360-byte window of the post-shrub region is a table of 41 records at stride 368, each with a roughly 43-byte tail whose fields are mostly zero. The surrounding 675 KB is not that table. What went wrong before: the original measurement was taken over a window where 368 held, then reported as a property of the region. That window holds 41 of 1841 records, and averaging it over the region flattens the peak to the noise floor. The re-verification checked the stride but not whether it applied uniformly, so it confirmed the number without testing the claim attached to it. |
||
|
ReaperOfVeriod
|
7992e9d8f3 |
Note that the SBlk period is not a record boundary
Range measured: level 0 SBlk, 24..19508, read at 4-byte alignment. The most common u32 values are 0x0 (2261), 0x1 (369), 0x7f (168), 0x5a000000 (125), 0x1eff7f7f (125), 0x1dfcc (124), 0x19fc0 (93), 0x7c34240d (91), 0x7c3821c8 (91). That does not look like a table of records with typed fields. A 40-byte layout of index, length and offset would not produce 0x1eff7f7f and 0x7c34240d a hundred times each, nor would those values survive a byte shift. So either the 40-byte period is not a record boundary, or the region is an encoded stream that happens to have a 40-byte statistical period. The 0x7f values point at the latter: in a varint encoding 0x7f is the largest value representable in seven bits, so it appears as a frequent saturated byte -- consistent with a coded stream, inconsistent with fixed-layout records. Left open. The 40-byte period is solid -- 27 of 27 levels, sharply isolated, multiples confirming -- and so is the alignment with four constant zero positions. But those two are in tension in a way nothing else in this project has been: a confirmed period that is not a record boundary. Someone attacking this format should not assume the record framing that the autocorrelation suggests. |
||
|
ReaperOfVeriod
|
e9c927994c |
Establish the SBlk 40-byte record layout
Range measured: level 0 SBlk, 24..19508, aligned at +0 -- 487 whole records. Method: for each of 40 candidate alignments, count byte positions that are near-constant across records, and keep the best. Alignment +0 wins with 10. Four positions are exactly constant and all zero: 7, 11, 31 and 35. Distinct values per position run from 1 to 155, with position 29 the most variable by a wide margin -- 155 against 21 to 48 elsewhere -- and positions 2, 10, 27, 34 and 39 low at 2 to 8. The first record carries the SBlk magic and the DAW tag rather than descriptor data, so it is a header rather than an entry. That matters before reading any field at a fixed offset. Not decoded: the layout is established and the constant positions are known, but the fields are not named and which of the 40 bytes are an index, a length and an offset is still open. |
||
|
ReaperOfVeriod
|
0f1a7f27c7 |
Resolve the SBlk discrepancy: 40 is the record, markers are nested
A fine scan settles it in favour of 40: stride 40 +0.454 stride 39 +0.266 stride 41 +0.215 stride 38 +0.214 The peak is sharp and isolated, nearly twice its immediate neighbours, and the multiples confirm it: 80 scores +0.376, 120 +0.394, 160 +0.340, while the multiples of 12 (24, 48, 96, 144) all sit at 0.21 to 0.24. So the record is 40 bytes and the 0x7f/0x47f markers at 12-byte spacing are a nested structure inside those records, not the record boundary. Their spacing is irregular -- 12, 24 and larger -- which is what one would expect of a field present in some records and absent in others. The previous commit's "they disagree" framing was itself wrong in one respect: the marker spacing never claimed to be the record size. Read as a claim about markers inside records rather than about the record itself, the two measurements were never in conflict, and I had set up a false dichotomy between them. |
||
|
ReaperOfVeriod
|
5df3914160 |
Record an unresolved discrepancy in the SBlk record size
Range measured: level 0 SBlk, 24..19508, 19484 bytes. The 0x7f/0x47f marker words sit at 12-byte spacing on 144 of roughly 180 occurrences, with 24-byte and larger gaps accounting for the rest. So there is a 12-byte structure. That contradicts the autocorrelation. If records were 12 bytes, stride 12 should peak, and it does not: stride 12 +0.237 stride 20 +0.175 stride 24 +0.237 stride 40 +0.454 stride 48 +0.239 40 is not a multiple of 12, so the measurements disagree rather than one explaining the other. Recorded as open rather than resolved in favour of either: the marker spacing says 12, the period says 40, and 40 being twice as strong is the more surprising of the two. Worth resolving before attempting a record layout, because a layout built on the wrong one will parse plausibly and be wrong. |
||
|
ReaperOfVeriod
|
d3373042b9 |
Confirm moby's 16-byte vertex by two independent methods
The part table established vertex_offset + 16 * vertex_count == next_part on 43467 of 43467 parts, which fixes the vertex stride at 16. Autocorrelation over the spans now reaches the same number without using that result. Range measured: moby class spans of 16 KB or more on level 0 -- 60 spans, bounded by consecutive class offsets, over their first 32768 bytes. Method: peak-stride autocorrelation against a matched control, strides 4 to 64. stride 16 43 spans stride 8 8 stride 48 4 stride 32 3 stride 24 2 16 dominates and it is the same 16 the part table produces from a completely different argument -- a tiling identity over part records rather than a periodicity measurement over span contents. Two unrelated methods agreeing is worth more than either alone. The autocorrelation would be vulnerable to the exact artifacts this project has already been bitten by: a dominant byte value inflates it, and a period that happens to divide the measurement window looks identical to a real one. Neither applies here, and a tiling identity cannot be inflated at all. This is the strongest form of evidence produced so far in this project -- a structural invariant confirmed by two methods that do not share a failure mode. |
||
|
ReaperOfVeriod
|
332dc63600 |
Measure tie class spans: 16-byte unit, 48-byte period, dominant not universal
Range measured: every tie class span of 16 KB or more on levels 0, 3, 6 and every third level after -- 386 spans, bounded by consecutive class offsets. Method: peak-stride autocorrelation with a matched control over the first 32768 bytes of each span. stride 48 285 spans stride 16 58 stride 64 29 stride 96 11 stride 24 2 stride 32 1 48 bytes dominates but is not universal: 58 spans peak at 16 against 285 at 48, so this is a majority rather than a constant. Every observed peak is a multiple of 16, suggesting a 16-byte base unit with 48 arising from how those units group rather than 48 being the record. This is weaker than the sound bank's 40-byte result and is recorded at the confidence it deserves. That one was sharply isolated -- 0.454 excess at 40 against 0.21 to 0.28 across strides 36 to 44 -- and identical on all 27 levels. Here the peak is 0.09 to 0.22 excess and moves between levels and spans, so the two should not be treated as the same kind of claim. |
||
|
ReaperOfVeriod
|
d1190a3343 |
Find a 40-byte period in the sound bank SBlk, on all 27 levels
Range measured: each level's SBlk region -- the descriptor table between sblk_offset and daw_offset, from the sound bank header rather than from core_data. 17-20 KB per level. Method: peak-stride autocorrelation with a matched control, the same bytes shuffled with the same seed on every level. The peak is at exactly 40 bytes on all 27 levels, every one with excess above 0.2. On level 0 the peak is sharply isolated: 0.454 excess at stride 40 against 0.207 to 0.278 for strides 36 through 44, so it is a period rather than a broad tendency. This is the first structural result about the sound bank beyond the container framing already committed. Record counts are 487, 443, 428, 494 and 447 whole records on levels 0, 1, 2, 13 and 26, and the region is not an exact multiple of 40 on any level -- 19484 leaves 4 bytes over -- so 40 is a period rather than a record size with no remainder. Unexplained: which byte positions are constant varies by level. Level 0 has 7, 11, 31 and 35 constant; level 1 has 15, 19, 23, 35 and 39; level 2 has 7, 23, 27 and 31. A fixed 40-byte layout would share its constant positions, so either the records have a variable tail or 40 is a period in something the records sit inside. Also measured: DAW shows no stride above threshold against its control, consistent with ADPCM nibble-packed audio and with the container framing being the only part worth decoding. gameplay.bin peaks at 16 bytes. gs_ram.bin has almost no stride structure but a lag-8 correlation of +0.503, so it is smooth rather than tabular. |
||
|
ReaperOfVeriod
|
51e81b497e |
Show the sky sub-object shape holds across levels
Range measured: sub-objects of the sky region on levels 0, 1, 3, 7, 13 and 26 -- 25 sub-objects, each over its whole length. Method: correlation at lag against the byte sequence, as before. lag 8 +0.53 .. +0.79 positive on all 25 lag 64 +0.16 .. +0.63 positive on all 25 lag 512 -0.13 .. +0.42 not universal lag 1024 -0.05 .. +0.37 not universal Lag 8 and lag 64 hold on every sub-object of every level tested. The longer lags do not, so the second scale seen in level 0's first sub-object is a property of that data rather than of the format. The sky sub-objects are all the same kind of thing: 8-byte units strongly related to their neighbours, with a further relation at 64 bytes. That is the first result about sky that holds everywhere rather than on one level. The 8-byte unit is still unnamed and the values still unidentified. |
||
|
ReaperOfVeriod
|
b5edbd1810 |
Show sky sub-object 0 is a slowly-varying sequence at two scales
Range measured: level 0, sky region +365760..+388880, 23120 bytes. Method: Pearson correlation between the byte sequence and itself at increasing lag, against a shuffled control. This asks whether consecutive records are related, which the position-wise distinct-value counts could not show. lag 1 +0.050 lag 256 -0.044 lag 4 +0.560 lag 512 +0.304 lag 8 +0.746 lag 1024 +0.247 lag 16 +0.666 lag 2048 +0.079 lag 32 +0.602 shuffled +0.003 lag 64 +0.432 lag 128 +0.140 The profile peaks at lag 8, decays to a zero crossing near 256, then rises again to a second smaller peak at 512 before decaying. So records are related within themselves and repeat every 64 records. This settles the earlier "no constant substructure" reading. The records share no fixed fields, but they are not independent either, and that was the thing the distinct-value counts were failing to capture. Slowly varying values, related within a record, repeating every 64 records, is the shape of a keyframe or lookup table -- which is what sky data should be, where a colour or fog parameter is sampled rather than computed per vertex. Still not decoded: nothing names the values or the sampling. But the shape is now known well enough for a decode attempt to aim at, rather than to hunt. |
||
|
ReaperOfVeriod
|
0ba790df1f |
Correct the sky sub-object 0 live-byte figure; record the byte 7 anomaly
The previous commit said "a mean of 6.99 of every 8 bytes are zero". That is inverted. 6.99 is the mean count of live bytes, and the records are nearly full rather than sparse. The live-byte histogram is 0:15, 2:1, 4:31, 5:238, 6:461, 7:1030, 8:1114. That error also invalidated the reasoning built on it in this session's next message, where the high zero-count was used to explain why the differential analysis found no constant positions. There was no zero-density to explain. The differential result stands -- all 8 positions vary -- but for the plain reason that the records are dense, not because of any artifact. New finding from re-examining it: byte 7 takes only 14 distinct values where bytes 0-6 take 142-204. Its common values are 0x00 (1378), 0xff (806) and 0x80 (592), which are the high bytes of signed 16-bit values. That is consistent with byte 7 being an MSB rather than ordinary data. Unexplained. Also tried and recorded as not helping: a signed 16-bit reading spans -32743..32633, near the full range, with no per-position structure (533, 371, 401 and 187 distinct at positions 0-3). |
||
|
ReaperOfVeriod
|
3855169d48 |
Characterise sky sub-object 0: sparse, with duplicated entries
Range measured: level 0, sky region +365760..+388880, 23120 bytes, read as 2890 records of 8. The matched-control autocorrelation puts the peak at stride 8 (0.217 real, 0.000 shuffled), so 8 is the unit to describe. Composition: 1185 distinct u32 values across 5780 words; a mean of 6.99 of every 8 bytes are zero, so most records are sparse; and 549 of 2890 adjacent pairs, 19%, are exact duplicates of each other. The most common 8-byte record appears 283 times and is d7d7d780 d7d7d780 -- a 4-byte value stored twice. Others follow the same shape: 00051980 00051980, 00082580 00082580. Not decoded. Sparsity with repeated entries is what a keyframe or index table looks like, but nothing here establishes that, and the duplicated-4-byte shape may just as easily be two adjacent records sharing a value. Recorded as a characterisation with its range rather than a finding, because that is what it is. The useful part is that the next attempt does not have to re-derive the composition to know whether a decode is plausible. |
||
|
ReaperOfVeriod
|
a82e4a711f |
Measure sky sub-objects across levels; the stride table was over-fitted
Range measured: every level's sky region, from the count field at +0x0c through the sub-object offsets at +0x20, each sub-object over its first 16384 bytes. Method: peak-stride autocorrelation per sub-object with a matched control -- the same bytes shuffled, seeded per level and index. Field note: the sub-object count is a u32 at +0x0c, not +0x0a. Reading +0x0a gives 327,680 on level 0, which is not a count. A brief probe used the wrong offset and read as though it were a count of 5; it got the right offsets by accident because the offset table starts at +0x20 either way. Level 0 has five sub-objects at 365760, 388880, 396432, 406784 and 414032, of lengths 23120, 7552, 10352, 7248 and 1536, each with structure well clear of its control. But the strides do not repeat across levels: obj 0: 4 (x6), 8 (x13), 32, 288 obj 1: 16 (x3), 32 (x6), 288 obj 2: 8, 16 (x5), 256, 288 obj 3: 8, 16 (x6), 216 obj 4: 8, 16 (x3), 512 (x4) So the per-index stride table read off level 0 alone was over-fitted to one level, and is withdrawn. What is consistent is weaker and more honest: every sub-object sits above its matched control, and most peak at 4, 8, 16 or 32, suggesting 16- or 32-bit records whose stride varies with content. Level 2's header is still genuinely different: its count field reads 196,617 and is unusable while every other level yields a plausible count. |
||
|
ReaperOfVeriod
|
98c0d1ab3c |
Re-verify the 368 stride with a control matched to its data
The sky autocorrelation turned out to be inflated by a control that did not match the data, so the 368 result was re-checked the same way rather than being left on the evidence I now know to be weak. Range measured: each level's post-shrub region start, first 65536 bytes. Method: as before, but the control is the same bytes shuffled rather than uniform random. Shuffling preserves the value distribution and destroys only the ordering, which is what a control for this statistic needs to do. Level 0's region is evenly distributed -- 256 distinct byte values, most common only 15.6% -- so uniform random was the appropriate control here, and the shuffled control independently confirms it, reading 0.0000 at strides 340, 368 and 4 against the real data's 0.1110 at 368. Recomputed across all levels with matched controls: 24 of 24 peak at exactly 368, every one with excess above 0.05. So the 368 period stands, now with a control appropriate to its data. The contrast is the useful part: the same statistic, asked the same question of two regions, gave a real answer in one and an inflated one in the other, and the only thing that separated them was whether the control matched the byte distribution. |
||
|
ReaperOfVeriod
|
462d0eda5a |
Correct the sky control and the image extent
The 256-byte period in the sky region stands, but the evidence I gave for it was badly chosen and one figure in it was wrong. The control was uniform random bytes, which read flat at 0.0000 and made the 0.772 match rate look like an enormous signal. It is not: 91.8% of the region's bytes are the same value, 0x07, so a stride-256 comparison matches trivially wherever both positions are 0x07. The correct control is the same bytes shuffled, which keeps the value distribution and destroys only the ordering. It reads 0.512 at stride 256, not 0.0000. The real excess is +0.26 -- a genuine signal, and much smaller than reported. The extent was wrong for the same reason. Measured against the matched control, the excess over the value distribution is: +0 .. +49152 +0.38 .. +0.45 +53248 +0.26 +57344 +0.07 +61440 -0.13 so the periodic data ends between +53248 and +57344, rows 205 to 224, not at +60160. The earlier number came from a raw threshold with no distribution control, and a byte value covering 91.8% of the data drags such a threshold to wherever the dominant run happens to stop. So: the period is real and cross-checked, the image dimensions are not. Any "256x236" from the previous commit should be discarded. The general lesson is the one AGENTS.md asks for and this missed -- a control has to match the thing being controlled for. Uniform random was the wrong reference for data with a dominant byte value, and it made a weak result look like a strong one. |
||
|
ReaperOfVeriod
|
10f797dcdc |
Find a 256-byte-row image at the start of the sky region
Range measured: level 0, 2192576..2615232, the 422,656-byte sky region. Periodicity start and end located by sliding an 8 KB window in 256-byte steps through the region and measuring the 256-stride match rate. Method: autocorrelation, then a periodicity scan. A random control of the same size reads flat at 0.0000 for every stride tested. The peak stride in the first 64 KB is 256 at a 0.772 match rate. Strides 255, 257 and 4 score nearly as high, which is what a genuine period looks like when sampled rather than assumed -- a useful contrast with the 148-byte "stride" that held for eight values and then broke. Across levels the peak is 256 on 21, 512 on 5 and 128 on 1. 512 and 128 are a harmonic and a sub-multiple of 256, so the fundamental period is 256 bytes. Within level 0 the periodicity holds from +0 to about +60160 and then stops: 236 rows of 256 bytes, i.e. a 256-wide by 236-tall image at one byte per pixel. The remaining ~362,240 bytes -- 86% of the region -- are not 256-periodic and stay unidentified. That is presumably what the layer offset table at +0x20 points into. The sky region is therefore not one thing. Its first ~60 KB is an image and is the tractable half; the rest is still open, as is the non-uniform header where level 2 differs from every other level. |
||
|
ReaperOfVeriod
|
d397a1ee34 |
Differential analysis finds no constant substructure at stride 368
Range measured: level 0, 16047040..16737344. 1875 whole 368-byte records from the region start, variation checked across the first 60. Method: the differential analysis that identified the shrub body -- compare records byte by byte, record which positions differ. Result: all 368 positions vary. No constant field, no padding, no header inside a record. The shrub body separated immediately into a varying payload and a constant two-byte tail; this format has nothing constant to separate. That is consistent with the autocorrelation peak rather than in tension with it. Checking which positions carry the 368-stride matches over 20000 positions gives 168 exact 4-byte repeats, distributed evenly at 0.84% across all four byte offsets. Whole 4-byte sequences recur at exactly 368 while no individual position is constant -- consistent with records containing entries that repeat at a fixed relative index, not with a fixed-layout record. So the 368 stride stands and the records stay unidentified. Worth recording that the shrub method failed here for a specific and predictable reason: it needs something constant to latch onto. Applying a method that worked elsewhere is not the same as it applying here, and saying so is cheaper than the next person assuming the technique transfers. |
||
|
ReaperOfVeriod
|
21791e8f5f |
Find a repeating structure at stride 368 in the region after shrub
Range measured: level 0, 16047040..16737344, sampled as the first 65536 bytes. Cross-checked on every level that has the region, each sampled the same way from its own region start. Method: autocorrelation. For each candidate stride, count how often a 4-byte sequence at offset i repeats at i + stride, as a fraction of comparable positions. A random control of the same size measured identically reads flat at 0.0000 for every stride, so the statistic does discriminate. stride 368 0.1110 stride 340 0.0935 strides 337-339 0.026-0.029 background random control 0.0000 Stride 368 is the strongest peak. On level 0, 340 is a secondary peak well clear of the background; the two differ by 28, unexplained. The cross-check is the part that matters: the peak stride was recomputed independently for every level with the region, and all 24 peak at exactly 368. One level's peak could be a property of that level's data; 24 of 24 agreeing is a format invariant. What a 368-byte record is remains unestablished. This is the same position the shrub body was in before differential analysis identified its records, and the same method is the obvious next step -- apply it to the 368 stride rather than continuing to look for headers. Chosen deliberately over the header search: autocorrelation found this in one pass with a control, where four header guesses found nothing. The shrub body only yielded to differential analysis too, and pattern-matching is what produced this session's four retractions. |
||
|
ReaperOfVeriod
|
690092dfb6 |
Disconfirm the ten-entry table lead from the region after shrub
Range measured: level 0, 16047040..16737344. Treating the ten entries' low halves as offsets from the region start lands on blocks exactly 148 bytes apart, the first nine of which begin 08 00 followed by a 16-bit id stepping by 0x40. That reads as a pointer table and it is a coincidence. Walking the same 148-byte stride across the rest of the region, only 14 of 4657 blocks start with 08 00. The first nine agree, then the pattern breaks completely -- id deltas go from a steady 0x40 to 0x801, 0xf510 and negative values. Eight consecutive values at a constant stride is much more than chance should produce and much less than a structure needs. The lead committed an entry ago is therefore disconfirmed, and the region remains unidentified: real, non-zero data, bounded, and unopened beyond its first ~150 bytes. The open question is unchanged and is where the next attempt should start: the 690,304 bytes on level 0 between the last shrub class and the constant trailer. |
||
|
ReaperOfVeriod
|
69b81af3b2 |
Expose the shrub body as 8-byte records, bounded per class
Turns the differential finding into code. Each shrub body record is 6 varying bytes plus 2 constant ones; body_records() hands them back raw and body_is_well_formed() checks the trailing pair. ShrubHeader::region now takes an explicit span_end. It has to: without it the data slice runs to the end of the lump, so the last class in each level absorbs whatever follows shrub and a per-class check ends up measuring its neighbours. That is the same unbounded-span mistake as before, caught this time by a failing assertion rather than by reading the docs afterwards. The disc test asserts well-formedness only for the 334 of 358 classes whose span ends at the next class. The last class in each level has no successor, so its span contains the 690 KB region after shrub and its body is not wholly records -- that is the open question, and asserting either way would be wrong. Shrub::span_end_is_a_successor records which case a span is. Records are summed only across bounded bodies. Summing the last class too reproduces the "3.5 million records" figure retracted two commits ago, to within rounding -- which is a useful check on how easy that mistake is to repeat by accident. 8,938 records across the 334 bounded bodies; 16 denormal reserved words; 358 headers naming their own class. |
||
|
ReaperOfVeriod
|
ed7e568231 |
Find a ten-entry table on a 148-byte stride after shrub
Range measured: level 0, 16047040..16737344, the 690,304 bytes between the last shrub class's body and the constant trailer. Nothing outside that range is read. Method: reading the bytes at +0x20 as u32 shows ten consecutive entries whose low halves increase by a constant 0x94 (148), followed by a zero. The high half is 0xf000 for the first nine and 0x0a40 for the tenth, so the table is ten entries and ends at +0x48, absolute 16047112. What the entries mean, what 148 refers to, and why there are ten are all unestablished. Ten matches no count already known for this level -- 238 moby classes, 74 tie, 7 shrub, 93 tfrag textures -- so it is not yet tied to anything else in the lump. Recording it as a lead rather than a finding. The stride is a real measurement on a bounded range, but a ten-entry table with an unexplained stride is exactly the shape that looks like a structure while being something else, and the rule in AGENTS.md is that a pattern gets looked at before it gets named. |
||
|
ReaperOfVeriod
|
352a4e5ad7 |
Locate the region after shrub, 690 KB on level 0
Range measured: each level's last shrub class data_offset field (class + 0x2c) through to the start of the constant trailer. Nothing outside that range is read. Method: the last shrub class's body is an array of 8-byte records whose final two bytes are 00 00. Scanning forward in 8-byte steps for the first record whose trailing pair differs locates the body's end. That is a search for a stated pattern rather than a statistic, so it either finds the boundary or says it did not. Result: on 24 of 24 levels the boundary lands exactly on an 8-byte record boundary, with body lengths of 192, 208, 224 or 240 bytes -- the same sizes seen when shrub classes were grouped and diffed. Level 0: last shrub class data 16046800 .. 16047040 240 bytes next region 16047040 .. 16737344 690,304 bytes constant trailer 16737344 .. 16969296 231,952 bytes So 690,304 bytes between the last shrub class and the trailer are a distinct region rather than shrub data. What is in them is not established. They are the leading candidate for moby_anim, now that collision..moby has been ruled out as texture. Worth recording how a wrong conclusion nearly happened here. An earlier probe searched for PS2 NaN filler at the start of this range, found none, and was about to report the body as having zero length. That would have been wrong: the last shrub class is a real class with real data, and NaN filler only appears in degenerate ones. Checking the actual byte pattern instead of concluding from a failed heuristic is what caught it. |
||
|
ReaperOfVeriod
|
6afcfa8608 |
Require bounds and method on every reported number
Four numbers in one session looked load-bearing and turned out to be artifacts: a format identified as VIF-packed on the strength of recurring byte patterns where a control put it on the random baseline, a body size computed over wrongly bounded spans, a "6.2 MB of unidentified region" that was texture data, and "3 unexplained blocks" that were a sampling artifact. All four shared one cause -- measuring before bounding, naming bytes before looking at them -- and none errored, which is why they survived long enough to be believed. Two of them needed follow-up commits to undo. The rule: any size, count or distribution comes with the byte range it describes and how it was measured. Ranges come from core_layout rather than from running to core_data.len(). A statistic that cannot separate your format from uniform random cannot tell you anything. And counting distinct byte values only tells you which blocks to look at, not what they are. Worth having in the working agreements rather than in anyone's memory. |
||
|
ReaperOfVeriod
|
8d57a09b76 |
Bound the last region by core_data's constant trailer
Comparing all 27 levels byte-for-byte from the end: the last 231,952 bytes (0x38A10) of every level's core_data are identical. Level-specific content therefore ends at len - 0x38A10, which is a real bound for the final region and is not core_data.len(). This closes shrub. Across the 24 levels that have a shrub region the bound gives 24,442,928 bytes total, and the disc test asserts on every level that the region ends strictly inside the lump rather than at its end. What the trailer is remains unestablished -- fixed data, or padding from a fixed-size decompression buffer -- but it is constant, which is all a bound needs. The data between the last shrub class and the trailer is high entropy: 183 to 256 distinct byte values per 4 KB block, mean 254, where texture data sits under 200. It is real content of unknown type and is the most likely home for moby_anim, now that the range between collision and moby has been ruled out as texture. A lump shorter than the trailer now yields end: None rather than a backwards bound, covered by a test. |
||
|
ReaperOfVeriod
|
8882d8fc26 |
Re-scan at finer granularity: collision..moby is entirely texture
The "3 of 21 blocks not texture-like" figure from the last commit was an artifact of sampling every 256 KB and missing everything between. Re-scanned at 4 KB granularity there are about 100 such blocks, and looking at them rather than only counting them, they are smooth gradient ramps -- sky and fog texture. A noisy gradient has a high distinct byte count without being code, which is exactly the mistake the byte-count heuristic invites. So collision..moby is entirely texture data. There is no separate moby_anim region there, and two claims about this range have now been wrong: the "6.2 MB of unknown data" and the "3 unexplained blocks". Both came from sizes and counts taken without looking at what they described. moby_anim is therefore still unlocated, but it is not here. The remaining candidate is the unresolved tail after the last shrub class -- ~925 KB of real, non-zero data on level 0 that nothing accounts for. Recording the two corrections together rather than one at a time, because the mechanism is the same in both: a heuristic count is not an identification. |
||
|
ReaperOfVeriod
|
db414b3d5f |
Correct: the gap between collision and moby is texture data
The previous commit claimed ~6.2 MB of unidentified data sat between the collision header and the first moby class, and speculated it might be moby_anim. That was wrong, and it came from quoting a size before bounding the thing it measured. Looking at the bytes rather than only the offsets: the range is dominated by 8-bit indexed texture data. Byte-value concentration distinguishes them cleanly -- a texture block draws on a small palette, code and collision data does not. 18 of 21 sampled 4 KB blocks fall under 200 distinct byte values, against 249 for tfrags and 209 for collision. The real layout on level 0: collision 2615232 .. 3518544 (header plus its declared 903,248 payload) textures 3518848 .. ~8832192 moby 8832192 .. The texture table's data_offset values stop at 5326208, which is what made the gap look genuine. Mip chains extend well past the last entry's offset, so sizing a region from a table of first-level offsets understates the data it points at. Three of the 21 blocks are not texture-like, so something is mixed in that is still unidentified. It is not 6.2 MB of moby_anim. This is the third number this session that looked load-bearing and turned out to rest on an unbounded measurement. The pattern is consistent enough to be worth stating plainly: sizes derived from tables or offsets are not evidence until the bytes underneath have been looked at. |
||
|
ReaperOfVeriod
|
8b98d388f7 |
Show region bounds in probe, which exposed an unnamed 6.2 MB region
Printing the region table next to the header fields makes the gap visible without having to remember to consult core_layout, and it paid for itself immediately. On level 0 the collision region comes out 6216960 bytes, from 2615232 to the first moby class at 8832192. Collision's own header declares a payload of 903,248. So roughly 6.2 MB sits between the collision header and the first moby class, and almost none of it is collision. The successor rule is correct -- a region does end where the next one begins -- but "the next" is not yet known there. collision..moby is really collision + <something unnamed> .. moby. The name is wrong, not the bound. Candidates for the space, rough order of likelihood: moby and tie model animation data, which is what moby_anim would need, then volumes, then per-class runtime structures. Tfrag texture texels are already placed and sit earlier in the lump. Worth resolving before starting moby_anim, since that format is the most likely tenant. Until then, any probe reading between the collision header and the first moby class is reading something the region table does not describe. |
||
|
ReaperOfVeriod
|
cb38b8c279 |
Compute core_data region bounds once, from successor boundaries
Region spans were being re-derived per format while probing, usually running from one known offset to the next with the last span running to core_data.len(). On level 0 that gave the final shrub class a 925,008-byte span absorbing most of the lump's tail, and every aggregate computed over it -- record counts, value distributions, "this looks random" -- was silently wrong. Nothing errored, which is what made it survive two commits and a correction. core_layout computes the layout once: each region ends where the next one begins, class regions start at the lowest non-zero geometry offset in their table, and a region whose successor is unknown gets end: None rather than a clamped guess. Zero class entries mean absent, not position zero -- the same trap occlusion and collision each fell into separately. The disc test asserts what the module is actually for: that the unknown is visible. Shrub's end is unresolved on the 24 levels that have a shrub region, and that is asserted rather than hidden. It also found that not every level has every region -- level 10 has neither occlusion nor shrub -- so the test checks ordering against the canonical sequence instead of demanding an identical set. Still open: where the shrub region ends. Texture texels cannot bound it; they sit before moby, not after shrub. See docs/formats/core-data-regions.md. |
||
|
ReaperOfVeriod
|
aec462df21 |
Record that shrub span bounds are wrong, and what that invalidates
A follow-up measurement contradicted the body size implied by the differential analysis, and the cause is bad span bounds rather than anything in the format. Shrub spans were bounded by the next shrub offset, with the last running to the end of core_data. On level 0 that gives the final shrub class a 925,008-byte span, absorbing most of the lump's tail. The moby-to-tie boundary is clean by comparison: last moby offset 13414592, first tie 13416192, exactly 1600 bytes apart. What that invalidates: - The 3.5M record / 28 MB body figure is an artifact of the oversized final span and should not be quoted. - The "payload looks uniform-random" measurement ran over the same contaminated range, so it says nothing about shrub geometry either way. - The differential result itself still stands -- it was computed on groups, and the groups used had bounded spans. The global aggregates built on it do not. What is needed is the offset ending the shrub region. It is not in the core index, and there is no obvious successor boundary the way moby->tie has one. This is the second confident-sounding number in this format that turned out to rest on a bad bound, the first being the VIF reading. Aggregates across many spans are only as good as the span bounds under them, and bounds are the easy thing to get wrong because nothing errors when you do. |
||
|
ReaperOfVeriod
|
52228c14ed |
Retract the VIF reading of the shrub body
The previous commit concluded the shrub body's 6-byte payload was VIF-packed, on the strength of recurring patterns like 7ea5 and 1ab6. That was pattern matching dressed up as a conclusion, and a control kills it. Decoding every 32-bit word of every shrub body as a VifCode: shrub bodies 24.9% unpack, 128 distinct cmds uniform random 25.1% unpack, 128 distinct cmds sound_bank (definitely not VIF) 19.0% unpack, 122 distinct cmds gs_ram (mixed) 13.6% unpack, 128 distinct cmds moby class 0 span (known VIF) 40.1% unpack, 105 distinct cmds Shrub sits on the random baseline and exercises all 128 possible command bytes, which is what unstructured data looks like. Worth being precise about what this does and does not show. The test is weak -- even the known-VIF region only reaches 40.1%, well short of a clean signal -- so this does not prove shrub is not VIF. It proves the claim was never established, which is the part that was wrong. Spotting three recurring byte values and naming them a format is the same mistake as the three stride guesses it was meant to replace, one level down. A hypothesis needs a discriminating test before it goes in a commit message, and I skipped that step because the finding was welcome. What still stands is the part that rests on differential analysis rather than recognition: 8-byte records, 6 varying and 2 constant, 144 varying bytes per group regardless of count. The payload's encoding is unknown. |
||
|
ReaperOfVeriod
|
572753e7c1 |
Find the shrub body stride by diffing spans instead of fitting it
Stride-fitting the shrub body failed three times, so this attacks it from the other side: group spans by (count, data_offset, tail length) and byte diff every member of each group. Positions that differ across a group are geometry; positions that never differ are structure. The answer is blunt. 144 bytes vary in every group, whether that group has count 1, 2, 3, 5 or 6 and a tail of 192, 208 or 240 bytes. The varying data is not sized by count at all, which is why three stride guesses were fitting the wrong thing -- and why they kept looking nearly plausible. Those 144 bytes fall in runs of 6, every 8 bytes. The body is an array of 8-byte records: 6 varying bytes and 2 constant. The payload is not half-floats -- decoding as half3 gives 245.25 and -168.375, which no shrub has. The patterns (7ea5, 1ab6, 43d9, af80) and their repetition are VIF-packed data, the same quadruple-word packing moby and tfrag use. Filler around it is the PS2 NaN 0x7fff0000 rather than zeros, which is why the body read as noise in the earlier dumps. So the open question is the VIF code assignment, not the stride, and the decoder written for tfrag is the obvious place to start. |
||
|
ReaperOfVeriod
|
517f0b63fa |
Read the shrub header floats, including a denormal the PS2 flushes
Extended the shrub header past the two self-identifying words. The first four words are floats -- finite on all 358 spans -- though they do not behave like a min/max bounding box, which orders correctly on only 146 of 358, so they are recorded as bounds rather than named. The word at +0x10 is positive everywhere, 3.0 to 20.0, exactly 5.0 on 281 of them. The word at +0x14 is the interesting one. It is zero on 342 spans and the bit pattern 04000000 on the other 16, which is a denormal of about 2.8e-45. The Emotion Engine has no denormals, so hardware reads all 358 as zero. That is only the second reading. My probe reported the field as 0.000 everywhere because it printed to three decimals, and the test caught it: level 7 class 7 fails a strict zero check. The lesson is that "exact" in a measurement means exact in the measured precision -- rounding in the output made a real distinction invisible. It also matches AGENTS.md on modelling the FPU rather than correcting it: this field looks like noise on disc and is a constant to the machine that reads it, and the parser accepts either rather than picking one. The shrub body past +0x2c is still not decoded. |
||
|
ReaperOfVeriod
|
39c364e874 | Note in the roadmap that shrub header fields are checked, not just reported | ||
|
ReaperOfVeriod
|
7107f395d0 |
Parse the shrub class header
Shrub is the third class-geometry family and is not the moby one: all 358 spans on the disc have a leading word of 0 or one too large to index a table into, so none contain a moby-style part table. The rule that holds for 43,467 of 43,467 moby parts holds for zero of them. Two header fields do self-identify, exactly, on all 358 spans: +0x24 equals the owning class entry's own o_class, and +0x2c is a data offset landing inside the span. Both are checked rather than merely reported -- a span naming the wrong class is a different thing, and quietly returning the mismatched value would hide that. The body past +0x2c is not decoded. It tiles at no stride consistent with the count at +0x28 (tails of 240, 240, 192 and 224 bytes against counts of 5, 6, 5 and 3), and the header's leading bytes are packed rather than float even though +0x10 lands exactly on 5.0f on 281 of 358 spans. Shrub class entries are 0x30 bytes rather than 0x20, because they carry billboard texture fields the other two tables lack; the parser records that so the stride is not re-derived. |
||
|
ReaperOfVeriod
|
f6bbbf8d5f |
Add gc-extract probe to dump structures through the real parser
Reading these formats by unpacking raw offsets out of core_index.bin has cost real time. The index interleaves single-word region offsets with count/offset pairs, so miscounting one shifts every field after it: that produced a page of implausible collision headers on all 27 levels, sent me chasing a tie-breakoff that turned out not to exist, and nearly had me "fix" a stride bug in code that was already right. probe dumps a level's core header and all three class tables through LevelCoreHeader, so a field name printed is the field name the parser uses. It also reports how many entries carry geometry inside core_data, which is what pins the class table strides rather than guessing at them: shrub reads 7/7 sane at 0x30 against 4/7 read at 0x20. Reading a structure by hand in a shell still works; this just stops it being the default, and stops the offset arithmetic being re-derived every time a new region is investigated. |
||
|
ReaperOfVeriod
|
c4ba84232a |
Record that tie does not reuse moby's part table
Checked the obvious hypothesis -- that tie, being the other class-geometry table, would share moby's part-table format -- and it does not. The tiling rule vertex_offset + 16 * vertex_count == next_part holds for 0 of 25,024 records across the 1,864 tie spans, against 43,467/43,467 for moby. Tie's second field also reads as an absolute offset rather than a span-relative one, which is the first sign the record shape differs. Past the verified 0x48-byte header the bytes are f32-shaped rather than the expected VIF marker, but they are not a fixed array either: the run of consecutive plausible floats varies from 1 to 6 across spans (822 spans have 6, 455 have 1, 373 have 2). So there is more header past 0x48 and it is not one array of distances -- the floats after that point are a different population from the LOD distances before it, including negative values. Recorded so the next attempt starts from that rather than re-trying the moby template. |
||
|
ReaperOfVeriod
|
fc4c8b427d | Mark shrub partial in the roadmap | ||
|
ReaperOfVeriod
|
3ca8522be1 |
Record shrub geometry findings and the per-table entry strides
The three class tables do not share an entry size. Measured by how many entries give a geometry offset inside core_data plus a plausible o_class, over all 27 levels: moby stride 0x20 5769 / 5796 tie stride 0x20 1864 / 1864 shrub stride 0x30 356 / 358 Reading shrub at 0x20 still yields 231 sane-looking entries out of 358, which is the hazard: wrong often enough to look right. The committed code already handles this -- ShrubClassEntry is 48 bytes and carries billboard texture fields the other two lack -- so this is measurement, not a fix. Shrub geometry is not moby's part-table format: all 358 spans have a leading word of 0 or one too large to index a table into, so none contain a moby-style part table. Two header fields do self-identify, both exact across all 358 spans: +0x24 equals the class entry's own o_class, and +0x2c is a data offset landing inside the span. +0x10 reads 5.0f on 281 of 358, so it is a value rather than a marker and carries no structural weight. Rest of the shrub header undecoded. |
||
|
ReaperOfVeriod
|
bbb10c678d |
Parse the moby per-class part table
A moby class span does not start with its geometry. The first u32 is the offset of a part table, and the table is 16-byte records of vertex offset, vertex count, an unestablished second count, the next part offset and flags. Parts tile: vertex_offset + 16 * vertex_count == next_part. The stride is never assumed anywhere -- it falls out of that identity, which is what pins the record size and the field boundaries. The identity holds for 43,467 of 43,467 part records across every moby class on all 27 levels, and the parser rejects any span that violates it rather than returning a plausible wrong answer. Two terminations are needed. A zero vertex_offset ends the run, but 172 records on the disc have a zero vertex_offset and a zero next_part while still carrying a vertex count. Stopping only on the first leaves 99.6% instead of 100% and quietly includes a second table as if it were parts. 1,270,802 vertices over 4,560 class spans; 116 spans carry no geometry. Vertices are 16 bytes of VIF-packed half-float quads cycling through four distinct words. Handed back raw -- unpacking them is separate work from finding them, and second_count is recorded but not guessed at. |
||
|
ReaperOfVeriod
|
1f96ef09b9 |
Record core index field order, sky and moby findings
Three things this session that are cheap to rediscover wrongly and expensive to rediscover wrongly twice. The core index interleaves single-word region offsets with count/offset pairs, so gs_ram and the three class tables are two words each while tfrags..collision are one. Treating the pairs as single words shifts every field after them, which is how collision got read as sky and produced a page of implausible collision headers on all 27 levels. Sky is not uniform. Most levels share a header shape, but level 2 carries extra fields and the count word at +0x0a over-reports against the number of real offset slots, so no single fixed header fits. Moby geometry starts with zero words rather than a header and then packed half-float VIF data. Not decoded; recorded so the next attempt starts from the data instead of re-deriving the layout. |
||
|
ReaperOfVeriod
|
dc941a8eec |
Parse the collision container framing
Collision sits inside core_data at the offset the core index gives, behind a 0x40-byte header that self-describes: it carries its own size and the payload size. That pairing is the invariant -- the declared header size is exactly 0x40 on all 27 levels and header + payload stays inside the lump. 41,481,520 payload bytes across the 19 levels that have any; 8 declare zero length, which is present-but-empty rather than absent, so the two are counted separately. The cells inside the payload are not decoded. They do not walk as fixed-stride cells, and treating faces as variable-length records stops partway into the first cell on level 0 rather than tiling the payload. The first cell reads as 11 verts and 33 faces under every hypothesis tried, which is exactly the kind of agreement that looks like a result and is not one -- three different strides all produce the same first cell and all of them are wrong. Left open rather than committed as a guess. |
||
|
ReaperOfVeriod
|
2dcc6e9710 |
Parse the sound bank container framing
A level's sound extent is two regions: an `SBlk` descriptor table and the sample data that follows it. Four u32 at the head describe the layout, and the invariant that establishes it is sblk_offset + sblk_size == daw_offset, which holds on all 27 levels. That is what pins the field meanings; the numbers alone are ambiguous about which of them is the offset. DAW is a format tag inside the SBlk header, not a region signature. It occurs exactly once in the entire extent, at sblk_offset + 12, naming the sample format -- the sample data itself starts at daw_offset with no signature at all, roughly 19 KB away on level 0. Reading it as a region magic is the obvious reading and it is wrong. daw_size never reaches the end of the file. The tail is positive on all 27 levels, 232 to 1,848 bytes, so the field describes less than the region it names; the test asserts that direction rather than inventing a size that fits. 523,720 SBlk and 37,649,008 DAW bytes across the 27 levels. The sample format inside DAW is not decoded. It looks like VAG/ADPCM from the nibble packing, and it has no bearing on drawing anything, so it stays open rather than getting a half-answer. |
||
|
ReaperOfVeriod
|
cfabe9a0ae |
Fix mask bit lookup to match its documented encoding
visible() documented its argument as `byte << 8 | 1 << bit` -- high byte selects the mask byte, low byte is already the shifted bit -- then shifted the low byte a second time as though it were a bit index. That is wrong twice over. It inverts the meaning of every bit argument, and for any low byte above 8 it panics in debug, because `1u8 << 32` is an overflow. Real data reaches it: the folded bits for the upper bytes of a mask are exactly the values that trip it. Treat the low byte as the bit mask, which is what the doc says and what the renderer needs. Added an assertion for a mask index that does not exist, since that was the case that exposed the mismatch. |
||
|
ReaperOfVeriod
|
632505a6bd |
Parse the occlusion PVS grid
A z -> y -> x tree of u16 node offsets over 4x4x4-unit cells, each populated cell naming one 128-byte mask of 1024 visibility bits that the renderer fills from the camera's cell each frame. Two sentinels carry meaning and one of them is easy to miss: a z or y slot of 0 means "no subtree", and an x slot of 0xffff means "no cell". Reading the latter as a mask index gives a grid whose highest mask is 65535 on every single level, which is the tell. The check that makes this real is the mask array. It starts at masks_offset and runs to (highest_mask + 1) * 0x80, and on every level with a grid that ends at the block boundary exactly -- widest unexplained tail across the disc is 0 bytes. A wrong tree shape or cell size would leave the highest mask index elsewhere entirely. 412,521 populated cells over 140,939 masks, across the 24 levels that have one. An offset of 0 means the level has no occlusion data at all, which is not the same as "at the start of the file" -- that is where the tfrags block lives, and parsing it as a grid produces a mask array that fits anyway, with 1600 bytes to spare. That is what made level 10 look plausible on the first pass; three levels (10, 24, 25) carry none. |
||
|
ReaperOfVeriod
|
79025b22d6 |
Decode the tie class header prefix that RAC1 and RAC2 agree on
Tie is the easy one of the three geometry subsystems -- instanced static props, no skeleton -- so it is the right place to start. RAC1's class header is 0x80 bytes. RAC2 agrees with it over the first 0x48 and diverges after. From 0x48 the RAC1 layout does not describe RAC2 data: lod_info at 0x50 reads back as noise, packet_count and texture_count are zero on every class examined, and the normals offset at 0x0c points 117,788,434 bytes into a 16.9 MB lump. Only the agreed-on prefix is decoded. What makes it trustworthy: * o_class at 0x44 equals the core index's class number for all 1,864 tie classes across all 27 levels. A field landing on the right number that often is not a coincidence, and it pins the header base. * The LOD distances are ordered 0 < near < mid < far for 1,784 of 1,864. The 80 exceptions are real, unexplained, and are reported rather than asserted away. Level 0's first tie class has distances 12.0 / 25.0 / 40.0, a plausible prop LOD band, and packet offsets 256, 20,592 and 26,048. The rest is deliberately not guessed at. A field order that produces plausible output and is wrong is the specific failure this project keeps recording, three times now: the gameplay extent, the tfrag record framing and the class tables were all "reasonable" and all wrong. |
||
|
ReaperOfVeriod
|
4c4a78d402 |
Identify the container the class tables point into: core_data itself
assets_compressed_size and assets_decompressed_size at index offsets 0x88 and 0x8c are 9,738,678 and 16,969,296 on level 0 -- exactly the sizes of core_data before and after decompression. There is no separate asset container. The "asset WAD" is core_data, and offset_in_asset_wad indexes the decompressed form directly. Checked rather than assumed, on all 27 levels: the decompressed size equals core_data's exact length, every class offset is either 0 or inside it, and the moby, tie and shrub spans do not overlap each other. An offset of 0 means the slot has no geometry, not the start of the lump; level 0's ninth moby class is 4909 at offset 0. Two assumptions worth discarding while here -- the tables are not sorted by class number (level 0's run 0, 1, 2, 14 ... 4909) and the offsets within a table are not monotonic. Neither is needed, but both look true from the first few entries. The chunk descriptors after the named extents are not it: they exist on only 3 of 27 levels, because level 0's four named extents tile the container exactly. An asset blob there would be missing almost everywhere. What sits at the offsets is raw GS/VIF packet data rather than a further container -- moby[0] begins 00 00 00 00 3f 16 02 55 6f 6f and tie[0] 00 01 00 00 70 50 00 00, which read as packet headers rather than a magic number. None of the three is decoded yet and this does not start that. Moby is skeletal and is the largest single piece left before the rest of a level can draw. Also: the pipeline now writes each level's chunk descriptors where they exist, and LevelCoreHeader decodes the asset lump's two sizes. |
||
|
ReaperOfVeriod
|
e1105543e4 |
Parse the level core class tables: moby, tie and shrub
These were ruled out for want of an entry size. The size is knowable exactly, because each table ends precisely where the next begins: moby 238 entries at 192, 32 bytes: 192 + 238*32 = 7808 = tie's tie 74 entries at 7808, 32 bytes: 7808 + 74*32 = 10176 = shrub's shrub 7 entries at 10176, 48 bytes: 10176 + 7*48 = 10512 = textures That closes on 27 of 27 levels, which is what makes it evidence rather than a coincidence: any wrong entry size breaks the chain immediately. It is the same "does this hold on every level" discriminator the project already uses for the container pair table. Across the disc: 5,796 moby, 1,864 tie and 358 shrub classes, plus the 1,595 tfrag textures. Class entries are 32 bytes -- asset offset, class number, two unknown words and 16 bytes of texture slots -- and a shrub entry adds 16 bytes of billboard slots for 48. The header now decodes through to the texture tables rather than skipping 0x18..0x30 wholesale, which also recovers the moby, tie and shrub texture ranges. Two more pairs sit between shrub textures and the textures base offset. Two things worth recording. ClassEntry::parse read its 16 texture bytes before its four i32s, because the struct literal that uses them comes after the loop that advances the cursor -- so every entry was parsed backwards. Struct literal field order does not reorder the reads that already happened. And every shrub class on level 0 carries all-zero billboard slots, so the dimensions are optional rather than something worth asserting on. This is the third instance of the same failure shape. The gameplay extent was "an unidentified compression" because it was parsed compressed; the tfrag record framing was "unrecoverable" because the entry size was never known. Both looked like properties of the format and were properties of the question. Also records that point lights are a runtime feature rather than missing work: every one of the 18,102 patches has point_lights == 0xffff and the eight-slot bank is filled by gameplay, so there is nothing for them to light in a static level render. And that PCSX2 is not installed, so the comparison loop cannot be built yet -- which is why lighting magnitudes stay unverified. |
||
|
ReaperOfVeriod
|
b8b4a1a3d3 |
Consolidate: tidy level.rs, correct stale docs, pin hardware models
No behaviour change. This is the pass over the last stretch of work to make sure the accumulated result hangs together rather than only each commit in isolation. Code. lit_color had grown a `let _ = (depth, lod_distances)` and a dead branch from repeated edits. The directional pass genuinely does not depend on distance, so the parameters are gone rather than discarded, with a note that the point lights -- not implemented -- will need them. lit_at folded into lit_color, which had become a wrapper around it. Checked for leftover scaffolding: only two env vars remain, both real (GC_ENGINE_SHOT and GC_ENGINE_DEBUG_BOUNDS); GC_ENGINE_UNLIT, GC_ENGINE_DEBUG_COLOR and GC_ENGINE_ABL were never actually written, because those edits silently failed to apply against rustfmt'd source. AGENTS.md. The crate table did not say that gc-engine owns hardware models. ps2.rs is the Emotion Engine's FPU and the lighting pass runs on it, so the boundary is now stated: gc-formats decodes inputs, gc-engine reproduces how the hardware computes with them. That is the distinction rule 2 depends on, and it was implicit. Docs. Three claims had been overtaken by what we learned and were actively misleading: * docs/formats/level-wad.md still said gameplay "does not decode with the WAD decoder". It does; it is a WAD stream. * docs/plan/level-assets.md carried a whole section, "a compression we have not identified", with a ruled-out list including "bit order", "header length" and "the WAD format at all". All three were ruled out against the wrong input. Replaced, with the failure mode kept: a uniform failure across 27 inputs was read as "the decoder is wrong" when it meant "the data was handed over uncompressed", and the positive control -- decompress and look -- was never run. * The same file still said the tfrag "record framing is not recovered". True, but it was a false lead that has since been superseded by the VIF1 framing. Kept as a recorded wrong turn rather than deleted. Also refreshed "Still open" to name what is actually outstanding: point lights, the GS register set, the guard-band clip test, the unread core_index class tables, and -- stated plainly -- that lighting magnitudes are unverified because there is still nothing to compare against. Verified after the changes: 148 tests pass, clippy and fmt clean, all 5,213 extracted files match the hash table, and level 0 still loads at 26,758 triangles with the same LOD split and fully lit. |
||
|
ReaperOfVeriod
|
bcf0d1f0cc |
Decompress the gameplay extent: the level light bank now parses
gameplay is a WAD stream, exactly like core_data. The extractor was writing it compressed, so the light bank framing read the WAD header's stream length as a light count: level 0's offset at +0x04 was 0xa0d and the "count" there was 698810368. The file says so in its first six bytes: 57 41 44 79 0d 0a -- "WADy". Decompressed, level 0's gameplay extent is 1,872,256 bytes, the offset at +0x04 is 0xc2b20, and the count there is 5. Five directional light sets. All 80,274 vertices of level 0 now go through LightTfrags, and the renderer's fallback line no longer appears. The frame's colour distribution goes from 295 to 737 distinct sampled colours and takes on the level's warm cast. This corrects a recorded negative result that was simply wrong. docs/roadmap.md carried "gameplay does not decode with the WAD LZ77 decoder, all 27 levels fail identically, so the input is wrong rather than the decoder", with bit order and header length also ruled out. That was true of the data being parsed and wrong about the cause: the decoder was never the problem, it was being handed a file that had not been through it. Same lesson as the origin bug, from the other direction. A recorded failure without a positive control is easy to record and easy to leave; the control here is trivial and should have been run first -- decompress and look. The renderer keeps counting and reporting how many vertices it lit, because a silent fallback to stored colours looks plausible and is wrong. Lighting magnitudes remain unverified: that needs comparison against reference frames via the PCSX2 loop, which is still to come. |
||
|
ReaperOfVeriod
|
65b5d366de |
Implement the PS2 float model and the LightTfrags pass
Two pieces, and the first is the reason the second can be trusted. ps2.rs is the EE's FPU on raw u32 bit patterns. It is not f32: there are no denormals (exponent 0 *is* zero), no NaN and no infinity (exponent 255 is an ordinary value, so MAX reads back in Rust as NaN -- a test asserts exactly that, because it is the sharpest available statement of why this module exists), every result truncates toward zero, the adder discards the small operand's bits more than one below the large one's LSB and returns the large operand unchanged across a 25-bit exponent gap, and rsqrt is sqrt-then-divide with truncation between the steps rather than one rounded operation. The lighting pass accumulates all of these across every vertex of every patch, so this is a visible difference rather than a rounding curiosity. tfrag_light.rs is the pass itself: PEXT5 base colour, the select word picking one light set or blending two by bits 8..15 / 256, blended directions re-normalised where unblended ones are not, and base + A.rgb*max(dA, dA*wA) + B.rgb*max(dB, dB*wB). Lighting indexes positions, not vertex-info entries, since a common entry and a LOD entry can share a position. TfragLight gains a `select` field for the u16 at +6. It was briefly reconstructed with an unsafe pointer cast over the struct, which is exactly the sort of thing that breaks silently the moment a field is reordered; it is now a plainly parsed field. The pass does not run yet. The light bank framing the reference uses -- a u32 offset at +0x04, an i32 count there -- gives offset 0xa0d and a count of 698810368 on RAC2 level 0, so RAC2's gameplay extent is not laid out that way. The renderer now counts the vertices it actually lit and prints it, because falling back silently to the stored colours produces a plausible but wrong picture: lighting: 0 of 80274 vertices shaded by LightTfrags; the rest fall back to the tfrag's stored colours (no usable light bank) That line is the current honest state, and the frame has not changed as a result. Where RAC2 keeps its light bank is the open question. |
||
|
ReaperOfVeriod
|
90f02890f8 |
Decode tfrag normals from the overlay's sine table
Normals are per-vertex, and the tfrag's light records store them as an azimuth and elevation *index* into a 256-entry sine table rather than as angles. This decodes that table and builds normals for the renderer. The reference reads the table from an ELF at BOOT_NORMAL_TABLE_VADDR (0x165500) and notes every level overlay carries a copy. The overlay has no ELF header -- it is a raw loadable image starting with code -- so there is no program header to map a virtual address through, and the offset is not fixed either: level 0 has the table at 0xb650c, level 5 at 0xcca30, level 26 at 0xb6284. So it is located by content. Every table starts with exactly (1.0, 0.0), angle zero, and the scan then requires all 256 entries to match cos(i*pi/128), sin(i*pi/128). Validating the whole curve is what makes the search safe: a lone (1.0, 0.0) pair of floats occurs by accident in a 2.7 MB executable, and a test plants one as a decoy. The values are read from the disc rather than recomputed, since those are the numbers the game indexes. A normal is (cos(az)cos(el), sin(az)cos(el), sin(el)) *negated* -- the stored normals point into the surface -- and then given the same Z-up to Y-up swap as positions, or the lighting comes out sideways. The azimuth and elevation fields are stored signed but index 0..=255, so entry 200 must be read as 200, not as -56; there is a test for that. Also frames and parses the level light bank from the gameplay extent, and the pipeline now writes overlay.bin. The per-vertex lighting evaluation is not written yet, so the renderer stays unlit by default. GC_ENGINE_LIT=1 substitutes a host directional light for inspection -- that is a stand-in, not the game's lighting, and it is deliberately not the default. The hash table is regenerated for the 27 new overlay.bin files: now 5,213 files and 5,762,497,559 bytes. |
||
|
ReaperOfVeriod
|
17c7e8ccc9 |
Draw tfrag per-vertex colour, unlit, and record why the frame is dark
The tfrag RGBA array now modulates the texture. It is indexed by *position*, not by vertex-info: several vertex-info entries can share a position, and the array is sized vert_count, which is a position count. Colour follows the same rule as the vertex itself. Morphing blends the position only -- the game keeps the high-detail colour -- while collapse replaces the vertex outright, colour included. Getting that the other way round would tint distant terrain. The material is unlit, and that is a decision rather than a shortcut. Level 0's colours really are dark (patch 0 runs (25,26,26) to (108,144,141)) and the game's lighting is what brightens them, so what is on screen is the true albedo. Worth recording: the obvious move -- add a directional light, or crank ambient brightness -- does nothing at all. With no normals the PBR path is degenerate, so the brightness change has no effect, and a host light would have hidden the gap rather than shown it. The colour data was always reaching the mesh; what is missing is lighting, not a material flag. Written up in plan/level-assets.md, including what LightTfrags actually needs: the light bank from the level's gameplay extent, the 256-entry normal table at BOOT_NORMAL_TABLE_VADDR (0x165500) which every level overlay carries, and the already-parsed azimuth/elevation light records. |
||
|
ReaperOfVeriod
|
192dcde247 |
Render a level: first drawable frame, and the origin bug it exposed
Adds gc-engine's binary and a level.rs that joins the format layer to the renderer: LOD selection and morphing per patch, then triangles grouped by texture so each texture is one draw rather than one per triangle. The PS2 world is Z-up at 1024 raw units per world unit, so the conversion is (x, y, z) -> (x, z, -y). The sign flip on the third axis turns the PS2's right-handed frame into Bevy's left-handed one rather than negating a determinant, which is what stops the level rendering mirrored. Morphing and collapse are kept distinct. Morphing blends a vertex toward the midpoint of its two parents across its tier's band; collapse replaces the vertex outright, position and UV, once it is past the far edge. Morphing is a blend, collapse is a cut, and conflating them makes distant terrain shimmer. Blending has to happen in raw units before the axis swap: mixing a converted parent with an unconverted child scales the blend by 1024. Level 0 renders as 26,758 textured triangles over 92 textures, correct UVs, with LOD selection live (26 patches at LOD 0, 650 at LOD 2 from the start camera). GC_ENGINE_SHOT writes one frame as a PPM and exits. Rendering immediately exposed a bug no test could. Positions are i16 offsets from a per-patch origin, and the obvious rule -- the last non-index STROW row -- is wrong on RAC2, which also uses STROW for per-vertex light records that come *after* the origin. Taking the last one yields [186, 186, 186, 186] and collapses the whole level onto the world origin. The entire level then fit in a 63-unit box and every test still passed: vert_count matched, tri_count matched, every parent and triangle index resolved, because all of those are relative to the same broken origin. Only a screenshot showed it. The rule that works is the STROW immediately preceding the first V3_16 position unpack; with it, level 0 spans x 86..444, y 46..211, z -477..40, matching the patch bounding spheres. That is worth more than the renderer: a parser can be completely self-consistent and still describe the wrong thing, and where a value is absolute rather than relative it has to be checked against something independent. The frame is flat and dark because there are no normals and no lighting yet. That is the next task -- the per-vertex light records -- not something to paper over with an exposure fudge. |
||
|
ReaperOfVeriod
|
71a3185fa5 |
Resolve tfrag LOD parent linkage: guard absent tiers by their counts
The four V4_8 parent arrays are classified by VU address, but the addresses collide: on level 0 patch 1, unk_indices_2_lod_01_addr == parent_indices_lod_0_addr == 291, and unk_indices_2_lod_0_addr == indices_addr == 294. Matching on address alone cannot separate them, and it fails silently -- the arrays still parse, they just hold the wrong numbers. Two corrections, and the first one is worth recording because it looked like a data problem rather than a code one: Each V4_8 element is 4 bytes and holds four indices, so an array of N entries is count*4 bytes, not count. An earlier analysis compared the unpack's element count against the position counts and concluded that 15,000 patches could not hold enough parents. That was the wrong unit; the data was fine. unk_06 and unk_0a suppress an empty tier. When the extra vertex-info count is zero there is no extra array and its address is simply the next tier's, so guarding those branches on the count lets the address fall through to the array that exists. That one change took resolved LOD-0 entries from 28,739 to 341,306. Result: of 886,841 vertex-info entries across all 27 levels, 516,492 are common and have no link by design, and all 370,349 LOD entries resolve -- 29,043 LOD-01 and 341,306 LOD-0 -- with zero failures. Each resolves to a real first parent, a LOD-01 parent that is always a common entry, and three positions inside the patch. A common entry never claims a parent. Morphing can now be completed: a patch knows which LOD to draw, how far to blend each vertex toward its parents, and where those parents are. |
||
|
ReaperOfVeriod
|
2a5307a157 |
Add tfrag LOD selection, morph weights and parent linkage
The block header's float at +0x08 gives L and the switch distances are [D0, D1, D2] = [6L, 4L, 2L] world units -- 192/128/64 on every level measured. The distance test uses the patch's bounding sphere truncated to integers against trunc(D * 1024), not the sphere's centre, so a patch straddling a threshold takes the coarser mode. Six modes result: LOD 2, LOD 1 collapse, LOD 1 morph, LOD 0 collapse, LOD 0 morph, LOD 0. A morph weight runs 0 to 1 across its tier's band. Past the far edge an entry is not blended but replaced by its first parent -- a collapse. The tested property is that distance never moves back to a finer mode, swept across the whole range, because that is what keeps terrain from popping as the camera moves. The four V4_8 parent arrays are classified by VU address and an unrecognised address is now an error, which is a real check: all 18,102 patches still parse. Every recorded parent resolves to real vertex info and real positions, and no common entry claims a parent. That is as far as it goes, and the limit is a finding rather than a rounding error. The addresses are not unique: on level 0 patch 1, unk_indices_2_lod_01_addr == parent_indices_lod_0_addr == 291, with unk_06 and unk_0a both zero, so nothing but unpacking order separates them. Across all 27 levels 57,782 of 374,349 LOD entries resolve, and 316,548 unresolved LOD-0 entries against 28,739 resolved is far too lopsided to be genuine -- either the arrays are misassigned where addresses collide, or RAC2's positions_lod_0_count means something other than RAC1's. Only 3,274 of 18,102 patches carry enough V4_8 bytes to cover positions_lod_01_count + positions_lod_0_count + extras. The vertex-info tiers themselves are confirmed: the addresses at +9 and +11 match the observed V4_16 unpacks and the three tier counts sum to vert_count on every patch, so the header layout is right and it is specifically the parent arrays that are open. Ruled out so far: address-only classification (collides) and one-parent-per-position (fails on 15,000 patches). Not yet tried: ordering by VU address instead of by declaration, and using unk_06/unk_0a to suppress an empty tier before falling through. Written up in plan/level-assets.md. |
||
|
ReaperOfVeriod
|
29fdf89f53 |
Decode level textures: 8-bit indices through a CSM1 palette
Adds texture.rs and makes the pipeline write gs_ram.bin, which it had been skipping. Nothing could be decoded before this: the CLUTs live there. A level texture is not pixels, it is palette indices -- width*height bytes at core_data[textures_base + data_offset], with colour coming from a 256-entry CLUT at gs_ram[palette * 0x100]. gs_ram is a byte range in the level's data extent header at offset 0x10, 518,656 bytes on level 0. Two PS2 conventions have to be honoured and neither is a design choice: within each 32-entry CLUT group the two middle 8-entry blocks are swapped (bits 3 and 4 of the index are exchanged, an involution), and alpha runs 0..=0x80 rather than 0..=255 with 0x80 fully opaque. The mip chain splits across two buffers: levels 0 and 1 are in the texture block with level 1 immediately after level 0, and levels 2 and 3 are resident in gs_ram at mipmap*0x100 and pad*0x100. Tfrag textures are square so every level halves evenly. Verification is two-layered, because every step here is a table lookup and a wrong CLUT remap still produces a well-formed image of the wrong thing. All 1,595 tfrag textures across all 27 levels decode; 96.8% are fully opaque and 96.3% use more than eight colours. On top of that, `gc-extract dump-texture` writes any texture or mip level as a PPM. Level 0 texture 0 comes back as a clean bevelled metal panel with rivets and specular highlights, and texture 74 as a vent grille whose punched windows are fully transparent. That tool is worth keeping: it is the cheapest way to tell a decode that is merely well-formed from one that is right. The hash table is regenerated for the 27 new gs_ram.bin files, now 5,186 files and 5,685,541,143 bytes. |
||
|
ReaperOfVeriod
|
48d9506cf1 |
Decode tfrag texture bindings, vertex colours and light records
Adds level_core.rs (the uncompressed core index header and texture table) and extends tfrag.rs with the data a draw call needs beyond geometry. An ad-gif is five A+D quadwords holding packed fields rather than GS registers; the level load rewrites them in place before drawing. On disc TEX0_1.data_lo indexes the level's texture table, TEX1_1.data_lo is the LOD bias K and data_hi the minification filter, and CLAMP_1 holds both wrap modes. Three things are easy to get wrong here: * K is a signed 12-bit value in 1/16 units, sign-extended by shifting the field up 4 and back down. Retail values are -137..-89 mip-scaled; reading it unsigned gives 1911 instead of -137. * TEX1_1.data_hi is MMIN, and 4 decodes as LinearMipmapNearest, not LinearMipmapLinear as the raw bit layout suggests. * RGBA and the light/normal records are not in the VIF lists at all. They sit at fixed offsets in the patch data block: rgba_size*4 colours at rgba_ofs, vert_count light records at light_ofs+0x10. The tfrag origin lives there too -- if no STROW row supplied one, the quadword at light_ofs is it. The texture table checks out on its own terms: 93 entries at index offset 0x2900 on level 0, and 0x2900 + 93*16 lands exactly on the next range. Verification: an ad-gif index outside the level's table would mean the geometry and the textures were misread independently, so every one is range-checked against real entries. All 27 levels: 47,950 ad-gifs over 18,102 patches, all in range, referencing 1429 of 1595 level texture entries (90%). Each patch's light array is exactly vert_count records and its RGBA array exactly rgba_size*4. Still open: LOD selection and morphing, and the texel data itself. |
||
|
ReaperOfVeriod
|
9459f15329 |
Parse tfrag topology: vertex info, strips, indices and triangles
Turns tfrag positions into drawable geometry. Every unpack in a patch's VIF lists is now classified by format *and* destination VU address, using the VU header at the head of the common list. That table is what tells the strips from the indices from the parent arrays: they are all V4_8 and only the address distinguishes them. Strip indices address the vertex-info array, not the position array. Several vertex-info entries can share a position, so an index resolves through vertex_info[i].vertex / 2 to reach a position. Vertex-info also carries UVs and the LOD parent link. A strip run of n vertices is a GS triangle strip of n - 2 triangles with alternating winding. The vertex count is biased by -128 when negative, and the first record of a list is always biased even when positive -- real patches open with -116 or -124, never a small positive number. tri_count lives in the patch header and cannot be derived from the lists without walking them, so reproducing it is a strong check: all 27 levels, 18,102 patches, 809,879 triangles, every patch matching its header, every triangle index inside its LOD's vertex-info range. Four wrong turns got there, each of which failed quietly rather than loudly: reading the element size from the immediate instead of the command, 2-byte payload padding instead of 4, ignoring the inline payload of STROW/STCOL/STMASK/MPG/DIRECT, and banking strips by arrival order instead of by LOD. Every one produced parseable, plausible geometry. Only the header's own triangle count caught them. Still open: ad-gif texture bindings, per-vertex RGBA and light records, and the LOD switch and morph rules. |
||
|
ReaperOfVeriod
|
36641b4cf8 |
Parse tfrag terrain blocks and decode all vertex positions
Adds gc-formats::tfrag and a gated real-disc test that decodes every terrain patch on all 27 levels. Two rules decide whether this works at all, and both were wrong on the first attempt: The vector format is encoded in the VIF command byte, not the immediate. Bits 2-3 are vn (component count minus one) and bits 0-1 are vl, giving element_size = ((32 >> vl) * (vn + 1)) / 8. Terrain positions are vn=2, vl=1 (command 0x69) at 6 bytes. Reading the format from the immediate's low nibble -- the more commonly quoted layout -- yields 12 or 16 bytes and misaligns everything after it. Unpack payloads pad to 4 bytes, and several non-unpack commands carry inline data: STROW/STCOL 16, STMASK 4, MPG num*8, DIRECT/DIRECTHL imm*16. Ignoring those desynchronises the walk, and a desynchronised walk still finds code words that look like positions. Positions are also not all in one list. vert_count is the total across three tiers, the LOD-2 list precedes the common one in offset order, and the LOD-0-only list starts at shared_ofs + lod_1_size*16 rather than at an offset of its own. Scanning only the common list matches about a quarter of patches, each exactly -- which is what made it convincing. The invariant is that every patch yields exactly its own vert_count. It caught both errors, so tfrag_positions returns None on a mismatch rather than partial geometry. All 27 levels: 18,102 patches, 821,439 vertices, zero mismatches. Still open: strips, triangle indices and the RGBA vertex-info arrays. Positions alone are not drawable. |
||
|
ReaperOfVeriod
|
3ec548efe8 |
Correct the terrain findings: region layout confirmed, record framing disproven
Two claims in the previous commit were wrong and are retracted here rather than
left standing for the next person to build on.
"382 records of 64 bytes" and "byte +3 is 0x48 in every record" were both
artefacts. The 0x48 at that offset is simply the most common byte in the region --
945 of 2000 sampled slots -- not a marker. Over 33,091 slots at that position
only 4,127 hold 0x48. The pattern holds for the first 382 records and then
breaks, which is exactly how a coincidence of a dense byte value masquerades as
structure.
"The coordinates are fixed-point s16 with a scale of 8" is also not supported.
It rested on one coincidence: 17,496 / 8 = 2,187, the planet radius. No grouping
of the record into coordinate triples produces points on a sphere of that radius.
Tested every stride from 2 to 16 bytes at every starting offset, with three and
four components; the best result is 5.0% of points on-sphere, which is chance.
What is now established, and holds on all 27 levels:
core_data region layout, as strictly increasing offsets
tfrag [0, occlusion)
occlusion [occlusion, sky)
sky [sky, collision)
collision [collision, textures_base)
Ruled out the competing reading where 0x0c/0x10/0x14 are sizes laid end to end:
the cumulative total misses textures_base on every level and the leading bytes
of each supposed region are not plausible structures.
Also new: terrain carries its texels inline. A scan for the PS2 texel pattern
18 18 18 80 finds 6,739 occurrences in 138 clusters starting at 0xd9f8, well
inside the tfrag region, so textures_base names something other than terrain
textures.
The vertex format remains unrecovered and the record says so. The next attempt
should look for a longer run of plausible vertices rather than re-deriving a
framing from one record.
|
||
|
ReaperOfVeriod
|
84d94337f7 |
Record the terrain record layout: 382 fixed 64-byte records
Continued on the tfrag terrain region and got a firm answer on the framing, which is what the next parser needs before it can read vertices. The region holds exactly 382 records of 64 bytes, from 0x40 to 0x5fc0; the 383rd breaks the pattern. Per-column analysis over 300 records separates marker bytes from data: +3 0x48 in 300/300 -- a marker, the high byte of the s16 at +2 +19,20,21,23,43,45 0x00 in 300/300 +52..55 ff 00 ff ff in 300/300 -- a fixed signature everything else varying Coordinates are fixed-point s16 with a scale of 8. That is not a guess: the value 17,496 appears in the first record and 17496 / 8 = 2187, the radius of a Ratchet & Clank planet to the unit. Read as f32 the data is nonsense, which is consistent with the earlier finding that no plausible float run exists anywhere in the 16.9 MB. Still open, and stated as open: which column is which of x, y and z, and whether 64 bytes is one vertex, two, or a header plus data. The ff 00 ff ff signature is more likely a record delimiter than a GS packet, since a VIF interpretation of the first tag does not chain. The next step is mechanical rather than exploratory: try every grouping of one record into 3-component vertices and keep the one whose points all fall within a sphere of radius ~2,187. Recorded in plan/level-assets.md. No code change. |
||
|
ReaperOfVeriod
|
85fa9d1432 |
Locate the tfrag terrain region and record what its records look like
The core header field at 0x08 is zero on all 27 levels, and the seven offsets 0x0c, 0x10, 0x14, 0x60, 0x64 and 0x68 are strictly increasing on every level. Reading them as a sequence of region boundaries gives the layout of core_data: tfrag [0, occlusion) occlusion [occlusion, sky) sky [sky, collision) collision [collision, textures_base) So tfrag = 0 is a real value rather than a missing field, and the terrain is the first 2.12 MB of level 0's geometry. The terrain begins with a short header followed by a repeating 64-byte record from 0x40. Established about those records: byte columns 3, 7, 11 and 15 are 0x48 in every record, which are high bytes of four s16 values giving a near-fixed Y on a planet of radius about 2,187; columns 16 and 17 increase steadily across records like a grid walk; and columns 48-51 carry a repeating ffff00ff pattern characteristic of GS packet headers. The record layout itself is not yet recovered, and the write-up says so rather than guessing. Ruled out and recorded: uncompressed f32 vertices (the longest plausible float run in 16.9 MB is a run of zeros in the collision block, so vertices are packed); GS packet tags at every position (0xE0..0xEF is frequent but uniform, which is what packed data looks like); and a 48-byte record stride (the pattern repeats at 64). Also confirmed this session: the core header base is offset 0 with certainty, because the compressed and decompressed sizes of core_data appear at the same offsets (0x88 and 0x8c) in all 27 levels' core_index. And the occlusion block at the header's 0x0c offset reads as an ascending u16 array of 65 values before restarting, consistent with per-cell visibility records. No code change: this is the reverse-engineering record for the next parser. 75 unit tests, clippy clean. |
||
|
ReaperOfVeriod
|
16c04db559 |
Fix level extents read as bytes, and extract the level geometry
A defect in the Phase 1 extractor, found by reading the data extent's contents: level extents are declared in sector counts, the generic G/ pair table uses byte counts, and level containers were being routed through the generic path. So LEVEL0.WAD's data extent -- 6,484 sectors -- was taken as 6,484 bytes. Every test still passed, because a truncated file is still a file. The extent's own header gave it away: it declares an uncompressed total of 13,277,920 bytes, and 13,277,920 / 6,484 is 2047.7. A ratio landing on 2048 is a unit error, not a compression ratio. Across 27 levels this had been discarding roughly 300 MB of level content. Level containers now go through the typed LevelHeader instead. Level 0 yields: data.bin 13,279,232 the data extent core_data.wad 9,738,678 the geometry, compressed core_data.bin 16,969,296 the geometry, decompressed core_index.bin 31,600 the geometry directory gameplay.bin 659,456 sound_bank.bin 1,503,232 core_data is a WAD LZ77 stream -- the format already implemented for the game -- and it decompresses cleanly. The first real level content this project has recovered. New: LevelDataHeader parses the data extent's own header, a table of byte ranges naming the level overlay executable, the geometry directory, the GS RAM image, the HUD banks and the geometry. Note this is a third meaning for a "size" field on the same disc, after sector counts in the level header and byte counts in the generic table; all three are documented together in level-data.md so the next parser does not rediscover this the hard way. The geometry directory's offsets were checked against the geometry rather than taken on trust: all twelve land inside the 16.9 MB of decompressed core_data, which is what confirms the header is being read correctly. The class tables themselves do not yet decode under the documented layout, and that is recorded as open work in plan/level-assets.md rather than papered over. 75 unit tests, 9 integration tests against the real disc. Hash table regenerated at 5,159 files and 5,671,915,799 bytes -- hashes only, no content. Extraction now writes to ~/Documents/gc-extracted rather than /tmp, which is 16 GB and did not have room for a 5.3 GB output. |
||
|
ReaperOfVeriod
|
a9e43e39c1 |
Parse the level container header, naming each level's four extents
The first typed structure read from real data. Every one of the 27 level containers begins with the same 0x60-byte header, which turns the generic pair table into named assets: data, sound_bank, gameplay and occlusion, plus three chunk descriptors. Two things about this header are worth stating because both are silent failures rather than errors. Sizes here are sector counts, not byte counts. The generic G/ pair table on the same disc uses byte counts, so both readings type-check and the wrong one is off by a factor of 2048. The tiling check settles it: read as sectors, the four extents tile every one of the 27 containers exactly; read as bytes they overlap each other. Level id is not container order. LEVEL21.WAD reports id 30, so a level was cut from the disc and its number left a gap. Anything keying off file order -- a save slot, the planet select screen -- would mislabel that planet. There is also a second trap in the same area: the ISO filesystem sorts names alphabetically, so LEVEL10 precedes LEVEL2, and any tool needing order has to sort numerically. The header's field at 0x04 is zero on all 27 levels. It is not the container size despite looking like one, so the code reads it as a runtime field and takes the container length from the ISO entry instead. Measured across all 27 levels, three invariants now hold in tests: extents tile in file order; extent coverage plus chunks reaches at least the end of the container (21 levels land exactly on its length, 6 leave 134 to 689 sectors of trailing data); and exactly one level's id differs from its container index. The coverage assertion is >= rather than == on purpose -- 21 levels end one sector past the file end because the header occupies sector 0, and asserting equality would truncate a level's tail. gameplay is left unparsed and that is recorded rather than hidden. It begins "WAD" plus a per-level byte and does not decode with the WAD LZ77 decoder: all 27 levels produce plausible output for the first packet and then fail on a rule the format's own encoder never breaks, which means the input is wrong rather than the decoder. docs/plan/level-assets.md records what is ruled out and what to try next. 66 unit tests, plus 7 integration tests against the real disc. The unpack hash table is unchanged, so this adds parsing without altering the pipeline. |
||
|
ReaperOfVeriod
|
027373e65b |
Add the disc pipeline: identify, unpack and verify
Phase 1 of the roadmap. The disc is now turned into verified asset files, and
the result is checked against a committed hash table rather than against
whatever happens to be on disk.
gc-extract identify disc.iso SCUS_972.68 v1.01 NTSC, ELF SHA-1 matches
gc-extract unpack disc.iso out/ 5135 files, 4,500,106,960 bytes, 10.8s
gc-extract verify out/ table.tsv OK: 5135 files match
All 92 G/*.WAD containers parse. MPEG1 and MPEG2 are raw video and pass
through whole; the other 90 are indexed containers whose extents are extracted
individually.
Formats recovered and written up in docs/formats/:
- iso9660.rs. R&C2 lists its assets in the filesystem, unlike R&C1, so a
directory walk is the primary asset index. Three details that all fail
quietly rather than loudly: directory records store every 32-bit field twice
(little-endian then big-endian), the CD001 signature is at sector offset 1
rather than 0x8001, and the directory flag at record offset 25 is
authoritative where the identifier's trailing slash is not.
- global_wad.rs. Container headers hold a table of extents. All 92 indexed
containers use {offset in sectors, size in bytes}. Measured, not assumed: the
sector-count reading places every entry of all 27 LEVEL<n> and all 27
AUDIO<n> containers out of range, while the byte reading places all of them
correctly. SCENE<n> is the one exception, a flat sector list whose extents
are sized by the next value.
- wad.rs. Insomniac's LZ77 stream: four flag-selected packet classes, with 0-3
trailing literal bytes per packet carried in unused low bits. This is not a
bit stream, which is why it resists the usual LZ77 reading.
- buf.rs. A bounds-checked reader, so untrusted 2003 disc data produces typed
errors rather than panics.
gc-extract carries the three stages and a SHA-1 implementation with FIPS 180-4
test vectors, so identification works without pulling in a crypto dependency.
Verification: 54 unit tests, plus four integration tests that run only when
GC_DISC names a disc. Those four check identification against the recorded
values, cross-check the Rust identifier against tools/identify-disc.py, parse
all 92 containers, and confirm a full unpack matches the committed table. The
committed table is 316 KB of sizes and SHA-1s for all 5,135 files: hashes only,
no game content, so it is safe to version and it turns "matches retail" into
something a test can assert.
docs/plan/disc-pipeline.md records two wrong turns, because both are easy to
repeat and both failed by producing plausible wrong output rather than an
error: the container table was read backwards, and the LZ77 stream was modelled
as a bit stream. The first survived because it happened to be correct on the
one container used for manual checking.
|
||
|
ReaperOfVeriod
|
a02f79a6d1 |
Initialise native Rust port of Ratchet & Clank: Going Commando
Sets up the foundation for reimplementing the PS2 game in Rust rather than emulating it. Nothing is implemented yet; this is the scaffold, the verified target and the working agreements that keep a long project coherent. Target disc, verified against a real image: SCUS_972.68 v1.01, USA Black Label, NTSC 60 Hz boot ELF 2,618,684 bytes, SHA-1 bacda860...d098a77, entry 0x00131AE8 The entry point independently matches the value recorded by the rac2-gc-decomp community reference, which cross-confirms the identification. This is the exact build the upstream research pins; Greatest Hits v2.00 is a different binary and is not the target. NTSC matters beyond label compatibility: PAL discs run 50 Hz, which would make chargeboots and other speed-sensitive movement run at 5/6 speed. Verifying the disc before writing movement code avoids discovering that in Phase 5. Contents: - Five-crate workspace (formats, extract, game, video, engine) on Bevy 0.19.1, matching the version ReRAC uses so its Graphics Synthesizer and VU work stays portable. `cargo check --workspace` is clean. - tools/identify-disc.py, which reproduces the identification against any disc image and distinguishes a truncated dump from a non-PS2 image. - .githooks/pre-commit, refusing game data from entering git history: disc image extensions, asset directories, container signatures at offset 0, and files over 50 MiB. Verified against seven cases including a disc image renamed to look like source. Now that this pushes to a remote, this matters more than it would have locally. - AGENTS.md, the operating agreements: no game data ever, reproduce the PS2 float model rather than approximating it, bound every read, verify against the real disc. - PROVENANCE.md, recording that this follows ReRAC (ISC) and learns from rac2-gc-decomp (MIT). Wrench is used as a written specification only and no GPLv3 code is copied, since that would relicense everything derived from it. RAC2Decomp is explicitly not used: its provenance is disputed in the upstream community reference, and it targets the wrong region. - docs/roadmap.md, phased so the riskiest unknowns are retired early. The two that matter: overlay relocation transfers from ReRAC and makes 27 level overlays tractable, while VU1 microcode is binary and is the hardest single item. Phase 5 is a deliberately scoped Veldin vertical slice as the go/no-go. |