ReaperOfVeriod/Going-Commando
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
129 Commits 1 Branch 0 Tags
54862b64ee1ed435ee73156e8c430fdfd7d726fe
Commit Graph
54 Commits
Author SHA1 Message Date
ReaperOfVeriod 54862b64ee Identify the tie record as RAC1's TiePacketHeader, and retract a formula 
The 16-byte tie record is R&C1's TiePacketHeader offset for offset: a data
pointer at +0, then R&C1's byte fields at +4 through +0x10. The identification
is not a resemblance but R&C1's own rule -- bfc_distance == 5 * shader_count --
holding on all 15,219 records of all 2,177 blocks on all 27 levels.

Measured over the same set, and reported rather than asserted where it is not
universal:

  bfc_distance == 5 * shader_count      15219 / 15219
  vert_ofs <= vert_size                 all
  scissor_size <= scissor_ofs           all
  control_size <= control_count         all
  color_ofs == vert_size                12640 / 15219
  color_ofs == vert_ofs + vert_size    0 / 15219
  color_count, strip_count, strip_vertex_count zero  15089 / 15077 / 15078

R&C1's rule for the colour offset holding on none of them is the interesting
line: the two games use that field differently, so it is named color_ofs for
what this game's majority reads as and the 2,579 exceptions are counted rather
than averaged away. TieRecord parses all twelve fields, invariants_hold asserts
only the universal four, and the disc test counts the rest.

A packet-length formula is withdrawn rather than softened. (vert_ofs +
vert_size + scissor_ofs + scissor_size) * 16 was measured at 61 of 61 pairs and
is not reproduced on the correct blocks -- 0 of 13,042. The block set was the
error, not the arithmetic: the script that produced the 61 read packets[2]
where packets[1] was meant, so it measured 461 blocks instead of the 2,177 the
class tables define, and the formula agreed perfectly with a population that
was not the tie blocks. TieRecord::packet_bytes goes with it, and the
retraction is recorded in the format doc and in the struct.

That misread also explains why the earlier RAM verification saw different
numbers for the two inputs -- same lesson as rule 3, arriving through a wrong
subscript instead of a wrong span.
 2026-10-13 04:22:38 +02:00
ReaperOfVeriod 4e90c282ad Make the tie record relocation a command, and verify it on two levels 
The relocation rule from the previous commit was established by a script that
lived in a shell history, which means the next person would have had to
rediscover it. It is now `gc-trace verify-tie-records <unpacked-dir> <level>
<class> <lod> <src>`: resolve the class, parse the block, search the RAM image
for the loaded record table, report what it found, and exit 0 only when every
record's pointer has become table + word and every field byte is verbatim.

Swept over every (class, lod) slot of two levels, from PCSX2 savestates:

  level 0: 87 slots with a block, 87 verified, 0 mismatched, 61 absent
  level 1: 104 slots with a block, 104 verified, 0 mismatched, 78 absent

So the rule holds on every LOD block of two levels the game had loaded, which
is what makes the record layout a finding rather than a reading of one block.

Writing the command also caught an off-by-one in the parser it exercises:
TieBlockHeader::records started its slice after the 16-byte header, but the
header IS the first record -- the table runs from the block start to
data_offset, and a block with data_offset 0x50 holds five records, the first
being the header itself and its pointer naming where the payload begins. Found
because the tool's first run relocated nothing while the same check in a script
had passed; the unit test fixture was wrong in the same way and is fixed.

Two operational notes now in the format doc: the live emulator is not a stable
input (verifying level 1 over PINE matched 104 of 104 while the user was on
that planet, and the same command minutes later found no matching copy because
they had returned to level 0 -- a miss there is not a counterexample, so the
report says "no copy matched" rather than "failed"), and a class index past the
table count is an error rather than a mismatch, so a sweep should stop at the
level's class count rather than a round number.
 2026-10-12 15:24:48 +02:00
ReaperOfVeriod e6eecb871f Decode the tie LOD block: it is a table of 16-byte records 
The tie block's word at +0x00 was read as an element count, and the reading
failed in a way worth recording: the payload length divided by it is a whole
number on 0 of 922 blocks, at any record size and any header up to 512 bytes.
That is the correct measurement of the wrong hypothesis -- the word never
described the payload.

Range measured: every LOD block of level 1 whose word is a non-zero multiple
of 16, plus the two read by hand in full.

Method: locate each block's record table in PCSX2's RAM by its twelve field
bytes, which are the only distinctive 12-byte run in a block, then compare
every record against its disc copy. Each block has several copies in RAM, and
the one that matches is the loaded one -- checking the first hit is what made
the first attempt report 0 of 16.

Result: on 16 of 16 blocks, every record's leading word has become
table_address + word, and the other twelve bytes are byte-identical to the
disc. So a block is [16-byte header][record table][vertex payload], the records
are 16 bytes -- a leading u32 pointer plus twelve bytes of fields -- and the
header's word at +0x00 is the byte offset from the block start to the payload,
that is, the length of the table.

That is what the multiple-of-16 was: the table holds 16-byte records. It is
never zero because there is always at least the first record, and its 42
distinct values are table lengths. Each record's pointer is relative to the
table start, relocated at load to table + word.

TieBlockHeader::data_offset and TieBlockHeader::records carry it, bound by the
offset, with TieRecord for the twelve field bytes, and the disc test asserts
the table is bounded on every block. What the records' fields name and what the
payload holds are still open -- the payload is known to be variable-length,
float-shaped, not LZ77 and not VIF1.
 2026-10-12 07:17:41 +02:00
ReaperOfVeriod c2af0528e3 Correct the chunk loader finding and decode a record field 
The loader note said every record gains a 02 00 prefix in RAM. It does not:
the word is the record's own leading field, already on the disc at every
record. What the loader changes is the pointer -- it addresses the word, two
bytes earlier than the disc's table -- and that pointer shift is what made the
bytes look inserted.

Measured, in order:

- the two bytes before a payload are identical in core_data and in RAM, record
  for record: 02 00 on level 0's 90-record chunk at 0x100b302, 08 00 on the
  level's first chunk. Nothing is inserted;
- the RAM table's low 24 bits are the payload's minus 2, consistently, so the
  record is [leading u16][two index bytes][payload] and the loaded table
  addresses the leading word. That is also why the index bytes read as +2 from
  the pointer -- the same +2 the earlier RAM result quotes;
- the word is a field, not a pad: uniform across a chunk's records on 1,439 of
  1,509 chunks (None on the 70 that vary, reported rather than averaged),
  fixed within all 71 chunk types, and only ever 1, 2, 4, 6, 8 or 10 -- 554
  chunks carry 2, 520 carry 4.

Two readings of the value were measured and disproved, so they are recorded as
disproved rather than left as candidates: it is not an entry count (the payload
divided by it leaves 57 distinct fractional "sizes", and 162 chunks do not
divide at all), and it is not the number of distinct leading bytes (agrees on 0
of 1,506 chunks).

Chunk gains `prefix: Option<u16>`, and the disc test asserts the uniformity,
the per-type fixity and the value set. The scratch example that produced the
per-type class-ID table is dropped now that the same scan is a committed test,
which also lands here: moby class IDs sit in one body slot per chunk type, with
the slot varying by type and the distinct >= 4 filter load-bearing -- without it
the top slots are constants that read as 305x because any value below the moby
count counts as a hit.
 2026-10-12 06:00:29 +02:00
ReaperOfVeriod 28680b4be3 Show the first two chunk-record bytes encode the record's own index 
Continuing from the phase counter. Within a chunk:

  i    :  0  1  2  3  4  5  6  7  8  9 10 11 ... 15 16 17 ...
  byte0:  0 16 32 48 64 80 96 112 128 144 160 176 ... 240  0 16 32
  byte1:  0  0  0  0  0  0  0  0  0  0  0  0 ...   0  1  1  1

byte0 is step*(j mod g) and byte1 is (j / g), where g = 256/step and
j = index + phase. The pair is the record's index in mixed radix: byte0 carries
the low digits scaled by step, byte1 the quotient.

Measured over all 27 levels: 1,235 of 1,509 chunks fit this on both bytes. Byte 0
alone fits 1,262, so the 27-chunk difference are chunks whose first byte follows
the rule and whose second does not. On 1,235 chunks a record knows its own
position and the two leading bytes hold no per-record information.

ResolvedChunk::index_rule returns the (step, phase) only when the whole chunk fits
on BOTH bytes, so the 274 that do not stay distinguishable and nothing skips a
leading byte on a rule that does not apply to it.

A number that moved when the implementation changed: a first probe reported
1,181; the committed method reports 1,235. The method is right. The probe
accumulated ok0/ok1 flags across candidate phases with an early-exit that could
skip a phase which would have matched both bytes, and it also disagreed with
bucket_rule on the byte-0 count (1,208 against 1,262), which is what gave it away.

Recorded because both numbers came from the same run and only the discrepancy
exposed the probe as wrong. The lesson is narrow: when a one-off probe and a
committed helper disagree, check the probe first.
 2026-10-12 04:25:44 +02:00
ReaperOfVeriod 79914c420e Show the chunk record's leading byte is a per-chunk phase counter; every chunk 
has a uniform record length

Follows the alignment result and settles what the leading byte is.

Within one chunk the leading byte steps by a constant and wraps:

  offsets: 1420 1748 2072 2392 2716 3044 3372 3700 4028 4352 ...
  byte0:      0  16   32   48   64   80   96  112  128  144 ...

Sweeping every (step, phase) over all 27 levels:

  fully explained: 1,262 of 1,509 chunks
  by step: 8 -> 51 chunks, 16 -> 554, 32 -> 520, 64 -> 137
  unexplained:  247 chunks across 11 chunk types

So on 1,262 chunks the leading byte carries no per-record information: it is a
deterministic function of the record's index and the chunk's phase, which is a
position marker rather than data. That is why the earlier "16 discrete values,
uniform counts" pattern appeared at all.

ResolvedChunk::bucket_rule returns the (step, phase) when the whole chunk fits and
None otherwise, so the 247 unexplained chunks stay distinguishable and nothing
skips a leading byte on the strength of a rule that does not apply to it.

Every chunk has a uniform record length: 1,509 of 1,509. Length varies BETWEEN
chunks of a type -- one type runs 76 to 104 bytes -- but never within one. This
withdraws the previous commit's "variable-length type", which came from pooling
minlen/maxlen across all of a type's records at once. That is the pooled-claim
error rule 3a exists to prevent, and it is the third time this session a pooled
measurement produced a claim that partitioning reversed. The type is not
variable-length; the chunk is fixed-length and different chunks differ.
 2026-10-12 04:21:32 +02:00
ReaperOfVeriod a9bf74fc10 Characterise all 77 chunk types: record byte 0 is a bucket index, not data 
Characterised every chunk type -- 77 of them, 1,509 chunks, 34,491 records, all 27
levels.

Byte 0 of every record is a small aligned index, not record content. It takes 16
discrete values per chunk type (0x00, 0x10, ... 0xf0) with near-uniform counts
WITHIN each type. Uniform weighting is the tell: a running byte offset would
weight the buckets unevenly, because records differ in size.

  type_id      recs  len  byte0
  0x007d7ec438 2430  328  0x00:162 0x10:162 0x20:162 ... 0xf0:135
  0x00429ec43d 2079  368  0x00:162 0x10:135 0x20:135 ... 0xf0:108
  0x00d76cc453  810  108  0x00:27  0x08:27  0x10:27  ... 0xe8:27   <- steps by 8

The alignment step is per type, not global. The leading byte is a multiple of 8 on
34,432 of 34,491 records but a multiple of 16 on only 33,349 -- the 1,142-record
difference falls in seven types, four of which step by 8. 59 records in three types
break even the 8-byte rule, and that is where the variable-length type lives:
0x00f2ef4275, records 76 to 104 bytes, no single stride to align to.

What the bucket indexes is NOT named. It is reported as CHUNK_RECORD_ALIGNMENT and
record_is_aligned rather than as a parsed field, because naming it would be a guess
while the invariant is solid. The test pins the exact exception set -- 59 records
across 3 types -- so a change there fails loudly.

A measurement I initially got wrong: the first version of the constant claimed a
16-byte multiple on 34,432 records. 34,432 is the multiple-of-8 count; the
multiple-of-16 count is 33,349. Right number, wrong label, and the test caught it.
That is the second time this session a correct number carried a wrong label, after
the tie four-value retraction. Both times the assertion did, which is the argument
for putting these numbers in tests rather than only in prose.

Still open: what the bucket indexes, and the remaining record fields.
 2026-10-12 04:18:55 +02:00
ReaperOfVeriod 291153a0d9 Find the tie LOD block terminator; packets == 0 means no block at that LOD 
parse_tie_block walks to a 0xf6 terminator that every real block carries, on
2177 of 2177 across all 27 levels, followed by zero padding. This is what makes
the blocks parseable: packets[1] bounds one on only 461 of 1864 classes, so the
terminator is the only general way to find one.

The terminator was worth a control, because 0xf6 could have been byte-frequency
coincidence. It occurs in payloads at 0.83x the uniform rate -- slightly LESS
often than chance -- while the last non-zero byte is 0xf6 on 909 of 922 blocks in
the first subset, against 17 of 922 for the same blocks cut at a random point. So
it is a terminator, not a common byte.

packets == 0 means the class has NO block at that LOD: of 3,728 (class, LOD)
slots, 2177 point at a block and 1551 are exactly zero.

Corrected header, over all 2177:

  +0x00  count    multiple of 16 on all; 42 distinct; never zero
  +0x02           zero on all
  +0x04           exactly four values: 0x501 (1069), 0xa02 (720), 0xf03 (350),
                  0x1404 (38)
  +0x06           16 distinct, never zero
  +0x08           180 distinct, never zero
  +0x0a           61 distinct, never zero
  +0x0c           125 distinct, never zero, clustered 0x043f..0x0443
  +0x0e           11 distinct, zero on 2162 -- the only non-universal field

+0x0c clusters around 1087-1091, close enough to a block length to be worth
stating, but it is NOT the length: the difference against the true payload length
scatters from -12 to -172.

A correction to the previous commit. It reported 3,728 blocks all terminating and
packed_texture taking 842 distinct values, and on that basis retracted the
four-value claim. Both numbers were wrong the same way: the scan did not skip
packets == 0, so it computed class_base + 0 and walked forward from the CLASS
BODY, finding a terminator there by accident. That contaminated 1551 of the 3728.

With the zeros excluded the slot takes exactly four values on all 2177 blocks.
The four-value claim was right; my retraction of it was wrong; and the retraction
was made on a number from the same contaminated set. A retraction is not
automatically better than the claim it retracts -- it needs its own bounds.

Still open: the payload's records. Established this round: blocks are
self-delimiting, absent LODs are marked by a zero packets word, and the header is
fully characterised. Not established: any record boundary inside the payload,
which stays variable-length.
 2026-10-12 04:13:52 +02:00
ReaperOfVeriod 02342de31b Decode the tie LOD block header; establish the payload is variable-length 
TieBlockHeader parses the block's own 16 bytes, measured over the 922 blocks
where "the next packets offset ends this block" holds, across all 27 levels. A
real-disc test asserts every figure.

  +0x00  count        multiple of 16 on 922/922; 16..2096; 72 distinct disc-wide
  +0x02               zero on 922/922
  +0x04               ONE OF FOUR VALUES DISC-WIDE: 0x501, 0xa02, 0xf03, 0x1404
  +0x06               15 distinct, 0x0206..0x0512
  +0x08               135 distinct, 0x0b05..0x4ee4
  +0x0a               45 distinct
  +0x0c               94 distinct, 0x010e..0x0553
  +0x0e               zero on 914/922

+0x04 is packed rather than a magnitude: four values across the whole disc with
the two bytes moving independently, low 1-4 and high 5/10/15/20 in lockstep. That
it is a texture index paired with a scale is a guess; that it is two fields is
not.

The count is not a record count -- block_size minus the header is divisible by it
on 0 of 922, for any record size and any header up to 512 bytes.

On the payload, a control was needed and changed the answer. Reading blocks as
3 x f32 triples, 53-64% of reads land within +/-5000 of origin, against 16% for
a uniform-random control of identical length. So the content is float-shaped and
genuinely not random.

But the rate is flat across every start offset and every stride tried, 53-65%
throughout. A statistic that cannot separate the right stride from the wrong one
cannot identify a stride -- rule 4's exact case. Flatness plus a count that does
not tile means the records are variable length.

That killed two readings. A VIF1 command walk consumes 0 of 88 blocks exactly,
so it is not a VIF1 CPU-format stream. wad::decompress rejects the block at every
start offset, so it is not WAD LZ77.

Scanning every (start, stride) pair from 0x10 to 0x7f for byte positions with a
small distinct-value vocabulary finds none: every position takes all 256 values
at every alignment. For float payload that is expected, so the scan found nothing
rather than confirming anything -- worth saying, because "no fields found" here is
uninformative, not evidence against fields.

Still open: what the payload holds. Established: variable-length, float-shaped,
non-random, not LZ77, not VIF1. Not established: any record boundary.
 2026-10-12 04:09:08 +02:00
ReaperOfVeriod 855e640f39 Establish the 16-byte tie record on measurable classes; packets[0] is a constant 
The previous commit reported "3580 non-empty LOD blocks compute as divisible by
16" and then said the figure should not be trusted, because it came from the
"next packets offset ends this block" rule. That rule is unsound: packets is
strictly increasing on only 461 of 1864 classes. On the other 1403 the two
offsets are unordered and cannot both be block starts into one ascending region.

Restricting to the 461 classes where the rule does hold, 922 of 922 consecutive
block sizes are a multiple of 16. That is the evidence for a 16-byte record, and
it is now scoped to the subset where it can be checked rather than claimed over
all 1864. The other 1403 are unmeasured, not measured-and-different.

packets[0] is a constant, not an offset: it reads 256 on 1864 of 1864 classes,
while packets[1] and packets[2] vary widely and land at different addresses in
core_data on every class. Read as a block start, 256 points just past the
0x48-byte header -- which is how it was being used, and exactly how a field gets
mistaken for a pointer. It is now TIE_PACKETS0 and nothing depends on it.

Still not decoded: what the 16 bytes hold. i16[0] at +4 takes 8969 distinct
values on level 0 and is not the two-valued field a 12-unit sample suggested; the
word at +0 is not a run offset.

The test pins all four numbers -- packets[0] constant, the 922/922 divisibility,
the 461/1864 ordering, and the 0/1864 monotonic disproof -- and asserts that the
ordering is *not* universal, so a future change there fails loudly instead of
quietly invalidating the record size.
 2026-10-12 03:50:42 +02:00
ReaperOfVeriod 60e3f129d8 Resolve both tie indirections on all 1,864 classes, and disprove a sample artifact 
resolve_tie_class walks the chain, which is two steps and both are easy to get
wrong:

  1. The tie class table lives in core_index.bin, not core_data.bin. On level 0
     the tables chain as moby at 192, 238 x 32 bytes later tie at 7808, 74 x 32
     later shrub at 10176, which closes. Reading the same table out of
     core_data.bin produces plausible garbage rather than an error.
  2. ClassEntry::offset_in_asset_wad addresses core_data.bin, despite the field
     name and despite core_data.wad sitting alongside it.

Verified: all 1864 tie classes resolve, o_class agreeing on every one, and all
5592 LOD block offsets land inside core_data.

Disproved, and this is the part worth reading: the word at +0 of a 16-byte unit
inside an LOD block. On the first twelve units of level 0's class 0 it rises
strictly -- 288, 1520, 2848, 3968, 5088, 6096, 7312, 8528, 9792, 10976, 12016,
13072 -- which reads exactly like a linked list, and the accompanying i16s looked
quantised, two distinct values in i16[0] and three in i16[1].

Across the population it is strictly increasing on 0 of 1864 blocks, and i16[0]
takes 8969 distinct values on level 0 alone rather than two. The sample was
twelve units out of 1,724,334. Both the monotonic reading and the quantisation
were artifacts of that sample.

That is the pooled-claim error AGENTS.md rule 3a exists to prevent, committed
almost verbatim: the first twelve units looked clean, and I was one step from
writing "u32[0] is a linked list of runs" into a format spec. The test now
asserts the disproof, so it cannot creep back.

Not established: the block's record size. 3580 non-empty LOD blocks compute as
divisible by 16, but some packets offsets run backwards, so the "next offset
delimits this block" end is wrong for them and that figure should not be trusted.
The tie layout stays partial: indirections and header done, records not.
 2026-10-12 03:47:49 +02:00
ReaperOfVeriod 02ced5ee83 Parse the per-level overlay: header, constant block, level data 
level_overlay was the least-characterised file in the level set -- no parser, no
located structure. It now has gc_formats::overlay and a real-disc test over all 27
levels:

  overlay: 27 levels; field_00 1735808, field_08 1;
  field_04 37540..=45308, field_0c 2763912..=3027336
  (17 below the file length, 10 above);
  +0x10..0x1220 identical on all

The header is four words. +0x0c is not the file length: it was recorded as one on
the strength of the first levels measured, where it sat near the file end, but
across all 27 it falls on both sides -- below on 17, above on 10 -- so it is an
ordinary level-specific word. The test asserts this, being the claim most worth
pinning.

The common prefix is 4 bytes, not the whole header: +0x00 is constant at 1735808
while +0x04 and +0x0c vary. Everything from +0x10 to +0x1220 -- 4624 bytes -- is
byte-identical on all 27 levels.

Still not named: what any of the four header words means, and how the
level-specific tail is structured. The module reports them as fields.

Also records in docs/plan/pcsx2-verification.md that the 64-byte RAM stride from
the previous commit is disproved. A 415 KB dump of level 1 finds the longest such
run at 27 records, not 384, and those are identity matrices -- 27 of them, one per
level. It is a per-level transform table, not the collision grid, and the 20.0
reported there was a coincidence.
 2026-10-12 03:32:02 +02:00
ReaperOfVeriod 2569cc06ab Decode the collision payload's leading two counts 
Range measured: the first four bytes of the payload on all 19 levels that carry
a non-empty one.

Result: two u16 at payload +0 and +2, both non-zero on all 19, ranging 1..74 and
13..112, products 53 to 3256.

  level  0  11 x 33  =   363   payload   903248
  level  1  24 x 16  =   384   payload  2800496
  level  2   1 x 53  =    53   payload  2315296
  level  7  40 x 35  =  1400   payload  4049232
  level  9  74 x 44  =  3256   payload  2518352
  level 13   2 x 64  =   128   payload  3333520
  level 26  27 x 112 =  3024   payload   652176

Reported as two counts, not as a named grid. They have the shape of collision
grid dimensions -- bounded, both non-zero, products in the hundreds to thousands
-- but nothing measured ties the product to the payload size or to any per-cell
stride. Level 2's 1 x 53 giving a 2.3 MB payload and level 26's 27 x 112 giving
652 KB shows the product does not size the payload, so either cells vary wildly
in size or these are not grid dimensions. Naming them width and depth would be a
reading, not a measurement.

After the two counts comes a run of u16 that stays below 8192 for 30 to 58
entries depending on level, always an even count, and is not ascending. That is
where the per-cell table most likely starts, and it is the next thing to
resolve.

Exposed as CollisionHeader::first and ::second. They live in the payload rather
than the header, so CollisionHeader::region fills them in and the bare
CollisionHeader::parse leaves them zero -- documented at the call site so the
difference is not mistaken for an absence.
 2026-10-12 02:58:46 +02:00
ReaperOfVeriod 5610d173b0 Narrow the collision header to two fields plus a third used by five levels 
Range measured: all 27 levels that carry collision, reading the 0x40-byte
header each.

Method: read +0x00, +0x04 and +0x08, then check +0x08..+0x40 for zero.

Result:

  u32@+0x00 is 0x40 on 27 of 27 -- the header is self-sizing
  u32@+0x04 is the payload size, 20 distinct values across the disc
  bytes +0x08..+0x40 zero on 22 of 27

Five levels carry a large non-zero word at +0x08:

  level  7  1248048        level 12  2541616
  level  9  1766832        level 13  3376592
                      level 14  4060640

So the header is not two fields and 56 bytes of padding: there is a third field
most levels leave at zero and five use. Whether it is a size, an offset or a
second block's length is not established, and nothing measured here separates
those readings. Exposed as CollisionHeader::field_08.

Correction: the collision header was previously described as self-describing
across its full 0x40, which is true for 22 levels and overstates the case for
the other five.

Still not decoded: the payload. It contains ascending u32 runs on levels 0 and
2 -- level 0 at +280 with deltas 12, 12, 12 then a jump, level 2 at +13868 with
deltas 204, 204, 200, 196 -- but irregular deltas fit neither a flat table nor a
simple stride, and level 1's apparent run is a byte-shift false positive.
Nothing about the payload is asserted.
 2026-10-12 02:49:25 +02:00
ReaperOfVeriod 51348b214d Report the sky reserved-word count in the test output 
20 of 149 sky sub-objects carry a non-zero word where the other 129 are zero,
so the reserved area is not uniformly padding. Reporting it rather than
asserting it, and clearing three clippy warnings from the unused counter.
 2026-10-12 02:43:40 +02:00
ReaperOfVeriod 681e7e62ef Parse the sky block: 149 sub-objects with a bounding sphere, and correct the count field 
Range measured: all 27 levels, each sky region from the core index's sky offset
to the next region.

Method: read the 32-bit offset table at +0x20, bounded by staying inside the
region, and parse each sub-object it names.

Result: 149 sub-objects across 27 levels. Each opens with a count, then a
bounding sphere of four floats at +0x10, then a texture offset at +0x20 and
dimensions at +0x24/+0x26:

  +000: 26 00 00 00 | 00 00 00 00 00 00 00 00 00 00 00 00
  +010: e0 8d b2 c1 | 9f d6 f6 3d | ef fc fb be | 1c 61 4e 41
  +020: 90 99 05 00 | 1d 00 | 1d 00 | 00 00 e8 00 ...

Count 1 to 38, width 3 to 81, radius 1.95 to 14.56, and the radius is positive
on 149 of 149 -- the same sphere shape and the same reasoning as the shrub class
header's.

Correction to an earlier claim: the word at +0x0c is not the sub-object count. It
reads 5 on level 0 where the table holds 6 entries, 3 on level 1 against 4, 7 on
level 3 against 8, and 196617 on level 2 against 4. Across the disc
offsets.len() == count+1 holds on 10 of 27 levels.

The field is reported as SkyHeader::field_0c and nothing depends on it; the
offset table enumerates the sub-objects and is self-limiting. That earlier claim
came from level 0 alone, where 5 against 6 looked like a consistent off-by-one.
It is not an off-by-one -- it is a field that happens to be small on most levels
and is not a count at all.

gc_formats::sky parses the header and the sub-objects. The gated test asserts the
sphere is finite, the radius positive where positive at all, the dimensions
non-zero, and the sub-object list non-empty on every level.
 2026-10-12 02:43:28 +02:00
ReaperOfVeriod 5ade83cf0a Parse the shrub packet header, a fixed signature on all 2056 disc packets 
The packets found by the +0x40 table have a header the whole disc agrees on.

Range measured: every packet on the disc -- 2056 of them, across the 334
successor-bounded spans -- read at each packet's span-relative offset.

  bytes 0-3   04 04 00 01   2056 / 2056
  bytes 4-7   zero           2056 / 2056
  byte  8     zero           2056 / 2056
  byte  11    0x05           2056 / 2056
  size % 16   zero           2056 / 2056

Zero exceptions on any of the five.

And the loader fills the rest in. Read back out of the running game the same
records carry byte 8 as a count (0x0b to 0x29 observed), byte 11 as one of four
tag values, and bytes 12-15 as an absolute EE RAM pointer where the disc has a
span-relative offset. Bytes 8-15 are rewritten wholesale on load.

That settles what the disc header is for. It carries no count of its own, and
the count the RAM copy has is the +0x40 table's count arriving by another
route. The disc header is a signature; the RAM header is populated state.

ShrubPacketHeader::parse and Shrub::packet_headers() expose this. parse returns
None unless all the invariants hold, rather than a default, so a packet that
does not match is reported as not-understood instead of half-interpreted.

The gated test asserts all four header invariants and the size alignment on
every packet of every bounded span.

This is the first piece of the mesh packet format represented in gc-formats;
everything known about the packet payloads so far is RAM measurement, since the
disc form of bytes 8-15 is rewritten at load.
 2026-10-12 02:35:00 +02:00
ReaperOfVeriod aa2b14ad66 The shrub mesh geometry is in the class blob after all 
The open question from the shrub decode was where the mesh lives: the blob
looked like header, palette and optional billboard, with the 333 classes
without a billboard having nothing to draw. Answering it took the running
game, not the disc.

Diffing a loaded shrub class against its disc bytes in the live PCSX2 (level
1, class 1) shows what the loader rewrites: a table at +0x40 of (offset,
size) pairs, whose offsets are span-relative on disc and absolute in RAM with
the sizes unchanged -- all 21 entries matching offset for offset and size for
size -- plus a relocated palette pointer at +0x2c and a pointer at +0x18,
zero on disc, that asserts the parser enforces.

So the blob is header, packet table, packets, [billboard], palette: the mesh
was here the whole time. Why it was missed is the part worth recording --
data_offset points at the END of the span, so reading the body as
data_offset..span_end accounts for the last few hundred bytes and leaves the
20 KB in front of it unexamined. A correct bound and a body small enough to
look plausible are together enough to hide a region.

Range measured: every shrub span on the disc, 358 over 24 levels, each bounded
by its successor class or the located chunk container.

Method: read count at +0x28, then the entries at +0x40, and require them to
tile -- the first packet at align16(0x40 + count*8), every entry exactly size
past the previous, and the last ending at the palette, or 0x40 short of it on
the 25 spans that carry a billboard.

Result: 358 of 358 tile exactly, holding 2,239 packets. That is what pins the
field meanings: count is the packet count (1..55 on the disc), data_offset is
the palette's offset, and the earlier "the body is not sized by count" note
was true for a reason nobody had understood -- the count sizes the packets.

Shrub::packets() exposes them and returns None rather than a partial list
when a table does not tile. Still open: the packet contents (a VIF reading was
already retracted once, so this needs a control rather than a pattern match),
and the 0x20-byte records the loader points at from +0x18.
 2026-10-12 02:02:01 +02:00
ReaperOfVeriod 69a8398f6e Verify the tfrag lighting magnitudes against the running game 
The harness that compares the port against the game's own memory existed but
had never been compiled: tools/trace did not build (27 errors, all in the
argument plumbing), and the tfrag_light.rs light-bank correction it motivated
was unverified. This finishes it and runs it, and the run finds two real bugs.

Result, on level 0 against three inputs -- the live emulator over PINE and two
PCSX2 savestates from the same level, agreeing byte for byte:

  headers matching the disc on invariant bytes: 833/833
  data pointers relocated as table + disc offset: 833/833
  light bank in RAM at 0x50ab20, byte-identical to the disc's
  vertices compared: 40600 (162400 colour bytes)
  equal: 162400 (100.00%)

The locator finds the level's tfrags block in RAM by tfrag 0's bounding
sphere and accepts a candidate only when every header matches the disc on the
bytes the game leaves alone and every data pointer is the relocated form, so
"the block was found" is itself a verification rather than an assumption. The
bank is searched in RAM by its exact section bytes, which is what settles its
framing: offset at +0x04, an i32 set count, twelve bytes of pad, then the sets.
The previous reading (offset at +0x00, count at +0x04) was wrong.

Two bugs the comparison found, both of which produced plausible output:

1. vertex_normal_bits negated x and y and left z positive, while
   NormalTable::normal negates all three. dA came out -0.707 where the game
   had +0.707, so max(dA, dA*wA) took the wrong branch and every vertex lit
   near its base colour: dark, but not wrong-dark. 95.88% agreement with the
   bug, 100.00% without.
2. A record whose select names a set the bank lacks drew transparent black.
   The game writes the record's base colour; 3,130 of level 0's 40,600
   vertices do this (select 0x000f, set 15 -- a runtime slot the level's
   directional bank does not have). The vertex is unlit, not invisible.

Both are now regression tests in the engine. The comparison prints the pass's
per-vertex terms (light_vertex_terms) for the first mismatches, because a
colour mismatch alone cannot be attributed and the comparison is worthless
without attribution -- that is how the second bug was identified from a
three-line read rather than guessed at.

Docs: docs/plan/pcsx2-verification.md records the harness, the result, the
two bugs and what to compare next; roadmap and level-assets.md stop carrying
lighting magnitudes as an open item. Findings reach tests only as numbers --
no test reads a savestate, a dump or a live emulator.
 2026-10-11 19:51:25 +02:00
ReaperOfVeriod 2c7555fe16 Decode the shrub class blob: a 24-normal palette and a billboard record 
The shrub body's payload encoding was the open question after the span-bound
work, and it answered itself once the R&C1 reference was in view: R&C1's shrub
class header keeps the field offsets R&C2 shows (o_class at 0x24, a count at
0x28, the data offset at 0x2c), which says which bytes to look at, and the disc
says what they hold.

Range measured: all 358 shrub class spans over the 24 levels that have them,
each bounded by its successor class or the located chunk container. Nothing
outside the spans is read.

Result. The body is a palette of 24 unit-length s16 normals -- s16 x, y, z and
a zero pad word per 8-byte record -- followed by all-zero padding records.
Three independent measurements, each of which fails for any other reading:

- all 8,592 live entries have squared length within 65,535 of 32767^2, and
  the bound is tight (the largest observed deviation is exactly 65,535). That
  is 32768^2 - 32767^2, which is what a unit vector scaled by 32768 produces
  when an axis-aligned normal quantises to -32768 and the positive axis clamps
  to 32767; components run the full s16 range and 1,765 entries are
  axis-aligned. A random s16 triple is off by about a billion, so one entry in
  fifteen thousand would pass by chance: all 8,592 pass.
- the padding records are always at index 24 or later and always all zero,
  never a live entry. That is what separates the palette from geometry, and it
  is the observation the three earlier stride-fitting attempts were missing.
- 17 classes carry a palette of 24 identical (0, 0, 1) entries. A position or
  index table cannot produce that; a normal palette for a flat-facing class
  must.

The header decodes as: sphere at +0x00 (centre + radius, radius positive on
all 358 spans, 1.72..41.57), size at +0x10, mode_bits and scratch at +0x14,
a zero word at +0x18, the billboard offset at +0x1c, and scale at +0x20. The
parser asserts the two invariants that hold without exception (scratch and the
reserved word are zero on all 358) plus a positive radius and scale, and
reports mode_bits rather than asserting it: it is 4 on 16 of the 358 spans.

The billboard offset is the strongest self-identification after o_class
because it is confirmed from a second file: it is non-zero on exactly the 25
classes whose core-index entry carries billboard texture fields, pairing
correctly both ways. At the offset sits R&C1's ShrubBillboard shape, register
block and all -- fade_distance, then TEX1_1 (K -233..-168, MMIN 4, address
0x14), TEX0_1 (index 1..4, address 0x06) and MIPTBP1_1 (address 0x34), all
three register addresses checked in the disc test.

Two corrections this produces, both recorded in docs/plan/level-assets.md
because the old readings were confidently wrong rather than merely
unverified:

- "reserved is 0 or a denormal" had the two u16 halves the wrong way round:
  the 16 non-zero spans are mode_bits = 4 with scratch = 0. As an f32 the word
  is still a denormal, so the PS2 reads zero either way -- but the field is a
  flag word, not a magnitude, which is exactly the FPU point worth making.
- "bounds" was tested as a min.xyz/max.xyz box, which holds on only 146 of
  358 spans, and the field was left unnamed as a result. It is a bounding
  sphere: the fourth float is positive on all 358 spans and the centres
  cluster near the class origin. The 146/358 figure was a property of the
  wrong question.

Still open: the three floats in the billboard record (they repeat
byte-identically across levels -- 16 distinct triples among 25 records, one on
four classes across levels 1, 4, 12 and 21 -- so they are shared art data
rather than dimensions), size, mode_bits, count, and where the shrub mesh
geometry lives. R&C1 puts the mesh packets in this blob; R&C2's blob holds
only the palette, so the 333 classes without a billboard must draw meshes from
somewhere else.
 2026-10-11 13:33:37 +02:00
ReaperOfVeriod ccea53807a Resolve the shrub region bound: the chunk container closes it 
The shrub region's end was the last unlocated bound in core_data. It was not
in the core index, and unlike moby->tie it had no successor boundary to chain
against, so the last shrub class in every level ran out to the lump's end and
the aggregates over it (the "3.5M records / 28MB of shrub body") were
artifacts of that oversized span.

Range measured: for each level, from the last shrub class's geometry offset to
the constant trailer. Nothing outside that range is read.

Method: scan forward from the last shrub class for the pair (a) the constant
two bytes f6 7c sitting immediately in front of (b) a chunk signature -- a
16-byte header ending 0xff, ten zero bytes, then four or more 0xf0-tagged
offsets. The two halves are each weak alone, and the halves are measured
separately: the marker pair occurs 25 to 103 times in a 5 to 17 MB lump, so
matching it alone picks dozens of wrong offsets; the chunk shape matches
random data often enough to need four tagged offsets before it is believed.

Result: on all 27 levels the marker sits exactly two bytes before the first
chunk header, and the two parts of the signature are never found apart. No
chunk signature occurs inside any last shrub body, on any level. The three
levels without shrub (10, 25, 26) carry the same marker before the container
that closes tie instead, which is what makes the locator a property of the
region rather than a shrub rule.

Consequences:

- All 358 shrub class bodies are bounded and are whole 8-byte records. That
  was not assertable while the last class in each level ran to the lump's end,
  and it is the check the boundary earns its keep for.
- 9,600 records in total, against the 8,938 previously measurable in
  successor-bounded spans; 662 of them sit in the 24 last-class bodies. The
  shrub region totals 2,033,776 bytes across the 24 levels that have one.
- The chunk container is 25,775,850 bytes across 27 levels (1,236 chunks,
  22,562 records). The container measurements were previously taken over "the
  900,000 bytes before the trailer", a window that did not align to its real
  extent: on level 0 it is 690,302 bytes, and 27 of 1,123 chunks were
  truncated at that invented edge. chunk_containers_resolve_on_every_level now
  reads the region from core_layout instead of measuring backwards from the
  end of the lump.

core_layout takes core_data now: the container's start is the one bound that
cannot be derived from the index. It gains the chunks region, so every byte of
level-specific core_data belongs to a named region. Where the signature is not
found the end stays None -- the region is left visible rather than clamped to
the tail.

Still open: the marker's meaning (constant across 27 levels, at exactly this
position, unnamed) and the shrub body payload's encoding.
 2026-10-11 06:20:48 +02:00
ReaperOfVeriod 10188416e2 Decode the chunk count field: header byte 14 is the record count 
Range measured: all 27 levels, 1123 chunks, over the 900000 bytes before the
constant trailer.

Method: compare header byte 14 against the number of offsets actually resolved
for that chunk.

Result: byte 14 equals the record count on 1123 of 1123 chunks, zero
mismatches. This is the first genuinely decoded field in this container rather
than a measured correlation.

The parser now truncates the offset table to header byte 14, so a stray
tagged word in a chunk's record data cannot silently extend the table and put
the count at odds with its own header.

Two more header bytes are near-constant and read as ASCII: byte 13 takes only
0x45 (1076 chunks) and 0x46 (47); byte 9 takes 0x44 (26), 0x45 (1084) and
0x46 (13). Those are D, E and F -- a three-letter tag at two byte positions
with different distributions. Recorded as suggestive, not established: the
other 14 header bytes are high-entropy and read as ASCII at no shift tried, so
the D/E/F may be incidental. Worth testing against a known type table.

Still open: the 14 opaque header bytes, whether bytes 9 and 13 are type tags,
which record fields are index/length/offset, and whether a chunk is moby_anim.
 2026-10-11 04:26:44 +02:00
ReaperOfVeriod 681333535a Add a bounds-checked parser for the post-shrub chunk container 
gc_formats::chunks parses the container that fills the region after the last
shrub class body and before the constant trailer:

  +0x00  16 bytes, last byte 0xff     (opaque, differs per chunk)
  +0x10  10 zero bytes
  +0x1a  N tagged offsets, top byte 0xf0

Each tagged offset is 0xF0000000 | low24, and a record is read at
chunk_start + low. Resolving that way gives records sharing a field layout,
which is what distinguishes the container from a coincidental byte pattern.

Fields are not named. The 16-byte header is opaque, the 0xf0 tag is
unexplained, and it is not established that a chunk is moby_anim -- a
49 x 368-byte table inside this region is byte-identical on all 27 levels,
which is evidence against that identification. What the parser provides is
chunk and record extraction, which is what the analysis established.

The gated test measures, over all 27 levels: 1123 chunks, 19546 records, 50
distinct strides, 12.4% of byte positions constant across records (per-chunk
range 2.8% to 75%). Chance gives a constant position probability of 1/256^n.

Three things the test got wrong first, recorded because they are the traps:

- A per-chunk threshold of 20% constant positions. Individual chunks run from
  2.8% to 75%; the aggregate is the invariant, not the chunk. Asserting 20%
  per chunk was asserting a property the format does not have.
- Requiring every offset to resolve. 27 chunks truncate at the window edge,
  because the 900000-byte window does not align to chunk boundaries. Counted
  now rather than asserted away.
- read_u32 in the chunk signature check was big-endian while Buf::u32 is
  little-endian. The tag lives in the high byte of a little-endian word, so
  the mismatch made every level report zero chunks while parse_chunk -- which
  goes through Buf -- was correct. Worth remembering that the two paths can
  disagree on a format this young.

Tests: 190 -> 191.
 2026-10-11 04:25:18 +02:00
ReaperOfVeriod 69b81af3b2 Expose the shrub body as 8-byte records, bounded per class 
Turns the differential finding into code. Each shrub body record is 6
varying bytes plus 2 constant ones; body_records() hands them back raw and
body_is_well_formed() checks the trailing pair.

ShrubHeader::region now takes an explicit span_end. It has to: without it
the data slice runs to the end of the lump, so the last class in each level
absorbs whatever follows shrub and a per-class check ends up measuring its
neighbours. That is the same unbounded-span mistake as before, caught this
time by a failing assertion rather than by reading the docs afterwards.

The disc test asserts well-formedness only for the 334 of 358 classes whose
span ends at the next class. The last class in each level has no successor,
so its span contains the 690 KB region after shrub and its body is not
wholly records -- that is the open question, and asserting either way would
be wrong. Shrub::span_end_is_a_successor records which case a span is.

Records are summed only across bounded bodies. Summing the last class too
reproduces the "3.5 million records" figure retracted two commits ago, to
within rounding -- which is a useful check on how easy that mistake is to
repeat by accident.

8,938 records across the 334 bounded bodies; 16 denormal reserved words;
358 headers naming their own class.
 2026-10-11 03:26:06 +02:00
ReaperOfVeriod 8d57a09b76 Bound the last region by core_data's constant trailer 
Comparing all 27 levels byte-for-byte from the end: the last 231,952 bytes
(0x38A10) of every level's core_data are identical. Level-specific content
therefore ends at len - 0x38A10, which is a real bound for the final region
and is not core_data.len().

This closes shrub. Across the 24 levels that have a shrub region the bound
gives 24,442,928 bytes total, and the disc test asserts on every level that
the region ends strictly inside the lump rather than at its end.

What the trailer is remains unestablished -- fixed data, or padding from a
fixed-size decompression buffer -- but it is constant, which is all a bound
needs.

The data between the last shrub class and the trailer is high entropy: 183 to
256 distinct byte values per 4 KB block, mean 254, where texture data sits
under 200. It is real content of unknown type and is the most likely home for
moby_anim, now that the range between collision and moby has been ruled out
as texture.

A lump shorter than the trailer now yields end: None rather than a backwards
bound, covered by a test.
 2026-10-11 03:09:21 +02:00
ReaperOfVeriod 8b98d388f7 Show region bounds in probe, which exposed an unnamed 6.2 MB region 
Printing the region table next to the header fields makes the gap visible
without having to remember to consult core_layout, and it paid for itself
immediately.

On level 0 the collision region comes out 6216960 bytes, from 2615232 to
the first moby class at 8832192. Collision's own header declares a payload
of 903,248. So roughly 6.2 MB sits between the collision header and the
first moby class, and almost none of it is collision.

The successor rule is correct -- a region does end where the next one
begins -- but "the next" is not yet known there. collision..moby is
really collision + <something unnamed> .. moby. The name is wrong, not
the bound.

Candidates for the space, rough order of likelihood: moby and tie model
animation data, which is what moby_anim would need, then volumes, then
per-class runtime structures. Tfrag texture texels are already placed and
sit earlier in the lump.

Worth resolving before starting moby_anim, since that format is the most
likely tenant. Until then, any probe reading between the collision header
and the first moby class is reading something the region table does not
describe.
 2026-10-11 03:03:46 +02:00
ReaperOfVeriod cb38b8c279 Compute core_data region bounds once, from successor boundaries 
Region spans were being re-derived per format while probing, usually
running from one known offset to the next with the last span running to
core_data.len(). On level 0 that gave the final shrub class a 925,008-byte
span absorbing most of the lump's tail, and every aggregate computed over
it -- record counts, value distributions, "this looks random" -- was
silently wrong. Nothing errored, which is what made it survive two
commits and a correction.

core_layout computes the layout once: each region ends where the next one
begins, class regions start at the lowest non-zero geometry offset in their
table, and a region whose successor is unknown gets end: None rather than a
clamped guess. Zero class entries mean absent, not position zero -- the
same trap occlusion and collision each fell into separately.

The disc test asserts what the module is actually for: that the unknown is
visible. Shrub's end is unresolved on the 24 levels that have a shrub
region, and that is asserted rather than hidden.

It also found that not every level has every region -- level 10 has neither
occlusion nor shrub -- so the test checks ordering against the canonical
sequence instead of demanding an identical set.

Still open: where the shrub region ends. Texture texels cannot bound it;
they sit before moby, not after shrub. See docs/formats/core-data-regions.md.
 2026-10-11 03:02:02 +02:00
ReaperOfVeriod 517f0b63fa Read the shrub header floats, including a denormal the PS2 flushes 
Extended the shrub header past the two self-identifying words. The first
four words are floats -- finite on all 358 spans -- though they do not
behave like a min/max bounding box, which orders correctly on only 146 of
358, so they are recorded as bounds rather than named. The word at +0x10
is positive everywhere, 3.0 to 20.0, exactly 5.0 on 281 of them.

The word at +0x14 is the interesting one. It is zero on 342 spans and the
bit pattern 04000000 on the other 16, which is a denormal of about 2.8e-45.
The Emotion Engine has no denormals, so hardware reads all 358 as zero.

That is only the second reading. My probe reported the field as 0.000
everywhere because it printed to three decimals, and the test caught it:
level 7 class 7 fails a strict zero check. The lesson is that "exact" in
a measurement means exact in the measured precision -- rounding in the
output made a real distinction invisible.

It also matches AGENTS.md on modelling the FPU rather than correcting it:
this field looks like noise on disc and is a constant to the machine that
reads it, and the parser accepts either rather than picking one.

The shrub body past +0x2c is still not decoded.
 2026-10-11 02:41:34 +02:00
ReaperOfVeriod 7107f395d0 Parse the shrub class header 
Shrub is the third class-geometry family and is not the moby one: all 358
spans on the disc have a leading word of 0 or one too large to index a
table into, so none contain a moby-style part table. The rule that holds
for 43,467 of 43,467 moby parts holds for zero of them.

Two header fields do self-identify, exactly, on all 358 spans: +0x24 equals
the owning class entry's own o_class, and +0x2c is a data offset landing
inside the span. Both are checked rather than merely reported -- a span
naming the wrong class is a different thing, and quietly returning the
mismatched value would hide that.

The body past +0x2c is not decoded. It tiles at no stride consistent with
the count at +0x28 (tails of 240, 240, 192 and 224 bytes against counts of
5, 6, 5 and 3), and the header's leading bytes are packed rather than
float even though +0x10 lands exactly on 5.0f on 281 of 358 spans.

Shrub class entries are 0x30 bytes rather than 0x20, because they carry
billboard texture fields the other two tables lack; the parser records that
so the stride is not re-derived.
 2026-10-11 02:38:35 +02:00
ReaperOfVeriod f6bbbf8d5f Add gc-extract probe to dump structures through the real parser 
Reading these formats by unpacking raw offsets out of core_index.bin has
cost real time. The index interleaves single-word region offsets with
count/offset pairs, so miscounting one shifts every field after it: that
produced a page of implausible collision headers on all 27 levels, sent
me chasing a tie-breakoff that turned out not to exist, and nearly had me
"fix" a stride bug in code that was already right.

probe dumps a level's core header and all three class tables through
LevelCoreHeader, so a field name printed is the field name the parser
uses. It also reports how many entries carry geometry inside core_data,
which is what pins the class table strides rather than guessing at them:
shrub reads 7/7 sane at 0x30 against 4/7 read at 0x20.

Reading a structure by hand in a shell still works; this just stops it
being the default, and stops the offset arithmetic being re-derived every
time a new region is investigated.
 2026-10-11 02:36:37 +02:00
ReaperOfVeriod bbb10c678d Parse the moby per-class part table 
A moby class span does not start with its geometry. The first u32 is the
offset of a part table, and the table is 16-byte records of vertex
offset, vertex count, an unestablished second count, the next part offset
and flags.

Parts tile: vertex_offset + 16 * vertex_count == next_part. The stride is
never assumed anywhere -- it falls out of that identity, which is what
pins the record size and the field boundaries. The identity holds for
43,467 of 43,467 part records across every moby class on all 27 levels,
and the parser rejects any span that violates it rather than returning a
plausible wrong answer.

Two terminations are needed. A zero vertex_offset ends the run, but 172
records on the disc have a zero vertex_offset and a zero next_part while
still carrying a vertex count. Stopping only on the first leaves 99.6%
instead of 100% and quietly includes a second table as if it were parts.

1,270,802 vertices over 4,560 class spans; 116 spans carry no geometry.

Vertices are 16 bytes of VIF-packed half-float quads cycling through four
distinct words. Handed back raw -- unpacking them is separate work from
finding them, and second_count is recorded but not guessed at.
 2026-10-11 02:29:57 +02:00
ReaperOfVeriod dc941a8eec Parse the collision container framing 
Collision sits inside core_data at the offset the core index gives, behind
a 0x40-byte header that self-describes: it carries its own size and the
payload size. That pairing is the invariant -- the declared header size is
exactly 0x40 on all 27 levels and header + payload stays inside the lump.

41,481,520 payload bytes across the 19 levels that have any; 8 declare
zero length, which is present-but-empty rather than absent, so the two are
counted separately.

The cells inside the payload are not decoded. They do not walk as
fixed-stride cells, and treating faces as variable-length records stops
partway into the first cell on level 0 rather than tiling the payload. The
first cell reads as 11 verts and 33 faces under every hypothesis tried,
which is exactly the kind of agreement that looks like a result and is
not one -- three different strides all produce the same first cell and all
of them are wrong. Left open rather than committed as a guess.
 2026-10-11 02:21:10 +02:00
ReaperOfVeriod 2dcc6e9710 Parse the sound bank container framing 
A level's sound extent is two regions: an `SBlk` descriptor table and the
sample data that follows it. Four u32 at the head describe the layout, and
the invariant that establishes it is sblk_offset + sblk_size == daw_offset,
which holds on all 27 levels. That is what pins the field meanings; the
numbers alone are ambiguous about which of them is the offset.

DAW is a format tag inside the SBlk header, not a region signature. It
occurs exactly once in the entire extent, at sblk_offset + 12, naming the
sample format -- the sample data itself starts at daw_offset with no
signature at all, roughly 19 KB away on level 0. Reading it as a region
magic is the obvious reading and it is wrong.

daw_size never reaches the end of the file. The tail is positive on all 27
levels, 232 to 1,848 bytes, so the field describes less than the region
it names; the test asserts that direction rather than inventing a size
that fits.

523,720 SBlk and 37,649,008 DAW bytes across the 27 levels.

The sample format inside DAW is not decoded. It looks like VAG/ADPCM from
the nibble packing, and it has no bearing on drawing anything, so it stays
open rather than getting a half-answer.
 2026-10-11 02:15:33 +02:00
ReaperOfVeriod cfabe9a0ae Fix mask bit lookup to match its documented encoding 
visible() documented its argument as `byte << 8 | 1 << bit` -- high byte
selects the mask byte, low byte is already the shifted bit -- then shifted
the low byte a second time as though it were a bit index.

That is wrong twice over. It inverts the meaning of every bit argument,
and for any low byte above 8 it panics in debug, because `1u8 << 32` is
an overflow. Real data reaches it: the folded bits for the upper bytes of
a mask are exactly the values that trip it.

Treat the low byte as the bit mask, which is what the doc says and what
the renderer needs. Added an assertion for a mask index that does not
exist, since that was the case that exposed the mismatch.
 2026-10-11 02:15:33 +02:00
ReaperOfVeriod 632505a6bd Parse the occlusion PVS grid 
A z -> y -> x tree of u16 node offsets over 4x4x4-unit cells, each
populated cell naming one 128-byte mask of 1024 visibility bits that the
renderer fills from the camera's cell each frame.

Two sentinels carry meaning and one of them is easy to miss: a z or y
slot of 0 means "no subtree", and an x slot of 0xffff means "no cell".
Reading the latter as a mask index gives a grid whose highest mask is
65535 on every single level, which is the tell.

The check that makes this real is the mask array. It starts at
masks_offset and runs to (highest_mask + 1) * 0x80, and on every level
with a grid that ends at the block boundary exactly -- widest
unexplained tail across the disc is 0 bytes. A wrong tree shape or cell
size would leave the highest mask index elsewhere entirely.

412,521 populated cells over 140,939 masks, across the 24 levels that
have one.

An offset of 0 means the level has no occlusion data at all, which is
not the same as "at the start of the file" -- that is where the tfrags
block lives, and parsing it as a grid produces a mask array that fits
anyway, with 1600 bytes to spare. That is what made level 10 look
plausible on the first pass; three levels (10, 24, 25) carry none.
 2026-10-11 02:11:30 +02:00
ReaperOfVeriod 79025b22d6 Decode the tie class header prefix that RAC1 and RAC2 agree on 
Tie is the easy one of the three geometry subsystems -- instanced
static props, no skeleton -- so it is the right place to start.

RAC1's class header is 0x80 bytes. RAC2 agrees with it over the first
0x48 and diverges after. From 0x48 the RAC1 layout does not describe
RAC2 data: lod_info at 0x50 reads back as noise, packet_count and
texture_count are zero on every class examined, and the normals offset
at 0x0c points 117,788,434 bytes into a 16.9 MB lump.

Only the agreed-on prefix is decoded. What makes it trustworthy:

* o_class at 0x44 equals the core index's class number for all 1,864 tie
  classes across all 27 levels. A field landing on the right number that
  often is not a coincidence, and it pins the header base.
* The LOD distances are ordered 0 < near < mid < far for 1,784 of 1,864.
  The 80 exceptions are real, unexplained, and are reported rather than
  asserted away.

Level 0's first tie class has distances 12.0 / 25.0 / 40.0, a plausible
prop LOD band, and packet offsets 256, 20,592 and 26,048.

The rest is deliberately not guessed at. A field order that produces
plausible output and is wrong is the specific failure this project keeps
recording, three times now: the gameplay extent, the tfrag record
framing and the class tables were all "reasonable" and all wrong.
 2026-10-11 02:06:39 +02:00
ReaperOfVeriod 4c4a78d402 Identify the container the class tables point into: core_data itself 
assets_compressed_size and assets_decompressed_size at index offsets
0x88 and 0x8c are 9,738,678 and 16,969,296 on level 0 -- exactly the
sizes of core_data before and after decompression. There is no separate
asset container. The "asset WAD" is core_data, and offset_in_asset_wad
indexes the decompressed form directly.

Checked rather than assumed, on all 27 levels: the decompressed size
equals core_data's exact length, every class offset is either 0 or inside
it, and the moby, tie and shrub spans do not overlap each other.

An offset of 0 means the slot has no geometry, not the start of the
lump; level 0's ninth moby class is 4909 at offset 0. Two assumptions
worth discarding while here -- the tables are not sorted by class number
(level 0's run 0, 1, 2, 14 ... 4909) and the offsets within a table are
not monotonic. Neither is needed, but both look true from the first few
entries.

The chunk descriptors after the named extents are not it: they exist on
only 3 of 27 levels, because level 0's four named extents tile the
container exactly. An asset blob there would be missing almost
everywhere.

What sits at the offsets is raw GS/VIF packet data rather than a further
container -- moby[0] begins 00 00 00 00 3f 16 02 55 6f 6f and tie[0]
00 01 00 00 70 50 00 00, which read as packet headers rather than a
magic number.

None of the three is decoded yet and this does not start that. Moby is
skeletal and is the largest single piece left before the rest of a level
can draw.

Also: the pipeline now writes each level's chunk descriptors where they
exist, and LevelCoreHeader decodes the asset lump's two sizes.
 2026-10-11 02:03:31 +02:00
ReaperOfVeriod e1105543e4 Parse the level core class tables: moby, tie and shrub 
These were ruled out for want of an entry size. The size is knowable
exactly, because each table ends precisely where the next begins:

  moby  238 entries at   192, 32 bytes:  192 + 238*32 =  7808 = tie's
  tie    74 entries at  7808, 32 bytes: 7808 +  74*32 = 10176 = shrub's
  shrub   7 entries at 10176, 48 bytes: 10176 +   7*48 = 10512 = textures

That closes on 27 of 27 levels, which is what makes it evidence rather
than a coincidence: any wrong entry size breaks the chain immediately.
It is the same "does this hold on every level" discriminator the project
already uses for the container pair table.

Across the disc: 5,796 moby, 1,864 tie and 358 shrub classes, plus the
1,595 tfrag textures. Class entries are 32 bytes -- asset offset, class
number, two unknown words and 16 bytes of texture slots -- and a shrub
entry adds 16 bytes of billboard slots for 48.

The header now decodes through to the texture tables rather than
skipping 0x18..0x30 wholesale, which also recovers the moby, tie and
shrub texture ranges. Two more pairs sit between shrub textures and the
textures base offset.

Two things worth recording. ClassEntry::parse read its 16 texture bytes
before its four i32s, because the struct literal that uses them comes
after the loop that advances the cursor -- so every entry was parsed
backwards. Struct literal field order does not reorder the reads that
already happened. And every shrub class on level 0 carries all-zero
billboard slots, so the dimensions are optional rather than something
worth asserting on.

This is the third instance of the same failure shape. The gameplay
extent was "an unidentified compression" because it was parsed
compressed; the tfrag record framing was "unrecoverable" because the
entry size was never known. Both looked like properties of the format
and were properties of the question.

Also records that point lights are a runtime feature rather than missing
work: every one of the 18,102 patches has point_lights == 0xffff and the
eight-slot bank is filled by gameplay, so there is nothing for them to
light in a static level render.

And that PCSX2 is not installed, so the comparison loop cannot be built
yet -- which is why lighting magnitudes stay unverified.
 2026-10-10 03:44:47 +02:00
ReaperOfVeriod b8b4a1a3d3 Consolidate: tidy level.rs, correct stale docs, pin hardware models 
No behaviour change. This is the pass over the last stretch of work to
make sure the accumulated result hangs together rather than only each
commit in isolation.

Code. lit_color had grown a `let _ = (depth, lod_distances)` and a dead
branch from repeated edits. The directional pass genuinely does not
depend on distance, so the parameters are gone rather than discarded,
with a note that the point lights -- not implemented -- will need them.
lit_at folded into lit_color, which had become a wrapper around it.
Checked for leftover scaffolding: only two env vars remain, both real
(GC_ENGINE_SHOT and GC_ENGINE_DEBUG_BOUNDS); GC_ENGINE_UNLIT,
GC_ENGINE_DEBUG_COLOR and GC_ENGINE_ABL were never actually written,
because those edits silently failed to apply against rustfmt'd source.

AGENTS.md. The crate table did not say that gc-engine owns hardware
models. ps2.rs is the Emotion Engine's FPU and the lighting pass runs
on it, so the boundary is now stated: gc-formats decodes inputs,
gc-engine reproduces how the hardware computes with them. That is the
distinction rule 2 depends on, and it was implicit.

Docs. Three claims had been overtaken by what we learned and were
actively misleading:

* docs/formats/level-wad.md still said gameplay "does not decode with
  the WAD decoder". It does; it is a WAD stream.
* docs/plan/level-assets.md carried a whole section, "a compression we
  have not identified", with a ruled-out list including "bit order",
  "header length" and "the WAD format at all". All three were ruled out
  against the wrong input. Replaced, with the failure mode kept: a
  uniform failure across 27 inputs was read as "the decoder is wrong"
  when it meant "the data was handed over uncompressed", and the
  positive control -- decompress and look -- was never run.
* The same file still said the tfrag "record framing is not recovered".
  True, but it was a false lead that has since been superseded by the
  VIF1 framing. Kept as a recorded wrong turn rather than deleted.

Also refreshed "Still open" to name what is actually outstanding:
point lights, the GS register set, the guard-band clip test, the
unread core_index class tables, and -- stated plainly -- that lighting
magnitudes are unverified because there is still nothing to compare
against.

Verified after the changes: 148 tests pass, clippy and fmt clean, all
5,213 extracted files match the hash table, and level 0 still loads at
26,758 triangles with the same LOD split and fully lit.
 2026-10-10 03:37:03 +02:00
ReaperOfVeriod bcf0d1f0cc Decompress the gameplay extent: the level light bank now parses 
gameplay is a WAD stream, exactly like core_data. The extractor was
writing it compressed, so the light bank framing read the WAD header's
stream length as a light count: level 0's offset at +0x04 was 0xa0d
and the "count" there was 698810368.

The file says so in its first six bytes: 57 41 44 79 0d 0a -- "WADy".

Decompressed, level 0's gameplay extent is 1,872,256 bytes, the offset at
+0x04 is 0xc2b20, and the count there is 5. Five directional light sets.

All 80,274 vertices of level 0 now go through LightTfrags, and the
renderer's fallback line no longer appears. The frame's colour
distribution goes from 295 to 737 distinct sampled colours and takes on
the level's warm cast.

This corrects a recorded negative result that was simply wrong.
docs/roadmap.md carried "gameplay does not decode with the WAD LZ77
decoder, all 27 levels fail identically, so the input is wrong rather
than the decoder", with bit order and header length also ruled out.
That was true of the data being parsed and wrong about the cause: the
decoder was never the problem, it was being handed a file that had not
been through it.

Same lesson as the origin bug, from the other direction. A recorded
failure without a positive control is easy to record and easy to leave;
the control here is trivial and should have been run first -- decompress
and look.

The renderer keeps counting and reporting how many vertices it lit,
because a silent fallback to stored colours looks plausible and is wrong.
Lighting magnitudes remain unverified: that needs comparison against
reference frames via the PCSX2 loop, which is still to come.
 2026-10-10 03:27:52 +02:00
ReaperOfVeriod 65b5d366de Implement the PS2 float model and the LightTfrags pass 
Two pieces, and the first is the reason the second can be trusted.

ps2.rs is the EE's FPU on raw u32 bit patterns. It is not f32: there are
no denormals (exponent 0 *is* zero), no NaN and no infinity (exponent 255
is an ordinary value, so MAX reads back in Rust as NaN -- a test asserts
exactly that, because it is the sharpest available statement of why this
module exists), every result truncates toward zero, the adder discards
the small operand's bits more than one below the large one's LSB and
returns the large operand unchanged across a 25-bit exponent gap, and
rsqrt is sqrt-then-divide with truncation between the steps rather than
one rounded operation. The lighting pass accumulates all of these across
every vertex of every patch, so this is a visible difference rather than
a rounding curiosity.

tfrag_light.rs is the pass itself: PEXT5 base colour, the select word
picking one light set or blending two by bits 8..15 / 256, blended
directions re-normalised where unblended ones are not, and
base + A.rgb*max(dA, dA*wA) + B.rgb*max(dB, dB*wB). Lighting indexes
positions, not vertex-info entries, since a common entry and a LOD entry
can share a position.

TfragLight gains a `select` field for the u16 at +6. It was briefly
reconstructed with an unsafe pointer cast over the struct, which is
exactly the sort of thing that breaks silently the moment a field is
reordered; it is now a plainly parsed field.

The pass does not run yet. The light bank framing the reference uses --
a u32 offset at +0x04, an i32 count there -- gives offset 0xa0d and a
count of 698810368 on RAC2 level 0, so RAC2's gameplay extent is not
laid out that way. The renderer now counts the vertices it actually lit
and prints it, because falling back silently to the stored colours
produces a plausible but wrong picture:

  lighting: 0 of 80274 vertices shaded by LightTfrags; the rest fall
  back to the tfrag's stored colours (no usable light bank)

That line is the current honest state, and the frame has not changed as
a result. Where RAC2 keeps its light bank is the open question.
 2026-10-10 03:21:52 +02:00
ReaperOfVeriod 90f02890f8 Decode tfrag normals from the overlay's sine table 
Normals are per-vertex, and the tfrag's light records store them as an
azimuth and elevation *index* into a 256-entry sine table rather than as
angles. This decodes that table and builds normals for the renderer.

The reference reads the table from an ELF at BOOT_NORMAL_TABLE_VADDR
(0x165500) and notes every level overlay carries a copy. The overlay has
no ELF header -- it is a raw loadable image starting with code -- so there
is no program header to map a virtual address through, and the offset is
not fixed either: level 0 has the table at 0xb650c, level 5 at 0xcca30,
level 26 at 0xb6284.

So it is located by content. Every table starts with exactly (1.0, 0.0),
angle zero, and the scan then requires all 256 entries to match
cos(i*pi/128), sin(i*pi/128). Validating the whole curve is what makes the
search safe: a lone (1.0, 0.0) pair of floats occurs by accident in a
2.7 MB executable, and a test plants one as a decoy. The values are read
from the disc rather than recomputed, since those are the numbers the game
indexes.

A normal is (cos(az)cos(el), sin(az)cos(el), sin(el)) *negated* -- the
stored normals point into the surface -- and then given the same Z-up to
Y-up swap as positions, or the lighting comes out sideways. The azimuth
and elevation fields are stored signed but index 0..=255, so entry 200
must be read as 200, not as -56; there is a test for that.

Also frames and parses the level light bank from the gameplay extent, and
the pipeline now writes overlay.bin. The per-vertex lighting evaluation
is not written yet, so the renderer stays unlit by default. GC_ENGINE_LIT=1
substitutes a host directional light for inspection -- that is a stand-in,
not the game's lighting, and it is deliberately not the default.

The hash table is regenerated for the 27 new overlay.bin files: now 5,213
files and 5,762,497,559 bytes.
 2026-10-10 03:09:07 +02:00
ReaperOfVeriod 17c7e8ccc9 Draw tfrag per-vertex colour, unlit, and record why the frame is dark 
The tfrag RGBA array now modulates the texture. It is indexed by
*position*, not by vertex-info: several vertex-info entries can share a
position, and the array is sized vert_count, which is a position count.

Colour follows the same rule as the vertex itself. Morphing blends the
position only -- the game keeps the high-detail colour -- while collapse
replaces the vertex outright, colour included. Getting that the other way
round would tint distant terrain.

The material is unlit, and that is a decision rather than a shortcut.
Level 0's colours really are dark (patch 0 runs (25,26,26) to
(108,144,141)) and the game's lighting is what brightens them, so what is
on screen is the true albedo.

Worth recording: the obvious move -- add a directional light, or crank
ambient brightness -- does nothing at all. With no normals the PBR path
is degenerate, so the brightness change has no effect, and a host light
would have hidden the gap rather than shown it. The colour data was
always reaching the mesh; what is missing is lighting, not a material
flag.

Written up in plan/level-assets.md, including what LightTfrags actually
needs: the light bank from the level's gameplay extent, the 256-entry
normal table at BOOT_NORMAL_TABLE_VADDR (0x165500) which every level
overlay carries, and the already-parsed azimuth/elevation light records.
 2026-10-10 03:02:52 +02:00
ReaperOfVeriod 192dcde247 Render a level: first drawable frame, and the origin bug it exposed 
Adds gc-engine's binary and a level.rs that joins the format layer to the
renderer: LOD selection and morphing per patch, then triangles grouped by
texture so each texture is one draw rather than one per triangle.

The PS2 world is Z-up at 1024 raw units per world unit, so the conversion is
(x, y, z) -> (x, z, -y). The sign flip on the third axis turns the PS2's
right-handed frame into Bevy's left-handed one rather than negating a
determinant, which is what stops the level rendering mirrored.

Morphing and collapse are kept distinct. Morphing blends a vertex toward the
midpoint of its two parents across its tier's band; collapse replaces the
vertex outright, position and UV, once it is past the far edge. Morphing is a
blend, collapse is a cut, and conflating them makes distant terrain
shimmer. Blending has to happen in raw units before the axis swap: mixing a
converted parent with an unconverted child scales the blend by 1024.

Level 0 renders as 26,758 textured triangles over 92 textures, correct UVs,
with LOD selection live (26 patches at LOD 0, 650 at LOD 2 from the start
camera). GC_ENGINE_SHOT writes one frame as a PPM and exits.

Rendering immediately exposed a bug no test could. Positions are i16 offsets
from a per-patch origin, and the obvious rule -- the last non-index STROW row --
is wrong on RAC2, which also uses STROW for per-vertex light records that come
*after* the origin. Taking the last one yields [186, 186, 186, 186] and
collapses the whole level onto the world origin. The entire level then fit in
a 63-unit box and every test still passed: vert_count matched, tri_count
matched, every parent and triangle index resolved, because all of those are
relative to the same broken origin. Only a screenshot showed it. The rule that
works is the STROW immediately preceding the first V3_16 position unpack; with
it, level 0 spans x 86..444, y 46..211, z -477..40, matching the patch
bounding spheres.

That is worth more than the renderer: a parser can be completely
self-consistent and still describe the wrong thing, and where a value is
absolute rather than relative it has to be checked against something
independent.

The frame is flat and dark because there are no normals and no lighting yet.
That is the next task -- the per-vertex light records -- not something to
paper over with an exposure fudge.
 2026-10-10 02:58:43 +02:00
ReaperOfVeriod 71a3185fa5 Resolve tfrag LOD parent linkage: guard absent tiers by their counts 
The four V4_8 parent arrays are classified by VU address, but the
addresses collide: on level 0 patch 1,
unk_indices_2_lod_01_addr == parent_indices_lod_0_addr == 291, and
unk_indices_2_lod_0_addr == indices_addr == 294. Matching on address
alone cannot separate them, and it fails silently -- the arrays still
parse, they just hold the wrong numbers.

Two corrections, and the first one is worth recording because it looked
like a data problem rather than a code one:

Each V4_8 element is 4 bytes and holds four indices, so an array of N
entries is count*4 bytes, not count. An earlier analysis compared the
unpack's element count against the position counts and concluded that
15,000 patches could not hold enough parents. That was the wrong unit;
the data was fine.

unk_06 and unk_0a suppress an empty tier. When the extra vertex-info
count is zero there is no extra array and its address is simply the next
tier's, so guarding those branches on the count lets the address fall
through to the array that exists. That one change took resolved LOD-0
entries from 28,739 to 341,306.

Result: of 886,841 vertex-info entries across all 27 levels, 516,492 are
common and have no link by design, and all 370,349 LOD entries resolve --
29,043 LOD-01 and 341,306 LOD-0 -- with zero failures. Each resolves to a
real first parent, a LOD-01 parent that is always a common entry, and
three positions inside the patch. A common entry never claims a parent.

Morphing can now be completed: a patch knows which LOD to draw, how far
to blend each vertex toward its parents, and where those parents are.
 2026-10-10 02:41:18 +02:00
ReaperOfVeriod 2a5307a157 Add tfrag LOD selection, morph weights and parent linkage 
The block header's float at +0x08 gives L and the switch distances are
[D0, D1, D2] = [6L, 4L, 2L] world units -- 192/128/64 on every level
measured. The distance test uses the patch's bounding sphere truncated
to integers against trunc(D * 1024), not the sphere's centre, so a
patch straddling a threshold takes the coarser mode. Six modes result:
LOD 2, LOD 1 collapse, LOD 1 morph, LOD 0 collapse, LOD 0 morph, LOD 0.

A morph weight runs 0 to 1 across its tier's band. Past the far edge an
entry is not blended but replaced by its first parent -- a collapse.

The tested property is that distance never moves back to a finer mode,
swept across the whole range, because that is what keeps terrain from
popping as the camera moves.

The four V4_8 parent arrays are classified by VU address and an
unrecognised address is now an error, which is a real check: all 18,102
patches still parse. Every recorded parent resolves to real vertex info
and real positions, and no common entry claims a parent.

That is as far as it goes, and the limit is a finding rather than a
rounding error. The addresses are not unique: on level 0 patch 1,
unk_indices_2_lod_01_addr == parent_indices_lod_0_addr == 291, with
unk_06 and unk_0a both zero, so nothing but unpacking order separates
them. Across all 27 levels 57,782 of 374,349 LOD entries resolve, and
316,548 unresolved LOD-0 entries against 28,739 resolved is far too
lopsided to be genuine -- either the arrays are misassigned where
addresses collide, or RAC2's positions_lod_0_count means something other
than RAC1's. Only 3,274 of 18,102 patches carry enough V4_8 bytes to
cover positions_lod_01_count + positions_lod_0_count + extras.

The vertex-info tiers themselves are confirmed: the addresses at +9 and
+11 match the observed V4_16 unpacks and the three tier counts sum to
vert_count on every patch, so the header layout is right and it is
specifically the parent arrays that are open. Ruled out so far:
address-only classification (collides) and one-parent-per-position (fails
on 15,000 patches). Not yet tried: ordering by VU address instead of by
declaration, and using unk_06/unk_0a to suppress an empty tier before
falling through. Written up in plan/level-assets.md.
 2026-10-10 02:36:40 +02:00
ReaperOfVeriod 29fdf89f53 Decode level textures: 8-bit indices through a CSM1 palette 
Adds texture.rs and makes the pipeline write gs_ram.bin, which it had
been skipping. Nothing could be decoded before this: the CLUTs live
there.

A level texture is not pixels, it is palette indices -- width*height
bytes at core_data[textures_base + data_offset], with colour coming from
a 256-entry CLUT at gs_ram[palette * 0x100]. gs_ram is a byte range in
the level's data extent header at offset 0x10, 518,656 bytes on level 0.

Two PS2 conventions have to be honoured and neither is a design choice:
within each 32-entry CLUT group the two middle 8-entry blocks are
swapped (bits 3 and 4 of the index are exchanged, an involution), and
alpha runs 0..=0x80 rather than 0..=255 with 0x80 fully opaque.

The mip chain splits across two buffers: levels 0 and 1 are in the
texture block with level 1 immediately after level 0, and levels 2 and 3
are resident in gs_ram at mipmap*0x100 and pad*0x100. Tfrag textures are
square so every level halves evenly.

Verification is two-layered, because every step here is a table lookup
and a wrong CLUT remap still produces a well-formed image of the wrong
thing. All 1,595 tfrag textures across all 27 levels decode; 96.8% are
fully opaque and 96.3% use more than eight colours. On top of that,
`gc-extract dump-texture` writes any texture or mip level as a PPM.
Level 0 texture 0 comes back as a clean bevelled metal panel with rivets
and specular highlights, and texture 74 as a vent grille whose punched
windows are fully transparent. That tool is worth keeping: it is the
cheapest way to tell a decode that is merely well-formed from one that
is right.

The hash table is regenerated for the 27 new gs_ram.bin files, now 5,186
files and 5,685,541,143 bytes.
 2026-10-10 02:30:56 +02:00
ReaperOfVeriod 48d9506cf1 Decode tfrag texture bindings, vertex colours and light records 
Adds level_core.rs (the uncompressed core index header and texture table)
and extends tfrag.rs with the data a draw call needs beyond geometry.

An ad-gif is five A+D quadwords holding packed fields rather than GS
registers; the level load rewrites them in place before drawing. On disc
TEX0_1.data_lo indexes the level's texture table, TEX1_1.data_lo is the
LOD bias K and data_hi the minification filter, and CLAMP_1 holds both
wrap modes.

Three things are easy to get wrong here:

* K is a signed 12-bit value in 1/16 units, sign-extended by shifting the
  field up 4 and back down. Retail values are -137..-89 mip-scaled;
  reading it unsigned gives 1911 instead of -137.
* TEX1_1.data_hi is MMIN, and 4 decodes as LinearMipmapNearest, not
  LinearMipmapLinear as the raw bit layout suggests.
* RGBA and the light/normal records are not in the VIF lists at all. They
  sit at fixed offsets in the patch data block: rgba_size*4 colours at
  rgba_ofs, vert_count light records at light_ofs+0x10. The tfrag origin
  lives there too -- if no STROW row supplied one, the quadword at
  light_ofs is it.

The texture table checks out on its own terms: 93 entries at index offset
0x2900 on level 0, and 0x2900 + 93*16 lands exactly on the next range.

Verification: an ad-gif index outside the level's table would mean the
geometry and the textures were misread independently, so every one is
range-checked against real entries. All 27 levels: 47,950 ad-gifs over
18,102 patches, all in range, referencing 1429 of 1595 level texture
entries (90%). Each patch's light array is exactly vert_count records and
its RGBA array exactly rgba_size*4.

Still open: LOD selection and morphing, and the texel data itself.
 2026-10-10 02:22:51 +02:00
ReaperOfVeriod 9459f15329 Parse tfrag topology: vertex info, strips, indices and triangles 
Turns tfrag positions into drawable geometry. Every unpack in a patch's
VIF lists is now classified by format *and* destination VU address, using
the VU header at the head of the common list. That table is what tells
the strips from the indices from the parent arrays: they are all V4_8
and only the address distinguishes them.

Strip indices address the vertex-info array, not the position array.
Several vertex-info entries can share a position, so an index resolves
through vertex_info[i].vertex / 2 to reach a position. Vertex-info also
carries UVs and the LOD parent link.

A strip run of n vertices is a GS triangle strip of n - 2 triangles with
alternating winding. The vertex count is biased by -128 when negative,
and the first record of a list is always biased even when positive --
real patches open with -116 or -124, never a small positive number.

tri_count lives in the patch header and cannot be derived from the lists
without walking them, so reproducing it is a strong check: all 27 levels,
18,102 patches, 809,879 triangles, every patch matching its header, every
triangle index inside its LOD's vertex-info range.

Four wrong turns got there, each of which failed quietly rather than
loudly: reading the element size from the immediate instead of the
command, 2-byte payload padding instead of 4, ignoring the inline
payload of STROW/STCOL/STMASK/MPG/DIRECT, and banking strips by arrival
order instead of by LOD. Every one produced parseable, plausible geometry.
Only the header's own triangle count caught them.

Still open: ad-gif texture bindings, per-vertex RGBA and light records,
and the LOD switch and morph rules.
 2026-10-10 02:15:32 +02:00
ReaperOfVeriod 36641b4cf8 Parse tfrag terrain blocks and decode all vertex positions 
Adds gc-formats::tfrag and a gated real-disc test that decodes every
terrain patch on all 27 levels.

Two rules decide whether this works at all, and both were wrong on the
first attempt:

The vector format is encoded in the VIF command byte, not the immediate.
Bits 2-3 are vn (component count minus one) and bits 0-1 are vl, giving
element_size = ((32 >> vl) * (vn + 1)) / 8. Terrain positions are vn=2,
vl=1 (command 0x69) at 6 bytes. Reading the format from the immediate's
low nibble -- the more commonly quoted layout -- yields 12 or 16 bytes
and misaligns everything after it.

Unpack payloads pad to 4 bytes, and several non-unpack commands carry
inline data: STROW/STCOL 16, STMASK 4, MPG num*8, DIRECT/DIRECTHL
imm*16. Ignoring those desynchronises the walk, and a desynchronised
walk still finds code words that look like positions.

Positions are also not all in one list. vert_count is the total across
three tiers, the LOD-2 list precedes the common one in offset order, and
the LOD-0-only list starts at shared_ofs + lod_1_size*16 rather than at
an offset of its own. Scanning only the common list matches about a
quarter of patches, each exactly -- which is what made it convincing.

The invariant is that every patch yields exactly its own vert_count. It
caught both errors, so tfrag_positions returns None on a mismatch rather
than partial geometry. All 27 levels: 18,102 patches, 821,439 vertices,
zero mismatches.

Still open: strips, triangle indices and the RGBA vertex-info arrays.
Positions alone are not drawable.
 2026-10-10 02:07:02 +02:00