a94704de18243b0eabbdae35eaa4c7aff8a3d707
65
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  ReaperOfVeriodandreaper
|
8030dbf006 |
close: tie LOD distance ordering exceptions - 80/1864 classes have mid=8192 far=8190
Tie class headers ordered near < mid < far on most classes. Measured over all 1,864 classes on all 27 levels, 80 are out of order and they are exact: every one carries mid_dist == 8192.0 and far_dist == 8190.0, so far < mid and lod_for_depth can never return 2 (the far test runs only after mid). Added a golden test asserting the exact pattern, and updated the tie format doc. Co-authored-by: reaper <reaper@users.noreply.github.com> |
||
|
ReaperOfVeriod
|
572cd1898d |
Record a defect packet slicing found: the tie block terminator is early
Range measured: every tie LOD block on the disc, 2,177 blocks. The last packet's end is known two ways. Its length comes from its record's packet_quadwords, and every other record's length is confirmed by the next record's pointer -- so only the last packet of each table is unconfirmed. Comparing its computed end against where parse_tie_block stopped: the last packet's computed end lands 2 to 16 bytes past the located 0xf6, on all 2,150 blocks measured. Two readings fit. Either the terminator was located too early -- inside the last packet's data -- or the last packet's length over-predicts by a small amount. The first is likelier: a length error on the last record of every table would have to be a coincidence. Neither is settled here, and the test asserts the measurement rather than the reading. Why the existing test missed it: "2,177 of 2,177 blocks end on 0xf6" is satisfied by a false terminator exactly as well as a true one -- it checks that the byte found IS a 0xf6, not that it is the block's end. A test that confirms the shape it walked to will pass on the wrong byte. The cost: the block extent is short by that much, so payload and padding under-report and anything computed from them is correspondingly short. The block starts are unaffected and RAM-confirmed. Recorded as a limitation with the measurement attached, not fixed. A terminator packet data cannot imitate is its own task, and guessing one would be the plausible-sounding alternative this project has been bitten by. A packets() accessor was tried and removed for the same reason: with the extent short it silently yields the wrong answer, and an API that is quietly wrong is worse than no API. |
||
|
ReaperOfVeriod
|
1cfe744501 |
Settle the tie packet length, and correct the retraction that hid it
Range measured: every tie LOD block on the disc -- 2,177 blocks, 13,042
consecutive record pairs.
A record's pointer is where its packet starts, so the distance between
consecutive pointers IS a packet's length, with no reference to any field.
Comparing that against the record's own fields:
packet_quadwords = vert_ofs + vert_size + scissor_size
12,927 of 12,927 record pairs whose strip_count is zero
0 of 115 record pairs whose it is not
The partition is exact: every record with a non-zero strip_count is an
exception and no record with a zero one is. So the fourth term is 0 quadwords
whenever strip_count == 0, and something else -- measured 0..19 quadwords, and
NOT equal to strip_count -- whenever it is not. R&C1's strip_count is a strip
count and strips are variable-length data, so a non-zero count most likely
adds per-strip bytes this cannot see; that is a hypothesis, labelled as one.
TieRecord gains packet_quadwords and packet_quadwords_are_known. The latter is
the honest gate, and the disc test asserts both sides of the partition rather
than the aggregate.
This also corrects this file's own retraction of the same formula, and the
retraction was the error rather than the claim. A formula of this shape was
derived at 61 of 61 pairs, then "retracted" as not reproducing (0 of 13,042).
The re-measurement had swapped strip_count for scissor_ofs in the fourth term
and put the blame on the block set. Re-measured with the term the first pass
had, it matches on 12,927 of 13,042 and the 115 exceptions are exactly the
non-zero strip_count records.
Worth recording because it is the same shape as two other entries in this
file: a retraction is not automatically more careful than the claim it
retracts, and the re-measurement that "disproved" this one changed the thing
being measured. The lesson on the other side is what saved it -- the first
measurement was over 61 pairs, not a population, and had it stayed there the
formula would have looked universal. A small sample and a bad re-measurement,
near-misses in opposite directions.
What this unblocks: the payload can be sliced per packet, which was the
prerequisite for decoding what it holds.
|
||
|
ReaperOfVeriod
|
4abce518e5 |
Decode the overlay's load base, from its own profiler table
Range measured: all 27 levels' overlays. The overlay carries the game's profiler menu definition -- a run of 8-byte padded names (normal, camera, cam+chr, volume, bars, render, mobys, ties, shrubs, cpu/gpu, occl, ...) followed by runs of absolute RAM addresses that point back at those same names. On every level the four addresses after normal/camera/cam+chr/volume, each minus its target name's overlay offset, give the same number: 1,735,792. That is field_00 - 16, where field_00 is the header word that read 1,735,808 on all 27 levels with nothing attached to it. The overlay is loaded with its 16-byte header stripped, so overlay offset X sits at RAM X + 1,735,792. OverlayHeader::ram_address carries it, and the disc test asserts the derivation holds on all 27 levels. Confirmed, not only derived: level 0's RAM 0x1acf80 holds "normal\0camera\0cam+chr\0volume\0", byte-identical to overlay 0x5310, and 0x5310 + 1735792 == 0x1acf80. All four names land where the overlay's own table says. `gc-trace verify-overlay-base` runs that check, savestate-gated the same way as the tie and tfrag ones. The overlay is relocated, not copied: only 37.67% of level 0's is byte-identical at the base, and the 256-entry sine table -- at overlay 0xb650c -- is not at 0xb650c + base. So the base locates the overlay's image, but individual tables inside it move. That is the shape of a loader applying relocations, and it means the base is necessary but not sufficient for reading the game's tables. The value of the base is that any overlay offset can now be carried to a RAM address, which is the prerequisite for reading the game's live level state -- the thing the tie relocation and tfrag lighting checks had to work around. Two things get deleted rather than finished. The collision u16 table was probed four ways -- magnitude distribution, level-independence, run starts, run lengths -- and every one came back level-dependent: the run-start pattern is identical on 1 of 19 levels. So the first 1024 bytes of the collision payload are data, not a fixed table, and nothing was decoded. Recorded as the negative it is rather than left as four half-findings. |
||
|
ReaperOfVeriod
|
55ea619b4c |
Decode the tie class header's +0x0c word: three LOD record counts
Range measured: every tie class on the disc, 1,864 classes over all 27 levels. Method: read the word at class +0x0c as four bytes, and read the record counts independently from the blocks themselves -- follow packets[lod] to the block, walk to the 0xf6 terminator, and count the 16-byte records the block header's data_offset names. Two paths to the same number, so the check is not circular. Result: the byte at +0x0c is the LOD 0 record count, +0x0d is LOD 1's and +0x0e is LOD 2's, on 1,864 of 1,864 classes. Classes with an absent LOD agree too: a zero packets[lod] gives a zero count and the byte is zero. This closes the file's oldest lie about the tie header. The comment in TieClassHeader::parse read "0x0c: the 64 light-slot normals offset. Read past, not decoded" -- inherited from R&C1's layout, where the word genuinely is an offset. On R&C2 it is not an offset at all; it is three counts, and R&C1 carries the same quadruple at +0x20. Skipping the field was the right action on the wrong information: the field was unknown only because R&C1's map was applied to R&C2's bytes. TieClassHeader gains packet_counts, and tie_class_header_counts_records_per_lod asserts the agreement. The fourth byte, +0x0f, is left unnamed. It ranges 1..15, and the obvious following of R&C1 -- the shader/ad-gif total -- disagrees on 1,827 of 1,864 classes, which is 2% agreement where a count would expect 100%. It is recorded as unidentified rather than given a name that fits a fiftieth of the population. |
||
|
ReaperOfVeriod
|
f1f0b517a5 |
Pin the extracted tree, which every disc test reads and nothing verified
data/gc-files.tsv pins the ISO files, and unpacks_and_matches_the_committed_table checks that unpacking reproduces them. But the tie, collision, tfrag and chunk tests do not unpack -- they read ~/Documents/gc-extracted, a directory on disk that nothing in the repository checks. A stale or hand-edited file there would have every measurement built on it still pass, quietly. Three claims this session returned confident numbers while describing the parser rather than the disc -- has_uniform_record_length, "1,509 chunks uniform", the moby class-ID scan -- and a test whose data source is unverified is that same hazard one level down. data/gc-extracted.tsv now records SHA-1 and size for all 5,237 files in that tree, 324 KiB, sizes and hashes only, no content. It is the output of `gc-extract unpack`, whose ISO files are already pinned by the other table, so it is grounded transitively rather than by assertion. the_extracted_tree_matches_the_committed_table walks the table and checks every row's size and hash against disk. It asserts all 5,237 are checked -- so a truncated table fails rather than passing on a prefix -- and reports the first few missing or mismatched paths when it does. Current result: 5,237 of 5,237 match. This does not retroactively validate the three retracted claims. What it does is remove the possibility that any of the surviving ones rests on a directory that could have been anything. 246 tests, clippy clean. |
||
|
ReaperOfVeriod
|
2d3880b624 |
The class-ID retraction goes further: the claim was wrong before the fix
Pinned down the one slot that still cleared 100x after the record-boundary fix,
and it is the artifact this section was written to guard against.
The discriminator the original version lacked: a slot that really carries moby
class IDs must SPAN the ID space, so some of its values land above a level's moby
count. The artifact does the opposite.
type slot enrichment distinct values ABOVE the moby count
0xd76cc453 6 305x 9 0
0xf2ef4275 34 64x 9 341
0xf2ef4275 7 46x 16 412
0xf2ef4275 14 34x 16 432
0xd76cc453 9 32x 28 702
The 305x slot's nine values are 1, 3, 4, 5, 10, 11, 12, 13, 15 -- all under a
level's ~215 moby classes, so every one scores as a hit and 65536/215 = 305x.
Nine small numbers is not a class-ID array. Every slot that actually spans the ID
space scores 10x to 64x, which is chance.
So the original claim's own numbers gave it away and it was not read that way:
"5 to 42 distinct values at 153x to 289x" is exactly the artifact's shape -- a
handful of small values, none above the count. The record-boundary fix was never
what made this claim wrong. It was wrong when written, and the retraction is
complete rather than the partial one I committed two turns ago.
The test now separates the two quantities: slots clearing 100x (currently 1,
expected to remain the artifact) and slots clearing 100x WHILE ALSO having values
above the moby count (currently 0, must stay empty). That second number is the
one that means something, and a slot appearing in it would be the first genuine
evidence of a class-ID array anywhere in the container.
Worth noting the per-SLOT bookkeeping: the count of values above the moby count
is keyed by (type, slot), not by type. A first attempt keyed it by type and the
artifact type passed the check on the strength of its ordinary slots -- 783 of its
own values sat above the count at other slots. The check has to be about the
individual slot, or one good slot launders a bad one.
245 tests, clippy clean.
|
||
|
ReaperOfVeriod
|
d159195510 |
General guard for the over-read class, and the audit showing it was unique
Searched for the pattern that caused the over-read -- a record length derived from
the first difference between two offsets:
grep -rn "\[1\] - \[0\]" crates/gc-formats/src/*.rs
crates/gc-formats/src/chunks.rs:357 (offsets[1] - offsets[0])
One hit, in the one parser that walks an explicit offset table.
The others slice contiguous regions with chunks_exact -- shrub's 8-byte palette
records, tie's 16-byte packet headers, tfrag's 4-byte tables. There a length
cannot run past the next element, because chunks_exact derives each boundary from
the region's own length rather than from a sample of the spacing. Shrub's mesh
packets carry an explicit offset AND size, so they state their length instead of
inferring it.
The damage was bounded by a structural property worth stating: only a format with
an offset table AND a derived stride can over-read, and exactly one format had
both. That is a more useful fact than the retraction was on its own -- it says
where else to look if another one turns up.
The guard is positive rather than a comment.
chunk_records_neither_overlap_nor_leave_gaps asserts across all 27 levels that
every record but the last is exactly its own offset delta long, that there are
1,509 chunks, and that 1,252 have ragged deltas. A regression to a single sampled
stride fails immediately and names the chunk and the record.
245 tests, clippy clean.
|
||
|
ReaperOfVeriod
|
92aa0fe49f |
Audit what the record-boundary fix contaminated: one retraction, one survivor
The fix changed record lengths in 1,252 of 1,509 chunks, so every committed claim
computed over record bytes needed re-checking rather than assuming. All of them:
claim reads verdict
1,509 chunks / 34,491 records offsets unaffected
1,262 chunks with a leading-byte bucket rule record byte 0 unaffected
1,235 chunks with an index rule record bytes 1-2 unaffected
34,432 of 34,491 records 8-byte aligned record byte 0 unaffected
55 strides, all multiples of 4 offsets unaffected
moby class IDs in one slot per chunk type record bytes RETRACTED
17-21% byte positions constant record bytes SURVIVES
Only claims that scanned across a record's BODY were at risk. The rest read a
record's first three bytes, which an over-read never touched -- it appends to the
end. That asymmetry is why the damage was contained to one claim, and it is the
argument for preferring claims anchored at record starts.
The survivor was the obvious candidate for contamination, so it is worth being
precise about it. Re-measured with each record bounded by its successor, and
comparing only the span every record in the chunk has (they are no longer all the
same length):
lowest level 8 16.7%
highest 20.0%
Still inside the documented band. Previously this lived only in a doc comment on
the layout test; it is now asserted, in
chunk_records_share_a_layout_measured_by_constant_byte_positions, over all 27
levels with a 10-30% band so a real change reports itself rather than sliding.
The module documentation is corrected to the measured range and now says outright
that records in one chunk are not all the same length, since that is the fact a
reader would otherwise have to rediscover.
244 tests, clippy clean.
|
||
|
ReaperOfVeriod
|
4fab4d0359 |
Fix the record boundaries after all, and retract a claim built on the bug
I reverted the record-boundary fix last commit on the grounds that "fixed-length
records plus inter-record padding" and "variable-length records" produce the same
offset table. That reasoning was wrong, and cheaply so.
stride is the FIRST offset delta. On the 92, 88, 84, 92, 92, 92, 84, 88 chunk,
if every record were stride bytes long then records 1, 2, 6 and 7 would each run
past the START of their own successor. Records are addressed by an explicit
offset table; they cannot overlap. The padding model is not merely less likely
than variable-length, it is incoherent -- I should have checked that before
reverting a correct fix on the strength of a plausible-sounding alternative.
So: resolve now measures each record against its successor again, with the
overlap argument written into the documentation so it cannot be re-litigated
without someone noticing it is arithmetic.
THE RETRACTION, which is the substantive part:
Three tests broke. Two were straightforward, but the third was not.
* constant_positions indexed records as if they were all the same length. Now
that they are not, it compared past the short ones. It takes the minimum
length instead.
* "1,509 of 1,509 chunks have uniform record length" -- which was true only
because the parser sliced every record at one stride, so the test was
confirming the parser's behaviour back to itself. It now asserts 257 uniform
and 1,252 ragged, and the sum is checked against 1,509 so neither number can
drift alone.
* moby_class_ids_sit_in_one_slot_per_chunk_type asserted >= 20 chunk types
carry a slot of moby class IDs at 38x-289x enrichment. It now finds 2, of
which 1 clears 100x. RETRACTED. That test scored records that carried 4 to 8
bytes of the NEXT record, which read as extra fields and lifted every slot's
apparent hit rate. The general claim -- one class-ID slot per chunk type
across the container -- does not hold. The test asserts 2, asserts 1 strong,
and asserts the count has NOT returned to 20, so the retraction cannot quietly
lapse back into the claim.
That is a committed finding withdrawn on the strength of a parser bug. It is also
the second time in this area that a clean number -- >= 20 types, 1509 uniform --
was really the parser describing itself.
243 tests, clippy clean.
|
||
|
ReaperOfVeriod
|
b9e9835639 |
A parser defect: one chunk type is not fixed-stride, and over-reads
[75 42 ef f2 71 b5] -- the signature that is both level-specific and
variable-length -- makes three chunks per level, 81 total:
stride 76 deltas [76, 76, 76, 76, 76, 76, 76, 76] uniform
stride 92 deltas [92, 88, 84, 92, 92, 92, 84, 88] NOT uniform
stride 104 deltas [104, 104, 104, ...] uniform
Identical on all 27 levels: 27 chunks of each shape, three shapes, no others.
So "this signature has strides {76, 92, 104}" is a misdescription. It has three
chunk shapes, one of which is not fixed-stride at all. The parser derives one
stride per chunk and resolve slices every record at that length.
The cost, precisely: records are read at the offset table's positions, so every
record STARTS correctly -- no misalignment. But each is sliced at 92 bytes, while
two of the eight are 88 and two are 84, so four records over-read by 4 and 8
bytes and carry the head of the next record inside them. Bounds checking holds;
this is a data-correctness bug, not a safety one. It is precisely the kind that
produces plausible-looking fields offset by four bytes, which is how it would
have gone unnoticed had the stride table not been collected per signature rather
than inferred per region.
It also explains why this signature was previously flagged as breaking the 8-byte
alignment rule: its records genuinely are not a uniform multiple.
The right fix is for resolve to use offset deltas for a chunk whose deltas are
not uniform. Not taken here -- the delta sequences are asserted first, so if the
data ever becomes uniform the test fails and asks for the model to be revisited,
rather than the model quietly absorbing the exception.
Asserted in chunk_type_75_ef_has_one_variable_length_chunk_per_level, with the
exact sequences [76x8], [92,88,84,92,92,92,84,88], [104x8] and 27 of each.
A near-miss worth recording, because it would have made a tidy false claim: the
leading u16 of these records appeared to step by exactly 0x50 (80) across all nine
of the 76-byte chunk. Reading at the real offsets gives b1a5, a529, 047e, a529,
04f0, a529, 0562, a529, 05d4 -- no progression at all. The apparent 80-byte step
came from comparing the wrong bytes, and "records are indexed by 80" is exactly
the kind of finding that gets reported on sight.
243 tests, clippy clean.
|
||
|
ReaperOfVeriod
|
af62032ddc |
Most of the chunk container is level-independent: 65 of 74 signatures
Comparing every chunk record against the same record index in the other 26
levels, across all 74 signatures:
65 of 74 every record byte-identical in all 27 levels
9 of 74 level-specific
0 ragged no signature appeared with different record counts per level
The nine that vary:
[5d 44 13] [75 42 ef] [7f 44 ee] [82 c4 2f] [96 c3 37]
[a2 44 a5] [c6 c3 ef] [df c3 ea] [f3 c4 b9]
This reframes the container. 65 of 74 chunk types are static tables the game ships
unchanged in every level -- constants and lookups, not level content. The largest
by record count are all in that group: [38 c4 7e] stride 328 with 90 records per
level, [a9 c4 d8] stride 308 with 90, [3d c4 9e] stride 368 with 77.
The consequence is about method, and it is worth being blunt about. For a
level-specific type, comparing levels is the strongest lever available: a field
that moves with the level is identifiable by correlating it with something known.
For the other 65 there is nothing to compare -- one copy of the data, repeated 27
times -- so they must be decoded structurally, from what the bytes are. A future
decoder that reaches for per-level comparison as its first tool will find it
useless on the majority of the container.
[75 42 ef] is both level-specific AND the one signature with three strides. A
type that is both variable-length and level-varying is the most informative one
here, and is where per-type field decoding should start.
How this was found, including the wrong turn:
The first look at [53 c4 6c d7 9a 43] (stride 108) showed 45 of 108 byte positions
varying WITHIN a chunk and 810 records all distinct -- which reads like per-record
data. Counting distinct values per u32 slot across all 810 records contradicted
that: 29 to 30 distinct values across 27 levels, i.e. about one per level.
Both readings were wrong because they measure different things. The 45 positions
vary between records within a level; the u32 counts were low because the same 30
values recur in every level. Comparing record i across levels settled it: all 30
records are byte-identical in all 27 levels, and zero byte positions vary across
levels.
Had the u32 distinct counts been the only measurement, the conclusion would have
been "one value per level" -- the opposite of the truth, describing a per-level
parameter table that does not exist. The within-chunk and across-level views had
to be taken together to get a true one.
Asserted in most_chunk_signatures_are_level_independent. 242 tests, clippy clean.
|
||
|
ReaperOfVeriod
|
59dba3e58d |
Chunk records: the stride table, and the exception pinned by name
Walked every chunk on all 27 levels, grouped by six-byte header signature:
1,506 chunks, 34,415 records, 74 distinct signatures
55 distinct strides, 52..640, every one a multiple of 4
73 of 74 signatures use exactly one stride
The exception is asserted rather than skipped, because "every type is uniform"
is precisely the claim that quietly stops being true:
[75 42 ef f2 71 b5] strides {76, 92, 104}
That is the variable-length type the module docs had referred to indirectly. It
is now pinned by signature and to those three strides exactly.
The 4-byte alignment is a result, not an artefact. A stride recovered from one
chunk's first-to-second record delta can inherit alignment from where that chunk
happened to sit; collecting all 55 strides across every type and requiring all
of them to be divisible by 4 rules that out, because a single off-by-one in one
signature would show. Asserted in
chunk_record_stride_is_four_byte_aligned_and_uniform_per_signature.
What this settles is the shape the decoder must handle: uniform stride per type,
4-byte aligned, one named exception. Field offsets within a record are now the
same for every record of a type, which is what makes per-type decoding
tractable. The fields themselves are still unnamed.
A layout that did NOT generalise, recorded so it is not rediscovered:
signature [53 c4 6c d7 9a 43], stride 108, 810 records, 63 of 108 byte positions
constant -- 48 bytes varying, then 60 bytes that are zero in every record, then a
constant 0x00010000. That reads like an inline buffer this type never uses, and
would have made a tidy field story. Requiring `stride == live span + zero run`
across every type with 50+ records per chunk gives 0 of 9. The zero tail belongs
to that one type and means nothing general.
Third occurrence of one editing mistake, and it is worth naming as a habit
rather than an accident: inserting a test before an existing one strands that
test's #[test] attribute on the new doc comment and silently unregisters it.
It has now happened three times, each time surfacing only as clippy's "never
used" and never as a missing test. Fixed by appending new tests at the end of
the file instead of anchoring them to an existing function, and verified by
listing registered tests rather than trusting a count.
241 tests, clippy clean, all five recent tests confirmed registered.
|
||
|
ReaperOfVeriod
|
28ab7e094b |
SBlk head characterised: four constants and one word that is not a count
sound_bank's head was parsed but never characterised. Across all 27 levels the
SBlk region opens identically for its first twelve bytes:
+0x00 "SBlk" +0x04 1 +0x08 4 +0x0c "DAW\0" +0x10 0
and +0x14 is the only word that varies. It varies with its low sixteen bits zero
on all 27, leaving 0x0092..0x0103 -- 146 to 259.
That is the shape of a record count, and it tracks the body length at close to
100 bytes per unit. It is not one:
body / field_14 91.2 .. 110.8 across 27 levels
any stride 0..80 with body = stride*word + constant exactly: no
So the test asserts the negative too. If a stride ever makes the relation exact
on all 27, sound_bank_sblk_head_is_constant_apart_from_one_word fails and asks
for the record size to be revisited. A near-miss at 100 bytes/record is exactly
what hardens into a name the next time somebody glances at it; the constants are
also checked individually so a real deviation names which one moved.
The DAW region begins with sixteen zero bytes then block-structured data with
16-byte periodicity and slowly varying nibbles -- consistent with block
compressed audio, not decoded. Recorded as such.
Two mistakes, both mine, both the same shape as earlier ones:
1. The constructor bound `format_tag` with a `let` BEFORE the two words that
precede it in the file, consuming bytes 4..8 and shifting every field after
it. The struct literal listed the names in the right order; the statements did
not. A scratch script reading the same bytes gave the opposite answer, which
is what exposed it. Fixed, with a comment, because "read the lets in file
order" gets undone otherwise.
2. Inserting the new test ahead of an existing one stranded that test's #[test]
on my doc comment -- silently unregistering
tie_vertex_element_is_16_bytes_with_fixed_texcoords. This is the SAME mistake
as two commits ago and I made it again. It again surfaced only as clippy's
"never used" rather than as a missing test. Restored and verified by listing
the registered tests rather than trusting the count.
240 tests, clippy clean, both tie vertex tests confirmed present.
|
||
|
ReaperOfVeriod
|
e86de3c73e |
Tie i16[0..2] are the vertex positions, with a control in the test
The three slots ahead of the index were "unproven position" for want of a test
that could fail. One is available on the disc alone, with the control built in.
LOD 1 and LOD 2 of a tie class are the same object at two levels of detail, so
anything that locates a vertex in that object's own geometry is shared between
the two blocks; a different class' LOD is an unrelated object. Intersecting the
sets of i16[0..2] triples both ways turns "is this a position" into a
measurement. Levels 0-5, ordinary-layout elements only:
same class, LOD1 vs LOD2 96.1% 107 classes, 107 nonzero
different class, control 0.4% 2,144 pairs
A factor of 240. Per level the same-class figure runs 94.4 / 97.9 / 98.1 / 96.7 /
96.8 for levels 0-4. The control stays under 1.0% except for one level-1 pair at
45.9% -- two classes sharing most of their geometry, which is what duplicate or
variant models should look like. Reported, not excluded, and the assertion is on
the mean rather than the max so it does not have to be.
Asserted in tie_vertex_slots_0_to_2_are_shared_positions, including the ratio
(same > 20 * diff) so a change that collapsed both numbers fails rather than
passing twice, and a nonzero-overlap check so no class can silently stop
agreeing with itself.
What this rules out is the point. An index, a handle or a per-vertex counter
would not be shared at 96% between two independently built LOD meshes, nor sit at
0.4% between two objects. With 10.7k-13.2k distinct values per slot over the
full i16 range, these are positions in the object's own frame. Scale and axis
order stay unnamed -- nothing here says centimetres, and axis order is
unconstrained because nothing here assumes one.
Two things worth recording about the process:
1. The diagnostic line printed "1.0%" for 99.4% because the format was
`{same:.1}%` with no `* 100`. The assertions compared fractions and were
always correct; only the readout was wrong. Same family as everything else in
this log -- the readout disagreeing with the claim.
2. Inserting the new test in front of an existing one stranded that test's
#[test] on my doc comment, which silently unregistered
tie_vertex_element_is_16_bytes_with_fixed_texcoords. It showed up as
"never used" in clippy, not as a missing test. Restored; both tie vertex
tests are registered and 239 tests run.
Element layout now reads:
+0x00 i16[0] position x +0x06 u8 index 0..=254
+0x02 i16[1] position y +0x07 u8 layout tag, 0 in 93.4%
+0x04 i16[2] position z +0x08 i16[4] texcoord u
+0x0a i16[5] texcoord v +0x0c/+0x0e i16[6], i16[7] unnamed
|
||
|
ReaperOfVeriod
|
06cf2fd46d |
Tie vertex slot 3 is a byte-sized index, not a coordinate
Splitting the 95,612 elements of levels 0-2 by i16[3]'s HIGH byte, which is the
layout selector:
0x00 89,341 93.4%
0x03 5,344 5.6%
144 other selector values 927 1.0%
146 selector values in all. Not a two-way switch, and the second case is not 0x01.
Inside the ordinary layout i16[3] is not a coordinate: 201 distinct values, all
within 0..=254, with 0 most common. That is an index. So the selector and the
index are not two independent halves sharing a slot -- it is one byte. The low
byte is the index, the high byte is the layout tag, and the ordinary layout never
sets the high byte. Asserted.
Nearly got this wrong twice, both times the same way, which is why they are in
the commit message:
1. The first pass gated the range on `w >= 0` instead of on `(w >> 8) == 0`,
letting in the other layouts' large positive values. That reported "564
distinct values in 0..=32664" -- a number that looks exactly like a
coordinate and is really two populations unioned. The selector byte is the
filter. The filter is part of the claim.
2. Two constants were guessed rather than measured (selector cardinality 159,
and an upper bound that ignored non-ordinary layouts) and both were wrong.
The cardinality is 146. Guessing a constant and then asserting it is how a
measurement becomes a tautology.
Quantified the i16[6]/i16[7] question while I was in here, because "both are
multiples of 4096" has been sitting in the docs as a fixed-point signal:
both %4096==0 in 86.5% of elements
expected if the two residues were independent: 84.3%
enrichment 1.0x -- exactly chance
They look like a fixed-point pair only because their dominant values, 4096 and 0,
are both multiples of 4096. Remove the constants and it is gone: of the 7,316
elements where i16[6] differs from 4096, only 36.7% are still multiples, and of
the 9,905 where i16[7] differs from 0, ONE is. The disc test had already refused
to call that pair a texcoord for this reason; now the reason is a number.
Remaining slots in the ordinary layout, recorded: i16[0..2] read as position but
are unproven (10.7k-13.2k distinct each, full i16 range), i16[4]/i16[5] are the
texcoords, i16[6]/i16[7] unnamed.
Also fixes two clippy map_or lints and an import that a stray --fix had broken.
238 tests, clippy clean.
|
||
|
ReaperOfVeriod
|
26a0918178 |
Gate the PCSX2 claims as tests; two corrections fall out of writing them
The audit two commits ago listed three claims as "not assertable here -- needs a
savestate". With PCSX2 running they are assertable, so they now are. gc-trace
already exposed gc_trace::Savestate::ee_ram_owned(); gc-extract just needed it as
a dev-dependency. Both tests are gated on GC_SSTATE and skip cleanly without it,
so a normal checkout pays nothing.
Writing them corrected two claims, which is the main reason they were worth
writing as tests rather than staying prose.
1. The collision payload is NOT "byte-for-byte identical". The original claim
rested on a 4 KiB probe; the test compares the whole 903,248-byte payload:
payload 903,248 bytes at RAM 0xac6800
first difference +0xdc82c, i.e. 903,212 bytes in
bytes that differ 6 (0.00%)
903,212 of 903,248 bytes are verbatim and 6 are not, all in the final 36.
"Verbatim" was an overclaim on a probe that never looked at the tail. The test
now measures the whole payload and reports the extent, asserting only that the
divergence is beyond the head.
2. The tfrag relocation base is the TABLE address, not the block base. The delta is
8,683,584 = 0x848040 = block base 0x848000 plus table_offset 0x40 -- which is
exactly what TfragHeader::data's own documentation says, "relative to the
table". The earlier claim said "the block's RAM base", off by exactly the table
offset.
Also: +0x10 and +0x38 do NOT share one constant. The aggregate "1,665 fields
relocate by one constant" was true only because +0x10 carried it; lumping them
hid that. The test asserts +0x10 and REPORTS +0x38's delta rather than assuming.
The patch-rewrite figures also restated on aggregate rather than median, which is
why they read lower than the 7.80%/8.12% medians recorded earlier:
patch bytes rewritten -- cannot morph 1.9% of 334, can morph 2.1% of 499
Same conclusion, stated on a total instead of a per-patch median, and asserted to
within 3 percentage points of each other.
Net: the audit table in docs/formats/core-data-regions.md now has no "prose only"
rows for disc-or-savestate-readable claims.
|
||
|
ReaperOfVeriod
|
9f938f47de |
Assert the tie vertex layout finding in a test, and audit which claims are asserted
The i16[3] high-byte finding -- the most recent and most subtle result of this stretch, and the one I corrected twice -- lived only in prose. Two commits is exactly how long a claim can sit unasserted before it drifts. It now has a test: tie_vertex_layout_is_selected_by_i16_3_high_byte which reproduces the figures exactly and, importantly, asserts both directions: (4,5) and (6,7) are above 0.85 and 0.80 in the ordinary group, (6,7) is BELOW 0.99 so it cannot silently become the constant I once mistook it for, (0,1) and (2,3) are under 0.20, and every pair in the non-zero group is under 0.15. A future change in either layout now fails loudly. Also audited the rest of this session's claims against the test suite and recorded the result in the format doc: tie vertex element is 16 bytes asserted layout selected by i16[3] HIGH byte asserted (this commit) blocks self-delimiting on 0xf6 asserted per-block packets == 0 means no block at that LOD asserted packets[0] == 256 on 1,864 of 1,864 asserted chunk record length uniform per chunk asserted moby class IDs in one body slot per type asserted collision payload verbatim in RAM NOT assertable here -- needs a savestate tfrag patch bytes all rewritten by LOD same morphing is unobservable in that image same The split is deliberate: disc-readable claims are asserted against the disc, savestate-dependent ones live in plan/pcsx2-verification.md with their method. What is not acceptable is a disc-readable claim living only in prose -- which is how the tie i16[3] finding spent two commits before it had a test, and how several older claims drifted before the roadmap reconciliation caught them. |
||
|
ReaperOfVeriod
|
180fc27b04 |
Locate the tie texcoord field at i16[4]/i16[5], and rule out the (6,7) artifact
Scoring each adjacent i16 slot pair of the 16-byte tie vertex for 12-bit fixedness over 36,501 level-0 vertices locates the fields: (i16[0], i16[1]) 1.1% not a texcoord (i16[2], i16[3]) 4.8% not a texcoord (i16[4], i16[5]) 73.6% <- the texcoords (i16[6], i16[7]) 76.6% <- the near-constant 4096/0 pair (6,7) scores HIGHER and is not a texcoord. It is the near-constant 4096, 0 pair: 4096 is 1.0 in 12-bit fixed and 0 is zero, so both are trivially multiples of 4096. That is the constant artifact again -- a fixed pair would score 100% anywhere. The field is (4,5), with (0,1) and (2,3) ruled out at 1.1% and 4.8%. The first four slots span the full i16 range with 10,803 to 13,421 distinct values each, so they are neither texcoords nor small fields. The vertex base is confirmed to be vert_ofs alone. Adding the shader or control region makes it worse -- vert_ofs only 73.6% bfc + vert_ofs 69.8% bfc + control_count*control_size + vert_ofs 66.7% -- so the 73.6% residual is not a mislocated base. Worth recording because vert_ofs (3..7) numerically overlaps the shader region (5*shader_count, 5..20 quadwords), which made a combined base look plausible; measurement says the simple one is right. The test asserts (4,5) above 60% and (0,1)/(2,3) below 5%/15%, and names the (6,7) artifact in its failure message so the higher-scoring pair cannot be mistaken for the answer later. |
||
|
ReaperOfVeriod
|
35b087c882 |
Establish the tie vertex element is 16 bytes, and correct the roadmap's tie entry
With TieRecord identified as RAC1's TiePacketHeader, each record's leading word points at its packet in the block payload. Following it and stepping vert_size elements from vert_ofs reaches the vertices: 95,612 of them on levels 0-2. Element size settled by comparison, not assumed. Requiring the texcoord pair to be 12-bit fixed -- both words multiples of 4096, which is how 0x1000 = 1.0 reads: element 16 bytes : 78,919 / 95,612 = 82.5% element 8 bytes : 45,661 / 95,612 = 47.8% element 12 bytes : 44,834 / 95,612 = 44.8% Sixteen wins by a factor of 1.7 over the nearest alternative. Observed layout, 8 x i16: the first three slots span the full i16 range with 10,803-13,421 distinct values each; i16[4] and [5] are the texcoords and are both multiples of 4096 on 82.5%; i16[6] reads 4096 on 85.8% and i16[7] reads 0 on 84.0%. Not established: what any field means, and why the texcoord rate is 82.5% rather than 100%. The residual tracks record shape -- (shaders 2, control 7, size 2) hits 60%, (shaders 1, control 18, size 5) hits 89% -- so it is not one bad block but something per-packet this reading does not capture. Worth recording that this is the OPPOSITE of what the disc alone concluded two commits ago. Reading the header word at +0x00 as an element count gave 0 of 922 blocks dividing; it is the byte offset to the payload, so the table it bounds has 16-byte records. The number was right and the reading was wrong, which is what the packet-header identification corrected. The roadmap's tie entry was also stale -- it still said "a 16-byte record is withdrawn", which the packet-header work had already reversed. Rewritten from the commits: both indirections, self-delimiting blocks, the TiePacketHeader identification with its 15,219/15,219 rule, the 16-byte vertex element, and what remains open. A test asserts the element size against both alternatives so the 82.5% figure cannot be quietly replaced by an assumption. |
||
|
ReaperOfVeriod
|
54862b64ee |
Identify the tie record as RAC1's TiePacketHeader, and retract a formula
The 16-byte tie record is R&C1's TiePacketHeader offset for offset: a data pointer at +0, then R&C1's byte fields at +4 through +0x10. The identification is not a resemblance but R&C1's own rule -- bfc_distance == 5 * shader_count -- holding on all 15,219 records of all 2,177 blocks on all 27 levels. Measured over the same set, and reported rather than asserted where it is not universal: bfc_distance == 5 * shader_count 15219 / 15219 vert_ofs <= vert_size all scissor_size <= scissor_ofs all control_size <= control_count all color_ofs == vert_size 12640 / 15219 color_ofs == vert_ofs + vert_size 0 / 15219 color_count, strip_count, strip_vertex_count zero 15089 / 15077 / 15078 R&C1's rule for the colour offset holding on none of them is the interesting line: the two games use that field differently, so it is named color_ofs for what this game's majority reads as and the 2,579 exceptions are counted rather than averaged away. TieRecord parses all twelve fields, invariants_hold asserts only the universal four, and the disc test counts the rest. A packet-length formula is withdrawn rather than softened. (vert_ofs + vert_size + scissor_ofs + scissor_size) * 16 was measured at 61 of 61 pairs and is not reproduced on the correct blocks -- 0 of 13,042. The block set was the error, not the arithmetic: the script that produced the 61 read packets[2] where packets[1] was meant, so it measured 461 blocks instead of the 2,177 the class tables define, and the formula agreed perfectly with a population that was not the tie blocks. TieRecord::packet_bytes goes with it, and the retraction is recorded in the format doc and in the struct. That misread also explains why the earlier RAM verification saw different numbers for the two inputs -- same lesson as rule 3, arriving through a wrong subscript instead of a wrong span. |
||
|
ReaperOfVeriod
|
e6eecb871f |
Decode the tie LOD block: it is a table of 16-byte records
The tie block's word at +0x00 was read as an element count, and the reading failed in a way worth recording: the payload length divided by it is a whole number on 0 of 922 blocks, at any record size and any header up to 512 bytes. That is the correct measurement of the wrong hypothesis -- the word never described the payload. Range measured: every LOD block of level 1 whose word is a non-zero multiple of 16, plus the two read by hand in full. Method: locate each block's record table in PCSX2's RAM by its twelve field bytes, which are the only distinctive 12-byte run in a block, then compare every record against its disc copy. Each block has several copies in RAM, and the one that matches is the loaded one -- checking the first hit is what made the first attempt report 0 of 16. Result: on 16 of 16 blocks, every record's leading word has become table_address + word, and the other twelve bytes are byte-identical to the disc. So a block is [16-byte header][record table][vertex payload], the records are 16 bytes -- a leading u32 pointer plus twelve bytes of fields -- and the header's word at +0x00 is the byte offset from the block start to the payload, that is, the length of the table. That is what the multiple-of-16 was: the table holds 16-byte records. It is never zero because there is always at least the first record, and its 42 distinct values are table lengths. Each record's pointer is relative to the table start, relocated at load to table + word. TieBlockHeader::data_offset and TieBlockHeader::records carry it, bound by the offset, with TieRecord for the twelve field bytes, and the disc test asserts the table is bounded on every block. What the records' fields name and what the payload holds are still open -- the payload is known to be variable-length, float-shaped, not LZ77 and not VIF1. |
||
|
ReaperOfVeriod
|
c2af0528e3 |
Correct the chunk loader finding and decode a record field
The loader note said every record gains a 02 00 prefix in RAM. It does not: the word is the record's own leading field, already on the disc at every record. What the loader changes is the pointer -- it addresses the word, two bytes earlier than the disc's table -- and that pointer shift is what made the bytes look inserted. Measured, in order: - the two bytes before a payload are identical in core_data and in RAM, record for record: 02 00 on level 0's 90-record chunk at 0x100b302, 08 00 on the level's first chunk. Nothing is inserted; - the RAM table's low 24 bits are the payload's minus 2, consistently, so the record is [leading u16][two index bytes][payload] and the loaded table addresses the leading word. That is also why the index bytes read as +2 from the pointer -- the same +2 the earlier RAM result quotes; - the word is a field, not a pad: uniform across a chunk's records on 1,439 of 1,509 chunks (None on the 70 that vary, reported rather than averaged), fixed within all 71 chunk types, and only ever 1, 2, 4, 6, 8 or 10 -- 554 chunks carry 2, 520 carry 4. Two readings of the value were measured and disproved, so they are recorded as disproved rather than left as candidates: it is not an entry count (the payload divided by it leaves 57 distinct fractional "sizes", and 162 chunks do not divide at all), and it is not the number of distinct leading bytes (agrees on 0 of 1,506 chunks). Chunk gains `prefix: Option<u16>`, and the disc test asserts the uniformity, the per-type fixity and the value set. The scratch example that produced the per-type class-ID table is dropped now that the same scan is a committed test, which also lands here: moby class IDs sit in one body slot per chunk type, with the slot varying by type and the distinct >= 4 filter load-bearing -- without it the top slots are constants that read as 305x because any value below the moby count counts as a hit. |
||
|
ReaperOfVeriod
|
28680b4be3 |
Show the first two chunk-record bytes encode the record's own index
Continuing from the phase counter. Within a chunk: i : 0 1 2 3 4 5 6 7 8 9 10 11 ... 15 16 17 ... byte0: 0 16 32 48 64 80 96 112 128 144 160 176 ... 240 0 16 32 byte1: 0 0 0 0 0 0 0 0 0 0 0 0 ... 0 1 1 1 byte0 is step*(j mod g) and byte1 is (j / g), where g = 256/step and j = index + phase. The pair is the record's index in mixed radix: byte0 carries the low digits scaled by step, byte1 the quotient. Measured over all 27 levels: 1,235 of 1,509 chunks fit this on both bytes. Byte 0 alone fits 1,262, so the 27-chunk difference are chunks whose first byte follows the rule and whose second does not. On 1,235 chunks a record knows its own position and the two leading bytes hold no per-record information. ResolvedChunk::index_rule returns the (step, phase) only when the whole chunk fits on BOTH bytes, so the 274 that do not stay distinguishable and nothing skips a leading byte on a rule that does not apply to it. A number that moved when the implementation changed: a first probe reported 1,181; the committed method reports 1,235. The method is right. The probe accumulated ok0/ok1 flags across candidate phases with an early-exit that could skip a phase which would have matched both bytes, and it also disagreed with bucket_rule on the byte-0 count (1,208 against 1,262), which is what gave it away. Recorded because both numbers came from the same run and only the discrepancy exposed the probe as wrong. The lesson is narrow: when a one-off probe and a committed helper disagree, check the probe first. |
||
|
ReaperOfVeriod
|
79914c420e |
Show the chunk record's leading byte is a per-chunk phase counter; every chunk
has a uniform record length Follows the alignment result and settles what the leading byte is. Within one chunk the leading byte steps by a constant and wraps: offsets: 1420 1748 2072 2392 2716 3044 3372 3700 4028 4352 ... byte0: 0 16 32 48 64 80 96 112 128 144 ... Sweeping every (step, phase) over all 27 levels: fully explained: 1,262 of 1,509 chunks by step: 8 -> 51 chunks, 16 -> 554, 32 -> 520, 64 -> 137 unexplained: 247 chunks across 11 chunk types So on 1,262 chunks the leading byte carries no per-record information: it is a deterministic function of the record's index and the chunk's phase, which is a position marker rather than data. That is why the earlier "16 discrete values, uniform counts" pattern appeared at all. ResolvedChunk::bucket_rule returns the (step, phase) when the whole chunk fits and None otherwise, so the 247 unexplained chunks stay distinguishable and nothing skips a leading byte on the strength of a rule that does not apply to it. Every chunk has a uniform record length: 1,509 of 1,509. Length varies BETWEEN chunks of a type -- one type runs 76 to 104 bytes -- but never within one. This withdraws the previous commit's "variable-length type", which came from pooling minlen/maxlen across all of a type's records at once. That is the pooled-claim error rule 3a exists to prevent, and it is the third time this session a pooled measurement produced a claim that partitioning reversed. The type is not variable-length; the chunk is fixed-length and different chunks differ. |
||
|
ReaperOfVeriod
|
a9bf74fc10 |
Characterise all 77 chunk types: record byte 0 is a bucket index, not data
Characterised every chunk type -- 77 of them, 1,509 chunks, 34,491 records, all 27 levels. Byte 0 of every record is a small aligned index, not record content. It takes 16 discrete values per chunk type (0x00, 0x10, ... 0xf0) with near-uniform counts WITHIN each type. Uniform weighting is the tell: a running byte offset would weight the buckets unevenly, because records differ in size. type_id recs len byte0 0x007d7ec438 2430 328 0x00:162 0x10:162 0x20:162 ... 0xf0:135 0x00429ec43d 2079 368 0x00:162 0x10:135 0x20:135 ... 0xf0:108 0x00d76cc453 810 108 0x00:27 0x08:27 0x10:27 ... 0xe8:27 <- steps by 8 The alignment step is per type, not global. The leading byte is a multiple of 8 on 34,432 of 34,491 records but a multiple of 16 on only 33,349 -- the 1,142-record difference falls in seven types, four of which step by 8. 59 records in three types break even the 8-byte rule, and that is where the variable-length type lives: 0x00f2ef4275, records 76 to 104 bytes, no single stride to align to. What the bucket indexes is NOT named. It is reported as CHUNK_RECORD_ALIGNMENT and record_is_aligned rather than as a parsed field, because naming it would be a guess while the invariant is solid. The test pins the exact exception set -- 59 records across 3 types -- so a change there fails loudly. A measurement I initially got wrong: the first version of the constant claimed a 16-byte multiple on 34,432 records. 34,432 is the multiple-of-8 count; the multiple-of-16 count is 33,349. Right number, wrong label, and the test caught it. That is the second time this session a correct number carried a wrong label, after the tie four-value retraction. Both times the assertion did, which is the argument for putting these numbers in tests rather than only in prose. Still open: what the bucket indexes, and the remaining record fields. |
||
|
ReaperOfVeriod
|
291153a0d9 |
Find the tie LOD block terminator; packets == 0 means no block at that LOD
parse_tie_block walks to a 0xf6 terminator that every real block carries, on
2177 of 2177 across all 27 levels, followed by zero padding. This is what makes
the blocks parseable: packets[1] bounds one on only 461 of 1864 classes, so the
terminator is the only general way to find one.
The terminator was worth a control, because 0xf6 could have been byte-frequency
coincidence. It occurs in payloads at 0.83x the uniform rate -- slightly LESS
often than chance -- while the last non-zero byte is 0xf6 on 909 of 922 blocks in
the first subset, against 17 of 922 for the same blocks cut at a random point. So
it is a terminator, not a common byte.
packets == 0 means the class has NO block at that LOD: of 3,728 (class, LOD)
slots, 2177 point at a block and 1551 are exactly zero.
Corrected header, over all 2177:
+0x00 count multiple of 16 on all; 42 distinct; never zero
+0x02 zero on all
+0x04 exactly four values: 0x501 (1069), 0xa02 (720), 0xf03 (350),
0x1404 (38)
+0x06 16 distinct, never zero
+0x08 180 distinct, never zero
+0x0a 61 distinct, never zero
+0x0c 125 distinct, never zero, clustered 0x043f..0x0443
+0x0e 11 distinct, zero on 2162 -- the only non-universal field
+0x0c clusters around 1087-1091, close enough to a block length to be worth
stating, but it is NOT the length: the difference against the true payload length
scatters from -12 to -172.
A correction to the previous commit. It reported 3,728 blocks all terminating and
packed_texture taking 842 distinct values, and on that basis retracted the
four-value claim. Both numbers were wrong the same way: the scan did not skip
packets == 0, so it computed class_base + 0 and walked forward from the CLASS
BODY, finding a terminator there by accident. That contaminated 1551 of the 3728.
With the zeros excluded the slot takes exactly four values on all 2177 blocks.
The four-value claim was right; my retraction of it was wrong; and the retraction
was made on a number from the same contaminated set. A retraction is not
automatically better than the claim it retracts -- it needs its own bounds.
Still open: the payload's records. Established this round: blocks are
self-delimiting, absent LODs are marked by a zero packets word, and the header is
fully characterised. Not established: any record boundary inside the payload,
which stays variable-length.
|
||
|
ReaperOfVeriod
|
02342de31b |
Decode the tie LOD block header; establish the payload is variable-length
TieBlockHeader parses the block's own 16 bytes, measured over the 922 blocks where "the next packets offset ends this block" holds, across all 27 levels. A real-disc test asserts every figure. +0x00 count multiple of 16 on 922/922; 16..2096; 72 distinct disc-wide +0x02 zero on 922/922 +0x04 ONE OF FOUR VALUES DISC-WIDE: 0x501, 0xa02, 0xf03, 0x1404 +0x06 15 distinct, 0x0206..0x0512 +0x08 135 distinct, 0x0b05..0x4ee4 +0x0a 45 distinct +0x0c 94 distinct, 0x010e..0x0553 +0x0e zero on 914/922 +0x04 is packed rather than a magnitude: four values across the whole disc with the two bytes moving independently, low 1-4 and high 5/10/15/20 in lockstep. That it is a texture index paired with a scale is a guess; that it is two fields is not. The count is not a record count -- block_size minus the header is divisible by it on 0 of 922, for any record size and any header up to 512 bytes. On the payload, a control was needed and changed the answer. Reading blocks as 3 x f32 triples, 53-64% of reads land within +/-5000 of origin, against 16% for a uniform-random control of identical length. So the content is float-shaped and genuinely not random. But the rate is flat across every start offset and every stride tried, 53-65% throughout. A statistic that cannot separate the right stride from the wrong one cannot identify a stride -- rule 4's exact case. Flatness plus a count that does not tile means the records are variable length. That killed two readings. A VIF1 command walk consumes 0 of 88 blocks exactly, so it is not a VIF1 CPU-format stream. wad::decompress rejects the block at every start offset, so it is not WAD LZ77. Scanning every (start, stride) pair from 0x10 to 0x7f for byte positions with a small distinct-value vocabulary finds none: every position takes all 256 values at every alignment. For float payload that is expected, so the scan found nothing rather than confirming anything -- worth saying, because "no fields found" here is uninformative, not evidence against fields. Still open: what the payload holds. Established: variable-length, float-shaped, non-random, not LZ77, not VIF1. Not established: any record boundary. |
||
|
ReaperOfVeriod
|
855e640f39 |
Establish the 16-byte tie record on measurable classes; packets[0] is a constant
The previous commit reported "3580 non-empty LOD blocks compute as divisible by 16" and then said the figure should not be trusted, because it came from the "next packets offset ends this block" rule. That rule is unsound: packets is strictly increasing on only 461 of 1864 classes. On the other 1403 the two offsets are unordered and cannot both be block starts into one ascending region. Restricting to the 461 classes where the rule does hold, 922 of 922 consecutive block sizes are a multiple of 16. That is the evidence for a 16-byte record, and it is now scoped to the subset where it can be checked rather than claimed over all 1864. The other 1403 are unmeasured, not measured-and-different. packets[0] is a constant, not an offset: it reads 256 on 1864 of 1864 classes, while packets[1] and packets[2] vary widely and land at different addresses in core_data on every class. Read as a block start, 256 points just past the 0x48-byte header -- which is how it was being used, and exactly how a field gets mistaken for a pointer. It is now TIE_PACKETS0 and nothing depends on it. Still not decoded: what the 16 bytes hold. i16[0] at +4 takes 8969 distinct values on level 0 and is not the two-valued field a 12-unit sample suggested; the word at +0 is not a run offset. The test pins all four numbers -- packets[0] constant, the 922/922 divisibility, the 461/1864 ordering, and the 0/1864 monotonic disproof -- and asserts that the ordering is *not* universal, so a future change there fails loudly instead of quietly invalidating the record size. |
||
|
ReaperOfVeriod
|
60e3f129d8 |
Resolve both tie indirections on all 1,864 classes, and disprove a sample artifact
resolve_tie_class walks the chain, which is two steps and both are easy to get
wrong:
1. The tie class table lives in core_index.bin, not core_data.bin. On level 0
the tables chain as moby at 192, 238 x 32 bytes later tie at 7808, 74 x 32
later shrub at 10176, which closes. Reading the same table out of
core_data.bin produces plausible garbage rather than an error.
2. ClassEntry::offset_in_asset_wad addresses core_data.bin, despite the field
name and despite core_data.wad sitting alongside it.
Verified: all 1864 tie classes resolve, o_class agreeing on every one, and all
5592 LOD block offsets land inside core_data.
Disproved, and this is the part worth reading: the word at +0 of a 16-byte unit
inside an LOD block. On the first twelve units of level 0's class 0 it rises
strictly -- 288, 1520, 2848, 3968, 5088, 6096, 7312, 8528, 9792, 10976, 12016,
13072 -- which reads exactly like a linked list, and the accompanying i16s looked
quantised, two distinct values in i16[0] and three in i16[1].
Across the population it is strictly increasing on 0 of 1864 blocks, and i16[0]
takes 8969 distinct values on level 0 alone rather than two. The sample was
twelve units out of 1,724,334. Both the monotonic reading and the quantisation
were artifacts of that sample.
That is the pooled-claim error AGENTS.md rule 3a exists to prevent, committed
almost verbatim: the first twelve units looked clean, and I was one step from
writing "u32[0] is a linked list of runs" into a format spec. The test now
asserts the disproof, so it cannot creep back.
Not established: the block's record size. 3580 non-empty LOD blocks compute as
divisible by 16, but some packets offsets run backwards, so the "next offset
delimits this block" end is wrong for them and that figure should not be trusted.
The tie layout stays partial: indirections and header done, records not.
|
||
|
ReaperOfVeriod
|
02ced5ee83 |
Parse the per-level overlay: header, constant block, level data
level_overlay was the least-characterised file in the level set -- no parser, no located structure. It now has gc_formats::overlay and a real-disc test over all 27 levels: overlay: 27 levels; field_00 1735808, field_08 1; field_04 37540..=45308, field_0c 2763912..=3027336 (17 below the file length, 10 above); +0x10..0x1220 identical on all The header is four words. +0x0c is not the file length: it was recorded as one on the strength of the first levels measured, where it sat near the file end, but across all 27 it falls on both sides -- below on 17, above on 10 -- so it is an ordinary level-specific word. The test asserts this, being the claim most worth pinning. The common prefix is 4 bytes, not the whole header: +0x00 is constant at 1735808 while +0x04 and +0x0c vary. Everything from +0x10 to +0x1220 -- 4624 bytes -- is byte-identical on all 27 levels. Still not named: what any of the four header words means, and how the level-specific tail is structured. The module reports them as fields. Also records in docs/plan/pcsx2-verification.md that the 64-byte RAM stride from the previous commit is disproved. A 415 KB dump of level 1 finds the longest such run at 27 records, not 384, and those are identity matrices -- 27 of them, one per level. It is a per-level transform table, not the collision grid, and the 20.0 reported there was a coincidence. |
||
|
ReaperOfVeriod
|
51348b214d |
Report the sky reserved-word count in the test output
20 of 149 sky sub-objects carry a non-zero word where the other 129 are zero, so the reserved area is not uniformly padding. Reporting it rather than asserting it, and clearing three clippy warnings from the unused counter. |
||
|
ReaperOfVeriod
|
681e7e62ef |
Parse the sky block: 149 sub-objects with a bounding sphere, and correct the count field
Range measured: all 27 levels, each sky region from the core index's sky offset to the next region. Method: read the 32-bit offset table at +0x20, bounded by staying inside the region, and parse each sub-object it names. Result: 149 sub-objects across 27 levels. Each opens with a count, then a bounding sphere of four floats at +0x10, then a texture offset at +0x20 and dimensions at +0x24/+0x26: +000: 26 00 00 00 | 00 00 00 00 00 00 00 00 00 00 00 00 +010: e0 8d b2 c1 | 9f d6 f6 3d | ef fc fb be | 1c 61 4e 41 +020: 90 99 05 00 | 1d 00 | 1d 00 | 00 00 e8 00 ... Count 1 to 38, width 3 to 81, radius 1.95 to 14.56, and the radius is positive on 149 of 149 -- the same sphere shape and the same reasoning as the shrub class header's. Correction to an earlier claim: the word at +0x0c is not the sub-object count. It reads 5 on level 0 where the table holds 6 entries, 3 on level 1 against 4, 7 on level 3 against 8, and 196617 on level 2 against 4. Across the disc offsets.len() == count+1 holds on 10 of 27 levels. The field is reported as SkyHeader::field_0c and nothing depends on it; the offset table enumerates the sub-objects and is self-limiting. That earlier claim came from level 0 alone, where 5 against 6 looked like a consistent off-by-one. It is not an off-by-one -- it is a field that happens to be small on most levels and is not a count at all. gc_formats::sky parses the header and the sub-objects. The gated test asserts the sphere is finite, the radius positive where positive at all, the dimensions non-zero, and the sub-object list non-empty on every level. |
||
|
ReaperOfVeriod
|
5ade83cf0a |
Parse the shrub packet header, a fixed signature on all 2056 disc packets
The packets found by the +0x40 table have a header the whole disc agrees on. Range measured: every packet on the disc -- 2056 of them, across the 334 successor-bounded spans -- read at each packet's span-relative offset. bytes 0-3 04 04 00 01 2056 / 2056 bytes 4-7 zero 2056 / 2056 byte 8 zero 2056 / 2056 byte 11 0x05 2056 / 2056 size % 16 zero 2056 / 2056 Zero exceptions on any of the five. And the loader fills the rest in. Read back out of the running game the same records carry byte 8 as a count (0x0b to 0x29 observed), byte 11 as one of four tag values, and bytes 12-15 as an absolute EE RAM pointer where the disc has a span-relative offset. Bytes 8-15 are rewritten wholesale on load. That settles what the disc header is for. It carries no count of its own, and the count the RAM copy has is the +0x40 table's count arriving by another route. The disc header is a signature; the RAM header is populated state. ShrubPacketHeader::parse and Shrub::packet_headers() expose this. parse returns None unless all the invariants hold, rather than a default, so a packet that does not match is reported as not-understood instead of half-interpreted. The gated test asserts all four header invariants and the size alignment on every packet of every bounded span. This is the first piece of the mesh packet format represented in gc-formats; everything known about the packet payloads so far is RAM measurement, since the disc form of bytes 8-15 is rewritten at load. |
||
|
ReaperOfVeriod
|
aa2b14ad66 |
The shrub mesh geometry is in the class blob after all
The open question from the shrub decode was where the mesh lives: the blob looked like header, palette and optional billboard, with the 333 classes without a billboard having nothing to draw. Answering it took the running game, not the disc. Diffing a loaded shrub class against its disc bytes in the live PCSX2 (level 1, class 1) shows what the loader rewrites: a table at +0x40 of (offset, size) pairs, whose offsets are span-relative on disc and absolute in RAM with the sizes unchanged -- all 21 entries matching offset for offset and size for size -- plus a relocated palette pointer at +0x2c and a pointer at +0x18, zero on disc, that asserts the parser enforces. So the blob is header, packet table, packets, [billboard], palette: the mesh was here the whole time. Why it was missed is the part worth recording -- data_offset points at the END of the span, so reading the body as data_offset..span_end accounts for the last few hundred bytes and leaves the 20 KB in front of it unexamined. A correct bound and a body small enough to look plausible are together enough to hide a region. Range measured: every shrub span on the disc, 358 over 24 levels, each bounded by its successor class or the located chunk container. Method: read count at +0x28, then the entries at +0x40, and require them to tile -- the first packet at align16(0x40 + count*8), every entry exactly size past the previous, and the last ending at the palette, or 0x40 short of it on the 25 spans that carry a billboard. Result: 358 of 358 tile exactly, holding 2,239 packets. That is what pins the field meanings: count is the packet count (1..55 on the disc), data_offset is the palette's offset, and the earlier "the body is not sized by count" note was true for a reason nobody had understood -- the count sizes the packets. Shrub::packets() exposes them and returns None rather than a partial list when a table does not tile. Still open: the packet contents (a VIF reading was already retracted once, so this needs a control rather than a pattern match), and the 0x20-byte records the loader points at from +0x18. |
||
|
ReaperOfVeriod
|
2c7555fe16 |
Decode the shrub class blob: a 24-normal palette and a billboard record
The shrub body's payload encoding was the open question after the span-bound work, and it answered itself once the R&C1 reference was in view: R&C1's shrub class header keeps the field offsets R&C2 shows (o_class at 0x24, a count at 0x28, the data offset at 0x2c), which says which bytes to look at, and the disc says what they hold. Range measured: all 358 shrub class spans over the 24 levels that have them, each bounded by its successor class or the located chunk container. Nothing outside the spans is read. Result. The body is a palette of 24 unit-length s16 normals -- s16 x, y, z and a zero pad word per 8-byte record -- followed by all-zero padding records. Three independent measurements, each of which fails for any other reading: - all 8,592 live entries have squared length within 65,535 of 32767^2, and the bound is tight (the largest observed deviation is exactly 65,535). That is 32768^2 - 32767^2, which is what a unit vector scaled by 32768 produces when an axis-aligned normal quantises to -32768 and the positive axis clamps to 32767; components run the full s16 range and 1,765 entries are axis-aligned. A random s16 triple is off by about a billion, so one entry in fifteen thousand would pass by chance: all 8,592 pass. - the padding records are always at index 24 or later and always all zero, never a live entry. That is what separates the palette from geometry, and it is the observation the three earlier stride-fitting attempts were missing. - 17 classes carry a palette of 24 identical (0, 0, 1) entries. A position or index table cannot produce that; a normal palette for a flat-facing class must. The header decodes as: sphere at +0x00 (centre + radius, radius positive on all 358 spans, 1.72..41.57), size at +0x10, mode_bits and scratch at +0x14, a zero word at +0x18, the billboard offset at +0x1c, and scale at +0x20. The parser asserts the two invariants that hold without exception (scratch and the reserved word are zero on all 358) plus a positive radius and scale, and reports mode_bits rather than asserting it: it is 4 on 16 of the 358 spans. The billboard offset is the strongest self-identification after o_class because it is confirmed from a second file: it is non-zero on exactly the 25 classes whose core-index entry carries billboard texture fields, pairing correctly both ways. At the offset sits R&C1's ShrubBillboard shape, register block and all -- fade_distance, then TEX1_1 (K -233..-168, MMIN 4, address 0x14), TEX0_1 (index 1..4, address 0x06) and MIPTBP1_1 (address 0x34), all three register addresses checked in the disc test. Two corrections this produces, both recorded in docs/plan/level-assets.md because the old readings were confidently wrong rather than merely unverified: - "reserved is 0 or a denormal" had the two u16 halves the wrong way round: the 16 non-zero spans are mode_bits = 4 with scratch = 0. As an f32 the word is still a denormal, so the PS2 reads zero either way -- but the field is a flag word, not a magnitude, which is exactly the FPU point worth making. - "bounds" was tested as a min.xyz/max.xyz box, which holds on only 146 of 358 spans, and the field was left unnamed as a result. It is a bounding sphere: the fourth float is positive on all 358 spans and the centres cluster near the class origin. The 146/358 figure was a property of the wrong question. Still open: the three floats in the billboard record (they repeat byte-identically across levels -- 16 distinct triples among 25 records, one on four classes across levels 1, 4, 12 and 21 -- so they are shared art data rather than dimensions), size, mode_bits, count, and where the shrub mesh geometry lives. R&C1 puts the mesh packets in this blob; R&C2's blob holds only the palette, so the 333 classes without a billboard must draw meshes from somewhere else. |
||
|
ReaperOfVeriod
|
ccea53807a |
Resolve the shrub region bound: the chunk container closes it
The shrub region's end was the last unlocated bound in core_data. It was not in the core index, and unlike moby->tie it had no successor boundary to chain against, so the last shrub class in every level ran out to the lump's end and the aggregates over it (the "3.5M records / 28MB of shrub body") were artifacts of that oversized span. Range measured: for each level, from the last shrub class's geometry offset to the constant trailer. Nothing outside that range is read. Method: scan forward from the last shrub class for the pair (a) the constant two bytes f6 7c sitting immediately in front of (b) a chunk signature -- a 16-byte header ending 0xff, ten zero bytes, then four or more 0xf0-tagged offsets. The two halves are each weak alone, and the halves are measured separately: the marker pair occurs 25 to 103 times in a 5 to 17 MB lump, so matching it alone picks dozens of wrong offsets; the chunk shape matches random data often enough to need four tagged offsets before it is believed. Result: on all 27 levels the marker sits exactly two bytes before the first chunk header, and the two parts of the signature are never found apart. No chunk signature occurs inside any last shrub body, on any level. The three levels without shrub (10, 25, 26) carry the same marker before the container that closes tie instead, which is what makes the locator a property of the region rather than a shrub rule. Consequences: - All 358 shrub class bodies are bounded and are whole 8-byte records. That was not assertable while the last class in each level ran to the lump's end, and it is the check the boundary earns its keep for. - 9,600 records in total, against the 8,938 previously measurable in successor-bounded spans; 662 of them sit in the 24 last-class bodies. The shrub region totals 2,033,776 bytes across the 24 levels that have one. - The chunk container is 25,775,850 bytes across 27 levels (1,236 chunks, 22,562 records). The container measurements were previously taken over "the 900,000 bytes before the trailer", a window that did not align to its real extent: on level 0 it is 690,302 bytes, and 27 of 1,123 chunks were truncated at that invented edge. chunk_containers_resolve_on_every_level now reads the region from core_layout instead of measuring backwards from the end of the lump. core_layout takes core_data now: the container's start is the one bound that cannot be derived from the index. It gains the chunks region, so every byte of level-specific core_data belongs to a named region. Where the signature is not found the end stays None -- the region is left visible rather than clamped to the tail. Still open: the marker's meaning (constant across 27 levels, at exactly this position, unnamed) and the shrub body payload's encoding. |
||
|
ReaperOfVeriod
|
10188416e2 |
Decode the chunk count field: header byte 14 is the record count
Range measured: all 27 levels, 1123 chunks, over the 900000 bytes before the constant trailer. Method: compare header byte 14 against the number of offsets actually resolved for that chunk. Result: byte 14 equals the record count on 1123 of 1123 chunks, zero mismatches. This is the first genuinely decoded field in this container rather than a measured correlation. The parser now truncates the offset table to header byte 14, so a stray tagged word in a chunk's record data cannot silently extend the table and put the count at odds with its own header. Two more header bytes are near-constant and read as ASCII: byte 13 takes only 0x45 (1076 chunks) and 0x46 (47); byte 9 takes 0x44 (26), 0x45 (1084) and 0x46 (13). Those are D, E and F -- a three-letter tag at two byte positions with different distributions. Recorded as suggestive, not established: the other 14 header bytes are high-entropy and read as ASCII at no shift tried, so the D/E/F may be incidental. Worth testing against a known type table. Still open: the 14 opaque header bytes, whether bytes 9 and 13 are type tags, which record fields are index/length/offset, and whether a chunk is moby_anim. |
||
|
ReaperOfVeriod
|
681333535a |
Add a bounds-checked parser for the post-shrub chunk container
gc_formats::chunks parses the container that fills the region after the last shrub class body and before the constant trailer: +0x00 16 bytes, last byte 0xff (opaque, differs per chunk) +0x10 10 zero bytes +0x1a N tagged offsets, top byte 0xf0 Each tagged offset is 0xF0000000 | low24, and a record is read at chunk_start + low. Resolving that way gives records sharing a field layout, which is what distinguishes the container from a coincidental byte pattern. Fields are not named. The 16-byte header is opaque, the 0xf0 tag is unexplained, and it is not established that a chunk is moby_anim -- a 49 x 368-byte table inside this region is byte-identical on all 27 levels, which is evidence against that identification. What the parser provides is chunk and record extraction, which is what the analysis established. The gated test measures, over all 27 levels: 1123 chunks, 19546 records, 50 distinct strides, 12.4% of byte positions constant across records (per-chunk range 2.8% to 75%). Chance gives a constant position probability of 1/256^n. Three things the test got wrong first, recorded because they are the traps: - A per-chunk threshold of 20% constant positions. Individual chunks run from 2.8% to 75%; the aggregate is the invariant, not the chunk. Asserting 20% per chunk was asserting a property the format does not have. - Requiring every offset to resolve. 27 chunks truncate at the window edge, because the 900000-byte window does not align to chunk boundaries. Counted now rather than asserted away. - read_u32 in the chunk signature check was big-endian while Buf::u32 is little-endian. The tag lives in the high byte of a little-endian word, so the mismatch made every level report zero chunks while parse_chunk -- which goes through Buf -- was correct. Worth remembering that the two paths can disagree on a format this young. Tests: 190 -> 191. |
||
|
ReaperOfVeriod
|
69b81af3b2 |
Expose the shrub body as 8-byte records, bounded per class
Turns the differential finding into code. Each shrub body record is 6 varying bytes plus 2 constant ones; body_records() hands them back raw and body_is_well_formed() checks the trailing pair. ShrubHeader::region now takes an explicit span_end. It has to: without it the data slice runs to the end of the lump, so the last class in each level absorbs whatever follows shrub and a per-class check ends up measuring its neighbours. That is the same unbounded-span mistake as before, caught this time by a failing assertion rather than by reading the docs afterwards. The disc test asserts well-formedness only for the 334 of 358 classes whose span ends at the next class. The last class in each level has no successor, so its span contains the 690 KB region after shrub and its body is not wholly records -- that is the open question, and asserting either way would be wrong. Shrub::span_end_is_a_successor records which case a span is. Records are summed only across bounded bodies. Summing the last class too reproduces the "3.5 million records" figure retracted two commits ago, to within rounding -- which is a useful check on how easy that mistake is to repeat by accident. 8,938 records across the 334 bounded bodies; 16 denormal reserved words; 358 headers naming their own class. |
||
|
ReaperOfVeriod
|
8d57a09b76 |
Bound the last region by core_data's constant trailer
Comparing all 27 levels byte-for-byte from the end: the last 231,952 bytes (0x38A10) of every level's core_data are identical. Level-specific content therefore ends at len - 0x38A10, which is a real bound for the final region and is not core_data.len(). This closes shrub. Across the 24 levels that have a shrub region the bound gives 24,442,928 bytes total, and the disc test asserts on every level that the region ends strictly inside the lump rather than at its end. What the trailer is remains unestablished -- fixed data, or padding from a fixed-size decompression buffer -- but it is constant, which is all a bound needs. The data between the last shrub class and the trailer is high entropy: 183 to 256 distinct byte values per 4 KB block, mean 254, where texture data sits under 200. It is real content of unknown type and is the most likely home for moby_anim, now that the range between collision and moby has been ruled out as texture. A lump shorter than the trailer now yields end: None rather than a backwards bound, covered by a test. |
||
|
ReaperOfVeriod
|
8b98d388f7 |
Show region bounds in probe, which exposed an unnamed 6.2 MB region
Printing the region table next to the header fields makes the gap visible without having to remember to consult core_layout, and it paid for itself immediately. On level 0 the collision region comes out 6216960 bytes, from 2615232 to the first moby class at 8832192. Collision's own header declares a payload of 903,248. So roughly 6.2 MB sits between the collision header and the first moby class, and almost none of it is collision. The successor rule is correct -- a region does end where the next one begins -- but "the next" is not yet known there. collision..moby is really collision + <something unnamed> .. moby. The name is wrong, not the bound. Candidates for the space, rough order of likelihood: moby and tie model animation data, which is what moby_anim would need, then volumes, then per-class runtime structures. Tfrag texture texels are already placed and sit earlier in the lump. Worth resolving before starting moby_anim, since that format is the most likely tenant. Until then, any probe reading between the collision header and the first moby class is reading something the region table does not describe. |
||
|
ReaperOfVeriod
|
cb38b8c279 |
Compute core_data region bounds once, from successor boundaries
Region spans were being re-derived per format while probing, usually running from one known offset to the next with the last span running to core_data.len(). On level 0 that gave the final shrub class a 925,008-byte span absorbing most of the lump's tail, and every aggregate computed over it -- record counts, value distributions, "this looks random" -- was silently wrong. Nothing errored, which is what made it survive two commits and a correction. core_layout computes the layout once: each region ends where the next one begins, class regions start at the lowest non-zero geometry offset in their table, and a region whose successor is unknown gets end: None rather than a clamped guess. Zero class entries mean absent, not position zero -- the same trap occlusion and collision each fell into separately. The disc test asserts what the module is actually for: that the unknown is visible. Shrub's end is unresolved on the 24 levels that have a shrub region, and that is asserted rather than hidden. It also found that not every level has every region -- level 10 has neither occlusion nor shrub -- so the test checks ordering against the canonical sequence instead of demanding an identical set. Still open: where the shrub region ends. Texture texels cannot bound it; they sit before moby, not after shrub. See docs/formats/core-data-regions.md. |
||
|
ReaperOfVeriod
|
517f0b63fa |
Read the shrub header floats, including a denormal the PS2 flushes
Extended the shrub header past the two self-identifying words. The first four words are floats -- finite on all 358 spans -- though they do not behave like a min/max bounding box, which orders correctly on only 146 of 358, so they are recorded as bounds rather than named. The word at +0x10 is positive everywhere, 3.0 to 20.0, exactly 5.0 on 281 of them. The word at +0x14 is the interesting one. It is zero on 342 spans and the bit pattern 04000000 on the other 16, which is a denormal of about 2.8e-45. The Emotion Engine has no denormals, so hardware reads all 358 as zero. That is only the second reading. My probe reported the field as 0.000 everywhere because it printed to three decimals, and the test caught it: level 7 class 7 fails a strict zero check. The lesson is that "exact" in a measurement means exact in the measured precision -- rounding in the output made a real distinction invisible. It also matches AGENTS.md on modelling the FPU rather than correcting it: this field looks like noise on disc and is a constant to the machine that reads it, and the parser accepts either rather than picking one. The shrub body past +0x2c is still not decoded. |
||
|
ReaperOfVeriod
|
7107f395d0 |
Parse the shrub class header
Shrub is the third class-geometry family and is not the moby one: all 358 spans on the disc have a leading word of 0 or one too large to index a table into, so none contain a moby-style part table. The rule that holds for 43,467 of 43,467 moby parts holds for zero of them. Two header fields do self-identify, exactly, on all 358 spans: +0x24 equals the owning class entry's own o_class, and +0x2c is a data offset landing inside the span. Both are checked rather than merely reported -- a span naming the wrong class is a different thing, and quietly returning the mismatched value would hide that. The body past +0x2c is not decoded. It tiles at no stride consistent with the count at +0x28 (tails of 240, 240, 192 and 224 bytes against counts of 5, 6, 5 and 3), and the header's leading bytes are packed rather than float even though +0x10 lands exactly on 5.0f on 281 of 358 spans. Shrub class entries are 0x30 bytes rather than 0x20, because they carry billboard texture fields the other two tables lack; the parser records that so the stride is not re-derived. |
||
|
ReaperOfVeriod
|
f6bbbf8d5f |
Add gc-extract probe to dump structures through the real parser
Reading these formats by unpacking raw offsets out of core_index.bin has cost real time. The index interleaves single-word region offsets with count/offset pairs, so miscounting one shifts every field after it: that produced a page of implausible collision headers on all 27 levels, sent me chasing a tie-breakoff that turned out not to exist, and nearly had me "fix" a stride bug in code that was already right. probe dumps a level's core header and all three class tables through LevelCoreHeader, so a field name printed is the field name the parser uses. It also reports how many entries carry geometry inside core_data, which is what pins the class table strides rather than guessing at them: shrub reads 7/7 sane at 0x30 against 4/7 read at 0x20. Reading a structure by hand in a shell still works; this just stops it being the default, and stops the offset arithmetic being re-derived every time a new region is investigated. |
||
|
ReaperOfVeriod
|
bbb10c678d |
Parse the moby per-class part table
A moby class span does not start with its geometry. The first u32 is the offset of a part table, and the table is 16-byte records of vertex offset, vertex count, an unestablished second count, the next part offset and flags. Parts tile: vertex_offset + 16 * vertex_count == next_part. The stride is never assumed anywhere -- it falls out of that identity, which is what pins the record size and the field boundaries. The identity holds for 43,467 of 43,467 part records across every moby class on all 27 levels, and the parser rejects any span that violates it rather than returning a plausible wrong answer. Two terminations are needed. A zero vertex_offset ends the run, but 172 records on the disc have a zero vertex_offset and a zero next_part while still carrying a vertex count. Stopping only on the first leaves 99.6% instead of 100% and quietly includes a second table as if it were parts. 1,270,802 vertices over 4,560 class spans; 116 spans carry no geometry. Vertices are 16 bytes of VIF-packed half-float quads cycling through four distinct words. Handed back raw -- unpacking them is separate work from finding them, and second_count is recorded but not guessed at. |
||
|
ReaperOfVeriod
|
dc941a8eec |
Parse the collision container framing
Collision sits inside core_data at the offset the core index gives, behind a 0x40-byte header that self-describes: it carries its own size and the payload size. That pairing is the invariant -- the declared header size is exactly 0x40 on all 27 levels and header + payload stays inside the lump. 41,481,520 payload bytes across the 19 levels that have any; 8 declare zero length, which is present-but-empty rather than absent, so the two are counted separately. The cells inside the payload are not decoded. They do not walk as fixed-stride cells, and treating faces as variable-length records stops partway into the first cell on level 0 rather than tiling the payload. The first cell reads as 11 verts and 33 faces under every hypothesis tried, which is exactly the kind of agreement that looks like a result and is not one -- three different strides all produce the same first cell and all of them are wrong. Left open rather than committed as a guess. |
||
|
ReaperOfVeriod
|
2dcc6e9710 |
Parse the sound bank container framing
A level's sound extent is two regions: an `SBlk` descriptor table and the sample data that follows it. Four u32 at the head describe the layout, and the invariant that establishes it is sblk_offset + sblk_size == daw_offset, which holds on all 27 levels. That is what pins the field meanings; the numbers alone are ambiguous about which of them is the offset. DAW is a format tag inside the SBlk header, not a region signature. It occurs exactly once in the entire extent, at sblk_offset + 12, naming the sample format -- the sample data itself starts at daw_offset with no signature at all, roughly 19 KB away on level 0. Reading it as a region magic is the obvious reading and it is wrong. daw_size never reaches the end of the file. The tail is positive on all 27 levels, 232 to 1,848 bytes, so the field describes less than the region it names; the test asserts that direction rather than inventing a size that fits. 523,720 SBlk and 37,649,008 DAW bytes across the 27 levels. The sample format inside DAW is not decoded. It looks like VAG/ADPCM from the nibble packing, and it has no bearing on drawing anything, so it stays open rather than getting a half-answer. |
||
|
ReaperOfVeriod
|
632505a6bd |
Parse the occlusion PVS grid
A z -> y -> x tree of u16 node offsets over 4x4x4-unit cells, each populated cell naming one 128-byte mask of 1024 visibility bits that the renderer fills from the camera's cell each frame. Two sentinels carry meaning and one of them is easy to miss: a z or y slot of 0 means "no subtree", and an x slot of 0xffff means "no cell". Reading the latter as a mask index gives a grid whose highest mask is 65535 on every single level, which is the tell. The check that makes this real is the mask array. It starts at masks_offset and runs to (highest_mask + 1) * 0x80, and on every level with a grid that ends at the block boundary exactly -- widest unexplained tail across the disc is 0 bytes. A wrong tree shape or cell size would leave the highest mask index elsewhere entirely. 412,521 populated cells over 140,939 masks, across the 24 levels that have one. An offset of 0 means the level has no occlusion data at all, which is not the same as "at the start of the file" -- that is where the tfrags block lives, and parsing it as a grid produces a mask array that fits anyway, with 1600 bytes to spare. That is what made level 10 look plausible on the first pass; three levels (10, 24, 25) carry none. |