4fab4d03592222b98d1d5340a87e24df7bbeb3cd
100
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 ReaperOfVeriod
|
4fab4d0359 |
Fix the record boundaries after all, and retract a claim built on the bug
I reverted the record-boundary fix last commit on the grounds that "fixed-length
records plus inter-record padding" and "variable-length records" produce the same
offset table. That reasoning was wrong, and cheaply so.
stride is the FIRST offset delta. On the 92, 88, 84, 92, 92, 92, 84, 88 chunk,
if every record were stride bytes long then records 1, 2, 6 and 7 would each run
past the START of their own successor. Records are addressed by an explicit
offset table; they cannot overlap. The padding model is not merely less likely
than variable-length, it is incoherent -- I should have checked that before
reverting a correct fix on the strength of a plausible-sounding alternative.
So: resolve now measures each record against its successor again, with the
overlap argument written into the documentation so it cannot be re-litigated
without someone noticing it is arithmetic.
THE RETRACTION, which is the substantive part:
Three tests broke. Two were straightforward, but the third was not.
* constant_positions indexed records as if they were all the same length. Now
that they are not, it compared past the short ones. It takes the minimum
length instead.
* "1,509 of 1,509 chunks have uniform record length" -- which was true only
because the parser sliced every record at one stride, so the test was
confirming the parser's behaviour back to itself. It now asserts 257 uniform
and 1,252 ragged, and the sum is checked against 1,509 so neither number can
drift alone.
* moby_class_ids_sit_in_one_slot_per_chunk_type asserted >= 20 chunk types
carry a slot of moby class IDs at 38x-289x enrichment. It now finds 2, of
which 1 clears 100x. RETRACTED. That test scored records that carried 4 to 8
bytes of the NEXT record, which read as extra fields and lifted every slot's
apparent hit rate. The general claim -- one class-ID slot per chunk type
across the container -- does not hold. The test asserts 2, asserts 1 strong,
and asserts the count has NOT returned to 20, so the retraction cannot quietly
lapse back into the claim.
That is a committed finding withdrawn on the strength of a parser bug. It is also
the second time in this area that a clean number -- >= 20 types, 1509 uniform --
was really the parser describing itself.
243 tests, clippy clean.
|
||
|
ReaperOfVeriod
|
5f0396e3d7 |
Measure every chunk's record offsets; the cause is ambiguous, so no fix
The previous commit fixed on one signature. Measuring all of them shows the
pattern is the norm. Taking delta = offset[i+1] - offset[i] against the chunk's
own first delta, levels 0-2, 2,953 deltas:
0 1,723 -4 549 +4 426
-8 146 +8 32
-12 71 +12 21
-16 34 +16 4
-20 27 +20 1
-24 13
Every deviation is a multiple of 4 and most are exactly +/-4; worst observed is 24
bytes in a 7-record chunk. 1,252 of 1,509 chunks have at least one, across 66 of
74 signatures.
WHY THIS DOES NOT BECOME A FIX
Two models produce this offset table and they are observationally identical:
1. records are genuinely variable-length, padded to 4 bytes, stride = largest
2. records are fixed at stride and the offset table carries inter-record
padding, so the gap to the next record is record + padding
Nothing in the offsets distinguishes them. Both say where records start. Both
produce 92, 88, 84, 92, 92, 92, 84, 88 for the case already discussed.
I wrote the change that slices each record against its successor -- model 1 -- and
then reverted it. Two things came out of writing it:
First, it broke three tests, which is the informative part. Those tests assert
"1,509 of 1,509 chunks have uniform record length" and they passed BECAUSE the
parser forced uniformity by slicing every record at a single stride. They were
asserting the parser's behaviour back at itself, and would have kept passing
forever regardless of what the disc contains.
Second, shipping it would have replaced one unverified model with another and
silently changed the record boundaries of 1,252 chunks on the strength of a
guess. The repo's own rule applies: a heuristic count is not an identification,
and here it is not even a count -- it is two equally good explanations of one
table.
WHAT WOULD SETTLE IT: a field inside the record whose end is independently
observable -- a terminator, a count implying a length, or a known value pinning
its position. Under model 2 the disputed bytes are padding and will be uniform or
zero; under model 1 they are data and will not be. For [75 42 ef] those 8 bytes
read a0 00 27 20 at both the tail of one slice and the head of the next, which is
arithmetic rather than evidence, since it is the same eight bytes seen twice.
The uniform model stays, the ambiguity is recorded in resolve()'s documentation
and here, and the 1,509-of-1,509 assertion stands as a statement about what the
parser does rather than about what the disc contains. 243 tests, clippy clean.
|
||
|
ReaperOfVeriod
|
b9e9835639 |
A parser defect: one chunk type is not fixed-stride, and over-reads
[75 42 ef f2 71 b5] -- the signature that is both level-specific and
variable-length -- makes three chunks per level, 81 total:
stride 76 deltas [76, 76, 76, 76, 76, 76, 76, 76] uniform
stride 92 deltas [92, 88, 84, 92, 92, 92, 84, 88] NOT uniform
stride 104 deltas [104, 104, 104, ...] uniform
Identical on all 27 levels: 27 chunks of each shape, three shapes, no others.
So "this signature has strides {76, 92, 104}" is a misdescription. It has three
chunk shapes, one of which is not fixed-stride at all. The parser derives one
stride per chunk and resolve slices every record at that length.
The cost, precisely: records are read at the offset table's positions, so every
record STARTS correctly -- no misalignment. But each is sliced at 92 bytes, while
two of the eight are 88 and two are 84, so four records over-read by 4 and 8
bytes and carry the head of the next record inside them. Bounds checking holds;
this is a data-correctness bug, not a safety one. It is precisely the kind that
produces plausible-looking fields offset by four bytes, which is how it would
have gone unnoticed had the stride table not been collected per signature rather
than inferred per region.
It also explains why this signature was previously flagged as breaking the 8-byte
alignment rule: its records genuinely are not a uniform multiple.
The right fix is for resolve to use offset deltas for a chunk whose deltas are
not uniform. Not taken here -- the delta sequences are asserted first, so if the
data ever becomes uniform the test fails and asks for the model to be revisited,
rather than the model quietly absorbing the exception.
Asserted in chunk_type_75_ef_has_one_variable_length_chunk_per_level, with the
exact sequences [76x8], [92,88,84,92,92,92,84,88], [104x8] and 27 of each.
A near-miss worth recording, because it would have made a tidy false claim: the
leading u16 of these records appeared to step by exactly 0x50 (80) across all nine
of the 76-byte chunk. Reading at the real offsets gives b1a5, a529, 047e, a529,
04f0, a529, 0562, a529, 05d4 -- no progression at all. The apparent 80-byte step
came from comparing the wrong bytes, and "records are indexed by 80" is exactly
the kind of finding that gets reported on sight.
243 tests, clippy clean.
|
||
|
ReaperOfVeriod
|
af62032ddc |
Most of the chunk container is level-independent: 65 of 74 signatures
Comparing every chunk record against the same record index in the other 26
levels, across all 74 signatures:
65 of 74 every record byte-identical in all 27 levels
9 of 74 level-specific
0 ragged no signature appeared with different record counts per level
The nine that vary:
[5d 44 13] [75 42 ef] [7f 44 ee] [82 c4 2f] [96 c3 37]
[a2 44 a5] [c6 c3 ef] [df c3 ea] [f3 c4 b9]
This reframes the container. 65 of 74 chunk types are static tables the game ships
unchanged in every level -- constants and lookups, not level content. The largest
by record count are all in that group: [38 c4 7e] stride 328 with 90 records per
level, [a9 c4 d8] stride 308 with 90, [3d c4 9e] stride 368 with 77.
The consequence is about method, and it is worth being blunt about. For a
level-specific type, comparing levels is the strongest lever available: a field
that moves with the level is identifiable by correlating it with something known.
For the other 65 there is nothing to compare -- one copy of the data, repeated 27
times -- so they must be decoded structurally, from what the bytes are. A future
decoder that reaches for per-level comparison as its first tool will find it
useless on the majority of the container.
[75 42 ef] is both level-specific AND the one signature with three strides. A
type that is both variable-length and level-varying is the most informative one
here, and is where per-type field decoding should start.
How this was found, including the wrong turn:
The first look at [53 c4 6c d7 9a 43] (stride 108) showed 45 of 108 byte positions
varying WITHIN a chunk and 810 records all distinct -- which reads like per-record
data. Counting distinct values per u32 slot across all 810 records contradicted
that: 29 to 30 distinct values across 27 levels, i.e. about one per level.
Both readings were wrong because they measure different things. The 45 positions
vary between records within a level; the u32 counts were low because the same 30
values recur in every level. Comparing record i across levels settled it: all 30
records are byte-identical in all 27 levels, and zero byte positions vary across
levels.
Had the u32 distinct counts been the only measurement, the conclusion would have
been "one value per level" -- the opposite of the truth, describing a per-level
parameter table that does not exist. The within-chunk and across-level views had
to be taken together to get a true one.
Asserted in most_chunk_signatures_are_level_independent. 242 tests, clippy clean.
|
||
|
ReaperOfVeriod
|
59dba3e58d |
Chunk records: the stride table, and the exception pinned by name
Walked every chunk on all 27 levels, grouped by six-byte header signature:
1,506 chunks, 34,415 records, 74 distinct signatures
55 distinct strides, 52..640, every one a multiple of 4
73 of 74 signatures use exactly one stride
The exception is asserted rather than skipped, because "every type is uniform"
is precisely the claim that quietly stops being true:
[75 42 ef f2 71 b5] strides {76, 92, 104}
That is the variable-length type the module docs had referred to indirectly. It
is now pinned by signature and to those three strides exactly.
The 4-byte alignment is a result, not an artefact. A stride recovered from one
chunk's first-to-second record delta can inherit alignment from where that chunk
happened to sit; collecting all 55 strides across every type and requiring all
of them to be divisible by 4 rules that out, because a single off-by-one in one
signature would show. Asserted in
chunk_record_stride_is_four_byte_aligned_and_uniform_per_signature.
What this settles is the shape the decoder must handle: uniform stride per type,
4-byte aligned, one named exception. Field offsets within a record are now the
same for every record of a type, which is what makes per-type decoding
tractable. The fields themselves are still unnamed.
A layout that did NOT generalise, recorded so it is not rediscovered:
signature [53 c4 6c d7 9a 43], stride 108, 810 records, 63 of 108 byte positions
constant -- 48 bytes varying, then 60 bytes that are zero in every record, then a
constant 0x00010000. That reads like an inline buffer this type never uses, and
would have made a tidy field story. Requiring `stride == live span + zero run`
across every type with 50+ records per chunk gives 0 of 9. The zero tail belongs
to that one type and means nothing general.
Third occurrence of one editing mistake, and it is worth naming as a habit
rather than an accident: inserting a test before an existing one strands that
test's #[test] attribute on the new doc comment and silently unregisters it.
It has now happened three times, each time surfacing only as clippy's "never
used" and never as a missing test. Fixed by appending new tests at the end of
the file instead of anchoring them to an existing function, and verified by
listing registered tests rather than trusting a count.
241 tests, clippy clean, all five recent tests confirmed registered.
|
||
|
ReaperOfVeriod
|
040386c0fb |
Volumes: the reference that did not generalise
Last commit said what would settle the volume candidate and implied one of those
was available. They are not. Recording the failure rather than moving on quietly.
Level 0's data.bin -- the level's file descriptor -- contains the value 799204
exactly once, at 0xab6360, and 799204 is byte-for-byte the offset of the 20-record
144-byte candidate table. A single 4-byte match in 13.3 MB has a chance
expectation of ~0.003, so it was worth checking rather than assuming.
It does not hold up. Each level's own best 144-byte table, searched in that same
level's own data.bin:
level candidate records refs (chance ~0.0035)
0 1,810,724 32 0
1 1,884,884 386 0
2 2,066,804 224 0
3 2,438,228 365 0
4 2,677,780 114 0
5 1,255,780 14 0
6 2,496,020 115 0
7 2,062,072 311 0
None. The level 0 hit at 0xab6360 also sits among high-entropy neighbours, and
level 0's own BEST table -- a different offset, 32 records, not the 20-record one
-- is not referenced either. So the hit does not generalise and is most likely
coincidence.
This retracts what the previous commit implied. Nothing links a 144-byte float
table to volumes: the table is reached by walking structure rather than by an
explicit pointer, so the discriminator I named does not exist, and the candidate
stands exactly where it did.
Recording the three routes that would actually settle it, all of them
unavailable: the game's own volume code (same dead end as the collision anchor
-- a GP-relative string search returns nothing); ground truth, such as a count of
level 0's trigger volumes from a map; or a correlation with a named class, since
gameplay.bin resolves ids through its dispatch records -- but no class name
containing "volume" exists on the disc, which is exactly why this is unavailable.
No test added, because there is no solid claim here to assert. 240 tests,
unchanged.
|
||
|
ReaperOfVeriod
|
ca277a0a01 |
Volumes: a candidate table found, explicitly not called volumes
Volumes were the only type never located, so this is a search report rather than
a result, and it is recorded as a candidate on purpose.
Ruled out first: no engine-side name exists. No level file contains "Volume"
outside UI text ("Music volume", "Volume de la musique") and Italian item
descriptions. The format cannot be found by label and has to be structural.
gameplay.bin is already-decompressed level data. Its head is a count (336 on
level 0) then 16-byte records from byte 416, carrying (type, pad, offset, id).
The id groups them -- each id has exactly two records pointing into two parallel
offset arrays -- and those offsets resolve into a text pool, the first ones
landing mid-string in dialogue ("it Megacorp Lab"). So that part of the file is
dialogue and class dispatch, not geometry.
The structural hit: a table of 144-byte float records. Level 0 has one at
799204, 20 records of which 19 are populated and one is an all-zero terminator,
with 34-35 of each record's 36 float slots world-plausible and +0x0c holding a
varying world position:
(206.5, 284.1, 116.0) (265.0, 445.3, 131.7) (244.0, 407.2, 128.0)
(219.3, 247.0, 107.5) (185.1, 282.6, 104.9) (260.8, 423.6, 130.9)
(248.2, 198.4, 51.8) (185.8, 282.7, 102.6) (247.0, 211.5, 100.6)
A 144-byte record with world positions and float radii is the shape a volume
trigger region should have, and a matching table is present in every level
examined (0-9), each preceded by an all-zero record, counts 23 to 612.
It is NOT called volumes. Nothing ties this table to volumes. Camera bounds,
spawn points and path nodes would present exactly the same thing: float records,
world coordinates, no label. The discriminating evidence has not been gathered,
and writing "volumes" in the roadmap because the shape fits would be the exact
error this log keeps catching.
Two detector bugs, both mine, both confident before they were caught:
1. The validity test accepted zero as a world-plausible float, so every
zero-filled region scored perfectly and the cross-level check returned runs of
225 to 1,514 records that were obviously padding. Requiring non-zero floats
(>=20 of 36) separated real tables from memory immediately.
2. The 144-byte stride came from four offsets I eyeballed as being 144 apart.
Gap 144 was not in the top eight gaps of the detector that found them -- it
survived because three consecutive observations agreed, not because the
measurement supported it. The corrected detector then found much longer runs,
which is the opposite of what the eyeball version implied.
Same shape as every other correction here: a clean small plausible number read as
conclusive because nothing was checked against a control that could disagree.
To actually settle it, the table needs to be correlated with something known --
if a count matches a level's visible trigger regions, or if an id in the record
resolves through the class table to a named object, that is the discriminator.
Until then volumes remain open and the roadmap says so.
|
||
|
ReaperOfVeriod
|
28ab7e094b |
SBlk head characterised: four constants and one word that is not a count
sound_bank's head was parsed but never characterised. Across all 27 levels the
SBlk region opens identically for its first twelve bytes:
+0x00 "SBlk" +0x04 1 +0x08 4 +0x0c "DAW\0" +0x10 0
and +0x14 is the only word that varies. It varies with its low sixteen bits zero
on all 27, leaving 0x0092..0x0103 -- 146 to 259.
That is the shape of a record count, and it tracks the body length at close to
100 bytes per unit. It is not one:
body / field_14 91.2 .. 110.8 across 27 levels
any stride 0..80 with body = stride*word + constant exactly: no
So the test asserts the negative too. If a stride ever makes the relation exact
on all 27, sound_bank_sblk_head_is_constant_apart_from_one_word fails and asks
for the record size to be revisited. A near-miss at 100 bytes/record is exactly
what hardens into a name the next time somebody glances at it; the constants are
also checked individually so a real deviation names which one moved.
The DAW region begins with sixteen zero bytes then block-structured data with
16-byte periodicity and slowly varying nibbles -- consistent with block
compressed audio, not decoded. Recorded as such.
Two mistakes, both mine, both the same shape as earlier ones:
1. The constructor bound `format_tag` with a `let` BEFORE the two words that
precede it in the file, consuming bytes 4..8 and shifting every field after
it. The struct literal listed the names in the right order; the statements did
not. A scratch script reading the same bytes gave the opposite answer, which
is what exposed it. Fixed, with a comment, because "read the lets in file
order" gets undone otherwise.
2. Inserting the new test ahead of an existing one stranded that test's #[test]
on my doc comment -- silently unregistering
tie_vertex_element_is_16_bytes_with_fixed_texcoords. This is the SAME mistake
as two commits ago and I made it again. It again surfaced only as clippy's
"never used" rather than as a missing test. Restored and verified by listing
the registered tests rather than trusting the count.
240 tests, clippy clean, both tie vertex tests confirmed present.
|
||
|
ReaperOfVeriod
|
bc5ebe559f |
Tie slots 3, 6, 7: two readings excluded, and why the LOD test stops working
Having established that i16[0..2] are positions, ran the same LOD-overlap method
across all eight slots to see whether it discriminates elsewhere. It does not,
and that is worth recording before someone reads the passing numbers as nine
confirmations:
slot same class control ratio
i16[0] 97.0% 3.0% 32x
i16[1] 96.7% 3.2% 30x
i16[2] 96.4% 6.0% 16x
i16[3] 75.3% 78.7% 1x
i16[4..7] 68-91% 52-79% 1x
Only i16[0..2] carry signal, because they are the only slots with tens of
thousands of distinct values. The rest draw from sets of 226 to 1,759 values, so
any two sets of that size overlap heavily regardless of what the field means. A
passing result from a method that has run out of discriminative power is not a
passing result.
Slot 3 excluded as a skin/joint index. It holds 201 values in 0..=254 and 50
distinct values within a single draw packet, so it is per-vertex, not per-material
-- which invites the obvious reading, since tie objects are skinned. Wrong:
mean pairwise distance, same slot-3 value 14,481
mean pairwise distance, different slot-3 value 13,795
ratio 0.95x
control: same position-x bucket 8,105
Vertices sharing a slot-3 value are marginally FARTHER apart than vertices
differing in it. The control halves the distance, so the measurement works; the
0.95x is on the wrong side of neutral, though by too little to claim much on its
own. Combined with no LOD sharing (75.3% against a 78.7% control), slot 3 is
per-vertex, uncorrelated with the geometry it sits in and with the detail level.
Recorded as not understood. A per-vertex animation phase fits the arithmetic and
is labelled a guess rather than offered as a reading.
Slots 6 and 7 are mostly not per-vertex. Distinct values per draw packet, median
over 1,804 records:
i16[0..2] 13-17 constant for the whole packet 0%
i16[4,5] 6, 4 0%
i16[6] 2 43%
i16[7] 4 34%
They take a single value across an entire packet in well over a third of records,
which matches the 91.8%/88.9% dominance of 4096 and 0. Unnamed.
So the tie element is: position, byte-sized index, texcoords, layout tag, and two
slots characterised but not named. Positions, texcoord placement, element size and
the layout selector are all asserted; the remaining three are not, and the reasons
they are not are now written down rather than left as silence.
On "is Phase 2 nearly done": 40 items closed, 9 open. The nine are the hard
ones -- collision cells blocked on the game's own code, volumes not located at
all, moby_anim ruled out but unexplained, chunk record fields unnamed. Progress
on the checklist is real but the remainder is not the easy tail.
|
||
|
ReaperOfVeriod
|
e86de3c73e |
Tie i16[0..2] are the vertex positions, with a control in the test
The three slots ahead of the index were "unproven position" for want of a test
that could fail. One is available on the disc alone, with the control built in.
LOD 1 and LOD 2 of a tie class are the same object at two levels of detail, so
anything that locates a vertex in that object's own geometry is shared between
the two blocks; a different class' LOD is an unrelated object. Intersecting the
sets of i16[0..2] triples both ways turns "is this a position" into a
measurement. Levels 0-5, ordinary-layout elements only:
same class, LOD1 vs LOD2 96.1% 107 classes, 107 nonzero
different class, control 0.4% 2,144 pairs
A factor of 240. Per level the same-class figure runs 94.4 / 97.9 / 98.1 / 96.7 /
96.8 for levels 0-4. The control stays under 1.0% except for one level-1 pair at
45.9% -- two classes sharing most of their geometry, which is what duplicate or
variant models should look like. Reported, not excluded, and the assertion is on
the mean rather than the max so it does not have to be.
Asserted in tie_vertex_slots_0_to_2_are_shared_positions, including the ratio
(same > 20 * diff) so a change that collapsed both numbers fails rather than
passing twice, and a nonzero-overlap check so no class can silently stop
agreeing with itself.
What this rules out is the point. An index, a handle or a per-vertex counter
would not be shared at 96% between two independently built LOD meshes, nor sit at
0.4% between two objects. With 10.7k-13.2k distinct values per slot over the
full i16 range, these are positions in the object's own frame. Scale and axis
order stay unnamed -- nothing here says centimetres, and axis order is
unconstrained because nothing here assumes one.
Two things worth recording about the process:
1. The diagnostic line printed "1.0%" for 99.4% because the format was
`{same:.1}%` with no `* 100`. The assertions compared fractions and were
always correct; only the readout was wrong. Same family as everything else in
this log -- the readout disagreeing with the claim.
2. Inserting the new test in front of an existing one stranded that test's
#[test] on my doc comment, which silently unregistered
tie_vertex_element_is_16_bytes_with_fixed_texcoords. It showed up as
"never used" in clippy, not as a missing test. Restored; both tie vertex
tests are registered and 239 tests run.
Element layout now reads:
+0x00 i16[0] position x +0x06 u8 index 0..=254
+0x02 i16[1] position y +0x07 u8 layout tag, 0 in 93.4%
+0x04 i16[2] position z +0x08 i16[4] texcoord u
+0x0a i16[5] texcoord v +0x0c/+0x0e i16[6], i16[7] unnamed
|
||
|
ReaperOfVeriod
|
06cf2fd46d |
Tie vertex slot 3 is a byte-sized index, not a coordinate
Splitting the 95,612 elements of levels 0-2 by i16[3]'s HIGH byte, which is the
layout selector:
0x00 89,341 93.4%
0x03 5,344 5.6%
144 other selector values 927 1.0%
146 selector values in all. Not a two-way switch, and the second case is not 0x01.
Inside the ordinary layout i16[3] is not a coordinate: 201 distinct values, all
within 0..=254, with 0 most common. That is an index. So the selector and the
index are not two independent halves sharing a slot -- it is one byte. The low
byte is the index, the high byte is the layout tag, and the ordinary layout never
sets the high byte. Asserted.
Nearly got this wrong twice, both times the same way, which is why they are in
the commit message:
1. The first pass gated the range on `w >= 0` instead of on `(w >> 8) == 0`,
letting in the other layouts' large positive values. That reported "564
distinct values in 0..=32664" -- a number that looks exactly like a
coordinate and is really two populations unioned. The selector byte is the
filter. The filter is part of the claim.
2. Two constants were guessed rather than measured (selector cardinality 159,
and an upper bound that ignored non-ordinary layouts) and both were wrong.
The cardinality is 146. Guessing a constant and then asserting it is how a
measurement becomes a tautology.
Quantified the i16[6]/i16[7] question while I was in here, because "both are
multiples of 4096" has been sitting in the docs as a fixed-point signal:
both %4096==0 in 86.5% of elements
expected if the two residues were independent: 84.3%
enrichment 1.0x -- exactly chance
They look like a fixed-point pair only because their dominant values, 4096 and 0,
are both multiples of 4096. Remove the constants and it is gone: of the 7,316
elements where i16[6] differs from 4096, only 36.7% are still multiples, and of
the 9,905 where i16[7] differs from 0, ONE is. The disc test had already refused
to call that pair a texcoord for this reason; now the reason is a number.
Remaining slots in the ordinary layout, recorded: i16[0..2] read as position but
are unproven (10.7k-13.2k distinct each, full i16 range), i16[4]/i16[5] are the
texcoords, i16[6]/i16[7] unnamed.
Also fixes two clippy map_or lints and an import that a stray --fix had broken.
238 tests, clippy clean.
|
||
|
ReaperOfVeriod
|
1fa1913e38 |
Reconcile the docs after the last three corrections
The last three commits changed things the documentation still asserted in the
old direction. Fixed:
roadmap
* 227 tests -> 238.
* The level_overlay bullet contradicted itself: it said "the header words are
not named" in the same breath as "+0x00 load base is 0x1A7C80". Now says
+0x00 IS the load base and is named, +0x0c is excluded as the .lit size by
sign (below file length on 16/27, by up to 8.9%) and remains unnamed, and the
unnamed remainder is stated as such.
* Section names aligned with the Phase 3 list (lvl.vtbl, lvl.camvtbl,
lvl.sndvtbl).
core-data-regions.md
* The claim-assertion audit still said the two PCSX2 claims were "NOT
assertable here -- needs a savestate". They are asserted now, gated on
GC_SSTATE, in the same file. Updated, with the two corrections that fell out
of writing them recorded right there.
* "that string is inside the collision routine" was never established. It is
in the string pool, yes; the reference to it is not found. Reworded to say
so, including the zero-hit result.
research log
* field_00 now carries its name and the .lit/$gp/program context.
* The field_0c falsification and the byte-order retraction are both recorded
here as well, so the log and the format file agree.
Nothing left asserting the withdrawn anchor, and no stale test counts.
|
||
|
ReaperOfVeriod
|
282e633a7c |
Withdraw the collision anchor: it was a byte-order bug in my own search
Last commit claimed exactly one GP-relative occurrence of the collision assert
string's $lo half, at 0x354cc8, and called it an unambiguous anchor. That was
wrong, and building the disassembler is what proved it.
The search compared word bytes 1..3 against the 16-bit immediate. The PS2's MIPS
is little-endian, so an I-type immediate sits in bits 15..0 -- word bytes 0 and
1. Bytes 1..3 are the opcode, $rs and $rt fields. The match was on those.
The instruction at the phantom hit is 0x24a89948:
addiu $t0, $a1, -0x66b8
whose real immediate is 0x9948 -- ten bytes from the 0x99a8 being searched for.
-0x66b8 merely looks like a plausible $gp offset, which is what made it
convincing. With the byte order corrected the count is ZERO, in 2,751,268 bytes.
So the anchor is not established and the assertion is withdrawn. What survives is
still worth having: the functions are named (Camera_CollPrimTest takes (prim,
grid)), $gp = 0x1AEFF0, .lit loads at 0x1A7C80, and the assert string is a unique
copy at overlay+0xd18.
Probably the code reaching that string does not address it through $gp, or
reaches a copy in another section. The decomp's assert-message-names.tsv gives
0x002db350 for level 0 -- inside .lit, but it disassembles to a function epilogue
(jr $ra / addiu $sp, $sp, 0x70), so those are code locations, not string
addresses, and they are not traced to the assert either.
tools/overlay-anchors now does the search with the correct byte order and
carries a comment on exactly the part that was wrong, because that is the part
that will be re-derived otherwise. It also disassembles around an address:
cargo run -q -p gc-overlay-anchors -- <overlay.bin> --disasm 0x2db350 0 64
The useful general lesson, which the roadmap already says and this is a second
demonstration of: a probe that returns a small, clean, narratively satisfying
number is the most dangerous kind of wrong result. "Exactly one hit in 2.75 MB"
read as conclusive. It was an artifact of comparing the wrong two bytes, and it
survived a commit because it fit the story.
|
||
|
ReaperOfVeriod
|
1f903efe23 |
An external decomp turns two overlay constants from measured into named
OpenRAC/rac2-gc-decomp is the same disc as ours, verified rather than assumed:
ours 3,828,350,976 9db2e33e...b1ce5
theirs 3,828,350,976 9db2e33e...b1ce5
Vendored read-only under vendor/ (already gitignored) and added to PROVENANCE.
It names two constants this parser had measured but never explained:
* overlay.bin +0x00 = 1,735,808 = 0x1A7C80, the .lit load base. It is constant
across all 27 levels because every level's literal section loads at the same
address. So the first header word is the load base of the section the file IS,
which also establishes that overlay.bin is that section -- it holds both code
and the debug string pool. $gp = 0x1AEFF0 comes with it, and that closes the
roadmap's open "$gp for this game" item.
The program around it is seven sections -- .lit .bss .data vtbl camvtbl sndvtbl
.text -- which is precisely the "seven-section parser and relocation" Phase 3
item for this format. Known target now rather than a guess.
I then tested the obvious follow-up and it is wrong: if +0x00 is the .lit load
base, +0x0c ought to be the .lit size. It is SMALLER than the file on 16 of 27
levels, by up to 8.9% (level11: file 3,151,128 vs 2,870,792). Excluded by sign,
not by a near miss. Still unnamed.
Collision: still blocked, but no longer blocked on tooling. The decomp names
Camera_CollPrimTest(void *prim, void *grid) without decompiling it, so the cell
layout is not handed over. What it does supply is $gp, and that is enough to
start. "Camera_CollPrimTest WARNING! - grid out of bounds! (RAR)" is at
overlay+0xd18, the only copy in the file, so:
string overlay+0xd18 -> RAM 0x1a8998; delta to $gp -0x6658; $lo 0x99a8
MIPS puts the $lo half in bytes 1..3 of a 4-byte instruction. Searching the whole
overlay instruction-aligned gives exactly ONE hit:
0x354cc8 (overlay+0x1ad048)
One occurrence in 2.75 MB of code is unambiguous: that is the instruction
materialising the assert string, and its enclosing function is
Camera_CollPrimTest. That is the route in, and it needs no symbols.
tools/overlay-anchors does this step, dependency-free. I started this with
capstone for the disassembly proper and dropped it: capstone 0.14 does not
re-export Mode/Endian from a public path, and an excluded-from-the-workspace
native dependency is a poor trade for a step that does not need one yet. The
anchor search is pure byte work, so it is pure Rust. No native dependency is
added anywhere, and the tool is an ordinary workspace member.
Surveyed the rest of that project against our open items: no volume structures,
no tie vertex layouts, no SBlk/DAW fields, no wavelet tables. It is a guide to
the code, not a replacement for reading it.
|
||
|
ReaperOfVeriod
|
26a0918178 |
Gate the PCSX2 claims as tests; two corrections fall out of writing them
The audit two commits ago listed three claims as "not assertable here -- needs a
savestate". With PCSX2 running they are assertable, so they now are. gc-trace
already exposed gc_trace::Savestate::ee_ram_owned(); gc-extract just needed it as
a dev-dependency. Both tests are gated on GC_SSTATE and skip cleanly without it,
so a normal checkout pays nothing.
Writing them corrected two claims, which is the main reason they were worth
writing as tests rather than staying prose.
1. The collision payload is NOT "byte-for-byte identical". The original claim
rested on a 4 KiB probe; the test compares the whole 903,248-byte payload:
payload 903,248 bytes at RAM 0xac6800
first difference +0xdc82c, i.e. 903,212 bytes in
bytes that differ 6 (0.00%)
903,212 of 903,248 bytes are verbatim and 6 are not, all in the final 36.
"Verbatim" was an overclaim on a probe that never looked at the tail. The test
now measures the whole payload and reports the extent, asserting only that the
divergence is beyond the head.
2. The tfrag relocation base is the TABLE address, not the block base. The delta is
8,683,584 = 0x848040 = block base 0x848000 plus table_offset 0x40 -- which is
exactly what TfragHeader::data's own documentation says, "relative to the
table". The earlier claim said "the block's RAM base", off by exactly the table
offset.
Also: +0x10 and +0x38 do NOT share one constant. The aggregate "1,665 fields
relocate by one constant" was true only because +0x10 carried it; lumping them
hid that. The test asserts +0x10 and REPORTS +0x38's delta rather than assuming.
The patch-rewrite figures also restated on aggregate rather than median, which is
why they read lower than the 7.80%/8.12% medians recorded earlier:
patch bytes rewritten -- cannot morph 1.9% of 334, can morph 2.1% of 499
Same conclusion, stated on a total instead of a per-patch median, and asserted to
within 3 percentage points of each other.
Net: the audit table in docs/formats/core-data-regions.md now has no "prose only"
rows for disc-or-savestate-readable claims.
|
||
|
ReaperOfVeriod
|
9f938f47de |
Assert the tie vertex layout finding in a test, and audit which claims are asserted
The i16[3] high-byte finding -- the most recent and most subtle result of this stretch, and the one I corrected twice -- lived only in prose. Two commits is exactly how long a claim can sit unasserted before it drifts. It now has a test: tie_vertex_layout_is_selected_by_i16_3_high_byte which reproduces the figures exactly and, importantly, asserts both directions: (4,5) and (6,7) are above 0.85 and 0.80 in the ordinary group, (6,7) is BELOW 0.99 so it cannot silently become the constant I once mistook it for, (0,1) and (2,3) are under 0.20, and every pair in the non-zero group is under 0.15. A future change in either layout now fails loudly. Also audited the rest of this session's claims against the test suite and recorded the result in the format doc: tie vertex element is 16 bytes asserted layout selected by i16[3] HIGH byte asserted (this commit) blocks self-delimiting on 0xf6 asserted per-block packets == 0 means no block at that LOD asserted packets[0] == 256 on 1,864 of 1,864 asserted chunk record length uniform per chunk asserted moby class IDs in one body slot per type asserted collision payload verbatim in RAM NOT assertable here -- needs a savestate tfrag patch bytes all rewritten by LOD same morphing is unobservable in that image same The split is deliberate: disc-readable claims are asserted against the disc, savestate-dependent ones live in plan/pcsx2-verification.md with their method. What is not acceptable is a disc-readable claim living only in prose -- which is how the tie i16[3] finding spent two commits before it had a test, and how several older claims drifted before the roadmap reconciliation caught them. |
||
|
ReaperOfVeriod
|
922248a4ec |
Tie: the element layout differs by i16[3] high byte, and (6,7) is a real pair
Two results, one of them a correction.
1. "(6,7) is just the constant 4096/0 pair, not a texcoord" -- wrong. It looked
like an artifact because a constant is trivially a pair of multiples of 4096.
But partitioned by i16[3]'s high byte it is a genuine fixed-point pair in its
own right:
i16[3] high byte ZERO (n = 89,341) divisor rate by slot pair
pair /1 /4 /16 /256
( 0, 1) 100.0% 11.5% 4.6% 3.3% baseline
( 2, 3) 100.0% 11.0% 4.5% 3.3% baseline
( 4, 5) 100.0% 92.9% 92.0% 91.8% the texcoords
( 6, 7) 100.0% 89.5% 87.3% 86.5% a SECOND fixed-point pair
If (6,7) were a constant it would score 100%, not 89.5%. The constant is the
common case WITHIN it, not the whole of it -- which is exactly the distinction
the constant-artifact lesson is about, and I applied it backwards last time.
So the ordinary element carries TWO fixed-point pairs, (4,5) and (6,7), with
(0,1) and (2,3) at baseline. What (6,7)'s role is remains open -- second UV
set, normal, or a zero-heavy fixed-point pair all fit.
2. The element LAYOUT differs between the three cases. Scoring all four slot
pairs within the non-zero high-byte group, no pair stands out:
i16[3] high byte NON-ZERO (n = 6,271)
pair /1 /4 /16 /256
( 0, 1) 100.0% 4.6% 3.5% 0.7%
( 2, 3) 100.0% 2.0% 0.6% 0.0%
( 4, 5) 100.0% 1.9% 0.3% 0.1%
( 6, 7) 100.0% 7.5% 5.7% 4.5%
Best is 7.5% against a 2-5% baseline. So for those 6,271 vertices the 16 bytes
are not laid out as position/texcoord/normal at all -- a different element
shape, selected by the high byte.
This also explains why the global slot-pair score looked ambiguous to begin with:
pooling the two layouts averages a 92% pair and a 7% pair into one number that
describes neither. Fourth time this session that partitioning produced the answer
rather than merely bounding it.
|
||
|
ReaperOfVeriod
|
ed3e85a0f2 |
Refine the tie texcoord finding: it is i16[3]'s HIGH byte that selects
The previous commit said i16[3] selects the texcoord encoding. That was right but loose -- it is specifically the HIGH BYTE, and the low byte is a separate thing. Grouped by the high byte across the same 95,612 vertices on levels 0-2: i16[3] high byte n % / 16 % / 256 0x00 89,341 92.0% 91.8% 0x03 5,344 0.0% 0.0% 0x04 71 0.0% 0.0% 0x40 58 5.2% 0.0% 0x02 47 2.1% 2.1% 0x19, 0x05, 0x20, ... the remaining 73 A high byte of zero is the ordinary case -- 89,341 of 95,612, 93.4% -- at 92%. EVERY other high byte scores at or near zero. The split is on the byte, not on the value: i16[3] of 1 and of 771 differ only in the high byte and behave completely differently. Within the all-zero case there is a third distribution: i16[3] == 0 exactly (5,851 vertices) scores 62% at 16 and 61% at 256, against 92% for 1..255. So three cases, selected by the high byte and then the low: high 0x00, low 0x00 5,851 a third distribution, 62% at 1/16 high 0x00, low 1..255 83,490 the ordinary case, 92% at 1/4096 high non-zero 6,271 0%, an encoding this reading does not see The 6,271 matches the figure quoted last commit for "vertices using an encoding this reading does not see", and the two group counts sum consistently with the committed test's 95,612 -- so this refines the finding rather than changing it. A measurement trap worth recording: the all-zero case is a SUBSET of high byte 0x00, so summing the per-high-byte groups double-counts it. An intermediate total came out at 100,792 against the true 94,941 for those groups; the committed test's figure is the one to trust. |
||
|
ReaperOfVeriod
|
e27c40e738 |
Explain the tie texcoord residual: i16[3] selects the encoding
The texcoords are 12-bit fixed on 82.5% of vertices, not 100%, and the residual had no explanation. Grouped by the vertex's i16[3] slot across 95,612 vertices on levels 0-2, the buckets separate completely: i16[3] n % / 4096 % / 256 negative 333 1.2% 1.2% == 0 5,851 29.2% 60.7% 1..63 28,884 92.7% 94.3% 64..255 54,606 92.3% 93.8% 256..511 8 0.0% 0.0% 512..1023 5,391 0.0% 0.0% >= 1024 539 0.2% 0.2% The residual is fully accounted for. 87.3% of vertices sit in 1..255 at about 92.5% fixedness, contributing 0.873 * 0.925 = 80.8 points; the 5,851 at i16[3] == 0 contribute 1.8 more. That is the 82.5% measured, to the point. The remaining ~6% are not 12-bit fixed at all. i16[3] >= 256, and negatives, score 0% at BOTH 4096 and 256 -- 6,271 vertices in an encoding this reading does not see. And i16[3] == 0 scores 29.2% at 4096 but 60.7% at 256, which is a third distribution rather than a weaker version of the same one. What this establishes: i16[3] is a per-vertex packed attribute whose value SELECTS HOW the texcoord pair is stored, rather than an independent field sitting beside the texcoords. 1-255 is the ordinary case, 0 is special, 256-and-up and negatives are a different encoding. What those three cases MEAN is still open -- texture index, normal index, flag or a combination -- but it is now a field whose role is known, which is more than "packed word". Worth noting the shape of the claim this replaces: "the texcoords are 12-bit fixed at i16[4]/i16[5], 73.6% of the time, unexplained". The field position was right; the 73.6% was a population average over three encodings. Partitioning by the one variable that separates them turns a mystery into a lookup -- rule 3a again, and the third time this session that partitioning a mystery produced the answer rather than merely bounding it. |
||
|
ReaperOfVeriod
|
3ee0b3d0dc |
Disprove that morphing causes the tfrag byte changes: LOD selection does
Read the whole tfrags block from RAM -- 4,379,200 bytes at 0x848000 -- and compared
every one of the 833 patches against its disc copy, grouped by whether it CAN morph:
patches with NO morphing entries 334 differing: max 10.59%, median 7.80%
patches WITH morphing entries 499 differing: max 10.49%, median 8.12%
patches 100% identical to disc 0/334 and 0/499
The two groups are indistinguishable. Morphing is NOT what changes the bytes, and
this disproofs the claim two commits ago that the 8.6% was "morphed positions with
topology and UVs left verbatim". If it were, the 334 patches with no morphing
entries would be untouched -- and none of them is.
Nor is it a pointer relocation: the differing bytes fall in short structured runs
(257-263, 272-277, 305 on patch 0) whose u32 deltas share no common value.
The explanation consistent with everything measured is that the loader REGENERATES
each patch's VU packet stream for the LOD tier it selected. That happens to every
patch, morphing or not -- and it is why morphing shows no signal.
This also settles the verification route, which the previous commit left as "needs
the camera distance at the savestate". It does not, and no camera distance would
help: this memory image shows the LOD-selected form rather than the morph state,
so there is nothing in it to solve a weight against. Verifying the derivation needs
a captured frame at a known view distance instead.
What is established now, all measured on the full population rather than inferred
from one patch:
- data and cube_ofs are relocated pointers, delta 0x848000, on all 833 records
- every patch's bytes are rewritten, ~8%, by LOD-driven packet regeneration
- morphing is not observable in this memory image
- 833 of 833 patches parse; 499 of 833 have morphing entries; 43,613 vertex-info
entries across the level
|
||
|
ReaperOfVeriod
|
aff977e8c5 |
Correct the tfrag pointer-field claim, and show morphing regenerates VU packets
Two corrections to the previous commit, both from continuing the RAM comparison.
1. The previous commit said the RAM comparison "names two fields the disc-side
work had left as data and an unnamed word". It does not. +0x10 is
TfragHeader::data and +0x38 is TfragHeader::cube_ofs, both already named in the
struct. What RAM adds is that both are POINTERS -- relocated by adding the
block's RAM base -- rather than plain offsets. That is still worth knowing, but
it is not a naming.
2. The previous commit attributed the 8.6% of rewritten bytes to "morphed
positions with topology and UVs left verbatim". That is wrong, and patch 1 shows
why. Comparing 4 KiB, disc 0xdf60 against RAM 0x855f60:
bytes differing 354 of 4,096 = 8.6%
contiguous runs 120
longest run 8 bytes
extent +0xc1 through the end of the window
Patch 1 has 83 positions and 83 vertex-info entries, of which 13 morph. If
morphing rewrote position fields alone, the differing bytes would be a 13/83 slice
of a 498-byte position array -- a couple of dozen bytes in one contiguous region.
Instead 354 bytes differ, spread over 120 short runs across the whole window.
So the loader is not patching positions in place: it REGENERATES the VU packet
stream for the current morph state. That is what morphing is on this hardware --
VU1 GIF packets, not a CPU-side vertex array.
Consequence for verification, and the reason this is worth committing rather than
just noting: the plan of "map RAM vertices to disc indices and solve for the weight"
does not work against this memory image. The positions are inside regenerated
packets, so there is no stable per-vertex correspondence to read a weight out of.
Verifying the morph weight against the original needs the CAMERA DISTANCE at the
savestate, so the weight can be computed from the derivation and compared with the
rendered result, rather than read back out of RAM.
Population figures gathered on the way: 833 of 833 patches parse, carrying 43,613
vertex-info entries; 499 of 833 patches have at least one morphing entry, and
patch 0 -- the first record -- is base-only with none.
Also fixed a measurement error mid-run: I compared patch 1 at disc 0xdf60 against
RAM 0x855f00, off by 0x60, and got "every byte differs". The corrected base is
0x855f60.
|
||
|
ReaperOfVeriod
|
93591a79de |
Verify the tfrag record framing against RAM: two pointer fields, morphed vertices
The tfrags block header -- table_offset 0x40, tfrag_count 833, lod_distance 32.0 -- hits once in the level-0 savestate at RAM 0x848000, so disc offset 0 maps to RAM 0x848000 and the block can be compared position for position. Result 1: exactly two fields per record are rewritten. Comparing all 833 64-byte table records field by field: records compared 833 u32 fields differing 1,665 = 833 x 2, and no others their offsets +0x10 and +0x38, on every record relocation delta 8,683,584 = 0x848000, on 833 of them So +0x10 and +0x38 are block-relative pointers and the loader adds the block's RAM base to each. Every other byte of all 833 records is verbatim. Six fields differ by 0xF80000 and 0x1F00000 instead -- three records pointing into a different buffer, recorded rather than averaged. This names two fields the disc-side work had left as `data` and an unnamed word. Result 2: behind the pointers, about 8.6% of bytes are rewritten. Following record 0's +0x10 to disc 0xd040 / RAM 0x855040 and comparing 16 KiB: 14,977 bytes identical, 1,407 differ = 8.6%. The differences spread across many field positions rather than concentrating, which is what morphed POSITIONS look like -- topology, UVs, counts and command lists stay verbatim while a fraction of the position bytes move. 8.6% is about half of a 16-byte vertex's six position bytes, consistent with roughly half the vertices in this patch mid-morph at the saved camera distance. Not established: that the morphed bytes match resolve_vertex's formula. That needs mapping RAM vertices to disc vertex indices and solving for the weight, which is the next step -- but the framing is now known well enough to attempt it, and the two pointer fields are confirmed rather than inferred. This also answers the "is morphing real?" question that made the earlier tfrags probe fail to match RAM: the tfrags block is not verbatim in RAM, because both its table pointers are relocated and its vertex positions are morphed. |
||
|
ReaperOfVeriod
|
6d0c1b6fd1 |
Close out tfrag on the disc side, and fix two stale "not recovered" claims
Picked tfrag morph targets from the open list and found the premise was wrong. There are no morph targets in this format. Morphing is a blend toward the midpoint of the vertex's two PARENT vertices -- g[parent1_position] and g[parent2_position] -- which are other vertices in the same tfrag. The engine implements exactly that in resolve_vertex, and it is covered by morphing_blends_toward_the_parent_midpoint plus morph_weight_runs_from_zero_to_one_across_its_band and collapse_distances_are_the_far_edge_of_each_band. So "morph targets unrecovered" was never a gap; it named a thing that does not exist in the format. Strip bookkeeping is likewise already there: TfragStrip carries vertex_count_and_flag, and tfrag_triangles walks the strip command lists and index arrays to build triangles. That was the other half of the same roadmap entry. Coverage, all passing: 36 tfrag tests, 9 texture tests, 23 gc-engine tests. The real-disc texture test decodes all 1595 level textures, with pixels and mip levels, so the module doc's "not yet recovered: the texel data the texture table points into" is also stale. Two stale claims corrected: 1. crates/gc-formats/src/tfrag.rs said "Not yet recovered: the LOD switch and morph rules in TfragProc, and the texel data the texture table points into. See docs/plan/tfrag.md." The rules are recovered and tested, the texels are decoded, and docs/plan/tfrag.md DOES NOT EXIST -- a dangling reference to a file that was never written. 2. docs/roadmap.md marked tfrag partial with "Still open: morph targets and the strip/chain bookkeeping". Neither is open. The honest replacement for both is what is genuinely not claimed: the LOD switch distances, morph weight and collapse edge are DERIVED from the data -- tiers are classified from the four V4_8 parent arrays, the weight runs zero to one across each tier's band -- but have not been checked against PCSX2 the way the lighting pass was. That is a verification gap, not a parsing gap, and it needs a frame captured at a known view distance rather than more disc-side measurement. |
||
|
ReaperOfVeriod
|
d56b439502 |
Exclude the collision segmentation theories on the full population, and name the
unblock Continuing on the verbatim payload with the grid known. Cells are not zero-separated. Counting maximal runs of non-zero u16 across the whole payload: level 0 363 cells 15,115 runs mean run 25.3 u16 level 1 384 cells 24,647 runs mean run 51.5 u16 level 2 53 cells 31,512 runs mean run 30.3 u16 Two to three orders of magnitude apart, so a zero run is not a cell boundary. Cells are not uniform blocks either: level 0 averages 2,488 bytes per cell and level 2 averages 43,684, which cannot both be one fixed-size record. The run lengths are not records. Their histogram has a long tail of near-power-of-two values -- 2,140, 1,184, 840, 816, 628, 604, 562 -- which is what you get when a run is bounded by a single zero byte inside otherwise dense data, not by a delimiter. With the 18%-zeros finding, the payload reads as dense data with sparse holes rather than a delimited sequence. So all three segmentation theories are now excluded on the full population rather than on a sample: not ascending u32 tables, not zero-run delimiters, not fixed stride. What would actually unblock this: overlay.bin IS the game's own code, and the collision parser is in it. Camera_CollPrimTest WARNING! - grid out of bounds! (RAR) was located in the debug string pool earlier, and that string is inside the collision routine. So the format is recoverable -- but not by measuring bytes. It needs disassembly of that routine, or an R&C2 reference for the collision cell record. Stating that plainly is worth more than another scan. The disc-side measurement space for collision is exhausted: the bytes are available, verbatim, the grid is known, and three independent segmentation theories have failed on the whole population. Further probing without the loader's own logic will not converge. |
||
|
ReaperOfVeriod
|
338af9b7bd |
Fix the collision grid dims, and disprove two sample artifacts in the payload
With the payload confirmed verbatim in RAM, the disc bytes can be read directly.
The grid dimensions are the first two u16 of the payload: level 0 is 11 x 33 =
363 cells, level 1 24 x 16 = 384, level 2 1 x 53 = 53. Level 2 being one cell wide
is worth noting -- the grids are not all similar in shape.
Two readings disproved, both by measuring the whole payload instead of a sample.
An ascending u32 offset table. A probe from payload +1024 reads f4ae0000 40af0000
c4b2000020b20000, which is three ascending values before the fourth goes down
(0xb2c4 to 0xb220). Read as a table, that was a three-value impression. Scanning
all three levels for maximal ascending u32 runs of 8 or more entries finds zero,
on every level.
A global alternating-zero stride. The head of level 0 shows an obvious (value, 0)
pattern from about i16[37]:
[32..] [1808, 1872, 2198, 0, 34, 86, 25848, 1, 25824, 1,
-13028, 0, -12284, 0, -12228, 0, -12716, 0, ...]
Measured over the whole payload, both alignments score far below what that sample
suggests: even-slot aligned 18% second-half zero, odd-slot aligned 13%. The
sample again.
What the global numbers do say is that the payload is heavily zero-bearing -- 18%
against a chance rate of about 0.0015% -- which is what a sparse or variable-length
encoding looks like, and consistent with the variable-length reading established
earlier. The 18-against-13 split is a real alignment preference but a weak one.
The cell layout is still undecoded. The useful position now is that the bytes can
be read as they are and that they are sparse and variable-length rather than
fixed-stride, so the next attempt should look for a terminator-delimited walk
through the sparse region rather than for a record size.
|
||
|
ReaperOfVeriod
|
d5e637930b |
Show the collision payload is verbatim in RAM, and retract the "not in RAM" claim
This overturns the premise collision work was blocked on. Searching the level-0 savestate for a probe from collision payload +1024 hits at 0xac6c00; a probe from +64 hits at 0xac6840. The difference is 0x3C0 = 960, which is exactly 1024 - 64, so the payload sits at a fixed relative address -- RAM 0xac6800 for the disc address 0x27e800 -- and reading it back is byte-for-byte the disc's own bytes: RAM 0xac6800 0b 00 21 00 b6 08 ad 08 a4 08 6b 08 38 08 63 07 disc 0x27e800 0b 00 21 00 b6 08 ad 08 a4 08 6b 08 38 08 63 07 Nothing is reshaped. That is the opposite of the tie block, where the loader relocates a pointer inside each record. The retraction. The previous note reads: "level 1's 24 x 16 grid dimensions appear in memory, level 0's 11 x 33 appear nowhere, so disc-side decoding has to work against the reshaped form." Level 0's dimensions ARE in RAM, at 0xac6800 -- 0b 00 and 21 00 are the first two bytes of the block, 11 and 33. The search that missed them looked for 0b 00 00 00 21 00 00 00, the dimensions as a pair of u32s. They are stored as two u16s, so that pattern never occurs. So the negative result was an artifact of the search pattern, not evidence about the data -- and it was then used as a premise for several rounds of disc-side work. Both lessons were already written down in AGENTS.md and both were re-learned: a negative from a byte search bounds the search, not the format; and the control here is one command -- read the predicted RAM address and compare it to the disc -- which would have shown the verbatim copy immediately. Also recorded while mapping it: the payload contains ascending u32 offset tables from +1024 onward, and its first two bytes are the grid dimensions, so level 0's grid is 11 x 33. The cell layout itself is still undecoded, but the blocker is gone: disc-side decoding no longer has to work against a reshaped form. |
||
|
ReaperOfVeriod
|
180fc27b04 |
Locate the tie texcoord field at i16[4]/i16[5], and rule out the (6,7) artifact
Scoring each adjacent i16 slot pair of the 16-byte tie vertex for 12-bit fixedness over 36,501 level-0 vertices locates the fields: (i16[0], i16[1]) 1.1% not a texcoord (i16[2], i16[3]) 4.8% not a texcoord (i16[4], i16[5]) 73.6% <- the texcoords (i16[6], i16[7]) 76.6% <- the near-constant 4096/0 pair (6,7) scores HIGHER and is not a texcoord. It is the near-constant 4096, 0 pair: 4096 is 1.0 in 12-bit fixed and 0 is zero, so both are trivially multiples of 4096. That is the constant artifact again -- a fixed pair would score 100% anywhere. The field is (4,5), with (0,1) and (2,3) ruled out at 1.1% and 4.8%. The first four slots span the full i16 range with 10,803 to 13,421 distinct values each, so they are neither texcoords nor small fields. The vertex base is confirmed to be vert_ofs alone. Adding the shader or control region makes it worse -- vert_ofs only 73.6% bfc + vert_ofs 69.8% bfc + control_count*control_size + vert_ofs 66.7% -- so the 73.6% residual is not a mislocated base. Worth recording because vert_ofs (3..7) numerically overlaps the shader region (5*shader_count, 5..20 quadwords), which made a combined base look plausible; measurement says the simple one is right. The test asserts (4,5) above 60% and (0,1)/(2,3) below 5%/15%, and names the (6,7) artifact in its failure message so the higher-scoring pair cannot be mistaken for the answer later. |
||
|
ReaperOfVeriod
|
35b087c882 |
Establish the tie vertex element is 16 bytes, and correct the roadmap's tie entry
With TieRecord identified as RAC1's TiePacketHeader, each record's leading word points at its packet in the block payload. Following it and stepping vert_size elements from vert_ofs reaches the vertices: 95,612 of them on levels 0-2. Element size settled by comparison, not assumed. Requiring the texcoord pair to be 12-bit fixed -- both words multiples of 4096, which is how 0x1000 = 1.0 reads: element 16 bytes : 78,919 / 95,612 = 82.5% element 8 bytes : 45,661 / 95,612 = 47.8% element 12 bytes : 44,834 / 95,612 = 44.8% Sixteen wins by a factor of 1.7 over the nearest alternative. Observed layout, 8 x i16: the first three slots span the full i16 range with 10,803-13,421 distinct values each; i16[4] and [5] are the texcoords and are both multiples of 4096 on 82.5%; i16[6] reads 4096 on 85.8% and i16[7] reads 0 on 84.0%. Not established: what any field means, and why the texcoord rate is 82.5% rather than 100%. The residual tracks record shape -- (shaders 2, control 7, size 2) hits 60%, (shaders 1, control 18, size 5) hits 89% -- so it is not one bad block but something per-packet this reading does not capture. Worth recording that this is the OPPOSITE of what the disc alone concluded two commits ago. Reading the header word at +0x00 as an element count gave 0 of 922 blocks dividing; it is the byte offset to the payload, so the table it bounds has 16-byte records. The number was right and the reading was wrong, which is what the packet-header identification corrected. The roadmap's tie entry was also stale -- it still said "a 16-byte record is withdrawn", which the packet-header work had already reversed. Rewritten from the commits: both indirections, self-delimiting blocks, the TiePacketHeader identification with its 15,219/15,219 rule, the 16-byte vertex element, and what remains open. A test asserts the element size against both alternatives so the 82.5% figure cannot be quietly replaced by an assumption. |
||
|
ReaperOfVeriod
|
54862b64ee |
Identify the tie record as RAC1's TiePacketHeader, and retract a formula
The 16-byte tie record is R&C1's TiePacketHeader offset for offset: a data pointer at +0, then R&C1's byte fields at +4 through +0x10. The identification is not a resemblance but R&C1's own rule -- bfc_distance == 5 * shader_count -- holding on all 15,219 records of all 2,177 blocks on all 27 levels. Measured over the same set, and reported rather than asserted where it is not universal: bfc_distance == 5 * shader_count 15219 / 15219 vert_ofs <= vert_size all scissor_size <= scissor_ofs all control_size <= control_count all color_ofs == vert_size 12640 / 15219 color_ofs == vert_ofs + vert_size 0 / 15219 color_count, strip_count, strip_vertex_count zero 15089 / 15077 / 15078 R&C1's rule for the colour offset holding on none of them is the interesting line: the two games use that field differently, so it is named color_ofs for what this game's majority reads as and the 2,579 exceptions are counted rather than averaged away. TieRecord parses all twelve fields, invariants_hold asserts only the universal four, and the disc test counts the rest. A packet-length formula is withdrawn rather than softened. (vert_ofs + vert_size + scissor_ofs + scissor_size) * 16 was measured at 61 of 61 pairs and is not reproduced on the correct blocks -- 0 of 13,042. The block set was the error, not the arithmetic: the script that produced the 61 read packets[2] where packets[1] was meant, so it measured 461 blocks instead of the 2,177 the class tables define, and the formula agreed perfectly with a population that was not the tie blocks. TieRecord::packet_bytes goes with it, and the retraction is recorded in the format doc and in the struct. That misread also explains why the earlier RAM verification saw different numbers for the two inputs -- same lesson as rule 3, arriving through a wrong subscript instead of a wrong span. |
||
|
ReaperOfVeriod
|
4e90c282ad |
Make the tie record relocation a command, and verify it on two levels
The relocation rule from the previous commit was established by a script that lived in a shell history, which means the next person would have had to rediscover it. It is now `gc-trace verify-tie-records <unpacked-dir> <level> <class> <lod> <src>`: resolve the class, parse the block, search the RAM image for the loaded record table, report what it found, and exit 0 only when every record's pointer has become table + word and every field byte is verbatim. Swept over every (class, lod) slot of two levels, from PCSX2 savestates: level 0: 87 slots with a block, 87 verified, 0 mismatched, 61 absent level 1: 104 slots with a block, 104 verified, 0 mismatched, 78 absent So the rule holds on every LOD block of two levels the game had loaded, which is what makes the record layout a finding rather than a reading of one block. Writing the command also caught an off-by-one in the parser it exercises: TieBlockHeader::records started its slice after the 16-byte header, but the header IS the first record -- the table runs from the block start to data_offset, and a block with data_offset 0x50 holds five records, the first being the header itself and its pointer naming where the payload begins. Found because the tool's first run relocated nothing while the same check in a script had passed; the unit test fixture was wrong in the same way and is fixed. Two operational notes now in the format doc: the live emulator is not a stable input (verifying level 1 over PINE matched 104 of 104 while the user was on that planet, and the same command minutes later found no matching copy because they had returned to level 0 -- a miss there is not a counterexample, so the report says "no copy matched" rather than "failed"), and a class index past the table count is an error rather than a mismatch, so a sweep should stop at the level's class count rather than a round number. |
||
|
ReaperOfVeriod
|
e6eecb871f |
Decode the tie LOD block: it is a table of 16-byte records
The tie block's word at +0x00 was read as an element count, and the reading failed in a way worth recording: the payload length divided by it is a whole number on 0 of 922 blocks, at any record size and any header up to 512 bytes. That is the correct measurement of the wrong hypothesis -- the word never described the payload. Range measured: every LOD block of level 1 whose word is a non-zero multiple of 16, plus the two read by hand in full. Method: locate each block's record table in PCSX2's RAM by its twelve field bytes, which are the only distinctive 12-byte run in a block, then compare every record against its disc copy. Each block has several copies in RAM, and the one that matches is the loaded one -- checking the first hit is what made the first attempt report 0 of 16. Result: on 16 of 16 blocks, every record's leading word has become table_address + word, and the other twelve bytes are byte-identical to the disc. So a block is [16-byte header][record table][vertex payload], the records are 16 bytes -- a leading u32 pointer plus twelve bytes of fields -- and the header's word at +0x00 is the byte offset from the block start to the payload, that is, the length of the table. That is what the multiple-of-16 was: the table holds 16-byte records. It is never zero because there is always at least the first record, and its 42 distinct values are table lengths. Each record's pointer is relative to the table start, relocated at load to table + word. TieBlockHeader::data_offset and TieBlockHeader::records carry it, bound by the offset, with TieRecord for the twelve field bytes, and the disc test asserts the table is bounded on every block. What the records' fields name and what the payload holds are still open -- the payload is known to be variable-length, float-shaped, not LZ77 and not VIF1. |
||
|
ReaperOfVeriod
|
c2af0528e3 |
Correct the chunk loader finding and decode a record field
The loader note said every record gains a 02 00 prefix in RAM. It does not: the word is the record's own leading field, already on the disc at every record. What the loader changes is the pointer -- it addresses the word, two bytes earlier than the disc's table -- and that pointer shift is what made the bytes look inserted. Measured, in order: - the two bytes before a payload are identical in core_data and in RAM, record for record: 02 00 on level 0's 90-record chunk at 0x100b302, 08 00 on the level's first chunk. Nothing is inserted; - the RAM table's low 24 bits are the payload's minus 2, consistently, so the record is [leading u16][two index bytes][payload] and the loaded table addresses the leading word. That is also why the index bytes read as +2 from the pointer -- the same +2 the earlier RAM result quotes; - the word is a field, not a pad: uniform across a chunk's records on 1,439 of 1,509 chunks (None on the 70 that vary, reported rather than averaged), fixed within all 71 chunk types, and only ever 1, 2, 4, 6, 8 or 10 -- 554 chunks carry 2, 520 carry 4. Two readings of the value were measured and disproved, so they are recorded as disproved rather than left as candidates: it is not an entry count (the payload divided by it leaves 57 distinct fractional "sizes", and 162 chunks do not divide at all), and it is not the number of distinct leading bytes (agrees on 0 of 1,506 chunks). Chunk gains `prefix: Option<u16>`, and the disc test asserts the uniformity, the per-type fixity and the value set. The scratch example that produced the per-type class-ID table is dropped now that the same scan is a committed test, which also lands here: moby class IDs sit in one body slot per chunk type, with the slot varying by type and the distinct >= 4 filter load-bearing -- without it the top slots are constants that read as 305x because any value below the moby count counts as a hit. |
||
|
ReaperOfVeriod
|
8147110377 |
Bring the roadmap's chunk section up to date
Four commits of chunk work -- uniform record length, the leading-byte phase counter, the two-byte record index, and the PCSX2 RAM findings -- were written to docs/formats/core-data-regions.md and docs/plan/pcsx2-verification.md but never reached the roadmap, because the scripted replacements kept missing their target text and I did not check that they had applied. That is the staleness complained about earlier recurring through a different route: not a missing edit, but a silently-failed one. The tie, collision and level_overlay entries had landed; only the chunk ones had not. Rewritten from the commits rather than from memory, and the numbers are the ones the tests assert: 1,509 chunks, 34,491 records, uniform length 1,509/1,509, phase counter 1,262/1,509, index rule 1,235/1,509, RAM prefix 02 00 on 90/90. |
||
|
ReaperOfVeriod
|
2c4bbdc600 |
Use PCSX2 RAM to recover the chunk loader's transformation
This is what RAM was needed for: nothing about the loader's transformation is recoverable from the disc. Anchor: a 32-byte probe from inside a chunk record in level 0's core_data, searched in the level-0 savestate. It hits at 0x18538a8 and the record's first bytes at 0x185387e -- exactly chunk_base + record_offset, no rescaling. So the chunk payload is copied VERBATIM; unlike collision and the shrub packet tables it is not expanded or repacked. Result 1: the offset table is rewritten to absolute EE addresses. disc: f000057c f00006b0 f00007e4 f0000914 RAM: 0xf185387c 0xf18539b0 0xf1853ae4 0xf1853c14 The 0xf0000000|low24 tag becomes 0xf1850000|low24 -- tag replaced by the chunk's load base, low 24 bits untouched. Consecutive deltas agree exactly (0x134, 0x134, 0x130), which is why the disc-side stride is approximate rather than constant. Result 2: every record gains a constant 2-byte prefix 02 00, and the rewritten table points at THAT rather than at the record data. Walking all 90 records from the RAM dump: prefix 02 00 on 90 of 90; the disc's index bytes preserved unchanged at +2/+3 on 90 of 90. Result 3, the useful one: the byte at +2 takes exactly 0, 16, 32, ... 240 -- the 16 values index_rule predicts for step = 16, phase = 0. The mixed-radix record index derived from the disc is corroborated on the running game's own copy of the chunk. That is a disc-derived claim validated against the game, which is the strongest form of check available here. On disc a chunk is [16-byte header][10 zero bytes][tagged offset table][records]. In RAM it is [16-byte header][10 zero bytes][absolute-pointer table][records each prefixed with 02 00]. The 02 00 is unexplained -- possibly a record type or an alignment the loader needs -- but it is constant, so like the two index bytes it carries no per-record information. |
||
|
ReaperOfVeriod
|
e2b12a2e52 |
Rule out a count field in chunk record bodies; correct a six-record impression
With the record framing settled (leading two bytes = record index), the body is bytes 2..len. Four probes, mostly negative, recorded so the next attempt does not repeat them. No count field. For every chunk type, every u16 and u32 position in the first 64 body bytes, at element strides 1,2,3,4,6,8,12,16: does the stored value equal (record_len - offset) / stride on 90%+ of records? No type produced a hit. The record does not carry an element count near its head. The body is dense, not float-dominated. For the largest type (2430 records, 326-byte bodies): f32 reads within +/-5000 of origin 74%, uniform-random control 49%, zero-byte fraction 13%. Above baseline but far short of the tie blocks (53-64% against 16%, with 43% zero bytes). There is an index-like region near the tail. In the largest type, 12 of 163 u16 slots have a maximum below 0x1000 -- too small for float or half-float, so those look like indices or small counts -- clustered at slots 111-121 and 130-131. A sample-size correction. Dumping six records of one chunk showed slots 112-119 zero on every record plus a long run of other constants, which reads as "the record is mostly a per-chunk template". Measured across the 1,171 chunks with at least 8 records: 15,422 of 164,390 u16 slots are constant across their chunk, 9.4%, and 0 of 1,171 chunks are entirely constant. So 9.4%, and no chunk is a template. Six records looked like a majority because that chunk's tail carries shared constants. Per type the figure ranges from 9.4% overall to 55.6% on one type, so it is reported per type as well as pooled -- the pooled figure alone would hide that spread, which is the same trap rule 3a describes. |
||
|
ReaperOfVeriod
|
28680b4be3 |
Show the first two chunk-record bytes encode the record's own index
Continuing from the phase counter. Within a chunk: i : 0 1 2 3 4 5 6 7 8 9 10 11 ... 15 16 17 ... byte0: 0 16 32 48 64 80 96 112 128 144 160 176 ... 240 0 16 32 byte1: 0 0 0 0 0 0 0 0 0 0 0 0 ... 0 1 1 1 byte0 is step*(j mod g) and byte1 is (j / g), where g = 256/step and j = index + phase. The pair is the record's index in mixed radix: byte0 carries the low digits scaled by step, byte1 the quotient. Measured over all 27 levels: 1,235 of 1,509 chunks fit this on both bytes. Byte 0 alone fits 1,262, so the 27-chunk difference are chunks whose first byte follows the rule and whose second does not. On 1,235 chunks a record knows its own position and the two leading bytes hold no per-record information. ResolvedChunk::index_rule returns the (step, phase) only when the whole chunk fits on BOTH bytes, so the 274 that do not stay distinguishable and nothing skips a leading byte on a rule that does not apply to it. A number that moved when the implementation changed: a first probe reported 1,181; the committed method reports 1,235. The method is right. The probe accumulated ok0/ok1 flags across candidate phases with an early-exit that could skip a phase which would have matched both bytes, and it also disagreed with bucket_rule on the byte-0 count (1,208 against 1,262), which is what gave it away. Recorded because both numbers came from the same run and only the discrepancy exposed the probe as wrong. The lesson is narrow: when a one-off probe and a committed helper disagree, check the probe first. |
||
|
ReaperOfVeriod
|
79914c420e |
Show the chunk record's leading byte is a per-chunk phase counter; every chunk
has a uniform record length Follows the alignment result and settles what the leading byte is. Within one chunk the leading byte steps by a constant and wraps: offsets: 1420 1748 2072 2392 2716 3044 3372 3700 4028 4352 ... byte0: 0 16 32 48 64 80 96 112 128 144 ... Sweeping every (step, phase) over all 27 levels: fully explained: 1,262 of 1,509 chunks by step: 8 -> 51 chunks, 16 -> 554, 32 -> 520, 64 -> 137 unexplained: 247 chunks across 11 chunk types So on 1,262 chunks the leading byte carries no per-record information: it is a deterministic function of the record's index and the chunk's phase, which is a position marker rather than data. That is why the earlier "16 discrete values, uniform counts" pattern appeared at all. ResolvedChunk::bucket_rule returns the (step, phase) when the whole chunk fits and None otherwise, so the 247 unexplained chunks stay distinguishable and nothing skips a leading byte on the strength of a rule that does not apply to it. Every chunk has a uniform record length: 1,509 of 1,509. Length varies BETWEEN chunks of a type -- one type runs 76 to 104 bytes -- but never within one. This withdraws the previous commit's "variable-length type", which came from pooling minlen/maxlen across all of a type's records at once. That is the pooled-claim error rule 3a exists to prevent, and it is the third time this session a pooled measurement produced a claim that partitioning reversed. The type is not variable-length; the chunk is fixed-length and different chunks differ. |
||
|
ReaperOfVeriod
|
a9bf74fc10 |
Characterise all 77 chunk types: record byte 0 is a bucket index, not data
Characterised every chunk type -- 77 of them, 1,509 chunks, 34,491 records, all 27 levels. Byte 0 of every record is a small aligned index, not record content. It takes 16 discrete values per chunk type (0x00, 0x10, ... 0xf0) with near-uniform counts WITHIN each type. Uniform weighting is the tell: a running byte offset would weight the buckets unevenly, because records differ in size. type_id recs len byte0 0x007d7ec438 2430 328 0x00:162 0x10:162 0x20:162 ... 0xf0:135 0x00429ec43d 2079 368 0x00:162 0x10:135 0x20:135 ... 0xf0:108 0x00d76cc453 810 108 0x00:27 0x08:27 0x10:27 ... 0xe8:27 <- steps by 8 The alignment step is per type, not global. The leading byte is a multiple of 8 on 34,432 of 34,491 records but a multiple of 16 on only 33,349 -- the 1,142-record difference falls in seven types, four of which step by 8. 59 records in three types break even the 8-byte rule, and that is where the variable-length type lives: 0x00f2ef4275, records 76 to 104 bytes, no single stride to align to. What the bucket indexes is NOT named. It is reported as CHUNK_RECORD_ALIGNMENT and record_is_aligned rather than as a parsed field, because naming it would be a guess while the invariant is solid. The test pins the exact exception set -- 59 records across 3 types -- so a change there fails loudly. A measurement I initially got wrong: the first version of the constant claimed a 16-byte multiple on 34,432 records. 34,432 is the multiple-of-8 count; the multiple-of-16 count is 33,349. Right number, wrong label, and the test caught it. That is the second time this session a correct number carried a wrong label, after the tie four-value retraction. Both times the assertion did, which is the argument for putting these numbers in tests rather than only in prose. Still open: what the bucket indexes, and the remaining record fields. |
||
|
ReaperOfVeriod
|
291153a0d9 |
Find the tie LOD block terminator; packets == 0 means no block at that LOD
parse_tie_block walks to a 0xf6 terminator that every real block carries, on
2177 of 2177 across all 27 levels, followed by zero padding. This is what makes
the blocks parseable: packets[1] bounds one on only 461 of 1864 classes, so the
terminator is the only general way to find one.
The terminator was worth a control, because 0xf6 could have been byte-frequency
coincidence. It occurs in payloads at 0.83x the uniform rate -- slightly LESS
often than chance -- while the last non-zero byte is 0xf6 on 909 of 922 blocks in
the first subset, against 17 of 922 for the same blocks cut at a random point. So
it is a terminator, not a common byte.
packets == 0 means the class has NO block at that LOD: of 3,728 (class, LOD)
slots, 2177 point at a block and 1551 are exactly zero.
Corrected header, over all 2177:
+0x00 count multiple of 16 on all; 42 distinct; never zero
+0x02 zero on all
+0x04 exactly four values: 0x501 (1069), 0xa02 (720), 0xf03 (350),
0x1404 (38)
+0x06 16 distinct, never zero
+0x08 180 distinct, never zero
+0x0a 61 distinct, never zero
+0x0c 125 distinct, never zero, clustered 0x043f..0x0443
+0x0e 11 distinct, zero on 2162 -- the only non-universal field
+0x0c clusters around 1087-1091, close enough to a block length to be worth
stating, but it is NOT the length: the difference against the true payload length
scatters from -12 to -172.
A correction to the previous commit. It reported 3,728 blocks all terminating and
packed_texture taking 842 distinct values, and on that basis retracted the
four-value claim. Both numbers were wrong the same way: the scan did not skip
packets == 0, so it computed class_base + 0 and walked forward from the CLASS
BODY, finding a terminator there by accident. That contaminated 1551 of the 3728.
With the zeros excluded the slot takes exactly four values on all 2177 blocks.
The four-value claim was right; my retraction of it was wrong; and the retraction
was made on a number from the same contaminated set. A retraction is not
automatically better than the claim it retracts -- it needs its own bounds.
Still open: the payload's records. Established this round: blocks are
self-delimiting, absent LODs are marked by a zero packets word, and the header is
fully characterised. Not established: any record boundary inside the payload,
which stays variable-length.
|
||
|
ReaperOfVeriod
|
02342de31b |
Decode the tie LOD block header; establish the payload is variable-length
TieBlockHeader parses the block's own 16 bytes, measured over the 922 blocks where "the next packets offset ends this block" holds, across all 27 levels. A real-disc test asserts every figure. +0x00 count multiple of 16 on 922/922; 16..2096; 72 distinct disc-wide +0x02 zero on 922/922 +0x04 ONE OF FOUR VALUES DISC-WIDE: 0x501, 0xa02, 0xf03, 0x1404 +0x06 15 distinct, 0x0206..0x0512 +0x08 135 distinct, 0x0b05..0x4ee4 +0x0a 45 distinct +0x0c 94 distinct, 0x010e..0x0553 +0x0e zero on 914/922 +0x04 is packed rather than a magnitude: four values across the whole disc with the two bytes moving independently, low 1-4 and high 5/10/15/20 in lockstep. That it is a texture index paired with a scale is a guess; that it is two fields is not. The count is not a record count -- block_size minus the header is divisible by it on 0 of 922, for any record size and any header up to 512 bytes. On the payload, a control was needed and changed the answer. Reading blocks as 3 x f32 triples, 53-64% of reads land within +/-5000 of origin, against 16% for a uniform-random control of identical length. So the content is float-shaped and genuinely not random. But the rate is flat across every start offset and every stride tried, 53-65% throughout. A statistic that cannot separate the right stride from the wrong one cannot identify a stride -- rule 4's exact case. Flatness plus a count that does not tile means the records are variable length. That killed two readings. A VIF1 command walk consumes 0 of 88 blocks exactly, so it is not a VIF1 CPU-format stream. wad::decompress rejects the block at every start offset, so it is not WAD LZ77. Scanning every (start, stride) pair from 0x10 to 0x7f for byte positions with a small distinct-value vocabulary finds none: every position takes all 256 values at every alignment. For float payload that is expected, so the scan found nothing rather than confirming anything -- worth saying, because "no fields found" here is uninformative, not evidence against fields. Still open: what the payload holds. Established: variable-length, float-shaped, non-random, not LZ77, not VIF1. Not established: any record boundary. |
||
|
ReaperOfVeriod
|
881fd48e9a |
Withdraw the 16-byte tie record claim: block size is a multiple of 16, not records
The previous commit inferred a 16-byte record from "922 of 922 consecutive block sizes are a multiple of 16". That inference does not hold, and the claim is withdrawn rather than softened. What survives: - Every one of those 922 blocks opens with a u32 that is positive and a multiple of 4, on 922 of 922, ranging 48 to 1744 with 17 distinct values across level 0's LOD0 blocks. The block opens with a count, not a size and not a pointer. - Reading that u32 as a byte size and chaining to the next block lands exactly on the next block start on 0 of 1383 blocks. Not a size chain. - Assuming [header][count][count uniform records] and solving for the record size gives no answer: no record size tiles the block for any header size up to 512 bytes. Not 16, not 12, not 8. - Reading the block as uniform 16-byte units gives fields spanning their entire legal range -- u32@0 from -2139029504 to 2139029587, u16@8 and u16@10 both 0..65535, and 8969 distinct values in i16@0 on level 0. Uniform units would not produce that. So the 16 is a property of the block size, not of a record. The earlier commit also claimed a record size "on the classes where it can be measured"; those classes are still correctly identified, but what they measure is the block size. The strict-increase disproof from two commits ago stands untouched and is what should have prevented this: the word at +0 was already known not to behave, so treating 16-byte units as records contradicted a measurement sitting in the same file. Divisibility is arithmetic; it is not evidence of a record boundary. Roadmap updated to say block sizes are established and the record layout is not, with the three measurements that rule the 16-byte reading out. |
||
|
ReaperOfVeriod
|
855e640f39 |
Establish the 16-byte tie record on measurable classes; packets[0] is a constant
The previous commit reported "3580 non-empty LOD blocks compute as divisible by 16" and then said the figure should not be trusted, because it came from the "next packets offset ends this block" rule. That rule is unsound: packets is strictly increasing on only 461 of 1864 classes. On the other 1403 the two offsets are unordered and cannot both be block starts into one ascending region. Restricting to the 461 classes where the rule does hold, 922 of 922 consecutive block sizes are a multiple of 16. That is the evidence for a 16-byte record, and it is now scoped to the subset where it can be checked rather than claimed over all 1864. The other 1403 are unmeasured, not measured-and-different. packets[0] is a constant, not an offset: it reads 256 on 1864 of 1864 classes, while packets[1] and packets[2] vary widely and land at different addresses in core_data on every class. Read as a block start, 256 points just past the 0x48-byte header -- which is how it was being used, and exactly how a field gets mistaken for a pointer. It is now TIE_PACKETS0 and nothing depends on it. Still not decoded: what the 16 bytes hold. i16[0] at +4 takes 8969 distinct values on level 0 and is not the two-valued field a 12-unit sample suggested; the word at +0 is not a run offset. The test pins all four numbers -- packets[0] constant, the 922/922 divisibility, the 461/1864 ordering, and the 0/1864 monotonic disproof -- and asserts that the ordering is *not* universal, so a future change there fails loudly instead of quietly invalidating the record size. |
||
|
ReaperOfVeriod
|
60e3f129d8 |
Resolve both tie indirections on all 1,864 classes, and disprove a sample artifact
resolve_tie_class walks the chain, which is two steps and both are easy to get
wrong:
1. The tie class table lives in core_index.bin, not core_data.bin. On level 0
the tables chain as moby at 192, 238 x 32 bytes later tie at 7808, 74 x 32
later shrub at 10176, which closes. Reading the same table out of
core_data.bin produces plausible garbage rather than an error.
2. ClassEntry::offset_in_asset_wad addresses core_data.bin, despite the field
name and despite core_data.wad sitting alongside it.
Verified: all 1864 tie classes resolve, o_class agreeing on every one, and all
5592 LOD block offsets land inside core_data.
Disproved, and this is the part worth reading: the word at +0 of a 16-byte unit
inside an LOD block. On the first twelve units of level 0's class 0 it rises
strictly -- 288, 1520, 2848, 3968, 5088, 6096, 7312, 8528, 9792, 10976, 12016,
13072 -- which reads exactly like a linked list, and the accompanying i16s looked
quantised, two distinct values in i16[0] and three in i16[1].
Across the population it is strictly increasing on 0 of 1864 blocks, and i16[0]
takes 8969 distinct values on level 0 alone rather than two. The sample was
twelve units out of 1,724,334. Both the monotonic reading and the quantisation
were artifacts of that sample.
That is the pooled-claim error AGENTS.md rule 3a exists to prevent, committed
almost verbatim: the first twelve units looked clean, and I was one step from
writing "u32[0] is a linked list of runs" into a format spec. The test now
asserts the disproof, so it cannot creep back.
Not established: the block's record size. 3580 non-empty LOD blocks compute as
divisible by 16, but some packets offsets run backwards, so the "next offset
delimits this block" end is wrong for them and that figure should not be trusted.
The tie layout stays partial: indirections and header done, records not.
|
||
|
ReaperOfVeriod
|
e5d267a810 |
Reconcile roadmap.md with the parsers that actually exist
Findings were going to docs/plan/level-assets.md and docs/formats/*.md while roadmap.md kept listing work as open that has been done for some time. The worst case was occlusion appearing twice: open at one line with "2 to 25 sectors, small enough to read whole", and done further down with 412521 cells over 140939 masks. The open line is deleted. Also corrected: - tfrag was listed flatly open while six tfrag entries beneath it were checked. Now partial: format, topology, textures, LOD, normals and colour are done; morph targets and strip/chain bookkeeping are not. - sky was lumped in with volumes as open. Sky is done -- 149 sub-objects across 27 levels -- so it has its own entry, keeping the word at +0x0c documented as not a count, since it disagrees with the table on most levels and reads 196617 on level 2 against 4 sub-objects. - level_overlay was open. It now has a parser and a 27/27 test, so it is partial rather than done: header, constant block and level data parse, +0x10..+0x1220 is identical on every level, and the word at +0x0c falls below the file length on 17 levels and above on 10. The header words are still unnamed and the Phase 3 seven-section parser is untouched. - sound_bank was flatly open. It is partial: container framing parses and bounds are verified on 27/27, but the SBlk record fields are not decoded and DAW shows no stride, consistent with ADPCM. - tie now records what is actually known, including the two indirection chains that have not been walked: class table entries point into core_data.wad, and the header's packets[0..3] resolve against the lump base. Level 0 has 74 tie, 238 moby and 7 shrub classes. - collision now separates the header, which is decoded, from the cell layout, which is not -- including the 17-entry table at +5932 against a 75-cell grid and the disproved 64-byte RAM stride. - moby_anim records the negative result: per-chunk-type record counts are level-independent, so nothing in the chunk set is keyed by moby class. The "Where we are" section now states plainly what has parsers and that what remains open is naming rather than locating, and names the five types with no decoded fields. Reading that list is how you would have known to ask. Phase 2 remains in progress. Volumes and moby_anim have no located structure and SBlk records, tie vertices and collision cells have located structure without decoded fields. |
||
|
ReaperOfVeriod
|
d8b8ccc6d4 |
Follow the gameplay.bin pointers: they address position and direction arrays
Where the pointers go. Following them shows what gameplay.bin indexes. The leading table is 39 words, all of which land inside the file, and three clusters dominate: near 150000, near 800000 and near 1050000. The region near 797808 is position data -- 31 consecutive plausible floats that group cleanly into xyz triples: 246.948 201.015 73.867 | 88.240 186.000 302.598 | 107.712 12.723 183.125 353.591 126.365 19.962 | 182.068 374.001 122.567 | 116.961 265.001 437.866 and the region near 797456 is direction data -- unit-length vectors: 0.969 0.945 0.733 | -0.120 0.433 0.250 | -0.866 0.000 0.267 So the file's leading words are a table of pointers into float arrays, not a flat list of small integers. The earlier reading of it as "u32 pointers holding nothing" was too quick: it is a pointer table into data that is present. Volumes remain unlocated. The position arrays are the right shape to bound a volume and nothing yet shows the volume records themselves. |
||
|
ReaperOfVeriod
|
b17ec17127 |
Rule out the moby_anim instance-table reading of the chunk types
moby_anim was the one post-shrub chunk type with a concrete reason to exist: an instance table would have to scale with the level's moby class count, so comparing per-type record counts against that count was the obvious test. It fails, and the way it fails is itself the result: per-type record counts are level-independent. The same types report the same counts on every level tried: type_id level0 level1 level2 level3 0x007d7ec438 90 90 90 90 0x00d3d8c4a9 90 90 90 90 0x00429ec43d 77 77 77 77 0x0053c8c4aa 77 77 77 77 0x007626c4a4 50 50 50 50 0x00fb88c462 47 47 47 47 Across all 27 levels there are 77 distinct chunk types, the largest at 2430 records total -- exactly 90 x 27. What that rules out: nothing in this chunk set is a per-moby-class instance table, because those counts would vary by level the way the moby class counts do. So whatever holds moby animation data is either keyed by something other than the class count, or it is not among these chunk types. Also visible: the types pair up. 0x007d7ec438 and 0x00d3d8c4a9 both report 90 on every level; 0x00429ec43d and 0x0053c8c4aa both report 77. Two chunk types carrying identical, constant record counts on every level is a signature of two views of the same fixed table rather than two independent tables that happen to agree. Still open: the pairings are described but not explained, and no type has been named. The Moby class-ID enrichment from earlier stands separately -- 9493 hits against 246.1 expected is real, and the class arrays are contiguous per type, but that identifies where classes are referenced, not which chunk is moby_anim. |
||
|
ReaperOfVeriod
|
02ced5ee83 |
Parse the per-level overlay: header, constant block, level data
level_overlay was the least-characterised file in the level set -- no parser, no located structure. It now has gc_formats::overlay and a real-disc test over all 27 levels: overlay: 27 levels; field_00 1735808, field_08 1; field_04 37540..=45308, field_0c 2763912..=3027336 (17 below the file length, 10 above); +0x10..0x1220 identical on all The header is four words. +0x0c is not the file length: it was recorded as one on the strength of the first levels measured, where it sat near the file end, but across all 27 it falls on both sides -- below on 17, above on 10 -- so it is an ordinary level-specific word. The test asserts this, being the claim most worth pinning. The common prefix is 4 bytes, not the whole header: +0x00 is constant at 1735808 while +0x04 and +0x0c vary. Everything from +0x10 to +0x1220 -- 4624 bytes -- is byte-identical on all 27 levels. Still not named: what any of the four header words means, and how the level-specific tail is structured. The module reports them as fields. Also records in docs/plan/pcsx2-verification.md that the 64-byte RAM stride from the previous commit is disproved. A 415 KB dump of level 1 finds the longest such run at 27 records, not 384, and those are identity matrices -- 27 of them, one per level. It is a per-level transform table, not the collision grid, and the 20.0 reported there was a coincidence. |
||
|
ReaperOfVeriod
|
bd012efe71 |
Find a 64-byte collision record stride in RAM, and record a savestate search pitfall
Searching the savestates for the collision grid dimensions as a u32 pair -- level
0's 11 x 33 and level 1's 24 x 16 -- puts level 1's at several RAM addresses and
level 0's nowhere. So the grid dimensions are not held in RAM as adjacent u32s,
and the collision payload is reshaped on load, as the shrub packet tables are.
A nearby region of level 1's RAM does show structure. At 0x12e048c and
0x12e04cc, 64 bytes apart:
28 01 00 00 | 92 ff 00 00 | 04 00 a0 41 | ...
296 -110 20.0f
Searching for that shared 12-byte tag finds a consecutive run at a 64-byte
stride in slot 2:
0x12df390
0x12df3d0 +0x40
0x12df410 +0x40
plus scattered singles elsewhere. Slot 1 has the tag at four addresses not on a
64-byte stride, so the two levels lay it out differently.
What this is: a 64-byte record carrying two small signed values and a float
reading exactly 20.0 -- the shape of a per-cell entry with coordinates and a
radius or half-extent, which is what the collision grid would hold.
What this is not: it is not decoded. Three consecutive records is not enough to
establish a stride, the tag matches only a few records rather than a whole grid,
and 20.0 may be coincidence. Level 1's grid is 384 cells; nothing here has been
shown to count to 384.
Method note: savestates are compressed on disk, so a byte scan of the file finds
nothing. All RAM search must go through gc-trace find --state, which decompresses
first -- scanning the file directly reports zero hits for a sequence that
demonstrably exists in memory.
|
||
|
ReaperOfVeriod
|
d3e1475f64 |
Characterise gameplay.bin as a flat pointer table, and record volumes as unlocated
Range measured: the first 40 u32 and the whole file, on levels 0 and 1 --
1872256 and 2006400 bytes.
Result: the file is a flat array of u32, opening with a small count then a run of
ascending offsets, terminating on a zero:
level 0 336 797472 798832 799136 464 150672 150688 314800 479920 ...
... 1847600 1845456 797808 1829056 160 1849088 1249808 1847616 0
level 1 496 799904 800128 800432 624 151152 151168 315680 481456 ...
... 1984496 1982400 800112 1966000 160 1988160 1303456 1984512 0
Both levels carry the same shape. The sentinel 160 appears at u32 index 34 on
both and the zero terminator at index 38 on both, with counts 336 against 496
and 464 against 624 at the head. The leading ~38 words are a structured header
whose layout is level-independent apart from its sizes.
The offsets cluster into groups -- near 800000, 150000, 1050000, 1800000 --
which is what a table of pointers into several buffers looks like.
Not decoded: what the pointers point into, what 160 is, and why the zero
terminator sits at the same index on both levels. No gc_formats module exists for
this yet; it is characterised, not parsed.
On volumes: nothing resembling volumes was found here. gameplay.bin is u32
pointers throughout and holds no float data in its first 64 words read as f32 --
59 of 64 are denormals, i.e. small integers reinterpreted. Volumes remain
unlocated on the disc; nothing in the regions examined so far presents itself as
one.
|
||
|
ReaperOfVeriod
|
575d1a5bdd |
Correct the overlay strings from a table to a pool
The previous commit described a "debug string table on 16-byte alignment". It is not a table. Dumping the 32 bytes before each run shows most are the tail of the previous string: +3104 "host0:r:/rc2/shock/wavetable.bin" 32 chars +3120 "ck/wavetable.bin" its own tail +3152 "/i5/shock/wavetable1.bin" +3168 "ble1.bin" its own tail and the 32-byte prefixes are 327 distinct out of 332 -- essentially all different, because they are just the preceding string's bytes. There is no per-record header here. It is one contiguous pool of debug text, and the 16-byte alignment is a property of how the pool was laid out, which makes continuations land on boundaries and read as separate entries. The count of 332 "strings" was therefore inflated by the same effect: it counts continuations as entries. The substance is unchanged and still useful -- the pool carries the game's own debug text, including "Camera_CollPrimTest WARNING! - grid out of bounds! (RAR)", which is direct evidence for reading the collision payload's two leading u16 as grid dimensions, and the wavetable.bin paths, which bear on the sound bank framing. Only the framing was wrong. The pool is located and counted but not parsed, and there is no gc_formats module for it. The +0x18/+0x20 names are section names but their structure is not established, and it is not shown that they index anything. |
||
|
ReaperOfVeriod
|
a5a33b9bc0 |
Locate level_overlay's sections and a 16-aligned debug string table
level_overlay was on the open list with no parser and no located structure. It now has a header naming two sections and a string table. Range measured: overlay.bin on levels 0, 1 and 2 -- 2751268, 2763668 and 2928424 bytes. Result 1: the file opens with section names. +000: 80 7c 1a 00 | dc 99 00 00 | 01 00 00 00 | 88 3b 2a 00 +010: 00 00 00 00 | 00 00 00 00 | "crap\n" | 00 00 00 00 00 +020: "system misc" 00 00 00 00 00 00 00 00 00 +030: 20 80 ff 40 | 00 60 cf 00 | 64 00 | 46 00 | 19 00 00 00 u32@+0x00 is 1735808 on all three levels, +0x08 is 1, and the bytes at +0x18 and +0x20 spell "crap" and "system misc". Result 2: a debug string table on 16-byte alignment. 332 such strings on level 0, 362 on level 1, 351 on level 2, first at +0x20 and last between 1.3 MB and 2.5 MB. They are the game's own debug text: host0:r:/rc2/shock/wavetable.bin host0:z:/i5/shock/wavetable1.bin unloaded sound bank %#010x Camera_CollPrimTest WARNING! - grid out of bounds! (RAR) render setup sky draw pre effects pre 2 effects Camera_CollPrimTest WARNING! - grid out of bounds! (RAR) is direct evidence for the collision reading. It is the game's own string, naming a grid and a collision test, and it is why the two leading counts in the collision payload are worth continuing to treat as grid dimensions despite not sizing the payload. The wavetable.bin paths also confirm the sound bank is loaded from named files, which bears on the SBlk/DAW framing. Not done: the string table is located and counted but not parsed -- the records preceding each string have not been identified, so there is no gc_formats module for it yet. The +0x18/+0x20 names are section names but their structure is not established, and it is not yet shown that they index anything. |
||
|
ReaperOfVeriod
|
e58ab0e815 |
Correct the collision table's extent: 17 entries, deltas rise then fall
+5932 is the table's start, not a point inside it: walking backwards, no predecessor satisfies "ascending with a delta under 4096". The run is 17 entries, not 19 -- the earlier 19 counted two entries past the jump. The deltas are 204 212 216 224 224 228 228 228 224 220 216 200 96 96 100 96 then a jump of 41484 to 190508. The deltas rise and then fall -- a symmetric profile, not a monotone one. That weakens the prefix-sum reading: a cumulative offset table has no reason to come back down, whereas cell contents that grow and shrink as the table walks across a level would. 17 against a 75-cell grid, and 75 - 17 = 58. No arithmetic relating the two has been found. The earlier guess that 19 related to 3x25 did not survive re-measurement and is not carried forward. Still open. 17 entries may not correspond to the grid at all -- the table may cover something else, or a subset of cells. Resolving it needs either the same offset found in other levels at a position the leading counts predict, or a reason for 17. Neither exists yet, and this is where the measurement trail goes cold on collision. |
||
|
ReaperOfVeriod
|
b66e77855c |
Locate a per-cell offset table in collision, on level 4
Range measured: longest strictly-ascending u32 run (delta under 65536) in each
level's collision payload, levels 0 to 5.
Level 4 is the clean case. Grid 3x25, payload 2660608, and at payload +5932 a
run of 19 entries whose values are all inside the payload, starting at 146012:
deltas 204, 212, 216, 224, 224, 228, 228
The deltas grow monotonically, which is the signature of a prefix sum over
records of increasing size -- what a per-cell offset table looks like when cell
contents vary.
The others are not:
level 0 11x33 +21232 17 8193, 24575, 8192, 8192 multiples of 8192,
coincidental on
dense data
level 1 24x16 +41452 29 32771, 57339, 24575 wild, no trend
level 2 1x53 +13752 49 24, 64, 396, 104 in-region but
irregular
level 3 19x31 +667152 18 769, 257, 257, 257 values ~5.2e8, far
outside the payload
level 4 3x25 +5932 19 204, 212, 216, 224 growing, in-region
Only level 4's run has both properties a prefix sum needs: values inside the
payload, and deltas that grow.
What this does not give: level 4's run is 19 entries against 75 grid cells, so
it is not one offset per cell -- either most cells share an offset, are empty,
or the table covers something other than the grid. Nothing measured picks between
those.
Next to check: whether 19 relates to 3x25 by any arithmetic, and whether the
same offset at +5932 exists in other levels at a position given by their leading
counts. Neither has been tried.
|
||
|
ReaperOfVeriod
|
2569cc06ab |
Decode the collision payload's leading two counts
Range measured: the first four bytes of the payload on all 19 levels that carry a non-empty one. Result: two u16 at payload +0 and +2, both non-zero on all 19, ranging 1..74 and 13..112, products 53 to 3256. level 0 11 x 33 = 363 payload 903248 level 1 24 x 16 = 384 payload 2800496 level 2 1 x 53 = 53 payload 2315296 level 7 40 x 35 = 1400 payload 4049232 level 9 74 x 44 = 3256 payload 2518352 level 13 2 x 64 = 128 payload 3333520 level 26 27 x 112 = 3024 payload 652176 Reported as two counts, not as a named grid. They have the shape of collision grid dimensions -- bounded, both non-zero, products in the hundreds to thousands -- but nothing measured ties the product to the payload size or to any per-cell stride. Level 2's 1 x 53 giving a 2.3 MB payload and level 26's 27 x 112 giving 652 KB shows the product does not size the payload, so either cells vary wildly in size or these are not grid dimensions. Naming them width and depth would be a reading, not a measurement. After the two counts comes a run of u16 that stays below 8192 for 30 to 58 entries depending on level, always an even count, and is not ascending. That is where the per-cell table most likely starts, and it is the next thing to resolve. Exposed as CollisionHeader::first and ::second. They live in the payload rather than the header, so CollisionHeader::region fills them in and the bare CollisionHeader::parse leaves them zero -- documented at the call site so the difference is not mistaken for an absence. |
||
|
ReaperOfVeriod
|
72bbd5da38 |
Rule out GS packet framing in tie, with a control that nearly fooled me
Range measured: level 0, the 73 tie span bodies (bytes +0x40 onward), 2585856 bytes pooled. Per-position distinct counts, an autocorrelation scan against a shuffled control, and a GS GIF tag test. Result 1: the bodies are dense. Every one of the 16 slot positions takes 245 to 256 distinct values, byte 15 zero on 81.3% and byte 7 zero on 78.8%. The best stride peak is 48 at +0.097, barely above 16 at +0.095 -- no period. Result 2: no GS GIF framing at 16-byte alignment. Testing whether byte 15's low three bits fall in 0..4, the GS tag range, gives 97.0%. That looked like a strong hit until the control was done properly. A uniform shuffle gives 62.6%, exactly 5/8, what a random byte predicts -- so the naive comparison reads as a 34-point excess. The distribution-matched null is 97.0%, identical to the observation. The whole excess was that byte 15 is 81% zero and (0 & 7) <= 4 is always true. Recomputed per alignment the null tracks the observation everywhere: align 0 97.0% align 4 75.7% align 8 96.7% align 2 75.5% align 12 89.3% This is the control rule earning its place a second time. A uniform shuffle is not a matched control when the data is skewed, and AGENTS.md already says so; the near-miss was reading 62.6% as a baseline without checking what the data's own distribution predicts. The tie bytes concentrate on 0, 1, 2, 3 and 63 -- suggestive in isolation -- and still carry no framing. What this rules out: the tie bodies are not GS GIF packet streams at 16-byte alignment and carry no measurable periodicity. With every slot position taking ~256 distinct values they are dense data -- fat vertices, as the roadmap expects -- and locating the vertex layout needs a reference for this game's instanced geometry format, not another stride scan. |
||
|
ReaperOfVeriod
|
86ff2e6052 |
Measure tie spans: a constant first word and three quantized floats
Range measured: all 1864 tie spans across 27 levels, read at each class entry's
geometry offset.
Method: read +0x00, +0x0c and the three f32 at +0x10, +0x14, +0x18, and count
distinct values per position.
Result:
u32@+0x00 takes exactly one value across all 1864 spans: 0x00000100, 256
+0x0c takes 83 distinct values, so it is not constant
the three floats take 103 distinct triples built from 42 distinct values,
and those values are round -- 4, 8, 9, 10, 12, 15, 16, 18, 20, 24, 25, 30
with only 10 of the 42 exact powers of two
Two level 0 spans open:
+000 00000100 00005070 000065c0 07050612
+010 41400000 41c80000 42200000 00009460
+000 00000100 00007ca0 0000a040 0605091e
+010 41400000 41c80000 42200000 0000ddf0
This rules out the bounding-box reading. A box would take continuous values;
these come from a 42-value set of round numbers, so they are authored or
quantised quantities -- a grid, a size class, or LOD thresholds -- and nothing
measured separates those.
Correction: these floats were previously read as a bounding-box minimum, from
level 0's first two spans happening to share (12.0, 25.0, 40.0). That triple
holds on 266 of 1864 spans, not all of them, and the value set is quantised.
Not decoded: what the three floats are, what 0x100 at +0x00 counts, and what
the 83 values at +0x0c are.
|
||
|
ReaperOfVeriod
|
32a24574be |
Identify data.bin as the level file descriptor, not geometry
data.bin is the largest per-level file -- 13.3 MB on level 0, 4.2 to 21.9 MB across the disc -- and had been characterised only as "4-byte structured". Range measured: the first 64 u32 of data.bin on levels 0, 1, 2 and 5, compared against the actual byte lengths of the other extracted level files. Method: read the first 64 u32 and look for any equal to the size of core_index.bin, gs_ram.bin, overlay.bin, gameplay.bin, sound_bank.bin or core_data.bin. Result: three exact matches, at the same indices on every level. u32 index 1 overlay.bin 2751268 2763668 2928424 2863676 u32 index 3 core_index.bin 31600 34016 33808 23536 u32 index 5 gs_ram.bin 518656 569344 513024 434176 Each is the exact byte length of the named file and each lands on the same index on all four levels. Three coincidences at fixed positions is not a coincidence. So data.bin is a descriptor for the level's other files, not geometry in its own right. The roadmap's "the large one: 4.2 to 21.9 MB of geometry per level" entry was wrong and is corrected. The intervening indices 0, 2 and 4 hold values that are not file sizes and their meaning is not established; they are plausibly offsets, but no cumulative-sum or offset+size relation has been checked yet. What follows the descriptor is still unknown, and the file is large enough that the descriptor is unlikely to be all of it. |
||
|
ReaperOfVeriod
|
5610d173b0 |
Narrow the collision header to two fields plus a third used by five levels
Range measured: all 27 levels that carry collision, reading the 0x40-byte
header each.
Method: read +0x00, +0x04 and +0x08, then check +0x08..+0x40 for zero.
Result:
u32@+0x00 is 0x40 on 27 of 27 -- the header is self-sizing
u32@+0x04 is the payload size, 20 distinct values across the disc
bytes +0x08..+0x40 zero on 22 of 27
Five levels carry a large non-zero word at +0x08:
level 7 1248048 level 12 2541616
level 9 1766832 level 13 3376592
level 14 4060640
So the header is not two fields and 56 bytes of padding: there is a third field
most levels leave at zero and five use. Whether it is a size, an offset or a
second block's length is not established, and nothing measured here separates
those readings. Exposed as CollisionHeader::field_08.
Correction: the collision header was previously described as self-describing
across its full 0x40, which is true for 22 levels and overstates the case for
the other five.
Still not decoded: the payload. It contains ascending u32 runs on levels 0 and
2 -- level 0 at +280 with deltas 12, 12, 12 then a jump, level 2 at +13868 with
deltas 204, 204, 200, 196 -- but irregular deltas fit neither a flat table nor a
simple stride, and level 1's apparent run is a byte-shift false positive.
Nothing about the payload is asserted.
|
||
|
ReaperOfVeriod
|
51348b214d |
Report the sky reserved-word count in the test output
20 of 149 sky sub-objects carry a non-zero word where the other 129 are zero, so the reserved area is not uniformly padding. Reporting it rather than asserting it, and clearing three clippy warnings from the unused counter. |
||
|
ReaperOfVeriod
|
681e7e62ef |
Parse the sky block: 149 sub-objects with a bounding sphere, and correct the count field
Range measured: all 27 levels, each sky region from the core index's sky offset to the next region. Method: read the 32-bit offset table at +0x20, bounded by staying inside the region, and parse each sub-object it names. Result: 149 sub-objects across 27 levels. Each opens with a count, then a bounding sphere of four floats at +0x10, then a texture offset at +0x20 and dimensions at +0x24/+0x26: +000: 26 00 00 00 | 00 00 00 00 00 00 00 00 00 00 00 00 +010: e0 8d b2 c1 | 9f d6 f6 3d | ef fc fb be | 1c 61 4e 41 +020: 90 99 05 00 | 1d 00 | 1d 00 | 00 00 e8 00 ... Count 1 to 38, width 3 to 81, radius 1.95 to 14.56, and the radius is positive on 149 of 149 -- the same sphere shape and the same reasoning as the shrub class header's. Correction to an earlier claim: the word at +0x0c is not the sub-object count. It reads 5 on level 0 where the table holds 6 entries, 3 on level 1 against 4, 7 on level 3 against 8, and 196617 on level 2 against 4. Across the disc offsets.len() == count+1 holds on 10 of 27 levels. The field is reported as SkyHeader::field_0c and nothing depends on it; the offset table enumerates the sub-objects and is self-limiting. That earlier claim came from level 0 alone, where 5 against 6 looked like a consistent off-by-one. It is not an off-by-one -- it is a field that happens to be small on most levels and is not a count at all. gc_formats::sky parses the header and the sub-objects. The gated test asserts the sphere is finite, the radius positive where positive at all, the dimensions non-zero, and the sub-object list non-empty on every level. |
||
|
ReaperOfVeriod
|
bbce492c91 |
Establish that the loader produces the RAM packet forms, not the disc
Everything previously reported about five-block sub-lists and fixed-point
payloads was read out of the running game. Checking the same packets on disc
settles what the disc actually stores.
Range measured: level 0, all 29 packets named by the +0x40 tables of the 6
bounded spans, read at their span-relative offsets; 1636 whole 16-byte records
after the 16-byte header.
Neither RAM form exists on disc:
disc RAM
00 00 00 30 per exactly 1 exactly 5
sub-list packet
blocks at 0/32/64 0 of 10 11 of 11
fixed-point byte3 0 of 19 100%
== 0x80
The 29 packets split 10 sub-list / 19 fixed-point by whether they contain
00 00 00 30 at all, and not one of the ten has the block structure, and not one
of the nineteen has the fixed-point marker.
The disc body is a compact 16-byte record stream:
+000: 04 04 00 01 | 00 00 00 00 | 00 00 00 05 | 00 80 0e 6c
+010: 02 00 00 00 | 05 00 00 00 | 32 00 00 00 | 0e 00 00 00
+020: 09 00 00 00 | 00 40 3e 30 | 12 04 00 00 | 05 00 00 00
+030: 09 00 00 00 | 00 40 3e 30 | 12 04 00 00 | 21 00 00 00
Small u32 fields and a repeating 00 40 3e 30 12 04 00 00. Per position over all
1636 records the same two outliers hold: byte 7 takes 4 distinct values and byte
15 takes 2, against 101 to 153 everywhere else.
Byte 7 is per-record, not per-packet: the first record of all 29 packets has
byte 7 = 0x00. The four-value vocabulary varies within a packet body rather than
tagging the packet, which is why grouping packets by it never separated the two
kinds.
Consequence for Phase 2: the payload is encoded on disc and expanded at load, so
a disc parser can bound it and check its signature but cannot read it as either
RAM form. Decoding it means reversing the expansion, not re-describing what the
loader produced. That is the honest boundary of what is known.
|
||
|
ReaperOfVeriod
|
5ade83cf0a |
Parse the shrub packet header, a fixed signature on all 2056 disc packets
The packets found by the +0x40 table have a header the whole disc agrees on. Range measured: every packet on the disc -- 2056 of them, across the 334 successor-bounded spans -- read at each packet's span-relative offset. bytes 0-3 04 04 00 01 2056 / 2056 bytes 4-7 zero 2056 / 2056 byte 8 zero 2056 / 2056 byte 11 0x05 2056 / 2056 size % 16 zero 2056 / 2056 Zero exceptions on any of the five. And the loader fills the rest in. Read back out of the running game the same records carry byte 8 as a count (0x0b to 0x29 observed), byte 11 as one of four tag values, and bytes 12-15 as an absolute EE RAM pointer where the disc has a span-relative offset. Bytes 8-15 are rewritten wholesale on load. That settles what the disc header is for. It carries no count of its own, and the count the RAM copy has is the +0x40 table's count arriving by another route. The disc header is a signature; the RAM header is populated state. ShrubPacketHeader::parse and Shrub::packet_headers() expose this. parse returns None unless all the invariants hold, rather than a default, so a packet that does not match is reported as not-understood instead of half-interpreted. The gated test asserts all four header invariants and the size alignment on every packet of every bounded span. This is the first piece of the mesh packet format represented in gc-formats; everything known about the packet payloads so far is RAM measurement, since the disc form of bytes 8-15 is rewritten at load. |
||
|
ReaperOfVeriod
|
af0b1aa949 |
Repeat the packet measurements per level, correcting two over-general claims
The five-block result was measured on level 0 only. Rule 3a asks for the same measurement per partition, so it was repeated on level 1's savestate before being written up as a property. Range measured: savestate slot 2 (level 1), RAM 0x22a000..0x28f52f, 415024 bytes, then six of its pointer-bearing records read to count*16. The six sampled pointers split 3/3 between the two payload kinds: 00949430 41 00 00 00 30 sub-list 009496c0 13 69 b0 bf 80 fixed-point 00957970 36 00 00 00 30 sub-list 00957bb0 10 58 7e 8d 80 fixed-point 0095b740 12 37 49 41 80 fixed-point 0095d6d0 57 00 00 00 30 sub-list Taking all six as sub-list gives 3/6 with five blocks. Taking only the three that actually open 00 00 00 30 gives 3/3, with the first three blocks at 0, 32, 64 in all three. The three fixed-point payloads have zero such blocks, as expected. The earlier 3/6 was my sampling, not a counterexample -- the six pointers were never all the same kind of object. Combined: 11 of 11 sub-list payloads across both levels have exactly five blocks, the first three at offsets 0, 32, 64. The fixed-point payloads have an exact signature: byte 3 of every 4-byte group is 0x80 in 100% of cases -- 52/52, 40/40, 48/48 over the first 512 bytes of the three level 1 samples, and the same across all nine level 0 payloads. That separates the two payload kinds without the record tag, which does not select them. Correction: level 1's record tags are not level 0's. The same scan finds 18 04 04 00 01 headers against 19, plus 6 02 01 00 01 against 10, and byte 11 is no longer constant: 0x30 on 10, 0x00 on 5, 0x20 on 2, 0xbc on 1, where level 0 had 0x30 on all 19. "Byte 11 is 0x30, constant" was a level-0 partition result written up as universal; across both levels it takes four values. Fixed-point payload entropy over all nine level 0 payloads is 5.51 to 5.89 bits per byte against 8.0 for uniform, and the mean of the s16 reading varies per payload from -82 to -907 -- structured, and consistent with vertex positions relative to a per-payload origin. Not established: what the five blocks are, and how the fixed-point data decodes to vertices. Its entropy is far below random, so it is compressed or permuted rather than stored plainly, and the permutation table seen once in 0x009376f0 remains unexplained. |
||
|
ReaperOfVeriod
|
186a700f32 |
Establish the sub-list payload as a fixed five-block shape
All eight sub-list payloads from level 0 were read to their exact extent (count * 16) and compared block by block. Range measured: savestate slot 1, the packets at 0x009376f0, 0x00939c50, 0x0093cc70, 0x00998880, 0x009c19c0, 0x009cf370, 0x009d2da0 and 0x009f5aa0, counts 21 to 41, each truncated to exactly count*16 bytes. Every payload contains exactly five blocks tagged 00 00 00 30, and the first three are always at offsets 0, 32 and 64: 009376f0 22 0, 32, 64, 184, 304 00939c50 36 0, 32, 64, 268, 400 0093cc70 23 0, 32, 64, 200, 288 00998880 36 0, 32, 64, 272, 404 009c19c0 25 0, 32, 64, 196, 300 009cf370 24 0, 32, 64, 204, 316 009d2da0 41 0, 32, 64, 308, 448 009f5aa0 21 0, 32, 64, 176, 280 Blocks 4 and 5 move with the payload; the first three do not. Eight payloads, eight different block-4/5 positions, and 0/32/64 in every case. Per-block fields constant across all eight: block 0 byte 4 varies 0x78..0xcd block 1 byte 4 = 0 in 8/8, trailer 00000000 in 8/8 block 2 byte 4 varies independently, bytes 16-20 zero in 8/8 block 3 byte 4 EQUALS BLOCK 0 in 8/8 block 4 byte 4 = 0 in 8/8 Block 2's word is unrelated to block 0's in every case, so the equality is not every block reading one shared field. Correction: the "byte permutation table" reported in the previous commit does not generalise. It appears in 0x009376f0 at offset 212, but a scan across all eight payloads finds no such run outside that one. It was one payload's block content, not a format feature. The five-block shape, the fixed 0/32/64 offsets and the block-0/block-3 equality are what survive eight-way comparison. Not established: what the five blocks are, why blocks 0 and 3 share a word, and what moves blocks 4 and 5 -- their offsets are not monotone in count (21 gives 280, 36 gives 400 and 404), so something inside the earlier blocks sets them. |
||
|
ReaperOfVeriod
|
3fa64b7003 |
Read a full sub-list payload: tagged blocks and a byte permutation table
With the extent known from the previous entry (count * 16), a whole sub-list
payload can be read. This is the packet at 0x009376f0, count 22, 352 bytes.
Range measured: savestate slot 1, 0x009376f0..0x0093784f.
The payload is a sequence of blocks, each opening 00 00 00 30, carrying a
repeating word and a short trailer:
00 00 00 30 | cd 00 00 00 cd 00 00 00 cd 00 00 00 cd 00 00 00
01 00 00 05 | 22 c1 01 6e | 00 00 00 00
00 00 00 30 | 00 00 00 45 00 00 00 45 1d 00 00 00 1d 00 00 00 1f
81 01 6d 00 | 00 00 00 00 | 00 00 a4 00
00 00 00 30 | bc c0 03 00 23 17 03 00 e5 dd 00 00 00 00 00 00
02 01 00 01 | c1 80 01 69 | 00 00 00 00
The repeating word differs per block (0xcd, 0x45, 0x03) and the trailer's
second word varies in its high bytes (22 c1 01 6e, 81 01 6d 00, c1 80 01 69).
Every block ends 6e, 69 or 6d at byte 23 -- the same 6e the 02 01 00 01 records
carried at bytes 8-9.
The payload contains a byte permutation table at 0x009377c4:
00 01 02 03 3e 3f 53 54 40 41 42 43 4e 4f 50 51 04 05 06 07 08 ...
Ascending byte values with a handful (3e 3f, 53 54, 40 41 42 43, 4e 4f 50 51)
lifted out of order ahead of the sequential run. Read as a table it maps each
of 0x00..0x39 to a slot -- the shape of a code-length or symbol-ordering table,
and the first thing in this format that is self-evidently a table rather than
data to be interpreted.
Not said: whether it is a Huffman-style code, an index reorder for the vertex
data that follows, or a palette permutation. The sequence continues to 39 then
23 c1 01 6e, after which the next block opens.
Corroboration: the fixed-point payload 0x00937850 begins exactly where this one
ends. The 22-record identity made the boundary exact and the next packet starts
on a payload boundary with nothing in between, so the two payload kinds sit
adjacent in RAM -- consistent with them being two records of the same blob.
|
||
|
ReaperOfVeriod
|
f1cf6cd164 |
Show the byte-8 count sizes sub-list payloads at exactly 16 bytes each
The previous entry left one test: compare the count at byte 8 against the payload extent. Pointer spacing gives the extent, since the packets lie in ascending order in EE RAM. Range measured: savestate slot 1, the 17 packets in RAM 0x22a000..0x24037f, each with its record address, byte-8 count and pointer, sorted by pointer. Result: eight packets sit exactly count*16 bytes after their predecessor -- a ratio of 16.00 in every one, no rounding. 0x22a6e8 count 22 ptr 009376f0 gap 352 16.00 0x22a728 count 36 ptr 00939c50 gap 576 16.00 0x22a768 count 23 ptr 0093cc70 gap 368 16.00 0x22a7a8 count 36 ptr 00998880 gap 576 16.00 0x22a7e8 count 25 ptr 009c19c0 gap 400 16.00 0x22a828 count 24 ptr 009cf370 gap 384 16.00 0x22a868 count 41 ptr 009d2da0 gap 656 16.00 0x22a8a8 count 21 ptr 009f5aa0 gap 336 16.00 Eight exact hits at a ratio that must be an integer is not luck. The count at byte 8 is the record count, and these payloads are 16 bytes per record. These are exactly the sub-list payloads -- 009376f0, 00939c50, 0093cc70, 00998880 and 009d2da0 were read directly in earlier passes and all open 00 00 00 30 followed by a repeated word. The other nine have ratios of 168, 419, 559, 31288, 7985, 3458, 764 and 6452 -- no shared factor, far too large to be a record count times anything. Either their payload is not followed immediately by the next packet in pointer order, or they are not 16 bytes per record. Ruled out either way: they do not follow the same rule. This resolves byte 8, measured two entries ago as "a count, 0x0b to 0x29" and now shown to size a payload exactly in eight independent cases within one level. Whether it also sizes the fixed-point payloads is the open half. Still open: why the fixed-point packets differ, and what the 16 bytes per sub-list record are -- the repeated word (0xcd, 0xc0, 0xc8, 0x78) differs per packet, so the records are not identical to each other. |
||
|
ReaperOfVeriod
|
db76319600 |
Find a second packet record magic interleaved with the first
Range measured: savestate slot 1, RAM 0x22a000..0x24037f. Result: a second four-byte record magic appears at comparable density beside 04 04 00 01 -- 02 01 00 01, ten occurrences against nineteen. The two interleave: every 02 01 00 01 sits between two 04 04 00 01 records. 0x22a6b8 04 04 00 01 13 00 00 30 70 6a 93 00 0x22a6c8 02 01 00 01 19 c0 4c 6e 00 00 10 00 00 00 00 00 0x22a6e8 04 04 00 01 16 00 00 30 f0 76 93 00 0x22a6f8 04 04 00 01 16 00 00 30 50 78 93 00 0x22a708 02 01 00 01 1e c0 58 6e 00 00 10 00 00 00 00 00 0x22a728 04 04 00 01 24 00 00 30 50 9c 93 00 The 02 01 00 01 records are 16 bytes like the others but carry no pointer -- bytes 12-15 are zero in all ten. Their fields are a count at 4-5 (0x14, 0x19, 0x1e), a constant 0xc0 at 6, a varying byte at 7 (0x30, 0x4c, 0x54, 0x58), 0x6e at 8-9, and 10 00 at 10-11. The 0x6e at bytes 8-9 matches the ".. c1 01 6e" the sub-list payloads carry at offset 20-23. That is a second correspondence, and unlike the one retracted in the previous commit it is a shared value rather than a shared position. Not settled: this still does not say what discriminates the two payload kinds among the 04 04 00 01 records, since those share the same magic and the same 0x30 at byte 11 and differ only in the count at byte 8. The 02 01 00 01 records sit between them rather than replacing them, so they are a companion structure rather than the missing discriminator. Next: compare the count at byte 8 against the payload's own length. |
||
|
ReaperOfVeriod
|
2d7de9f13e |
Correct the packet tag mapping: the tag does not select the payload
The previous commit mapped the disc tag vocabulary onto the three payload classes. A wider scan says that is wrong. Range measured: savestate slot 1, RAM 0x22a000..0x24037f -- 91008 bytes, 20 times the span scanned before. Result: 19 packet headers, and all 19 carry byte 11 = 0x30. There are no 0x6d or 0x80 tags anywhere in this range. Reading six payloads: rec 0022a6b8 tag 30 -> 8d 8c 80 80 89 87 7c 80 ... fixed point rec 0022a6e8 tag 30 -> 00 00 00 30 cd 00 00 00 ... sub-list rec 0022a6f8 tag 30 -> 92 90 84 80 94 94 86 80 ... fixed point rec 0022a728 tag 30 -> 00 00 00 30 c0 00 00 00 ... sub-list rec 0022a738 tag 30 -> 4c 4c 4b 80 4c 4c 4b 80 ... fixed point rec 0022a768 tag 30 -> 00 00 00 30 78 00 00 00 ... sub-list Records of identical tag alternate between two payload kinds. The tag does not select the payload. What went wrong: the previous entry took the 0x30 and 0x6d that head the payloads to be the disc-side tag vocabulary, and matched them to payload classes by that correspondence. The disc vocabulary at record byte 7 (0x00, 0x30, 0x6d, 0x80) and RAM record byte 11 are different fields that merely overlap in two values. Three payloads each made the coincidence look like a rule. What is actually established about the RAM header: 0-3 04 04 00 01, constant 4-7 zero, constant 8 a count, 0x0b to 0x29 across the 19 records 9-10 zero, constant 11 0x30, constant on all 19 in this range 12-15 an EE RAM pointer Byte 8 is the finding that survives: a small count varying per record, and the one field here doing arithmetic rather than tagging. Still open: what 0x30 at byte 11 means, what discriminates payload kinds given the tag does not, and 0x80, which has no sample in 91 KB of RAM. |
||
|
ReaperOfVeriod
|
0b1793d6af |
Map shrub packet tags onto three payload kinds
Dereferenced all nine pointers in the level 0 table that carry the 0x30 tag at byte 11 and an EE pointer at 12..15, reading 32 bytes each. They fall into exactly three classes, three payloads each, with no overlap. A, Q16.15 fixed-point vertex data (3 of 9): every high byte 0x80 or 0x7c, repeating on an 8-byte boundary, values differing per payload while the 4e 4e 4c 80 tail is shared by two. 00936a70 8d 8c 80 80 89 87 7c 80 ... 4e 4e 4c 80 0093cde0 8a 8a 7f 80 92 90 84 80 ... 4e 4e 4c 80 009cf4f0 6e 6f 68 80 6e 6f 68 80 ... 54 54 51 80 B, a 0x30 sub-list with a repeated word (3 of 9), structurally identical apart from that word (0xcd, 0xc0, 0xc8), all carrying 01 00 00 05 at offset 16 and a ".. c1 01 6e" at 20. 009376f0 00 00 00 30 cd 00 00 00 cd 00 00 00 ... 00998880 00 00 00 30 c0 00 00 00 c0 00 00 00 ... 009d2da0 00 00 00 30 c8 00 00 00 c8 00 00 00 ... C, an index/attribute list (3 of 9), both opening 00 c0 05 6d, then a count, then a run of small ascending u16. 00939700 00 c0 05 6d 3e 00 00 00 01 00 00 00 ... 009c1540 00 c0 05 6d 36 00 00 00 01 00 00 00 ... So the tag vocabulary measured on disc at record byte 7 -- 0x00, 0x30, 0x6d, 0x80 -- selects the payload: 0x30 class B, the 00 00 00 30 sub-list 0x6d class C, the 00 c0 05 6d attribute list 0x00 class A, the fixed-point vertex data 0x30 heads the class B payloads and 0x6d heads the class C payloads. That was flagged as a single coincidence two commits back; three payloads each makes it a rule. 0x80 has no sample among these nine and its meaning stays open. Correction: the record spacing in this table is not uniformly 0x30. The find output shows 0x30, 0x10, 0x30 between consecutive packet headers, and only 9 of the 5632 bytes scanned carry both the 0x30 tag and an EE pointer. The earlier "0x30 spacing" was inferred from a truncated hit list. |
||
|
ReaperOfVeriod
|
ad70e4c4be |
Follow shrub packet pointers: adjacent records hold unrelated payloads
The header pointers found in the previous commit are live, so they can be dereferenced. Two records 0x30 apart in the level 0 table at 0x22a6b8 were followed. Range measured: savestate slot 1, reading 0x00936a70 (from the record at 0x22a6b8) and 0x009376f0 (from the record at 0x22a6e8), 64 bytes each. The two payloads have nothing in common. 0x00936a70 is fixed-point vertex data: 8d 8c 80 80 89 87 7c 80 8d 8c 80 80 89 87 7c 80 4e 4e 4c 80 4e 4e 4c 80 4e 4e 4c 80 4e 4e 4c 80 repeating Every high byte is 0x80 or 0x7c and the pattern repeats on an 8-byte boundary. As s16 these are large negatives in a narrow band around -32500, consistent with Q16.15 fixed point rather than f32, and with a stride of 4 s16 per vertex. 0x009376f0 is a tagged sub-list, not vertex data: 00 00 00 30 cd 00 00 00 cd 00 00 00 cd 00 00 00 cd 00 00 00 01 00 00 05 22 c1 01 6e 00 00 00 00 Structurally different: small integers, a repeating 0xcd, 0x30 tags in the high byte. Nothing here is position data. This supports the type reading of the header: the table is a tagged collection where the tag selects the payload kind, not a uniform array. It also rules out the alternative that the 16-byte tiling was one uniform record format with varying fields -- two adjacent records with unrelated payloads is not that. Still open: the mapping from tag value to payload kind. The 0x30 appearing in record bytes 8-11 in RAM also heads the second payload, which is suggestive but is one coincidence so far and is deliberately not written up as a link. |
||
|
ReaperOfVeriod
|
3854e52019 |
Show the loader rewrites shrub packet headers into RAM pointers
Answering the "what is the tag byte" question needed an anchor. Searching live RAM from scratch was the wrong way to get one: 04 04 00 01 matches unrelated data at 0x1ac988 and there is no way to tell a real hit from a coincidental one. The savestates are the anchor -- slot 1 is level 0, and searching one gives candidates that can be checked rather than guessed at. Range measured: savestate slot 1, ~/.config/PCSX2/sstates/SCUS-97268 (38996035).01.p2s, records at 0x22a6b8 onward, read with gc-trace read --state. Result: 40+ records beginning 04 04 00 01 00 00 00 00 at 0x30 spacing -- the packet framing identified on disc -- with the second half of each header rewritten: disc: 04 04 00 01 | 00 00 00 00 | 00 00 00 05 | 00 80 0e 6c RAM: 04 04 00 01 | 00 00 00 00 | 13 00 00 30 | 70 6a 93 00 Bytes 0-7 unchanged; bytes 8-15 are not. On disc bytes 12-13 are the constant 00 80 and byte 15 the constant 6c; in RAM bytes 12-15 hold a pointer -- 0x00936a70, 0x009376f0, 0x00937850, 0x00939c50 in consecutive records, all inside EE RAM and ascending across the table as a walk of consecutive objects would be. Byte 11 becomes 0x30 in every record seen, against a constant 0x05 on disc, and bytes 8-9 hold a varying small value (0x13, 0x16, 0x24). So the packet table at +0x40 is what the loader walks: span-relative on disc, absolute in RAM. This was inferred from one class before; these 40+ records show it directly. Consequence for the tag bytes: the vocabularies at record bytes 7 and 15 are real, but byte 15 sits in a field the loader overwrites with a pointer. Its disc values are read as data by something rather than stored as a tag. Not settled: whether byte 7's vocabulary is also rewritten, and whether 0x6d marks payload the loader expands. That needs these records compared one by one against their disc spans, which is cheap now that the anchor exists. |
||
|
ReaperOfVeriod
|
d1aec9f866 |
Show shrub packet records are two tagged 8-byte halves, fixed length
Range measured: level 0, the 1636 sixteen-byte records following the header across all 29 packets. Per-byte distinct counts, not pooled. Method: count distinct values and zero frequency at each of the 16 positions, then tabulate the joint distribution of the two low-entropy positions. Result: fourteen positions are uniformly varied, 101 to 153 distinct values, zero in 13% to 58% of records. Two are not: byte 7 4 distinct: 0 (1432), 0x30 (135), 0x6d (40), 0x80 (29) byte 15 2 distinct: 0 (1618), 0x6d (18) Both sit in the last byte of an 8-byte half, eight bytes apart, and both take values from a small closed vocabulary rather than the full byte range. The joint distribution shows no record where both are non-zero, and 0x6d appears in either position but never both: (0,0) 1414 (0,0x6d) 18 (0x30,0) 135 (0x6d,0) 40 (0x80,0) 29 Each 8-byte half carries a tag or subtype byte in its last position, from a small closed set. The shared 0x6d is what suggests one vocabulary used twice rather than two unrelated fields. This answers the question the previous commit left open. There is no per-record length field: the best candidate offsets hold the value 16 on at most 13% of records, which is the chance rate for small integers, not a self-describing record. The records are therefore fixed 16 bytes, and the tiling is not a coincidence of variable-length records that happen to be multiples of 16. Still open: what the four and two tag values mean, and what the 14 varied bytes encode. The tag vocabulary is small enough to enumerate against known PS2 primitive lists, which is the reference to bring in next. |
||
|
ReaperOfVeriod
|
e6dd60ec3a |
Characterise shrub mesh packets: 16-byte header, 16-byte tiling
The packet table decode left the packet contents open, with a warning that a VIF reading had already been retracted once and needed a control rather than a pattern match. This is that measurement. Range measured: level 0, the 29 packets of the 6 successor-bounded shrub spans, 26640 bytes. Per-packet; nothing pooled. Method: autocorrelation within the largest packet against a distribution matched shuffled control, then per-byte distinct counts across all 29 packets. Result 1: all 29 packet sizes are exact multiples of 16, and the internal stride scan peaks at 16 (+0.283) with 32, 48, 64, 80 and 128 all elevated behind it -- the signature of a record array, not a coincidental spacing. Result 2: record 0 is a 16-byte header. Constant across all 29 packets at bytes 0-3 (04 04 00 01), 4-11 (zero), 12-13 (00 80) and 15 (6c). Bytes 14 and 16-23 vary. The 04 04 00 01 prefix looks like a magic or format version. Result 3: header byte 14 is echoed in the body. It takes values 06, 07, 0e and 11 across packets, and the same value appears at offset 12 of record 1 in every packet sampled. A real repeated field, not two small values coinciding. Not established: no header field matches the record count. The packet holds 1056/16 = 66 records and none of the candidate u32 or u16 header offsets at 8, 10, 12, 14, 16, 18 or 20 equals 66, 65 or 67 on more than half the packets. So the count is either computed from a size field elsewhere in the span, or the records are variable length and merely 16-byte aligned. That distinction is unresolved and matters. Fixed 16-byte records make the payload a flat table; variable records make the 16-byte alignment a coincidence of every record being a multiple of 16 long. The stride scan cannot separate the two, because both peak at 16. The retracted VIF reading is not revived: the tiling is uniform across all 29 packets with no command-shaped tail, and a VIF packet would show a changing command byte last rather than the constant 6c at header byte 15. Next: read the record count from the packet table in the span rather than from the packet header, which is what would separate the two readings. |
||
|
ReaperOfVeriod
|
aa2b14ad66 |
The shrub mesh geometry is in the class blob after all
The open question from the shrub decode was where the mesh lives: the blob looked like header, palette and optional billboard, with the 333 classes without a billboard having nothing to draw. Answering it took the running game, not the disc. Diffing a loaded shrub class against its disc bytes in the live PCSX2 (level 1, class 1) shows what the loader rewrites: a table at +0x40 of (offset, size) pairs, whose offsets are span-relative on disc and absolute in RAM with the sizes unchanged -- all 21 entries matching offset for offset and size for size -- plus a relocated palette pointer at +0x2c and a pointer at +0x18, zero on disc, that asserts the parser enforces. So the blob is header, packet table, packets, [billboard], palette: the mesh was here the whole time. Why it was missed is the part worth recording -- data_offset points at the END of the span, so reading the body as data_offset..span_end accounts for the last few hundred bytes and leaves the 20 KB in front of it unexamined. A correct bound and a body small enough to look plausible are together enough to hide a region. Range measured: every shrub span on the disc, 358 over 24 levels, each bounded by its successor class or the located chunk container. Method: read count at +0x28, then the entries at +0x40, and require them to tile -- the first packet at align16(0x40 + count*8), every entry exactly size past the previous, and the last ending at the palette, or 0x40 short of it on the 25 spans that carry a billboard. Result: 358 of 358 tile exactly, holding 2,239 packets. That is what pins the field meanings: count is the packet count (1..55 on the disc), data_offset is the palette's offset, and the earlier "the body is not sized by count" note was true for a reason nobody had understood -- the count sizes the packets. Shrub::packets() exposes them and returns None rather than a partial list when a table does not tile. Still open: the packet contents (a VIF reading was already retracted once, so this needs a control rather than a pattern match), and the 0x20-byte records the loader points at from +0x18. |
||
|
ReaperOfVeriod
|
c83f5104d2 |
Document the PCSX2 test setup
PCSX2 was set up and working but existed only in this machine's filesystem and in people's heads, so every comparison looked like a one-off observation rather than something re-runnable. The setup section in docs/plan/pcsx2-verification.md now records where the emulator, the BIOS, the disc, the savestates and the PINE socket are, how a state is captured, and which command the disc integration tests need -- and AGENTS.md and README.md point at it from their orientation sections. One operational detail worth spelling out: the emulator's directory is lowercase (pcsx2), which is not what people type. |
||
|
ReaperOfVeriod
|
bbdb51e2f9 |
Verify level 1's tfrag lighting against the game
Savestate slot 2 is the second planet, so the comparison runs on a level with a different geometry extent, light bank and runtime-set population: level1: 401/401 headers, 401/401 pointers, bank byte-identical vertices compared: 26470 (105880 colour bytes) -- equal: 100.00% The live emulator agrees with the savestate byte for byte. Level 1 is a different planet, so the two fixes in the previous commit are not level-0 coincidences: the normal's z sign and the unlit-record base colour both hold across 268,280 colour bytes on two levels. |
||
|
ReaperOfVeriod
|
69a8398f6e |
Verify the tfrag lighting magnitudes against the running game
The harness that compares the port against the game's own memory existed but had never been compiled: tools/trace did not build (27 errors, all in the argument plumbing), and the tfrag_light.rs light-bank correction it motivated was unverified. This finishes it and runs it, and the run finds two real bugs. Result, on level 0 against three inputs -- the live emulator over PINE and two PCSX2 savestates from the same level, agreeing byte for byte: headers matching the disc on invariant bytes: 833/833 data pointers relocated as table + disc offset: 833/833 light bank in RAM at 0x50ab20, byte-identical to the disc's vertices compared: 40600 (162400 colour bytes) equal: 162400 (100.00%) The locator finds the level's tfrags block in RAM by tfrag 0's bounding sphere and accepts a candidate only when every header matches the disc on the bytes the game leaves alone and every data pointer is the relocated form, so "the block was found" is itself a verification rather than an assumption. The bank is searched in RAM by its exact section bytes, which is what settles its framing: offset at +0x04, an i32 set count, twelve bytes of pad, then the sets. The previous reading (offset at +0x00, count at +0x04) was wrong. Two bugs the comparison found, both of which produced plausible output: 1. vertex_normal_bits negated x and y and left z positive, while NormalTable::normal negates all three. dA came out -0.707 where the game had +0.707, so max(dA, dA*wA) took the wrong branch and every vertex lit near its base colour: dark, but not wrong-dark. 95.88% agreement with the bug, 100.00% without. 2. A record whose select names a set the bank lacks drew transparent black. The game writes the record's base colour; 3,130 of level 0's 40,600 vertices do this (select 0x000f, set 15 -- a runtime slot the level's directional bank does not have). The vertex is unlit, not invisible. Both are now regression tests in the engine. The comparison prints the pass's per-vertex terms (light_vertex_terms) for the first mismatches, because a colour mismatch alone cannot be attributed and the comparison is worthless without attribution -- that is how the second bug was identified from a three-line read rather than guessed at. Docs: docs/plan/pcsx2-verification.md records the harness, the result, the two bugs and what to compare next; roadmap and level-assets.md stop carrying lighting magnitudes as an open item. Findings reach tests only as numbers -- no test reads a savestate, a dump or a live emulator. |
||
|
ReaperOfVeriod
|
2c7555fe16 |
Decode the shrub class blob: a 24-normal palette and a billboard record
The shrub body's payload encoding was the open question after the span-bound work, and it answered itself once the R&C1 reference was in view: R&C1's shrub class header keeps the field offsets R&C2 shows (o_class at 0x24, a count at 0x28, the data offset at 0x2c), which says which bytes to look at, and the disc says what they hold. Range measured: all 358 shrub class spans over the 24 levels that have them, each bounded by its successor class or the located chunk container. Nothing outside the spans is read. Result. The body is a palette of 24 unit-length s16 normals -- s16 x, y, z and a zero pad word per 8-byte record -- followed by all-zero padding records. Three independent measurements, each of which fails for any other reading: - all 8,592 live entries have squared length within 65,535 of 32767^2, and the bound is tight (the largest observed deviation is exactly 65,535). That is 32768^2 - 32767^2, which is what a unit vector scaled by 32768 produces when an axis-aligned normal quantises to -32768 and the positive axis clamps to 32767; components run the full s16 range and 1,765 entries are axis-aligned. A random s16 triple is off by about a billion, so one entry in fifteen thousand would pass by chance: all 8,592 pass. - the padding records are always at index 24 or later and always all zero, never a live entry. That is what separates the palette from geometry, and it is the observation the three earlier stride-fitting attempts were missing. - 17 classes carry a palette of 24 identical (0, 0, 1) entries. A position or index table cannot produce that; a normal palette for a flat-facing class must. The header decodes as: sphere at +0x00 (centre + radius, radius positive on all 358 spans, 1.72..41.57), size at +0x10, mode_bits and scratch at +0x14, a zero word at +0x18, the billboard offset at +0x1c, and scale at +0x20. The parser asserts the two invariants that hold without exception (scratch and the reserved word are zero on all 358) plus a positive radius and scale, and reports mode_bits rather than asserting it: it is 4 on 16 of the 358 spans. The billboard offset is the strongest self-identification after o_class because it is confirmed from a second file: it is non-zero on exactly the 25 classes whose core-index entry carries billboard texture fields, pairing correctly both ways. At the offset sits R&C1's ShrubBillboard shape, register block and all -- fade_distance, then TEX1_1 (K -233..-168, MMIN 4, address 0x14), TEX0_1 (index 1..4, address 0x06) and MIPTBP1_1 (address 0x34), all three register addresses checked in the disc test. Two corrections this produces, both recorded in docs/plan/level-assets.md because the old readings were confidently wrong rather than merely unverified: - "reserved is 0 or a denormal" had the two u16 halves the wrong way round: the 16 non-zero spans are mode_bits = 4 with scratch = 0. As an f32 the word is still a denormal, so the PS2 reads zero either way -- but the field is a flag word, not a magnitude, which is exactly the FPU point worth making. - "bounds" was tested as a min.xyz/max.xyz box, which holds on only 146 of 358 spans, and the field was left unnamed as a result. It is a bounding sphere: the fourth float is positive on all 358 spans and the centres cluster near the class origin. The 146/358 figure was a property of the wrong question. Still open: the three floats in the billboard record (they repeat byte-identically across levels -- 16 distinct triples among 25 records, one on four classes across levels 1, 4, 12 and 21 -- so they are shared art data rather than dimensions), size, mode_bits, count, and where the shrub mesh geometry lives. R&C1 puts the mesh packets in this blob; R&C2's blob holds only the palette, so the 333 classes without a billboard must draw meshes from somewhere else. |
||
|
ReaperOfVeriod
|
ccea53807a |
Resolve the shrub region bound: the chunk container closes it
The shrub region's end was the last unlocated bound in core_data. It was not in the core index, and unlike moby->tie it had no successor boundary to chain against, so the last shrub class in every level ran out to the lump's end and the aggregates over it (the "3.5M records / 28MB of shrub body") were artifacts of that oversized span. Range measured: for each level, from the last shrub class's geometry offset to the constant trailer. Nothing outside that range is read. Method: scan forward from the last shrub class for the pair (a) the constant two bytes f6 7c sitting immediately in front of (b) a chunk signature -- a 16-byte header ending 0xff, ten zero bytes, then four or more 0xf0-tagged offsets. The two halves are each weak alone, and the halves are measured separately: the marker pair occurs 25 to 103 times in a 5 to 17 MB lump, so matching it alone picks dozens of wrong offsets; the chunk shape matches random data often enough to need four tagged offsets before it is believed. Result: on all 27 levels the marker sits exactly two bytes before the first chunk header, and the two parts of the signature are never found apart. No chunk signature occurs inside any last shrub body, on any level. The three levels without shrub (10, 25, 26) carry the same marker before the container that closes tie instead, which is what makes the locator a property of the region rather than a shrub rule. Consequences: - All 358 shrub class bodies are bounded and are whole 8-byte records. That was not assertable while the last class in each level ran to the lump's end, and it is the check the boundary earns its keep for. - 9,600 records in total, against the 8,938 previously measurable in successor-bounded spans; 662 of them sit in the 24 last-class bodies. The shrub region totals 2,033,776 bytes across the 24 levels that have one. - The chunk container is 25,775,850 bytes across 27 levels (1,236 chunks, 22,562 records). The container measurements were previously taken over "the 900,000 bytes before the trailer", a window that did not align to its real extent: on level 0 it is 690,302 bytes, and 27 of 1,123 chunks were truncated at that invented edge. chunk_containers_resolve_on_every_level now reads the region from core_layout instead of measuring backwards from the end of the lump. core_layout takes core_data now: the container's start is the one bound that cannot be derived from the index. It gains the chunks region, so every byte of level-specific core_data belongs to a named region. Where the signature is not found the end stays None -- the region is left visible rather than clamped to the tail. Still open: the marker's meaning (constant across 27 levels, at exactly this position, unnamed) and the shrub body payload's encoding. |
||
|
ReaperOfVeriod
|
4d1e277a15 |
Exclude f32 and u16 readings of the shrub body record
Range measured: level 0 shrub bodies, 164 records of 8 bytes, following the
previous commit.
The six varying bytes are not an f32 pair:
f32 @0 158/164 finite, range up to ~1e35. Not positions or normals. The 6
non-finite values (3.7%) are far above the 0.05% random bytes would
give, so the field is structured but is not an f32.
f32 @4 164/164 finite, range 0 to 9.18e-5. Looks promising until you notice
bytes 6 and 7 are always zero, which pins it to a coarse grid and
explains the tiny range. Not independent evidence.
u16 @0 85 distinct, range 0..65534
u16 @2 90 distinct, range 0..65534
u16 @4 80 distinct, range 0..65512
The u16 fields are not small counts or indices; they span essentially the whole
range.
Excluded: two packed u32, an f32 pair, and narrow u16 fields. The six bytes
remain uninterpreted.
Worth noting the f32 @4 case as a trap: a plausible-looking finite non-negative
float range that was entirely an artifact of the two constant zero bytes
pinning it. Checking the constant bytes before believing a float reading is now
the order to work in.
|
||
|
ReaperOfVeriod
|
7c0df0f893 |
Characterise shrub body records on level 0 as a per-level partition
Range measured: level 0, shrub class table, 7 entries at core index offset 10176. Six classes are bounded by a successor; the seventh runs to the end of core_data and is excluded because its span contains the whole post-shrub region. Method: bound each span by the next class geometry offset, take the body from data_offset to the span end, split into 8-byte records. Result: 164 records across 6 bounded bodies, lengths 192 twice, 208 twice, 224, 240. Per byte over those 164 records: byte 0: 74 distinct, 60 zero byte 4: 72 distinct, 35 zero byte 1: 58 distinct, 65 zero byte 5: 59 distinct, 38 zero byte 2: 76 distinct, 43 zero byte 6: 1 distinct, 164 zero byte 3: 62 distinct, 48 zero byte 7: 1 distinct, 164 zero Bytes 6 and 7 are zero in every record; bytes 0 to 5 take 58 to 76 distinct values with only 21 to 39% zero, so uniformly varied rather than sparse. Read as u32 the fields are not plausible: u32[0] spans 0 to 4294871316 and no field is monotone across the body. The six bytes are not two packed 32-bit values. This settles the six-varying-plus-two-constant shape on level 0 as a per-level partition, rather than only in aggregate over the disc's 8938 records, which is what rule 3a asks for. Two trailing zero bytes are padding or a flag pair. Unresolved: whether the six bytes are six u8, an f32 plus two u16, or something else. The u32 reading is excluded but no positive reading is established. Process note: the first attempt used ClassEntry and a hand-computed o_class instead of ShrubClassEntry, and every ShrubHeader::region call failed. The parser was fine; the caller was not. |
||
|
ReaperOfVeriod
|
efe4c06be6 |
Show the chunk header does not encode the record stride
All 90 two-byte windows of header[0..14] were tested: none maps to a unique stride across every group. There is no simple two-byte field to read the record size from. The stride must therefore come from the offset table, by differencing the first two tagged offsets, which is what the parser already does. This validates that choice as more than convenience. Given three chunks sharing one header with three different strides, a dispatch that assumed "header implies size" would be wrong for the whole type-ID coverage on at least one layout. Reading the stride from the offsets is the only interpretation consistent with every measurement so far. Negative result recorded deliberately: it rules out a whole family of hypotheses (stride encoded in the header) rather than narrowing one. |
||
|
ReaperOfVeriod
|
1d84562e88 |
Resolve the colliding header: three types share 16 bytes
Header 7542eff271b522a79e45348cca4509ff was reported with three strides, 76, 92 and 104. It is not one type with a variable stride, and not a misparse. It appears three times per level, always with record count 9, always in the same order at the same offsets within the region: offset stride count first last 863842 92 9 928 1640 867554 76 9 912 1520 872866 104 9 1000 1832 The offsets are 3712 apart each time and the strides differ, so these are three separate chunks sharing 16 header bytes. Consequence for the type ID: 64 distinct headers for 66 distinct record layouts, not 64 for 64. The header is nearly a type ID but does not uniquely determine the layout. Not excluded: a distinguishing byte sits outside the 16 captured, or the layout varies within a type. This does not change "63 of 64 headers imply one stride", which remains true, but the mapping is not a bijection and should not be used as one. Worth checking before the parser is extended to dispatch on header alone. |
||
|
ReaperOfVeriod
|
4c4175de7f |
Add a rule requiring measurements to be partitioned before reporting
Rule 3 required bounds and a method for every number, but not that the measurement be partitioned. Three errors in one stretch were all of that kind, and all three passed the control check rule 3 asks for -- because a matched control bounds significance while saying nothing about scope: - a 368-byte period attributed to a whole region, when it was one chunk's stride among ~15. Averaging the strides put the peak at the noise floor, which read as "no structure" rather than "wrong scope". - a ten-entry table disconfirmed, when it existed at a fixed stride among several. - a moby class array reported at u16 slots 129-140, which was the largest-stride chunk type dominating a pooled scan. Per type, all 26 place their array elsewhere. In each case the aggregate was arithmetically correct and the conclusion was wrong. The re-run that caught all three was identical: group by the thing that varies, then look. Rule 3a now requires stating the partition before the number, reporting per partition or saying explicitly that a claim is pooled and unverified, treating a structure found in one partition as unproven until every partition agrees, and retracting rather than softening a result that changes under partitioning. Renumbered the following rules. |
||
|
ReaperOfVeriod
|
b13901a7e7 |
Correct the moby class array offset: it is per chunk type
The previous commit reported moby class IDs at u16 slots 129 through 140. That was a pooling artifact. Repeating the scan per chunk type shows every type places its class array at a different offset: aac4 129-134 82c4 104-109 4ec4 106-111 b8c4 120-125 9943 82-87 79c4 111-115 3245 24-29 and so on across all 26 types Slots 129-140 were simply the type with the largest stride dominating a scan that pooled every type together. The previous commit flagged this as unverified; the check confirmed the flag. What survives is stronger than what it replaced. In every one of the 26 types the top hits form a contiguous run of consecutive u16 slots. The real invariant is therefore per-type: each chunk type has its own class array, at its own offset, of its own length. That fits the header-implies-one-stride result -- each type is a distinct record layout -- but the class array is a per-type field, not a common one at a fixed offset. The moby reference itself is unaffected. The 38.6x excess was measured across all records and does not depend on which slots were counted. |
||
|
ReaperOfVeriod
|
ddfbc6eb33 |
Find moby class references in chunk records at 38x chance
Range measured: level 0, the 900000 bytes before the trailer; 28 chunks, 475 records, 67780 u16 slots. Reference set: the 238 distinct o_class values from level 0's moby class table, read through LevelCoreHeader. Method: scan every u16 in every record, count membership in the moby class set, and compare against the chance rate of 238/65536 per slot. Result: 9493 hits against 246.1 expected, a 38.6x excess. The hits are concentrated, not spread. Slots 129 through 140 -- twelve consecutive u16, bytes 258 to 280 of the record -- carry 105 to 188 hits each. Nothing else comes close except a separate cluster at slot 38. Every chunk contributes in proportion to its record count. A contiguous twelve-value array that is mostly moby class identifiers is what a model or instance definition listing referenced mobies would look like. Enrichment is not identification. A 38x excess over chance is strong evidence these are moby class IDs rather than incidental small integers, but it does not establish that the chunk is moby_anim: an instance table, a model list and an animation table would all reference class IDs. The format and the meaning of slots 129-140 are both still unnamed. Also unverified: whether the twelve slots are always the same fields. The offsets come from the largest records, and this scan pooled all chunk types together, so a type with a different stride could place them elsewhere. Process note: the first version of this test extracted the moby class table by hand and got 67 entries instead of 238, which made every chunk look like a hit and would have supported any conclusion. Reading it through LevelCoreHeader was what made the measurement meaningful. |
||
|
ReaperOfVeriod
|
379c12f8ec |
Identify the chunk header as a type ID that determines the record stride
Range measured: all 27 levels, 1123 chunks. Method: group chunks by their 16-byte header; for each distinct header, collect the set of strides seen across every level it appears on. Result: only 64 distinct headers across 1123 chunks. The header is a type identifier, not a per-chunk descriptor -- the same 16 bytes recur on every level. And 63 of those 64 imply exactly one stride, so header to record layout is very nearly a function. The exception is header 7542eff271b522a79e45348cca4509ff, seen with strides 76, 92 and 104. Either three types share a header or one is a misparse. This also settles the previous entry's open question about D/E/F being incidental. Grouping by header bytes 9 and 13: DE 26 chunks, 2 strides (184, 344) EE 1050 chunks, 46 strides EF 34 chunks, 2 strides (268, 640) FF 13 chunks, 1 stride (520) Rare tags are tightly coupled to their layouts -- FF is a single stride across all 13 chunks -- while EE is heterogeneous. That is what a type tag looks like. Consequence: a chunk can be dispatched on its header to a record layout, and 63 of 64 headers give that layout's size without inspecting a record. First point in this container where a field does real work rather than correlating. Still open: what the 64 types mean, which record fields are index/length/offset, and whether any type is moby_anim. |
||
|
ReaperOfVeriod
|
10188416e2 |
Decode the chunk count field: header byte 14 is the record count
Range measured: all 27 levels, 1123 chunks, over the 900000 bytes before the constant trailer. Method: compare header byte 14 against the number of offsets actually resolved for that chunk. Result: byte 14 equals the record count on 1123 of 1123 chunks, zero mismatches. This is the first genuinely decoded field in this container rather than a measured correlation. The parser now truncates the offset table to header byte 14, so a stray tagged word in a chunk's record data cannot silently extend the table and put the count at odds with its own header. Two more header bytes are near-constant and read as ASCII: byte 13 takes only 0x45 (1076 chunks) and 0x46 (47); byte 9 takes 0x44 (26), 0x45 (1084) and 0x46 (13). Those are D, E and F -- a three-letter tag at two byte positions with different distributions. Recorded as suggestive, not established: the other 14 header bytes are high-entropy and read as ASCII at no shift tried, so the D/E/F may be incidental. Worth testing against a known type table. Still open: the 14 opaque header bytes, whether bytes 9 and 13 are type tags, which record fields are index/length/offset, and whether a chunk is moby_anim. |
||
|
ReaperOfVeriod
|
681333535a |
Add a bounds-checked parser for the post-shrub chunk container
gc_formats::chunks parses the container that fills the region after the last shrub class body and before the constant trailer: +0x00 16 bytes, last byte 0xff (opaque, differs per chunk) +0x10 10 zero bytes +0x1a N tagged offsets, top byte 0xf0 Each tagged offset is 0xF0000000 | low24, and a record is read at chunk_start + low. Resolving that way gives records sharing a field layout, which is what distinguishes the container from a coincidental byte pattern. Fields are not named. The 16-byte header is opaque, the 0xf0 tag is unexplained, and it is not established that a chunk is moby_anim -- a 49 x 368-byte table inside this region is byte-identical on all 27 levels, which is evidence against that identification. What the parser provides is chunk and record extraction, which is what the analysis established. The gated test measures, over all 27 levels: 1123 chunks, 19546 records, 50 distinct strides, 12.4% of byte positions constant across records (per-chunk range 2.8% to 75%). Chance gives a constant position probability of 1/256^n. Three things the test got wrong first, recorded because they are the traps: - A per-chunk threshold of 20% constant positions. Individual chunks run from 2.8% to 75%; the aggregate is the invariant, not the chunk. Asserting 20% per chunk was asserting a property the format does not have. - Requiring every offset to resolve. 27 chunks truncate at the window edge, because the 900000-byte window does not align to chunk boundaries. Counted now rather than asserted away. - read_u32 in the chunk signature check was big-endian while Buf::u32 is little-endian. The tag lives in the high byte of a little-endian word, so the mismatch made every level report zero chunks while parse_chunk -- which goes through Buf -- was correct. Worth remembering that the two paths can disagree on a format this young. Tests: 190 -> 191. |
||
|
ReaperOfVeriod
|
8b0a2545a3 |
Confirm chunk offsets address real records on all 27 levels
Range measured: each level's post-shrub region, 900000 bytes before the constant trailer. Method: per chunk, read tagged offsets as 0xF0000000 | low24, take stride as the first delta, load one record per offset at chunk_start + low, and count byte positions constant across that chunk's records. Result: 1096 chunks and roughly 13000 records resolve on all 27 levels. Every level shows 17-21% of byte positions constant across its own records. Level 0: 27 chunks, 466 records, 20.8% constant. The stride-368 chunk (77 records) has 26 constant positions of 368; the stride-340 chunk has 25 of 340. Constant positions cluster at the record end -- 267-276 for stride 368, 258-268 for stride 340 -- so a footer, not scattered padding. The tag is uniform: every tagged offset has top byte 0xF0 on all levels. This settles the previous entry's framing. Resolving at chunk_start + low yields records with ~20% constant positions, which does not happen by chance, so the offsets genuinely address records and those records share a layout. Still not decoded: the 16-byte header, the 0xF0 tag, which fields are index/length/offset, and whether a chunk is moby_anim. A parser can extract chunks and records bounds-checked today; naming the fields needs a domain argument this project has not found. |
||
|
ReaperOfVeriod
|
1d6cddf3bb |
Identify the post-shrub region as a chunk container
Range measured: level 0 post-shrub, 16047040..16737344, 690304 bytes. Method: scan for a 16-byte header ending 0xff, then 10 zero bytes, then a run of u32 whose top nibble is 0xF. Result: 28 chunk headers on level 0, present on all 27 levels. Each chunk is +0x00 16 bytes, last byte 0xff (opaque, differs per chunk) +0x10 10 zero bytes +0x1a N tagged offsets, top nibble 0xF Within a chunk the offsets step by a chunk-specific stride. On level 0: 148, 340, 368, 288, 320, 236, 272, 384, 116, 160, 250, 580, 576. Near-constant within a chunk, different between chunks. This resolves two earlier results, both of which were half-right. The 368-byte period is one chunk's stride, not a region property. That is why the region-wide autocorrelation is flat: ~15 strides averaging to noise. The 15360-byte window where 368 scored +0.315 is the third chunk at region+30066, whose stride is 368. The "ten-entry table" earlier disconfirmed is real. Chunks carry 5 to 30 tagged offsets, 10 most common on level 0. It was disconfirmed because I tested for a fixed 148 stride across a whole region; the table exists, but 148 is one chunk's stride among many. Both errors were the same one: reporting a local measurement as a regional property. Twice now. The control methodology caught neither, because a matched control bounds significance but says nothing about scope. Not decoded: the 16-byte header, the 0xF tag, where chunk data lives, and what any chunk means. Framing established, contents not. |
||
|
ReaperOfVeriod
|
c60340b1de |
Correct the post-shrub 368 stride: real, but in one 15 KB window
The claim that the post-shrub region has a 368-byte period throughout is wrong. Re-measured over the whole region the curve is flat and near noise: stride 8 +0.027 stride 12 +0.023 stride 148 +0.009 stride 340 +0.018 stride 368 +0.018 stride 632 +0.020 Nothing peaks at 368, and the 0.1110 recorded earlier does not reproduce under any sampling convention tried. Where the structure is: scanning 8 KB windows at 1 KB steps for a 368-excess above 0.15 finds one substantial region, region+41984..+57344, 15360 bytes, plus a 4 KB patch at region+650240. Within that window 368 is unambiguous -- +0.315 against +0.284 at 736, with 364, 372 and 732 all at noise. 41 whole records, mean 16 distinct values per position, and nine positions exactly constant and zero: 329-336 and 340. Twenty-five positions have 3 or fewer distinct values, clustered in 325-349, which is a tail rather than scattered padding. Corrected statement: a 15360-byte window of the post-shrub region is a table of 41 records at stride 368, each with a roughly 43-byte tail whose fields are mostly zero. The surrounding 675 KB is not that table. What went wrong before: the original measurement was taken over a window where 368 held, then reported as a property of the region. That window holds 41 of 1841 records, and averaging it over the region flattens the peak to the noise floor. The re-verification checked the stride but not whether it applied uniformly, so it confirmed the number without testing the claim attached to it. |
||
|
ReaperOfVeriod
|
7992e9d8f3 |
Note that the SBlk period is not a record boundary
Range measured: level 0 SBlk, 24..19508, read at 4-byte alignment. The most common u32 values are 0x0 (2261), 0x1 (369), 0x7f (168), 0x5a000000 (125), 0x1eff7f7f (125), 0x1dfcc (124), 0x19fc0 (93), 0x7c34240d (91), 0x7c3821c8 (91). That does not look like a table of records with typed fields. A 40-byte layout of index, length and offset would not produce 0x1eff7f7f and 0x7c34240d a hundred times each, nor would those values survive a byte shift. So either the 40-byte period is not a record boundary, or the region is an encoded stream that happens to have a 40-byte statistical period. The 0x7f values point at the latter: in a varint encoding 0x7f is the largest value representable in seven bits, so it appears as a frequent saturated byte -- consistent with a coded stream, inconsistent with fixed-layout records. Left open. The 40-byte period is solid -- 27 of 27 levels, sharply isolated, multiples confirming -- and so is the alignment with four constant zero positions. But those two are in tension in a way nothing else in this project has been: a confirmed period that is not a record boundary. Someone attacking this format should not assume the record framing that the autocorrelation suggests. |
||
|
ReaperOfVeriod
|
e9c927994c |
Establish the SBlk 40-byte record layout
Range measured: level 0 SBlk, 24..19508, aligned at +0 -- 487 whole records. Method: for each of 40 candidate alignments, count byte positions that are near-constant across records, and keep the best. Alignment +0 wins with 10. Four positions are exactly constant and all zero: 7, 11, 31 and 35. Distinct values per position run from 1 to 155, with position 29 the most variable by a wide margin -- 155 against 21 to 48 elsewhere -- and positions 2, 10, 27, 34 and 39 low at 2 to 8. The first record carries the SBlk magic and the DAW tag rather than descriptor data, so it is a header rather than an entry. That matters before reading any field at a fixed offset. Not decoded: the layout is established and the constant positions are known, but the fields are not named and which of the 40 bytes are an index, a length and an offset is still open. |
||
|
ReaperOfVeriod
|
0f1a7f27c7 |
Resolve the SBlk discrepancy: 40 is the record, markers are nested
A fine scan settles it in favour of 40: stride 40 +0.454 stride 39 +0.266 stride 41 +0.215 stride 38 +0.214 The peak is sharp and isolated, nearly twice its immediate neighbours, and the multiples confirm it: 80 scores +0.376, 120 +0.394, 160 +0.340, while the multiples of 12 (24, 48, 96, 144) all sit at 0.21 to 0.24. So the record is 40 bytes and the 0x7f/0x47f markers at 12-byte spacing are a nested structure inside those records, not the record boundary. Their spacing is irregular -- 12, 24 and larger -- which is what one would expect of a field present in some records and absent in others. The previous commit's "they disagree" framing was itself wrong in one respect: the marker spacing never claimed to be the record size. Read as a claim about markers inside records rather than about the record itself, the two measurements were never in conflict, and I had set up a false dichotomy between them. |
||
|
ReaperOfVeriod
|
5df3914160 |
Record an unresolved discrepancy in the SBlk record size
Range measured: level 0 SBlk, 24..19508, 19484 bytes. The 0x7f/0x47f marker words sit at 12-byte spacing on 144 of roughly 180 occurrences, with 24-byte and larger gaps accounting for the rest. So there is a 12-byte structure. That contradicts the autocorrelation. If records were 12 bytes, stride 12 should peak, and it does not: stride 12 +0.237 stride 20 +0.175 stride 24 +0.237 stride 40 +0.454 stride 48 +0.239 40 is not a multiple of 12, so the measurements disagree rather than one explaining the other. Recorded as open rather than resolved in favour of either: the marker spacing says 12, the period says 40, and 40 being twice as strong is the more surprising of the two. Worth resolving before attempting a record layout, because a layout built on the wrong one will parse plausibly and be wrong. |
||
|
ReaperOfVeriod
|
d3373042b9 |
Confirm moby's 16-byte vertex by two independent methods
The part table established vertex_offset + 16 * vertex_count == next_part on 43467 of 43467 parts, which fixes the vertex stride at 16. Autocorrelation over the spans now reaches the same number without using that result. Range measured: moby class spans of 16 KB or more on level 0 -- 60 spans, bounded by consecutive class offsets, over their first 32768 bytes. Method: peak-stride autocorrelation against a matched control, strides 4 to 64. stride 16 43 spans stride 8 8 stride 48 4 stride 32 3 stride 24 2 16 dominates and it is the same 16 the part table produces from a completely different argument -- a tiling identity over part records rather than a periodicity measurement over span contents. Two unrelated methods agreeing is worth more than either alone. The autocorrelation would be vulnerable to the exact artifacts this project has already been bitten by: a dominant byte value inflates it, and a period that happens to divide the measurement window looks identical to a real one. Neither applies here, and a tiling identity cannot be inflated at all. This is the strongest form of evidence produced so far in this project -- a structural invariant confirmed by two methods that do not share a failure mode. |
||
|
ReaperOfVeriod
|
332dc63600 |
Measure tie class spans: 16-byte unit, 48-byte period, dominant not universal
Range measured: every tie class span of 16 KB or more on levels 0, 3, 6 and every third level after -- 386 spans, bounded by consecutive class offsets. Method: peak-stride autocorrelation with a matched control over the first 32768 bytes of each span. stride 48 285 spans stride 16 58 stride 64 29 stride 96 11 stride 24 2 stride 32 1 48 bytes dominates but is not universal: 58 spans peak at 16 against 285 at 48, so this is a majority rather than a constant. Every observed peak is a multiple of 16, suggesting a 16-byte base unit with 48 arising from how those units group rather than 48 being the record. This is weaker than the sound bank's 40-byte result and is recorded at the confidence it deserves. That one was sharply isolated -- 0.454 excess at 40 against 0.21 to 0.28 across strides 36 to 44 -- and identical on all 27 levels. Here the peak is 0.09 to 0.22 excess and moves between levels and spans, so the two should not be treated as the same kind of claim. |
||
|
ReaperOfVeriod
|
d1190a3343 |
Find a 40-byte period in the sound bank SBlk, on all 27 levels
Range measured: each level's SBlk region -- the descriptor table between sblk_offset and daw_offset, from the sound bank header rather than from core_data. 17-20 KB per level. Method: peak-stride autocorrelation with a matched control, the same bytes shuffled with the same seed on every level. The peak is at exactly 40 bytes on all 27 levels, every one with excess above 0.2. On level 0 the peak is sharply isolated: 0.454 excess at stride 40 against 0.207 to 0.278 for strides 36 through 44, so it is a period rather than a broad tendency. This is the first structural result about the sound bank beyond the container framing already committed. Record counts are 487, 443, 428, 494 and 447 whole records on levels 0, 1, 2, 13 and 26, and the region is not an exact multiple of 40 on any level -- 19484 leaves 4 bytes over -- so 40 is a period rather than a record size with no remainder. Unexplained: which byte positions are constant varies by level. Level 0 has 7, 11, 31 and 35 constant; level 1 has 15, 19, 23, 35 and 39; level 2 has 7, 23, 27 and 31. A fixed 40-byte layout would share its constant positions, so either the records have a variable tail or 40 is a period in something the records sit inside. Also measured: DAW shows no stride above threshold against its control, consistent with ADPCM nibble-packed audio and with the container framing being the only part worth decoding. gameplay.bin peaks at 16 bytes. gs_ram.bin has almost no stride structure but a lag-8 correlation of +0.503, so it is smooth rather than tabular. |
||
|
ReaperOfVeriod
|
51e81b497e |
Show the sky sub-object shape holds across levels
Range measured: sub-objects of the sky region on levels 0, 1, 3, 7, 13 and 26 -- 25 sub-objects, each over its whole length. Method: correlation at lag against the byte sequence, as before. lag 8 +0.53 .. +0.79 positive on all 25 lag 64 +0.16 .. +0.63 positive on all 25 lag 512 -0.13 .. +0.42 not universal lag 1024 -0.05 .. +0.37 not universal Lag 8 and lag 64 hold on every sub-object of every level tested. The longer lags do not, so the second scale seen in level 0's first sub-object is a property of that data rather than of the format. The sky sub-objects are all the same kind of thing: 8-byte units strongly related to their neighbours, with a further relation at 64 bytes. That is the first result about sky that holds everywhere rather than on one level. The 8-byte unit is still unnamed and the values still unidentified. |