ReaperOfVeriod/Going-Commando
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Resolve the shrub region bound: the chunk container closes it

Browse Source 
The shrub region's end was the last unlocated bound in core_data. It was not
in the core index, and unlike moby->tie it had no successor boundary to chain
against, so the last shrub class in every level ran out to the lump's end and
the aggregates over it (the "3.5M records / 28MB of shrub body") were
artifacts of that oversized span.

Range measured: for each level, from the last shrub class's geometry offset to
the constant trailer. Nothing outside that range is read.

Method: scan forward from the last shrub class for the pair (a) the constant
two bytes f6 7c sitting immediately in front of (b) a chunk signature -- a
16-byte header ending 0xff, ten zero bytes, then four or more 0xf0-tagged
offsets. The two halves are each weak alone, and the halves are measured
separately: the marker pair occurs 25 to 103 times in a 5 to 17 MB lump, so
matching it alone picks dozens of wrong offsets; the chunk shape matches
random data often enough to need four tagged offsets before it is believed.

Result: on all 27 levels the marker sits exactly two bytes before the first
chunk header, and the two parts of the signature are never found apart. No
chunk signature occurs inside any last shrub body, on any level. The three
levels without shrub (10, 25, 26) carry the same marker before the container
that closes tie instead, which is what makes the locator a property of the
region rather than a shrub rule.

Consequences:

- All 358 shrub class bodies are bounded and are whole 8-byte records. That
  was not assertable while the last class in each level ran to the lump's end,
  and it is the check the boundary earns its keep for.
- 9,600 records in total, against the 8,938 previously measurable in
  successor-bounded spans; 662 of them sit in the 24 last-class bodies. The
  shrub region totals 2,033,776 bytes across the 24 levels that have one.
- The chunk container is 25,775,850 bytes across 27 levels (1,236 chunks,
  22,562 records). The container measurements were previously taken over "the
  900,000 bytes before the trailer", a window that did not align to its real
  extent: on level 0 it is 690,302 bytes, and 27 of 1,123 chunks were
  truncated at that invented edge. chunk_containers_resolve_on_every_level now
  reads the region from core_layout instead of measuring backwards from the
  end of the lump.

core_layout takes core_data now: the container's start is the one bound that
cannot be derived from the index. It gains the chunks region, so every byte of
level-specific core_data belongs to a named region. Where the signature is not
found the end stays None -- the region is left visible rather than clamped to
the tail.

Still open: the marker's meaning (constant across 27 levels, at exactly this
position, unnamed) and the shrub body payload's encoding.
This commit is contained in:
ReaperOfVeriod
2026-10-11 06:20:48 +02:00
parent 4d1e277a15
commit ccea53807a
8 changed files with 681 additions and 117 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/formats/core-data-regions.md
+69 -14
View File
@@ -27,8 +27,10 @@ than inventing their own.
| `sky` | `sky` | `collision` |
| `collision`| `collision` | first moby class |
| `moby` | first moby | first tie class |
| `tie` | first tie | first shrub class |
| `shrub` | first shrub | lump length − 0x38A10 |
| `tie` | first tie | first shrub class, |
| | | else chunk container |
| `shrub` | first shrub | chunk container |
| `chunks` | chunk container| lump length − 0x38A10|
Class regions start at the lowest non-zero geometry offset in their class
table. A zero entry means *absent*, not position zero -- the same trap the
@@ -43,26 +45,79 @@ Comparing all 27 levels byte-for-byte from the end: the last **231,952 bytes
therefore ends at `len - 0x38A10`, and the final region is bounded there rather
than at `core_data.len()`.
This is what closes shrub. Across the 24 levels that have a shrub region it
gives **24,442,928 bytes** total, and it is the same bound the disc test
asserts on every level.
This is what closes the chunk container. The regions it separates from level
content are measured on the disc: **2,033,776 bytes of shrub region across the
24 levels that have one, and 25,775,850 bytes of chunk container** across all
27. The disc test asserts the bound on every level.
What the trailer *is* is not established. It may be fixed data, or padding
from a fixed-size decompression buffer. Either way it is constant, which is
all that is needed to bound the tail.
## Open: what was in the region before it
## Found: the chunk container closes the class region
The offset ending the shrub region is not in the core index.
The offset ending the shrub region is not in the core index, and the class
table that precedes the container carries no successor offset for it -- unlike
moby and tie, whose boundaries fall out of the tables. Texture texels do not
bound it either: they sit before moby, not after shrub. So the boundary is
located in `core_data` itself.
Texture texels do **not** bound it. They sit before moby, not after shrub: on
level 0 the texel range is 3539328..5326208 while the last shrub class starts
at 16044288, leaving ~925 KB of real, non-zero data after the last shrub class
that nothing yet accounts for.
**Range measured:** for each level, from the last shrub class's geometry
offset to the constant trailer. Nothing outside that range is read.
`core_layout` reports `end: None` for shrub on the 24 levels that have one, and
the disc test asserts it, so the gap stays visible rather than being papered
over by clamping to the end of the lump.
**Method:** scan forward from the last shrub class for the pair (a) the
constant two bytes `f6 7c`, sitting immediately in front of (b) a chunk
signature -- a 16-byte header ending `0xff`, ten zero bytes, then four or more
`0xf0`-tagged offsets. `gc_formats::chunks::chunk_region_start`.
Each half of that signature is weak on its own, which is why the halves are
measured separately:
- the pair `f6 7c` occurs **25 to 103 times** in a 5 to 17 MB lump, so
matching it alone picks dozens of wrong offsets;
- the chunk shape matches random data often enough that `parse_chunk` demands
four tagged offsets before a signature is believed.
**Result: on all 27 levels the pair sits exactly two bytes before the first
chunk header.** The two are never found apart. This is the boundary, and
`core_layout` closes shrub with it -- `shrub`'s end is the container's start,
and the container's own end is the trailer, so every byte of level-specific
`core_data` now belongs to a named region.
Two controls, both asserted by the disc test `chunk_container_closes_the_last_class_region`:
- **No signature inside the body.** Scanning the last shrub body for a bare
chunk signature finds none, on any level -- so the boundary search was not
matching something it should have skipped, and the marker's job is
corroboration rather than load-bearing disambiguation.
- **The three levels without shrub** (10, 25, 26) carry the same marker two
bytes before the container. There the container closes *tie* instead, which
is what makes the locator a property of the region rather than a shrub
rule.
On level 0:
| what | range | bytes |
|---|---|---|
| last shrub class data | 16046800 .. 16047040 | 240 |
| marker `f6 7c` | 16047040 .. 16047042 | 2 |
| **chunk container** | 16047042 .. 16737344 | **690,302** |
| constant trailer | 16737344 .. 16969296 | 231,952 |
The marker's meaning is not established. It is constant across all 27 levels
and it appears at exactly this position, but whether it is a terminator, an end
count or a tag is unknown, so it is not named.
**Correction this produces.** The chunk-container measurements recorded below
were taken over "the 900,000 bytes before the constant trailer" -- a window
wide enough to contain the container on every level but not aligned to its
true extent. Chunks sitting at the window's edge were truncated (27 of 1,123),
and the region's size was never correctly stated. Both are now bounded
properly: on level 0 the container is 690,302 bytes, not ~900,000, and the
25,775,850 bytes of chunk data across 27 levels are the measured total rather
than a windowed sample. The `chunk_containers_resolve_on_every_level` test now
reads the region from `core_layout` instead of measuring backwards from the
end of the lump.
## Correction: it is mostly texture data, not an unnamed region
docs/plan/level-assets.md
+43
View File
@@ -961,3 +961,46 @@ This is the second time in this format that a confident aggregate hid a bad
bound -- the first was the VIF reading. The pattern is worth noting: aggregates
across many spans are only as trustworthy as the span bounds underneath them,
and those are the easy thing to get wrong because nothing errors.
## Shrub span bounds: resolved (2026-10-11, later still)
The offset that ends the shrub region is the **start of the chunk container**,
and it is found by signature rather than read from the core index, which has no
successor offset for shrub.
**Range measured:** for each level, from the last shrub class's geometry offset
to the constant trailer. Nothing outside that range is read.
**Method:** scan forward from the last shrub class for two bytes `f6 7c`
sitting immediately in front of a chunk signature (16-byte header ending `0xff`,
ten zero bytes, four or more `0xf0`-tagged offsets). Each half of that
signature is weak alone -- the marker pair occurs 25 to 103 times per lump --
and together they are exact.
**Result: on all 27 levels the marker sits exactly two bytes before the first
chunk header.** The two parts of the signature are never found apart, and no
chunk signature occurs inside any last shrub body. The three levels without
shrub (10, 25, 26) carry the same marker before the container that closes
*tie*, so the locator is a property of the region, not a shrub rule.
`gc_formats::chunks::chunk_region_start` implements it and
`regions::core_layout` closes shrub with it.
Consequences, all measured over the new bounds:
- **All 358 shrub class bodies are bounded and well formed** -- every one is
whole 8-byte records. That was not assertable while the last class in each
level ran out to the lump's end, and it is the check the boundary earns.
- **9,600 records in total**, against the 8,938 previously measurable in
successor-bounded spans; 662 of them sit in the 24 last-class bodies, 24 to
30 per level. The retracted "3.5 million records / 28 MB of shrub body"
figure stays retracted: the correct total for the shrub region is
**2,033,776 bytes** across the 24 levels that have one.
- The chunk container is **25,775,850 bytes across 27 levels** (1,236 chunks,
22,562 records), not an ~690-900 KB afterthought per level. The earlier
container measurements were taken over "the 900,000 bytes before the
trailer", a window that did not align to the container's real extent: on
level 0 it is 690,302 bytes, and 27 of 1,123 chunks were truncated at that
invented edge. The chunk test now reads the region from `core_layout`.
What is still not known: the marker's meaning (constant across 27 levels, at
exactly this position, unnamed), and the shrub body payload's encoding.
docs/roadmap.md
+16 -3
View File
@@ -1,6 +1,6 @@
# Roadmap
Status as of **2026-10-09**.
Status as of **2026-10-11**.
A from-scratch native Rust reimplementation of Ratchet & Clank: Going Commando
(PS2). Each phase ends in a **gate** — something checkable, not a feeling that
@@ -57,6 +57,14 @@ early, while they are still cheap to retire.
through `LightTfrags`. The renderer still counts and reports how many
vertices it lit, because a silent fallback to stored colours would look
plausible and be wrong.
- [x] **The shrub region is bounded, and with it the whole lump.** Shrub's end
was the last unlocated region bound: it is the start of the post-class
chunk container, found by a marker-paired signature in `core_data` (the
index carries no successor offset). The container closes shrub on the
24 shrub levels and tie on the three without. All 358 shrub class bodies
are now bounded and are whole 8-byte records, so shrub measurements are
trustworthy for the first time -- the "3.5M records" figure from the
unbounded final span is replaced by a measured 9,600.
- [ ] Lighting *magnitudes* are unverified: they need comparison against
reference frames, which needs the PCSX2 comparison loop.
- [ ] Then the GS register set and the guard-band clip test.
@@ -153,6 +161,9 @@ Phase 1 hands over the extents; this phase works out what each one contains.
- [x] Region layout of `core_data` established: tfrag, occlusion, sky, collision
and `textures_base` are offsets, strictly increasing on all 27 levels.
Terrain is `[0, occlusion)` and carries inline texels.
- [x] Region layout **closed**: the chunk container's marker-paired signature
ends the last class region, and the constant trailer ends the container.
Every byte of level-specific `core_data` now belongs to a named region.
- [x] **tfrag vertex format: recovered.** The tfrags block header gives a table
offset (`0x40`) and a patch count; each patch is a 0x40-byte header whose
`data` offset is relative to the table, not to the block. Positions are
@@ -209,8 +220,10 @@ Phase 1 hands over the extents; this phase works out what each one contains.
Geometry header parsed and both fields **checked**, not merely reported:
`+0x24` equals the entry's own `o_class`, and `+0x2c` is an in-span data
offset, on all 358 spans. Shrub does **not** use moby's part-table format.
The body past `+0x2c` is not decoded — it tiles at no stride consistent
with the count at `+0x28`.
The body is **8-byte records, 6 varying bytes and 2 constant**, and now
that every span is bounded by the located chunk container, all 358
bodies are whole records — 9,600 in total. The payload's encoding is
still unknown, and so is the 2-byte marker that closes the region.
- [x] `moby` — per-class model geometry. A class span opens with a `u32`
pointing at a part table of 16-byte records; parts tile exactly, with
`vertex_offset + 16 * vertex_count == next_part` holding for **43,467 of