The shrub mesh geometry is in the class blob after all
The open question from the shrub decode was where the mesh lives: the blob looked like header, palette and optional billboard, with the 333 classes without a billboard having nothing to draw. Answering it took the running game, not the disc. Diffing a loaded shrub class against its disc bytes in the live PCSX2 (level 1, class 1) shows what the loader rewrites: a table at +0x40 of (offset, size) pairs, whose offsets are span-relative on disc and absolute in RAM with the sizes unchanged -- all 21 entries matching offset for offset and size for size -- plus a relocated palette pointer at +0x2c and a pointer at +0x18, zero on disc, that asserts the parser enforces. So the blob is header, packet table, packets, [billboard], palette: the mesh was here the whole time. Why it was missed is the part worth recording -- data_offset points at the END of the span, so reading the body as data_offset..span_end accounts for the last few hundred bytes and leaves the 20 KB in front of it unexamined. A correct bound and a body small enough to look plausible are together enough to hide a region. Range measured: every shrub span on the disc, 358 over 24 levels, each bounded by its successor class or the located chunk container. Method: read count at +0x28, then the entries at +0x40, and require them to tile -- the first packet at align16(0x40 + count*8), every entry exactly size past the previous, and the last ending at the palette, or 0x40 short of it on the 25 spans that carry a billboard. Result: 358 of 358 tile exactly, holding 2,239 packets. That is what pins the field meanings: count is the packet count (1..55 on the disc), data_offset is the palette's offset, and the earlier "the body is not sized by count" note was true for a reason nobody had understood -- the count sizes the packets. Shrub::packets() exposes them and returns None rather than a partial list when a table does not tile. Still open: the packet contents (a VIF reading was already retracted once, so this needs a control rather than a pattern match), and the 0x20-byte records the loader points at from +0x18.
This commit is contained in:
@@ -1663,6 +1663,7 @@ fn shrub_headers_name_their_own_class_on_every_level() {
|
||||
let mut norm2_deviation = 0i64;
|
||||
let mut billboards = 0usize;
|
||||
let mut billboard_entries = 0usize;
|
||||
let mut packets_total = 0usize;
|
||||
|
||||
for n in 0..27 {
|
||||
let dir = data.join(format!("level{n}"));
|
||||
@@ -1787,6 +1788,21 @@ fn shrub_headers_name_their_own_class_on_every_level() {
|
||||
}
|
||||
padding_records += s.padding_record_count();
|
||||
|
||||
// The packets: the table at +0x40 tiles the span from
|
||||
// align16 of its own end to the palette, or to the
|
||||
// billboard when the class has one. This is what puts the
|
||||
// mesh geometry in the blob, and it is the identity that
|
||||
// makes `count` the packet count rather than a guess.
|
||||
let packets = s.packets().unwrap_or_else(|| {
|
||||
panic!("level{n} class {i}: packet table does not tile")
|
||||
});
|
||||
assert_eq!(
|
||||
packets.len(),
|
||||
s.header.count as usize,
|
||||
"level{n} class {i}: packet count disagrees with the header"
|
||||
);
|
||||
packets_total += packets.len();
|
||||
|
||||
// The billboard: present exactly when the class entry
|
||||
// carries billboard texture fields.
|
||||
let entry_has_billboard = e.billboard_width != 0 || e.billboard_height != 0;
|
||||
@@ -1840,6 +1856,7 @@ fn shrub_headers_name_their_own_class_on_every_level() {
|
||||
all {bounded_bodies} bodies bounded, {bounded_records} records, \
|
||||
{palette_records} of them palette normals and {padding_records} zero padding; \
|
||||
{billboards} billboards, all with texture fields; \
|
||||
{packets_total} packets across all blobs; \
|
||||
mode_bits {mode_hist:?}, radius {radius_range:?}, scale {scale_range:?}, \
|
||||
largest palette deviation from unit length {norm2_deviation} of a possible 65535"
|
||||
);
|
||||
|
||||
@@ -142,8 +142,9 @@ pub struct ShrubHeader {
|
||||
/// The class this geometry belongs to; equals the class entry's
|
||||
/// `o_class` on all 358 spans on the disc.
|
||||
pub o_class: u32,
|
||||
/// A count. Meaning not established (1..55 on the disc). It is *not* the
|
||||
/// body size: bodies hold 24 normals plus padding regardless.
|
||||
/// A count. It is the **packet count** -- the length of the table at
|
||||
/// `+0x40` -- which the packet tiling proves (see [`Shrub::packets`]).
|
||||
/// Ranges 1..55 on the disc; unrelated to the palette's length.
|
||||
pub count: u32,
|
||||
/// Offset of the geometry data from the start of the span. Lands inside
|
||||
/// the span on all 358 spans on the disc.
|
||||
@@ -201,6 +202,16 @@ pub struct Shrub<'a> {
|
||||
pub span_end_is_a_successor: bool,
|
||||
}
|
||||
|
||||
/// One mesh packet: where it sits in the span and how long it is.
|
||||
///
|
||||
/// The offsets are relative to the start of the span, like every other
|
||||
/// pointer in a class blob.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub struct ShrubPacket {
|
||||
pub offset: usize,
|
||||
pub size: usize,
|
||||
}
|
||||
|
||||
impl<'a> Shrub<'a> {
|
||||
/// The palette's 8-byte records, raw.
|
||||
pub fn body_records(&self) -> impl Iterator<Item = &'a [u8]> {
|
||||
@@ -212,6 +223,51 @@ impl<'a> Shrub<'a> {
|
||||
self.data.len() / 8
|
||||
}
|
||||
|
||||
/// The mesh packets, as the packet table at `+0x40` names them.
|
||||
///
|
||||
/// The table is `count` entries of `(offset, size)`, both relative to the
|
||||
/// span start. What makes it a table rather than a guess is that the
|
||||
/// entries **tile the blob**: the first packet starts at
|
||||
/// `align16(0x40 + count * 8)` -- the table, rounded up to 16 -- every
|
||||
/// entry is exactly `size` past the previous one, and the last ends at the
|
||||
/// palette, or exactly 0x40 short of it on the spans that carry a
|
||||
/// billboard record in between. That holds on **358 of 358** spans on the
|
||||
/// disc, and it is what pins the field meanings: `count` is the packet
|
||||
/// count, and `data_offset` is the palette's offset.
|
||||
///
|
||||
/// Returns `None` when the entries do not tile, rather than handing back
|
||||
/// a partial list: a table that stops making sense means this is not
|
||||
/// laid out the way this thinks it is.
|
||||
pub fn packets(&self) -> Option<Vec<ShrubPacket>> {
|
||||
let count = self.header.count as usize;
|
||||
// A count that cannot be read out of the span is not a count.
|
||||
if self.raw.len() < 0x40 + count.saturating_mul(8) {
|
||||
return None;
|
||||
}
|
||||
let mut out = Vec::with_capacity(count);
|
||||
let mut cursor = (0x40 + count.saturating_mul(8) + 0xf) & !0xf;
|
||||
for i in 0..count {
|
||||
let at = 0x40 + i * 8;
|
||||
let entry = self.raw.get(at..at + 8)?;
|
||||
let offset = u32::from_le_bytes(entry[..4].try_into().ok()?) as usize;
|
||||
let size = u32::from_le_bytes(entry[4..].try_into().ok()?) as usize;
|
||||
if offset != cursor {
|
||||
return None;
|
||||
}
|
||||
cursor += size;
|
||||
out.push(ShrubPacket { offset, size });
|
||||
}
|
||||
// The packets end where the palette begins, leaving room for the
|
||||
// billboard record when the class has one.
|
||||
let palette = self.header.data_offset as usize;
|
||||
let expected_end = if self.header.billboard_offset > 0 {
|
||||
palette.checked_sub(0x40)?
|
||||
} else {
|
||||
palette
|
||||
};
|
||||
(cursor == expected_end).then_some(out)
|
||||
}
|
||||
|
||||
/// The palette: the first [`SHRUB_NORMAL_COUNT`] records, parsed.
|
||||
///
|
||||
/// Returns fewer only if the body is shorter than the palette, which does
|
||||
@@ -535,6 +591,89 @@ mod tests {
|
||||
assert!(!s.normals().any(|n| n.is_unit()));
|
||||
}
|
||||
|
||||
/// A span laid out the way a disc blob is: header, packet table, packets,
|
||||
/// then the palette at the end.
|
||||
fn span_with_packets(palette_at: usize) -> Vec<u8> {
|
||||
let mut v = vec![0u8; BASE + palette_at + SHRUB_NORMAL_COUNT * 8];
|
||||
v[BASE + 0x0c..BASE + 0x10].copy_from_slice(&12.5f32.to_le_bytes());
|
||||
v[BASE + 0x10..BASE + 0x14].copy_from_slice(&6.0f32.to_le_bytes());
|
||||
v[BASE + 0x20..BASE + 0x24].copy_from_slice(&0.25f32.to_le_bytes());
|
||||
v[BASE + 0x24..BASE + 0x28].copy_from_slice(&1072u32.to_le_bytes());
|
||||
v[BASE + 0x2c..BASE + 0x30].copy_from_slice(&(palette_at as u32).to_le_bytes());
|
||||
for (i, n) in (0..SHRUB_NORMAL_COUNT).map(fixture_normal).enumerate() {
|
||||
let at = BASE + palette_at + i * 8;
|
||||
for (c, v_c) in n.iter().enumerate() {
|
||||
let q = (v_c * 32767.0).round() as i16;
|
||||
v[at + c * 2..at + c * 2 + 2].copy_from_slice(&q.to_le_bytes());
|
||||
}
|
||||
}
|
||||
v
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_packets_tile_the_blob() {
|
||||
// Three packets tiling from align16(0x40 + 3*8) = 0x60 to the palette.
|
||||
let palette_at = 0x300usize;
|
||||
let mut c = span_with_packets(palette_at);
|
||||
c[BASE + 0x28..BASE + 0x2c].copy_from_slice(&3u32.to_le_bytes());
|
||||
let entries = [
|
||||
(0x60usize, 0x40usize),
|
||||
(0xa0, 0x30),
|
||||
(0xd0, palette_at - 0xd0),
|
||||
];
|
||||
for (i, (o, sz)) in entries.iter().enumerate() {
|
||||
let at = BASE + 0x40 + i * 8;
|
||||
c[at..at + 4].copy_from_slice(&(*o as u32).to_le_bytes());
|
||||
c[at + 4..at + 8].copy_from_slice(&(*sz as u32).to_le_bytes());
|
||||
}
|
||||
let s = ShrubHeader::region(&c, BASE as i32, 1072, c.len())
|
||||
.unwrap()
|
||||
.expect("present");
|
||||
let packets = s.packets().expect("the entries tile");
|
||||
assert_eq!(packets.len(), 3);
|
||||
assert_eq!(packets[0].size, 0x40);
|
||||
assert_eq!(packets[2].size, palette_at - 0xd0);
|
||||
// The palette is untouched by the packets sitting before it.
|
||||
assert_eq!(s.normals().count(), SHRUB_NORMAL_COUNT);
|
||||
assert!(s.normals().all(|n| n.is_unit()));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_billboard_record_sits_between_the_packets_and_the_palette() {
|
||||
// With a billboard, the packets end 0x40 short of the palette.
|
||||
let palette_at = 0x300usize;
|
||||
let mut c = span_with_packets(palette_at);
|
||||
c[BASE + 0x28..BASE + 0x2c].copy_from_slice(&1u32.to_le_bytes());
|
||||
// One entry: the table ends at 0x48, aligned up to 0x50.
|
||||
let size = palette_at - 0x40 - 0x50;
|
||||
c[BASE + 0x40..BASE + 0x44].copy_from_slice(&0x50u32.to_le_bytes());
|
||||
c[BASE + 0x44..BASE + 0x48].copy_from_slice(&(size as u32).to_le_bytes());
|
||||
// A billboard offset inside the span, in its last 0x40 bytes.
|
||||
c[BASE + 0x1c..BASE + 0x20].copy_from_slice(&((palette_at - 0x40) as u32).to_le_bytes());
|
||||
let s = ShrubHeader::region(&c, BASE as i32, 1072, c.len())
|
||||
.unwrap()
|
||||
.expect("present");
|
||||
let packets = s.packets().expect("the entry stops at the billboard");
|
||||
assert_eq!(packets.len(), 1);
|
||||
assert_eq!(packets[0].size, palette_at - 0x40 - 0x50);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_packet_table_that_does_not_tile_is_rejected() {
|
||||
let palette_at = 0x300usize;
|
||||
let mut c = span_with_packets(palette_at);
|
||||
c[BASE + 0x28..BASE + 0x2c].copy_from_slice(&2u32.to_le_bytes());
|
||||
// The second entry starts mid-packet, not where the first ended.
|
||||
c[BASE + 0x40..BASE + 0x44].copy_from_slice(&0x60u32.to_le_bytes());
|
||||
c[BASE + 0x44..BASE + 0x48].copy_from_slice(&0x40u32.to_le_bytes());
|
||||
c[BASE + 0x48..BASE + 0x4c].copy_from_slice(&0x61u32.to_le_bytes());
|
||||
c[BASE + 0x4c..BASE + 0x50].copy_from_slice(&0x40u32.to_le_bytes());
|
||||
let s = ShrubHeader::region(&c, BASE as i32, 1072, c.len())
|
||||
.unwrap()
|
||||
.expect("present");
|
||||
assert!(s.packets().is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_billboard_record_is_read_and_checked() {
|
||||
let mut c = span_with_tail(0x40 + 8);
|
||||
|
||||
@@ -1123,12 +1123,58 @@ three classes on different levels carry identical `fade`, floats, K and scale
|
||||
blobs themselves are shared between levels, which is the same behaviour the
|
||||
chunk container showed with its byte-identical tables.
|
||||
|
||||
### The packets: the mesh geometry is in the blob (2026-10-11, later)
|
||||
|
||||
The open question above -- where the mesh geometry lives -- answered itself
|
||||
once the running game was consulted. The running PCSX2 loads shrub classes
|
||||
into RAM, and diffing a loaded blob against its disc bytes shows what the
|
||||
loader rewrites:
|
||||
|
||||
- **`+0x40` holds a table of `count` entries of `(offset, size)`**, both
|
||||
relative to the span start on disc, rewritten to absolute addresses in RAM
|
||||
with the sizes unchanged. Verified on level 1 class 1: all 21 entries match
|
||||
the disc offset for offset and the size for size after adding the blob's
|
||||
RAM base.
|
||||
- **`+0x2c` (the palette offset) is relocated** the same way, to an absolute
|
||||
pointer.
|
||||
- **`+0x18` -- zero on disc, asserted by the parser -- is written with a
|
||||
pointer at load**, to a table of 0x20-byte records that look instance-shaped
|
||||
(a position, a size, a count, and a `0xffff` light slot). Not established;
|
||||
recorded because it is where the next question starts.
|
||||
|
||||
So the blob is header, packet table, packets, [billboard], palette -- the same
|
||||
shape as R&C1's, and the mesh was here the whole time. The reason it was
|
||||
missed is worth writing down: **`data_offset` points at the *end* of the
|
||||
span**, so reading the body as `data_offset..span_end` accounts for the last
|
||||
few hundred bytes and leaves the 20 KB in front of it unexamined. A bound that
|
||||
is correct and a body that is small enough to look plausible are together
|
||||
enough to hide a region.
|
||||
|
||||
**Range measured:** every shrub class span on the disc -- 358 spans over 24
|
||||
levels, each bounded by its successor class or the located chunk container.
|
||||
|
||||
**Method:** for each span, read `count` from `+0x28`, then the entries at
|
||||
`+0x40`, and require them to tile: the first packet starts at
|
||||
`align16(0x40 + count * 8)` -- the table, rounded up to 16 -- every entry is
|
||||
exactly `size` past the previous one, and the last ends at the palette, or
|
||||
exactly 0x40 short of it on the spans that carry a billboard record in
|
||||
between.
|
||||
|
||||
**Result: 358 of 358 tile exactly, holding 2,239 packets.** No span's
|
||||
deviations: the first-packet offsets, the sizes and the ends all agree with
|
||||
the rule. That is what pins the field meanings -- `count` is the **packet
|
||||
count** (1..55 on the disc), and `data_offset` is the palette's offset -- and
|
||||
it is why the earlier "the body is not sized by count" observation was true
|
||||
for a reason that had not been understood: the body is not sized by the count
|
||||
because the count sizes the packets.
|
||||
|
||||
`gc_formats::shrub::Shrub::packets()` exposes them, and returns `None` rather
|
||||
than a partial list when a table does not tile.
|
||||
|
||||
### Still open
|
||||
|
||||
- The three billboard floats, `size`, `mode_bits`, and `count`.
|
||||
- **Where shrub mesh geometry lives.** R&C1 keeps the class's mesh packets in
|
||||
the blob between the header and the palette; R&C2's blob is header, palette
|
||||
and optional billboard only -- 24 normals and nothing else. The 333 classes
|
||||
without a billboard still have to draw a mesh, and it is not here. The
|
||||
candidates are the chunk container that follows shrub and the gameplay
|
||||
extent, and neither is decoded.
|
||||
- The three billboard floats, `size`, and `mode_bits` (4 on 16 spans).
|
||||
- The packet contents. They are VIF command lists in R&C1; the R&C2 packets
|
||||
have not been decoded, and the VIF reading of a shrub body was already
|
||||
retracted once, so this needs a control, not a pattern match.
|
||||
- The 0x20-byte records the loader points at from `+0x18.
|
||||
|
||||
@@ -235,6 +235,11 @@ Phase 1 hands over the extents; this phase works out what each one contains.
|
||||
three floats in the billboard record, `size`, `mode_bits`, `count`, and
|
||||
**where the mesh geometry lives** — R&C1 puts the mesh packets in this
|
||||
blob and R&C2's blob holds only the palette.
|
||||
- [x] **The shrub mesh geometry is in the class blob.** A packet table at
|
||||
`+0x40` names `count` packets between the table and the palette, and
|
||||
they tile on **358 of 358 spans** (2,239 packets), which makes `count`
|
||||
the packet count. Confirmed against the running game, which relocates
|
||||
the table's offsets to absolute addresses and `data_offset` likewise.
|
||||
- [x] `moby` — per-class model geometry. A class span opens with a `u32`
|
||||
pointing at a part table of 16-byte records; parts tile exactly, with
|
||||
`vertex_offset + 16 * vertex_count == next_part` holding for **43,467 of
|
||||
|
||||
Reference in New Issue
Block a user