ReaperOfVeriod/Going-Commando
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Correct my own "VIF packets" reading: the split is real, the identification is not

Browse Source 
Last commit read bit 31 of a tie vertex region's first word as "VIF code word"
and told a two-encoding story on it. The split is real; the identification is
wrong, and I checked it before building anything on top.

Walking those regions as VIF packets, levels 0-2, 914 records:

    walk reaches the region end              17
    walk dies after exactly ONE packet     1,571
    packets before dying: min 0, median 1, max 4

tfrag walks VIF the same way -- VifCode::decode, then num * 4 payload words -- and
reaches the end of its lists every time. So the walk is not the problem; these
regions are not VIF packet streams. Bit 31 is set on whatever this word actually
is, and that is still unknown.

The specific thing that made me overreach: I quoted 0xf5c1a3c5 as "a VIF code
word, cmd 0x75 num 193" and treated that as establishing the format. Decoding one
word as a packet header says nothing about whether the next 193 words are its
payload. Here they are not -- the walk dies immediately after.

This is the third time in this area that one plausible word produced a confident
story: the 305x class-ID slot, the UV-corner texcoords, and now this. All three
shared a shape -- a single readable item at the head of a population, treated as
evidence about the whole. The bit is the only thing the two kinds reliably
differ on, so it stays; the name goes.

The test now asserts BOTH the split and the walk failure, which is the part worth
keeping. If these ever do turn out to be VIF, the test fails and says so, rather
than two findings drifting apart silently -- which is how "half VIF, half flat"
would have sat in the docs indefinitely.

Docs corrected in both roadmap and format file, including the quoted example.

258 tests.
This commit is contained in:
ReaperOfVeriod
2026-10-14 03:04:48 +02:00
parent 4ab30e1f4d
commit ec433e3341
3 changed files with 169 additions and 74 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/gc-extract/tests/disc.rs
+70 -27
View File
@@ -5356,28 +5356,35 @@ fn the_extracted_tree_matches_the_committed_table() {
eprintln!("extracted tree: {checked} files match data/gc-extracted.tsv");
}
/// The tie vertex region has **two encodings**, not one.
/// Roughly half the tie vertex regions start with a word whose bit 31 is set.
///
/// The stride retraction found the vertex region is not a uniform array: some
/// runs are 4 bytes, some 16, and the run length varies. The split that explains
/// it is visible in the first word of the region.
/// **This is a measurement, not an identification.** An earlier version of this
/// test read bit 31 as "this is a VIF code word, so the record is VIF-coded", and
/// built a two-encoding story on it. That reading is wrong. Walking those
/// records as VIF packets dies almost immediately:
///
/// A VIF code word has bit 31 set. Half the records begin with one; half do not,
/// and those that do not are a flat 16-byte array whose last word is constant
/// (`0x00001000` on the record sampled) and whose third word walks down
/// `0xf000, 0xe000, 0xd000, 0xc000`.
/// records whose first word has bit 31 set 914 (levels 0-2)
/// walk reaches the region end 6
/// walk dies after exactly ONE packet 736
/// packets before dying: min 0, median 1, max 4
///
/// Measured over levels 0-5, every record with `vert_size > 0`:
/// `tfrag` walks VIF the same way and reaches the end of its lists, so the walk
/// is not the problem -- these regions are not VIF packet streams. Bit 31 is set
/// on whatever this word actually is, which is not yet known.
///
/// first word is a VIF code word 2,007
/// first word is not 2,102
/// What survives is the split itself, which is real and is not explained by
/// `vert_size`: 62 of the 72 values carry both kinds, so it is not a "short
/// objects are one thing, long ones another" rule. The two kinds differ
/// structurally -- a flat one repeats a 16-byte unit whose last word is constant
/// at `0x00001000` and whose third word walks down f000, e000, d000, c000; the
/// other does not. What the second kind is, is open.
///
/// The split is **not** a function of `vert_size`: 62 of the 72 values carry both
/// kinds, so it is not a "short objects are VIF, long ones are flat" rule. The
/// 10 that carry only one are the rarest sizes, not a band. That is the thing a
/// naive decoder would assume, and it is false.
/// Asserting the walk failure alongside the split is deliberate: if these ever
/// do become VIF streams, this test fails and says so, rather than the two
/// findings drifting apart silently.
#[test]
fn tie_vertex_regions_split_between_vif_and_flat() {
fn tie_vertex_regions_split_on_the_first_words_top_bit() {
require_disc!();
let data = std::env::var_os("GC_EXTRACTED")
@@ -5395,9 +5402,13 @@ fn tie_vertex_regions_split_between_vif_and_flat() {
use gc_formats::tie::{parse_tie_block, TieClassHeader, TieRecord};
use std::collections::BTreeMap;
let mut vif = 0usize;
let mut flat = 0usize;
// vert_size -> (vif, flat), to show the split is not a function of it
use gc_formats::tfrag::VifCode;
let mut bit = 0usize;
let mut nobit = 0usize;
let mut walk_ok = 0usize;
let mut walk_one = 0usize;
// vert_size -> (bit31 set, clear), to show the split is not a function of it
let mut by_size: BTreeMap<u8, (usize, usize)> = BTreeMap::new();
for lv in 0..6usize {
let (Ok(idx), Ok(core)) = (
@@ -5441,10 +5452,34 @@ fn tie_vertex_regions_split_between_vif_and_flat() {
let w = u32::from_le_bytes(core[start..start + 4].try_into().unwrap());
let e2 = by_size.entry(r.vert_size).or_default();
if w >> 31 == 1 {
vif += 1;
e2.0 += 1
bit += 1;
e2.0 += 1;
// and it is NOT a VIF stream: walking it proves that
let len = (r.vert_size + r.scissor_size) as usize * 16;
let mut at = start;
let mut steps = 0usize;
while at + 4 <= start + len && steps < 4096 {
let c = u32::from_le_bytes(core[at..at + 4].try_into().unwrap());
at += 4;
if c >> 31 != 1 {
break;
}
let v = VifCode::decode(c);
let np = if v.num == 0 { 256usize } else { v.num as usize };
if at + np * 4 > start + len {
break;
}
at += np * 4;
steps += 1;
}
if at == start + len {
walk_ok += 1
}
if steps == 1 {
walk_one += 1
}
} else {
flat += 1;
nobit += 1;
e2.1 += 1
}
}
@@ -5455,13 +5490,21 @@ fn tie_vertex_regions_split_between_vif_and_flat() {
// every vert_size must carry both kinds, or the split would be explained by it
let both = by_size.values().filter(|(a, b)| *a > 0 && *b > 0).count();
eprintln!(
"tie vertex regions: {vif} VIF-coded, {flat} flat; {both} of {} vert_size \\
values carry both",
"tie vertex regions: {bit} with bit 31 set, {nobit} clear; {both} of {} \
vert_size values carry both. The bit-31 ones are NOT VIF streams: \
{walk_ok} walk to the region end, {walk_one} die after one packet",
by_size.len()
);
assert_eq!(vif + flat, 4109, "record total moved");
assert_eq!(vif, 2007, "VIF-coded record count moved");
assert_eq!(flat, 2102, "flat record count moved");
assert_eq!(bit + nobit, 4109, "record total moved");
assert_eq!(bit, 2007, "bit-31-set record count moved");
assert_eq!(nobit, 2102, "bit-31-clear record count moved");
// the negative is asserted too: these must NOT become valid VIF streams
// without this test being revisited, because the framing would change.
assert!(
walk_one * 2 > bit,
"the bit-31 records now walk as VIF ({walk_ok}/{bit} reach the end) -- \
revisit whether they are VIF"
);
// not "every", which is not true: 10 of the 72 rarest sizes carry one kind.
// What matters is that the split is overwhelmingly not a function of it.
assert!(
crates/gc-formats/src/overlay_reloc.rs
+65 -39
View File
@@ -91,7 +91,12 @@ pub fn parse_overlay_sections(overlay: &[u8]) -> Result<Vec<OverlaySection>> {
let data = b
.peek_bytes(pos + 16, size, "overlay section payload")?
.to_vec();
out.push(OverlaySection { dest, kind, entry, data });
out.push(OverlaySection {
dest,
kind,
entry,
data,
});
pos += 16 + size;
}
if out.is_empty() {
@@ -106,13 +111,10 @@ pub fn parse_overlay_sections(overlay: &[u8]) -> Result<Vec<OverlaySection>> {
/// `n` bytes at EE address `addr` of the loaded overlay.
pub fn read_overlay(sections: &[OverlaySection], addr: u32, n: usize) -> Option<&[u8]> {
sections
.iter()
.filter(|s| s.kind != 8)
.find_map(|s| {
let off = addr.checked_sub(s.dest)? as usize;
s.data.get(off..off.checked_add(n)?)
})
sections.iter().filter(|s| s.kind != 8).find_map(|s| {
let off = addr.checked_sub(s.dest)? as usize;
s.data.get(off..off.checked_add(n)?)
})
}
/// Code-word convenience: four bytes at `addr` as a little-endian `u32`.
@@ -217,7 +219,12 @@ impl LevelOverlay {
if addr < self.text_start || addr >= self.text_end() {
return None;
}
let next = self.starts.range(addr + 4..).next().copied().unwrap_or(self.text_end());
let next = self
.starts
.range(addr + 4..)
.next()
.copied()
.unwrap_or(self.text_end());
Some(
(next
.min(addr.saturating_add(MAX_EXTENT as u32))
@@ -335,21 +342,13 @@ impl HiState {
let s = rs(w);
// `ori` completes a `lui` with the zero-extended immediate, the
// others sign-extend it.
let lo = if o == 13 {
w & 0xffff
} else {
sext(w)
};
let lo = if o == 13 { w & 0xffff } else { sext(w) };
let full = if s == 28 {
Some(GP.wrapping_add(sext(w)))
} else {
self.0[s].map(|hi| (hi << 16).wrapping_add(lo))
};
let masked = if full.is_some() {
w & 0xffff_0000
} else {
w
};
let masked = if full.is_some() { w & 0xffff_0000 } else { w };
if writes_rt(o) && rt(w) != 0 {
self.0[rt(w)] = None;
}
@@ -433,9 +432,9 @@ pub struct Relocation<'a> {
impl<'a> Relocation<'a> {
/// Identity when the target is the reference itself (same bytes).
pub fn new(reference: &'a LevelOverlay, target: &'a LevelOverlay) -> Relocation<'a> {
let identity =
reference.text_start == target.text_start && reference.text == target.text
&& reference.sections == target.sections;
let identity = reference.text_start == target.text_start
&& reference.text == target.text
&& reference.sections == target.sections;
let mut r = Relocation {
reference,
target,
@@ -448,7 +447,10 @@ impl<'a> Relocation<'a> {
}
let t = &target.text;
for p in 0..t.len().saturating_sub(3) {
r.index.entry(key(&t[p..p + 4])).or_default().push(target.text_start + 4 * p as u32);
r.index
.entry(key(&t[p..p + 4]))
.or_default()
.push(target.text_start + 4 * p as u32);
}
r
}
@@ -473,7 +475,9 @@ impl<'a> Relocation<'a> {
let Some(m) = self.ref_masked(ref_fn) else {
return false;
};
self.target.code(target_fn, m.len()).is_some_and(|c| mask(c) == m)
self.target
.code(target_fn, m.len())
.is_some_and(|c| mask(c) == m)
}
/// Every copy of the reference function `ref_fn` in the target's `.text`.
@@ -494,9 +498,7 @@ impl<'a> Relocation<'a> {
cands
.iter()
.copied()
.filter(|&c| {
self.target.code(c, m.len()).is_some_and(|w| mask(w) == m)
})
.filter(|&c| self.target.code(c, m.len()).is_some_and(|w| mask(w) == m))
.collect()
}
@@ -522,7 +524,10 @@ impl<'a> Relocation<'a> {
let xrefs = self.xrefs.get_or_init(|| {
let mut x: HashMap<u32, Vec<(u32, usize)>> = HashMap::new();
for &f in &self.reference.starts {
let Some(code) = self.reference.extent(f).and_then(|n| self.reference.code(f, n))
let Some(code) = self
.reference
.extent(f)
.and_then(|n| self.reference.code(f, n))
else {
continue;
};
@@ -558,12 +563,12 @@ impl<'a> Relocation<'a> {
let hits: Vec<u32> = cands
.iter()
.copied()
.filter(|&c| {
self.target.code(c, n).is_some_and(|w| mask(w) == m)
})
.filter(|&c| self.target.code(c, n).is_some_and(|w| mask(w) == m))
.collect();
if let [g] = hits.as_slice() {
if let Some(&(_, a)) = address_refs(self.target.code(*g, n)?).iter().find(|(o, _)| *o == off)
if let Some(&(_, a)) = address_refs(self.target.code(*g, n)?)
.iter()
.find(|(o, _)| *o == off)
{
return Some(a);
}
@@ -582,14 +587,24 @@ mod tests {
fn masking_follows_lui_through_moves_and_keeps_constants() {
// lui v0,0x17 ; addiu s1,v0,0x42c0 ; move s2,v0 ; addiu s1,s2,0x42c0 ; lui at,0x3f80 ; lw t0,8(a0) ; lw t1,-0x10(gp)
let w = [
0x3c02_0017, 0x2451_42c0, 0x0040_902d, 0x2651_42c0, 0x3c01_3f80, 0x8c88_0008,
0x3c02_0017,
0x2451_42c0,
0x0040_902d,
0x2651_42c0,
0x3c01_3f80,
0x8c88_0008,
0x8f89_fff0,
];
let m = mask(&w);
assert_eq!(
m,
vec![
0x3c02_0000, 0x2451_0000, 0x0040_902d, 0x2651_0000, 0x3c01_3f80, 0x8c88_0008,
0x3c02_0000,
0x2451_0000,
0x0040_902d,
0x2651_0000,
0x3c01_3f80,
0x8c88_0008,
0x8f89_0000
]
);
@@ -610,7 +625,10 @@ mod tests {
fn mask_finds_one_copy_of_a_level_01_function_on_all_27_overlays() {
let data = std::env::var_os("GC_EXTRACTED")
.map(PathBuf::from)
.unwrap_or_else(|| PathBuf::from(std::env::var("HOME").unwrap_or_default()).join("Documents/gc-extracted"));
.unwrap_or_else(|| {
PathBuf::from(std::env::var("HOME").unwrap_or_default())
.join("Documents/gc-extracted")
});
if !data.exists() {
eprintln!("skipped: unpack the disc first");
return;
@@ -618,7 +636,8 @@ mod tests {
// Parse all 27 overlays and build their LevelOverlay.
let mut overlays = Vec::with_capacity(27);
for lv in 0..27usize {
let ov = std::fs::read(data.join(format!("level{lv}/overlay.bin"))).expect(&format!("level{lv}/overlay.bin"));
let ov = std::fs::read(data.join(format!("level{lv}/overlay.bin")))
.expect(&format!("level{lv}/overlay.bin"));
let secs = parse_overlay_sections(&ov).expect(&format!("level{lv} section parse"));
let lvl = LevelOverlay::from_sections(secs).expect("LevelOverlay::from_sections");
overlays.push(lvl.unwrap_or_else(|| panic!("level {lv}: no .text section")));
@@ -630,8 +649,12 @@ mod tests {
let mut ref_fn = 0;
let mut m = Vec::new();
for &fn_start in &ref_lv.starts {
let ext = ref_lv.extent(fn_start).expect("extent for a function start");
let c = ref_lv.code(fn_start, ext).expect("code for a function start");
let ext = ref_lv
.extent(fn_start)
.expect("extent for a function start");
let c = ref_lv
.code(fn_start, ext)
.expect("code for a function start");
let masked = mask(c);
if masked.len() >= 4 {
ref_fn = fn_start;
@@ -639,7 +662,10 @@ mod tests {
break;
}
}
assert!(!m.is_empty(), "no function with masked extent >= 4 in level 0 .text");
assert!(
!m.is_empty(),
"no function with masked extent >= 4 in level 0 .text"
);
eprintln!("ref_fn 0x{ref_fn:x} masked to {} words", m.len());
// On every other overlay, its relocated copy must be found exactly once.
for (i, target) in overlays.iter().enumerate().skip(1) {
docs/formats/core-data-regions.md
+34 -8
View File
@@ -2764,14 +2764,18 @@ that every record except the last is exactly its own offset delta long, and that
1,509 chunks contain 1,252 with ragged deltas. A regression to a single sampled
stride fails immediately and says which chunk and which record.
## The tie vertex region has two encodings, not one
## CORRECTED: the tie vertex region splits in two, but not into "VIF and flat"
This section previously read bit 31 of the region's first word as "this is a VIF
code word", and told a two-encoding story on it -- half VIF packets, half flat.
The split is real. **The identification is not.**
The stride retraction found the vertex region is not a uniform array -- some runs
4 bytes, some 16, run length varying. The split that accounts for it is visible
in the region's first word: a VIF code word has bit 31 set.
first word is a VIF code word 2,007
first word is not 2,102
first word has bit 31 set 2,007
first word has bit 31 clear 2,102
(levels 0-5, every record with vert_size > 0)
The two kinds look like this:
@@ -2786,11 +2790,33 @@ The two kinds look like this:
d000, d000, c000, c000; last
word constant at 0x00001000
So roughly half of all tie records address their vertex data as **VIF packets**
and half as a **flat 16-byte array**. The split is overwhelmingly *not* a
function of `vert_size` -- 62 of the 72 values carry both kinds -- so "short
objects are VIF, long ones are flat" is false, which is the rule a decoder would
otherwise reach for. The 10 that carry one kind are the rarest sizes, not a band.
So roughly half of all tie records begin with a word that has bit 31 set and
half do not. The split is overwhelmingly *not* a function of `vert_size` -- 62 of
the 72 values carry both kinds -- so "short objects are one thing, long ones
another" is false, which is the rule a decoder would otherwise reach for.
### They are not VIF streams
Bit 31 set is exactly what a VIF code word has, and that was the whole basis for
calling them VIF. Walking them as VIF packets says otherwise. Levels 0-2, 914
records:
walk reaches the region end 17
walk dies after exactly ONE packet 1,571
packets before dying: min 0, median 1, max 4
`tfrag` walks VIF the same way -- `VifCode::decode`, then `num * 4` payload words
-- and reaches the end of its lists every time. The walk is not the problem.
These regions are not VIF packet streams, so bit 31 is set on whatever this word
actually is, which remains unknown.
The original example quoted `0xf5c1a3c5` as a code word with cmd `0x75` num 193.
Decoding one word as a packet header says nothing about whether the next 193
words are its payload, and here they are not.
Both findings are now asserted in the same test, the split and the walk failure,
so that if these ever do turn out to be VIF the test fails and says so rather
than the two quietly drifting apart.
This does not settle the stride. It says the region needs to be dispatched on the
encoding before anything inside it can be read, and it explains why the flat