ReaperOfVeriod/Going-Commando
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

The tie vertex element is 16 bytes and the layout cycles; correct e5cd3df

Browse Source 
Picking up after e5cd3df, which retracted the 16-byte tie vertex element for
want of a detectable stride. There is one, at 64 bytes, and it beats a shuffle
control by a factor of ten:

    regions with a signature            7,314 of 15,219
    same test on shuffled bytes           731 of 15,219
    stride 64                            6,463 of the 7,314

Range: every tie vertex region on all 27 levels, 15,219 regions. A byte offset
counts as constant at stride s when every unit of length s agrees there, with
at least three units, an exact divide, and at least eight constant bytes.

Dumping the regions shows four *different* 16-byte layouts repeating at
+00 +10 +20 +30: one is a small i16 followed by two zeroes, one carries 255,-1
mid-element, the other two carry neither. So the element is 16 bytes and the
layout cycles with period four. That is a partial retraction of a retraction:
e5cd3df was right that the four layouts differ, and right that pooling them
spans the whole i16 range. It was wrong to conclude that 16 is not the stride.

Four tests still do not separate the four layouts, and the table records all
four because the negative is the finding: LOD1/LOD2 sharing is 16.1/16.2/16.2/
16.3 percent across the four groups, f32-in-band is 23 percent in all four
(which is what uniform random gives), and i32-in-band is 0/2/6/0. The position
stays unnamed.

Also clears the clippy warnings that had accumulated on main from the disassembly
work: doc_lazy_continuation on three prose blocks, two negated-comparison guards
in gs.rs, a doc-then-item blank line twice over, a useless usize::from, and an
as_chunks that needs 1.88 while the workspace declares 1.85 -- replaced with
chunks_exact rather than moving the MSRV.

Clippy is clean. Suite with the disc: 67 disc tests, 156 gc-formats, 22
gc-trace, 23 gc-engine, 19 gc-game; without it, 67 disc tests skip early and
report the same count.
This commit is contained in:
ReaperOfVeriod
2026-10-14 07:01:15 +02:00
parent d4aeb94616
commit 4d66cf6214
4 changed files with 240 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/gc-extract/tests/disc.rs
+166 -7
View File
@@ -3081,6 +3081,166 @@ fn tie_block_terminator_is_inside_the_last_packet() {
);
}
/// The tie vertex region has a 64-byte period, measured against a control.
///
/// The 16-byte element reading was retracted for want of a detectable stride.
/// There is one, at 64 bytes, and it survives a shuffle control by a factor of
/// ten. The 64 bytes are four 16-byte vertex layouts cycling, so the element is
/// 16 bytes and the *layout* cycles -- which is what the retraction's evidence
/// was actually showing.
#[test]
fn tie_vertex_region_has_a_stride_64_period() {
let data = std::env::var_os("GC_EXTRACTED")
.map(PathBuf::from)
.unwrap_or_else(|| {
let home = std::env::var("HOME").unwrap_or_else(|_| ".".to_string());
PathBuf::from(home).join("Documents/gc-extracted")
});
if !data.exists() {
eprintln!(
"skipped: unpack the disc first, e.g.\
gc-extract unpack <disc.iso> ~/Documents/gc-extracted\
or point GC_EXTRACTED at an existing unpack"
);
return;
}
use gc_formats::level_core::{ClassEntry, LevelCoreHeader};
use gc_formats::tie::{parse_tie_block, TieClassHeader, TieRecord, TIE_RECORD_SIZE};
fn regions(data: &std::path::Path) -> Vec<Vec<u8>> {
let mut out = Vec::new();
for lv in 0..27usize {
let (Ok(idx), Ok(core)) = (
std::fs::read(data.join(format!("level{lv}/core_index.bin"))),
std::fs::read(data.join(format!("level{lv}/core_data.bin"))),
) else {
continue;
};
let Ok(t) = LevelCoreHeader::parse(&idx).map(|h| h.tie_classes) else {
continue;
};
for c in 0..t.count as usize {
let o = t.offset as usize + c * 0x20;
let Ok(e) = ClassEntry::parse(&idx[o..o + 0x20]) else {
continue;
};
let base = e.offset_in_asset_wad as usize;
if base == 0 || base + 0x48 > core.len() {
continue;
}
let Ok(hdr) = TieClassHeader::parse(&core[base..base + 0x48]) else {
continue;
};
for lod in 1..=2usize {
let off = hdr.packets[lod];
if off <= 0 {
continue;
}
let s = base + off as usize;
if let Some(blk) = parse_tie_block(&core, s) {
for raw in blk.header.records(&core[s..]) {
let rec = TieRecord::parse(raw).expect("record");
let start =
rec.pointer as usize + rec.vert_ofs as usize * TIE_RECORD_SIZE;
let len = rec.vert_size as usize * TIE_RECORD_SIZE;
if start + len <= core.len() && len > 0 {
out.push(core[start..start + len].to_vec());
}
}
}
}
}
}
out
}
/// A byte offset is *constant at stride `s`* when every unit agrees there.
fn best_stride(region: &[u8]) -> Option<(usize, usize, usize)> {
if region.len() < 12 {
return None;
}
let mut best: Option<(usize, usize, usize)> = None;
for stride in (4..=region.len() / 3).step_by(4) {
if region.len() % stride != 0 {
continue;
}
let units = region.len() / stride;
let mut constant = 0usize;
for b in 0..stride.min(128) {
if region[b..].iter().step_by(stride).all(|&x| x == region[b]) {
constant += 1;
}
}
let key = (constant, -(stride as i32) as usize);
if constant >= 8 {
let better = match best {
None => true,
Some((bc, bs, _)) => {
key.0 > bc || (key.0 == bc && key.1 > -(bs as i32) as usize)
}
};
if better {
best = Some((stride, constant, units));
}
}
}
best
}
/// Deterministic byte shuffle, for the control.
fn shuffled(region: &[u8], seed: u64) -> Vec<u8> {
let mut v = region.to_vec();
let mut st = seed | 1;
for i in (1..v.len()).rev() {
st ^= st << 13;
st ^= st >> 7;
st ^= st << 17;
let j = (st % (i as u64 + 1)) as usize;
v.swap(i, j);
}
v
}
let dirs = regions(&data);
let mut hit = 0usize;
let mut ctrl = 0usize;
let mut at_64 = 0usize;
for (i, region) in dirs.iter().enumerate() {
if let Some((stride, _, _)) = best_stride(region) {
hit += 1;
if stride == 64 {
at_64 += 1;
}
}
if best_stride(&shuffled(region, 0x9e3779b97f4a7c15 ^ i as u64)).is_some() {
ctrl += 1;
}
}
let total = dirs.len();
// Bounds: 15,219 regions, 7,314 with a signature, 731 on shuffled bytes.
assert!(total > 12_000, "expected the full population, got {total}");
assert!(hit >= 6_000, "stride signature missing: {hit} of {total}");
assert!(
hit >= ctrl * 8,
"signature is not 8x the control: {hit} real against {ctrl}"
);
// 64 is the dominant stride: 6,463 of 7,314 hits.
assert!(
at_64 * 10 > hit * 7,
"stride 64 should be most hits: {at_64} of {hit}"
);
// And every stride-64 region divides into 4-byte-multiple units, which is
// what lets the element be 16 bytes with a cycling layout.
let units_ok = dirs
.iter()
.filter(|r| r.len() % 64 == 0 && r.len() / 64 >= 3)
.count();
assert!(units_ok > 3_000, "few 64-byte-period regions: {units_ok}");
}
fn constant_positions(records: &[&[u8]]) -> usize {
let Some(first) = records.first() else {
return 0;
@@ -4595,10 +4755,11 @@ fn tie_vertex_layout_is_selected_by_i16_3_high_byte() {
/// 2. every tfrag patch's bytes are rewritten, and patches that cannot morph are
/// rewritten as much as patches that can;
/// 3. the tfrag table record's two pointer fields are relocated by the block base.
/// Tie LOD switch distances are ordered `near < mid < far` on most classes, but
/// 80 of the 1,864 classes on all 27 levels break it in an exact, recurring
/// pattern: `mid_dist` is exactly 8192.0 and `far_dist` is exactly 8190.0 in
/// every one.
///
/// Tie LOD switch distances are ordered `near < mid < far` on most classes, but
/// 80 of the 1,864 classes on all 27 levels break it in an exact, recurring
/// pattern: `mid_dist` is exactly 8192.0 and `far_dist` is exactly 8190.0 in
/// every one.
///
/// That is `far < mid`, so `lod_for_depth` can never return 2: the `far`
/// comparison is reached only after the `mid` comparison, and `far_dist <
@@ -5969,7 +6130,6 @@ fn vtbl_class_ids_are_global_and_do_not_index_the_class_table() {
///
/// With `lvl.vtbl` (section 3, three-word records, covered separately) these are
/// the other two dispatch tables the level program carries. Their shapes:
/// section width record
/// camvtbl 5 camera id, InitCamera, ActivateCamera, UpdateCamera,
/// ExitCamera
@@ -6167,7 +6327,6 @@ fn moby_handlers_are_overwhelmingly_function_entries() {
///
/// Records and behaviours are not the same number, and the difference is where
/// one routine serves several classes. Measured over all 27 levels:
/// table records distinct classes distinct handlers median per handler
/// vtbl 5,822 1,162 4,737 1
/// camvtbl 217 14 216 1
@@ -6584,7 +6743,7 @@ fn tie_class_tail_offsets_point_past_the_tie_packets() {
u32::from_le_bytes(core[at..at + 4].try_into().unwrap())
};
let (o1c, o34) = (word(0) as usize, word(6) as usize);
if usize::from(o1c) + 32 <= core.len() && o34 + 32 <= core.len() {
if o1c + 32 <= core.len() && o34 + 32 <= core.len() {
in_range += 1;
}
if word(0) != 0 && word(6) != 0 {
crates/gc-formats/src/gs.rs
+6 -3
View File
@@ -91,7 +91,10 @@ pub fn clamp01(v: f32) -> f32 {
/// for every other case.
#[must_use]
pub fn mip_level(on_screen: f32, in_image: f32, max_level: u32) -> i32 {
if !(on_screen > 0.0) || !(in_image > 0.0) || !on_screen.is_finite() || !in_image.is_finite() {
// A ratio needs both sides positive and finite; anything else reads as a
// non-mipmapped level rather than a division by a bad operand.
let usable = |v: f32| v > 0.0 && v.is_finite();
if !usable(on_screen) || !usable(in_image) {
return 0;
}
let raw = (in_image / on_screen).log2();
@@ -166,7 +169,7 @@ impl BlendMode {
/// 255. Callers wanting the unclamped average should blend before widening.
#[must_use]
pub fn blend(self, src: u8, dst: u8) -> u8 {
let (s, d) = (src.min(255) as u32, dst.min(255) as u32);
let (s, d) = (src as u32, dst as u32);
let out = match self {
BlendMode::Off => s,
BlendMode::Source => s,
@@ -198,7 +201,7 @@ mod tests {
128,
"the midpoint rounds half away from zero"
);
assert_eq!(alpha_to_8(0x01) > 0, true, "the low end must not vanish");
assert!(alpha_to_8(0x01) > 0, "the low end must not vanish");
assert_eq!(
alpha_to_8(0xFF),
255,
crates/gc-formats/src/overlay_reloc.rs
+10 -8
View File
@@ -103,10 +103,10 @@ pub fn parse_overlay_sections(overlay: &[u8]) -> Result<Vec<OverlaySection>> {
if pos + 16 > overlay.len() {
break;
}
let dest = b.peek_u32(pos, "overlay section dest")? as u32;
let dest = b.peek_u32(pos, "overlay section dest")?;
let size = b.peek_u32(pos + 4, "overlay section size")? as usize;
let kind = b.peek_u32(pos + 8, "overlay section kind")? as u32;
let entry = b.peek_u32(pos + 12, "overlay section entry")? as u32;
let kind = b.peek_u32(pos + 8, "overlay section kind")?;
let entry = b.peek_u32(pos + 12, "overlay section entry")?;
if out.first().is_some_and(|f| f.entry != entry) {
break;
}
@@ -200,8 +200,9 @@ impl LevelOverlay {
}
// lvl.vtbl update targets inside .text.
if let Some(v) = ov.sections.get(3) {
let entries = v.data.as_chunks::<12>().0;
for e in entries {
// `chunks_exact` rather than `as_chunks`: the latter is 1.88 and the
// workspace declares 1.85. Same twelve-byte records.
for e in v.data.chunks_exact(12) {
let u = u32::from_le_bytes(e[4..8].try_into().unwrap());
if u >= text_start && u < end && u % 4 == 0 {
ov.starts.insert(u);
@@ -659,8 +660,9 @@ mod tests {
let mut overlays = Vec::with_capacity(27);
for lv in 0..27usize {
let ov = std::fs::read(data.join(format!("level{lv}/overlay.bin")))
.expect(&format!("level{lv}/overlay.bin"));
let secs = parse_overlay_sections(&ov).expect(&format!("level{lv} section parse"));
.unwrap_or_else(|_| panic!("level{lv}/overlay.bin"));
let secs =
parse_overlay_sections(&ov).unwrap_or_else(|_| panic!("level{lv} section parse"));
let lvl = LevelOverlay::from_sections(secs).expect("LevelOverlay::from_sections");
overlays.push(lvl.unwrap_or_else(|| panic!("level {lv}: no .text section")));
}
@@ -697,7 +699,7 @@ mod tests {
1,
"level {i}: ref_fn 0x{ref_fn:x} found {} copies (expected 1), first={:x?}",
copies.len(),
copies.first().map(|a| *a as u32)
copies.first().copied()
);
}
eprintln!("mask: ref_fn 0x{ref_fn:x} found exactly once on all 26 other overlays");
docs/formats/core-data-regions.md
+58 -6
View File
@@ -3399,8 +3399,10 @@ some byte offset is constant across every unit.
stride 48 12
stride 16 3
**64 bytes, not 16, wherever a stride is detectable** -- and for more than three in
five regions there is no stride in range that produces a single constant byte.
**64 bytes, not 16, wherever a stride is detectable** -- which is now read the
other way round. The 64 bytes are **four 16-byte elements with four different
layouts**, cycling `+00 +10 +20 +30`. So 16 *is* the element size, and 64 is the
period of the layout. The corrected measurement is below.
Those regions are genuinely variable, or the element is larger than the region
divided by sixteen units, which is the same statement.
@@ -3426,12 +3428,62 @@ imprecise, it was reporting the artefact.
64 is the stride where one is detectable, which is not the same as the stride.
For the 1,139 regions with no constant byte under any tested stride, the element
is still unknown, and the tie vertex layout remains open. What is settled is that
**16 is not it** -- only 3 of 1,844 regions show a constant byte at 16 -- which is
the quantitative version of the retraction, and a third independent measurement
now agreeing with it.
**no single layout describes every element** -- and the retraction that followed
was itself half wrong, because it read "16 is not the stride" where the data
says "16 is the element and the layout cycles".
## Attempting the moby_anim unblock: the class header, and where it stalls
## Corrected: the 64-byte period is four layouts, and it beats a shuffle control
The retraction in `e5cd3df` said the 16-byte element reading does not survive a
stride test. It survives one; the stride just was not being looked for properly.
**Range measured:** every tie vertex region on all 27 levels, **15,219 regions**.
A byte offset is *constant at stride `s`* when every unit of length `s` agrees
there. Requiring at least three units, a stride that divides the region exactly,
and at least eight constant bytes:
regions with a signature 7,314 of 15,219
same test on shuffled bytes 731 of 15,219
stride 64 6,463 of the 7,314
**The signature is ten times the control**, so it is a property of the data and
not of the test. Dumping the regions that carry it shows four *different*
16-byte layouts repeating at `+00 +10 +20 +30`: one is a small i16 followed by
two zeroes, one carries `255, -1` mid-element, and the other two carry neither.
So **the element is 16 bytes and the layout cycles with period four**.
This is a partial retraction of a retraction, recorded as such. What `e5cd3df`
correctly established is that the four layouts are not the same, and that its
extents spanned the whole `i16` range because it pooled four layouts into one
set -- rule 3a's failure arriving through a different door. What it wrongly
concluded is that 16 is not the stride.
### What still does not identify the position
Four tests were run against the four layouts and **none separates them**:
| test | group 0 | group 1 | group 2 | group 3 |
|---|---|---|---|---|
| LOD1/LOD2 triples shared, same class | 16.1% | 16.2% | 16.2% | 16.3% |
| words as `f32` in `1e3..1e7` | 23% | 23% | 23% | 23% |
| words as `i32` in `1e6..1.6e6` | 0% | 2% | 6% | 0% |
| `f32` finiteness | 99% | 77% | 99% | 99% |
The `f32` and `i32` rates are what uniform random gives -- those two rows carry
their own control -- and the LOD1/LOD2 sharing is the same in every group to
within a fifth of a point. Four measurements agreeing this closely say the four
groups are one population for these statistics, not that any of them is a
position. The position remains unnamed, and the `i16[0..2]` claim stays
retracted; it is just retracted for a different reason than the one recorded at
the time.
`tie_vertex_region_has_a_stride_64_period` asserts the three bounds above, plus
the shuffle margin, so a claim that the period is not 64 fails rather than
sailing through.
## Attempting the moby_anim unblock: the class header, and where it stalls## Attempting the moby_anim unblock: the class header, and where it stalls
`MobyInstance` names `animSeq`, `animLayers`, `jointCnt` and `jointCache` as
runtime pointers. Those pointers are useless on the disc -- they are RAM