Resolve both tie indirections on all 1,864 classes, and disprove a sample artifact
resolve_tie_class walks the chain, which is two steps and both are easy to get
wrong:
1. The tie class table lives in core_index.bin, not core_data.bin. On level 0
the tables chain as moby at 192, 238 x 32 bytes later tie at 7808, 74 x 32
later shrub at 10176, which closes. Reading the same table out of
core_data.bin produces plausible garbage rather than an error.
2. ClassEntry::offset_in_asset_wad addresses core_data.bin, despite the field
name and despite core_data.wad sitting alongside it.
Verified: all 1864 tie classes resolve, o_class agreeing on every one, and all
5592 LOD block offsets land inside core_data.
Disproved, and this is the part worth reading: the word at +0 of a 16-byte unit
inside an LOD block. On the first twelve units of level 0's class 0 it rises
strictly -- 288, 1520, 2848, 3968, 5088, 6096, 7312, 8528, 9792, 10976, 12016,
13072 -- which reads exactly like a linked list, and the accompanying i16s looked
quantised, two distinct values in i16[0] and three in i16[1].
Across the population it is strictly increasing on 0 of 1864 blocks, and i16[0]
takes 8969 distinct values on level 0 alone rather than two. The sample was
twelve units out of 1,724,334. Both the monotonic reading and the quantisation
were artifacts of that sample.
That is the pooled-claim error AGENTS.md rule 3a exists to prevent, committed
almost verbatim: the first twelve units looked clean, and I was one step from
writing "u32[0] is a linked list of runs" into a format spec. The test now
asserts the disproof, so it cannot creep back.
Not established: the block's record size. 3580 non-empty LOD blocks compute as
divisible by 16, but some packets offsets run backwards, so the "next offset
delimits this block" end is wrong for them and that figure should not be trusted.
The tie layout stays partial: indirections and header done, records not.
This commit is contained in:
@@ -2584,3 +2584,116 @@ fn overlay_header_and_constant_block_are_stable() {
|
||||
+0x10..0x{OVERLAY_CONSTANT_BLOCK_END:x} identical on all"
|
||||
);
|
||||
}
|
||||
|
||||
/// The two tie indirections resolve on every class on every level, and the
|
||||
/// LOD block offsets land inside `core_data`.
|
||||
///
|
||||
/// See `gc_formats::tie`. This pins the resolution chain and the header; the
|
||||
/// block's internal layout is *not* decoded, and one tempting reading of it is
|
||||
/// disproved below.
|
||||
#[test]
|
||||
fn tie_classes_resolve_through_both_indirections() {
|
||||
require_disc!();
|
||||
|
||||
let data = std::env::var_os("GC_EXTRACTED")
|
||||
.map(PathBuf::from)
|
||||
.unwrap_or_else(|| {
|
||||
let home = std::env::var("HOME").unwrap_or_else(|_| ".".to_string());
|
||||
PathBuf::from(home).join("Documents/gc-extracted")
|
||||
});
|
||||
if !data.exists() {
|
||||
eprintln!("skipped: unpack the disc first");
|
||||
return;
|
||||
}
|
||||
|
||||
use gc_formats::level_core::LevelCoreHeader;
|
||||
use gc_formats::tie::resolve_tie_class;
|
||||
|
||||
let mut classes = 0usize;
|
||||
let mut o_class_agree = 0usize;
|
||||
let mut lod_blocks = 0usize;
|
||||
let mut lod_blocks_in_range = 0usize;
|
||||
let mut monotonic_u32_0 = 0usize;
|
||||
|
||||
for n in 0..27 {
|
||||
let idx = std::fs::read(data.join(format!("level{n}/core_index.bin"))).expect("core_index");
|
||||
let core = std::fs::read(data.join(format!("level{n}/core_data.bin"))).expect("core_data");
|
||||
let h = LevelCoreHeader::parse(&idx).expect("core header");
|
||||
let t = h.tie_classes;
|
||||
if t.count == 0 {
|
||||
continue;
|
||||
}
|
||||
for c in 0..t.count as usize {
|
||||
let r = resolve_tie_class(&idx, &core, t, c).expect("tie class should resolve");
|
||||
classes += 1;
|
||||
if r.header.o_class == r.entry.o_class {
|
||||
o_class_agree += 1;
|
||||
}
|
||||
|
||||
let base = r.entry.offset_in_asset_wad as usize;
|
||||
for (i, p) in r.header.packets.iter().enumerate() {
|
||||
let abs = base.wrapping_add(*p as usize);
|
||||
if abs <= core.len() {
|
||||
lod_blocks_in_range += 1;
|
||||
}
|
||||
lod_blocks += 1;
|
||||
if *p >= 0 && abs + 16 <= core.len() {
|
||||
// Is the word at the block start strictly increasing across
|
||||
// 16-byte units? It looked like it on the first twelve units
|
||||
// of one class. It is not a property of the format.
|
||||
let next = if i + 1 < 3 {
|
||||
r.header.packets[i + 1]
|
||||
} else {
|
||||
*p
|
||||
};
|
||||
if next > *p {
|
||||
let end = base + next as usize;
|
||||
if end <= core.len() {
|
||||
let n16 = (end - abs) / 16;
|
||||
let mut prev = 0i64;
|
||||
let mut ok = true;
|
||||
for k in 0..n16 {
|
||||
let q = abs + k * 16;
|
||||
let a =
|
||||
u32::from_le_bytes(core[q..q + 4].try_into().unwrap()) as i64;
|
||||
if a <= prev {
|
||||
ok = false;
|
||||
break;
|
||||
}
|
||||
prev = a;
|
||||
}
|
||||
if ok && n16 > 0 {
|
||||
monotonic_u32_0 += 1
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
assert!(classes >= 1800, "only {classes} tie classes resolved");
|
||||
assert_eq!(
|
||||
o_class_agree,
|
||||
classes,
|
||||
"header o_class disagreed with the class entry on {} of {classes}",
|
||||
classes - o_class_agree
|
||||
);
|
||||
assert_eq!(
|
||||
lod_blocks_in_range,
|
||||
lod_blocks,
|
||||
"{} of {lod_blocks} LOD block offsets fell outside core_data",
|
||||
lod_blocks - lod_blocks_in_range
|
||||
);
|
||||
assert_eq!(
|
||||
monotonic_u32_0, 0,
|
||||
"the word at +0 of a 16-byte tie unit is strictly increasing on {monotonic_u32_0} blocks; \
|
||||
if that ever becomes true the disproved reading below needs revisiting"
|
||||
);
|
||||
|
||||
eprintln!(
|
||||
"tie: {classes} classes resolved through both indirections, o_class agreeing on all; \
|
||||
{lod_blocks} LOD block offsets all inside core_data; \
|
||||
strictly-increasing +0 on 0 of {classes} blocks (disproved)"
|
||||
);
|
||||
}
|
||||
|
||||
@@ -25,7 +25,8 @@
|
||||
//! * the LOD distances are ordered `0 < near < mid < far`: 315 of 320. The
|
||||
//! five exceptions are not yet explained and are worth chasing.
|
||||
|
||||
use crate::buf::{Buf, Result};
|
||||
use crate::buf::{Buf, Error, ErrorKind, Result};
|
||||
use crate::level_core::ClassEntry;
|
||||
|
||||
/// Bytes of a tie class header that RAC2 and RAC1 agree on.
|
||||
pub const TIE_CLASS_HEADER_MATCHED: usize = 0x48;
|
||||
@@ -91,6 +92,81 @@ impl TieClassHeader {
|
||||
}
|
||||
}
|
||||
|
||||
/// A tie class resolved through both of its indirections.
|
||||
///
|
||||
/// The chain is two steps and both are easy to get wrong, so it is worth
|
||||
/// stating plainly:
|
||||
///
|
||||
/// 1. The class **table** lives in the level's `core_index.bin`, not in
|
||||
/// `core_data.bin`. On level 0 the tables chain as moby at `192`,
|
||||
/// 238 x 32 bytes later tie at `7808`, 74 x 32 later shrub at `10176`,
|
||||
/// which closes. Reading the same table out of `core_data.bin` yields
|
||||
/// plausible-looking garbage rather than an error.
|
||||
/// 2. [`ClassEntry::offset_in_asset_wad`] addresses `core_data.bin`, despite the
|
||||
/// field's name and despite `core_data.wad` existing alongside it.
|
||||
///
|
||||
/// With both handled, all **1,864** tie classes on the disc resolve to a header
|
||||
/// whose `o_class` matches the class entry's own.
|
||||
#[derive(Debug, Clone, Copy, PartialEq)]
|
||||
pub struct ResolvedTieClass {
|
||||
/// The index entry this came from.
|
||||
pub entry: ClassEntry,
|
||||
/// Its index within the level's tie table.
|
||||
pub index: usize,
|
||||
/// The class header, at `offset_in_asset_wad` in `core_data`.
|
||||
pub header: TieClassHeader,
|
||||
}
|
||||
|
||||
/// Resolve one tie class. `class_index` counts from zero.
|
||||
///
|
||||
/// `core_index` is the level's `core_index.bin`; `core_data` is the
|
||||
/// decompressed `core_data.bin`. Both are bounds-checked.
|
||||
pub fn resolve_tie_class(
|
||||
core_index: &[u8],
|
||||
core_data: &[u8],
|
||||
classes: crate::level_core::ArrayRange,
|
||||
class_index: usize,
|
||||
) -> Result<ResolvedTieClass> {
|
||||
const ENTRY: usize = 32;
|
||||
let start = classes.offset as usize + class_index * ENTRY;
|
||||
if classes.offset < 0 || start.checked_add(ENTRY).is_none() || start + ENTRY > core_index.len()
|
||||
{
|
||||
return Err(Error {
|
||||
offset: start,
|
||||
context: "tie class table entry",
|
||||
kind: ErrorKind::OutOfBounds {
|
||||
needed: ENTRY,
|
||||
available: core_index.len().saturating_sub(start),
|
||||
},
|
||||
});
|
||||
}
|
||||
if class_index >= classes.count as usize {
|
||||
return Err(Error {
|
||||
offset: start,
|
||||
context: "tie class index",
|
||||
kind: ErrorKind::Invalid("class index past the table count"),
|
||||
});
|
||||
}
|
||||
let entry = ClassEntry::parse(&core_index[start..start + ENTRY])?;
|
||||
let base = entry.offset_in_asset_wad as usize;
|
||||
if base < TIE_CLASS_HEADER_MATCHED || base + TIE_CLASS_HEADER_MATCHED > core_data.len() {
|
||||
return Err(Error {
|
||||
offset: base,
|
||||
context: "tie class header",
|
||||
kind: ErrorKind::OutOfBounds {
|
||||
needed: TIE_CLASS_HEADER_MATCHED,
|
||||
available: core_data.len().saturating_sub(base),
|
||||
},
|
||||
});
|
||||
}
|
||||
let header = TieClassHeader::parse(&core_data[base..base + TIE_CLASS_HEADER_MATCHED])?;
|
||||
Ok(ResolvedTieClass {
|
||||
entry,
|
||||
index: class_index,
|
||||
header,
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
@@ -1097,3 +1097,34 @@ six bytes remain uninterpreted.
|
||||
hand-computed `o_class` offset instead of `ShrubClassEntry`, and every
|
||||
`ShrubHeader::region` call failed. The parser was not wrong; the caller was.
|
||||
The working call shape was already in the disc test.
|
||||
|
||||
## tie: both indirections, and one reading disproved
|
||||
|
||||
**The chain, which is two steps and easy to get wrong.** The tie class **table**
|
||||
lives in the level's `core_index.bin`, not `core_data.bin`. On level 0 the three
|
||||
tables chain as moby at `192`, 238 x 32 bytes later tie at `7808`, 74 x 32 later
|
||||
shrub at `10176`, which closes. Reading the same table out of `core_data.bin`
|
||||
produces plausible garbage rather than an error -- `offset_in_asset_wad` comes
|
||||
back as 1,217,208,842, which is outside every buffer, so the mistake announces
|
||||
itself if you look. Then `ClassEntry::offset_in_asset_wad` addresses
|
||||
**`core_data.bin`**, despite its name and despite `core_data.wad` sitting
|
||||
alongside it.
|
||||
|
||||
**Verified:** all **1,864** tie classes resolve, `o_class` agreeing on every one,
|
||||
and all **5,592** LOD block offsets land inside `core_data`.
|
||||
|
||||
**Disproved:** the word at `+0` of a 16-byte unit inside an LOD block. On the
|
||||
first twelve units of level 0's class 0 it rises strictly -- 288, 1520, 2848,
|
||||
3968, 5088, 6096, 7312, 8528, 9792, 10976, 12016, 13072 -- which reads exactly
|
||||
like a linked list, and the accompanying `i16`s looked quantised (two distinct
|
||||
values in `i16[0]`, three in `i16[1]`).
|
||||
|
||||
Across the whole population it is strictly increasing on **0 of 1,864** blocks,
|
||||
and `i16[0]` takes **8,969** distinct values on level 0 alone rather than two.
|
||||
The sample was twelve units out of 1,724,334. Both the monotonic reading and the
|
||||
quantisation were artifacts of that sample, and the disproof is asserted in
|
||||
`crates/gc-extract/tests/disc.rs` so it cannot creep back.
|
||||
|
||||
**Not established:** the block's record size. 3,580 non-empty LOD blocks compute
|
||||
as divisible by 16, but some `packets` offsets run backwards, so the "next offset
|
||||
delimits this block" end is wrong for them and the figure should not be trusted.
|
||||
|
||||
+15
-5
@@ -241,11 +241,21 @@ Phase 1 hands over the extents; this phase works out what each one contains.
|
||||
- [~] `tfrag` — vertex format, topology, texture bindings, per-vertex colour
|
||||
and normals, LOD selection and parent indices are all recovered and
|
||||
verified. **Still open:** morph targets and the strip/chain bookkeeping.
|
||||
- [ ] `tie` — class table and the 0x48-byte header agree with R&C1; the
|
||||
**vertex layout is not decoded**. The class table entries are pointers
|
||||
into `core_data.wad`, and the header's `packets[0..3]` LOD offsets are
|
||||
resolved against the lump base -- neither chain has been walked end to
|
||||
end yet. Level 0 has 74 tie classes, 238 moby and 7 shrub.
|
||||
- [~] `tie` — **both indirections now resolve.** `resolve_tie_class` walks the
|
||||
chain: the class table is in `core_index.bin`, and the entry's
|
||||
`offset_in_asset_wad` addresses `core_data.bin` (not `core_data.wad`,
|
||||
despite the field name). All **1,864** tie classes resolve, with
|
||||
`o_class` agreeing on every one, and all **5,592** LOD block offsets land
|
||||
inside `core_data`.
|
||||
- [ ] The tie **block layout is not decoded.** LOD0 blocks are not a uniform
|
||||
stride of records: 3,580 non-empty blocks compute as divisible by 16,
|
||||
but that figure is not trustworthy because some `packets` offsets run
|
||||
backwards, so the "next offset" end is wrong for them. One reading is
|
||||
**disproved**: the word at `+0` of a 16-byte unit is strictly increasing
|
||||
on **0 of 1,864** blocks, despite looking exactly like a linked list on
|
||||
the first twelve units of one class. A 12-unit sample of a 1.7M-unit
|
||||
population is the pooled-claim error rule 3a exists to prevent; the test
|
||||
now asserts the disproof so it cannot creep back.
|
||||
- [~] `shrub` — class table done (48-byte entries, not 32; reading shrub at
|
||||
32 still gives 231 sane-looking entries of 358, which is the hazard).
|
||||
Geometry header parsed and checked field by field on all 358 spans:
|
||||
|
||||
Reference in New Issue
Block a user