ReaperOfVeriod/Going-Commando
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Correct the chunk loader finding and decode a record field

Browse Source 
The loader note said every record gains a 02 00 prefix in RAM. It does not:
the word is the record's own leading field, already on the disc at every
record. What the loader changes is the pointer -- it addresses the word, two
bytes earlier than the disc's table -- and that pointer shift is what made the
bytes look inserted.

Measured, in order:

- the two bytes before a payload are identical in core_data and in RAM, record
  for record: 02 00 on level 0's 90-record chunk at 0x100b302, 08 00 on the
  level's first chunk. Nothing is inserted;
- the RAM table's low 24 bits are the payload's minus 2, consistently, so the
  record is [leading u16][two index bytes][payload] and the loaded table
  addresses the leading word. That is also why the index bytes read as +2 from
  the pointer -- the same +2 the earlier RAM result quotes;
- the word is a field, not a pad: uniform across a chunk's records on 1,439 of
  1,509 chunks (None on the 70 that vary, reported rather than averaged),
  fixed within all 71 chunk types, and only ever 1, 2, 4, 6, 8 or 10 -- 554
  chunks carry 2, 520 carry 4.

Two readings of the value were measured and disproved, so they are recorded as
disproved rather than left as candidates: it is not an entry count (the payload
divided by it leaves 57 distinct fractional "sizes", and 162 chunks do not
divide at all), and it is not the number of distinct leading bytes (agrees on 0
of 1,506 chunks).

Chunk gains `prefix: Option<u16>`, and the disc test asserts the uniformity,
the per-type fixity and the value set. The scratch example that produced the
per-type class-ID table is dropped now that the same scan is a committed test,
which also lands here: moby class IDs sit in one body slot per chunk type, with
the slot varying by type and the distinct >= 4 filter load-bearing -- without it
the top slots are constants that read as 305x because any value below the moby
count counts as a hit.
This commit is contained in:
ReaperOfVeriod
2026-10-12 06:00:29 +02:00
parent 8147110377
commit c2af0528e3
4 changed files with 324 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/gc-extract/tests/disc.rs
+222
View File
@@ -2355,6 +2355,98 @@ fn chunk_container_closes_the_last_class_region() {
);
}
/// The chunk record's leading word is fixed per type.
///
/// A record is `[leading u16][two index bytes][payload]`. Measured on the
/// disc: the word is the same at every record of a chunk on 1,439 of 1,509
/// chunks (`None` on the 70 that vary, which the parser reports rather than
/// averaging), and fixed within every chunk type, taking only the values
/// 1, 2, 4, 6, 8 and 10.
///
/// Not named. What the value counts is not established, and two readings were
/// measured and disproved: it is not the payload's entry count -- the payload
/// length divided by it leaves 57 distinct fractional "sizes" -- and it is not
/// the number of distinct leading bytes, which agrees on 0 of 1,506 chunks.
/// See `docs/formats/core-data-regions.md`.
#[test]
fn chunk_record_leading_word_is_fixed_per_type() {
require_disc!();
let data = std::env::var_os("GC_EXTRACTED")
.map(PathBuf::from)
.unwrap_or_else(|| {
let home = std::env::var("HOME").unwrap_or_else(|_| ".".into());
PathBuf::from(home).join("Documents/gc-extracted")
});
if !data.exists() {
eprintln!("skipped: unpack the disc first");
return;
}
use gc_formats::chunks::parse_all_chunks;
use std::collections::{BTreeMap, BTreeSet};
// header -> the words its chunks carry, and the value histogram. Chunks
// whose own records disagree are counted apart rather than folded in.
let mut by_type: BTreeMap<[u8; 16], BTreeSet<u16>> = BTreeMap::new();
let mut histogram: BTreeMap<u16, usize> = BTreeMap::new();
let mut levels = 0usize;
let mut chunks = 0usize;
let mut varying = 0usize;
for n in 0..27 {
let core = match std::fs::read(data.join(format!("level{n}/core_data.bin"))) {
Ok(c) => c,
Err(_) => continue,
};
let found = parse_all_chunks(&core).expect("chunks should parse");
if found.is_empty() {
continue;
}
levels += 1;
for c in &found {
chunks += 1;
match c.chunk.prefix {
Some(p) => {
by_type.entry(c.chunk.header).or_default().insert(p);
*histogram.entry(p).or_default() += 1;
}
None => varying += 1,
}
}
}
assert!(levels >= 25, "only {levels} levels carry chunks");
assert!(chunks > 1400, "only {chunks} chunks parsed");
// The word is uniform within a chunk on all but a small minority.
let uniform = chunks - varying;
assert!(
uniform > 1400,
"only {uniform} of {chunks} chunks carry a uniform leading word"
);
// Fixed within every type: no type's chunks carry two different words.
let varying_types: Vec<_> = by_type.iter().filter(|(_, v)| v.len() > 1).collect();
assert!(
varying_types.is_empty(),
"{} types carry more than one leading word",
varying_types.len()
);
// The word is never zero and never large, and it is not the record count:
// 621 chunks carry 2, 520 carry 4, and the rest are spread thin.
assert!(
histogram.keys().all(|&p| p > 0 && p <= 16),
"prefix values {histogram:?}"
);
let two = histogram.get(&2).copied().unwrap_or(0);
let four = histogram.get(&4).copied().unwrap_or(0);
assert!(two > 500, "only {two} chunks carry the word 2");
assert!(four > 400, "only {four} chunks carry the word 4");
eprintln!(
"chunk leading word: {uniform} of {chunks} chunks uniform ({varying} vary), \
{} types, values {histogram:?}",
by_type.len()
);
}
/// Byte positions that hold the same value in every supplied record.
fn constant_positions(records: &[&[u8]]) -> usize {
let Some(first) = records.first() else {
@@ -2973,3 +3065,133 @@ fn chunk_records_start_16_byte_aligned() {
unaligned.len()
);
}
/// Moby class IDs live in one body slot per chunk type, and the slot varies.
///
/// Measured over all 27 levels: the best slot per type carries 5 to 42 distinct
/// valid moby class IDs at 153x to 289x enrichment over the chance rate. The
/// `distinct >= 4` filter is load-bearing -- without it the top slots are
/// constants that read as 305x because any value below the moby count counts as
/// a hit. See `docs/formats/core-data-regions.md`.
#[test]
fn moby_class_ids_sit_in_one_slot_per_chunk_type() {
require_disc!();
let data = std::env::var_os("GC_EXTRACTED")
.map(PathBuf::from)
.unwrap_or_else(|| {
let home = std::env::var("HOME").unwrap_or_else(|_| ".".to_string());
PathBuf::from(home).join("Documents/gc-extracted")
});
if !data.exists() {
eprintln!("skipped: unpack the disc first");
return;
}
use gc_formats::chunks::parse_all_chunks;
use gc_formats::level_core::LevelCoreHeader;
use std::collections::{BTreeMap, BTreeSet};
// (type, slot) -> (nonzero, hits, expected, distinct)
let mut stat: BTreeMap<(u32, usize), (u64, u64, f64, usize)> = BTreeMap::new();
let mut seen: BTreeMap<(u32, usize), BTreeSet<u16>> = BTreeMap::new();
for n in 0..27 {
let core = std::fs::read(data.join(format!("level{n}/core_data.bin"))).expect("core_data");
let idx = std::fs::read(data.join(format!("level{n}/core_index.bin"))).expect("core_index");
let nm = LevelCoreHeader::parse(&idx)
.expect("hdr")
.moby_classes
.count as u64;
if nm == 0 {
continue;
}
for c in parse_all_chunks(&core).expect("chunks") {
let t = u32::from_le_bytes([
c.chunk.header[0],
c.chunk.header[1],
c.chunk.header[2],
c.chunk.header[3],
]);
let len = c.records[0].len();
if c.records.iter().any(|r| r.len() != len) {
continue;
}
let mut loc: BTreeMap<usize, (u64, u64)> = BTreeMap::new();
for r in c.records.iter() {
for k in 1..len / 2 {
let v = u16::from_le_bytes([r[k * 2], r[k * 2 + 1]]);
if v == 0 {
continue;
}
seen.entry((t, k)).or_default().insert(v);
let e = loc.entry(k).or_default();
e.0 += 1;
if (v as u64) < nm {
e.1 += 1
}
}
}
for (k, (nn, hh)) in loc {
let e = stat.entry((t, k)).or_default();
e.0 += nn;
e.1 += hh;
e.2 += nn as f64 * (nm as f64 / 65536.0);
}
}
}
for (key, s) in seen.iter() {
stat.entry(*key).or_default().3 = s.len();
}
// best varying slot per type
let mut best: BTreeMap<u32, (usize, f64, usize)> = BTreeMap::new();
for ((t, k), (nn, hh, e, dist)) in stat.iter() {
if *nn < 8 || *e < 1.0 || *dist < 4 {
continue;
}
let r = (*hh as f64) / (*e);
if best.get(t).is_none_or(|(_, ce, _)| r > *ce) {
best.insert(*t, (*k, *hh as f64 / *e, *dist));
}
}
assert!(
best.len() >= 20,
"only {} chunk types carry a varying class-ID slot",
best.len()
);
let strong: Vec<_> = best.values().filter(|(_, r, _)| *r >= 100.0).collect();
assert!(
strong.len() >= 20,
"only {} types reach 100x enrichment",
strong.len()
);
let mut all: Vec<_> = best
.iter()
.map(|(t, (k, r, dist))| (*t, *k, *r, *dist))
.collect();
all.sort_by(|a, b| b.2.partial_cmp(&a.2).unwrap());
assert!(!all.is_empty(), "no varying class-ID slots found");
let mn = all
.iter()
.map(|a| a.2)
.fold(f64::MAX, |g: f64, v: f64| f64::min(g, v));
let mx = all
.iter()
.map(|a| a.2)
.fold(0.0f64, |g: f64, v: f64| f64::max(g, v));
let dmin = all.iter().map(|a| a.3).min().unwrap();
let dmax = all.iter().map(|a| a.3).max().unwrap();
eprintln!(
"chunk class-ID slots: {} types; best-slot enrichment {mn:.0}x..{mx:.0}x; \
distinct IDs per slot {dmin}..{dmax}",
best.len()
);
eprintln!(
" strongest (type, byte, ratio, distinct): {:?}",
&all[..all.len().min(6)]
);
}