Add a rule requiring measurements to be partitioned before reporting

Rule 3 required bounds and a method for every number, but not that the
measurement be partitioned. Three errors in one stretch were all of that kind,
and all three passed the control check rule 3 asks for -- because a matched
control bounds significance while saying nothing about scope:

- a 368-byte period attributed to a whole region, when it was one chunk's
  stride among ~15. Averaging the strides put the peak at the noise floor,
  which read as "no structure" rather than "wrong scope".
- a ten-entry table disconfirmed, when it existed at a fixed stride among
  several.
- a moby class array reported at u16 slots 129-140, which was the
  largest-stride chunk type dominating a pooled scan. Per type, all 26 place
  their array elsewhere.

In each case the aggregate was arithmetically correct and the conclusion was
wrong. The re-run that caught all three was identical: group by the thing that
varies, then look.

Rule 3a now requires stating the partition before the number, reporting per
partition or saying explicitly that a claim is pooled and unverified, treating
a structure found in one partition as unproven until every partition agrees,
and retracting rather than softening a result that changes under partitioning.

Renumbered the following rules.
This commit is contained in:
2026-10-11 04:32:27 +02:00
parent b13901a7e7
commit 4c4175de7f
+36 -2
View File
@@ -65,6 +65,40 @@ All four shared one cause: measuring before bounding, and naming bytes before
looking at them. None of them errored, which is why they survived long enough looking at them. None of them errored, which is why they survived long enough
to be believed. to be believed.
### 3a. Partition the measurement, or you have not measured it.
A number computed over a heterogeneous population describes nobody. Report a
measurement **per partition** -- per level, per chunk type, per record size --
before reporting it in aggregate.
Three errors in one stretch were all of this kind, and all three survived the
control check that rule 3 asks for, because a matched control bounds
*significance* while saying nothing about *scope*:
- a 368-byte "period" attributed to a whole region, when it was the stride of
one chunk among ~15; averaging them put the peak at the noise floor, which
read as "no structure" rather than "wrong scope";
- a ten-entry table disconfirmed, when it existed at a fixed stride among
several;
- a moby class array reported at `u16` slots 129-140, which was simply the
largest-stride chunk type dominating a pooled scan. Per type, every one of
the 26 places its array somewhere else.
In every case the aggregate number was arithmetically correct and the
conclusion drawn from it was wrong. The re-run that caught all three was the
same each time: group by the thing that varies, then look.
- State the partition before the number, not after.
- If a claim is made over a pooled set, also report it for each partition, or
say explicitly that it is pooled and unverified.
- A structure found in one partition is not a property of the whole until
every partition agrees.
- When a result changes under partitioning, the pooled result was not a weak
version of the truth. It was a different and wrong claim. Retract it rather
than soften it.
### 4. Controls, and heuristic counts that are not identifications.
Concretely: Concretely:
- Derive ranges from `gc_formats::regions::core_layout`, not by running from - Derive ranges from `gc_formats::regions::core_layout`, not by running from
@@ -76,13 +110,13 @@ Concretely:
- A heuristic count is not an identification. Count distinct byte values, then - A heuristic count is not an identification. Count distinct byte values, then
*look* at the blocks that stand out before naming them. *look* at the blocks that stand out before naming them.
### 4. Verify against the real disc, not against a guess. ### 5. Verify against the real disc, not against a guess.
A parser is correct when it round-trips your own disc's bytes, checked by hash. A parser is correct when it round-trips your own disc's bytes, checked by hash.
Commit size-and-hash tables (never content) so a test can assert "this matches Commit size-and-hash tables (never content) so a test can assert "this matches
retail" rather than "this matches whatever was on my disk". retail" rather than "this matches whatever was on my disk".
### 5. Bound every read. ### 6. Bound every read.
Asset data is untrusted input from a 2003 disc format. Parsers return typed Asset data is untrusted input from a 2003 disc format. Parsers return typed
errors; they never panic and never read past the end of a buffer. errors; they never panic and never read past the end of a buffer.