Add a rule requiring measurements to be partitioned before reporting
Rule 3 required bounds and a method for every number, but not that the measurement be partitioned. Three errors in one stretch were all of that kind, and all three passed the control check rule 3 asks for -- because a matched control bounds significance while saying nothing about scope: - a 368-byte period attributed to a whole region, when it was one chunk's stride among ~15. Averaging the strides put the peak at the noise floor, which read as "no structure" rather than "wrong scope". - a ten-entry table disconfirmed, when it existed at a fixed stride among several. - a moby class array reported at u16 slots 129-140, which was the largest-stride chunk type dominating a pooled scan. Per type, all 26 place their array elsewhere. In each case the aggregate was arithmetically correct and the conclusion was wrong. The re-run that caught all three was identical: group by the thing that varies, then look. Rule 3a now requires stating the partition before the number, reporting per partition or saying explicitly that a claim is pooled and unverified, treating a structure found in one partition as unproven until every partition agrees, and retracting rather than softening a result that changes under partitioning. Renumbered the following rules.
This commit is contained in:
@@ -65,6 +65,40 @@ All four shared one cause: measuring before bounding, and naming bytes before
|
||||
looking at them. None of them errored, which is why they survived long enough
|
||||
to be believed.
|
||||
|
||||
### 3a. Partition the measurement, or you have not measured it.
|
||||
|
||||
A number computed over a heterogeneous population describes nobody. Report a
|
||||
measurement **per partition** -- per level, per chunk type, per record size --
|
||||
before reporting it in aggregate.
|
||||
|
||||
Three errors in one stretch were all of this kind, and all three survived the
|
||||
control check that rule 3 asks for, because a matched control bounds
|
||||
*significance* while saying nothing about *scope*:
|
||||
|
||||
- a 368-byte "period" attributed to a whole region, when it was the stride of
|
||||
one chunk among ~15; averaging them put the peak at the noise floor, which
|
||||
read as "no structure" rather than "wrong scope";
|
||||
- a ten-entry table disconfirmed, when it existed at a fixed stride among
|
||||
several;
|
||||
- a moby class array reported at `u16` slots 129-140, which was simply the
|
||||
largest-stride chunk type dominating a pooled scan. Per type, every one of
|
||||
the 26 places its array somewhere else.
|
||||
|
||||
In every case the aggregate number was arithmetically correct and the
|
||||
conclusion drawn from it was wrong. The re-run that caught all three was the
|
||||
same each time: group by the thing that varies, then look.
|
||||
|
||||
- State the partition before the number, not after.
|
||||
- If a claim is made over a pooled set, also report it for each partition, or
|
||||
say explicitly that it is pooled and unverified.
|
||||
- A structure found in one partition is not a property of the whole until
|
||||
every partition agrees.
|
||||
- When a result changes under partitioning, the pooled result was not a weak
|
||||
version of the truth. It was a different and wrong claim. Retract it rather
|
||||
than soften it.
|
||||
|
||||
### 4. Controls, and heuristic counts that are not identifications.
|
||||
|
||||
Concretely:
|
||||
|
||||
- Derive ranges from `gc_formats::regions::core_layout`, not by running from
|
||||
@@ -76,13 +110,13 @@ Concretely:
|
||||
- A heuristic count is not an identification. Count distinct byte values, then
|
||||
*look* at the blocks that stand out before naming them.
|
||||
|
||||
### 4. Verify against the real disc, not against a guess.
|
||||
### 5. Verify against the real disc, not against a guess.
|
||||
|
||||
A parser is correct when it round-trips your own disc's bytes, checked by hash.
|
||||
Commit size-and-hash tables (never content) so a test can assert "this matches
|
||||
retail" rather than "this matches whatever was on my disk".
|
||||
|
||||
### 5. Bound every read.
|
||||
### 6. Bound every read.
|
||||
|
||||
Asset data is untrusted input from a 2003 disc format. Parsers return typed
|
||||
errors; they never panic and never read past the end of a buffer.
|
||||
|
||||
Reference in New Issue
Block a user