Add a rule requiring measurements to be partitioned before reporting
Rule 3 required bounds and a method for every number, but not that the measurement be partitioned. Three errors in one stretch were all of that kind, and all three passed the control check rule 3 asks for -- because a matched control bounds significance while saying nothing about scope: - a 368-byte period attributed to a whole region, when it was one chunk's stride among ~15. Averaging the strides put the peak at the noise floor, which read as "no structure" rather than "wrong scope". - a ten-entry table disconfirmed, when it existed at a fixed stride among several. - a moby class array reported at u16 slots 129-140, which was the largest-stride chunk type dominating a pooled scan. Per type, all 26 place their array elsewhere. In each case the aggregate was arithmetically correct and the conclusion was wrong. The re-run that caught all three was identical: group by the thing that varies, then look. Rule 3a now requires stating the partition before the number, reporting per partition or saying explicitly that a claim is pooled and unverified, treating a structure found in one partition as unproven until every partition agrees, and retracting rather than softening a result that changes under partitioning. Renumbered the following rules.
This commit is contained in:
@@ -65,6 +65,40 @@ All four shared one cause: measuring before bounding, and naming bytes before
|
|||||||
looking at them. None of them errored, which is why they survived long enough
|
looking at them. None of them errored, which is why they survived long enough
|
||||||
to be believed.
|
to be believed.
|
||||||
|
|
||||||
|
### 3a. Partition the measurement, or you have not measured it.
|
||||||
|
|
||||||
|
A number computed over a heterogeneous population describes nobody. Report a
|
||||||
|
measurement **per partition** -- per level, per chunk type, per record size --
|
||||||
|
before reporting it in aggregate.
|
||||||
|
|
||||||
|
Three errors in one stretch were all of this kind, and all three survived the
|
||||||
|
control check that rule 3 asks for, because a matched control bounds
|
||||||
|
*significance* while saying nothing about *scope*:
|
||||||
|
|
||||||
|
- a 368-byte "period" attributed to a whole region, when it was the stride of
|
||||||
|
one chunk among ~15; averaging them put the peak at the noise floor, which
|
||||||
|
read as "no structure" rather than "wrong scope";
|
||||||
|
- a ten-entry table disconfirmed, when it existed at a fixed stride among
|
||||||
|
several;
|
||||||
|
- a moby class array reported at `u16` slots 129-140, which was simply the
|
||||||
|
largest-stride chunk type dominating a pooled scan. Per type, every one of
|
||||||
|
the 26 places its array somewhere else.
|
||||||
|
|
||||||
|
In every case the aggregate number was arithmetically correct and the
|
||||||
|
conclusion drawn from it was wrong. The re-run that caught all three was the
|
||||||
|
same each time: group by the thing that varies, then look.
|
||||||
|
|
||||||
|
- State the partition before the number, not after.
|
||||||
|
- If a claim is made over a pooled set, also report it for each partition, or
|
||||||
|
say explicitly that it is pooled and unverified.
|
||||||
|
- A structure found in one partition is not a property of the whole until
|
||||||
|
every partition agrees.
|
||||||
|
- When a result changes under partitioning, the pooled result was not a weak
|
||||||
|
version of the truth. It was a different and wrong claim. Retract it rather
|
||||||
|
than soften it.
|
||||||
|
|
||||||
|
### 4. Controls, and heuristic counts that are not identifications.
|
||||||
|
|
||||||
Concretely:
|
Concretely:
|
||||||
|
|
||||||
- Derive ranges from `gc_formats::regions::core_layout`, not by running from
|
- Derive ranges from `gc_formats::regions::core_layout`, not by running from
|
||||||
@@ -76,13 +110,13 @@ Concretely:
|
|||||||
- A heuristic count is not an identification. Count distinct byte values, then
|
- A heuristic count is not an identification. Count distinct byte values, then
|
||||||
*look* at the blocks that stand out before naming them.
|
*look* at the blocks that stand out before naming them.
|
||||||
|
|
||||||
### 4. Verify against the real disc, not against a guess.
|
### 5. Verify against the real disc, not against a guess.
|
||||||
|
|
||||||
A parser is correct when it round-trips your own disc's bytes, checked by hash.
|
A parser is correct when it round-trips your own disc's bytes, checked by hash.
|
||||||
Commit size-and-hash tables (never content) so a test can assert "this matches
|
Commit size-and-hash tables (never content) so a test can assert "this matches
|
||||||
retail" rather than "this matches whatever was on my disk".
|
retail" rather than "this matches whatever was on my disk".
|
||||||
|
|
||||||
### 5. Bound every read.
|
### 6. Bound every read.
|
||||||
|
|
||||||
Asset data is untrusted input from a 2003 disc format. Parsers return typed
|
Asset data is untrusted input from a 2003 disc format. Parsers return typed
|
||||||
errors; they never panic and never read past the end of a buffer.
|
errors; they never panic and never read past the end of a buffer.
|
||||||
|
|||||||
Reference in New Issue
Block a user