AGENTS 3b: never report a test count a filter could have hidden a failure in
Rule 3b, requested after a commit went out with a failing test in it.
cargo test -q --workspace | grep "^test result: ok" | awk '{s += $4}'
That sums the green test binaries and ignores the red one. A broken build still
prints a number, and the number looks entirely reasonable -- six healthy counts
added together is not a number anyone questions. I used that filtered sum as the
evidence of health in commit messages for the better part of a session, and it
was reporting green on a build that was not.
What actually happened is the part worth keeping. The count came back lower than
the previous run -- 100 instead of 278. That is a genuine signal and I explained
it away twice as an environment difference, the disc not being set, before
checking. Both explanations were plausible. Neither was tested. Only the
discrepancy was noticed, and noticing it is what surfaced the failure.
So the rule covers both halves: do not commit or push with a failing test, and do
not derive the number you report with a filter that drops the failing binaries.
The second half is the one that is easy to get wrong again, because
`cargo test -q` is quiet, the grep looks like a reasonable way to total up, and
the total is the only part anyone reads.
Also records the mistake as a mechanism rather than a symptom. Same defect as
rules 3 and 3a, one level up: a quantity that should have had one definition had
two, and nothing compared them.
Added a line to "Working rhythm" pointing at it, since that is the list someone
reads before committing.
This commit is contained in:
@@ -97,6 +97,47 @@ same each time: group by the thing that varies, then look.
|
||||
version of the truth. It was a different and wrong claim. Retract it rather
|
||||
than soften it.
|
||||
|
||||
### 3b. Never report a test count that a filter could have hidden a failure in.
|
||||
|
||||
Do not commit or push with a failing test. If one fails, it is fixed or reverted
|
||||
in the same commit — never "fixed in the next one".
|
||||
|
||||
The same rule governs the *number* you report for the suite. Do not derive it
|
||||
with a filter that drops the failing binaries:
|
||||
|
||||
```sh
|
||||
# WRONG. Sums only the binaries that passed. A red build still prints a number,
|
||||
# and the number looks fine.
|
||||
cargo test -q --workspace | grep "^test result: ok" | awk '{s += $4} END {print s}'
|
||||
```
|
||||
|
||||
A commit was pushed with a broken test because of exactly this. The filtered sum
|
||||
reported a healthy count, because `^test result: ok` matches the six green test
|
||||
binaries and ignores the seventh, which was red. The giveaway was that the count
|
||||
came back lower than the previous run's, and the difference was mistaken for
|
||||
"the disc tests were skipped" rather than "one of them failed". It was read as
|
||||
an environment difference and moved on.
|
||||
|
||||
Use the exit status, or count the lines that are not green:
|
||||
|
||||
```sh
|
||||
cargo test --workspace # then read every "test result:" line
|
||||
cargo test --workspace && echo ok # the exit status is the real answer
|
||||
```
|
||||
|
||||
Concretely:
|
||||
|
||||
- Run the full suite with the disc present when the change touches disc-decoding
|
||||
code, and say so in the commit body. A skipped test and a passing one both
|
||||
report success.
|
||||
- If the reported count moves between runs, find out why before writing anything
|
||||
down. A drop means either an environment difference or a failure, and the two
|
||||
look identical from the summary line.
|
||||
- Never quote a number you computed rather than read.
|
||||
|
||||
This is the same defect as rules 3 and 3a, one level up: a quantity that should
|
||||
have had one definition had two, and nothing compared them.
|
||||
|
||||
### 4. Controls, and heuristic counts that are not identifications.
|
||||
|
||||
Concretely:
|
||||
|
||||
Reference in New Issue
Block a user