diff --git a/AGENTS.md b/AGENTS.md index 2390ebb..10b37df 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -97,6 +97,47 @@ same each time: group by the thing that varies, then look. version of the truth. It was a different and wrong claim. Retract it rather than soften it. +### 3b. Never report a test count that a filter could have hidden a failure in. + +Do not commit or push with a failing test. If one fails, it is fixed or reverted +in the same commit — never "fixed in the next one". + +The same rule governs the *number* you report for the suite. Do not derive it +with a filter that drops the failing binaries: + +```sh +# WRONG. Sums only the binaries that passed. A red build still prints a number, +# and the number looks fine. +cargo test -q --workspace | grep "^test result: ok" | awk '{s += $4} END {print s}' +``` + +A commit was pushed with a broken test because of exactly this. The filtered sum +reported a healthy count, because `^test result: ok` matches the six green test +binaries and ignores the seventh, which was red. The giveaway was that the count +came back lower than the previous run's, and the difference was mistaken for +"the disc tests were skipped" rather than "one of them failed". It was read as +an environment difference and moved on. + +Use the exit status, or count the lines that are not green: + +```sh +cargo test --workspace # then read every "test result:" line +cargo test --workspace && echo ok # the exit status is the real answer +``` + +Concretely: + +- Run the full suite with the disc present when the change touches disc-decoding + code, and say so in the commit body. A skipped test and a passing one both + report success. +- If the reported count moves between runs, find out why before writing anything + down. A drop means either an environment difference or a failure, and the two + look identical from the summary line. +- Never quote a number you computed rather than read. + +This is the same defect as rules 3 and 3a, one level up: a quantity that should +have had one definition had two, and nothing compared them. + ### 4. Controls, and heuristic counts that are not identifications. Concretely: