Spec checker (built in-house)

in-house build, fourth scoring round, 2026-08-20

We sat with the spec-review work of the people who commissioned us, then rewrote in Python what a person had been checking by eye. The test asked how much it actually catches, and whether it flags clean documents that are fine

1. (1)
   PreparePlace 35 regression samples, 4 carried over from the previous round, 5 newly built, 6 for the run test, and 15 real documents. The samples came from a different party than the one under test.
2. (2)
   RunFeed the samples one at a time, 65 runs in total, capturing the exit code and the list of flags each time.
3. (3)
   MatchMatch against the answer key. A defect not raised counts as a miss; a clean document flagged counts as a false positive. Anything that appears only in a warning and never in the result counts as a miss.
4. (4)
   ReadRead the six counts — detected, missed, downgraded, false positive, out of scope, bad input — and the exit code.

25planted defects it caught

22 / 25

Four rounds of scoring put it at grade 4, short of the grade 1 its job requires. Across 65 runs it caught 22 planted defects, missed 3, and wrongly flagged 3 clean documents.

Grade 4 · stopped at Detects

It used to be grade 1. On 19 August the hash on the issued report no longer matched the code in the ledger, and the grade was pulled from 1 to none. The ledger hash had been swapped to the current code while the grade and history stayed put — and that was our own hand. Four re-scorings followed; the third dropped it to grade 5, because a single full-width space in a table header made three checks vanish while the tool reported nothing and exited 0. Fixing that brought it back to grade 4.

Proven1 rungs not run

Grade 2 is not met — three false positives remain — and grade 1 was never attempted. Limits in the tool itself also stand: it cannot tell a product photo from a logo, so it only counts them, and it misses a blank country-of-origin when only some of several items are blank.

2026-08-20