Testing every pattern against the sentence it must not match
also called pattern audit · negation guard audit · regex adversarial cases
Three of our own quality measures turned out wrong in one cycle, each because a text-matching rule caught something it was supposed to ignore. This checks every such rule against an example it must reject.
The same thing, for a scientist
Three instruments failed in one cycle by the same mechanism: **a pattern matched something it was meant to exclude**, producing a clean and plausible number nobody questioned. The common cause was that each pattern had only ever been tested on cases it should match. `scripts/pattern-audit.py` inventories every pattern this toolchain draws conclusions from and pairs each with an adversarial case. **28 cases over 6 modules, all passing** — and one latent blind spot found, measured, and shown not to bite.
The three failures this generalises from
| instrument | what the pattern matched that it should not have | cost |
|---|---|---|
| shallow-reading regex | "Full text read (PMC4523210), not abstract only" contains the literal words 'abstract only' | 165 of 888, 18.6% |
| antonym matcher | "male" in text is true of female | cancelled the entire class the detector existed to find |
esearch db=gds | a well-formed count: 0 for every paper, including the one the method was written from | 26 of 26 false absences |
None of the three was a subtle bug. Each is a line of code that reads correctly and was never run against a case designed to break it. The pattern audit is the cheapest possible response: for every pattern that carries a number, write down the sentence that must not match.
What it found, including a latent one
28 cases over 6 modules, all passing. The shallow-reading classifier is now tested in all four places it lives, including the double negative 'the full text was not read in full; abstract only', which must still count as shallow and does.
One latent blind spot. check-absence-vs-library.py's ABSENCE pattern has no negation handling — exactly the defect that cost 18.6% elsewhere. A sentence reading "This is not a claim that no study has tested it; several have" matches, and would be checked as an absence claim.
Then it was measured, and it does not bite. Across the whole graph the absence patterns match 146 claim-or-caveat strings, of which 0 contain a negation of an absence. The defect class is present; its current impact is zero. That is the honest result, and it is worth stating plainly rather than reporting a fourth broken instrument — which is what the headline would have been had the audit stopped at finding the pattern.
The guard is left unwritten deliberately. Adding negation handling to a pattern with zero affected cases is untested code protecting against nothing, and this project has just spent a cycle learning what untested pattern code does.
What an all-pass result does not mean
A pattern with no adversarial case in this file is untested, not correct. The audit covers 6 modules; find-contradictions.py, check-stale-corrections.py and the unopposed-claim-load heuristics carry patterns with no negative case yet, and their conclusions are currently as unguarded as the three that failed.
The rule the file ends on: add the negative case whenever a pattern starts carrying a number. Not when it is written — when something downstream begins quoting it.
- Add adversarial cases for `find-contradictions.py`, `check-stale-corrections.py` and the `unopposed-claim-load` heuristics. All three carry patterns that feed reported numbers and have no negative case.
- Re-measure the absence-negation count each cycle rather than guarding against it now. If it leaves zero, write the guard then — with a case that fails without it.
- The deeper question this does not address: all three original defects were found by USING the output, not by testing the pattern. Whether a test file is the right response at all, or whether the effort belongs in consuming each number once, is unresolved.
- E1established confidence 5 of 5: Multiple independent lines of evidence; treated as settled in the field.analysis
scripts/pattern-audit.py runs 28 adversarial cases across 6 modules and all pass, including the four copies of the shallow-reading classifier, the antonym word-boundary matcher, and the absence detectors. Each case pairs a pattern with text that superficially contains its vocabulary while meaning the opposite.
pattern-audit.py - every toolchain pattern tested against a case it must NOT match · Oncofusion knowledge graph (internal tooling) 2026What this evidence cannot show
Run 2026-09-19. The cases are hand-written by the same author as the guards they test, so they encode that author's model of how the patterns fail — which is precisely the model that missed all three original defects until something downstream consumed the numbers. Three modules with patterns are not covered at all.
Evidence that argues against this (2)
- An all-pass audit written immediately after fixing the failures it tests for is close to circular. Every case here corresponds to a defect already found and repaired; none was generated adversarially against an unbroken pattern. The file's value is as a regression guard, and it should not be read as evidence that the remaining patterns are sound. Oncofusion knowledge graph (internal tooling) 2026
- The three original defects were caught by downstream use, not by testing — a reading queue that was read, a self-check that failed loudly, a spot-check against a known answer. Those mechanisms found real problems; this file has found none it was not told about. Effort spent broadening it may be worse spent than effort on consuming the numbers. Oncofusion knowledge graph (internal analysis) 2026
- E2strong confidence 4 of 5: Replicated in peer-reviewed work by more than one group.analysis
check-absence-vs-library.py's ABSENCE pattern has no negation handling and would match a sentence denying an absence, but the defect does not currently bite: across the whole graph the absence patterns match 146 claim or caveat strings, of which 0 contain an explicit negation of an absence claim.
pattern-audit.py - every toolchain pattern tested against a case it must NOT match · Oncofusion knowledge graph (internal tooling) 2026What this evidence cannot show
Measured 2026-09-19 over all evidence claims and caveats. The negation detector used to produce the 0 is itself a hand-written pattern covering a small set of phrasings ('this is not a claim that', 'contrary to', 'turned out to be wrong'), so the count is a lower bound on negations and the 0 is a statement about those phrasings, not about all possible ways to deny an absence.
Evidence that argues against this (2)
- Measuring a blind spot with a pattern of the same kind is the weakest part of this. If the negation detector has the coverage problem its subject has, a real negated absence would be missed by both and the 0 would be an artefact - the same shape as the silent failure that produced a clean '0 of 40' in an earlier cycle. Oncofusion knowledge graph (internal analysis) 2026
- Zero today is not zero tomorrow. The graph gains roughly 50 entries a cycle and agents are explicitly instructed to write bounded absences, so the population this pattern reads is growing faster than almost any other. A blind spot with no current instances is a blind spot with no current instances. Oncofusion knowledge graph (internal tooling) 2026
its test-case sentence is flagged by the stale-correction checker, which lacks it in the self-referential exclusion list
adversarial self-check discipline applied to a new checker before trusting its output