A detector for cross-paper contradictions, and the measurement that says it does not work
also called opposed claims · find-opposed-claims · cross-paper contradiction detection
The graph was holding two opposite answers to the same question in different places, and nothing noticed. So we built something to notice. Then we tested it on the case we already knew the answer to, and it failed: the real contradiction came 76th in a list of 200, below two hundred pairs that were not contradictions at all.
The same thing, for a scientist
`scripts/find-opposed-claims.py` looks for claims in different entries that describe the same quantity, cite **different** sources, and split an antonym pair — the class `find-contradictions.py` cannot see, since it keys on a shared PMID. Tested against the known sex-effect contradiction: **at every setting short enough to read, the true pair is absent; at the settings that contain it, it ranks 76th and 101st of ~200, and 0 of the top 10 are true.** Two defects in the detector surfaced on the way, one of which made its first output 90% noise while looking entirely reasonable.
The grid
| rare-word index depth | overlap floor | pairs flagged | known case found |
|---|---|---|---|
| 8 | 0.30 | 1 | no |
| 8 | 0.20 | 10 | no |
| 8 | 0.15 | 18 | no |
| 20 | 0.30 | 6 | no |
| 20 | 0.20 | 62 | no |
| 20 | 0.15 | 199 | yes (ranks 76, 101) |
| 40 | 0.30 | 10 | no |
| 40 | 0.20 | 112 | no |
| 40 | 0.15 | 336 | yes (ranks 134, 179) |
The shape is the finding. There is no setting that is both readable and correct. Tuning until the output is pleasant selects exactly the configuration that misses the case the tool was built for — and nothing in the output would have said so, which is why the script now carries a self-check that fails loudly.
Two defects in the detector, and what they cost
1. Substring matching. "male" in text is true when the text contains female. Every male/female pair silently cancelled — the exact shape the script exists to find. Fixing it to whole-word matching removed 9 of the original 10 flagged pairs as artefacts of the same bug class: up inside upon and group, gain inside against. The first version's output was 90% noise and read as a perfectly plausible short list.
2. The optimisation is the blindness. Candidate pairs come from a blocking index on each claim's rarest words, so the scan does not compare 4,019 claims pairwise. But two claims about the same quantity from different papers share only common words. The known pair shares exactly five — age, cns, infants, kmt2a, sex — for an overlap of 0.182. No rare-word bucket brings them together.
Both defects produced clean, plausible, wrong output. That is the third time in three cycles.
The same lesson as the claim checker, from the other side
verify-claims.py scores a claim against its abstract by IDF-weighted overlap, and its own entry records the blindness that matters: a claim that reuses an abstract's words while reversing its meaning scores high, because overlap is not entailment.
This script fails for the mirror-image reason. Two claims that contradict each other are usually written in different vocabulary, because they come from different papers with different framings — so overlap is low exactly when the contradiction is real. One instrument is fooled by shared words without shared meaning; the other is defeated by shared meaning without shared words.
Overlap measures wording. Contradiction is a relation between meanings. This project has now built two instruments on the first hoping to reach the second and measured both failing. The structured route — extract (quantity, population, direction) from each claim and compare directions within a matched quantity — would make the sex case trivial: (prognosis, infant ALL, male worse) against (prognosis, infant ALL, female worse). That is a schema change, not a script.
- Find a second known contradiction. Recall measured at n=1 is barely measured, and the grid above would mean something quite different if a second case ranked third.
- Structure the claims. (quantity, population, direction) triples would make this class trivially detectable and would also serve the partner-provenance audit, which needed exactly the same thing — a per-entry fact aggregated across the graph.
- Deduplicate identical claim text before pairing, and re-measure. The ranks may improve a great deal for one line of code.
- Run the detector's output past a reader once. 200 pairs is an afternoon, and the two true ones are in there — the objection to this analysis is that nobody has read past rank 10.
- E1established confidence 5 of 5: Multiple independent lines of evidence; treated as settled in the field.analysis
Tested against a contradiction whose answer was already known, the opposed-claims detector places the true pair at ranks 76 and 101 of about 200 flagged pairs, with 0 of the top 10 true. At the three settings whose output is under 20 pairs, the true pair is absent entirely. Grid run over 4,019 claims in 882 entries on 2026-09-19.
Opposed-claims detection: can lexical overlap find a contradiction between two papers that share no identifier? · Oncofusion knowledge graph (internal analysis) 2026What this evidence cannot show
The ground truth is a single case, so this measures recall at n=1 and precision only over the ten pairs actually read. A second known case would be worth more than any refinement of the script. Pair counts vary by one or two between runs at a boundary floor; the ranks quoted are from the default setting.
Evidence that argues against this (2)
- A queue that contains the answer at rank 76 is not useless — 200 pairs is an afternoon's reading, and no other instrument in this project surfaces the class at all. Calling this a failure sets the bar at 'ranks it first', which no lexical method was ever going to clear. Oncofusion knowledge graph (internal analysis) 2026
- The ten pairs read were the ten highest-scoring, which is the region where a ranking-based method should look best. Judging precision there and generalising to the other 190 overstates how bad the tail is, in the same way that reading only the top of a reading queue overstates how good it is. Oncofusion knowledge graph (internal tooling) 2026
- E2established confidence 5 of 5: Multiple independent lines of evidence; treated as settled in the field.analysis
Whole-word matching removed 9 of the detector's original 10 flagged pairs. The first implementation matched antonyms as substrings, so 'male' matched inside 'female', 'up' inside 'upon' and 'group', and 'gain' inside 'against' — cancelling the male/female class outright while generating a plausible-looking short list that was 90% artefact.
Opposed-claims detection: can lexical overlap find a contradiction between two papers that share no identifier? · Oncofusion knowledge graph (internal analysis) 2026What this evidence cannot show
Counted by running the same code path before and after the one-line change on the same graph snapshot. Both runs are reproducible from analyses/opposed-claims/run.py.
Evidence that argues against this (2)
- The bug was caught within minutes because the script was run against a case with a known answer, which is this project's standing rule rather than luck. A detector without a self-check would have shipped the 90%-noise version and its short, readable output would have been taken as evidence the graph was consistent. Oncofusion knowledge graph (internal analysis) 2026
- One-line string-matching bugs are not a deep finding about instrument design, and framing them alongside the blocking-index limitation flatters the analysis. The second defect is structural; the first is a typo with a large blast radius. Oncofusion knowledge graph (internal tooling) 2026
- E3strong confidence 4 of 5: Replicated in peer-reviewed work by more than one group.analysis
The graph's own redundancy inflates the queue. At the default setting, one claim appears in 11 flagged pairs and the seven top-ranked pairs are the same duplicated MLL1 SET-domain sentence — present verbatim in seven entries — paired against a single KDM4 claim. A single lexical coincidence is multiplied by how often the graph repeats itself.
Opposed-claims detection: can lexical overlap find a contradiction between two papers that share no identifier? · Oncofusion knowledge graph (internal analysis) 2026What this evidence cannot show
Counted from the default-setting report on 2026-09-19. Duplication of a claim across entries is usually legitimate here: the same primary result genuinely supports several entries, and the project requires the citation to be repeated rather than cross-referenced.
Evidence that argues against this (1)
- This says as much about the detector's scoring as about the graph. Deduplicating identical claim text before pairing would remove the inflation without touching the duplication itself, so presenting it as a property of the graph is a choice about where to put the blame. Oncofusion knowledge graph (internal analysis) 2026