The counter-note defect detector mistakes cross-referencing for unsupported assertion, which is why the different-source rate looked twice as bad
also called ledger item L · different-source defect triage · counter-note support
A tool here tries to spot counter-arguments that state specific facts the paper they cite does not actually contain, and it reported that counter-arguments citing a different paper were about twice as likely to be defective as ones citing the same paper - but nobody had read the thirty-one it flagged. Reading them, almost none is defective: the 'unsupported facts' are mostly gene names, absent from the cited abstract because the whole purpose of those notes is to bring in knowledge from elsewhere, so the tool was penalising exactly the behaviour that makes a counter-argument worth having.
The same thing, for a scientist
The by-population analysis scored 287 different-source counter-notes and found a 10.8% defect-shape rate against 5.6% same-source, recording the comparison as confounded. The 31 flagged rows were never read. Reading them: **91 of 118 flagged 'unsupported specifics' are gene symbols** (KMT2A, AFF1, WDR5, RBBP5, DOT1L), a further 25 are dataset numbers, assay concentrations or cell-line names. Dataset/methods tokens explain 7 of 31 rows outright. The remaining 24 are **synthesis arguments** - notes whose function is to say 'this does not generalise, because X and Y behave differently' - and naming genes absent from the cited abstract is what that argument IS. The detector penalises cross-referencing, which is structurally commoner in the different-source population by construction.
What the 31 turned out to be
| token class flagged as an 'unsupported specific' | n |
|---|---|
| gene symbol | 91 |
| methods concentration or dose | 11 |
| dataset number (DepMap Chronos-like) | 9 |
| cell line name | 3 |
| bare number | 2 |
| other | 2 |
My first hypothesis was wrong and is recorded as such. I expected the different-source population to be inflated by numbers from datasets - DepMap scores an abstract can never contain. That explains only 7 of 31 rows (23%). The real driver is gene symbols.
Reading the notes shows why that is not a defect. Representative flags:
protein-eps15-domainsE3, score 0.10, flagged for AFF1, MLLT3, MLLT1, DOT1L: 'The dimerisation model does not extend to the dominant KMT2A partners. AFF1, MLLT3 and MLLT1 act by recruiting the super elongation complex and DOT1L...'mechanism-dot1l-catalysis-independent-functionE4, flagged for AF9, ENL, AHD, AF10: a note observing that a PPI-disruption strategy inherits a partner-specificity problem.hyp-oligomerisation-interfaces-as-mutation-resistant-targetsE4: 'This evidence item undercuts the hypothesis it is filed under.'
Every one of these is a scope or generalisation argument. Its job is to import knowledge the cited paper does not contain and set it against the claim. The genes are absent from the cited abstract because that is the point of the note. A detector that requires a counter-note's factual tokens to appear in the abstract it cites is requiring counter-notes not to cross-reference.
And that explains the 10.8% against 5.6% without any quality difference. A note citing a different paper is, by construction, more likely to be doing cross-referencing than one inheriting the parent's citation. The earlier analysis called the comparison confounded; this identifies the confounder.
What this means for the instrument, and what it does not
The detector is not useless. It was built because 6 genuine defects in the same-source population passed every other checker, and it found them. What it cannot do is compare populations, because its false-positive rate depends on how much cross-referencing a population does - and that is precisely what differs between them.
The honest statement of the original finding stands and gets stronger: citing a different paper does not predict better support. The medians were already near-identical (0.219 against 0.230). The defect-shape gap, which was the one number that looked like a real difference, is now explained away.
A fix exists and is not free. The detector could exempt tokens that are gene symbols with entries elsewhere in this graph - a cross-reference to something the graph already knows is not an unsupported assertion. That would cut the flag count sharply and would also hide a real defect of the form 'a note invents a gene's involvement', which is a failure mode this project has seen. The trade is recorded here rather than made silently.
Nothing in the 31 was corrected, because on reading, none of the sixteen examined needed correcting. That is a finding, not an omission.
- Do not use this to conclude the graph's counter-evidence is sound. It concludes that one detector's population comparison was an artefact - the six genuine same-source defects it originally found were real, and nothing here says how many defects the different-source population contains.
- Should the detector exempt gene symbols that have entries elsewhere in this graph? It would cut false positives sharply and would hide notes that invent a gene's involvement.
- Are the cross-referenced facts in those 24 notes themselves sourced elsewhere in the graph? This triage did not check, and that is the question that would turn a generous reading into a verified one.
- E1moderate confidence 3 of 5: One strong study, or several smaller consistent ones.analysis
Of 118 tokens the counter-note support scorer flagged as unsupported specifics across the 31 different-source defect-shaped notes, 91 are gene symbols, 11 are assay concentrations or doses, 9 are DepMap-like gene-effect numbers and 3 are cell-line names; dataset and methods tokens account for only 7 of the 31 rows, and the remainder are notes making scope or generalisation arguments in which the named genes are absent from the cited abstract by design.
different-source-defect-triage: classification and reading of the 31 flagged different-source counter-notes · Oncofusion knowledge graph (internal analysis) 2026What this evidence cannot show
Run 2026-09-20. **Token classification is coarse regex**: a Chronos-like score cannot be told from any other two-decimal negative number, and a gene symbol cannot be told from an acronym - and this graph has just documented that REST, EFS and PDF are all both. **Sixteen of the 31 notes were read in full, not all 31**, so the characterisation is a reading of the majority rather than a census. Moderate for both reasons. 'Structurally explained' means the scorer's flag carries no information about that note - **it does not mean the note is correct**, since a note can cite real numbers and still misread them.
Evidence that argues against this (1)
- Calling all 24 non-dataset rows 'synthesis arguments' is the same generous reading this project has warned against elsewhere: a note that names AFF1 and MLLT1 to argue a model does not generalise is doing legitimate cross-referencing only if those claims about AFF1 and MLLT1 are themselves sourced somewhere. This triage did NOT check whether each cross-referenced fact is supported elsewhere in the graph, so it cannot exclude that some of these notes assert unsourced biology in passing. Oncofusion knowledge graph (internal analysis) 2026
- E2moderate confidence 3 of 5: One strong study, or several smaller consistent ones.analysis
Counter-notes citing a different source from the claim they oppose show a median support score of 0.230 against 0.219 for same-source notes, with defect-shape rates of 10.8% and 5.6% respectively across 287 and 286 scored notes drawn from populations of 4,371 and 650.
counter-note-support-by-population: support scores for same-source and different-source counter-notes under one instrument · Oncofusion knowledge graph (internal analysis) 2026What this evidence cannot show
From the prior cycle's analysis, re-read on 2026-09-20. **Both populations were sampled, not censused** (287 of 4,371 and 286 of 650), so the different-source figure rests on a 6.6% sample. The medians being near-identical is the robust part; the defect-shape gap is the fragile part, and this analysis now attributes it to the detector rather than to the notes.
Evidence that argues against this (1)
- A 6.6% sample of the different-source population could understate defect concentration if defects cluster in particular beats, which the sampling was not designed to detect. The conclusion that the gap is an artefact rests on reading the flagged rows, not on the sampling being adequate. Oncofusion knowledge graph (internal analysis) 2026
SAME LESSON, THIRD INSTRUMENT: what makes the graph good - careful scoping, cross-referencing - is what makes the detector fail.
STRENGTHENS ITS CONCLUSION: the one number that still looked like a real difference between the populations - 10.8% against 5.6% defect shape - is an artefact of the detector penalising cross-referencing, which the different-source population does more of by construction.
THE SIX GENUINE DEFECTS THAT MOTIVATED THE DETECTOR WERE REAL; what this shows is that its false-positive rate depends on how much a population cross-references, so it cannot compare populations.
SAME WEEK, SAME SHAPE: a token classifier cannot tell a gene symbol from an acronym, which is why REST, EFS and PDF ranked at the top of the other instrument.