start / graph / ana-the-same-source-counter-notes-are-mostly-the-authors-own-caveats

Analysisstrongking

The standing worry about same-source counter-evidence, read at n=50: 84% are the author's own caveat or a point of logic, 16% could have cited independent literature, and none of fifty is unsupported

also called same-source counter-notes · counter-argument quality sweep · ledger priority 1 · counter-evidence independence

When this project records an objection to a finding, it sometimes cites the same paper the finding came from, and we have worried for many cycles that this is a weakness. Reading fifty of them at random shows it usually is not: 84% are the study's own stated limitation or a plain point of logic, which the paper itself is the right source for.

The same thing, for a scientist

`docs/coverage.md` priority 1 has asked since cycle 6 for the same-source counter-notes to be separated into those quoting an author's own caveat and those that could have cited independent literature and did not, noting that this 'needs reading, not a checker'. Measured across 2,241 node files: **1,023 of 7,934 counter-notes carrying a citation object (12.9%) cite the same PMID as the claim they argue against.** **Fifty** were read at seed 66 against a four-way taxonomy fixed in the script BEFORE reading. Result: **A, the author's own caveat, 24; B, a logical or methodological objection needing no citation, 18; C, independent literature existed and was not cited, 8; D, an unsupported assertion, 0.** So **42 of 50 (84%, Wilson 95% CI 71.5-91.7) are legitimate by construction**, and the same-source pattern is mostly correct practice rather than a defect. The residue is real but bounded: C is 16% (CI 8.3-28.5), which is **85 to 292 notes** in the population. And **zero unsupported in fifty puts the upper 95% bound on the worst category at 7.1%**, or at most about 73 notes.

Go deeper

Why the count was never the question

Counter-argument coverage in this graph reached 100% on all three measures and stopped being informative - the ledger says so itself. What remained was a quality question that no checker could answer, because the distinction is semantic: a counter-note citing the same paper as its claim is correct when the objection is the study's own stated limitation, and weak when it asserts something about the wider field that a different paper would support.

This project has had nine instruments find their own construction before they found a defect, including a truncation heuristic that flagged 1,717 titles and was wrong about essentially all of them. So the sampler here deliberately classifies nothing. It selects at a fixed seed and prints each note beside the claim it argues against, because a counter-note cannot be judged without the claim.

The taxonomy, fixed before reading

class	meaning	is same-source a defect?
A	the note reports a limitation the cited paper itself states	No - the paper is the right and often the only source for its own caveat
B	a logical or methodological objection - n=1, no control arm, the assay cannot distinguish X from Y	No - needs no citation at all
C	asserts something about the wider field that a different paper would support	Yes - this is what the ledger was asking about
D	states a fact that is neither in the cited paper nor a matter of logic	Yes, worse

What the fifty actually were

A - the author's own caveat (24 of 50). The largest class, and same-source is correct in every one. risk-mixed-donor-chimerism-in-children-under-three: the same study found no impact of age or busulfan exposure on graft rejection, GVHD, transplant-related mortality or sinusoidal obstruction syndrome. concept-who-gets-transplanted-and-whether-it-helps quotes the authors' own concluding sentence verbatim against the entry's own favourable numbers. target-mcl1-with-bcl2-and-dexamethasone-infant-kmt2a-all: this paper's own data contradict the simple version - S63845 sensitivity did not correlate with MCL1 protein or mRNA. In each case the cited paper is the only source that could support the objection.

B - logic, no citation possible (18 of 50). mechanism-calm-af10-om-lz-dot1l-recruitment: a higher-order assembly demonstrated with engineered constructs is not the same as showing it at endogenous stoichiometry. gene-dpy30: a wild-type-allele requirement shown for MLL-AF9 in mouse need not hold for KMT2A::AFF1. hyp-fusion-is-exempt-from-restraint-not-free-of-it argues that calling two observations independent overstates it because both run through the same axis and the same paper supplies one arm - a note that attacks its own entry's independence claim.

C - independent literature existed (8 of 50). localization-af9-cytoplasmic-pool-and-hsp90-dependent-nuclear-residence invokes that AF9 475-568 is the AHD, the same surface DOT1L, AFF1/AF4 and BCOR bind competitively - true, and sourced to other papers. mechanism-mef2c-s222-predicts-menin-inhibitor-response invokes menin inhibitors' differentiation syndrome and QTc liabilities, which are documented elsewhere. ana-phase2-model-resistance-mutation asserts no study shows that cfDNA-guided switching improves outcome, which is an absence claim about the wider field and needs a bounded search rather than the parent citation. gene-faap100 attaches a number from this graph's own DepMap sweep to a paper's citation block.

D - unsupported (0 of 50). None asserted a fact that was neither in the cited paper nor a matter of logic.

The number the ledger carries is not this number, and both are right

The ledger says 327 same-source counter-notes on PAPER claims; a later triage reported 785 same-PMID counter-notes; this measures 1,014. These are three different populations measured at three different corpus sizes - the graph has roughly doubled since the first figure - and the definitions differ in whether they restrict to claims whose evidence type is a paper. Recorded here so that the next reader does not treat the movement as a contradiction. The ratio is the stable quantity: same-source notes are about an eighth of all counter-notes carrying a citation object.

Facts and figures

node files walked	2241
counter notes with a citation object	7934
same source notes	1023
same source fraction PCT	12.9
sample n	50
seed	66
taxonomy fixed before reading	true
legitimate A plus B PCT	84
legitimate wilson 95 ci PCT	71.5 to 91.7
C wilson 95 ci PCT	8.3 to 28.5
C extrapolated note range	85 to 292 of 1,023
D upper 95 bound PCT	7.1
why the ledger number differs	the ledger's 327 and a later triage's 785 are different populations at different corpus sizes; the stable quantity is the ratio, about one eighth
reproducible	analyses/counter-note-same-source-taxonomy/sample.py
measured	2026-09-22, king, cycle 66, n=50
D extrapolated max notes	73

classification

classification — the numbers behind the chart

{
  "A_authors_own_caveat": 24,
  "B_logic_or_method": 18,
  "C_independent_literature_uncited": 8,
  "D_unsupported": 0
}

superseded n12 run same cycle

{
  "why_recorded": "the n=12 version of this entry was committed earlier in cycle 66 with an open question saying the interval was too wide to plan against. Keeping both makes the movement auditable and shows which way a small sample erred.",
  "legitimate": "75.0% (CI 46.8-91.1) at n=12  ->  84.0% (CI 71.5-91.7) at n=50",
  "C": "25.0% (CI 8.9-53.2) at n=12  ->  16.0% (CI 8.3-28.5) at n=50",
  "D_upper_bound": "24.3% at n=12  ->  7.1% at n=50",
  "direction": "the n=12 sample OVERSTATED the defect rate, and its most misleading figure was the D bound: 0 of 12 was consistent with ~247 unsupported notes, 0 of 50 with at most ~73"
}

Still open

PARTLY ANSWERED IN THE SAME CYCLE. This asked whether the C rate holds at n=100; it was re-read at n=50 and the estimate moved from 25% to 16% with the interval halving. What remains is the same question at n=100-200, which would separate 'about one note in six' from 'about one in twelve' - the difference between a queue of ~85 and ~292 items.
Are the notes that carry a DIFFERENT identifier from their claim any better? This analysis measured only the same-source population, and the project's earlier sweep suggested the independent-looking column may be where a defective note hides, because attaching any second identifier moves it there. The comparison has never been run.
Can a reviewer who did not write the notes reproduce this classification? The legitimate rate of 75% was assigned by the person who wrote many of the notes, and inter-rater agreement on the A/B/C boundaries is unmeasured.
Are the eight C notes worth repairing, or is the right fix to the METRIC? Each would take a few minutes to attach a second identifier to - but the earlier sweep in this project found that attaching a second identifier is exactly the motion that moves a defective note into the 'independent' column without improving it. Repairing them would raise the independence figure and might teach the corpus the wrong lesson.

Every source behind this entry (1)

E1strong confidence 4 of 5: Replicated in peer-reviewed work by more than one group.computation_on_this_corpus
Across 2,241 node files the graph holds 7,934 counter-evidence notes carrying a citation object, of which 1,023 (12.9%) cite the same PMID as the evidence item they argue against. A random sample of 50 at seed 66, read against a taxonomy fixed in the script before reading, classifies as: author's own caveat 24, logical or methodological objection 18, independent literature available and not cited 8, unsupported assertion 0. Legitimate by construction (A+B) is 42 of 50, 84%, Wilson 95% CI 71.5 to 91.7%.
Random-sample read of 50 same-source counter-evidence notes against a pre-fixed four-way taxonomy · king, Oncofusion knowledge graph, cycle 66 · analyses/counter-note-same-source-taxonomy/sample.py 2026
What this evidence cannot show
**n=50, which narrows the question without closing it.** The C fraction of 16% has a Wilson 95% CI of 8.3 to 28.5%, i.e. **85 to 292 notes** in the population - still a factor of three. Zero D findings in fifty puts the upper 95% bound at **7.1%**, about 73 notes, which is a real bound but not zero. **Classification was done by the same person who wrote many of the notes**, which is a conflict the fixed-in-advance taxonomy limits but does not remove; inter-rater agreement is unmeasured. Roughly eight of the fifty sat on an A/B or a B/C boundary and were assigned by judgement - the RAD6 paralogue note, the Drosophila Ash1 note and the CTCF-loss-impairs-many-leukaemias note among them - and a different reader could move two or three between classes without being wrong.
Evidence that argues against this (2)
- The sampling frame itself embeds an assumption that may understate the problem. It defines same-source as note PMID equalling claim PMID, so a note that inherited its parent's citation and then had a DIFFERENT identifier attached later falls outside the population entirely - and the earlier counter-evidence sweep in this project found exactly that pattern, where attaching any second identifier would have moved a defective note into the 'independent' column. The population measured here is therefore the one where the defect is easiest to see, not the one where it is most likely to hide.
- A high legitimate rate is also what you would expect if the notes were written by someone who knew the metric, which is the case here. The finding that the same-source pattern is mostly correct practice is consistent both with the notes being good and with the author selecting objections that the source paper happens to support. Only a reviewer who did not write them can distinguish those.

Connected to

supports →The 82.9% population scored twice as many defect-shaped notes as the same-source one, and reading six of them says the comparison is confounded by our own DepMap numbers
the independence metric measures whether a counter-note's citation DIFFERS from the claim's; this reads the same-source half and finds 9 of 12 legitimate, so differing is not the thing that makes a note good
related to →Reading the same-source counter-evidence: 77 notes adjudicated, and the defect is not the one the metric measures
the earlier sweep found six defective notes all same-source by accident; this samples the same-source population at random and classifies by WHY the source is shared
supports →Counter-argument reuse across 8,080 notes is 1.04x and only 1.6% are class-level objections that ignore their own claim: the ledger's standing quality worry was measured on 53 notes and does not generalise
a second random-sample read of counter-evidence quality reaching the same direction: the standing worry is largely not borne out, and the residue is narrower than feared
related to →One laboratory holds up 92 entries, which counting papers could not show
both measure how independent this graph's supporting sources really are - that one by clustering PMIDs into laboratories, this one by asking whether a shared PMID was the right choice
related to →577 citations carried a bare PMID with no title, so the audit built to catch a fabricated title could not see them: the third dropped denominator in this project
methodological sibling: that audit's denominator leaked, so this one prints sample + unsampled == pool and states the Wilson bound on its zero finding