Counter-argument reuse across 8,080 notes is 1.04x and only 1.6% are class-level objections that ignore their own claim: the ledger's standing quality worry was measured on 53 notes and does not generalise
also called counter-note boilerplate census · counter-argument quality · class-level objections · 100% coverage caveat
This project requires every claim to carry a published argument against it, and reached 100% coverage a while ago - but immediately warned that a high score might just mean the same generic objection pasted everywhere. Measured across all 8,080 of them, that turns out not to be what happened: 95% are one-off texts written for their own claim, and the generic ones are mostly generic for a good reason.
The same thing, for a scientist
The ledger records counter-argument coverage at 100% of entries and evidence items and attaches a standing caveat: *"35 of the last 53 notes use a class-level objection - single-source, lineage-split - rather than a claim-specific one... 100% coverage is not 6,706 individually reasoned arguments."* **That caveat was written from a 53-note sample and does not hold over the corpus.** Two independent measurements over all **8,080 counter-notes in 1,749 entries**: **1. Reuse is negligible.** Exact duplicates after normalisation: **302 notes (3.7%)** across 72 distinct texts. Near-duplicates at token-set Jaccard >= 0.75: **378 notes (4.7%)** in 97 clusters. **7,702 notes are singletons**, giving at most 7,799 distinct arguments for 8,080 notes - a **reuse factor of 1.04x**. **2. Class-level objections are a fifth of the corpus and mostly not the failure mode.** **21.3%** invoke a named template (lineage/system-mismatch 7.0%, murine 6.4%, abstract-only 4.3%, database/by-similarity 3.3%, underpowered 2.0%, single-source 1.0%, preprint 0.3%). But invoking a template is not the defect - **ignoring the claim while doing so** is. Cross-tabulating template use against IDF-weighted engagement with the note's own claim: | | n | % | |---|---:|---:| | **template + no engagement (the worry)** | **130** | **1.6%** | | template + engages specifically | 1,592 | 19.7% | | no template + no engagement | 557 | 6.9% | | no template + engages specifically | 5,801 | **71.8%** | **And the 1.6% is itself an overcount.** Reading the ten lowest-engagement template notes: most are correct, specific objections that happen to paraphrase rather than repeat the claim's nouns. Separately, **55.8% of counter-notes cite a different paper than the claim they answer**, 10.1% cite the same paper, and 0.9% carry no citation.
Why 53 notes gave the wrong answer, and why that is the interesting part
The ledger's sample was "the last 53 notes" - the most recently written ones. That is not a random sample of the corpus, and the direction of its bias is predictable in hindsight.
Late-cycle notes are disproportionately written by the coordinator during integration passes, onto entries whose claims are already well defended, where the remaining honest objection genuinely is a class-level one. They are also written under time pressure at the end of a cycle. The recent tail is the worst-case slice of the corpus, and it was mistaken for the corpus.
This is the fourth instrument-scope error in this project inside two cycles, and it is a new variety. The others were about what a check walks: a fetch failure leaving a denominator, 577 citations with no title to score, 663 PMIDs a collector never reached. This one is about what a sample represents. A measurement can cover its whole denominator honestly and still generalise from the wrong slice of it.
The practical consequence is that a standing weak spot should be retired. The ledger has carried 'upgrade the class-level counter-notes' as queued work for several cycles. On these numbers the queue is 130 notes at the outside and probably a few dozen after reading, not a corpus-wide quality problem - and there are better uses of a beat.
Where the reuse that does exist is correct, and the one place it is not
The most-reused arguments are reused because the same objection is the right one:
- x12 and x11: "A reviewed database record fixes the canonical sequence and primary symbol on the date it was read, and nothing more." This appears against claims that ARE accession lookups. The correct caveat for twelve accession claims is the same caveat twelve times.
- x10: domain boundaries differ between InterPro, Pfam, SMART and crystal constructs - against
protein-*-domainsentries, where it is the governing limitation of every one. - x9: "A curated interaction inherits every limitation of the single experiment behind it" - against IntAct-derived entries, which is exactly the population it bounds.
One cluster is doing something different and better. An 8-note cluster carries "This is the decisive objection to the whole WRAD thread and it should be read before any of it: the SET domain of MLL1 - the catalytic activity that WRAD exists to stimulate - is dispensable..." That is not boilerplate; it is one strong argument deliberately propagated across the thread it refutes, which is the behaviour a knowledge graph should want. A reuse metric cannot tell it apart from copy-paste, and that is a limit of the metric rather than a finding about the graph.
The honest statement: reuse in this corpus is low, and where it exists it is mostly either the correct shared caveat or a deliberately propagated argument. Neither is the failure the coverage caveat feared.
Both instruments found their own false positives before they found a defect
The engagement metric is lexical and its low tail is dominated by paraphrase. The ten worst-scoring template notes were read. Examples scoring 0.000:
gene-dpy30#E4: "A co-dependency correlation can be produced entirely by two genes being required in the same lineage rather than in the same process." Against a DepMap correlation claim. This is precisely the right objection - and this project builtanalyses/lineage-confound-myeloid-tf-dependencies/because of exactly that confound.mark-h3k79me3#E1: "Distributivity is established on bulk chromatin and says nothing about a locus where the enzyme is held in place by a fusion protein." Specific, mechanistic, and scores zero because it shares no rare tokens with its claim.
So the 130 is a ceiling on a ceiling. The measurement is reported anyway, because a bounded overcount is more useful than no number - but it must not be quoted as 130 defects.
And the duplicate census cannot see reworded boilerplate at all, which is why it was not reported alone. A class-level objection paraphrased each time is unique text and scores as a distinct argument. Reporting the 1.04x reuse factor as if it answered the ledger's question would have been answering an easier question than the one asked - the second script exists for that reason.
| n counter notes | 8080 |
|---|---|
| n entries | 1749 |
| exact duplicate notes | 302 |
| near duplicate notes | 378 |
| singletons | 7702 |
| distinct arguments upper bound | 7799 |
| reuse factor | 1.04 |
| engagement median | 0.161 |
| ledger claim being corrected | 35 of the last 53 notes use a class-level objection rather than a claim-specific one |
templates PCT — the numbers behind the chart
{
"any": 21.3,
"lineage_system_mismatch": 7,
"murine": 6.4,
"abstract_only": 4.3,
"database_by_similarity": 3.3,
"underpowered": 2,
"single_source": 1,
"preprint": 0.3
}cross tab — the numbers behind the chart
{
"template_and_no_engagement": 130,
"template_and_engages": 1592,
"no_template_and_no_engagement": 557,
"no_template_and_engages": 5801
}source independence — the numbers behind the chart
{
"different_paper_than_claim": 4512,
"same_paper": 815,
"no_citation": 76
}- Do not quote '1.04x reuse' as showing the counter-arguments are sound. It shows they are textually distinct. Eight thousand individually-worded weak objections would score identically, and neither instrument here reads for soundness.
- Do not quote '130 class-level notes' as a defect count. Reading the worst ten found mostly correct objections that paraphrase rather than repeat their claim's nouns, so the real number is lower by an unmeasured amount.
- Do not use this to argue counter-evidence work is finished. It argues the OPPOSITE of a corpus-wide problem, which frees effort for the specific queues that remain - the 815 same-paper objections and the stale-correction queue - rather than retiring the topic.
- Are the 130 template-plus-no-engagement notes actually weak? Reading ten found mostly false positives. Reading all 130 is a couple of hours and would convert a ceiling into a count.
- Is the 8-note WRAD cluster a pattern worth encouraging? One strong argument deliberately propagated across the thread it refutes is good practice that a reuse metric penalises. If so, the graph may want a way to mark a propagated argument as such.
- Does the recent-tail bias apply to other quality measures taken from recent work? The sampling error here was not about denominators but about which slice was read, and nothing in this project currently checks for it.
- Do counter-notes exist outside node JSON - in edge or figure files? Not walked here, the same collector-scope question that hid 663 PMIDs until the citation collector was widened.
- E1strong confidence 4 of 5: Replicated in peer-reviewed work by more than one group.analysis
Across all 8,080 counter_evidence notes in 1,749 node entries, 302 (3.7%) are exact duplicates after normalisation and 378 (4.7%) fall in a near-duplicate cluster at token-set Jaccard 0.75, leaving 7,702 singleton notes and a reuse factor of 1.04 notes per distinct argument.
counter-note-boilerplate-census/run.py: near-duplicate clustering of every counter_evidence note in the graph · Oncofusion knowledge graph (internal analysis) 2026What this evidence cannot show
Run 2026-09-20 over node JSON only. **This is a FLOOR on reuse**: a class-level objection reworded each time is unique text and counts as distinct, so the true number of distinct arguments is lower than 7,799. Digits are stripped before comparison so two notes differing only in a count collapse together - deliberate, but it cannot separate a per-entry quantitative objection from boilerplate with a number swapped in. Near-duplicate search blocks on the 4 rarest tokens per note and skips tokens in more than 400 notes, so errors run toward finding FEWER duplicates than exist.
Evidence that argues against this (1)
- A LOW REUSE FACTOR IS NOT EVIDENCE THE ARGUMENTS ARE GOOD, only that they are textually distinct. Eight thousand individually-worded weak objections would score exactly as well as eight thousand strong ones on this measurement. The companion class-level analysis exists because this number cannot answer the question the ledger asked, and even that one is lexical rather than semantic. Neither instrument reads for soundness, and nothing here should be quoted as showing the counter-arguments are correct. Oncofusion knowledge graph (internal analysis) 2026
- E2moderate confidence 3 of 5: One strong study, or several smaller consistent ones.analysis
21.3% of counter-notes invoke a named class-level template, but only 130 notes (1.6%) both invoke one and score below 0.05 IDF-weighted engagement with their own claim, while 71.8% invoke no template and engage specifically - so the ledger's caveat that class-level objections dominate, drawn from a 53-note sample of the most recently written notes, does not generalise to the corpus.
counter-note-boilerplate-census/class_level.py: class-level template detection cross-tabulated against engagement with the note's own claim · Oncofusion knowledge graph (internal analysis) 2026What this evidence cannot show
**Moderate, not strong, because engagement is lexical and its low tail is dominated by paraphrase.** Reading the ten lowest-scoring template notes found most to be correct specific objections that reuse none of the claim's rare tokens, so 130 is an overcount of unknown size and must not be quoted as a defect count. The template list was built from the ledger's own examples plus reading, so it finds the families it was told to look for and misses any nobody has named - the 21.3% is a floor. Notes whose evidence item carries no claim sentence are excluded, so this denominator is stated rather than silently dropped.
Evidence that argues against this (1)
- THE LEDGER'S SAMPLE WAS NOT WRONG ABOUT ITSELF. 35 of the last 53 notes really may have been class-level; this analysis does not re-examine that sample and has no reason to dispute it. What it shows is that the recent tail is not representative - late-cycle notes are written onto already-well-defended claims where a class-level objection is often all that honestly remains. So the right reading is that the ledger observed a real property of recent work and generalised it, not that it observed nothing. Oncofusion knowledge graph (internal analysis) 2026
THE SAME FAMILY OF ERROR, A NEW VARIETY: those were about what a check WALKS (a dropped denominator, an unreachable region); this is about which SLICE a sample represents. A measurement can cover its denominator honestly and still generalise from the wrong part.
THE OTHER AXIS OF THE SAME QUESTION: independence of the SOURCE behind an objection, where this measures distinctness of the objection itself. 55.8% of counter-notes cite a different paper than the claim they answer.
THE COMPANION RULE ON HOW TO READ A QA PERCENTAGE: confidence grades the claim rather than the source, and likewise counter-argument coverage counts notes rather than arguments.
a second random-sample read of counter-evidence quality reaching the same direction: the standing worry is largely not borne out, and the residue is narrower than feared