Oncofusion

start / graph / ana-forty-two-percent-of-the-full-text-rise-is-document-length-and-my-own-entry-overstated-it

Analysisstrong confidence 4 of 5: Replicated in peer-reviewed work by more than one group.

The length-matched control I demanded of myself: 42% of the abstract-to-full-text rise is reproduced by an unrelated same-field paper, so the specificity I claimed for it was roughly half what I reported

also called is the full-text rise just document length · same-field decoy control · own 0.571 vs mean decoy 0.365 · the CDK9 entry is its own internal control

Yesterday I showed that reading whole papers instead of their summaries roughly tripled how well citations matched their claims, and then wrote down the obvious objection: a longer document shares more words with anything. Scoring each claim against unrelated papers of similar length shows about two-fifths of that improvement was the document's length rather than its contents.

The same thing, for a scientist

Cycle 133 reported the median claim-to-paper overlap rising from **0.217 (abstract)** to **0.571 (full text)** and used it to confirm the U4 explanation. I wrote the objection into that entry's own `counter_evidence` - *"a rise is exactly what a roughly fifty times longer document would produce regardless of whether the support is present"* - and into its `open_questions` as *"add a length-matched control"*. **This is that control, and it deflates my own headline.** **Design.** Each of the 20 claims was scored against its own paper's full text **and** against the full text of the other five papers in the pool. Those decoys are a deliberately **hard** baseline: all six are chromatin/leukaemia papers, several on menin, KMT2A or H3K4 methylation, of comparable length (6,788-15,423 words). A decoy from an unrelated field would have flattered the result [E1]. **The decomposition, which is the result:** | | median overlap | |---|---:| | abstract | **0.217** | | **mean unrelated same-field paper** | **0.365** | | own paper, full text | **0.571** | | best of five decoys | 0.500 | **Of the +0.354 rise, +0.148 (42%) is reproduced by papers that are not the cited one**, leaving **+0.206 (58%)** paper-specific. The margin over the *best* decoy is only **+0.108** [E1]. 🔵 **The pre-registered C1 branch still fires, and it is worth something.** Own-paper overlap beats **every** same-field decoy for **12 of 20** claims. Against a conservative decoy set that is a real signal - it just is not the signal I described. 🔴 **So cycle 133's framing was wrong and is amended.** "Full text triples the low scores" implied the tripling was about the right paper. The correct statement is **own 0.571 against 0.365 for an arbitrary same-field paper**, and the honest headline is the **+0.108 median margin over the best decoy**, not the +0.354 rise. 🟠 **And the two instruments disagree, which I nearly got wrong.** I expected the pairs failing this decoy test to be exactly the seven the cycle-133 distinctive-term check flagged. **They are not: 4 of the 8 decoy failures were flagged, and so were 4 of the 12 that pass.** The tests measure different things and neither adjudicates a citation alone. `mechanism-nup98-fg-condensate-assay-audit` has the third-best gap (**+0.285**) while failing the term check - correct in both cases, because its claim is *"the study reports **no** FRAP"* and everything in it except FRAP is highly specific to that paper.

Go deeper
An internal control I did not design, and it is the strongest evidence here

One entry - mechanism-cdk9-and-menin-inhibition-are-synergistic-not-sub-additive - contributes two claims citing the same paper (PMID 40979068), and they land at opposite ends of the table:

claimabstractownbest decoygap
"A PubMed query on 2026-09-20 combining revumenib, SNDX-5613, VTP-50469 ... returned exactly one record"0.1720.2760.345-0.069
the substantive synergy finding0.3140.8860.514+0.372

Same entry, same paper, same tokeniser, opposite verdicts - and each is the verdict the claim's content predicts. A claim describing this project's own search terms does worse than an unrelated paper; a claim describing the paper's result beats every decoy by the largest margin in the set.

That is the clearest available demonstration that the measure has genuine resolution, and it arrived by accident rather than by design. The three supplement-derived Zhou claims (gene-nlrc5, gene-ciita, ana-the-transactivator-split...) all sit on the negative side, which is what "the support is in a file the text does not contain" should look like.

What the worst performer tells us

absence-partner-stratified-response-in-menin-inhibitor-trials scores 0.182 against its own paper and 0.455 against the best decoy - a gap of -0.273, the worst in the set.

Read the claim and this is unsurprising rather than alarming: it says outcome reporting after menin-inhibitor failure has begun to appear, which is the natural place a partner-stratified analysis could be added retrospectively. That is a forward-looking statement about what the literature does not yet contain, attached to a paper as an existence proof that the reporting has started. It is the absence-claim failure mode again, in a different dress: the instrument looks for the claim's content in the paper, and the claim's content is largely about the paper's absence of content.

Five of the eight decoy failures are of this family - self-referential, forward-looking, or supplement-derived. That is not a defect in the citations; it is the measure being asked a question it cannot answer.

Facts and figures
scriptanalyses/is-the-full-text-rise-just-document-length/run.py
statusrun
run date2026-09-24
pool size6
decoys per claim5
claims scored20
median abstract0.217
median mean decoy0.365
median best decoy0.5
median own fulltext0.571
median gap own minus best decoy0.108
claims beating every decoy12
rise total0.354
rise length and topic component0.148
rise paper specific component0.206
length and topic share PCT42
preregistered branch firedC1, but the margin is +0.108 over the best decoy rather than the +0.354 rise cycle 133 implied
agreement with the distinctive term checkpoor: 4 of 8 decoy failures were term-flagged, and 4 of 12 decoy passes were also term-flagged
document lengths words
409790686,78840979068: 6788413515137,20041351513: 7200391580679,44039158067: 94403898961512,14438989615: 121444019371512,91740193715: 129173416306915,42334163069: 15423
document lengths words — the numbers behind the chart
{
  "34163069": 15423,
  "38989615": 12144,
  "39158067": 9440,
  "40193715": 12917,
  "40979068": 6788,
  "41351513": 7200
}
measurements
[
  {
    "quantity": "median claim-to-paper overlap by comparison target",
    "value": {
      "abstract": 0.217,
      "unrelated_same_field_paper": 0.365,
      "best_of_five_decoys": 0.5,
      "cited_paper_full_text": 0.571
    },
    "unit": "fraction of claim content words present in the scored text",
    "subject": "20 claims from the lowest-scoring tail of the cycle-132 run",
    "universe": "6 papers, each claim scored against its own paper and 5 same-field decoys"
  },
  {
    "quantity": "share of the abstract-to-full-text rise reproduced by an unrelated same-field paper",
    "value": 42,
    "unit": "percent",
    "subject": "the median rise of +0.354",
    "universe": "the same 20 claims; an upper bound because the decoys are topically adjacent by construction"
  }
]
Still open
  1. Run the same control with decoys drawn at random from the corpus's 1,908 cited papers rather than from the low-scoring tail. The current decoys are topically adjacent by construction, which makes 42% an upper bound on the confound and leaves the absolute baseline unestablished.
  2. Hold claim length constant when comparing self-referential to substantive claims. The CDK9 internal control is confounded with claim wordiness and is a single accidental pair.
  3. The decoy test and the distinctive-term test agree on only half the flagged pairs. Neither is a citation verdict on its own, and no combined rule has been specified - that rule is what a future audit actually needs.
Every source behind this entry (2)
  1. E1strong confidence 4 of 5: Replicated in peer-reviewed work by more than one group.

    Scoring 20 claims against their own paper's full text and against five same-field decoy papers of comparable length (6,788 to 15,423 words) gives median overlaps of 0.217 for abstracts, 0.365 for an unrelated same-field paper, 0.500 for the best of five decoys and 0.571 for the cited paper; own-paper overlap beats every decoy for 12 of 20 claims with a median margin of +0.108.

    What this evidence cannot show

    Run 2026-09-24 (cycle 134). Only six papers were available, so each claim has five decoys and the decoy distribution is coarse - 'beats every decoy' is a rank statistic over five values and cannot support a p-value. Same-field decoys make the test conservative for C1 but mean a failure may reflect vocabulary overlap between near-duplicate subjects rather than a problem with the citation. Document lengths differ by more than twofold across the pool and are reported per paper rather than matched exactly, so 'length-matched' describes the intent and not a strict design.

    Evidence that argues against this (2)
    • The decoy pool is drawn from the papers already implicated in the low-scoring tail, which are topically adjacent to one another by construction. This inflates decoy scores and therefore UNDERSTATES paper specificity - so the 42% length-and-topic share is an upper bound on the confound, and the true paper-specific component may be larger than 58%. Symmetrically, no truly random decoy set was tried, so the absolute level of the 0.365 baseline is not established for the corpus as a whole.
    • A word-set overlap measure rewards claims that quote a paper's vocabulary and penalises claims that paraphrase it, independent of whether the paper supports them. Two of the three highest gaps are claims written with the paper open; that is a property of drafting style, not of citation quality, and this control does not separate the two.
  2. E2strong confidence 4 of 5: Replicated in peer-reviewed work by more than one group.

    Within one entry, two claims citing the same paper (PMID 40979068) land at opposite ends of the decoy comparison: the claim describing this project's own PubMed query scores 0.276 against its own paper versus 0.345 for the best decoy (gap -0.069), while the substantive synergy claim scores 0.886 versus 0.514 (gap +0.372), the largest margin in the set.

    PMID 40979068DOI
    What this evidence cannot show

    The stored title was refetched from PubMed in cycle 132 and matched verbatim; the DOI is recorded from the corpus's own citation block and was not independently re-resolved in this run. This is a single paired observation, not a designed experiment - it is reported because it happened to be present, and one pair cannot establish the measure's sensitivity in general.

    Evidence that argues against this (1)
    • The two claims differ in length and in how much technical vocabulary they contain, not only in kind, so the contrast is confounded with claim wordiness. A designed version would hold claim length constant across the self-referential and substantive categories, which this accidental pairing does not.
Connected to