Oncofusion

start / graph / ana-full-text-raises-the-low-overlap-scores-and-the-seven-that-stay-low-are-four-instrument-blind-spots

Analysisstrong confidence 4 of 5: Replicated in peer-reviewed work by more than one group.

U4 was assumed, so I tested it: full text raises the low claim-to-paper overlap scores with a +0.108 median margin over same-field decoys (42% of the raw rise is document length), and every one of the seven pairs that stays low is a blind spot in my own instrument rather than a misattached citation

also called does full text explain the low overlap scores · testing U4 · abstract 0.217 to full text 0.571 · the four blind spots of a distinctive-term check

Yesterday I explained away the worst citation scores by saying the evidence must be in the body of the paper rather than its summary, and wrote down that I had assumed this rather than checked it. Checking it showed the scores do improve for the right paper, though about two-fifths of the improvement is just that whole papers are longer, and the handful that stayed low turned out to be faults in my measuring tool rather than bad citations.

The same thing, for a scientist

🟠 **AMENDED 2026-09-24 (cycle 134): the length confound this entry declared has now been measured, and it accounts for 42% of the rise reported below.** This entry's own `counter_evidence` said the abstract-to-full-text rise was what a fifty-times longer document would produce regardless of support, and its `open_questions` asked for a length-matched control. The control was run. Scoring each claim against **five same-field decoy papers** of comparable length: | | median overlap | |---|---:| | abstract | **0.217** | | **an unrelated same-field paper** | **0.365** | | the cited paper | **0.571** | | best of five decoys | 0.500 | **So +0.148 of the +0.354 rise (42%) is reproduced by papers that are not the cited one.** The paper-specific margin over the *best* decoy is **+0.108**, and own-paper overlap beats every decoy for **12 of 20** claims. 🔴 **What is corrected:** the framing below - that full text "roughly triples" the low scores - implied the tripling was about the **right** paper. It is not. The defensible statement is *own 0.571 against 0.365 for an arbitrary same-field paper*, and the headline number should be the **+0.108 median margin**, not the +0.354 rise. 🔵 **What survives:** the pre-registered C1 branch fires - the rise does track the right paper for a majority of claims against a conservative decoy set - and **the part of this entry that carried the weight was never the overlap number.** It was the reading of all seven flagged pairs and the four blind spots, which the control does not touch. Note also that the control and the distinctive-term check **agree on only half the flagged pairs** (4 of 8 decoy failures were term-flagged; so were 4 of 12 passes). See `ana-forty-two-percent-of-the-full-text-rise-is-document-length-and-my-own-entry-overstated-it`. --- Cycle 132 scored 141 claim/citation pairs against **abstracts**, found a minimum of 0.000, read the lowest five, and attributed the low scores to the pre-registered **U4 - "abstracts only: a claim supported in a figure or supplementary table will look unsupported here."** That entry's own `open_questions` then recorded, in writing, that **U4 was assumed rather than tested**. This tests it. **Retrieval was complete.** All **6 distinct PMIDs** in the 26-pair tail resolved to PMCIDs and all 6 yielded full text; the 3 DOIs are `NOT_APPLICABLE` (a PDB deposit has no full text). Census: **FULLTEXT 6 / NOT_OPEN_ACCESS 0 / FETCH_FAILED 0 / NOT_APPLICABLE 3**, summing to 9 [E1]. 🔵 **U4 is confirmed as the general mechanism.** Scored against full text instead of abstracts, on the same tokeniser: | | abstract | full text | |---|---:|---:| | median overlap, 20 pairs | **0.217** | **0.571** | | pairs rising by >=0.15 | | **17 of 20** | The largest movers go from near-nothing to near-total: `resistance-q125-a-polycomb-deficient-cell-is...` 0.273 -> **1.000**, `hyp-paralogue-buffering-hides-dependencies` 0.200 -> **0.914**, `mechanism-cdk9-and-menin-inhibition...` 0.314 -> **0.886** [E1]. 🔴 **But the pre-registered T2 branch fired, and T2's decision rule was wrong.** Seven pairs still lack their distinctive claim terms anywhere in the full text. T2 declared those "candidate misattachments". **I read all seven. None is a misattachment.** Four distinct causes, none of which my pre-registration anticipated: | cause | pairs | evidence | |---|---:|---| | **Support is in an unretrievable supplement** | 3 | All three cite PMID 39158067 for group assignments in *"Zhou's published 9,292-locus menin and H2AK119ub target set"*, with coordinates (`chr16:10877200`, `chr16:56989555`). The paper has **7 supplementary files** including Supplemental Table 4; **none is retrievable** [E2]. | | **Gene-symbol alias** | 2 | The corpus writes **BOD1L1**, the official symbol (NCBI Gene **259282**, *biorientation of chromosomes in cell division 1 like 1*, alias **BOD1L**). The paper is titled *"**BOD1L** mediates chromatin binding..."*. My check found `BOD1L` in three other pairs and scored `BOD1L1` as absent [E3]. | | **The claim asserts an ABSENCE** | 1 | `mechanism-nup98-fg-condensate-assay-audit` claims *"The Nature study reports **no** FRAP"*. FRAP being absent from the full text **confirms** the claim. My instrument scored a verified absence as a failure. | | **The claim is about this project's own query** | 1 | *"A PubMed query on 2026-09-20 combining revumenib, SNDX-5613, VTP-50469 ... returned exactly one record"*. Those are the **search terms**, not the paper's contents. | 🟠 **So the honest verdict is neither T1 nor T2 as written.** U4 holds for 17 of 20 pairs as a measured fact; the remaining three genuinely untestable pairs are untestable for a *different* reason than U4 named - not "the support is in a supplement" but **"the support is in a supplement I cannot open"**. That distinction is the whole difference between a verified citation and an unverifiable one, and cycle 132 did not make it.

Go deeper
Two instrument failures inside one run, both of which would have produced a confident wrong answer

1. A convenience parameter returned HTTP 400 for every identifier. The first version appended &tool=oncofusion-kg&email=oncofusion to the NCBI ID converter. oncofusion is not an address, and the endpoint rejects the entire request with a 400 - batched or single. The run printed FULLTEXT 0 / NOT_OPEN_ACCESS 0 / FETCH_FAILED 6. Isolated by testing the variants: no email works, email=<a real address> works, email=oncofusion is a 400.

Only the pre-registered D1/D2 split stopped that from becoming a result. Had failures been merged with absences, the run would have reported "none of these papers is open access" - a false statement about six papers, produced by a parameter I added to be polite.

2. 0 supplementary files for all six papers, when one demonstrably has seven. The scan required xlink:href on the <supplementary-material> element. In PMC's XML the href sits on a nested <media> child. The corrected scan finds 22 files across the 6 papers.

And a third, smaller: two supplements first reported as REACHABLE (21396 bytes) and (21400 bytes) - a .pptx and a .pdf four bytes apart. Both were HTTP 200 responses serving a text/html landing page, and both 404'd on retry. The check now requires the body to be the file (PK.. or %PDF, not HTML). A 200 is not a file.

What a distinctive-term check cannot do, stated so the next one is built differently

A term-presence check over full text is a better instrument than abstract overlap, and it still cannot adjudicate a citation on its own. It fails on:

  1. Claims of absence. "The study reports no FRAP" is refuted by finding FRAP and confirmed by not finding it. The check has the polarity exactly backwards and no way to know.
  2. Symbol aliases. Official symbol in the corpus, legacy symbol in the paper. This project already records query the old gene symbol too as a rule and the instrument did not implement it.
  3. Support outside the retrievable text - supplements, figures, deposited tables.
  4. Self-referential method claims, where the claim describes the project's own search rather than the paper.

All four appeared in a 20-pair sample, which suggests they are common rather than exotic. A future version should take, per claim, an explicit expected_terms and expected_absent_terms list written by the beat that made the claim - turning an inference by the auditor into an assertion by the author.

Facts and figures
scriptanalyses/does-full-text-explain-the-low-overlap-scores/run.py
statusrun
run date2026-09-24
tail tested26
of scored141
distinct pmids6
distinct dois3
pairs with fulltext20
overlap median abstract0.217
overlap median fulltext0.571
pairs rising at least 0 1517
pairs missing distinctive terms7
misattachments found0
supplementary files seen22
supplementary files retrieved0
preregistered branch firedT2, whose decision rule was under-specified; on reading, none of the seven is a misattachment
retrieval census
NOT_OPEN_ACCESS0.000NOT_OPEN_ACCESS: 0FETCH_FAILED0.000FETCH_FAILED: 0NOT_APPLICABLE3.000NOT_APPLICABLE: 3FULLTEXT6.000FULLTEXT: 6
blind spots
claim_asserts_an_absence1.000claim_asserts_an_absence: 1claim_is_about_this_proje…1.000claim_is_about_this_projects_own_query: 1gene_symbol_alias2.000gene_symbol_alias: 2support_in_unretrievable_…3.000support_in_unretrievable_supplement: 3
retrieval census — the numbers behind the chart
{
  "FULLTEXT": 6,
  "NOT_OPEN_ACCESS": 0,
  "FETCH_FAILED": 0,
  "NOT_APPLICABLE": 3
}
blind spots — the numbers behind the chart
{
  "support_in_unretrievable_supplement": 3,
  "gene_symbol_alias": 2,
  "claim_asserts_an_absence": 1,
  "claim_is_about_this_projects_own_query": 1
}
instrument failures this run
[
  "email=oncofusion made the NCBI ID converter return HTTP 400 for every identifier; first census read FULLTEXT 0 / FETCH_FAILED 6",
  "supplement scan required xlink:href on <supplementary-material> where PMC puts it on a nested <media>; reported 0 files for a paper with 7",
  "two supplements reported REACHABLE were HTTP 200 text/html landing pages and 404 on retry"
]
measurements
[
  {
    "quantity": "median claim-to-paper overlap, abstract vs full text",
    "value": [
      0.217,
      0.571
    ],
    "unit": "fraction of claim content words present in the scored text",
    "subject": "the 20 lowest-scoring claim/citation pairs whose papers yielded full text",
    "universe": "the 26-pair tail of the 141 pairs scored in cycle 132; NOT the corpus"
  },
  {
    "quantity": "supplementary files retrieved from PMC",
    "value": 0,
    "unit": "count",
    "subject": "the first three supplementary files of each of the 6 papers",
    "universe": "22 supplementary files listed across those 6 papers"
  }
]
amended cycle 134
{
  "median_mean_same_field_decoy": 0.365,
  "median_best_decoy": 0.5,
  "length_and_topic_share_of_rise_pct": 42,
  "paper_specific_margin_over_best_decoy": 0.108,
  "claims_beating_every_decoy": "12 of 20",
  "note": "the 0.217 -> 0.571 rise reported here is 42% reproducible with an unrelated same-field paper; the framing 'full text triples the low scores' overstated specificity and is corrected. The seven-pair reading and the four blind spots are unaffected."
}
Still open
  1. Try the publisher-hosted supplement behind each DOI. This run bounds only what PMC will serve, and the three unverifiable claims all hinge on a Blood supplemental table that the publisher may serve directly.
  2. Add a length-matched control: score each claim against a RANDOM paper's full text. Without it, the rise from 0.217 to 0.571 is confounded with document length and the counter-evidence on E1 stands unanswered.
  3. Have beats declare expected_terms and expected_absent_terms per claim. All four blind spots found here would be removed by an assertion from the author instead of an inference by the auditor.
Every source behind this entry (3)
  1. E1strong confidence 4 of 5: Replicated in peer-reviewed work by more than one group.

    Across the 26 lowest-scoring claim/citation pairs from the cycle-132 run, all 6 distinct PMIDs resolved to PMCIDs and yielded full text, and scoring the 20 resulting pairs against full text instead of abstracts raised the median overlap from 0.217 to 0.571, with 17 of 20 pairs rising by at least 0.15.

    What this evidence cannot show

    Run 2026-09-24 (cycle 133). Full text was served by NCBI efetch db=pmc; EuropePMC returned HTTP 503 or 500 for every identifier throughout the run, which is why two independent routes are used - one service being down is not evidence about a paper. Overlap uses the same tokeniser as the abstract run so the two medians are comparable, but it remains a crude word-set measure. The tail tested is 26 of 141 pairs, chosen as the lowest scorers; the median of the tail is not the median of the corpus.

    Evidence that argues against this (1)
    • A rise in overlap is exactly what one would expect from scoring against a document roughly fifty times longer, regardless of whether the support is present: a longer text contains more words and therefore intersects more of any claim. This run did NOT include a length-matched or shuffled-paper control, so the rise establishes that the words are present in the full text and does not by itself establish that the paper supports the claim. The distinctive-term check is the part that carries actual weight, and it is qualitative.
  2. E2strong confidence 4 of 5: Replicated in peer-reviewed work by more than one group.

    The six papers carry 22 supplementary files between them and none could be retrieved: eight URLs return HTTP 200 with content-type text/html and a body of roughly 21,400 bytes beginning with a doctype declaration, and the remainder return HTTP 404; the PMC Open Access service returns 404 for PMC11561541.

    What this evidence cannot show

    Only the first three supplementary files per paper were probed, so 'none retrieved' is established for the probed subset and is an inference for the rest. Two routes were tried: the PMC /bin/ path and the PMC OA service. Publisher-hosted copies behind the DOI were NOT tried and may well work - this bounds what is reachable from PMC, not what is reachable at all.

    Evidence that argues against this (1)
    • Unreachable is not absent and not unsupported. The citing entries state the support was read directly from supplemental Table 4 at the time, and one of them records independent confirmation by a second beat. This census therefore cannot be read as evidence against those claims; it establishes only that they cannot be re-verified by this route now, which is a statement about the archive rather than about the corpus.
  3. E3established confidence 5 of 5: Multiple independent lines of evidence; treated as settled in the field.

    BOD1L1 is the official NCBI gene symbol (Gene ID 259282, 'biorientation of chromosomes in cell division 1 like 1') with BOD1L listed among its aliases, while PMID 38989615 is titled 'BOD1L mediates chromatin binding and non-canonical function of H3K4 methyltransferase SETD1A', so the corpus and the paper use different symbols for the same gene.

    PMID 38989615DOI
    What this evidence cannot show

    Gene record fetched from NCBI Gene on 2026-09-24 by symbol query restricted to human; the title was fetched from PubMed esummary the same day. The DOI is recorded from the corpus's own stored citation and was not independently re-resolved in this run.

    Evidence that argues against this (1)
    • An alias mismatch explains why the term check failed, but it does not by itself establish that the paper supports the specific quantitative claims attached to it - a 618-sgRNA tiling screen in MV4-11 finding a dependency confined to the N-terminal Shg1 domain. That remains on the original beat's reading; this evidence item only removes the false alarm.
Connected to