Oncofusion

start / graph / ana-the-grading-rule-applied-and-orcid-rejected

Analysisunrated confidence 0 of 5: unrated

The grading rule applied to twenty citations, and ORCID tested as a clustering key and rejected: zero coverage before 2010

also called grading rule applied · ORCID coverage test

We wrote a rule for how confident to be in each piece of evidence, then actually applied it. We also tested whether a standard author identifier could tell laboratories apart, and it cannot - the older papers do not carry one.

The same thing, for a scientist

Two open questions from `concept-the-confidence-grading-rule` and `ana-how-many-laboratories-is-this-graph-standing-on`, both closed. **Applied the rule to 20 citations across 4 papers**, chosen as the confidence splits with claim overlap above 0.8 - the band where two graded claims are near-identical text and one grade must be wrong. **ORCID is not usable** as a replacement for surname clustering: across 238 papers from the graph's fifteen largest groups, ORCID appears on **10.2% of author slots** and **0% of pre-2010 papers**. And a third open question was simply wrong: `find-contradictions.py` **already** filters confidence splits to spread >= 2 with claim overlap >= 0.30, so it was never reporting the expected case.

Go deeper
Applying the rule, and the direction it moved things

The rule: a result within the reported cohort is established; a generalisation from it is strong; an inference beyond the reported numbers is moderate or weak.

PMID 37099340, the blinatumomab infant trial, is cited 17 times and had the tiers partly inverted. Its own primary results - 30 infants, 2-year disease-free survival 81.6% against 49.4% historical, primary safety endpoint met - were graded strong in several entries, while the background epidemiology the paper quotes in its introduction - "most relapses occur during treatment", "3-year EFS below 40%" - was graded established in one and moderate in three others.

Fourteen regraded: twelve up to established (the trial's own results) and two down to moderate (background it cites rather than measures). One was left alone because its claim mixes the trial's result with its background, which is the case the rule does not cover.

Three further papers above 0.8 overlap, all resolving the same way - a result within the reported study is established: PMID 8134107 (the 1994 EPS15/AF-1p cloning, two entries), PMID 12620411 (FLT3 mutations in 5 of 30, one entry), PMID 37679565 (CRISPR-ChIP partitioning of H3K79, three entries).

Confidence splits fell from 23 to 20, and the remaining flag on 37099340 dropped from 1.00 to 0.44 claim overlap - the identical-text pairs are gone and what is left is two genuinely different claims from one paper, which the rule says is expected.

ORCID cannot replace the surname proxy, and the reason is the graph's own age

ana-how-many-laboratories-is-this-graph-standing-on clusters papers by folded last author and names the obvious fix: "re-cluster using ORCID or PubMed's own author disambiguation rather than surname-plus-initial. That replaces a proxy with an identifier." Tested on the 238 papers belonging to the fifteen largest groups:

erapaperswith any ORCID
pre-2010740 (0%)
2010-20191077 (7%)
2020+5737 (65%)
all23844 (18%)

Across all author slots, ORCID is present on 10.2%.

So the fix would make the measurement worse, not better. This graph's foundational literature is pre-2010 - the KMT2A cloning papers, the menin work, the recombinome series that begins in 1998 - and that is exactly where the concentration sits. Clustering on ORCID would silently split every pre-2010 laboratory into singletons and inflate apparent independence, which is the opposite of the correction the open question was asking for.

The honest position: surname-plus-initial remains the best available key for this corpus, with its known contamination mode, and the 12.8% overcount stands as a lower bound.

Still open
  1. Raise the claim-overlap threshold in find-contradictions.py from 0.30 to about 0.6 and see how many of the remaining 20 splits survive. One case suggests 0.44 is a false positive and one case is not a calibration.
  2. Test a hybrid clustering key - ORCID where deposited, folded surname otherwise - which would sharpen the post-2020 tail without abandoning the pre-2010 corpus.
  3. Have someone who did not write the grading rule check the eighteen upgrades. A rule that mostly raises confidence, applied by its author, is the shape of a problem.
Every source behind this entry (3)
  1. E1established confidence 5 of 5: Multiple independent lines of evidence; treated as settled in the field.analysis

    Applying the grading rule to the four confidence splits with claim overlap above 0.8 regraded 20 citations across PMIDs 37099340, 8134107, 12620411 and 37679565, moving eighteen up to established and two down to moderate, and reduced reported confidence splits from 23 to 20.

    Measuring confidence-grade consistency per cited paper across the graph · Oncofusion knowledge graph (internal analysis) 2026
    What this evidence cannot show

    A count of this project's own edits on 2026-09-20, so established as a fact about the corpus. **The regrade for PMID 37099340 was applied by regular expression** - claims matching trial-result tokens went to established, claims matching background-epidemiology tokens went to moderate - which is mechanical and therefore reproducible, and also blind to any claim whose wording falls outside both token sets. One such claim was detected and left alone; others may not have been detected.

    Evidence that argues against this (1)
    • **Eighteen upgrades against two downgrades is a suspicious ratio** for a rule written by the same author who then applied it. The direction is defensible - a trial's own primary endpoint genuinely is a fact about what the trial found - but a rule that mostly raises confidence deserves scrutiny from someone who did not write it. Oncofusion knowledge graph (internal) 2026
  2. E2strong confidence 4 of 5: Replicated in peer-reviewed work by more than one group.database

    ORCID identifiers appear on 10.2% of author slots and 18% of papers across the 238 papers belonging to the graph's fifteen largest research groups, with 0% coverage before 2010, 7% for 2010-2019 and 65% for 2020 onwards.

    What this evidence cannot show

    Measured 2026-09-20 from PubMed efetch XML. **The sample is the largest groups' papers, not a random sample of the graph** - those groups skew older, which is precisely why they are largest, so overall graph coverage is probably somewhat higher than 18%. PubMed carries ORCID only where the publisher deposited it, so this measures deposition rather than whether the author has an ORCID.

    Evidence that argues against this (1)
    • A 65% coverage rate for 2020+ means ORCID would work for the recent literature, so a hybrid key - ORCID where present, surname where not - is available and was not tested. It would help least where the concentration is worst, but 'unusable' overstates it; 'unusable as a sole key for this corpus' is the accurate claim. Oncofusion knowledge graph (internal analysis) 2026
  3. E3established confidence 5 of 5: Multiple independent lines of evidence; treated as settled in the field.analysis

    find-contradictions.py already restricts reported confidence splits to those at least two grades apart with claim-word overlap of at least 0.30, so it was never reporting the adjacent-grade case that the grading rule says is expected.

    Measuring confidence-grade consistency per cited paper across the graph · Oncofusion knowledge graph (internal analysis) 2026
    What this evidence cannot show

    Read from the script on 2026-09-20. The 0.30 overlap threshold is loose enough to admit genuinely different claims from one paper - the remaining 37099340 flag sits at 0.44 and is two different claims - so the report still contains false positives, just not the ones the open question alleged.

    Evidence that argues against this (1)
    • This corrects an open question this project wrote about its own tooling, which is the third time in this stretch that a complaint about an instrument dissolved on reading the instrument. The pattern argues for reading the code before proposing to change it. Oncofusion knowledge graph (internal) 2026
Connected to