Oncofusion

start / graph / ana-twenty-five-of-thirty-six-self-measuring-analyses-were-computed-against-a-graph-that-no-longer-exists

Analysismoderate confidence 3 of 5: One strong study, or several smaller consistent ones.

Twenty-five of thirty-six analyses that measure this graph were computed against a graph that no longer exists, and seven stamp a date that never moves

also called analysis staleness · stale stored results · hardcoded generated date

This project keeps small programs that measure itself, and their saved answers are quoted later as if still true. Most of them were computed when the graph was a third of its current size, and one such answer had quietly reversed while it was still being cited as a live problem.

The same thing, for a scientist

**Why this was built.** Re-running `analyses/core-complex-interactome-gap` this cycle — a cheap instrument nobody had re-run for several cycles — did not merely refresh a number, it **inverted a conclusion**. Both that entry and `docs/coverage.md` stated that the worst-covered bait was *"the one the therapeutic hypothesis is about: MEN1, 4 of 30"*. MEN1 is now **17 of 30 (56.7%), the second-best covered**, and the worst is KMT2A's own partner list at 41 of 101. Ordinary tranche work had closed the gap while the paragraph went on being quoted as a live weak spot [E2]. **The distinction the instrument turns on.** An analysis of EXTERNAL data — DepMap, FAERS, HPA — does not go stale because this graph grew. An analysis that reads `data/graph/nodes` measures *this graph*, so every entry written afterwards makes its stored result less true. `scripts/check-analysis-staleness.py` reports only the second class, and detects the class by reading each script rather than from a list, because a hand-maintained list is the thing that goes stale next. **The measurement.** Staleness is not wall-clock age — a two-week-old result over an unchanged graph is fine. It is how far the graph moved: the commit that last wrote `results.json` gives a node count at that moment, against 1,954 now. **25 of 36 self-measuring analyses have drifted at least 15%**, and four were computed against a graph less than a third of the present size: `structural-tractability-ligandability` (528 nodes, +270%), `noncoding-gap-ledger` (531, +268%), `unopposed-claim-load` (719, +172%), `source-independence-by-lab` (737, +165%) [E1]. **A second defect found on the way.** **Seven stored results carry a HARDCODED `generated` date**, so the field never moves when the data does and reads as provenance it does not have: `literature-recall`, `load-bearing-citations`, `literature-recall-narrowed`, `abstract-only-recoverable`, `reading-depth`, `normal-tissue-exposure`, `aff1-interactome-gap`. Two of them state a date *later* than the commit that wrote them. **Twenty-two more hardcode the field and happen to still agree — latent, not clean, because the next re-run makes each one wrong.** `core-complex-interactome-gap` was an eighth such case and is fixed [E2]. **One apparent case is not a defect, and separating it mattered.** `source-independence-by-lab` also shows a date mismatch, but its script *computes* the date with `time.strftime`; the disagreement only means `results.json` was re-run after being committed. Reporting all eight as hardcoded would have been a false accusation against the one script doing it correctly, so the reporter classifies by cause [E1]. **This instrument failed the way it was built to catch, on its first run.** The detector originally matched only the literal string `data/graph/nodes` and therefore missed `core-complex-interactome-gap` — the very analysis that prompted it — because that script builds the path with `os.path.join(ROOT, "data", "graph", "nodes")`. The idiomatic spelling was invisible, so the first report said **13 of 22** where the truth is **25 of 36**: a detector that finds the sloppy form and misses the clean one undercounts precisely the careful code.

Facts and figures
scriptscripts/check-analysis-staleness.py
reportdocs/analysis-staleness-report.json
graph nodes now1954
self measuring analyses36
drifted 15pct or more25
worst driftstructural-tractability-ligandability, computed over 528 node files against 1,954 now (+270%)
hardcoded generated but latent22
date mismatch but benignsource-independence-by-lab -- computes the date with time.strftime, so the mismatch only means results.json was re-run after its commit. Classified separately so the one script doing it correctly is not accused.
how staleness is definedChange in node-file count between the commit that last wrote results.json and HEAD. Not wall-clock age, because a two-week-old result over an unchanged graph is fine.
the instruments own first failureThe detector matched only the literal 'data/graph/nodes' and missed scripts using os.path.join(ROOT, 'data', 'graph', 'nodes') -- including the analysis that prompted the whole exercise. First run reported 13 of 22; the truth is 25 of 36. The fixed pattern matches the POSIX path, the Windows path and the os.path.join segment form.
what this does NOT showDrift does not make an analysis wrong. It makes it unverified against the current corpus. Several drifted analyses will return the same answer, and a few are about one beat that has not grown at all.
shown for fusionfalse
absence classunclassified-not-checked
first drifted analysis re run and it was UNCHANGEDanalyses/unopposed-claim-load, drifted +172% (719 node files at its commit against 1,954 now), was re-run 2026-09-22 and returns 0 unopposed of 9,286 evidence items -- the same conclusion it recorded at 719 nodes. So the first drifted analysis actually re-run did NOT change, which is direct evidence for this entry's own counter-argument that drift risks manufacturing busywork. It is also the expected result for this particular measure: counter-argument coverage has been at its 100% ceiling for several cycles and the ledger already records that it has stopped discriminating, so an unchanged answer here is what a saturated metric looks like rather than evidence that drift is harmless in general. The instrument's value is unproven until a drifted analysis whose measure is NOT at a ceiling is re-run.
a false positive I declared and then disprovedI first called noncoding-gap-ledger a false positive of my own instrument, reasoning that it reads data/graph/nodes/genome/ which held 24 files at its commit and holds 24 now, so its +268% had to be contamination from other beats. Then I re-ran it and all six headline numbers had moved. The analysis also selects every node of type noncoding_element wherever it is filed, so it genuinely reads the whole corpus. A directory count is a proxy for a selection predicate and can be wrong in either direction. The lesson is narrow and worth keeping: re-run the analysis, do not reason about its scope from its paths.
five re runs and the pattern they showFive drifted analyses have now been re-run rather than argued about. COUNTS all moved; RATIOS mostly held. evidence-redundancy: all 7 numeric keys changed (3,585 -> 9,320 evidence items) but restatement_fraction went 0.1869 -> 0.1849, essentially unchanged. load-bearing-citations: all 9 changed, including the figure docs/coverage.md quotes -- sources appearing in counter-evidence and nowhere as support, 467 -> 823. system-of-record: both keys changed. noncoding-gap-ledger: all six headline numbers changed. structural-tractability-ligandability: 91 -> 415 accessions and the no-drug-like-ligand fraction 53.8% -> 61.7% -- a RATIO that did move, and in the unwelcome direction. unopposed-claim-load alone was unchanged, and its measure is at a 100% ceiling. So the practical rule is: a stored PROPORTION from this corpus is usually still good, a stored COUNT usually is not, and the exception is any proportion whose denominator was a curated subset rather than the whole graph.
the counts move ratios hold rule is QUALIFIED 2026 09 22Three more drifted analyses were re-run in cycle 54 and TWO OF THEIR RATIOS MOVED, which qualifies the rule this entry derived from the first five. absence-claims-bounded-or-not: absence assertions 1,878 -> 3,445, bounded 127 -> 276, and the BOUNDED FRACTION 0.0676 -> 0.0801 -- an 18% relative improvement, not noise. cross-entry-contradiction-screen: claims screened 4,053 -> 5,910, candidates 619 -> 1,076, and the CANDIDATE FRACTION 0.1527 -> 0.1821, moving in the worse direction. non-gene-antigen-blind-spot: only the denominator moved. SO THE RULE NEEDS ITS CONDITION STATED: a ratio holds when the measure is SATURATED (like counter-argument coverage at its 100% ceiling) or when corpus COMPOSITION is stable. It moves when composition is changing -- and composition has been changing here deliberately, because recent cycles wrote bounded absences under rule 80, which is exactly why the bounded fraction rose. 'Counts move, ratios hold' was drawn from five analyses of which four were composition-stable; it is a useful default and not a law.
a substantive finding from the re runOnly 276 of 3,445 absence assertions in this corpus are BOUNDED -- 8.0%, leaving 3,169 unbounded. The fraction is improving (6.8% -> 8.0%) but the absolute number of unbounded assertions nearly doubled, because the corpus grew faster than the discipline spread. This is the 'absence of testing is not a negative result' rule measured rather than asserted, and it is a bigger standing weak spot than the counter-argument column that replaced it: counter-argument coverage sits at 100% and discriminates nothing, while absence-bounding sits at 8% and discriminates everything.
i tried to explain away the 8 percent and could notHaving quoted '8.0% of absence assertions are bounded', I suspected the figure was inflated by sentences that cannot or need not be bounded, and added a kind classifier to test that: META (the sentence COUNTS absences, e.g. '41 sentences assert that something has not been done' -- bounding it is a category error), CORPUS-INTERNAL (the absence is about THIS graph, falsifiable by grep rather than by a literature query), and LITERATURE (the only class where an unbounded absence is a real defect). THE EXPLANATION FAILED. All three kinds bound at nearly the same rate: literature 250/3,072 = 8.1%, corpus-internal 28/364 = 7.7%, meta 1/11 = 9.1%, against 8.1% overall. The 8% is not a measurement artefact; it is the practice. I was wrong to suspect it and the decomposition was worth doing precisely because it could have explained the number away and did not.
what the decomposition DID buyIt sharpened the actionable list from 9 to 3. Unbounded absences sitting inside an evidence[].claim AND of the literature kind: 3 -- absence-no-menin-inhibitor-occupancy-dataset-in-an-infant-derived-kmt2a-aff1-line evidence[1], gene-med26 evidence[1], gene-kmt2a evidence[5]. The other six were meta-counts or absences about this corpus. And the measure is RESPONSIVE: gene-men1 and hyp-ncbaf-not-canonical-baf-is-the-kmt2a-dependency were on the list of 9 this cycle, were bounded with fetched records and controls, and have dropped off it.
hardcoded generated and already wrong
[
  "literature-recall",
  "load-bearing-citations",
  "literature-recall-narrowed",
  "abstract-only-recoverable",
  "reading-depth",
  "normal-tissue-exposure",
  "aff1-interactome-gap"
]
three drifted analyses actually re run
{
  "unopposed-claim-load (+172%)": "UNCHANGED. 0 unopposed of 9,286 evidence items, the same conclusion it recorded at 719 nodes. Expected, because counter-argument coverage has been at its 100% ceiling for several cycles; an unchanged answer is what a saturated metric looks like.",
  "noncoding-gap-ledger (+268%)": "ALL SIX HEADLINE NUMBERS CHANGED: 23 -> 26 entries, 18 -> 21 entries with absence claims, 41 -> 55 absence claims total, 12 -> 18 HIGH, 14 -> 17 MEDIUM, 15 -> 20 LOW. So 12 of this beat's absence claims were graded HIGH when the stored result was written and 18 are now, which is the kind of number a cycle would quote.",
  "structural-tractability-ligandability (+270%)": "SUBSTANTIALLY CHANGED. Coverage went 91 -> 415 accessions (324 added, 0 removed, so the new set is a clean superset and the comparison is licensed). Median solved fraction 46.7% -> 58.8%, proteins with at least one drug-like co-crystal ligand 42 -> 159, proteins with none 53.8% -> 61.7%. The newly covered proteins are LESS tractable than the original 91, which is what one would expect if the original set was the well-studied core -- and it means the graph's druggability picture is worse than the stored figure implied, not better.",
  "verdict": "Two of three re-runs changed materially and the third was at a metric ceiling. The instrument is worth running before quoting any stored number."
}
How this could bite us back
  • This entry criticises this project's own instruments and could be used to argue that any inconvenient stored number is stale. It should not be: drift is a prompt to re-run, not a licence to discard. Where a drifted analysis is quoted in a conclusion that matters, the honest move is to re-run it or to date the quotation, not to drop it.
  • Nothing here touches a clinical or therapeutic claim. It is a provenance audit of internal measurement code.
Still open
  1. Five of the 25 drifted analyses have now been re-run: four changed materially, one was at a metric ceiling, and the pattern is that absolute counts move while ratios mostly hold. The remaining 20 are unaudited, and the ones worth re-running first are those whose ratio has a curated-subset denominator, since that is where structural-tractability's 53.8% -> 61.7% shift came from.
  2. Can the instrument detect amendment churn as well as growth? A cycle that rewrites 200 entries without adding any leaves drift at zero while invalidating every stored result, and that is the shape of several recent cycles here.
  3. Should `generated` be computed everywhere by convention, and should a checker enforce it? Twenty-nine of 36 self-measuring analyses hardcode it.
Every source behind this entry (2)
  1. E1moderate confidence 3 of 5: One strong study, or several smaller consistent ones.analysis

    Of 36 analyses in this repository whose scripts read data/graph/nodes, 25 have a stored results.json last committed when the node corpus was at least 15% smaller than its current 1,954 files, four of them at less than a third of the current size (528, 531, 719 and 737 nodes). Seven stored results carry a hardcoded `generated` date that already disagrees with the commit that wrote them, 22 more hardcode it and currently agree, and one further mismatch is benign because that script computes the date.

    What this evidence cannot show

    Node-file count is a crude proxy for the graph an analysis actually read: an analysis scoped to one beat is unaffected by growth in another, so a high drift percentage is a prompt to check rather than a defect in itself. The instrument is also blind to AMENDMENTS -- a cycle that rewrites two hundred existing entries without adding any moves every stored result while leaving drift at zero, which is precisely the shape of several recent cycles in this project and is the reason this is a reporter rather than a gate. The self-measuring test is a source-text match, so a script that reaches the corpus by a path this pattern does not know would still be missed, exactly as the first version missed the os.path.join form.

    Evidence that argues against this (2)
    • Drift is not a defect and this framing risks manufacturing busywork. Most of these analyses will return the same answer on a re-run, because the questions they ask are about proportions rather than counts and the corpus has grown fairly evenly. The one inverted case that motivated the instrument may be unrepresentative: interactome coverage is the measure most directly moved by writing per-partner entries, which is exactly what recent cycles have been doing, so it is the analysis most likely to drift and the least likely to be typical.
    • A hardcoded `generated` date is harmless as long as nobody re-runs the script, and 22 of the 29 hardcoded cases currently agree with their commit. Calling them 'latent' is a judgement rather than a finding. The counter-argument is that this project HAS now re-run two of them in one cycle and was misled once, so the latency is short.
  2. E2moderate confidence 3 of 5: One strong study, or several smaller consistent ones.analysis

    Re-running the same instrument over the same six baits gives 89 of 217 curated human partners with their own entry (41.0%), up from 46 (21.2%), with MEN1 moving from 4 of 30 to 17 of 30 (56.7%) and KMT2A itself becoming the worst-covered bait at 41 of 101 (40.6%). The union partner count returned as exactly 217 both times, which is what licenses the comparison.

    What this evidence cannot show

    IntAct's contents move between runs, so part of any change belongs to the database rather than to this graph; the union returning exactly 217 partners both times is the check that licenses attributing the delta to our own entries, and without that check the comparison would be worthless. The measure counts the EXISTENCE of a per-protein entry, not its depth or quality, so 41.0% is structural coverage and not a saturation claim. This script was itself one of the hardcoded-date cases and stamped 2026-09-20 onto the fresh numbers, which is why the refresh was nearly misread as the original measurement.

    Evidence that argues against this (1)
    • The measure is partly self-referential: writing one entry per curated partner is exactly the activity it rewards, so a doubling of the percentage records effort rather than understanding. And the original conclusion survives the refresh -- 119 of 217 partners still have no entry, so 41.0% still does not support the ledger's grade of 'saturated for phase 1' on the interactome beat. Only the magnitude changed.
Connected to