Oncofusion

start / graph / ana-the-two-h2ak119ub-conventions-flip-sign-on-half-of-all-ratios-but-materially-on-only-two-of-68

Analysismoderate confidence 3 of 5: One strong study, or several smaller consistent ones.

Enumerating every H2AK119ub number in 168 analysis result files instead of hand-picking fourteen: the two conventions flip sign on 35 of 68 comparable ratios but MATERIALLY on only 2, the exhaustive list finds exactly one real single-convention quotation against two arithmetic coincidences, and the statistic cycle 105 cited as proof they agree is one of the two places they materially disagree

also called the exhaustive H2AK119ub audit · 51.5% of flips are noise crossing 1.0 · two material disagreements out of 68 · the third convention that was not one

A chemical mark in a public dataset was processed two different ways, and earlier work worried the two versions might contradict each other. Checking every number the project has ever computed from that mark shows the two versions do land on opposite sides of no-change about half the time, but nearly always by a hair - only two out of sixty-eight comparisons differ by enough to matter. Awkwardly, one of those two is the very comparison the previous cycle used as its example of the two versions agreeing.

The same thing, for a scientist

**Statement.** Cycle 105 filed a caveat against itself: its H2AK119ub fingerprint list was **fourteen hand-picked values from two beats**, so an entry quoting any other convention-specific number was invisible. This discharges it. **275,309 numeric leaves across 168 analysis result files** were walked and every statistic matched to its twin under the other convention by key path. **RESULT 1 - the conventions flip sign constantly and matter rarely.** Of **72 matched ratio statistics, 68 comparable: 33 agree in direction and 35 disagree.** That is **51.5%** - far above cycle 105's impression. But **only 2 of the 35 flips are MATERIAL**, with both sides outside the 10% floor. The other 33 are values like **0.9940 against 1.0035**: noise crossing 1.0, not contradiction. **The material disagreement rate is 2 of 68, 2.9%.** **RESULT 2 - and the two material ones are an embarrassment for cycle 105.** They are | statistic | `PAIRED` | `REP2` | |---|---:|---:| | `K119_C1_VTP` Q1 | **1.1325** | **0.8842** | | `K119_NT_VTP` Q3 | **1.1238** | **0.8878** | **The first is precisely the pair cycle 105 quoted as its proof that the conventions agree.** That entry wrote *1.4450 to 1.1325 and 1.0151 to 0.8842, both falling* - true of the **change**, false of the **level**: 1.1325 says the mark is enriched at Q1, 0.8842 says it is depleted. **Cycle 105 picked, as its illustration of agreement, one of the only two places in the deposit where they materially disagree.** Its conclusion survives on a different statistic - the two entries it flagged quoted the **aggregate** 1.4450, whose REP2 companion 1.0151 is also above 1 - and its entry is corrected accordingly. **RESULT 3 - the exhaustive list found one real fault, and two coincidences.** Three hits, reviewed in context as `U2` requires. **GENUINE**: `ana-q125-gse260742-deposits-a-mouse-sample-in-an-hg38-container` quotes `R_H2AK119ub(SGC/DMSO)` at `S_FLAG` as **1.4932** with no convention named; amended to carry its REP2 companion **1.3855**. **COINCIDENCE x2**: `0.7746` in a Q130 entry is an **H3K79me2** value from GSE127507/MOLM13, and `1.0333` is an **H3K4me3 matched null**. **A false-positive rate of 2 in 3 flagged hits**, which is the honest bound on this method. **The method does carry signal, and it was measured rather than asserted.** Shifting every fingerprint by +0.0003 - same magnitudes, numbers no H2AK119ub computation produced - gives **2 entries and 4 hits against the real list's 20 entries and 110 hits, a 10.0x enrichment.**

Go deeper
Three defects in my own instrument, in the order they were caught

1. The ambiguity filter suppressed the signal, not the noise. U1 as first written excluded any value also appearing under a non-mark path anywhere. Across ~10^5 leaves almost every 4-dp value collides, so it dropped 1,054 of 1,290 values - including 13 of the 14 fingerprints cycle 105 had verified by eye - and the corpus search reported 0 entries carrying both conventions where cycle 105 had found 13. Replaced by convention-specificity plus a narrow exclusion of values within 0.005 of 1.0 or 0.0.

2. The denominator contained statistics that cannot fail the test. The first rate, 14.6% over 323 pairs, was computed over a set including /tracks/ coverage metadata and /dmso_share/ and /absolute_share_change/ values - shares, which sit below 1 by construction. A test asking whether both lie on the same side of 1.0 scores every share pair AGREE for free. Restricting to ratios cut the denominator to 68 and moved the rate to 51.5%. The contaminated denominator understated it by more than threefold - the same defect class as cycle 103's D9 absorbs 619%.

3. The script ate its own output. It walked analyses/*/results.json including its own, and the denominator drifted between runs - 66 pairs at 59.1%, then 68 at 60.3%. With self excluded, a run from a deleted output and a re-run agree exactly. The drift is fixed; which key fed back was not isolated, and that is recorded rather than asserted.

The third convention that was not one, and had already been decided

Fixing a regex that mapped paired and rep1 onto the same normalised key - silently overwriting one with the other on 572 keys - surfaced REP1 as an apparent third convention, disagreeing with PAIRED on 92.3% of 26 ratio pairs, 16 of them material. That looked like the finding of the cycle.

It is not a convention. GSM8123407-412, titled H2AK119ub rep1 in GEO, are FLAG ChIPs - their depositors' own filenames read ChIP_FLAG_* - established in cycle 101. A Polycomb mark and the fusion protein disagreeing is the expected result.

And the corpus had already acted on it. Q125's criterion I4_rep1_channel_identity reads, verbatim: FAIL - rep1 is NOT established to be H2AK119ub; it is dropped from every H2AK119ub criterion. The question was settled five cycles before this beat rediscovered it. REP1 is excluded here for the same reason. PAIRED and REP2 are processing states, not replicates: PAIRED ChIPs are .filt.bam.tdf normalised against .filt.bam.tdf inputs, REP2 are bare .tdf. The 60.3% figure computed on the corrupted PAIRED column is withdrawn.

Facts and figures
questiondoes an exhaustive H2AK119ub fingerprint list find entries the hand-picked one missed, and how often do the conventions really disagree?
dischargesthe largest caveat cycle 105 filed against itself
results files walked168
numeric leaves275309
distinct values under mark paths1290
convention specific fingerprints105
new fingerprints beyond the 14 hand picked92
matched ratio pairs72
comparable68
agree in direction33
disagree in sign35
sign disagreement rate PCT51.5
MATERIAL disagreements2
material rate PCT2.9
materiality floor0.1
withdrawnthe 60.3% figure, computed on a PAIRED column corrupted by a regex that mapped paired and rep1 onto one key
absence classmeasured-not-curated
bite backThis entry licenses continued use of H2AK119ub from GSE260742 on the strength of a 2-in-68 material disagreement rate. That rate is computed against a floor borrowed from genotype variation in a different comparison, and over statistics that are not independent. If the true convention-level floor is tighter than 10%, the material count rises and this entry will have been the document that waved the mark through.
the two material
[
  {
    "statistic": "K119_C1_VTP R_by_quartile Q1",
    "PAIRED": 1.1325,
    "REP2": 0.8842
  },
  {
    "statistic": "K119_NT_VTP R_by_quartile Q3",
    "PAIRED": 1.1238,
    "REP2": 0.8878
  }
]
corpus
{
  "committed_node_files": 2477,
  "mentioning_the_mark": 123,
  "quote_both": 13,
  "single_convention_named": 2,
  "flagged": 3,
  "genuine": 1,
  "coincidences": 2,
  "false_positive_rate": "2 of 3 flagged hits"
}
the genuine fault
{
  "id": "ana-q125-gse260742-deposits-a-mouse-sample-in-an-hg38-container-that-clipped-2-76-percent-of-its-own-data",
  "value": "1.4932 (PAIRED), R_H2AK119ub(SGC/DMSO) at S_FLAG",
  "rep2_companion": 1.3855,
  "action": "amended to carry both"
}
negative control
{
  "design": "every fingerprint shifted by +0.0003",
  "real": "20 entries, 110 hits",
  "decoy": "2 entries, 4 hits",
  "enrichment": 10
}
REP1 IS NOT A CONVENTION
{
  "what_it_is": "GSM8123407-412, titled 'H2AK119ub rep1' in GEO, are FLAG ChIPs (cycle 101)",
  "already_decided": "Q125 criterion I4_rep1_channel_identity: FAIL - rep1 is NOT established to be H2AK119ub; it is dropped from every H2AK119ub criterion",
  "what_paired_and_rep2_are": "processing states - PAIRED is .filt.bam.tdf normalised against .filt.bam.tdf inputs, REP2 is bare .tdf"
}
Still open
  1. Build the decoy control from OTHER MARKS values rather than arbitrary shifts. Both false positives were numbers from H3K79me2 and H3K4me3, which is the collision the current control does not model.
  2. Does the 10% materiality floor survive? It is borrowed from input pairs bounding genotype variation. GSE299100 same-condition replicate would give a floor measured from the right quantity, and it would move this entry headline.
  3. Enumerate the 72 retained ratio pairs by hand once. The ratio filter is a regex over key names and nobody has checked what it excluded.
  4. Do H3K27me3, H3K4me3 or H3K79me2 have two processing states in this deposit? Still unchecked after two cycles of asking - and this beat shows the check is cheap now that the matching code exists.
  5. STOP AUDITING THIS DEPOSIT. Three tooling faults were found against one corpus fault. The blocked measurement is the same-condition noise floor, not another pass over H2AK119ub.
Every source behind this entry (1)
  1. E1moderate confidence 3 of 5: One strong study, or several smaller consistent ones.analysis

    Across 168 analysis results.json files and 275,309 numeric leaves, 1,290 distinct values sit under an H2AK119ub key path and 105 are convention-specific. Of 72 matched PAIRED/REP2 ratio statistics, 68 are comparable: 33 agree in direction, 35 disagree in sign (51.5%), and 2 disagree materially with both sides outside the 10% floor (2.9%). The two material disagreements are K119_C1_VTP Q1 (PAIRED 1.1325, REP2 0.8842) and K119_NT_VTP Q3 (PAIRED 1.1238, REP2 0.8878). Searching 2,477 committed node files at git HEAD, 123 mention the mark, 13 carry both conventions, 2 name the single convention they use, and 3 were flagged: one genuine (1.4932, amended) and two arithmetic coincidences (0.7746, an H3K79me2 value; 1.0333, an H3K4me3 null). A decoy list shifted by +0.0003 hits 2 entries and 4 times against the real list 20 entries and 110 hits, a 10.0x enrichment.

    What this evidence cannot show

    THE 68 COMPARABLE PAIRS ARE NOT INDEPENDENT. They are quartiles, deciles and aggregates drawn from the same handful of contrasts in one deposit, so 51.5% is a rate over STATISTICS and not over experiments; a single badly behaved contrast can contribute many rows. The 10% materiality floor is BORROWED - cycle 102 measured it from input pairs, which bound genotype variation, and it is used here to bound convention variation, a different quantity. The ratio filter is a regex over key names, so a ratio named in some other style is silently excluded from the denominator and an oddly named non-ratio could enter it; the 72 retained pairs were not enumerated by hand. The corpus search remains a 4-dp string match with a measured 2-in-3 false-positive rate among flagged hits, so it screens, it does not adjudicate. AGREEMENT IS NOT ACCURACY: the two conventions share every upstream step except processing state, so agreeing does not make either number right.

    Evidence that argues against this (4)
    • The headline cuts both ways and the framing chosen here is a judgement. Reported as '51.5% of ratios flip sign' the deposit looks unusable; reported as '2 of 68 flip materially' it looks fine. Both are the same data. I chose materiality because a flip between 0.9940 and 1.0035 is not a disagreement about biology - but the 10% floor that makes that judgement is borrowed from the wrong quantity, and a floor of 3% would move several of the 33 trivial flips into the material column and change the verdict.
    • The negative control tests the wrong null for the strongest conclusion. Decoys shifted by +0.0003 measure how often an ARBITRARY number of similar magnitude appears in the corpus. They do not measure how often a number from a DIFFERENT MARK in the same deposit appears - which is exactly what both false positives turned out to be. A decoy set drawn from H3K27me3 and H3K79me2 values would be the harder and more appropriate control, and the 10.0x enrichment would probably fall.
    • This is the third consecutive cycle in which the thing corrected was something I wrote one to five cycles earlier, and the second in which a contaminated denominator inverted a verdict before it shipped. The instrument found one real fault in the corpus and three real faults in itself. An honest reading is that the fault rate in this thread TOOLING now exceeds the fault rate it detects in the CORPUS, and that the marginal return on further auditing of this deposit is low compared with measuring the same-condition noise floor that four separate results are still waiting on.
    • Calling the REP1 episode a near miss overstates my own diligence. The reason I checked what REP1 was is that the number looked too strong, not that I followed a rule. Had PAIRED-vs-REP1 come out at 30% instead of 92.3% it would probably have shipped as a third convention, and the Q125 criterion that already settled it would have gone unread for a second time.
Connected to