A Jaccard index is not a result until its null knows that active promoters cluster
If you compare two lists of busy genes, they will overlap a lot just because the same genes are busy in everybody. The 'expected by chance' number has to allow for that, or every comparison looks like a discovery.
The same thing, for a scientist
The textbook chance expectation for the overlap of two sets of sizes a and b drawn from a universe of N is `ab/N`, and the corresponding Jaccard is `(ab/N)/(a+b-ab/N)`. That formula assumes the N elements are exchangeable. In a promoter-level chromatin assay they are not: promoter activity, accessibility and read recovery are strongly shared across samples, so any two samples' call sets are both drawn preferentially from the same active minority. Measured on GSE252378, correcting for this raises the chance floor by 1.73x in an unfiltered 19,404-promoter universe. A coverage filter does most of the same work - in a 27-way `>=20 fragments` intersection the correction is only 1.02x - so a beat with neither a filter nor a matched null is exposed twice, and a beat that quotes excess-over-chance without saying which floor it used cannot be re-derived.
The three floors, and when they differ
- Independence
ab/N. Valid only if every element of the universe is equally likely to be in either set. Almost never true in genomics. - Covariate-matched permutation. Bin the universe on whatever drives set membership other than the biology under test - here promoter coverage, which is simultaneously an activity proxy and the denominator that sets statistical power - and permute labels within bins. Preserves set sizes and the covariate distribution; breaks only the identity correspondence.
- The empirical distribution over comparable pairs. What do OTHER pairs of the same kind give? This is the only floor that answers 'is this pair special', and it is the one both earlier beats reached for and mis-specified in opposite ways.
All three are needed. (1) tells you the arithmetic floor, (2) tells you the structural floor, (3) tells you whether your pair is unusual. A result that clears (1) and (2) can still be the base rate under (3), which is exactly what happened here.
The corollaries that bit in practice
- A chance floor is a property of a PAIR, not of a cohort. A cohort median of
ab/Nover 351 pairs whose set sizes range from 0 to 2,282 is the floor for none of them. Dividing one pair's observation by it overstated an excess 4-fold. - Equal set sizes hide the problem. A quantile calling rule fixes both sets at 5% of the universe, so every pair has the same independence floor and the floor never has to be thought about. That is convenience, not correctness: the structural correction is largest exactly there (1.73x), because the unfiltered universe still contains the inactive majority.
- Coverage filtering and null matching are partial substitutes, so a beat that adds a filter may correctly conclude its independence null was adequate, and be wrong the moment the filter is relaxed.
How to satisfy this in one line each
Report, for every pairwise overlap: the universe size, the comparable fraction, both set sizes, the independence floor for THAT pair, the matched-permutation floor for THAT pair with its seed and draw count, and the distribution of the same statistic over comparable pairs. Six numbers. Every one of them was missing from at least one of the two entries this beat adjudicated.
- Excess-over-chance in target-set overlap is how this corpus argues that a fusion has a conserved, druggable target programme. If those excesses are systematically 2-4x overstated, the case for a fusion-directed rather than patient-directed target list is weaker than it reads.
- What is the right stratification for a comparison of ENHANCER sets rather than promoter sets, where coverage is far less shared across samples? The correction may be larger there, not smaller.
- Is there a corpus-wide audit possible of every reported overlap statistic and whether it names a pair-specific floor?
- E1established confidence 5 of 5: Multiple independent lines of evidence; treated as settled in the field.paper
Permutation frameworks for genomic region association exist because uniform placement is an invalid null in a structured genome, and they operate by restricting the randomisation space.
regioneR: an R/Bioconductor package for the association analysis of genomic regions based on permutation tests · Gel B, Diez-Villanueva A, Serra E, Buschbeck M, Peinado MA, Malinverni R · Bioinformatics 32:289-291 2015What this evidence cannot show
regioneR randomises positions on the genome rather than labels over a fixed feature set; the label-permutation form used here is the discrete analogue.
Evidence that argues against this (1)
- Restricting the randomisation space is itself a modelling choice, and a restriction chosen after seeing the answer can be tuned; pre-registration of the stratification variable is the only real protection. Heger A et al. 2013
- E2established confidence 5 of 5: Multiple independent lines of evidence; treated as settled in the field.paper
A simulation framework for interval association must allow each set's density to depend on external variables, because the alternative - independent uniform placement - is known to be wrong.
GAT: a simulation framework for testing the association of genomic intervals · Heger A, Webber C, Goodson M, Ponting CP, Lunter G · Bioinformatics 29:2046-2048 2013What this evidence cannot show
GAT's worked covariates are isochore structure and chromosome identity, not assay coverage; the extension to coverage is by argument.
Evidence that argues against this (1)
- If coverage is partly a consequence of the signal being measured, stratifying on it removes real signal and makes the corrected excess conservative rather than merely correct. Gel B et al. 2015
- E3strong confidence 4 of 5: Replicated in peer-reviewed work by more than one group.dataset
Measured on the KMT2A oncoprotein N/C promoter calls of GSE252378, the structure-respecting floor exceeds the independence floor by 1.73x in an unfiltered promoter universe and by 1.02x in a 27-way coverage intersection.
GSE252378: AutoCUT&RUN of KMT2A-rearranged primary leukemia samples (KMT2A N- and C-terminal antibodies, ENL, DOT1L, H3K4me3, IgG) · Janssens DH, et al. · NCBI Gene Expression Omnibus 2024What this evidence cannot show
One deposit, one assay type, one feature set. The size of the correction in other data is unknown and the 1.73x should not be carried elsewhere as a constant.
Evidence that argues against this (1)
- A coverage-filtered universe makes the correction almost vanish, which could be read as an argument that filtering is sufficient and matched nulls are unnecessary - but the filter also removed 76% of promoters and changed which biology was visible. Janssens DH et al. 2024
the worked implementation of the second floor
the worked case of a cohort median used as a pair's floor
the promoter-level form of the same failure
here the quantity that varies is promoter coverage
the pair-specificity half of the null requirement
extends q146's clustering caveat by giving the direction of the error for an exclusion pattern
- ← related toA partner-exclusive promoter set needs a label-permutation null, not an independence one
the jaccard-null concept is the same family of failure
the same clustering fact q146 recorded, measured by decile