A gene-set permutation is anti-conservative for a protein complex, and the fix is to permute the other margin
also called complex correlation · anti-conservative permutation · line-label permutation
Checking whether a group of genes stands out by comparing it against random groups is only fair if the genes are independent, and the proteins in a machine are not. The fix is to shuffle which cell lines you call the test group instead of which genes, and when that fix is applied here the result survives.
The same thing, for a scientist
The standard gene-set permutation asks whether a set's mean rank beats that of random sets of the same size drawn from the scored population. Its null assumes exchangeable, independent members. Physically interacting subunits have correlated gene effects — that is a well-known property of CRISPR screen data and the whole basis of co-essentiality analysis — so a 26-gene complex has far fewer than 26 effective degrees of freedom. The permutation null therefore has too little variance and the test is **anti-conservative for exactly the configuration being tested**. `ana-effect-size-vs-z-reranking` states this objection about its own p = 5e-05 and does not address it; a p-value known to be biased in the direction of the claim it supports is not usable as stated [E1]. **The fix is to permute the other margin.** Hold the gene set completely fixed and randomise which lines constitute the index arm, stratified to reproduce the real 7-myeloid/4-lymphoid composition. Every gene-gene correlation, every line's screen quality and the lineage mix survive untouched; only index membership varies. Run at 5,000 draws this gives p = 0.0026 for the 26-gene FA/BRCA set, 0.0004 for the eight obligate core subunits and 0.0002 for the eight headline genes, against p = 0.092 for the ACTB offset control [E2]. The enrichment is not an artefact of the independence assumption.
The bias, stated precisely
Let the observed statistic be the mean rank of a set of k genes. Under the gene-label null, k genes are drawn independently, so the variance of their mean rank falls as 1/k. If the real k genes are correlated with pairwise correlation ρ, the variance of their mean falls only to roughly (1 + (k-1)ρ)/k times the single-gene variance. For a complex with ρ around 0.3 and k = 26, that is about eight times the variance the null assumes.
The observed mean rank is then compared against a null that is far too narrow, and a modest shift is declared extreme. The direction of the error is always the same: toward significance, for any set whose members are correlated. Protein complexes, paralogue families and pathway members are all in that category, and they are also exactly the sets people test.
This is not a hypothetical concern about this project's result. It is the concern that ana-effect-size-vs-z-reranking raised about its own p = 5e-05, in its own caveat, and then left standing.
What a proper test looks like
Three options, in ascending order of how much they cost and how much they are worth.
1. Rotation or variance-inflation correction. Estimate the between-member correlation from the data and widen the null accordingly, as competitive gene-set tests such as CAMERA do for expression data. Cheap, but it needs an estimate of ρ that is itself noisy at k = 26 with 54 lines.
2. Sample-label permutation — the one that was run. Keep the gene set fixed; permute which samples (here, cell lines) are the index arm. This is the standard remedy and the right one, because it makes the null preserve the entire gene-gene covariance structure by construction: the same 26 genes, with the same correlations, are simply asked about a different set of lines. Stratifying the draw by lineage additionally holds the index arm's composition fixed, so the null is not contaminated by lineage. Cost: one loop 2.
3. Effective-set-size collapse. Reduce the complex to one composite score per module and test the modules. Most conservative, and it throws away the per-gene structure that turned out to matter here — see mech-fa-core-complex-heterogeneity-kmt2a, where the disagreement between subunits is the finding.
The result of option 2, at 5,000 stratified draws of 7 myeloid + 4 lymphoid lines from pools of 32 and 21:
| set | n | observed mean d | p |
|---|---|---|---|
| all FA/BRCA | 26 | -0.157 | 0.0026 |
| obligate FA core | 8 | -0.249 | 0.0004 |
| the eight headline genes | 8 | -0.357 | 0.0002 |
| MEN1 (positive control) | 1 | -0.859 | 0.0002 |
| ACTB (offset control) | 1 | -0.182 | 0.092 |
Note what happened to the p-values: they got larger, from 5e-05 to between 0.0002 and 0.0026. That is the bias being removed, and the result surviving anyway.
What the line permutation still cannot do, and a caution for reuse
It cannot separate a KMT2A effect from anything else those eleven lines share, because it redraws lines rather than genotypes. A null built by shuffling sample labels answers 'are these particular samples unusual for this gene set' and nothing more precise than that.
It also has a subtler limit worth recording for anyone who reuses this machinery: because the draws come from a pool of 53 lineage-labelled leukaemia lines and the real index arm is 11 of them, a proportion of every draw overlaps the real arm. That makes the null slightly conservative — the opposite bias to the one being corrected, and the safe direction, but it means the p-values are not exact.
The transferable rule. Any time this project tests a set of genes that are physically or functionally coupled — a complex, a paralogue family, a pathway — the gene-label permutation alone is not enough, and a sample-label permutation should be run beside it. Several entries in this graph rest on gene-set permutations over complexes and were not tested this way.
| kind | compensation |
|---|---|
| severity | real bias, measured and corrected for this result; likely uncorrected elsewhere in the graph |
| bias direction | always toward significance, for any set whose members are correlated |
| variance inflation sketch | var(mean of k correlated genes) is about (1 + (k-1)rho)/k times single-gene variance; at k=26 and rho=0.3 that is roughly 8x what the independence null assumes |
| mitigation | run a sample-label permutation beside every gene-set permutation over a complex, paralogue family or pathway |
affects
[ "ana-effect-size-vs-z-reranking", "ana-fa-pathway-lineage-and-kmt2a-specificity" ]
remedies
[ "sample-label (line-label) permutation with the gene set held fixed — run here", "variance-inflation / rotation correction as in competitive gene-set tests", "collapse the complex to one composite score per module" ]
- Correcting a statistic in the direction that weakens it and finding it survives is the best possible outcome, and also the easiest to over-read: the corrected p is still a statement about eleven cell lines.
- What is the actual between-subunit correlation of gene effect for the FA core complex across the DepMap leukaemia panel, and how large is the variance inflation in practice?
- Which other gene-set permutation results in this graph rest on correlated sets and have never been sample-permuted?
- Should the project adopt a standing rule that a gene-set p is not reportable for a complex without a sample-label companion?
- E1strong confidence 4 of 5: Replicated in peer-reviewed work by more than one group.dataset
The analysis that produced the FA/BRCA p = 5e-05 states in its own caveat that a permutation over gene sets does not control for correlation in gene effect between physically interacting subunits, which makes the test anti-conservative for any complex.
Effect-size versus z re-ranking of the genome-scale lineage-consistency sweep, with a permutation test of the Fanconi/FA-BRCA pathway · Oncofusion knowledge graph, analyses/effect-size-vs-z-reranking; DepMap Chronos via Open Targets 2026What this evidence cannot show
The analysis names the problem and quantifies nothing about it — no estimate of the between-subunit correlation, no corrected p. The size of the bias in that particular test is still unmeasured.
Evidence that argues against this (1)
- The analysis also ran a second null restricted to the 1,579 genes actually depleted in the comparator arm, which narrows the pool and is a partial mitigation; it does not address correlation, but it is not true that no correction was attempted. Oncofusion knowledge graph, analyses/effect-size-vs-z-reranking 2026
- E2moderate confidence 3 of 5: One strong study, or several smaller consistent ones.dataset
A 5,000-draw stratified line-label permutation — gene set fixed, index arm redrawn as 7 myeloid + 4 lymphoid leukaemia lines from pools of 32 and 21 — gives p = 0.0026 for the 26-gene FA/BRCA set, 0.0004 for the eight obligate core subunits, 0.0002 for the eight headline genes, 0.0002 for MEN1 and 0.092 for the ACTB offset control.
Is the Fanconi signal lineage-wide, fusion-specific, and separable from pan-essentiality? Lineage-split, subtype-control and line-label permutation re-test of the FA/BRCA enrichment · Oncofusion knowledge graph, analyses/fa-pathway-lineage-and-fusion-specificity; DepMap Chronos via Open Targets 2026What this evidence cannot show
Draws are taken from all lineage-labelled leukaemia lines including the real KMT2A-rearranged ones, so a fraction of each draw overlaps the observed arm and the null is slightly conservative. The pools are 32 myeloid and 21 lymphoid lines, so the number of distinct draws is large but the lines are recycled heavily.
Evidence that argues against this (1)
- Permuting sample labels answers only whether these particular lines are unusual for this gene set. It cannot attribute the result to KMT2A rearrangement rather than to anything else the eleven lines share, and it inherits every limitation of the cell-line panel. Ben-David U et al. 2018
tested: line-label permutation preserves subunit correlations and the result holds
tested: line-label permutation preserves subunit correlations and the result holds
the p = 5e-05 is anti-conservative for a complex by the analysis's own caveat; corrected it becomes 0.0002 to 0.0026 and survives