Three rounds of rank-32 LoRA self-training on Qwen3-8B. Zero detectable gains. Worse: self-training degraded problems the base model already solved, at measurably elevated rates. That is the central result of "Phantom Gains: Auditing Self-Improvement Against a Measured Null," posted to arXiv on August 20, 2026 by Xu, Yan, Chen, and Kechadi. The paper identifies seven measurement failures in standard self-improvement evaluation, each capable alone of inverting a reported finding.
| # | Failure | Mechanism | Effect on Reported Results |
|---|---|---|---|
| 1 | Single greedy decode | Inference-batching artifacts produce per-problem changes on a frozen model | Manufactures apparent gains/losses where no learning occurred |
| 2 | No measured null distribution | Gain/loss ledger differences two noisy estimates with no noise floor reference | Real acquisitions indistinguishable from random fluctuation |
| 3 | Natural threshold repair | Calibrates the threshold against a null that is itself noise-contaminated | Null remains non-zero on frozen comparisons; repair does not correct it |
| 4 | Expansion statistic without null | Statistic applied to frozen model returns 0.280 (implies learning) | Assigns genuine-skill-acquisition score where no training occurred |
| 5 | Aggregate accuracy tracking | A new gain on one problem numerically offsets a regression elsewhere | Corruption of previously solved problems is hidden by the mean |
| 6 | Insufficient baseline replicates | Most multi-arm studies lack enough frozen-model eval passes to build a reliable null | Capability-change statistics built on an untrustworthy noise floor |
| 7 | Mean-accuracy auditing | Problem-level transitions are invisible in aggregate metrics | Regressions stay hidden; only net accuracy change is reported |
The issue: teams shifted from mean accuracy to tracking per-problem gains and losses across training rounds. This granular view reveals more, but introduces a compounding problem. Gain/loss ledgers are built by differencing two noisy estimates—pre- and post-training pass rates per problem. Noise in either estimate propagates directly into the capability signal. Without a separately measured null distribution, you cannot distinguish a real acquisition from random fluctuation.
The authors ran a frozen Qwen3-8B control through the identical evaluation pipeline as trained arms. Any measured change is, by definition, an artifact. A ledger built on single greedy decode manufactures capability changes on the untrained model—an artifact of inference batching, not learning. The expansion statistic, designed to separate genuine skill acquisition from sharpening of partial knowledge, assigns the frozen model a rate of 0.280. That score implies learning where none occurred.
The standard "natural threshold repair" does not fix this. The null distribution remains non-zero when estimated across frozen comparisons in a multi-arm study. This repair calibrates against noise-contaminated signal.
The paper proposes a per-problem exact test against a pooled baseline, run under false-discovery-rate control. Applied to held-out replicates, it detects nothing—the right answer when the model has not changed. It remains stable across changes to multiple-testing correction, error rate, and pool size. A statistic that shifts with FDR thresholds is not measuring capability.
With a credible null in place, the distillation-versus-self-training comparison becomes clear. External distillation moves problems the base model rarely reaches; three forms of self-training do not. A regression testing whether this asymmetry reflects something structural about self-training rejects the null at p < 10⁻⁸, but the direction contradicts self-training advocates. The asymmetry results from distillation's larger overall gain, not evidence of different problem-space targeting. On problems the base model never reaches, the evidence is inconclusive.
| Training Method | Moves Rarely-Reached Problems | Overall Gain | Degrades Solved Problems | Asymmetry vs. Self-Training |
|---|---|---|---|---|
| External distillation | Yes | Larger | Not detected above noise | p < 10⁻⁸ (regression) |
| Self-training — form 1 | No | Not detected | Yes — above measured noise | — |
| Self-training — form 2 | No | Not detected | Yes — above measured noise | — |
| Self-training — form 3 | No | Not detected | Yes — above measured noise | — |
| Frozen base (control) | N/A | Zero by design | Artifact only | Expansion statistic = 0.280 |
The corruption finding demands immediate action for teams running iterative self-training. Self-training degrades problems the base model solves at baseline, at rates above measured noise. Teams tracking aggregate accuracy miss this—a new gain on one problem numerically offsets a regression elsewhere. Transition-level auditing exposes the regression; mean-accuracy auditing hides it.
The practical constraint: building a valid null requires baseline replicates—multiple evaluation passes on the unmodified model, run under identical conditions as trained arms. Most multi-arm studies have enough data to construct this null without new experiments. Few run enough replicates to do it reliably.
The architect's takeaway: before shipping any self-training pipeline, run your frozen base model through the full eval stack at least as many times as each trained arm, and build capability-change statistics against that measured null—not against assumed zero.