SPADE Agents Learn By Teaching Themselves New Tasks

Every verifiable-reward RL system for language agents hits the same wall: the environment pool stops growing. Hand-curated tasks exhaust; statically synthesized ones freeze the goal distribution at creation time; frozen-verifier pipelines can't adapt once the model surpasses the hardest preset task. SPADE, published on arXiv on August 19 by an 18-author team from UW, CMU, and collaborators, attacks that bottleneck by merging environment designer and reasoning agent into a shared policy.

A single LLM plays two roles. As Environment Designer, it writes complete Python programs—executable MDPs with reset()/step() interfaces, state transitions, reward functions, and verification code—grounded on passages from a pretraining corpus and an accumulated memory of prior generated environments. As Reasoning Agent, it steps through those programs. Both roles update one shared policy via GRPO; there is no separate designer or agent model.

The reward signal prevents collapse through hint-based regret: the gap between the agent's episode return with a privileged hint and without it. High regret marks the learning frontier—tasks the agent solves with help but not alone. Low regret plus high returns mark mastery; low regret plus low returns mark an intractable environment the designer is penalized for producing. This three-way distinction stops SPADE from degenerating into a pure adversary that generates unsolvable tasks.

Ablations show the critical dependencies. Remove corpus grounding and the designer can't anchor new environments to real concepts. Remove environment memory and it regenerates redundant tasks. Disable designer training entirely and the baseline saturates. In games, SPADE improves through all 400 training steps while fixed-environment baselines plateau. Curriculum progression is visible in released logs: at step 4, an environment runs 173 lines with 11 state variables and the 30B agent solves 31 of 32 attempts in 3.7 turns average; at step 304, an environment runs 1,012 lines with 13 state variables and the same agent uses all 12 allowed turns on every one of 32 attempts.

At 30B scale (Qwen3-30B-A3B-Instruct), SPADE delivers a +5.3 average gain over the strongest fixed-environment baseline across eight held-out benchmarks covering math, science, code, and procedural reasoning—none appearing in any generated training environment. Tool-use shows larger gains: +5.7 on BFCL-v4 multi-turn and +13.9 on ACEBench-Agent. One games run produced 4,976 distinct environments; the 30B tool-use run generated 4,023 environments over 105 steps, each simulating a backend with an OpenAI function-calling schema. The margin over fixed-environment baselines grows with model scale—tested at 4B, 8B, and 30B—suggesting the approach strengthens as the underlying model improves.

The training stack is production-weight: distributed RL via Slime, inference through SGLang, policy updates via Megatron-LM, and per-role advantage normalization in GRPO. Evaluation uses RLVE, the Berkeley Function Calling Leaderboard, and PRIME utilities. Code is open source at github.com/spade-rl/spade (Python 3.10–3.12, pip-installable); launchers require Megatron-converted checkpoints, a grounding corpus JSONL, and a W&B key.

For teams building long-horizon agent systems at the ceiling of their current task pool, SPADE's shared-policy dual-role design is the most concrete published path to a self-expanding curriculum. The +13.9 ACEBench-Agent number is the signal to watch—tool-use benchmarks are where most production agent failures appear, and that gap is large enough to justify infrastructure investment in Slime plus Megatron even before the approach matures.