Agents score 0.166 on Navers Lab training algorithm benchmark

A new benchmark from Navers Lab and Einsia.AI quantifies a hard limit: LLM agents cannot rewrite training algorithms to produce better models. AI4AI-Bench, published August 20, tests 6 systems across 29 configurations on 10 frozen research repositories and finds a mean score of 0.166 on a normalized scale where 0.1 is the baseline algorithm the repo already ships and 1.0 is the task optimum. The best single system reaches 0.250 — closing less than a fifth of the gap between current and possible.

The benchmark design is adversarial to shortcuts. Each repository covers a distinct family of training algorithms — objectives, update rules, regularization schedules — and is frozen so the agent cannot swap datasets or tune hyperparameters to win. The agent gets 4 hours on one B300 GPU to read code, propose changes, and test ideas against a cheap proxy metric. It produces a source-code patch and nothing else: no cached weights, no retained state. That patch is applied in a fresh container and the full training run executes for up to 12 hours, keeping at most three checkpoints. A fixed evaluator, hidden from the agent during its 4-hour window, scores the result against the original algorithm run under identical conditions.

Most agents never change how the model learns. They move loss weights, adjust batch sizes, or do nothing substantive. Submissions that actually edit the learning algorithm average 0.226 on the benchmark scale; submissions that do not average 0.126. That 0.100-point gap is the entire delta between a meaningful attempt and inertia.

Higher reasoning configurations take the "edits the learning algorithm" fraction from 8% to 64% of submissions and lift the mean score from 0.094 to 0.196. More thinking buys willingness to go into the training code, not ability to improve it once there. No tested system demonstrates reliable algorithmic invention.

The paper places this benchmark in a specific theoretical slot. Algorithmic improvements — Adam, layer normalization, DPO, GRPO — compound across every subsequent training run. Systems-level gains are bounded by hardware rooflines. Data-level gains are bounded by finite human text and follow power-law diminishing returns. Algorithmic design is different: a better objective changes the compute-capability exchange rate for all future runs, including the run that produces the next agent. If recursive self-improvement compounds, this is where it must originate. Current agents cannot drive it.

Existing benchmarks have not measured this. MLE-Bench rewards feature engineering and ensembling while the learning algorithm stays a library call the agent never touches. PostTrainBench's primary levers are data assembly and initialization. RSIBench-Data (Evolvent AI / National University of Singapore) deliberately freezes the post-training stack so only data-centric decisions vary — and frontier agents improve on their first attempt in only 58.33% of settings, with 78.26% of continued searches ending below their own peak score. Frontis-MA1-35B (Horizon Research / Frontis.AI / Tsinghua) pushes MLE engineering results to 71.21% Medal Average on MLE-Bench Lite, but that task domain is prediction competitions — not training algorithm invention.

For teams evaluating agentic automation of their ML stack: agents can optimize around a training algorithm but cannot replace the researcher who designs it. Algorithmic design remains the one RSI level that requires a human.