IBM Research published a study this week on ALTK-Evolve, a framework that distills an agent's own past trajectories into reusable behavioral guidelines and injects them at inference time—no weight updates, no human annotation. Optimal agentic memory depends on model size and capacity. Miscalibration wastes tokens or kills performance.
The team tested three memory configurations on AppWorld, a 585-task benchmark spanning 168 standard and 417 challenge tasks across nine simulated apps. The baseline used no memory. "Full guideline set" injected every mined guideline at each ReAct step. "Curated retrieval" delivered a fixed core plus per-task selections. All guidelines came from the training-split pool; nothing from test data was used.
Three patterns emerged. Strong models with spare capacity gained the most from full guidelines. DeepSeek-V3.2 (671B MoE) gained 9.5 percentage points in task completion, with the bandwidth to absorb edge-case lessons.
Smaller and mid-tier models showed the opposite pattern. Large guideline sets degraded performance or broke even while consuming tokens. Curated retrieval proved both most accurate and cheapest. gpt-oss-120b (117B MoE) gained 16.1 percentage points using curated retrieval at +5% additional tokens in context. The same model given full guidelines gained less at roughly 50% more tokens.
Already-saturated models showed no measurable gain. GLM-5 (745B MoE) produced flat results for either configuration. IBM researchers flagged this as the "saturated pattern" and left interpretation open: the model may have hit its ceiling on these tasks, guidelines may not address its failure modes, or it may not apply guidance effectively.
| Model | Size | Capacity pattern | Best memory config | Peak task-completion gain | Token overhead (best config) | Full-guideline token overhead |
|---|---|---|---|---|---|---|
| DeepSeek-V3.2 | 671B MoE | Capacity-rich | Full guidelines | +9.5 pp | — | — |
| gpt-oss-120b | 117B MoE | Mid-tier | Curated retrieval | +16.1 pp | +5% | ~+50% |
| GLM-5 | 745B MoE | Saturated | Neither | ~0 pp | — | — |
This work differs from prior memory benchmarks by centering production cost. Curated retrieval is simultaneously most accurate and cheapest for mid-tier models. Prompt caching makes full guidelines affordable for frontier models in batch workloads. The entire learning loop—trajectory collection, guideline extraction, consolidation, injection—runs without weight updates, which is why ALTK-Evolve ran identically across all eight models tested.
Parameter count alone does not predict which pattern a model falls into. Benchmark headroom, context window size, architecture, guideline quality, and task distribution all interact. IBM Research says separating those factors is ongoing work. For production deployments, run a calibration pass on representative tasks before choosing a memory configuration for a specific model.
AppWorld tasks are bounded, well-structured, and binary-scored—ideal for measurement but structurally different from open-ended production workloads that run over longer horizons with fuzzier success criteria. Transferability is not established by this study.
For teams building inference infrastructure: curated retrieval with a tight core guideline set is the sweet spot for any model outside the frontier tier. Measure token overhead before and after adding memory. If overhead exceeds 10–15% without corresponding task completion gains on held-out evals, the model is likely saturated or being flooded. This paper provides the diagnostic categories; calibration still requires your own eval data.