The week agents became a measurable engineering discipline—and the silicon underneath them stopped being a monopoly.
Eighty-eight percent.
That's the proportion of AI agent projects that never reach production. Not projects that disappoint. Projects that are abandoned before a single real user interacts with them.
This is Edition 22 of ai|expert. The week agents became a measurable engineering discipline—and the silicon underneath them stopped being a monopoly.
We start with the problem that dominated the week in research: the gap between benchmark and production in agentic systems isn't editorial—it's structural. And two papers and one framework release tried to close that gap at the same time, from completely different angles.
LangChain opened up on Wednesday. They published Harbor as open-source—a benchmark runner that trades text comparison for real-state inspection. The agent manipulates a Docker environment defined by Dockerfile or Compose. A test.sh script checks artifacts and API side effects. The question stops being "does the answer look right?" and becomes "did the filesystem change the way it should?" That fundamentally changes what counts as passing. [ref: langchains-agent-benchmarking-]
The numbers are dense. Harbor-Index has eighty-two tasks distilled from more than six thousand candidates across fifty-four benchmarks—covering software engineering, research, data analysis, and long-horizon tool use. The τ³-bench contributes thirty multi-turn tasks with simulated users, where scoring validates real results rather than surface dialogue coherence. ContextBench adds thirty retrieval tasks, each with complete corpus sent inside the sandbox, forcing the agent to locate and combine information rather than rely on parametric knowledge or an external vector database. [ref: langchains-agent-benchmarking-]
The two-layer architecture is what makes the operation viable. The "lite" subset runs eight times faster and six times cheaper than the full suite. Daily commits run through lite. Release decisions run through complete. That's not theoretical—it was that separation that caught the regression when LangChain removed to-do-list middleware and compressed the system prompt in Deep Agents 0.7. The benchmark detected the problem before release. That's the use case that justifies all the infrastructure overhead.
The overhead is real and needs to be budgeted. LangChain didn't publish latency per task, cost in dollars per session, or GPU consumption for the full suite. Each new task requires Dockerfile, test script, and deterministic environment setup. The integration cost is substantially higher than a prompt eval. And what was published is LangChain's internal release engineering—not evidence of adoption in production by external customers. Architects should seek cost curves and regression rates at customer scale before importing the stack.
I agree with the caveat, but I don't diminish the standard. The separation between fast suite for commits and heavy suite for release gates is what's missing in most teams operating agents today. They either don't evaluate, or they evaluate everything with the same weight. That's the operational deficit.
And KDD '26 arrived with the diagnosis from the other side of the same problem. A tutorial from Grace Hui Yang and collaborators from Bloomberg, Bayer, Salesforce AI Research, and Microsoft Research—four organizations with direct production experience—mapped in detail the failure modes that static leaderboards don't capture. [ref: agents-in-the-wild-bridging-th]
The numbers that anchor the argument: eighty-eight percent of agent projects never reach production, according to CIO Research 2025. Gartner projects that more than forty percent of those remaining will be canceled by 2027. The failure modes named by the tutorial—silent corruption of tool calls, cascading errors in agent graphs, retry loops that consume budget while appearing successful—are exactly what final output tests don't detect. A corrupted argument in step two can silently poison every step that follows. [ref: agents-in-the-wild-bridging-th]
The operational detail the tutorial adds is cross-agent observability. The OpenTelemetry GenAI semconv reached stable status at semconv 1.29+. But tracking state across agents in a multi-agent graph remains an unsolved engineering problem within industrial latency and compute budgets. And multi-agent coordination introduces failure modes that monolithic systems avoid entirely: deadlocks between collaborating agents, synchronization overhead, and absence of secure inter-agent communication protocols.
There's an operational heuristic here that Zylos.ai documented with concrete numbers. A code agent can consume two dollars in API calls on a good day and forty dollars on a bad day trapped in a retry loop. Any session costing more than five times the median for that feature type becomes a correctness incident—not a billing issue. Above fifty times the median is uncontrolled loop. Cost as a correctness signal is the second lens missing from most teams.
The combined conclusion from both papers: state-based artifact evaluation, session cost as a correctness signal, and cross-agent tracing via stable semconv. Three practical contributions from the week for anyone operating agents in production today.
The third piece of research this week goes in the opposite direction. It's not about measuring the agent's external behavior—it's about trusting what the model says about itself. About interpretability as a foundation for safety.
And the paper's answer is direct: don't trust it yet. A paper from Hiskias Dingeto audited the Natural Language Autoencoder from Anthropic on Qwen-2.5-7B—the layer twenty checkpoint available on HuggingFace, fine-tuned from Qwen-2.5-7B-Instruct. The result: roughly two percent of specific claims in the explanations are genuinely grounded in activation. The rest—approximately ninety-eight percent—survive by tracking the gist of the input without carrying verifiable facts. The explanation looks faithful. It isn't. [ref: activation-explanations-fail-f]
And the failure mode isn't a rare edge case. In synthetic tests with exact ground truth, the standard training recipe produced what the paper calls "private codes"—false language that the reconstructor learned to depend on—in five of five runs. False by construction, with high reconstruction scores.
The problem is structural in the objective. The reconstruction objective rewards sufficiency: the explanation needs to contain enough information to regenerate the activation. But the objective doesn't penalize false additions. That holds regardless of scale or model. The insensitivity follows from the objective itself.
The paper proposes RECAP—Readable Encodings via Co-trained Auxiliary Predictors. Linear heads trained alongside the model keep designated content decodable. A probe trained with RECAP reaches AUC of zero point ninety-six at distinguishing true claims from false—against zero point eighty-two from baseline. Under adversarial editing designed to suppress eighty-seven percent of the lying penalty while preserving reconstruction score, the RECAP probe still detects deception with AUC zero point ninety-five.
The control probe, under that same adversarial condition, collapses to zero point fifty-one—indistinguishable from chance. The cost of RECAP: an additional zero point zero zero one nats of training. Technically cheap. But with one critical operational constraint: RECAP requires co-training during initial training. It can't be added to already-published weights—including the public Qwen-2.5-7B NLA checkpoint in use now. [ref: activation-explanations-fail-f]
Any team using activation explanations today to audit steering vectors or concept probes is limited to post-hoc audit protocols. Which are manual diagnostic steps—not automated filters.
And there's an implication that runs across all three research blocks this week. We're building agent evaluations on interpretability stacks that, by this paper, may give us plausible and false claims at the same time. Reconstruction loss alone isn't a sufficient gate for fidelity. Any security layer built on activation verbalizers needs claim-by-claim verification—not just vector similarity.
Measuring is necessary. But the measuring instrument can also lie.
While the software side renegotiated its contracts with reality, the hardware side renegotiated its contract with NVIDIA. And that renegotiation has numbers.
DAC 2026 was this week's measurement point for the compute side. Submissions grew more than twenty-six percent year-over-year in both tracks—Research and Engineering. Forty percent of the technical program focused on AI and chip design. But the number that matters is projection: custom ASIC shipments will grow forty-four point six percent CAGR in 2026—more than double the sixteen point one percent for merchant GPUs. NVIDIA's share of inference accelerators specific to domain—where two-thirds of all AI compute is concentrated—is expected to fall from more than ninety percent today to twenty to thirty percent by 2028. [ref: diy-ai-chips-enterprises-desig]
The DAC Engineering Track—which only accepts production results, not research prototypes—is where the substance lies. Samsung presented a pre-silicon emulation environment using Deep Q-Networks with dueling enhancements and double-DQN and prioritized experience replay, to optimize SoC QoS parameters in throughput, latency, and power. Eliminating manual tuning of arbitration and bandwidth allocation. That's RL applied to hardware design, not text generation.
IBM presented two papers that deserve separate attention. The first: agentic verification flows using MCP servers to ingest design specifications, HDL, waveforms, and coverage bases—with estimated reduction of fifteen to forty percent in manual effort for hardware verification triage on IBM Z. The second paper is the most operationally dramatic: an MCP framework that generates EDA utilities from user specifications using reusable MCP blocks, reducing structural verification tool development from four person-weeks estimated to less than thirty minutes. [ref: diy-ai-chips-enterprises-desig]
Four person-weeks to thirty minutes isn't incremental acceleration. It's a different category of impact—and it's in a domain where design cycles cost millions and verification errors cost entire tape-outs.
And that has external validation beyond DAC. Midjourney migrated inference from NVIDIA GPUs to Google TPUs, reducing monthly compute costs from two point one million dollars to seven hundred thousand dollars—a sixty-five percent reduction. Morgan Stanley estimates that Amazon will ship one point five million Trainium chips in 2026. XPUs—custom accelerators—should lead data center spending growth in 2026 at twenty-two percent, surpassing GPUs at nineteen percent.
But DAC also exposed the real friction that this transition carries. TSMC is operating the three-nanometer node at one hundred percent capacity, with demand three times current supply. Physical substrates for custom silicon are already in structural shortage. And the cost advantage of forty to sixty-five percent from ASICs—attractive at hyperscaler scale running billions of queries per day—looks very different for a company running tens of thousands of queries per week. Break-even requires substantial inference volume.
And there's the dynamic that one DAC observer named collective myopia. Samsung, NVIDIA, Meta, and OpenAI are each building similar AI layers on top of licensed EDA engines—effective and nearsighted at once. Each giant rebuilds the same plumbing in private. Lessons don't propagate between them.
The week's most unlikely partner was Cerebras and AMD.
Disaggregated inference: AMD Helios, with EPYC and Instinct MI400, processes high-throughput prompts and prefill. Cerebras's Wafer Scale Engine handles token generation, where memory bandwidth is the dominant bottleneck. The two companies' claim: five times more tokens per second per watt than competing approaches. Available on Cerebras Cloud in the second half of 2026. [ref: cerebras-amd-partnership-signa]
With the obligatory caveat: this benchmark comes from AMD Performance Labs and Cerebras, internal modeling from July 2026. No independent validation, no MLPerf, no live production traces. Absolute latency, p99 tail behavior, cost per million tokens—not published. And the interconnect between the two silicon domains—protocol, bandwidth, and physical topology for KV-cache state transfer—hasn't been disclosed yet. That's the unknown that determines whether the architecture works in practice.
The strategic context is real regardless of benchmark. NVIDIA acquired Groq assets for twenty billion dollars in December for low-latency technology. Cerebras has a separate deal with OpenAI worth more than ten billion dollars to deliver seven hundred fifty megawatts of compute by 2028. The AMD partnership is Cerebras's third bet in an inference market that's clearly restructuring outside the NVIDIA monopoly. [ref: cerebras-amd-partnership-signa]
Anyone betting on Cerebras-AMD architecture today is betting on an interconnect that nobody has described yet. The pattern—prefill on dense GPU, decode on Wafer Scale Engine—has architectural logic. The real cost per query doesn't.
Silicon is fragmenting. Regulation is trying to control where the fragments can flow.
The executive order of July 20, effective January 1, 2027, eliminates supply-chain exemptions for semiconductors from adversarial nations in defense contracts. The twenty-five percent tariff under Section 232 on imported advanced AI chips is the immediate cost impact. And potential two-year export restrictions for NVIDIA's Blackwell silicon to adversarial nations compound the pressure on any AI team focused on defense. [ref: us-defense-supply-chain-order-]
The operational requirement is specific and burdensome. Prime contractors and subcontractors at all levels must map critical supply chains from raw materials through finished products—an "indentured Bill of Materials" linking software and firmware dependencies to physical components, manufacturers, and countries of origin. Significant risks reported in fifteen days. Confidential corrective action plans in forty-five days.
The Department of Defense has one hundred eighty days to develop mapping policy and an additional ninety days to issue regulations. But contractors must already track administrative access, data hosting, development locations, and beneficial ownership now—not in 2027. The preparation window is smaller than it looks.
The 2026 NDAA bans "Covered AI" from defense and intelligence contracts—primarily DeepSeek and parent High Flyer, plus any model developed by entities with twenty percent or more indirect ownership from those sources or from China, Russia, Iran, or North Korea. Subcontractors must certify compliance as a contract condition, with exposure to False Claims Act penalties for inaccurate submissions. [ref: us-defense-supply-chain-order-]
And here's the detail most teams will discover too late: the definition of critical supply chain includes cloud providers, managed service providers, and software developers. Multi-cloud abstraction layers are direct compliance surface. Every GPU cluster and model weight file must be treated as a Bill of Materials item from the first purchase order.
Proving provenance across four subcontractor levels under False Claims Act scrutiny isn't a problem you solve after deploy. The Department of Defense was also instructed to use AI to analyze contractor submissions and identify single points of failure—which makes the supply-chain maps themselves a high-value attack surface.
Policy is being used to solve a hardware problem. And hardware is being redesigned to escape policy. Both moves are happening at the same time.
To close, three cases where the week moved from paper into real production. We start with Anthropic auditing its own operations.
Claude Tag—the code agent integrated into Anthropic's Slack—closes sixty-five percent of product engineering pull requests from the Claude Code team. In parallel, the Claude Code system prompt was reduced by eighty percent. These two numbers together are the clearest signal yet that frontier models require less scaffolding—not more. [ref: claude-tag-closes-65-of-anthro]
The mechanism was detailed in a transcribed fireside chat by Simon Willison, with engineers Cat Wu and Thariq Shihipar. Claude Tag runs in "auto" mode—no human approval for each tool call—in Anthropic's public Slack channels, generating diffs openly and automating code review for outer product layers. Human review reserved for critical changes. What wasn't published: the exact criteria that distinguish routine changes from critical ones.
The most important paradigm inversion isn't in PR count—it's in prompts. On models like Fable 5 and Opus 4.8, adding examples to the system prompt now degrades quality. "Don't do X, don't do Y" lists also. The recipe that stabilized old models actively hurts new ones. Any system in production stabilized by few-shot prompting and restriction lists will face a breaking re-architecture when migrating to this generation of models.
The timeline from idea to production compressed from six to twelve months to around one week. What they call internally "ant fooding"—testing features on employees before any external rollout—is the internal retention gate for launch. Features that don't gain internal traction don't reach broad rollout.
Thariq Shihipar inverted the Mythical Man-Month: rewrites are now the right approach. A disciplined test suite makes rewrite safer than preserving legacy code when the codebase is the only specification document. And it was the benchmark—not the engineer's intuition—that justified deleting legacy scaffolding in 0.7. [ref: claude-tag-closes-65-of-anthro]
With the data provenance caveat: sixty-five percent comes from the team that built the tool. Not from other Anthropic teams. Not from external companies. Economic inference, GPU burn rate, rate of PRs requiring human amendment after merge—not published.
The second case is the FDA.
In two months of launch, ELSA—the FDA's internal generative AI platform—went from less than one percent to eighty-five percent daily adoption across the agency's sixteen thousand employees. The underlying data layer, Halo, processes approximately one petabyte of documents and hundreds of gigabytes per day from fifty to sixty sources across all eight FDA centers on a single Databricks stack with Unity Catalog applying table-level access controls. [ref: fdas-ai-platform-achieves-85-d]
What made this adoption possible was what happened before. The CDER—the FDA's drug center—spent five years validating the Databricks foundation before the other seven centers followed. Two-month adoption came after the foundation was proven. Not the reverse. That's the sequencing lesson most rapid adoption cases omit.
The most concrete operational result: drug application reviewers now consult three to four million pages of initial material records in roughly three minutes. The same task took days before consolidation. Data sharing between centers that took four to five days was replaced by real-time streaming. And employees build hundreds of new agents per week via MCP servers on top of Unity Catalog—without writing queries. [ref: fdas-ai-platform-achieves-85-d]
What the FDA didn't disclose is critical for any architect wanting to use this case as a reference: inference latency, cost per token, GPU hours, the specific base models powering ELSA, and—most importantly—the evaluation harness, hallucination detection, or red-teaming protocols for a platform influencing pre-market drug regulatory decisions.
In an environment where corrupted data extraction can skew a drug review, the absence of published human-in-the-loop protocols isn't a communication gap—it's real risk. The FDA migrated from isolated chatbots to a unified agent factory. The evaluation stack that justifies that trust stays undisclosed. Before citing the FDA as an adoption reference, an architect in regulated industry should ask exactly that question.
And the third case closes the loop between hardware and real application.
Bristol Myers Squibb is deploying eight Vera Rubin NVL72 DGX systems in a second DGX SuperPOD—the first life sciences company to acquire Vera Rubin silicon at scale. The efficiency claim is ten times more performance per megawatt over the predecessor cluster. The earlier cluster was already saturated—in production with large-scale predictions for large molecules and building proprietary foundation models. [ref: bristol-myers-squibb-builds-ai]
The clinical impact is the number that anchors this. Robert Plenge at BMS indicates that AI tools have already reduced the time needed to produce drugs for clinical trials by twenty to thirty percent—with expectations to reach fifty percent. Screening throughput grew from roughly ten to dozens of compounds. And a candidate for sickle cell disease is in early trials produced using this methodology.
The transferable pattern is called "Predict First". Cheap inference filters candidates before expensive wet-lab synthesis. Researchers submit jobs in natural language—target identification, molecule ranking by multi-parameter optimization—before any physical validation. The pattern is applicable in any regulated domain where physical validation is the dominant cost. It's not pharma-specific. [ref: bristol-myers-squibb-builds-ai]
What's missing from the published model: BMS didn't disclose queue depth, preemption policies, or latency distributions for the natural language job dispatcher. Integration of a single data plane—breaking constraints from prior acquisition sites—is a migration that typically costs more than the hardware itself. And the "Predict First" gating method has no public evaluation harness, leaving the false-negative rate for viable compounds as an unknown.
BMS operates in a compliance domain that makes those details unlikely to be published. But the "Predict First" principle—cheap compute as a gate before expensive physical validation—is the pattern that exits this case and enters your architecture.
Five layers, one motion. Measure the agent by the state it leaves in the world, not by the text it produces. Question what the model says about itself before building security on top of it. Choose the right silicon without locking into a single vendor. Operate inside regulation that treats GPU clusters as Bill of Materials items with traceable provenance. And ship results in production domains where the cost of error is clinical. Edition returns Friday. Good work.