LangChain Labs and Fireworks partnered to fine-tune a Qwen-3.5-35B model on detecting "Perceived Error" in production agent traces—identifying when users think the assistant made a mistake or generated output needing correction. The fine-tuned model matches or exceeds frontier model (Claude Opus, GPT-5.5) accuracy while running 100x cheaper. On the training dataset (chat-langchain), the model reached 96.1% accuracy, and critically, it transferred to a different domain (Fleet, a no-code agent builder) with 90.8% accuracy, demonstrating generalization.
LangSmith processes billions of tokens daily across production traces. The goal was to cost-effectively mine signals from every trace while maintaining frontier performance. The team used two internal datasets (chat-langchain: 885 examples; Fleet: 911 examples) with multi-turn traces, combined model-assisted and human review labeling, and trained with LoRA on Fireworks' managed SFT infrastructure. The fine-tuned model also transfers across domains better than the baseline Qwen, suggesting "perceived error" is a genuinely general-purpose evaluator signal.
For practitioners, this demonstrates a repeating pattern: fine-tuning small open models on domain-specific data can match or exceed frontier model performance at vastly lower cost. For agent-heavy products, detecting perceived error in real-time traces enables continuous improvement loops without frontier model API bills.