Together AI released autoscaling for its Dedicated Model Inference endpoints, allowing deployments to scale based on inference-specific metrics including in-flight requests, time-to-first-token (TTFT), GPU utilization, and token throughput. Unlike traditional CPU-style autoscaling metrics that can misrepresent LLM serving load, these metrics directly reflect request queue pressure and batching efficiency. Builders can set replica bounds, pick a metric and target, and tune scale-up and scale-down windows to manage the tradeoff between under-provisioning (sharp p95 latency degradation) and over-provisioning (paying for idle GPUs).
The platform applies a proportional control loop: if you target 8 in-flight requests per replica and observe 16, the system scales to 2x replicas (within bounds). Timing windows prevent oscillation—scale-up windows are kept short (costs only a few replica-minutes if false) while scale-down windows stay longer (false scale-downs trigger cold starts). Together AI provides eight autoscaling metrics: concurrency-driven (inflight_requests, a safe default), SLO-driven (TTFT p95, latency percentiles), and efficiency-driven (cost-first). A single PATCH command sets min/max replicas, windows, metric, target, and percentile.
For architects: GPU cost structure now mirrors airline utilization economics. LLM serving degrades nonlinearly when request queues back up; autoscaling on leading indicators (in-flight count) rather than lagging ones (utilization) is critical for production SLOs. Builders should tune autoscaling aggressively on in-flight requests for chat workloads (high batching) and more conservatively for long-context agent traffic. Cold start delays (minutes to pull and warm model weights) mean the autoscaler must scale early on leading signals, not react to observed latency spikes.