Serving at Scale

Chapter 27 made a single model run fast on a single machine. But a real product serves thousands of requests per second, from users around the world, around the clock, reliably, within a budget. That is no longer a model problem — it is a SYSTEMS problem. This final chapter of Part VI is about the engineering that turns an optimized model into a dependable, scalable SERVICE.

A demo serving one request at a time hides nearly everything that matters in production. At scale, you must handle many concurrent users, spread load across many expensive GPUs, survive hardware failures without downtime, meet latency promises even under load spikes, roll out new models without breaking anything, and keep the enormous compute bill under control. None of this is about the model itself; all of it determines whether the product works.

Everything in production serving balances three forces, building on Chapter 27's metrics: LATENCY (responses must be fast enough for users), THROUGHPUT (the system must handle the total load), and COST (it must fit a budget). These pull against each other — lower latency often means more cost, higher throughput can hurt latency — and the art of serving at scale is balancing all three under real-world conditions.

The API is how everyone — your own apps, external developers, other services — talks to the model. A well-designed API is the foundation of a usable service; a poorly-designed one creates friction forever. Let us cover the essentials of designing an LLM API.

For interactive use, STREAMING is essential. Instead of waiting for the entire response (which could be many seconds for a long answer), the API streams tokens to the user AS THEY ARE GENERATED. The user sees text appearing immediately, dramatically improving perceived responsiveness — the response starts at the TTFT (Chapter 27), not after the full generation. Almost every interactive LLM product streams.

One GPU cannot serve millions of users. Production systems run many REPLICAS — copies of the model on many GPUs — with a LOAD BALANCER spreading requests across them and an AUTOSCALER adding or removing replicas as demand changes. This is the backbone of scalable serving.

A load balancer sits in front of the replicas and routes each incoming request to one of them. The goal is to keep all replicas evenly busy — no replica overloaded while others sit idle. Naive strategies like round-robin (rotate through replicas) work poorly for LLMs because requests vary wildly in cost (a 10-token reply vs a 2,000-token reply). LLM-aware load balancing routes based on actual LOAD — current queue depth, KV-cache usage, or number of active requests per replica.

Demand is not constant — it spikes during the day, drops at night, surges with viral events. AUTOSCALING adds replicas when demand rises (to maintain latency) and removes them when demand falls (to save cost). The challenge for LLMs: spinning up a new replica is SLOW — loading a large model onto a GPU can take minutes — so autoscaling must be PREDICTIVE (scale up before the spike, based on trends) rather than purely reactive, or users hit slow responses while new capacity warms up.

A production service makes PROMISES about its performance — a Service Level Agreement (SLA) — and must measure whether it keeps them. Defining and meeting SLAs is central to running a dependable service, and it requires the right metrics.

An SLA defines the level of service users can expect, typically covering: AVAILABILITY (uptime, e.g. 99.9%), LATENCY (e.g. 'p95 time-to-first-token under 500ms'), and sometimes throughput or error rates. The service is engineered, monitored, and over-provisioned to meet these targets. Falling short has consequences — unhappy users, broken contracts, financial penalties.

A crucial lesson: measure latency with PERCENTILES, not averages. The average hides the bad cases — if 99% of requests are fast but 1% take 30 seconds, the average looks fine while some users have a terrible experience. The p95 (95th percentile) and p99 (99th percentile) latencies capture the TAIL — how bad the slow requests are. SLAs are written in percentiles ('p99 latency < 2s') because they reflect the worst experiences real users actually have.

Beyond simple load balancing, a sophisticated serving system ROUTES requests intelligently and SCHEDULES them to balance competing goals. With diverse requests (short and long, cheap and expensive, free and paid) and a heterogeneous fleet (different GPUs, different models), good routing and scheduling are key to both performance and cost.

Routing decides WHICH replica or model handles a request. Smart routing considers: which replica is least loaded, which already has the relevant prefix cached (route a follow-up to the same replica — 'cache-aware routing'), which model size suits the request (route easy queries to a small fast model, hard ones to a large model — 'model routing'), and the request's priority tier. Good routing can dramatically improve both latency and cost.

Within a replica, the SCHEDULER decides the order requests are served and how they share the GPU (building on continuous batching from Chapter 27). It balances fairness (no request starves), priority (paid or interactive requests first), and efficiency (keep the batch full). A key concern is preventing a few huge requests from monopolizing the GPU and starving everyone else — so schedulers may preempt, chunk, or cap long generations to protect the tail latency of short ones.

Models are updated regularly — improved versions, fine-tunes, fixes. But a new model can behave differently, and a bad update can degrade quality or break downstream systems for millions of users at once. MODEL VERSIONING and SAFE ROLLOUT practices let you change models without risking the whole product.

Users and downstream systems may depend on a model's specific behaviour. Silently swapping in a new model can break prompts that were tuned for the old one, change output formats that code parses, or shift quality in unexpected ways. VERSIONING means each model version is named and addressable, old versions remain available for a transition period, and clients can pin to a specific version. This lets the system evolve without yanking the ground out from under anyone.

How do you know a new model or prompt is actually BETTER, not just different? Offline benchmarks (Chapter 21) help, but the real test is how it performs with REAL users on REAL traffic. A/B TESTING and production evaluation answer this rigorously, building on the evaluation discipline from Part IV.

In an A/B test, you serve the OLD model (A) to one randomly-chosen group of users and the NEW model (B) to another, then compare outcomes — user satisfaction, task completion, engagement, thumbs-up rates, retention. Because users are randomly assigned, differences in outcomes can be attributed to the model change. This is the gold standard for deciding whether a change genuinely improves the product, not just the benchmark.

Production evaluation goes beyond benchmark scores to real outcomes: explicit feedback (thumbs up/down, ratings), implicit signals (did the user retry, rephrase, or abandon?), task completion, latency, and cost. The key discipline — echoing the 'distrust the proxy' lesson of Chapter 23 — is to measure what actually MATTERS to users, not just what is easy to measure. A model that scores higher on a benchmark but frustrates real users is not an improvement.

At scale, failures are not exceptional — they are constant. GPUs fail, networks hiccup, replicas crash, dependencies time out, traffic spikes. A reliable service is designed to keep working DESPITE failures, not to assume they won't happen. Fault tolerance is what separates a service that has occasional bad days from one users can depend on.

A crucial reliability idea: when the system is overloaded or partially failing, DEGRADE GRACEFULLY rather than collapse. Under extreme load, it is better to serve everyone a slightly-worse response (a smaller faster model, a cached answer, a shorter generation) than to serve some users perfectly while others get errors or time out. A service that bends under pressure beats one that breaks.

LLM serving is expensive — GPUs cost a lot, and at scale the bill is enormous. Cost optimization is not an afterthought; it can be the difference between a viable product and an unprofitable one. Fortunately, many levers reduce cost, layering on top of the per-request optimizations of Chapter 27.

The fundamental unit of LLM serving cost is COST PER TOKEN (or per request). Every optimization ultimately aims to lower it: quantization and batching lower the GPU cost of producing each token; routing and caching avoid producing tokens with an expensive model when a cheaper path suffices. Tracking cost per token across the fleet, and per feature, reveals where the money goes and where optimization pays off most.

A production service must be OBSERVABLE — you must be able to see what it is doing, detect problems quickly, and diagnose them. Monitoring and observability are what let you operate a service reliably, catch issues before users do, and understand what is happening across a large fleet.

Monitoring data feeds two things: DASHBOARDS (live views of the system's health for humans to inspect) and ALERTS (automatic notifications when something crosses a threshold — latency spiking, errors rising, a replica down). Good alerting catches problems early, ideally before users notice; good dashboards let engineers diagnose and resolve them quickly. The aim is to know about and fix issues faster than users experience them.

Beyond system metrics (latency, errors), production LLM serving must monitor OUTPUT QUALITY — which is harder. Watch for spikes in refusals (the model suddenly declining too much), output anomalies, drops in user feedback, and shifts in behaviour after a deployment. Quality regressions can be subtle and invisible to system metrics: the service is 'up' and fast, but the model's answers got worse. Monitoring quality signals — not just whether requests succeed — is essential and distinctive to ML serving.

A public LLM service faces security, privacy, and abuse challenges beyond performance and reliability. Serving at scale means defending against misuse, protecting user data, and enforcing usage policies — responsibilities that grow with the service's reach.

When a service handles many users' or organizations' data ('multi-tenant'), strict ISOLATION is essential: one tenant's data, cache entries, and context must never leak to another. This shapes caching (don't share caches across tenants carelessly), logging (be careful what you store), and data handling (encryption, retention limits, deletion on request). Privacy is both an ethical obligation and, increasingly, a legal requirement.

Rate limiting protects the service from overload and abuse, and controls cost. By capping how many requests or tokens a user can consume per time window, the service prevents any single user (malicious or buggy) from overwhelming capacity or running up unbounded cost. Tiered quotas (more for paying customers) also implement the business model. Rate limiting is a basic but essential layer of any public LLM API.

Let us assemble the whole chapter — and much of Part VI — into a picture of a complete production LLM serving stack, from the user's request to the response and back.

Exercises 1–10 are pen-and-paper or design; 11–20 require code.

Further reading: “Orca: A Distributed Serving System for Transformer-Based Generative Models” (Yu et al., 2022). The vLLM, TensorRT-LLM, and Ray Serve documentation for production serving. Google's Site Reliability Engineering (SRE) book for SLAs, monitoring, and reliability principles. “The Tail at Scale” (Dean & Barroso, 2013) for tail-latency engineering. Literature on A/B testing and online experimentation (e.g. Kohavi et al.). Cloud providers' guidance on GPU autoscaling and cost optimization for inference workloads.

You have now taken a model all the way from raw mathematics to a deployed, scalable, multi-modal, tool-using service. Across six Parts you have built the foundations (Part I), classical methods (Part II), the Transformer (Part III), pretraining (Part IV), alignment (Part V), and deployment (Part VI). Part VII — Frontier Techniques — turns to the cutting edge and the open horizon: Mixture-of-Experts architectures that scale models efficiently (Chapter 32), long-context and memory methods that extend how much a model can attend to (Chapter 33), agents and multi-agent systems that push tool use to its limits (Chapter 34), and the open problems that remain unsolved at the frontier of the field (Chapter 35). Having mastered how LLMs work and how to deploy them, you are ready to explore where they are going — and to contribute to what comes next.