AI inference infrastructure: designing for latency, cost, and reliability

Production inference is the part of an AI system that turns a trained model into a dependable service. It receives a request, prepares inputs, selects a model and device, schedules computation, returns an answer, and records enough evidence to explain what happened. The model weights are important, but they are only one component. Queueing, memory, networking, batching, release management, and failure handling often determine whether users experience a responsive product or a timeout-prone demo.

This article is scoped to information available on 20 October 2026. It focuses on the engineering decisions behind online inference, especially transformer and large-language-model (LLM) services, while noting where the same ideas apply to image, speech, embedding, and classical predictive models. Vendor capabilities and open-source project behavior change quickly; standards, papers, and product documentation are therefore treated differently from general design principles.

Start with the service contract, not the accelerator

Begin by writing the contract that the inference service must satisfy. “Fast” is not a useful target. Specify the request mix, the maximum prompt and output lengths, the required quality, the availability objective, and separate latency objectives for interactive and background traffic.

For an interactive text endpoint, useful measures include time to first token (TTFT), the delay between streamed output tokens, total response time, and the proportion of requests that finish within each target. TTFT includes admission, queueing, prompt processing (often called prefill), and the first decode step. Decode then produces output tokens one at a time. A user may tolerate a moderate total duration if the first token appears quickly, but a long pause between tokens feels like a stalled connection. A batch summarisation job has almost the opposite priority: it can accept queueing if it maximises completed tokens per dollar.

Define a request budget that can be decomposed. For example:

• 20 ms for authentication, input validation, and routing;
• 80 ms for queueing and scheduling at the target percentile;
• a TTFT objective for prompt processing;
• a time-between-tokens objective for streaming;
• a hard output-token and wall-clock limit; and
• an error budget for overload, dependency failures, and invalid requests.

The percentile matters. A p50 result describes a typical request; p95 or p99 shows what a user sees during contention. Capacity planning based only on averages hides queue growth. Measure the distribution by model, route, tenant, prompt-length bucket, output-length bucket, and response mode. Keep the metric definitions stable: “latency” should say whether it includes network transit, queue time, tokenisation, streaming flushes, and post-processing.

What a production serving system actually contains

A serving path normally has an API gateway, an admission controller, a scheduler, a model runtime, device workers, and telemetry. The gateway authenticates the caller, enforces quotas, validates the schema, and assigns a request or trace identifier. The admission controller rejects work that cannot be completed within policy, such as an overlong prompt or a request that would exceed a tenant’s token budget. The scheduler puts admitted work into model-specific queues and decides when to form a batch.

The runtime loads a model artifact, maps its tensors to a device, executes kernels, manages intermediate state, and converts outputs into the API format. For an LLM, it also maintains a key-value (KV) cache: the attention keys and values already computed for a sequence. Device workers may be replicated on one host or distributed across hosts. A control plane manages versions, rollout, health, and capacity; the data plane handles requests.

Open the full-size diagram

This separation is a design choice, not a universal standard. KServe, for example, exposes a Kubernetes custom resource and a common deployment model for predictive and generative inference, including model revisions, health checks, networking, autoscaling, and canary capability (KServe documentation). Triton, vLLM, and TensorRT-LLM are runtime or serving components with different interfaces and feature sets. An OpenAI-compatible HTTP shape may simplify client migration, but it does not make the scheduling, token accounting, safety, or failure semantics identical across implementations.

Choose an accelerator for the workload you have

An accelerator is not selected by parameter count alone. The relevant workload includes arithmetic precision, memory capacity and bandwidth, interconnect speed, batch shape, context length, concurrency, and the mix of prefill and decode. A smaller model with long contexts can consume more memory than a larger model with short prompts because each active sequence needs KV-cache space. A high-throughput offline job can exploit a device that would be unsuitable for a latency-sensitive endpoint.

For dense transformer inference, prefill is often compute-intensive because many input tokens are processed in parallel. Decode is commonly more memory- and communication-sensitive because each active sequence advances by one token per iteration. Mixture-of-experts models change the arithmetic and communication pattern again. Embedding and classification services may be dominated by matrix multiplication with stable, batchable shapes. Image and speech workloads have their own preprocessing and transfer bottlenecks.

Compare candidate devices with the actual model and a representative request trace. Record throughput, TTFT, time-between-tokens, tail latency, device memory headroom, power, startup time, and cost per useful output token. Include host CPU, memory, storage, and network costs. For multi-device inference, test the communication path, not merely the sum of single-device numbers. Tensor or pipeline parallelism can make a model fit, but collective communication and synchronisation can erase a theoretical compute advantage. TensorRT-LLM documents multiple parallelism and optimisation paths, but its behavior is specific to its supported software and NVIDIA-oriented stack (TensorRT-LLM documentation); it is not a general guarantee for every accelerator.

Use a benchmark as a comparison instrument, not a promise. MLPerf rules define workloads, quality targets, load-generation scenarios, and reporting constraints so that submitted results have a common baseline (MLPerf Inference rules). A closed benchmark result still may not match a service with different prompts, output lengths, network overhead, safety checks, or traffic bursts. Reproduce a vendor or community result only when the model, precision, batch policy, software versions, and measurement boundaries match.

A practical selection sequence is: first, eliminate devices that cannot hold the model plus worst-case KV cache; second, eliminate devices that miss quality or tail-latency targets; third, compare capacity and cost at the target load; and only then consider peak tokens per second. Leave memory headroom for fragmentation, runtime workspaces, transient batches, and a rolling model update.

Batching is a queueing policy, not a free speed switch

Batching combines independent requests so that a device can reuse loaded weights and execute larger tensor operations. It usually improves utilisation and throughput, but it adds waiting and can make one long request hold shorter requests hostage. The right question is not “what is the biggest batch?” but “how much queueing can this class of request spend to buy useful device work?”

For conventional stateless models, dynamic batching waits briefly for requests and forms a batch. Triton’s documented configuration includes maximum batch size, optional preferred sizes, queue delay, priority levels, queue limits, and timeout actions (Triton dynamic batching guide). A short, measured queue delay can raise throughput. A large delay can turn a healthy system into a latency failure even while the accelerator looks busy. Preferred batch sizes make sense only when a particular runtime profile is materially faster; they should not be assumed to help every model.

Autoregressive generation requires a more specialised policy. Each request performs repeated decode iterations and completes at a different time. Continuous or iteration-level batching lets requests enter and leave between iterations instead of waiting for an entire request batch to finish. The PagedAttention paper describes how vLLM manages the growing KV cache with paging-like blocks and reports its own 2023 evaluation against earlier systems; the result is evidence about that paper’s implementation and test conditions, not a universal multiplier (PagedAttention paper). Current vLLM documentation lists continuous batching, chunked prefill, prefix caching, and multiple hardware backends as supported features, but individual model and hardware combinations still require testing (vLLM documentation).

Prefill and decode compete for the same device. An eager long prefill can interrupt active decodes and create a generation stall. Chunked prefill limits the amount of prompt work admitted in one step, allowing decode work to continue. Sarathi-Serve describes this throughput-versus-tail-latency problem and evaluates chunked-prefill scheduling under specific models and hardware (Sarathi-Serve paper). The engineering choice is a policy: reserve capacity for decodes, cap prompt chunks, or run separate prefill and decode pools when the operational complexity and traffic volume justify it.

Separate queues by service class. A short interactive request should not wait behind an offline request with a very long context. Apply weighted fairness or explicit priority carefully; priority without a limit can starve lower classes. Use admission control before the GPU queue: once estimated work exceeds the remaining deadline, reject, defer, or route the request rather than accepting it and producing a predictable timeout.

Quantisation is a measured quality and systems decision

Quantisation represents weights, activations, or cache values with fewer bits than the training representation. It can reduce memory footprint and bandwidth, increase the number of requests that fit on a device, and sometimes improve throughput. It can also reduce accuracy, change output distributions, complicate kernel support, and make numerical failures appear only on specific languages, code patterns, long contexts, or safety prompts.

Distinguish weight-only quantisation from weight-and-activation quantisation. Weight-only methods may reduce model storage and memory traffic while retaining higher-precision activations. Activation quantisation can deliver larger arithmetic gains on hardware with suitable low-precision instructions, but its calibration and outlier behavior require more care. KV-cache quantisation is a separate choice because it affects every active sequence and can change long-context quality.

GPTQ is an example of post-training, weight-oriented quantisation. Its paper reports experiments on 3- and 4-bit weights and specific end-to-end speedups on particular GPUs, while also exploring more aggressive settings (GPTQ paper). Those measurements should not be copied into a capacity plan. Quantisation is only beneficial when the selected runtime has an efficient kernel for the selected format; a smaller file can still run slowly if it causes dequantisation overhead or falls back to an unfused path.

Treat each precision as a release candidate. Evaluate task accuracy, calibration, refusal and safety behavior, structured-output validity, multilingual behavior, hallucination-sensitive cases, and long-context performance. Compare not just a mean score but the worst important slice. Keep an unquantised or higher-precision fallback for critical routes. Version the calibration data and record the exact runtime kernel path. A five percent quality regression may be acceptable for a low-risk classification queue and unacceptable for a regulated decision-support workflow; there is no infrastructure-only answer.

Caching saves work, but creates correctness and privacy obligations

There are several different caches, and they should not be conflated.

Weight and engine caches store downloaded model files, compiled kernels, or serialised execution engines. They reduce cold-start time and bandwidth. Key them by an immutable artifact digest, runtime, driver, device capability, and compilation options. Never let a rolling deployment reuse an engine built for an incompatible driver or architecture without validation.

KV caches store attention state for active sequences. A server can page or share cache blocks, evict inactive sessions, or offload state to host memory. More cache space can increase concurrency, but it also competes with model weights and workspace memory. Set per-request context limits and a global cache budget. Track cache hit, allocation, eviction, and out-of-memory events. Do not assume that a runtime’s cache policy is portable: vLLM and TensorRT-LLM expose different controls and implementation details.

Prefix caches reuse the computed state for a stable prompt prefix, such as a long system instruction. They are useful when the same prefix recurs and the prefix is safe to share. The cache key must include the model revision, tokenizer and relevant generation context. A prefix containing tenant data must not be shared across tenants. Authentication and authorisation still apply after a cache hit.

Response caches return a previous completed result. They are appropriate only when the request is deterministic enough and freshness, identity, and policy allow reuse. Include the full semantic input in the key: model and prompt version, parameters, tools, locale, tenant policy, and data freshness. Avoid caching outputs that depend on private context, a changing database, or per-user permissions. Log whether an answer came from a response cache so evaluation and incident analysis do not mistake a cache hit for fresh inference.

Caching changes the work being measured. The current MLPerf rules, for example, prohibit cross-query caching in the relevant benchmark context while allowing normal within-query optimisations; a performance harness must follow its own rules rather than quietly benefiting from a response cache (MLPerf Inference rules). In production, define cache correctness and invalidation before optimising hit rate.

Route requests by capability, state, and deadline

Routing is more than load balancing across identical pods. A router may choose a model tier, precision, accelerator pool, region, tenant queue, or fallback. It should know which model revision is loaded, how much memory is free, what queue delay is expected, whether a session is stateful, and whether a request has a deadline.

Use capability-based routing. An image request should not land on a text-only worker; a request requiring a tool or structured decoder should land only on a runtime that supports that feature. For a stateful sequence, route subsequent messages to the worker holding its state or use an explicit external state store. Triton’s sequence batcher is documented for stateful models where requests in a sequence must remain associated with the same model instance, in contrast to its stateless dynamic batcher (Triton dynamic batching guide).

Use weighted traffic splitting for canaries, but keep a stable control group. Compare quality, tail latency, error rates, token usage, and cost, not just HTTP success. A model that returns 200 responses with malformed JSON is not healthy. Drain a worker before removal, allow active streams to finish or close them with an explicit retry-safe status, and make retries idempotent. Blind retries can multiply expensive generation and create an overload loop.

A useful routing score is estimated completion time under the request’s deadline, subject to policy constraints. It can incorporate queue delay, prompt length, expected output length, model capacity, and locality. Keep the scoring model simple enough to debug. If its telemetry is stale, fail closed to a conservative route rather than sending traffic to a worker that may be full.

Observe the path the user pays for

Device utilisation is necessary but insufficient. A GPU at 95% utilisation can be healthy, or it can be spending its time on oversized prefills while interactive decodes miss their tail objective. Monitor request rate, admitted and rejected requests, queue depth and age, batch size, batch formation delay, TTFT, time-between-tokens, end-to-end latency, output tokens, cancellations, timeouts, and error reasons. Break them down by model, revision, route, tenant, precision, context length, and response mode.

Measure queue time separately from model execution. For an LLM, record prefill and decode work separately when the runtime exposes it. Record cache hit/miss and eviction, device memory, host memory, network transfer, model load time, and cold-start rate. Cost dashboards should combine token counts with accelerator-seconds and show cost per successful request and per useful output token. “Tokens per second” without input/output definition is ambiguous.

OpenTelemetry’s GenAI semantic-conventions project defines a shared vocabulary for generative-AI spans, metrics, and events, including client inference and model-related attributes (OpenTelemetry GenAI semantic conventions). Treat it as a schema to adopt deliberately, not as permission to record every prompt. Prompt and completion content can contain secrets or personal data. Prefer metadata, hashed identifiers, sampled redacted payloads, and access-controlled exemplars. Separate operational traces from approved quality-review datasets.

Runtime metrics should be correlated with request traces. Triton exposes Prometheus metrics for request and GPU statistics; its documented counters distinguish requests, inferences, executions, pending work, and failures, which helps infer batching and backlog behavior (Triton metrics documentation). Alert on symptoms first: rising p99 TTFT, queue age, deadline misses, error ratio, or out-of-memory events. Use utilisation and temperature as diagnostic signals rather than user-facing SLOs.

Evaluate the system, not only the model

Evaluation has two tracks. Offline evaluation tests quality and safety on a fixed, versioned set. Include representative traffic, adversarial inputs, boundary lengths, languages, tool calls, schema-constrained outputs, and cases where a refusal or abstention is correct. Store inputs securely, define the scoring method, and compare the candidate with the currently deployed baseline. Human review is still useful for criteria that automatic metrics cannot capture, but reviewers need a rubric and a way to inspect model version and context.

Online evaluation tests the service under realistic load. Replay a trace with the same prompt-length and output-length distribution, arrival burstiness, cancellation pattern, and tenant mix. Measure cold and warm paths. Test single-user latency, saturated capacity, partial device loss, model reload, dependency timeout, cache eviction, and a noisy neighbor. Use load generation that respects streaming and cancellation; a client that waits for full bodies can produce misleading TTFT numbers.

A benchmark run must state the boundary. Does it include tokenisation, safety filtering, network serialization, queueing, and model loading? Does “throughput” mean requests, input tokens, output tokens, or completed conversations? The MLPerf rules are useful precisely because they specify scenarios and constraints, but a production test should add the product’s own workload and SLOs. Never publish a single benchmark number without model, precision, hardware, software, request shape, concurrency, percentile, and quality target.

Gate a release on both quality and capacity. A candidate that gains 8% throughput but increases p99 TTFT by 40% may be a win for batch traffic and a regression for chat. A quantised model that passes aggregate accuracy but fails a safety slice should not be silently routed to all users. Keep a rollback artifact warm enough to meet the recovery objective.

Secure the inference boundary

Protect the endpoint as an expensive, stateful service. Authenticate callers, authorise model and tool access, enforce tenant isolation, validate content types and lengths, rate-limit requests, and cap input and output tokens. Put hard limits on concurrent streams and total work per request. These controls address both accidental overload and deliberate cost attacks.

Prompt injection, sensitive-information disclosure, improper output handling, excessive agency, system-prompt leakage, and unbounded consumption are among the risks listed in the OWASP 2025 LLM and GenAI application guidance (OWASP GenAI risks). Infrastructure cannot solve all application risks, but it can constrain blast radius. Keep tools behind explicit allow-lists and typed interfaces. Treat model output as untrusted data before it reaches a browser, SQL interpreter, shell, workflow engine, or downstream API. Require separate authorisation for consequential actions; an LLM should not be the authority that grants itself access.

Encrypt transport, protect keys, restrict model and telemetry storage, and separate operator roles. Keep secrets out of prompts and traces. Apply retention and deletion rules to inputs, outputs, caches, and debug samples. Ensure that a cache namespace includes tenant and policy context. Scan model artifacts and containers, verify provenance and digests, and stage upgrades in an isolated environment. Security requirements differ by jurisdiction and sector; the NIST Generative AI Profile is a voluntary risk-management companion to AI RMF 1.0, not a universal legal checklist (NIST AI RMF Generative AI Profile).

Plan for abuse telemetry. A sudden increase in maximum-length prompts, repeated retries, unusual tool requests, or cache-bypass patterns should be visible without exposing content. Security controls must have a measured failure mode: if the policy service is unavailable, choose whether to fail closed, serve a restricted model, or return a clear error. Document that decision per endpoint.

Control cost with capacity and graceful degradation

Cost is a systems metric. The main drivers may include accelerator time, replica idle time, model transfer, storage, network egress, CPU preprocessing, observability, and the number of output tokens. A lower per-token rate can still cost more if it encourages longer prompts, retries, or low cache hit rates. Report cost per successful request and per accepted output token by route and tenant.

Use controls that preserve the product’s core value:

1. Limit work early. Enforce maximum input and output tokens, deadlines, concurrency, and tool-call budgets before admission. Return a useful validation error rather than starting work that cannot finish.
2. Shape traffic. Separate interactive, batch, and premium queues. Use queue delay and batch limits appropriate to each class. Schedule offline work in capacity valleys only when the business can tolerate delay.
3. Reduce unnecessary computation. Reuse safe prefixes, cache deterministic results, trim repeated context, and choose a smaller model for simple intents. Measure the quality cost of every shortcut.
4. Autoscale on meaningful signals. CPU utilisation is a poor proxy for GPU queue age or token backlog. Kubernetes HPA can scale from resource, object, or external metrics and applies stabilisation behavior, but the controller follows the metric supplied to it; it does not infer TTFT or model quality by itself (Kubernetes HPA documentation). Use queue age, pending work, token rate, and warm capacity where the platform supports them.
5. Keep a floor of warm capacity. Scale-to-zero saves money but adds model-load latency and creates a cold-start failure mode. Use it for tolerant background workloads, not automatically for an interactive critical path.
6. Set a spending circuit breaker. Per-tenant budgets, global token ceilings, and anomaly alerts should stop runaway traffic. The circuit breaker should preserve a restricted health or emergency route.

Graceful degradation should be explicit and ordered. Under rising load, first stop admitting work that exceeds its deadline. Then reduce optional output length, disable expensive tools or secondary reranking, route simple requests to a smaller validated model, and reserve the highest-quality model for protected traffic. If the primary model is unhealthy, return a safe cached answer only where freshness and identity permit; otherwise return a bounded, honest error. Do not silently substitute a model whose quality or data-handling policy is unknown.

Load-test the degradation ladder. Verify that queues drain after load falls, retries do not amplify traffic, streaming connections close predictably, and a degraded response is labelled for clients and telemetry. Reliability is not “the model never fails”; it is a controlled response that keeps the service within its safety and cost boundaries when a dependency, device, or model revision fails.

A concrete implementation and decision framework

A small team can make these decisions in four passes.

Pass one: define the workload. Capture a representative trace with request arrival times, prompt and output lengths, cancellation, tenant, region, and endpoint. Remove or protect sensitive content. Define quality slices, SLOs, cost ceiling, and unacceptable failure modes. Decide which requests may be degraded.

Pass two: establish a reproducible baseline. Pin one model, tokenizer, precision, runtime, driver, device, and configuration. Run warm and cold tests at low load and increasing concurrency. Report p50, p95, and p99 TTFT, time-between-tokens, total latency, throughput, error ratio, memory headroom, and cost. Confirm quality before tuning throughput.

Pass three: vary one system lever at a time. Sweep batch size and queue delay; then test continuous batching or chunked prefill; then compare precision; then test cache policies; then compare device and parallelism layouts. Keep a run manifest. Reject any change that improves a headline number while violating an important quality or tail-latency gate. For each candidate, repeat the test with short and long contexts and with bursty arrivals.

Pass four: make operations part of the release. Add dashboards, traces, budget alerts, overload tests, rollback drills, artifact provenance, and a degradation policy. Canary the model and runtime separately when possible. Drain and recover a worker. Exercise an out-of-memory event, a slow policy dependency, a full queue, a failed accelerator, and a cache namespace mistake. Record who can change routing, limits, model artifacts, and retention.

The output should be a short decision record: workload assumptions; quality threshold; chosen model and precision; device and replica shape; batching policy; cache rules; routing and fallback; SLOs; cost formula; security controls; telemetry; and rollback trigger. This document is more valuable than a claim that a particular server is “high performance,” because it exposes the assumptions that determine whether the choice remains valid.

Conclusion

Reliable inference comes from coordinating model, memory, scheduler, device, router, and operations. Choose hardware from measured workload behavior. Treat batching as a queueing policy, quantisation as a quality-controlled release, and caching as a correctness and privacy boundary. Measure queueing and tail latency separately from device utilisation. Evaluate the complete request path under realistic bursts, and make security, cost limits, and degradation behavior explicit before launch.

The most portable principle is simple: optimise for a stated service contract, then verify every optimisation against that contract. Runtime features will change, accelerators will change, and model architectures will change. A disciplined workload trace, reproducible benchmark, bounded scheduler, privacy-aware telemetry, and tested fallback path remain useful across those changes.

References and further reading

• vLLM documentation
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
• TensorRT-LLM documentation
• Triton Inference Server dynamic batching
• Triton Inference Server metrics
• MLPerf Inference rules
• OpenTelemetry GenAI semantic conventions
• NIST AI RMF Generative AI Profile
• OWASP GenAI and LLM Top 10 risks
• KServe
• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers