LLM Serving/fundamentals

LLM serving means running a trained model so that real users and real systems can use it. A model file on disk does nothing for a product. It becomes useful only when it is loaded into a runtime, exposed through an API, protected with access rules, monitored in production, and tuned for speed, cost and reliability. All of that together is serving.

If training builds the engine, serving runs the transport network around it. Most teams building AI products today start from a pretrained model, so their main engineering job is to serve that model well. That is also where infrastructure and backend teams spend most of their time.

Training vs serving

Why serving is the work most developers do

Look at a typical week of AI product work and almost all of it is serving:

• ›Integrating model APIs into product features
• ›Building prompt templates and guardrails
• ›Managing request and response formats
• ›Adding retries, timeouts and fallback logic
• ›Monitoring latency, token usage and errors
• ›Running deployments and rollbacks, and responding to incidents

How an LLM becomes an API

A practical request path has eight steps. The first three happen once, when the server starts. The other five happen for every request, which is why they decide your latency and your bill.

Model runtime vs application layer

A common mistake is to put all the logic in one place. Better systems split the work between two layers with a clear boundary between them.

• ›Model runtime: tokenization and decoding, inference scheduling, GPU and CPU placement, the token generation loop and runtime memory management. This is what tools such as Ollama and vLLM provide.
• ›Application layer: authentication and authorization, business logic, prompt assembly and validation, response shaping, safety checks and policy rules, and observability for product behaviour.

A clear boundary makes each side easier to scale and to debug. You can swap a runtime, or run several of them, without rewriting product code. You can also change a prompt or a policy without touching GPU configuration.

Where you see it

The same serving fundamentals sit behind very different products: end-user chat assistants, coding copilots in IDEs, internal support copilots, customer service reply tools, workflow automation, and classification and extraction pipelines.

Why serving quality matters

• ›User experience. Users leave quickly when responses are slow or inconsistent.
• ›Cost. Inefficient serving can multiply your infrastructure spend.
• ›Scalability. Growth needs stable performance under higher concurrency.
• ›Reliability. Production systems need predictable behaviour during failures and traffic spikes.

Common serving failure points

Teams tend to hit the same problems early:

• ›Long request queues during traffic spikes
• ›Timeouts that don't match between the application and the runtime
• ›No visibility into token usage
• ›Prompts that keep growing as chat history accumulates
• ›No fallback path when the runtime fails

The rest of this guide explains the basics that prevent most of them.

Inference is running a trained model on new input to produce output. Every time a user sends a prompt, your system runs inference. The basic flow is short: prompt → tokenization → model processing → next-token prediction → response.

Step by step

1. 01A prompt arrives from a user or another service.
2. 02The request is validated: size limits, required fields and so on.
3. 03The text is tokenized into token IDs.
4. 04The model processes the full input context.
5. 05The next token is predicted from the current context.
6. 06That token is added to the output.
7. 07Steps 5 and 6 repeat until a stop condition is reached.
8. 08The output is returned in full or streamed.

Why inference is expensive

• ›Models are large and need a lot of memory.
• ›Generating token by token repeats the computation many times.
• ›Long inputs add processing work before any output can start.
• ›Hardware is expensive, so it must stay busy to be cost-efficient.

If inference is not optimized, cost and latency go up together.

Why optimization matters

Good optimization improves several things at once: a faster first token, a faster full response, lower cost per request, higher throughput on the same hardware, and more predictable performance under load. That predictability gives teams confidence when traffic grows. Optimization is not a final polish step. It is a core part of serving design.

A real chatbot request

A user types: My payment failed twice, what should I check first? From the user's side this is one message and one reply. Behind it:

1. 01The API receives the request.
2. 02The system adds its role instructions and safety rules.
3. 03Conversation history may be added.
4. 04The full input is tokenized.
5. 05The runtime starts inference.
6. 06The first output token is generated.
7. 07More tokens are generated one at a time.
8. 08The application streams the output to the user.
9. 09Latency and token counts are recorded as metrics.

Prefill and decode

Runtimes split inference into two phases, and you will see these terms everywhere:

• ›Prefill: process the full input context before generation starts. It runs over all prompt tokens at once, so it is compute-heavy.
• ›Decode: generate the output one token at a time. Each step is small, but there are many of them.

This is why both sizes matter. Long prompts make prefill slower and delay the first token. Long outputs make decode slower and delay the last one.

LLMs don't read sentences the way people do. They process tokens. A token is a small unit of text: a whole word, part of a word, a punctuation mark or a pattern of spaces. Tokenization is the process of converting text into these units, and then into integer IDs the model can compute with.

Why tokens instead of words?

Language is messy and varied. Tokens give the model a consistent numeric format that:

• ›works across many languages
• ›handles rare words by splitting them into known pieces
• ›keeps the vocabulary to a manageable size
• ›supports efficient processing at runtime

The same text can split differently in different models. One tokenizer might produce Hello world, while another produces Hel lo world. This is normal, and it is why token counts for the same prompt vary between models.

Why token counts matter in production

Token count directly drives latency, cost, the maximum prompt size and the maximum output size. That is why production teams watch token usage closely.

The context window

The context window is the maximum number of tokens a model can handle in a single request. It covers the input tokens (system prompt, user prompt, history, retrieved documents) and the output tokens the model generates. If the total goes over the limit, the request has to be changed.

Take a model with an 8K-token context window:

Staying inside the limit

• ›Drop the oldest conversation turns
• ›Summarize earlier history
• ›Remove repeated instructions
• ›Cap the maximum output tokens
• ›Route the request to a model with a larger context window

LLMs generate text one token at a time in a repeated loop. Each new token is fed back as input for the next one. This is called autoregressive generation.

1. 01Receive the request. The API gets the prompt text, generation settings and any conversation history. Validation runs here.
2. 02Convert text to tokens. The tokenizer turns the input into token IDs that the model can process.
3. 03Process the context. The model reads the input tokens and builds its internal state. This is prefill, and it is heavy when the input is long.
4. 04Predict the next token. The model scores possible next tokens based on the context so far, and one is chosen.
5. 05Emit the token. The token is added to the output. With streaming on, it can be sent to the client straight away.
6. 06Repeat until done. The loop continues until a stop token appears, a stop rule matches, the output token limit is reached, or the request times out or is cancelled.

Why text appears one token at a time

Each token depends on the tokens before it, so the model cannot skip ahead to the final paragraph. This explains three things you see in practice: long responses take longer, streaming feels natural, and output token limits strongly affect latency.

What you notice in production

• ›Long prompts delay the first output token.
• ›Longer outputs increase tail latency.
• ›High concurrency can slow down each individual request.
• ›Small prompt changes can change token usage a lot.
• ›Streaming improves perceived speed even when total time stays the same.

Not every inference workload is interactive. You usually choose between real-time and batch modes, or use both.

Real-time inference

Use it when a user is actively waiting for an answer: chat apps, interactive assistants and AI features inside SaaS products. It needs a low latency target, an immediate response path and strong reliability for users.

Batch inference

Use it when requests can be queued and processed later: summarizing large sets of documents, nightly classification pipelines and bulk data enrichment. Throughput is the main goal, higher latency per item is acceptable, and cost efficiency at scale is better.

Which mode to use

• ›Real-time when interaction quality is the priority, any delay hurts the experience, and the product is conversational or assistive.
• ›Batch when the work can run asynchronously, volume matters more than immediate response, and output can be used later.
• ›Hybrid, which is common in production: a real-time path for user chat, and a batch path for background summarizing, indexing or tagging. Users get a fast experience and infrastructure costs stay under control.

Operational tips

These two metrics drive almost every production serving decision.

Latency

Latency is how long one request takes. For user-facing products it has a strong effect on satisfaction. Report it as percentiles rather than an average:

• ›p50: the typical user experience.
• ›p95: the slow edge that many users still hit.
• ›p99: the worst normal case.

Throughput

Throughput is how much work the system finishes in a time window, usually measured as requests per second (RPS) or tokens per second (TPS). For LLM serving, TPS often says more about runtime efficiency than RPS, because one request can produce ten tokens or ten thousand.

Why they compete

A setting that helps one metric often hurts the other:

• ›Serving requests at once in small batches can improve latency but lower throughput.
• ›Bigger batches use the hardware better and raise throughput, but can increase latency.
• ›Higher output token limits may improve answer quality, but raise both latency and cost.

Balancing them in production

Good systems set both goals explicitly: a user latency target and a cost or throughput target. Teams then tune the queue policy, concurrency limits, output token caps and model routing to meet both. Serving is an engineering discipline of constant balancing.

The KV cache is one of the most important practical ideas in modern LLM serving.

The problem

During generation, every new token depends on all the tokens before it. Without caching, the model would redo the attention work for the whole context at every step. Generation would be slower, compute costs higher and concurrency lower.

The solution

The KV cache stores the attention keys and values already computed for earlier tokens. The model reuses that work instead of recomputing it, so each decode step only has to process the newest token.

Benefits

• ›Faster token generation
• ›Less repeated computation
• ›Better scalability under concurrent load
• ›More consistent response times under pressure

The catch: memory

The KV cache is powerful, but it uses memory, and that memory grows with context length and with the number of concurrent sessions. Production teams have to balance session concurrency, context size and available GPU memory.

Why vLLM leans on it

vLLM is widely used because it manages KV cache memory very efficiently. Less memory is wasted, so more sessions fit on the same GPU, which means faster serving and higher concurrency. This is one of the main reasons vLLM performs well in production.

Quantization lowers the numeric precision used to store a model's weights. Using fewer bits per value shrinks the memory footprint and often speeds up inference. Common formats are FP16 (16-bit floating point), INT8 (8-bit integer) and INT4 (4-bit integer). As precision drops, memory use usually drops with it.

Why it matters for serving

Quantization can be the difference between a model that doesn't fit on your hardware and one that runs smoothly at an acceptable speed. Freeing memory also leaves more room for the KV cache, which can raise throughput.

Trade-offs

• ›Benefit: lower memory use
• ›Benefit: faster inference in many cases
• ›Benefit: larger models fit on smaller hardware
• ›Cost: possible loss of output quality
• ›Cost: you need task-level testing to confirm quality

Choosing a format

1. 01Start with a stable precision, often FP16.
2. 02Measure latency, throughput and quality.
3. 03Try INT8 and compare output quality on real tasks.
4. 04Consider INT4 only if memory pressure is high and quality stays acceptable.

Always decide with tests on your own workload, not assumptions.

Streaming means returning output as it is generated, instead of waiting for the full answer. A traditional response goes straight from waiting to the full response. A streamed response sends the first token, then the next, and so on until it is complete.

You have seen it in the typing effect of chat assistants, in customer support AI tools and in real-time writing and coding assistants.

Benefits

• ›Lower perceived waiting time
• ›More user trust during long responses
• ›Easy cancellation when the user wants to stop
• ›Early partial output can already be useful

Even when total generation time is the same, streaming usually feels faster. That is why many production chat systems stream by default.

Inference

Running a model on new input to generate output.

Token

A piece of text the model processes.

Context window

The maximum number of tokens allowed in one request, input and output combined.

Prefill

Processing the whole input before the first output token.

Decode

Generating output tokens one at a time.

Latency

The time taken to complete one request.

Throughput

The amount of work done per unit of time (RPS or TPS).

TTFT

Time to first token: how long until output starts.

KV cache

Stored attention keys and values reused during generation.

Quantization

A lower-precision format that reduces memory and can improve speed.

Streaming

Sending output incrementally while generation is still running.

Batch inference

Grouped or offline processing focused on throughput.

Real-time inference

Interactive processing focused on low latency.

• ›LLM serving turns trained models into usable APIs for real products, and it is where most engineering work happens.
• ›Inference runs a prompt through tokenization, model processing and token generation. Input size and output size both affect cost and speed.
• ›Tokens are the units models process. The context window is a hard limit shared by input and output.
• ›Output is generated one token at a time in a loop, so prefill sets time to first token and decode sets total time.
• ›Real-time inference optimizes for responsiveness. Batch inference optimizes for throughput. Many systems use both.
• ›Throughput and latency compete. Set explicit targets for both and track p95, p99 and TTFT.
• ›The KV cache removes repeated computation, but its memory use limits concurrency.
• ›Quantization cuts memory use and often speeds up inference, at some risk to quality, so test it on your own workload.
• ›Streaming improves perceived performance and is now standard for interactive AI products.

Dennis K is an AI infrastructure engineer who helps teams take language models from prototype to production. That includes choosing and deploying a runtime such as Ollama or vLLM, building OpenAI-compatible APIs, scaling with Ray Serve and Docker, sizing GPUs, and tuning latency, throughput and cost.

• ›LLM deployment and self-hosted inference
• ›Latency, throughput and GPU cost optimization
• ›KV cache, quantization and capacity planning
• ›Production monitoring, fallbacks and reliability

If your model is slow, expensive or hard to scale, hire Dennis or book a consultation below.