Latency and Cost Math for LLM Applications: Predict Your Bill

Your chatbot demo cost a fraction of a cent per question. Then you launched, and the monthly bill arrived with a number that made your stomach drop. At the same time, users complain that answers feel slow. Both problems come from the same place, and the math behind them is surprisingly simple.

• Demo: fraction of a cent
• Production: thousands of dollars

In the last episode we covered the Model Context Protocol, a standard way to hook tools into models. Today we put numbers on everything. We will derive a latency formula, build a cost model for a RAG chatbot and for an agent, learn the levers that cut both, and write a small Python calculator.

• Latency formula
• Cost model
• Optimization levers
• Python calculator

Every LLM application spends two budgets: time and money. Both are paid in tokens. But input tokens and output tokens behave very differently, and that difference is the key to this whole lesson. If you remember one idea today, remember that a token in is not the same as a token out.

• Both are counted in tokens
• Input and output behave differently

First term: time to first token, or TTFT. It is the delay between sending your request and receiving the very first piece of output. It includes network time, any queueing at the provider, and the model reading your entire prompt. It decides whether an app feels responsive or frozen.

• TTFT
• Request sent to first token back

Second term: inter-token latency, the gap between one output token and the next. Suppose it is twenty-five milliseconds. Then the model produces one divided by zero point zero two five, which is forty tokens per second for that single stream. Humans read around five to ten tokens per second, so forty feels fast.

• Gap between consecutive tokens
• Tokens per second = 1 / gap

Careful, because tokens per second means two things. Per-user speed is how fast one stream flows. Throughput is the total across all concurrent requests on a server. If a GPU serves thirty-two streams at forty tokens per second each, throughput is twelve hundred eighty. Operators love throughput; your user only feels per-user speed.

• Tokens/sec per request
• Tokens/sec across the server

Now money. Providers charge per million tokens, with separate prices for input and output. Throughout this video I will use hypothetical prices: three dollars per million input tokens and fifteen dollars per million output tokens. Real prices vary by model and change often, so always check your provider. The pattern, output costing several times more than input, is common.

• Priced per million tokens
• Output costs several times more

Recall the KV cache lesson. Every request has two phases. In prefill, the model processes all prompt tokens in parallel and fills the cache. In decode, it generates the answer one token at a time, each step reading the cache and producing a single new token.

• Prefill: read the prompt
• Decode: write the answer

Prefill is compute-bound. Thousands of tokens flow through the weights together as large matrix multiplications, so the GPU's arithmetic units stay busy. That is why prefill time grows roughly with prompt length, and why a long retrieved context raises your time to first token.

• Many tokens at once
• Big matrix multiplies
• GPU math units busy

Decode is memory-bound. To make one token, the GPU must stream essentially all the model weights from memory. As an illustration, an eight billion parameter model in sixteen-bit precision is about sixteen gigabytes. With roughly two terabytes per second of bandwidth, that is about eight milliseconds per token, whatever the arithmetic speed.

• One token per step
• Reload all weights each step
• Time ≈ bytes / bandwidth

Now the formula. Total latency is approximately TTFT plus N times t, where N is the number of output tokens and t is the time per output token. TTFT covers network, queue and prefill. The second term covers decode. Notice that output length multiplies the per-token time, while prompt length only touches TTFT.

• latency ≈ TTFT + N × t
• N = output tokens
• t = per-token time

Worked example, with illustrative numbers. TTFT is zero point four seconds. Per-token time is twenty-five milliseconds, and the answer has three hundred tokens. Decode takes three hundred times zero point zero two five, which is seven point five seconds. Add zero point four and the total is seven point nine seconds. Decoding is about ninety-five percent of the wait.

• TTFT = 0.4 s
• t = 25 ms, N = 300
• Total ≈ 7.9 s

This formula gives a practical rule. Halving the answer to one hundred fifty tokens drops latency to about four point one five seconds. Doubling the prompt barely moves the total, as long as TTFT stays small. So for speed, ask for concise outputs. Prompt length matters more for cost, as we will see.

• Cut output length first
• Prompt length mostly hits TTFT

The cost of one request is input tokens times the input price, plus output tokens times the output price. Convert prices to per-token by dividing by one million. Three dollars per million is three millionths of a dollar per token. Simple, but the details in each term are where bills grow.

• cost = in × p_in + out × p_out
• Prices per token = price / 1,000,000

Where do input tokens come from? In a RAG chatbot, the prompt stacks several parts. A system prompt of five hundred tokens. Four retrieved chunks of four hundred tokens each, so sixteen hundred. Conversation history of four hundred. And the user question of one hundred. That totals twenty-six hundred input tokens, far more than the question itself.

• System prompt
• History
• Retrieved context
• User question

Now price it. Input: twenty-six hundred tokens times three millionths is zero point zero zero seven eight dollars. Output: three hundred tokens times fifteen millionths is zero point zero zero four five. Together, about one point two three cents per request. Again, these prices are hypothetical, chosen to make the arithmetic clear.

• Input: 2,600 × $3/M = $0.0078
• Output: 300 × $15/M = $0.0045
• Total ≈ $0.0123

Look at the split. Output is only about ten percent of the tokens, but roughly thirty-seven percent of the cost. If your answers get twice as long, the bill climbs noticeably even though the prompt did not change. That is why ignoring output pricing is such a common budgeting error.

• Output: 10% of tokens
• Output: 37% of cost

Scale it up. Say each user sends about forty requests a month. Forty times zero point zero one two three is forty-nine cents per user per month. With ten thousand active users, that is four hundred thousand requests and about forty-nine hundred twenty dollars monthly. Small per-request cost, real monthly bill.

• Per request → per user → per month

Now see what moves the bill. Retrieve eight chunks instead of four and input grows by sixteen hundred tokens, adding about zero point zero zero four eight dollars per request, so the month climbs to roughly seven thousand dollars. Double the answer length and output adds another zero point zero zero four five. Every design choice is a cost choice.

• Retrieve 8 chunks, not 4
• Answers 2× longer

The same chatbot has a latency profile too. Prefill over twenty-six hundred tokens fits inside that zero point four second TTFT, then decode runs for seven and a half seconds. Without streaming the user stares at a blank screen for nearly eight seconds. With streaming, words appear after under half a second.

• Prefill of 2,600 tokens
• Decode of 300 tokens

Agents are different. In the ReAct loop, each step calls the model again with the whole conversation so far: system prompt, tools, the task, every earlier thought and every tool result. The context keeps growing, and you pay for all of it again at each step.

• Every step carries full history
• Tool results stay in context

Assume the first call has twelve hundred input tokens. At each step the model writes about one hundred fifty tokens, and a tool returns six hundred tokens. So the context grows by seven hundred fifty per step. Input at step k is twelve hundred plus seven hundred fifty times k minus one.

• Initial prompt: 1,200 tokens
• Each step adds 150 out + 600 tool result

Five steps give inputs of twelve hundred, nineteen fifty, twenty-seven hundred, thirty-four fifty, and forty-two hundred. The sum is thirteen thousand five hundred input tokens. A naive estimate of five times twelve hundred would say six thousand. Reality is more than double, because every step re-reads everything before it.

• 1,200 + 1,950 + 2,700 + 3,450 + 4,200
• Total input = 13,500

Cost the run. Thirteen thousand five hundred input tokens at three millionths is four point oh five cents. Seven hundred fifty output tokens at fifteen millionths is about one point one cents. The run costs roughly five point two cents, over four times a single RAG answer, for a single task.

• Input: 13,500 × $3/M = $0.0405
• Output: 750 × $15/M = $0.01125
• Total ≈ $0.052

In general, total input is N times I-zero plus g times N times N minus one, divided by two. The second term grows with the square of the number of steps. At ten steps the input is forty-five thousand seven hundred fifty tokens, over three times the five step run, for twice the steps. Long agent loops get expensive quickly, and slow too, since each step adds its own TTFT and decode.

• Total input = N·I0 + g·N(N−1)/2
• 10 steps → 45,750 tokens

Lever one is the plainest: send fewer tokens. Tighten the system prompt, use a reranker to keep the best two chunks rather than four, and summarize old history. Cutting the prompt from twenty-six hundred to about thirteen fifty roughly halves input cost, and it often improves quality by removing distracting text.

• Trim system prompt
• Fewer, better chunks
• Summarize history

Lever two is prompt or prefix caching. Many providers let you reuse the computed state of an identical prompt beginning, billing cached tokens at a discount and skipping some prefill work. Prefix must match exactly, so put stable content first and variable content last. Check your provider's actual discount and rules.

• Reuse a repeated prefix
• Cached input billed at a discount

Apply this to the agent, with a hypothetical cached price of thirty cents per million. Each step re-sends the earlier context, so only the first twelve hundred and four new slices of seven hundred fifty are fresh: forty-two hundred tokens at full price, nine thousand three hundred cached. Input becomes about one and a half cents. With output, the run costs about two point seven cents, nearly half.

• Only new tokens pay full price
• Cost drops to about $0.027

Lever three: not every question needs your biggest model. A small model, or a classifier acting as a router, can handle the easy majority, escalating only hard cases to a large model. Smaller models also have lower per-token time, since there are fewer weights to read. Measure quality on your own evals before routing.

• Easy queries → small model
• Hard queries → large model

Lever four is a bundle. Streaming does not shorten total latency, but it slashes perceived latency. Batch endpoints, where offered, often give discounts for jobs that can wait. And exact or semantic response caching returns stored answers for repeated questions at nearly no cost. Semantic caches risk returning a wrong answer, so set a strict similarity threshold.

• Stream for perceived speed
• Batch offline jobs
• Cache repeated answers

Lever five relates to the reasoning models from the chain-of-thought lesson. Thinking tokens are generated tokens, so they are billed at output price and add to the N in our latency formula. A question answered with two thousand reasoning tokens costs and waits far more than the visible answer suggests. Use a thinking budget, and reserve deep reasoning for hard tasks.

• Reasoning tokens are output tokens
• Set a thinking budget

Time for code. Write a function called estimate that takes input tokens, output tokens, the two prices per million, a TTFT, and a per-token time. It returns cost as input times price in plus output times price out, each divided by a million, and latency as TTFT plus output tokens times per-token time. Keep prices as parameters, never hard-coded.

• Pure Python
• Prices as parameters

Extend it for agents. Keep a context variable that starts at the initial prompt size. In a loop over steps, add that context to total input, add the output length to total output, then grow context by the output plus the tool result. Print a table each step, and you will watch the quadratic growth appear.

• Loop over steps
• Grow context each step

Now measure real systems. Wrap your provider-neutral call_llm with streaming enabled. Record the start with a performance counter. On the first chunk, store the elapsed time as TTFT. On each later chunk, store the gap since the previous one. At the end, compute total time, number of chunks, and output tokens divided by decode time.

• Record TTFT
• Record inter-token gaps
• Compute tokens/sec

One timing means nothing. Run hundreds of requests with realistic prompts and store every sample in a list. Then compute percentiles from the list, not only the mean. That prepares us for the mistakes that cause the most pain in production.

• Run hundreds of requests
• Store every sample

Mistake one: budgeting from prompt size alone. Output tokens cost more per token and also drive decode time, so they hit both budgets. Set maximum output limits, ask for concise formats, and count output tokens in every estimate. Include reasoning tokens, too.

• Output priced higher
• Output also sets decode time

Mistake two: averaging. Suppose ninety-five requests take three seconds and five take twenty. The mean is three point eight five seconds, a number almost nobody experiences. Long outputs, queueing, retries and agent loops create heavy tails. Track p50, p95 and p99, and set targets on those.

• Use p50, p95, p99
• Tail latency drives complaints

A few more traps. Retries multiply cost, so count them. Agents without a step limit can loop and burn money. And prices and speeds change, so keep them in configuration, then recheck regularly. Always set budgets and alerts per user, so one runaway session cannot surprise you.

• Forgetting retries
• Ignoring agent step limits
• Stale prices

Recap. Latency is about TTFT plus output tokens times per-token time, with prefill compute-bound and decode memory-bound. Cost is input tokens times input price plus output tokens times output price. Agents re-send context, so cost grows with the square of steps. Cut tokens, cache prefixes, route models, stream, and measure p95 and p99.

• latency ≈ TTFT + N × t
• cost = in·p_in + out·p_out
• Agents grow quadratically
• Watch p95, p99

Since decode time is roughly weight bytes divided by memory bandwidth, an obvious question follows: what if the weights were smaller? That is quantization, our next episode, covering INT8, INT4, GPTQ and AWQ.

• Shrink the weights
• Faster decode, lower memory