Measuring Cost and Latency Per Request

You cannot manage what you do not measure. This lesson covers the specific metrics every LLM product should track from day one and why unit-level measurement matters more than aggregate.

Two Levels of Measurement

Aggregate. Total cost this month, total requests this month, average latency across all requests. Easy to get from the provider's dashboard.

Per-request. Cost of this specific request, latency of this specific request, tokens used by this specific request, broken down by tool and model call.

Aggregate tells you how the bill compares to last month. Per-request tells you what is driving the bill and how to shape it.

You need both. If you only have aggregate, you cannot answer "which feature costs the most" or "which user segment is expensive". If you only have per-request, you cannot see the wood for the trees.

What to Capture Per Request

Minimum viable per-request metrics:

• Cost in ZAR (or your reporting currency). Sum across every model call and paid tool call in the request.
• Latency in ms. From first token in to last token out.
• Total tokens. Input + output. Broken down by call.
• Tool call count. How many tools were invoked.
• Model call count. How many round trips to the model.

Extra useful:

• Feature or intent tag. Which of your product's features handled this request.
• User cohort. Free tier vs paid, new vs established.
• Success/failure flag. Did the request achieve its goal.

All of these should be attributes on the top-level span in your tracing system (Module 4). Every framework's tracing tool can then aggregate them any way you want.

Latency Breakdown

Total latency is not one thing. Decompose it:

• Network latency. Time from your server to the model provider and back.
• Model latency. Time the model actually spent generating tokens.
• Tool latency. Time your tools took to run.
• Framework overhead. Time spent in your agent loop code between model calls.

Different bottlenecks need different fixes. If model latency is 90 percent of your budget, prompt caching and model choice matter most. If tool latency dominates, your database and external APIs need attention. If framework overhead is a big share, you probably have inefficient code between calls.

Cost Breakdown

Same treatment for cost:

• Model tokens. By far the largest component for most products.
• Cached tokens. If using prompt caching, these are much cheaper than fresh tokens — track separately.
• Paid tool calls. Search APIs, external LLM services, paid databases.
• Infrastructure. Server compute is usually negligible against LLM calls but worth checking.

Track cost per feature so you can see which parts of the product are expensive. Sometimes a rarely-used feature is disproportionately expensive; that is important to know before the bill arrives.

Aggregation Beyond Averages

Averages hide outliers. Track distributions:

• P50 latency. Median request. What the typical user experiences.
• P90, P95, P99. The slowest 10, 5, 1 percent. Long tails are what users complain about.
• Max. The worst single request in the window. Often this is a stuck-loop trace worth investigating.

Same for cost:

• Median cost per request.
• P95 cost per request. Are the top 5 percent unusually expensive?
• Cost of the top 1 percent of runs as a fraction of total spend. Often surprisingly high.

If your top 1 percent of runs are 20 percent of your spend, capping outlier cost (via iteration limits, token budgets) is your biggest lever.

Where to Store This

Two common patterns:

Traces plus derived metrics. Every trace has cost and latency attributes. Your tracing system aggregates. Simple. Downside: analytical queries over long time windows may be slow.

Traces plus a metrics database. Ship aggregated metrics (per-minute, per-user, per-feature) to a time-series database (Prometheus, InfluxDB, cloud provider metrics). Query fast, alert easily. Downside: another system to run.

Start with traces alone. Add a metrics DB when you need faster queries or more sophisticated alerting.

What to Alert On

Basic alerts every LLM product should have:

• Error rate above X percent for Y minutes
• P95 latency above X seconds for Y minutes
• Cost per hour above X (catches runaway loops fast)
• Total daily cost projection above X (catches slow-burn issues)

Set thresholds based on your baseline, not on absolutes. Alert if things worsen from normal, not if they exceed some fixed number.

Key Takeaway

Track both aggregate and per-request metrics. For each request capture cost, latency, token counts, tool call count, model call count and a feature tag. Decompose latency and cost into components so you know which bottleneck to attack. Watch distributions, not just averages: P95 and P99 tell the true user story. Alert on rate-of-change from your baseline, not on absolutes.