Computing org-wide tok/s without trusting the invoice

A conventional network sniffer will see nothing: the traffic is encrypted. Measurement happens at the chokepoint, not on the wire.

Let's clear the false lead first. Pointing tcpdump or Wireshark at the link yields encrypted bytes and nothing else: all traffic to model APIs runs over TLS, and without the session key there are no tokens to count. A passive analyzer is the wrong tool.

Three approaches actually work. A forward proxy that terminates TLS — your applications point at it, it sees requests in the clear and relays them. Client-side instrumentation via OpenTelemetry's GenAI conventions, which means touching every application. Or eBPF capture hooking SSL_write before encryption — powerful, but awkward to operate.

For most organizations the proxy is the right trade. LiteLLM Proxy is open source, speaks both Anthropic's and OpenAI's protocols, and asks applications for nothing more than a changed base URL. It exposes litellm_input_tokens_metric, litellm_output_tokens_metric and litellm_total_tokens_metric to Prometheus, plus litellm_spend_metric for spend, labelled by model, team, key alias, and user.

Here's the part that catches people out: none of those metrics is a rate. They are cumulative counters. Throughput is derived at read time. In PromQL, sum(rate(litellm_output_tokens_metric[1m])) gives generated tokens per second across the whole organization, live. Add by (team) to break it down.

Size against the peak, not the average. A monthly average flattens precisely the moment that constrains you. max_over_time(sum(rate(litellm_output_tokens_metric[1m]))[30d:]) returns peak throughput over thirty days: that's the figure to reserve, and it is routinely three to five times the average.

Two warnings before putting this in production. A proxy becomes a single point of failure on the path of every request — plan for redundancy on day one. And it adds latency: measure its own with litellm_overhead_latency_metric rather than assuming it negligible.

What this instrumentation gives you that no console will: concurrency, peak, and per-team breakdown, in real time. That is exactly the set of numbers you need to decide whether reserved throughput costs you less than usage-based billing — and, if so, how much to reserve.