The 6× gap: reconciling an LLM cost log with the bill

In early April 2026, while checking a document-processing job for a client, I found that the provider’s bill was about six times higher than my application’s cost log. The investigation pointed to requests that timed out and were retried. Fixing the accounting took changes across several parts of the request pipeline.

That mattered because I use the cost log to price work, set budgets and tell a client what a job will cost.

The job

The job translated a long technical document, one page per request. It used OpenAI’s GPT-5.4 mini, and for a time it ran at the model’s highest reasoning effort (xhigh), so each page sent a large prompt and then reasoned for a long time. In hindsight, that setting may not have been the right choice for the task in the first place.

The requests went through my doc-processing-toolkit, a Python framework for AI agents that I designed, built and maintain. It wrapped each request in a retry decorator from the tenacity library: five attempts with exponential back-off. When a request succeeded, the toolkit priced the usage block in the response, using rates from its model registry (based on each provider’s published pricing), and wrote the cost to the job’s files.

Reconciling the log against the bill

I used the provider’s bill as the reference for reconciliation, and treated the log as the thing to check. The bill had limits of its own. The provider’s usage API returns usage in time buckets grouped by model, not per request, so it shows what a day cost but not which page caused it. I decided not to set up an admin key for it, because the job’s own logs had finer detail.

The logs showed a pattern straight away. Page after page had an APITimeoutError followed by a retry. On one page, the first attempt ran for about 30 minutes and timed out, the next four did the same, and the page failed. The same pattern repeated across many pages.

Those timeouts and retries, and the total on the bill, are what I observed. The link between them is an inference. On each timed-out attempt, the model would have read the whole prompt and could have reasoned for up to half an hour before the connection dropped, and that work would have been billed. The toolkit never received a response, so it had no usage block to price and recorded nothing. Then tenacity retried, and that attempt went unrecorded too. Because the bill only gives totals by model and time bucket, it can’t confirm what any single attempt cost.

For this job I wrote a one-off reconciliation script that backed up the existing records and corrected them. It counted the timeouts on each page and spread the unexplained remainder (the bill minus the log) evenly across them, using an assumed split of the cost (one-third input, two-thirds output) to estimate token counts, and it marked the new figures as estimates.

The corrected figures are still approximate; the more important work was changing the toolkit so later jobs wouldn’t need a script like it.

What caused the gap

Three problems in the toolkit combined:

1. The requests were blocking. The OpenAI, Google and Fireworks calls didn’t stream, and Anthropic streamed on some paths but not all. In a blocking call, the response and its usage block arrive together at the end, or not at all, so a request that died at minute 29 left nothing to count.
2. Retries weren’t logged. The retry decorator discarded each failed attempt, so the toolkit could only ever cost the last one.
3. Long reasoning runs kept hitting a 30-minute timeout, and three of the four provider clients had no explicit timeout set.

What I changed in April

I wrote a remediation plan and iterated on it with AI before changing any code, as I do for larger changes (the process is in Spec-first, test-first). It had three parts.

• Streaming on all four providers. Every provider call now streams internally. Agent.run() callers get the same result object as before, and a new Agent.stream() exposes the events. Each chunk resets the HTTP read timer, which reduces idle timeouts. Even a stream that fails partway leaves behind the tokens it had already delivered. That is partial evidence rather than a complete usage report (a request can still time out before its first chunk arrives), but it’s far better than the blocking calls, which left nothing to count.
• Explicit client timeouts, with a two-hour read timeout as a safety net behind streaming.
• Retry-cost accounting. A RetryAccumulator hooks into tenacity’s before_sleep callback and records each failed attempt: its tokens, its cost, where the token counts came from (token_source), the error and the duration. The run’s cost includes those attempts, and retry_count and retry_cost show how much of the total they account for.

The plan also settled some decisions before any code was written. Streaming is always on, with no opt-out flag, because a targeted fix for one provider is easier to maintain than a global switch. Cost alerts were left to client applications, which can watch the stream’s per-iteration cost events and stop early.

The plan set the toolkit’s accounting policy for a failed attempt as well. Here it is with the “rejected before stream” rule that was added later:

# The toolkit's accounting policy for a failed attempt: (input tokens, output tokens) to bill.
def bill_failed_attempt(error, partial_output_tokens, input_estimate):
    if rejected_before_stream(error):   # an HTTP error response, or no connection was made
        return 0, 0                     # assumed not billed
    if partial_output_tokens > 0:       # generation had started
        return input_estimate, partial_output_tokens
    if is_billable_error(error):        # e.g. a timeout before any chunk arrived
        return input_estimate, 0        # assume the prompt was read and billed
    return 0, 0

These are assumptions about how providers bill, not proof of what a given provider charged. A request rejected with an HTTP error response, or one that never connected, is assumed to cost nothing. A stream that had started is charged its estimated input plus the output tokens it delivered. An error the toolkit treats as billable, such as a timeout before the first chunk, is charged the estimated input only. Figures built from an estimate are flagged as estimates, as described below.

The streaming branch merged on 18 April, after a first phase that wrote acceptance criteria and placeholder tests before any implementation code. Two follow-up fixes came within a week. Five days after the merge, I found that each provider’s top-level error handler returned zero tokens and zero cost, which threw away whatever the stream consumer had counted. A Claude stream of about four minutes that closed early had been billed by Anthropic and recorded by the toolkit as free. The first of the two fixes made the error handlers return what the stream consumer had counted. The next day I found that the accumulator and the hook existed, and every provider function accepted an accumulator, but no caller ever created one. Retry accounting was effectively switched off, and cost still covered only the successful attempt.

These April changes, the streaming work and its two follow-up fixes, are the ones that relate to the original 6× discrepancy.

Later accounting defects

Between August and October I found more defects in how the toolkit records cost. They were real errors, but none of them was shown to be part of the April gap. They fall into three groups.

Attempts the toolkit didn’t see or didn’t count

The OpenAI and Anthropic SDKs retry failed requests themselves by default, which I found months after the April work. Each of tenacity’s five attempts could therefore hide up to two SDK retries: in the worst case, 15 requests, of which the toolkit saw 5. Release 0.4.0 builds the SDK clients with max_retries=0, and tenacity now also retries the statuses the SDKs used to handle. The worst case went from 15 requests, 5 of them visible, to 5 requests, all of them visible.

In October, 0.5.2 fixed a new form of the April failure: a truncated OpenAI reply was re-sent unchanged five times, the run failed, and cost read 0.0 for five billed requests.

Failed attempts priced wrongly

In September, a documentation review found that retry_cost was always 0.0. The hook recorded each failed attempt’s tokens but copied its cost from a field that every provider set to zero for a broken stream. Release 0.4.0 prices each attempt at registry rates, at the served tier and with the cached share, and flags it as estimated when its input count is the pre-flight estimate.

The same release fixed the opposite error. A request the provider rejected before streaming anything had been charged its estimated input; it now costs zero and is labelled rejected_before_stream. Over-reporting cost is an accounting bug too. Earlier, release 0.3.0 had fixed a report total that counted retries twice, adding retry cost on top of a cost that already included it.

Prices and usage that didn’t match the provider’s billing

In August, the toolkit started keeping a cost record for each request. A run’s cost is now the sum of its requests’ costs, because tiered pricing isn’t linear and summed tokens can cross a price threshold that no single request crossed. A later fix applied the same rule to failed attempts, where pricing two completed 150k- and 160k-token requests as one had doubled their cost.

Other mismatches were in the details. Streamed Gemini priority requests were priced at the standard rate and flagged as downgraded, because the toolkit read the served tier from a header that the streaming endpoint never sends. Anthropic cache writes were priced at the cache lifetime (TTL) requested, not the TTLs actually served. Release 0.5.2 changed Claude refusals to match how Anthropic bills them, since some refusal categories are billed and others aren’t. Then 0.5.4 started pricing the thinking tokens on Gemini image calls, which had never been counted; in the release check they were about an eighth of the call’s cost.

Testing cost tracking

Over those six months (April to October 2026), I came to treat cost tracking as a feature with its own specification and tests.

The tests need to use the data shapes providers actually return. In one Google bug, the toolkit built its token-breakdown keys by calling str() on an enum. That produced "MediaModality.AUDIO" where the pricing function looked for "AUDIO", so it found no audio tokens, and audio input on Google tool-calling runs was priced at the text rate. The pricing engine’s own tests stayed green the whole time, because they called it with hand-written keys that the provider code never produced.

Golden files freeze what goes on the wire. What the toolkit sends decides what the provider charges for: the service tier, the cache controls and the thinking settings. The toolkit keeps more than 100 golden files of request bodies. Each is produced by running a real Agent session against a client that records the exact request body, with no network involved. If a commit changes a body that was meant to stay the same, the test fails with a diff. Keeping the golden files current is ongoing work, because providers change their APIs over time, and I’m looking at ways to refresh them regularly.

Closed lists keep the toolkit in step with its dependencies. One test compares the toolkit’s table of OpenAI stream failure codes with the SDK’s own list. When the SDK adds a code, the test fails until someone decides whether that code is retried or fails at once.

Small live probes check the assumptions that fixtures can’t. In one probe on 23 September 2026 (gpt-5-nano hosting gpt-image-2 at 1024×1024, one render per quality setting), OpenAI’s published per-quality price table for image generation was 39% too high at one quality setting and 40% too low at another. The toolkit now prices those calls from the usage the API reports, and falls back to the table only when no usage is reported, flagging the cost as estimated.

Estimates are labelled as estimates. Every run reports a token_source (api_response, stream_partial, rejected_before_stream or estimated) and a cost_estimated flag. When the accounting policy treats an attempt as billed but the toolkit couldn’t measure its usage, it reports a flagged estimate rather than $0.00. The flag says a figure is estimated; it doesn’t say how far off it might be, so reconciling against the bill is still worth doing. I’m considering more automated checks to make that routine. The toolkit docs describe the current cost model.

A checklist for tracking LLM costs

• Reconcile your cost log against the provider’s bill regularly, by model and by day. The bill may only give totals, so per-request detail has to come from your own logs.
• Log every attempt, not only the successful ones. A failed request can still be billed if the model did any work.
• Count your retry layers. Your SDK, your HTTP client and your own code may all retry; keep one, and log it.
• Stream long generations. Streaming reduces idle timeouts and leaves partial usage when a stream fails, though it doesn’t guarantee a complete record.
• Price each request at the tier it was served, not from summed tokens or the tier you asked for.
• Label estimates as estimates, test the cost code with the data shapes the provider actually returns, and keep those shapes up to date, because they change.