Rounak Zoeb · Writing

A platform for AI agents that handle long-running tasks: architecture and roadmap

Mon, 19 Oct 2026 00:00:00 GMT

I’m working with a co-developer on a platform for AI agents that can run tasks and maintain small applications. The initial design covers local and cloud execution, permissions and persistent state. This post explains the choices we’ve made so far and what we plan to build first.

The idea is a personal AI assistant that can run locally, in the cloud or both, and take on longer-running jobs rather than single chat sessions. Examples include a weekly research summary, monitoring an inbox, or a small app for managing engineering documents that an agent builds and keeps up to date. Its working title is AI App Builder.

Status, October 2026: the initial design is drafted, and the core is in early development. The platform builds on my existing agent framework, doc-processing-toolkit, and tool library, doc-processing-std-tools.

The first workflow: a document register

On engineering projects, a document register is the running list of a project’s documents, such as drawings, specifications and reports, with their numbers, revisions and status. It’s the first specialised application we plan to build.

Registers are part of engineering information management: keeping the information about a plant or other asset correct, complete and traceable, from design and construction into operation. A large project produces thousands of documents, and many go through several revisions. If the register falls behind the documents, it no longer shows which ones are missing, late or superseded. That matters most at handover, when the contractor has to show that every required document has been delivered.

Much of this work is still done by hand. The information is spread across a document management system, many spreadsheets, shared drives and email; document numbers and revisions sit in title blocks, cover sheets and file names; and each project’s register has its own layout. The AI features that document management vendors have added mostly work on the data inside their own products, so keeping the spreadsheet registers in step with the documents is still clerical work for people.

General AI assistants, such as ChatGPT and Claude, can read a few documents in a chat and answer questions about them. A register asks for more: it covers more documents than fit in one conversation, it has to be kept current for the life of the project as new revisions arrive, every value has to be traceable to its source, and on many projects the documents can’t leave the client’s own systems.

The plan is to give the agent a folder of project documents and have it build a register application from them. A master document register, a transmittal register (what was sent to whom, and when) and a handover view would all be configurations of that one application. Each field the AI fills in would carry a value, a confidence score and a citation that opens the source passage. Low-confidence rows would go to a review queue, and corrections would be kept as training data.

Most of what that needs is in the software around the model. In the design, each document is parsed once into numbered blocks that keep their page and position, and the agent works through them a batch at a time, so the size of the document set isn’t limited by how much text a model can take in at once (its context window). Choosing what goes into the model’s context at each step, and in what form, is called context engineering. Here the model would see the blocks it’s working on, the register’s columns and the project’s numbering rules, and little else. Each value it returns points to a block, so its citation leads to an exact place on the page without relying on the model to quote its source. Documents that haven’t changed aren’t read again. Different steps can use different models, and before a task starts, the platform checks which models the project’s documents may be sent to.

This one workflow needs most of what the platform is meant to provide, which is why it’s a good test of the design:

The register is built across many documents and has to stay up to date, so it’s a long-running task that must survive interruptions and pick up where it stopped.
The result is an application with its own data, interface and version history, which people review and correct.
Keeping it current means noticing when new documents arrive, without paying for a model call every few minutes to check.
It works with project documents, so permissions and approvals matter, and each model call has a cost that should be recorded.

Later workflows

After the register, we plan to add workflows that keep running in the background:

monitoring a folder, a website, a data feed or a running process, and acting when something meaningful changes;
monitoring an inbox, sorting what arrives and drafting replies for a person to approve;
longer email tasks, where you forward the agent some documents, it emails you a question when it’s unsure, and it carries on when you reply.

General assistants already offer scheduled tasks and connectors to other tools. In AI App Builder, each of these workflows is a persistent task with its own history, permissions and cost record, and its results can go into an application that people review. A monitor calls a model only when something has changed, and gives it just the context that change needs.

Where it stands

On the roadmap, Stage 1 (initial design and architecture) is current, and Stage 2 (the core agent framework and security) is next. We plan to keep the code private until the first release.

The agent loop comes from my own framework, doc-processing-toolkit, with its companion tool library, doc-processing-std-tools. AI App Builder builds a more general-purpose platform on top of both. The framework sits behind an internal interface that can be augmented or replaced later (more on that under open questions).

As in my other projects, coding agents do most of the implementation, and I set the direction, review and sign off (how that works is in Spec-first, test-first). The repository is being set up so agents can work from it directly, with repository knowledge as the system of record and architecture rules enforced mechanically: a short map file, a docs/ folder as the source of truth, and plans checked in beside the code. Later we’d like agents to pick up tasks themselves from a task board, raise pull requests and review each other’s work, along the lines OpenAI described in its harness-engineering write-up.

Tasks and applications

A language model is like a car’s engine. It provides the power, but on its own it doesn’t take anyone anywhere. The rest of the car decides how that power is used: the steering and brakes, the fuel system, the instruments that tell the driver what’s happening, and the safety systems that matter when something goes wrong. For an AI agent, the rest of the car is the harness: the tools it can use, the memory it keeps, the permissions it works within, the checks on its output, and the interface people use to steer it.

A more powerful engine in a car with weak brakes isn’t safer to drive, and an agent built on a stronger model isn’t more useful without a harness that can direct and check it. Most of our work on AI App Builder is on the harness. New engines arrive every few months, so the car has to take a more powerful one without being rebuilt: the provider layer lets us change models, and the tools, permissions, checks and cost limits stay in place whichever engine is fitted.

Two ideas sit at the centre of the design: persistent tasks and applications that agents build.

Proposed platform architecture. Design stage; the features shown are planned.View full size

A task is any piece of work the platform runs for you. A chat, a scheduled agent such as a weekly research summary, a monitor watching a folder and an email thread are all the same kind of object, so the same recording, checkpointing and resuming apply to all of them. Every fact about a task goes into one event log, which is the single source of truth.

Runs are designed to be durable. We plan to use DBOS, a library for durable workflows, on SQLite locally and Postgres in the cloud. Each agent step is a checkpoint, so an interrupted task resumes from the last completed step; the trade-off is that a step cut off part-way runs again from its start. When an unattended task needs a person’s approval, the request is stored as a record, so nothing has to stay running in memory while it waits.

An application is something the agent builds and keeps for you. It has its own data, interface and version history, and it can be edited and shared. The agent can also create storage for an application, on the user’s machine or in the cloud (on AWS, for example), within the limits its permissions allow. Examples range from simple tools, such as a research digest, to specialised ones like the document register. The design sorts every feature into one of three kinds:

a small core capability of the platform;
a thin, removable capability pack, such as email, voice or document registers;
an application the agent builds.

A CRM (customer relationship management tool), a study helper and a register tracker are meant to be applications. If the agent can’t build one of them, we’d rather improve the core than add bespoke platform code, because a special case would hide the gap. We use this check when reviewing proposed features, to stay true to what the product is for. The range of things it can do is meant to come from the applications agents build, while the platform itself stays small; each special case added to the platform would bring its own storage, interface and ways of failing for two people to maintain.

How an application’s interface is made

In generative UI, the model produces the interface as well as its content. There are roughly three kinds:

Pre-built components that the model chooses. Developers build the cards, forms and charts, and the model decides which to show and fills them with data. This is safe and consistent, but limited to what someone thought of in advance.
An interface the model describes. The model writes a structured description, such as a JSON layout of known components, and the application draws it. This is more flexible, but still limited to the set of components.
An interface the model writes. The model writes the interface itself, as HTML, CSS and JavaScript, and it runs in a sandbox. It can produce any interface, but it’s the hardest of the three to keep consistent, accessible and secure.

Most examples of the third kind, such as Google’s generative UI research and Claude’s artifacts, create an interface for a single answer. In AI App Builder an application is kept, versioned, shared and maintained by its agent over months, so its interface has to last.

AI App Builder is designed for the third kind, full generative UI. An application’s interface is written by the agent that builds it and can take whatever shape the task needs, instead of being assembled from a fixed set of parts. We’ll still provide about a dozen trusted components, such as tables, charts, forms, and document and citation viewers, because today’s models do better work with them and they make small edits cheap. They work like driver aids for today’s engines: they keep quality up while models improve. Each time a new model is released, an evaluation re-runs the same editing tasks and shows how much of that help is still needed. As with the rest of the harness, we’re building the car for more capable engines than today’s.

Full generative UI asks more of the platform than a renderer:

Each application runs in its own sandboxed origin.
Every version is kept, so a bad edit is one step from undone.
You can point at part of the interface and ask for a change.
Before a change is saved, the platform renders it at three screen sizes and checks it for errors, accessibility problems and calls the application isn’t allowed to make.

The design isolates applications from each other and from the platform. Each one gets its own origin on a separate domain and runs in a sandboxed iframe with a strict content security policy. It reaches the platform only through a message bridge modelled on MCP Apps. (MCP, the Model Context Protocol, is an open standard for connecting AI models to tools and data; MCP Apps extends it to interactive interfaces.) An application doesn’t hold API keys, it runs with the permissions of whoever is viewing it, and the paid calls it makes are metered. That’s more set-up per application than running everything in one page, but the aim is to keep a faulty or malicious application away from other applications and from the user’s credentials.

Versions are append-only, so undoing a change means pointing back to an earlier version. We considered leaving version history for later, but it’s a major feature that makes a real difference to the experience, so it’s in from the start.

Local, cloud and hybrid

The platform is designed to run in three ways: locally on the user’s machine, with local or hosted models; in the cloud; or hybrid, where one task uses both local and cloud resources. Different tasks can use different AI models.

The aim is to hand a task between environments where that makes sense, for example starting on a laptop and continuing in the cloud. In the design, one environment is in charge of a task at a time, and a handoff happens between agent steps, at the same checkpoints used for recovery, rather than in the middle of a step. The trade-off is that a handoff waits for the current step to finish. Processes move at checkpoints for now; moving arbitrary running processes is a later question.

Permissions and costs

The design puts a policy layer in front of tool calls. It checks each call by risk and context, then allows it, refuses it or asks a person for approval. It fails closed: if it can’t decide, the call doesn’t go ahead. Low-risk reads would go through, while risky actions would wait for a person. Other parts of the design:

file and code tools run in a sandbox, with outbound network access off by default;
the model sees placeholders instead of secrets, and an outbound proxy swaps in the real values, so that credentials stay out of the model’s context;
a budget guard reserves money before each call and settles it afterwards.

Stage 2 is meant to build these alongside the agent runtime, so that no agent runs without the sandbox and permission checks.

Cost tracking is part of the design from the start. It records one usage row per API request, priced when the request happens and not re-priced later, so a past run keeps the cost it was charged at. Because a retry is its own request, it gets its own row; The 6× gap describes what happened when timed-out requests and retries were left out of my cost log. Traces are for debugging, a ledger is for money, and a hash-chained log, where each entry includes a hash of the previous one, is for audit. At first, any users will bring their own model API keys.

Roadmap

The roadmap has five stages. The order is indicative and the stages can overlap; features such as voice and computer use may be developed alongside others. We’ve deliberately not put dates on it while the roadmap is still being tuned. The design work produced far more ideas for a first version than two people can build quickly, even with AI assistance, so the roadmap is ordered by scope.

Development roadmap. The sequence is indicative and may change as development progresses.View full size

Stage	Includes
1. Initial design and architecture (now)	Product requirements and core workflows; initial architecture and technical design; the existing agent framework and tooling; the development and validation approach
2. Core agent framework and security (next)	The agent runtime and model selection; sandboxed execution and tool integrations; permissions, approvals and usage controls; execution history and cost tracking
3. Persistent and hybrid execution	Persistent task execution and recovery; local, cloud and hybrid execution, with task handoffs; desktop integration and computer use; voice input and output, and live interaction
4. Application generation and editing	Persistent applications with their own data; application generation and visual editing; version history, rollback and sharing; multi-user collaboration
5. Automation and initial applications	Scheduled agents, triggers and monitoring; agents with dedicated email inboxes, including external correspondence; meeting assistance; the first applications, for research, information monitoring and engineering document management

A few decisions sit behind the order:

Security comes before persistence. No agent should run without the sandbox and permission checks, so they’re built in Stage 2, alongside the runtime.
Monitoring means an agent watching something, such as a folder, a website, an inbox or a data feed, and acting when it changes. It’s planned as a cost-saving design. A cheap, deterministic probe (ordinary code, with no model involved) would compute a fingerprint of whatever it watches, a short value that changes when the content does, and only a meaningful change would start an agent run. A folder watcher that called a model every minute would make 1,440 model calls a day. Some monitors will still need a model to judge whether a change matters, so the aim is to use one only where it’s needed.
Each agent will be able to have its own email address that you can cc, forward to or message. Correspondence with external contacts is planned to run within per-thread permissions and approvals, and the design treats their messages as data rather than instructions. Like the other policies, that has to be implemented and tested before it counts as protection against prompt injection, where text in a message tries to redirect the agent.
Meeting assistance covers capturing information, summaries and follow-up actions. We’re still working out the details of how it will work.
We plan to end each stage with a test on real work, such as an interrupted task resuming where it stopped, or a prompt injection failing to reach a credential.

Open questions

Some parts of the design will only be settled by building them:

how far handoffs between local and cloud execution can go in practice (passing a task between the user’s machine and the cloud while it’s under way);
which monitors can stay deterministic (checked by ordinary code, with no model call), and which will need a model;
how meeting assistance should work;
whether my own agent framework stays at the core. The design records when we’d replace it: if maintaining it costs more than it saves.

Feedback welcome

I’ll post updates as development progresses. If you work with engineering document registers, I’d like to hear which register tasks take up most of your time, since that’s what the first application needs to get right. For any feedback or thoughts, please get in touch through the contact page.

Keeping an LLM app current: what a 'model string update' really involved

Mon, 12 Oct 2026 00:00:00 GMT

I expected updating the models in my Japanese flashcard app to be a small maintenance task. Checking the integration exposed problems with retries, uploaded images and cost tracking. This is what I changed, and how I tested it.

The app is my Japanese Flashcard Generator (read about the project). It turns a photo of a study page into flashcards for Anki, a spaced-repetition flashcard app. I use it for my own Japanese study, and it’s open source. The update described here went live on 9 October 2026.

It’s a small Streamlit app (Streamlit is a Python framework for simple web apps), and each run has four steps:

A Gemini call checks that the photo is a usable study page.
LLMWhisperer runs OCR (optical character recognition) on it.
A second Gemini call cross-checks the OCR text against the image and returns the cards as JSON, which Pydantic validates. Built-in few-shot examples, sample pages that show the model what to produce, can guide this step.
The app exports a comma-separated text file for Anki and shows what the run cost.

The pipeline after the October 2026 update.View full size

When the model is retired under you

The app’s default and fallback model, gemini-2.0-flash, was shut down on 1 June 2026. Five of the eight models in the app’s dropdown had been shut down or retired, including the default. The other three still work, but only for some people: Google’s changelog entry of 18 September 2026 limits the Gemini 2.5 models to “users who have actively used them in the past”, while saying they aren’t deprecated.

One detail surprised me. In October, models.get("gemini-2.0-flash") still returned metadata, even though the model was missing from models.list and documented as shut down. A start-up check that asked “does this model exist?” would have passed. The deprecations page was the reliable source.

More than a model-string change

I started with a read-only review of the integration, which produced a written plan checked against Google’s docs, the installed SDK and offline HTTP probes. A coding agent then made the changes, and I reviewed and merged them. (I describe this way of working in Spec-first, test-first.)

The plan’s answer to “is it mainly the model strings?” was “No, but almost.” Changing only the model names would have left the app running Gemini 3 with:

a sampling setting the docs warn against;
no working retries;
a crash whenever a prompt was blocked;
a clean-up step that deleted every file in the user’s Gemini project.

Gemini 3 wants different settings

Google’s Gemini 3 guidance is to remove temperature, top_p and top_k, and it warns that setting temperature below 1.0 “may lead to unexpected behavior, such as looping or degraded performance”. My suitability check sent 0.1.

Gemini 3 also replaces thinking budgets with thinking levels, and the details vary by model. On 3.8 Flash, the minimal level returns an error, and sending a level and a budget together returns a 400. So the app now builds its request settings per model family. Gemini 3 models get no temperature and an explicit thinking_level on each call, so a change to Google’s default can’t quietly change the app’s behaviour. The legacy 2.5 models keep their old settings.

The retry that never ran

The app wrapped its Gemini call in a tenacity retry decorator, whose predicate looked for google.api_core exceptions and gRPC status codes. The current google-genai SDK raises its own errors, with HTTP status codes, so the predicate didn’t match them. The errors didn’t reach the decorator anyway. The function’s catch-all handler caught them and returned an error dictionary, so the decorator never saw an exception. And in the SDK version the app now uses, nothing retries unless you configure it to. I reproduced it: a 429 (rate-limit) response produced exactly one call.

The fix was to use the SDK’s own HttpRetryOptions, with four attempts on 408, 429 and 5xx responses and a 180-second timeout, and delete the old retry code. That also removed two direct dependencies. A test now drives the real SDK client through a mocked transport that returns a 429 and then a 200, and checks there were exactly two calls.

A clean-up that removed the wrong files

Images went to Gemini through its Files API. After each call, a finally: block listed every file in the API key’s project and deleted all of them.

On the public, bring-your-own-key version of the app, that’s the visitor’s project. In a reproduction, two unrelated files that were already in the project were deleted. Nothing in the app looked wrong while it happened, and files belonging to the visitor’s other apps would simply have disappeared.

The app no longer uses the Files API. Images now go inline, inside the request itself. In the tests, the fake client’s Files API raises an error if anything touches it, so the delete-all can’t quietly come back.

Sending images inline

Sending images inline raised the question of request size. Google’s image-understanding guide limits a request with inline images to 20 MB in total, prompt included. Its newer file-input guide allows inline payloads of up to 100 MB (50 MB for PDFs), after an increase Google announced in January 2026. I kept the app well under the stricter figure.

The old pre-processing wrote PNGs of about 6–10.5 MB each. A Vocabulary run sends the three built-in example images plus the study page, so a request came to about 27–37 MB. I switched to JPEG, with the longest edge at 3,072 px, quality 90 and transparency flattened onto white. One Vocabulary run (three example images plus one study page), measured before and after the change and recorded in the pull request, gave these results:

the request payload fell from 34.8 MB to 3.0 MB;
pre-processing time fell from 45.7 s to 2.1 s.

The pre-processing time is the app’s own image preparation before the request is sent. It doesn’t include the time Gemini takes to respond. The app has no request-size cap of its own: each image is now resized to about 2 MB or less. (The 20 MB upload limit in the next section is per file, and it’s there to protect the upload box.)

For the Gemini 3 models the app uses, the default image resolution already matches the “high” setting, so I didn’t need to set one.

One Vocabulary run, before and after the switch to resized JPEGs. The marker is the 20 MB figure in Google's image guide; pre-processing time covers the app's own image preparation, not Gemini's response.View full size

Hardening the public upload box

The live app is public, so anyone can upload a file to it. One risk is a decompression bomb, a small file that expands enormously when decoded: a 20 KB PNG can decode to about 483 MB, before any API call or key check runs. Uploads now go through size, format and pixel-count checks on the file header, before anything is decoded. The limits are:

JPEG and PNG only;
20 MB per file (MAX_UPLOAD_MB = 20) and 10 files per run;
120 megapixels for JPEGs, which are decoded at reduced scale, and 40 megapixels for PNGs.

In the smoke test the app rejected a bomb PNG, a GIF renamed .png, a 21 MB file and an 11-file batch, all before any API call.

The other problem was location data. The OCR step received the original upload, metadata and all, and phone photos often carry GPS coordinates; my own test photos did. My condition for fixing this was that image quality mustn’t change. The app now strips EXIF, XMP, comments and trailing data, and copies the compressed image data byte for byte, keeping the orientation and colour profile. The decoded pixels are identical.

A security review of the public repository after the update found no committed secrets and nothing rated high severity.

I also added a table of the generated cards above the Download button, and clicking Download no longer clears it.

Prices that change on a set date

At the time of writing (October 2026), Google’s pricing page lists Gemini 3.8 Flash and 3.6 Flash on the Standard tier at US$0.75 per million input tokens and US$3.75 per million output tokens until 31 December 2026. That price is scheduled to double on 1 January 2027, to US$1.50 and US$7.50. With one price per model, the app’s cost readout would show half the real cost of every run on those models from New Year’s Day. Gemini 3.5 Flash-Lite has no announced change.

Each entry in the app’s price registry can now carry price_changes with an effective_from date. Thinking tokens are billed at the output rate. I left out the -latest model aliases, because a cost readout can’t know which model an alias points to.

In the paid smoke test on 9 October 2026, a Vocabulary run (the three example pages plus one study page) cost about US$0.02 on 3.8 Flash and about US$0.0065 on 3.5 Flash-Lite, at October 2026 prices. Those are Gemini token charges only; the OCR ran on LLMWhisperer’s free tier. In October 2026 that tier gives 100 pages a day with no card required. Beyond it, LLMWhisperer charges per page according to the processing mode, from US$0.001 for digital PDFs up to US$0.015 for hard scans with forms and tables. A hand calculation matched the app’s readout exactly (US$0.001723 for one suitability call). The whole paid smoke test cost US$0.32, against a cap of US$2.

Testing with fakes

The update added 37 automated tests: 33 offline tests and 4 headless Streamlit UI tests. The offline tests use a fake Gemini client, a mocked HTTP layer, a fake OCR client and images built in memory, so they need no API keys. I chose fakes because the tests then make no real API calls, so they run quickly, cost nothing and give the same result every time. They’re in the repository’s tests folder. When the first version of the suite ran against the original code, 11 tests failed, which confirmed the bugs before any fix was written.

The suite has limits. It runs on Python 3.10 and 3.12, but it isn’t in CI yet. Because it uses fakes, it tests the plumbing, meaning the app’s own logic around the API calls. It can’t judge the quality of the cards. Pydantic confirms the structure of the output, but whether the Japanese is correct still depends on the live smoke test and on a person reading the cards.

What a second code review found

The second reviewer was GitHub Copilot’s review agent, which reviews pull requests on GitHub automatically. It raised five distinct findings, and each was reproduced before I decided what to do. Four were real bugs, and each fix came with a regression test that fails without it:

Transparent greyscale and palette PNGs turned black in the copy sent to Gemini.
The metadata strip kept JFIF thumbnails.
A prompt of exactly 200,000 tokens was billed at the higher price tier.
Empty OCR text passed the status check, so Gemini was asked for cards from nothing.

The fifth pointed out that the registry’s rate-limit section covered only the old models. I left it alone. Those limits depend on the account’s usage tier, so filling them in would have meant inventing numbers.

Updating both versions of the app

The repository has two long-lived branches. main uses my own API keys. app_public asks visitors for theirs, and it’s the branch Streamlit Community Cloud deploys. Both needed the same fixes, so one pull request (#11) went into main, and a second (#12) merged the same change set into app_public, resolving the conflicts once. Both branches now share the same commits.

One change set, two branches, two reviewed pull requests.View full size

Still open: the few-shot leak

With example images switched on, the legacy Gemini 2.5 Pro returned exactly the seven kanji from the built-in example, not the ones on the page. 2.5 Flash-Lite mixed the two sets.

A/B runs showed the same leak with the old pre-processing, and no leak with the examples switched off, so the bug predates this update. The Gemini 3 models weren’t affected in testing and seem much less susceptible; the 2.5-series issue isn’t fixed and is probably a model limitation. The options are to send the examples as separate conversation turns, or to switch them off for the 2.5 models. Because Google now limits the 2.5 models to users who have used them before, the app labels them “legacy”.

Looking back

I now treat the model list as a dependency with an expiry date. Preview models can be retired at two weeks’ notice, and the API’s metadata endpoint still described a model that had been shut down, so I check the deprecations page. Prices need dates too, because a single price per model is accurate only until the next change. The two worst bugs, a retry path that never ran and a clean-up that deleted files the app didn’t own, were both silent. Fake clients and mocked transports were what exposed them.

I also kept the app’s surface small: fewer dependencies, no Files API and no model aliases. Google now calls generateContent “legacy” but still fully supports it, so I kept it rather than switching APIs in the same change.

As of 18 October 2026, the hosted app still runs on Python 3.10, which reached end of life on 1 October 2026. The tests also pass on 3.12, and I plan to move the hosted app to it.

The 6× gap: reconciling an LLM cost log with the bill

Mon, 05 Oct 2026 00:00:00 GMT

In early April 2026, while checking a document-processing job for a client, I found that the provider’s bill was about six times higher than my application’s cost log. The investigation pointed to requests that timed out and were retried. Fixing the accounting took changes across several parts of the request pipeline.

That mattered because I use the cost log to price work, set budgets and tell a client what a job will cost.

The job

The job translated a long technical document, one page per request. It used OpenAI’s GPT-5.4 mini, and for a time it ran at the model’s highest reasoning effort (xhigh), so each page sent a large prompt and then reasoned for a long time. In hindsight, that setting may not have been the right choice for the task in the first place.

The requests went through my doc-processing-toolkit, a Python framework for AI agents that I designed, built and maintain. It wrapped each request in a retry decorator from the tenacity library: five attempts with exponential back-off. When a request succeeded, the toolkit priced the usage block in the response, using rates from its model registry (based on each provider’s published pricing), and wrote the cost to the job’s files.

Reconciling the log against the bill

I used the provider’s bill as the reference for reconciliation, and treated the log as the thing to check. The bill had limits of its own. The provider’s usage API returns usage in time buckets grouped by model, not per request, so it shows what a day cost but not which page caused it. I decided not to set up an admin key for it, because the job’s own logs had finer detail.

The logs showed a pattern straight away. Page after page had an APITimeoutError followed by a retry. On one page, the first attempt ran for about 30 minutes and timed out, the next four did the same, and the page failed. The same pattern repeated across many pages.

Those timeouts and retries, and the total on the bill, are what I observed. The link between them is an inference. On each timed-out attempt, the model would have read the whole prompt and could have reasoned for up to half an hour before the connection dropped, and that work would have been billed. The toolkit never received a response, so it had no usage block to price and recorded nothing. Then tenacity retried, and that attempt went unrecorded too. Because the bill only gives totals by model and time bucket, it can’t confirm what any single attempt cost.

One page: five timed-out attempts, and nothing recorded by the toolkit. The billed lane is inferred from the provider's totals, and attempt lengths are approximate.View full size

For this job I wrote a one-off reconciliation script that backed up the existing records and corrected them. It counted the timeouts on each page and spread the unexplained remainder (the bill minus the log) evenly across them, using an assumed split of the cost (one-third input, two-thirds output) to estimate token counts, and it marked the new figures as estimates.

The corrected figures are still approximate; the more important work was changing the toolkit so later jobs wouldn’t need a script like it.

What caused the gap

Three problems in the toolkit combined:

The requests were blocking. The OpenAI, Google and Fireworks calls didn’t stream, and Anthropic streamed on some paths but not all. In a blocking call, the response and its usage block arrive together at the end, or not at all, so a request that died at minute 29 left nothing to count.
Retries weren’t logged. The retry decorator discarded each failed attempt, so the toolkit could only ever cost the last one.
Long reasoning runs kept hitting a 30-minute timeout, and three of the four provider clients had no explicit timeout set.

What I changed in April

I wrote a remediation plan and iterated on it with AI before changing any code, as I do for larger changes (the process is in Spec-first, test-first). It had three parts.

Streaming on all four providers. Every provider call now streams internally. Agent.run() callers get the same result object as before, and a new Agent.stream() exposes the events. Each chunk resets the HTTP read timer, which reduces idle timeouts. Even a stream that fails partway leaves behind the tokens it had already delivered. That is partial evidence rather than a complete usage report (a request can still time out before its first chunk arrives), but it’s far better than the blocking calls, which left nothing to count.
Explicit client timeouts, with a two-hour read timeout as a safety net behind streaming.
Retry-cost accounting. A RetryAccumulator hooks into tenacity’s before_sleep callback and records each failed attempt: its tokens, its cost, where the token counts came from (token_source), the error and the duration. The run’s cost includes those attempts, and retry_count and retry_cost show how much of the total they account for.

The plan also settled some decisions before any code was written. Streaming is always on, with no opt-out flag, because a targeted fix for one provider is easier to maintain than a global switch. Cost alerts were left to client applications, which can watch the stream’s per-iteration cost events and stop early.

The plan set the toolkit’s accounting policy for a failed attempt as well. Here it is with the “rejected before stream” rule that was added later:

# The toolkit's accounting policy for a failed attempt: (input tokens, output tokens) to bill.
def bill_failed_attempt(error, partial_output_tokens, input_estimate):
    if rejected_before_stream(error):   # an HTTP error response, or no connection was made
        return 0, 0                     # assumed not billed
    if partial_output_tokens > 0:       # generation had started
        return input_estimate, partial_output_tokens
    if is_billable_error(error):        # e.g. a timeout before any chunk arrived
        return input_estimate, 0        # assume the prompt was read and billed
    return 0, 0

These are assumptions about how providers bill, not proof of what a given provider charged. A request rejected with an HTTP error response, or one that never connected, is assumed to cost nothing. A stream that had started is charged its estimated input plus the output tokens it delivered. An error the toolkit treats as billable, such as a timeout before the first chunk, is charged the estimated input only. Figures built from an estimate are flagged as estimates, as described below.

The streaming branch merged on 18 April, after a first phase that wrote acceptance criteria and placeholder tests before any implementation code. Two follow-up fixes came within a week. Five days after the merge, I found that each provider’s top-level error handler returned zero tokens and zero cost, which threw away whatever the stream consumer had counted. A Claude stream of about four minutes that closed early had been billed by Anthropic and recorded by the toolkit as free. The first of the two fixes made the error handlers return what the stream consumer had counted. The next day I found that the accumulator and the hook existed, and every provider function accepted an accumulator, but no caller ever created one. Retry accounting was effectively switched off, and cost still covered only the successful attempt.

These April changes, the streaming work and its two follow-up fixes, are the ones that relate to the original 6× discrepancy.

Later accounting defects

Between August and October I found more defects in how the toolkit records cost. They were real errors, but none of them was shown to be part of the April gap. They fall into three groups.

Attempts the toolkit didn’t see or didn’t count

The OpenAI and Anthropic SDKs retry failed requests themselves by default, which I found months after the April work. Each of tenacity’s five attempts could therefore hide up to two SDK retries: in the worst case, 15 requests, of which the toolkit saw 5. Release 0.4.0 builds the SDK clients with max_retries=0, and tenacity now also retries the statuses the SDKs used to handle. The worst case went from 15 requests, 5 of them visible, to 5 requests, all of them visible.

In October, 0.5.2 fixed a new form of the April failure: a truncated OpenAI reply was re-sent unchanged five times, the run failed, and cost read 0.0 for five billed requests.

Failed attempts priced wrongly

In September, a documentation review found that retry_cost was always 0.0. The hook recorded each failed attempt’s tokens but copied its cost from a field that every provider set to zero for a broken stream. Release 0.4.0 prices each attempt at registry rates, at the served tier and with the cached share, and flags it as estimated when its input count is the pre-flight estimate.

The same release fixed the opposite error. A request the provider rejected before streaming anything had been charged its estimated input; it now costs zero and is labelled rejected_before_stream. Over-reporting cost is an accounting bug too. Earlier, release 0.3.0 had fixed a report total that counted retries twice, adding retry cost on top of a cost that already included it.

Prices and usage that didn’t match the provider’s billing

In August, the toolkit started keeping a cost record for each request. A run’s cost is now the sum of its requests’ costs, because tiered pricing isn’t linear and summed tokens can cross a price threshold that no single request crossed. A later fix applied the same rule to failed attempts, where pricing two completed 150k- and 160k-token requests as one had doubled their cost.

Other mismatches were in the details. Streamed Gemini priority requests were priced at the standard rate and flagged as downgraded, because the toolkit read the served tier from a header that the streaming endpoint never sends. Anthropic cache writes were priced at the cache lifetime (TTL) requested, not the TTLs actually served. Release 0.5.2 changed Claude refusals to match how Anthropic bills them, since some refusal categories are billed and others aren’t. Then 0.5.4 started pricing the thinking tokens on Gemini image calls, which had never been counted; in the release check they were about an eighth of the call’s cost.

Where cost data leaked on its way through the toolkit, and when each leak was fixed. Only the April fixes relate to the original incident; the others were found later.View full size

Testing cost tracking

Over those six months (April to October 2026), I came to treat cost tracking as a feature with its own specification and tests.

The tests need to use the data shapes providers actually return. In one Google bug, the toolkit built its token-breakdown keys by calling str() on an enum. That produced "MediaModality.AUDIO" where the pricing function looked for "AUDIO", so it found no audio tokens, and audio input on Google tool-calling runs was priced at the text rate. The pricing engine’s own tests stayed green the whole time, because they called it with hand-written keys that the provider code never produced.

Golden files freeze what goes on the wire. What the toolkit sends decides what the provider charges for: the service tier, the cache controls and the thinking settings. The toolkit keeps more than 100 golden files of request bodies. Each is produced by running a real Agent session against a client that records the exact request body, with no network involved. If a commit changes a body that was meant to stay the same, the test fails with a diff. Keeping the golden files current is ongoing work, because providers change their APIs over time, and I’m looking at ways to refresh them regularly.

Closed lists keep the toolkit in step with its dependencies. One test compares the toolkit’s table of OpenAI stream failure codes with the SDK’s own list. When the SDK adds a code, the test fails until someone decides whether that code is retried or fails at once.

Small live probes check the assumptions that fixtures can’t. In one probe on 23 September 2026 (gpt-5-nano hosting gpt-image-2 at 1024×1024, one render per quality setting), OpenAI’s published per-quality price table for image generation was 39% too high at one quality setting and 40% too low at another. The toolkit now prices those calls from the usage the API reports, and falls back to the table only when no usage is reported, flagging the cost as estimated.

Estimates are labelled as estimates. Every run reports a token_source (api_response, stream_partial, rejected_before_stream or estimated) and a cost_estimated flag. When the accounting policy treats an attempt as billed but the toolkit couldn’t measure its usage, it reports a flagged estimate rather than $0.00. The flag says a figure is estimated; it doesn’t say how far off it might be, so reconciling against the bill is still worth doing. I’m considering more automated checks to make that routine. The toolkit docs describe the current cost model.

A checklist for tracking LLM costs

Reconcile your cost log against the provider’s bill regularly, by model and by day. The bill may only give totals, so per-request detail has to come from your own logs.
Log every attempt, not only the successful ones. A failed request can still be billed if the model did any work.
Count your retry layers. Your SDK, your HTTP client and your own code may all retry; keep one, and log it.
Stream long generations. Streaming reduces idle timeouts and leaves partial usage when a stream fails, though it doesn’t guarantee a complete record.
Price each request at the tier it was served, not from summed tokens or the tier you asked for.
Label estimates as estimates, test the cost code with the data shapes the provider actually returns, and keep those shapes up to date, because they change.

Spec-first, test-first: how I build AI systems with AI coding agents

Mon, 28 Sep 2026 00:00:00 GMT

I use coding agents across most of my current projects. I set the direction and make the architectural decisions: what I want to build, what it needs to do and what it doesn’t. Then I plan the work, steer the agents while they implement it, review the changes and decide what ships. This is the workflow I’ve settled on, including the parts that still need my attention.

Most of the examples come from the doc-processing-toolkit, a Python framework for AI agents that I designed, built and maintain. It started as plumbing for document-processing agents and is now a general framework for building agents of many kinds. It gives applications one interface to models from Anthropic, OpenAI, Google and Fireworks, and handles the parts most agents need, such as cost tracking, connecting to tools through MCP servers (MCP, the Model Context Protocol, is a standard way to connect agents to tools), and keeping long conversations within a model’s limits. Its documentation site is public. Other examples come from doc-processing-std-tools, a library of MCP servers designed mainly to supply tools to agents built with the toolkit, and from my Japanese flashcard app.

I mainly use Claude Code for implementation and OpenAI Codex for review and critique, though Codex writes some code too. GitHub Copilot’s cloud agent often does a second review of the pull requests.

Setting up the repository

I think of the repository instructions, tooling and checks around the agents as a harness. Its job is to give each agent session the context it needs, so the result doesn’t depend entirely on what I remember to put in a prompt.

Most of my repositories have a set of instruction files, CLAUDE.md, which Claude Code reads; the other agents, Codex and GitHub Copilot, read the same files as AGENTS.md. In the toolkit there’s a main guide, plus separate guides for the code’s structure, the list of supported models, the tests and the documentation. The main guide is written as rules. A few of them:

Model additions are JSON-only.
Costs are billed per request and by served tier.
Cached tokens are a subset of input tokens.
A behaviour change a deployed client could notice ships with a kill switch.
Never push unless explicitly asked.
Stage an explicit file list. Never `git add -A` or `git add .`

In plain terms: adding a model means adding an entry to a data file, with no new code. Costs are worked out for each request, at the price tier the provider actually used, and cached input is counted within the input total. A change that an application already using the toolkit could notice comes with a kill switch: a setting that turns the new behaviour off again. Agents don’t push to the shared repository unless asked, and they commit only the files they changed, named one by one.

The last two exist because several agents can work in the same copy of the repository at once. Each one commits only its own files, and treats files it didn’t touch as someone else’s work in progress. Some rules come from incidents: one bans a particular git shortcut because it once quietly undid a feature that had already been committed.

I also use git worktrees a lot. A worktree is an extra working copy of the same repository, on its own branch, so agents working in separate worktrees can run at the same time without touching each other’s files.

The guides also record things about the design that aren’t obvious from the code. For example, the code that runs an agent’s back-and-forth with the model exists in four near-identical copies, one for each provider. So the guide says a change to it has to be made in all four; otherwise it would work with one provider and quietly not with the others. I’ve decided to merge the four copies later, and the guide says not to start that as a side effect of another task.

There is also a documentation agent, set up for each of the three tools. It started in February 2026 as a GitHub Copilot agent and was ported to Claude Code and Codex in April. After a code change, it checks that the descriptions in the code still match it, tidies the formatting, rebuilds the documentation site until it builds without warnings, regenerates the READMEs and the documentation bundle for AI tools, and runs the unit tests.

Work I decide to postpone goes into a numbered technical-debt register, a list of known shortcuts and deferred work, so nothing deferred gets lost.

What the agents read, and the checks their work has to pass, in the toolkit's repository.View full size

Decisions go into the plan

Larger pieces of work start with a written plan. I write the first draft, then work through it with an agent, usually Claude Code, until it’s final. I review each version by writing my comments into the file itself, inside HTML comment tags (), and the agent revises the plan in response. That works better than describing the changes in a prompt: the agent can find every comment with a text search such as grep, and each comment sits beside the text it’s about, so the agent gets that context with it. For reviews of long content, where typing would take too long, I sometimes record my comments as audio and pass the recording to the agent, which transcribes it and acts on the comments. A plan sets out the problem and the evidence for it, the design, the parts of the code it changes, the risks, how it will be tested and how we’ll know it’s finished. It also lists the questions I need to answer before any code is written, and the agent has to bring me anything else it’s unsure of. Once the plan is final, the agent sets up a progress tracker, and the work is done in batches.

One example followed the cost problem described in The 6× gap. The fix changed how the toolkit receives a model’s response: as a stream, piece by piece, so a request that fails part-way still leaves a partial record of what it used. Before any code was written, the plan settled that streaming would always be on, with no setting to turn it off, and that alerts about high costs would be left to the applications built on the toolkit. It also came with written briefs for the sub-agents that did the work. A sub-agent is a separate agent session that the main session hands a task to.

I use sub-agents wherever it’s reasonable. Each one works in its own context (an agent session’s working memory), so the main agent, the one I’m working with, keeps its context for coordinating the work. I’ve also tried set-ups in which one agent is run over and over on the same task with no coordinator (the idea behind the “Ralph loop”), and built some custom GitHub Copilot agents to test it. I settled on one main agent coordinating as many sub-agents as the work needs. For large research tasks, Claude Code’s workflows have been very useful: the main agent writes the coordination of the other agents as a small program.

For one toolkit release (0.5.4), the plan went through several rounds of review, and my answers are kept in a table in the plan. For example, everything would ship as one release, the remaining security items would wait for the next one, and three corrections to the toolkit’s output would ship without kill switches.

The plan for my flashcard app listed my open questions, the things I’d already ruled out (so they wouldn’t be reopened), and fixed rules for the agent: don’t force-push (which can overwrite work already shared), don’t touch the file that holds the API keys, commit named files only, and stop at a spending limit. Rules like these can also be enforced with hooks: scripts that run automatically before or after the agent uses a tool. A hook that runs before an action can block it, so hooks can stop an agent reading certain files or taking certain actions.

I plan this way because an agent that meets an open question halfway through the work will usually pick a plausible answer and carry on, and that choice is easy to miss among many changed lines. Answering the questions first, and having the agent raise new ones, puts the decisions in writing, where I can check the result against them.

Tests that catch drift

Where it makes sense, I write tests before the code or alongside it, and I design them to fail when behaviour drifts: changes without anyone intending it.

For the streaming change, the first phase wrote 91 acceptance criteria (conditions the finished work has to meet) and 74 placeholder tests before any of the new code existed. Later phases replaced each placeholder with a real test.

Golden files are saved copies of exactly what the toolkit sends to a provider for a given model and set of options. The toolkit has 112 of them, holding more than 1,000 recorded requests, and any change to what it sends for an existing model makes a test fail and shows the difference. Before one larger change to the requests sent to Anthropic, a single commit saved the existing requests first, so the change could be checked against them. When the 0.5.4 release added a model, regenerating the golden files created nine new files and changed none of the existing ones. Keeping these recordings current is ongoing work, because providers change their APIs, and the recordings, along with the recorded responses the unit tests use, have to match what the providers currently accept and return.

Some tests name every model that has a particular capability. Adding Claude Haiku 5.5 made three of them fail until the new model was added to each list. That’s intended: the failure makes someone decide whether the new model belongs on each list. A similar test fails when a provider’s official Python library is at a new major version that hasn’t been tested with the toolkit, so an upgrade can’t slip through unchecked.

For a bug fix, I want a regression test that fails without the fix. If the test fails before the fix and passes after it, it really detects the bug, so it will catch the bug if it ever comes back. In the flashcard app’s review round (the full story is in Keeping an LLM app current), each of the five new tests failed without its fix and passed with it. On a doc-processing-std-tools release, an independent reviewer confirmed that eight new tests failed on the version before the fix.

The test guide records the unit-test result at each release. At doc-processing-toolkit 0.5.4, measured on 9 October 2026, the unit tests every change has to pass came to 9,834 passed and 8 skipped. A test can be marked to be skipped under certain conditions, for example when an optional dependency isn’t installed. The guide treats those 8 skips as the baseline and says a skip count well above it should be investigated, because it can mean tests are being quietly switched off.

These unit tests don’t contact any provider. They check the toolkit’s own logic, such as the requests it builds, the list of models and the cost calculations, using the golden files and recorded or made-up responses in place of the providers. How the providers actually respond is covered by the live tests, which make real API calls and are described below.

Implementation and review

The main agent (typically Claude Code for me) implements the plan in batches. Each batch follows the same order: implement, run the unit tests, run a quick live test against a real model, then commit. When the release is ready, the pull request goes through review.

The agents can call on each other. I’ve set up a skill for Claude Code that lets it talk to Codex, and a skill for Codex that lets it talk to Claude Code (a skill is a packaged set of instructions an agent loads when it needs them). Both can also ask GitHub Copilot’s review agent for a review, through Copilot’s command-line tool.

A separate agent (a Claude sub-agent, Codex or Copilot) reviews the pull request. Each finding is reproduced first, and then I decide whether to fix it, answer it or record it for later. On doc-processing-std-tools, Copilot found that a security check, which stops tools reaching private network addresses, relied on Python’s own list of reserved addresses, and on Python 3.10 that list treats some of them as public. The fix writes the official list into the code, so every Python version gives the same answer. Not every finding becomes a fix: one finding on the flashcard app was answered with a reason instead, because acting on it “would mean inventing numbers”.

I sometimes use Codex as a critic for Claude, and the other way round. In August I ran five Codex reviews of one release before it shipped: one of the whole release, and four of separate areas (the provider connections and cost calculations, the code that runs agents, new modules, and security). The fixes went into one commit; one of them restored charges for cached input that were lost when a request took a fallback route. The remaining findings were questions about the toolkit’s security assumptions, which went into the technical-debt register for me to decide.

On one doc-processing-std-tools release (0.6.1), three fresh agent sessions reviewed the changes one after another, without editing anything. The first found a way past the confirmation step that’s meant to stop an agent controlling the desktop without approval. The second checked the fix and found longer ways of writing the same input that still got through. The third reported the release ready to merge. Several rounds of review don’t prove the code is correct. Here, the gap in the first fix was found only because a second review tested it.

Documentation reviews sometimes find bugs in the code as well as out-of-date text. In September, one found that the toolkit’s figure for what retried requests cost had always been zero.

Checks against real APIs

The unit tests run offline. The live tests call the providers’ real APIs, through scripts that choose which models to test; testing every model on every run would cost 13 times as much.

I give live checks a budget for each project. The flashcard plan told the agent to stop and ask if the app’s cost readout went over US$2, or if any single run went over US$0.50. The app’s live API tests, including re-runs after the review fixes, cost US$0.32 in total. The doc-processing-toolkit 0.5.4 live checks cost about a dollar, itemised check by check in the release record. Those limits suit small projects like these. The part that carries over is setting a limit, and a point where the agent stops and asks, before it starts making paid calls.

Live checks find problems that mocks (made-up stand-ins for the provider APIs) can’t. The live checks for doc-processing-toolkit 0.5.4 found that Gemini’s image models return JPEG images, while the code saved every file as .png.

A check on 9 October 2026 sent a 60 MB clip inside a single request to Google’s gemini-nano-banana-2.1 model. Google’s file-input guide allows up to 100 MB sent this way (50 MB for PDFs), while the plan had assumed the 20 MB limit from older pages. The toolkit still switches to Google’s separate file-upload service above 20 MB, as a safety margin; its Multimedia docs describe how that works.

Another check, on claude-haiku-5-5 and also on 9 October, measured how the model counts tokens: the units providers bill by, and in which a model’s limits are measured. A 737,000-character prompt of repeated text, expected to come to about 101,000 tokens, came to 264,555, because the model counts that kind of repetitive text at about 2.8 characters per token. A prompt sized by its character count can therefore cost much more than planned, or not fit at all.

The loop for one release, from plan to tag.View full size

Implementation records

Each release ends with an implementation record: the decisions applied, the work on each item with its commits, test evidence, live checks with their cost, departures from the plan, issues found and follow-ups. Anything that wasn’t verified is marked “not verified”, so the record separates what was measured from what was assumed.

The records get corrected too. In one, a live check had counted a simulated key press as a success because the tool sending it reported success, although the key press had been ignored; after review, a corrected check watched for the key press itself. In another, a pull request description had overstated a check of the user interface, and the record notes the correction.

The final sign-off on the overall architecture and the code is mine.

What still needs my judgement

The agents write much of the code, and their reviews often raise design questions. The decisions are still mine. That starts with understanding what a client needs, setting the overall vision, and deciding what to prioritise and how things should work. With agents doing more of the implementation, that part of my role matters more than before. I don’t need to be able to explain every line of the code, but I do need to be able to explain how the system works and why I chose that design over the alternatives.

In the toolkit, the decisions cover the architecture: how each provider is connected, keeping the list of models as data, how each request’s cost is calculated, and where hooks and the shortening of long conversations plug in. They also cover scope: what goes into a release, what becomes a technical-debt item and what’s ruled out. For example, I put off moving to a new Gemini API, postponed two toolkit features (harness hooks and scheduled agents), and decided that streaming couldn’t be switched off.

Security trade-offs are mine as well. The toolkit’s MCP servers are designed to run on a machine the user trusts. I chose to document that limit, with a warning not to open the servers to untrusted clients, and left the product as it is. I also decide which findings block a release and which go into the register with an outline of the fix. A change that an application already using the toolkit could notice ships with a kill switch unless I waive it, and waivers are written into the changelog.

Some parts of the work haven’t become easier. Unclear requirements still depend on me noticing them before an agent settles them in a plausible but wrong direction, and a plan only covers the questions I think to ask. Judging review findings takes knowledge of the product, because a finding can be correct about the code and still be the wrong change to make. Realistic evaluation is hard to automate: the unit tests run against recorded and made-up responses, and the live checks are small, capped samples, so the token count and Google’s actual size limit only showed up against the real APIs. Fitting a change into the whole system needs care too, because a change that looks small can affect all four providers, applications already in use, or a security boundary.