Keeping an LLM app current: what a 'model string update' really involved

I expected updating the models in my Japanese flashcard app to be a small maintenance task. Checking the integration exposed problems with retries, uploaded images and cost tracking. This is what I changed, and how I tested it.

The app is my Japanese Flashcard Generator (read about the project). It turns a photo of a study page into flashcards for Anki, a spaced-repetition flashcard app. I use it for my own Japanese study, and it’s open source. The update described here went live on 9 October 2026.

It’s a small Streamlit app (Streamlit is a Python framework for simple web apps), and each run has four steps:

1. A Gemini call checks that the photo is a usable study page.
2. LLMWhisperer runs OCR (optical character recognition) on it.
3. A second Gemini call cross-checks the OCR text against the image and returns the cards as JSON, which Pydantic validates. Built-in few-shot examples, sample pages that show the model what to produce, can guide this step.
4. The app exports a comma-separated text file for Anki and shows what the run cost.

When the model is retired under you

The app’s default and fallback model, gemini-2.0-flash, was shut down on 1 June 2026. Five of the eight models in the app’s dropdown had been shut down or retired, including the default. The other three still work, but only for some people: Google’s changelog entry of 18 September 2026 limits the Gemini 2.5 models to “users who have actively used them in the past”, while saying they aren’t deprecated.

One detail surprised me. In October, models.get("gemini-2.0-flash") still returned metadata, even though the model was missing from models.list and documented as shut down. A start-up check that asked “does this model exist?” would have passed. The deprecations page was the reliable source.

More than a model-string change

I started with a read-only review of the integration, which produced a written plan checked against Google’s docs, the installed SDK and offline HTTP probes. A coding agent then made the changes, and I reviewed and merged them. (I describe this way of working in Spec-first, test-first.)

The plan’s answer to “is it mainly the model strings?” was “No, but almost.” Changing only the model names would have left the app running Gemini 3 with:

• a sampling setting the docs warn against;
• no working retries;
• a crash whenever a prompt was blocked;
• a clean-up step that deleted every file in the user’s Gemini project.

Gemini 3 wants different settings

Google’s Gemini 3 guidance is to remove temperature, top_p and top_k, and it warns that setting temperature below 1.0 “may lead to unexpected behavior, such as looping or degraded performance”. My suitability check sent 0.1.

Gemini 3 also replaces thinking budgets with thinking levels, and the details vary by model. On 3.8 Flash, the minimal level returns an error, and sending a level and a budget together returns a 400. So the app now builds its request settings per model family. Gemini 3 models get no temperature and an explicit thinking_level on each call, so a change to Google’s default can’t quietly change the app’s behaviour. The legacy 2.5 models keep their old settings.

The retry that never ran

The app wrapped its Gemini call in a tenacity retry decorator, whose predicate looked for google.api_core exceptions and gRPC status codes. The current google-genai SDK raises its own errors, with HTTP status codes, so the predicate didn’t match them. The errors didn’t reach the decorator anyway. The function’s catch-all handler caught them and returned an error dictionary, so the decorator never saw an exception. And in the SDK version the app now uses, nothing retries unless you configure it to. I reproduced it: a 429 (rate-limit) response produced exactly one call.

The fix was to use the SDK’s own HttpRetryOptions, with four attempts on 408, 429 and 5xx responses and a 180-second timeout, and delete the old retry code. That also removed two direct dependencies. A test now drives the real SDK client through a mocked transport that returns a 429 and then a 200, and checks there were exactly two calls.

A clean-up that removed the wrong files

Images went to Gemini through its Files API. After each call, a finally: block listed every file in the API key’s project and deleted all of them.

On the public, bring-your-own-key version of the app, that’s the visitor’s project. In a reproduction, two unrelated files that were already in the project were deleted. Nothing in the app looked wrong while it happened, and files belonging to the visitor’s other apps would simply have disappeared.

The app no longer uses the Files API. Images now go inline, inside the request itself. In the tests, the fake client’s Files API raises an error if anything touches it, so the delete-all can’t quietly come back.

Sending images inline

Sending images inline raised the question of request size. Google’s image-understanding guide limits a request with inline images to 20 MB in total, prompt included. Its newer file-input guide allows inline payloads of up to 100 MB (50 MB for PDFs), after an increase Google announced in January 2026. I kept the app well under the stricter figure.

The old pre-processing wrote PNGs of about 6–10.5 MB each. A Vocabulary run sends the three built-in example images plus the study page, so a request came to about 27–37 MB. I switched to JPEG, with the longest edge at 3,072 px, quality 90 and transparency flattened onto white. One Vocabulary run (three example images plus one study page), measured before and after the change and recorded in the pull request, gave these results:

• the request payload fell from 34.8 MB to 3.0 MB;
• pre-processing time fell from 45.7 s to 2.1 s.

The pre-processing time is the app’s own image preparation before the request is sent. It doesn’t include the time Gemini takes to respond. The app has no request-size cap of its own: each image is now resized to about 2 MB or less. (The 20 MB upload limit in the next section is per file, and it’s there to protect the upload box.)

For the Gemini 3 models the app uses, the default image resolution already matches the “high” setting, so I didn’t need to set one.

Hardening the public upload box

The live app is public, so anyone can upload a file to it. One risk is a decompression bomb, a small file that expands enormously when decoded: a 20 KB PNG can decode to about 483 MB, before any API call or key check runs. Uploads now go through size, format and pixel-count checks on the file header, before anything is decoded. The limits are:

• JPEG and PNG only;
• 20 MB per file (MAX_UPLOAD_MB = 20) and 10 files per run;
• 120 megapixels for JPEGs, which are decoded at reduced scale, and 40 megapixels for PNGs.

In the smoke test the app rejected a bomb PNG, a GIF renamed .png, a 21 MB file and an 11-file batch, all before any API call.

The other problem was location data. The OCR step received the original upload, metadata and all, and phone photos often carry GPS coordinates; my own test photos did. My condition for fixing this was that image quality mustn’t change. The app now strips EXIF, XMP, comments and trailing data, and copies the compressed image data byte for byte, keeping the orientation and colour profile. The decoded pixels are identical.

A security review of the public repository after the update found no committed secrets and nothing rated high severity.

I also added a table of the generated cards above the Download button, and clicking Download no longer clears it.

Prices that change on a set date

At the time of writing (October 2026), Google’s pricing page lists Gemini 3.8 Flash and 3.6 Flash on the Standard tier at US$0.75 per million input tokens and US$3.75 per million output tokens until 31 December 2026. That price is scheduled to double on 1 January 2027, to US$1.50 and US$7.50. With one price per model, the app’s cost readout would show half the real cost of every run on those models from New Year’s Day. Gemini 3.5 Flash-Lite has no announced change.

Each entry in the app’s price registry can now carry price_changes with an effective_from date. Thinking tokens are billed at the output rate. I left out the -latest model aliases, because a cost readout can’t know which model an alias points to.

In the paid smoke test on 9 October 2026, a Vocabulary run (the three example pages plus one study page) cost about US$0.02 on 3.8 Flash and about US$0.0065 on 3.5 Flash-Lite, at October 2026 prices. Those are Gemini token charges only; the OCR ran on LLMWhisperer’s free tier. In October 2026 that tier gives 100 pages a day with no card required. Beyond it, LLMWhisperer charges per page according to the processing mode, from US$0.001 for digital PDFs up to US$0.015 for hard scans with forms and tables. A hand calculation matched the app’s readout exactly (US$0.001723 for one suitability call). The whole paid smoke test cost US$0.32, against a cap of US$2.

Testing with fakes

The update added 37 automated tests: 33 offline tests and 4 headless Streamlit UI tests. The offline tests use a fake Gemini client, a mocked HTTP layer, a fake OCR client and images built in memory, so they need no API keys. I chose fakes because the tests then make no real API calls, so they run quickly, cost nothing and give the same result every time. They’re in the repository’s tests folder. When the first version of the suite ran against the original code, 11 tests failed, which confirmed the bugs before any fix was written.

The suite has limits. It runs on Python 3.10 and 3.12, but it isn’t in CI yet. Because it uses fakes, it tests the plumbing, meaning the app’s own logic around the API calls. It can’t judge the quality of the cards. Pydantic confirms the structure of the output, but whether the Japanese is correct still depends on the live smoke test and on a person reading the cards.

What a second code review found

The second reviewer was GitHub Copilot’s review agent, which reviews pull requests on GitHub automatically. It raised five distinct findings, and each was reproduced before I decided what to do. Four were real bugs, and each fix came with a regression test that fails without it:

• Transparent greyscale and palette PNGs turned black in the copy sent to Gemini.
• The metadata strip kept JFIF thumbnails.
• A prompt of exactly 200,000 tokens was billed at the higher price tier.
• Empty OCR text passed the status check, so Gemini was asked for cards from nothing.

The fifth pointed out that the registry’s rate-limit section covered only the old models. I left it alone. Those limits depend on the account’s usage tier, so filling them in would have meant inventing numbers.

Updating both versions of the app

The repository has two long-lived branches. main uses my own API keys. app_public asks visitors for theirs, and it’s the branch Streamlit Community Cloud deploys. Both needed the same fixes, so one pull request (#11) went into main, and a second (#12) merged the same change set into app_public, resolving the conflicts once. Both branches now share the same commits.

Still open: the few-shot leak

With example images switched on, the legacy Gemini 2.5 Pro returned exactly the seven kanji from the built-in example, not the ones on the page. 2.5 Flash-Lite mixed the two sets.

A/B runs showed the same leak with the old pre-processing, and no leak with the examples switched off, so the bug predates this update. The Gemini 3 models weren’t affected in testing and seem much less susceptible; the 2.5-series issue isn’t fixed and is probably a model limitation. The options are to send the examples as separate conversation turns, or to switch them off for the 2.5 models. Because Google now limits the 2.5 models to users who have used them before, the app labels them “legacy”.

Looking back

I now treat the model list as a dependency with an expiry date. Preview models can be retired at two weeks’ notice, and the API’s metadata endpoint still described a model that had been shut down, so I check the deprecations page. Prices need dates too, because a single price per model is accurate only until the next change. The two worst bugs, a retry path that never ran and a clean-up that deleted files the app didn’t own, were both silent. Fake clients and mocked transports were what exposed them.

I also kept the app’s surface small: fewer dependencies, no Files API and no model aliases. Google now calls generateContent “legacy” but still fully supports it, so I kept it rather than switching APIs in the same change.

As of 18 October 2026, the hosted app still runs on Python 3.10, which reached end of life on 1 October 2026. The tests also pass on 3.12, and I plan to move the hosted app to it.