AI
Two model providers, one endpoint
A versioned Gemini model name retired and started returning 404 on a key that worked fine, for a model half the tutorials online still reference. What that outage taught me about aliases, fallback, and treating conversation history as untrusted input.
The chat widget on this site stopped answering. Not slowly, not intermittently: every request failed. The API key was valid, the quota was untouched, the code had not changed, and the model name in the request was one the documentation still listed and half the tutorials online still use.
It had been retired. Not removed from the docs, not flagged in the dashboard, just no longer available to new users, which turns out to include a key that had never called that particular version before. The error was a 404 on a model that, as far as every search result was concerned, existed.
That is a specific and searchable kind of pain, and it changed how I structure calls to model providers.
Prefer the rolling alias
Model providers publish both pinned versions and rolling aliases. Pinning is the instinct a careful engineer brings from package management, where an unpinned dependency is how you get a surprise on a Friday.
Model versions do not behave like package versions. A pinned package stays on your disk forever. A pinned model is a name on somebody else's infrastructure, and it has an expiry date whether or not you were told about it. Pinning does not protect you from change; it converts a gradual behaviour change into a sudden total outage, and it schedules that outage for a moment you will not be watching.
// Rolling alias maintained by Google, which avoids breaking again when
// versioned models retire
const GEMINI_MODEL = 'gemini-flash-latest'The trade is real and worth stating honestly. An alias can shift under you: quality, latency and formatting can all move without a deploy on your side, and if your prompts are finely tuned to one version's quirks, that is a genuine risk. Pin when a silent behaviour change would be worse than downtime, which is a real category, mostly involving structured output and evaluation baselines. For a conversational feature where the failure mode of a bad version is a slightly different tone, and the failure mode of a retired version is an endpoint that returns nothing, the alias is plainly the better bet.
Retrying a retired name never succeeds
Most resilience code is retry logic, and retry logic is built for transient failures: a timeout, a blip, a rate limit. It is exactly the wrong response to a model that no longer exists. You will retry three times with exponential backoff, take three times as long, and fail anyway.
What works is falling back to something structurally different:
if (model === 'gemini') {
try {
return await generateWithGemini(context, message, history)
} catch (error) {
// Gemini can fail on retired model names, quota, or key issues.
// Fall back to OpenAI so the visitor still gets an answer.
console.error('Gemini error, falling back to OpenAI:', error)
return generateWithOpenAI(context, message, history)
}
}
return generateWithOpenAI(context, message, history)The same context, the same history, the same question, handed to a different vendor. The visitor gets an answer. They do not know or care which model produced it, and nothing about the experience communicates that anything went wrong.
The catch is deliberately broad, and I would defend that here. Retired model, revoked key, exhausted quota, regional outage, malformed response: from the caller's point of view these are all the same event, which is this provider cannot answer right now. Distinguishing them buys nothing at the call site, because the action is identical.
Retry when the same call might work next time. Fall back when it will not. A lot of resilience code is retry logic applied to failures that are not transient.
One prompt, thin adapters
Fallback only works if both providers can genuinely be handed the same job. The moment you maintain a Gemini prompt and an OpenAI prompt, they drift, and your fallback path becomes the one nobody has tested since the day it was written.
So there is one system prompt and one context string, and each provider gets a thin adapter that maps the conversation into whatever shape its SDK wants. For OpenAI that is a flat message array with the system prompt at the front. For Gemini the system prompt is a separate systemInstruction field and the history is a different shape with different role names.
// Gemini requires history to start with a user turn
const firstUserIndex = history.findIndex(m => m.role === 'user')
const geminiHistory =
firstUserIndex === -1
? []
: history.slice(firstUserIndex).map(m => ({
role: m.role === 'assistant' ? ('model' as const) : ('user' as const),
parts: [{ text: m.content }],
}))That leading user turn requirement is the sort of thing you discover in production. A history that starts with an assistant message, which is exactly what you get if the widget opens with a greeting, is rejected outright. Slicing from the first user turn handles it without the rest of the code needing to know.
The general point: the differences between provider SDKs are irritating enough to need handling and nowhere near substantial enough to justify two prompts. Keep the expensive, tuned, hard-to-reproduce part shared, and let the adapters absorb the shape mismatch.
Conversation history is user input
This deserves more attention than it gets, because the architecture that causes it is extremely common.
A stateless chat endpoint has no memory, so the client sends the conversation history with every request. That is a reasonable design. It also means the history is not a record of what was said. It is a JSON array that arrived from a browser, and anyone can post whatever they like in it, including turns labelled assistant that the assistant never produced.
The consequence is that an attacker can put words in the model's mouth before it answers. Fabricate a previous assistant turn agreeing to something, or claiming a policy, or stating a price, and the model will treat it as its own prior statement and continue consistently with it. No prompt injection into the message is needed; the injection is in the history.
const MAX_HISTORY_MESSAGES = 12
const MAX_HISTORY_MESSAGE_LENGTH = 2000
function sanitizeHistory(history: unknown): HistoryMessage[] {
if (!Array.isArray(history)) return []
return history
.filter((m): m is HistoryMessage =>
!!m &&
typeof m === 'object' &&
(m.role === 'user' || m.role === 'assistant') &&
typeof m.content === 'string' &&
m.content.trim().length > 0)
.slice(-MAX_HISTORY_MESSAGES)
.map(m => ({ role: m.role, content: m.content.slice(0, MAX_HISTORY_MESSAGE_LENGTH) }))
}Validate the shape, allow only the two roles you expect, require non-empty strings, cap the count, and clamp the length of each. Note what the map does: it rebuilds each message from scratch with only the two fields that matter, rather than passing the caller's object through. Any extra properties smuggled alongside are discarded rather than forwarded to a provider SDK that might interpret them.
This does not solve the fundamental problem. You genuinely cannot verify a client-supplied history, and if that matters for your application, the history has to live server side under a session key. What sanitising does is bound the damage: no unbounded payloads, no unexpected roles, no thousand-turn conversation running up a bill, no arbitrary object graph reaching the SDK.
Bounding the window is a cost decision and a latency one
Those two constants are not only a security control. Twelve messages at up to two thousand characters each is roughly six thousand tokens of history in the worst case, on top of the system prompt and the retrieved context, on every single request.
Prompt tokens are charged and they are also latency: a longer prompt takes longer to process before the first token comes back, which the visitor experiences directly as the widget feeling slow. Capping the window is one line that improves cost and perceived speed together, and it costs you almost nothing, because the useful context in a support-style conversation is nearly always in the last few turns.
Where this is still weak
Two honest gaps, both of which I would fix before recommending this shape for anything with real traffic.
The fallback is one directional. If Gemini is chosen, OpenAI backs it up. If OpenAI is chosen, or if the fallback itself fails, the request fails. A proper implementation would try the other provider in both directions and only give up when both are down.
And there is no circuit breaker. When a model name has been retired, every request pays the full cost of a failed call before falling back, which doubles latency for every visitor until somebody notices and changes a constant. A counter that trips after a few consecutive failures and routes straight to the fallback for a few minutes would remove that, and it is not much code.
What to take from this
- Prefer the rolling alias unless a silent behaviour change would hurt more than an outage. A pinned model is a scheduled failure.
- Fall back to a different provider rather than retrying the same one. Retries cannot fix a name that no longer exists.
- Catch broadly at the provider boundary. Retired model, bad key, exhausted quota and an outage all mean the same thing to the caller.
- Keep one prompt and one context string. Put the SDK differences in thin adapters.
- Expect provider quirks like a required leading user turn, and absorb them in the adapter.
- Treat client-supplied history as untrusted input. Filter roles, require non-empty strings, cap count and length, and rebuild each object.
- Bound the conversation window. It is a cost control, a latency control and a safety control at once.
- Add a circuit breaker, so a persistent failure does not tax every request with a doomed call.
Underneath all of it is one product decision, not an engineering one: a visitor should never see an error because a vendor retired a model name. They did not choose the provider and they cannot act on the failure. Everything above is what it takes to make that promise, and most of it is a few dozen lines. The retrieval half of the same endpoint, and why short follow-ups break it, is a separate article.