Every panel this product generates, every synthesis, every trade-off test, every in-character answer, comes from a call to a language model API. For a long time that meant one provider, one API key, and an unspoken bet that it would stay up. It didn't, and the way it failed taught us more than the uptime itself would have.
The failure wasn't the provider being down in any obvious way. It was a credit-exhausted API key returning an error explicitly marked isRetryable: true — technically true (retrying eventually succeeds, once someone tops up the balance), practically useless, because the AI SDK's own retry logic took that flag at face value and quietly retried the same doomed call, with backoff, before the failure ever reached anything that could actually do something about it.
“A retry against the same dead provider isn't resilience. It's just slower failure.
The fix has two parts, and they only work together. First, every generation call now goes through its own multi-provider, multi-model fallback chain — Gemini, then Groq, then OpenAI, then OpenRouter, each with several model tiers — so a failing provider doesn't fail the request, it just moves to the next candidate in line. Second, we turned off the SDK's own default retry-with-backoff on every one of those calls, because our own fallback loop already is the retry strategy; the SDK quietly retrying first was pure added latency stacked underneath it, invisible until we actually went looking.
Gemini sits first in that chain because it's the fastest and cheapest, which also makes it the one most worth protecting — a single Gemini key hitting its own rate limit used to mean falling all the way through to slower providers for a problem that was really just "one key is out of quota," not "every provider is down." So the chain now spreads load across multiple Gemini keys on the same model tier before it ever drops to a worse one: every key gets a shot at the best available quality first, and only degrades in quality once every key at that tier is genuinely exhausted.
None of this was visible until the logging caught up to it. The original error logs were freeform strings — readable one at a time, unreadable as a pattern across dozens of calls in a row. Every attempt now logs one structured line: which provider, which model, which key, how long it took, what happened.
“The number that mattered wasn't in the error message. It was in how many identical error messages there were.
What this doesn't fix: if every provider we've integrated is genuinely down at once, or every key across every provider is exhausted, there's still no answer to give. That's a real, if unlikely, failure mode, and no amount of fallback chaining removes it — it just makes the far more common failure (one key, one provider, temporarily unavailable) invisible to the person waiting on a response, instead of the whole product's problem.