ai-gateway is an OpenAI-compatible gateway I built that sits in front of several LLM providers. Its main promise is simple: if one provider goes down, requests fall back to another. It has unit tests for fallback, retries and a circuit breaker, and they all passed.

So I tested the promise directly: run real load through the gateway, break the primary provider in the middle, and measure what clients actually see.

The setup

• Two mock upstream providers, primary and secondary, each answering in a fixed 200 ms
• One model alias with priority fallback: try primary, then secondary
• Provider timeout_ms: 1000. Overall request deadline: 5000 ms
• Load: 10 concurrent clients for 60 s, split into 15 s healthy → 30 s primary broken → 15 s recovered
• Each scenario run with the circuit breaker on and off

Three ways to break the primary:

| Scenario | What happens to the primary | |---|---| | kill | process killed with SIGKILL, so connections are refused | | hang | accepts the request and never answers | | 5xx | answers immediately with HTTP 503 |

The result: crashes were fine, hangs were a disaster

The kill scenario looked perfect: 0% errors, fallback to the secondary in about 0.2 s.

The hang scenario with the breaker off did not:

| Run | Error rate during the outage | Requests completed in 30 s | |---|---|---| | hang, breaker off | 100% | 60 | | healthy baseline | 0% | ~720 |

Every single request failed, even though the secondary provider was completely healthy.

Finding the bug

A refused connection fails instantly, so the gateway moves on to the secondary right away. A hung provider never fails. It just holds the connection open. The gateway was supposed to give up on the primary after timeout_ms (1 s) and try the secondary with the remaining time.

It turned out timeout_ms was parsed from the config and never used. Each attempt got the whole remaining request deadline, so the hung primary burned all 5 seconds and fallback never got a turn. The unit tests didn't catch it because none of them simulated a provider that hangs instead of failing.

The fix is one small function: cap each attempt at the provider's timeout, but never past the overall deadline.

def _attempt_budget(self, provider: str, remaining: float) -> float:
    """Seconds one non-streaming attempt may take: the provider cap, never past the deadline."""
    cap = self.attempt_timeouts.get(provider)
    return remaining if cap is None else min(cap, remaining)

I also added a regression test for exactly this case: a hung provider must be cut off by its timeout and fall back.

After the fix

| Scenario | Breaker | Error rate | Outage latency p50 / p95 | |---|---|---|---| | hang | off | 0% | 1206 / 1211 ms | | hang | on | 0% | 206 / 1206 ms | | kill | on | 0% | 208 / 215 ms | | 5xx | on | 0% | 209 / 217 ms |

No request failed in any scenario. Two more things the numbers showed:

1. The timeout fixes errors; the circuit breaker fixes latency. With the breaker off, every request during a hang still pays about 1.2 s (the 1 s timeout, then the secondary). With it on, only 83 requests paid that cost before the circuit opened, and then the median went back to about 206 ms.

2. The breaker takes 5–10 seconds to open, not instantly. It trips on the failure ratio over a 10 s window, and that window still holds the successful requests from before the outage. It only opens once enough of them age out. Hangs are the slowest (9.7 s), because each failure is only recorded after its 1 s timeout. A shorter window or a "N failures in a row" trigger would open it faster, at the cost of more false alarms.

Caveats

These are single runs on a busy laptop (Intel i5-1240P), with the load generator on the same machine. They show behaviour, not production capacity. Still to measure: repeated runs with error bars, and the Redis-backed breaker shared across several gateway processes. The raw per-request results and the script are in the repo (bench/outage.py, bench/results/).

Takeaway

Unit tests checked that fallback works. Only a failure-injection benchmark checked that it works for the failure that actually happens. Crashes fail fast and are easy. Hangs quietly eat your whole time budget. If your system has a fallback path, break the primary in every way you can think of, under load, and look at what clients see.