Fallback routing: designing for provider outages

Provider outages are not edge cases. Every major LLM API provider — OpenAI, Anthropic, Google, Cohere — has experienced multi-hour degradations in the past 18 months. If your AI features depend on a single provider, your features inherit that provider’s reliability ceiling. For user-facing applications, that ceiling is lower than most engineering teams realize until they’ve been paged at 2am for an outage they didn’t cause.

Fallback routing is the architectural answer. Instead of a hard dependency on a single provider, your gateway defines an ordered sequence of targets. The gateway attempts the first, and if it fails, transparently retries the next. Application code changes nothing.

The anatomy of an outage

Provider degradations come in three forms, and your fallback configuration needs to handle all three:

Hard errors — the provider returns 5xx responses immediately. These are easy to detect and respond to within milliseconds.

Soft errors — the provider accepts the request, throttles it, and returns a 429 rate-limit error after a delay. These waste time before the fallback triggers.

Timeout degradations — the provider is technically available but running slowly. Responses arrive, but P95 latency is 10–30 seconds instead of 1–3. These are the hardest to handle because the provider isn’t technically failing — it’s just slow.

Each error type requires different timeout configuration to handle effectively.

Configuring fallback routing in ManyLayers Gateway

Fallback routing is configured at the alias level. An alias is a named routing target that your applications reference; the gateway resolves the alias to a specific model at request time.

aliases:
  chat-production:
    routes:
      - target: openai/gpt-4o
        weight: 100
        on_error:
          fallback_to: anthropic/claude-sonnet-4-5
          retry_on: [429, 500, 502, 503, 504]
          timeout_ms: 6000
      - target: anthropic/claude-sonnet-4-5
        on_error:
          fallback_to: self-hosted/llama-3-1-70b
          timeout_ms: 10000

The timeout_ms on each target controls how long the gateway waits before treating a slow response as a timeout and triggering the fallback. For interactive chat, 6,000ms is an appropriate first-target timeout. For batch processing, 30,000ms may be more appropriate.

The retry_on list controls which HTTP status codes trigger a fallback. Include 429 — rate limit — in addition to 5xx errors. Providers frequently return 429 during degradations when their infrastructure is under stress, even if the request doesn’t exceed your account’s rate limit.

Handling timeouts: the hardest case

Timeout degradations are the most dangerous scenario. If your timeout_ms is set too high, users wait many seconds before the fallback kicks in. If it’s set too low, you generate unnecessary fallback traffic on providers that are healthy but momentarily slow.

The right approach is to set timeout_ms based on your P99 latency baseline, not your P50. If your primary provider typically returns P99 latency of 3,500ms, a 5,000ms timeout gives 1,500ms of buffer before triggering a fallback. During a degradation where P99 climbs to 12,000ms, the fallback triggers after 5 seconds — an acceptable user experience for most interactive applications.

Measure your provider’s P99 latency in the Analytics dashboard under Analytics → Latency → By Target over a trailing 30-day window. Use the 95th percentile of that P99 distribution as your timeout baseline.

Fallback chain depth

How many fallbacks are enough? For most production workloads, a two-level chain (primary + one fallback) covers 99%+ of outage scenarios. Provider-wide outages severe enough to take down two major providers simultaneously are vanishingly rare.

A three-level chain makes sense when:

• One of your fallback targets is a self-hosted model with potentially lower reliability than a cloud provider
• Your workload has latency headroom that makes a longer retry chain acceptable
• You’re routing to providers with overlapping infrastructure (e.g., multiple Azure-hosted providers could degrade simultaneously during an Azure region event)

aliases:
  resilient-chat:
    routes:
      - target: openai/gpt-4o
        on_error:
          fallback_to: anthropic/claude-sonnet-4-5
          timeout_ms: 5000
      - target: anthropic/claude-sonnet-4-5
        on_error:
          fallback_to: self-hosted/llama-3-1-70b
          timeout_ms: 8000
      - target: self-hosted/llama-3-1-70b
        timeout_ms: 15000

What to observe during a provider outage

When a provider degrades, the ManyLayers Gateway audit log and metrics endpoint record every fallback event with the original target, the error code, and the resolved fallback target. This lets you reconstruct the impact of an outage: which requests fell back, how long the degradation lasted, and whether the fallback delivered comparable quality.

In Analytics → Events, filter for event_type: fallback to see all fallback-triggered requests. During an active provider incident, the fallback rate on the affected target will spike — this is your early warning signal. Configure an alert in Platform Settings → Alerts to fire when the fallback rate on any alias exceeds 5% over a 5-minute window.

The application layer contract

Applications that use ManyLayers Gateway don’t need to know anything about fallback routing. They send requests to the gateway endpoint, reference an alias name, and receive a response. The resolved provider, the fallback chain, and any retry logic are invisible.

This separation of concerns is the key architectural benefit. When you add a new provider to your fallback chain, no application code changes. When a provider raises prices and you reprioritize your routing order, no application code changes. The gateway owns the routing logic; applications own the application logic.

Provider reliability is not your fault, but it is your problem. Fallback routing makes it a problem you’ve already solved before the next outage happens.