Canary routing for LLM traffic: safe model upgrades without downtime

Upgrading a production LLM is nerve-wracking. A new model checkpoint might be faster and cheaper, but it can also silently regress on edge cases your evals didn’t cover. Canary routing lets you validate a new model against real traffic before flipping the switch for everyone.

The problem with big-bang model upgrades

Most teams do model upgrades the dangerous way: they test the new model in a staging environment, decide it looks good enough, update the API call to reference the new model identifier, and deploy. If something goes wrong, they roll back — which requires another deployment, another review cycle, and meanwhile production has been degraded for however long it took to notice.

The staging environment test has a fundamental limitation: it doesn’t reflect the full distribution of production traffic. Your staging tests covered the happy path and a handful of edge cases. Production traffic covers everything your users actually do, including edge cases you’ve never thought to test.

Canary routing bridges this gap. You run the new model against real production traffic at low volume — say, 5% — while the established model handles the remaining 95%. If quality metrics hold, you ramp up. If they don’t, you set the canary weight back to zero. No deployment required in either direction.

What canary routing means for LLM gateways

In traditional software deployments, a canary release sends a small percentage of requests to a new version while the rest continue hitting the stable version. The same principle applies to LLM traffic, but with an extra wrinkle: model outputs aren’t binary pass/fail. You need to watch latency, token costs, guardrail hit rates, and downstream user signals simultaneously.

ManyLayers Gateway implements weighted routing at the alias level. You define a named alias — say, chat-production — and split it across two or more backend model targets with percentage weights. The gateway resolves every incoming request against the alias at runtime, with no changes required on the client side.

aliases:
  chat-production:
    routes:
      - target: openai/gpt-4o
        weight: 95
      - target: openai/gpt-4o-mini
        weight: 5

Running a canary rollout

Step 1 — Establish a baseline. Before touching weights, run your eval suite against the current model and record P50/P95 latency, cost per 1,000 requests, and guardrail trigger rate. These become your rollback thresholds.

Step 2 — Start narrow. Set the new model to 5%. At meaningful request volumes (a few thousand per hour), five percent gives you statistical significance within 30–60 minutes for latency and cost metrics, while limiting user exposure if something is wrong.

Step 3 — Watch the right signals. ManyLayers Gateway emits structured JSON logs for every request, tagged with the resolved alias, target model, token counts, and latency. Pipe these into your observability stack and create dashboards split by the route_target field. If P95 latency on the canary bucket climbs above your baseline threshold, you have a clear rollback trigger.

Step 4 — Promote or roll back. Promoting is a one-line config change: raise the new model’s weight to 100 and remove the old target. Rolling back is identical — set the old model back to 100. Because the change is at the gateway alias level, no application code is touched, and the change propagates in under a second.

Ramp schedule: from 5% to 100%

A typical canary ramp for a model upgrade looks like this:

• Day 1: 5% canary. Watch for hard errors and catastrophic latency regression. Any issue here triggers immediate rollback.
• Day 3: 20% canary. At this weight, quality regressions on narrow prompt types become statistically visible. Review the eval pipeline results.
• Day 7: 50% canary. At parity, cost and latency differences are clearly measurable. Make the promote/rollback decision.
• Day 10: 100% if metrics hold.

Don’t compress this schedule unless you have very high traffic volumes (where statistical significance accrues faster) or time-sensitive business reasons. The value of a canary ramp is patience — it gives quality regressions time to surface.

Combining canary routing with evals

ManyLayers Gateway can forward canary-bucket requests to ManyLayers Workspace’s eval pipeline in parallel with serving the response. This lets you score outputs against your golden dataset continuously during the rollout, without adding latency for end users. If the eval pass rate drops below your configured threshold, a webhook can trigger an automatic weight reset.

Configure this in Evals → Pipelines → [Pipeline] → Routing Integration. Set shadow_eval: true and link the pipeline to the canary alias. The eval pipeline scores the canary responses asynchronously — no latency impact for users, and you get a continuous quality signal throughout the ramp.

Practical notes

• Sticky sessions: If your users expect session consistency (e.g., a multi-turn chat that must stay on the same model), use the sticky_by option to hash on user ID. The canary percentage still applies at the cohort level, not per request.
• Cost budgets: Set per-alias budget limits so an unexpectedly verbose new model can’t spike your bill before you notice.
• Audit log: Every resolved route is recorded in the immutable audit log with the alias, target, and weight snapshot at request time. This makes post-incident reconstruction straightforward.
• Minimum traffic requirements: Canary routing doesn’t work well for very low-volume aliases. At fewer than a few hundred requests per day, statistical significance takes weeks to accumulate. For low-traffic aliases, run offline evals against the full request archive instead.

Canary routing won’t eliminate all risk from model upgrades, but it turns a binary “big bang” cutover into a measurable, reversible ramp — which is exactly what production AI infrastructure needs.