Choosing a routing strategy

The ManyLayers Gateway supports four routing strategies, and choosing the wrong one for your workload is one of the most common configuration mistakes we see in new deployments. This guide walks through each strategy, when to use it, and the configuration patterns that make each one reliable in production.

Understanding what you’re optimizing for

Before configuring any routing strategy, answer three questions about each workload:

1. Is latency the primary concern? For user-facing chat, every 100ms of added latency is perceptible. For batch document processing, a 2-second response time is irrelevant.
2. Is cost the primary concern? Internal tooling with high volume benefits dramatically from routing to cheaper models. External-facing products may absorb the cost to ensure quality.
3. What happens when the primary provider is unavailable? Transient provider outages are a fact of life. Your routing strategy should specify fallback behavior explicitly.

Single routing

Single routing sends every request to one configured target. It is the right starting point for new deployments and for workloads where you have strong evidence that one model outperforms alternatives.

aliases:
  document-summarizer:
    routes:
      - target: anthropic/claude-haiku-3-5
        weight: 100

Single routing is not naïve — the Gateway still applies all guardrails, PII firewall, semantic cache checks, and audit logging. “Single” refers to the routing decision, not the feature set.

Use single routing when: you have a validated model choice, your workload is latency-sensitive, and you want predictable behavior. Add a fallback strategy once you’ve validated the primary route.

Fallback routing

Fallback routing defines an ordered list of targets. The Gateway attempts the first target, and if it returns an error (5xx, timeout, or rate limit), it automatically retries the next target in the list.

aliases:
  customer-support:
    routes:
      - target: openai/gpt-4o
        weight: 100
        fallback_to: anthropic/claude-sonnet-4-5
    fallback_timeout_ms: 8000

Fallback routing is the minimum resilience baseline for any production workload. Provider outages, rate limits, and transient errors are common enough that a manual retry — requiring a human to notice the failure and reconfigure — is operationally unacceptable.

The fallback_timeout_ms parameter controls how long the Gateway waits for the primary target before switching. For interactive workloads, set this to 3,000–5,000ms. For batch workloads, 15,000–30,000ms is more appropriate — you don’t want to fall back prematurely on a slow but working response.

Canary routing

Canary routing splits traffic across multiple targets by weight. Use it during model evaluation, version upgrades, or cost optimization experiments.

aliases:
  chat-production:
    routes:
      - target: openai/gpt-4o
        weight: 90
      - target: openai/gpt-4o-mini
        weight: 10

The primary use case is safe model upgrades: run the new model at 5–10% of traffic, observe latency and quality metrics, and ramp up only when the data supports it. The Gateway records the resolved target on every request in the audit log, so you can slice analytics by target and compare distributions.

Canary routing can be combined with fallback: each target in a canary split can have its own fallback. A request that resolves to the 10% target and then receives a provider error will fall back to the configured fallback for that target, not necessarily to the 90% target.

Conditional routing

Conditional routing evaluates request attributes at runtime to select a target. It supports the most complex logic but requires the most careful configuration.

aliases:
  smart-router:
    conditional:
      - condition: "metadata.priority == 'high'"
        target: openai/gpt-4o
      - condition: "prompt_tokens < 500"
        target: openai/gpt-4o-mini
      - condition: "metadata.team == 'legal'"
        target: self-hosted/llama-3-1-70b
      - default:
          target: anthropic/claude-haiku-3-5

Conditions are evaluated top-to-bottom. The first matching condition determines the target. Always include a default target — a request that matches no condition returns an error without one.

Common conditional routing use cases:

• Priority lanes: high-priority requests (flagged by the calling application via metadata) route to a faster, more capable model; low-priority background jobs route to a cheaper model.
• Prompt-size routing: short prompts route to cost-optimized models; long-context requests route to models with large context windows.
• Team-based routing: the legal team routes to a self-hosted model for data residency; other teams route to cloud providers.
• Compliance routing: requests tagged with a data classification label route to providers under a specific contractual framework.

Combining strategies

Strategies compose. A canary split can include conditional logic within each branch. Fallback can be layered on top of conditional routing. Build complexity incrementally — start with single routing, add fallback, then introduce canary splits for evaluation, and graduate to conditional routing when you have a validated routing logic that can’t be expressed by the simpler strategies.

What to measure after deploying a routing strategy

Regardless of which strategy you choose, track these metrics in the Analytics dashboard:

• Error rate by target — a spike in 5xx errors on a specific target is the earliest indicator of a provider issue.
• P95 latency by target — latency divergence across targets in a canary split reveals the performance tradeoff before it affects most users.
• Cost per request by target — validates that cost-optimization routing is actually reducing spend.
• Fallback rate — a high fallback rate means your primary target is unreliable and you should reconsider the primary choice.

Routing strategy is not a one-time configuration decision. Review it whenever you add a new provider, when a model is deprecated, or when your workload characteristics change significantly. The Gateway makes reconfiguration fast — changes propagate in under a second — so iteration cost is low.