Platform / Routing & Performance

Intelligent routing across every provider and model.

Fallback chains, canary splits, conditional routing, and semantic caching — all evaluated in under a millisecond before the request leaves your infrastructure. Built into the gateway, zero additional services required.

Read the docs Talk to us

Fallback Routing

Define ordered provider chains. When a provider returns an error or times out, ManyLayers immediately retries the next entry in the chain — transparent to the calling client. Configurable per retry budget and timeout threshold.

  • Health checks exclude degraded providers before routing begins
  • Retry budgets prevent cascading cost spikes on sustained outages
  • Fallback chains configurable per model alias or team

Fallback Routing — config excerpt

01 Health checks exclude degraded providers before routing begins
02 Retry budgets prevent cascading cost spikes on sustained outages
03 Fallback chains configurable per model alias or team

Canary Routing

Shift a percentage of live traffic to a new model or provider before committing to a full rollout. ManyLayers splits requests deterministically by weight, logs outcomes separately, and lets you ramp traffic based on observed quality.

  • Weight-based splitting at any granularity (1%–99%)
  • Separate cost and latency tracking per canary target
  • Promote or roll back canary with a single config change

Canary Routing — config excerpt

01 Weight-based splitting at any granularity (1%–99%)
02 Separate cost and latency tracking per canary target
03 Promote or roll back canary with a single config change

Conditional Routing

Route requests based on team ID, model alias, request metadata, or content properties. A single ManyLayers endpoint can simultaneously route engineering to GPT-4o, finance to Claude, and batch jobs to a local open-weight model.

  • Route by team, API key, model alias, or custom header
  • Combine with fallback chains — conditions nest cleanly
  • Audit log records which routing rule matched each request

Conditional Routing — config excerpt

01 Route by team, API key, model alias, or custom header
02 Combine with fallback chains — conditions nest cleanly
03 Audit log records which routing rule matched each request

Semantic Cache

Embedding-based cache intercepts requests semantically similar to a previously seen prompt. Rather than an exact string match, ManyLayers computes a cosine similarity against cached embeddings and returns a stored response when the threshold is met.

  • Configurable similarity threshold per cache namespace
  • Cache entries stored in your own Postgres instance
  • Typical hit rates of 20–40% on enterprise chat workloads

Semantic Cache — config excerpt

01 Configurable similarity threshold per cache namespace
02 Cache entries stored in your own Postgres instance
03 Typical hit rates of 20–40% on enterprise chat workloads
Strategy matrix

Routing strategies and their compatibility.

Strategy Multi-provider Canary Conditional Cache-compatible Latency impact
Fallback chain ✓ — — ✓ None on hit; retry on fail
Canary split ✓ ✓ — ✓ None — parallel not sequential
Conditional route ✓ ✓ ✓ ✓ Sub-millisecond evaluation
Semantic cache ✓ — — N/A Eliminates provider RTT on hit

Route intelligently across every provider and model.

Read the docs Talk to us