Platform / Model Performance

Measure and improve model quality, cost, and latency continuously.

Latency budgets that trigger automatic routing adjustments, semantic caching to eliminate redundant provider calls, canary analysis on live traffic, and ELO leaderboards so every model change is quantified, not guessed.

Book a demo Routing details

Latency Budgets

Define maximum acceptable latency thresholds per model alias, team, or request class. When a provider's rolling P95 exceeds the budget, the routing layer automatically deprioritises or excludes that provider until it recovers.

  • P50, P95, and P99 latency tracked per provider and model
  • Configurable thresholds trigger automatic routing adjustments
  • Latency SLO dashboards exportable to Grafana or Datadog

Latency Budgets — key behaviours

01 P50, P95, and P99 latency tracked per provider and model
02 Configurable thresholds trigger automatic routing adjustments
03 Latency SLO dashboards exportable to Grafana or Datadog

Semantic Caching

An embedding-based cache intercepts requests semantically similar to previously answered prompts. Rather than an exact string match, ManyLayers computes cosine similarity against cached prompt embeddings and returns stored responses above the configured threshold.

  • Similarity threshold configurable per cache namespace and team
  • Cache entries stored in your own Postgres instance — no shared cache
  • Hit rates of 20–40% typical on enterprise chat and support workloads

Semantic Caching — key behaviours

01 Similarity threshold configurable per cache namespace and team
02 Cache entries stored in your own Postgres instance — no shared cache
03 Hit rates of 20–40% typical on enterprise chat and support workloads

Canary Analysis

Route a percentage of live traffic to a new model or provider version. Canary targets emit separate cost, latency, and error-rate metrics. Promote when the canary meets your quality bar; roll back with a single config change.

  • Weight-based traffic splitting at any granularity from 1% to 99%
  • Separate Prometheus metrics stream per canary target
  • Automatic rollback rules configurable on error rate or latency breach

Canary Analysis — key behaviours

01 Weight-based traffic splitting at any granularity from 1% to 99%
02 Separate Prometheus metrics stream per canary target
03 Automatic rollback rules configurable on error rate or latency breach

ELO Evals & Leaderboards

Run automated evaluations on any model or prompt change. Pairwise comparisons are scored and ranked on an ELO leaderboard so quality changes are quantified, not guessed. Eval datasets are private and stay in your environment.

  • Eval suites run on any model alias — frontier or self-hosted
  • ELO scores update after each eval run, tracking trends over time
  • Eval results linked to the deployment or fine-tuning job that triggered them

ELO Evals & Leaderboards — key behaviours

01 Eval suites run on any model alias — frontier or self-hosted
02 ELO scores update after each eval run, tracking trends over time
03 Eval results linked to the deployment or fine-tuning job that triggered them
Capability matrix

Performance capabilities and their interactions.

Capability What is tracked Triggers routing Eval-compatible Cache-compatible
Latency budgets P50/P95/P99 ✓ — ✓
Semantic cache Hit rate / miss rate ✓ — N/A
Canary analysis Per-target P99 + error rate ✓ ✓ ✓
ELO evals ELO score + delta — N/A —

Start measuring and improving model performance today.

Book a demo See routing details