Running evals your team will trust

Most AI teams have a version of the same problem: they run evals, get a number, and then argue about whether the number means anything. The eval methodology is informal, the dataset is stale, and the process lives in a notebook that one person maintains. When it’s time to decide whether to upgrade a model, no one trusts the output.

This guide walks through building an eval program in ManyLayers Workspace that your team will actually rely on for routing decisions.

Why evals fail and how to fix them

Evals fail for three reasons: the dataset doesn’t reflect actual traffic, the judging is inconsistent, and the process isn’t connected to any real decision.

ManyLayers Workspace addresses all three. The eval dataset is drawn directly from gateway traffic — real prompts from real users. Judging uses a combination of human reviewers and LLM-as-judge, with a structured rubric that produces consistent outcomes. And the eval pipeline connects directly to canary routing, so eval results have a clear consequence: a model with a strong ELO ranking gets promoted to higher traffic weight.

Step 1: Define what you’re evaluating

Before configuring anything, answer: what does “good” look like for this workload? The answer depends on the task:

• Factual Q&A: Is the answer correct? Does it cite a verifiable source?
• Code generation: Does the code run? Does it follow the style guide?
• Summarization: Does the summary capture the key points? Is it an appropriate length?
• Classification: Does the label match the ground truth?

Write these criteria as a rubric — a structured set of dimensions with clear pass/fail or scored criteria. A rubric is the foundation of consistent judging.

Step 2: Build your golden dataset

A golden dataset is a curated set of prompts with known-good responses. It serves as your ground truth for regression testing: when you evaluate a new model, you compare its outputs against the golden answers.

In ManyLayers Workspace, go to Evals → Datasets → New Dataset. You have three ways to populate it:

1. Manual curation: write representative prompts and ideal responses from scratch. Slow but high-quality. Start with 50–100 examples.
2. Traffic sampling: sample from gateway logs. Workspace can surface a random sample of past requests for review — you approve or edit each one to add it to the dataset. This is the fastest way to build a dataset that reflects real usage.
3. Import: upload a JSONL file with {"prompt": "...", "ideal_response": "..."} records.

Maintain the golden dataset as a living document. Add new examples when you encounter prompt patterns that weren’t well-represented, and retire examples that test capabilities no longer relevant to your deployment.

Step 3: Configure the eval pipeline

The eval pipeline orchestrates how prompts are sent to models, how responses are judged, and how ELO scores are updated.

Navigate to Evals → Pipelines → New Pipeline and configure:

pipeline:
  name: chat-model-eval
  dataset: golden-support-prompts
  models:
    - openai/gpt-4o
    - anthropic/claude-sonnet-4-5
    - self-hosted/llama-3-1-70b
  judging:
    mode: hybrid
    llm_judge: openai/gpt-4o
    rubric: support-accuracy-rubric
    human_review_sample_rate: 0.15
  scoring:
    algorithm: elo
    k_factor: 32
    initial_rating: 1000

The hybrid judging mode uses the LLM judge for most comparisons and routes 15% of comparisons to human reviewers. This keeps the pipeline scalable while using human judgment to calibrate and validate the LLM judge’s decisions.

Step 4: Set up the rubric

A rubric in ManyLayers Workspace is a structured template that the LLM judge uses to evaluate comparisons. Navigate to Evals → Rubrics → New Rubric. A good rubric for a support assistant might look like:

You are evaluating two AI assistant responses to a customer support question. The customer’s question is: {prompt}

Response A: {response_a} Response B: {response_b}

Evaluate on these dimensions:

1. Accuracy: Which response is more factually correct?
2. Completeness: Which response more fully addresses the question?
3. Clarity: Which response is easier to understand?
4. Appropriate scope: Which response avoids unnecessary information?

Select A or B, and provide one sentence explaining your primary reason.

Test the rubric on 20–30 known comparisons where you already know the preferred answer. If the judge agrees with your ground truth less than 80% of the time, revise the rubric before scaling up.

Step 5: Connect evals to canary routing

This is what makes evals actionable. In Evals → Pipelines → [Pipeline] → Routing Integration, connect the pipeline to a gateway alias. When the ELO ranking reaches statistical significance (configurable threshold, typically after 150+ comparisons), Workspace will flag the top-ranked model for promotion.

Configure the promotion workflow:

routing_integration:
  alias: chat-production
  auto_promote: false        # require human approval
  promote_threshold: 120     # ELO point advantage over current production model
  canary_start_weight: 10    # start promoted model at 10% traffic
  shadow_eval: true          # continue collecting evals during canary phase

Setting auto_promote: false keeps a human in the loop. The platform sends a notification when a model qualifies for promotion; a platform admin reviews the ELO ranking and approves the canary rollout. This maintains human oversight while removing the friction of manual model research.

Step 6: Stratify by task type

A single ELO ranking masks important variation. A model that dominates at summarization may underperform at structured data extraction. Create separate pipelines for each distinct task type your deployment handles.

In practice, most teams need two to four pipelines: one per major use case. Tag gateway requests with a task type (via request metadata) so you can filter the golden dataset and eval comparisons by task. This gives you a ranking per task type, which maps directly to routing decisions for task-specific aliases.

Reading and acting on results

Check the Evals → Leaderboard dashboard weekly. Look for:

• Models with 100+ point ELO leads that haven’t been promoted yet — these are candidates for a canary.
• Rapid ELO changes in either direction — a model dropping quickly may indicate a provider-side update that degraded quality.
• Low comparison counts on new models — add them to a higher traffic weight in the eval split to accelerate ranking convergence.

The leaderboard is a decision support tool, not an autonomous decision maker. Use it to inform routing changes, but always review the underlying comparison samples before promoting a model to production. The samples are available in Evals → Comparisons, filterable by model pair and outcome.