Evals & testing

Four ways to know a routing decision was right: compare models side by side, evaluate them on your own prompts, shadow-test alternatives against live traffic, and get told when a model drifts.

Compare models

In the Playground, switch to Compare models, pick up to five and run one prompt on all of them at once. You get every answer side by side with a judge score, latency and cost, and badges for the best answer, the cheapest and the fastest. Calls run on your provider keys at the normal routing fee.

Eval sets from your own traffic

The Evaluations page builds a test set out of prompts you have actually sent.

1. Mine my traffic samples representative prompts from recent successful requests, spread across task clusters, so the set isn't all one kind of work.
2. Add your own prompts by hand, and switch any of them on or off.
3. Run an eval calls each chosen model on every active prompt and has the judge score the answers.

Results show per model — score, answers, average latency, cost, errors — and as a score-by-cluster grid, so you can see that a cheap model matches on extraction but not on reasoning.

A run costs the routing fee per call — prompts × models — and runs in the background; progress is shown live.

Shadow testing

Shadow testing answers "would a different model have done better?" without risking a single user-facing response.

After the real answer has already been returned, a background job re-runs the same prompt on the shadow model, judges both answers and records the winner. Latency for your users is unaffected. The report shows runs, shadow win rate, ties and the cost difference per pairing — a model that wins often and costs less is a candidate to promote.

Shadow calls use tokens on your provider keys but cost no credits, and a daily cap keeps them bounded.

Benchmark

The benchmark runs one prompt set through every strategy and reports them side by side, so a routing claim becomes a table you can publish:

• Each model on its own — your frontier model and your cheap one
• A random eligible model — the honest "no routing" control
• Cheapest-with-escalation — the strongest naive baseline
• LinuxAir at several price settings, for a cost-versus-quality curve

For every strategy you get quality, cost per 1,000 requests, cost versus baseline, quality delta in points, p50 and p95 latency, routing overhead and errors — plus the same table as Markdown to paste into a post or a customer email.

Run it from Evaluations → Benchmark, or on the server:

routiq.bench.run_cli  workspace='WS-0001'  source='evals'  lambdas=[0, 0.35, 0.8]

Use your eval set for judge-scored prompts, or the shared probe set for deterministic scoring with no judge involved.

Routing overhead

Every request records how long routing itself took — embedding the prompt and choosing a model — separately from the provider call. The Benchmark tab reports p50, p95 and embedding time so you can state an honest number. With an in-process embedding model this is typically single-digit milliseconds; with a hosted embedding endpoint it includes a network round trip.

Drift alerts

Providers sometimes change a model without changing its id. Every re-probe compares the new quality with the previous run; a move beyond the threshold flags the model in the interface and emails your alert address. Routing has already adapted by then — the alert tells you why your mix suddenly shifted.

Model health

Each call is recorded in a short rolling window per model. If the error rate crosses the threshold, that model is taken out of rotation for a cooldown instead of being retried on every request, and shows as Degraded with live latency on its card. It returns automatically.

Exports and metrics

• Usage CSV — the Requests page exports the full log: timestamps, model, tokens, cost, baseline, savings, credits, latency, cache hits, escalations, judge scores and feedback.
• Prometheus — GET /api/method/routiq.metrics.prometheus with a bearer token (set by your administrator) exposes requests, errors, cache hits, escalations, cost, savings, tokens, latency and per-model gauges.