Evals & testing

Four ways to know a routing decision was right: compare models side by side, evaluate them on your own prompts, shadow-test alternatives against live traffic, and get told when a model drifts.

Compare models

In the Playground, switch to Compare models, pick up to five and run one prompt on all of them at once. You get every answer side by side with a judge score, latency and cost, and badges for the best answer, the cheapest and the fastest. Calls run on your provider keys at the normal routing fee.

Eval sets from your own traffic

The Evaluations page builds a test set out of prompts you have actually sent.

1. Mine my traffic samples representative prompts from recent successful requests, spread across task clusters, so the set isn't all one kind of work.
2. Add your own prompts by hand, and switch any of them on or off.
3. Run an eval calls each chosen model on every active prompt and has the judge score the answers.

Results show per model — score, answers, average latency, cost, errors — and as a score-by-cluster grid, so you can see that a cheap model matches on extraction but not on reasoning.

A run costs the routing fee per call — prompts × models — and runs in the background; progress is shown live.

Shadow testing

Shadow testing answers "would a different model have done better?" without risking a single user-facing response.

After the real answer has already been returned, a background job re-runs the same prompt on the shadow model, judges both answers and records the winner. Latency for your users is unaffected. The report shows runs, shadow win rate, ties and the cost difference per pairing — a model that wins often and costs less is a candidate to promote.

Shadow calls use tokens on your provider keys but cost no credits, and a daily cap keeps them bounded.

Drift alerts

Providers sometimes change a model without changing its id. Every re-probe compares the new quality with the previous run; a move beyond the threshold flags the model in the interface and emails your alert address. Routing has already adapted by then — the alert tells you why your mix suddenly shifted.

Model health

Each call is recorded in a short rolling window per model. If the error rate crosses the threshold, that model is taken out of rotation for a cooldown instead of being retried on every request, and shows as Degraded with live latency on its card. It returns automatically.

Exports and metrics

• Usage CSV — the Requests page exports the full log: timestamps, model, tokens, cost, baseline, savings, credits, latency, cache hits, escalations, judge scores and feedback.
• Prometheus — GET /api/method/routiq.metrics.prometheus with a bearer token (set by your administrator) exposes requests, errors, cache hits, escalations, cost, savings, tokens, latency and per-model gauges.