Routing & policy
How LinuxAir predicts quality, guarantees it within your tolerance, and chooses the best value — plus every setting that shapes the decision.
The routing space
A shared probe set of about 86 prompts across eleven categories — math, reasoning, code, extraction, JSON formatting, classification, factual recall, multilingual, instruction following, writing and summarisation — is embedded and grouped with k-means into K clusters (8 by default). The cluster centres form the routing space shared by every model.
Model fingerprints
When a model is added, it answers every probe once. Each answer is scored automatically — exact match, contains, numeric tolerance, regular expression, JSON keys, or an LLM judge for open-ended tasks. The model's error in cluster k is smoothed toward its overall error so that small clusters don't mislead:
Predicting quality for a prompt
The prompt is embedded and softly assigned to its nearest clusters:
Because some fingerprints rest on few measurements, LinuxAir also estimates uncertainty and routes on a pessimistic score:
The quality guard
Only models whose pessimistic quality is within the tolerance τ of the best model are eligible:
τ defaults to 0.005 + min(0.12, 0.06 · λ). On hard prompts — when even the best model's LCB is below 0.70 — τ is halved, so difficult work goes to stronger models.
Choosing the best value
- C(h) — estimated cost on a log scale across eligible models, in [0, 1]. Log scaling means a 2× price difference matters equally at cents or dollars.
- L(h) — median latency relative to the slowest eligible model, in [0, 1].
- Estimated cost uses the prompt's token count (characters ÷ 4) and the model's typical answer length measured during probing.
Models that are worse and more expensive than another candidate never win. Hard constraints — context window, max_cost, min_quality, allow and exclude lists — are applied before scoring.
Worked example
A structured-extraction prompt with the default Best value policy (λ = 0.35, τ = 2.6 pts):
| Model | Quality q | LCB | Est. cost | Result |
|---|---|---|---|---|
| Frontier model | 94.9% | 93.7% | $0.00410 | Eligible, lower value |
| Small fast model | 94.1% | 92.9% | $0.00025 | Chosen — 0.8 pts below the best, 16× cheaper |
| Open-weights 70B | 90.6% | 89.1% | $0.00034 | Not considered — 4.6 pts below the best |
Switch the same prompt to la/quality and τ shrinks to 0.5 pts, so the frontier model wins.
Modes and settings
| Setting | Where | Effect |
|---|---|---|
| Mode / price weight λ | Routing policy, API key, la.lambda, model alias | How eagerly to take a cheaper model among near-equals. |
| Quality tolerance τ | Routing policy (custom), la.tolerance | Max predicted quality given up. Overrides the λ-derived default. |
| Minimum quality | Routing policy, la.min_quality | Hard floor on predicted quality. |
| Speed preference μ | Routing policy, la.latency_weight | Prefer faster models among near-equals. |
| Answer checks | Routing policy, la.cascade | Judge and escalate weak answers. |
Precedence: per-request la fields → model alias → API key override → workspace policy.
Capability-aware routing
Right after probing, each model is tested for two things: can it call a tool, and can it honour a strict JSON response format? When a request sends tools or a JSON response_format, only models that passed are considered — if none do, the API says so instead of returning a broken answer. Detected capabilities show on the model card and can be overridden by hand.
Semantic cache
Before any provider call, the prompt's embedding is compared with recent prompts from your workspace. A match above the similarity threshold (default 97%), with the same model and generation settings and within the TTL, returns the earlier answer immediately — no tokens, no credits, flagged as cached in the response. Send la: {"cache": false} to bypass it.
Answer checks and escalation
After the chosen model answers, a judge model scores it (0–10) when any of these is true:
- the chosen model is more than 4 pts below the best predicted model (it was a trade-down);
- its predicted quality is under 55%;
- its uncertainty σ is above 0.10.
If the score is below 6/10, the request is retried once on the highest-quality model not yet tried, and the weak result is fed back as negative evidence. Checks are skipped for streaming and tool calls. Thresholds are configurable by the operator.
Failover and model health
If a provider returns an error, LinuxAir tries the next candidate in ranked order — up to two failovers by default. Failed requests don't consume credits.
A rolling window also tracks each model's error rate. If it crosses the threshold, that model is skipped entirely for a cooldown period rather than retried on every request, and appears as Degraded in the interface until it recovers.
Online learning
Each model keeps a live error per cluster, updated from feedback with an exponential moving average (α = 0.08), weighted by how strongly the prompt belonged to each cluster. The router blends live and probe error in proportion to evidence:
Signals come from thumbs up/down (dashboard or /v1/feedback), failed answer checks, implicit retries — the same prompt sent again within two minutes — and eval runs on your own prompts.
Getting sharp predictions
text-embedding-3-small) and a small, fast judge model in Routiq Settings, rebuild clusters, then re-probe your models. See Self-hosting.