Routing · menu
Core
Routing
A pinned model is always honored. When an application defers with model: "auto", Arbr decides which model serves the request, using only the rules and policies a human has enabled. It always explains why.
Routing modes
Routing has three modes, set globally in the dashboard:
| Mode | Behaviour |
|---|---|
| Off | Deferred requests go to the default model. No automated substitution. |
| Cost guardrail | Cheap task types go to a lighter target model. Everything else uses the default. |
| AI routing | A human-approved, AI-generated task→model policy decides, informed by a task classification of each request. |
The precedence ladder
Every request resolves top to bottom. The first rule that applies wins.
| Order | Stage | routingDecision |
|---|---|---|
| 1 | Budget enforcement. A breached cap can block (429) or downgrade to a lighter model. Outranks even an explicit pin. | budget |
| 2 | Explicit pin. A specific model is served as-is, skipping all policy. Works for unregistered pass-through models too. | explicit passthrough |
| 3a | Cache. An exact-match hit on a recent identical request. | cache |
| 3b | Human rules. A rule an operator enabled matches, by task type, application, or workflow. | rule |
| 3c | Automated routing. The cost guardrail or the AI policy chooses. | auto ai |
| 3d | Default. Nothing else applied. | passthrough |
| 4 | Fallback. The chosen provider errored, so Arbr retries another live one (scope configurable). | fallback |
The full routingDecision enum is explicit, passthrough, rule, auto, ai, cache, fallback, budget. Every served request carries the value that decided it, so nothing is a black box.
Task classification
To route by task, Arbr first classifies each deferred request. It records how it did so in classifiedBy:
| Value | Meaning |
|---|---|
| provided | The caller passed taskType explicitly. Always trusted. |
| keyword | A deterministic keyword heuristic matched the prompt. |
| ai | A short LLM call classified it, only under AI routing and only when the keyword pass is inconclusive. Cached by prompt. |
A cost guardrail moves these default cheap task types to a lighter model: classification, extraction, summarisation, translation, faq, and support response.
Difficulty-aware routing
Under AI routing, the classifier also scores difficulty from 1 to 10, so an easy request and a hard one with the same task type can route differently:
- Easy (≤3) → a cheaper model.
- Normal (4–7) → the default for that task.
- Hard (≥8) → a stronger model.
A confidence below 0.5 is ignored, so a low-confidence guess never overrides the base policy.
Explainability
Alongside routingDecision, Arbr logs a routingExplain object that captures the non-obvious "why": which rule matched, which policy source and base were used, the default scope, and any later override such as a fallback or a budget downgrade. The dashboard narrates the decision from these fields.
Arbr never reroutes autonomously. auto follows only the rules and policies a human enabled, and every one is reversible from the dashboard in seconds. Routing decides on task type and difficulty, not embeddings or output-quality scoring. Evaluation (see Recommendations) measures whether a candidate holds up, but it doesn't feed routing automatically. A person always approves.
Rule conditions, the AI policy generator, and the cost-guardrail internals are covered in the routing reference on GitHub ↗.