Gateway
One endpoint your app talks to, no matter which AI provider actually answers.
Native chat endpoint
A single POST /v1/chat call handles routing, budgets, safety checks, and logging for you.
OpenAI-compatible endpoint
Point any tool already built for OpenAI's API at Arbr instead. Same request and response shape, including streaming.
Streaming responses
Get tokens back as they're generated instead of waiting for the whole answer.
Tool / function calling
Models can call your app's functions mid-conversation, passed through faithfully.
Embeddings endpoint
Turn text into vectors for search or similarity, through the same gateway.
Observe-only ingestion
Already calling providers directly? Report what happened after the fact, in bulk, so it still shows up in your dashboards.
Realtime voice proxy
Relays a live voice/WebSocket session to OpenAI's Realtime API while still counting the tokens used.
Model & provider discovery
Ask Arbr what models and providers are currently connected and ready to use.
Providers & models
Over 100 providers and 3,000 models, tracked in one place with current pricing.
Eight native provider adapters
OpenAI, Anthropic, Google Gemini, Amazon Bedrock, DeepSeek, Moonshot, xAI, and Groq are wired in directly, with each one's own API shape handled for you.
Around thirty providers pre-configured
Mistral, Cohere, Together, Fireworks, Perplexity, Cerebras, OpenRouter, Azure AI, Databricks, Vertex AI and more are ready to connect with just a key, no endpoint setup.
Any OpenAI-compatible endpoint
Point Arbr at any other OpenAI-compatible base URL and it works, no code change needed.
Live catalog of 100+ providers and 3,000+ models
Arbr pulls the full public LiteLLM catalog, so every model from every listed provider is available with its pricing and capabilities (vision, reasoning, tool use) kept current automatically.
Automatic fallback on failure
If a call fails, Arbr can retry with the same provider, try another live provider, or stop, whichever you choose.
Self-healing on first boot
If the model catalog is ever empty, Arbr refills it itself, with no manual step required.
Smart routing
Decide, automatically or by hand, which model actually answers a request.
AI-generated routing policy
Arbr can learn which model is the best fit for each kind of task and route accordingly.
Cost guardrail mode
Automatically downgrades cheap, low-risk task types to a lighter model to save money.
Human-authored routing rules
Write your own exact "if this, use that model" rules, with no AI guesswork, fully reversible.
Pinned models are never touched
If your app explicitly asks for a specific model, Arbr always honors it. Automation only kicks in for "auto" requests.
Caching & token limits
Skip a repeat AI call entirely when it's safe to, and stop runaway responses.
Exact-match cache
An identical request gets the identical answer back instantly, for a short window.
Semantic response cache
Requests that mean the same thing, even if worded differently, can reuse a cached answer.
Max-output-tokens guardrail
Caps how long a single response is allowed to be, so one runaway reply can't blow up your bill.
Guardrails & safety
Catch bad inputs going in and bad content coming out.
Prompt injection detection
Flags and blocks messages trying to hijack the model with hidden instructions or jailbreak attempts.
Output content guardrails
Blocks a response outright if it matches words or patterns you've banned, per app if you like.
PII masking in stored logs
Personal information in prompts and responses is redacted before it's ever saved to the database.
PII masking in live responses
Can also strip personal information out of the answer actually sent back to the caller, not just the logs.
Recommendations
Arbr watches real traffic and tells you where you're overpaying.
Overspend detection
Spots when an expensive model is being used for work a cheaper one could handle just as well.
Per-app or org-wide suggestions
Surfaces a recommendation scoped to one application if the pattern is local, or the whole org if it's everywhere.
Real projected savings
Every suggestion shows the actual dollar saving, calculated from your real usage, not a guess.
A visible lifecycle for every suggestion
Each recommendation shows exactly where it stands, whether building evidence, being tested, rolling out, live, or dismissed, and what to do next.
"Almost-savings" surfaced too
Also shows genuine savings opportunities that weren't caught because a task type wasn't yet marked as "cheap", with a one-click fix.
Realised vs. promised savings
After a change goes live, Arbr compares what was actually saved against what was projected, honestly.
Proving it's safe (evaluation)
No cheaper model gets used until it's been tested against your own traffic.
Test datasets from real traffic
Builds a test set out of your own past requests, not generic benchmarks, so the test actually reflects your work.
Balanced, deduplicated sampling
Picks a representative spread of examples instead of a lucky (or unlucky) handful of similar ones.
Offline replay against a candidate model
Reruns your real historical requests through the proposed cheaper model to see how it actually performs.
AI judge with a strict scorecard
A second AI grades correctness, completeness, formatting, and safety, and flags anything seriously wrong.
Bias-resistant judging
The judge doesn't know which answer is the "new" one, and a different-family model checks high-stakes cases so it isn't grading its own homework.
Second-opinion pass
A "worse" verdict gets a skeptical second look before it's allowed to sink a candidate.
Risk-based pass bars
Higher-stakes task types must clear a stricter bar, a bigger sample and near-zero critical failures, before they can pass.
Shadow testing on live traffic
Mirrors real live requests to the candidate model in the background, with zero effect on what the customer actually sees.
Spend and error caps during testing
Testing automatically pauses itself if it starts costing too much or erroring too often.
Rolling out & rolling back
Turn a proven candidate on gradually, with an automatic escape hatch.
Gradual canary rollout
Sends only a small, controlled slice of traffic to the new model at first, never touching requests pinned to a specific model.
Automatic rollback on trouble
Watches error rate, latency, and savings continuously and reverts to the old model by itself if anything breaches the limits you set.
One-click promote to 100%
Once a canary has proven itself, make it the permanent, fully-enabled routing rule with one action.
Manual rollback, always available
A human can revert a rollout at any time, no waiting for an automatic trigger.
Exportable evidence report
A complete paper trail for one recommendation, covering the test results, the rollout, the approvals, and the outcome, downloadable as JSON or a readable document.
Zero-key demo mode
A one-command demo walks through the entire story, from opportunity to test to rollout to rollback, using realistic synthetic data and no real API keys.
Budgets & governance
Hard limits and org-wide switches, so spend can never quietly run away.
Spend caps by app, provider, department, model, or end user New
Set a budget scoped as narrowly or broadly as you need, down to a single end user, or org-wide.
Three ways to enforce a cap
Choose to just get alerted, automatically downgrade to a cheaper model, or hard-block requests once a cap is hit.
Early-warning alerts
Get notified well before a budget is actually breached, not just at the moment it happens.
Per-user usage alerts New
Warn, or hard-cap, an individual end user as they approach their limit, and fire a webhook your app can turn into an in-product "you're at 80% of your usage" notice.
Org-wide maintenance kill-switch
One toggle instantly stops all AI traffic with a custom message, for planned maintenance or an incident.
Automatic data retention & cleanup
Old request logs are deleted automatically after a retention window you choose.
Webhook alerts
Sends a real-time notification to Slack or any webhook URL when a budget or error-rate threshold is crossed.
Analytics & reporting
See exactly what you're spending, on what, and whether it's working.
Usage & spend overview
Headline numbers at a glance: total cost, request volume, tokens, success rate, and cache hit rate.
Break down by any dimension
Slice spend and usage by application, team, workflow, model, provider, task type, or user.
Cost and usage trends over time
Day-by-day or hour-by-hour charts showing where spend is heading.
Spend in your own currency New
Show all costs in INR, EUR, GBP and more, converted live from up-to-date exchange rates. Amounts stay stored in USD, so nothing about billing changes, and the rate falls back to the last good value if the feed is briefly unavailable.
Embeddable usage widget New
Drop a live per-user usage chart straight into your own product with a single iframe, with no rebuild and no admin key, gated by a read-only usage token.
Measured, realised savings
Tracks actual savings from requests that got served by a cheaper model than originally asked for.
Trust level on every dollar saved
Shows which savings came from a fully-tested rule versus a human override versus no gate at all.
Per-provider health
Live error rate and response speed for each connected provider.
Latency percentiles
Not just averages. See typical, 95th- and 99th-percentile response times.
Arbr's own overhead, kept separate
Arbr's internal AI calls (classification, testing, judging) are tracked but never mixed into your customer-facing numbers.
Identity & access
Control who can see and change what, with a real name attached to every action.
Three permission levels
Viewer, operator, and administrator roles, each unlocking more control.
Sign in with your company's login
Supports standard single sign-on (OIDC) against Okta, Google Workspace, Auth0, or any compatible provider.
Works behind an identity-aware proxy
Can trust identity already verified by Google IAP or a reverse proxy in front of it, no separate login screen.
Break-glass master key
A backup admin key always works for emergency or automated access, in any login mode.
Every action tied to a real person
Audit entries show exactly who did what, never a generic "admin."
User management
Administrators can list, promote, disable, and re-enable teammates without touching a config file.
Simple first-time setup
A one-time command mints your very first administrator account.
Keys & security
Credentials that are hard to leak and easy to rotate.
Per-app gateway API keys
Issue a separate key to each application, shown only once, so you always know what's calling in.
Read-only usage tokens New
Issue a scoped token that can read only its own usage numbers, with no ability to make AI calls and no admin access, so an app or an end user can see spend without your master key.
Per-key rate limits
Cap how many requests per minute any single key can make.
Key expiry & one-click rotation
Set keys to expire automatically, and replace a key instantly without breaking the app that uses it.
Self-service key rotation New
A key's own holder can rotate or revoke it themselves, using the key as proof, without needing an operator or admin. The old key stops working immediately.
Provider keys encrypted at rest
Your OpenAI/Anthropic/etc. keys are encrypted in the database and never sent back to the browser.
Cloud secret-manager support
Point any credential at a secret stored in Google Cloud Secret Manager instead of typing it in directly (AWS/Azure documented, easy to add).
Rotate a secret with no downtime
Update a secret in your cloud vault and Arbr picks up the new value automatically, no restart needed.
Refuses to start if misconfigured
In production, Arbr won't boot at all if a required secret can't be reached, and it says so with a clear, actionable error.
CSRF protection
Blocks forged requests that try to ride along on a logged-in browser session.
Admin API rate limiting
Throttles the admin API itself so a leaked key or a runaway script can't hammer the database.
Full audit trail
Every meaningful change is logged and can be exported as a CSV for compliance or review.
Running it reliably
The unglamorous stuff that makes Arbr something you can actually run in production.
Separate "is it alive" and "is it ready" checks
A load balancer can tell the difference between a crashed instance and one that's just draining traffic before a restart.
Graceful shutdown
On restart, Arbr finishes in-flight requests and gives traffic a moment to move elsewhere before it actually stops.
One-command backup
Back up the entire database with a single script.
Guided, verified restore
Restore from a backup with a confirmation step and an automatic health check afterward.
Export your whole configuration
Download every setting, rule, and budget as one file, never your provider keys.
Import configuration back
Restore that same configuration onto a fresh instance in one step.
One-click support bundle
Generates a diagnostics package for support, covering version, settings, disk space, and recent activity, with no private data or conversation content inside.
Disk-full protection
Deployments refuse to proceed if the server is nearly out of disk space, instead of failing mid-way.
Log rotation built in
Container logs are capped automatically so they can't quietly fill up the disk over time.
Safe, gated deploys
New versions only go live after passing tests, and roll themselves back automatically if the new version doesn't come up healthy.
Observability
Plug Arbr into the monitoring tools you already use.
Distributed tracing export
Sends a trace for every request to your existing tracing tool (Datadog, Jaeger, Honeycomb, and similar), off by default.
Safe by default
Prompts, responses, and credentials are never included in exported traces unless you explicitly turn that on.
Adjustable on the fly
Turn tracing on or off, or change how much of it gets sampled, right from the dashboard, with no restart.
Trace continuity
If a request already arrives with a trace attached, Arbr's span nests inside it instead of starting a new, disconnected one.
The web console
Every screen in the dashboard, and what it's for.
Overview
The homepage: spend, usage, and trend at a glance.
Applications
Metrics per connected app, down to individual requests.
Recommendations
Every cost-saving suggestion and where it stands.
Routing
Routing mode, your rules, and the AI policy, all in one place.
Model Evals
Test results, shadow campaigns, and canary rollouts.
Budgets
Every spend cap and how close it is to breaching.
Governance
The kill switch, privacy settings, and org-wide guardrails.
Audit
A searchable log of every admin action taken.
Users
Manage teammates and their permission level.
Models
Connect providers, browse the model catalog, add custom ones.
Settings
Gateway keys, read-only usage tokens, display currency, and the default provider/model.
In-app docs
A built-in walkthrough of every screen, with copy-pasteable examples.
Sign-in
Admin-key or company-login sign-in screen, depending on how it's configured.
SDKs, integrations & demos
Get connected quickly, whatever you're building with.
JavaScript SDK
A zero-dependency Node.js client for calling Arbr from your app.
Python SDK
A zero-dependency Python client, with sync and async support and an optional LangChain adapter.
LibreChat integration
Add Arbr as a custom endpoint in LibreChat with no code changes.
OpenCode integration
Route agentic coding traffic through Arbr from OpenCode.
NVIDIA (build.nvidia.com) integration
Connect NVIDIA-hosted models the same way as any other OpenAI-compatible provider.
One-command demo
A single Docker Compose command seeds a realistic dashboard with synthetic data, no provider keys required.