Core

Gateway

One endpoint stands between your applications and every model provider. Point a client at it and every call gains routing, cost logging, attribution, and budget enforcement. Nothing in your code changes beyond the base URL.

Arbr exposes several surfaces on one port (default 4100). Two of them accept chat traffic: a native endpoint with full attribution and a drop-in OpenAI-compatible endpoint. There are also embeddings, a realtime voice proxy, and an observe-only ingestion endpoint.

Authentication

Data-plane calls authenticate with a gateway API key (prefix ab_), sent as a bearer token. Keys are optional until an administrator turns on Require API keys, after which anonymous calls are rejected.

Authorization: Bearer ab_...

A key binds attribution: the application, department, and rate limit tied to it override whatever the request body claims. See Gateway API keys.

Native endpoint: POST /v1/chat

The Arbr-native endpoint. It accepts business metadata alongside the messages for full attribution, task classification, and routing.

Body fields

Example

curl -X POST http://localhost:4100/v1/chat \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer ab_...' \
  -d '{
    "application": "support-chat",
    "workflow": "ticket-triage",
    "department": "Support",
    "taskType": "classification",
    "model": "claude-haiku-4-5",
    "provider": "anthropic",
    "messages": [
      { "role": "system", "content": "Classify the ticket in one word." },
      { "role": "user",   "content": "My card was declined at checkout." }
    ],
    "maxTokens": 50
  }'

Response

{
  "requestId": "a1b2c3d4-...",
  "text": "billing",
  "model": "claude-haiku-4-5",
  "modelRequested": "claude-haiku-4-5",
  "provider": "anthropic",
  "routingDecision": "explicit",
  "classifiedBy": "provided",
  "cacheHit": false,
  "usage": {
    "inputTokens": 28,
    "outputTokens": 1,
    "totalTokens": 29,
    "cachedReadTokens": 0
  }
}

model is what was served, and modelRequested is what the app asked for. Storing both is what makes realised savings measurable. routingDecision explains why this model was chosen (see Routing).

OpenAI-compatible: POST /v1/chat/completions

A drop-in replacement for the OpenAI chat API. Any client that speaks the OpenAI spec works unchanged: the official SDKs, LangChain's ChatOpenAI, LibreChat, OpenWebUI. Just change the base URL and use a gateway key.

from openai import OpenAI

client = OpenAI(
    base_url="https://arbr.yourcompany.com/v1",
    api_key="ab_your_gateway_key",
)

resp = client.chat.completions.create(
    model="auto",                       # let Arbr route
    messages=[{"role": "user", "content": "Hello"}],
)
print(resp.choices[0].message.content)

The response is a standard chat.completion object. To pin a provider for a pass-through model, pass it in extra_body:

client.chat.completions.create(
    model="deepseek-chat",
    messages=[...],
    extra_body={"provider": "deepseek"},
)

Streaming

Set "stream": true for Server-Sent Events. You get chat.completion.chunk lines terminated by data: [DONE], exactly as OpenAI streams. One caveat: output guardrails and response transforms don't apply to streamed responses, though the input max-tokens clamp still does. Behind nginx, disable proxy buffering for the streaming location.

Embeddings: POST /v1/embeddings

OpenAI-compatible embeddings, routed to OpenAI (text-embedding-3-small / -large) or Gemini (gemini-embedding-001) by model ID. Logged with taskType: "embedding".

curl -X POST http://localhost:4100/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{ "model": "text-embedding-3-small", "input": "hello world" }'

Pass dimensions to request a shorter vector (Gemini maps it to outputDimensionality). It has to match your index size. The response is { object: "list", data: [{ embedding: [...] }], usage }.

Realtime voice: WS /v1/realtime

A transparent WebSocket proxy to the OpenAI Realtime API. Clients connect with a gateway key, Arbr injects the real provider key, and it relays every frame unmodified. When the session closes, it logs audio and text token counts and the session duration.

wss://arbr.yourcompany.com/v1/realtime?model=gpt-4o-realtime-preview

Observe-only ingestion: POST /v1/ingest

Report metadata for calls that happened elsewhere, such as a LiteLLM callback, so their cost and usage show up in Arbr without routing through it. Send a batch of up to 500 events. Only requestId and model are required per event, and cost is derived server-side. Ingested records count toward budgets for visibility but never trigger enforcement, and they're idempotent per key on requestId.

Discovery & health