Why Your AI Agents Need a Local LLM Gateway

A year ago most developers used one AI coding assistant against one model. Today a typical setup looks different: Codex for one task, Claude Code for another, Gemini CLI or OpenCode for a third, plus a script or two calling a model directly. Each agent has its own configuration file, its own idea of which protocol to speak, and its own copy of your API keys.

That works until you want to change something. Trying a new model means editing four config files. A provider outage stops every agent that points at it. Nobody can tell you what last week cost, or which agent spent it.

A local LLM gateway fixes this by putting one endpoint on your machine between all your agents and all your providers. This article explains what a local gateway does, the five problems it solves, how it compares to other approaches, and how to set one up with AutoJev.

What is a local LLM gateway?

A local LLM gateway is a small server that runs on your computer, listens on localhost, and forwards model requests from your agents to upstream providers. Instead of pointing each agent at a provider, you point every agent at the gateway:

Codex / Claude Code / Gemini CLI / OpenCode / Hermes / your scripts
                              │
                 http://127.0.0.1:9527/v1
                              │
                    AutoJev local gateway
                              │
        Fixed model / Load balancing / Intelligent selection / Failover
                              │
        OpenAI / DeepSeek / OpenRouter / Ollama / any compatible API

The gateway decides which model handles each request, converts between API formats when needed, retries on failure, and records what happened. Because it runs locally, requests go straight from your machine to your providers using your own keys — there is no third-party relay in the path.

AutoJev exposes these endpoints on 127.0.0.1:9527:

1. Routing: the right model for each request

Not every request deserves your most expensive model. Renaming a variable and designing a security-sensitive migration are different jobs. A gateway lets you express that once, as routes, instead of hard-coding a model into each agent.

In AutoJev a route is a group of candidate models with a strategy, exposed to agents as autojev/<route-id>:

• Fixed model — always the same model.
• Load balancing — new sessions are spread by weight within the highest priority group.
• Intelligent selection — a decision preference picks the model:
  • Lower latency decides locally from measured speed tests, without any external call.
  • Lower cost picks the cheapest candidate that meets the capability requirements.
  • Higher quality and Balanced ask the OpenRouter Jev decision model, using your own OpenRouter key. Only routing metadata is sent, never your conversation. If Jev is unavailable or its confidence is below 0.5, the route falls back to weighted load balancing.

Session affinity keeps every request in a session on the same model while it remains eligible, so an agent does not switch models halfway through a refactor.

Speed routing only works if you measure speed. AutoJev's model speed tests record first-content latency and output tokens per second, run on a schedule (every 30 minutes by default) and keep up to 30 samples per model.

Learn more: smart routing, AI coding agents, cost control.

2. Failover: one provider outage should not stop your work

Every provider has bad days: rate limits, expired keys, overloaded regions. If an agent points directly at one provider, its failure is your failure.

A gateway can try another candidate. AutoJev fails over on connection errors and HTTP 401, 402, 403, 408, 429 and 5xx — but only before any output has started. That rule matters for agents: replaying a half-finished stream could duplicate text or run the same tool call twice, so a response that has started is never retried on another model.

Defaults you can tune:

• Up to 4 attempts per request, each candidate tried at most once.
• 3 consecutive failures put a channel into cooldown; 401, 402, 403 and 429 trigger cooldown immediately.
• 30-second cooldown (1–3600 s), honouring the provider's Retry-After.
• After cooldown, one probe request is let through before the channel is marked healthy.

Learn more: failover and load balancing.

3. Protocol conversion: any agent, any model

Agents speak different protocols. Codex speaks OpenAI Responses. Claude Code speaks Anthropic Messages. Most providers and local servers offer Chat Completions. Without conversion, your choice of agent limits your choice of model.

AutoJev converts in all six directions among those three formats, including streaming, function tools and tool calls with stable IDs, Responses custom tools, images, usage and errors. A model's API type describes its upstream; the gateway handles whatever the agent sends.

It is honest about limits. Conversion cannot add tools, vision, reasoning or context to a model that lacks them. Features without an equivalent — previous_response_id, hosted tools like web search, audio and file content — are rejected rather than silently dropped, and provider-specific thinking blocks are omitted.

Two walkthroughs show this in practice:

• How to use DeepSeek (or any model) in Claude Code
• Run Codex CLI with any model: OpenRouter, DeepSeek, Ollama

Learn more: protocol conversion.

4. Observability: know what your agents are doing

When requests are scattered across agents and providers, basic questions are hard to answer. Which model answered that request? Why was it slow? What did the team's agents cost this week?

A gateway sees every request, so it can answer them. AutoJev records:

• Logs — time, agent, model, provider, route ID, input/output/cache tokens, first-byte and total time, estimated cost and status, filterable by date, agent, provider and status. The detail view lists every upstream attempt, so you can see a failover happen.
• Usage — requests, tokens, cache read and write, cache hit rate, estimated cost, success rate and average duration, with a daily chart and breakdowns by agent, model and provider.

A debug console lets you send test requests to any model or route in conversation or cURL mode and inspect the selected model, timing, tokens, cost and raw JSON.

Costs are estimates based on the prices you configure and the usage providers return. Learn more: usage and logs, debug console, model evaluation, and the LLM cost calculator.

5. Privacy: keep the gateway on your machine

A gateway sees everything your agents send, so where it runs matters.

AutoJev is local-first:

• The gateway listens on localhost only.
• Requests go directly from your machine to your providers with your own keys. There is no AutoJev cloud relay and no AutoJev account, and AutoJev adds no markup.
• Request logs store metadata only. Full prompts, responses and API keys are not persisted in request logs.
• When intelligent selection uses Jev (quality or balanced), only routing metadata is sent — not conversation content.
• Local models via Ollama never leave your machine at all; a route can prefer local models when they qualify.

One honest caveat: configuration and provider keys live in ~/.autojev/autojev.db, and keys are currently stored unencrypted. Protect that file like any other credential store.

Learn more: local models.

One-click agent setup, reversible by design

A gateway is only useful if connecting agents to it is painless. Editing TOML, JSON and YAML by hand for every agent is exactly the work a gateway should remove.

AutoJev detects installed agents and connects them in one click, writing each agent's native configuration:

Every connection backs up the original file first. Disconnecting restores it while keeping later edits that do not conflict. Stop and restore configurations reverts every agent at once. Other tools can use custom agents or the Connection parameters dialog, which produces a base URL, a placeholder key and a model or route ID:

OPENAI_BASE_URL=http://127.0.0.1:9527/v1
OPENAI_API_KEY=autojev-local-my-script

Learn more: one-click agent setup, agent config generator.

How a local gateway compares to other approaches

There are several ways to manage models across agents, and they solve different problems:

• Hosted model aggregators give you one API key for many models, with routing on their servers. A local gateway can use them as one provider among others — AutoJev ships an OpenRouter preset. See AutoJev vs OpenRouter.
• Config switchers change which provider an agent points at, but do not sit in the request path, so they cannot route per request, fail over or convert protocols. AutoJev can import providers from CC Switch. See AutoJev vs CC Switch and CC Switch alternative.
• Self-hosted proxy servers are powerful for teams running shared infrastructure, but you operate them as a service. See AutoJev vs LiteLLM.
• Single-agent routers focus on one client. See AutoJev vs Claude Code Router and Claude Code Router alternative.

AutoJev's position is narrow on purpose: a desktop app for one developer's machine, with a GUI for providers, models, routes and agents, and one local endpoint shared by every agent.

Set it up

1. Install AutoJev from the download page — macOS, Windows x64 or Linux x64. It is free and open source (MIT license).
2. Add providers under Providers: OpenAI, DeepSeek, OpenRouter, Ollama, or any OpenAI-compatible or Anthropic-compatible endpoint. Or import from CC Switch or Termany.
3. Add models with the upstream API type and prices, then run a speed test.
4. Create routes such as fast, cheap and quality.
5. Connect agents under Agents, send a test from Debug, and watch Logs.

AutoJev is in early development, and features may change between releases. The documentation covers every setting in detail.

Summary

As soon as you run more than one agent or more than one model, a local LLM gateway pays for itself. It gives every agent one endpoint, routes each request to an appropriate model, survives provider failures, converts between Responses, Chat Completions and Messages, shows you exactly what happened and what it cost, and keeps your traffic between your machine and your providers.

Explore the features: smart routing · one-click agent setup · protocol conversion · model speed tests · failover and load balancing · usage and logs · debug console · provider management