Feature · Failover & load balancing
LLM failover and load balancing that keeps your agents running
A rate limit or a provider outage shouldn't stop a coding session. AutoJev spreads traffic across models by priority and weight, keeps each session on one model, and quietly moves to the next candidate when a request fails before any output has started.
- Weighted pools with priority groups
- Failover on connection errors, 401/402/403/408/429, 5xx
- Cooldowns that honour Retry-After
- Sticky sessions — no mid-task model hopping
Automatic failover
autojev/fast
- 1
Attempt 1 · openai/gpt-5-mini
HTTP 429 · Retry-After 30s
- 2
Cooldown
openai/gpt-5-mini · 30s
Attempt 2 · deepseek/deepseek-chat
200 · streaming
One provider is a single point of failure
Long agent sessions hit rate limits, expired credits and the occasional 503. When the only model your agent knows about goes down, the whole session stalls until you notice and edit the config.
With AutoJev, the agent asks for a route such as autojev/auto. Behind it you list several candidate models — from different providers if you like — and decide how traffic is shared and what happens when one of them fails.
How load balancing works
Priority groups
Every candidate has a priority from 0 to 1,000,000. Higher numbers go first; lower groups only take traffic when the higher group has no available model.
Weights
Within a priority group, new sessions are shared by weight (1–10,000). Set 1:3 and the second model gets roughly three times as many new sessions.
Sticky sessions
When a request carries a session identity, follow-up turns stay on the same model, so caches stay warm and the agent's behaviour stays consistent.
When AutoJev fails over — and when it doesn't
Failover applies to named load-balancing and intelligent-selection routes. Each candidate is tried at most once per request.
Defaults: up to 4 attempts per request (1–16), a 30-second cooldown (1–3600 s) and a failure threshold of 3 (1–20), all adjustable in Settings.
A failover, step by step
The first-choice model returns 429. AutoJev records the attempt, tries the next candidate, and streams that answer back. The agent sees one successful response; the log shows both attempts.
Automatic failover
autojev/fast
- 1
Attempt 1 · openai/gpt-5-mini
HTTP 429 · Retry-After 30s
- 2
Cooldown
openai/gpt-5-mini · 30s
Attempt 2 · deepseek/deepseek-chat
200 · streaming
Cooldown and recovery
- A cooled-down model is skipped by new requests until the cooldown ends
- If the provider sends Retry-After (seconds or an HTTP date), AutoJev uses it, capped at one hour
- When the cooldown ends, a single probe request is allowed through; a complete successful response restores the model
- A late success from an older request can't clear a newer cooldown
- Sessions that moved to a backup can stay there — AutoJev doesn't force-migrate in-flight work
- Channel health is shown in Settings and can be reset manually
Build a resilient route
- 1
Create a load-balancing route
Under Routes, add a route with ID auto and choose Load balancing.
- 2
Add candidates with priority and weight
Give your preferred models the highest priority and split them by weight. Add a backup model from a different provider at a lower priority.
- 3
Tune timeouts if needed
Defaults are 10 s to connect, 60 s to receive response headers and 120 s of stream idle time. Adjust them, along with attempts and cooldowns, in Settings.
- 4
Point agents at autojev/auto
Connect agents from the Agents page. Open Logs → Details to see every upstream attempt with its provider, model, HTTP status and timing.
Why no retries after streaming starts
Once an agent has received part of an answer, silently restarting on another model could duplicate text or repeat tool actions. AutoJev only fails over before the first byte reaches the client, and reports mid-stream interruptions as errors.
Frequently asked questions
What is LLM failover?
Failover sends a request to a backup model when the first choice fails. AutoJev does this automatically for connection errors and HTTP 401, 402, 403, 408, 429 and 5xx responses, as long as no output has been sent to the agent yet.
How does weighted load balancing decide which model to use?
Only the highest available priority group is used. Within it, new sessions are distributed in proportion to each model's weight. Small samples won't match the ratio exactly, but it evens out over many sessions.
Does AutoJev respect Retry-After headers?
Yes. When a failing response includes Retry-After, AutoJev cools the model down for that long, up to one hour, instead of its default cooldown.
Will failover make my agent repeat a tool call?
No. AutoJev never replays a response that has already started streaming, so a tool call that reached the agent won't be issued again by another model.
Can I fail over between providers using different APIs?
Yes. Candidates can use Chat Completions, Responses or Anthropic Messages; AutoJev converts the request for whichever model ends up handling it, within the protocol conversion limits.
Keep exploring
Updated 2026-10-14
Give every agent the right model.
Install AutoJev, add the providers you already pay for, and connect your agents in one click. Everything stays on your machine.
Free & open source · macOS · Windows · Linux