AutoJev
Download

Feature · Speed tests

LLM speed tests: first-token latency and tokens per second, measured locally

Provider benchmarks don't tell you how fast a model is from your desk, today, with your key. AutoJev measures it — first-content latency and output speed for every model — and uses the numbers to route latency-sensitive work.

  • First-content latency + output tok/s
  • 3 streaming requests per model per test
  • Scheduled every 30 min by default
  • Feeds the "Lower latency" routing preference
Download AutoJevStar on GitHub

Models

▶ Speed test
ModelFirst tokenSpeed test
deepseek-chat620 ms58 t/s
claude-sonnet940 ms71 t/s
gpt-5-mini780 ms96 t/s
qwen3:14b210 ms42 t/s

Why measure speed yourself

Time to first token (TTFT) and output speed vary by provider, region, time of day and the proxy you sit behind. A model that is quick for someone else can feel sluggish in your agent loop.

AutoJev tests from the same machine and network your agents use, through the same outbound proxy settings. Results show up next to each model in the Models table as latency and output rate on one line, with sample count, success rate and timestamp on hover.

What a speed test measures

MetricMeaning
First contentMilliseconds until the first real content arrives. Role-only events, usage events and heartbeats don't count. Lower is faster.
Output speedOutput tokens per second after the first content. Shown as unknown if the provider doesn't report complete token usage.
Success rateShare of samples that completed successfully, used to penalise unreliable models.
SamplesUp to 30 per model, kept across restarts. Expired samples are labelled explicitly.

How a test runs

  • Select models and click test — or let AutoJev test all enabled models if none are selected
  • Selected models are tested concurrently; each gets 3 sequential streaming requests
  • Fixed short prompt, 128-token output limit and a 20-second total timeout
  • Progress is shown live and a batch can be cancelled; only one batch runs at a time
  • Real streaming requests from your agents also update measurements after they complete
  • Tests consume provider API usage, so they appear on your provider bill

Scheduled tests keep numbers fresh

On by default

Automatic speed tests run every 30 minutes while the app is running, including from the tray. Adjust the interval between 5 and 120 minutes in Settings → General.

Only what's stale

The scheduler checks once a minute and skips models with samples newer than the interval, so you don't pay to re-test models your agents just used.

Keyed to the real endpoint

Samples are tied to provider, upstream model ID, endpoint and protocol. Change the endpoint or protocol and old samples no longer count.

No prompts stored

Samples keep timing, token counts, success and a timestamp — never prompt text or credentials.

How the "Lower latency" preference uses the numbers

For an intelligent route set to Lower latency, AutoJev ranks candidates locally and never calls the Jev decision model.

  1. 1

    Filter

    Keep enabled, healthy candidates that fit the request's context, tool and vision needs and the task's desired tier.

  2. 2

    Group

    Keep the highest available priority group.

  3. 3

    Pick recent samples

    Use at most five samples from the last 30 minutes, preferring real requests in the same context band (<4K, 4K–32K, ≥32K input tokens), with synthetic probes as a cold-start estimate.

  4. 4

    Score

    Drop candidates without measured latency or below 50% success. Estimate total time from median first-content latency plus expected output at median tok/s, divided by success rate.

  5. 5

    Fall back gracefully

    With no comparable measurements, the route uses load balancing; scheduled tests then fill the gap. Existing sessions keep their model while eligible.

Short probes, honest caveat

Synthetic probes use a short prompt, so they can't guarantee which model is fastest for a long-context request. That's why AutoJev prefers measurements from your real requests in a similar context band whenever they exist.

Frequently asked questions

What is TTFT and why does it matter for coding agents?

Time to first token is how long you wait before a model starts answering. Agents make many short calls in a loop, so a few hundred milliseconds per call adds up across a session.

How do I measure tokens per second for an LLM?

In AutoJev, open Models, select the models and run a speed test. Each gets three streaming requests; output tokens per second is calculated after the first content arrives, using the token usage the provider reports.

Do speed tests cost money?

Yes, a little. Each test sends three short streaming requests with a 128-token output limit per model through your own provider key.

Can I turn off automatic speed tests?

Yes. Scheduled tests are on by default every 30 minutes; you can switch them off or set an interval from 5 to 120 minutes in Settings → General.

Do speed measurements get sent anywhere?

They are stored locally. When an intelligent route uses the balanced or quality preference, recent performance figures are included in the routing metadata sent to the Jev decision model.

Keep exploring

FeaturesSmart model routingExpose routes like autojev/fast to every agent. Load-balance models or let Jev pick by cost, quality, speed or balance.Use casesFor evaluating modelsMeasured latency and tokens/s, a debug console for identical prompts, and per-model stats from real agent traffic.FeaturesFailover & load balancingWeighted pools, priority groups and sticky sessions, with automatic failover on 429s and 5xx errors and Retry-After cooldowns.FeaturesDebug consoleTest models and routes in conversation or cURL mode and inspect the selected model, timing, tokens, cost and raw JSON.Use casesFor local-first developersPrefer Ollama models, fail over to the cloud when needed. Localhost gateway, metadata-only logs, no account.

Updated 2026-10-14

Give every agent the right model.

Install AutoJev, add the providers you already pay for, and connect your agents in one click. Everything stays on your machine.

Download AutoJevStar on GitHub

Free & open source · macOS · Windows · Linux