Use case · Model evaluation

Evaluate models on your real coding workflow, not a leaderboard

Benchmarks tell you how a model did on someone else’s tasks. AutoJev lets you measure models where it matters — inside the agents you use every day — with speed tests, a debug console, logs and per-model usage numbers.

First-content latency and output tokens/s per model
Debug console with conversation and cURL modes
Per-model success rate, duration, tokens and cost
Swap the model behind an agent without editing configs

Download AutoJev Star on GitHub

Models

▶ Speed test

ModelFirst tokenSpeed test

deepseek-chat620 ms58 t/s

claude-sonnet940 ms71 t/s

gpt-5-mini780 ms96 t/s

qwen3:14b210 ms42 t/s

Why model comparisons are hard

Every provider reports differently

Each dashboard has its own latency and usage view, if any. Numbers from different providers are not comparable.

Your network and region matter

A model that is fast in someone’s benchmark can be slow from your location, at your time of day.

Switching is friction

Trying model B in your agent means rewriting its config, then rewriting it back. So most comparisons never happen.

Vibes are not data

“It felt slower today” is hard to act on without request-level timings and success rates.

How AutoJev turns it into data

All your models sit behind one gateway, so every request — from speed tests, the debug console or your agents — is measured the same way. Speed tests send three short streaming requests per model and record first-content latency and output tokens/s; AutoJev keeps up to 30 samples per model and can re-test automatically every 30 minutes (5–120 minutes configurable). Real streaming requests from your agents update the measurements too.

Because agents talk to AutoJev rather than to a provider, switching the model behind an agent is a selection in the app, not a config edit. And Logs and Usage put every model on the same table: tokens, first byte, total time, estimated cost, status and success rate.

Evaluation recipe: compare two models in a week

1
Add the candidates
Add both models — from any providers — with accurate prices, tier, capability flags and context window. Run the provider test for each.
2
Run speed tests
On Models, select both and run a speed test. Each gets three streaming requests with a fixed short prompt and a 128-token output limit. Hover a result to see sample count, success rate and timestamp.
3
Send identical prompts in the debug console
Open Debug, pick model A, send your prompt; then pick model B and send the same prompt. Each run shows the selected model, timing, tokens, estimated cost and raw JSON. Switch the API type to check how each model behaves over Chat, Responses or Messages, add up to 4 attachments (≤4 MB each) or custom parameters.
4
Split real traffic
Create a load-balancing route with both models at the same priority and equal weight, and connect your agent to it. Session affinity keeps each conversation on one model, so sessions are cleanly attributable.
text
Route: autojev/ab-test Strategy: Load balancing Candidates: provider-a/<model_a> priority 1 weight 1 provider-b/<model_b> priority 1 weight 1
5
Read the results
In Usage, break down by model: requests, tokens, cache hit rate, estimated cost, success rate and average duration. In Logs, filter by model and open a request to see its upstream attempts.

The debug console

Conversation mode for quick back-and-forth, cURL mode for exact request bodies. Each live test starts a new session.

Debug

autojev/quality

Summarize the failing test and propose a fix.

The test fails because the date parser ignores the timezone offset. Normalize to UTC before comparing…

Modelopenrouter/claude-sonnet

First token842 ms

First byte / total3.9 s

Tokens in / out1,204 / 318

Est. cost$0.0084

Reproduce a request outside the app

cURL mode posts JSON to the local gateway. The same request works from your terminal — just change the model to compare.

Same prompt, different model

for model in deepseek/deepseek-chat openrouter/anthropic/claude-sonnet-4; do
  curl -s http://127.0.0.1:9527/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer autojev-local-eval' \
    -d "{\"model\": \"$model\", \"messages\": [{\"role\": \"user\", \"content\": \"Refactor this function to be pure\"}]}" \
    -o /dev/null -w "$model %{time_total}s\n"
done

What you can measure

Signal	Where
First-content latency, output tokens/s	Models (speed tests + real streaming requests)
First byte and total time per request	Logs
Input, output and cache tokens; estimated cost	Logs, Usage, Debug
Success rate and average duration per model	Usage
Upstream attempts and failover	Log details
Exact request and response JSON	Debug console

Measure what you can trust

Speed tests use a short fixed prompt, so they cannot guarantee the fastest model for long-context requests; real requests fill that gap. Tests and live debug requests use your provider quota, and output rate is unknown when a provider does not report complete token usage.

Frequently asked questions

How does AutoJev measure model speed?

Each speed test sends three sequential streaming requests per model with a fixed short prompt, a 128-token output limit and a 20-second timeout, recording first-content latency and output tokens/s. Up to 30 samples are kept per model.

Can I compare two models side by side?

Run the same prompt against each model in the debug console, or use the cURL mode, and compare timing, tokens and cost. For real workloads, split agent traffic with an equal-weight load-balancing route and compare models in Usage.

Do speed tests cost money?

Yes, a little: they are real requests to your providers. Scheduled tests skip models with fresh samples, and you can turn scheduling off or set the interval between 5 and 120 minutes.

How do I switch the model behind Codex or Claude Code quickly?

Select a different model or route for the agent on the Agents page and reconnect, or change the models inside the route the agent uses. No manual config editing is needed.

Are prompts stored for later review?

No. Request logs keep metadata only. The debug console shows the full response for the request you just sent, but it is not persisted in request logs.

Keep exploring

FeaturesModel speed testsMeasure first-content latency and output tokens per second on your own network, on a schedule, and route by the results.FeaturesDebug consoleTest models and routes in conversation or cURL mode and inspect the selected model, timing, tokens, cost and raw JSON.FeaturesUsage & request logsTokens, latency and estimated cost for every request, broken down by agent, model and provider. Metadata only, stored locally.FeaturesFailover & load balancingWeighted pools, priority groups and sticky sessions, with automatic failover on 429s and 5xx errors and Retry-After cooldowns.Use casesFor cutting agent costsRoute routine turns to cheaper models, set a cost baseline, and see estimated spend by agent and model. No markup.

Updated 2026-10-14

Give every agent the right model.

Install AutoJev, add the providers you already pay for, and connect your agents in one click. Everything stays on your machine.