AutoJev

Jev Computer Use: Browser and Desktop Agent Patterns | AutoJev

Jev computer use applies a decision model inside a browser or desktop control loop. Code observes the interface, builds a bounded set of valid actions, Jev chooses an operation or target, and a deterministic executor performs only the selected action.

Jev does not itself click a button, move a mouse or type a password. The automation framework owns observation, grounding, execution, permissions, stop conditions and verification.

Why Jev fits computer use

Many computer-use steps are selections rather than open-ended writing:

  • Choose click, scroll, wait, type, done or blocked.
  • Select one visible DOM or accessibility element.
  • Decide whether a proposed click is reversible or needs confirmation.
  • Score whether the goal is complete after the interface changes.

These questions have dynamic but enumerable candidates. A Jev Choice can return a complete distribution over those candidates, while Noul can represent “is the goal complete?” or “does this action require confirmation?” as a probability.

Free text still needs a writer or user input. A hybrid agent can use Jev for the operation and target, then call a generative model only when a text field needs a new value.

Browser-use architecture

A DOM-grounded Jev browser agent can use this loop:

  1. Read interactive elements and assign stable indexes.
  2. Remove hidden, disabled and incompatible targets.
  3. Send the goal, recent action summary and bounded operation list as State.
  4. Ask one Choice for the operation and parallel Choices for compatible targets.
  5. Execute only the target associated with the selected operation.
  6. Observe the new page and verify progress before continuing.

Browser Use's Jev Ultrafast is a public example of this pattern. Its repository says Jev selects the operation and indexed DOM element in one request, while a small text model is used only for TYPE_TEXT. The project publishes its own measurements and traces; those results describe that implementation and should be reproduced before being used as a production forecast.

Desktop computer-use architecture

Desktop automation may combine OCR and the operating system's accessibility tree:

  1. Capture the active window and read relevant text.
  2. Enumerate labeled controls exposed through accessibility APIs.
  3. Merge observations into a bounded candidate list with locations and roles.
  4. Ask Jev to select the next action and target.
  5. Apply confidence thresholds and confirmation policy.
  6. Execute through a platform adapter, then capture the next state.

awlevin/typesafe-computer-use demonstrates this approach on macOS. Its README documents OCR, accessibility state, deterministic actions, replay files and stop rules. It also explains the tradeoff: reasoning that a general vision model performs implicitly must be represented explicitly in state or deterministic code.

Example Jev computer-use decision

json
{
  "state": {
    "goal": "Open the billing page without changing the subscription",
    "page": "Account settings",
    "elements": [
      { "id": 12, "role": "link", "label": "Billing" },
      { "id": 18, "role": "button", "label": "Cancel subscription" },
      { "id": 21, "role": "button", "label": "Save changes" }
    ],
    "recent_actions": ["Opened account settings"]
  },
  "questions": {
    "operation": {
      "type": "choice",
      "instructions": "Choose the next bounded operation that advances the goal.",
      "criteria": {
        "click": "A listed element safely advances the goal.",
        "wait": "The interface is still changing.",
        "done": "The goal is already complete.",
        "blocked": "No safe listed action advances the goal."
      }
    },
    "click_target": {
      "type": "choice",
      "instructions": "If clicking is selected, choose the target.",
      "criteria": {
        "12": "Billing link",
        "18": "Cancel subscription button",
        "21": "Save changes button"
      }
    },
    "needs_confirmation": {
      "type": "noul",
      "instructions": "Would the proposed action make a consequential account change?"
    }
  }
}

The executor must ignore click_target unless operation is click. It should also deny elements that disappeared after observation and ask the user before actions that cross the application's confirmation threshold.

Safety boundaries for Jev computer use

  • Ground every target. Never allow the model to invent coordinates, selectors or actions outside the observed candidate set.
  • Re-check before execution. Pages can change between observation and action; verify that the selected element still exists and has the expected role.
  • Separate recommendation from authority. A high-confidence choice is not permission to purchase, publish, delete or change credentials.
  • Protect secrets. Passwords, session cookies, API keys and private screen regions should not enter model State.
  • Use deterministic stop rules. Set step limits, repeated-action detection, timeouts and a fail-closed path for unavailable decisions.
  • Keep evidence. Store redacted observations, candidate lists, decisions and executed actions for replay and evaluation.

Where AutoJev fits

AutoJev can provide the decision calls and guardrail layer, but it is not a browser driver. Application code can use the Jev API for custom operation and target questions, the tool-call guardrail before consequential actions, and a completion review after the browser reports success.

Explore more community implementations in the Jev ecosystem guide, or review Jev games for another example of bounded actions inside an external control loop.