What is Computer Use? A Complete Guide to AI Agents That Control Computers
Computer use is the ability of an AI model to operate a computer through its graphical interface: reading the screen, moving the mouse, typing. Here is how the loop works, which models support it in 2026, and where the benchmarks stand.
Computer use is the ability of an AI model to operate a computer through its graphical interface instead of an API. The model gets a screenshot, decides on an action, and emits it as a mouse or keyboard event. Then it takes another screenshot. That loop is the entire idea, and it means an agent can drive any application a person can drive, including software that exposes no API at all.
What is Computer Use?
A computer use agent does not call functions on the application it is operating. It looks at a rendered desktop, decides that the button it wants is at (450, 320), and clicks there.
The term entered common use in October 2024, when Anthropic shipped computer use as a public beta on Claude 3.5 Sonnet: the first frontier model to offer it. Two years later it is a standard capability across Anthropic, OpenAI, Google, and open source, and on Anthropic's own API it has left beta.
How Computer Use Agents Work
The agent runs a perception-action loop. Capture the screen, interpret it, choose an action, execute it, capture again. Four pieces do the work.
Vision. A multimodal model reads the screenshot and identifies what is on screen: buttons, fields, menus, dialogs, error text. This is the same model that does the reasoning, which is why computer use arrived when it did rather than in 2020.
Grounding. Deciding to "click submit" is not enough. Something has to convert that intent into a pixel coordinate. Some stacks let the frontier model emit coordinates directly; others hand off to a smaller grounding model tuned for UI elements, which is usually faster and cheaper per step.
Action execution. The chosen action becomes a synthetic input event: a cursor move and click, a keystroke sequence, a scroll. In a sandboxed desktop this goes through the display server rather than physical hardware.
Planning. A useful task takes tens of steps, and any one of them can land somewhere unexpected. The planner holds the goal across steps, notices when the screen does not match the expected result, and replans. Most real-world failures happen here, not in vision.
Every step costs a screenshot in and a decision out. A 40-step task is 40 model calls with images attached, which is the dominant term in both latency and price.
Computer Use vs Traditional Automation
| Characteristic | Computer Use Agents | Traditional Automation (RPA) |
|---|---|---|
| Interface interaction | Visual interpretation of the GUI | Requires API access or DOM selectors |
| Adaptation to changes | Tolerates UI updates it can still read | Breaks when selectors move; requires reprogramming |
| Software compatibility | Any application with a GUI | Limited to supported integrations |
| Error handling | Can reason about a failure and retry differently | Rigid; fails on unexpected conditions |
| Context understanding | Maintains goal awareness across steps | Executes fixed sequences |
| Determinism | Non-deterministic; same task can take different paths | Deterministic and repeatable |
| Speed per action | Seconds (screenshot plus inference per step) | Milliseconds |
The tradeoff is honest in both directions. If a stable API exists for the job, call the API. Computer use earns its cost where no API exists, where the interface changes faster than an integration can be maintained, or where the task spans several unrelated applications.
Which Models Support Computer Use in 2026
Anthropic. The computer_toolset_20260801 toolset is generally available with no beta header, on claude-opus-5, claude-sonnet-5, claude-opus-4-8, claude-fable-5, and claude-mythos-5. That version ships on the Claude API and Google Cloud. Bedrock, Claude Platform on AWS, and Microsoft Foundry are still on the older computer_20251124 tool, which requires the computer-use-2025-11-24 beta header.
OpenAI. Computer use runs on the gpt-5.6 line. The dedicated computer-use-preview model was shut down on 2026-07-23, with gpt-5.6-terra listed as its replacement.
Google. Gemini 3.5 Flash has computer use built into the main model line as of 2026-06-24. The standalone gemini-2.5-computer-use-preview-10-2025 is now a legacy preview.
Open source. Simular's Agent S3 is the current release of the Agent S line (pip install gui-agents).
If you are following a tutorial written before 2026, check the model id before you copy it. Claude 3.7 Sonnet was retired on 2026-02-19 and requests to it now fail. GPT-4V (gpt-4-vision-preview) was shut down on 2024-12-06. Both still appear in a lot of published example code.
Benchmark Progress on OSWorld
OSWorld is the standard measure: real applications on a real desktop, scored on task completion rather than on step accuracy.
When OSWorld was published in April 2024, the best model scored 12.24%. Anthropic's Claude 3.5 Sonnet reached 14.9% in the screenshot-only category that October, against a next-best 7.8%. Agent S2 hit 34.5% in March 2025. Agent S3, released in October 2025, reached 62.6% as a single agent and 69.9% with Behavior Best-of-N wide scaling.
In December 2025, Simular reported 72.6%, crossing the 72.36% human baseline from the original OSWorld paper. That took two years from the benchmark's publication.
The benchmark itself has moved twice: OSWorld-Verified landed on 2025-07-28 and OSWorld 2.0 on 2026-06-26. Scores are not comparable across those revisions, so treat any leaderboard number without a version attached as unusable.
Getting Started with Computer Use
You need three things: a model with computer use support, a desktop for it to operate, and a way to issue instructions and read results.
The desktop is the part people underestimate. An agent that clicks arbitrary things needs to be isolated from anything you care about, needs a consistent starting state so runs are comparable, and needs to boot fast enough that you are not paying for idle machines between tasks. Orgo runs cloud computers for AI agents and covers that layer. If the work never leaves a web page, compare cloud desktops against headless browsers first: the browser is cheaper and runs at higher concurrency.
A minimal Python example:
from orgo import Computer
computer = Computer()
computer.prompt("Open Firefox and search for weather in Seattle")prompt() runs the full screenshot-action loop for you. If you are writing your own agent logic, drive the primitives directly:
computer.screenshot() # Capture current state
computer.left_click(450, 320) # Click at coordinates
computer.type("search query") # Type text
computer.key("ctrl+enter") # Press key combination
computer.bash("ls -la") # Execute shell commandTwo practical notes from running these loops. First, bash() is often the correct tool even in a GUI task: if the goal is to move a file, do not click through a file manager. Second, cap your step count. An agent that has taken 60 actions without finishing is usually stuck in a loop, and the cheapest fix is to fail and restart rather than let it keep paying for screenshots.
Where Computer Use Goes Next
The human baseline has been crossed on the benchmark, which mostly moves the interesting questions elsewhere: cost per completed task, latency per step, and reliability over long sessions where a single misread dialog compounds into a wrong end state. Grounding models are getting faster, planners are getting better at noticing they are off track, and providers are folding computer use into their main model lines rather than shipping it as a separate preview.
The durable point is the one that was true in 2024. Software that will never expose an AI-friendly API is still the majority of software, and an agent that can read a screen does not need one.