Computer use models compared: Claude, OpenAI, Gemini, and Agent S3 on Orgo
How Claude, OpenAI, Gemini, and Agent S3 differ when they drive the same Orgo computer.
TL;DR
Orgo has an integration guide for four computer use options: Anthropic Claude (Messages API, the computer_toolset_20260801 toolset), OpenAI (Responses API, the computer tool), Google Gemini (Interactions API, the computer_use tool), and Agent S3, Simular's open-source framework (Use any model). Each one drives the same kind of Orgo computer through the Orgo API, so switching models changes your agent loop, not your infrastructure.
Two differences matter most in practice:
- Coordinates. Claude and OpenAI answer in the pixels of the screenshot you send. Gemini answers on a 0-999 grid that you scale to the screen. Agent S3 emits
pyautoguicode that runs inside the computer, so your loop translates nothing, but itswidthandheightsettings must match the computer's screen. - Who runs the loop. Orgo's OpenAI-compatible
POST /v1/chat/completionsendpoint runs the loop for you with Orgo's ownorgo-agentengine or one of the Claude models it accepts. To use an OpenAI or Gemini model you choose, or Agent S3, you run the loop in your own code (Create chat completion).
Orgo does not publish a computer use ranking of these models. Test them on your own task.
What is a computer use model?
A computer use model operates software through its interface. It reads a screenshot, chooses an action such as a click or a keystroke, and your harness carries out the action and sends back a new screenshot. Turning “click Save” into screen coordinates is called grounding. Some models return coordinates directly; other systems pair a planning model with a separate grounding model (What is computer use?).
“Model” is a loose word here. On Gemini 3.x, computer use is a built-in tool rather than a separate model (Gemini computer use). OpenAI's computer tool is part of the Responses API, and Claude declares a toolset on the Messages API. Agent S3 is a framework that pairs a reasoning model with a grounding model (Agent S3 guide).
Orgo sits underneath all of them. It provides cloud computers for AI agents: a full desktop with a browser and the standard userland, reachable over HTTP and VNC. Orgo provides the computer, and its hosted endpoint can also run the agent loop for you (see below).
How do Claude, OpenAI, Gemini, and Agent S3 compare on Orgo?
| Claude | OpenAI | Gemini | Agent S3 | |
|---|---|---|---|---|
| Who ships it | Anthropic | OpenAI | Simular, open source (gui-agents) | |
| API surface | Messages API | Responses API | Interactions API | Python framework (gui_agents.s3) that calls the models you configure |
| Models to start with | claude-sonnet-5-5 for most agents, claude-opus-5-5 for the hardest tasks | gpt-6.1-sol or gpt-6-astra | gemini-3.8-flash | gpt-5-2025-08-07 for reasoning plus ui-tars-1.5-7b for grounding |
| Tool declaration | computer_toolset_20260801, plus bash_20250124 for a shell. No beta header | {"type": "computer"}, no configuration | computer_use with environment: "desktop" | None. The agent emits code |
| Coordinates returned | Screenshot pixels | Screenshot pixels | A 0-999 grid | None for your loop to translate: pyautogui code, scaled to OSWorldACI's width and height |
| What your handler does | Passes coordinates straight to Orgo's click | Passes coordinates straight to Orgo's click | Scales coordinates to the screen size | Runs the code through POST /computers/{id}/exec |
| Screenshot you send | Base64 image, unresized | Data URL with "detail": "original", unresized | PNG, unresized | Raw image bytes (response_format=binary), unresized |
| Actions per turn | Several member calls | A batch in actions[] | Several function_call steps | One code action per step |
| Shell | bash tool, mapped to computer.bash | No shell action in the computer tool | No shell action in the desktop environment | Optional coding agent, which runs on the machine running the script, not on the Orgo computer |
| Orgo can run the loop for you | Yes, with one of the Claude models POST /v1/chat/completions accepts | Not with a model you pick. The default orgo-agent runs on GPT-6 Astra, but OpenAI models are not selectable | No, bring your own loop | No, bring your own loop |
| Older path | computer_20251124 or computer_20250124, with display size fields and a beta header. claude-sonnet-5-5 and claude-opus-5-5 accept only the toolset on the Claude API | computer_use_preview on computer-use-preview, with display_width and display_height | gemini-2.5-computer-use-preview-10-2025 with generate_content, which Google's computer use docs no longer list | Agent S2 under gui_agents.s2 |
Sources: Claude guide, OpenAI guide, Gemini guide, Agent S3 guide, Use any model, Create chat completion, OpenAI computer use, Gemini computer use, and the Agent S README.
Model lists change quickly. As of October 2026, OpenAI's computer use docs show the computer tool with gpt-6.1-sol, and Google's docs name gemini-3.8-flash as the recommended model for computer use. Check the provider's current list before you pin one.
The table compares integration shape, not quality. Orgo does not publish a computer use ranking of these models, and this page does not make one.
Which computer use model should I pick?
Start from the stack you already run and how much loop code you want to own:
- You want the least code. Use Orgo's computer use API. Point the OpenAI SDK at
https://www.orgo.ai/api/v1, pass acomputer_id, chooseorgo-agent(Orgo's computer use engine) or a Claude model, and Orgo runs the agent loop on its side, then returns the final assistant message in chat completions format (Orgo computer use API). - You want full control of a Claude loop. Call the Messages API yourself with
computer_toolset_20260801. Orgo's Claude guide suggestsclaude-sonnet-5-5for most agents andclaude-opus-5-5for the hardest multi-step desktop tasks (Claude guide). - You already build on the OpenAI Responses API. Use the
computertool. It takes no configuration and returns a batch of actions per turn (OpenAI guide). As of October 2026, OpenAI's docs recommend a code execution integration over thecomputertool for GPT-6 Astra, and say thecomputertool remains supported (OpenAI computer use). - You build on Gemini. Use the
computer_usetool withenvironment: "desktop", and treat the scaling constants as load-bearing (Gemini guide). - You want open source and swappable parts. Agent S3 lets you choose the reasoning model and the grounding model separately, and adds Behavior Best-of-N, which runs several attempts at a task and has a judge pick the one that completed it (Agent S3 on Orgo).
Because the Orgo computer is the same in every case, you can try one loop, keep the computer, and switch the loop later.
Why do clicks land in the wrong place?
Because the coordinate space the model answers in does not match the computer's real screen. Every Orgo guide warns about the same failure: downscale a screenshot, or scale against the wrong resolution, and every click lands short of where the model aimed. Nothing errors. The agent looks like it is confidently clicking the wrong thing (Use any model).
Screen sizes differ across Orgo computers. A Linux computer created without a template or a resolution starts at 1280x720 (1280x720x24). A macOS computer defaults to 1920x1080. A computer launched from a template takes the template's resolution, and some curated templates use 1920x1080. You can also choose a resolution when you create a computer, or resize a screen later. So read the real size before you scale anything: it is the pixel size of an unresized screenshot, and on Linux GET /computers/{id}/screens reports each screen's width and height.
Leave the screenshot call's scale at its default of 1. A smaller scale returns a smaller image. Claude and OpenAI then answer in that smaller image's pixels, so every click lands short unless you map the coordinates back (Screenshot).
| Integration | What to send | How to avoid misclicks |
|---|---|---|
| Claude, current toolset | The screenshot, unresized | Pass coordinates straight through. If you resize anyway, divide them by your scale factor before you click (Claude guide) |
| Claude, older tools | The screenshot, unresized | Set display_width_px and display_height_px to the computer's width and height (Claude guide) |
OpenAI, computer tool | The screenshot, unresized, with "detail": "original" | Pass coordinates straight through (OpenAI guide) |
| OpenAI, preview tool | The screenshot, unresized | Set display_width and display_height to the computer's width and height (OpenAI guide) |
| Gemini | A PNG screenshot | Scale every coordinate: x / 1000 * width and y / 1000 * height, including start_x, start_y, end_x, and end_y on drags (Gemini guide) |
| Agent S3 | Raw image bytes, unresized | Leave grounding_width and grounding_height at the grounding model's documented values, not the computer's resolution. The Agent S README pairs UI-TARS-1.5-7B with 1920x1080 (Agent S3 guide). Set OSWorldACI's width and height to the computer's screen size, since the guide's script leaves them at 1920x1080 (Agent S3 on Orgo) |
What changes in my code when I switch models?
The Orgo side stays the same. The handler that turns model output into Orgo calls is what you rewrite:
| Handler piece | Claude | OpenAI | Gemini | Agent S3 |
|---|---|---|---|---|
| Read the action | tool_use blocks, with the member name in name and no action field | computer_call with an actions[] batch | function_call steps in interaction.steps | A pyautogui code string from agent.predict |
| Keys | computer.key(text) | Map keypress keys to X11 keysym names, for example ENTER to Return | Join hotkey keys with + and lowercase them | Inside the emitted code |
| Scroll | scroll_direction and scroll_amount | scroll_y // 100 as the amount | magnitude_in_pixels // 100 as the amount | Inside the emitted code |
| Return the result | One tool_result per tool_use, all in one user message, each computer result echoing toolset_name | A computer_call_output with the new screenshot | One function_result per call_id, each with a screenshot | The next predict call, with a new screenshot |
| Chain turns | Append to messages | previous_response_id | previous_interaction_id | Your step loop |
Sources: Claude guide, OpenAI guide, Gemini guide, Agent S3 guide. Orgo scrolls up or down only, so the Claude, OpenAI, and Gemini guides convert the model's scroll value into an Orgo amount, and a Gemini scroll left or right has no Orgo equivalent (Use any model).
Two Orgo-side notes apply to the guides' examples. They take screenshots over HTTP rather than through the Orgo SDK, and the Claude, OpenAI, and Gemini examples scroll over HTTP too, because SDK versions before 0.0.47 (Python) and 1.0.22 (TypeScript) cannot read the stored-image path the screenshot API returns by default, and their scroll() takes no coordinates. Those older SDKs also default to 2 GB of RAM, which POST /computers rejects with 400, so the examples that create a computer pass ram and cpu (Use any model). The TypeScript SDK is version 1 of the orgo npm package; from version 2, orgo on npm is the CLI, so pin orgo@1 (Claude guide).
Can Orgo run the computer use loop for me?
Yes, with Orgo's orgo-agent engine or one of the Claude models it accepts. POST https://www.orgo.ai/api/v1/chat/completions runs the agent loop on Orgo's side against the computer you name in computer_id, then returns the final assistant message. It follows OpenAI's chat completions format, so an OpenAI SDK pointed at https://www.orgo.ai/api/v1 can call it, with computer_id sent as an extra body field (Create chat completion).
What matters when you compare models:
- Default. Leave out
modeland Orgo runsorgo-agent(Orgo's computer use engine on GPT-6 Astra, fast tier) on Linux, Windows, macOS, and iPhone computers. It runsclaude-sonnet-5on Android computers, and whenever you send your own Anthropic key inx-anthropic-key. - Accepted models.
orgo-agent,orgo-agent-opus-5.5(the same engine on Claude Opus 5.5),claude-sonnet-5,claude-opus-5.5,claude-opus-5,claude-opus-4.8,claude-sonnet-4.6, andclaude-opus-4.6. Anything else returns400 invalid_model. The twoorgo-agentmodels do not run on Android computers (400 model_unsupported_os). Claude Sonnet 5.5 is not on the list, so run it in your own loop. - ID spelling. This endpoint uses dotted IDs such as
claude-sonnet-4.6. Anthropic's own API uses hyphens, such asclaude-sonnet-4-6.
OpenAI and Gemini models are not selectable here, even though orgo-agent runs on OpenAI's GPT-6 Astra. To use an OpenAI or Gemini model you choose, or Agent S3, run the loop yourself and use the Orgo API for computer actions. For threads, step limits, streaming, billing, and errors, see Orgo computer use API.
How should I evaluate computer use models for my own task?
Test the task you actually need to run. Benchmarks such as OSWorld help compare systems under defined conditions, but a result from a different setup is not a direct comparison, and a benchmark score is not a production success rate. For your own application, measure completed tasks, time and cost per completed task, and how often a person has to step in, and save failed runs as well as successful ones (What is computer use?).
A fair test gives every model the same starting state. On Orgo you can prepare one computer and copy it for each run (Pause, fork, or clone):
- Clone (
POST /computers/{id}/clone) copies the disk (files, installed apps, browser sessions) into a new computer that starts from a fresh boot. Use it when your task starts from a clean desktop. - Fork (
POST /computers/{id}/fork) also copies live memory, so open apps and windows start where the source was. It works on running Linux computers only.
Each copy is a new computer and counts against the workspace owner's plan (Clone computer). Templates are the other route to a reproducible starting computer: one file describes the setup, and every launch starts from the same build. Publishing your own template requires a Scale plan or higher (Orgo templates and triggers).
FAQ
Which computer use models work with Orgo? Claude, OpenAI, and Gemini each have an Orgo guide, and Agent S3 is the worked open-source example. Any model that outputs structured actions such as click, type, and screenshot can follow the same pattern (Use any model).
Do Claude and OpenAI need a display size on the request?
No. Claude's computer_toolset_20260801 and OpenAI's computer tool take no display size and return coordinates in the pixel space of the screenshots you send. Only the older tool versions declare one (Claude guide, OpenAI guide).
Why does Gemini need extra code?
Gemini returns coordinates on a 0-999 grid, so you scale every coordinate to the computer's real width and height before you click. Send it PNG screenshots, as Google's examples and Orgo's guide do; the screenshot call takes format=png (Gemini guide).
Is Agent S3 a model?
No. It is Simular's open-source framework, distributed as gui-agents. It pairs a reasoning model with a grounding model and emits pyautogui code that Orgo runs inside the computer (Agent S3 guide).
Can I use GPT or Gemini with Orgo's hosted chat completions endpoint?
Not directly. The endpoint accepts orgo-agent, orgo-agent-opus-5.5, and a fixed list of Claude models, and returns 400 invalid_model for anything else. The default orgo-agent runs on OpenAI's GPT-6 Astra, but you cannot name a GPT or Gemini model there. To drive Orgo with one, run your own loop with the OpenAI guide or Gemini guide.
Do I need a different Orgo computer for each model? No. Every integration uses the same kind of computer and the same Orgo API, so one computer can be driven by different models over time (Use any model).
Which model is most accurate? Orgo does not publish a computer use ranking. Check each provider's current documentation and benchmark write-ups, then test your own task.