How to run Agent S3 on an Orgo cloud computer

Agent S3 is Simular's open-source computer use agent, distributed as the gui-agents Python package. Out of the box it controls the machine it runs on. With a small executor class, it controls an Orgo computer instead: Agent S3 plans and grounds each step, and Orgo is the desktop where the screenshots come from and the actions run.

This post follows the Agent S3 guide in the Orgo docs and notes two things to check or fix in its script. For the previous generation, see Running Agent S2 on a remote desktop.

What is Agent S3 with Orgo?

Agent S3 is the third generation of Simular's Agent S framework, and the current open-source one as of October 2026 (gui-agents 0.3.x). Compared with Agent S2, it drops the manager and worker hierarchy, adds a native coding agent that can write and run code alongside GUI actions, and adds Behavior Best-of-N, which runs several attempts at a task and has a judge pick the one that completed it.

Orgo provides cloud computers for AI agents: a full desktop with a browser, reachable over HTTP and VNC. In this setup Orgo is the computer and you bring the agent: Agent S3, Claude, OpenAI, Gemini or your own loop. Linux and Windows computers are self-serve; Mac and iPhone are available by request.

Running Agent S3 with Orgo means the agent's decisions stay in your Python process, while every screenshot and every action happens on the Orgo computer.

Code written for Agent S2 imports from gui_agents.s2. Agent S3 lives under gui_agents.s3, and its top-level class is AgentS3. Update both when you upgrade.

For background on agents that operate interfaces, see What is computer use?. For when a full desktop is worth it over a browser, see Cloud desktops vs. headless browsers.

How does Agent S3 differ from Agent S2 and provider computer use?

Use Agent S3 for new gui-agents projects. The S2 post is only useful if you maintain code from that generation.

Local or remote: where does Agent S3 run?

The agent_s command that ships with the package always drives the machine it runs on. To control an Orgo computer, use the guide's agent_s3.py script with USE_CLOUD_ENVIRONMENT=true.

How does coordinate space work with Agent S3 on Orgo?

Agent S3 never hands you a click coordinate to translate. It emits pyautogui code with the coordinates already in it, and RemoteExecutor runs that code on the computer through POST /computers/{id}/exec. The scaling happens inside Agent S3: for each point the grounding model returns, OSWorldACI multiplies x by width / grounding_width and y by height / grounding_height. Both pairs have to fit your setup.

• width and height are the screen the clicks land on. They are arguments to OSWorldACI and default to 1920x1080, and Agent S's own CLI sets them to your screen's size. A Linux computer created without a template or a resolution boots at 1280x720, and GET /computers/{id}/screens lists each screen with its width and height. The guide's create_agent does not pass them, so they stay at 1920x1080. To change them, add width= and height= to its OSWorldACI(...) call.
• grounding_width and grounding_height are the coordinate space the grounding model answers in. Agent S's CLI describes them as the size of the screenshot after the model's processor rescales it. The Agent S README pairs UI-TARS-1.5-7B with 1920x1080 and UI-TARS-72B with 1000x1000.

Test one click before a real task: ask Agent S3 to click something you can see, and watch where the pointer lands in the Orgo dashboard. If clicks miss by more the further the target is from the top-left corner, the two pairs don't fit together. Send screenshots at full size, as the guide does, and if a prompt mentions a resolution, make it the computer's real one.

How do you set up Agent S3 on Orgo?

These steps follow the Agent S3 guide. gui-agents supports Python 3.9 through 3.12.

1. Install the packages

pip install gui-agents pyautogui python-dotenv orgo

gui-agents uses pytesseract, and the upstream README asks you to install Tesseract too (brew install tesseract on macOS).

2. Set your API keys

export OPENAI_API_KEY=your_openai_api_key
export HF_TOKEN=your_huggingface_token
export ORGO_API_KEY=your_orgo_api_key  # remote mode only

OPENAI_API_KEY is for the main model. HF_TOKEN is for a grounding model hosted on Hugging Face Inference Endpoints. ORGO_API_KEY is for the Orgo API at https://www.orgo.ai/api; keys start with sk_live_, and you only need one when USE_CLOUD_ENVIRONMENT=true. Sign up, then create a key under API keys in the Orgo dashboard.

3. Add the remote executor

The guide's agent_s3.py has two executors with the same three methods: screenshot(), exec(code) and destroy(). LocalExecutor uses your own screen. RemoteExecutor is the Orgo half:

import os
import requests
from orgo import Computer
 
 
class RemoteExecutor:
    def __init__(self):
        # Pass a size: SDKs before 0.0.47 default to 2 GB, which the API rejects.
        self.computer = Computer(ram=4, cpu=1)
        self.platform = "linux"
        # The Linux image ships python3 without pip or pyautogui, and the
        # actions Agent S3 emits import pyautogui. Install it once.
        self.computer.bash(
            "apt-get update -qq && apt-get install -y -qq python3-pip && "
            "pip3 install --break-system-packages -q pyautogui"
        )
 
    def screenshot(self):
        # The image bytes themselves, at full size.
        response = requests.get(
            f"https://www.orgo.ai/api/computers/{self.computer.computer_id}/screenshot",
            params={"response_format": "binary"},
            headers={"Authorization": f"Bearer {os.environ['ORGO_API_KEY']}"},
        )
        response.raise_for_status()
        return response.content
 
    def exec(self, code):
        result = self.computer.exec(code)
        if not result["success"]:
            raise Exception(result.get("error", "Execution failed"))
        if result["output"]:
            print(f"Output: {result['output']}")
 
    def destroy(self):
        self.computer.destroy()

Three things to know about it:

• Size. Orgo Python SDK versions before 0.0.47 send 2 GB when you name no size, and the API refuses that: RAM must be 4, 8, 16, 32 or 64 GB. That is why the example names one. From 0.0.47, leaving out both ram and cpu gives you 4 GB and 1 vCPU.
• Screenshots. Agent S3 needs image bytes. response_format=binary returns them directly, whatever SDK version you have. The SDK's screenshot helpers before 0.0.47 cannot read the stored-image path the endpoint returns by default.
• Cleanup. The guide's script calls destroy() when it exits, which deletes the computer and everything on it. Remove that call if you want to keep the computer.

Computer(ram=4, cpu=1) creates a computer on your plan, so this path needs a paid plan (pricing). A free account can't create plan computers. Its $5 signup credit runs one Medium Linux computer by the hour instead: launch that computer from the dashboard, connect to it with Computer(computer_id="your_computer_id"), and remove the destroy() call.

The complete agent_s3.py, with LocalExecutor, create_agent and the task loop, is in the Agent S3 guide.

4. Run a task

# Local: controls your own machine
python agent_s3.py "Open Chrome and search for weather"
 
# Remote: controls an Orgo computer
USE_CLOUD_ENVIRONMENT=true python agent_s3.py "Open Chrome"
 
# Interactive: asks for tasks until you type exit
python agent_s3.py

Watch the first remote run in the Orgo dashboard and check the result yourself. Agent S3 ends a task by returning DONE or FAIL as its action, and Agent S's own CLI stops on either. The guide's run_task stops only on an empty action or after MAX_STEPS, so it sends DONE to the computer as Python, gets an error back and keeps going. Add this check inside its loop, after the empty-action check:

if action[0] in ("DONE", "FAIL"):
    print(action[0])
    return action[0] == "DONE"

After any failed action, the guide's loop also replaces your task with "The previous action failed. Try a different approach." If you want the agent to keep the task in view, include it in that message.

The package's own CLI, agent_s, drives the machine it runs on, not an Orgo computer:

agent_s \
    --provider openai \
    --model gpt-5-2025-08-07 \
    --ground_provider huggingface \
    --ground_url http://localhost:8080 \
    --ground_model ui-tars-1.5-7b \
    --grounding_width 1920 \
    --grounding_height 1080

5. Optional: turn on the coding agent

Pass a LocalEnv from gui_agents.s3.utils.local_env as OSWorldACI(env=...) to turn on the coding agent. It runs code on the host running your script. With RemoteExecutor, that host is your laptop, not the Orgo computer, and Simular warns that the coding agent executes arbitrary Python and Bash with your user's permissions. Leave env=None unless that is what you want.

What environment variables does Agent S3 use?

The guide's agent_s3.py reads these:

Point GROUNDING_URL at wherever your grounding model runs, such as a Hugging Face Inference Endpoint serving UI-TARS-1.5-7B. The model defaults are the pairing the Orgo guide and the Agent S README recommend as of October 2026. Model IDs change upstream, so check that your providers still serve them.

What is in the Agent S3 architecture?

• Native coding agent (new in S3): writes and runs code alongside GUI actions, so tasks better done in a shell stop being click sequences.
• Behavior Best-of-N (new in S3): runs several attempts at a task, summarizes each as a behavior narrative, and has a judge pick the one that completed it. Its pieces are in the package's gui_agents.s3.bbon module; the agent_s CLI and the guide's script run Agent S3 without it.
• Mixture of grounding (from S2): routes actions to specialized visual grounding models to locate UI elements. It was introduced in the Agent S2 paper and is still part of the design.
• Cross-platform: runs on macOS, Windows and Linux. In local mode each platform's setup applies: Accessibility access for your terminal on macOS, possibly an Administrator terminal on Windows, and python3-tk and python3-dev on Linux.

For benchmark results, see Simular's Agent S3 writeup and the OSWorld leaderboard.

How does this compare with Claude, OpenAI and Gemini computer use?

Orgo is the same computer whichever agent drives it:

• Claude: Run Claude computer use on Orgo
• OpenAI: OpenAI computer use with Orgo
• Gemini: Gemini computer use with Orgo
• Side by side: Computer use models compared

Pick Agent S3 when you want Simular's open-source loop (a coding agent and a separate grounding model) on an Orgo desktop. Pick a provider's computer use tool when you want one model to read the screen and return actions directly.

What should you know before you run it?

• You run Agent S3 yourself. Orgo provides the computer, and you pay your model providers directly for Agent S3's model calls.
• The remote example needs a paid plan as written. On a free account, connect to your hourly computer by id instead.
• Check the guide's script in two places. Test one click to confirm that OSWorldACI's width and height fit your grounding settings, and make the loop stop on DONE or FAIL.
• The coding agent runs on your machine when you pass LocalEnv with RemoteExecutor.
• The guide's RemoteExecutor targets Linux. It sets platform = "linux" and installs pyautogui with apt-get.
• One screen. Agent S is designed for a single monitor, and the executor works on the computer's boot screen.
• The guide's URL still ends in agent-s2, though the page covers Agent S3. Its video was recorded against Agent S2; the written imports and class names are the Agent S3 ones.

FAQ

Is Agent S3 the same as Agent S2? No. Agent S3 imports from gui_agents.s3 and its class is AgentS3. It drops the S2 manager and worker hierarchy and adds a coding agent and Behavior Best-of-N.

Do I need to translate click coordinates on Orgo? Not yourself. Agent S3 emits pyautogui code with the coordinates in it, scaled by OSWorldACI's width and height (1920x1080 unless you pass them) and the grounding model's grounding_width and grounding_height. Set those for your computer and test one click first.

What size should I pass to Computer()? The guide uses ram=4, cpu=1. The API accepts 4, 8, 16, 32 or 64 GB of RAM; your plan sets the largest you can create, and computers past 32 GB are set up on request. Name a size on SDK versions before 0.0.47, which default to 2 GB; the API rejects that.

Can I run it on a free account? Not as written, because Computer(ram=4, cpu=1) creates a computer on a plan. A free account's $5 signup credit runs one Medium Linux computer by the hour. Launch it from the dashboard, pass its id to Computer(computer_id=...), and remove the destroy() call so the script doesn't delete it.

Where does the coding agent run in remote mode? On the host running your Python script, typically your laptop, not on the Orgo computer. Leave env=None unless that is what you want.

Should new projects still follow the Agent S2 post? No. Use this page and the Agent S3 guide. Keep the Agent S2 post for maintaining S2 code.

Where are benchmark results? In Simular's Agent S3 writeup, the Agent S repository and the OSWorld leaderboard.