Set Up Agent S2: An Open-Source Computer Use Agent

Agent S2 is an open-source computer use agent from Simular. It screenshots your desktop, decides what to click, and drives the mouse and keyboard. This guide runs it locally, then on a cloud desktop.

Two defaults are broken as written:

• gui-agents now installs 0.3.x, the Agent S3 line. Pin it to 0.2.x to get Agent S2.
• The example repo's grounding model, claude-3-7-sonnet-20250219, was retired on 2026-02-19 and returns an error. Set GROUNDING_MODEL yourself.

The agent is experimental and slow. Expect to supervise it.

What is Agent S2?

A compositional framework: one model plans, a second finds pixel coordinates on screen. Simular published it in April 2025 at 34.5% on OSWorld over 50 steps, the leading result then. Agent S3 has since scored 69.9% at 100 steps, and Simular now reports 72.6% for Agent S, past the 72.36% human baseline. The s2 path is still maintained upstream.

How Agent S2 Works

1. Capture a screenshot
2. The planning model picks an action, such as click firefox
3. The grounding model returns coordinates for that target
4. Python clicks at (x, y)
5. Repeat until done or the step budget runs out

Mixture-of-grounding routes each target to a visual, textual, or structural expert; the textual one is OCR, hence the Tesseract prerequisite.

The example repo pairs GPT-4o for planning with Claude for grounding. That is the repo's choice, not Simular's: upstream Agent S recommends gpt-5-2025-08-07 for planning and a self-hosted UI-TARS-1.5-7B for grounding, which drops the Anthropic dependency.

Quickstart

Prerequisites

• Python 3.10 to 3.12 (gui-agents declares >=3.9, <=3.12)
• Tesseract: brew install tesseract, or apt install tesseract-ocr
• An OpenAI API key and an Anthropic API key

Step 1: Install

git clone https://github.com/spencerkinney/agent-s2-example.git
cd agent-s2-example
python3 -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate

requirements.txt ships unpinned. Edit the bare gui-agents line to gui-agents>=0.2,<0.3, then install:

pip install -r requirements.txt

Step 2: Configure API Keys

cp .env.example .env

OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
GROUNDING_MODEL=claude-sonnet-5

The last line is required: the repo's default is retired, and claude-sonnet-5 is Anthropic's suggested successor.

Step 3: macOS Permissions

The agent needs Accessibility access to move the cursor. Under System Settings, Privacy & Security, Accessibility, add the app you launch the script from.

Step 4: Run

python agent_s2.py

Commands that work well:

"Open Chrome"
"Create a folder called projects on the desktop"
"Search for weather in San Francisco"

Type exit to quit.

Run Agent S2 on a Cloud Desktop

Point the agent at a remote Linux desktop instead of your own machine. Orgo runs cloud computers for AI agents, and the orgo client is already a repo dependency. Sign up at orgo.ai/start, add two lines to .env, and rerun.

ORGO_API_KEY=your_orgo_key
USE_CLOUD_ENVIRONMENT=true

Configuration

Model Selection

# Split roles (the repo's default pairing)
AGENT_MODEL=gpt-4o                  # Planning
AGENT_MODEL_TYPE=openai
GROUNDING_MODEL=claude-sonnet-5     # Grounding
GROUNDING_MODEL_TYPE=anthropic
 
# Or use Claude for both
AGENT_MODEL=claude-sonnet-5
AGENT_MODEL_TYPE=anthropic

Step Budget and Pacing

MAX_STEPS=20      # Default: 10. Raise for multi-step tasks.
STEP_DELAY=0.1    # Default: 0.5s. Lower to run faster.

Troubleshooting

"Module not found"

The virtual environment is not active. See Step 1.

macOS: "Permission denied"

Terminal needs Accessibility permissions. See Step 3.

The agent clicks the wrong location

Grounding runs in a fixed coordinate space scaled to your display. Set scaling to 100% and use a standard resolution such as 1920x1080.

Safety

Agent S2 has full control of the mouse and keyboard. Watch every session, and use cloud desktop mode for anything you would not want run against your own filesystem.

Additional Resources

• Agent S2 paper
• Agent-S repository
• Example code for this guide