Gemini computer use with Orgo

Gemini computer use with Orgo means connecting Google's computer_use tool to an Orgo cloud computer. Gemini looks at a screenshot and returns a batch of actions, and your loop runs them on a real Linux desktop that keeps running between tasks.

As of October 2026, Google recommends gemini-3.8-flash for computer use, called through the Interactions API (Gemini computer use). For an Orgo computer, you declare the tool with environment: "desktop". Coordinates are the part that fails without an error: Gemini answers on a 0-999 grid, and your code has to scale every x and y to the screen's real size. This post covers that, a complete loop, and what changed from the 2.5 preview model. The Orgo docs have a Gemini guide as well.

What is Gemini computer use with Orgo?

Gemini computer use is a tool you turn on in a request to a supported Gemini model, not a separate model. The model sees a screenshot and returns function_call steps such as click, type, or scroll. Your code carries them out and sends back a fresh screenshot. Google's computer use page lists gemini-3.8-flash as the recommended model, and also lists gemini-4-argon, gemini-3.5-flash-lite, and gemini-3-flash-preview.

Orgo provides cloud computers for AI agents: a full desktop with a browser and the standard userland, reachable over HTTP and VNC, that runs until you stop it (Orgo docs). In this setup, Orgo provides the computer and you bring the model loop, whether that is Gemini, Claude, OpenAI, Agent S3, or your own. If you would rather send a task and let Orgo run the agent, see the Orgo computer use API.

So Gemini decides what to do, and the Orgo computer is where the actions land. This post assumes a Linux computer.

How is Gemini computer use different from Claude and OpenAI on Orgo?

All three run against the same Orgo computer. What changes is the API, the tool, and the coordinate system. As of October 2026:

Sources: Gemini computer use, Claude computer use tool, OpenAI computer use, and Orgo's OpenAI guide.

If you port a Claude or OpenAI loop to Gemini, the API calls change, and so does every handler that takes a coordinate. For a wider comparison that includes Agent S3, see Computer use models compared. Run OpenAI computer use on Orgo and the Claude guide cover the other two loops.

Why do Gemini clicks land in the wrong place?

Because the scaling does not match the screen. Gemini returns coordinates on a 0-999 grid, and there is no display size to declare on the request. Your code is the only thing turning that grid into pixels, with x / 1000 * width and y / 1000 * height (Gemini computer use).

Get width and height wrong and nothing errors. The agent just clicks the wrong thing. Scale with 1024x768 on a 1280x720 screen and every horizontal position comes out 20% short: a button at x=1200 gets clicked near x=960.

A hard-coded size is fragile because Orgo computers do not all share one:

• A Linux computer created without a template or a resolution boots at 1280x720 (Create computer).
• A computer made from a template takes the template's size. The Hermes Agent, OpenClaw, and Coding templates boot at 1920x1080.
• A macOS computer defaults to 1920x1080.
• On Linux, a screen can be resized while the computer runs (Resize a screen).

The reliable fix is to scale against the screenshot you just sent Gemini, since its size is the screen's size:

import base64
import io
 
from orgo import Computer
 
computer = Computer(computer_id="your-computer-id")  # reads ORGO_API_KEY
 
def screenshot():
    """Return the screen as base64 PNG, plus its (width, height)."""
    image = computer.screenshot()  # a PIL image
    buffer = io.BytesIO()
    image.save(buffer, format="PNG")
    return base64.b64encode(buffer.getvalue()).decode(), image.size
 
def to_pixels(x, y, size):
    width, height = size
    return int(x / 1000 * width), int(y / 1000 * height)

Scale x, y, start_x, start_y, end_x, and end_y on every action that has them. If you call the HTTP API instead of the SDK, GET /computers/{id}/screenshot?response_format=base64 returns width and height alongside the image (Take screenshot).

How do I set up Gemini computer use on an Orgo computer?

1. Install the packages. Google's computer use docs call for google-genai 2.7.0 or later. The code here also needs Orgo's Python SDK 0.0.47 or later, the first release whose screenshot() reads the API's base64 response and whose scroll() takes a position: pip install "orgo>=0.0.47" "google-genai>=2.7.0".
2. Set your keys. Export ORGO_API_KEY and GEMINI_API_KEY. Orgo keys begin with sk_live_.
3. Get a computer ID. It is the id that Create computer returns, or the last part of the computer's page URL in the dashboard, https://www.orgo.ai/computers/{id}. Pass it as Computer(computer_id="...").
4. Declare the tool. Use {"type": "computer_use", "environment": "desktop", "enable_prompt_injection_detection": True}. Pick desktop, since an Orgo computer is a full desktop. Prompt injection detection is off unless you turn it on.
5. Write a system instruction for the desktop. Tell Gemini to double-click desktop icons, since a single click only selects them. Send it with every request: the Interactions API applies system_instruction and tools to one interaction only, and previous_interaction_id carries over only the conversation history (Interactions API).
6. Run the loop below until a turn comes back with no function calls.

What does the Gemini agent loop look like?

Each turn has the same five steps (Gemini computer use):

1. Request. Send the task and a PNG screenshot with client.interactions.create(...).
2. Actions. Gemini returns function_call steps in interaction.steps. Each one has an id, a name, and arguments, and a turn can hold several.
3. Execute. Run every call in order on the Orgo computer.
4. Screenshot. Send back one function_result per call, with call_id set to that call's id, each carrying the new screenshot.
5. Repeat. Chain turns with previous_interaction_id until no call comes back. The final answer is in the text blocks of a model_output step.

Every action's arguments include an intent string that says why Gemini chose it. Log it: when the agent does something odd, it tells you what Gemini thought it was doing.

Here is the whole loop, using the screenshot() and to_pixels() helpers above:

import json
import os
import time
 
from google import genai
 
client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
 
MODEL = "gemini-3.8-flash"
TOOLS = [{
    "type": "computer_use",
    "environment": "desktop",
    "enable_prompt_injection_detection": True,
}]
SYSTEM = (
    "You are controlling an Ubuntu desktop. Double-click desktop icons and "
    "file browser items to open them. Single-click menus, buttons, and "
    "taskbar icons. Press Enter to submit."
)
 
def keysym(name):
    # On Linux, Orgo's key endpoint takes case-sensitive X11 keysym names.
    return {"enter": "Return", "backspace": "BackSpace", "escape": "Escape"}.get(name.lower(), name)
 
def run(name, args, size):
    if name in ("click", "double_click", "right_click"):
        x, y = to_pixels(args["x"], args["y"], size)
        {"click": computer.left_click,
         "double_click": computer.double_click,
         "right_click": computer.right_click}[name](x, y)
    elif name == "type":
        computer.type(args["text"])
        if args.get("press_enter"):
            computer.key("Return")
    elif name == "press_key":
        computer.key(keysym(args["key"]))
    elif name == "hotkey":
        computer.key("+".join(keysym(key) for key in args["keys"]))
    elif name == "scroll":
        if args["direction"] not in ("up", "down"):
            return {"error": "Only up and down scrolling is supported."}
        x, y = to_pixels(args["x"], args["y"], size)
        clicks = max(1, int(args.get("magnitude_in_pixels", 300)) // 100)
        computer.scroll(args["direction"], clicks, x=x, y=y)
    elif name == "drag_and_drop":
        x1, y1 = to_pixels(args["start_x"], args["start_y"], size)
        x2, y2 = to_pixels(args["end_x"], args["end_y"], size)
        computer.drag(x1, y1, x2, y2)
    elif name == "wait":
        computer.wait(args.get("seconds", 1))
    elif name != "take_screenshot":
        return {"error": f"Unsupported action: {name}"}
    return {}
 
def needs_confirmation(args):
    decision = args.get("safety_decision") or {}
    return decision.get("decision") == "require_confirmation"
 
task = "Open Chrome and search for 'gemini computer use'"
image, size = screenshot()
interaction = client.interactions.create(
    model=MODEL,
    system_instruction=SYSTEM,
    tools=TOOLS,
    input=[
        {"type": "text", "text": task},
        {"type": "image", "data": image, "mime_type": "image/png"},
    ],
)
 
for turn in range(20):
    calls = [step for step in interaction.steps if step.type == "function_call"]
    if not calls:
        print(" ".join(
            block.text
            for step in interaction.steps if step.type == "model_output"
            for block in step.content if block.type == "text"
        ))
        break
 
    results = []
    for call in calls:
        print(f"{call.name}: {call.arguments.get('intent', '')}")
        confirm = needs_confirmation(call.arguments)
        if confirm and input(f"Allow {call.name}? [y/N] ").lower() != "y":
            raise SystemExit("Stopped: the action was not approved.")
        try:
            result = run(call.name, call.arguments, size)
        except Exception as error:
            result = {"error": str(error)}
        if confirm:
            result["safety_acknowledgement"] = True
        results.append((call, result))
 
    time.sleep(1)  # let the screen settle
    image, size = screenshot()
    interaction = client.interactions.create(
        model=MODEL,
        system_instruction=SYSTEM,
        tools=TOOLS,
        previous_interaction_id=interaction.id,
        input=[
            {
                "type": "function_result",
                "name": call.name,
                "call_id": call.id,
                "result": [
                    {"type": "text", "text": json.dumps(result)},
                    {"type": "image", "data": image, "mime_type": "image/png"},
                ],
            }
            for call, result in results
        ],
    )
 
# The computer keeps running after the loop ends.

Which Gemini actions map to which Orgo calls?

Two details are worth knowing:

• Key names. On Linux, Orgo passes key names to the computer unchanged, so they must be X11 keysym names, which are case-sensitive: Return, BackSpace, Escape. A misspelled key returns a 500 (Press key). Log what Gemini sends and map any name that differs, as keysym() does above.
• Scrolling. Orgo scrolls up or down in wheel clicks, so the loop treats Gemini's magnitude_in_pixels (300 by default) as a hint and converts it to clicks. Always pass x and y. On Linux the scroll endpoint moves the pointer before it scrolls, and a missing coordinate moves it to 0 (Scroll). Gemini can also ask to scroll left or right, which Orgo does not do.

The desktop environment also has triple_click, middle_click, move, mouse_down, mouse_up, key_down, and key_up. The loop above answers those with an error result. To keep Gemini from choosing actions your loop does not handle, list them in the tool's excluded_predefined_functions (Gemini computer use).

How do Gemini safety confirmations work on Orgo?

An action's arguments can include a safety_decision. When its decision is require_confirmation, ask your user before running the action. If they approve, run it and set safety_acknowledgement to true in that call's result. If they decline, do not run it. Google's docs also describe decisions that block an action and say your client should halt when one comes back (Gemini computer use). The loop above checks only for require_confirmation, so add that stop before you leave it running unattended.

The Orgo side of safety is isolation. The agent works inside its own cloud computer rather than on your laptop, so its clicks, downloads, and typing stay on that computer.

Should I migrate from gemini-2.5-computer-use-preview?

Yes. Google's deprecations page gives July 28, 2026 as the shutdown date for gemini-2.5-computer-use-preview-10-2025, and that date has passed. The replacement it lists is gemini-3.8-flash (Gemini deprecations).

If your code names gemini-3.7-flash, it keeps working: the same deprecations page says Google routes that ID to gemini-3.8-flash.

The Orgo side changes little when you migrate. The computer and the 0-999 scaling stay the same, and your handler only has to match the new action names.

What goes wrong most often?

• Desktop icons do not open. Gemini is single-clicking them. Put the double-click rule in system_instruction, and send it with every request, not only the first.
• Clicks land in the wrong place. The size you scale against does not match the screen. Scale against the size of the screenshot you sent.
• A key press returns 500. The key name is not an X11 keysym, or its case is wrong, such as return for Return.
• Scrolling happens in the top-left corner. A scroll request to the API had no x and y.
• Missing API key. Set both ORGO_API_KEY and GEMINI_API_KEY.

For other Orgo API errors, the troubleshooting guide maps common errors to the next step.

When is Gemini on Orgo the right fit?

Pick it when you want Google's model driving a full desktop, not just a browser tab, and you want that computer to persist between runs. Browser-only tools give an agent a tab. Orgo gives it a whole computer. See Cloud desktops vs. headless browsers for that tradeoff.

Orgo works with any model, so the same computer can run Gemini today and Claude or OpenAI tomorrow, and you can compare them on your own tasks. The Use any model guide covers the pattern.

FAQ

Is Gemini computer use a separate model? No. It is a tool you turn on in the request. As of October 2026, Google recommends gemini-3.8-flash for it, through the Interactions API.

What resolution does an Orgo computer use? A Linux computer created without a template or a resolution boots at 1280x720. The Hermes Agent, OpenClaw, and Coding templates boot at 1920x1080, and macOS computers default to 1920x1080. Scale Gemini's coordinates against the size of each screenshot rather than a constant.

What image format should I send? PNG, as Google's computer use examples do. Orgo's Linux screenshots are PNG already, and the screenshot() helper re-encodes whatever it gets as PNG.

Should I use the desktop or browser environment? desktop. An Orgo computer is a full desktop. browser adds page actions such as navigate, go_back, and go_forward, and Google's browser example also returns the current URL in each result.

Does the computer shut down when the loop ends? No. By default an Orgo computer keeps running until you stop, suspend, or delete it. computer.destroy() deletes it. To keep its state for later, stop or suspend it instead. See How do you pause, fork, or clone an AI agent's computer?

Does Orgo work with other computer use models? Yes. Claude, OpenAI, Agent S3, or your own loop can drive the same computer, and the Hermes Agent and OpenClaw templates run an agent on the computer itself.

Related

• What is computer use?
• Orgo computer use API: run an AI agent with one request
• How to run Agent S3 on an Orgo cloud computer