How do you watch what your AI agent is doing in real time?

TL;DR

Every Linux or Windows computer on Orgo exposes four live WebSocket streams on one host, all authenticated with the same per-computer password:

• Screen (/ws/websockify): the live desktop over VNC, for a human to watch or take over.
• Events (/ws/events): window focus, windows opening and closing, clipboard, processes starting and stopping, screen size, audio streams, and idle or active state, pushed as JSON the moment they happen.
• Terminal (/ws/terminal): an interactive root shell (a full PTY) you can attach to and read.
• Audio (/ws/audio): the computer's speaker as raw PCM in 20 ms frames.

The URL pattern is wss://www.orgo.ai/desktops/{id}/ws/{stream}?token={password}. Your server reads the password with your API key from GET /computers/{id}/vnc-password and never ships the API key to a browser (Events WebSocket, Terminal WebSocket, Audio WebSocket, Embed computers).

For most agent monitoring, start with the Events stream. It is small, structured, has no idle timeout, and tells you what the agent is doing ("focus moved to Google Chrome") without decoding pixels.

What does "watching an AI agent in real time" mean?

Real-time agent monitoring is following what an autonomous agent does on its computer while it does it, rather than reading a log after the run. On a desktop agent, the useful signals are the screen itself, which window has focus, which programs start and stop, what lands on the clipboard, what the shell prints, and whether the desktop has gone quiet.

On Orgo, a computer is a full desktop with a browser, reachable over HTTP and VNC (llms.txt). Orgo provides the computer, not the agent, so these streams work the same whether Claude computer use, OpenAI's CUA, Hermes Agent, OpenClaw, or your own loop is driving it. New to agents on a desktop? Start with What is computer use?

Which stream should I use?

Sources: Embed computers, Events WebSocket, Terminal WebSocket, Audio WebSocket, Take screenshot.

How do I connect to a computer's live streams?

Two steps, and the first one belongs on your server.

1. Read the computer password with your API key.

curl https://www.orgo.ai/api/computers/$COMPUTER_ID/vnc-password \
  -H "Authorization: Bearer $ORGO_API_KEY"

The response carries password. GET /computers/{id} also returns it as vnc_password. It can change, for example when a computer is started again from its archive or after an API key that could read it is revoked, so fetch it for each session instead of storing it. While a change is in progress the call answers 503 with Retry-After: 30; wait and ask again (Get VNC password, Embed computers). This endpoint takes the computer's UUID only.

2. Open the stream with the password as token.

wss://www.orgo.ai/desktops/{id}/ws/events?token={password}

Build the URL from the computer's id (UUID). Its instance_id also resolves, but it changes when the computer is started from its archive or wakes from storage (Terminal WebSocket).

Three credential rules that trip people up:

• The password goes in ?token=, not a header. On the events, terminal and audio streams, an Authorization header is accepted by Orgo but not forwarded to the computer, so the socket opens and then closes with 1000.
• An API key does not work here. On those streams an sk_ key in ?token= fails the same way, and from a browser (any connection with an Origin header) it is refused with 4001.
• The password is root on that computer. The same value opens the terminal and the screen, so only hand it to people and pages that are allowed to control the computer (Embed computers).

Only the owner and Full access members can read the password. A Can chat member gets 403, and so do an MCP sign-in and a computer's own identity token (Get VNC password). Signed in to Orgo, a Can chat member can still watch the screen and open the audio stream with their session; the terminal and events streams close with 4001 for them. Despite the name, those members can't chat with the workspace's agents either. That takes Full access.

How do I get a live feed of what the agent is doing?

Use the Events WebSocket. A new connection receives nothing until it sends a subscribe message, so you opt in to only the types you need (Events WebSocket).

{ "type": "subscribe", "event_types": ["window_focus", "process_start", "process_stop", "idle", "active"] }

Each event then arrives as:

{
  "type": "event",
  "event": {
    "type": "window_focus",
    "timestamp": "2026-03-03T12:00:00.123Z",
    "data": { "window_id": "52428804", "window_name": "Google Chrome" }
  }
}

The Linux catalog and how often each is checked:

Source: Events WebSocket.

What to know before you build on it:

• Events describe changes, not state. Connecting does not replay what you missed. Watchers run for the life of the computer, independent of your connection.
• Names are not validated. A typo in event_types still returns subscribed / ok and then delivers nothing.
• Slow consumers lose events. The per-connection buffer is 256 events; extra events are dropped, not queued. Hand them off to a queue quickly.
• file_change is not delivered on Linux. To notice a finished download, poll the folder with Execute bash instead.
• Windows emits less. A Windows computer sends only window_focus, clipboard, and file_change, and Windows file_change events are delivered for the Administrator's Desktop and Downloads folders.
• No idle timeout. Unlike the terminal and audio streams, a long-lived events connection is not closed for inactivity.

How do I log every agent action to my own system?

Run a small watcher next to your agent loop that writes each event as a JSON line. This one fetches the password again before every connect, re-subscribes (subscriptions are per connection and are not restored), and backs off exponentially, as the docs recommend.

It does not give up on 4001. The password can change while the watcher runs, and a socket that offers an old password, or one that is being changed, closes with 4001 (an open socket closes with 4003 'Screen password changed'). Fetching the password again and retrying with backoff is the fix. It also keeps going when the password fetch or the connect fails, instead of crashing.

import asyncio, json, os
import requests, websockets
 
API = "https://www.orgo.ai/api"
COMPUTER_ID = os.environ["COMPUTER_ID"]  # the computer's UUID
HEADERS = {"Authorization": f"Bearer {os.environ['ORGO_API_KEY']}"}
EVENT_TYPES = ["window_focus", "window_open", "window_close",
               "process_start", "process_stop", "idle", "active"]
FINAL = {4004, 4005}  # no such computer, computer locked
 
class TryLater(Exception):
    def __init__(self, seconds: float):
        self.seconds = seconds
 
def computer_password() -> str:
    r = requests.get(f"{API}/computers/{COMPUTER_ID}/vnc-password",
                     headers=HEADERS, timeout=15)
    if r.status_code == 429 or r.status_code >= 500:  # 503 while it changes
        raise TryLater(float(r.headers.get("Retry-After") or 0))
    r.raise_for_status()  # 401, 403, 404: fix the key, its scope, or the id
    return r.json()["password"]
 
def log(**fields) -> None:
    print(json.dumps(fields), flush=True)
 
async def watch() -> None:
    delay = 2
    while True:
        wait = 0.0
        try:
            password = await asyncio.to_thread(computer_password)
            url = f"wss://www.orgo.ai/desktops/{COMPUTER_ID}/ws/events?token={password}"
            async with websockets.connect(url) as ws:
                await ws.send(json.dumps({"type": "subscribe", "event_types": EVENT_TYPES}))
                async for raw in ws:
                    msg = json.loads(raw)
                    if msg["type"] == "event":
                        delay = 2
                        print(json.dumps(msg["event"]), flush=True)
        except TryLater as e:
            wait = e.seconds
            log(password="try later", retry_after=wait)
        except requests.HTTPError:
            raise  # a refused key will not fix itself
        except requests.RequestException as e:
            log(password_fetch_failed=repr(e))
        except websockets.ConnectionClosed as e:
            code = e.rcvd.code if e.rcvd else None
            log(closed=code)
            if code in FINAL:
                raise
            # 4001 and 4003 too: the next pass fetches the current password
        except (websockets.InvalidHandshake, OSError, TimeoutError) as e:
            log(connect_failed=repr(e))
        await asyncio.sleep(max(wait, delay))
        delay = min(delay * 2, 60)
 
asyncio.run(watch())

A process_start for chrome followed by a run of window_focus events is a readable trail of what the agent opened and in what order. An idle event during a step that should be busy is a cheap stuck-agent signal.

For a trace of each agent run rather than of its desktop, Orgo also integrates with Latitude, which records each run's steps, tool calls, model, duration, token counts, and status. Orgo sets it up for OpenClaw and Hermes Agent on Linux; automatic setup never sends prompts, replies, files, or screen content.

How do I watch a long command the agent started?

Attach to the Terminal WebSocket and read its output frames. You get a real PTY, so colors and curses apps render, and you can send input frames ("\r" is Enter) when you need to step in (Terminal WebSocket).

wss://www.orgo.ai/desktops/{id}/ws/terminal?token={password}&cols=120&rows=40&session=build

• Use session so the shell survives you. A session name of 1 to 32 characters (A-Z, a-z, 0-9, -, _) keeps the shell running after a disconnect; reconnecting with the same name re-attaches, long job still going. Without it, the shell is killed when the socket closes.
• Idle is traffic in either direction. A build that keeps printing keeps the socket open. After 30 minutes of silence it closes with 4008; reconnect with the same session.
• Watch for the error frame. If the shell cannot start, an error frame arrives followed by a normal 1000 close, so a client that only reads close codes never learns why.

To run one command and get its result, Execute bash is simpler: it waits up to 200 seconds by default, and timeout is clamped to 1 to 300.

Can I hear what is playing on the agent's computer?

Yes. The Audio WebSocket streams the computer's speaker output: browser media, system alerts, application audio (Audio WebSocket).

The server first sends a started frame with the settings actually in use; size your playback from it. Browsers need a user gesture before an AudioContext plays. The audio itself does not count as activity. The socket stays open while anything else on the computer's shared idle timer is active (see How does the idle timer work?), and a ping from you counts, so a lone audio socket should ping at least every 30 minutes or it closes with 4008. The docs recommend a ping every 30 seconds on every stream to catch dead connections early.

When are screenshots better than a stream?

When you need a record, not a live view. GET /computers/{id}/screenshot returns one image per call, as a stored URL by default, or inline with response_format=base64 or binary. You can pick png, jpeg, or webp, set quality (1 to 100, default 80), and scale down. For a small image the docs suggest ?response_format=binary&format=webp&quality=75&scale=0.5 (Take screenshot).

A common pattern: let the Events stream trigger a screenshot. Take one on every window_focus and you get an audit trail of each screen the agent touched, without recording video. The agent's own loop already uses screenshots the same way (The Orgo computer use API).

How do I keep live streams connected?

Read the close code. It is the only signal a refused connection gives you; an embedded VNC screen that fails just stays black (Embed computers).

A URL that is not a /desktops/{id}/ws/... path gets no close code at all: it is refused before the socket opens.

Sources: the close code tables on Events, Terminal, and Audio, checked against Orgo's WebSocket proxy.

How does the idle timer work?

The screen, terminal and audio sockets close with 4008 after 30 minutes without activity. The events socket has no idle timer. The timer is shared, not per socket:

• All password sockets on a computer share one timer. Every socket opened with the computer password (an embed, an SDK, your watcher) reads the same clock. A signed-in Orgo user gets a separate clock for their own sockets.
• What counts as activity: any frame a client sends on the terminal or audio socket (a ping included), any terminal output, any screen frame from the viewer other than a plain "send the next frame" request (keys, pointer, clipboard), and opening a new screen, terminal or audio socket.
• What doesn't count: the screen's pixels and the audio data.

So a terminal streaming a long build keeps an embedded screen open too, while a screen an agent is busy redrawing, with nobody watching and nothing else open, still closes with 4008 after 30 minutes. Treat 4008 as a clean close and reconnect on the next action.

One more case: a restart keeps the computer's VNC password, but open VNC and terminal WebSockets drop during the reboot, so reconnect afterwards (Restart computer). How do you pause, fork, or clone an AI agent's computer? covers what each lifecycle call does to a running computer.

How do I show the live screen inside my own app?

Point a noVNC client at wss://www.orgo.ai/desktops/{id}/ws/websockify?token={password}, with the password fetched by your server. The full walkthrough, including the black-screen close codes, is in How to embed a live AI agent computer in your app. The screen uses the shared idle timer above, so pixels alone don't keep it open.

A Linux computer can also run extra screens (Screens). Stream one by adding ?screen={screen id} to the same URL, for example /ws/websockify?screen=screen-101&token={password}. Every screen uses the computer's one password.

FAQ

Can I watch my AI agent's screen live on Orgo? Yes. A Linux or Windows computer streams its screen over wss://www.orgo.ai/desktops/{id}/ws/websockify?token={password}, and any websockify-compatible VNC client such as noVNC can show it (Embed computers).

How do I know which app my agent is using right now? Subscribe to window_focus on the Events WebSocket. Each event carries the window's id and title, for example Google Chrome (Events WebSocket).

Why do I receive no events after connecting? Either you have not sent a subscribe message, or an event name is misspelled. Unknown names are accepted silently and deliver nothing. On Linux, file_change is not delivered.

Can I use my API key as the WebSocket token? No. Use the computer password from GET /computers/{id}/vnc-password. On the events, terminal and audio streams, an API key closes the socket with 1000 right after it opens, or with 4001 from a browser (Terminal WebSocket).

Do I miss events while I am disconnected? Yes. Events are not replayed on reconnect, and a slow consumer drops events past a 256-event buffer.

Can I watch the agent's terminal without killing its job when I disconnect? Yes, connect with a session name. The shell survives disconnection and you re-attach with the same name (Terminal WebSocket).

Does this work on Windows computers? The screen, terminal (PowerShell), and audio streams do. Events on Windows are limited to window_focus, clipboard, and file_change (Events WebSocket).

Related

• What is computer use?
• How to embed a live AI agent computer in your app
• The Orgo computer use API
• How do you pause, fork, or clone an AI agent's computer?
• Scale AI agent computers to thousands

Sources

• Events WebSocket, Terminal WebSocket, Audio WebSocket
• Get VNC password, Take screenshot, Execute bash, Restart computer
• Embed computers, Screens, Latitude, llms.txt