Vercel Labs' agent-desktop gives AI agents the accessibility tree, not the screenshot

Vercel Labs has published agent-desktop, a computer-use tool that lets an AI agent see and operate desktop apps on macOS, Windows, and Linux by reading the OS accessibility tree rather than taking screenshots. It ships as a CLI and a C library, runs 56 operational commands out of a declared 60, and speaks MCP over stdio so an agent harness can call it as a tool. The repo is Apache-2.0. The premise is the news: no pixel matching, no vision model, no screenshot loop. The agent gets JSON with a ref for every element it can act on, acts on a ref, reads the UI again, repeats.

That last sentence is the whole design. The loop lives in your agent. agent-desktop has no model inside. For anyone who has watched a computer-use demo stall because a vision model misread a button at 2x scaling, this is the correction the category has needed. Accessibility trees are boring, old, and exact. Screenshots are impressive, new, and approximate.

What the tool actually does

A snapshot returns an envelope with a version, a snapshot_id, a ref_count, and a tree. Each actionable element carries a ref_id like @s8f3k2p9:e8, scoped to the snapshot that produced it. A bare @e8 fails with INVALID_ARGS. If the UI moved, an action returns STALE_REF or AMBIGUOUS_TARGET, and the agent takes a new snapshot. That is the retry discipline baked into the protocol, and it is more honest than most agent frameworks about the fact that desktop state changes under you.

The command surface is broad and specific. Observe: snapshot, find, get, is, screenshot, list-surfaces. Act on a ref: click, double-click, triple-click, right-click, type, set-value, clear, focus, select, toggle, check, uncheck, expand, collapse, scroll, scroll-to. Keyboard and mouse: press, hover, drag, mouse-move, mouse-click, mouse-wheel, plus key-down, key-up, mouse-down, mouse-up. Apps and windows: launch, close-app, list-apps, list-windows, list-displays, focus-window, resize-window, move-window, minimize, maximize, restore, open-system-surface. Notifications: list-notifications, dismiss-notification, dismiss-all-notifications, notification-action. Clipboard, wait, sessions, trace, cursor overlay, system.

Four commands refuse to run: key-down, key-up, mouse-down, mouse-up. They stay disabled until a daemon can release a held key or button. That is a small detail with a large implication. Someone at Vercel Labs thought about the agent that presses a modifier and never lets go, and decided to ship the refusal rather than the footgun. Actions are headless by default, with --headed for physical input, so an agent does not steal your cursor while it works.

The platform matrix is the real constraint

macOS 13 or later on Intel or Apple Silicon, via the AX API. Windows 10 1809 or later on x64 or ARM64, via UI Automation. Linux on Ubuntu 24.04 or later, x86_64 or aarch64, X11 or Wayland, via AT-SPI. Each binary is built for one OS and talks to that OS’s accessibility API through a PlatformAdapter. Each platform adapter builds only on its own OS, so the CI has to scope cargo to host packages.

This is where the enthusiasm should cool. The accessibility tree is not a uniform abstraction. AX, UI Automation, and AT-SPI disagree about roles, names, and what counts as actionable. The repo ships agent-desktop skills get platform precisely because the OS you run on changes the rules. An agent that works on Finder may behave differently on a Windows app built with a canvas that exposes nothing to UI Automation. Electron and Flutter apps are the known soft spot for this entire approach. The tree is only as good as the app’s accessibility implementation, and plenty of desktop software treats accessibility as an afterthought.

Vercel Labs is upfront about the permissions cost. On macOS, agent-desktop permissions --guide walks you through the Accessibility grant. That grant is broad. An agent with it can read and drive nearly any app on the machine.

The security note is the most important paragraph

Buried in the README, before the install instructions: “Snapshots, clipboard reads, screenshots, command output, and debug artifacts can contain private desktop content. Keep them out of public logs and uploads.”

Read that twice. Every snapshot is a JSON dump of what is on screen, element names and all. Every clipboard read is whatever the user last copied. Every screenshot is the screen. If an agent harness logs tool calls to a remote service, or a developer pastes a trace into a GitHub issue, private desktop content leaves the machine. The tool tells you this. It does not enforce it. There is no redaction layer in the envelope, no field-level scrubbing, no default that strips text from snapshots before they hit a log.

For a category that has spent two years arguing about prompt injection and data exfiltration, this is the threat model that matters for desktop agents. The agent does not need to be malicious. The logging pipeline is enough. A trace command exists for debugging, and traces are exactly the artifact most likely to end up somewhere public.

The agent does not need to be malicious. The logging pipeline is enough.

Why this matters for agent builders

The computer-use conversation has been dominated by vision. Screenshot in, coordinates out, a model that learns to click. That approach is expensive per step, slow, and fragile to resolution and theming. agent-desktop proposes the opposite: give the agent a structured, addressable view of the UI and let a cheaper model or plain code drive it. The model does not need to see pixels. It needs to know that element @s8f3k2p9:e8 is a tree item named “Documents” with a Click action available.

That reframing has consequences. It makes computer use testable, because a ref is a deterministic handle and a stale ref is a typed error rather than a silent misfire. It makes it cheap, because a JSON tree is fewer tokens than an image. It makes it portable across model vendors, because nothing in the loop depends on a specific vision model. And it makes it brittle in a different place: the accessibility tree, which app developers have spent decades under-investing in.

The MCP surface is the distribution play. agent-desktop mcp serves every command as a tool, named after the command (desktop_snapshot, desktop_click, desktop_session_start), and global flags like --session, --headed, or --trace apply to every call. Any MCP-capable harness can wire it in with a few lines of config. Vercel Labs also ships skills installable via npx skills add, plus an AGENTS.md snippet that tells the agent to read the desktop and platform guides before first use. That is a small but telling choice: the tool assumes the agent needs onboarding, not just a tool schema.

The C library, libagent_desktop_ffi, matters for anyone embedding this in a non-JavaScript runtime. The README shows a Python ctypes example calling ad_init, ad_adapter_create, and ad_adapter_destroy, with an ABI version check. In-process access means no subprocess per action, which is the difference between a usable loop and a slow one at scale.

What to watch

Two things. First, whether the accessibility-tree approach holds up on the apps people actually use, the Electron chat clients and Flutter dashboards that expose thin trees. If it does not, the vision camp gets its rebuttal. Second, whether anyone builds the redaction layer the README implies but does not ship. An agent that can drive your desktop and log what it sees is a data-loss tool until someone puts a filter between the snapshot and the upload. Vercel Labs shipped the capability and wrote the warning. The next commit that matters will come from somewhere else.