What is computer use in AI?

Computer use is an AI agent's ability to operate a computer the same way a person would: it looks at the screen through screenshots, interprets what they show, and acts by moving the cursor, clicking, and typing on the keyboard, instead of talking to the application through a dedicated programming interface. Anthropic introduced this mode with Claude's computer use tool, in research preview on macOS since March 2026. Alternatives are already available: OpenAI exposes a computer-using agent model through its API, and Google offers Gemini Computer Use, after shutting down its Project Mariner and folding its capabilities in. The difference from an agent that calls a service's API directly is clear: computer use requires no dedicated integration and works on any application a person would know how to use, but it is slower and less reliable, because every action depends on correctly interpreting an image, not on a structured call with a predictable response.

Screenshots, not structured calls

The mechanism is a loop: the model receives a screenshot of the screen, reasons about what it shows and returns an action, move the cursor to these coordinates, click, type this text, press this key. The program hosting the model executes the action on the operating system or the browser and takes a new screenshot, which becomes the input for the next step. There is no shared data structure between model and application, unlike classic tool calling: the model works on the same visual representation a human user would see, with all the universality that gives it and all the fragility that comes with it.

When it beats API integration

An API remains the more reliable choice whenever it exists: structured, predictable, checkable responses before acting. Computer use earns its place where an API does not exist or is not reachable: legacy desktop applications, terminals of older enterprise systems, software from a vendor that exposes no programmatic integration. A concrete case is a company that needs to automate data entry into an AS/400 system still running in production, with no room to rewrite it in the short term: an agent with computer use can read an incoming document and fill in the on-screen form the way an operator would, while the company plans genuine modernization on a longer horizon.

The risk it carries

Giving an agent control of the screen, mouse and keyboard means granting it the same range of action as a logged-in human user, far broader than the minimal set of operations a well-designed API integration would allow. This opens the same family of risks already known for agentic browsers: a malicious element shown on screen, a popup, hidden text inside an open document, can be interpreted by the model as an instruction to execute. It should therefore be treated as a permissions problem and a narrowly scoped process automation, with human confirmation on irreversible actions, not as a zero-risk productivity tool.

Why it matters for decision makers

Computer use does not replace API integration where one is available: it remains the last resort, not the first choice, because it costs more in reliability and oversight. It is worth the investment when the cost of rewriting or integrating a legacy system far exceeds the cost of operating it through a supervised graphical interface, with minimal permissions and a well-defined scope of action from day one.