computer-use/setup

12 notes, read out of this brain and free to use. Each one was extracted from a source and is re-checked against its exam.

Computer Use tool setup with AI SDK

Set up the Computer Tool using anthropic.tools.computer_20250124() with parameters displayWidthPx and displayHeightPx. The tool requires an execute function that handles screenshot captures (returning type 'image' with PNG data) and computer actions (mouse movements, clicks, typing). A toModelOutput() function converts text and image responses into format the model can process.

Computer Use execute function implementation pattern

The execute function in the computer tool must handle the 'screenshot' action by returning an object with type 'image' and data property containing the screenshot. For other actions (cursor movement, clicking, typing), delegate to a function like executeComputerAction(action, coordinate, text). You must implement the actual low-level operating system interactions yourself.

Computer Use full implementation code example

```ts import { anthropic } from '@ai-sdk/anthropic'; import { getScreenshot, executeComputerAction } from '@/utils/computer-use'; const computerTool = anthropic.tools.computer_20250124({ displayWidthPx: 1920, displayHeightPx: 1080, execute: async ({ action, coordinate, text }) => { switch (action) { case 'screenshot': { return { type: 'image', data: getScreenshot(), }; } default: { return executeComputerAction(action, coordinate, text); } } }, toModelOutput({ output }) { return typeof output === 'string' ? [{ type: 'text', text: output }] : [{ type: 'file-data', data: output.data, mediaType: 'image/png' }]; }, }); ```

Using Computer Tool with generateText

```ts const result = await generateText({ model: 'anthropic/claude-sonnet-4-20250514', prompt: 'Move the cursor to the center of the screen and take a screenshot', tools: { computer: computerTool }, }); console.log(result.text); ``` This example shows one-shot text generation with Computer Tool for a simple task.

Using Computer Tool with streamText

```ts const result = streamText({ model: 'anthropic/claude-sonnet-4-20250514', prompt: 'Open the browser and navigate to vercel.com', tools: { computer: computerTool }, }); for await (const chunk of result.textStream) { console.log(chunk); } ``` This example shows streaming responses with Computer Tool to receive updates in real-time.

Configure multi-step agentic Computer Use with stopWhen

```ts import { isStepCount } from 'ai'; const stream = streamText({ model: 'anthropic/claude-sonnet-4-20250514', prompt: 'Open the browser and navigate to vercel.com', tools: { computer: computerTool }, stopWhen: isStepCount(10), }); ``` The stopWhen parameter with isStepCount(10) allows the model to perform multiple steps without user intervention by automatically sending tool results back to trigger subsequent generations. Adjust the step count based on task complexity.

Combine all three Computer Use tools example

```ts const computerTool = anthropic.tools.computer_20250124({...}); const bashTool = anthropic.tools.bash_20250124({ execute: async ({ command, restart }) => execSync(command).toString() }); const textEditorTool = anthropic.tools.textEditor_20250124({ execute: async ({ command, path, file_text, insert_line, new_str, insert_text, old_str, view_range }) => { switch(command) { return executeTextEditorFunction({ command, path, fileText: file_text, insertLine: insert_line, newStr: new_str, insertText: insert_text, oldStr: old_str, viewRange: view_range }); } } }); const response = await generateText({ model: 'anthropic/claude-sonnet-4-20250514', prompt: "Create a new file called example.txt, write 'Hello World' to it, and run 'cat example.txt' in the terminal", tools: { computer: computerTool, bash: bashTool, str_replace_editor: textEditorTool, }, }); ``` This example shows combining all three Computer Use tools in a single request for complex workflows. The textEditor tool parameters are: command (required), path (required), file_text, insert_line, new_str, insert_text, old_str, and view_range.

Install AI SDK and Anthropic provider for Computer Use

To use Computer Use with the AI SDK, install both the AI SDK core package and the Anthropic provider using: pnpm add ai @ai-sdk/anthropic

Computer Use requires custom execute function implementation

Computer Use tools in the AI SDK are predefined interfaces that require your own implementation of the execution layer. While the SDK provides type definitions and structure, you must implement: (1) A controlled environment for Computer Use execution, (2) Core functionality like mouse control and keyboard input, (3) Screenshot capture and processing, (4) Rules and limits for how Claude can interact with your system.

Anthropic reference implementation for Computer Use

Anthropic provides a reference implementation at https://github.com/anthropics/anthropic-quickstarts/tree/main/computer-use-demo that includes: (1) A containerized environment configured for safe Computer Use, (2) Ready-to-use Python implementations of Computer Use tools, (3) An agent loop for API interaction and tool execution, (4) A web interface for monitoring and control. This serves as a foundation before building custom solutions.

AI SDK Computer Use Template with Next.js

Vercel provides an AI SDK Computer Use Template at https://github.com/vercel-labs/ai-sdk-computer-use that demonstrates working Computer Use implementation with Next.js and the AI SDK.

Computer Use guide

The AI SDK provides a guide on how to get started with Claude's Computer Use capabilities.

Give your agent this brain