Claude Computer Use 2.0 Deep Dive

Overview

Anthropic's Claude Computer Use capabilities have evolved significantly with Claude 4.6 Opus, released in July 2026. Computer Use 2.0 introduces enhanced visual grounding, multi-application orchestration, long-horizon task execution, and new safety controls.

This tutorial provides a comprehensive guide to building desktop automation agents with Claude Computer Use 2.0, from basic screenshot analysis to complex multi-step workflows.

---

What's New in Computer Use 2.0

Feature	v1.0	v2.0
Screenshot Resolution	640x480	Up to 1920x1080
Multi-App Support	Single app	Full desktop
Keyboard/Mouse Precision	Basic	Pixel-perfect
Long-Horizon Tasks	Limited	128K reasoning tokens
Safety Controls	Basic	Policy-based guardrails
Multi-Modal Context	Screenshots only	Screenshots + window metadata

---

Core Concepts

1. The Computer Use Agent Loop

Computer Use works through a feedback loop:

1. Observe: Claude takes a screenshot of the current state
2. Plan: Claude decides what action to take next
3. Act: Claude sends keyboard/mouse commands
4. Repeat: The loop continues until the task is complete

import anthropic
from anthropic.types import ContentBlock

client = anthropic.Anthropic()

def computer_use_agent(task_description: str, max_steps: int = 20):
    messages = [{"role": "user", "content": task_description}]
    
    for _ in range(max_steps):
        response = client.messages.create(
            model="claude-4-6-opus",
            max_tokens=128000,
            messages=messages,
            tools=[{
                "type": "computer_20250624",
                "name": "computer",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "action": {
                            "type": "string",
                            "enum": [
                                "screenshot",
                                "key",
                                "type",
                                "mouse_move",
                                "left_click",
                                "left_click_drag",
                                "right_click",
                                "double_click",
                                "scroll",
                            ],
                        },
                    },
                },
            }],
        )
        
        messages.append({"role": "assistant", "content": response.content})
        
        # Execute tool calls and get new screenshot
        for tool_use in [c for c in response.content if c.type == "tool_use"]:
            result = execute_computer_action(tool_use)
            messages.append({
                "role": "user",
                "content": [{"type": "tool_result", "tool_use_id": tool_use.id, "content": result}],
            })
        
        if response.stop_reason == "end_turn":
            break
    
    return messages

2. Available Actions

# Keyboard actions
actions = {
    "key": "Press a single key (e.g., 'Enter', 'Tab', 'Escape')",
    "type": "Type text (e.g., 'Hello World')",
    "hotkey": "Press key combinations (e.g., 'Command+Shift+4')",
}

# Mouse actions
actions = {
    "mouse_move": "Move cursor to (x, y) coordinates",
    "left_click": "Click left mouse button",
    "right_click": "Click right mouse button",
    "double_click": "Double-click at current position",
    "left_click_drag": "Click and drag to (x, y) coordinates",
    "scroll": "Scroll up/down by pixels",
}

# Observation
actions = {
    "screenshot": "Capture current screen state",
}

3. Coordinate System

Computer Use 2.0 provides window-relative coordinates:

response = client.messages.create(
    model="claude-4-6-opus",
    messages=[{"role": "user", "content": "Click the Save button"}],
    tools=[computer_use_tool],
    computer_use={
        "display_height": 1080,
        "display_width": 1920,
        "coordinate_system": "window_relative",  # New in v2.0
    },
)

---

Building a Multi-App Automation Agent

Let's build an agent that works across multiple applications:

from dataclasses import dataclass
from enum import Enum

class AppState(Enum):
    BROWSER = "browser"
    IDE = "ide"
    TERMINAL = "terminal"
    FILE_MANAGER = "file_manager"

@dataclass
class AutomationTask:
    description: str
    target_apps: list[AppState]
    success_criteria: str

class MultiAppAgent:
    def __init__(self):
        self.client = anthropic.Anthropic()
        self.current_app = None
        self.task_history = []
    
    async def execute_task(self, task: AutomationTask):
        messages = [{"role": "user", "content": task.description}]
        
        for step in range(50):
            response = self.client.messages.create(
                model="claude-4-6-opus",
                max_tokens=128000,
                messages=messages,
                tools=[self._get_computer_use_tool()],
            )
            
            for tool_use in [c for c in response.content if c.type == "tool_use"]:
                action = tool_use.input
                
                # App switching
                if action.get("action") == "switch_app":
                    await self._switch_app(action["app_name"])
                
                result = await self._execute_action(action)
                messages.append({
                    "role": "user",
                    "content": [{"type": "tool_result", "tool_use_id": tool_use.id, "content": result}],
                })
            
            if response.stop_reason == "end_turn":
                break
        
        return self._verify_task(task)
    
    async def _switch_app(self, app_name: str):
        # Implement app switching logic
        pass
    
    def _verify_task(self, task: AutomationTask):
        # Check if success criteria are met
        pass

---

Advanced Patterns

1. Task Decomposition with Reasoning

Claude 4.6 Opus's extended reasoning (up to 128K tokens) enables complex task planning:

async def decompose_task(task_description: str):
    response = client.messages.create(
        model="claude-4-6-opus",
        max_tokens=128000,
        messages=[{
            "role": "user",
            "content": f"Break this into steps:\n{task_description}\n\nReturn a JSON array of steps.",
        }],
        response_format={"type": "json_object"},
    )
    return json.loads(response.content[0].text)

2. Safety Guardrails

v2.0 introduces policy-based guardrails:

from anthropic.types import GuardrailConfig

guardrail_policy = GuardrailConfig(
    actions=[
        {"type": "deny", "pattern": "delete"},  # Block destructive actions
        {"type": "deny", "pattern": "rm -rf /"},  # Block dangerous commands
    ],
    max_retries=3,
)

response = client.messages.create(
    model="claude-4-6-opus",
    messages=[{"role": "user", "content": "Clean up old files"}],
    tools=[computer_use_tool],
    guardrail_policy=guardrail_policy,
)

3. Human-in-the-Loop Approval

Critical actions require human approval:

class HumanApprovalAgent:
    async def execute_with_approval(self, action):
        if action.get("action") in ["delete", "command_execute"]:
            if not await self.request_approval(action):
                return {"status": "blocked", "reason": "Requires approval"}
        
        return await self._execute_action(action)

---

Integration with Other Tools

1. MCP Integration

Use MCP servers to enhance Computer Use with additional context:

from mcp import ClientSession

async def computer_use_with_mcp(task: str):
    async with ClientSession() as mcp_client:
        # Get context from MCP
        files = await mcp_client.tools.call("list_files", {"path": "/workspace"})
        
        # Combine with Computer Use
        response = client.messages.create(
            model="claude-4-6-opus",
            messages=[{
                "role": "user",
                "content": f"Task: {task}\nContext: {files}\n\nUse Computer Use to execute.",
            }],
            tools=[computer_use_tool],
        )

2. Browser Automation

Combine with browser tools for web automation:

import playwright.async_api as pw

async def browser_computer_use(url: str):
    async with pw.playwright() as p:
        browser = await p.chromium.launch()
        page = await browser.new_page()
        await page.goto(url)
        
        # Use Computer Use on the browser window
        screenshot = await page.screenshot()
        # Process with Claude
        await page.close()

---

Pros

• ✅ Enhanced visual grounding with pixel-perfect accuracy
• ✅ Multi-application orchestration across desktop
• ✅ 128K reasoning tokens for long-horizon tasks
• ✅ Policy-based safety guardrails
• ✅ Window-relative coordinate system
• ✅ Integration with MCP and other tools

Cons

• ❌ Requires Claude 4.6 Opus (expensive)
• ❌ Desktop automation is inherently risky
• ❌ No native support for mobile devices
• ❌ Latency due to screenshot-action loop
• ❌ Not suitable for high-frequency automation

When to Use

• Desktop automation workflows
• Multi-app task coordination
• Complex UI interactions
• Testing and QA automation
• Digital assistant applications

Resources

• Anthropic Computer Use Docs
• Claude 4.6 Opus Announcement
• Computer Use Cookbook
• Safety Best Practices