Building Desktop Automation with Anthropic Computer Use

Overview

This tutorial walks you through building a desktop automation agent using Anthropic's Computer Use feature. You'll learn how to create an AI agent that can interact with desktop applications through visual perception and direct manipulation—essentially automating GUI tasks just like a human would.

By the end of this tutorial, you'll have a working agent that can open applications, fill out forms, navigate menus, and perform multi-step workflows across different desktop applications.

Prerequisites

• Python 3.10+ or Node.js 18+
• Anthropic API key (Claude 3.5 Sonnet access)
• Basic understanding of Python or TypeScript
• A desktop environment (Windows, macOS, or Linux)

Step 1: Set Up Your Environment

Python Setup

# Install the Anthropic SDK
pip install anthropic

# Install screen capture utilities
pip install pyautogui pillow

# For Windows
pip install pywin32

# For macOS
pip install pyobjc-framework-Quartz

Get Your API Key

1. Go to Anthropic Console
2. Create an API key
3. Set it as an environment variable:

export ANTHROPIC_API_KEY="your-api-key-here"

Step 2: Create the Computer Use Tool

The Computer Use tool allows Claude to control your desktop. Here's the implementation:

Python Implementation

import anthropic
import pyautogui
import base64
from io import BytesIO
from PIL import Image

class ComputerUseAgent:
    def __init__(self, api_key: str):
        self.client = anthropic.Anthropic(api_key=api_key)
        self.width, self.height = pyautogui.size()
        
    def screenshot(self) -> str:
        """Capture current screen and return base64."""
        screenshot = pyautogui.screenshot()
        buffer = BytesIO()
        screenshot.save(buffer, format="PNG")
        return base64.b64encode(buffer.getvalue()).decode()
    
    def get_cursor_position(self) -> tuple:
        """Get current cursor position."""
        return pyautogui.position()
    
    def move_mouse(self, x: int, y: int):
        """Move mouse to coordinates."""
        pyautogui.moveTo(x, y, duration=0.5)
    
    def click(self, button: str = "left"):
        """Click mouse button."""
        if button == "left":
            pyautogui.click()
        elif button == "right":
            pyautogui.rightClick()
        elif button == "double":
            pyautogui.doubleClick()
    
    def type_text(self, text: str):
        """Type text using keyboard."""
        pyautogui.write(text)
    
    def press_key(self, key: str):
        """Press a single key."""
        pyautogui.press(key)
    
    def press_keys(self, keys: list):
        """Press multiple keys (e.g., Ctrl+C)."""
        pyautogui.hotkey(*keys)
    
    def execute_action(self, action: dict):
        """Execute a computer use action."""
        action_type = action.get("action")
        
        if action_type == "screenshot":
            return {"base64_image": self.screenshot()}
        
        elif action_type == "cursor_position":
            x, y = self.get_cursor_position()
            return {"coordinate": [x, y]}
        
        elif action_type == "mouse_move":
            coord = action.get("coordinate", [0, 0])
            self.move_mouse(coord[0], coord[1])
            return {"success": True}
        
        elif action_type == "left_click":
            self.click("left")
            return {"success": True}
        
        elif action_type == "right_click":
            self.click("right")
            return {"success": True}
        
        elif action_type == "double_click":
            self.click("double")
            return {"success": True}
        
        elif action_type == "left_click_drag":
            coord = action.get("coordinate", [0, 0])
            pyautogui.dragTo(coord[0], coord[1], duration=0.5)
            return {"success": True}
        
        elif action_type == "type":
            text = action.get("text", "")
            self.type_text(text)
            return {"success": True}
        
        elif action_type == "key":
            keys = action.get("keys", [])
            if len(keys) > 1:
                self.press_keys(keys)
            else:
                self.press_key(keys[0])
            return {"success": True}
        
        return {"error": f"Unknown action: {action_type}"}

Step 3: Define the Tool Schema

COMPUTER_TOOL = {
    "name": "computer",
    "description": "Perform mouse clicks, keyboard input, and screenshot capture",
    "input_schema": {
        "type": "object",
        "properties": {
            "action": {
                "type": "string",
                "enum": [
                    "mouse_move", "left_click", "right_click", 
                    "double_click", "left_click_drag", "screenshot",
                    "cursor_position", "type", "key"
                ],
                "description": "The action to perform"
            },
            "coordinate": {
                "type": "array",
                "items": {"type": "integer"},
                "maxItems": 2,
                "description": "X, Y coordinates for mouse actions"
            },
            "text": {
                "type": "string",
                "description": "Text to type"
            },
            "button": {
                "type": "string",
                "enum": ["left", "right"],
                "description": "Mouse button for click actions"
            },
            "keys": {
                "type": "array",
                "items": {"type": "string"},
                "description": "Keys to press (e.g., ['ctrl', 'c'])"
            }
        },
        "required": ["action"]
    }
}

Step 4: Build the Agent Loop

class DesktopAutomationAgent:
    def __init__(self, api_key: str):
        self.client = anthropic.Anthropic(api_key=api_key)
        self.agent = ComputerUseAgent(api_key)
        self.messages = []
    
    def run(self, task: str, max_steps: int = 20) -> str:
        """Execute a desktop automation task."""
        self.messages = [{"role": "user", "content": task}]
        
        for step in range(max_steps):
            # Take screenshot
            screenshot_base64 = self.agent.screenshot()
            
            # Build message with image
            user_content = [
                {
                    "type": "image",
                    "source": {
                        "type": "base64",
                        "media_type": "image/png",
                        "data": screenshot_base64
                    }
                },
                {
                    "type": "text",
                    "text": f"Current step: {step + 1}. {task}"
                }
            ]
            
            self.messages.append({"role": "user", "content": user_content})
            
            # Get Claude's response
            response = self.client.messages.create(
                model="claude-3-5-sonnet-20241022",
                max_tokens=1024,
                tools=[COMPUTER_TOOL],
                messages=self.messages
            )
            
            # Check for tool use
            tool_use = None
            for content in response.content:
                if content.type == "tool_use":
                    tool_use = content
                    break
            
            if not tool_use:
                # No more actions needed
                return response.content[0].text
            
            # Execute the action
            action_input = tool_use.input
            result = self.agent.execute_action(action_input)
            
            # Add tool result to conversation
            self.messages.append({
                "role": "assistant",
                "content": response.content
            })
            self.messages.append({
                "role": "user",
                "content": [
                    {
                        "type": "tool_result",
                        "tool_use_id": tool_use.id,
                        "content": str(result)
                    }
                ]
            })
        
        return "Max steps reached"

# Usage
if __name__ == "__main__":
    agent = DesktopAutomationAgent(api_key="your-api-key")
    result = agent.run("Open Chrome and navigate to google.com")
    print(result)

Step 5: Test with Real Tasks

Task 1: Open an Application

agent.run("Open the calculator application")

What happens:

1. Claude sees the desktop
2. Identifies the Start menu or dock
3. Clicks to open application launcher
4. Types "calculator" to search
5. Clicks on the Calculator app
6. Confirms the calculator is open

Task 2: Fill Out a Form

agent.run("Fill out the contact form with name 'John Doe', email 'john@example.com', and submit it")

What happens:

1. Claude identifies form fields
2. Clicks on the name field
3. Types "John Doe"
4. Clicks on the email field
5. Types "john@example.com"
6. Locates and clicks the submit button

Task 3: Multi-Application Workflow

agent.run("Open Excel, create a new spreadsheet with columns 'Name' and 'Score', add 3 rows of data, then save it as 'test.xlsx'")

Step 6: Add Safety Features

Confirmation Before Critical Actions

def execute_with_confirmation(self, action: dict, critical_actions: list = ["delete", "format", "close"]):
    """Ask for confirmation before critical actions."""
    if action.get("action") in critical_actions:
        print(f"⚠️  Confirmation required: {action}")
        response = input("Proceed? (y/n): ")
        if response.lower() != "y":
            return {"error": "Action cancelled by user"}
    return self.execute_action(action)

Action Logging

import logging
from datetime import datetime

logging.basicConfig(filename='automation.log', level=logging.INFO)

def log_action(self, action: dict, result: dict):
    """Log all actions for audit trail."""
    logging.info(f"{datetime.now()}: Action={action}, Result={result}")

Screen Region Limiting

def restrict_to_region(self, action: dict, region: tuple):
    """Restrict mouse actions to a specific screen region."""
    if action.get("action") in ["mouse_move", "left_click", "right_click"]:
        coord = action.get("coordinate", [0, 0])
        x, y = coord
        rx, ry, rw, rh = region
        
        if not (rx <= x <= rx + rw and ry <= y <= ry + rh):
            return {"error": f"Action outside allowed region: {coord} not in {region}"}
    
    return self.execute_action(action)

Best Practices

1. Start Simple: Begin with single-step tasks before attempting complex workflows
2. Use Descriptive Prompts: Be specific about what you want accomplished
3. Handle Errors Gracefully: Implement retry logic for failed actions
4. Monitor Progress: Log all actions for debugging and audit
5. Test Incrementally: Verify each step works before chaining them together
6. Respect Rate Limits: Add delays between actions if needed

Troubleshooting

Issue: Claude Can't Find UI Elements

Solution: Improve the prompt with more context:

"Look for a button with the text 'Submit' in the bottom right corner"

Issue: Actions Are Too Fast

Solution: Add delays in your action execution:

import time
time.sleep(0.5)  # Wait for UI to respond

Issue: Wrong Coordinates

Solution: Use relative positioning or improve visual recognition:

# Instead of absolute coordinates, describe the location
"Click on the button that says 'Save' near the top of the window"

Next Steps

• Explore the Anthropic Computer Use documentation
• Try the MCP Computer Use server for Claude Desktop integration
• Combine with other tools for powerful automation workflows
• Contribute to the growing Computer Use ecosystem

Resources

• Anthropic Computer Use Docs
• Claude 3.5 Sonnet Announcement
• pyautogui Documentation
• MCP Computer Use Server