CUA (Computer Use Agents)

Overview

CUA (Computer Use Agents) is an open-source infrastructure project designed for building, training, and evaluating AI agents that can interact with desktop environments. It provides the foundational building blocks — sandboxes, SDKs, and benchmarks — needed to create agents that can control full desktops on macOS, Linux, and Windows.

With over 18,000 GitHub stars, CUA has emerged as one of the most important infrastructure projects for the computer-use agent ecosystem, complementing frameworks like Anthropic's Computer Use SDK and OpenHands.

Features

• Cross-Platform Support: Works on macOS, Linux, and Windows desktop environments
• Sandboxed Execution: Safe, isolated environments for running agent-controlled actions
• Standardized Benchmarks: Built-in evaluation suite for measuring agent performance
• SDK Integration: Clean APIs for agent-to-environment communication
• Training Infrastructure: Tools and datasets for training computer-use agents

Installation

git clone https://github.com/trycua/cua
cd cua
pip install -e .

Quick Start

from cua import Sandbox, AgentClient

# Initialize a sandboxed desktop environment
sandbox = Sandbox(platform="linux")  # or "macos", "windows"

# Connect your agent to the sandbox
client = AgentClient(sandbox)

# Send a command for the agent to execute
result = client.execute("open browser and search for 'AI agents'")

# Get screenshot and state information
screenshot = client.get_screenshot()
state = client.get_state()

Core Concepts

Sandboxes

CUA provides containerized sandbox environments that isolate agent actions from the host system. This is critical for safety — agents can explore, click, and type without risking damage to the host machine.

Benchmarks

The project includes standardized benchmarks for evaluating agent performance on real desktop tasks:

• Navigation Tasks: Opening applications, navigating menus
• Search Tasks: Finding and retrieving information
• Form-Filling Tasks: Completing web forms and data entry
• Multi-Step Workflows: Complex sequences of desktop actions

SDK Architecture

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   Agent     │────▶│    SDK      │────▶│  Sandbox    │
│  (LLM-based)│     │  (CUA SDK)  │     │ (Desktop)   │
└─────────────┘     └─────────────┘     └─────────────┘

Advanced Features

Multi-Step Workflows

from cua import Workflow

workflow = Workflow(sandbox)
workflow.add_step("open browser")
workflow.add_step("navigate to example.com")
workflow.add_step("fill form with data")
workflow.add_step("submit and verify")

result = workflow.run()

Screenshot-Based Interaction

# Agent receives screenshots for visual understanding
screenshot = client.get_screenshot(format="png")
# Process with vision model
vision_response = vision_model.analyze(screenshot)

Action Validation

# Validate actions before execution
if client.validate_action("click", element_id="submit-btn"):
    client.execute_action("click", element_id="submit-btn")

Examples

Automated Form Filling

from cua import Sandbox, AgentClient

sandbox = Sandbox(platform="linux")
client = AgentClient(sandbox)

# Navigate to form
client.navigate_to("https://example.com/form")

# Fill form fields
client.fill_field("name", "John Doe")
client.fill_field("email", "john@example.com")
client.fill_field("message", "Hello, this is a test.")

# Submit
client.click("submit-btn")

Desktop Automation Testing

from cua import Sandbox, AgentClient, Benchmark

sandbox = Sandbox(platform="windows")
client = AgentClient(sandbox)

# Run benchmark suite
benchmark = Benchmark.from_suite("desktop_tasks")
results = benchmark.evaluate(client)

print(f"Success rate: {results.success_rate:.2%}")
print(f"Average steps: {results.avg_steps}")

Pros

• ✅ Cross-platform — works on macOS, Linux, and Windows
• ✅ Built-in sandboxing — safe isolated execution
• ✅ Standardized benchmarks — fair agent comparison
• ✅ Open-source — active community development
• ✅ Essential infrastructure — foundational for computer-use agents

Cons

• ❌ Relatively new — evolving APIs and documentation
• ❌ Complex setup — requires sandbox configuration
• ❌ Limited benchmark coverage — specialized tasks may not be covered
• ❌ Sandbox overhead — performance depends on underlying technology

When to Use

• Building computer-use agents that need desktop control
• Evaluating and benchmarking agent performance
• Research on agent-desktop interaction
• Training agents on desktop automation tasks
• Creating safe sandboxed environments for agent testing

Resources

• GitHub Repository
• Documentation
• Related: Anthropic Computer Use SDK
• Related: OpenHands