Add production-grade monitoring to your AI agents with AgentOps, from zero-code instrumentation to advanced debugging.

Building AI Agents with AgentOps: Monitoring and Debugging Guide

Introduction

AgentOps is an AI agent monitoring and debugging platform that helps you track, analyze, and optimize your AI agent applications. It provides zero-code instrumentation for major frameworks like CrewAI, AutoGen, LangChain, and LangGraph. In this tutorial, you'll learn how to integrate AgentOps into your AI agent projects.

Prerequisites

An AgentOps account (free tier available at agentops.ai)
Basic understanding of AI agent frameworks
Python 3.8+

Installation

pip install agentops

Quick Start

Step 1: Initialize AgentOps

Add AgentOps to your project with minimal code:

import agentops

# Initialize with your API key
agentops.init(api_key="your-api-key")

# Your agent code here
from crewai import Agent, Task, Crew

# All agent activity is automatically tracked!

That's it! AgentOps automatically instruments your agent framework and starts tracking all activity.

Step 2: Run Your Agent

# Example: CrewAI agent
researcher = Agent(
    role="Researcher",
    goal="Find the latest information about AI agents",
    backstory="You are an expert researcher",
    verbose=True
)

task = Task(
    description="Research the top 3 AI agent frameworks in 2026",
    agent=researcher
)

crew = Crew(
    agents=[researcher],
    tasks=[task]
)

result = crew.kickoff()
print(result)

Step 3: View Your Dashboard

Visit your AgentOps dashboard to see:

Real-time agent activity
Conversation traces
Cost breakdowns
Performance metrics
Error tracking

Core Concepts

Sessions

A session represents a single agent execution or conversation. Each session captures:

All LLM calls
Tool executions
Agent decisions
Timing and costs

Events

Events are individual actions within a session:

LLM Call: Model selection, prompt, completion, tokens, cost
Tool Call: Tool name, inputs, outputs, errors
Agent Step: Agent role, decision, reasoning

Advanced Features

Cost Budgets

Set budget limits to control spending:

agentops.set_budget(
    max_cost=10.00,  # $10 per session
    alert_threshold=0.8  # Alert at 80% of budget
)

Custom Metrics

Track custom metrics for your specific use case:

agentops.record_metric(
    name="response_quality",
    value=0.85,
    tags=["quality", "evaluation"]
)

Comparison View

Compare different agent versions or configurations:

# Run variant A
agentops.start_session(tags=["variant-a"])
result_a = run_agent(config_a)
agentops.end_session()

# Run variant B
agentops.start_session(tags=["variant-b"])
result_b = run_agent(config_b)
agentops.end_session()

# View comparison in dashboard

Team Management

Invite team members and manage access:

# In your AgentOps dashboard
# Settings > Team > Invite members

Framework-Specific Integration

CrewAI

import agentops
agentops.init()

from crewai import Agent, Task, Crew

# Zero-code instrumentation — just import and it works
researcher = Agent(
    role="Researcher",
    goal="Research AI trends",
    backstory="Expert AI researcher"
)

task = Task(
    description="Write a report on AI agent trends",
    agent=researcher
)

crew = Crew(agents=[researcher], tasks=[task])
crew.kickoff()

AutoGen

import agentops
agentops.init()

from autogen import AssistantAgent, UserProxyAgent

# AutoGen is automatically instrumented
assistant = AssistantAgent("assistant", llm_config=llm_config)
user_proxy = UserProxyAgent("user_proxy", human_input_mode="NEVER")

user_proxy.initiate_chat(
    assistant,
    message="Analyze this data and provide insights"
)

LangChain

import agentops
agentops.init()

from langchain.agents import initialize_agent, Tool
from langchain.llms import OpenAI

# LangChain agents are automatically tracked
tools = [
    Tool(name="Search", func=search_function),
    Tool(name="Calculator", func=calculator_function)
]

agent = initialize_agent(
    tools,
    llm=OpenAI(),
    agent="zero-shot-react-description"
)

agent.run("What is the population of Tokyo?")

LangGraph

import agentops
agentops.init()

from langgraph.graph import StateGraph

# LangGraph execution is traced
builder = StateGraph(AgentState)
builder.add_node("agent", agent_node)
builder.add_node("tools", tool_node)

app = builder.compile()
result = app.invoke({"messages": [...]})

Debugging with AgentOps

Finding Errors

When an error occurs, AgentOps captures:

Full stack trace
State at time of error
All preceding actions
LLM prompts and completions

# Example: Debug a failed tool call
# 1. Go to AgentOps dashboard
# 2. Find the failed session
# 3. Click on the error event
# 4. See full context: inputs, state, preceding actions

Performance Analysis

Identify bottlenecks in your agent:

# View timing breakdown in dashboard
# - LLM call latency
# - Tool execution time
# - Agent decision time
# - Total session duration

Cost Optimization

Analyze costs and optimize:

# View cost breakdown by:
# - Model (GPT-4 vs GPT-3.5)
# - Agent role
# - Tool usage
# - Session type

# Recommendations from AgentOps:
# - Switch to cheaper models for simple tasks
# - Cache frequent queries
# - Reduce context window size

Best Practices

1. Use Tags for Organization

# Tag by environment
agentops.tag("production")

# Tag by version
agentops.tag("agent-v2.1")

# Tag by use case
agentops.tag("customer-support", "billing")

2. Set Budget Alerts

# Prevent runaway costs
agentops.set_budget(max_cost=50.00)
agentops.set_alert_threshold(0.8)

3. Compare Variants Systematically

# A/B test agent configurations
for variant in ["v1", "v2", "v3"]:
    agentops.start_session(tags=[f"variant-{variant}"])
    result = run_agent(get_config(variant))
    agentops.end_session()

4. Monitor in Production

# Enable production monitoring
agentops.init(
    api_key="your-key",
    environment="production",
    enable_tracing=True
)

Complete Example: Production Agent with Monitoring

import agentops
agentops.init(
    api_key="your-api-key",
    environment="production"
)

from crewai import Agent, Task, Crew
from crewai_tools import SerperDevTool

# Set up tools
search_tool = SerperDevTool()

# Define agents
researcher = Agent(
    role="Senior Research Analyst",
    goal="Uncover cutting-edge developments in AI",
    backstory="""You are an expert at understanding 
    the AI landscape. You have deep knowledge of 
    recent developments and can identify trends.""",
    tools=[search_tool],
    verbose=True,
    allow_delegation=False
)

writer = Agent(
    role="Tech Content Strategist",
    goal="Craft compelling content on AI developments",
    backstory="""You are a content strategist who 
    translates complex technical concepts into 
    engaging content for a technical audience.""",
    verbose=True,
    allow_delegation=True
)

# Define tasks
research_task = Task(
    description="""Conduct comprehensive research 
    on AI agent frameworks released in 2026.""",
    expected_output="""A detailed research report 
    with 5 key findings and sources.""",
    agent=researcher
)

writing_task = Task(
    description="""Create a blog post based on 
    the research findings.""",
    expected_output="""A well-structured blog post 
    of 1000+ words with proper citations.""",
    agent=writer
)

# Create crew
crew = Crew(
    agents=[researcher, writer],
    tasks=[research_task, writing_task],
    verbose=2
)

# Execute with monitoring
result = crew.kickoff()

# Add custom metrics
agentops.record_metric(
    name="content_quality_score",
    value=evaluate_quality(result),
    tags=["production", "blog-post"]
)

print(result)

Resources

Summary

In this tutorial, you learned:

How to initialize AgentOps with zero-code instrumentation
How to track sessions, events, and tags
How to use cost budgets and custom metrics
Framework-specific integration for CrewAI, AutoGen, LangChain, and LangGraph
How to debug errors and analyze performance
Best practices for production monitoring

AgentOps makes it easy to add production-grade monitoring to your AI agent applications with minimal code changes. Start with the free tier and scale up as your needs grow.