LiveKit Agents

Overview

LiveKit Agents is a framework for building realtime, programmable AI participants that run on servers. Unlike chat-only agent frameworks, LiveKit Agents are designed for multi-modal, low-latency voice and video conversations — agents that can "see, hear, and understand" in real time.

With 12,000+ GitHub stars and a rapidly growing community, LiveKit Agents fills a gap that traditional agent frameworks (CrewAI, LangGraph, AutoGen) leave open: real-time conversational agents that participate in WebRTC rooms just like a human would. A JavaScript/TypeScript version of the SDK is also under active development alongside the Python core.

LiveKit Agents is relevant to the agentic platform landscape because voice-first agents are becoming a primary interface for consumer-facing AI. The framework's native MCP support (one-line integration) also positions it as a bridge between the realtime agent world and the broader MCP server ecosystem.

Features

• Multi-modal agent SDK: build agents that process audio, video, and text in a single conversational loop
• Flexible model pipeline: mix and match any STT, LLM, TTS, and Realtime API in a composable pipeline (OpenAI, Deepgram, Cartesia, and more)
• Built-in job scheduling with dispatch APIs for scaling agents across multiple rooms
• WebRTC client support across major platforms (web, iOS, Android, desktop)
• Telephony integration for outbound and inbound phone calls with AI agents
• RPC and Data APIs for client-to-agent bidirectional communication within rooms
• Semantic turn detection using transformer models — knows when a speaker is finished without fixed timeout thresholds
• Native MCP support with one-line integration — agents can call any MCP server
• Built-in test framework with AI judges for evaluating agent behavior
• Fully open-source under Apache-2.0 license (model detection licensed under LiveKit Model License)

Installation

pip install "livekit-agents[openai,deepgram,cartesia]"

For the JavaScript/TypeScript SDK:

npm install @livekit/agents

Quick Start

import asyncio
import os
from livekit.agents import Agent, JobContext, WorkerOptions, CLI
from livekit.agents.pipeline import VoiceAgent

async def entrypoint(ctx: JobContext):
    agent = VoiceAgent(
        stt=os.getenv("DEEPGRAM_API_KEY"),
        llm=os.getenv("OPENAI_API_KEY"),
        tts=os.getenv("CARTESIA_API_KEY"),
    )
    await ctx.connect()
    room = ctx.room
    room.on("participant_connected", lambda p: agent.start(room, p))
    agent.start(room)

if __name__ == "__main__":
    CLI(
        App(WorkerOptions(entrypoint_fnc=entrypoint)),
        description="LiveKit Voice Agent",
    ).run()

Start the worker and join a room via python myagent.py console for terminal testing.

Core Concepts

• Agent as a room participant: each agent joins a LiveKit room like any other participant, receiving and publishing audio/video tracks
• Pipeline architecture: STT → LLM → TTS form a composable pipeline where each stage can be swapped independently
• Semantic turn detection: instead of waiting for silence, a transformer model determines when a speaker has finished — reducing awkward pauses in conversation
• Server-side agents: the agent logic runs on your server, while clients (browsers, phones) only need to handle WebRTC media

Advanced Features

• Multi-agent handoff: route conversations between specialized agents (e.g., a menu agent hands off to an order agent)
• Push-to-talk systems for multiple simultaneous users
• Outbound calling agents that proactively call phone numbers
• Video avatar integration for agents with visual presence
• MCP server integration: agents call external MCP servers for tools like GitHub access, database queries, or API calls with a single configuration line
• AI judges test framework for automated quality evaluation of agent responses

Examples

Scenario	Description
Voice restaurant ordering	Multi-agent pipeline with menu → order → confirmation handoff
Push-to-talk call center	Multiple agents handle concurrent inbound calls
Outbound appointment reminders	Agent dials numbers and confirms appointments by voice
Video avatar assistant	Agent with real-time video presence in a LiveKit room
Text-only chat agent	Same SDK, audio tracks disabled
Live transcription service	STT-only pipeline streaming transcript to room

Pros

• ✅ Purpose-built for real-time voice agents — not bolted onto a chat framework
• ✅ Rich plugin ecosystem for STT, LLM, and TTS providers
• ✅ Native MCP support connects realtime agents to the broader tool ecosystem
• ✅ Built-in test framework with AI judges
• ✅ Active development with strong community (12,000+ stars)
• ✅ Apache-2.0 license with production-ready deployment options
• ✅ Excellent documentation and starter templates

Cons

• ❌ Requires a LiveKit server (self-hosted or LiveKit Cloud) as infrastructure
• ❌ More complex setup than chat-only frameworks — WebRTC, STT, TTS all need configuration
• ❌ Python-centric; JavaScript SDK still maturing
• ❌ Rapidly evolving — some API surfaces may change between versions

When to Use

Use LiveKit Agents when you are building real-time voice or video conversational agents — call centers, voice assistants, video avatars, or any scenario where low-latency audio is the primary interface. Use a traditional agent framework like CrewAI or LangGraph when your agent is primarily text-based or task-orchestration focused. Use LiveKit when the conversation speed of voice matters more than the complexity of multi-step reasoning chains.

Resources

• Official Documentation
• GitHub Repository
• LiveKit Cloud