AgentVoice

Overview

AgentVoice is a real-time voice library for AI agents, launched October 7, 2026 (74 HN points) as the general-purpose successor to the voice-agent experiments of the August 2026 roundup. Where the August line was a set of demo applications, AgentVoice is a maintained library that turns any agent framework's tool-calling loop into a low-latency voice pipeline.

The library's core claim: you should be able to bolt AgentVoice onto an existing agent (LangGraph, CrewAI, OpenAI Agents SDK, PydanticAI) and get a voice interface in under 100 lines of code. No separate voice agent runtime, no bespoke prompt engineering, no framework fork. The agent is still your primary artifact — the voice is a transport layer.

Under the hood, AgentVoice chains three components: a streaming STT model (Whisper Live, Deepgram Nova-3, or OpenAI Whisper-2), a TTS model (ElevenLabs Flash-2, OpenAI TTS-1, or a local Kokoro voice), and a transport adapter that bridges the two with the agent's tool-calling loop. The orchestration layer is what makes it different from a simple STT-LLM-TTS pipeline — it handles interruption, barge-in, turn-taking, and tool results streamed as audio mid-utterance.

Features

• Streaming STT — Whisper Live, Deepgram Nova-3, OpenAI Whisper-2, or Groq-hosted Whisper; partial transcripts arrive within 150 ms of speech
• Streaming TTS — ElevenLabs Flash-2, OpenAI TTS-1, Kokoro local, or Azure Neural TTS; first audio byte within 200 ms of the model's first token
• Barge-in / interruption — agents stop speaking and re-listen when the user starts talking over them
• Tool-result streaming — long-running tool calls stream partial results back as audio mid-utterance, so the agent doesn't freeze waiting for the tool
• Turn-taking detection — voice activity detection decides when the user has finished; agents wait rather than interrupting
• Emotional tone tagging — speaker diarization and prosody analysis feeds the LLM so responses can match tone
• Framework adapters — drop-in adapters for LangGraph, CrewAI, OpenAI Agents SDK, PydanticAI, and AgentHub IDE
• Local-first capable — all components (STT, TTS, ASR) have local alternatives for fully on-prem deployments
• Recording and replay — every conversation is recorded with timestamps, suitable for QA and agent evaluation
• MCP server — AgentVoice ships an MCP server so existing agents can add voice without code changes

Installation

# Python
pip install agentvoice

# Node.js
npm install agentvoice

# Rust
cargo add agentvoice

Environment variables for the hosted backends:

export ELEVENLABS_API_KEY=***
export DEEPGRAM_API_KEY=***
export OPENAI_API_KEY=***

Quick Start

from agentvoice import VoiceAgent
from openai import OpenAI

# Wrap an existing OpenAI-compatible agent with voice
voice = VoiceAgent(
    agent=OpenAI(),
    stt="deepgram/nova-3",
    tts="elevenlabs/flash-2",
    interruption="auto",
)

# Voice agent is now a transport for any OpenAI-compatible agent
voice.start(port=8765)

For LangGraph:

from agentvoice.langgraph import VoiceGraph
from langgraph.prebuilt import create_react_agent

graph = create_react_agent("gpt-5.6", tools)
voice_graph = VoiceGraph(graph, stt="whisper-live", tts="elevenlabs/flash-2")
voice_graph.start()

Core Concepts

Barge-In Handling

Real-time voice is hard because humans interrupt. AgentVoice exposes three strategies:

• interruption="auto" — default; agent stops on detected user speech and resumes when user silence is detected
• interruption="manual" — only stop on explicit user command ("stop", "wait", or a hotword)
• interruption="never" — agent speaks to completion; use only for scripted or pre-recorded output

Each strategy is implemented by monitoring the STT partial transcript; once a user phoneme is detected during agent speech, the TTS stream is silenced and the agent's turn is interrupted.

Tool-Result Streaming

Most voice agents have an awkward behavior: they say "let me look that up" and then freeze for several seconds while a tool executes. AgentVoice streams tool results as audio. A long search returns partial results every 200 ms; the agent picks up each chunk and speaks it. The user hears continuous speech; the agent never freezes.

@voice.tool
async def search(query: str):
    for chunk in search_stream(query):
        yield chunk  # streamed back to TTS as it arrives

MCP Voice Server

AgentVoice ships an MCP server (agentvoice-mcp) that exposes voice as a tool. Existing agents using MCP 2.2 can call agentvoice:reply and agentvoice:listen without any framework-specific code. This is how Operator 2.0-style general consumer agents can add voice without rewriting the agent.

Advanced Features

• Emotion-aware responses — prosody analysis feeds the model context so the agent matches tone; a sad user gets a softer response
• Speaker diarization — multi-party conversations automatically attributed; agents respond to the last speaker
• Wake word — optional keyword detection (default "hey agent"); the agent stays asleep until spoken to
• Local STT/TTS — Whisper.cpp and Kokoro-82M for fully on-prem deployments
• Audio memory — recent audio frames are kept in a rolling buffer for context-sensitive responses ("what did I just say?")
• Cost controls — per-turn budget caps to prevent runaway voice loops

MCP Configuration

{
  "mcpServers": {
    "agentvoice": {
      "command": "agentvoice-mcp",
      "args": ["--stt", "deepgram/nova-3", "--tts", "elevenlabs/flash-2"],
      "env": {
        "ELEVENLABS_API_KEY": "***",
        "DEEPGRAM_API_KEY": "***"
      }
    }
  }
}

Examples

A real-time voice coding assistant:

from agentvoice import VoiceAgent
from openai import OpenAI
from agentvoice.tools import shell, filesystem

voice = VoiceAgent(
    agent=OpenAI(),
    stt="whisper-live",
    tts="elevenlabs/flash-2",
    tools=[shell, filesystem],
    interruption="auto",
)
voice.start()
# Agent now listens on the microphone and can execute shell commands by voice

A voice customer-support agent:

from agentvoice import VoiceAgent
from langchain_openai import ChatOpenAI

agent = VoiceAgent(
    agent=ChatOpenAI(model="gpt-5.6"),
    stt="openai/whisper-2",
    tts="openai/tts-1",
    tools=[lookup_order, escalate_to_human],
)
agent.start()

Pros

• Framework-agnostic — works with any OpenAI-compatible agent
• Real barge-in support, not just a simple STT-LLM-TTS chain
• Tool-result streaming avoids the "let me look that up" freeze
• Local-first option with Whisper.cpp + Kokoro for on-prem deployments
• MCP server allows voice integration without framework code changes
• Recording/replay built in for QA

Cons

• Adds 300–800 ms of latency compared to text-only interactions (STT + TTS round-trip)
• ElevenLabs and Deepgram require separate paid accounts
• Wake word support is basic; production-quality wake words still need a custom model
• Emotion detection is experimental and noisy on non-English audio

When to Use

• Building voice interfaces for existing agents
• Needing low-latency barge-in for conversational assistants
• Multi-agent voice pipelines where agents talk to each other
• Local-first voice deployments that need on-prem STT/TTS

Resources

• Website: https://agentvoice.dev
• GitHub: https://github.com/agentvoice/agentvoice
• Docs: https://agentvoice.dev/docs
• MCP Server: https://agentvoice.dev/mcp