Langfuse

Overview

Langfuse is an open-source LLM engineering platform that helps teams build better AI applications through comprehensive tracing, evaluation, and debugging. It provides a visual interface for observing LLM calls, tracking costs, evaluating quality, and collaborating on improvements.

Unlike proprietary alternatives, Langfuse can be self-hosted for full data control, making it ideal for teams with privacy or compliance requirements.

Features

• Full Request Tracing: Visualize every LLM call with inputs, outputs, and metadata
• Cost Tracking: Monitor spending across all models and providers
• Evaluation Framework: Define custom metrics and automated evaluations
• Dataset Management: Create test datasets for regression testing
• Prompt Management: Version and A/B test your prompts
• Self-Hostable: Run on your own infrastructure with Docker
• Wide Integration: LangChain, LlamaIndex, OpenAI, Anthropic, and more

Installation

Self-Hosted

docker compose up -d

Cloud (SaaS)

Sign up at langfuse.com

SDK Installation

pip install langfuse

Quick Start

from langfuse import Langfuse
import os

langfuse = Langfuse(
    public_key=os.environ["LANGFUSE_PUBLIC_KEY"],
    secret_key=os.environ["LANGFUSE_SECRET_KEY"],
    host="https://cloud.langfuse.com"
)

# Create a trace
trace = langfuse.trace(
    name="chat-session",
    input={"message": "Hello, how are you?"}
)

# Create a generation
generation = trace.generation(
    name="gpt-4o-response",
    model="gpt-4o",
    input={"role": "user", "content": "Hello, how are you?"},
    output={"role": "assistant", "content": "I'm doing well, thank you!"},
    usage={"totalTokens": 50}
)

# Add a score (evaluation)
trace.score(
    name="human-feedback",
    value=5,
    comment="Great response!"
)

langfuse.flush()

Integration with LangChain

from langfuse.langchain import CallbackHandler
from langchain_openai import ChatOpenAI

handler = CallbackHandler()

llm = ChatOpenAI(
    model="gpt-4o",
    callbacks=[handler]
)

response = llm.invoke("Hello!")

Features

Tracing

Every LLM interaction is automatically traced with:

• Input and output content
• Model and provider information
• Token usage and cost
• Latency metrics
• Custom metadata

Evaluation

# Define a custom evaluator
def hallucination_check(trace):
    # Check for hallucinated content
    return {"score": 0.9, "comment": "Low hallucination risk"}

# Run evaluation
langfuse.evaluate(
    trace_id=trace.id,
    evaluator=hallucination_check
)

Datasets

# Create a dataset
dataset = langfuse.create_dataset(name="qa-test-set")

# Add items
dataset.create_item(
    input={"question": "What is the capital of France?"},
    expected_output={"answer": "Paris"}
)

# Run against dataset
for item in dataset.items:
    result = my_agent.run(item.input)
    dataset.create_run(
        trace_id=result.trace_id,
        observation_id=result.generation_id
    )

Pros

• ✅ Open-source with self-hosting option
• ✅ Comprehensive tracing and debugging
• ✅ Strong evaluation and dataset features
• ✅ Active community and frequent updates
• ✅ Good documentation and examples
• ✅ No vendor lock-in

Cons

• ❌ Requires infrastructure to self-host
• ❌ Some advanced features on paid plan
• ❌ Smaller ecosystem than LangSmith
• ❌ Learning curve for evaluation setup

Use Cases

Use Case	Why Langfuse
Self-hosted Observability	Full data control with open-source platform
Compliance-heavy Apps	Keep LLM data on your infrastructure
Cost Tracking	Monitor spending across all LLM providers
Team Collaboration	Share traces and evaluations with team
Production Monitoring	Real-time visibility into LLM behavior

Comparison with Alternatives

Feature	Langfuse	LangSmith	Arize Phoenix	Helicone
Paradigm	Open-source + SaaS	SaaS	Open-source	SaaS
Self-hostable	✅ Yes	❌ No	✅ Yes	⚠️ Limited
LangChain Native	✅ Yes	✅ Yes	✅ Yes	⚠️ Limited
Evaluation Framework	✅ Good	✅ Comprehensive	⚠️ Basic	❌ No
Prompt Management	✅ Yes	✅ Yes	❌ No	❌ No
Learning Curve	Medium	Medium	Low	Low
Best for	Privacy-focused teams	LangChain users	Local dev	API monitoring

Best Practices

1. Set up tracing early — Capture data from the start for better insights
2. Define custom scores — Create metrics that match your quality criteria
3. Use datasets for testing — Build regression test suites with representative data
4. Monitor costs proactively — Set up alerts for budget thresholds
5. Version your prompts — Track changes and their impact on quality
6. Collect feedback systematically — Use user feedback to improve models

Troubleshooting

Issue	Solution
Traces not appearing	Verify API keys and host configuration
Self-hosting fails	Check Docker ports (3000, 5432, 6379) are available
Evaluation errors	Ensure evaluator returns proper score format
Slow dashboard	Reduce trace retention period, use sampling
Integration issues	Check SDK version compatibility

Resources

• Official Documentation
• GitHub Repository
• Self-Hosting Guide
• Discord Community