LangSmith

Overview

LangSmith is an end-to-end development platform by Anthropic (formerly LangChain) for building, testing, and monitoring LLM applications. It provides comprehensive tooling for the entire AI application lifecycle — from development and debugging to production monitoring and evaluation.

LangSmith was originally developed by the LangChain team and has become the de facto standard for LLM application observability. It integrates seamlessly with LangChain, LangGraph, and other frameworks, offering deep tracing capabilities that let developers understand exactly what their LLM applications are doing at every step.

Features

• Tracing: Automatic tracing of LLM calls, chain executions, and agent actions with full context
• Evaluation: Built-in evaluation framework with custom evaluators and dataset management
• Debugging: Interactive debugging interface with step-by-step chain visualization
• Dataset Management: Version-controlled datasets for testing and fine-tuning
• Production Monitoring: Real-time monitoring of production LLM applications with alerts
• Prompt Management: Version control and A/B testing for prompts
• Feedback Collection: Automated and manual feedback collection from end users
• Cost Tracking: Detailed cost analytics across all LLM calls and providers

Installation

pip install langsmith

Then set your API key:

export LANGCHAIN_API_KEY="your-api-key"
export LANGCHAIN_TRACING_V2="true"

Quick Start

from langsmith import Client

# Initialize the client
client = Client()

# Log a trace manually
with client.trace("My LLM Call", inputs={"prompt": "Hello, how are you?"}) as trace:
    # Your LLM call here
    response = "I'm doing well, thank you!"
    trace.outputs = {"response": response}

Core Concepts

Traces

Every LLM interaction is captured as a trace, which includes:

• Inputs: What was sent to the LLM
• Outputs: What the LLM returned
• Metadata: Model used, tokens consumed, latency, costs
• Child Runs: Nested calls within chains and agents

Datasets

Datasets are versioned collections of input-output pairs used for:

• Regression testing
• Fine-tuning data preparation
• Benchmarking model performance

Evaluators

Custom or built-in evaluators that score LLM outputs:

• Exact Match: String comparison
• Similarity: Embedding-based semantic similarity
• Hallucination Detection: Fact-checking against context
• Custom LLM Evaluators: Use an LLM to evaluate another LLM

Advanced Features

Prompt Management

from langsmith import Client

client = Client()

# Create a prompt version
prompt = client.create_prompt_version(
    "my-prompt",
    prompt="Translate to French: {text}",
    metadata={"version": "1.0"}
)

# Use in production
prompt = client.pull_prompt("my-prompt")

Dataset Evaluation

from langsmith.evaluation import evaluate

def my_llm_app(inputs):
    return {"output": "translated text"}

results = evaluate(
    my_llm_app,
    data="my-dataset",
    evaluators=["exact_match"],
    metadata={"version": "1.0"}
)

Production Monitoring

from langsmith import Client

client = Client()

# Get production runs
runs = client.list_runs(
    project_name="production",
    filter="latency_ms > 1000"
)

# Get feedback metrics
feedback = client.get_feedback_stats("production")

Examples

Debugging a LangChain Chain

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langsmith import traceable

@traceable
def format_docs(docs):
    return "\n\n".join([d.page_content for d in docs])

@traceable
def retrieve(query):
    # Your retrieval logic
    return [{"page_content": "Sample document"}]

chain = (
    {"context": retrieve | format_docs, "question": RunnablePassthrough()}
    | ChatPromptTemplate.from_template("Context: {context}\nQuestion: {question}")
    | ChatOpenAI(model="gpt-4")
)

# All calls are automatically traced in LangSmith
response = chain.invoke("What is AI?")

Custom Evaluator

from langsmith.schemas import Example, Run

def relevance_evaluator(run: Run, example: Example) -> dict:
    """Check if the output is relevant to the question."""
    question = example.inputs["question"]
    answer = run.outputs["answer"]
    
    # Simple heuristic or use an LLM
    relevance_score = len(answer) > 50  # Example heuristic
    
    return {
        "key": "relevance",
        "score": relevance_score,
        "comment": "Answer is sufficiently detailed" if relevance_score else "Answer too brief"
    }

Pros

• ✅ Comprehensive tracing with minimal code changes
• ✅ Deep integration with LangChain ecosystem
• ✅ Powerful evaluation framework with custom evaluators
• ✅ Production-ready monitoring and alerting
• ✅ Dataset versioning and management
• ✅ Cost tracking across all providers
• ✅ Active development and strong community

Cons

• ❌ Primarily designed for LangChain (limited for non-LangChain apps)
• ❌ SaaS platform (self-hosting not available)
• ❌ Paid tiers for production use
• ❌ Can be overwhelming for simple applications
• ❌ Proprietary platform (vendor lock-in concerns)

Use Cases

Use Case	Why LangSmith
Production Monitoring	Real-time visibility into LLM application behavior
Regression Testing	Dataset-based testing to catch performance drops
Prompt Optimization	A/B test prompts and track improvements
Cost Management	Track and optimize LLM spending
Team Collaboration	Share traces and evaluations across teams

Comparison with Alternatives

Feature	LangSmith	Langfuse	Arize Phoenix	Helicone
Paradigm	SaaS	Open-source + SaaS	Open-source	SaaS
Self-hostable	❌ No	✅ Yes	✅ Yes	⚠️ Limited
LangChain Native	✅ Yes	✅ Yes	✅ Yes	⚠️ Limited
Evaluation Framework	✅ Comprehensive	✅ Good	⚠️ Basic	❌ No
Prompt Management	✅ Yes	✅ Yes	❌ No	❌ No
Learning Curve	Medium	Medium	Low	Low
Best for	LangChain users	Privacy-focused teams	Local dev	API monitoring

Best Practices

1. Trace everything — Enable tracing from day one for baseline data
2. Use datasets for testing — Build regression test suites early
3. Define custom evaluators — Score outputs based on your quality criteria
4. Monitor costs regularly — Set up alerts for spending thresholds
5. Version your prompts — Track prompt changes and their impact
6. Collect user feedback — Use feedback scores to improve models

Troubleshooting

Issue	Solution
Traces not appearing	Check LANGCHAIN_TRACING_V2 is set to "true"
Evaluation failures	Verify evaluator function returns proper dict
Dataset sync issues	Ensure dataset name matches exactly
High latency	Check network connectivity to LangSmith API
Cost tracking wrong	Verify model names match provider naming

Resources

• Official Documentation
• GitHub Repository
• LangChain Documentation
• Community Forum