Helicone

Overview

Helicone is an open-source LLM gateway and observability platform that provides a unified interface for monitoring, analyzing, and optimizing LLM application performance. It acts as a middleware layer between your application and LLM providers, capturing detailed telemetry without requiring code changes.

Helicone's key differentiator is its open-source nature — you can self-host it for complete data control and privacy, or use their managed cloud service. It supports all major LLM providers (OpenAI, Anthropic, Google, Cohere, etc.) through a single unified API endpoint.

Features

• Unified API: Single endpoint for all LLM providers with automatic routing
• Request Logging: Detailed logging of all LLM requests and responses
• Cost Tracking: Real-time cost analytics across all providers and models
• Rate Limiting: Built-in rate limiting with per-key and per-model limits
• Caching: Intelligent response caching to reduce costs and latency
• Prompt Versioning: Track and version prompts across deployments
• Team Collaboration: Multi-user access with role-based permissions
• Alerts & Webhooks: Custom alerts for latency spikes, cost thresholds, and errors
• Self-Hosted Option: Full open-source deployment for data sovereignty

Installation

Cloud Version

Sign up at helicone.ai and get your API key.

export HELICONE_API_KEY="your-api-key"

Self-Hosted (Docker)

docker run -d \
  --name helicone \
  -p 8080:8080 \
  -e HELICONE_API_KEY="your-api-key" \
  helicone/helicone:latest

Quick Start

Using the Helicone Gateway

import openai

# Point to Helicone instead of OpenAI directly
client = openai.OpenAI(
    api_key="sk-...",
    base_url="https://api.helicone.ai/v1"
)

# Add Helicone headers for tracking
response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Hello!"}],
    headers={
        "Helicone-Property-AppName": "my-app",
        "Helicone-Property-UserId": "user-123"
    }
)

Using the OpenAI SDK with Helicone

import openai

client = openai.OpenAI(
    api_key="sk-...",
    base_url="https://api.helicone.ai/v1",
    default_headers={
        "Helicone-Auth": "Bearer sk-helicone-...",
        "Helicone-Property-Environment": "production"
    }
)

Core Concepts

Gateway

Helicone's gateway sits between your application and LLM providers:

Your App → Helicone Gateway → OpenAI/Anthropic/Google/etc.

All requests pass through the gateway, which captures telemetry and applies policies.

Properties

Custom properties let you tag requests for filtering and analysis:

headers = {
    "Helicone-Property-UserId": "user-123",
    "Helicone-Property-Plan": "premium",
    "Helicone-Property-Endpoint": "/api/chat"
}

Caching

Enable caching to automatically reuse identical prompts:

headers = {
    "Helicone-Cache-Enabled": "true",
    "Helicone-Cache-Key": "user-pref-v1"  # Custom cache key
}

Advanced Features

Rate Limiting

# helicone.yaml
rate_limits:
  - key: "user"
    limit: 100
    window: "1m"
  - key: "model:gpt-4"
    limit: 10
    window: "1m"

Cost Budgets

# Set monthly budget alerts
client.post(
    "https://api.helicone.ai/v1/budgets",
    json={"limit": 100, "currency": "USD", "period": "month"}
)

Custom Analytics

# Query your data
response = client.get(
    "https://api.helicone.ai/v1/requests",
    params={
        "filter": "properties.userId = 'user-123'",
        "sort": "-createdAt"
    }
)

Examples

Multi-Provider Routing

# Route to cheapest available provider
client = openai.OpenAI(
    base_url="https://api.helicone.ai/v1",
    default_headers={
        "Helicone-Provider": "openai",  # or "anthropic", "google", etc.
        "Helicone-Property-UseCase": "summarization"
    }
)

Response Caching

# Cache identical prompts for 24 hours
response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "What is 2+2?"}],
    headers={
        "Helicone-Cache-Enabled": "true",
        "Helicone-Cache-Secret": "my-cache-secret",
        "Helicone-Cache-Age": "86400"  # 24 hours in seconds
    }
)

A/B Testing Prompts

# Route 50% to prompt version A, 50% to version B
response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": prompt}],
    headers={
        "Helicone-Property-Prompt-Version": "v1",
        "Helicone-Property-Experiment": "prompt-ab-test"
    }
)

Pros

• ✅ Open-source with self-hosting option
• ✅ Supports all major LLM providers through unified API
• ✅ Zero-code integration (just change base URL)
• ✅ Built-in caching reduces costs significantly
• ✅ Comprehensive cost tracking and budget alerts
• ✅ Fine-grained rate limiting
• ✅ Privacy-focused (self-hosted option)
• ✅ Active development and growing community

Cons

• ❌ Cloud version has usage limits on free tier
• ❌ Self-hosting requires infrastructure management
• ❌ Less deep integration than LangSmith with LangChain
• ❌ No built-in evaluation framework
• ❌ No dataset management features

When to Use

• Multi-provider LLM applications — Unified API for OpenAI, Anthropic, Google, etc.
• Cost-sensitive applications — Caching and detailed cost analytics
• Privacy-conscious deployments — Self-host for data control
• Rate limit management — Built-in rate limiting across providers
• Production monitoring — Real-time observability without code changes

Use Cases

Use Case	Why Helicone
Multi-Provider Routing	Single API for OpenAI, Anthropic, Google, Cohere
Cost Optimization	Caching reduces costs by 30-50%
Privacy-First Monitoring	Self-host for complete data control
Rate Limit Management	Built-in limits across all providers

Comparison with Alternatives

Feature	Helicone	LangSmith	AgentOps	Portkey
Open Source	✅ Yes	❌ No	❌ No	⚠️ Partial
Self-Hosted	✅ Yes	⚠️ Limited	❌ No	✅ Yes
Unified API	✅ Yes	⚠️ Partial	❌ No	✅ Yes
Caching	✅ Yes	❌ No	❌ No	✅ Yes
Rate Limiting	✅ Yes	❌ No	❌ No	✅ Yes
Zero Code	✅ Yes	⚠️ Manual	✅ Yes	✅ Yes
Learning Curve	Low	Medium	Low	Low
Best for	Self-hosted gateway	LangChain deep dive	Multi-framework	Enterprise gateway

Best Practices

1. Enable caching early — Set Helicone-Cache-Enabled: true for cost savings
2. Use custom properties — Tag requests with userId, environment, useCase
3. Set rate limits — Configure per-key and per-model limits
4. Monitor costs — Use budget alerts to prevent overspending
5. Self-host for privacy — Deploy locally for complete data control

Troubleshooting

Issue	Solution
Requests not tracked	Verify Helicone headers are sent correctly
Cache not working	Check cache secret matches across requests
Rate limit errors	Increase limits or implement retry logic
Self-host fails	Verify Docker ports 8080 are available

Resources

• Official Documentation
• GitHub Repository
• Self-Hosting Guide
• API Reference