LiteLLM

Overview

LiteLLM is an open-source library that provides a unified API for 100+ LLM providers. It allows you to call any LLM model using a consistent interface, handling provider-specific differences, rate limiting, caching, and fallbacks automatically.

Widely adopted in production environments, LiteLLM serves as the glue layer between your application and multiple LLM providers, enabling seamless model switching without code changes.

Features

• Unified API for 100+ Providers: OpenAI, Anthropic, Google, Azure, AWS Bedrock, and more
• Built-in Rate Limiting: Automatic request throttling per provider
• Automatic Fallbacks: Fail over to alternative models/providers on errors
• Request/Response Caching: Reduce costs and latency with intelligent caching
• Usage Tracking: Monitor costs across all providers in one place
• Self-Hostable Proxy: Run your own LLM gateway for data privacy
• OpenTelemetry Integration: Full observability with tracing

Installation

pip install litellm

Quick Start

from litellm import completion

# Works with any provider using the same interface
response = completion(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}]
)

# Switch to Anthropic with no code changes
response = completion(
    model="claude-3-5-sonnet-20241022",
    messages=[{"role": "user", "content": "Hello!"}]
)

# Use Google Gemini
response = completion(
    model="gemini/gemini-1.5-pro",
    messages=[{"role": "user", "content": "Hello!"}]
)

Configuration

Environment Variables

export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
export GOOGLE_API_KEY="..."

Proxy Server Setup

# Install the proxy
pip install litellm[proxy]

# Run the proxy server
litellm --model gpt-4o --api_key $OPENAI_API_KEY

config.yaml

model_list:
  - model_name: my-gpt-model
    litellm_params:
      model: gpt-4o
      api_key: os.environ/OPENAI_API_KEY
  - model_name: my-claude-model
    litellm_params:
      model: claude-3-5-sonnet-20241022
      api_key: os.environ/ANTHROPIC_API_KEY

Advanced Features

Fallbacks

response = completion(
    model="gpt-4o",
    messages=messages,
    fallbacks=["claude-3-5-sonnet-20241022", "gemini/gemini-1.5-pro"]
)

Caching

from litellm import completion, cache

cache.init_cache()

response = completion(
    model="gpt-4o",
    messages=messages,
    cache={"no-cache": False}
)

Rate Limiting

from litellm import completion

response = completion(
    model="gpt-4o",
    messages=messages,
    num_retries=3,
    retry_delay=1
)

Providers Supported

Provider	Model Examples
OpenAI	gpt-4o, gpt-4o-mini, o1
Anthropic	claude-3-5-sonnet, claude-3-opus
Google	gemini-1.5-pro, gemini-2.0-flash
AWS Bedrock	titan, claude, llama
Azure OpenAI	gpt-4, gpt-35-turbo
Cohere	command-r, command-r-plus
Together AI	llama-3, qwen
DeepSeek	deepseek-coder, deepseek-chat

Pros

• ✅ Switch providers without code changes
• ✅ Battle-tested in production at scale
• ✅ Excellent cost management and tracking
• ✅ Open-source with active community
• ✅ Can be self-hosted for data privacy
• ✅ Handles rate limiting and retries automatically

Cons

• ❌ Not a full agent framework (gateway layer only)
• ❌ Some provider-specific features may not translate
• ❌ Proxy server adds infrastructure complexity
• ❌ Configuration can be extensive for advanced use

When to Use

• Using multiple LLM providers in your application
• Need fallback strategies for production reliability
• Want to track and control LLM costs centrally
• Building self-hosted LLM infrastructure
• Need rate limiting and caching out of the box

Use Cases

Use Case	Why LiteLLM
Multi-Provider Apps	Single API for 100+ LLM providers
Cost Management	Centralized tracking and budgeting
Production Reliability	Automatic fallbacks and rate limiting
Self-Hosted Gateway	Complete data control with proxy

Comparison with Alternatives

Feature	LiteLLM	Helicone	Portkey	OpenAI Router
Providers	✅ 100+	✅ All major	✅ 100+	❌ OpenAI only
Unified API	✅ Yes	✅ Yes	✅ Yes	❌ No
Caching	✅ Yes	✅ Yes	✅ Yes	❌ No
Rate Limiting	✅ Yes	✅ Yes	✅ Yes	❌ No
Fallbacks	✅ Yes	❌ No	✅ Yes	❌ No
Self-Hosted	✅ Yes	✅ Yes	⚠️ Limited	❌ No
Learning Curve	Low	Low	Low	Low
Best for	Multi-provider	Observability	Enterprise	OpenAI focus

Best Practices

1. Configure fallbacks early — Set up model fallbacks for production reliability
2. Enable caching — Reduce costs with intelligent request caching
3. Use proxy for privacy — Self-host for complete data control
4. Track costs centrally — Monitor spending across all providers
5. Set rate limits — Prevent API abuse with per-provider limits

Troubleshooting

Issue	Solution
Provider not working	Check API key format and provider-specific params
Fallback not triggering	Verify error conditions match fallback criteria
Cache not working	Initialize cache before making requests
Proxy fails to start	Check port availability and config.yaml syntax

Resources

• Official Documentation
• GitHub Repository
• Proxy Documentation
• Discord Community