Fireworks AI

Overview

Fireworks AI is a high-performance inference platform optimized for serving large language models with exceptional speed and efficiency. It specializes in running open-source models at scale with proprietary inference optimizations that deliver up to 10x faster inference compared to standard deployments.

Fireworks AI's key differentiator is its custom inference engine built from the ground up for LLM serving. The platform uses techniques like speculative decoding, continuous batching, and custom kernels to achieve industry-leading throughput and latency. It supports a wide range of open-source models and provides both API-based serving and self-hosted options.

Features

• High-Performance Inference: Proprietary engine with 5-10x speedup
• Open-Source Model Library: Llama, Mistral, Qwen, Yi, and more
• OpenAI-Compatible API: Easy migration from OpenAI
• Speculative Decoding: Accelerated generation using draft models
• Continuous Batching: Optimized batch processing for high throughput
• Multi-Model Serving: Serve multiple models on shared infrastructure
• Fine-Tuning: Support for LoRA and full fine-tuning
• Self-Hosting: Deploy on your own infrastructure
• Enterprise Features: SSO, audit logs, dedicated support

Installation

pip install fireworks-ai

Quick Start

Using the Fireworks Python Client

from fireworks.client import Fireworks

client = Fireworks(api_key="your-api-key")

response = client.chat.completions.create(
    model="accounts/fireworks/models/llama-v3p1-8b-instruct",
    messages=[{"role": "user", "content": "Explain the theory of relativity"}]
)

print(response.choices[0].message.content)

Using the OpenAI-Compatible API

import openai

client = openai.OpenAI(
    api_key="your-fireworks-api-key",
    base_url="https://api.fireworks.ai/inference/v1"
)

response = client.chat.completions.create(
    model="accounts/fireworks/models/mixtral-8x7b-instruct",
    messages=[{"role": "user", "content": "Hello!"}]
)

Core Concepts

Models

Fireworks AI hosts a curated selection of optimized models:

Model	Parameters	Throughput
Llama 3.1 8B	8B	~150 tokens/s
Llama 3.1 70B	70B	~60 tokens/s
Mixtral 8x7B	47B	~80 tokens/s
Qwen 2.5 72B	72B	~55 tokens/s

Pricing

Fireworks AI offers competitive per-token pricing:

Llama 3.1 8B: $0.20 / 1M input, $0.40 / 1M output
Llama 3.1 70B: $0.90 / 1M input, $0.90 / 1M output
Mixtral 8x7B: $0.50 / 1M input, $0.50 / 1M output

Advanced Features

Speculative Decoding

response = client.chat.completions.create(
    model="accounts/fireworks/models/llama-v3p1-8b-instruct",
    messages=[{"role": "user", "content": "Write a poem"}],
    extra_body={
        "speculative_decoding": True,
        "draft_model": "accounts/fireworks/models/llama-v3p1-8b-instruct"
    }
)

Continuous Batching

# Fireworks automatically uses continuous batching
# No additional configuration needed
response = client.chat.completions.create(
    model="accounts/fireworks/models/llama-v3p1-70b-instruct",
    messages=[{"role": "user", "content": f"Question {i}"}]
    for i in range(100)  # Batch of 100 requests
)

Fine-Tuning with LoRA

from fireworks.apis import fine_tuning

# Upload training data
file = client.files.create(
    file=open("training_data.jsonl", "rb"),
    purpose="fine-tune"
)

# Create fine-tuning job
fine_tune = fine_tuning.create(
    model="accounts/fireworks/models/llama-v3p1-8b-instruct",
    training_file=file.id,
    hyperparameters={
        "n_epochs": 3,
        "batch_size": 4,
        "lora_rank": 16
    }
)

Examples

High-Throughput Batch Inference

from fireworks.client import Fireworks
import asyncio

client = Fireworks(api_key="your-api-key")

async def generate(prompt):
    response = client.chat.completions.create(
        model="accounts/fireworks/models/llama-v3p1-8b-instruct",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=100
    )
    return response.choices[0].message.content

# Process 100 requests concurrently
prompts = [f"Explain concept {i}" for i in range(100)]
results = asyncio.gather(*[generate(p) for p in prompts])

Streaming with Low Latency

stream = client.chat.completions.create(
    model="accounts/fireworks/models/llama-v3p1-8b-instruct",
    messages=[{"role": "user", "content": "Write a technical article"}],
    stream=True,
    extra_body={"min_tokens_to_stream": 1}  # Stream immediately
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

RAG with Fast Retrieval

from fireworks.client import Fireworks

client = Fireworks(api_key="your-api-key")

def rag_query(embeddings_db, query, top_k=5):
    # Retrieve relevant documents
    relevant_docs = embeddings_db.search(query, top_k=top_k)
    context = "\n".join([d.text for d in relevant_docs])
    
    # Generate answer
    response = client.chat.completions.create(
        model="accounts/fireworks/models/llama-v3p1-70b-instruct",
        messages=[
            {"role": "system", "content": "Answer based on context only"},
            {"role": "user", "content": f"Context: {context}\n\nQuestion: {query}"}
        ],
        temperature=0.0
    )
    return response.choices[0].message.content

Pros

• ✅ Industry-leading inference speed (5-10x faster)
• ✅ Proprietary inference optimizations
• ✅ Competitive pricing for high-volume use
• ✅ OpenAI-compatible API
• ✅ Speculative decoding for faster generation
• ✅ Self-hosting option available
• ✅ Good enterprise support
• ✅ Active development and improvements

Cons

• ❌ Only open-source models available
• ❌ Less model variety than Together AI
• ❌ Fine-tuning still maturing
• ❌ Self-hosting requires significant infrastructure
• ❌ Smaller community than established platforms

When to Use

• Latency-sensitive applications — Fastest inference available
• High-volume production — Cost-effective at scale
• Real-time applications — Low latency streaming
• Enterprise deployments — Dedicated support and SLAs
• Self-hosting requirements — Deploy on your own infrastructure

Use Cases

Use Case	Why Fireworks AI
Low-Latency Inference	5-10x faster than standard deployments
High-Volume Production	Cost-effective at scale with competitive pricing
Real-Time Applications	Streaming with minimal latency
Open-Source Models	Best-in-class serving for Llama, Mistral, Qwen

Comparison with Alternatives

Feature	Fireworks AI	Together AI	Anyscale	Groq
Speed	✅ 5-10x faster	✅ Fast	⚠️ Standard	✅ Fastest
Open-Source Models	✅ Yes	✅ Yes	⚠️ Limited	⚠️ Limited
Proprietary Models	❌ No	⚠️ Limited	❌ No	❌ No
Self-Hosting	✅ Yes	❌ No	❌ No	❌ No
Fine-Tuning	✅ Yes	✅ Yes	❌ No	❌ No
Pricing	Competitive	Competitive	Higher	Higher
Learning Curve	Low	Low	Medium	Low
Best for	Speed + OSS	Model variety	Enterprise	Ultra-low latency

Best Practices

1. Use speculative decoding — Enable for 2x+ speedup on compatible models
2. Batch requests — Use continuous batching for high throughput
3. Stream responses — Enable streaming for better perceived latency
4. Choose right model — Match model size to your latency/cost requirements
5. Cache embeddings — Reduce redundant API calls for RAG applications

Troubleshooting

Issue	Solution
Slow inference	Enable speculative decoding or use smaller model
Rate limit errors	Implement retry with exponential backoff
Model not found	Check model name format in Fireworks model library
High latency	Use streaming and reduce max_tokens

Resources

• Official Documentation
• API Reference
• Model Library
• GitHub SDK