Together AI

Overview

Together AI is a cloud platform that provides fast, cost-effective inference for open-source large language models. It offers a unified API compatible with OpenAI's interface, making it easy to switch between models and providers. Together AI specializes in serving popular open-source models like Llama, Mistral, Qwen, and more with optimized inference infrastructure.

Together AI's key advantage is its focus on open-source models — it provides production-grade inference for models that would otherwise require significant infrastructure investment. Their servers are optimized for throughput and latency, making them suitable for both development and production workloads.

Features

• Open-Source Model Library: Access to 100+ open-source models (Llama, Mistral, Qwen, Yi, etc.)
• OpenAI-Compatible API: Drop-in replacement for OpenAI API
• High Performance: Optimized inference with up to 10x faster throughput
• Cost-Effective: Significantly lower pricing than proprietary models
• Fine-Tuning: Built-in fine-tuning service for custom models
• Function Calling: Native support for tool use and function calling
• Streaming: Real-time token streaming for interactive applications
• Batch Inference: Support for batch processing large workloads

Installation

pip install together

Quick Start

Using the Together Python Client

from together import Together

client = Together(api_key="your-api-key")

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Explain quantum computing in simple terms"}]
)

print(response.choices[0].message.content)

Using the OpenAI-Compatible API

import openai

client = openai.OpenAI(
    api_key="your-together-api-key",
    base_url="https://api.together.xyz/v1"
)

response = client.chat.completions.create(
    model="mistralai/Mixtral-8x7B-Instruct-v0.1",
    messages=[{"role": "user", "content": "Hello!"}]
)

Core Concepts

Models

Together AI hosts a wide variety of open-source models:

Model	Parameters	Use Case
Llama 3.1 8B	8B	General purpose
Llama 3.1 70B	70B	Complex reasoning
Mixtral 8x7B	47B	High-quality outputs
Qwen 2.5 72B	72B	Multilingual
Yi-1.5 34B	34B	Balanced performance

Pricing

Together AI offers competitive pricing based on tokens:

Llama 3.1 8B: $0.20 / 1M input tokens, $0.40 / 1M output tokens
Llama 3.1 70B: $0.88 / 1M input tokens, $0.88 / 1M output tokens
Mixtral 8x7B: $0.60 / 1M input tokens, $0.60 / 1M output tokens

Advanced Features

Function Calling

from together import Together

client = Together(api_key="your-api-key")

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string", "description": "City name"}
                },
                "required": ["location"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "What's the weather in Tokyo?"}],
    tools=tools,
    tool_choice="auto"
)

Fine-Tuning

from together import Together

client = Together(api_key="your-api-key")

# Upload training data
file = client.files.create(
    file=open("training_data.jsonl", "rb"),
    purpose="fine-tune"
)

# Create fine-tuning job
fine_tune = client.fine_tuning.create(
    training_file=file.id,
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    hyperparameters={
        "n_epochs": 3,
        "batch_size": 4,
        "learning_rate_multiplier": 1.0
    }
)

Streaming

stream = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Write a story"}],
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

Examples

RAG Application

from together import Together

client = Together(api_key="your-api-key")

def query_rag(context, question):
    prompt = f"""You are a helpful assistant. Answer the question based on the context.

Context: {context}

Question: {question}

Answer:"""

    response = client.chat.completions.create(
        model="meta-llama/Meta-Llama-3.1-8B-Instruct",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.1,
        max_tokens=512
    )
    return response.choices[0].message.content

Multi-Turn Conversation

conversation = [
    {"role": "system", "content": "You are a helpful coding assistant."},
    {"role": "user", "content": "How do I sort a list in Python?"},
    {"role": "assistant", "content": "You can use the sorted() function or list.sort() method..."},
    {"role": "user", "content": "What's the difference?"}
]

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    messages=conversation,
    max_tokens=256
)

Pros

• ✅ Extensive open-source model library
• ✅ OpenAI-compatible API (easy migration)
• ✅ Significantly lower costs than proprietary models
• ✅ High throughput and low latency
• ✅ Built-in fine-tuning service
• ✅ Function calling support
• ✅ Free tier available for development
• ✅ Good documentation and community

Cons

• ❌ Only open-source models (no proprietary models)
• ❌ Less consistent quality than top proprietary models
• ❌ Fine-tuning has limitations compared to full training
• ❌ No self-hosting option
• ❌ Model availability can change

When to Use

• Cost-sensitive applications — Much cheaper than GPT-4/Claude
• Open-source model experimentation — Access to many models
• Development and testing — Free tier for prototyping
• Fine-tuning custom models — Built-in fine-tuning service
• Multilingual applications — Strong non-English model support

Use Cases

Use Case	Why Together AI
Cost-Sensitive Production	Significantly cheaper than GPT-4/Claude
Open-Source Experimentation	Access to 100+ models for testing
Fine-Tuning	Built-in service for custom model adaptation
Multilingual Apps	Strong non-English model support

Comparison with Alternatives

Feature	Together AI	Fireworks AI	Anyscale	Groq
Open-Source Models	✅ 100+	✅ Many	⚠️ Limited	⚠️ Limited
Proprietary Models	❌ No	❌ No	❌ No	❌ No
OpenAI-Compatible	✅ Yes	✅ Yes	✅ Yes	✅ Yes
Fine-Tuning	✅ Yes	✅ Yes	❌ No	❌ No
Speed	✅ Fast	✅ 5-10x faster	⚠️ Standard	✅ Fastest
Pricing	Competitive	Competitive	Higher	Higher
Learning Curve	Low	Low	Medium	Low
Best for	OSS variety	Speed	Enterprise	Ultra-low latency

Best Practices

1. Use OpenAI-compatible API — Drop-in replacement for easy migration
2. Choose right model — Match model size to your quality/cost needs
3. Enable streaming — Better perceived latency for interactive apps
4. Leverage fine-tuning — Adapt models to your specific domain
5. Monitor costs — Track token usage across model experiments

Troubleshooting

Issue	Solution
Model not found	Check model name format in Together model library
Rate limit errors	Implement retry with exponential backoff
Slow inference	Use smaller model or enable batching
Fine-tuning fails	Verify training data format and size requirements

Resources

• Official Documentation
• API Reference
• Model Library
• GitHub SDK