omlix

Overview

omlx (pronounced "Omlix") is a high-performance LLM inference server optimized for Apple Silicon, combining continuous batching, SSD caching, and Metal-accelerated computation to deliver production-grade throughput on consumer-grade hardware. With approximately 19,700 GitHub stars, omlix has become the go-to inference solution for developers who want to run large language models locally on Macs without sacrificing performance.

Unlike generic inference servers like vLLM or TGI that target GPU clusters, omlx is built specifically for the Apple Silicon architecture — taking full advantage of the unified memory architecture (UMA), Neural Engine, and fast SSD subsystems that define modern Macs. It enables running models like Llama 3 70B, Mixtral 8x22B, and Qwen 72B on a MacBook Pro or Mac Studio that would otherwise require expensive GPU servers.

omlix positions itself at the intersection of local-first AI and production inference. It's designed for AI agent developers who need a local model backend that can serve multiple concurrent requests without the latency and cost of cloud APIs — a critical requirement for agentic applications that make dozens or hundreds of model calls per task.

Features

• Apple Silicon native: Built from the ground up for M-series chips with Metal and MPS acceleration
• Continuous batching: Dynamic request batching for maximum throughput without increasing latency
• SSD caching: Automatic KV cache offloading to SSD for large-context workloads
• Memory-efficient quantization: FP8, INT8, and INT4 quantization optimized for Apple Silicon
• Unified memory support: Leverages Apple's UMA architecture for models up to the full system RAM
• REST and OpenAI-compatible API: Drop-in replacement for OpenAI API in agent frameworks
• Multi-model serving: Serve multiple models simultaneously with automatic memory management
• Streaming support: Token-by-token streaming for real-time agent interactions
• Structured outputs: Native support for JSON schema and tool call formatting
• GPU fallback: Graceful degradation when RAM pressure forces model offloading

Installation

# Install via pip
pip install omlix

# Or via Homebrew
brew install omlix

# Verify installation
omlix --version

Quick Start

# Download and start a model
omlix pull llama3-70b
omlix serve llama3-70b --port 8080

# Test with curl (OpenAI-compatible API)
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3-70b",
    "messages": [{"role": "user", "content": "Hello"}],
    "stream": true
  }'

Integration with Agent Frameworks

# LangChain integration
from langchain_omlx import OmlxChat

llm = OmlxChat(
    model="llama3-70b",
    base_url="http://localhost:8080/v1",
    temperature=0.7,
    max_tokens=4096,
)

response = llm.invoke("Explain continuous batching in LLM inference")

# OpenAI SDK integration
from openai import OpenAI

client = OpenAI(
    api_key="not-needed",
    base_url="http://localhost:8080/v1",
)

response = client.chat.completions.create(
    model="llama3-70b",
    messages=[{"role": "user", "content": "Hello"}],
    stream=True,
)

Core Concepts

Continuous Batching

omlx implements continuous batching (also known as iterative batching), where:

1. New requests are added to the batch at every token generation step
2. Finished requests are removed from the batch immediately
3. The batch size varies dynamically based on request patterns

This is fundamentally different from static batching, where the batch size is fixed at request time. Continuous batching can improve throughput by 3-10x for typical workloads.

SSD KV Cache Offloading

For models with long context windows (128K+ tokens), the KV cache can consume significant memory. omlix automatically offloads cold KV cache entries to the SSD, using the fast NVMe storage in Apple Silicon Macs as an effective memory extension. This enables running 128K+ context workloads on Macs with "only" 64GB or 128GB of RAM.

# Configure cache offloading
omlx serve llama3-70b \
  --max-context 128000 \
  --cache-ssd \
  --cache-ssd-path /tmp/omlx-cache

Unified Memory Architecture (UMA)

Apple Silicon's UMA means GPU and CPU share the same physical memory. For LLM inference, this means the entire system RAM is available for model weights and KV cache — no separate VRAM to manage. A Mac with 192GB of RAM can run models that would require multiple A100 GPUs in a traditional setup.

omlx manages this memory transparently:

• Model weights stay in RAM (potentially with quantization)
• Active KV cache entries stay in RAM for speed
• Cold KV cache entries offload to SSD
• The Neural Engine handles attention computation when beneficial

Advanced Features

Multi-Model Serving

# Serve multiple models simultaneously
omlx serve \
  --models llama3-70b,mixtral-8x22b,qwen2-72b \
  --max-total-vram 128g \
  --eviction-policy lru

omlx manages the total memory budget across all models, automatically evicting inactive models when memory pressure requires.

Quantization

# Serve with INT4 quantization (maximum memory savings)
omlix serve llama3-70b --quantization int4

# Serve with FP8 quantization (best quality/size balance)
omlix serve llama3-70b --quantization fp8

Quantization	Size Reduction	Quality Impact	Best For
FP16	None	None	Maximum quality
FP8	~50%	Minimal	Production use
INT8	~50%	Small	Cost-sensitive
INT4	~75%	Moderate	Maximum compression

Structured Outputs

omlx supports constrained decoding for structured outputs:

from pydantic import BaseModel

class AgentAction(BaseModel):
    tool: str
    arguments: dict
    reasoning: str

response = client.chat.completions.create(
    model="llama3-70b",
    messages=[{"role": "user", "content": "What should I do next?"}],
    response_format={"type": "json_schema", "json_schema": AgentAction.model_json_schema()},
)

Examples

Example 1: Local Agent Backend

import openai
from langgraph.prebuilt import create_react_agent

client = openai.OpenAI(base_url="http://localhost:8080/v1", api_key="sk-local")

model = client
agent = create_react_agent(model, tools=[search_tool, calculator_tool])

result = agent.invoke({"messages": [{"role": "user", "content": "What is 2+2 and what is the weather in Tokyo?"}]})

Example 2: High-Throughput Batch Processing

# Process 1000 prompts with continuous batching
omlix batch ./prompts.jsonl \
  --model llama3-70b \
  --output ./results.jsonl \
  --concurrency 32

Example 3: Fine-Tuning Pipeline

from omlix import Trainer

trainer = Trainer(
    model="llama3-70b",
    dataset="./fine-tune-data.jsonl",
    output_dir="./fine-tuned",
    quantization="int8",
    max_steps=1000,
)
trainer.fit()

Pros

• ✅ Apple Silicon optimized: Purpose-built for the architecture, not a port
• ✅ SSD caching: Unique feature enables large context on limited RAM
• ✅ OpenAI-compatible API: Drop-in integration with existing agent frameworks
• ✅ Continuous batching: Production-grade throughput on consumer hardware
• ✅ Multi-model serving: Cost-effective model experimentation
• ✅ Strong community: ~20K stars, active development
• ✅ Local-first: No cloud dependencies, data stays on your machine

Cons

• ❌ Apple-only: No support for Windows, Linux, or Intel Macs
• ❌ No distributed inference: Single-machine only; no multi-GPU or multi-node support
• ❌ Limited model ecosystem: Supports major open models, but less than vLLM/Ollama
• ❌ Newer project: Less battle-tested than established alternatives
• ❌ Requires Apple Silicon: Not viable for organizations standardizing on other hardware

When to Use

• Running LLMs locally on a Mac: When you need to run large models without cloud costs
• Agent development: When building AI agents that require many model calls per task
• Privacy-sensitive applications: When data cannot leave your local machine
• Cost reduction: When cloud API costs are prohibitive for high-throughput use cases
• Offline development: When working in environments without reliable internet
• Model experimentation: When you need to try many models quickly and cheaply

Resources

• GitHub Repository
• Documentation

Related Projects

• Ollama — Local LLM inference (cross-platform)
• LM Studio — Desktop LLM server
• vLLM — High-throughput GPU inference
• MLX — Apple's ML framework