AirLLM

Overview

AirLLM is an inference runtime that enables running massive language models on low-end GPUs — including 70B models on a single 4 GB GPU, Llama 3.1 405B on 8 GB, and even the Kimi K3 (2.8T) on under 4 GB. The core innovation is a layer-at-a-time loading strategy: instead of keeping the entire model in VRAM, AirLLM streams only one layer (or one expert in MoE models) onto the GPU at any moment.

With 26,000+ GitHub stars, AirLLM addresses one of the most persistent barriers to running frontier models locally: memory. It does so without quantization, distillation, or pruning — preserving full model fidelity.

AirLLM is directly relevant to the local-first agent movement: as tools like OpenWorker (Andrew Ng) and Hermes Agent push agents onto the desktop, developers need ways to run capable models without cloud API costs. AirLLM fills that gap.

Features

• Layer-at-a-time streaming: only one model layer occupies VRAM at any time, reducing memory requirements from total model size to a single layer's size
• MoE support via per-expert streaming: sparse Mixture-of-Experts models stream one expert at a time, enabling trillion-parameter models on consumer hardware
• FP8 support and 4-bit / 8-bit block-wise quantization for a 3× inference speedup with almost negligible accuracy loss
• Prefetching: overlaps disk loading with GPU computation to hide I/O latency
• No quantization required by default: run full-precision models on tiny GPUs
• Cross-platform: works on Linux and macOS (Apple Silicon)
• Broad model support: Llama, Qwen, DeepSeek, Mistral & Mixtral, Phi, Gemma, ChatGLM, Baichuan, InternLM, Yi
• Apache-2.0 license — free and open-source

Installation

pip install airllm

Quick Start

from airllm import AutoModel

# Load a large model that would normally require far more VRAM
model = AutoModel.from_pretrained("Qwen/Qwen3-32B")

input_text = ['What is the capital of United States?']
input_tokens = model.tokenizer(
    input_text, return_tensors="pt",
    return_attention_mask=False, truncation=True, max_length=128
)
output = model.generate(
    input_tokens['input_ids'].cuda(), max_new_tokens=20
)
print(model.tokenizer.decode(output.sequences[0]))

Core Concepts

• Streaming vs. full load: Traditional inference loads the entire model into GPU memory before it runs. AirLLM instead treats the model as a stream, loading one layer at a time from disk — the VRAM requirement depends on a single layer's size, not the total model.
• MoE streaming: For Mixture-of-Experts models like Kimi K3 (2.8T), only one expert is loaded per forward pass, collapsing the 2.8T footprint into a few GB.
• Quantization as a speed knob, not a fidelity knob: Traditional quantization trades accuracy for smaller footprint. AirLLM already solves footprint via streaming, so quantization is optional and used purely to speed up disk reads.

Advanced Features

• 3× speedup via block-wise quantization, with minimal accuracy regression
• FP8 tensor support for hardware that benefits from mixed precision
• Prefetch pipeline to overlap disk I/O with GPU compute, eliminating loading stalls
• AutoModel API mirrors the Hugging Face transformers interface for minimal migration effort

Examples

Model	Parameters	VRAM Required
Qwen3-32B	32B	4 GB
Llama 3.1 405B	405B	8 GB
DeepSeek-V3	671B	12 GB
Kimi K3	2.8T	3.72 GB

Pros

• ✅ Runs models far beyond GPU memory limits with no accuracy loss
• ✅ No quantization, distillation, or pruning required
• ✅ MoE models run on consumer-grade hardware
• ✅ Familiar AutoModel API for easy adoption
• ✅ Cross-platform (Linux + macOS Apple Silicon)
• ✅ 26,000+ stars, actively maintained

Cons

• ❌ Streaming from disk is slower than full GPU load — latency is higher than traditional inference
• ❌ Throughput-per-second is lower than optimized runtimes for models that fit in memory
• ❌ Requires a fast storage drive (NVMe recommended) to keep I/O from becoming the bottleneck
• ❌ Not a production inference solution for high-throughput serving

When to Use

Use AirLLM when you want to experiment with or run frontier-scale models locally and your GPU memory is the limiting factor. It is ideal for local agent setups, research prototyping, and private inference where cloud API costs or privacy concerns make remote inference impractical. It is not designed for high-throughput production serving where latency per token matters more than memory constraints.

Resources

• GitHub Repository
• PyPI Package