Turbo-Fieldfare

Overview

Turbo-Fieldfare is a Swift + Metal runtime that enables Gemma 4 26B-A4B (a 26-billion-parameter frontier-class model) to run on any M-series Apple Silicon Mac using only ~2 GB of RAM. The project achieved this milestone through expert-streaming from SSD rather than speculative decoding.

With 829 Hacker News points on release, Turbo-Fieldfare represents a breakthrough in memory-efficient LLM inference on consumer hardware. The model never loads the full 14.3 GB checkpoint into memory — it keeps only the shared 1.35 GB core and FP16 KV cache resident, streaming individual experts from SSD as needed.

Features

• Expert-Streaming Runtime: Streams only the MoE experts needed per token from SSD, never materializing the full model
• 4-bit Quantized Weights with 8-bit router for maximum compression
• Custom Metal Kernels: Hand-optimized quantized operations for Apple GPU
• Expert Caching: Cached expert activations reduce repeated SSD reads
• Three Interfaces: Native Mac app, CLI, and OpenAI-compatible API server
• Chunked Prefill: Processes long inputs incrementally to avoid OOM
• Measured Decode Speed: 5.1–6.3 tok/s on 8GB M2 MacBook Air; 31–35 tok/s on 24GB M5 Pro

Installation

Prerequisites:

• macOS 26 with Metal 4
• Xcode 26
• Swift 6.2+
• M-series Apple Silicon Mac

git clone https://github.com/drumih/turbo-fieldfare.git
cd turbo-fieldfare
swift build -c release

The model downloads on first run via a streaming installer that never materializes the full source checkpoint.

Quick Start

Mac App:

.build/release/TurboFieldfareMac

CLI Chat:

swift run -c release TurboFieldfareCLI \
  --model scratch/gemma4.gturbo \
  --messages-file messages.json

CLI Raw Completion:

swift run -c release TurboFieldfareCLI \
  --model scratch/gemma4.gturbo \
  --prompt "The capital of France is" \
  --max-new 64 \
  --temperature 0

OpenAI-Compatible Server

.build/release/TurboFieldfareServer \
  --model scratch/gemma4.gturbo

Use with any OpenAI-compatible client:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
response = client.chat.completions.create(
    model="gemma4-26b",
    messages=[{"role": "user", "content": "Hello"}]
)
print(response.choices[0].message.content)

Core Concepts

• Gemma 4 26B-A4B Architecture: A Mixture-of-Experts model with 26B total parameters but only ~3.88B active per token
• Streaming Expert Loading: Instead of loading all 8 experts into memory simultaneously, the runtime loads one expert per token from SSD, enabling the full 26B model to run with minimal resident memory
• Gturbo Format: A custom model format optimized for the streaming runtime, created on first run

Pros

• ✅ Runs a 26B frontier model on consumer hardware (2 GB RAM)
• ✅ No need for cloud API calls — fully local and private
• ✅ Multiple interfaces (app, CLI, server)
• ✅ Measured 5+ tok/s on M2 MacBook Air (usable for interactive chat)
• ✅ Apache 2.0 license
• ✅ No Python dependencies

Cons

• ❌ Apple Silicon only (no Windows/Linux support)
• ❌ Requires macOS 26, Metal 4, Xcode 26, Swift 6.2+
• ❌ Slow on older/smaller Macs (~5 tok/s)
• ❌ Single model (Gemma 4 26B only)
• ❌ Model weights remain under original license terms

When to Use

Turbo-Fieldfare is ideal for:

• Private local inference on sensitive data that shouldn't leave your machine
• Developers on a tight budget who need frontier-class model access without API costs
• Offline scenarios with no internet access
• Apple Silicon users who want to experiment with 26B models

Resources

• GitHub Repository
• Hacker News Discussion