TurboVec

Overview

TurboVec is a Rust-based vector index built on the TurboQuant quantization algorithm, designed for high-performance RAG (Retrieval-Augmented Generation) systems. With approximately 16,100 GitHub stars, it has quickly become one of the most popular alternatives to FAISS for embedding storage and similarity search.

TurboVec's defining characteristic is its aggressive compression: it achieves 16× storage compression at 2-bit quantization while maintaining competitive recall rates through hand-written SIMD kernels and a novel length-renormalization scoring technique. The project provides Python bindings via maturin, making it accessible to the broader ML community.

The project positions itself as a drop-in replacement for existing vector stores in LangChain, LlamaIndex, Haystack, and Agno, allowing teams to upgrade their RAG pipelines without changing application code.

Features

• Online ingest: Add vectors directly with no training step, parameter tuning, or rebuilds
• SIMD-optimized search: Hand-written kernels for ARM (NEON SDOT/SMMLA) and x86 (AVX-512 VNNI, AVX2)
• Extreme compression: 16× at 2-bit; e.g., a 1536-dim float32 vector goes from 6,144 bytes to 384 bytes
• Incremental saves: sync(path) persists only what changed — crash-safe with one fsync per call
• In-kernel filtering: Pass ID allowlists or slot bitmasks to search() for filter at search time
• Pure local: No managed service; data never leaves your machine
• Length-renormalized scoring: Removes quantization bias at zero search-time cost
• TQ+ calibration: Optional per-coordinate calibration improves recall up to +2.2pp at @1
• Python bindings: Full Python API via TurboQuantIndex and IdMapIndex

Installation

pip install turbovec

For the Rust crate:

[dependencies]
turbovec = "0.1"

Quick Start

from turbovec import TurboQuantIndex

# Create a new index for 768-dim vectors
index = TurboQuantIndex(dim=768, bits=2)

# Add vectors (online ingest, no training needed)
vectors = [[0.1, 0.2, ...], [0.3, 0.4, ...]]  # shape: (n, 768)
index.add(vectors)

# Search
query = [0.15, 0.25, ...]  # shape: (768,)
results = index.search(query, top_k=10)

# Filter at search time
results = index.search(query, top_k=10, id_allowlist=[0, 2, 5])

# Persist incrementally
index.sync("my_index.bin")

Core Concepts

TurboQuant Algorithm

TurboQuant is an integer quantization algorithm that compresses float32 vectors to 2-bit or 4-bit representations. Unlike product quantization (used by FAISS), TurboQuant uses a single codebook and relies on length-renormalization at search time to preserve vector similarity — eliminating the need for codebook training.

Online Ingest

New vectors can be added at any time without rebuilding the index. This is a key differentiator from FAISS IVF indexes, which require a training step and periodic rebuilds.

Length-Renormalized Scoring

Scalar quantization introduces a downward bias in computed distances — TurboVec corrects this at search time by renormalizing based on the original vector lengths stored during indexing.

Advanced Features

TQ+ Calibration

TQ+ applies per-coordinate calibration using a one-time sample of vectors, improving recall at @1 by up to +2.2 percentage points without any retraining:

index.calibrate(sample_vectors)  # one-time, no retraining

IdMapIndex for Stable External IDs

For applications requiring stable external identifiers and O(1) removal:

from turbovec import IdMapIndex

index = IdMapIndex(dim=768, bits=2)
index.add("user_123", vector)
index.add("user_456", vector2)
index.remove("user_123")  # O(1) removal

Framework Integrations

Drop-in replacements for major RAG frameworks:

# LangChain
from langchain_community.vectorstores import TurboVec
vectorstore = TurboVec(embedding=embeddings, index_path="rag_index.bin")

# LlamaIndex
from llama_index.vector_stores import TurboVecVectorStore

Examples

Air-Gapped RAG Deployment

index = TurboQuantIndex(dim=1536, bits=2)  # for text-embedding-3-large
for doc in documents:
    vec = embed(doc.text)
    index.add(vec)

index.sync("/secure/internal/rag_index.bin")

Multi-Modal Retrieval with Filtering

# Search only within user's permission scope
user_docs = [100, 101, 102, 103, 104]  # IDs this user can access
results = index.search(query_vec, top_k=20, id_allowlist=user_docs)

Pros

• 3.4× faster than FAISS at 4-bit quantization
• 16× compression dramatically reduces storage and memory costs
• Pure local — ideal for air-gapped or privacy-sensitive RAG deployments
• Drop-in replacement for LangChain, LlamaIndex, Haystack, and Agno
• Online ingest eliminates the need for batch training
• Active Rust development with Python bindings

Cons

• Relatively new project with a smaller community than FAISS
• Extreme quantization (2-bit) may not suit all use cases requiring high recall
• TQ+ calibration requires a one-time sample of representative vectors
• Rust ecosystem less mature than Python for ML tooling
• No managed/cloud hosting option — purely self-hosted

When to Use

TurboVec is ideal for:

• RAG systems needing aggressive compression to fit large embedding corpora in limited memory
• Air-gapped or on-premises deployments where data cannot leave the local machine
• Teams already using LangChain, LlamaIndex, Haystack, or Agno
• Production systems where incremental indexing without downtime is critical

Consider alternatives like FAISS or Pinecone if you need a managed cloud service, require sub-bit quantization with different tradeoffs, or need a more battle-tested solution at scale.

Resources

• GitHub Repository
• Python Package
• TurboQuant Paper