BentoML

Overview

BentoML is an open-source platform for building, packaging, and deploying machine learning models and AI applications at scale. It provides a unified interface for model serving, enabling developers to deploy models from any framework (PyTorch, TensorFlow, scikit-learn, Hugging Face, etc.) with consistent APIs and production-ready infrastructure.

BentoML's core innovation is the "Bento" — a standardized, self-contained deployment package that includes the model, code, dependencies, and configuration. This abstraction simplifies the complex process of taking a model from development to production, handling everything from environment management to horizontal scaling.

Features

• Framework Agnostic: Support for PyTorch, TensorFlow, JAX, scikit-learn, Hugging Face, XGBoost, and more
• Bento Packaging: Self-contained deployment units with all dependencies
• High-Performance Serving: Optimized inference servers with batch processing and streaming
• Auto-Scaling: Kubernetes-native auto-scaling with GPU support
• Model Registry: Version control and lifecycle management for models
• Multi-Model Serving: Serve multiple models in a single deployment
• Real-Time & Batch: Support for both real-time inference and batch processing
• Observability: Built-in metrics, logging, and tracing integration
• Cloud Native: Deploy to Kubernetes, AWS, GCP, Azure, or BentoML Cloud

Installation

pip install bentoml

Quick Start

Define a Service

# service.py
import bentoml
import numpy as np
from PIL import Image

# Load a model from the BentoML model store
model = bentoml.pytorch.get("resnet50:latest").to("cuda")

@bentoml.service(
    resources={"gpu": 1, "cpu": 2},
    traffic={"timeout": 10},
)
class ImageClassifier:
    @bentoml.api
    async def predict(self, image: Image.Image) -> dict:
        input_tensor = preprocess(image)
        with torch.no_grad():
            output = model(input_tensor)
        return {"prediction": postprocess(output)}

Build a Bento

bentoml build service.py

This creates a bentoml directory with your packaged application.

Run Locally

bentoml serve image_classifier:latest

Deploy to Kubernetes

bentoml containerize image_classifier:latest
docker tag image_classifier:latest myregistry/image_classifier:latest
docker push myregistry/image_classifier:latest

# Deploy with bentoml deploy
bentoml deploy image_classifier --platform kubernetes

Core Concepts

Models

BentoML's model store provides versioned model management:

import bentoml

# Save a model to the store
bentoml.pytorch.save("resnet50", model, metadata={"arch": "resnet50"})

# Load a model
model = bentoml.pytorch.get("resnet50:v1.2.3")

Services

Services define the inference API:

@bentoml.service
class MyService:
    @bentoml.api(input=bentoml.numpy(), output=bentoml.json())
    def predict(self, input_data):
        return self.model(input_data)

Bentos

Bentos are the deployment units:

my_bento/
├── service.py          # Service definition
├── models/             # Model references
├── requirements.txt    # Dependencies
├── bentofile.yaml      # Build configuration
└── .dockerignore

Advanced Features

Batch Inference

@bentoml.api(
    input=bentoml.numpy(),
    batch=True,
    batch_max_latency_ms=100,
    batch_max_batch_size=32
)
async def predict_batch(self, inputs):
    return self.model(torch.stack(inputs))

Multi-Model Serving

@bentoml.service
class MultiModelService:
    def __init__(self):
        self.classifier = bentoml.pytorch.get("resnet50:latest")
        self.detector = bentoml.onnx.get("yolo:latest")

    @bentoml.api
    async def classify(self, image):
        return self.classifier.predict(image)

    @bentoml.api
    async def detect(self, image):
        return self.detector.predict(image)

Custom Dockerfile

# bentofile.yaml
docker:
  python_version: "3.11"
  setup_sh: "setup.sh"
  dockerfile_template: "docker/Dockerfile"

GPU Support

@bentoml.service(
    resources={"gpu": 1, "gpu_type": "nvidia-a100"},
    traffic={"timeout": 300}
)
class GPUService:
    ...

Examples

Hugging Face Model Serving

import bentoml
from transformers import AutoModelForSequenceClassification, AutoTokenizer

@bentoml.service(
    resources={"cpu": 2, "memory": "4Gi"},
    traffic={"timeout": 60}
)
class TextClassifier:
    def __init__(self):
        self.model_id = "distilbert-base-uncased-finetuned-sst-2-english"
        self.tokenizer = AutoTokenizer.from_pretrained(self.model_id)
        self.model = AutoModelForSequenceClassification.from_pretrained(self.model_id)

    @bentoml.api(input=bentoml.str(), output=bentoml.json())
    async def classify(self, text: str) -> dict:
        inputs = self.tokenizer(text, return_tensors="pt")
        with torch.no_grad():
            outputs = self.model(**inputs)
        return {"label": outputs.logits.argmax().item(), "score": float(torch.softmax(outputs.logits, dim=-1)[0][1])}

LLM Serving with vLLM

import bentoml
from vllm import LLM

@bentoml.service(
    resources={"gpu": 1, "memory": "16Gi"},
    traffic={"timeout": 120}
)
class LLMService:
    def __init__(self):
        self.llm = LLM(model="meta-llama/Llama-2-7b-hf")

    @bentoml.api(input=bentoml.str(), output=bentoml.str())
    async def generate(self, prompt: str) -> str:
        outputs = self.llm.generate(prompt)
        return outputs[0].outputs[0].text

Pros

• ✅ Framework-agnostic (supports all major ML frameworks)
• ✅ Production-ready serving with auto-scaling
• ✅ Unified packaging with Bento abstraction
• ✅ Kubernetes-native deployment
• ✅ GPU support with auto-scaling
• ✅ Open-source with active community
• ✅ Built-in observability and monitoring
• ✅ Model registry with version control

Cons

• ❌ Steeper learning curve than simple HTTP servers
• ❌ Requires Kubernetes knowledge for production
• ❌ Larger deployment footprint than lightweight alternatives
• ❌ Less focused on LLM-specific features
• ❌ BentoML Cloud is a paid service

When to Use

• Production ML model serving — Enterprise-grade deployment
• Multi-framework environments — Unified serving for PyTorch, TF, etc.
• GPU-intensive workloads — Auto-scaling with GPU support
• Kubernetes deployments — Native K8s integration
• Model lifecycle management — Version control and A/B testing

Use Cases

Use Case	Why BentoML
Production ML Serving	Enterprise-grade deployment with auto-scaling
Multi-Framework Environments	Unified serving for PyTorch, TensorFlow, scikit-learn
GPU Workloads	Auto-scaling with GPU support for inference
Model Registry	Version control and lifecycle management

Comparison with Alternatives

Feature	BentoML	TorchServe	TF Serving	Seldon
Framework Support	✅ All major	⚠️ PyTorch only	⚠️ TensorFlow only	✅ Multiple
Bento Packaging	✅ Yes	❌ No	❌ No	⚠️ Partial
Kubernetes Native	✅ Yes	✅ Yes	✅ Yes	✅ Yes
GPU Auto-scaling	✅ Yes	⚠️ Limited	⚠️ Limited	✅ Yes
Model Registry	✅ Yes	❌ No	❌ No	✅ Yes
Learning Curve	Medium	Medium	Medium	High
Best for	Multi-framework	PyTorch focus	TensorFlow focus	Enterprise MLOps

Best Practices

1. Define services early — Structure your service with clear API endpoints
2. Use Bento packaging — Always build Bentos for consistent deployments
3. Configure resources properly — Set GPU/CPU/memory based on model requirements
4. Enable batching — Use batch inference for better throughput
5. Test locally first — Use bentoml serve before deploying to Kubernetes

Troubleshooting

Issue	Solution
Bento build fails	Check bentofile.yaml configuration and dependencies
GPU not detected	Verify CUDA drivers and NVIDIA container toolkit
Model loading slow	Use model store caching and pre-load models
Kubernetes deployment fails	Check resource limits and GPU node availability

Resources

• Official Documentation
• GitHub Repository
• BentoML Cloud
• Deployment Guide

---

Last updated: May 2026