Complete guide to deploying ML models with BentoML, from basic service definition to Kubernetes deployment.

Deploying ML Models with BentoML: From Development to Production

Introduction

BentoML is an open-source platform for building, packaging, and deploying machine learning models and AI applications at scale. It provides a unified interface for model serving with consistent APIs and production-ready infrastructure. In this tutorial, you'll learn how to deploy ML models using BentoML.

Prerequisites

Python 3.8+
Basic understanding of ML models (PyTorch, TensorFlow, scikit-learn, etc.)
Docker installed (for production deployment)

Installation

pip install bentoml

Core Concepts

Models

BentoML's model store provides versioned model management:

import bentoml

# Save a model to the store
bentoml.pytorch.save("resnet50", model, metadata={"arch": "resnet50"})

# Load a model
model = bentoml.pytorch.get("resnet50:v1.2.3")

Services

Services define the inference API:

import bentoml

@bentoml.service
class MyService:
    @bentoml.api(input=bentoml.numpy(), output=bentoml.json())
    async def predict(self, input_data):
        return self.model(input_data)

Bentos

Bentos are the deployment units — self-contained packages with everything needed for deployment:

my_bento/
├── service.py          # Service definition
├── models/             # Model references
├── requirements.txt    # Dependencies
├── bentofile.yaml      # Build configuration
└── .dockerignore

Step 1: Define Your Service

Create a service.py file:

import bentoml
import torch
import numpy as np
from PIL import Image

# Load model from BentoML model store
model = bentoml.pytorch.get("resnet50:latest").to("cuda")

@bentoml.service(
    resources={"gpu": 1, "cpu": 2},
    traffic={"timeout": 10},
)
class ImageClassifier:
    def __init__(self):
        self.model = model
        self.classes = load_imagenet_classes()

    @bentoml.api(
        input=bentoml.images(),
        output=bentoml.json()
    )
    async def predict(self, image: Image.Image) -> dict:
        # Preprocess
        input_tensor = preprocess(image).to("cuda")
        
        # Inference
        with torch.no_grad():
            output = self.model(input_tensor)
        
        # Postprocess
        prediction = postprocess(output, self.classes)
        
        return {"prediction": prediction}

Step 2: Build a Bento

bentoml build service.py

This creates a bentoml directory with your packaged application:

ls bentoml/
# image_classifier:20260530_123456_abc123/

Step 3: Run Locally

bentoml serve image_classifier:latest

Visit http://localhost:3000 to see the interactive API documentation.

Step 4: Test Your API

Using curl

curl -X POST http://localhost:3000/predict \
  -H "Content-Type: image/png" \
  --data-binary @test_image.png

Using Python

import requests
from PIL import Image

response = requests.post(
    "http://localhost:3000/predict",
    files={"image": open("test_image.png", "rb")}
)
print(response.json())

Using the OpenAPI Client

from openapi_client import DefaultApi, Configuration

config = Configuration()
config.host = "http://localhost:3000"
api = DefaultApi(api_client=config)

result = api.predict(image=open("test_image.png", "rb"))
print(result)

Advanced: Batch Inference

For high-throughput scenarios, use batch inference:

@bentoml.api(
    input=bentoml.numpy(),
    batch=True,
    batch_max_latency_ms=100,
    batch_max_batch_size=32
)
async def predict_batch(self, inputs: np.ndarray) -> np.ndarray:
    """Process multiple inputs in a single batch."""
    with torch.no_grad():
        outputs = self.model(torch.stack(inputs))
    return outputs.numpy()

Advanced: Multi-Model Serving

Serve multiple models in a single deployment:

@bentoml.service
class MultiModelService:
    def __init__(self):
        self.classifier = bentoml.pytorch.get("resnet50:latest")
        self.detector = bentoml.onnx.get("yolo:latest")

    @bentoml.api
    async def classify(self, image: Image.Image) -> dict:
        return {"classification": self.classifier.predict(image)}

    @bentoml.api
    async def detect(self, image: Image.Image) -> dict:
        return {"detections": self.detector.predict(image)}

Advanced: Hugging Face Model Serving

import bentoml
from transformers import AutoModelForSequenceClassification, AutoTokenizer

@bentoml.service(
    resources={"cpu": 2, "memory": "4Gi"},
    traffic={"timeout": 60}
)
class TextClassifier:
    def __init__(self):
        self.model_id = "distilbert-base-uncased-finetuned-sst-2-english"
        self.tokenizer = AutoTokenizer.from_pretrained(self.model_id)
        self.model = AutoModelForSequenceClassification.from_pretrained(self.model_id)

    @bentoml.api(input=bentoml.str(), output=bentoml.json())
    async def classify(self, text: str) -> dict:
        inputs = self.tokenizer(text, return_tensors="pt")
        with torch.no_grad():
            outputs = self.model(**inputs)
        label = outputs.logits.argmax().item()
        score = float(torch.softmax(outputs.logits, dim=-1)[0][label])
        return {"label": label, "score": score}

Advanced: LLM Serving with vLLM

import bentoml
from vllm import LLM

@bentoml.service(
    resources={"gpu": 1, "memory": "16Gi"},
    traffic={"timeout": 120}
)
class LLMService:
    def __init__(self):
        self.llm = LLM(model="meta-llama/Llama-2-7b-hf")

    @bentoml.api(input=bentoml.str(), output=bentoml.str())
    async def generate(self, prompt: str) -> str:
        outputs = self.llm.generate(prompt)
        return outputs[0].outputs[0].text

Deployment

Deploy to Kubernetes

# Containerize the bento
bentoml containerize image_classifier:latest

# Tag and push to your registry
docker tag image_classifier:latest myregistry/image_classifier:latest
docker push myregistry/image_classifier:latest

# Deploy with bentoml deploy
bentoml deploy image_classifier --platform kubernetes

Deploy to AWS

bentoml deploy image_classifier --platform aws

Deploy to GCP

bentoml deploy image_classifier --platform gcp

Deploy to Azure

bentoml deploy image_classifier --platform azure

Custom Dockerfile

For advanced customization, create a custom Dockerfile:

# bentofile.yaml
docker:
  python_version: "3.11"
  setup_sh: "setup.sh"
  dockerfile_template: "docker/Dockerfile"

# docker/Dockerfile
FROM bentoml/image:3.11-cuda12

# Add custom dependencies
RUN pip install custom-package

# Add custom startup script
COPY setup.sh /app/setup.sh
RUN chmod +x /app/setup.sh

GPU Support

Specify GPU requirements:

@bentoml.service(
    resources={
        "gpu": 1,
        "gpu_type": "nvidia-a100"
    },
    traffic={"timeout": 300}
)
class GPUService:
    ...

Observability

BentoML includes built-in observability:

# Metrics are automatically exposed
# Visit /metrics for Prometheus metrics

# Logging is structured and searchable
# Integrate with your logging infrastructure

Best Practices

1. Use Model Versioning

# Always version your models
bentoml.pytorch.save("resnet50", model, metadata={"version": "v1.2.3"})

# Load specific versions in production
model = bentoml.pytorch.get("resnet50:v1.2.3")

2. Set Appropriate Timeouts

@bentoml.service(
    traffic={
        "timeout": 30,  # 30 seconds per request
        "max_queue_size": 100
    }
)

3. Use Batch Inference for High Throughput

@bentoml.api(
    batch=True,
    batch_max_latency_ms=100,  # Max 100ms wait for batch
    batch_max_batch_size=32    # Max 32 items per batch
)

4. Monitor Resource Usage

# Check GPU memory usage
import torch
print(f"GPU memory allocated: {torch.cuda.memory_allocated() / 1024**2:.2f} MB")

Complete Example: Production-Ready Image Classifier

import bentoml
import torch
import numpy as np
from PIL import Image
from typing import List, Dict

# Preprocessing function
def preprocess(image: Image.Image) -> torch.Tensor:
    transform = transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize(mean=[0.485, 0.456, 0.406], 
                            std=[0.229, 0.224, 0.225])
    ])
    return transform(image).unsqueeze(0)

# Postprocessing function
def postprocess(output: torch.Tensor, classes: List[str]) -> Dict:
    probabilities = torch.softmax(output, dim=1)
    top5 = torch.topk(probabilities, 5)
    
    return {
        "predictions": [
            {"class": classes[i], "probability": float(p)}
            for i, p in zip(top5.indices[0], top5.values[0])
        ]
    }

@bentoml.service(
    resources={"gpu": 1, "cpu": 2},
    traffic={"timeout": 30},
)
class ImageClassifier:
    def __init__(self):
        self.model = bentoml.pytorch.get("resnet50:production").to("cuda")
        self.model.eval()
        self.classes = self._load_imagenet_classes()

    def _load_imagenet_classes(self) -> List[str]:
        # Load ImageNet class labels
        import json
        with open("imagenet_classes.json") as f:
            return json.load(f)

    @bentoml.api(
        input=bentoml.images(),
        output=bentoml.json()
    )
    async def predict(self, image: Image.Image) -> dict:
        input_tensor = preprocess(image).to("cuda")
        
        with torch.no_grad():
            output = self.model(input_tensor)
        
        return postprocess(output, self.classes)

    @bentoml.api(
        input=bentoml.images(),
        output=bentoml.json(),
        batch=True,
        batch_max_latency_ms=200,
        batch_max_batch_size=16
    )
    async def predict_batch(self, images: List[Image.Image]) -> List[dict]:
        tensors = torch.stack([preprocess(img) for img in images]).to("cuda")
        
        with torch.no_grad():
            outputs = self.model(tensors)
        
        return [
            postprocess(output.unsqueeze(0), self.classes)
            for output in outputs
        ]

Resources

Summary

In this tutorial, you learned:

How to define a BentoML service with @bentoml.service
How to build a Bento with bentoml build
How to run locally with bentoml serve
Advanced features: batch inference, multi-model serving, GPU support
How to deploy to Kubernetes, AWS, GCP, and Azure
Best practices for production deployment

BentoML provides a standardized, production-ready way to deploy ML models with consistent APIs and scalable infrastructure. Start with local development and gradually add production features as your needs grow.