bentoml /BentoML
The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
★ 8,941⑂ 1,065👁 80⑃ 4,312● 1,186▤ 195
+0
Daily
8,941
stars
1,065
forks
80
watchers
4,312
prs
1,186
issues
195
releases
README
Unified Model Serving Framework
🍱 Build model inference APIs and multi-model serving systems with any open-source or custom AI models. 👉 Join our forum!
What is BentoML?
BentoML is a Python library for building online serving systems optimized for AI apps and model inference.
- 🍱 Easily build APIs for Any AI/ML Model. Turn any model inference script into a REST API server with just a few lines of code and standard Python type hints.
- 🐳 Docker Containers made simple. No more dependency hell! Manage your environments, dependencies and model versions with a simple config file. BentoML automatically generates Docker images, ensures reproducibility, and simplifies how you deploy to different environments.
- 🧭 Maximize CPU/GPU utilization. Build high performance inference APIs leveraging built-in serving optimization features like dynamic batching, model parallelism, multi-stage pipeline and multi-model inference-graph orchestration.
- 👩💻 Fully customizable. Easily implement your own APIs or task queues, with custom business logic, model inference and multi-model composition. Supports any ML framework, modality, and inference runtime.
- 🚀 Ready for Production. Develop, run and debug locally. Seamlessly deploy to production with Docker containers or Kubernetes.
Getting started
Install BentoML:
Requires Python≥3.9
pip install -U bentoml
Define APIs in a service.py file.
python
import bentoml
@bentoml.service(
image=bentoml.images.Image(python_version="3.11").python_packages("torch", "transformers"),
)
class Summarization:
def __init__(self) - None:
import torch
from transformers import pipeline
device = "cuda" if torch.cuda.is_available() else "cpu"
self.pipeline = pipeline('summarization', device=device)
@bentoml.api(batchable=True)
def summarize(self, texts: list[str]) - list[str]:
results = self.pipeline(texts)
return [item['summary_text'] for item in results]
💻 Run locally
Install PyTorch and Transformers packages to your Python virtual environment.
bash
pip install torch transformers additional dependencies for local run
Run the service code locally (serving at http://localhost:3000 by default):
bash
bentoml serve
You should expect to see the following output.
[INFO] [cli] Starting production HTTP BentoServer from "service:Summarization" listening on http://localhost:3000 (Press CTRL+C to quit)
[INFO] [entry_service:Summarization:1] Service Summarization initialized
Now you can run inference from your browser at http://localhost:3000 or with a Python script:
python
import bentoml
with bentoml.SyncHTTPClient('http://localhost:3000') as client:
summarized_text: str = client.summarize([bentoml.__doc__])[0]
print(f"Result: {summarized_text}")
🐳 Deploy using Docker
Run bentoml build to package necessary code, models, dependency configs into a Bento - the standardized deployable artifact in BentoML:
bash
bentoml build
Ensure Docker is rTop issues
#4117bug: containerize filed30 comments · 2023-08-11#3669bug: aiohttp.client_exceptions.ServerDisconnectedError: Server disconnected19 comments · 2023-03-14#3787bug: Http Server is not worked with Pytorch-Lightning without 'del sys.modules["prometheus_client"]'18 comments · 2023-04-25#5426bug: Uneven request distribution across workers in BentoML service11 comments · 2025-08-06#5466feature: support for pylock.toml7 comments · 2025-09-12#3463feature: Allow for New relic integration7 comments · 2023-01-20#3200bug: Batching async run issue 7 comments · 2022-11-08#2693feat: ONNX-mlir6 comments · 2022-07-05
Discussions
#5748BentoML CLI included in AgentManager's tool catalog0 comments · 2026-10-18#5635MCP to help OSS devs understand bentoml/BentoML0 comments · 2026-06-12#5623image tiling for deployment2 comments · 2026-05-31#5574ORCH — CLI runtime for orchestrating AI agent teams (Claude Code, Codex, Cursor)0 comments · 2026-03-17#5545Posting content SSE conform with bentoml3 comments · 2026-02-05#5422Help needed:: Guidance on Deploying to Modern BentoCloud2 comments · 2025-07-27#5367Provide a predictable container path for secrets3 comments · 2025-05-21#5325BentoML code evolution2 comments · 2025-04-18
Latest release
v1.4.40 v1.4.40 2026-10-23Security: eMudhra intermediate CA compromise - \certifi=2026.10.19\ is now required. It drops trust in the compromised eMudhra/emSign CAs. - Generated Dockerfiles and local \bentoml build\ install \uv==0.13.4\ with \--require-hashes\ whenever uv is missing or older (uv < 0.13.4 bundles the compromised CAs). pip 26.2.2 is used where supported. (5757) Action for users: run \bentoml build\ again with 1.4.40, then redeploy, so your bento image picks up the fixed certifi and uv. Also in this release - Agent skills for creating BentoML projects and for deploying to Kubernetes and EC2 (5705, 568