NVIDIA /TensorRT-LLM
TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.
★ 15,079⑂ 2,936👁 119⑃ 17,811● 4,252▤ 96
+4
Daily
+77
Weekly
+277
Monthly
15,079
stars
2,936
forks
119
watchers
17,811
prs
4,252
issues
96
releases
README
TensorRT LLM =========================== TensorRT LLM optimizes inference for LLMs and Visual Gen models with specialized kernels for common operations, an efficient runtime, and a pythonic framework that enables you to customize and extend the system. Architecture | Performance | Examples | Documentation | Roadmap --- Tech Blogs [11/11] Dense DWDP: Extending Distributed Weight Data Parallelism to Dense Weights on Vera Rubin NVL72 ✨ ➡️ link [11/03] ISER-MoE: In-Switch Expert Replication for Load-Balanced MoE Prefill ✨ ➡️ link [10/14] KV Cache Compression in TensorRT LLM ✨ ➡️ link [09/22] GVR V2: Self-Sampling and Multi-Thresholding for Faster Exact Top-K ✨ ➡️ link [09/02] Accelerating Video Generation with GEMM Quantization, Attention Quantization and Skip Softmax Attention in TensorRT-LLM ✨ ➡️ link [08/20] Evaluating Agentic Serving with Trace Replay and Job-Level Metrics ✨ ➡️ link [07/17] DeepSeek-V4 on NVIDIA Blackwell: Model-Specific and Agentic-Workload Optimizations in TensorRT LLM ✨ ➡️ link [07/01] Scaling Video Generation Across NVL72 Rack with TensorRT-LLM ✨ ➡️ link [05/15] Joint Optimization of Agent Applications and TensorRT-LLM ✨ ➡️ link [04/03] Tuning CUDA Graph Batch Sizes for Higher Output Throughput ✨ ➡️ link [04/03] DWDP: Distributed Weight Data Parallelism for High-Performance LLM Inference on NVL72 ✨ ➡️ link [03/16] Optimizing MoE Communication with One-Sided AlltoAll Over NVLink ✨ ➡️ link [03/04] Sparse Attention in TensorRT LLM ✨ ➡️ link [02/06] Accelerating Long-Context Inference with Skip Softmax Attention ✨ ➡️ link [01/09] Optimizing DeepSeek-V3.2 on NVIDIA Blackwell GPUs ✨ ➡️ link Previous Blogs [10/13] Scaling Expert Parallelism in TensorRT LLM (Part 3: Pushing the Performance Boundary) ✨ ➡️ link [09/26] Inference Time Compute Implementation in TensorRT LLM ✨ ➡️ link [09/19] Combining Guided Decoding and Speculative Decoding: Making CPU and GPU Cooperate Seamlessly ✨ ➡️ link [08/29] ADP Balance Strategy ✨ ➡️ link [08/05] Running a High-Performance GPT-OSS-120B Inference Server with TensorRT LLM ✨ ➡️ link [08/01] Scaling Expert Parallelism in TensorRT LLM (Part 2: Performance Status and Optimization) ✨ ➡️ link [07/26] N-Gram Speculative Decoding in TensorRT LLM ✨ ➡️ link [06/19] Disaggregated Serving in TensorRT LLM ✨ ➡️ link [06/05] Scaling Expert Parallelism in TensorRT LLM (Part 1: Design and Implementation of Large-scale EP) ✨ ➡️ link [05/30] Optimizing DeepSeek R1 Throughput on NVIDIA Blackwell GPUs: A Deep Dive for Developers ✨ ➡️ link [05/23] DeepSeek R1 MTP Implementation and Optimization ✨ ➡️ link [05/16] Pushing Latency Boundaries: Optimizing DeepSeek-R1 Performance on NVIDIA B200 GPUs ✨ ➡️ link Latest News [04/03] 🎨 TensorRT LLM now supports diffusion models for visual generation ➡️ link Previous News
Top issues
#2953Running into free(): double free detected in tcache 2 when using trtllm-bench in a multi-node scenario32 comments · 2025-03-19#5581Running DeepSeek R1 on 8x RTX 6000 PRO24 comments · 2025-06-29#3334Support cpu inference with scaffolding21 comments · 2025-04-07#2338Whisper Encoder issues with Executor API18 comments · 2024-10-15#12071[Bug]: BeamSearch with beam_width 128 Encountered an error in cudaMemcpy2DAsync16 comments · 2026-03-10#14500`1.3.0rc15`: trtllm-serve hangs in fused_moe/quantization.py during cold-start on GB10 / sm_121 — `1.3.0rc12` works with identical config15 comments · 2026-05-24#9705[Usage]: How to perform Wide Expert Parallelism (Wide EP) inference testing for Qwen3-235B-A22B-FP8 on 8 GPUs using TensorRT LLM?13 comments · 2025-12-04#3701[Call for contributions] Feature parity between TensorRT and PyTorch backend - Quantization13 comments · 2025-04-20
Discussions
#16360SM120 (RTX PRO 6000) MoE backend gaps: MoeConfig.backend schema missing MegaMoE entries, and no EP-capable path for this hardware0 comments · 2026-07-14#15508Pre-inference context optimization: reducing prompt tokens before GPU0 comments · 2026-06-20#15476CUDA cudaMemcpyAsync invalid argument error when using Triton instance_group multi-instance load balancing with TensorRT-LLM BLS model on RTX 50902 comments · 2026-06-18#14696Hi Jhao-Ting0 comments · 2026-05-28#14425Scaffolding idea1 comments · 2026-05-21#12948Is there context parallelism method using exact algorithm for MLA prefill in pytorch backend?0 comments · 2026-04-11#12308Showcase / question: a board-proven offline language runtime on ESP32-C3, and whether this hints at a more specialized future for language inference0 comments · 2026-03-18#12164How can I do tool calling for gpt-oss-120B0 comments · 2026-03-12
Latest release
v1.2.1 v1.2.1 2026-04-20Highlights - Fixed Issue - Fixed an issue that caused KV cache corruption (12770) - Infrastructure Changes - Upgraded xgrammar and flashinfer (12811)