LMDeploy
Toolkit for compressing, deploying, and serving large language models with optimized inference.
About
LMDeploy is a toolkit from InternLM for compressing, deploying, and serving large language models. It provides persistent batching, blocked KV cache, weight-only and KV quantization, and high-performance CUDA kernels. The platform ships two engines, TurboMind and a PyTorch backend, and supports multi-model deployment across machines and GPUs.
Reviews (0)
Leave a Review
No reviews yet. Be the first to review!
Details
- Category
- LLM Inference & Serving
- Price
- Free
- Platform
- Local/Desktop
- Difficulty
- Intermediate (3/5)
- License
- Apache-2.0
- Added
- May 7, 2026
Related Tools
Port of Meta's LLaMA model in C/C++ for efficient CPU inference
High-throughput LLM serving engine with PagedAttention
Minimalist ML framework in Rust by Hugging Face for fast inference.
Optimized inference library for running quantized LLMs on consumer GPUs.
Open-source ChatGPT alternative that runs 100% offline on your computer.
Fast LLM inference on consumer GPUs using neuron-aware sparse computation.