LLM Inference & Serving AI Tools
Open-source tools and runtimes for running large language models locally or serving them via API endpoints.
Self-hosting a model means taking on the work a hosted API hides: loading weights, managing the KV cache, batching concurrent requests, and quantizing to fit the memory actually available. People arrive here for one of three reasons: data that cannot leave the building, per-token costs that stopped making sense at volume, or a need to work offline. The field has settled into roughly three camps. Local single-user runtimes such as Ollama and llama.cpp lean on GGUF quantization and CPU offload so a large model fits in consumer memory. Throughput servers such as vLLM and SGLang assume a GPU that stays fed, using paged attention and continuous batching to maximize aggregate tokens per second across many concurrent sessions. Above both sits a gateway layer that normalizes provider APIs behind a single client so local and remote endpoints stay interchangeable. The separating tradeoff is latency on one request against throughput across hundreds. A sensible order is Ollama first to get a model answering, llama.cpp when quantization and sampling details start to matter, and vLLM once real concurrency arrives. The trap to check early is that the runtime license and the weight license are separate documents. vLLM and llama.cpp are permissive, but the checkpoint loaded into them may carry a community license with user caps or a non-commercial clause, and that is the one governing the deployment. The other floor is hardware: the paged-attention servers expect a recent high-VRAM NVIDIA card and degrade sharply below it.