inference

SGLang is a high-performance serving framework for large language models and multimodal models.

LMCache: Supercharge Your LLM with the Fastest KV Cache Layer

Open-source inference server and production cluster for all the models your agent needs.

Native LLM inference server for Apple Silicon. OpenAI + Anthropic API compatible. No Python. Includes MLX Core macOS app with chat, agent mode, and tool calling.

Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM

Evidence-aware on-prem LLM inference sizing and TCO calculator

Reame — CPU-first LLM inference server on llama.cpp: disk KV cache, self-regulating speculation, generation archive, interleaved multi-user, the Conclave. Your hardware, your realm.