vllm

LMCache: Supercharge Your LLM with the Fastest KV Cache Layer

Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM

The Unified Latent-State Memory Fabric (UL-SMF) is a hardware-software co-designed memory compression fabric that solves the memory bottleneck in long-context Transformer inference. By combining FSQ with dynamic 16-dimensional latent mapping, UL-SMF compresses Key-Value (KV) cache tensors by up to 384x while maintaining >94% semantic retention.