rai

rai

CPU-only LLM inference engine in pure Rust — 4-bit quantized models, hand-written AVX2 kernels, speculative decoding, and a local HTTP/MCP server. No GPU, no Python runtime.

16 1
Apache-2.0
last commit 2026-09-19
Website Source
Share:

About

Run language models on your own computer's CPU: one-step installers for Windows, macOS and Linux, a local chat page and API, 4-bit models, pure Rust. No GPU, no Python, nothing leaves your machine.

rai website preview

Languages

Contributors1

No features listed.

Comments Theme
slug: rai