local-llm

Qwen3.8-27B on a single RTX 3090 with vLLM: ~1,000 tok/s at 64 concurrent (int8 tensor-core GEMMs, fp16 DeltaNet state), ~114 tok/s single-user at default sampling / ~124 greedy (MTP drafts, own-output draft vocab, calibrated int4 lm_head, split-KV verify attention), 150k-262k context; patches, requant scripts, benchmarks

Your iPhone helps your Mac run a 27B model: faster prompt reading and more context over a USB-C cable

The free open source Grammarly alternative offline writing assistant assist with grammar, rewriting & tone entirely on your machine.

Benchmark your local LLMs: speed, memory, and quality, in one command. TUI leaderboard for Ollama, LM Studio, llama.cpp, and vLLM.

Run Qwen3.6-35B-A3B locally on a 16 GB Apple Silicon Mac

Catches silent CPU fallback and mislabeled tok/s in local LLMs. llama.cpp and ollama, one file, no deps.