gigatoken

gigatoken

Language model tokenization at GB/s

1.6k 52508 HN points
MIT
last commit 2026-07-22
Source
Share:

About

Language model tokenization at GB/s

Languages

Contributors1

No features listed.

Comments Theme
Install
pip
Cargo
slug: gigatoken