The tools and repos trending this week, in your inbox.
Free. No spam. Unsubscribe any time.
I animated what GPT-4o's tokenizer does to "strawber... — Trendlair
Discover/I animated what GPT-4o's tokenizer does to "strawberry": real token ids, a tiny BPE run, and why the space belongs to the word (6 min video) [D]
article
I animated what GPT-4o's tokenizer does to "strawberry": real token ids, a tiny BPE run, and why the space belongs to the word (6 min video) [D]