Simon Willison personal blog
Understanding GPT tokenizers
Large language models such as GPT-3/4, LLaMA and PaLM work in terms of tokens. They take text, convert it into tokens (integers), then predict which tokens should come next. Playing …
- Points
- 432
- HN comments
- 130
- Submissions
- 1
- Cited by
- 0 commenters
AI & Machine LearningProgramming Languages2023
On Hacker News
| Submitted | Title on HN | Points | Comments |
|---|---|---|---|
| Jun 8, 2023 | Understanding GPT tokenizers by simonw | 432 | 130 |
