MIT HAN Lab company blog
How attention sinks keep language models stable
We discovered why language models catastrophically fail on long conversations: when old tokens are removed to save memory, models produce complete gibberish. We found models dump massive attention onto the first few tokens as "attention sinks"—places to park unused attention since softmax requires weights to sum to 1…
- Points
- 219
- HN comments
- 36
- Submissions
- 1
- Cited by
- 0 commenters
AI & Machine LearningSoftware Engineering2025
On Hacker News
| Submitted | Title on HN | Points | Comments |
|---|---|---|---|
| Aug 8, 2025 | How attention sinks keep language models stable by pr337h4m | 219 | 36 |
