Nick Yoder personal blog

Language models pack billions of concepts into 12k dimensions

by Nicholas Yoder nickyoder.com published 2025-02-208 min read

In a recent 3Blue1Brown video series on transformer models, Grant Sanderson posed a fascinating question: How can a relatively modest embedding space of 12,288 dimensions (GPT-3) accommodate millions of distinct real-world concepts? The answer lies at the intersection of high-dimensional geometry and a remarkable…

Points

368

HN comments

142

Submissions

1

Cited by

0 commenters

AI & Machine LearningMath & Algorithms2025

Related posts from the same era

Statement on US government directive to suspend access to Fable 5 and Mythos 5

Anthropic20263,158 points2,314 commentscited by 11 details

GPT-4o

OpenAI Blog20243,138 points2,366 commentscited by 16 details

Astra can read and write weights to improve Qwen

RRikiya Takehi20262,953 points896 comments details

HackerTube: YouTube built from videos recommended by HN

karpathy20262,883 points641 comments details

Claude Fable 5

Anthropic20262,626 points2,159 commentscited by 10 details

Claude Code is steganographically marking requests

Thereallo20262,445 points750 commentscited by 3 details