Skip to content

Giles Thomas personal blog

Writing an LLM from scratch, part 13 – attention heads are dumb

gilesthomas.com published 2025-05-0814 min read

A pause to take stock: realising that attention heads are simpler than I thought explained why we do the calculations we do.

Read on gilesthomas.com → Discuss on HN (67) Archived copy

Points: 351
HN comments: 67
Submissions: 1
Cited by: 0 commenters

AI & Machine Learning Software Engineering 2025

On Hacker News

Submitted	Title on HN	Points	Comments
May 8, 2025	Writing an LLM from scratch, part 13 – attention heads are dumb by gpjt	351	67

More from Giles Thomas

All 9 posts →

The maths you need to start understanding LLMs

2025616 points120 comments AI & Machine Learning details

LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

2025540 points121 comments AI & Machine Learning details

Writing an LLM from scratch, part 8 – trainable self-attention

2025380 points31 comments AI & Machine Learning details

It’s still worth blogging in the age of AI

2025333 points223 comments Life, Learning & Writing details

The benefits of learning in public

2025311 points95 comments Life, Learning & Writing details

Writing an LLM from scratch, part 22 – training our LLM

2025254 points10 comments AI & Machine Learning details

Related posts from the same era

Statement on US government directive to suspend access to Fable 5 and Mythos 5

Anthropic20263,158 points2,314 commentscited by 11 details

GPT-4o

OpenAI Blog20243,138 points2,366 commentscited by 16 details

Astra can read and write weights to improve Qwen

Rikiya Takehi20262,959 points899 comments details

HackerTube: YouTube built from videos recommended by HN

karpathy20262,886 points641 comments details

Claude Fable 5

Anthropic20262,626 points2,159 commentscited by 10 details

Claude Code is steganographically marking requests

Thereallo20262,445 points750 commentscited by 3 details