1
LLM from scratch, part 28 – training a base model from scratch on an RTX 3090
I felt like it should be possible to train a GPT-2 small level model on my own hardware using modern tools and open datasets from scratch. It was!
Giles Thomas
Currently mostly AI because in 2026 of course it is. Grinding through the nuts and bolts of LLMs and trying to work out why this stuff works...
- Popular posts
- 9
- Total points
- 3,212
- HN comments
- 1,014
- Rank
- #438 personal
Visit gilesthomas.com → RSS feedOn Hacker News Feb 2025 – Jun 2026
AI & Machine Learning 6Software Engineering 4Networking & Internet 2Life, Learning & Writing 2Infrastructure & DevOps 2Math & Algorithms 1
2
Writing an LLM from scratch, part 8 – trainable self-attention
Moving on from a toy self-attention mechanism, it's time to find out how to build a real trainable one. Following Sebastian Raschka's book 'Build a Large Language Model (from Scratch)'. Part 8/??
3
Writing an LLM from scratch, part 13 – attention heads are dumb
A pause to take stock: realising that attention heads are simpler than I thought explained why we do the calculations we do.
4
Writing an LLM from scratch, part 22 – training our LLM
Finally, we train an LLM! The final part of Chapter 5 of Build an LLM (from Scratch) runs the model on real text, then loads OpenAI’s GPT-2 weights for comparison.