Shyam's Blog

Beyond self-attention: How a small language model predicts the next token

A deep dive into the internals of a small transformer model to learn how it turns self-attention calculations into accurate predictions for the next token.