Document my learning notes.
Visit lilianweng.github.io → RSS feedOn Hacker News Oct 2018 – Aug 2026
AI & Machine Learning 13Math & Algorithms 5Systems & Performance 3Software Engineering 3Life, Learning & Writing 2Tools & Productivity 1
[Updated on 2022-03-13: add expert choice routing.] [Updated on 2022-06-10]: Greg and I wrote a shorted and upgraded version of this post, published on OpenAI Blog: “Techniques for Training Large Neural Networks”
[Updated on 2023-01-24: add a small section on Distillation.] Large transformer models are mainstream nowadays, creating SoTA results for a variety of tasks. They are powerful but very expensive to train and use. The extremely high inference cost, in both time and memory, is a big bottleneck for adopting a powerful…