sidsite personal blog
Notes on training BERT from scratch on an 8GB consumer GPU
I trained a BERT model (Devlin et al, 2019) from scratch on my desktop PC (which has a Nvidia 3060 Ti 8GB GPU). The model architecture, tokenizer, and trainer all came from Hugging Face libraries, and my contribution was mainly setting up the code, setting up the data (~20GB uncompressed text), and leaving my computer…
- Points
- 171
- HN comments
- 43
- Submissions
- 1
- Cited by
- 0 commenters
AI & Machine LearningSoftware Engineering2023
On Hacker News
| Submitted | Title on HN | Points | Comments |
|---|---|---|---|
| Jun 1, 2023 | Notes on training BERT from scratch on an 8GB consumer GPU by montebicyclelo | 171 | 43 |
