Andrew Chan personal blog
Crawling a billion web pages in just over 24 hours, in 2025
For some reason, nobody's written about what it takes to crawl a big chunk of the web in a while: the last point of reference I saw was Michael Nielsen's post from 2012 .
- Points
- 199
- HN comments
- 58
- Submissions
- 1
- Cited by
- 0 commenters
Systems & PerformanceInfrastructure & DevOps2026
On Hacker News
| Submitted | Title on HN | Points | Comments |
|---|---|---|---|
| Feb 23, 2026 | Crawling a billion web pages in just over 24 hours, in 2025 by pseudolus | 199 | 58 |
