Andrew Chan

andrewkchan.dev

I'm a software engineer living in San Francisco. Previously I was at South Park Commons and Figma , and graduated from UC Berkeley in 2019. This is my blog where I post about things I'm interested in, mostly tech-related.

Popular posts: 4
Total points: 1,642
HN comments: 315
Rank: #1,092 personal

Visit andrewkchan.dev → On Hacker News Dec 2023 – Feb 2026

Systems & Performance 2 AI & Machine Learning 2 Software Engineering 1 Math & Algorithms 1 Infrastructure & DevOps 1 Games & Graphics 1

Most points Most discussed Most cited Newest

Crawling a billion web pages in just over 24 hours, in 2025

2026199 points58 comments Systems & Performance details

For some reason, nobody's written about what it takes to crawl a big chunk of the web in a while: the last point of reference I saw was Michael Nielsen's post from 2012 .