New

The latest blog posts to take off on Hacker News. 185 posts.

Infrastructure & DevOps
2023
3

Indexing a billion pages

It’s two years since we launched Mwmbl, the open source, non-profit search engine, on Boxing Day 2021. A good time to take stock of where we are and where we’re going. We’ve indexed over 100 million pages Thanks to our volunteers, who crawl the web using the Firefox extension and command line script, we’re crawling up…