Best of 2025

The blog posts Hacker News loved most in 2025. 499 posts.

Web & Frontend
345

I was wrong about robots.txt

Recently, I wrote an article about my journey in learning about robots.txt and its implications on the data rights in regards to what I write in my blog. I was confident that I wanted to ban all the crawlers from my website. Turned out there was an unintended consequence that I did not account for. My LinkedIn posts…