New

The latest blog posts to take off on Hacker News. 6,220 posts.

Any topic
2025
3147

I was wrong about robots.txt

Recently, I wrote an article about my journey in learning about robots.txt and its implications on the data rights in regards to what I write in my blog. I was confident that I wanted to ban all the crawlers from my website. Turned out there was an unintended consequence that I did not account for. My LinkedIn posts…