Evgenii Pendragon personal blog
I was wrong about robots.txt
Recently, I wrote an article about my journey in learning about robots.txt and its implications on the data rights in regards to what I write in my blog. I was confident that I wanted to ban all the crawlers from my website. Turned out there was an unintended consequence that I did not account for. My LinkedIn posts…
- Points
- 171
- HN comments
- 204
- Submissions
- 1
- Cited by
- 0 commenters
Web & FrontendTools & Productivity2025
On Hacker News
| Submitted | Title on HN | Points | Comments |
|---|---|---|---|
| Jul 17, 2025 | I was wrong about robots.txt by EPendragon | 171 | 204 |
