Internet Archive Blog company blog
Robots.txt meant for search engines don’t work well for web archives
Robots.txt files were invented 20+ years ago to help advise “robots," mostly search engine web crawlers, which sections of a web site should be crawled and indexed for search. Many sites use their robots.txt files to improve their SEO (search engine optimization) by excluding duplicate content like print versions of…
- Points
- 295
- HN comments
- 143
- Submissions
- 1
- Cited by
- 43 commenters
Society, Policy & CultureWeb & Frontend2017
On Hacker News
| Submitted | Title on HN | Points | Comments |
|---|---|---|---|
| Apr 21, 2017 | Robots.txt meant for search engines don’t work well for web archives by r721 | 295 | 143 |
Cited in comments
Linked from 52 Hacker News comments by 43 different people, from Apr 2017 to May 2026.
Most recent:
- ycombinete · May 20, 2026
- joecool1029 · May 13, 2026
- danrl · Mar 21, 2026
- 1gn15 · Oct 6, 2025
- walski · Sep 1, 2025
- joecool1029 · Mar 20, 2025
