Robots.txt meant for search engines don’t work well for web archives
Robots.txt files were invented 20+ years ago to help advise “robots," mostly search engine web crawlers, which sections of a web site should be crawled and indexed for search. Many sites use their robots.txt files to improve their SEO (search engine optimization) by excluding duplicate content like print versions of…