Common Crawl Blog

commoncrawl.org

We build and maintain an open repository of web crawl data that can be accessed and analyzed by anyone.

Popular posts: 3
Total points: 448
HN comments: 99
Rank: #3,022 company

Visit commoncrawl.org → On Hacker News Dec 2011 – Nov 2013

Databases & Data 3 Software Engineering 1

Most points Most discussed Most cited Newest

MapReduce for the Masses: Zero to Hadoop in 5 minutes with Common Crawl

2011109 points26 comments Databases & Data details

Common Crawl aims to change the big data game with our repository of over 40 terabytes of high-quality web crawl information into the Amazon cloud, the net total of 5 billion crawled pages.