1
102TB of New Crawl Data Available
We are very please to announce that new crawl data is now available! The data was collected in 2013, contains approximately 2 billion web pages and is 102TB in size (uncompressed).
Common Crawl Blog
We build and maintain an open repository of web crawl data that can be accessed and analyzed by anyone.
- Popular posts
- 3
- Total points
- 448
- HN comments
- 99
- Rank
- #3,022 company
Visit commoncrawl.org → On Hacker News Dec 2011 – Nov 2013
2
MapReduce for the Masses: Zero to Hadoop in 5 minutes with Common Crawl
Common Crawl aims to change the big data game with our repository of over 40 terabytes of high-quality web crawl information into the Amazon cloud, the net total of 5 billion crawled pages.
3
A Look Inside Our 210TB 2012 Web Corpus
Want to know more detail about what data is in the 2012 Common Crawl corpus without running a job? Now you can thanks to Sebastian Spiegler!