"So, how much experience do you have with Big Data and Hadoop?" they asked me. I told them that I use Hadoop all the time, but rarely for jobs larger than a few TB. I'm basically a big data neophite - I know the concepts, I've written code, but never at scale.
London, 1772. I have been asked by my superiors to give a brief demonstration of the surprising effectiveness of even the simplest techniques of the new-fangled Social Networke Analysis in the pursuit of those who would seek to undermine the liberty enjoyed by His Majesty’s subjects. This is in connection with the…
I've recently completed a project to compile publicly-available NFL play-by-play data. It took a while, but now it's ready. The resulting ...
Blog article by Jeroen Janssens. Sep 19, 2013. 21 min read. Data science is OSEMN (pronounced as awesome). That is, it involves Obtaining, Scrubbing, Exploring, Modelling, and iNterpreting data. As a data scientist, I spend quite a bit of time on the command-line, especially when there’s data to be obtained, scrubbed…
I returned back home from PGCon.EU-2013 conference, Dublin, Ireland, where we gave two talks - Next Generation of GIN and Binary storage for nested data structures and application to hstore data type . These projects as well as visiting the conference (I, Teodor Sigaev and Alexander Korotkov) were…
An analysis and infographic of 10,000 porn stars and adult films, revealing what the average adult performer looks like and does on film.
Cassandra is a Dynamo system; like Riak, it divides a hash ring into a several chunks, and keeps N replicas of each chunk on different nodes. It uses tunable quorums, hinted handoff, and active anti-entropy to keep replicas up to date. Unlike the Dynamo paper and some of its peers, Cassandra eschews vector clocks in…
A beginner's guide to getting started with web scraping using Python and BeautifulSoup.
While at Waza this year, I had a chance to talk to my friend Guillaume Roques. In addition to talking about SalesForce, we took advantage of our mutual .NET backgrounds to discuss Microsoft. We did the typically uncompromising praise of The Gu and how far Azure has come along in the last 18 months… and of course we…
The command line tools that ship with any Unix-like system are extremely handy for many data tasks. Here are some of the ones that I've found most useful.
Connecting to a database 1 2 $ psql postgres # the default database $ psql database_name Connecting as a specific user 1 2 $ psql postgres john $ …
Update, August 2014: see Hacking Academia , my most recent reflection on this subject.
Stephen Wolfram shares interesting Facebook data analysis finds from the Data Donor program of Wolfram|Alpha Personal Analytics for Facebook.
A few weeks ago I criticized a proposal by Antirez for a hypothetical linearizable system built on top of Redis WAIT and a strong coordinator. I showed that the coordinator he suggested was physically impossible to build, and that anybody who tried to actually implement that design would run into serious problems. I…
Nowadays Python is probably the programming language of choice (besides R) for data scientists for prototyping, visualization, and running data analyses on small and medium sized data sets. And...
A Bloom filter is a data structure designed to tell you, rapidly and memory-efficiently, whether an element is present in a set.
