Posts that commenters keep linking to, ranked by how many different people have cited them in Hacker News comments. 253 posts.
We use MySQL for storing all of the data in FriendFeed . Our database has grown a lot as our user base has grown. We now store over 250 million entries and a bunch of other data, from comments and "likes" to friend lists.
Randy Olson charts out the percentage of Bachelor's degrees conferred to women (by major) and makes some startling findings.
This is the introduction to a four-part series of posts detailing evidence of fraud in four academic papers co-authored by Harvard Business School Professor Francesca Gino. In 2021, we and a team o…
I had an opportunity to meet Ted Neward at TechEd this year. Ted, among other things, famously coined the phrase “Object-Relational mapping is the Vietnam of our industy” in late 2004. It’s a scary analogy, but an apt one. I’ve seen developers struggle for years with the
An in-depth exploration of a new class of expressive, performant, and intuitive CRDTs, together with macOS and iOS sample code.
This is the text version of a talk I gave on October 1, 2015, at the Strata+Hadoop World conference in New York City. The video version is here (20 mins).
Continuing this week’s “making an article so I don’t have to keep typing it” ZFS series… here’s why you should stop using RAIDZ, and start using mirror vdevs instead.
Previously on Jepsen , we introduced the problem of network partitions . Here, we demonstrate that a few transactions which “fail” during the start of a partition may have actually succeeded.
"So, how much experience do you have with Big Data and Hadoop?" they asked me. I told them that I use Hadoop all the time, but rarely for jobs larger than a few TB. I'm basically a big data neophite - I know the concepts, I've written code, but never at scale.
This post is the first of many to come on Apache Arrow, pandas, pandas2, and the general trajectory of my work in recent times and into the foreseeable future. This is a bit of a read and overall fairly technical, but if interested I encourage you to take the time to work through it.
This post is co-authored with a team of researchers who have chosen to remain anonymous. They uncovered most of the evidence reported in this post. These researchers are not connected in any way to…
Packages that need to read cdb files should incorporate the necessary portions of the cdb library rather than relying on an external cdb library.
After a temporary stay in Japan to focus on my health and personal responsibilities, I am fully recovered, ready, and excited to work abroad again.
The big Cassandra vs Mongodb vs CouchDB vs Redis, vs Riak vs HBase vs Couchbase (ex-Membase) vs OrientDB vs Aerospike vs Neo4j vs Hypertable vs ElasticSearch vs Accumulo vs VoltDB vs Scalaris vs RethinkDB vs Kyoto Tycoon comparison. While SQL databases are insanely useful tools, their monopoly in the last decades is…
This blog post has been translated into Russian , Japanese , Chinese , and Chinese again . For more detail on problems with CAP, and a proposal for an alternative, please see my paper A Critique of the CAP Theorem .
I’m often surprised that people continually have fundamental misconceptions about how distributed systems behave. I myself shared many of these misconceptions, so I try not to demean or dismiss but rather educate and enlighten, hopefully while sounding less preachy than that just did. I continue to learn only by…
A forum moderated by author and artist Edward Tufte, including discussion and questions dealing with information design.
In two weeks we’ll present a paper on the Dynamo technology at SOSP, the prestigious biannual Operating Systems conference. Dynamo is internal technology developed at Amazon to address the need for an incrementally scalable, highly-available key-value storage system. The technology is designed to give its users the…
This is pretty important change, as it has the chance to cause performance issues in some cases.
Previously on Jepsen , we explored two-phase commit in Postgres . In this post, we demonstrate Redis losing 56% of writes during a partition.
I saw Martin Kleppmann’s talk a few weeks ago about his approach to realtime editing with CRDTs, and I felt a deep sense of despair. Maybe all the work I’ve been doing for the past decade won’t be part of the future after all, because Martin’s
Writing a sqlite clone from scratch in C
In the previous post, we discovered the potential for data loss in RabbitMQ clusters . In this oft-requested installation of the Jepsen series, we’ll look at etcd : a new contender in the CP coordination service arena. We’ll also discuss Consul ’s findings with Jepsen.
Every Bitcoin transaction is stored in the distributed database known as the Bitcoin blockchain. However, people have found ways to hack ...
Previously in Jepsen , we discussed Redis . In this post, we’ll see MongoDB drop a phenomenal amount of data. See also: followup analyses of 2.6.7 and 3.4.0-rc3 .
Please note: our followup analysis of 3.4.0-rc3 revealed additional faults in MongoDB’s replication algorithms which could lead to the loss of acknowledged documents–even with Majority Write Concern, journaling, and fsynced writes.
In this Jepsen report, we’ll verify RethinkDB’s support for linearizable operations using majority reads and writes, and explore assorted read and write anomalies when consistency levels are relaxed. This work was funded by RethinkDB, and conducted in accordance with the Jepsen ethics policy .
