Jepsen aims to improve the safety of distributed databases, queues, consensus systems, etc. We maintain an open source library for safety testing, and publish free, in-depth analyses of specific systems. In each analysis we explore whether the system lives up to its documentation’s claims, file new bugs, and suggest…
Visit jepsen.io → On Hacker News Feb 2017 – Oct 2026
Databases & Data 31Systems & Performance 30Math & Algorithms 1
PostgreSQL is a major open-source relational database with a 23-year history and a broad range of features. While Jepsen’s work has traditionally focused on distributed systems, our tooling is readily applicable to traditional, single-node databases. In this report, we present the results of applying Jepsen’s…
AWS’s Sergey Melnik, along with HN commenters matashii and Ants Aasma, have identified the cause of Long Fork in PostgreSQL clusters. As described a 2013 mailing list post , the order in which a PostgreSQL primary makes transactions visible is determined by an in-memory lock. Secondaries, however, make transactions…
2020-05-26: MongoDB identified a bug in the transaction retry mechanism which they believe was responsible for the anomalies observed in this report; a patch is scheduled for 4.2.8.
NATS is a popular streaming system. Producers publish messages to streams , and consumers subscribe to those streams, fetching messages from them. Regular NATS streams are allowed to drop messages. However, NATS has a subsystem called JetStream , which uses the Raft consensus algorithm to replicate data among nodes…
Datomic is a general-purpose database intended for systems of record. In many ways, Datomic is unusual. At any instant in time, the state of the database is represented by a set of [entity, attribute, value] ( EAV ) triples, known as datoms . Each datom declares that some entity (like a person) has a particular…
MySQL needs little introduction. Over the last 28 years it has become one of the most widely deployed SQL databases. MySQL is primarily used for online transaction processing (OLTP) workloads, but is also deployed as a part of OLAP and queuing systems.
Modern applications often spread state across a variety of systems: databases, caches, queues, data warehouses, search indices, and so on. Coordinating these systems is challenging: caches are often stale, the analytics platform might lag days behind the primary database, the search index might lose data, and…
CockroachDB is a distributed SQL database, loosely patterned after Google Spanner and designed for semi-synchronous networks. It speaks the PostgreSQL wire protocol , supports a reasonable dialect of SQL , and combines replication for durability with transparent scale-out sharding for large tables. Its most…
In 2012, Thomson, Diamond, Weng, et al. published Calvin: Fast Distributed Transactions for Partitioned Database Systems : a transactional protocol optimized for geographic replication. Each Calvin cluster is comprised of multiple replicas , where a replica is a collection of nodes which store a complete copy of the…
2017-10-07: IMap is not the only datatype supporting merge: ICache is mergable as well.
TigerBeetle is an Online Transactional Processing (OLTP) database built for double-entry accounting with a strong emphasis on safety and speed. It builds on the Viewstamped Replication (VR) consensus protocol to offer Strong Serializable consistency. Unlike general-purpose databases, TigerBeetle stores only accounts…
MongoDB is a long-time user of Jepsen . Over the past three years, Jepsen has performed several analyses for MongoDB, and MongoDB has integrated an expansive Jepsen test suite into their CI system. In March, MongoDB requested Jepsen perform an analysis on a previously untested configuration: sharded clusters. We also…
In the past year and a half, MongoDB has put a good deal of work into improved read safety, enabled by the adoption of their new replication protocol and the WiredTiger storage engine. Dirty reads were addressed in 3.2 by introducing a majority read concern , and stale reads were addressed with the introduction of the…
2024-11-13 : An earlier version of this report asserted that consumers with auto-commit enabled could lose data by automatically committing offsets returned by the most recent poll. This claim was based in part on Confluent’s documentation , which states:
The etcd key-value store is a distributed system intended for use as a coordination primitive. Like Zookeeper and Consul , etcd stores a small volume of infrequently-updated state ( by default, up to 8 GB ) in a key-value map, and offers strict-serializable reads, writes and micro-transactions across the entire…
Dgraph is a graph database which aims to provide scalable, highly-available, and snapshot-isolated transactions over a labeled directed graph, while minimizing network communication for performance. Conceptually, Dgraph stores a set of (entity, attribute, value) triples. Entities (also known as subjects), are compact…
RavenDB is a distributed document database which repeatedly advertises its support for ACID transactions. 1 It’s intended for OLTP workloads, and offers a variety of ETL paths for exporting data to other systems. Its transactional API revolves around a session handle, which “represents a single business transaction on…
Redis is a fast single-threaded data structure server, commonly used as a cache, scratchpad, queue, or coordination mechanism between distributed applications. It offers operations over a broad array of datatypes , including binary blobs, lists, sets, sorted sets, maps, geohashes, counters, channels, streams, and…
Redpanda is a Kafka -compatible distributed streaming system based around append-only logs. Compared to Kafka, Redpanda aims to offer users lower latencies and reduced operational complexity. It uses the Raft consensus algorithm internally, rather than depending on a separate installation of Zookeeper . Redpanda…
2019-04-10: The long fork test used in this analysis contained a bug which caused it to (in many cases) fail to identify long fork anomalies—though we did find instances of long fork using other tests. We have re-checked Dgraph 1.0.13 with a corrected checker, and its long fork tests still pass.
Tendermint is a server and a protocol for building linearizable (or sequentially consistent) byzantine fault-tolerant applications. Tendermint validators accept transactions from clients over HTTP, and replicate them to the other validators in the cluster, forming a totally ordered sequence of transactions. Each…
Scylla is a distributed wide-column 1 database which originated as a C++ port of Cassandra, aiming for improved performance. It supports both Cassandra- and DynamoDB -compatible APIs, and is intended for high-throughput, low-latency workloads, including analytics, messaging, and other time-series data.
In 2010, Google engineers Daniel Peng & Frank Dabek published Large-scale Incremental Processing Using Distributed Transactions and Notifications , which described Percolator: a Google-internal database for random access transactional workloads where low latencies are not required. Percolator uses a global…
Radix DLT (Distributed Ledger Technology) is a distributed ledger: a serializable log of transactions over a state machine, along with mechanisms for querying that state. Throughout this report, “Radix” refers to the Radix DLT software .
Jepsen analyzes the safety properties of distributed systems–most notably, identifying violations of consistency models. But what are consistency models? What phenomena do they allow? What kind of consistency does a given program really need?
Aerospike , formerly Citrusleaf, is a distributed, sharded document store designed for high-throughput, low-latency applications. Each record is a map of keys to values, which may be either primitive types (integers, strings, bytes, etc.) or nested collections like lists and maps. Secondary indices are also available…
Etcd is a coordination service designed to offer Strict Serializable reads, writes, and micro-transactions over a small, in-memory key-value store. In 2022, Jepsen re-tested etcd as a part of a private engagement covering an assortment of databases. We reported several issues to the etcd team during that engagement…
MariaDB is a popular open source SQL database which began as a fork of MySQL . MariaDB Galera Cluster is an active-active replication system for MariaDB which allows reads and writes on every node. In 2015 this author analyzed MariaDB with Galera Cluster and found that although Galera claimed to offer Snapshot…
2019-04-10: The long fork test used in this analysis contained a bug which caused it to (in many cases) fail to identify long fork anomalies. We have re-checked YugaByte DB 1.1.15.0-b16 with a corrected checker, and its long fork tests still pass.
2019-09-05: YugaByte’s blog post states YugaByte DB “passes Jepsen tests”. We feel obligated to state that YugaByte DB’s Jepsen test suite does not pass, though it may in the future. Race conditions in YugaByte DB’s schema system can cause correctness errors. For example, inserting rows into a freshly-created table…