On 2026-05-30, the Mastodon server I run started getting a flood of implausible signup requests. We think these are likely bots because our server rules and signup form clearly tell users to talk about their interest in gay leather. Most people sign up with something like “42 yo levis enthusiast into piss” or “Hi! I’m…
Visit aphyr.com → RSS feedOn Hacker News May 2013 – Apr 2026
Systems & Performance 24Databases & Data 23Software Engineering 13Society, Policy & Culture 8Career & Management 6Programming Languages 5
If you want to get a job as a software witch, you’re going to have to pass a whiteboard interview. We all do them, as engineers–often as a part of our morning ritual, along with arranging a beautiful grid of xterms across the astral plane, and compulsively running ls in every nearby directory–just in case things have…
This is a long article, so I've broken it up into a series of posts, listed below. You can also read the full work as a PDF or EPUB .
Every day my phone buzzes. It is a neighborhood group: four people were kidnapped at the corner drugstore. A friend a mile away sends a Slack message: she was at the scene when masked men assaulted and abducted two people on the street. A plumber working on my pipes is distraught, and I find out that two of his…
This is a long article, so I've broken it up into a series of posts, listed below. You can also read the full work as a PDF or EPUB .
Please note: our followup analysis of 3.4.0-rc3 revealed additional faults in MongoDB’s replication algorithms which could lead to the loss of acknowledged documents–even with Majority Write Concern, journaling, and fsynced writes.
Long ago, on Svalbard, when you were a young witch of forty-three, your mother took your unscarred wrists in her hands, and spoke:
People keep asking why Jepsen is written in Clojure , so I figure it’s worth having a referencable answer. I’ve programmed in something like twenty languages. Why choose a Weird Lisp?
In the formless days, long before the rise of the Church, all spells were woven of pure causality, all actions were permitted, and death was common. Many witches were disfigured by their magicks, found crumpled at the center of a circle of twisted, glass-eaten trees, and stones which burned unceasing in the pooling…
Back in 2023 I went through an inter-state move with Flat Rate Movers, LTD. , a.k.a. Flat Rate Moving, US DOT 488466, MC 254356. They bungled the job, damaged my stuff and my house, and then refused to talk to me or their own insurance company. I placed dozens of calls, wrote letters, emails, nothing worked. I finally…
This post covers Elasticsearch 1.1.0. In the months since its publication, Elasticsearch has added a comprehensive overview of correctness issues and their progress towards fixing some of these bugs.
Previously, on Jepsen, we reviewed Elasticsearch’s progress in addressing data-loss bugs during network partitions. Today, we’ll see Aerospike 3.5.4, an “ACID database”, react violently to a basic partition.
In this Jepsen report, we’ll verify RethinkDB’s support for linearizable operations using majority reads and writes, and explore assorted read and write anomalies when consistency levels are relaxed. This work was funded by RethinkDB, and conducted in accordance with the Jepsen ethics policy .
This story is not practical advice. For me, it’s closing the book on an almost two-year saga. For you, I hope it’s an enjoyable bit of bureaucratic schadenfreude. For Anthem, I hope it’s the subject of a series of painful but transformative meetings. This is not an isolated event. I’ve had dozens of struggles with…
This is a long article, so I've broken it up into a series of posts, listed below. You can also read the full work as a PDF or EPUB .
In the last Jepsen post, we found that RethinkDB could lose data when a network partition occurred during cluster reconfiguration. In this analysis, we’ll show that although VoltDB 6.3 claims strict serializability, internal optimizations and bugs lead to stale reads, dirty reads, and even lost updates; fixes are now…
This is not at all news, but it comes up often enough that I think there should be a concise explanation of the problem. People, myself included, like to say that POSIX time, also known as Unix time, is the number of seconds since the Unix epoch , which was 1970-01-01 at 00:00:00.
Previously, on Jepsen, we saw Chronos fail to run jobs after a network partition . In this post, we’ll see MariaDB Galera Cluster allow transactions to read partially committed state.
This is a long article, so I've broken it up into a series of posts, listed below. You can also read the full work as a PDF or EPUB .
This is a long article, so I've broken it up into a series of posts, listed below. You can also read the full work as a PDF or EPUB .
Previously, on Jepsen , we demonstrated stale and dirty reads in MongoDB . In this post, we return to Elasticsearch, which loses data when the network fails, nodes pause, or processes crash.
This is a long article, so I've broken it up into a series of posts, listed below. You can also read the full work as a PDF or EPUB .
Cassandra is a Dynamo system; like Riak, it divides a hash ring into a several chunks, and keeps N replicas of each chunk on different nodes. It uses tunable quorums, hinted handoff, and active anti-entropy to keep replicas up to date. Unlike the Dynamo paper and some of its peers, Cassandra eschews vector clocks in…
This guide aims to introduce newcomers and experienced programmers alike to the beauty of functional programming, starting with the simplest building blocks of software. You’ll need a computer, basic proficiency in the command line, a text editor, and an internet connection. By the end of this series, you’ll have a…
Chronos is a distributed task scheduler (cf. cron) for the Mesos cluster management system. In this edition of Jepsen , we’ll see how simple network interruptions can permanently disrupt a Chronos+Mesos cluster
Aisha’s hands rattle you. They float gently in front of her shoulders, wrists cocked back. One sways cheerfully as she banters with the hiring manager—her lacquered nails a cyan mosaic over ochre palms. They flit, then hover momentarily as the two women arrange lunch. When the door closes, Aisha slaps her fingertips…
RabbitMQ is a distributed message queue, and is probably the most popular open-source implementation of the AMQP messaging protocol. It supports a wealth of durability, routing, and fanout strategies, and combines excellent documentation with well-designed protocol extensions. I’d like to set all these wonderful…
This was my first successful craft, which lifted off, flew a short distance from the pad, then (after several botched attempts and the deaths of several Kerbal pilots) touched down.
A few weeks ago I criticized a proposal by Antirez for a hypothetical linearizable system built on top of Redis WAIT and a strong coordinator. I showed that the coordinator he suggested was physically impossible to build, and that anybody who tried to actually implement that design would run into serious problems. I…
In the previous post, we discovered the potential for data loss in RabbitMQ clusters . In this oft-requested installation of the Jepsen series, we’ll look at etcd : a new contender in the CP coordination service arena. We’ll also discuss Consul ’s findings with Jepsen.
I’ve been discussing Jepsen and partition tolerance with Peter Bailis over the past few weeks, and I’m honored to present this post as a collaboration between the two of us. We’d also like to extend our sincere appreciation to everyone who contributed their research and experience to this piece.