On December 21, 2018, the United States House of Representatives voted to enact H.R. 4174, the Foundations for Evidence-Based Policymaking Act of 2017, in a historic win for open government in the …
This'll be my first blog post on the internet, ever. Hopefully it's interesting and accurate. Please point out any mistakes if you see any!
A handy rule of thumb for creating a confidence interval for the probability of an event you haven't seen happen yet.
In my last post, I wrote about my adventure of requesting metadata for both phone calls and emails from the City of Chicago Office of the Mayor. The work there - and its associated frustration - sent me down a path of sending requests throughout the US to both learn whether these sorts of problems are systemic…
(Spanner vs. Calvin, Part 2) [TL;DR I wrote a post in 2017 that discussed Spanner vs. Calvin that focused on performance differences. Th...
How not to structure your database-backed web applications: a study of performance bugs in the wild Yang et al., ICSE’18 This is a fascinating study of the problems people get into when using…
The goal of the numpy exercises is to serve as a reference as well as to get you to apply numpy beyond the basics. The questions are of 4 levels of difficulties with L1 being the easiest to L4 being the hardest.
The internet is full of dogmatic commandments for choosing and using keys in relational databases. At times it verges on a holy war: should you use natural or artificial keys? Auto-incrementing integers, UUIDs?
In the summer of 2006, I was lucky enough to be an intern at Google. At the time, Google had an internal tool called gsearch that acted as if it ran grep over all the files in the Google source tree and printed the results. Of course, that implementation would be fairly slow, so what gsearch actually did was talk to a…
Postgres 11 is almost here, in fact the latest beta shipped today, and it features a lot of exciting improvements. If you want to get the full list of features it is definitely worth checking out the release notes, but for those who don’t read the release notes I put together a run down of some what I consider the…
At JupyterCon, I learned three things: reactive notebooks are the future, Jupyter is the new Bash, and data science is a gateway drug.
Decision trees are the fundamental building block of gradient boosting machines and Random Forests(tm), probably the two most popular machine learning models for structured data. Visualizing decision trees is a tremendous aid when learning how these models work and when interpreting models. Unfortunately, current…
Introduction This article gives an overview of LocustDB [1], a new and extremely fast open-source analytics database built in Rust. Part 1 gives some background on analytical query systems and my g…
Some applications, such as todo lists, need to maintain a user-defined order of items. The challenge is that the order is arbitrary and can change when the user rearranges items:
Thoughts on FOSS and fintech, layered by Mahmoud Hashemi
On Software, life and everything in-between
How we manage schema changes at Discourse minimizing downtime
Hitting the limit for maximum allowed connections is a common operational problem in Postgres. Here we look at a few techniques for managing connections and making efficient use of those that are available.
I was trying to run a simple join query on MariaDB (MySQL) and its performance was horrendous. Here’s how I cut down the query’s run time from over 380 hours to under 12 hours by executing part of it with two simple Unix commands.
I made a New Year's resolution: every plot I make during 2018 will contain uncertainty estimates. Nine months in and I have learned a lot, so I put together a summary of some of the most useful methods.
Event sourcing and CQRS gained a lot of popularity recently. The advantages are obvious and they share a very peculiar symbiosis with the…
There’s fast and there’s fast. This post is about Anna*, a key/value database design from our team at Berkeley that’s got phenomenal speed and buttery smooth scaling, with an unpr…
How a seemingly minor enhancement in Postgres 11 fills one of the system’s biggest operational holes.
In the past few years, PostgreSQL has advanced a lot in terms of features, performance, and scalability for many-core systems. However, o...
Flattening a single very hot vertical Redis node into a horizontal Redis Cluster at Stripe.
