1081
Transforming Wikipedia into a cultural knowledge quiz
Building “Test Your Culture” by taking the world’s largest encyclopedia… and combining it with data science
New
The latest blog posts to take off on Hacker News. 1,888 posts.
1082
Using parallel sequential scan in PostgreSQL
Parallel sequential scan is the first parallel access method in PostgreSQL and is introduced in version 9.6. The committer of this featu...
1083
The City of Seattle Accidentally Gave Me 32M Emails for $40
In my last post, I wrote about my adventure of requesting metadata for both phone calls and emails from the City of Chicago Office of the Mayor. The work there - and its associated frustration - sent me down a path of sending requests throughout the US to both learn whether these sorts of problems are systemic…
1084
How to Manage Connections Efficiently in Postgres, or Any Database
Hitting the limit for maximum allowed connections is a common operational problem in Postgres. Here we look at a few techniques for managing connections and making efficient use of those that are available.
1085
1086
1087
The hacker's guide to uncertainty estimates
I made a New Year's resolution: every plot I make during 2018 will contain uncertainty estimates. Nine months in and I have learned a lot, so I put together a summary of some of the most useful methods.
1088
SQL as an API language
Datasette (previously) is my open source tool for exploring and publishing structured data. There are a lot of ideas embedded in Datasette. I realized that I haven’t put many of …
1089
Data Factories
Facebook and Google and other advertising businesses are data factories, and regulation will be most effective if it lets users look inside
1090
How to visualize decision trees
Decision trees are the fundamental building block of gradient boosting machines and Random Forests(tm), probably the two most popular machine learning models for structured data. Visualizing decision trees is a tremendous aid when learning how these models work and when interpreting models. Unfortunately, current…
1091
The design and implementation of modern column-oriented database systems
The design and implementation of modern column-oriented database systems Abadi et al., Foundations and trends in databases, 2012 I came here by following the references in the Smoke paper we looked…
1092
Postgres 11 – A First Look
Postgres 11 is almost here, in fact the latest beta shipped today, and it features a lot of exciting improvements. If you want to get the full list of features it is definitely worth checking out the release notes, but for those who don’t read the release notes I put together a run down of some what I consider the…
1093
NewSQL databases fail to guarantee consistency and I blame Spanner
(Spanner vs. Calvin, Part 2) [TL;DR I wrote a post in 2017 that discussed Spanner vs. Calvin that focused on performance differences. Th...
1094
A Day in the Life of Americans
I wanted to see how daily patterns emerge at the individual level and how a person’s entire day plays out. So I simulated 1,000 of them.
1095
1096
1097
Who Wrote the Anti-Trump New York Times Op-Ed? Using Tidytext
Like a lot of people, I was intrigued by “I Am Part of the Resistance Inside the Trump Administration”, an anonymous New York Times op-ed written by a “senior official in the Trump administration”. And like many data scientists, I was curious about what role text mining could play.
1098
“Stateless” sync using version vectors
Isolating cloud sync and persistent storage in an app architecture.
1099
Learning Python for Social Scientists (2015)
I’ve compiled a list of Python tutorials and annotated analyses. I've tried to list pages that are accessible to social scientists with little background in Python and/or machine learning.
1100
1101
A Missing Link in Postgres 11: Fast Column Creation with Defaults
How a seemingly minor enhancement in Postgres 11 fills one of the system’s biggest operational holes.
1102
A little bit of the one-time MacOS version still lingers in ZFS
You're probably reading this page because you've attempted to access some part of my blog (Wandering Thoughts) or CSpace , the wiki thing it's part of. Unfortunately you're using a browser version that my anti-crawler precautions consider suspicious, most often because it's too old (most often this applies to versions…
1103
The Future of Notebooks: Lessons from JupyterCon
At JupyterCon, I learned three things: reactive notebooks are the future, Jupyter is the new Bash, and data science is a gateway drug.
1104
What they don’t tell you about event sourcing
Event sourcing and CQRS gained a lot of popularity recently. The advantages are obvious and they share a very peculiar symbiosis with the…
1105
How to Read 100s of Millions of Records per Second from a Single Disk
This article gives an overview of the implementation and performance of two recent additions to LocustDB, an extremely fast open-source analytics database built in Rust. The first addition is suppo…
1106
Using FOIA Data and Unix to halve major source of parking tickets
This'll be my first blog post on the internet, ever. Hopefully it's interesting and accurate. Please point out any mistakes if you see any!
1107
1108
How I slashed a SQL query's runtime with two Unix commands
I was trying to run a simple join query on MariaDB (MySQL) and its performance was horrendous. Here’s how I cut down the query’s run time from over 380 hours to under 12 hours by executing part of it with two simple Unix commands.
1109
How Numba and Cython speed up Python code
A high-level overview of Numba and Cython architectures
1110
How to Analyze Billions of Records per Second on a Single Desktop PC
Introduction This article gives an overview of LocustDB [1], a new and extremely fast open-source analytics database built in Rust. Part 1 gives some background on analytical query systems and my g…