SQLite is an embedded SQL database. It’s extremely easy to setup, buildable as a single C file with libraries existing for basically all common programming languages. It doesn’t need any server setup or configuration since the SQL logic is run in the host process, and the database consists of only
This is a very brief post announcing a fascinating discovery. It appears to be possible to use the cosine similarity approach powering explore2.marginalia.nu as a substitute for the link graph in an eigenvector-based ranking algorithm (i.e. PageRank). The original PageRank algorithm can be conceptualized as a…
Undo and redo have been a staple operation of text editors probably since the first typo was ever made, yet there has not been a lot of innovation around refining the idea of what undo and redo could be. Let’s explore what I mean…
Apache Kafka has been the most popular open source event streaming system for many years and it continues to grow in popularity. Within the wider ecosystem there are other open source and source available competitors to Kafka such as Apache Pulsar, NATS Streaming, Redis Streams, RabbitMQ and more re
Erik van Zwet, Sander Greenland, Guido Imbens, Simon Schwab, Steve Goodman, and I write :
Data Engineering is booming and many are looking for a cheat sheet to get started. Do you really need to learn 200 tools before you get your first job?
tl;dr: part 1 explains what an explanation algorithm is, and part 2 describes an open source SQL data differ.
I think simple ultra-high speed lossy (or near-lossless) image codecs, built from the new generation of fast LZ codecs, are going to become ...
How to get ACID transactions between almost any data stores, with the Epoxy protocol.
Spoiler alert: the answer is maybe! Although, my inclusion of the word “actually” betrays my bias. Vector databases are having their day right now. Three different vector DB companies have raised money on valuations up to $700 million (paywall link). Surprisingly, their rise in popularity is not for their “original”…
Have you seen this legendary SQL iceberg meme? Let's talk about it while wearing our PostgreSQL hat!
Prompted by your blog post this morning, I attach a plot from Figure 3 of They Still Haven’t Told You showing overnight and intraday returns to AIG (with logarithmic vertical scale, updated with data through the end of October).
The other day at work, we had a situation where we suspected a thread leak in one particular service, i.e. code which continuously starts new threads, without taking care of ever stopping them again. …
Demystifying Apache Arrow - some observations from a data scientist. Learning more about a tool that can filter and aggregate two billion rows on a laptop in two seconds
While playing Factorio the other day, I was struck by the many similarities with Apache Kafka.
Notes on personal libraries, collections and small indexes on the web
When the JPEG folks were working on image compression, they didn't create a better or faster LZ. Instead they developed new approaches. I se...
Benchmarks & Tips for Big Data, Hadoop, AWS, Google Cloud, PostgreSQL, Spark, Python & More...
I demonstrate a widely applicable pattern which integrates Prolog as a critical component in a data science analysis. Analytic methods are used to generate properties about the data under study and Prolog is used to reason about the data via the generated properties. The post includes some examples of piece-wise…
I’m often surprised that people continually have fundamental misconceptions about how distributed systems behave. I myself shared many of these misconceptions, so I try not to demean or dismiss but rather educate and enlighten, hopefully while sounding less preachy than that just did. I continue to learn only by…
3 interesting uses of DISTINCT in PostgreSQL
On an episode of Postgres FM , the hosts had a (very brief) discussion of data encryption in PostgreSQL. While Postgres FM is a podcast well worth a subscribe, the hosts aren’t data security experts, and so as someone who builds a queryable database encryption system , I found the coverage to be somewhat… lacking. I…
Important: This page summarizes data scaling only, using tokens to parameters as a ratio, and as derived from large language models like GPT-3, Chinchilla, and beyond, linked to the Compute-Optimal scaling laws like Kaplan and Hoffmann/Chinchilla. Please note that compute scaling laws are outside the scope of my…
Languages that suffer success often have to do so by selling out and adding features that go against some of the original purposes of their design. SQL is a...
Interactive post about the frequency of bank failures over time
In an era where the costs for cloud services continue to soar, I want to outline how our data team has successfully managed to run the infrastructure of our energy trading company within the first year using only two PostgreSQL database instances - each equipped with just two CPU cores and four GB of RAM. Despite this…
The cost for something that can be replicated free and open source is absurd. The Fortune 100 have a use case for these companies, the rest…
Personal website of Phil Booth
