151
Subtleties of SQLite Indexes
Understanding query planner quirks yielded a ~35% speedup.
New
The latest blog posts to take off on Hacker News. 1,888 posts.
152
A SQL Heuristic: ORs Are Expensive
But this is slow! With 1,000,000 applications and 1000 users uniformly distributed on both columns, it takes over 100ms. 1
153
Redis is fast – I'll cache in Postgres
There are books & many articles online, like this one arguing for using Postgres for everything. I thought I’d take a look at one use case - using Postgres instead of Redis for caching. I work with APIs quite a bit, so I’d build a super simple HTTP server that responds with data from that cache. I’d start from Redis…
154
How AWS S3 serves 1 petabyte per second on top of slow HDDs
Learn how Amazon built the backbone of the modern web that scales to 1 PB/s and 150M QPS on commodity hard drives
155
156
Ultra efficient vector extension for SQLite
Making vector search fast, memory-efficient, and natural in SQLite.
157
158
DSM Disorders Disappear in Statistical Clustering of Psychiatric Symptoms (2024)
Whither Major Depression?
159
Disk Utility still can't check and repair APFS volumes and containers (2021)
Don’t waste time trying to discover which files are still open when you want to check or repair an APFS disk and Disk Utility throws an error. Here’s the solution.
160
161
Which colours dominate movie posters and why?
I analysed 58,687 movie posters from the past century to see which colours dominate, how their use has shifted over time and what each hue signals to audiences.
162
Removing newlines in FASTA file increases ZSTD compression ratio by 10x
First released with Zstandard 1.3.2 in 2017, the --long range match finder increases the compressor’s search window to at least 128MiB, improving deduplication inside large files. This optional feature had substantial performance overheads at launch, but various optimisations have since brought its performance within…
163
164
Rails on SQLite: new ways to cause outages
This post was originally given as a talk for Friendly.rb and Kaigi on Rails. The slides are also available. Between Litestack and the Rails 8 trifecta of Solid Cable, Solid Cache, and Solid Queue, it’s easier than ever to spin up a Rails app that doesn’t need a database service, or a redis service, or a file storage…
165
Strong Eventual Consistency – The Big Idea Behind CRDTs
Missing the Forest for the Sequence Trees.
166
Oldest recorded transaction
The oldest recorded transaction was in 3100 BC
167
SQL needed structure
The data on the page is presented in a hierarchichal structure. The movie page contains a director, a list of genres, a list of actors, and each actor in the list contains a list of characters they played in the movie. You can't sensibly fit all of this into a single flat structure like a relation.
168
How to build vector tiles from scratch
Building MVT tiles from scratch
169
Poor man's bitemporal data system in SQLite and Clojure
On trying to mash up SQLite with ideas stolen from Accountants, Clojure, Datomic, XTDB, Rama, and Local-first-ers, to satisfy Henderson's Tenth Law. Viz., to make a sufficiently complicated data system containing an ad-hoc, informally-specified, bug-ridden, slow implementation of half of a bitemporal database…
170
How big are our embeddings now and why?
Embedding sizes and architectures have changed remarkably over the past 5 years
171
SQLite's documentation about its durability properties is unclear
Is SQLite durable by default? What settings guarantee durability? The documentation and even comments from its creator give conflicting answers.
172
How to install TrueNAS on a Raspberry Pi
Now that Joel0 in the TrueNAS community has created a fork of TrueNAS that runs on Arm, I thought I'd give it a spin—on a Raspberry Pi. I currently run an Ampere Arm server in my rack with Linux and ZFS as my primary storage server, and a Raspberry Pi with four SATA SSDs and ZFS as backup replica in my studio. My…
173
Charting Form Ds to roughly see the state of venture capital “fund” raising
Above is a graph that displays the amount of Form Ds filed, where the entity (read: company/firm) name contains the phrases "fund I", "fund II", "fund III", and "fund IV". The x-axis is not the prettiest, but it is broken down by quarter. You can see that the line for "fund I" sees by far the greatest peak around…
174
SQLite (with WAL) doesn't do `fsync` on each commit under default settings
SQLite doesn’t do fsync unless specified.
175
Materialized views are obviously useful
As programmers we spend a lot of time shuttling data back and forth between different systems and transforming it from one format to another. Sometimes it gets pretty miserable!
176
Why was Apache Kafka created?
The story behind how LinkedIn created Apache Kafka
177
The two versions of Parquet
The adoption of Parquet version 2 is limited due to the lack of support in the ecosystem, which affects its evolution despite its improvements in compression and performance.
178
Faster Index I/O with NVMe SSDs
The Marginalia Search index has been partially rewritten to perform much better, using new data structures designed to make better use of modern hardware. This post will cover the new design, and will also touch upon some of the unexpected and unintuitive performance characteristics of NVMe SSDs when it comes to read…
179
Lessons learned from building a sync-engine and reactivity system with SQLite
Welcome to my personal blog! I use it to share what I'm currently learning or thinking about, usually on topics related to technology, business, and health.
180
Writing a storage engine for Postgres: An in-memory table access method (2023)
Writing a storage engine for Postgres: an in-memory Table Access Method