121
Kafka at the low end: how bad can it get?
There is oft-quoted advice that Kafka does poorly as a job queue. I’ve experienced this myself, and I wanted to formalize it a bit.
Topics /
Databases & Data
The most popular blog posts about databases & data on Hacker News. 148 posts.
122
Decomposing Transactional Systems
Every transactional system must execute, order, validate, and persist transactions.
123
124
Optimizing Datalog for the GPU
Datalog primer and a neat data structure
125
The principles of database design, or, the Truth is out there
Every software project needs to represent the reality of the business he is embedded in. The way we can represent reality as limited rational beings is through propositions, i.e, declarative statements that affirm or deny something about reality. When a collection of such propositions is stored in a computer system…
126
MySQL transactions per second vs. fsyncs per second (2020)
Just wondering how many transactions or writes per second MySQL can handle? While it depends on many factors, fundamentally, about as many transactions as MySQL can commit to disk per second. A modern disk can do ~1000 fsyncs per second , but MySQL will group multiple writes with each fsync. An okay rule-of-thumb…
127
128
Dumb statistical models, always making people look bad
This is Jessica. In my Prediction for Decision-making class last quarter we talked about human predictions and decisions relative to algorithmic ones. This led to a discussion about why it’s often hard to demonstrate the value of human knowledge once you have a decent statistical model.
129
Sparrow, a modern C++ implementation of the Apache Arrow columnar format
A Modern C++ Implementation of the Apache Arrow Columnar Format
130
Siren Call of SQLite on the Server
At Terrateam, we are big fans of Fly.io. The service is hosted there and it’s served us well. Just deploy your TOML file, get your infrastructure, do something else with the rest of your day. One of the interesting sides of Fly is that they invest heavily in server-side SQLite. They’ve written a number of blog posts…
131
Database Protocols Are Underwhelming
If you’ve been in this trade for a while, you have probably seen dozens of debates on the merits and problems of SQL as a relational database query language. As an ORM maintainer, I have a few gripes with SQL, but overall it is workable, and anyway, it has so much inertia that there’s no point fantasizing about a…
132
How big are our embeddings now and why?
Embedding sizes and architectures have changed remarkably over the past 5 years
133
Your Supabase is public if you turn off RLS
I was chatting with a close friend of mine and he sent me a link to his new SaaS that he's developing.
134
Harder, Better, Faster, Stronger Version of Uber H3 in Rust
Since this article will probably have more reach than my usual rambling, It'll be written in English (but let's not make a habit of it).
135
How to build vector tiles from scratch
Building MVT tiles from scratch
136
Disk Utility still can't check and repair APFS volumes and containers (2021)
Don’t waste time trying to discover which files are still open when you want to check or repair an APFS disk and Disk Utility throws an error. Here’s the solution.
137
Selective async commits in PostgreSQL – balancing durability and performance
Safely leverage PostgreSQL's asynchronous commit for significant performance gains
138
139
SQLite (with WAL) doesn't do `fsync` on each commit under default settings
SQLite doesn’t do fsync unless specified.
140
Derivation and Intuition behind Poisson distribution
Made in Notion, the connected workspace where AI agents and teams organize docs, projects and knowledge together.
141
Weighting an average to minimize variance
How would you allocate money to a set of independent assets with to minimize volatility if some assets are more volatile than others?
142
Extending That XOR Trick to Billions of Rows
Learn how to extend the classic XOR trick to find thousands of missing values using Invertible Bloom Filters
143
SQLite transactions and virtual tables
In my previous post I introduced SQL virtual tables and how to use them in Go—registering modules per connection, defining schemas, and querying external sources as regular tables.
144
Wrong ways to use the databases, when the pendulum swung too far
Disclaimer: This story took place when I was a junior developer for a previous employer ages ago. Some details might be fuzzy, and goes without saying this is entirely from my own POV so take everything here with humor and a grain of salt.
145
Building agents using streaming SQL queries
Table of Contents Agents Need to Interact With LLMs Agents Should Be Event-Driven Agents Need Context Agents Require Memory When SQL Is Not Enough Parting Thoughts AI Agents have improved in leaps …
146
How/why to sweep async tasks under a Postgres table
I like slim and stupid servers, where each endpoint wraps a very dumb DB query. Dumb queries are fast. Fast queries make websites smooth and snappy. Keep those click/render loops sacred. Sweep complexity under a task table.
147
148
Writing a storage engine for Postgres: An in-memory table access method (2023)
Writing a storage engine for Postgres: an in-memory Table Access Method