121
Inside a Ferroelectric RAM Chip
Ferroelectric memory (FRAM) is an interesting storage technique that stores bits in a special "ferroelectric" material. Ferroelectric memory...
Topics /
Databases & Data
The most popular blog posts about databases & data on Hacker News. 165 posts.
122
Lossless Log Aggregation – Reduce Log Volume by 99% Without Dropping Data
Reduce Log Volume by 99% Without Dropping Data
123
DELETEs Are Difficult
Why are database DELETEs complex and inefficient? Uncover challenges from physical data removal to autovacuum, and find strategies for managing large deletions.
124
Large language model data pipelines and Common Crawl
This article provides a short introduction to the pipeline used to create the data to train large language models (LLMs) such as LLaMA using Common Crawl (CC).
125
How to think about creating a dataset for LLM fine-tuning evaluation
I summarise the kinds of evaluations that are needed for a structured data generation task.
126
Array languages vs. the curse of the spreadsheet
It has been said that the most popular programming language in the world is the spreadsheet (and in particular these days, Excel). There...
127
128
Let's consign CAP to the cabinet of curiosities
Brewer’s CAP theorem, and Gilbert and Lynch’s formalization of it , is the first introduction to hard trade-offs for many distributed systems engineers. Going by the vast amounts of ink and bile spent on the topic, it is not unreasonable for new folks to conclude that it’s an important, foundational, idea.
129
Lisp Query Notation (LQN)
Query language and terminal utility for transforming various data formats
130
UUID Benchmark War
This month’s PGSQL Phriday #015 topic is about UUIDs, hosted by Lætitia Avrot. Lætitia has called for a debate. No, no, no. I say let’s have an all-out war. A benchmark war. I have deci…
131
Setting up PostgreSQL for running integration tests
Setting up PostgreSQL for tests is crucial for achieving performance and reliability. In this article, I'll explain how to set up PostgreSQL for tests and discuss some common pitfalls to avoid.
132
Why is Snapshot Isolation not enough?
Let's talk about why Snapshot Isolation is not enough for a general purpose concurrency control. Snapshot Isolation has been adap...
133
SQLite does not do checksums
SQLite does not do checksums by default. Disk corruptions go silently unnoticed.
134
DeWitt and Stonebraker's "MapReduce: A major step backwards" (2009)
In January 2008 David DeWitt and Michael Stonebraker published an article criticising MapReduce. As far as I can find, the article is no lon...
135
Postgres locks explorer
All of the data in this page is derived from these tests , so is true by some definition of true. Notable is this test - given an arbitrary SQL statement, it determines via runtime inspection which locks are taken.
136
Notebooks Are McDonalds of Code
You can come to McDonalds and order a salad, but you won't. Same with notebooks, you can write NASA-production-grade software in a notebook, but most likely you won't. Notebooks make you lazy, and encourage bad practices.
137
Thoughts on PostgreSQL in 2024
A look ahead at some ongoing projects in PostgreSQL in 2024 and beyond
138
Analyzing Spotify Stream History
I recently learned Spotify provides downloads for users’ streaming history. For me, this is over 10 years worth of data, so at the very least it seemed like a good nostalgia trip. This post covers some of my analysis of my personal export, and hopefully is a good starting place if you’re interested in exploring your…
139
The time keepers: pg_cron and pg_timetable
Need to schedule tasks in PostgreSQL? Compare pg_cron for simple in-database scheduling with pg_timetable for complex workflows and multi-cluster orchestration.
140
Why I Wrote Data Science for Crime Analysis with Python (2023)
To help crime analysts get started with python, I am early releasing Data Science for Crime Analysis with Python, an introductory book for analysts.
141
Breaking Bell's Inequality with Monte Carlo Simulations in Python
The article explains the Bell inequality using Monte Carlo simulations in Python, and shows how non-local action-at-a-distance can be used to break it with entangled qubits.
142
Imagining a personal data pipeline
I've been thinking a lot about personal data lately: where it's stored, how to extract it, and what to do with it. Here's where I landed.
143
Fundamentals of Object Storage in Elixir
I did a livestream where I talked about Object Storage. The how and why. The Bad Old Days. And also the neat and interesting stuff just beyond the basics. I figured I'd cover that in text as well.
144
145
Ask questions of SQLite databases and CSV/JSON files in your terminal
I built a new plugin for my sqlite-utils CLI tool that lets you ask human-language questions directly of SQLite databases and CSV/JSON files on your computer. It’s called sqlite-utils-ask. Here’s …
146
Your database skills are not 'good to have' (2023)
And a hateful ode to Object-Relational Mappers
147
Proof of the Singular Value Decomposition
The existence claim for the singular value decomposition (SVD) is quite strong: “Every matrix is diagonal, provided one uses the proper bases for the domain and range spaces” (Trefethen & Bau III, 1997) . MIT professor Gilbert Strang has a wonderful lecture on the SVD, and he includes an existence proof for the SVD…
148
The "it" in AI models is the dataset
I’ve been at OpenAI for almost a year now. In that time, I’ve trained a lot of generative models. More than anyone really has any right to train. As I’ve spent these hours observing the effects of tweaking various model configurations and h
149
Django SQLite Production Config
Optimize your Django SQLite configuration for production.
150
Versioned finite-state machines with Postgres (2019)
Inspired by Felix Geisendorfer blog post I implemented a database FSM (Finite-State Machine) with Postgresql. I brought some improvements to Felix’s implementation but before reading the following I recommend you to read carefully the original post.