331
The problem with lying is keeping track of all the lies
Or why clear consistency guarantees are how to stay sane when programming distributed systems.
Topics /
Databases & Data
The most popular blog posts about databases & data on Hacker News. 350 posts.
332
VideoPrism: A foundational visual encoder for video understanding
Posted by Long Zhao, Senior Research Scientist, and Ting Liu, Senior Staff Software Engineer, Google Research An astounding number of videos are av...
333
Finding memory leaks in Postgres C code
I spent the last week looking for a memory leak in Postgres’s WAL Sender process. I spent a few days getting more acquainted with Valgrind and gcc/clang sanitizers, but ultimately got nowhere useful with them. Finally, I stumbled on the memleak program from the bcc tools collection which led me right to the source…
334
First Contact with SQLite
I've been playing around with SQLite the last couple of days. I thought I knew a little about SQLite, but I didn't, and am getting my remedial education through an accelerated gauntlet. Some of what …
335
Historically, 4NF explanations are needlessly confusing
4NF presented in a straightforward way, with some investigation of history of teaching 4NF.
336
How much memory does a graph take?
We need a fast way to query multiple potentially large sets of data on-demand at interactive speeds. Sometimes the easiest solution to a hard problem is to build the right tool for the job.
337
A Logic Language for Distributed SQL Queries
A comprehensive look at the design of Polar, and how we enabled it to operate over distributed data.
338
Type information for faster Python C extensions
Update: The paper version of this post is accepted at PLDI SOAP 2024. Take a look at the preprint (PDF).
339
340
Building an open data pipeline in 2024
Using Iceberg allows us to pick the optimal "big data" compute environment for the specific requirements we have. There's no need to limit yourself to a single solution.
341
How to Get or Create in PostgreSQL
And why it is so easy to get wrong...
342
Thinking about high-quality human data
[Special thank you to Ian Kivlichan for many useful pointers (E.g. the 100+ year old Nature paper “Vox populi”) and nice feedback. 🙏 ] High-quality data is the fuel for modern data deep learning model training. Most of the task-specific labeled data comes from human annotation, such as classification task or RLHF…
343
From Datalog to SVG
Exporting graph data to text file and SVG with Datalog
344
Do we need to store all that telemetry?
In my last post I talked about why modern observability has become so expensive. At the end of the post I posit a question: What if by default we never send any telemetry at all? In this post I’m goin
345
Displaying Content as a Graph
Most web content is displayed as a strict hierarchy, tree-based or otherwise. What if it wasn't?An exploration into the advantages of graph-based content design, the numerous pitfalls, and a few case studies along the way.
346
347
The CAP theorem. The Bad, the Bad, & the Ugly
The CAP theorem is too simplistic and too widely misunderstood to be of much use for characterizing systems.
348
349
32k context length text embedding models
TL;DR – We are excited to announce voyage-3 and voyage-3-lite embedding models, advancing the frontier of retrieval quality, latency, and cost. voyage-3 outperforms OpenAI v3 large by 7.55% on aver…
350
Rediscovering Transaction Processing from History and First Principles
In July 2020, I stumbled into a fundamental limitation in the general-purpose database design for transaction processing.