721
SQL as an API language
Datasette (previously) is my open source tool for exploring and publishing structured data. There are a lot of ideas embedded in Datasette. I realized that I haven’t put many of …
Topics /
Databases & Data
The most popular blog posts about databases & data on Hacker News. 1,888 posts.
722
How I Reduced My DB Server Load by 80%
Database load can be a silent performance killer. I’ve been optimizing the query performance of a web app I run designed to get people involved in open sourc...
723
How to get transactions between almost any data stores
How to get ACID transactions between almost any data stores, with the Epoxy protocol.
724
Do you need a vector database?
Spoiler alert: the answer is maybe! Although, my inclusion of the word “actually” betrays my bias. Vector databases are having their day right now. Three different vector DB companies have raised money on valuations up to $700 million (paywall link). Surprisingly, their rise in popularity is not for their “original”…
725
Explaining the Postgres iceberg
Have you seen this legendary SQL iceberg meme? Let's talk about it while wearing our PostgreSQL hat!
726
The often-overlooked random forest kernel
Ryan Marcus, assistant professor at the University of Pennsylvania. Using machine learning to build the next generation of data systems.
727
Bayesian Machine Learning Explained
So you know the Bayes rule. How does it relate to machine learning? It can be quite difficult to grasp how the puzzle pieces fit together - we know …
728
In S3 simplicity is table stakes
From simple object storage to sophisticated table management, builders have always shaped S3's evolution. Andy Warfield discusses why making complex systems simple remains our north star at AWS.
729
The end of big data
Databricks, Snowflake, and the end of an overhyped era.
730
Please stop calling databases CP or AP
This blog post has been translated into Russian , Japanese , Chinese , and Chinese again . For more detail on problems with CAP, and a proposal for an alternative, please see my paper A Critique of the CAP Theorem .
731
Hash indexes are faster than btree indexes?
PostgreSQL have supported Hash Index for a long time, but they are not much used in production mainly because they are not durable. Now, ...
732
733
Implementing State Machines in PostgreSQL
Finite-state machine (FSM) is a wonderful model of computation that has many practical applications. One of my favorites is turning business logic into simple FSMs. For example consider the following requirements for an order management system:
734
Loading CSV File at the Speed Limit of the NVMe Storage
I plan to write a series of articles to discuss some simple but not embarrassingly parallel algorithms. These will have practical usages and would most likely be on many-core CPUs or CUDA GPUs. Today’s is the first one to discuss a parallel algorithm implementation for CSV file parser.
735
DeepSeek's smallpond: Bringing Distributed Computing to DuckDB
DeepSeek is pushing DuckDB beyond its single-node roots with smallpond, a new, simple approach to distributed compute. But does it solve the scalability challenge—or introduce new trade-offs?
736
They say that stocks go down during the day and up at night
Prompted by your blog post this morning, I attach a plot from Figure 3 of They Still Haven’t Told You showing overnight and intraday returns to AIG (with logarithmic vertical scale, updated with data through the end of October).
737
738
FoundationDB: A distributed unbundled transactional key value store
This paper from Sigmod 2021 presents FoundationDB, a transactional key-value store that supports multi-key strictly serializable transacti...
739
Cross-Database Queries in SQLite
I released Datasette 0.55 and sqlite-utils 3.6 this week with a common theme across both releases: supporting cross-database joins. Cross-database queries in Datasette SQLite databases are single files on disk. …
740
Finding Java Thread Leaks with JDK Flight Recorder and a Bit of SQL
The other day at work, we had a situation where we suspected a thread leak in one particular service, i.e. code which continuously starts new threads, without taking care of ever stopping them again. …
741
Demystifying Apache Arrow (2020)
Demystifying Apache Arrow - some observations from a data scientist. Learning more about a tool that can filter and aggregate two billion rows on a laptop in two seconds
742
Building rqlite 9.0: Cutting disk usage by half
rqlite is a lightweight, open-source, distributed relational database written in Go. It is built on the Raft consensus protocol and uses SQLite as its storage engine. Development of 9.0 has begun and aims to reduce disk usage by approximately 50%. This goal will be achieved through a high-level design overhaul…
743
744
GpuScan and SSD-To-GPU Direct DMA
An article for none-Japanese readers....What I'm recently working on is a feature to load data blocks from NVMe-SSD to GPU using peer-to-peer DMA. It allows to bypass the CPU/RAM under a series of data loading process, thus, also allows to reduce number of expensive data copy. Once data blocks are l…
745
The Internals of Postgres – Concurrency Control
Concurrency Control is a mechanism that maintains atomicity and isolation, which are two properties of the ACID, when multiple transactions run concurrently in a database.
746
Understanding Kafka with Factorio (2019)
While playing Factorio the other day, I was struck by the many similarities with Apache Kafka.
747
Analysis of Product Hunt products from 2014 to 2021
An analysis of Product Hunt, 2014 - 2021
748
Introduction to Window Functions in SQL
It is interesting that many people working with data have no clue about window functions in SQL. During a long period of time instead of using window functions, I preferred coding in Python and panda…
749
750