211
Using machine learning to choose compression algorithms
30% less disk required compared to using the single best strategy
Topics /
Databases & Data
The most popular blog posts about databases & data on Hacker News. 272 posts.
212
213
A Predictive Database
What is a predictive database? A database that lets you query predictions from structured tabular data with no model training step. Learn how predictive databases work, when to use them, and how they compare to traditional ML.
214
Scribe: Transporting petabytes per hour via a distributed, buffered queueing
Our hardware infrastructure comprises millions of machines, all of which generate logs that we need to process, store, and serve. The total size of these logs is several petabytes every hour. The o…
215
Sorting in Japanese – An Unsolved Problem (2011)
Sorting Japanese is not only difficult—it’s an unsolved problem. This seems hard to believe if you are not familiar with the complexities of processing Japanese digitally. But what is trivially easy in English is impossible in Japanese, even with the amount of computer power we have available today.
216
Amazon Aurora PostgreSQL Serverless
The database is usually the most critical part of a software architecture and managing databases, especially relational ones, has never been easy. For this reason, we created Amazon Aurora Serverless, an auto-scaling version of Amazon Aurora that automatically starts up, shuts down and scales up or down based on your…
217
The Holes in the Map: England’s Unregistered Land
Coding and writing by Anna Powell-Smith. Around 15% of the freehold land in England & Wales is unregistered. What this means is that if you go to the Land Registry and ask them ‘Who owns …
218
219
220
Understanding Bloom Filters with Pharo
From Moldable Development to Media for Thought
221
Zabbix, Time Series Data and TimescaleDB
Every monitoring system has to deal with three kinds of performance-related challenges. Firstly, a good monitoring system must receive, process and record incoming data very quickly. Every microsecond counts here. This may not seem obvious from the start but when your system becomes large enough all the tiny fractions…
222
Building a Relational Database Using Kafka
In a previous post, I showed how Kafka can be used as the persistent storage for an embedded key-value store, called KCache. Once you have a key-value store, it can be used as the basis for other m…
223
Challenges in Implementing a Full-Text-Search Engine
While Full-text-search is famous for search through documents at a faster rate, there are other interesting ways in which these systems handle scale, synonyms, abbreviations, misspellings etc., we will
224
Faster hash joiner with vectorized execution
How we built a 40x faster hash joiner using vectorized execution.
225
226
CockroachDB's Consistency Model
CockroachDB's consistency model fits somewhere between serializability and linerarizability. We're proposing a new marketing phrase for CRDB's guarantees: no stale reads.
227
228
Investigating the 900GB “Collection #1-5” password leaks with R
This post shows how to process gigabytes of unstructured data with the Spark framework and using the R language.
229
Applied Monotonicity: A Brief History of CRDTs in Riak
RiakAround 2013, Riak was at the forefront of Conflict-Free Replicated Data Type (CRDT) development with the engineers at Basho Technologies working hard to integrate efficient, usable CRDTs from t...
230
231
PostgreSQL DBaaS vendor comparison and calculator
This is a comparison of PostgreSQL DBaaS providers with an interactive calculator.
232
Distributed consensus revised
Distributed consensus revised Howard, PhD thesis Welcome back to a new term of The Morning Paper! To kick things off, I’m going to start by taking a look at Dr Howard’s PhD thesis, ‘Distributed con…
233
Jepsen: YugaByte DB 1.1.9
2019-04-10: The long fork test used in this analysis contained a bug which caused it to (in many cases) fail to identify long fork anomalies. We have re-checked YugaByte DB 1.1.15.0-b16 with a corrected checker, and its long fork tests still pass.
234
Noise Mapping with KSQL, a Raspberry Pi and a Software-Defined Radio
Our new cat, Snowy, is waking early. She is startled by the noise of jets flying over our house. Can I determine which plane is upsetting her by utilizing open source streaming solutions (Apache Kafka®), KSQL (the streaming SQL engine for Apache Kafka) a…
235
Solving a linear optimization problem on incentive allocation
At Lyft, scientists solve all kinds of optimization problems. While solvers can come in handy, there are times when complicated or…
236
Squashed Bugs, with Failure Rate Heatmaps
TiDB is the distributed SQL database built for AI agents. Vectors, transactions, and analytics in one engine with ACID guarantees and elastic scale.
237
Beyond L2 Loss – How We Experiment with Loss Functions
Estimating expected time of arrival (ETA) is crucial to what we do at Lyft. Estimates go directly to riders and drivers using our apps, as…
238
239
Using the linear distance operator in Postgres 12 to find the closest match
Experience zero data sprawl with EDB Postgres AI. Capable of handling 30B+ trades a day and managing 50 TB.
240