211
Airflow 2.0
We're proud to announce that Apache Airflow 2.0.0 has been released.
Topics /
Databases & Data
The most popular blog posts about databases & data on Hacker News. 269 posts.
212
How to get stock fundamentals data with Python
Learn how to download fundamentals data with Python using the yahoo_fin package, including P/E and P/S ratios, return on equity, and more.
213
214
215
CockroachDB 20.1
We want to give you a database that allows you to quickly and simply build and manage data-intensive applications in the cloud—without worrying about the performance, scale or resilience of those apps. To support that mission, our latest release includes updates that let you build fast and build to last.
216
217
Migrating a 40TB SQL Server Database
Initially, I wasn’t sure whether to write about this migration project, but when I randomly asked if people would be interested, the response was overwhelming. This was a long, kind of boring, very repetitive, and at times incredibly frustrating project, but I learned a lot, and maybe someone else will learn from this…
218
Vector tiles in QGIS 3.14
The new QGIS 3.14 version adds support for the native loading of vector tiles. The easiest way to load them is via the recently released plugin.
219
220
NYC taxi meter and options pricing
An experiment analyzing the NYC taxi dataset through the eyes of an options trader.
221
MariaDB Temporal Data Tables
Complete system-versioned tables: WITH SYSTEM VERSIONING syntax, FOR SYSTEM_TIME AS OF/BETWEEN/ALL queries, and ROW_START/ROW_END columns.
222
Migrating Large Heroku Postgres Instances to AWS Aurora Without Downtime
In this article I discuss a general process I used recently to migrate a large multi-terabyte Heroku Postgres Database from the Heroku Platform to Amazon Aurora Postgres on a live Heroku based application architecture with near zero downtime and builtin failover during the process.
223
Domain Logic and SQL (2003)
A long-form article entitled: "Domain Logic and SQL"
224
CQRS: Command Query Responsibility Segregation (2011)
CQRS (Command Query Responsibility Segregation) is the notion that you can use a different model to update information than the model you use to read information
225
Feature Casualties of Large Databases
Databases shed important RDMS features as they get big. Examining why this tends to be the case, and some ideas for preventing it.
226
Loading NumPy arrays from disk: mmap() vs. Zarr/HDF5
Learn how to load larger-than-memory NumPy arrays from disk using either mmap() (using numpy.memmap), or the very similar Zarr and HDF5 file formats.
227
Diffing Coronaviruses
Comparing the genome sequences of the Wuhan coronavirus and a bat coronavirus using `diff` and NCBI BLAST reveals an 89% similarity.
228
AWS Managed Workflows for Apache Airflow
As the volume and complexity of your data processing pipelines increase, you can simplify the overall process by decomposing it into a series of smaller tasks and coordinate the execution of these tasks as part of a workflow. To do so, many developers and data engineers use Apache Airflow, a platform created by the…
229
Thread-Per-Core Buffer Management for a modern storage system
Redpanda powers an Agentic Data Plane and Data Streaming platform for real-time performance, AI innovation, and simplified operations.
230
231
232
ZFS: Use mirror vdevs, not RAIDZ
Continuing this week’s “making an article so I don’t have to keep typing it” ZFS series… here’s why you should stop using RAIDZ, and start using mirror vdevs instead.
233
234
Mendoza: Use stack machines to compute efficient JSON diffs
Mendoza is a new, super efficient format for expressing differences between JSON documents.
235
Immutable Databases
Like blockchain but without blockchain.
236
Writing a Postgres Foreign Data Wrapper for Clickhouse in Go
Postgres(hereinafter mentioned as PG) is a pretty cool database with lots of nice features, one of them little known ones is the ability of having Foreign data wrappers (hereinafter mentioned as FDWs). Clickhouse(hereinafter mentioned as CH) is another amazing database with an altogether different set of features…
237
How to build your own feature store for ML
We have many conversations with companies and organizations who are deciding between building their own feature store and buying one....
238
239
Bulk loading into PostgreSQL: Options and comparison
You have a file, possibly a huge CSV, and you want to import its content into your database. There are lots of options to do this but how would you decide which one to use. More often than not the question is how much time would the bulk load would take. I found my self doing the same
240
Analyzing the Limits of Connection Scalability in Postgres
Why it’s important to improve Postgres connection scalability, plus an analysis on the limits of connection scalability today.