3931
Building an open data pipeline in 2024
Using Iceberg allows us to pick the optimal "big data" compute environment for the specific requirements we have. There's no need to limit yourself to a single solution.
Topics /
Databases & Data
The most popular blog posts about databases & data on Hacker News. 4,224 posts.
3932
Tracing Military Command Chains Through Time and Location
Case Study ⸺ Using novel technology to analyze groundbreaking chain of command research
3933
How Coderwall grew over 50% with an A/B test
Turn SEO rankings into revenue. RankScience drives growth across Google, ChatGPT, and AI platforms for startups.
3934
3935
3936
How to Get or Create in PostgreSQL
And why it is so easy to get wrong...
3937
Announcing Spark 1.3
Apache Spark 1.3 introduces the DataFrame API, enhancing data processing with new features across streaming, ML, and SQL.
3938
3939
3940
3941
How Swiss tables work in Go built-in map
Go 1.24 replaced the built-in map’s bucket-based runtime with Swiss Tables. This article explains groups, control bytes, H1 and H2, probing, table growth, directories, deletion, load factor, and the experimental split-group layout.
3942
Multidimensional analysis of plot arcs in thousands of TV and movie scripts
Note: a somewhat more complete and slightly less colloquial, but eminently more citeable, version of this work is in the Proceedings of the ...
3943
Stock Option Pricing Inference
I’m a stock market enthusiast 📈, so lately I have spent my spare time on the topic of stock option pricing, and I was curious to find out whether or not I could build a script that accurately reflected real market option prices. It turns out I came close, and learned a lot along the way, which I share in this article.
3944
Fixing Bad Data in Datomic
The free database designed for immutable values and a time-aware information model.
3945
Math versus Dirty Data
At Google, our organization designs, owns, and maintains a number of optimization models that automate the planning of Google’s datacenter growth and health. As is pretty standard in supply chain optimization and planning, these models are often integer linear programs. It’s a core competency of operations research…
3946
EnclaveDB: A secure database using SGX
EnclaveDB: A secure database using SGX Priebe et al., IEEE Security & Privacy 2018 This is a really interesting paper (if you’re into this kind of thing I guess!) bringing together the security…
3947
3948
Building agents using streaming SQL queries
Table of Contents Agents Need to Interact With LLMs Agents Should Be Event-Driven Agents Need Context Agents Require Memory When SQL Is Not Enough Parting Thoughts AI Agents have improved in leaps …
3949
3950
Pgvector 0.5.0 Feature Highlights and how tos
An overview of new features in pgvector 0.5.0, including HNSW, parallel index builds, and more!
3951
Thinking about high-quality human data
[Special thank you to Ian Kivlichan for many useful pointers (E.g. the 100+ year old Nature paper “Vox populi”) and nice feedback. 🙏 ] High-quality data is the fuel for modern data deep learning model training. Most of the task-specific labeled data comes from human annotation, such as classification task or RLHF…
3952
Understanding Raft Consensus
Recently I was digging deeper into Raft , an important algorithm in the field of distributed systems. Raft is a consensus algorithm , meaning that it is designed to facilitate a set of computers agreeing on a state of the world (more on exactly how the state of the world is represented later), even when communications…
3953
From Datalog to SVG
Exporting graph data to text file and SVG with Datalog
3954
Why Did So Many Startups Choose MongoDB?
NoSQL databases were the future. MongoDB was the database for “modern” web engineers and used by countless startups. What happened?
3955
3956
3957
Database Version Control with Liquibase
Introduction to managing DB shcema changes with Liquibase
3958
Do we need to store all that telemetry?
In my last post I talked about why modern observability has become so expensive. At the end of the post I posit a question: What if by default we never send any telemetry at all? In this post I’m goin
3959
3960