The latest blog posts to take off on Hacker News. 1,888 posts.
Many groups are working on running sql queries in incremental/streaming systems . Query planning in this context is not a well understood problem. This post is a quick pointer towards the many open problems. It's not intended to be complete or authoritative, because I haven't engaged with the problem much beyond…
Benchmarks & Tips for Big Data, Hadoop, AWS, Google Cloud, PostgreSQL, Spark, Python & More...
Capture all changes to an application state as a sequence of events.
Benchmarks & Tips for Big Data, Hadoop, AWS, Google Cloud, PostgreSQL, Spark, Python & More...
Apple just announced that they will use “perceptual hashing” to detect illegal photos on iPhones. I have some experience to share on this technology.
A small, cool fact about SQLite is that its columns are flexibly typed. You can store any value in any column! Try it out: sqlite> create table t1 (v1 int, v2 varchar(10)); sqlite> insert into t1 values ("abc", "this string is longer than 10 characters!"); sqlite> select * from t1; abc|this string is longer than 10…
Some researchers in France put together a comparison showing lots of ways you could implement realtime collaborative editing (like Google Docs). They implemented lots of algorithms - CRDTs and OT algorithms and stuff. And they benchmarked them all to see how they perform. (Cool!!) Some algorithms worked reasonably…
I was writing a tiny website to display statistics of how much sponsored content a Youtube creator has over time when I noticed that I often write a small tool as a website that queries some data from a database and then displays it in a graph, a table, or similar. But if you want to use a
I have to admit, my research blog posts are getting longer and longer. From one side I find it genuinely encouraging, because if one gets so much information just by scratching the topic, imagine what’s hidden beneath the surface! One university professor once said “what could be interesting in databases?”, and it…
The other week, I stumbled on the post Inserting One Billion Rows in SQLite Under A Minute , which was a funny coincidence, as I was just in...
This is a chronicle of my experiment where I set out to insert 1B rows in SQLite
I recently had cause to take a beginners course in R - a language I'm fairly familiar with. One of the other students had never used it before, so we were buddied up in order for me to show them the ropes. The first lesson of R is always the same. Read a CSV, manipulate it a bit, draw a graph. We did it all without…
This post explains how Linux memory overcommit and OOM-killer behavior can turn a single runaway query into a full PostgreSQL crash, and why many managed cloud database services fail to disable overcommit the way self-hosted PostgreSQL recommends.
But SQL is the only widely-used implementation of the relational model, and it is:
You are brought into a startup to run their three-person data team. This is a story about teams and organization, and how you spend a year getting the team to a good place.
It covers how to make a time-series database engine from scratch
One of the easiest ways to start visualizing data is to turn a table into a heatmap: every cell gets a colour, the higher the number the brighter the colour. Unfortunately, this is often a fairly unrewarding exercise, yielding graphics that look like plaid or tartan fabric. Part of the problem is that the rows and…
On today's show, I'm talking to Richard Hipp about surviving becoming core infrastructure for the world. SQLite is everywhere. It's in your web browser, it's in your phone, it's probably in your car, and it's definitely in commercial planes. It's where your iMessages and WhatsApp messages are stored, and if you do a…
Modelling hierarchical/team/categorical/tag data with arbitrary depths.
Text classification is a very frequent use case for machine learning (ML) and natural language processing (NLP). It’s used for things like spam detection in emails, sentiment analysis for social media posts, or intent detection in chat bots. In this series I am going to compare several libraries that can be used to…
I'm super enthusiastic about Julia after running this comparison of Julia vs Numpy vs Fortran, for performance and code simplicity.
The new sqlite-utils memory command can import CSV and JSON data directly into an in-memory SQLite database, combine and query it using SQL and output the results as CSV, JSON …
Schema migrations are tricky, especially when planned downtime is not an option. Changes need to be carefully designed to stay backwards compatible so that…
In August, I set out to improve the machine learning ecosystem for Ruby and wasn’t sure where it would go. Over the next 5 months, I ended up...
