Many groups are working on running sql queries in incremental/streaming systems . Query planning in this context is not a well understood problem. This post is a quick pointer towards the many open problems. It's not intended to be complete or authoritative, because I haven't engaged with the problem much beyond…
Writing an ETL script is easy; making it production-ready and maintainable is much harder. Why to use an open-source EL(T) framework instead.
A visual explanation of multi-level modeling
Memory allocator is an important part of the system, so choosing the right allocator for a workload can give huge benefits. Here is a story of how we decreased service memory usage by almost three times.
PostgreSQL subtransactions (nested transactions) may cause multiple performance and scalability issues: higher rates of XID growth and higher risks of transaction ID wraparound, performance degradation when more than PGPROC_MAX_CACHED_SUBXIDS (64) are used in a session, drastic performance slowdowns when…
Jialin Li, Ellis Michael, Naveen Kr. Sharma, Adriana Szekeres, and Dan R. K. Ports, University of Washington
Efficient SQL data transformations & real-time analytics with dbt + Materialize: a powerful operational data warehouse combo.
A misconception that I held for a really long time was that systems that use SQL (Structured Query Language) are by definition relational databases, and systems that don’t (e.g., MongoDB, Cassandra) are non-relational. It’s easy to understand whether a database uses SQL or not, but what does it mean when we say a…
How we built an authorization system based on Google Zanzibar
An example of how you can encrypt Postgres data at rest using PGO, the open source Postgres Operator from Crunchy Data using AWS EBS gp2 volumes.
Huge swathes of people, in Europe and North America in particular, spend their entire working lives performing tasks they secretly believe…
Skylar Advisor combines real-time observability data, rich operational context, and customer-owned knowledge to deliver explainable AI guidance for IT operations.
Capture all changes to an application state as a sequence of events.
Struggling with IndexedDB performance? Discover hidden bottlenecks, advanced tuning techniques, and next-gen storage like the File System Access API.
Edit – I posted this on Hackernews and got some valuable feedback. Many brought up the fact that you should be able to reuse the internal state of the compressor instead of recompressing the training …
Lately I’ve been trying to learn more about Amazon Web Services (AWS) and some other popular technologies. Since I work at a large tech company, I spend a lot of my professional time using pr…
Results dashboard with the yearly (unofficial) Advent of Code participant survey
A look at the quarterly and lifetime failure rates of 175,443 drives, including a comparison of failure rates of HDD and SSD boot drives.
Product quantization (PQ) is a popular method for dramatically compressing high-dimensional vectors to use 97% less memory, and for making nearest-neighbor search speeds 5.5x faster in our tests.
The UK faces an epidemic of telephone scams. Fraudsters are constantly calling people up pretending to be their bank. But how can you be sure the number displayed on your screen in genuine? You can't. The telecom system is hopelessly insecure and shouldn't be trusted for anything more complicated than dialling the…
In this blog, we describe how we leverage the authentication and authorization support in Open Distro for Elasticsearch to make elasticsearch a...
A machine learning AI-esque robot challenges off on 156 trivia questions against tens of thousands of Water Cooler Trivia participants.
A surge involving the rapidly-transmitting Delta variant in heavily vaccinated countries has led to much hand-wringing that the vaccines are not effective against Delta, or vaccine effectivenss wanes after 4-6 months. This has fueled anti-vaccine sentiment suggesting the vaccines are not working, and causing much…
