Scaling request logging with ClickHouse, Kafka, and Vector
How we solved request logging at scale by moving from MariaDB to ClickHouse, Kafka, and Vector after our deprecated database engine couldn't keep up with billions of monthly requests.
Topics /
The most popular blog posts about infrastructure & devops on Hacker News. 4,375 posts.
How we solved request logging at scale by moving from MariaDB to ClickHouse, Kafka, and Vector after our deprecated database engine couldn't keep up with billions of monthly requests.
Learn what 768K day is and what it may mean for businesses that rely on the Internet to operate.
No single cloud fits all. Explore how we build a more flexible future with hybrid cloud.
I always smile when I hear a web performance speaker say this. I forget who said it first, but I’ve heard it numerous times at conferences and meetups over the past few years. It’s true! Caching is critical for making web pages faster. I’ve written extensively about caching:
Pixie Blog
Artwork courtesy of the Jenkins project. At Slack we manage a sophisticated Jenkins infrastructure to continuously build and test our mobile apps before release. We have hundreds of jobs running in a variety of different environments. One day something very odd happened — our Jenkins UI stopped working although the…
So, your machine now needs minutes to boot before you can ssh in where it used to be seconds before the Debian Buster update?
GitLab first became the standard for self hosting git with two-thirds of the market, then became the next generation CI system, and the next step is creating Auto DevOps.
We've been working on isolating the 1,500 services that power Monzo. The sheer size and complexity of our platform made this difficult. But by protecting us against compromised services, it makes Monzo more secure.
Amazon Elastic File System (Amazon EFS) was announced in early 2015 and became generally available in 2016. We launched EFS in order to make it easier for you to build applications that need shared access to file data. EFS is (and always has been) simple and serverless: you simply create a file system, attach it […]
🇬🇧 Article in English We always need to customize our deployment with Kubernetes and, I don’t know why but the main tool around for now is HELM which throws away all the logic we learn on docker and Kubernetes. Here I will introduce to you an alternative called Kustomize ❤️
Today we announce Consul, a solution for service discovery and configuration. Consul is completely distributed, highly available, and scales to thousands of nodes and services across multiple datacenters. Some concrete problems Consul solves: finding the services applications need (database, queue, mail server, etc.)…
We’re pleased to bring code search to you with the newly revamped Cloud Source Repositories in beta availability.
If you're building an application that needs to scale—and we all tell ourselves that we are—then at some point you have to figure out if it does or not. This is where load testing comes in: if you want to see whether or not your application can handle scale, just generate scale and see if it can handle it! It sounds…
Service mesh is a blazing hot topic in software engineering right now and rightfully so. I think it’s extremely promising technology and I’d love to see it widely adopted (in cases where it truly makes sense). Yet it remains shrouded in mystery for many people and even those who are familiar with it have trouble…
In Ukraine suicide drones strike critical infrastructure regularly. Some outages are caused directly by the attacks - destroyed power plants, substations, transformers, power lines, some are scheduled in advance - power grid wasn’t designed to work while some of its power plants and transformers are destroyed and are…
Purism makes premium phones, laptops, mini PCs and servers running free software on PureOS. Purism products respect people's privacy and freedom while protecting their security.
Discover the tech stack used by Uploadcare. Explore 45 tools across 4 categories.
Everything at Google, from Search to Gmail, is packaged and run in a Linux container. Each week we launch more than 2 billion container inst...
Microservice architecture paradigm fits well within the Unix’s first principle/philosophy: “Make one program do one thing well”. We l...
What nginx logs actually prove about AI traffic from ChatGPT, Claude, and Perplexity, plus why Google's AI crawlers leave a different signal than the others.
Docker announces significant upgrades to its subscription plans, delivering more value, flexibility, and tools for customers of all sizes.
Understanding our data and usage patterns was the real key.