Aubrey Tatarowicz, Carlo Curino, Evan P. C. Jones , and Samuel Madden. Lookup Tables: Fine-Grained Partitioning for Distributed Databases . In ICDE , Washington, DC, USA, Apr. 2012. ( pdf )
Visit evanjones.ca → RSS feedOn Hacker News Jan 2012 – Nov 2023
Systems & Performance 10Networking & Internet 2Software Engineering 1Life, Learning & Writing 1Infrastructure & DevOps 1Hardware & Electronics 1
I think it is safe to state that hash tables (e.g. maps in Go, dicts in Python, HashMap in Java, etc.) are far more common than ordered data structures such as trees for in-memory data structures. One data point is that in a talk about C++ hash table optimization at Google ( available in Abseil ), the speaker noted…
I was wondering: how often do nanosecond timestamps collide on modern systems? The answer is: very often, like 5% of all samples, when reading the clock on all 4 physical cores at the same time. As a result, I think it is unsafe to assume that a raw nanosecond timestamp is a unique identifier.
You can't safely use the C setenv() or unsetenv() functions in a program that uses threads. Those functions modify global state, and can cause other threads calling getenv() to crash. This also causes crashes in other languages that use those C standard library functions, such as Go's os.Setenv ( Go issue ) and Rust's…
Nearly all programs are written to access virtual memory addresses, which the CPU must translate to physical addresses. These translations are usually fast because the mappings are cached in the CPU's Translation Lookaside Buffer (TLB) . Unfortunately, virtual memory on x86 has used a 4 kiB page size since the 386 was…
Durability is the guarantee that data can be accessed after a failure. It seems like this should be very simple: either your system provides durable data storage, or it does not. However, durability is not a binary yes/no property, and instead should be defined as the kinds of failures you want your data to survive…
Recently at Bluecore I ran into a familiar situation: during brief periods of overload, all requests to a server were timing out. It was a catastrophic failure, where the server completely stopped serving. We added more capacity to prevent overload, but we also wanted to fix the server. Instead of exploding, it should…
Google's gRPC is an RPC system that supports many languages, and is relatively widely used. I think its popularity is due to being used for parts of Docker and Kubernetes. I think gRPC is mostly fine, but it is surprisingly easy to screw up by misconfiguring it. Part of that is because RPC systems are challenging to…
I defended my Ph.D. thesis on September 7th, 2011. Technically, I'm still not done for a few complicated reasons, but at this point I am willing to declare victory and say I've graduated. Like most other major events in life, this has caused me to reflect on the past four years, when I quit my job at Google to become…
TL;DR: The JVM by default exports statistics by mmap -ing a file in /tmp (hsperfdata). On Linux, modifying a memory mapped file can block until disk I/O completes, which can be hundreds of milliseconds. Since the JVM modifies these statistics during garbage collection and safepoints, this causes pauses that are…
The UNIX fork() system call seems like an elegant way to process tasks in parallel. It creates a copy of the current process, allowing tasks to read the same memory, without needing fine-grained synchronization. Unfortunately, you need to use it extremely carefully in large programs because fork in a multi-threaded…
At Twitter, a team had a unusual failure where corrupt data ended up in memcache. The root cause appears to have been a switch that was corrupting packets. Most packets were being dropped and the throughput was much lower than normal, but some were still making it through. The hypothesis is that occasionally the…