1
When your data doesn’t fit in memory: the basic techniques
You can process data that doesn’t fit in memory by using four basic techniques: spending money, compression, chunking, and indexing.
Itamar Turner-Trauring
Helping you deploy with confidence, ship higher quality code, and speed up your application.
- Popular posts
- 13
- Total points
- 3,459
- HN comments
- 1,689
- Rank
- #392 personal
Visit pythonspeed.com → RSS feedOn Hacker News May 2019 – Aug 2026
Software Engineering 10Systems & Performance 5Infrastructure & DevOps 5Databases & Data 3Security & Privacy 1Tech Industry & Business 1
2
Simple Dockerfile examples are often broken by default
Most Dockerfile examples for Python you’ll find on the Web are broken. And that’s a problem.
3
Alpine makes Python Docker builds slower, and images larger
Alpine Linux is often recommended as a smaller, faster Docker base image. But if you’re using Python, it will slow down your build and make your image larger.
4
Copying data is wasteful, mutating data is dangerous
Copying data wastes memory, and modifying/mutating data can lead to bugs. Learn how to implement a compromise between the two in Python: hidden mutability.
5
A deep dive into the official Docker image for Python
The “official” Python Docker image is useful, but to actually understand why, and to use it correctly, it’s worth understanding how exactly it’s constructed.
6
How many CPU cores can you use in parallel?
Figuring out how much parallelism your program can use is surprisingly tricky.
7
Faster Python calculations with Numba
Python-based calculations, especially those that use NumPy, can run much faster by using the Numba library.
8
Loading Pydantic models from JSON without running out of memory
Pydantic’s JSON loading uses a huge amount of memory; here’s how to reduce it.
9
Measuring memory usage in Python: it’s tricky
Measuring your Python program’s memory usage is not as straightforward as you might think. Learn two techniques, and the tradeoffs between them.
10
Faster floating point math with Rust's new API
Compilers have a hard time optimizing floating point math. Rust has a new API to fix this.