1
When your data doesn’t fit in memory: the basic techniques
You can process data that doesn’t fit in memory by using four basic techniques: spending money, compression, chunking, and indexing.
Itamar Turner-Trauring
Helping you deploy with confidence, ship higher quality code, and speed up your application.
- Popular posts
- 13
- Total points
- 3,459
- HN comments
- 1,689
- Rank
- #392 personal
Visit pythonspeed.com → RSS feedOn Hacker News May 2019 – Aug 2026
Software Engineering 10Systems & Performance 5Infrastructure & DevOps 5Databases & Data 3Security & Privacy 1Tech Industry & Business 1
2
Loading Pydantic models from JSON without running out of memory
Pydantic’s JSON loading uses a huge amount of memory; here’s how to reduce it.
3
Loading NumPy arrays from disk: mmap() vs. Zarr/HDF5
Learn how to load larger-than-memory NumPy arrays from disk using either mmap() (using numpy.memmap), or the very similar Zarr and HDF5 file formats.