Spotify Engineering company blog

Spotify Optimized the Largest Dataflow Job Ever for Wrapped 2020

engineering.atspotify.com

In this post we’ll discuss how Spotify optimized and sped up elements from our largest Dataflow job, , for using a technique called Sort Merge Bucket (SMB) join. We’ll present the design and implementation of SMB and how we incorporated it into our data pipelines.

Points

196

HN comments

125

Submissions

1

Cited by

0 commenters

Databases & DataInfrastructure & DevOps2021

Related posts from the same era

Hosting SQLite databases on GitHub Pages or any static file hoster

Pphiresky20211,812 points314 commentscited by 26 details

I'm all-in on server-side SQLite

Fly.io Blog20221,353 points565 commentscited by 16 details

AWS announces forks of Elasticsearch and Kibana

AWS Blog20211,314 points935 comments details

CRDTs are the future

Joseph Gentle20201,181 points357 commentscited by 12 details

Things I wished more developers knew about databases

Jaana Dogan20201,011 points445 comments details

SQLite is not a toy database

Anton Zhiyanov2021986 points484 comments details