New

The latest blog posts to take off on Hacker News. 21,819 posts.

Software Engineering
Any year
4147

The Birth of Parquet

15 years ago (2007-2011) I was at Yahoo! working with Map/Reduce and Apache Pig, which was the better Map/Reduce at the time. The Dremel paper just came out and, as everything I worked with seemed to be inspired from Google papers, I read it. I could see it applying to what we were doing at Yahoo! and this was to…

4160

A 100x speedup with unsafe Python

We're going to speed up some numpy code by 100x using "unsafe Python." Which is not quite the same as unsafe Rust, but it's a bit similar, and I'm not sure what else to call it... you'll see. It's not something you'd use in most Python code, but it's handy on occasion, and I think it shows "the nature of Python” from…

4165

Using a LLM to compress text

Introduction Large language models are trained on huge datasets of text to learn the relationships and contexts of words within larger documents. These relationships are what allows the model to generate text. Recently I've read concerns about LLMs being trained on copyrighted text and reproducing it. This got me…