Attention Is Off By One
Let’s fix these pesky Transformer outliers using Softmax One and QuietAttention.
Articles, software, calculators, and opinions.
Visit evanmiller.org → RSS feedOn Hacker News Apr 2010 – Jul 2023
Programming Languages 8Software Engineering 6Math & Algorithms 5Tech Industry & Business 3Web & Frontend 2Tools & Productivity 2
Let’s fix these pesky Transformer outliers using Softmax One and QuietAttention.
Users are rating items on your website. How do you know what the highest-rated items are?
Part of my work involves the mild reverse-engineering of binary file formats. I say “mild” because usually other people do all of the actual work; I just have to figure out what an extra flag field or two means, and I then take as much credit as possible for the discovery on my blog.
The problem with most programming languages is they’re designed by language geeks, who tend to worry about things that I don’t much care for. Safety, type systems, homoiconicity, and so forth. I’m sure these things are great, but when I’m messing around with a new project for fun, my two concerns are 1) making it work…
Nothing! Just kidding, you’ll see a lot of distressed messages from tasks that passed out wait for the phone to ring, and woke up wearing someone else’s OS thread. 1
When someone says “7 is greater than 5”, you declare that they’re really the same.
I will attempt to convince you, in two lines of code, that Elixir is more interesting than any programming language you’ve ever used.
Erlang is a twenty-five-year-old programming language that has yet to win a popularity contest, and almost certainly will never win any medals for speed, let alone any tiaras for syntactic beauty. The language is slow, awkward, and ugly. Refactoring Erlang code is a pain.
The programming profession is blessed with a number of gifted essayists. Today I will pick on three of my favorites — Eric Raymond, Paul Graham, and Steve Yegge — because they all seem to disagree about why (and whether) mathematics is relevant to the practicing programmer. Their attitudes might be summarized as:
[ Update , Feb. 5, 2020: Perl 6 is now called Raku. I’ve updated the review title, but left the review body intact. ]
UPDATE, 5/7/2013 : Some off-hand remarks in this article about HTML5 drawing speeds generated some controversy in the hackersphere. To read more, see Adrian Holovaty's post In Defense of <canvas> , and my response, Hackers and <canvas> .
A review of Rust, the programming language built for safety – not necessarily for comfort.
I admit to feeling a twinge of excitement last summer when I read that Paul Graham was leaving his jillion-dollar job to write more essays. It felt like the thinking man’s equivalent of LeBron James returning home to the Cleveland Cavaliers to play ball. I don’t know; it’s just a good human story when people decide to…
A few things about Reddit’s hot formula have always bothered me. The formula has obviously been a success when it comes to setting the Internet on fire, but I have to wonder:
C has a reputation for being inflexible. But did you know you can change the argument order of C functions if you don't like them?
If you’ve ever needed to do basic statistical calculations for an A/B test, your options traditionally have been:
When I first learned to drive a car, shortly after turning 15, my mother insisted that I learn on my father’s Mazda Miata: a silver, two-door convertible with a soft-top roof.
In the movie Avatar , there’s this big badass bird-brained pterodactyl thing called a Toruk that the main character must learn to ride in order to regain the trust of the blue people. As a general rule, Toruks do not like to be ridden, but if you fight one, subdue it, and then link your Blue Man ponytail to the…
An essay that bids farewell to x87 – a computing architecture too long for this world.
Do adventure games have dead ends? The author of The Teeny Tiny Mansion proves the absence of dead ends in that game via formal verification (in particular, with the help of the cbmc model-checking program ). Here I’ll demonstrate how to prove the same thing via Markov theory. The Markov approach requires more…
Are you falling victim to the peeking problem? Learn how to design and run an experiment properly.