Posts that commenters keep linking to, ranked by how many different people have cited them in Hacker News comments. 79 posts.
Stephen Wolfram explores the broader picture of what's going on inside ChatGPT and why it produces meaningful text. Discusses models, training neural nets, embeddings, tokens, transformers, language syntax.
I sometimes see people refer to neural networks as just “another tool in your machine learning toolbox”. They have some pros and cons, they…
One piece of slang that has long embodied the short attention span Internet age is TL;DR, short for “too long; didn’t read.” With the explosion of generative AI tools, we’re rapidly entering the age of TL;DW: “too long, didn’t write.” A January survey from Fishbowl found that 40% of nearly 12,000 workers have used…
Fictional short story about Clippy & AI hard takeoff scenarios grounded in contemporary ML scaling, self-supervised learning, reinforcement learning, and meta-learning research literature.
Here, I provide an in-depth analysis of GPUs for deep learning/machine learning and explain what is the best GPU for your use-case and budget.
I find this literature irritating and opaque. This is at least somewhat because I do not yet understand it well, and there's too much of it. But clearly I need to wrap my head around it, before I become technically obsolete. My scare quotes in the title of these notes thus derive in part from jealousy and fear. But…
It’s well-known at this point that GPT-4/GPT-3.5-turbo is non-deterministic, even at temperature=0.0. This is an odd behavior if you’re used to dense decoder-only models, where temp=0 should imply greedy sampling which should imply full determinism, because the logits for the next token should be a pure function of…
Wolfram plugin gives computationally accurate answers to ChatGPT queries using Wolfram's data knowledgebase and language. Custom visualizations are given as well. Stephen Wolfram explains how it works.
I procrastinated a deep dive into transformers for a few years. Finally the discomfort of not knowing what makes them tick grew too great for me. Here is that dive.
Yesterday, Sam Altman and Greg Brockman were fired from the Board of Directors of OpenAI.
On GPT-3: meta-learning, scaling, implications, and deep theory. The scaling hypothesis: neural nets absorb data & compute, generalizing and becoming more Bayesian as problems get harder, manifesting new abilities even at trivial-by-global-standards-scale. The deep learning revolution has begun as foretold.
I really want an AI assistant: a Large Language Model powered chatbot that can answer questions and perform actions for me based on access to my private data and tools. …
AI folklore tells a story about a neural network trained to detect tanks which instead learned to detect time of day; investigating, this probably never happened.
A machine learning researcher writes me in response to yesterday’s post, saying:I still think GPT-2 is a brute-force statistical pattern matcher which blends up the internet and gives you bac…
Implementing a GPT model from scratch in NumPy.
Here we go again. I started another company. The money is in the bank.
Proving you're a human on a web flooded with generative AI content
Prompt Engineering, also known as In-Context Prompting, refers to methods for how to communicate with LLM to steer its behavior for desired outcomes without updating the model weights. It is an empirical science and the effect of prompt engineering methods can vary a lot among models, thus requiring heavy…
In this post, I’ll iteratively optimize an implementation of matrix multiplication written in CUDA.My goal is not to build a cuBLAS replacement, but to deepl...
[LinkedIn discussion, Twitter thread]
Accessing Wolfram|Alpha's computational knowledge with ChatGPT--an ideal combination of precise computation with human-like expression of ideas. Stephen Wolfram explains how.
Written in collaboration with Joseph Miller. See the discussion of this post over on LessWrong.
I am publishing this because many people are asking me how I did it, so I will explain. When I talk about a model, I'm talking about a huggingface…
Embeddings are a really neat trick that often come wrapped in a pile of intimidating jargon. If you can make it through that jargon, they unlock powerful and exciting techniques …
The end of a dramatic weekend in tech is that OpenAI has split and Microsoft is partnered with one and has hired the other; this is the ultimate failure case of what should have been a for-profit c…
( This is the text of a talk I delivered at the Next Frontiers Applied Fiction Day in Stuttgart on Friday November 10th, 2023. Note: early draft, contains some typos, I'll fix them next week when I get home. )
Discussions: Hacker News (64 points, 3 comments), Reddit r/MachineLearning (219 points, 18 comments) Translations: Simplified Chinese, French, Korean, Russian, Turkish This year, we saw a dazzling application of machine learning. The OpenAI GPT-2 exhibited impressive ability of writing coherent and passionate essays…
Yesterday, President Joe Biden declared that “it remains to be seen” whether Artificial Intelligence (AI) is “dangerous.” I would beg to differ. I have been writing about th…
