Here are articles I've written on the inner workings of modern AI. Each article is a "companion" to a video in the YouTube series on AI created by 3Blue1Brown.
-
Transformers, the tech behind LLMs
We explore transformers, the core invention behind the explosion in modern AI heralded by the release of ChatGPT in November 2022. At a very high level, a (chatbot) transformer is merely a function that produces a ranking of most likely next words in response to an input prompt. However, the prerequesite of understanding how to represent words from human languages- in a concrete mathematical form that encodes semantics and meaning- is quite daunting. We start with this, investigating how word representation has was achieved in 2013 by "embedding" words as vectors into a high-dimensional "space", then discussing the entertaining $\text{queen} - \text{king} = \text{woman} - \text{man}$ example, and showing how to compute how much two word representations align.
Only after this can we approach the attention mechanism, the heart of the transformer. Getting a glimpse at how it takes advantage of the high-dimensionality of the embedding space to enrich word representations with context is pretty magical, and the key step. We end by explaining how the alignment computation is used along with the $\text{softmax}$ function on enriched vectors to produce the desired probability distribution.
Naturally, most of you will want to read about transformers first. Great! You will get a strong sense of how transformers work. But for a complete understanding, I do recommend reading the below articles. You would read them first if you were being super strict about prerequesites.
-
But what is a neural network?
We pose the problem of image recognition and reason about how we can build an AI to solve it. The kind of AI we end up considering is a neural network- a composition of many little computation machines called neurons. We explain why it is plausible for a network like this to solve the problem, and additionally why it is natural for a successful network to be arranged in a layered structure. Then we go in-depth with some formulas, using the example of edge detection to illustrate why neurons operate on their inputs by computing weighted sums, and why sigmoid functions are used for normalization (or "squishification"- as 3Blue1Brown puts it).
-
Gradient descent, how neural networks learn
We introduce the core principle behind modern AI: training. We learn how when we have training data we can measure the error rate of a neural network with its cost function, or loss function. Then we qualitatively describe how in a process called gradient descent, we can compute the gradient of the cost function and use it to nudge the configuration of the network so that the loss function is decremented as efficiently as possible.
-
Backpropagation, intuitively
With the broad strokes of neural networks sketched by the previous two articles and videos, we focus on the gradient descent algorithm. We explain how computing the gradient of the entire network's cost function reduces to the problem of computing the gradients of cost functions associated with individual neurons, in a recursive manner that starts at the last layer and advances towards the first. This amounts to an intuitive description of the backpropagation algorithm.
-
Backpropagation calculus
In this "all the details" version of the previous article, we derive the standard backpropagation equations from the ground up. The approach is of course more formula-heavy than the one in the previous article; suprisingly, it is also organizationally more direct. After we derive a verbose form of the backpropagation equations, we introduce the standard notation $\delta^{(\ell)}_j$, which makes them more concise.
-
Extra: summary article
This article summarizes all of the content presented in the previous articles. It also adds some new content. We refine an idea that was alluded to in the very first article, which is that the core idea behind neural networks is to take advantage of the fact that patterns are made up of subpatterns. We also give a "vectorized" version of the formula for the activation of a neuron.
-
Extra: computational universality
The "companion articles" are inspired by 3Blue1Brown's video series. And 3Blue1Brown's video series is inspired by
Michael Nielsen's excellent book on neural networks. The main concept from Nielsen's book
not present in the video series is that of
computational universality, which addresses the question: "How can we be sure that a neural network can solve any computational problem?" This article explores that topic.