The Mathematics You Need Before the Models Make Sense

Papers about models are dense with symbols, and the first reaction is often to assume the whole subject requires years of study before it becomes readable. It does not. A working understanding rests on a small set of ideas, each understood well enough to check your own work. Most of the rest can be learned as you go, once there is a reason to need it.

Vectors and the dot product

A vector is an ordered list of numbers. Read it either as a point in space or as an arrow from the origin, whichever helps. A model input such as a short list of measurements, or the numerical representation of a word, is a vector with many entries. The intuition you need is that vectors can be added, scaled and compared.

The dot product is the comparison. Multiply matching entries and add the results. The number is large and positive when two vectors point in similar directions, near zero when they are unrelated in that sense, and negative when they point away from each other. Much of the similarity search behind retrieval systems is a dot product or a close relative of it, applied to thousands of vectors at once.

Matrices as maps

A matrix is most useful as a function that takes a vector and returns another vector. Multiplying a matrix by a vector means each output entry is a weighted sum of the input entries, with the weights taken from one row. Keep track of shapes as you go. A matrix with m rows and n columns takes a vector of length n and returns one of length m. Many bugs in model code that look mysterious turn out to be shape mismatches. A matrix with two rows and three columns expects a vector of length three, and multiplying it by a vector of length two fails. The error message is usually the first clue to what the matrix was built to accept.

For a two-by-two matrix, test your understanding by picturing what it does to the unit square. The square becomes a parallelogram. The area of that parallelogram is the absolute value of the determinant, and a determinant of zero means the map squashes space onto a line or a point, so it cannot be reversed. A diagonal matrix with entries 2 and 3 makes this concrete. It stretches horizontal lengths by 2 and vertical lengths by 3, so the square becomes a rectangle of area 6, which matches the determinant.

In many neural network layers, most of the arithmetic is a matrix multiplied by a vector, followed by a simple non-linear function applied to each entry. If you understand that single step, the rest of the architecture is largely repetition at different sizes.

Probability as a way of scoring possibilities

A language model's output is a probability distribution over possible next tokens. Three ideas carry most of the weight here. A probability is a number between zero and one, and the values across all outcomes add up to one. A conditional probability is the chance of one thing given another. An expectation is a weighted average, where each value is weighted by its probability.

The loss commonly used for classification is built from these ideas. If the correct answer is assigned a probability of 0.9, the loss is small. If it is assigned 0.01, the loss is large. The logarithm in that penalty turns products of probabilities into sums, which is why it appears everywhere, and at first you can treat it as bookkeeping. A model that generates text samples its next token according to these probabilities, so the same input can produce different outputs on different runs.

Gradients and small steps

A derivative measures how a function changes when its input changes a little. A gradient is the same idea for a function of many variables: it points in the direction in which the function increases fastest. Training a model is largely a matter of nudging its parameters a small distance opposite the gradient, many times, with the step size set by a learning rate. The step size is a real trade-off. If it is too large, the parameters overshoot the minimum and the loss can rise. If it is too small, progress is slow and expensive. It is usually settled by trying a few values and watching the loss over a short run.

You can check this with a few lines of code. Take a function you can differentiate by hand, and compare the hand result with a numerical estimate.

def f(x):
    return x * x

h = 1e-6
estimate = (f(3 + h) - f(3)) / h
print(estimate)  # about 6, matching the derivative 2x at x = 3

If the two agree, the idea is working in your hands. If they disagree, you have found a misunderstanding cheaply.

What to skip and how to know you understand

Leave the formal proofs of convergence, measure theory and detailed spectral methods aside until a specific paper asks for them. They are real mathematics, but they are rarely needed to use models well. A sensible order is vectors and dot products, then matrix-vector multiplication, then one-variable derivatives, then partial derivatives, then probability distributions and expectations. Apply each step to a small example before moving on. Nothing in this order is wasted. Attention mechanisms in language models are built from the same dot products, weighted sums and probabilities, so the early ideas keep paying off long after the basics are secure.

The test of understanding is prediction. Before computing, say what a matrix will do to a direction, or whether a gradient step will increase or decrease a loss. Explain the dot product in a sentence a non-specialist could follow. If you can do those things without looking up a formula, the idea is yours. If you can only reproduce the symbols, go back to one example and work it again with real numbers.