Gradient descent update

Mathematical modelAI / MLMathematics

\htmlClass{sjt-theta1}{\theta_{t+1}} = \htmlClass{sjt-theta}{\theta_t} - \htmlClass{sjt-eta}{\eta}\,\htmlClass{sjt-grad}{\nabla_\theta} \htmlClass{sjt-L}{L(\theta_t)}

Tap a symbol above, or choose one here:

\theta_{t+1}

Next parametersRosetta entry →

Meaning

All the model’s weights after this update step.

Its job in this equation

The output: where the model moves to.

Picture it

Your position after taking a step.

Read it, progressively

1. Gist

Take a small step downhill, then repeat.

What it does not establish

It guarantees nothing about reaching a global minimum, about generalisation, or about what the model learns. It is a schematic: treat it as the idea behind production optimisers, not their specification. Do not read it as a description of biological learning or of evolution, which do not compute gradients this way.

Play with it

Gradient descent landscape

Learning rate

\eta

0.10Momentum0.00

step t0

θ(-2.60, 1.40)

L(θ)6.3870

‖∇L‖8.409

Each step applies $\theta_{t+1} = \theta_t - \eta\,\nabla_\theta L(\theta_t)$ : compute the slope at the current point and move a distance $\eta$ times that slope, downhill. In the narrow valley a large $\eta$ overshoots across the steep walls and zig-zags or diverges, while a small one crawls along the floor. Momentum (a running average of past steps) is one of the fixes used by real optimisers such as Adam. A real model has billions of θ dimensions rather than two, and the gradient is estimated from minibatches, so this is an idealisation.

Loading interactive…

§ Module 6: The Mathematics Inside Modern AI Next equation: Scaled dot-product attention →