Softmax with temperature

EstablishedAI / MLMathematics

\htmlClass{sjt-p}{p_i} = \frac{\htmlClass{sjt-num}{e^{z_i / T}}}{\htmlClass{sjt-den}{\sum_{j} e^{z_j / T}}}

Tap a symbol above, or choose one here:

p_i

Probability of option i

Meaning

For a language model, the probability of token i being next.

Its job in this equation

Output: a proper distribution (non-negative, sums to 1).

Picture it

The share of the vote option i gets.

Read it, progressively

1. Gist

Turn a list of arbitrary scores into probabilities, with a dial (temperature) for how decisive to be.

What it does not establish

Softmax probabilities are not calibrated confidence by default. A model can be "99% sure" and wrong. Temperature changes sampling, not knowledge.

Play with it

Token probabilities

The transformer was introduced in a paper titled “Attention Is All You▁

Need

95.9%z=9.1

Want

1.9%z=5.2

Get

0.9%z=4.4

Have

0.6%z=4.1

Know

0.3%z=3.2

Require

0.2%z=2.9

need

0.1%z=2.4

Deserve

0.1%z=1.6

Temperature

T

1.00

Entropy of next-token distribution0.33 bits

Loss if true token is “Need”0.042 nats

T → 0 behaves likeargmax (greedy)

A model outputs one score (logit) $z_i$ per vocabulary token. Softmax, $p_i = e^{z_i/T} / \sum_j e^{z_j/T}$ , turns the scores into probabilities that sum to 1. Temperature $T$ rescales the scores before the exponent: low T sharpens the distribution, high T flattens it. These eight logits were written by hand for teaching, and a real vocabulary has tens of thousands of tokens. The training loss on this step would be $-\ln p(\text{“ Need”})$ .

Loading interactive…

§ Module 6: The Mathematics Inside Modern AI Next equation: The Boltzmann distribution →