---
generator: contextjamming-llm-context
title: "Neural Networks & Transformers Explainer"
url: https://www.contextjamming.com/neuralnets
markdown_url: https://www.contextjamming.com/neuralnets.md
section: "Infographics & Dashboards"
content_type: "Infographic"
author: "Bret Kerr"
publisher: "Context Jamming (ACRA Insight LLC)"
date_published: 2026-07-17
language: en-US
word_count: 317
---

> **Neural Networks & Transformers Explainer** — Infographics & Dashboards, Context Jamming. This is a plain-Markdown copy of https://www.contextjamming.com/neuralnets, prepared for LLM context windows, agents and retrieval.
>
> Cite as: Bret Kerr, "Neural Networks & Transformers Explainer," *Context Jamming*, July 17, 2026, https://www.contextjamming.com/neuralnets

Gradient Descent & Optimization

## Visualizing the Loss Landscape & Neuron Activations

In a neural network, learning is the process of minimizing a cost function. We visualize this cost as a **loss landscape** in high-dimensional parameter spaces. By taking steps proportional to the negative gradient (w ← w - η ∇C), the weights roll down the curve. Watch how the learning rate η affects this trajectory: normal values converge smoothly, while a learning rate above 0.08 oscillates wildly and climbs up the landscape (divergence!). Simultaneously, observe the output layer activations converging onto the target digit **"3"**.

Current Epoch

0

Simulation steps run

Mean Squared Error (C)

62.9200

↓ -0.0000

Active Parameters

13,002

MNIST 784→16→16→10

Ready to run

### The Loss Landscape (2D Projection)

The red line plots the loss function C(w) = 13(w - 2)^2 + 0.1. The yellow dot is the network's weight parameter. Notice how high learning rates overshoot the minimum.

### Activation Distribution (Digit Output Neurons)

As the network minimizes loss, the output activation of target neuron **"3"** should rise to 1.0 (indicating confidence), while wrong classes converge to 0.

Attention Mechanisms & Semantic Embeddings

## Demystifying Self-Attention & High-Dimensional Superposition

In transformer architectures, words are mapped to a high-dimensional semantic space (word embeddings). Through **Self-Attention**, tokens communicate. Each token generates a *Query*, *Key*, and *Value*. By computing dot products (softmax(Q K^T / √dk)), tokens route information. Observe how the word **"bank"** shifts its embedding vector: originally ambiguous, attention weights pool context from **"river"**, pulling the representation into a hydrology-related semantic space.

Embedding Dimensions (dmodel)

12,288

GPT-3 scale vector space

Attention Heads (h)

96

Parallel routing pipelines

Softmax Output Range

\[0.0, 1.0\]

Attention weights normalization

### Self-Attention Weights: α = softmax(Q K^T / √dk)

Attention weights representing how the query token (from the dropdown) looks at the keys in the sequence: **"The bank of the river"**. Note the strong link between "bank" and "river".

### Vector Superposition & Contextual Update

Compare the original static dictionary embedding (**teal**) with the attention-updated contextual representation (**yellow**) along key semantic dimensions.