Module 5 · suggested week 5

Complexity and Systems Thinking

Simple rules can produce surprising collective behaviour.

~2.5 focused hoursBuilds on modules 1, 4 (advisory)

0 of 8 sections read

What you will be able to do

Recognise the core vocabulary of complex systems (feedback, networks, nonlinearity, emergence, power laws, phase transitions, criticality) and learn why a shared shape across systems is where an investigation starts, not where it ends.

\propto

Proportional to

\alpha

Alpha

\log

Logarithm

e^{x}

Exponential

\Delta

Delta (change / difference)

\sum

Summation (capital sigma)

\infty

Infinity

\approx

Approximately equal

What “complex” means here

In ordinary speech "complex" means complicated. In complexity science it means something narrower and more interesting: many interacting parts whose collective behaviour is not obvious from the parts alone. A jet engine is complicated: thousands of parts, but each designed to do one thing, and the whole is the sum of its specified functions. An ant colony, a market, a brain or a large language model in training is complex. The interactions, not the blueprint, produce the behaviour.

The working vocabulary is small: feedback (outputs loop back to become inputs), nonlinearity (doubling the cause does not double the effect), networks (who interacts with whom), emergence (collective patterns no part encodes), and scaling (how properties change with size). This module gives each one an equation shape you can recognise on the page.

Established

Systems of simple interacting units can produce large-scale order that no unit encodes: flocking, traffic jams, magnetisation, segregation patterns. This is demonstrated in both mathematical models and laboratory physics.

Feedback: the loop that runs the world

Negative feedback pushes a system back toward a set point. A thermostat, blood glucose regulation, a predator population that crashes when prey runs out. Positive feedback amplifies deviations: a microphone howl, a bank run, compound interest, a viral post.

The simplest negative-feedback rule looks like this:

Discrete negative feedbackMathematical model

x_{t+1} = x_t + k\,(x^{*} - x_t)

Next state = current state plus a correction proportional to the gap between the target $x^*$ and where you are now. $k$ is the gain: how hard the system pushes back.

let x = 30;            // start far from target
const target = 37, k = 0.3;
for (let t = 0; t < 20; t++) {
  x = x + k * (target - x);  // close 30% of the gap each step
}

Does not say: Real controllers have delays, saturation and noise. With a delay, a high gain makes the system overshoot and oscillate, which the interactive below lets you see.

Negative feedback · homeostasis with delay

Gain (correction strength)0.40Sensor delay0 steps

behaviourReturns smoothly

final temperature37.0 °C

A toy temperature regulator: each step it senses the error from the set point (37 °C) and corrects by gain × error, using a reading that is delay steps old. Low gain returns smoothly. High gain overshoots. Add delay and even moderate gain oscillates, because the system keeps correcting an error that is already gone. Flip to positive feedback and the error runs away. The block diagram (sense → compare → act → sense) recurs in thermostats, glucose–insulin regulation and agentic AI loops (act → observe → correct). That shared structure is a level-2 connection, the mathematics of control theory. The mechanisms (neurons and hormones, relays, LLM calls) differ completely. Numbers are illustrative, not physiological.

Loading interactive…

Raise the gain, then add a delay. Fast correction plus late information produces oscillation, the same signature seen in over-steering drivers, boom-bust supply chains and poorly tuned agent loops that act on stale state.

Networks: who connects to whom

A network (or graph) is a set of nodes and the edges between them. Proteins that bind, neurons that synapse, web pages that link, people who follow each other, tokens that attend to each other. The single most informative statistic is often the degree distribution $p(k)$ : the fraction of nodes with exactly $k$ connections.

In a network where every possible edge appears independently with the same small probability (an Erdős–Rényi random graph), degrees cluster around an average and very-high-degree hubs are vanishingly rare. In many real networks, hubs are common, and the degree distribution falls off roughly like a power of $k$ .

Growing a network · who gets the links?

nodes2 / 150

largest hub degree1

mean degree1.00

Each new node links to 1 existing node. With uniform random attachment every node is equally likely to be chosen, and degrees stay fairly even. With preferential attachment (Barabási–Albert) the chance is proportional to current degree. The rich get richer, a few hubs emerge, and the degree distribution develops a heavy tail. Be careful: a roughly straight line on a log-log plot does notidentify the mechanism. Several different processes produce heavy tails, and Clauset, Shalizi & Newman (2009) showed that many claimed power laws fail careful statistical tests. With 150 nodes this is a picture, not a measurement.

Loading interactive…

Grow the same number of nodes two ways. Random attachment gives a narrow, bell-like degree histogram. Preferential attachment ("the rich get richer": new nodes link to already-popular nodes) gives hubs and a heavy tail that looks roughly straight on log-log axes.

Mathematical model

Preferential attachment, in which new nodes connect to existing nodes with probability proportional to their degree, provably generates a power-law degree distribution in the model. Whether any particular real network grew this way is a separate empirical question.

Power laws: the straight line on log-log paper

Recognition

Logarithms straighten power laws

Take $y = A x^{-\alpha}$ and take logs of both sides: $\log y = \log A - \alpha \log x$ . That is the equation of a straight line in the variables $\log x$ and $\log y$ , with slope $-\alpha$ . Every log-log plot in a scaling paper is this one move.

y = A x^{-\alpha} \;\Longleftrightarrow\; \log y = \log A - \alpha\log x

// slope on log-log axes recovers the exponent
const A = 5, alpha = 0.7;
const y = (x) => A * Math.pow(x, -alpha);
const slope = (Math.log(y(1000)) - Math.log(y(10))) /
              (Math.log(1000) - Math.log(10));   // → -0.7

Power-law explorer

L(N) = 1.70 + 12.0\,N^{-0.080}

α, exponent0.080A, scale12.0L∞, irreducible floor1.70Probe model size N10^9.0

L at probe3.987

L at 10× params3.602

10× buys (reducible part)×0.832

10× buys (absolute)−0.385

Illustrative constants, not a fit to real models. On log–log axes the reducible part, $AN^{-\alpha}$ , is a straight line with slope −α. The full curve bends toward the floor $L_\infty$ , the loss that no amount of scale removes in this model. On linear axes nearly all of the change happens at the far left, which is why scaling papers plot on log axes.

Loading interactive…

Switch between linear and log-log axes. On linear axes a power law looks like a ski slope. On log-log axes the pure power law is a straight line, and the constant floor bends it flat at large N.

Power laws are called scale-free because they have no characteristic size. Zoom in or out and the shape is the same, only rescaled. An exponential $e^{-x/x_0}$ has a built-in scale $x_0$ beyond which things become rare fast. A power law decays slowly, so extreme events (giant cities, huge earthquakes, mega-hubs) are far more common than a bell-curve intuition expects.

Go deeper: How power laws are actually tested

Clauset, Shalizi and Newman (2009) set out the now-standard procedure: estimate the exponent by maximum likelihood (not by drawing a line through a log-log histogram), estimate where the power-law regime begins, test goodness of fit, and compare against alternatives such as the log-normal and the stretched exponential. When they reanalysed two dozen published datasets, several "power laws" did not survive. Least-squares fits on log-log plots can produce confident-looking exponents for data that are not power-law at all.

Same shape, different machines

Here is the central lesson of the module. At least four quite different mechanisms produce power-law-looking data:

Four routes to a power law

	Mechanism	Where it shows up	What generates the exponent	Evidence status
Preferential attachment	Rich-get-richer growth	Citation networks, web links (proposed)	Growth rule: link probability ∝ current degree	Proven in the model; contested per dataset
Criticality	A system tuned to a phase transition	Magnets at the Curie point; fluids at the critical point	Symmetry and dimension, which explains why it is universal across materials	Established in statistical physics
Multiplicative processes	Random growth by percentages, with a floor or reset	City sizes, firm sizes, wealth (proposed)	Balance between multiplicative growth and the boundary	Well-understood models; the data are often near log-normal
Neural scaling	Test loss vs. parameters, data, compute	Language models (Kaplan et al. 2020) and other domains	Proposed: how a model resolves a data manifold of some intrinsic dimension (Sharma & Kaplan 2020; Bahri et al. 2021)	Empirical regularity with partial theory

Established

A power-law form alone does not identify its generating mechanism. Different processes produce statistically similar power laws, so the shape is underdetermined by the data.

Source: Clauset, Shalizi & Newman 2009 (arXiv:0706.1062)

Consider an ecologist who finds that species abundances follow a heavy tail, a physicist who measures magnetisation fluctuations near the Curie temperature, and an ML team that plots test loss against parameter count. All three get straight lines on log-log axes. A tempting synthesis says: "Ecosystems, magnets and neural networks are all critical systems obeying the same universal law."

The mathematical structure, $y \propto x^{-\alpha}$ , really is shared. That is a Level 2 connection. The mechanism is not shown to be shared: the magnet's exponent comes from symmetry and dimensionality at a phase transition; the loss curve's exponent has a proposed explanation in terms of data-manifold dimension; the species-abundance tail has multiple competing ecological explanations. To claim Level 3, you would need a mechanism that makes a distinguishing prediction (a specific exponent, a specific cutoff, a specific response to perturbation) and that prediction would need to survive a test.

Phase transitions and criticality

Heat a magnet and at a precise temperature (the Curie point) it loses its magnetisation. Water boils at a sharp temperature. These are phase transitions: smooth changes in a control parameter (temperature) produce abrupt changes in a collective property (magnetisation, density).

The Ising model is physics' fruit fly for this. Each site on a grid holds a spin, up or down. Neighbouring spins prefer to align; temperature jostles them randomly. Nothing in the rule says "make a magnet", yet below a critical temperature large aligned domains form, and above it they dissolve. Exactly at the critical temperature, domains of all sizes appear, and many quantities follow power laws.

2D Ising model · a phase transition you can watch

Temperature T2.27 (T/Tc 1.00)

magnetisation m0.000

phasenear critical

Each cell is a spin (up = dark, down = light) that prefers to match its four neighbours. Temperature T sets how often a spin flips against its neighbours (Metropolis algorithm). Below the critical temperature large aligned domains form and the lattice magnetises. Above it, noise wins. Lars Onsager solved the 2D square lattice exactly in 1944: T_c = 2/ln(1+√2) ≈ 2.269 (in units where J = k_B = 1). Near T_c, clusters of all sizes coexist: that is criticality. The energy function here has the same form as a Hopfield network (1982) and other energy-based models. That is shared mathematical structure; it does not make a neural network a magnet. A 64×64 lattice is small, so the transition is smeared.

Loading interactive…

Drag the temperature through the critical value (≈2.27 in the model’s units). Below it, big domains of one colour take over; above it, the grid is salt-and-pepper noise; near it, clusters of every size coexist. This is a real Metropolis simulation of the 2D Ising model on a small grid, so finite-size effects blur the transition slightly.

Ising energyMathematical model

E = -J \sum_{\langle i,j \rangle} s_i\, s_j

Add up, over every pair of neighbouring sites $\langle i,j\rangle$ , the product of their spins ( $+1$ or $-1$ ). Aligned neighbours contribute $-J$ (lower energy), misaligned contribute $+J$ . The system prefers low energy; temperature lets it explore higher-energy states.

let E = 0;
for (const [i, j] of neighbourPairs) {
  E += -J * spin[i] * spin[j];   // Σ is just this loop
}

Does not say: A deliberately minimal model. Its power comes from universality: very different materials share its critical exponents.

Inspect symbol by symbol →

Established

Universality: near a continuous phase transition, systems with very different microscopic details share the same critical exponents if they share symmetry and dimensionality. This is one of the deepest established results of 20th-century physics, explained by the renormalization group.

Self-organisation, constraints and optimisation

Self-organisation is order arising from local interactions without a central planner: termite mounds, sand ripples, Schelling’s segregation model, synchronised fireflies. Constraints shape what order is possible: conservation laws, limited energy, limited bandwidth, network topology.

Many self-organising systems can be described as if they minimise or maximise some quantity: energy, free energy, travel time, prediction error. This "as if" is powerful mathematically. Optimisation is a lingua franca across physics, biology and ML. But "can be described as minimising X" is weaker than "is driven by a mechanism that minimises X." A ball rolling downhill and a gradient-descent optimiser both reduce a quantity. Only one of them computes a gradient.

Analogy

Calling an ecosystem, an economy or a society "an optimiser" is usually a Level 1 analogy. Unless the objective is specified, measurable, and shown to predict behaviour better than alternatives, the description adds vocabulary rather than explanation.

Go deeper: Dynamical systems in one paragraph

A dynamical system is a rule for how a state changes: $x_{t+1} = f(x_t)$ in discrete time or $dx/dt = f(x)$ in continuous time. Its long-run behaviour can settle to a fixed point (thermostat), cycle (predator–prey), or wander chaotically while staying bounded (weather). Training a neural network is a dynamical system on the space of weights; gradient descent is the rule $f$ . That is a Level 2 framing: it lets you import tools (stability analysis, fixed points) without implying training "is" weather.

Reading complexity claims in the wild

When you meet a complexity-flavoured claim (in Quanta, on arXiv, or from an LLM), run it through four questions:

What is the measured quantity, and over what range? Two decades of data is a weak power law; seven is a strong one.
What alternatives were tested? A power law that beat a log-normal in a likelihood test is worth more than a pretty line.
Is there a mechanism, and does it predict something the shape alone does not? An exponent value, a cutoff, a response to intervention.
Is "emergence", "criticality" or "self-organisation" doing explanatory work, or decorative work?

Empirical regularity

Neural language-model loss fell as a smooth power law in parameters, data and compute over many orders of magnitude in the settings Kaplan et al. measured. This is an empirical regularity. Its exponents differ by setting, and later work (Hoffmann et al. 2022) revised how to allocate compute between parameters and data.

Source: Kaplan et al. 2020 (arXiv:2001.08361)

For the full scaling-law equation, see the scaling-law inspector. For an exercise in reading the paper itself, open the Kaplan 2020 paper guide.

Connections, with their levels

PhysicsLevel 3 · Shared mechanism

Criticality and universality

Near continuous phase transitions, power laws arise with exponents set by symmetry and dimension. This is the gold standard of a shared mechanism across different materials.

Why this level: Within physics, renormalization-group theory explains why different materials share exponents, and the predictions have been measured. Extending "criticality" to brains or AI is a separate, much weaker claim.

BiologyLevel 2 · Shared math

Heavy tails in ecology and gene networks

Species-abundance distributions and some molecular interaction networks show heavy tails. Several mechanisms (neutral drift, niche partitioning, duplication-divergence growth) have been proposed.

Why this level: Shared statistical form, with competing mechanistic explanations. Fit quality is debated dataset by dataset, so this is not Level 3.

AI / MLLevel 2 · Shared math

Neural scaling laws

Loss vs. model size, data and compute follows power laws in the measured ranges. Theory links the exponent to the intrinsic dimension of the data manifold.

Why this level: Same mathematical form as physical power laws. The proposed mechanism (manifold resolution) is ML-specific and is not the mechanism behind magnetic criticality.

ComplexityLevel 2 · Shared math

Feedback in agentic systems

Gain and delay explain oscillation in thermostats, supply chains and agent loops that act on stale information.

Why this level: The control-theory equations genuinely apply to any loop with gain and delay. Whether a given LLM agent behaves like a linear controller has to be checked, not assumed.

Skeptic’s corner: Before you believe a "universal" pattern

Complexity science is especially prone to beautiful over-generalisation because its central objects (power laws, networks, emergence) really do recur everywhere. Recurrence of form is cheap; recurrence of mechanism is expensive to demonstrate.

How many orders of magnitude does the power law span, and was it fit by maximum likelihood or a line through a log-log plot?
Were alternatives (log-normal, exponential cutoff) tested?
Are the exponents the same, and was that sameness predicted in advance?
Is "emergent" or "critical" explaining something, or labelling it?
What intervention would distinguish the proposed mechanism from the alternatives?

Watch & explore

CourseMelanie Mitchell · Santa Fe Institute Complexity Explorer

Introduction to Complexity

No prerequisites, and it covers dynamics, chaos, fractals, information, self-organisation, agent-based models and networks, the same toolkit as Module 5.

Watch for: The units on networks and scaling. Notice how often Mitchell separates "this is a model" from "this is how nature works".