Calculus and Gradients

What it is and why it matters

You're in a shower with one knob. The water is too cold. You don't need a formula for water temperature; you need two answers: which way to turn the knob, and how much the temperature changes per turn. Turn, check, turn again.

Training a model is the same, except the "temperature" is the model's error (the loss) and there isn't one knob but millions of them, the weights. Calculus gives both answers for every knob at once:

• The derivative says how fast the loss changes when one weight moves a little.
• The gradient collects that answer for every weight into one vector.
• The chain rule lets you compute it through a long chain of steps, which is exactly what a neural network is.

You won't solve integrals in this course. You need slopes, and the habit of checking them.

How it works

Derivative = slope. For a function f(w), the derivative f'(w) is the change in f divided by a tiny change in w. You can always measure it by nudging:

f'(w) ≈ (f(w + h) − f(w − h)) / (2h)        with h tiny, like 0.00001

Positive slope: increasing w increases the loss, so move w down. Negative slope: move w up. Zero: you're at a flat spot, maybe the bottom.

A tiny model. Prediction p = w × x. For one example with x = 2 and the right answer 10, the loss is the squared error: loss = (w·x − 10)². At w = 3 the prediction is 6, the error is −4, the loss is 16.

Chain rule. The loss is a chain: w → p = w·x → err = p − 10 → loss = err². Each step has a simple local slope: d(loss)/d(err) = 2·err, d(err)/d(p) = 1, d(p)/dw = x. The chain rule says: multiply the local slopes along the chain.

d(loss)/dw = 2·err × 1 × x = 2·(−4) × 1 × 2 = −16

That's the whole of backpropagation (Part 3): run the chain forward to get values, then walk it backward multiplying slopes.

Partial derivatives and the gradient. With two knobs, p = w·x + b, you ask the slope question for each knob separately, holding the other still. Those are partial derivatives, ∂loss/∂w and ∂loss/∂b. Put them side by side and you have the gradient, a vector that points in the direction where the loss rises fastest. Training steps the opposite way.

Visual

Practical example

The chain-rule gradient compared with the nudging method at three values of w, a check that the derivative really predicts the change in loss, and a gradient with two knobs.

# A one-weight "model": prediction = w * x. One example: x = 2, the right answer y = 10.
x, y = 2.0, 10.0

def loss(w, b=0.0):
    return (w * x + b - y) ** 2

def grad_w(w, b=0.0):
    # chain rule: d/dw (err^2) = 2*err * d(err)/dw = 2*err * x
    return 2 * (w * x + b - y) * x

def grad_b(w, b=0.0):
    return 2 * (w * x + b - y) * 1

def numeric(f, w, h=1e-5):
    # slope measured by nudging w a tiny bit both ways
    return (f(w + h) - f(w - h)) / (2 * h)

for w in [3.0, 5.0, 6.0]:
    print(f"w={w:.1f}  loss={loss(w):5.2f}  grad by rule={grad_w(w):6.2f}  grad by nudging={numeric(loss, w):6.2f}")

# The derivative predicts how the loss changes for a small step.
w, step = 3.0, 0.1
print(f"step w {w:.1f} -> {w + step:.1f}: predicted loss {loss(w) + grad_w(w) * step:.2f}, actual {loss(w + step):.2f}")

# Two knobs: the gradient is a vector of partial derivatives.
w, b = 3.0, 0.0
print(f"gradient at (w={w:.1f}, b={b:.1f}) = [{grad_w(w, b):.1f}, {grad_b(w, b):.1f}]")
print("both negative -> increase both w and b to lower the loss")

package main

import "fmt"

// A one-weight "model": prediction = w * x. One example: x = 2, the right answer y = 10.
const x, y = 2.0, 10.0

func loss(w, b float64) float64 { e := w*x + b - y; return e * e }

// chain rule: d/dw (err^2) = 2*err * d(err)/dw = 2*err * x
func gradW(w, b float64) float64 { return 2 * (w*x + b - y) * x }
func gradB(w, b float64) float64 { return 2 * (w*x + b - y) * 1 }

// slope measured by nudging w a tiny bit both ways
func numeric(f func(float64) float64, w float64) float64 {
	h := 1e-5
	return (f(w+h) - f(w-h)) / (2 * h)
}

func main() {
	l := func(w float64) float64 { return loss(w, 0) }
	for _, w := range []float64{3, 5, 6} {
		fmt.Printf("w=%.1f  loss=%5.2f  grad by rule=%6.2f  grad by nudging=%6.2f\n", w, l(w), gradW(w, 0), numeric(l, w))
	}

	// The derivative predicts how the loss changes for a small step.
	w, step := 3.0, 0.1
	fmt.Printf("step w %.1f -> %.1f: predicted loss %.2f, actual %.2f\n", w, w+step, l(w)+gradW(w, 0)*step, l(w+step))

	// Two knobs: the gradient is a vector of partial derivatives.
	fmt.Printf("gradient at (w=%.1f, b=%.1f) = [%.1f, %.1f]\n", 3.0, 0.0, gradW(3, 0), gradB(3, 0))
	fmt.Println("both negative -> increase both w and b to lower the loss")
}

Output:

w=3.0  loss=16.00  grad by rule=-16.00  grad by nudging=-16.00
w=5.0  loss= 0.00  grad by rule=  0.00  grad by nudging=  0.00
w=6.0  loss= 4.00  grad by rule=  8.00  grad by nudging=  8.00
step w 3.0 -> 3.1: predicted loss 14.40, actual 14.44
gradient at (w=3.0, b=0.0) = [-16.0, -8.0]
both negative -> increase both w and b to lower the loss

Reading it:

• The rule (chain rule) and nudging agree at every w. This is called a gradient check, and it's how you test hand-written gradients in the Part 3 project.
• At w = 5 the loss is 0 and the slope is 0: the bottom. At w = 3 the slope is −16 (go up), at w = 6 it's +8 (go down).
• Stepping w by 0.1, the slope predicted the loss would drop to 14.40; it actually dropped to 14.44. The derivative is a straight-line guess, accurate for small steps and wrong for big ones. That's why training takes small steps (next lesson).
• With two knobs the gradient is [−16, −8]: both negative, so increase both. w gets the bigger push because its effect is multiplied by x = 2.

In production

• Nobody writes gradients by hand at scale. PyTorch and JAX record the chain of operations and run the chain rule for you (autograd). You still need this lesson to debug when gradients become 0, NaN or huge.
• Vanishing and exploding gradients. Multiplying many local slopes can shrink to almost 0 (slopes below 1) or blow up (slopes above 1). Deep networks fight this with careful activations and normalisation (Part 3).
• Nudging is the test, not the method. Numerical derivatives need two loss evaluations per weight, impossible for a billion weights, and they lose precision if h is too small. Use them only to check autograd or custom code.
• Non-smooth points. Functions like ReLU have a corner at 0 with no single slope. Frameworks just pick a value there, and it works fine in practice.

Key takeaway

The derivative tells you which way and how fast the loss changes when one weight moves; the gradient is that for every weight; the chain rule gets it by multiplying local slopes backward through the computation.

Practice

Take loss = (w·x − y)² with x = 3 and y = 6. Compute the gradient at w = 1 with the chain rule, then check it with the nudging formula (by hand or in code). Which way should w move, and where is the minimum?