Backpropagation derivation using Leibniz notation

This page presents a derivation/proof of backpropagation derivation using Leibniz notation. Leibniz notation is the most common notation for presenting backpropagation, but it is somewhat complicated due to its blurring of the function/value distinction and its reliance on functional relationships being implicit. Those who prefer function notation may wish to refer to backpropagation derivation using function notation instead of (or in addition to) this page.

Most of the notation on this page is borrowed from Michael Nielsen's book.

Throughout this page, let $n(l)$ be the number of neurons in the $l$th layer of the neural network.

Theorem.

Proof. The cost function $C$ depends on $w_{jk}^{l}$ only through the activation of the $j$th neuron in the $l$th layer, i.e. on the value of $a_{j}^{l}$. Thus we can use the chain rule to expand:

$${\frac {\partial C}{\partial w_{jk}^{l}}}={\frac {\partial C}{\partial a_{j}^{l}}}{\frac {\partial a_{j}^{l}}{\partial w_{jk}^{l}}}$$

We know that ${\frac {\partial a_{j}^{l}}{\partial w_{jk}^{l}}}=\sigma '(z_{j}^{l})a_{k}^{l-1}$ because $a_{j}^{l}=\sigma (z_{j}^{l})=\sigma \left(\sum _{k=1}^{n(l-1)}w_{jk}^{l}a_{k}^{l-1}+b_{j}^{l}\right)$. We have used the chain rule again here.

In turn, $C$ depends on $a_{j}^{l}$ only through the activations of the $(l+1)$th layer. Thus we can write (using the chain rule once again):

$${\frac {\partial C}{\partial a_{j}^{l}}}=\sum _{i=1}^{n(l+1)}{\frac {\partial C}{\partial a_{i}^{l+1}}}{\frac {\partial a_{i}^{l+1}}{\partial a_{j}^{l}}}$$

Backpropagation works recursively starting at the later layers. Since we are trying to compute ${\frac {\partial C}{\partial a_{j}^{l}}}$ for the $l$th layer, we can assume inductively that we have already computed ${\frac {\partial C}{\partial a_{i}^{l+1}}}$.

It remains to find ${\frac {\partial a_{i}^{l+1}}{\partial a_{j}^{l}}}$. But $a_{i}^{l+1}=\sigma (z_{i}^{l+1})=\sigma \left(\sum _{j=1}^{n(l)}w_{ij}^{l+1}a_{j}^{l}+b_{i}^{l+1}\right)$ so we have

$${\frac {\partial a_{i}^{l+1}}{\partial a_{j}^{l}}}=\sigma '(z_{i}^{l+1})w_{ij}^{l+1}$$

Putting all this together, we obtain

$${\begin{aligned}{\frac {\partial C}{\partial w_{jk}^{l}}}&={\frac {\partial C}{\partial a_{j}^{l}}}{\frac {\partial a_{j}^{l}}{\partial w_{jk}^{l}}}\\&=\left(\sum _{i=1}^{n(l+1)}{\frac {\partial C}{\partial a_{i}^{l+1}}}{\frac {\partial a_{i}^{l+1}}{\partial a_{j}^{l}}}\right)\sigma '(z_{j}^{l})a_{k}^{l-1}\\&=\left(\sum _{i=1}^{n(l+1)}{\frac {\partial C}{\partial a_{i}^{l+1}}}\sigma '(z_{i}^{l+1})w_{ij}^{l+1}\right)\sigma '(z_{j}^{l})a_{k}^{l-1}\end{aligned}}$$

Let us verify that we can calculate the right-hand side. By induction hypothesis, we can calculate ${\frac {\partial C}{\partial a_{i}^{l+1}}}$. We calculate $z_{i}^{l+1}$, $z_{j}^{l}$, and $a_{k}^{l-1}$ during the forward pass through the network. Finally, $w_{ij}^{l+1}$ is just a weight in the network, so we already know its value.