Backpropagation derivation using Leibniz notation

This page presents a derivation/proof of backpropagation derivation using Leibniz notation. Leibniz notation is the most common notation for presenting backpropagation, but it is somewhat complicated due to its blurring of the function/value distinction and its reliance on functional relationships being implicit. Those who prefer function notation may wish to refer to backpropagation derivation using function notation instead of (or in addition to) this page.

Most of the notation on this page is borrowed from Michael Nielsen's book.

Theorem statement

Let $N$ be a neural network with $L$ layers and $n(l)$ be the number of neurons in layer $l$ for $l\in \{1,\ldots ,L\}$. For $l\in \{2,\ldots ,L\}$, $k\in \{1,\ldots ,n(l-1)\}$, and $j\in \{1,\ldots ,n(l)\}$ let $w_{jk}^{l}\in \mathbf {R}$ be the weight from the $k$th neuron in the $(l-1)$th layer to the $j$th neuron in the $l$th layer. Let $z_{j}^{l}=\sum _{k=1}^{n(l-1)}w_{jk}^{l}a_{k}^{l-1}+b_{j}^{l}$ and let $a_{j}^{l}=\sigma (z_{j}^{l})$, where $\sigma :\mathbf {R} \to \mathbf {R}$ is the sigmoid function. Let $C={\frac {1}{2}}\sum _{j=1}^{n(L)}(y_{j}-a_{j}^{L})^{2}$ be a cost function. Then we can calculate the partial derivatives ${\frac {\partial C}{\partial w_{jk}^{l}}}$ and ${\frac {\partial C}{\partial b_{j}^{l}}}$ starting from the later layers. Specifically, we have

$${\frac {\partial C}{\partial w_{jk}^{l}}}=\left(\sum _{i=1}^{n(l+1)}{\frac {\partial C}{\partial a_{i}^{l+1}}}\sigma '(z_{i}^{l+1})w_{ij}^{l+1}\right)\sigma '(z_{j}^{l})a_{k}^{l-1}$$

and

$${\frac {\partial C}{\partial b_{j}^{l}}}=???$$

Proof

We induct on the layer number $l$, starting at $l=L$. For the base case, we have

$${\frac {\partial C}{\partial w_{jk}^{L}}}={\frac {\partial C}{\partial a_{j}^{L}}}{\frac {\partial a_{j}^{L}}{\partial w_{jk}^{L}}}={\frac {\partial C}{\partial a_{j}^{L}}}\sigma '(z_{j}^{L})a_{k}^{L-1}$$

We also have

$${\frac {\partial C}{\partial a_{j}^{L}}}=a_{j}^{L}-y_{j}$$

The cost function $C$ depends on $w_{jk}^{l}$ only through the activation of the $j$th neuron in the $l$th layer, i.e. on the value of $a_{j}^{l}$. Thus we can use the chain rule to expand:

$${\frac {\partial C}{\partial w_{jk}^{l}}}={\frac {\partial C}{\partial a_{j}^{l}}}{\frac {\partial a_{j}^{l}}{\partial w_{jk}^{l}}}$$

We know that ${\frac {\partial a_{j}^{l}}{\partial w_{jk}^{l}}}=\sigma '(z_{j}^{l})a_{k}^{l-1}$ because $a_{j}^{l}=\sigma (z_{j}^{l})=\sigma \left(\sum _{k=1}^{n(l-1)}w_{jk}^{l}a_{k}^{l-1}+b_{j}^{l}\right)$. We have used the chain rule again here.

In turn, $C$ depends on $a_{j}^{l}$ only through the activations of the $(l+1)$th layer. Thus we can write (using the chain rule once again):

$${\frac {\partial C}{\partial a_{j}^{l}}}=\sum _{i=1}^{n(l+1)}{\frac {\partial C}{\partial a_{i}^{l+1}}}{\frac {\partial a_{i}^{l+1}}{\partial a_{j}^{l}}}$$

Backpropagation works recursively starting at the later layers. Since we are trying to compute ${\frac {\partial C}{\partial a_{j}^{l}}}$ for the $l$th layer, we can assume inductively that we have already computed ${\frac {\partial C}{\partial a_{i}^{l+1}}}$.

It remains to find ${\frac {\partial a_{i}^{l+1}}{\partial a_{j}^{l}}}$. But $a_{i}^{l+1}=\sigma (z_{i}^{l+1})=\sigma \left(\sum _{j=1}^{n(l)}w_{ij}^{l+1}a_{j}^{l}+b_{i}^{l+1}\right)$ so we have

$${\frac {\partial a_{i}^{l+1}}{\partial a_{j}^{l}}}=\sigma '(z_{i}^{l+1})w_{ij}^{l+1}$$

Putting all this together, we obtain

$${\begin{aligned}{\frac {\partial C}{\partial w_{jk}^{l}}}&={\frac {\partial C}{\partial a_{j}^{l}}}{\frac {\partial a_{j}^{l}}{\partial w_{jk}^{l}}}\\&=\left(\sum _{i=1}^{n(l+1)}{\frac {\partial C}{\partial a_{i}^{l+1}}}{\frac {\partial a_{i}^{l+1}}{\partial a_{j}^{l}}}\right)\sigma '(z_{j}^{l})a_{k}^{l-1}\\&=\left(\sum _{i=1}^{n(l+1)}{\frac {\partial C}{\partial a_{i}^{l+1}}}\sigma '(z_{i}^{l+1})w_{ij}^{l+1}\right)\sigma '(z_{j}^{l})a_{k}^{l-1}\end{aligned}}$$

Let us verify that we can calculate the right-hand side. By induction hypothesis, we can calculate ${\frac {\partial C}{\partial a_{i}^{l+1}}}$. We calculate $z_{i}^{l+1}$, $z_{j}^{l}$, and $a_{k}^{l-1}$ during the forward pass through the network. Finally, $w_{ij}^{l+1}$ is just a weight in the network, so we already know its value.