Backpropagation derivation using Leibniz notation

This page presents a derivation/proof of backpropagation derivation using Leibniz notation. Leibniz notation is the most common notation for presenting backpropagation, but it is somewhat complicated due to its blurring of the function/value distinction and its reliance on functional relationships being implicit. Those who prefer function notation may wish to refer to backpropagation derivation using function notation instead of (or in addition to) this page.

Most of the notation on this page is borrowed from Michael Nielsen's book.

Theorem statement

Let $N$ be a neural network with $L$ layers and $n(l)$ be the number of neurons in layer $l$ for $l\in \{1,\dots ,L\}$. For $l\in \{2,\dots ,L\}$, $k\in \{1,\dots ,n(l-1)\}$, and $j\in \{1,\dots ,n(l)\}$ let $w_{jk}^l\in \mathbf{R}$ be the weight from the $k$th neuron in the $(l-1)$th layer to the $j$th neuron in the $l$th layer. Let $z_j^l=\sum _{k=1}^{n(l-1)}{w}_{jk}^l{a}_k^{l-1}+b_j^l$ and let $a_j^l=\sigma (z_j^l)$, where $\sigma :\mathbf{R}\to \mathbf{R}$ is the sigmoid function. Let $C=\frac{1}{2}\sum _{j=1}^{n(L)}(y_j-a_j^L{)}^2$ be a cost function. Then we can calculate the partial derivatives $\frac{\partial C}{\partial w_{jk}^l}$ and $\frac{\partial C}{\partial b_j^l}$ starting from the later layers. Specifically, we have

$$\frac{\partial C}{\partial w_{jk}^l}=(\sum _{i=1}^{n(l+1)}\frac{\partial C}{\partial a_i^{l+1}}{\sigma }^{\prime }(z_i^{l+1})w_{ij}^{l+1}){\sigma }^{\prime }(z_j^l)a_k^{l-1}$$

and

$$\frac{\partial C}{\partial b_j^l}=???$$

Proof

We induct on the layer number $l$, starting at $l=L$. For the base case, we have

$$\frac{\partial C}{\partial w_{jk}^L}=\frac{\partial C}{\partial a_j^L}\frac{\partial a_j^L}{\partial w_{jk}^L}=\frac{\partial C}{\partial a_j^L}{\sigma }^{\prime }(z_j^L)a_k^{L-1}$$

We also have

$$\frac{\partial C}{\partial a_j^L}=a_j^L-y_j$$

The cost function $C$ depends on $w_{jk}^l$ only through the activation of the $j$th neuron in the $l$th layer, i.e. on the value of $a_j^l$. Thus we can use the chain rule to expand:

$$\frac{\partial C}{\partial w_{jk}^l}=\frac{\partial C}{\partial a_j^l}\frac{\partial a_j^l}{\partial w_{jk}^l}$$

We know that $\frac{\partial a_j^l}{\partial w_{jk}^l}={\sigma }^{\prime }(z_j^l)a_k^{l-1}$ because $a_j^l=\sigma (z_j^l)=\sigma (\sum _{k=1}^{n(l-1)}{w}_{jk}^l{a}_k^{l-1}+b_j^l)$. We have used the chain rule again here.

In turn, $C$ depends on $a_j^l$ only through the activations of the $(l+1)$th layer. Thus we can write (using the chain rule once again):

$$\frac{\partial C}{\partial a_j^l}=\sum _{i=1}^{n(l+1)}\frac{\partial C}{\partial a_i^{l+1}}\frac{\partial a_i^{l+1}}{\partial a_j^l}$$

Backpropagation works recursively starting at the later layers. Since we are trying to compute $\frac{\partial C}{\partial a_j^l}$ for the $l$th layer, we can assume inductively that we have already computed $\frac{\partial C}{\partial a_i^{l+1}}$.

It remains to find $\frac{\partial a_i^{l+1}}{\partial a_j^l}$. But $a_i^{l+1}=\sigma (z_i^{l+1})=\sigma (\sum _{j=1}^{n(l)}{w}_{ij}^{l+1}{a}_j^l+b_i^{l+1})$ so we have

$$\frac{\partial a_i^{l+1}}{\partial a_j^l}={\sigma }^{\prime }(z_i^{l+1})w_{ij}^{l+1}$$

Putting all this together, we obtain

$$\begin{array}{rr}\frac{\partial C}{\partial w_{jk}^l}& =\frac{\partial C}{\partial a_j^l}\frac{\partial a_j^l}{\partial w_{jk}^l}\\ & =(\sum _{i=1}^{n(l+1)}\frac{\partial C}{\partial a_i^{l+1}}\frac{\partial a_i^{l+1}}{\partial a_j^l})\sigma \text{'}(z_j^l)a_k^{l-1}\\ & =(\sum _{i=1}^{n(l+1)}\frac{\partial C}{\partial a_i^{l+1}}{\sigma }^{\prime }(z_i^{l+1})w_{ij}^{l+1})\sigma \text{'}(z_j^l)a_k^{l-1}\end{array}$$

Let us verify that we can calculate the right-hand side. By induction hypothesis, we can calculate $\frac{\partial C}{\partial a_i^{l+1}}$. We calculate $z_i^{l+1}$, $z_j^l$, and $a_k^{l-1}$ during the forward pass through the network. Finally, $w_{ij}^{l+1}$ is just a weight in the network, so we already know its value.