Backpropagation derivation using Leibniz notation

This page presents a derivation/proof of backpropagation derivation using Leibniz notation. Leibniz notation is the most common notation for presenting backpropagation, but it is somewhat complicated due to its blurring of the function/value distinction and its reliance on functional relationships being implicit. Those who prefer function notation may wish to refer to backpropagation derivation using function notation instead of (or in addition to) this page.

Most of the notation on this page is borrowed from Michael Nielsen's book.

Theorem. Let $N$ be a neural network with $L$ layers and $n(l)$ be the number of neurons in layer $l$ for $l\in \{1,\dots ,L\}$. a cost function $C=\frac{1}{2}\sum _{j=1}^n(L)</div{>}^{″}Proof{.}^{″}Thecostfunction<math>C$ depends on $w_{jk}^l$ only through the activation of the $j$th neuron in the $l$th layer, i.e. on the value of $a_j^l$. Thus we can use the chain rule to expand:

$$\frac{\partial C}{\partial w_{jk}^l}=\frac{\partial C}{\partial a_j^l}\frac{\partial a_j^l}{\partial w_{jk}^l}$$

We know that $\frac{\partial a_j^l}{\partial w_{jk}^l}={\sigma }^{\prime }(z_j^l)a_k^{l-1}$ because $a_j^l=\sigma (z_j^l)=\sigma (\sum _{k=1}^{n(l-1)}{w}_{jk}^l{a}_k^{l-1}+b_j^l)$. We have used the chain rule again here.

In turn, $C$ depends on $a_j^l$ only through the activations of the $(l+1)$th layer. Thus we can write (using the chain rule once again):

$$\frac{\partial C}{\partial a_j^l}=\sum _{i=1}^{n(l+1)}\frac{\partial C}{\partial a_i^{l+1}}\frac{\partial a_i^{l+1}}{\partial a_j^l}$$

Backpropagation works recursively starting at the later layers. Since we are trying to compute $\frac{\partial C}{\partial a_j^l}$ for the $l$th layer, we can assume inductively that we have already computed $\frac{\partial C}{\partial a_i^{l+1}}$.

It remains to find $\frac{\partial a_i^{l+1}}{\partial a_j^l}$. But $a_i^{l+1}=\sigma (z_i^{l+1})=\sigma (\sum _{j=1}^{n(l)}{w}_{ij}^{l+1}{a}_j^l+b_i^{l+1})$ so we have

$$\frac{\partial a_i^{l+1}}{\partial a_j^l}={\sigma }^{\prime }(z_i^{l+1})w_{ij}^{l+1}$$

Putting all this together, we obtain

$$\begin{array}{rr}\frac{\partial C}{\partial w_{jk}^l}& =\frac{\partial C}{\partial a_j^l}\frac{\partial a_j^l}{\partial w_{jk}^l}\\ & =(\sum _{i=1}^{n(l+1)}\frac{\partial C}{\partial a_i^{l+1}}\frac{\partial a_i^{l+1}}{\partial a_j^l})\sigma \text{'}(z_j^l)a_k^{l-1}\\ & =(\sum _{i=1}^{n(l+1)}\frac{\partial C}{\partial a_i^{l+1}}{\sigma }^{\prime }(z_i^{l+1})w_{ij}^{l+1})\sigma \text{'}(z_j^l)a_k^{l-1}\end{array}$$

Let us verify that we can calculate the right-hand side. By induction hypothesis, we can calculate $\frac{\partial C}{\partial a_i^{l+1}}$. We calculate $z_i^{l+1}$, $z_j^l$, and $a_k^{l-1}$ during the forward pass through the network. Finally, $w_{ij}^{l+1}$ is just a weight in the network, so we already know its value.

References