Backpropagation derivation using Leibniz notation

Throughout this page, let $n(l)$ be the number of neurons in the $l$th layer of the neural network.

The cost function $C$ depends on $w_{jk}^l$ only through the activation of the $j$th neuron in the $l$th layer, i.e. on the value of $a_j^l$. Thus we can use the chain rule to expand:

$$\frac{\partial C}{\partial w_{jk}^l}=\frac{\partial C}{\partial a_j^l}\frac{\partial a_j^l}{\partial w_{jk}^l}$$

We know that $\frac{\partial a_j^l}{\partial w_{jk}^l}={\sigma }^{\prime }(z_j^l)a_k^{l-1}$ because $a_j^l=\sigma (z_j^l)=\sigma (\sum _{k=1}^{n(l-1)}{w}_{jk}^l{a}_k^{l-1}+b_j^l)$. We have used the chain rule again here.

In turn, $C$ depends on $a_j^l$ only through the activations of the $(l+1)$th layer. Thus we can write (using the chain rule once again):

$$\frac{\partial C}{\partial a_j^l}=\sum _{i=1}^{n(l+1)}\frac{\partial C}{\partial a_i^{l+1}}\frac{\partial a_i^{l+1}}{\partial a_j^l}$$

Backpropagation works recursively starting at the later layers. Since we are trying to compute $\frac{\partial C}{\partial a_j^l}$ for the $l$th layer, we can assume inductively that we have already computed $\frac{\partial C}{\partial a_i^{l+1}}$.

It remains to find $\frac{\partial a_i^{l+1}}{\partial a_j^l}$. But $a_i^{l+1}=\sigma (z_i^{l+1})=\sigma ({\sum }_j{w}_{ij}^{l+1}{a}_j^l+b_i^{l+1})$ so we have

$$\frac{\partial a_i^{l+1}}{\partial a_j^l}={\sigma }^{\prime }(z_i^{l+1})w_{ij}^{l+1}$$

Putting all this together, we obtain

$$\begin{array}{rr}\frac{\partial C}{\partial w_{jk}^l}& =\frac{\partial C}{\partial a_j^l}\frac{\partial a_j^l}{\partial w_{jk}^l}\\ & =(\sum _{i=1}^{n(l+1)}\frac{\partial C}{\partial a_i^{l+1}}\frac{\partial a_i^{l+1}}{\partial a_j^l})\sigma \text{'}(z_j^l)a_k^{l-1}\\ & =(\sum _{i=1}^{n(l+1)}\frac{\partial C}{\partial a_i^{l+1}}{\sigma }^{\prime }(z_i^{l+1})w_{ij}^{l+1})\sigma \text{'}(z_j^l)a_k^{l-1}\end{array}$$