Backpropagation derivation using Leibniz notation: Difference between revisions

Revision as of 22:28, 8 November 2018

The cost function $C$ depends on $w_{j k}^{l}$ only through the activation of the $j$ th neuron in the $l$ th layer, i.e. on the value of $a_{j}^{l}$ . Thus we can use the chain rule to expand:

$\frac{\partial C}{\partial w_{j k}^{l}} = \frac{\partial C}{\partial a_{j}^{l}} \frac{\partial a_{j}^{l}}{\partial w_{j k}^{l}}$

We know that $\frac{\partial a_{j}^{l}}{\partial w_{j k}^{l}} = σ^{'} (z_{j}^{l}) a_{k}^{l - 1}$ because $a_{j}^{l} = σ (z_{j}^{l}) = σ (\sum_{k} w_{j k}^{l} a_{k}^{l - 1} + b_{j}^{l})$ . We have used the chain rule again here.

In turn, $C$ depends on $a_{j}^{l}$ only through the activations of the $(l + 1)$ th layer. Thus we can write:

$\frac{\partial C}{\partial a_{j}^{l}} = \sum_{i \in {1, \dots, n (l + 1)}} \frac{\partial C}{\partial a_{i}^{l + 1}} \frac{\partial a_{i}^{l + 1}}{a_{j}^{l}}$

where $n (l + 1)$ is the number of neurons in the $(l + 1)$ th layer.

@@ Line 5: / Line 5: @@
 We know that <math>\frac{\partial a^l_j}{\partial w^l_{jk}} = \sigma'(z^l_j)a^{l-1}_k</math> because <math>a^l_j = \sigma(z^l_j) = \sigma\left(\sum_k w^l_{jk}a^{l-1}_k + b^l_j\right)</math>. We have used the chain rule again here.
-In turn, <math>C</math> depends on <math>a^l_j</math> only through the activations of the <math>(l+1)</math>th layer.
+In turn, <math>C</math> depends on <math>a^l_j</math> only through the activations of the <math>(l+1)</math>th layer. Thus we can write:
+<math>\frac{\partial C}{\partial a^l_j} = \sum_{i \in \{1,\ldots,n(l+1)\}} \frac{\partial C}{\partial a^{l+1}_i} \frac{\partial a^{l+1}_i}{a^l_j}</math>
+where <math>n(l+1)</math> is the number of neurons in the <math>(l+1)</math>th layer.