Derivative of a quadratic form

Let $A\in {\mathcal{M}}_{n,n}(\mathbf{R})$ be an $n$ by $n$ real-valued matrix, and let $f:{\mathbf{R}}^n\to \mathbf{R}$ be defined by $f(x)=x^{\mathrm{T}}Ax$. On this page, we calculate the derivative of $f$.

Straightforward method

Let $A=(a_{ki})$ and $x=(x_1,\dots ,x_n)$.

We expand

$x^{\mathrm{T}}Ax=x^{\mathrm{T}}\left(\begin{array}{c}\sum _{i=1}^n{a}_{1i}{x}_i\\ ⋮\\ \sum _{i=1}^n{a}_{ni}{x}_i\end{array}\right)=\sum _{k=1}^n{x}_k\sum _{i=1}^n{a}_{ki}{x}_i$

Now we find the partial derivative of the above with respect to $x_j$. To distinguish the constants from the variable, it makes sense to split the sum:

$\sum _{k=1}^n{x}_k\sum _{i=1}^n{a}_{ki}{x}_i=x_j\sum _{i=1}^n{a}_{ji}{x}_i+{\sum }_{k\ne j}{x}_k\sum _{i=1}^n{a}_{ki}{x}_i=x_j(a_{jj}{x}_j+{\sum }_{i\ne j}{a}_{ji}{x}_i)+{\sum }_{k\ne j}{x}_k(a_{kj}{x}_j+{\sum }_{i\ne j}{a}_{ki}{x}_i)$

Using the definition of the derivative

This is an expanded version of the answer at [1].

The derivative is the linear transformation $L$ such that:

$\underset{x\to x_0;x\ne x_0}{lim}\frac{|f(x)-(f(x_0)+L(x-x_0))|}{|x-x_0|}=0$

Using our function, this is:

$\underset{x\to x_0;x\ne x_0}{lim}\frac{|x^{\mathrm{T}}Ax-x_0^{\mathrm{T}}A{x}_0-L(x-x_0)|}{|x-x_0|}=0$

Defining $h=x-x_0$, we have $x=x_0+h$ and

$\frac{|(x_0+h{)}^{\mathrm{T}}A(x_0+h)-x_0^{\mathrm{T}}A{x}_0-L(h)|}{\left|h\right|}$

Focusing on the subexpression $(x_0+h{)}^{\mathrm{T}}A(x_0+h)$, since $A$ is a matrix, it is a linear transformation, so we obtain $(x_0+h{)}^{\mathrm{T}}(A{x}_0+Ah)$. Since the transpose of a sum is the sum of the transposes, we have $(x_0^{\mathrm{T}}+h^{\mathrm{T}})(A{x}_0+Ah)$. Now using linearity we have $x_0^{\mathrm{T}}A{x}_0+h^{\mathrm{T}}A{x}_0+x_0^{\mathrm{T}}Ah+h^{\mathrm{T}}Ah$.

Now the fraction is

$\frac{|x_0^{\mathrm{T}}A{x}_0+h^{\mathrm{T}}A{x}_0+x_0^{\mathrm{T}}Ah+h^{\mathrm{T}}Ah-x_0^{\mathrm{T}}A{x}_0-L(h)|}{\left|h\right|}=\frac{|h^{\mathrm{T}}A{x}_0+x_0^{\mathrm{T}}Ah+h^{\mathrm{T}}Ah-L(h)|}{\left|h\right|}$

Focusing on $h^{\mathrm{T}}A{x}_0$, it is a real number so taking the transpose leaves it unchanged: $h^{\mathrm{T}}A{x}_0=(h^{\mathrm{T}}A{x}_0{)}^{\mathrm{T}}=x_0^{\mathrm{T}}{A}^{\mathrm{T}}h$.

Now the fraction is

$\frac{|x_0^{\mathrm{T}}{A}^{\mathrm{T}}h+x_0^{\mathrm{T}}Ah+h^{\mathrm{T}}Ah-L(h)|}{\left|h\right|}=\frac{|x_0^{\mathrm{T}}(A^{\mathrm{T}}+A)h+h^{\mathrm{T}}Ah-L(h)|}{\left|h\right|}$

In the numerator, $h^{\mathrm{T}}Ah$ is a higher order term that will disappear when taking the limit, so the linear transformation we are looking for must be $L(h)=x_0^{\mathrm{T}}(A^{\mathrm{T}}+A)h$. Since $A$ is symmetric, we have $A^{\mathrm{T}}+A=2A$ and $L(h)=2x_0^{\mathrm{T}}Ah$.