Derivative of a quadratic form

Let $A\in {\mathcal{M}}_{n,n}(\mathbf{R})$ be an $n$ by $n$ real-valued matrix, and let $f:{\mathbf{R}}^n\to \mathbf{R}$ be defined by $f(x)=x^{\mathrm{T}}Ax$. On this page, we calculate the derivative of $f$.

Using the definition of the derivative

The derivative is the linear transformation $L$ such that:

$\underset{x\to x_0;x\ne x_0}{lim}\frac{|f(x)-(f(x_0)+L(x-x_0))|}{|x-x_0|}=0$

Using our function, this is:

$\underset{x\to x_0;x\ne x_0}{lim}\frac{|x^{\mathrm{T}}Ax-x_0^{\mathrm{T}}A{x}_0-L(x-x_0)|}{|x-x_0|}=0$

Defining $h=x-x_0$, we have $x=x_0+h$ and

$\frac{|(x_0+h{)}^{\mathrm{T}}A(x_0+h)-x_0^{\mathrm{T}}A{x}_0-L(h)|}{\left|h\right|}$

Focusing on the subexpression $(x_0+h{)}^{\mathrm{T}}A(x_0+h)$, since $A$ is a matrix, it is a linear transformation, so we obtain $(x_0+h{)}^{\mathrm{T}}(A{x}_0+Ah)$. Since the transpose of a sum is the sum of the transposes, we have $(x_0^{\mathrm{T}}+h^{\mathrm{T}})(A{x}_0+Ah)$. Now using linearity we have $x_0^{\mathrm{T}}A{x}_0+h^{\mathrm{T}}A{x}_0+x_0^{\mathrm{T}}Ah+h^{\mathrm{T}}Ah$.

Now the fraction is

$\frac{|x_0^{\mathrm{T}}A{x}_0+h^{\mathrm{T}}A{x}_0+x_0^{\mathrm{T}}Ah+h^{\mathrm{T}}Ah-x_0^{\mathrm{T}}A{x}_0-L(h)|}{\left|h\right|}=\frac{|h^{\mathrm{T}}A{x}_0+x_0^{\mathrm{T}}Ah+h^{\mathrm{T}}Ah-L(h)|}{\left|h\right|}$

Focusing on $h^{\mathrm{T}}A{x}_0$, it is a real number so taking the transpose leaves it unchanged: $h^{\mathrm{T}}A{x}_0=(h^{\mathrm{T}}A{x}_0{)}^{\mathrm{T}}=x_0^{\mathrm{T}}{A}^{\mathrm{T}}h$.

Now the fraction is

$\frac{|x_0^{\mathrm{T}}{A}^{\mathrm{T}}h+x_0^{\mathrm{T}}Ah+h^{\mathrm{T}}Ah-L(h)|}{\left|h\right|}=\frac{|x_0^{\mathrm{T}}(A^{\mathrm{T}}+A)h+h^{\mathrm{T}}Ah-L(h)|}{\left|h\right|}$

In the numerator, $h^{\mathrm{T}}Ah$ is a higher order term that will disappear when taking the limit, so the linear transformation we are looking for must be $L(h)=x_0^{\mathrm{T}}(A^{\mathrm{T}}+A)h$. Since $A$ is symmetric, we have $A^{\mathrm{T}}+A=2A$ and $L(h)=2x_0^{\mathrm{T}}Ah$.