Derivative of a quadratic form

Let $A\in {\mathcal{M}}_{n,n}(\mathbf{R})$ be an $n$ by $n$ real-valued matrix, and let $f:{\mathbf{R}}^n\to \mathbf{R}$ be defined by $f(x)=x^{\mathrm{T}}Ax$. On this page, we calculate the derivative of $f$.

Straightforward method

This method is the most straightforward, and involves breaking apart the matrix and vector into components and performing the differentiation. While straightforward, it appears messy due to the indices involved.

Let $A=(a_{ki})$ and $x=(x_1,\dots ,x_n)$.

We expand

$x^{\mathrm{T}}Ax=x^{\mathrm{T}}\left(\begin{array}{c}\sum _{i=1}^n{a}_{1i}{x}_i\\ ⋮\\ \sum _{i=1}^n{a}_{ni}{x}_i\end{array}\right)=\sum _{k=1}^n{x}_k\sum _{i=1}^n{a}_{ki}{x}_i$

Now we find the partial derivative of the above with respect to $x_j$. To distinguish the constants from the variable, it makes sense to split the sum:

$\sum _{k=1}^n{x}_k\sum _{i=1}^n{a}_{ki}{x}_i=x_j\sum _{i=1}^n{a}_{ji}{x}_i+{\sum }_{k\ne j}{x}_k\sum _{i=1}^n{a}_{ki}{x}_i=x_j(a_{jj}{x}_j+{\sum }_{i\ne j}{a}_{ji}{x}_i)+{\sum }_{k\ne j}{x}_k(a_{kj}{x}_j+{\sum }_{i\ne j}{a}_{ki}{x}_i)$

Now distributing we have

$a_{jj}{x}_j^2+({\sum }_{i\ne j}{a}_{ji}{x}_i)x_j+{\sum }_{k\ne j}(a_{kj}{x}_k{x}_j+x_k{\sum }_{i\ne j}{a}_{ki}{x}_i)=a_{jj}{x}_j^2+({\sum }_{i\ne j}{a}_{ji}{x}_i)x_j+({\sum }_{k\ne j}{a}_{kj}{x}_k)x_j+{\sum }_{k\ne j}{x}_k{\sum }_{i\ne j}{a}_{ki}{x}_i$

It is now easy to do the differentiation. We have

$2a_{jj}{x}_j+{\sum }_{i\ne j}{a}_{ji}{x}_i+{\sum }_{k\ne j}{a}_{kj}{x}_k$

Since the matrix is symmetric, $a_{kj}=a_{jk}$ so ${\sum }_{k\ne j}{a}_{kj}{x}_k={\sum }_{k\ne j}{a}_{jk}{x}_k={\sum }_{i\ne j}{a}_{ji}{x}_i$. The final equality follows because $k$ is just an indexing variable and we are free to rename it. But now the derivative becomes

$2a_{jj}{x}_j+2{\sum }_{i\ne j}{a}_{ji}{x}_i=2\sum _{i=1}^n{a}_{ji}{x}_i$

Using the definition of the derivative

This is an expanded version of the answer at [1].

The derivative is the linear transformation $L$ such that:

$\underset{x\to x_0;x\ne x_0}{lim}\frac{|f(x)-(f(x_0)+L(x-x_0))|}{|x-x_0|}=0$

Using our function, this is:

$\underset{x\to x_0;x\ne x_0}{lim}\frac{|x^{\mathrm{T}}Ax-x_0^{\mathrm{T}}A{x}_0-L(x-x_0)|}{|x-x_0|}=0$

Defining $h=x-x_0$, we have $x=x_0+h$ and

$\frac{|(x_0+h{)}^{\mathrm{T}}A(x_0+h)-x_0^{\mathrm{T}}A{x}_0-L(h)|}{\left|h\right|}$

Focusing on the subexpression $(x_0+h{)}^{\mathrm{T}}A(x_0+h)$, since $A$ is a matrix, it is a linear transformation, so we obtain $(x_0+h{)}^{\mathrm{T}}(A{x}_0+Ah)$. Since the transpose of a sum is the sum of the transposes, we have $(x_0^{\mathrm{T}}+h^{\mathrm{T}})(A{x}_0+Ah)$. Now using linearity we have $x_0^{\mathrm{T}}A{x}_0+h^{\mathrm{T}}A{x}_0+x_0^{\mathrm{T}}Ah+h^{\mathrm{T}}Ah$.

Now the fraction is

$\frac{|x_0^{\mathrm{T}}A{x}_0+h^{\mathrm{T}}A{x}_0+x_0^{\mathrm{T}}Ah+h^{\mathrm{T}}Ah-x_0^{\mathrm{T}}A{x}_0-L(h)|}{\left|h\right|}=\frac{|h^{\mathrm{T}}A{x}_0+x_0^{\mathrm{T}}Ah+h^{\mathrm{T}}Ah-L(h)|}{\left|h\right|}$

Focusing on $h^{\mathrm{T}}A{x}_0$, it is a real number so taking the transpose leaves it unchanged: $h^{\mathrm{T}}A{x}_0=(h^{\mathrm{T}}A{x}_0{)}^{\mathrm{T}}=x_0^{\mathrm{T}}{A}^{\mathrm{T}}h$.

Now the fraction is

$\frac{|x_0^{\mathrm{T}}{A}^{\mathrm{T}}h+x_0^{\mathrm{T}}Ah+h^{\mathrm{T}}Ah-L(h)|}{\left|h\right|}=\frac{|x_0^{\mathrm{T}}(A^{\mathrm{T}}+A)h+h^{\mathrm{T}}Ah-L(h)|}{\left|h\right|}$

In the numerator, $h^{\mathrm{T}}Ah$ is a higher order term that will disappear when taking the limit, so the linear transformation we are looking for must be $L(h)=x_0^{\mathrm{T}}(A^{\mathrm{T}}+A)h$. Since $A$ is symmetric, we have $A^{\mathrm{T}}+A=2A$ and $L(h)=2x_0^{\mathrm{T}}Ah$.