User:IssaRice/Chain rule proofs

a differentiable function looks locally like a linear transformation. if you compose two differentiable functions, then it seems pretty obvious that locally, that composed map also looks like a linear transformation. but which linear transformation? well, you first apply the linear transformation that locally approximates the inner function. then you land in some target space, and you find what the outer function locally looks like at the place you land. and that's all the chain rule says.

So Folland actually says this on page 109: "Since the product of two matrices gives the composition of the linear transformations defined by those matrices, the chain rule just says that the linear approximation of a composition is the composition of the linear approximations." But... putting this short paragraph away after a proof, and only after you've already discussed two versions of the chain rule in previous chapters, is in my opinion not trying hard enough to communicate the central message. This should be in bold flashing letters at the TOP of the FIRST discussion of the chain rule.

Using Newton's approximation

Main idea

The main idea of using Newton's approximation to prove the chain rule is that since f is differentiable at $x_{0}$ we have the approximation $f(x)\approx f(x_{0})+f'(x_{0})(x-x_{0})$ when $x$ is near $x_{0}$. Similarly since g is differentiable at $f(x_{0})$ we have the approximation $g(y)\approx g(f(x_{0}))+g'(f(x_{0}))(y-f(x_{0}))$ when $y$ is near $f(x_{0})$. Since f is differentiable at $x_{0}$, it is continuous there also, so we know that $f(x)$ is near $f(x_{0})$ whenever $x$ is near $x_{0}$. This allows us to substitute $f(x)$ into $y$ whenever $x$ is near $x_{0}$. So we get

$${\begin{aligned}g(f(x))&\approx g(f(x_{0}))+g'(f(x_{0}))(f(x)-f(x_{0}))\\&\approx g(f(x_{0}))+g'(f(x_{0}))(f'(x_{0})(x-x_{0}))\end{aligned}}$$

Thus we get $g\circ f(x)\approx g\circ f(x_{0})+g'(f(x_{0}))f'(x_{0})(x-x_{0})$, which is what the chain rule says.

Slightly more formally:

$f(x)=f(x_{0})+f'(x_{0})(x-x_{0})+o(x-x_{0})$ when $x$ is near $x_{0}$

$g(y)=g(f(x_{0}))+g'(f(x_{0}))(y-f(x_{0}))+o(y-f(x_{0}))$ when $y$ is near $f(x_{0})$

$f$ is continuous at $x_{0}$ so $f(x)$ is near $f(x_{0})$ whenever $x$ is near $x_{0}$.

Thus if $x$ is near $x_{0}$

$${\begin{aligned}g(f(x))&=g(f(x_{0}))+g'(f(x_{0}))(f'(x_{0})(x-x_{0})+o(x-x_{0}))+o(f(x)-f(x_{0}))\\&=g(f(x_{0}))+g'(f(x_{0}))f'(x_{0})(x-x_{0})+g'(f(x_{0}))o(x-x_{0})+o(f(x)-f(x_{0}))\end{aligned}}$$

to complete the proof, we just need to show that the error $g'(f(x_{0}))o(x-x_{0})+o(f(x)-f(x_{0}))$ is $o(x-x_{0})$. The former term clearly is. The latter term is $o(f'(x_{0})(x-x_{0})+o(x-x_{0}))$ so it is as well.

Proof

We want to show $g\circ f$ is differentiable at $x_{0}$ with derivative $L:=g'(f(x_{0}))f'(x_{0})$. By Newton's approximation, this is equivalent to showing that for every $\epsilon >0$ there exists $\delta >0$ such that

$$|g\circ f(x)-(g\circ f(x_{0})+L(x-x_{0}))|\leq \epsilon |x-x_{0}|$$

whenever $|x-x_{0}|\leq \delta$. So let $\epsilon >0$.

Now we do some algebraic manipulation. Write

$$g(y)=g(y_{0})+g'(y_{0})(y-y_{0})+E_{g}(y,y_{0})$$

where $E_{g}(y,y_{0}):=g(y)-(g(y_{0})+g'(y_{0})(y-y_{0}))$. This holds for every $y\in Y$. Since $f(x)\in Y$ we thus have

$$g(f(x))=g(f(x_{0}))+g'(f(x_{0}))(f(x)-f(x_{0}))+E_{g}(f(x),f(x_{0}))$$

Similarly write

$$f(x)=f(x_{0})+f'(x_{0})(x-x_{0})+E_{f}(x,x_{0})$$

where $E_{f}(x,x_{0}):=f(x)-(f(x_{0})+f'(x_{0})(x-x_{0}))$.

Substituting the expression for $f(x)$ in the expression for $g(f(x))$ we get

$${\begin{aligned}g(f(x))&=g(f(x_{0}))+g'(f(x_{0}))(f'(x_{0})(x-x_{0})+E_{f}(x,x_{0}))+E_{g}(f(x),f(x_{0}))\\&=g(f(x_{0}))+g'(f(x_{0}))f'(x_{0})(x-x_{0})+g'(f(x_{0}))E_{f}(x,x_{0})+E_{g}(f(x),f(x_{0}))\end{aligned}}$$

we can rewrite this as $g\circ f(x)-(g\circ f(x_{0})+L(x-x_{0}))=g'(f(x_{0}))E_{f}(x,x_{0})+E_{g}(f(x),f(x_{0}))$

Thus our goal now is to show $|g'(f(x_{0}))E_{f}(x,x_{0})+E_{g}(f(x),f(x_{0}))|\leq \epsilon |x-x_{0}|$.

But by the triangle inequality it suffices to show $|g'(f(x_{0}))E_{f}(x,x_{0})|+|E_{g}(f(x),f(x_{0}))|\leq \epsilon |x-x_{0}|$.

$|g'(f(x_{0}))E_{f}(x,x_{0})|\leq |g'(f(x_{0}))|\epsilon _{1}|x-x_{0}|$ where we are free to choose $\epsilon _{1}$.

To get the bound for $|E_{g}(f(x),f(x_{0}))|$ (using Newton's approximation), we need to make sure $|f(x)-f(x_{0})|$ is small. But by continuity of $f$ at $x_{0}$ we can do this.

$${\begin{aligned}|E_{g}(f(x),f(x_{0}))|&\leq \epsilon _{2}|f(x)-f(x_{0})|\\&=\epsilon _{2}|f'(x_{0})(x-x_{0})+E_{f}(x,x_{0})|\\&\leq \epsilon _{2}(|f'(x_{0})||x-x_{0}|+|x-x_{0}|)\\&=\epsilon _{2}(|f'(x_{0})|+1)|x-x_{0}|\end{aligned}}$$

where again we are free to choose $\epsilon _{2}$.

TODO: can we do this same proof but without using the error term notation?

TODO: somehow Folland does this without explicitly using continuity of f; i need to understand if he's using it implicitly somehow or he's actually proving it when bounding $|\mathbf {h} |$ using $|u|$

old proof

Since $g$ is differentiable at $y_{0}$, we know $g'(y_{0})$ is a real number, and we can write

$$g(y)=g(y_{0})+g'(y_{0})(y-y_{0})+[g(y)-(g(y_{0})+g'(y_{0})(y-y_{0}))]$$

(there is no magic: the terms just cancel out)

If we define $E_{g}(y,y_{0}):=g(y)-(g(y_{0})+g'(y_{0})(y-y_{0}))$ we can write

$$g(y)=g(y_{0})+g'(f(x_{0}))(y-y_{0})+E_{g}(y,y_{0})$$

Newton's approximation says that $|E_{g}(y,y_{0})|\leq \epsilon |y-y_{0}|$ as long as $|y-y_{0}|\leq \delta$.

Since $f$ is differentiable at $x_{0}$, we know that it must be continuous at $x_{0}$. This means we can keep $|f(x)-y_{0}|\leq \delta$ as long as we keep $|x-x_{0}|\leq \delta '$.

Since $f(x)\in Y$ and $|f(x)-y_{0}|\leq \delta$, this means we can substitute $y=f(x)$ and get

$$g(f(x))=g(y_{0})+g'(f(x_{0}))(f(x)-y_{0})+E_{g}(f(x),y_{0})$$

Now we use the differentiability of $f$. We can write

$$f(x)=f(x_{0})+f'(x_{0})(x-x_{0})+[f(x)-(f(x_{0})+f'(x_{0})(x-x_{0}))]$$

Again, we can define $E_{f}(x,x_{0}):=f(x)-(f(x_{0})+f'(x_{0})(x-x_{0}))$ and write this as

$$f(x)=f(x_{0})+f'(x_{0})(x-x_{0})+E_{f}(x,x_{0})$$

Now we can substitute this into the expression for $g(f(x))$ to get

$$g(f(x))=g(y_{0})+g'(f(x_{0}))(f'(x_{0})(x-x_{0})+E_{f}(x,x_{0}))+E_{g}(f(x),f(x_{0}))$$

where we have canceled out two terms using $f(x_{0})=y_{0}$.

Thus we have

$$g(f(x))=g(y_{0})+g'(f(x_{0}))f'(x_{0})(x-x_{0})+[g'(f(x_{0}))E_{f}(x,x_{0})+E_{g}(f(x),f(x_{0}))]$$

We can write this as

$$(g\circ f)(x)-((g\circ f)(x_{0})+L(x-x_{0}))=[g'(f(x_{0}))E_{f}(x,x_{0})+E_{g}(f(x),f(x_{0}))]$$

where $L:=g'(f(x_{0}))f'(x_{0})$. Now the left hand side looks like the expression in Newton's approximation. This means to show $g\circ f$ is differentiable at $x_{0}$, we just need to show that $|g'(f(x_{0}))E_{f}(x,x_{0})+E_{g}(f(x),f(x_{0}))|\leq \epsilon |x-x_{0}|$.

The stuff in square brackets is our "error term" for $g\circ f$. Now we just need to make sure it is small, even after dividing by $|x-x_{0}|$.

But f is differentiable at $x_{0}$, so by Newton's approximation,

$$|g'(f(x_{0}))E_{f}(x,x_{0})|\leq |g'(f(x_{0}))|\epsilon _{1}|x-x_{0}|$$

we also have

$$|E_{g}(f(x),f(x_{0}))|\leq \epsilon _{2}|f(x)-f(x_{0})|=\epsilon _{2}|f'(x_{0})(x-x_{0})+E_{f}(x,x_{0})|$$

We can bound this from above using the triangle inequality:

$${\begin{aligned}|E_{g}(f(x),f(x_{0}))|&\leq \epsilon _{2}|f'(x_{0})(x-x_{0})|+\epsilon _{2}|E_{f}(x,x_{0})|\\&\leq \epsilon _{2}|f'(x_{0})||x-x_{0}|+\epsilon _{2}\epsilon _{1}|x-x_{0}|\end{aligned}}$$

Now we can just choose $\epsilon _{1},\epsilon _{2}$ small enough.

Limits of sequences

Main idea

Let $(x_{n})_{n=1}^{\infty }$ be a sequence taking values in $X\setminus \{x_{0}\}$ that converges to $x_{0}$. Then we want to write

$${\frac {g(f(x_{n}))-g(f(x_{0}))}{x_{n}-x_{0}}}={\frac {g(f(x_{n}))-g(f(x_{0}))}{f(x_{n})-f(x_{0})}}\cdot {\frac {f(x_{n})-f(x_{0})}{x_{n}-x_{0}}}$$

Now use the limit laws to conclude that the limit is $g'(f(x_{0}))\cdot f'(x_{0})$. The problem is that $f(x_{n})-f(x_{0})$ can be zero even when $x_{n}\neq x_{0}$.

Proof

Let $(x_{n})_{n=1}^{\infty }$ be a sequence taking values in $X\setminus \{x_{0}\}$ that converges to $x_{0}$.

Define a function $\phi :Y\to \mathbf {R}$ by

$$\phi (y):={\begin{cases}{\frac {g(y)-g(f(x_{0}))}{y-f(x_{0})}}&{\text{if }}y\neq f(x_{0})\\g'(f(x_{0}))&{\text{if }}y=f(x_{0})\end{cases}}$$

The idea is that we want to say ${\frac {g(f(x_{n}))-g(f(x_{0}))}{f(x_{n})-f(x_{0})}}$ is going to $g'(f(x_{0}))$, so we just define it at the undefined points to already be at that limit.

Now we have

$${\frac {g(f(x_{n}))-g(f(x_{0}))}{x_{n}-x_{0}}}=\phi (f(x_{n}))\cdot {\frac {f(x_{n})-f(x_{0})}{x_{n}-x_{0}}}$$

for all $x_{n}$. (Why? Consider the cases $f(x_{n})=f(x_{0})$ and $f(x_{n})\neq f(x_{0})$ separately.)

Differentiability of $g$ at $f(x_{0})$ says that if $(y_{n})_{n=1}^{\infty }$ is a sequence taking values in $Y\setminus \{y_{0}\}$ that converges to $f(x_{0})$, then ${\frac {g(y_{n})-g(f(x_{0}))}{y_{n}-f(x_{0})}}\to g'(f(x_{0}))$ as $n\to \infty$. What if $(y_{n})_{n=1}^{\infty }$ is instead a sequence taking values in $Y$? Then we can say $\phi (y_{n})\to g'(f(x_{0}))$ as $n\to \infty$. To show this, let $\epsilon >0$.

Now we can find $N\geq 1$ such that for all $n\geq N$, if $y_{n}\neq f(x_{0})$, then $|\phi (y_{n})-g'(f(x_{0}))|=\left\vert {\frac {g(y_{n})-g(f(x_{0}))}{y_{n}-f(x_{0})}}-g'(f(x_{0}))\right\vert \leq \epsilon$. (TODO: I think here we need to break off into two cases: one where there's an infinite number of $y_{n}$ such that $y_{n}\neq f(x_{0})$, and one where there's only a finite number so that eventually the sequence is all just $y_{n}=f(x_{0})$. Only in the former case can we find $N$, by considering the subsequence that isn't equal to $f(x_{0})$, but this is not a problem because in the latter case the sequence's tail is already at the place where we need it to be, so we don't even need to find $N$. The question is, is there some more elegant way to do this that doesn't break off into cases?)

But this means if $n\geq N$, then we have two cases: either $y_{n}\in Y$ and $y_{n}\neq f(x_{0})$, in which case $|\phi (y_{n})-g'(f(x_{0}))|\leq \epsilon$ as above, or else $y_{n}=f(x_{0})$, in which case $\phi (y_{n})=g'(f(x_{0}))$ so $|\phi (y)-g'(f(x_{0}))|=0\leq \epsilon$.

Differentiability of $f$ at $x_{0}$ implies continuity of $f$ at $x_{0}$, so this means that $f(x_{n})\to f(x_{0})$ as $n\to \infty$. Since $f(x_{n})\in Y$ for each $n\geq 1$, we can use $(f(x_{n}))_{n=1}^{\infty }$ as our sequence in $Y$ to conclude that as $n\to \infty$ we have $\phi (f(x_{n}))\to g'(f(x_{0}))$.

Now by the limit laws

$${\begin{aligned}\lim _{n\to \infty }{\frac {g(f(x_{n}))-g(f(x_{0}))}{x_{n}-x_{0}}}&=\left(\lim _{n\to \infty }\phi (f(x_{n}))\right)\left(\lim _{n\to \infty }{\frac {f(x_{n})-f(x_{0})}{x_{n}-x_{0}}}\right)\\&=g'(f(x_{0}))f'(x_{0})\end{aligned}}$$

Since the sequence $(x_{n})_{n=1}^{\infty }$ was arbitrary, we can conclude that $\lim _{x\to x_{0};\,x\in X\setminus \{x_{0}\}}{\frac {g(f(x))-g(f(x_{0}))}{x-x_{0}}}=g'(f(x_{0}))f'(x_{0})$.

${\frac {g(f(x_{n}))-g(f(x_{0}))}{f(x_{n})-f(x_{0})}}\to g'(f(x_{0}))$

TODO: Tao says that division by zero occurs when $f'(x_{0})=0$, which seems strange to me.