← Notes

Taylor Expansion

Why Taylor Expansion

Many functions we care about are too complicated to work with directly, but can be approximated locally by a polynomial. A Taylor expansion builds that polynomial from the value and derivatives of the function at a chosen expansion point.

Single-Variable Taylor Expansion

For a scalar function \(f(x)\), expanded around \(x=a\),

\[ f(x) = f(a) + f'(a)(x-a) + \frac{f''(a)}{2!}(x-a)^2 + \cdots = \sum_{n=0}^{\infty} \frac{f^{(n)}(a)}{n!}(x-a)^n. \]

Each derivative captures another aspect of the local shape:

The approximation is local: it assumes the derivatives measured at the expansion point remain informative nearby.

Multivariate Taylor Expansion

For a scalar function

\[ f:\mathbb{R}^n\rightarrow\mathbb{R}, \]

choose an expansion point \(x_0\) and write a nearby point as

\[ x=x_0+h, \qquad h=x-x_0. \]

The vector \(h\) describes both the direction and distance from the expansion point. The second-order approximation is

\[ f(x_0+h) \approx f(x_0) + \nabla f(x_0)^\top h + \frac{1}{2} h^\top H_f(x_0)h + O(\|h\|^3). \]

The three visible pieces are the starting value, the linear response, and the curvature response.

Concrete Example

Consider

\[ f(x,y)=x^2+xy+2y^2. \]

Approximate the function at \(x=(1,1.1)\) using an expansion around \(x_0=(1,1)\). The displacement is

\[ h = x-x_0 = \begin{bmatrix} 0\\ 0.1 \end{bmatrix}. \]

The starting value is

\[ f(1,1)=1^2+(1)(1)+2(1)^2=4. \]

The gradient is

\[ \nabla f(x,y) = \begin{bmatrix} 2x+y\\ x+4y \end{bmatrix}, \qquad \nabla f(1,1) = \begin{bmatrix} 3\\ 5 \end{bmatrix}. \]

Hence the first-order response is

\[ \nabla f(1,1)^\top h = \begin{bmatrix}3&5\end{bmatrix} \begin{bmatrix}0\\0.1\end{bmatrix} = 0.5, \]

giving the first-order approximation

\[ f(1,1.1)\approx 4+0.5=4.5. \]

The Hessian is

\[ H_f = \begin{bmatrix} 2&1\\ 1&4 \end{bmatrix}. \]

The second-order term is

\[ \frac{1}{2}h^\top H_f h = \frac{1}{2}(0.04) = 0.02. \]

Therefore,

\[ f(1,1.1) \approx 4+0.5+0.02 = 4.52. \]