← Notes

The Gradient

Definition

For a scalar-valued function of several variables,

\[ f(x_1,\ldots,x_n), \]

the gradient is the vector of partial derivatives:

\[ \nabla f = \begin{bmatrix} \frac{\partial f}{\partial x_1}\\ \frac{\partial f}{\partial x_2}\\ \vdots\\ \frac{\partial f}{\partial x_n} \end{bmatrix}. \]

If \(f:\mathbb{R}^n\rightarrow\mathbb{R}\), then \(\nabla f:\mathbb{R}^n\rightarrow\mathbb{R}^n\): at every input point, the gradient produces a vector in the same input space.

Interpretation

The gradient collects the local rates of change of the function in each coordinate direction. Its key properties are:

Linear Response

For a small displacement \(h\), the first-order change in the function is

\[ \Delta f \approx \nabla f(x_0)^\top h. \]

This dot product combines the local rate of change in each coordinate with how far we move in that coordinate. In the Taylor expansion,

\[ f(x_0+h) \approx f(x_0) + \nabla f(x_0)^\top h. \]

Example

For

\[ f(x,y)=x^2+y^2, \]

the gradient is

\[ \nabla f(x,y) = \begin{bmatrix} 2x\\ 2y \end{bmatrix}. \]

The resulting vector field points radially away from the origin, which matches the fact that \(x^2+y^2\) increases fastest by moving directly away from the origin.