Gradient Descent

Definition. An iterative optimization algorithm that moves parameters in the direction that most reduces a loss function, using the gradient of that function.

Key Points

• The gradient $\nabla L(\theta)$ points in the direction of steepest increase, so we step in the opposite direction.
• The learning rate $\eta$ controls step size — too large and it diverges, too small and it converges slowly.
• Variants: batch, stochastic (SGD), and mini-batch gradient descent differ in how much data is used per update.

Formula

$$\theta_{t+1} = \theta_t - \eta \nabla L(\theta_t)$$

Example

For $L(\theta) = \theta^2$, the gradient is $\nabla L(\theta) = 2\theta$. Starting at $\theta_0 = 4$ with $\eta = 0.1$:

$$\theta_1 = 4 - 0.1(8) = 3.2$$

Each step shrinks $\theta$ toward the minimum at $\theta = 0$.

Notes / Gotchas

• A learning rate that's too high can cause the loss to oscillate or diverge instead of converge.
• Plain gradient descent can get stuck in local minima or saddle points — momentum and adaptive methods (Adam, RMSProp) help with this.