StudyBrixFree worksheets, study guides and courses

Artificial intelligence · University

Gradient descent and learning-rate choices

Gradient descent uses θnew=θ−η∇L(θ) with a positive step size η. The negative gradient is locally downhill for a differentiable objective.

Make a worksheet

Prerequisites

Derivatives and vector arithmetic.

Study notes

Update

Gradient descent uses θnew=θ−η∇L(θ) with a positive step size η. The negative gradient is locally downhill for a differentiable objective.

Learning rate

A step that is too large can overshoot or diverge. A very small step can be slow; no one value is suitable for every objective.

Batches

Full-batch and minibatch methods use different gradient estimates. Stochastic variation can affect the optimization trajectory.

Checking

A finite-difference check can help detect derivative implementation errors on small examples. It is a numerical diagnostic with step-size limitations.

Worked examples

For θ=4, gradient 6 and η=0.1, find the next gradient-descent parameter.

4−0.1×6=3.4.

3.4

For L(w)=(w−2)², find the gradient at w=5.

2(w−2)=6.

6

Check your understanding

  1. What does subtracting a positive multiple of the gradient aim to do locally?

    1. Remove all data
    2. Always maximize loss
    3. Reduce the differentiable loss
    4. Guarantee a global optimum in any problem

  2. Which can occur with an excessively large learning rate?

    1. Overshooting or divergence
    2. Guaranteed exact convergence in one step
    3. A negative dataset size
    4. Automatic data cleaning

  3. Why does a small gradient not always prove a global minimum?

  4. For f(w)=w², use central difference [f(2.1)−f(1.9)]/0.2. Find the result.

Answers and marking guidance

Answers and marking notes
  1. Reduce the differentiable loss

    The negative gradient is a local descent direction for sufficiently small steps.

  2. Overshooting or divergence

    Large steps can leave the local descent regime.

  3. It may indicate a local minimum, saddle, flat region or poor scaling. Global optimality needs stronger structure or analysis.

    Credit stationary-point alternatives.

  4. 4

    (4.41−3.61)/0.2=4.

Further reading and review references

These are reference links, not copied exercises, institutional endorsement or proof of independent review.

AI-assisted preview; independent educational review required. Level labels describe prerequisites, not accreditation or official curriculum alignment.