StudyBrixFree worksheets, study guides and courses

Artificial intelligence · lesson 2 of 4

Gradient descent and learning-rate choices

Gradient descent uses θnew=θ−η∇L(θ) with a positive step size η. The negative gradient is locally downhill for a differentiable objective.

University8 practice questionsNotes + worked examples
Take the quizStudy guideRevision sheet

Learn the ideas

Prerequisites: Derivatives and vector arithmetic.

Update

Gradient descent uses θnew=θ−η∇L(θ) with a positive step size η. The negative gradient is locally downhill for a differentiable objective.

Learning rate

A step that is too large can overshoot or diverge. A very small step can be slow; no one value is suitable for every objective.

Batches

Full-batch and minibatch methods use different gradient estimates. Stochastic variation can affect the optimization trajectory.

Checking

A finite-difference check can help detect derivative implementation errors on small examples. It is a numerical diagnostic with step-size limitations.

Worked examples

Example 1

For θ=4, gradient 6 and η=0.1, find the next gradient-descent parameter.

4−0.1×6=3.4.

3.4

Example 2

For L(w)=(w−2)², find the gradient at w=5.

2(w−2)=6.

6

Check your understanding

8 questions

Decimals or fractions are accepted for numeric answers. Proofs and written responses use sample answers and require human marking. This is practice, not a protected examination.

1 For θ=4, gradient 6 and η=0.1, find the next gradient-descent parameter.

Show answer and explanation

3.4

4−0.1×6=3.4.

2 For L(w)=(w−2)², find the gradient at w=5.

Show answer and explanation

6

2(w−2)=6.

3 What does subtracting a positive multiple of the gradient aim to do locally?

Show answer and explanation

Reduce the differentiable loss

The negative gradient is a local descent direction for sufficiently small steps.

4 Which can occur with an excessively large learning rate?

Show answer and explanation

Overshooting or divergence

Large steps can leave the local descent regime.

5 Why does a small gradient not always prove a global minimum?

Sample answer and marking guidance

It may indicate a local minimum, saddle, flat region or poor scaling. Global optimality needs stronger structure or analysis.

Credit stationary-point alternatives.

6 For f(w)=w², use central difference [f(2.1)−f(1.9)]/0.2. Find the result.

Show answer and explanation

4

(4.41−3.61)/0.2=4.

7 What does a minibatch gradient use?

Show answer and explanation

A subset of training examples for that step

It estimates a training gradient from a batch.

8 Why should a gradient check avoid declaring success from only one point?

Sample answer and marking guidance

A bug can cancel at a special point or parameter setting. Use varied small test cases and appropriate tolerances.

Credit multiple points and numerical tolerance.

Further reading and review references

These are reference links, not copied exercises, institutional endorsement or proof of independent review.