# Gradient descent and learning-rate choices

> Gradient descent uses θnew=θ−η∇L(θ) with a positive step size η. The negative gradient is locally downhill for a differentiable objective.

Course: [Mathematics of machine learning](https://studybrix.com/en/courses/machine-learning-mathematics/index.md)
Level: university
Prerequisites: Derivatives and vector arithmetic.

## Notes

### Update

Gradient descent uses θnew=θ−η∇L(θ) with a positive step size η. The negative gradient is locally downhill for a differentiable objective.

### Learning rate

A step that is too large can overshoot or diverge. A very small step can be slow; no one value is suitable for every objective.

### Batches

Full-batch and minibatch methods use different gradient estimates. Stochastic variation can affect the optimization trajectory.

### Checking

A finite-difference check can help detect derivative implementation errors on small examples. It is a numerical diagnostic with step-size limitations.

## Practice questions

### 1. For θ=4, gradient 6 and η=0.1, find the next gradient-descent parameter.

Answer: 3.4

Explanation: 4−0.1×6=3.4.

### 2. For L(w)=(w−2)², find the gradient at w=5.

Answer: 6

Explanation: 2(w−2)=6.

### 3. What does subtracting a positive multiple of the gradient aim to do locally?

A. Remove all data
B. Always maximize loss
C. Reduce the differentiable loss
D. Guarantee a global optimum in any problem

Answer: Reduce the differentiable loss

Explanation: The negative gradient is a local descent direction for sufficiently small steps.

### 4. Which can occur with an excessively large learning rate?

A. Overshooting or divergence
B. Guaranteed exact convergence in one step
C. A negative dataset size
D. Automatic data cleaning

Answer: Overshooting or divergence

Explanation: Large steps can leave the local descent regime.

### 5. Why does a small gradient not always prove a global minimum?

Sample answer / marking guidance: It may indicate a local minimum, saddle, flat region or poor scaling. Global optimality needs stronger structure or analysis.

Explanation: Credit stationary-point alternatives.

### 6. For f(w)=w², use central difference [f(2.1)−f(1.9)]/0.2. Find the result.

Answer: 4

Explanation: (4.41−3.61)/0.2=4.

### 7. What does a minibatch gradient use?

A. Only the final test labels
B. A subset of training examples for that step
C. Only model file names
D. No loss function

Answer: A subset of training examples for that step

Explanation: It estimates a training gradient from a batch.

### 8. Why should a gradient check avoid declaring success from only one point?

Sample answer / marking guidance: A bug can cancel at a special point or parameter setting. Use varied small test cases and appropriate tolerances.

Explanation: Credit multiple points and numerical tolerance.

## Further reading

- https://developers.google.com/machine-learning/crash-course
- https://cs229.stanford.edu/

Original AI-assisted educational draft; not independently teacher-reviewed. References support further study, not copied exercises, affiliation or curriculum accreditation.