Artificial intelligence · University

Neural layers, activations and parameter counts

A dense layer computes z=Wx+b followed by an activation. Dimensions must agree; each output has its own weighted combination.

Prerequisites

Vectors, matrix multiplication and functions.

A dense layer computes z=Wx+b followed by an activation. Dimensions must agree; each output has its own weighted combination.

Nonlinear activations allow compositions that are not just one affine map. Stacking affine layers alone collapses to another affine map.

For d inputs and h outputs, a dense layer with one bias per output has dh+h parameters.

ReLU(z)=max(0,z). It is piecewise linear and nondifferentiable at zero; implementations choose a convention there.

3×4+4=16.

max(0,−2)=0.

For weights (2,−1), input (3,4) and bias 1, find the preactivation.
What happens if several affine layers are stacked with no nonlinear activation?
1. Their composition is another affine map
2. They necessarily output zero
3. They automatically become any possible function
4. They have no parameters
Which dimensions must match in Wx?
1. The rows of W and word count
2. Nothing ever
3. Only the file sizes
4. The columns of W and entries of x
Why does a larger parameter count not guarantee better generalization?

Answers and marking notes

3
2×3−1×4+1=3.
Their composition is another affine map
Affine compositions remain affine.
The columns of W and entries of x
Matrix multiplication requires compatible inner dimensions.
Capacity, data quality, regularization, optimization and task structure all matter. More parameters alone do not establish unseen-data performance.
Credit multiple relevant factors and evaluation.