StudyBrixFree worksheets, study guides and courses

Artificial intelligence · lesson 1 of 4

Neural layers, activations and parameter counts

A dense layer computes z=Wx+b followed by an activation. Dimensions must agree; each output has its own weighted combination.

University8 practice questionsNotes + worked examples
Take the quizStudy guideRevision sheet

Learn the ideas

Prerequisites: Vectors, matrix multiplication and functions.

Layer

A dense layer computes z=Wx+b followed by an activation. Dimensions must agree; each output has its own weighted combination.

Nonlinearity

Nonlinear activations allow compositions that are not just one affine map. Stacking affine layers alone collapses to another affine map.

Parameters

For d inputs and h outputs, a dense layer with one bias per output has dh+h parameters.

ReLU

ReLU(z)=max(0,z). It is piecewise linear and nondifferentiable at zero; implementations choose a convention there.

Worked examples

Example 1

A dense layer has 3 inputs and 4 outputs with one bias per output. Count parameters.

3×4+4=16.

16

Example 2

Find ReLU(−2).

max(0,−2)=0.

0

Check your understanding

8 questions

Decimals or fractions are accepted for numeric answers. Proofs and written responses use sample answers and require human marking. This is practice, not a protected examination.

1 A dense layer has 3 inputs and 4 outputs with one bias per output. Count parameters.

Show answer and explanation

16

3×4+4=16.

2 Find ReLU(−2).

Show answer and explanation

0

max(0,−2)=0.

3 For weights (2,−1), input (3,4) and bias 1, find the preactivation.

Show answer and explanation

3

2×3−1×4+1=3.

4 What happens if several affine layers are stacked with no nonlinear activation?

Show answer and explanation

Their composition is another affine map

Affine compositions remain affine.

5 Which dimensions must match in Wx?

Show answer and explanation

The columns of W and entries of x

Matrix multiplication requires compatible inner dimensions.

6 Why does a larger parameter count not guarantee better generalization?

Sample answer and marking guidance

Capacity, data quality, regularization, optimization and task structure all matter. More parameters alone do not establish unseen-data performance.

Credit multiple relevant factors and evaluation.

7 A layer has 5 inputs, 2 outputs and no biases. Count weights.

Show answer and explanation

10

5×2=10.

8 Explain why saying ReLU has derivative one everywhere is incorrect.

Sample answer and marking guidance

Its derivative is zero for negative inputs and one for positive inputs, and the mathematical derivative is undefined at zero.

Credit both regions and the corner.

Further reading and review references

These are reference links, not copied exercises, institutional endorsement or proof of independent review.