# Neural layers, activations and parameter counts

> A dense layer computes z=Wx+b followed by an activation. Dimensions must agree; each output has its own weighted combination.

Course: [Neural networks, attention and embeddings](https://studybrix.com/en/courses/deep-learning/index.md)
Level: university
Prerequisites: Vectors, matrix multiplication and functions.

## Notes

### Layer

A dense layer computes z=Wx+b followed by an activation. Dimensions must agree; each output has its own weighted combination.

### Nonlinearity

Nonlinear activations allow compositions that are not just one affine map. Stacking affine layers alone collapses to another affine map.

### Parameters

For d inputs and h outputs, a dense layer with one bias per output has dh+h parameters.

### ReLU

ReLU(z)=max(0,z). It is piecewise linear and nondifferentiable at zero; implementations choose a convention there.

## Practice questions

### 1. A dense layer has 3 inputs and 4 outputs with one bias per output. Count parameters.

Answer: 16

Explanation: 3×4+4=16.

### 2. Find ReLU(−2).

Answer: 0

Explanation: max(0,−2)=0.

### 3. For weights (2,−1), input (3,4) and bias 1, find the preactivation.

Answer: 3

Explanation: 2×3−1×4+1=3.

### 4. What happens if several affine layers are stacked with no nonlinear activation?

A. Their composition is another affine map
B. They necessarily output zero
C. They automatically become any possible function
D. They have no parameters

Answer: Their composition is another affine map

Explanation: Affine compositions remain affine.

### 5. Which dimensions must match in Wx?

A. The rows of W and word count
B. Nothing ever
C. Only the file sizes
D. The columns of W and entries of x

Answer: The columns of W and entries of x

Explanation: Matrix multiplication requires compatible inner dimensions.

### 6. Why does a larger parameter count not guarantee better generalization?

Sample answer / marking guidance: Capacity, data quality, regularization, optimization and task structure all matter. More parameters alone do not establish unseen-data performance.

Explanation: Credit multiple relevant factors and evaluation.

### 7. A layer has 5 inputs, 2 outputs and no biases. Count weights.

Answer: 10

Explanation: 5×2=10.

### 8. Explain why saying ReLU has derivative one everywhere is incorrect.

Sample answer / marking guidance: Its derivative is zero for negative inputs and one for positive inputs, and the mathematical derivative is undefined at zero.

Explanation: Credit both regions and the corner.

## Further reading

- https://developers.google.com/machine-learning/crash-course
- https://cs229.stanford.edu/

Original AI-assisted educational draft; not independently teacher-reviewed. References support further study, not copied exercises, affiliation or curriculum accreditation.