Preporato

Forward pass of a two-layer network

Easy10 min10 hidden tests

A delivery app scores each order with a small neural network. Each order is a row of features. The first layer maps the features to hidden units and applies ReLU, which keeps positive values and turns negative ones into 0. The second layer maps the hidden units to the outputs with no activation. Write mlp_forward(X, W1, b1, W2, b2) that returns the outputs for a batch of orders. For the one order [1.0, 2.0] in the first example, the hidden units come out as 2.0, 1.5 and 0.0, and the output is 5.1.

Change the order’s features

relu(x W₁ + b₁) W₂ + b₂
Positive weightNegative weightUnit on

Hidden sums [2, 1.5, −1.3], after ReLU [2, 1.5, 0]. Score = 2×1 + 1.5×2 + 0×(−1) + 0.1 = 5.1. Hidden unit 3 is off: a negative sum becomes 0, so it adds nothing.

Inputs 1 and 2. Hidden 2, 1.5, 0. Score 5.1.

Examples

The third hidden unit is negative before ReLU, so it adds nothing

mlp_forward([[1.0, 2.0]], [[1.0, -1.0, 0.5], [0.5, 1.0, -1.0]], [0.0, 0.5, 0.2], [[1.0], [2.0], [-1.0]], [0.1])

returns [[5.1]]

Three orders, two outputs each

mlp_forward([[1.0, 0.0], [0.0, 1.0], [-1.0, -1.0]], [[2.0, -1.0], [1.0, 3.0]], [0.0, 0.0], [[1.0, 0.0], [0.0, 1.0]], [0.5, -0.5])

returns [[2.5, -0.5], [1.5, 2.5], [0.5, -0.5]]

Requirements

  • X has shape (n, d). W1 has shape (d, h) and b1 shape (h,). W2 has shape (h, k) and b2 shape (k,). Lists or arrays are both fine.
  • Compute relu(X @ W1 + b1) @ W2 + b2, where relu(z) is max(z, 0) for each entry.
  • Add the bias before the ReLU. The output layer has no activation.
  • Return a float NumPy array of shape (n, k), even when n or k is 1.
The idea behind it

A multilayer perceptron (MLP) is a stack of layers. Each layer multiplies its input by a weight matrix, adds a bias and usually applies an activation. ReLU, short for rectified linear unit, is the most common activation: relu(z) = max(z, 0). Without an activation, two layers collapse into one linear map, because a product of matrices is still a matrix. ReLU bends the map, so the network can learn curved decision boundaries. The last layer usually has no activation. Its raw outputs feed a loss, a softmax or a sigmoid. With a batch stored as rows, one matrix product handles every example at once: (n, d) @ (d, h) gives (n, h). The same pattern runs inside every transformer block, where it is called the feed-forward or MLP layer.

pythonCtrl+Enter runs
import numpy as np

def mlp_forward(X, W1, b1, W2, b2):
"""Outputs of a two-layer ReLU network for a batch.

X: (n, d) inputs, one row per example
W1: (d, h) first-layer weights, b1: (h,) first-layer biases
W2: (h, k) second-layer weights, b2: (k,) second-layer biases
Returns a float array of shape (n, k): relu(X @ W1 + b1) @ W2 + b2.
"""
raise NotImplementedError

Stuck? Reveal one rung at a time

Worked solution

Give it a real try first. Then read the solution and why each line is there.