MMiDS 8.3: Self-Assessment Quiz

How is the concatenation of vectors \( \mathbf{v}_1 \) and \( \mathbf{v}_2 \) denoted?

• a) \( \mathbf{v}_1 \oplus \mathbf{v}_2 \)
• b) \( \mathbf{v}_1 \otimes \mathbf{v}_2 \)
• c) \( \mathbf{v}_1 \parallel \mathbf{v}_2 \)
• d) \( (\mathbf{v}_1, \mathbf{v}_2) \)

In the backpropagation algorithm, what does the 'forward pass' compute?

• a) The adjoints \( p_i \) for each layer \( i \).
• b) The gradients \( q_i \) for the parameters of each layer \( i \).
• c) The function values \( z_i \) and the Jacobians \( A_i, B_i \) for each layer \( i \).
• d) The final gradient \( \nabla f(w) \) with respect to all parameters.

What is the purpose of the 'backward pass' in the backpropagation algorithm?

• a) To compute the function values \( z_i \) for each layer \( i \) from the input \( x \).
• b) To compute the Jacobians \( A_i, B_i \) for each layer \( i \) using the fundamental recursion.
• c) To compute the adjoints \( p_i \) and the gradients \( q_i \) for each layer \( i \) using the fundamental recursion.
• d) To compute the final output \( \ell(z_{L+1}) \) of the progressive function.

What is the computational complexity of the backpropagation algorithm in terms of the number of layers \( L \) and the matrix dimensions \( m \)?

• a) \( \mathcal{O}(Lm) \)
• b) \( \mathcal{O}(Lm^2) \)
• c) \( \mathcal{O}(Lm^2d) \)
• d) \( \mathcal{O}(Lm^3d) \)

In the context of progressive functions, what is the significance of the matrices \( A_i \) and \( B_i \)?

• a) They represent the Jacobians of the layer functions with respect to the inputs and parameters, respectively.
• b) They are the intermediate values computed during the forward pass.
• c) They are the adjoints used in the backpropagation algorithm.
• d) They are the matrices of parameters for each layer.

In the context of progressive functions, which of the following best describes the role of the vector \( w_i \)?

• a) The input to the \( i \)-th layer.
• b) The output of the \( i \)-th layer.
• c) The parameters specific to the \( i \)-th layer.
• d) The concatenation of parameters from all layers up to \( i \).

What is the significance of the vectors \( p_i \) computed in the backward pass of backpropagation?

• a) They represent the gradients of the loss function with respect to the parameters \( w_i \).
• b) They represent the Jacobians of the \( i \)-th layer with respect to its input \( z_i \).
• c) They represent the partial derivatives of the loss function with respect to the layer inputs \( z_i \).
• d) They represent the concatenation of all parameter gradients up to layer \( i \).