MMiDS 3.3: Self-Assessment Quiz

Which of the following is the correct definition of a global minimizer \( \mathbf{x}^* \) of a function \( f: \mathbb{R}^d \to \mathbb{R} \)?

a) \( f(\mathbf{x}) \geq f(\mathbf{x}^*) \) for all \( \mathbf{x} \) in some open ball around \( \mathbf{x}^* \).
b) \( f(\mathbf{x}) \geq f(\mathbf{x}^*) \) for all \( \mathbf{x} \in \mathbb{R}^d \).
c) \( \nabla f(\mathbf{x}^*) = \mathbf{0} \).
d) \( \mathbf{v}^T H_f(\mathbf{x}^*) \mathbf{v} > 0 \) for all \( \mathbf{v} \in \mathbb{R}^d \).

Which of the following statements about descent directions is true?

a) A descent direction for a function \( f \) at a point \( \mathbf{x}_0 \) is any direction in which \( f \) increases.
b) A descent direction for a function \( f \) at a point \( \mathbf{x}_0 \) is any direction in which \( f \) does not change.
c) A descent direction for a function \( f \) at a point \( \mathbf{x}_0 \) is any direction in which \( f \) decreases.
d) A descent direction for a function \( f \) at a point \( \mathbf{x}_0 \) is always the direction of steepest descent.

Which of the following is the correct definition of a local minimizer \( \mathbf{x}^* \) of a function \( f: \mathbb{R}^d \to \mathbb{R} \)?

a) \( f(\mathbf{x}^*) \leq f(\mathbf{x}) \) for all \( \mathbf{x} \in \mathbb{R}^d \).
b) \( f(\mathbf{x}^*) < f(\mathbf{x}) \) for all \( \mathbf{x} \in \mathbb{R}^d \).
c) There exists \( \delta > 0 \) such that \( f(\mathbf{x}^*) \leq f(\mathbf{x}) \) for all \( \mathbf{x} \in B_\delta(\mathbf{x}^*) \setminus \{\mathbf{x}^*\} \).
d) There exists \( \delta > 0 \) such that \( f(\mathbf{x}^*) < f(\mathbf{x}) \) for all \( \mathbf{x} \in B_\delta(\mathbf{x}^*) \setminus \{\mathbf{x}^*\} \).

Which of the following statements about stationary points is true?

a) All stationary points are local minimizers.
b) All local minimizers are stationary points.
c) A stationary point is a point where the gradient of the function is nonzero.
d) A stationary point is always a saddle point.

Which of the following is a necessary condition for a point \( \mathbf{x}^* \) to be a local minimizer of a continuously differentiable function \( f: \mathbb{R}^d \to \mathbb{R} \)?

a) \( \nabla f(\mathbf{x}^*) \neq \mathbf{0} \).
b) \( \nabla f(\mathbf{x}^*) = \mathbf{0} \).
c) \( \mathbf{H}_f(\mathbf{x}^*) \) is positive definite.
d) \( \mathbf{H}_f(\mathbf{x}^*) \) is negative definite.

Consider a continuously differentiable function \( f: \mathbb{R}^d \to \mathbb{R} \). If \( \mathbf{v} \in \mathbb{R}^d \) is a descent direction for \( f \) at \( \mathbf{x}_0 \), then which of the following is true?

a) \( \nabla f(\mathbf{x}_0)^T \mathbf{v} > 0 \).
b) \( \nabla f(\mathbf{x}_0)^T \mathbf{v} = 0 \).
c) \( \nabla f(\mathbf{x}_0)^T \mathbf{v} < 0 \).
d) None of the above.

Let \( f: \mathbb{R}^d \to \mathbb{R} \) be continuously differentiable at \( \mathbf{x}_0 \). The directional derivative of \( f \) at \( \mathbf{x}_0 \) in the direction \( \mathbf{v} \in \mathbb{R}^d \) is NOT given by:

a) \( \frac{\partial f(\mathbf{x}_0)}{\partial \mathbf{v}} = \nabla f(\mathbf{x}_0)^T \mathbf{v} \).
b) \( \frac{\partial f(\mathbf{x}_0)}{\partial \mathbf{v}} = \mathbf{v}^T \nabla f(\mathbf{x}_0) \).
c) \( \frac{\partial f(\mathbf{x}_0)}{\partial \mathbf{v}} = \mathbf{v}^T \mathbf{H}_f(\mathbf{x}_0) \mathbf{v} \).
d) \( \frac{\partial f(\mathbf{x}_0)}{\partial \mathbf{v}} = \lim_{h \to 0} \frac{f(\mathbf{x}_0 + h\mathbf{v}) - f(\mathbf{x}_0)}{h} \).

Let \( f: \mathbb{R}^d \to \mathbb{R} \) be twice continuously differentiable at \( \mathbf{x}_0 \). The second directional derivative of \( f \) at \( \mathbf{x}_0 \) in the direction \( \mathbf{v} \in \mathbb{R}^d \) is given by:

a) \( \frac{\partial^2 f(\mathbf{x}_0)}{\partial \mathbf{v}^2} = \lim_{h \to 0} \frac{\nabla f(\mathbf{x}_0 + h\mathbf{v}) - \nabla f(\mathbf{x}_0)}{h} \).
b) \( \frac{\partial^2 f(\mathbf{x}_0)}{\partial \mathbf{v}^2} = \lim_{h \to 0} \frac{1}{h} [\frac{\partial f(\mathbf{x}_0 + h\mathbf{v})}{\partial \mathbf{v}} - \frac{\partial f(\mathbf{x}_0)}{\partial \mathbf{v}}] \).
c) \( \frac{\partial^2 f(\mathbf{x}_0)}{\partial \mathbf{v}^2} = \mathbf{v}^T \mathbf{H}_f(\mathbf{x}_0) \mathbf{v} \).
d) Both b and c.

Let \( f: \mathbb{R}^d \to \mathbb{R} \) be twice continuously differentiable. If \( \nabla f(\mathbf{x}_0) = \mathbf{0} \) and \( \mathbf{H}_f(\mathbf{x}_0) \) is positive definite, then \( \mathbf{x}_0 \) is:

a) A global minimizer of \( f \).
b) A local minimizer of \( f \), but not necessarily a strict local minimizer.
c) A strict local minimizer of \( f \).
d) A saddle point of \( f \).

What is the Lagrangian function used for in constrained optimization?

a) It transforms a constrained optimization problem into an unconstrained one.
b) It provides a way to find the feasible set of the constrained problem.
c) It directly gives the optimal solution to the constrained problem.
d) It is used to calculate the gradient of the objective function.

In the method of Lagrange multipliers, what is the geometric interpretation of the condition \( \nabla f(\mathbf{x}^*) + \sum_{i=1}^l \lambda_i^* \nabla h_i(\mathbf{x}^*) = \mathbf{0} \)?

a) The gradient of the objective function is parallel to the gradient of each constraint function.
b) The gradient of the objective function is orthogonal to the subspace of first-order feasible directions at \( \mathbf{x}^* \).
c) The gradient of the objective function is zero at \( \mathbf{x}^* \).
d) The Lagrange multipliers are all equal to zero.

Consider the optimization problem \( \min_{\mathbf{x}} f(\mathbf{x}) \) subject to \( h(\mathbf{x}) = \mathbf{0} \), where \( f: \mathbb{R}^d \to \mathbb{R} \) and \( h: \mathbb{R}^d \to \mathbb{R}^\ell \) are continuously differentiable. Let \( \mathbf{x}^* \) be a local minimizer and assume that the vectors \( \nabla h_i(\mathbf{x}^*), i \in [\ell] \), are linearly independent. According to the Lagrange Multipliers theorem, which of the following must be true?

a) \( \nabla f(\mathbf{x}^*) = \mathbf{0} \).
b) \( \nabla f(\mathbf{x}^*) + \sum_{i=1}^\ell \lambda_i^* \nabla h_i(\mathbf{x}^*) = \mathbf{0} \) for some \( \boldsymbol{\lambda}^* \in \mathbb{R}^\ell \).
c) \( h(\mathbf{x}^*) = \mathbf{0} \).
d) Both b and c.

Consider the optimization problem \( \min_{\mathbf{x}} f(\mathbf{x}) \) subject to \( h(\mathbf{x}) = \mathbf{0} \), where \( f: \mathbb{R}^d \to \mathbb{R} \) and \( h: \mathbb{R}^d \to \mathbb{R}^\ell \) are continuously differentiable. Let \( \mathbf{x}^* \) be a local minimizer satisfying the first-order necessary conditions with Lagrange multipliers \( \boldsymbol{\lambda}^* \). The subspace of first-order feasible directions at \( \mathbf{x}^* \) is defined as:

a) \( F_h(\mathbf{x}^*) = \{\mathbf{v} \in \mathbb{R}^d: \nabla h_i(\mathbf{x}^*)^T \mathbf{v} = 0, \forall i \in [\ell]\} \).
b) \( F_h(\mathbf{x}^*) = \{\mathbf{v} \in \mathbb{R}^d: \nabla h_i(\mathbf{x}^*)^T \mathbf{v} > 0, \forall i \in [\ell]\} \).
c) \( F_h(\mathbf{x}^*) = \{\mathbf{v} \in \mathbb{R}^d: \nabla f(\mathbf{x}^*)^T \mathbf{v} = 0\} \).
d) \( F_h(\mathbf{x}^*) = \{\mathbf{v} \in \mathbb{R}^d: \nabla f(\mathbf{x}^*)^T \mathbf{v} + \sum_{i=1}^\ell \lambda_i^* \nabla h_i(\mathbf{x}^*)^T \mathbf{v} = 0\} \).

Which of the following is a correct statement of Taylor's Theorem (to second order) for a twice continuously differentiable function \( f: D \to \mathbb{R} \), where \( D \subseteq \mathbb{R}^d \), at an interior point \( \mathbf{x}_0 \in D \)?

a) For any \( \mathbf{x} \in B_\delta(\mathbf{x}_0) \), \( f(\mathbf{x}) = f(\mathbf{x}_0) + \nabla f(\mathbf{x}_0)^T (\mathbf{x} - \mathbf{x}_0) + \frac{1}{2}(\mathbf{x} - \mathbf{x}_0)^T \mathbf{H}_f(\mathbf{x}_0 + \xi(\mathbf{x} - \mathbf{x}_0))(\mathbf{x} - \mathbf{x}_0) \) for some \( \xi \in (0,1) \).
b) For any \( \mathbf{x} \in B_\delta(\mathbf{x}_0) \), \( f(\mathbf{x}) = f(\mathbf{x}_0) + \nabla f(\mathbf{x}_0 + \xi(\mathbf{x} - \mathbf{x}_0))^T (\mathbf{x} - \mathbf{x}_0) + \frac{1}{2}(\mathbf{x} - \mathbf{x}_0)^T \mathbf{H}_f(\mathbf{x}_0 + \xi(\mathbf{x} - \mathbf{x}_0))(\mathbf{x} - \mathbf{x}_0) \).
c) For any \( \mathbf{x} \in B_\delta(\mathbf{x}_0) \), \( f(\mathbf{x}) = f(\mathbf{x}_0) + \nabla f(\mathbf{x}_0 + \xi(\mathbf{x} - \mathbf{x}_0))^T (\mathbf{x} - \mathbf{x}_0) \).
d) For any \( \mathbf{x} \in B_\delta(\mathbf{x}_0) \), \( f(\mathbf{x}) = f(\mathbf{x}_0) + \frac{1}{2}(\mathbf{x}_0 + \xi(\mathbf{x} - \mathbf{x}_0))^T \mathbf{H}_f(\mathbf{x}_0)(\mathbf{x}_0 + \xi(\mathbf{x} - \mathbf{x}_0)) \).