MMiDS 6.2: Self-Assessment Quiz

Which of the following is NOT an example of an exponential family of distributions?

• a) Bernoulli
• b) Categorical
• c) Uniform
• d) Multivariate Gaussian

In the exponential family form \(p_{\boldsymbol{\theta}}(\mathbf{x}) = h(\mathbf{x}) \exp(\boldsymbol{\theta}^T \boldsymbol{\phi}(\mathbf{x}) - A(\boldsymbol{\theta}))\), what does \(A(\boldsymbol{\theta})\) represent?

• a) The sufficient statistic
• b) The log-partition function
• c) The canonical parameter
• d) The base measure

Given \(n\) independent samples \(X_1, \ldots, X_n\) from a parametric family \(p_{\theta^*}\) with unknown \(\theta^* \in \Theta\), the maximum likelihood estimator \(\hat{\theta}_{\mathrm{MLE}}\) is defined as:

• a) \(\hat{\theta}_{\mathrm{MLE}} \in \arg\max \left\{ \prod_{i=1}^n p_\theta(X_i) : \theta \in \Theta \right\}\)
• b) \(\hat{\theta}_{\mathrm{MLE}} \in \arg\min \left\{ \prod_{i=1}^n p_\theta(X_i) : \theta \in \Theta \right\}\)
• c) \(\hat{\theta}_{\mathrm{MLE}} \in \arg\max \left\{ \sum_{i=1}^n p_\theta(X_i) : \theta \in \Theta \right\}\)
• d) \(\hat{\theta}_{\mathrm{MLE}} \in \arg\min \left\{ \sum_{i=1}^n p_\theta(X_i) : \theta \in \Theta \right\}\)

What is the Kullback-Leibler divergence \(\mathrm{KL}(\mathbf{p} \parallel \mathbf{q})\) between two probability distributions \(\mathbf{p}\) and \(\mathbf{q}\)?

• a) \(\sum_{i} p_i \log(p_i / q_i)\)
• b) \(\sum_{i} q_i \log(q_i / p_i)\)
• c) \(\sum_{i} (p_i - q_i)^2\)
• d) \(\sum_{i} \left| p_i - q_i \right|\)

What is the relationship between the maximum likelihood estimator and the Kullback-Leibler divergence?

• a) The MLE minimizes the KL divergence between the true and estimated distributions
• b) The MLE maximizes the KL divergence between the true and estimated distributions
• c) The MLE is unrelated to the KL divergence
• d) The MLE minimizes the KL divergence between the estimated and true distributions

Under certain conditions, the maximum likelihood estimator is guaranteed to converge to the true parameter as the number of samples grows. This property is known as:

• a) Statistical efficiency
• b) Statistical sufficiency
• c) Statistical consistency
• d) Statistical unbiasedness

For exponential families, the maximum likelihood estimator (if it exists) solves which of the following equations?

• a) \(\nabla A(\boldsymbol{\theta}) = 0\)
• b) \(\nabla Z(\boldsymbol{\theta}) = 0\)
• c) \(\mathbb{E}[\boldsymbol{\phi}(\mathbf{X})] = \frac{1}{n} \sum_{i=1}^n \boldsymbol{\phi}(\mathbf{X}_i)\)
• d) \(\mathrm{Var}[\boldsymbol{\phi}(\mathbf{X})] = \frac{1}{n} \sum_{i=1}^n (\boldsymbol{\phi}(\mathbf{X}_i) - \mathbb{E}[\boldsymbol{\phi}(\mathbf{X})])^2\)

In a generalized linear model, the maximum likelihood estimator \(\hat{\mathbf{w}}_{\mathrm{MLE}}\) solves the equation:

• a) \(\sum_{i=1}^n \mathbf{x}_i \mu(\mathbf{w}; \mathbf{x}_i) = \sum_{i=1}^n \mathbf{x}_i \phi(y_i)\)
• b) \(\sum_{i=1}^n \mathbf{x}_i \mu(\mathbf{w}; \mathbf{x}_i) = \sum_{i=1}^n y_i \phi(\mathbf{x}_i)\)
• c) \(\sum_{i=1}^n \mu(\mathbf{w}; \mathbf{x}_i) = \sum_{i=1}^n \phi(y_i)\)
• d) \(\sum_{i=1}^n \mu(\mathbf{w}; \mathbf{x}_i) = \sum_{i=1}^n y_i\)

In logistic regression, which distribution is used for the outcome variable?

• a) Normal distribution
• b) Poisson distribution
• c) Bernoulli distribution
• d) Exponential distribution