Lecture 004Regression strikes backEdward Rubin04 February 20201 / 42

Admin2 / 42

Admin

Today

In-class

A roadmap (where are we going?)
Linear regression and model selection

3 / 42

Admin

Upcoming

Readings

Today
- ISL Ch. 3 and 6.1
Next
- ISL Ch. 6 and 4

Problem sets

Due tonight! (How did it go?)
Next: After we finish this set of notes

4 / 42

Roadmap

Where are we?

We've essentially covered the central topics in statistical learning†

Prediction and inference
Supervised vs. unsupervised methods
Regression and classification problems
The dangers of overfitting
The bias-variance tradeoff
Model assessment
Holdouts, validation sets, and cross validation††
Model training and tuning
Simulation

† Plus a few of the "basic" methods: OLS regression and KNN.
†† And the bootstrap!

5 / 42

Roadmap

Where are we going?

Next, we will cover many common machine-learning algorithms, e.g.,

Decision trees and random forests
SVM
Neural nets
Clustering
Ensemble techniques

6 / 42

Roadmap

Where are we going?

Next, we will cover many common machine-learning algorithms, e.g.,

Decision trees and random forests
SVM
Neural nets
Clustering
Ensemble techniques

But first, we return to good old linear regression—in a new light...

Linear regression
Variable/model selection and LASSO/Ridge regression
Plus: Logistic regression and discriminant analysis

6 / 42

Roadmap

Why return to regression?

Motivation 1
We have new tools. It might help to first apply them in a familiar setting.

7 / 42

Roadmap

Why return to regression?

Motivation 1
We have new tools. It might help to first apply them in a familiar setting.

Motivation 2
We have new tools. Maybe linear regression will be (even) better now?

7 / 42

Roadmap

Why return to regression?

Motivation 1
We have new tools. It might help to first apply them in a familiar setting.

Motivation 2
We have new tools. Maybe linear regression will be (even) better now?

Motivation 3

many fancy statistical learning approaches can be seen as generalizations or extensions of linear regression.

Source: ISL, p. 59; emphasis added

7 / 42

Linear regression8 / 42

Linear regression

Regression regression

Linear regression "fits" coefficients $\color{#e64173}{\beta}_0,\, \ldots,\, \color{#e64173}{\beta}_p$ for a model $\begin{align} \color{#FFA500}{y}_i = \color{#e64173}{\beta}_0 + \color{#e64173}{\beta}_1 x_{1,i} + \color{#e64173}{\beta}_2 x_{2,i} + \cdots + \color{#e64173}{\beta}_p x_{p,i} + \varepsilon_i \end{align}$ and is often applied in two distinct settings with fairly distinct goals:

Causal inference estimates and interprets the coefficients.
Prediction focuses on accurately estimating outcomes.

Regardless of the goal, the way we "fit" (estimate) the model is the same.

9 / 42

Linear regression

Fitting the regression line

As is the case with many statistical learning methods, regression focuses on minimizing some measure of loss/error.

$\begin{align} e_i = \color{#FFA500}{y_i} - \color{#6A5ACD}{\hat{y}_i} \end{align}$

10 / 42

Linear regression

Fitting the regression line

As is the case with many statistical learning methods, regression focuses on minimizing some measure of loss/error.

$\begin{align} e_i = \color{#FFA500}{y_i} - \color{#6A5ACD}{\hat{y}_i} \end{align}$

Linear regression uses the L2 loss function—also called residual sum of squares (RSS) or sum of squared errors (SSE)

$\begin{align} \text{RSS} = e_1^2 + e_2^2 + \cdots + e_n^2 = \sum_{i=1}^n e_i^2 \end{align}$

Specifically: OLS chooses the $\color{#e64173}{\hat{\beta}_j}$ that minimize RSS.

10 / 42

Linear regression

Performance

There's a large variety of ways to assess the fit† of linear-regression models.

† or predictive performance

() $\begin{align} \text{RSE}=\sqrt{\dfrac{1}{n-p-1}\text{RSS}}=\sqrt{\dfrac{1}{n-p-1}\sum_{i=1}^{n} \left( y_i - \hat{y}_i \right)^2} \end{align}$

() $\begin{align} R^2 = \dfrac{\text{TSS} - \text{RSS}}{\text{TSS}} = 1 - \dfrac{\text{RSS}}{\text{TSS}} \quad \text{where} \quad \text{TSS} = \sum_{i=1}^{n} \left( y_i - \overline{y} \right)^2 \end{align}$

11 / 42

Linear regression

Performance and overfit

As we've seen throughout the course, we need to be careful not to overfit.

12 / 42

Linear regression

Performance and overfit

As we've seen throughout the course, we need to be careful not to overfit.

provides no protection against overfitting—and actually encourages it. $\begin{align} R^2 = 1 - \dfrac{\text{RSS}}{\text{TSS}} \end{align}$ RSS $\downarrow$ and TSS is unchanged. Thus, R2 increases.

12 / 42

Linear regression

Performance and overfit

As we've seen throughout the course, we need to be careful not to overfit.

penalizes additional variables: $\begin{align} \text{RSE}=\sqrt{\dfrac{1}{n-p-1}\text{RSS}} \end{align}$ RSS $\downarrow$ but $p$ increases. Thus, RSE's change is uncertain.

12 / 42

Example

Let's see how R2 and RSE perform with 500 very weak predictors.

To address overfitting, we can compare in- vs. out-of-sample performance.

13 / 42

In-sample R2 mechanically increases as we add predictors.
Out-of-sample R2 does not.

14 / 42

In-sample R2 mechanically increases as we add predictors.
Out-of-sample R2 does not.

15 / 42

What about RSE? Does its penalty help?

16 / 42

Despite its penalty for adding variables, in-sample RSE still can overfit,
as evidenced by out-of-sample RSE.

17 / 42

Despite its penalty for adding variables, in-sample RSE still can overfit,
as evidenced by out-of-sample RSE.

18 / 42

Linear regression

Penalization

RSE is not the only way to penalization the addition of variables.†

† We'll talk about other penalization methods (LASSO and Ridge) shortly.

Adjusted R2 is another classic solution.

$\begin{align} \text{Adjusted }R^2 = 1 - \dfrac{\text{RSS}\color{#6A5ACD}{/(n - p - 1)}}{\text{TSS}\color{#6A5ACD}{/(n-1)}} \end{align}$

Adj. R2 attempts to "fix" R2 by adding a penalty for the number of variables.

19 / 42

Linear regression

Penalization

RSE is not the only way to penalization the addition of variables.†

† We'll talk about other penalization methods (LASSO and Ridge) shortly.

Adjusted R2 is another classic solution.

$\begin{align} \text{Adjusted }R^2 = 1 - \dfrac{\text{RSS}\color{#6A5ACD}{/(n - p - 1)}}{\text{TSS}\color{#6A5ACD}{/(n-1)}} \end{align}$

Adj. R2 attempts to "fix" R2 by adding a penalty for the number of variables.

$\text{RSS}$ always decreases when a new variable is added.

19 / 42

Linear regression

Penalization

RSE is not the only way to penalization the addition of variables.†

† We'll talk about other penalization methods (LASSO and Ridge) shortly.

Adjusted R2 is another classic solution.

$\begin{align} \text{Adjusted }R^2 = 1 - \dfrac{\text{RSS}\color{#6A5ACD}{/(n - p - 1)}}{\text{TSS}\color{#6A5ACD}{/(n-1)}} \end{align}$

Adj. R2 attempts to "fix" R2 by adding a penalty for the number of variables.

$\text{RSS}$ always decreases when a new variable is added.
$\color{#6A5ACD}{\text{RSS}/(n-p-1)}$ may increase or decrease with a new variable.

19 / 42

However, in-sample adjusted R2 still can overfit.
Illustrated by out-of-sample R2.

20 / 42

However, in-sample adjusted R2 still can overfit.
Illustrated by out-of-sample adjusted R2.

21 / 42

Model selection

A better way?

R2, adjusted R2, and RSE each offer some flavor of model fit, but they appear limited in their abilities to prevent overfitting.

22 / 42

Model selection

A better way?

R2, adjusted R2, and RSE each offer some flavor of model fit, but they appear limited in their abilities to prevent overfitting.

We want a method to optimally select a (linear) model—balancing variance and bias and avoiding overfit.

22 / 42

Model selection

A better way?

R2, adjusted R2, and RSE each offer some flavor of model fit, but they appear limited in their abilities to prevent overfitting.

We want a method to optimally select a (linear) model—balancing variance and bias and avoiding overfit.

We'll discuss two (related) methods today:

chooses a (sub)set of our $p$ potential predictors
fits a model using all $p$ variables but "shrinks" its coefficients

22 / 42

Model selection

Subset selection

In subset selection, we

whittle down the $p$ potential predictors (using some magic/algorithm)
estimate the chosen linear model using OLS

23 / 42

Model selection

Subset selection

In subset selection, we

whittle down the $p$ potential predictors (using some magic/algorithm)
estimate the chosen linear model using OLS

How do we do the whittling (selection)?

23 / 42

Model selection

Subset selection

In subset selection, we

whittle down the $p$ potential predictors (using some magic/algorithm)
estimate the chosen linear model using OLS

How do we do the whittling (selection)? We've go options.

Best subset selection fits a model for every possible subset.
Forward stepwise selection starts with only an intercept and tries to build up to the best model (using some fit criterion).
starts with all $p$ variables and tries to drop variables until it hits the best model (using some fit criterion).
Hybrid approaches are what their name implies (i.e., hybrids).

23 / 42

Model selection

Best subset selection

Best subset selection is based upon a simple idea: Estimate a model for every possible subset of variables; then compare their performances.

24 / 42

Model selection

Best subset selection

Best subset selection is based upon a simple idea: Estimate a model for every possible subset of variables; then compare their performances.

Q So what's the problem? (Why do we need other selection methods?)

24 / 42

Model selection

Best subset selection

Best subset selection is based upon a simple idea: Estimate a model for every possible subset of variables; then compare their performances.

Q So what's the problem? (Why do we need other selection methods?)
"a model for " can mean $\left( 2^p \right)$ of models.

24 / 42

Model selection

Best subset selection

Best subset selection is based upon a simple idea: Estimate a model for every possible subset of variables; then compare their performances.

Q So what's the problem? (Why do we need other selection methods?)
"a model for " can mean $\left( 2^p \right)$ of models.

E.g.,

10 predictors $\rightarrow$ 1,024 models to fit
25 predictors $\rightarrow$ >33.5 million models to fit
100 predictors $\rightarrow$ ~1.5 trillion models to fit

24 / 42

Model selection

Best subset selection

Best subset selection is based upon a simple idea: Estimate a model for every possible subset of variables; then compare their performances.

Q So what's the problem? (Why do we need other selection methods?)
"a model for " can mean $\left( 2^p \right)$ of models.

E.g.,

10 predictors $\rightarrow$ 1,024 models to fit
25 predictors $\rightarrow$ >33.5 million models to fit
100 predictors $\rightarrow$ ~1.5 trillion models to fit

Even with plentiful, cheap computational power, we can run into barriers.

24 / 42

Model selection

Best subset selection

Computational constraints aside, we can implement best subset selection as

Define $\mathcal{M}_0$ as the model with no predictors.
For $k$ in 1 to $p$ :
- Fit every possible model with $k$ predictors.
- Define $\mathcal{M}_k$ as the "best" model with $k$ predictors.
Select the "best" model from $\mathcal{M}_0,\,\ldots,\,\mathcal{M}_p$ .

As we've seen, RSS declines (and R increases) with $p$ , so we should use a cross-validated measure of model performance in step 3.†

† Back to our distinction between test vs. training performance.

25 / 42

Model selection

Example dataset: `Credit`

We're going to use the Credit dataset from ISL's R package ISLR.

26 / 42

Model selection

Example dataset: `Credit`

We're going to use the Credit dataset from ISL's R package ISLR.

ID	Income	Limit	Rating	Cards	Age	Education	Gender	Student	Married	Ethnicity	Balance
1	14.891	3606	283	2	34	11	Male	No	Yes	Caucasian	333
2	106.025	6645	483	3	82	15	Female	Yes	Yes	Asian	903
3	104.593	7075	514	4	71	11	Male	No	No	Asian	580
4	148.924	9504	681	3	36	11	Female	No	No	Asian	964
5	55.882	4897	357	2	68	16	Male	No	Yes	Caucasian	331
6	80.18	8047	569	4	77	10	Male	No	No	Caucasian	1151
7	20.996	3388	259	2	37	12	Female	No	No	African American	203
8	71.408	7114	512	2	87	9	Male	No	No	Asian	872
9	15.125	3300	266	5	66	13	Female	No	No	Caucasian	279
10	71.061	6819	491	3	41	19	Female	Yes	Yes	African American	1350

The Credit dataset has 400 observations on 12 variables.

26 / 42

Model selection

Example dataset: `Credit`

We need to pre-process the dataset before we can select a model...

income	limit	rating	cards	age	education	i_female	i_student	i_married	i_asian	i_african_american	balance
14.891	3606	283	2	34	11	0	0	1	0	0	333
106.025	6645	483	3	82	15	1	1	1	1	0	903
104.593	7075	514	4	71	11	0	0	0	1	0	580
148.924	9504	681	3	36	11	1	0	0	1	0	964
55.882	4897	357	2	68	16	0	0	1	0	0	331
80.18	8047	569	4	77	10	0	0	0	0	0	1151
20.996	3388	259	2	37	12	1	0	0	0	1	203
71.408	7114	512	2	87	9	0	0	0	1	0	872
15.125	3300	266	5	66	13	1	0	0	0	0	279
71.061	6819	491	3	41	19	1	1	1	0	1	1350

Now the dataset on has 400 observations on 12 variables (2,048 subsets).

27 / 42

28 / 42

29 / 42

Model selection

Best subset selection

From here, you would

Estimate cross-validated error for each $\mathcal{M}_k$ .
Choose the $\mathcal{M}_k$ that minimizes the CV error.
Train the chosen model on the full dataset.

30 / 42

Model selection

Best subset selection

Warnings

Computationally intensive
Selected models may not be "right" (squared terms with linear terms)
You need to protect against overfitting when choosing across $\mathcal{M}_k$
Also should worry about overfitting when $p$ is "big"
Dependent upon the variables you provide

Benefits

Comprehensive search across provided variables
Resulting model—when estimated with OLS—has OLS properties
Can be applied to other (non-OLS) estimators

31 / 42

Model selection

Stepwise selection

Stepwise selection provides a less computational intensive alternative to best subset selection.

The basic idea behind stepwise selection

Start with an arbitrary model.
Try to find a "better" model by adding/removing variables.
Repeat.
Stop when you have the best model. (Or choose the best model.)

32 / 42

Model selection

Stepwise selection

Stepwise selection provides a less computational intensive alternative to best subset selection.

The basic idea behind stepwise selection

Start with an arbitrary model.
Try to find a "better" model by adding/removing variables.
Repeat.
Stop when you have the best model. (Or choose the best model.)

The two most-common varieties of stepwise selection:

starts with only intercept $\left( \mathcal{M}_0 \right)$ and adds variables
starts with all variables $\left( \mathcal{M}_p \right)$ and removes variables

32 / 42

Model selection

Forward stepwise selection

The process...

Start with a model with only an intercept (no predictors), $\mathcal{M}_0$ .
For $k=0,\,\ldots,\,p$ :
- Estimate a model for each of the remaining $p-k$ predictors, separately adding the predictors to model $\mathcal{M}_k$ .
- Define $\mathcal{M}_{k+1}$ as the "best" model of the $p-k$ models.
Select the "best" model from $\mathcal{M}_0,\,\ldots,\, \mathcal{M}_p$ .

33 / 42

Model selection

Forward stepwise selection

The process...

Start with a model with only an intercept (no predictors), $\mathcal{M}_0$ .
For $k=0,\,\ldots,\,p$ :
- Estimate a model for each of the remaining $p-k$ predictors, separately adding the predictors to model $\mathcal{M}_k$ .
- Define $\mathcal{M}_{k+1}$ as the "best" model of the $p-k$ models.
Select the "best" model from $\mathcal{M}_0,\,\ldots,\, \mathcal{M}_p$ .

What do we mean by "best"?
2: best is often RSS or R2.
3: best should be a cross-validated fit criterion.

33 / 42

Forward stepwise selection with caret in R

train_forward = train(
  y = credit_dt[["balance"]],
  x = credit_dt %>% dplyr::select(-balance),
  trControl = trainControl(method = "cv", number = 5),
  method = "leapForward",
  tuneGrid = expand.grid(nvmax = 1:11)
)

34 / 42

Forward stepwise selection with caret in R

train_forward = train(
  y = credit_dt[["balance"]],
  x = credit_dt %>% dplyr::select(-balance),
  trControl = trainControl(method = "cv", number = 5),
  method = "leapForward",
  tuneGrid = expand.grid(nvmax = 1:11)
)

N vars	RMSE	R2	MAE
1	232.57	0.745	175.2
2	163.13	0.874	121.9
3	103.31	0.950	83.8
4	101.04	0.952	81.8
5	99.32	0.954	79.6
6	99.68	0.953	80.0
7	99.96	0.953	80.4
8	99.99	0.953	80.4
9	99.85	0.953	80.2
10	99.79	0.953	80.2

34 / 42

Model selection

Backward stepwise selection

The process for backward stepwise selection is quite similar...

35 / 42

Model selection

Backward stepwise selection

The process for backward stepwise selection is quite similar...

Start with a model that includes all $p$ predictors: $\mathcal{M}_p$ .
For $k=p,\, p-1,\, \ldots,\,1$ :
- Estimate $k$ models, where each model removes exactly one of the $k$ predictors from $\mathcal{M}_k$ .
- Define $\mathcal{M}_{k-1}$ as the "best" of the $k$ models.
Select the "best" model from $\mathcal{M}_0,\,\ldots,\, \mathcal{M}_p$ .

35 / 42

Model selection

Backward stepwise selection

The process for backward stepwise selection is quite similar...

Start with a model that includes all $p$ predictors: $\mathcal{M}_p$ .
For $k=p,\, p-1,\, \ldots,\,1$ :
- Estimate $k$ models, where each model removes exactly one of the $k$ predictors from $\mathcal{M}_k$ .
- Define $\mathcal{M}_{k-1}$ as the "best" of the $k$ models.
Select the "best" model from $\mathcal{M}_0,\,\ldots,\, \mathcal{M}_p$ .

What do we mean by "best"?
2: best is often RSS or R2.
3: best should be a cross-validated fit criterion.

35 / 42

Backward stepwise selection with caret in R

train_backward = train(
  y = credit_dt[["balance"]],
  x = credit_dt %>% dplyr::select(-balance),
  trControl = trainControl(method = "cv", number = 5),
  method = "leapBackward",
  tuneGrid = expand.grid(nvmax = 1:11)
)

36 / 42

Backward stepwise selection with caret in R

train_backward = train(
  y = credit_dt[["balance"]],
  x = credit_dt %>% dplyr::select(-balance),
  trControl = trainControl(method = "cv", number = 5),
  method = "leapBackward",
  tuneGrid = expand.grid(nvmax = 1:11)
)

N vars	RMSE	R2	MAE
1	233.06	0.743	177.6
2	165.41	0.871	124.9
3	104.30	0.949	83.8
4	99.88	0.954	79.5
5	99.40	0.954	79.4
6	99.41	0.954	79.4
7	99.64	0.954	79.5
8	100.02	0.953	79.7
9	100.00	0.953	79.9
10	99.84	0.954	79.7

36 / 42

Note: forward and backward step. selection can choose different models.

37 / 42

Model selection

Stepwise selection

Notes on stepwise selection

Less computationally intensive (relative to best subset selection)
- With $p=20$ , BSS fits 1,048,576 models.
- With $p=20$ , foward/backward selection fits 211 models.
There is no guarantee that stepwise selection finds the best model.
Best is defined by your fit criterion (as always).
Again, cross validation is key to avoiding overfitting.

38 / 42

Model selection

Criteria

Which model you choose is a function of how you define "best".

39 / 42

Model selection

Criteria

Which model you choose is a function of how you define "best".

And we have many options...

39 / 42

Model selection

Criteria

Which model you choose is a function of how you define "best".

And we have many options... We've seen RSS, (R)MSE, RSE, MA, R2, Adj. R2.

39 / 42

Model selection

Criteria

Which model you choose is a function of how you define "best".

And we have many options... We've seen RSS, (R)MSE, RSE, MA, R2, Adj. R2.

Of course, there's more. Each the $d$ predictors differently. $\begin{align} C_p &= \frac{1}{n} \left( \text{RSS} + \color{#6A5ACD}{2 d \hat{\sigma}^2 }\right) \\[1ex] \text{AIC} &= \frac{1}{n\hat{\sigma}^2} \left( \text{RSS} + \color{#6A5ACD}{2 d \hat{\sigma}^2 }\right) \\[1ex] \text{BIC} &= \frac{1}{n\hat{\sigma}^2} \left( \text{RSS} + \color{#6A5ACD}{\log(n) d \hat{\sigma}^2 }\right) \end{align}$

39 / 42

Model selection

Criteria

$C_p$ , $AIC$ , and $BIC$ all have rigorous theoretical justifications... the adjusted $R^2$ is not as well motivated in statistical theory

ISL, p. 213

In general, we will stick with cross-validated criteria, but you still need to choose a selection criterion.

40 / 42

Sources

These notes draw upon

An Introduction to Statistical Learning (ISL)
James, Witten, Hastie, and Tibshirani

41 / 42

42 / 42

Help

Keyboard shortcuts

↑, ←, Pg Up, k

Go to previous slide

↓, →, Pg Dn, Space, j

Go to next slide

Home

Go to first slide

End

Go to last slide

Number + Return

Go to specific slide

b / m / f

Toggle blackout / mirrored / fullscreen mode

Clone slideshow

Toggle presenter mode

Restart the presentation timer

?, h

Toggle this help

Lecture 004

Regression strikes back

Edward Rubin

04 February 2020

Admin

Admin

Today

Admin

Admin

Upcoming

Roadmap

Where are we?

Roadmap

Where are we going?

Roadmap

Where are we going?

Roadmap

Why return to regression?

Roadmap

Why return to regression?

Roadmap

Why return to regression?

Linear regression

Linear regression

Regression regression

Linear regression

Fitting the regression line

Linear regression

Fitting the regression line

Linear regression

Performance

Linear regression

Performance and overfit

Linear regression

Performance and overfit

Linear regression

Performance and overfit

Example

Linear regression

Penalization

Linear regression

Penalization

Linear regression

Penalization

Model selection

A better way?

Model selection

A better way?

Model selection

A better way?

Model selection

Subset selection

Model selection

Subset selection

Model selection

Subset selection

Model selection

Best subset selection

Model selection

Best subset selection

Model selection

Best subset selection

Model selection

Best subset selection

Model selection

Best subset selection

Model selection

Best subset selection

Model selection

Example dataset: Credit

Model selection

Example dataset: Credit

Model selection

Example dataset: Credit

Model selection

Best subset selection

Model selection

Best subset selection

Model selection

Stepwise selection

Example dataset: `Credit`

Example dataset: `Credit`

Example dataset: `Credit`