Topic 14: Prediction Methods

class: center, middle, inverse, title-slide

.title[
# Topic 14: Prediction Methods
]
.subtitle[
## Part 2: Learning with tidymodels
]
.author[
### Nick Hagerty <br> ECNS 460/560 <br> Montana State University
]
.date[
### <br> .smallest[*Adapted from <a href="https://github.com/edrubin/EC524W21">“Prediction and machine-learning in econometrics”</a> by Ed Rubin, used with permission. Slides 1-38 are excluded from this resource’s overall CC license.]
]

---

name: toc

# Table of contents

1. [Setup and splitting](#tidymodels)

1. [Recipes](#recipes)

1. [Workflows](#workflows)

1. [Tuning](#tuning)

1. [Prediction](#prediction)

1. [Dependence](#dependence)

---
layout: true
# Setting up tidymodels

---
class: inverse, middle
name: tidymodels

---

`tidymodels` is a set of packages that makes it easy to set up, estimate, tune, and use predictive models.
- There are many other approaches; this is just one.

Load the following packages:

```r
library(pacman)
p_load(
  tidyverse, modeldata, skimr, janitor, summarytools,
  tidymodels, magrittr, glmnet, ISLR
)
```

---

Let's try to **predict people's credit scores,** using other information about them and their credit cards. We'll work with the `Credit` dataset from the `ISLR` package.

.scroll-output-75[
  .smallest[

```r
# Load the credit dataset
data(Credit)
# Take a look at the data
skim(Credit)
```

<table style='width: auto;'
      class='table table-condensed'>
<caption>Data summary</caption>
<tbody>
  <tr>
   <td style="text-align:left;"> Name </td>
   <td style="text-align:left;"> Credit </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Number of rows </td>
   <td style="text-align:left;"> 400 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Number of columns </td>
   <td style="text-align:left;"> 12 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> _______________________ </td>
   <td style="text-align:left;">  </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Column type frequency: </td>
   <td style="text-align:left;">  </td>
  </tr>
  <tr>
   <td style="text-align:left;"> factor </td>
   <td style="text-align:left;"> 4 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> numeric </td>
   <td style="text-align:left;"> 8 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> ________________________ </td>
   <td style="text-align:left;">  </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Group variables </td>
   <td style="text-align:left;"> None </td>
  </tr>
</tbody>
</table>

**Variable type: factor**

**Variable type: numeric**

---
layout: true
# Split the data

---

The first thing we need to do is set aside a (randomly selected) subset of our data for testing.
* Remember, **testing is separate from (cross-)validation.** We only look at the test set once we're happy with our predictive model.

We can use some functions from the `rsample` package within `tidymodels`:

```r
# Set a seed (ensures reproducible results)
set.seed(12345)
# Create an 80/20 split by random sampling
credit_split = Credit |> initial_split(prop = 0.8)
# Grab each subset
credit_train = credit_split |> training()
credit_test  = credit_split |> testing()
credit_split
```

```
#> <Training/Testing/Total>
#> <320/80/400>
```

---
layout: true
# Recipes

---
name: recipes
class: inverse, middle

---

One advantage of `tidymodels` is that it makes it really easy to **clean** a lot of variables at once.

Examples of operations in which this might be useful:
* standardize *all* your numeric variables
* create dummies for *all* your categorical variables
* remove variables that are perfectly determined by other variables
* impute missing values

---

First, we **define the recipe.** This step is like listing the ingredients.
* The syntax is like `lm()`, but we aren't estimating anything yet.
* We're going to predict `Rating`, so it goes on the left-hand side.
* `.` means "all other variables".
* No transformations yet - we're just defining the roles of each variable.

```r
recipe_all = recipe(Rating ~ ., data = credit_train)
recipe_all
```

---

.smaller[
We now can **add cleaning steps** to the recipe to pre-process our data. The tidymodels galaxy has [many](https://recipes.tidymodels.org/reference/index.html) possible steps for
* imputation: `step_impute_mean()`, `step_impute_mode()`
* transformation: `step_log()`, `step_poly()`, `step_mutate()`
* dummying and discretization: `step_dummy()`, `step_discretize()`
* and many other things: `step_center()`, `step_normalize()`, `step_lag()`

To apply any of these steps, you just need to tell the `step_*()` function which variables you want it to target:
* All: `all_vars()`
* Role: `all_predictors()` or `all_outcomes()` or `has_role()`
* Type: `all_nominal()` or `all_numeric()`
* Variable names, or selectors (e.g., `starts_with()` or `contains()`)
* Unselect using the minus sign (-).

If you have an "ID" variable, you can give it an "ID" role using `update_role(id_var, new_role = "ID")`.
]

---

**Add cleaning steps:** Let's write a step that creates indicator variables for each of our predictors that are categorical or string variables.

```r
recipe_all |> step_dummy(all_nominal_predictors())
```

---

**Prep and juice:** To get the processed data frame, we need to add two more things to our recipe:
* `prep()` does the prep work (estimating means for imputation).
* `juice()` applies the preprocessing to the training dataset.

Now the original categorical variables are replaced by indicators for each of their levels.

```r
credit_clean = recipe_all |> 
  step_dummy(all_nominal_predictors()) |> 
  prep() |>
  juice()
credit_clean |> skim()
```

---

Let's set up and apply a more complex recipe:

.small[

```r
credit_recipe = 
  # Define the recipe: Rating predicted by all other vars in credit_train
  recipe(Rating ~ ., data = credit_train) |>
  # Define ID variable (so it's not used in the model)
  update_role(ID, new_role = "ID") |>
  # Impute missing values (use means) for numeric predictors
  step_impute_mean(all_numeric_predictors()) |> 
  # Impute missing values (by k-nearest neighbors) for categorical predictors
  step_impute_knn(all_nominal_predictors(), neighbors = 5) |>
  # Create polynomial terms for numeric predictors
  step_poly(all_numeric_predictors(), degree=2) |>
  # Create indicators for categorical predictors
  step_dummy(all_nominal_predictors()) |>
  # Create interactions (all possible first-degree terms)
  step_interact(~all_predictors():all_predictors()) |>
  # Remove predictors with near-zero variance (improves stability)
  step_nzv(all_predictors())
# Create the cleaned dataset
credit_clean = credit_recipe |> prep() |> juice()
# Skim the cleaned (full) dataset
credit_clean |> skim()
```

<table style='width: auto;'
      class='table table-condensed'>
<caption>Data summary</caption>
<tbody>
  <tr>
   <td style="text-align:left;"> Name </td>
   <td style="text-align:left;"> credit_clean </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Number of rows </td>
   <td style="text-align:left;"> 320 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Number of columns </td>
   <td style="text-align:left;"> 145 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> _______________________ </td>
   <td style="text-align:left;">  </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Column type frequency: </td>
   <td style="text-align:left;">  </td>
  </tr>
  <tr>
   <td style="text-align:left;"> numeric </td>
   <td style="text-align:left;"> 145 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> ________________________ </td>
   <td style="text-align:left;">  </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Group variables </td>
   <td style="text-align:left;"> None </td>
  </tr>
</tbody>
</table>

**Variable type: numeric**

<table>
 <thead>
  <tr>
   <th style="text-align:left;"> skim_variable </th>
   <th style="text-align:right;"> n_missing </th>
   <th style="text-align:right;"> complete_rate </th>
   <th style="text-align:right;"> mean </th>
   <th style="text-align:right;"> sd </th>
   <th style="text-align:right;"> p0 </th>
   <th style="text-align:right;"> p25 </th>
   <th style="text-align:right;"> p50 </th>
   <th style="text-align:right;"> p75 </th>
   <th style="text-align:right;"> p100 </th>
   <th style="text-align:left;"> hist </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> ID </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 196.50 </td>
   <td style="text-align:right;"> 117.56 </td>
   <td style="text-align:right;"> 1.00 </td>
   <td style="text-align:right;"> 91.75 </td>
   <td style="text-align:right;"> 196.00 </td>
   <td style="text-align:right;"> 296.00 </td>
   <td style="text-align:right;"> 400.00 </td>
   <td style="text-align:left;"> ▇▇▇▆▇ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Rating </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 355.36 </td>
   <td style="text-align:right;"> 155.36 </td>
   <td style="text-align:right;"> 93.00 </td>
   <td style="text-align:right;"> 244.50 </td>
   <td style="text-align:right;"> 344.00 </td>
   <td style="text-align:right;"> 442.75 </td>
   <td style="text-align:right;"> 982.00 </td>
   <td style="text-align:left;"> ▆▇▃▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Income_poly_1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.06 </td>
   <td style="text-align:right;"> -0.06 </td>
   <td style="text-align:right;"> -0.04 </td>
   <td style="text-align:right;"> -0.02 </td>
   <td style="text-align:right;"> 0.02 </td>
   <td style="text-align:right;"> 0.22 </td>
   <td style="text-align:left;"> ▇▂▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Income_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.06 </td>
   <td style="text-align:right;"> -0.07 </td>
   <td style="text-align:right;"> -0.04 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.03 </td>
   <td style="text-align:right;"> 0.31 </td>
   <td style="text-align:left;"> ▇▆▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Limit_poly_1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.06 </td>
   <td style="text-align:right;"> -0.09 </td>
   <td style="text-align:right;"> -0.04 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.03 </td>
   <td style="text-align:right;"> 0.22 </td>
   <td style="text-align:left;"> ▆▇▃▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Limit_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.06 </td>
   <td style="text-align:right;"> -0.04 </td>
   <td style="text-align:right;"> -0.04 </td>
   <td style="text-align:right;"> -0.02 </td>
   <td style="text-align:right;"> 0.02 </td>
   <td style="text-align:right;"> 0.44 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Cards_poly_1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.06 </td>
   <td style="text-align:right;"> -0.08 </td>
   <td style="text-align:right;"> -0.04 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.05 </td>
   <td style="text-align:right;"> 0.21 </td>
   <td style="text-align:left;"> ▇▆▅▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Cards_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.06 </td>
   <td style="text-align:right;"> -0.04 </td>
   <td style="text-align:right;"> -0.04 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.43 </td>
   <td style="text-align:left;"> ▇▂▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Age_poly_1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.06 </td>
   <td style="text-align:right;"> -0.10 </td>
   <td style="text-align:right;"> -0.04 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:right;"> 0.14 </td>
   <td style="text-align:left;"> ▆▇▇▇▂ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Age_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.06 </td>
   <td style="text-align:right;"> -0.06 </td>
   <td style="text-align:right;"> -0.05 </td>
   <td style="text-align:right;"> -0.02 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:right;"> 0.28 </td>
   <td style="text-align:left;"> ▇▃▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Education_poly_1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.06 </td>
   <td style="text-align:right;"> -0.15 </td>
   <td style="text-align:right;"> -0.04 </td>
   <td style="text-align:right;"> 0.01 </td>
   <td style="text-align:right;"> 0.05 </td>
   <td style="text-align:right;"> 0.12 </td>
   <td style="text-align:left;"> ▂▅▇▇▂ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Education_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.06 </td>
   <td style="text-align:right;"> -0.05 </td>
   <td style="text-align:right;"> -0.04 </td>
   <td style="text-align:right;"> -0.03 </td>
   <td style="text-align:right;"> 0.03 </td>
   <td style="text-align:right;"> 0.26 </td>
   <td style="text-align:left;"> ▇▂▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Balance_poly_1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.06 </td>
   <td style="text-align:right;"> -0.06 </td>
   <td style="text-align:right;"> -0.05 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:right;"> 0.18 </td>
   <td style="text-align:left;"> ▇▅▃▂▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Balance_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.06 </td>
   <td style="text-align:right;"> -0.06 </td>
   <td style="text-align:right;"> -0.05 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.05 </td>
   <td style="text-align:right;"> 0.39 </td>
   <td style="text-align:left;"> ▇▃▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Gender_Female </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.50 </td>
   <td style="text-align:right;"> 0.50 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.50 </td>
   <td style="text-align:right;"> 1.00 </td>
   <td style="text-align:right;"> 1.00 </td>
   <td style="text-align:left;"> ▇▁▁▁▇ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Student_Yes </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.11 </td>
   <td style="text-align:right;"> 0.31 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 1.00 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Married_Yes </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.62 </td>
   <td style="text-align:right;"> 0.49 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 1.00 </td>
   <td style="text-align:right;"> 1.00 </td>
   <td style="text-align:right;"> 1.00 </td>
   <td style="text-align:left;"> ▅▁▁▁▇ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Ethnicity_Asian </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.25 </td>
   <td style="text-align:right;"> 0.44 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 1.00 </td>
   <td style="text-align:right;"> 1.00 </td>
   <td style="text-align:left;"> ▇▁▁▁▃ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Ethnicity_Caucasian </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.49 </td>
   <td style="text-align:right;"> 0.50 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 1.00 </td>
   <td style="text-align:right;"> 1.00 </td>
   <td style="text-align:left;"> ▇▁▁▁▇ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Income_poly_1_x_Income_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.07 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Income_poly_1_x_Limit_poly_1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.05 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Income_poly_1_x_Limit_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.01 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.09 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Income_poly_1_x_Cards_poly_1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.03 </td>
   <td style="text-align:left;"> ▁▇▂▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Income_poly_1_x_Cards_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.02 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:left;"> ▁▇▃▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Income_poly_1_x_Age_poly_1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.03 </td>
   <td style="text-align:left;"> ▁▇▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Income_poly_1_x_Age_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.06 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Income_poly_1_x_Education_poly_1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.02 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.01 </td>
   <td style="text-align:left;"> ▁▁▇▅▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Income_poly_1_x_Education_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.01 </td>
   <td style="text-align:left;"> ▁▇▅▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Income_poly_1_x_Balance_poly_1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Income_poly_1_x_Balance_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.01 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.08 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Income_poly_1_x_Gender_Female </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:right;"> -0.06 </td>
   <td style="text-align:right;"> -0.02 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.21 </td>
   <td style="text-align:left;"> ▅▇▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Income_poly_1_x_Student_Yes </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.02 </td>
   <td style="text-align:right;"> -0.05 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.21 </td>
   <td style="text-align:left;"> ▁▇▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Income_poly_1_x_Married_Yes </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.05 </td>
   <td style="text-align:right;"> -0.06 </td>
   <td style="text-align:right;"> -0.03 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.22 </td>
   <td style="text-align:left;"> ▆▇▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Income_poly_1_x_Ethnicity_Asian </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.03 </td>
   <td style="text-align:right;"> -0.06 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.21 </td>
   <td style="text-align:left;"> ▂▇▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Income_poly_1_x_Ethnicity_Caucasian </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:right;"> -0.06 </td>
   <td style="text-align:right;"> -0.02 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.22 </td>
   <td style="text-align:left;"> ▅▇▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Income_poly_2_x_Limit_poly_1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.01 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.07 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Income_poly_2_x_Limit_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.14 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Income_poly_2_x_Cards_poly_1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.01 </td>
   <td style="text-align:left;"> ▁▂▇▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Income_poly_2_x_Cards_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.02 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.03 </td>
   <td style="text-align:left;"> ▁▇▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Income_poly_2_x_Age_poly_1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Income_poly_2_x_Age_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.01 </td>
   <td style="text-align:right;"> -0.02 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.09 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Income_poly_2_x_Education_poly_1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.03 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.02 </td>
   <td style="text-align:left;"> ▁▁▇▃▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Income_poly_2_x_Education_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.02 </td>
   <td style="text-align:left;"> ▁▇▃▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Income_poly_2_x_Balance_poly_1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.06 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Income_poly_2_x_Balance_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.01 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.12 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Income_poly_2_x_Gender_Female </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:right;"> -0.07 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.30 </td>
   <td style="text-align:left;"> ▇▂▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Income_poly_2_x_Student_Yes </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.02 </td>
   <td style="text-align:right;"> -0.07 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.30 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Income_poly_2_x_Married_Yes </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.05 </td>
   <td style="text-align:right;"> -0.07 </td>
   <td style="text-align:right;"> -0.02 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.01 </td>
   <td style="text-align:right;"> 0.31 </td>
   <td style="text-align:left;"> ▇▃▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Income_poly_2_x_Ethnicity_Asian </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.03 </td>
   <td style="text-align:right;"> -0.07 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.30 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Income_poly_2_x_Ethnicity_Caucasian </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:right;"> -0.07 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.31 </td>
   <td style="text-align:left;"> ▇▂▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Limit_poly_1_x_Limit_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.01 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.10 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Limit_poly_1_x_Cards_poly_1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.03 </td>
   <td style="text-align:left;"> ▁▇▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Limit_poly_1_x_Cards_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:left;"> ▇▃▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Limit_poly_1_x_Age_poly_1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.03 </td>
   <td style="text-align:left;"> ▅▇▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Limit_poly_1_x_Age_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.06 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Limit_poly_1_x_Education_poly_1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.02 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.01 </td>
   <td style="text-align:left;"> ▁▁▇▂▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Limit_poly_1_x_Education_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.02 </td>
   <td style="text-align:left;"> ▁▃▇▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Limit_poly_1_x_Balance_poly_1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Limit_poly_1_x_Balance_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.01 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.09 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Limit_poly_1_x_Gender_Female </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:right;"> -0.09 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.18 </td>
   <td style="text-align:left;"> ▂▇▂▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Limit_poly_1_x_Student_Yes </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.02 </td>
   <td style="text-align:right;"> -0.09 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.12 </td>
   <td style="text-align:left;"> ▁▁▇▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Limit_poly_1_x_Married_Yes </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:right;"> -0.09 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.01 </td>
   <td style="text-align:right;"> 0.22 </td>
   <td style="text-align:left;"> ▂▇▂▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Limit_poly_1_x_Ethnicity_Asian </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.03 </td>
   <td style="text-align:right;"> -0.09 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.18 </td>
   <td style="text-align:left;"> ▁▇▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Limit_poly_1_x_Ethnicity_Caucasian </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:right;"> -0.09 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.22 </td>
   <td style="text-align:left;"> ▂▇▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Limit_poly_2_x_Cards_poly_1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.02 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.03 </td>
   <td style="text-align:left;"> ▁▇▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Limit_poly_2_x_Cards_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.02 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:left;"> ▁▇▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Limit_poly_2_x_Age_poly_1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.06 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Limit_poly_2_x_Age_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.01 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.12 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Limit_poly_2_x_Education_poly_1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.02 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.03 </td>
   <td style="text-align:left;"> ▁▁▇▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Limit_poly_2_x_Education_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.02 </td>
   <td style="text-align:left;"> ▁▅▇▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Limit_poly_2_x_Balance_poly_1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.01 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.08 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Limit_poly_2_x_Balance_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.01 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.17 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Limit_poly_2_x_Gender_Female </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:right;"> -0.04 </td>
   <td style="text-align:right;"> -0.02 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.25 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Limit_poly_2_x_Student_Yes </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.02 </td>
   <td style="text-align:right;"> -0.04 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.12 </td>
   <td style="text-align:left;"> ▁▇▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Limit_poly_2_x_Married_Yes </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.05 </td>
   <td style="text-align:right;"> -0.04 </td>
   <td style="text-align:right;"> -0.03 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.44 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Limit_poly_2_x_Ethnicity_Asian </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.03 </td>
   <td style="text-align:right;"> -0.04 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.25 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Limit_poly_2_x_Ethnicity_Caucasian </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:right;"> -0.04 </td>
   <td style="text-align:right;"> -0.02 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.44 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Cards_poly_1_x_Cards_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.01 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.09 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Cards_poly_1_x_Age_poly_1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.02 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.01 </td>
   <td style="text-align:left;"> ▁▁▇▃▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Cards_poly_1_x_Age_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.02 </td>
   <td style="text-align:left;"> ▁▇▅▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Cards_poly_1_x_Education_poly_1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.01 </td>
   <td style="text-align:left;"> ▁▂▇▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Cards_poly_1_x_Education_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.02 </td>
   <td style="text-align:left;"> ▁▇▃▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Cards_poly_1_x_Balance_poly_1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.02 </td>
   <td style="text-align:left;"> ▂▇▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Cards_poly_1_x_Balance_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.03 </td>
   <td style="text-align:left;"> ▁▇▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Cards_poly_1_x_Gender_Female </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:right;"> -0.08 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.17 </td>
   <td style="text-align:left;"> ▂▇▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Cards_poly_1_x_Married_Yes </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:right;"> -0.08 </td>
   <td style="text-align:right;"> -0.04 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.17 </td>
   <td style="text-align:left;"> ▃▇▂▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Cards_poly_1_x_Ethnicity_Asian </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.03 </td>
   <td style="text-align:right;"> -0.08 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.17 </td>
   <td style="text-align:left;"> ▁▇▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Cards_poly_1_x_Ethnicity_Caucasian </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:right;"> -0.08 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.21 </td>
   <td style="text-align:left;"> ▂▇▂▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Cards_poly_2_x_Age_poly_1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.04 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.02 </td>
   <td style="text-align:left;"> ▁▁▁▇▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Cards_poly_2_x_Age_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.03 </td>
   <td style="text-align:left;"> ▂▇▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Cards_poly_2_x_Education_poly_1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.02 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.02 </td>
   <td style="text-align:left;"> ▁▁▇▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Cards_poly_2_x_Education_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.02 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.02 </td>
   <td style="text-align:left;"> ▁▁▇▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Cards_poly_2_x_Balance_poly_1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.03 </td>
   <td style="text-align:left;"> ▇▂▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Cards_poly_2_x_Balance_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.03 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.05 </td>
   <td style="text-align:left;"> ▁▇▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Cards_poly_2_x_Gender_Female </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.03 </td>
   <td style="text-align:right;"> -0.04 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.25 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Cards_poly_2_x_Married_Yes </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:right;"> -0.04 </td>
   <td style="text-align:right;"> -0.04 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.25 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Cards_poly_2_x_Ethnicity_Asian </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.03 </td>
   <td style="text-align:right;"> -0.04 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.25 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Cards_poly_2_x_Ethnicity_Caucasian </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:right;"> -0.04 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.43 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Age_poly_1_x_Age_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.02 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:left;"> ▁▇▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Age_poly_1_x_Education_poly_1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.01 </td>
   <td style="text-align:left;"> ▁▂▇▂▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Age_poly_1_x_Education_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.02 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.01 </td>
   <td style="text-align:left;"> ▁▁▇▃▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Age_poly_1_x_Balance_poly_1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.02 </td>
   <td style="text-align:left;"> ▂▇▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Age_poly_1_x_Balance_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.05 </td>
   <td style="text-align:left;"> ▇▃▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Age_poly_1_x_Gender_Female </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:right;"> -0.10 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.11 </td>
   <td style="text-align:left;"> ▁▂▇▂▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Age_poly_1_x_Married_Yes </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:right;"> -0.10 </td>
   <td style="text-align:right;"> -0.03 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.01 </td>
   <td style="text-align:right;"> 0.14 </td>
   <td style="text-align:left;"> ▂▂▇▂▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Age_poly_1_x_Ethnicity_Asian </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.03 </td>
   <td style="text-align:right;"> -0.10 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.10 </td>
   <td style="text-align:left;"> ▁▁▇▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Age_poly_1_x_Ethnicity_Caucasian </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:right;"> -0.10 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.14 </td>
   <td style="text-align:left;"> ▁▂▇▂▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Age_poly_2_x_Education_poly_1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.02 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.02 </td>
   <td style="text-align:left;"> ▁▂▇▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Age_poly_2_x_Education_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.02 </td>
   <td style="text-align:left;"> ▁▇▃▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Age_poly_2_x_Balance_poly_1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.05 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Age_poly_2_x_Balance_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.01 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.11 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Age_poly_2_x_Gender_Female </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:right;"> -0.06 </td>
   <td style="text-align:right;"> -0.02 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.18 </td>
   <td style="text-align:left;"> ▃▇▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Age_poly_2_x_Married_Yes </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:right;"> -0.06 </td>
   <td style="text-align:right;"> -0.03 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.28 </td>
   <td style="text-align:left;"> ▇▂▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Age_poly_2_x_Ethnicity_Asian </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.03 </td>
   <td style="text-align:right;"> -0.06 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.13 </td>
   <td style="text-align:left;"> ▂▇▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Age_poly_2_x_Ethnicity_Caucasian </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:right;"> -0.06 </td>
   <td style="text-align:right;"> -0.02 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.28 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Education_poly_1_x_Education_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.01 </td>
   <td style="text-align:right;"> -0.04 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.02 </td>
   <td style="text-align:left;"> ▁▁▁▇▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Education_poly_1_x_Balance_poly_1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.01 </td>
   <td style="text-align:left;"> ▁▁▇▃▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Education_poly_1_x_Balance_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.02 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.03 </td>
   <td style="text-align:left;"> ▁▃▇▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Education_poly_1_x_Gender_Female </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:right;"> -0.15 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.01 </td>
   <td style="text-align:right;"> 0.12 </td>
   <td style="text-align:left;"> ▁▁▇▂▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Education_poly_1_x_Married_Yes </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:right;"> -0.15 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.03 </td>
   <td style="text-align:right;"> 0.12 </td>
   <td style="text-align:left;"> ▁▂▇▃▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Education_poly_1_x_Ethnicity_Caucasian </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:right;"> -0.15 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.10 </td>
   <td style="text-align:left;"> ▁▁▇▂▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Education_poly_2_x_Balance_poly_1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.02 </td>
   <td style="text-align:left;"> ▁▃▇▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Education_poly_2_x_Balance_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.02 </td>
   <td style="text-align:left;"> ▁▇▅▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Education_poly_2_x_Gender_Female </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:right;"> -0.05 </td>
   <td style="text-align:right;"> -0.03 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.26 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Education_poly_2_x_Married_Yes </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:right;"> -0.05 </td>
   <td style="text-align:right;"> -0.03 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.26 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Education_poly_2_x_Ethnicity_Caucasian </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:right;"> -0.05 </td>
   <td style="text-align:right;"> -0.03 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.26 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Balance_poly_1_x_Balance_poly_2 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.07 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Balance_poly_1_x_Gender_Female </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:right;"> -0.06 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.15 </td>
   <td style="text-align:left;"> ▃▇▂▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Balance_poly_1_x_Student_Yes </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.02 </td>
   <td style="text-align:right;"> -0.06 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.14 </td>
   <td style="text-align:left;"> ▁▇▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Balance_poly_1_x_Married_Yes </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:right;"> -0.06 </td>
   <td style="text-align:right;"> -0.02 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.01 </td>
   <td style="text-align:right;"> 0.18 </td>
   <td style="text-align:left;"> ▅▇▂▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Balance_poly_1_x_Ethnicity_Asian </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.03 </td>
   <td style="text-align:right;"> -0.06 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.15 </td>
   <td style="text-align:left;"> ▁▇▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Balance_poly_1_x_Ethnicity_Caucasian </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:right;"> -0.06 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.18 </td>
   <td style="text-align:left;"> ▃▇▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Balance_poly_2_x_Gender_Female </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:right;"> -0.06 </td>
   <td style="text-align:right;"> -0.01 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.26 </td>
   <td style="text-align:left;"> ▇▂▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Balance_poly_2_x_Student_Yes </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.02 </td>
   <td style="text-align:right;"> -0.06 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.21 </td>
   <td style="text-align:left;"> ▁▇▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Balance_poly_2_x_Married_Yes </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.05 </td>
   <td style="text-align:right;"> -0.06 </td>
   <td style="text-align:right;"> -0.03 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.39 </td>
   <td style="text-align:left;"> ▇▂▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Balance_poly_2_x_Ethnicity_Asian </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.03 </td>
   <td style="text-align:right;"> -0.06 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.26 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Balance_poly_2_x_Ethnicity_Caucasian </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:right;"> -0.06 </td>
   <td style="text-align:right;"> -0.02 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.39 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Gender_Female_x_Student_Yes </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.07 </td>
   <td style="text-align:right;"> 0.25 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 1.00 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Gender_Female_x_Married_Yes </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.31 </td>
   <td style="text-align:right;"> 0.46 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 1.00 </td>
   <td style="text-align:right;"> 1.00 </td>
   <td style="text-align:left;"> ▇▁▁▁▃ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Gender_Female_x_Ethnicity_Asian </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.14 </td>
   <td style="text-align:right;"> 0.35 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 1.00 </td>
   <td style="text-align:left;"> ▇▁▁▁▂ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Gender_Female_x_Ethnicity_Caucasian </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.23 </td>
   <td style="text-align:right;"> 0.42 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 1.00 </td>
   <td style="text-align:left;"> ▇▁▁▁▂ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Student_Yes_x_Married_Yes </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.05 </td>
   <td style="text-align:right;"> 0.22 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 1.00 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Student_Yes_x_Ethnicity_Caucasian </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.05 </td>
   <td style="text-align:right;"> 0.22 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 1.00 </td>
   <td style="text-align:left;"> ▇▁▁▁▁ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Married_Yes_x_Ethnicity_Asian </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.17 </td>
   <td style="text-align:right;"> 0.38 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 1.00 </td>
   <td style="text-align:left;"> ▇▁▁▁▂ </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Married_Yes_x_Ethnicity_Caucasian </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 0.32 </td>
   <td style="text-align:right;"> 0.47 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 1.00 </td>
   <td style="text-align:right;"> 1.00 </td>
   <td style="text-align:left;"> ▇▁▁▁▃ </td>
  </tr>
</tbody>
</table>
]

---
layout: true
# Workflows

---
name: workflows
class: inverse, middle

---

After preprocessing, the next step is to **define a model.** This includes:
* The **model specification** (the general type of model).
* The **engine** (the specific function/package).

Here's a simple linear regression using `lm()`:

```r
model_lm = 
  linear_reg() |>
  set_engine("lm")
```

---

Last, we want to **set up any resampling** methods we're going to apply.

Define folds for 5-fold cross-validation:

```r
credit_cv = credit_train |> vfold_cv(v = 5)
tidy(credit_cv)
```

```
#> # A tibble: 1,600 × 3
#>      Row Data     Fold 
#>    <int> <chr>    <chr>
#>  1     1 Analysis Fold2
#>  2     1 Analysis Fold3
#>  3     1 Analysis Fold4
#>  4     1 Analysis Fold5
#>  5     2 Analysis Fold1
#>  6     2 Analysis Fold2
#>  7     2 Analysis Fold3
#>  8     2 Analysis Fold5
#>  9     3 Analysis Fold1
#> 10     3 Analysis Fold2
#> # ℹ 1,590 more rows
```

---

Now we can **put it all together** with `workflow()`.

```r
fit_lm_cv =
  workflow() |>                 # Define a workflow
  add_model(model_lm) |>        # Choose the model
  add_recipe(credit_recipe) |>  # Clean the data
  fit_resamples(credit_cv)       # Estimate & cross-validate
# Check the performance
fit_lm_cv |> collect_metrics()
```

```
#> # A tibble: 2 × 6
#>   .metric .estimator   mean     n std_err .config             
#>   <chr>   <chr>       <dbl> <int>   <dbl> <chr>               
#> 1 rmse    standard   26.9       5 3.35    Preprocessor1_Model1
#> 2 rsq     standard    0.969     5 0.00675 Preprocessor1_Model1
```

---

To see the performance within each fold:

```r
fit_lm_cv |> collect_metrics(summarize = F)
```

```
#> # A tibble: 10 × 5
#>    id    .metric .estimator .estimate .config             
#>    <chr> <chr>   <chr>          <dbl> <chr>               
#>  1 Fold1 rmse    standard      39.3   Preprocessor1_Model1
#>  2 Fold1 rsq     standard       0.955 Preprocessor1_Model1
#>  3 Fold2 rmse    standard      28.5   Preprocessor1_Model1
#>  4 Fold2 rsq     standard       0.949 Preprocessor1_Model1
#>  5 Fold3 rmse    standard      22.6   Preprocessor1_Model1
#>  6 Fold3 rsq     standard       0.980 Preprocessor1_Model1
#>  7 Fold4 rmse    standard      20.7   Preprocessor1_Model1
#>  8 Fold4 rsq     standard       0.980 Preprocessor1_Model1
#>  9 Fold5 rmse    standard      23.5   Preprocessor1_Model1
#> 10 Fold5 rsq     standard       0.979 Preprocessor1_Model1
```

---
layout: false
class: inverse, middle
# Tuning

---
layout: true
# Lasso regression example

---
name: tuning

In lasso regression, we don't just want to estimate one model, we also want to **tune** the penalty `\(\lambda\)`. That means:
- Estimate the same regression for different values of `\(\lambda\)`.
- Choose the one that performs best in cross-validation.

Remember, we also need to standardize all our predictors beforehand.
* Lucky for us, R's `glmnet` engine standardizes everything automatically by default.

---

The first change to make is the **model:**

```r
model_lasso = 
  linear_reg(penalty = tune(), mixture = 1) |> 
  set_engine("glmnet")
```

- `penalty = tune()` tells R what parameter we want to tune.
- Set `mixture = 1` means `\(\alpha=1\)` (for ridge, set `mixture = 0`).
- Use the `glmnet` engine.

---

**Then,** instead of using `fit_resamples()`, we're going to `tune_grid()`.

For that we need to define a set of `\(\lambda\)`'s to try, spanning several orders of magnitude. Try this:

```r
lambdas = 10 ^ seq(from = 5, to = -2, length = 1e3)
```

**Now** we can calculate cross-validated MSE for each value of `\(\lambda\)`:

```r
workflow_lasso = workflow() |>
  add_model(model_lasso) |>
  add_recipe(credit_recipe)
lasso_cv = workflow_lasso |>
  tune_grid(
    credit_cv,
    grid = data.frame(penalty = lambdas),
    metrics = metric_set(rmse)
  )
```

---

Which values of `\(\lambda\)` performed best?

```r
lasso_cv |> show_best()
```

```
#> # A tibble: 5 × 7
#>   penalty .metric .estimator  mean     n std_err .config                
#>     <dbl> <chr>   <chr>      <dbl> <int>   <dbl> <chr>                  
#> 1    1.17 rmse    standard    10.6     5   0.325 Preprocessor1_Model0296
#> 2    1.19 rmse    standard    10.6     5   0.321 Preprocessor1_Model0297
#> 3    1.21 rmse    standard    10.6     5   0.318 Preprocessor1_Model0298
#> 4    1.15 rmse    standard    10.6     5   0.328 Preprocessor1_Model0295
#> 5    1.13 rmse    standard    10.6     5   0.332 Preprocessor1_Model0294
```

---

Let's plot the results for all values of `\(\lambda\)`:

```r
autoplot(lasso_cv, metric = "rmse")
```

---
layout: false
# Ridge example

What if we want to use **ridge** instead of lasso? Just switch `mixture = 0`.

```r
model_ridge = 
  linear_reg(penalty = tune(), mixture = 0) |> 
  set_engine("glmnet")
workflow_ridge = workflow() |>
  add_model(model_ridge) |>
  add_recipe(credit_recipe)
credit_cv = credit_train |> vfold_cv(v = 5)
ridge_cv = workflow_ridge |>
  tune_grid(
    credit_cv,
    grid = data.frame(penalty = lambdas),
    metrics = metric_set(rmse)
  )
autoplot(ridge_cv, metric = "rmse")
ridge_cv |> show_best()
```

---
# Elasticnet example

We can use `tune()` to cross validate both `\(\alpha\)` and `\(\lambda\)`.

.note[Note] You need to consider all combinations of the two parameters.
<br>This combination can create *a lot* of models to estimate.

1,000 values of `\(\lambda\)` `\(\times\)` 1,000 values of `\(\alpha\)` =
- 1,000,000 models to estimate
- 5,000,000 if you're doing 5-fold CV

To keep your computer from going forever, use a **coarser grid** (i.e., try fewer values).

---
layout: false
class: clear, middle

**Cross-validating elasticnet:**

```r
# Ranges of λ and α
lambdas = 10 ^ seq(from = 5, to = -2, length = 1e2)
alphas = seq(from = 0, to = 1, by = 0.1)

# Define the elasticnet model
model_net = linear_reg(penalty = tune(), mixture = tune()) |>
  set_engine("glmnet")

# Define the workflow
workflow_net = workflow() |>
  add_recipe(credit_recipe) |>
  add_model(model_net)

# CV elasticnet with our ranges of lambdas and alphas
cv_net = 
  workflow_net |>
  tune_grid(
    credit_cv,
    grid = expand_grid(mixture = alphas, penalty = lambdas),
    metrics = metric_set(rmse)
  )
```

---
layout: false
class: clear, middle

**Alternative approach:** `grid_regular()` automatically chooses sensible values of your parameters to try.

```r
# Define the elasticnet model
model_net = linear_reg(penalty = tune(), mixture = tune()) |>
  set_engine("glmnet")

# Define the workflow
workflow_net = workflow() |>
  add_recipe(credit_recipe) |>
  add_model(model_net)

# CV elasticnet with grid_regular()
cv_net = 
  workflow_net |>
  tune_grid(
    credit_cv,
*   grid = grid_regular(mixture(), penalty(), levels=5:5),
    metrics = metric_set(rmse)
  )
```

---
layout: true
# Prediction

---
name: prediction
class: inverse, middle

---

After tuning the penalty through cross-validation, we need to:
1. **Tell** R which model we're choosing to carry forward.
1. **Fit** the model on the *entire* training dataset (not just 4 of 5 folds).
1. **Make predictions** in the testing dataset.

---

Tell R which model we're choosing:

.smaller[

```r
final_lasso = 
  workflow_lasso |>
  finalize_workflow(select_best(lasso_cv, metric = "rmse"))
final_lasso
```

```
#> ══ Workflow ════════════════════════════════════════════════════════════════════
#> Preprocessor: Recipe
#> Model: linear_reg()
#> 
#> ── Preprocessor ────────────────────────────────────────────────────────────────
#> 6 Recipe Steps
#> 
#> • step_impute_mean()
#> • step_impute_knn()
#> • step_poly()
#> • step_dummy()
#> • step_interact()
#> • step_nzv()
#> 
#> ── Model ───────────────────────────────────────────────────────────────────────
#> Linear Regression Model Specification (regression)
#> 
#> Main Arguments:
#>   penalty = 1.16698981861715
#>   mixture = 1
#> 
#> Computational engine: glmnet
```
]

---

Fit the model and make predictions in the test set:

```r
final_fit_lasso = final_lasso |> last_fit(credit_split)
final_fit_lasso |> collect_metrics()
```

```
#> # A tibble: 2 × 4
#>   .metric .estimator .estimate .config             
#>   <chr>   <chr>          <dbl> <chr>               
#> 1 rmse    standard      10.8   Preprocessor1_Model1
#> 2 rsq     standard       0.996 Preprocessor1_Model1
```

---

.smaller[

Now let's take a look at the coefficients from the best model:

```r
coefs = final_fit_lasso |>
  extract_fit_parsnip() |>
  tidy()
```

<div class="datatables html-widget html-fill-item" id="htmlwidget-e1e1f66cf4272a822be1" style="width:100%;height:auto;"></div>
<script type="application/json" data-for="htmlwidget-e1e1f66cf4272a822be1">{"x":{"filter":"none","vertical":false,"data":[["1","2","3","4","5","6","7","8","9","10","11","12","13","14","15","16","17","18","19","20","21","22","23","24","25","26","27","28","29","30","31","32","33","34","35","36","37","38","39","40","41","42","43","44","45","46","47","48","49","50","51","52","53","54","55","56","57","58","59","60","61","62","63","64","65","66","67","68","69","70","71","72","73","74","75","76","77","78","79","80","81","82","83","84","85","86","87","88","89","90","91","92","93","94","95","96","97","98","99","100","101","102","103","104","105","106","107","108","109","110","111","112","113","114","115","116","117","118","119","120","121","122","123","124","125","126","127","128","129","130","131","132","133","134","135","136","137","138","139","140","141","142","143","144"],["(Intercept)","Income_poly_1","Income_poly_2","Limit_poly_1","Limit_poly_2","Cards_poly_1","Cards_poly_2","Age_poly_1","Age_poly_2","Education_poly_1","Education_poly_2","Balance_poly_1","Balance_poly_2","Gender_Female","Student_Yes","Married_Yes","Ethnicity_Asian","Ethnicity_Caucasian","Income_poly_1_x_Income_poly_2","Income_poly_1_x_Limit_poly_1","Income_poly_1_x_Limit_poly_2","Income_poly_1_x_Cards_poly_1","Income_poly_1_x_Cards_poly_2","Income_poly_1_x_Age_poly_1","Income_poly_1_x_Age_poly_2","Income_poly_1_x_Education_poly_1","Income_poly_1_x_Education_poly_2","Income_poly_1_x_Balance_poly_1","Income_poly_1_x_Balance_poly_2","Income_poly_1_x_Gender_Female","Income_poly_1_x_Student_Yes","Income_poly_1_x_Married_Yes","Income_poly_1_x_Ethnicity_Asian","Income_poly_1_x_Ethnicity_Caucasian","Income_poly_2_x_Limit_poly_1","Income_poly_2_x_Limit_poly_2","Income_poly_2_x_Cards_poly_1","Income_poly_2_x_Cards_poly_2","Income_poly_2_x_Age_poly_1","Income_poly_2_x_Age_poly_2","Income_poly_2_x_Education_poly_1","Income_poly_2_x_Education_poly_2","Income_poly_2_x_Balance_poly_1","Income_poly_2_x_Balance_poly_2","Income_poly_2_x_Gender_Female","Income_poly_2_x_Student_Yes","Income_poly_2_x_Married_Yes","Income_poly_2_x_Ethnicity_Asian","Income_poly_2_x_Ethnicity_Caucasian","Limit_poly_1_x_Limit_poly_2","Limit_poly_1_x_Cards_poly_1","Limit_poly_1_x_Cards_poly_2","Limit_poly_1_x_Age_poly_1","Limit_poly_1_x_Age_poly_2","Limit_poly_1_x_Education_poly_1","Limit_poly_1_x_Education_poly_2","Limit_poly_1_x_Balance_poly_1","Limit_poly_1_x_Balance_poly_2","Limit_poly_1_x_Gender_Female","Limit_poly_1_x_Student_Yes","Limit_poly_1_x_Married_Yes","Limit_poly_1_x_Ethnicity_Asian","Limit_poly_1_x_Ethnicity_Caucasian","Limit_poly_2_x_Cards_poly_1","Limit_poly_2_x_Cards_poly_2","Limit_poly_2_x_Age_poly_1","Limit_poly_2_x_Age_poly_2","Limit_poly_2_x_Education_poly_1","Limit_poly_2_x_Education_poly_2","Limit_poly_2_x_Balance_poly_1","Limit_poly_2_x_Balance_poly_2","Limit_poly_2_x_Gender_Female","Limit_poly_2_x_Student_Yes","Limit_poly_2_x_Married_Yes","Limit_poly_2_x_Ethnicity_Asian","Limit_poly_2_x_Ethnicity_Caucasian","Cards_poly_1_x_Cards_poly_2","Cards_poly_1_x_Age_poly_1","Cards_poly_1_x_Age_poly_2","Cards_poly_1_x_Education_poly_1","Cards_poly_1_x_Education_poly_2","Cards_poly_1_x_Balance_poly_1","Cards_poly_1_x_Balance_poly_2","Cards_poly_1_x_Gender_Female","Cards_poly_1_x_Married_Yes","Cards_poly_1_x_Ethnicity_Asian","Cards_poly_1_x_Ethnicity_Caucasian","Cards_poly_2_x_Age_poly_1","Cards_poly_2_x_Age_poly_2","Cards_poly_2_x_Education_poly_1","Cards_poly_2_x_Education_poly_2","Cards_poly_2_x_Balance_poly_1","Cards_poly_2_x_Balance_poly_2","Cards_poly_2_x_Gender_Female","Cards_poly_2_x_Married_Yes","Cards_poly_2_x_Ethnicity_Asian","Cards_poly_2_x_Ethnicity_Caucasian","Age_poly_1_x_Age_poly_2","Age_poly_1_x_Education_poly_1","Age_poly_1_x_Education_poly_2","Age_poly_1_x_Balance_poly_1","Age_poly_1_x_Balance_poly_2","Age_poly_1_x_Gender_Female","Age_poly_1_x_Married_Yes","Age_poly_1_x_Ethnicity_Asian","Age_poly_1_x_Ethnicity_Caucasian","Age_poly_2_x_Education_poly_1","Age_poly_2_x_Education_poly_2","Age_poly_2_x_Balance_poly_1","Age_poly_2_x_Balance_poly_2","Age_poly_2_x_Gender_Female","Age_poly_2_x_Married_Yes","Age_poly_2_x_Ethnicity_Asian","Age_poly_2_x_Ethnicity_Caucasian","Education_poly_1_x_Education_poly_2","Education_poly_1_x_Balance_poly_1","Education_poly_1_x_Balance_poly_2","Education_poly_1_x_Gender_Female","Education_poly_1_x_Married_Yes","Education_poly_1_x_Ethnicity_Caucasian","Education_poly_2_x_Balance_poly_1","Education_poly_2_x_Balance_poly_2","Education_poly_2_x_Gender_Female","Education_poly_2_x_Married_Yes","Education_poly_2_x_Ethnicity_Caucasian","Balance_poly_1_x_Balance_poly_2","Balance_poly_1_x_Gender_Female","Balance_poly_1_x_Student_Yes","Balance_poly_1_x_Married_Yes","Balance_poly_1_x_Ethnicity_Asian","Balance_poly_1_x_Ethnicity_Caucasian","Balance_poly_2_x_Gender_Female","Balance_poly_2_x_Student_Yes","Balance_poly_2_x_Married_Yes","Balance_poly_2_x_Ethnicity_Asian","Balance_poly_2_x_Ethnicity_Caucasian","Gender_Female_x_Student_Yes","Gender_Female_x_Married_Yes","Gender_Female_x_Ethnicity_Asian","Gender_Female_x_Ethnicity_Caucasian","Student_Yes_x_Married_Yes","Student_Yes_x_Ethnicity_Caucasian","Married_Yes_x_Ethnicity_Asian","Married_Yes_x_Ethnicity_Caucasian"],[355.3449258456262,33.12873400179186,0,2681.400654481593,0,79.18099530086822,0,0,0,0,0,43.79041856457034,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1.110637865473197,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,12.56646086310496,0,0,0,0,0,0,0,0,-13.4094303791777,0,0,-5.80938789121881,0,3.036853439911538,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1.819420110735615,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],[1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147]],"container":"<table class=\"display\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>term<\/th>\n      <th>estimate<\/th>\n      <th>penalty<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"dom":"tp","columnDefs":[{"className":"dt-right","targets":[2,3]},{"orderable":false,"targets":0},{"name":" ","targets":0},{"name":"term","targets":1},{"name":"estimate","targets":2},{"name":"penalty","targets":3}],"order":[],"autoWidth":false,"orderClasses":false}},"evals":[],"jsHooks":[]}</script>

]

---

How about listing only the non-zero coefficients?

```r
coefs_nonzero = coefs |>
  filter(estimate != 0)
```

<div class="datatables html-widget html-fill-item" id="htmlwidget-c2e63392ab83ab44e5ac" style="width:100%;height:auto;"></div>
<script type="application/json" data-for="htmlwidget-c2e63392ab83ab44e5ac">{"x":{"filter":"none","vertical":false,"data":[["1","2","3","4","5","6","7","8","9","10","11"],["(Intercept)","Income_poly_1","Limit_poly_1","Cards_poly_1","Balance_poly_1","Income_poly_2_x_Ethnicity_Caucasian","Cards_poly_1_x_Married_Yes","Cards_poly_2_x_Gender_Female","Cards_poly_2_x_Ethnicity_Caucasian","Age_poly_1_x_Education_poly_1","Education_poly_2_x_Ethnicity_Caucasian"],[355.3449258456262,33.12873400179186,2681.400654481593,79.18099530086822,43.79041856457034,1.110637865473197,12.56646086310496,-13.4094303791777,-5.80938789121881,3.036853439911538,1.819420110735615],[1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147,1.166989818617147]],"container":"<table class=\"display\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>term<\/th>\n      <th>estimate<\/th>\n      <th>penalty<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"dom":"tp","columnDefs":[{"className":"dt-right","targets":[2,3]},{"orderable":false,"targets":0},{"name":" ","targets":0},{"name":"term","targets":1},{"name":"estimate","targets":2},{"name":"penalty","targets":3}],"order":[],"autoWidth":false,"orderClasses":false}},"evals":[],"jsHooks":[]}</script>

---

Finally, we can plot the predictions against the true values:

```r
final_fit_lasso |> collect_predictions() |>
  ggplot(aes(x=.pred, y=Rating)) + 
  geom_abline(slope=1) +
  geom_point(color="blue")
```

---
layout: true
# Dependence

---
class: inverse, middle

---

So far, we've treated each observation as independent of each other observation. But exceptions are everywhere:
- Individuals in a household (cross-sectional data)
- Days of stock returns (time series data)
- Observations of the same county over time (panel data)
- Pixels of satellite imagery (spatial data)

For cases like these, we need to modify the cross-validation procedures to take **dependence** into account.
- We'll come back to methods for doing this soon.

---
name: dependence

.b[Resampling methods] assume something similar to independence: our resampling must match the original sampling procedure.

If observations are "linked" but we resample independently, CV may break.

If we have .hi-pink[repeated observations] on individuals `\(i\)` through time `\(t\)` (i.e., time series or panel data):
- It's pretty likely `\(y_{i,t}\)` and `\(y_{i,t+1}\)` are related—and maybe `\(y_{i,t+\ell}\)`.
- Initial sample may draw individuals `\(i\)`, but .it[standard] CV ignores time.

In other case, .hi-pink[some individuals are linked with other individuals], *e.g.*,
- `\(y_{i,t}\)` and `\(y_{j,t}\)` my be correlated if `\(i\)` and `\(j\)` live togother
- Also: `\(y_{i,t}\)` and `\(y_{j,t+\ell}\)` could be correlated

---

In other words: Spatial or temporal dependence between observations .b[breaks the separation between training and testing samples].

Breaking this separation train-test separation leads us back to

- .b[Overfitting] the sample—since training and testing samples are linked

- .b[Overestimating model performance]—the estimated test MSE will be more of a training MSE.

.b[Solutions] to this problem involve .hi-pink[matching the resampling process] to the .hi-pink[original sampling and underlying dependence].

---
## 1. Cross-sectional spatial data

.small[
Observations near each other in space are probably related to each other. We may want to resample blocks of larger areas, not individual observations.

- **Training/testing split?** Use `group_initial_split()` instead of `initial_split()` to keep groups of observations together when splitting the data.

- **Cross-validation:** Use `group_vfold_cv()` instead of `vfold_cv()` to resample by grabbing entire blocks at once (e.g., all counties for a given state, instead of treating each county as independent). Example:
]

```r
# 5-fold CV grabbing entire states at once
cv_block = df_train |> group_vfold_cv(group=state, v=5)
```

---
## 2. Panel data

.small[
Repeated observations for the same unit (person, county, etc.) are related to each other. We want to resample entire units at a time, not allow data from the same unit to show up in both training & testing data.

- **Training/testing split?** Use `group_initial_split()`, setting the `group` argument to the unit ID variable.

- **Cross-validation:** Use `group_vfold_cv()`, setting the `group` argument to the unit ID variable.

*Time series or panel?* Let `\(N=\)` number of units, `\(T=\)` number of time periods. Generally, if `\(N>T\)` you have panel data; if `\(T>N\)` you have time series data.
]

---
## 3. Time series data

.small[
Later observations may "remember" (contain information about) earlier observations. We need to train *only* on data that comes *before* the testing or validation set.

- **Training/testing split:** Use `initial_time_split()` instead of `initial_split()`. This makes your training data come first in time, and your training data last, instead of random sampling.

- **Cross-validation:** Use `sliding_index()` (or related functions [here](https://rsample.tidymodels.org/reference/slide-resampling.html)) instead of `vfold_cv()` to resample by choosing points at which we will train in the "past" and validate in the "future" ([more info](https://medium.com/@soumyachess1496/cross-validation-in-time-series-566ae4981ce4)). Example:
]

```r
# This generates folds with 7 days for training, 1 day for assessment.
cv_time = df_train |> sliding_index(index=DATE_VAR, lookback=7, assess_stop=1)
```