Multiple regression
Load the fraud_data
dataset using the load
command. The dataset is located in the folder ./data
(assuming you are in the homework folder) and contains 1000 cases of employee fraud in financial trading companies.
The columns of this dataset are:
- damage: damage caused in USD
- gender: 0 = female, 1 = male
- promoted: whether the employee was promoted in the past 5 years (0 = no, 1 = yes)
- years_experience: number of years of overall job experience in this or a related position
Task: multiple regression recap
Build a regression model that models the damage cause through the employees’ gender and promotion status.
summary(gender_promotion_model)
Call:
lm(formula = damage ~ gender + promoted, data = fraud_data)
Residuals:
Min 1Q Median 3Q Max
-642288 -272725 -18084 241855 1242517
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 625579 26363 23.729 < 2e-16 ***
gender 67377 27629 2.439 0.014915 *
promoted -89986 23091 -3.897 0.000104 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 354400 on 997 degrees of freedom
Multiple R-squared: 0.02141, Adjusted R-squared: 0.01944
F-statistic: 10.91 on 2 and 997 DF, p-value: 2.065e-05
What is the mean absolute error (MAE) of your model?
#your code here
mean(abs(gender_promotion_model$residuals))
[1] 289068.9
Now add the additional predictor variable years_experience
. How does this affect your model’s MAE? Did you expect this?
mean(abs(gender_promotion_experience_model$residuals))
[1] 2433.641
Finally, build the full model (i.e. include all main effects and interaction effects). Plot the residuals of that model against the observed values.
Task: multiple regression model selection
You can decide empirically, which combination of predictors results in the best model fit.
Start by building both a “null” model (i.e. only the intercept) and a “full” model (also called the saturated model).
Now start from the full model and use backwards stepwise regression. Which model does this procedure result in?
#your code here
step(full_model, direction = 'backward')
Start: AIC=16043.46
damage ~ gender * promoted * years_experience
Df Sum of Sq RSS AIC
<none> 9133491739 16044
- gender:promoted:years_experience 1 26638535 9160130274 16044
Call:
lm(formula = damage ~ gender * promoted * years_experience, data = fraud_data)
Coefficients:
(Intercept) gender promoted years_experience
10927.96 39970.33 -70278.04 39996.66
gender:promoted gender:years_experience promoted:years_experience gender:promoted:years_experience
1353.27 15.39 33.15 -97.70
Do the same with forward stepwise regression. Does this result in the same model?
step(full_model, direction = 'forward'
, scope = list(lower = null_model, upper = full_model))
Start: AIC=16043.46
damage ~ gender * promoted * years_experience
Call:
lm(formula = damage ~ gender * promoted * years_experience, data = fraud_data)
Coefficients:
(Intercept) gender promoted years_experience
10927.96 39970.33 -70278.04 39996.66
gender:promoted gender:years_experience promoted:years_experience gender:promoted:years_experience
1353.27 15.39 33.15 -97.70
Finally, try to implement the bidirectional stepwise regression (hint: use “both” for the direction
argument).
step(null_model, direction = 'both'
, scope=list(upper=full_model))
Start: AIC=25577.03
damage ~ 1
Df Sum of Sq RSS AIC
+ years_experience 1 1.2650e+14 1.4629e+12 21108
+ promoted 1 1.9925e+12 1.2597e+14 25563
+ gender 1 8.3196e+11 1.2713e+14 25572
<none> 1.2797e+14 25577
Step: AIC=21107.7
damage ~ years_experience
Df Sum of Sq RSS AIC
+ promoted 1 1.1886e+12 2.7429e+11 19436
+ gender 1 3.0356e+11 1.1594e+12 20877
<none> 1.4629e+12 21108
- years_experience 1 1.2650e+14 1.2797e+14 25577
Step: AIC=19435.71
damage ~ years_experience + promoted
Df Sum of Sq RSS AIC
+ gender 1 2.6509e+11 9.2038e+09 16043
<none> 2.7429e+11 19436
+ promoted:years_experience 1 1.1208e+08 2.7418e+11 19437
- promoted 1 1.1886e+12 1.4629e+12 21108
- years_experience 1 1.2570e+14 1.2597e+14 25563
Step: AIC=16043.13
damage ~ years_experience + promoted + gender
Df Sum of Sq RSS AIC
+ promoted:years_experience 1 3.8363e+07 9.1655e+09 16041
<none> 9.2038e+09 16043
+ gender:years_experience 1 4.6917e+06 9.1991e+09 16045
+ gender:promoted 1 9.8374e+05 9.2028e+09 16045
- gender 1 2.6509e+11 2.7429e+11 19436
- promoted 1 1.1502e+12 1.1594e+12 20877
- years_experience 1 1.2522e+14 1.2523e+14 25559
Step: AIC=16040.95
damage ~ years_experience + promoted + gender + years_experience:promoted
Df Sum of Sq RSS AIC
<none> 9.1655e+09 16041
+ gender:years_experience 1 4.5954e+06 9.1609e+09 16042
+ gender:promoted 1 6.8637e+05 9.1648e+09 16043
- years_experience:promoted 1 3.8363e+07 9.2038e+09 16043
- gender 1 2.6502e+11 2.7418e+11 19437
Call:
lm(formula = damage ~ years_experience + promoted + gender +
years_experience:promoted, data = fraud_data)
Coefficients:
(Intercept) years_experience promoted gender years_experience:promoted
10780.57 40009.16 -69184.04 40153.60 -46.09
What do these findings tell you?
Task: multiple regression model comparison
You will often end up with different models of the same outcome variable. If these models are nested (i.e. one model can be derived from the other by removing model parameters), then you can use inferential statistics to determine whether one model is significantly worse than another.
In R, you can use the anova
function to conduct an analysis of variance on two models to determine whether a simpler model is significantly worse than a more complex model.
Perform the model comparison test between (1) the full model vs the null model, (2) a ‘gender-only’ model and a ‘gender + promotion’ model, and (3) between the full model and the ‘gender + promotion’ model.
# 3
anova(gender_promotion_model, full_model)
Analysis of Variance Table
Model 1: damage ~ gender + promoted
Model 2: damage ~ gender * promoted * years_experience
Res.Df RSS Df Sum of Sq F Pr(>F)
1 997 1.2523e+14
2 992 9.1335e+09 5 1.2522e+14 2719998 < 2.2e-16 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Rank the models from best to worst (use equal ranks if there is no significant difference).
Ranks come here:
- full model
- gender promotion model
- gender model
- null model
Logistic regression
Often you want to build a model to either understand relationships in the data, make predictions, or both, about an outcome variable that is scored as 0/1, present/not present, arrested/not arrested, etc. If such an outcome variable has only two levels, we also speak of a binary or dichotomous outcome.
Regression models can be applied in this context too. To understand what the special issue with binary outcome variables is, let’s have a look at a dataset.
Load the attack_data
dataset from ./data
. We use this dataset to revise concepts from the lecture. This dataset represents whether or not a website was hacked and the number of attempted hacking attacks Columns are:
- hacked: 0 = no, 1 = yes
- attempts
Task: logistic regression - link function
Suppose you want to model the relationship between hacked
and attempts
. If you look at the plot, you see that these data do not stem from a normal distribution:
A distribution of a variable that can take only 0 and 1,
We can also look at the ‘raw’ data to get a better understanding of the relationship between the variables hacked
and attempts
:
So let’s start with what we know from regression modelling and ‘fit’ an ordinary linear model:
ordinary_model = glm(hacked ~ attempts
, data = attack_data
, family = gaussian
)
summary(ordinary_model)
Call:
glm(formula = hacked ~ attempts, family = gaussian, data = attack_data)
Deviance Residuals:
Min 1Q Median 3Q Max
-1.01593 -0.20825 0.03781 0.21227 1.17248
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -2.871e-01 1.856e-02 -15.47 <2e-16 ***
attempts 1.349e-03 3.212e-05 41.99 <2e-16 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
(Dispersion parameter for gaussian family taken to be 0.08599304)
Null deviance: 237.456 on 999 degrees of freedom
Residual deviance: 85.821 on 998 degrees of freedom
AIC: 388.39
Number of Fisher Scoring iterations: 2
Note that a GLM with family “gaussian” is identical to a normal linear model. This is because the ordinary linear model assumes that the outcome variable is normally distributed (i.e. follows a Gaussian distribution).
To see what might be problematic, have a look at the predicted values:
You will notice that the values predicted through the model are (1) not only 1s and 0s and (2) exceed 1 and are even smaller than 0. Clearly, for a dataset where the outcome variable can only take the value 0 and 1, this is an inadequate way to model the data.
You can also look at the actual regression line fitted to the raw data:
So we need a solution to that issue.
Luckily, there is a way to transform the 1/0 outcome variable to a continuous variable so that the model can make predictions on a continuous scale.
A neat way to do this, is the logit function that performs the following steps:
- it assumes that each case has a probability of being 1 or 0
Let’s do this for a sequence fro 0.0 to 1.0 in steps of 0.1
This brings the transformed outcome variable from a 0,1 scale to a 0:1 scale (note: the :
reads as “to”).
- it transforms these probabilities to the odds:
odds = P/(1-P)
This transforms the outcome variable to a scale ranging from 0:Inf.
- it calculates the logarithm of the odds (if you need to recap the concept of a logarithm, plese take a look at this video tutorial). More specifically, we take the natural logarithm.
You can see that now we transformed the outcome variable further to a scale ranging from -Inf:Inf.
It is this logit function that is used to transform the outcome variable from a 1,0 discrete range to a continuous range from -Inf:Inf.
You will see this in action further below…
In the GLM function, you can specify this by using the family =
argument and setting it to “binomial” (since our outcome variable stems from a binomial distribution).
Task: logistic regression - fitting the GLM
Build a logistic regression model that models whether or not a website was hacked through the number of attacks:
attack_model = glm(hacked ~ attempts
, data = attack_data
, family = 'binomial')
summary(attack_model)
Call:
glm(formula = hacked ~ attempts, family = "binomial", data = attack_data)
Deviance Residuals:
Min 1Q Median 3Q Max
-3.3705 -0.1961 -0.0257 0.2012 4.1173
Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -9.841211 0.710059 -13.86 <2e-16 ***
attempts 0.016064 0.001142 14.07 <2e-16 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
(Dispersion parameter for binomial family taken to be 1)
Null deviance: 1335.7 on 999 degrees of freedom
Residual deviance: 407.5 on 998 degrees of freedom
AIC: 411.5
Number of Fisher Scoring iterations: 7
Task: logistic regression - interpreting the model
Take a look at the model summary. Remember what the logit model does? If we model the “log-odds”, then the coefficients (what we call the intercept and slope in linear regression) need to be interpreted as such.
But because the log-odds are hard to interpret, we want to transform them back to the more interpretable odds.
From the video above, you will have learned that you can reverse the natural logarithm by taking e to the power of the logarithm.
Do this transformation:
exp(attack_model$coefficients)
(Intercept) attempts
5.321283e-05 1.016194e+00
What does this yield (i.e. how do you intepret these values)?
What would the interpretation of these findings look like in your own words?
–> https://stats.idre.ucla.edu/other/mult-pkg/faq/general/faq-how-do-i-interpret-odds-ratios-in-logistic-regression/
Task: logistic regression - curve fitting
Similar to the “line-fitting” of linear regression, we can also look at the fitted model visually.
You can see that the model (= the curve) predicts values exclusively in the 0:1 range. However, you can also see that while the majority of the predicted values is either 0 or 1, some values are in between (e.g. around attempts == 500). This is the reason why you need thresholds if you want to ascertain the accuracy of such a model. In Year 3, you will learn about applications of this thresholding for logistic regression in machine learning.
You can see the relationship between fitted values (i.e. probabilities of a case being in on eof the two outcome classes - hacked vs not hacked - given a certain number of attempts) and the observed values:
LS0tCnRpdGxlOiAiU29sdXRpb25zOiBNdWx0aXBsZSByZWdyZXNzaW9uLCBsb2dpc3RpYyByZWdyZXNzaW9uIgphdXRob3I6ICJCIEtsZWluYmVyZywgSSB2YW4gZGVyIFZlZ3QiCnN1YnRpdGxlOiBEZXB0IG9mIFNlY3VyaXR5IGFuZCBDcmltZSBTY2llbmNlLCBVQ0wKb3V0cHV0OgogIGh0bWxfbm90ZWJvb2s6IGRlZmF1bHQKICBwZGZfZG9jdW1lbnQ6IGRlZmF1bHQKLS0tCgoqKlNPTFVUSU9OUyoqCgojIyBBaW1zIG9mIHRoaXMgbm90ZWJvb2sKCi0gbXVsdGlwbGUgcmVncmVzc2lvbgogICAgLSByZWNhcCAKICAgIC0gbW9kZWwgc2VsZWN0aW9uCiAgICAtIG1vZGVsIGNvbXBhcmlzb24KLSBsb2dpc3RpYyByZWdyZXNzaW9uCiAgICAtIGxpbmsgZnVuY3Rpb24KICAgIC0gZml0dGluZyB0aGUgR0xNCiAgICAtIGludGVycHJldGluZyB0aGUgbW9kZWwKICAgIC0gbW9kZWwgY29tcGFyaXNvbgoKIyMgUmVxdWlyZW1lbnRzCgpXZSBhc3N1bWUgdGhhdCB5b3UgaGF2ZToKCi0gcmVhZCB0aGUgcmVxdWlyZWQgbGl0ZXJhdHVyZSBmb3Igd2Vla3MgMS0zCi0gcmV2aXNlZCB0aGUgbGVjdHVyZXMKLSBjb21wbGV0ZWQgdGhlIGludHJvZHVjdG9yeSBSIHR1dG9yaWFscyAoMTIgc3RlcHMgJiBIb3cgdG8gc29sdmUgcHJvYmxlbXMpIGFzIHdlbGwgYXMgdGhlIHR1dG9yaWFsIGZyb20gd2VlayAyCi0gY29tcGxldGVkIHRoZSBob21ld29yayBpbiB0aGlzIG1vZHVsZSBzbyBmYXIKLSByZXBsaWNhdGVkIHRoZSBjb2RlIGZyb20gdGhlIGxlY3R1cmVzIChpZiBjb25jZXB0cy9SIGltcGxlbWVudGF0aW9uIGlzIHVuY2xlYXIpCgpJZiB5b3Ugc3RydWdnbGUgd2l0aCBiYXNpY3Mgb2YgUiwgeW91IG1heSBhbHNvIGZpbmQgdGhpcyBvbmxpbmUgYm9vayB1c2VmdWw6IFtodHRwczovL2Jvb2tkb3duLm9yZy9uZHBoaWxsaXBzL1lhUnJyL10oaHR0cHM6Ly9ib29rZG93bi5vcmcvbmRwaGlsbGlwcy9ZYVJyci8pCgotLS0KCiMjIE11bHRpcGxlIHJlZ3Jlc3Npb24KCkxvYWQgdGhlIGBmcmF1ZF9kYXRhYCBkYXRhc2V0IHVzaW5nIHRoZSBgbG9hZGAgY29tbWFuZC4gVGhlIGRhdGFzZXQgaXMgbG9jYXRlZCBpbiB0aGUgZm9sZGVyIGAuL2RhdGFgIChhc3N1bWluZyB5b3UgYXJlIGluIHRoZSBob21ld29yayBmb2xkZXIpIGFuZCBjb250YWlucyAxMDAwIGNhc2VzIG9mIGVtcGxveWVlIGZyYXVkIGluIGZpbmFuY2lhbCB0cmFkaW5nIGNvbXBhbmllcy4KClRoZSBjb2x1bW5zIG9mIHRoaXMgZGF0YXNldCBhcmU6CgotIGRhbWFnZTogZGFtYWdlIGNhdXNlZCBpbiBVU0QKLSBnZW5kZXI6IDAgPSBmZW1hbGUsIDEgPSBtYWxlCi0gcHJvbW90ZWQ6IHdoZXRoZXIgdGhlIGVtcGxveWVlIHdhcyBwcm9tb3RlZCBpbiB0aGUgcGFzdCA1IHllYXJzICgwID0gbm8sIDEgPSB5ZXMpCi0geWVhcnNfZXhwZXJpZW5jZTogbnVtYmVyIG9mIHllYXJzIG9mIG92ZXJhbGwgam9iIGV4cGVyaWVuY2UgaW4gdGhpcyBvciBhIHJlbGF0ZWQgcG9zaXRpb24KCiMjIyBUYXNrOiBtdWx0aXBsZSByZWdyZXNzaW9uIHJlY2FwCgpCdWlsZCBhIHJlZ3Jlc3Npb24gbW9kZWwgdGhhdCBtb2RlbHMgdGhlIGRhbWFnZSBjYXVzZSB0aHJvdWdoIHRoZSBlbXBsb3llZXMnIGdlbmRlciBhbmQgcHJvbW90aW9uIHN0YXR1cy4KCmBgYHtyfQojeW91ciBjb2RlIGhlcmUKbG9hZCgnLi9kYXRhL2ZyYXVkX2RhdGEuUkRhdGEnKQpnZW5kZXJfcHJvbW90aW9uX21vZGVsID0gbG0oZGFtYWdlIH4gZ2VuZGVyICsgcHJvbW90ZWQsIGRhdGEgPSBmcmF1ZF9kYXRhKQpzdW1tYXJ5KGdlbmRlcl9wcm9tb3Rpb25fbW9kZWwpCmBgYAoKV2hhdCBpcyB0aGUgbWVhbiBhYnNvbHV0ZSBlcnJvciAoTUFFKSBvZiB5b3VyIG1vZGVsPwoKYGBge3J9CiN5b3VyIGNvZGUgaGVyZQptZWFuKGFicyhnZW5kZXJfcHJvbW90aW9uX21vZGVsJHJlc2lkdWFscykpCmBgYAoKTm93IGFkZCB0aGUgYWRkaXRpb25hbCBwcmVkaWN0b3IgdmFyaWFibGUgYHllYXJzX2V4cGVyaWVuY2VgLiBIb3cgZG9lcyB0aGlzIGFmZmVjdCB5b3VyIG1vZGVsJ3MgTUFFPyBEaWQgeW91IGV4cGVjdCB0aGlzPwoKYGBge3J9CiN5b3VyIGNvZGUgaGVyZQpnZW5kZXJfcHJvbW90aW9uX2V4cGVyaWVuY2VfbW9kZWwgPSBsbShkYW1hZ2UgfiBnZW5kZXIgKyBwcm9tb3RlZCArIHllYXJzX2V4cGVyaWVuY2UsIGRhdGEgPSBmcmF1ZF9kYXRhKQptZWFuKGFicyhnZW5kZXJfcHJvbW90aW9uX2V4cGVyaWVuY2VfbW9kZWwkcmVzaWR1YWxzKSkKYGBgCgpGaW5hbGx5LCBidWlsZCB0aGUgZnVsbCBtb2RlbCAoaS5lLiBpbmNsdWRlIGFsbCBtYWluIGVmZmVjdHMgYW5kIGludGVyYWN0aW9uIGVmZmVjdHMpLiBQbG90IHRoZSByZXNpZHVhbHMgb2YgdGhhdCBtb2RlbCBhZ2FpbnN0IHRoZSBvYnNlcnZlZCB2YWx1ZXMuCgpgYGB7cn0KI3lvdXIgY29kZSBoZXJlCmZ1bGxfbW9kZWwgPSBsbShkYW1hZ2UgfiBnZW5kZXIqcHJvbW90ZWQqeWVhcnNfZXhwZXJpZW5jZSwgZGF0YSA9IGZyYXVkX2RhdGEpCnBsb3QoZnJhdWRfZGF0YSRkYW1hZ2UsIGZ1bGxfbW9kZWwkcmVzaWR1YWxzKQpgYGAKCiMjIyBUYXNrOiBtdWx0aXBsZSByZWdyZXNzaW9uIG1vZGVsIHNlbGVjdGlvbgoKWW91IGNhbiBkZWNpZGUgZW1waXJpY2FsbHksIHdoaWNoIGNvbWJpbmF0aW9uIG9mIHByZWRpY3RvcnMgcmVzdWx0cyBpbiB0aGUgYmVzdCBtb2RlbCBmaXQuCgpTdGFydCBieSBidWlsZGluZyBib3RoIGEgIm51bGwiIG1vZGVsIChpLmUuIG9ubHkgdGhlIGludGVyY2VwdCkgYW5kIGEgImZ1bGwiIG1vZGVsIChhbHNvIGNhbGxlZCB0aGUgc2F0dXJhdGVkIG1vZGVsKS4KCmBgYHtyfQojeW91ciBjb2RlIGhlcmUKbnVsbF9tb2RlbCA9IGxtKGRhbWFnZSB+IDEsIGRhdGEgPSBmcmF1ZF9kYXRhKQpgYGAKCk5vdyBzdGFydCBmcm9tIHRoZSBmdWxsIG1vZGVsIGFuZCB1c2UgYmFja3dhcmRzIHN0ZXB3aXNlIHJlZ3Jlc3Npb24uIFdoaWNoIG1vZGVsIGRvZXMgdGhpcyBwcm9jZWR1cmUgcmVzdWx0IGluPwoKYGBge3J9CiN5b3VyIGNvZGUgaGVyZQpzdGVwKGZ1bGxfbW9kZWwsIGRpcmVjdGlvbiA9ICdiYWNrd2FyZCcpCiMgSXQgcmVzdWx0cyBpbiB0aGUgZnVsbCBtb2RlbCAobWFpbiBlZmZlY3RzIGFuZCBhbGwgaW50ZXJhY3Rpb25zKQpgYGAKCkRvIHRoZSBzYW1lIHdpdGggZm9yd2FyZCBzdGVwd2lzZSByZWdyZXNzaW9uLiBEb2VzIHRoaXMgcmVzdWx0IGluIHRoZSBzYW1lIG1vZGVsPwoKYGBge3J9CiN5b3VyIGNvZGUgaGVyZQpzdGVwKGZ1bGxfbW9kZWwsIGRpcmVjdGlvbiA9ICdmb3J3YXJkJwogICAgICwgc2NvcGUgPSBsaXN0KGxvd2VyID0gbnVsbF9tb2RlbCwgdXBwZXIgPSBmdWxsX21vZGVsKSkKIyBUaGUgZm9yd2FyZCBzdGVwd2lzZSByZWdyZXNzaW9uIGFsc28gcmVzdWxzIGluIHRoZSBmdWxsIG1vZGVsCmBgYAoKRmluYWxseSwgdHJ5IHRvIGltcGxlbWVudCB0aGUgYmlkaXJlY3Rpb25hbCBzdGVwd2lzZSByZWdyZXNzaW9uIChoaW50OiB1c2UgImJvdGgiIGZvciB0aGUgYGRpcmVjdGlvbmAgYXJndW1lbnQpLgoKYGBge3J9CiN5b3VyIGNvZGUgaGVyZQpzdGVwKG51bGxfbW9kZWwsIGRpcmVjdGlvbiA9ICdib3RoJwogICAgICwgc2NvcGU9bGlzdCh1cHBlcj1mdWxsX21vZGVsKSkKIyBCaWRpcmVjdGlvbmFsIHN0ZXB3aXNlIHJlZ3Jlc3Npb24gcmVzdWx0cyBpbiBhIG1vZGVsIHdpdGggYWxsIG1haW4gZWZmZWN0cyBhbmQgYW4gaW50ZXJhY3Rpb24gYmV0d2VlbiBleHBlcmllbmNlIGFuZCBwcm9tb3Rpb24KYGBgCgpXaGF0IGRvIHRoZXNlIGZpbmRpbmdzIHRlbGwgeW91PwoKCiMjIyBUYXNrOiBtdWx0aXBsZSByZWdyZXNzaW9uIG1vZGVsIGNvbXBhcmlzb24KCllvdSB3aWxsIG9mdGVuIGVuZCB1cCB3aXRoIGRpZmZlcmVudCBtb2RlbHMgb2YgdGhlIHNhbWUgb3V0Y29tZSB2YXJpYWJsZS4gSWYgdGhlc2UgbW9kZWxzIGFyZSBuZXN0ZWQgKGkuZS4gb25lIG1vZGVsIGNhbiBiZSBkZXJpdmVkIGZyb20gdGhlIG90aGVyIGJ5IHJlbW92aW5nIG1vZGVsIHBhcmFtZXRlcnMpLCB0aGVuIHlvdSBjYW4gdXNlIGluZmVyZW50aWFsIHN0YXRpc3RpY3MgdG8gZGV0ZXJtaW5lIHdoZXRoZXIgb25lIG1vZGVsIGlzIHNpZ25pZmljYW50bHkgd29yc2UgdGhhbiBhbm90aGVyLgoKSW4gUiwgeW91IGNhbiB1c2UgdGhlIGBhbm92YWAgZnVuY3Rpb24gdG8gY29uZHVjdCBhbiBhbmFseXNpcyBvZiB2YXJpYW5jZSBvbiB0d28gbW9kZWxzIHRvIGRldGVybWluZSB3aGV0aGVyIGEgc2ltcGxlciBtb2RlbCBpcyBzaWduaWZpY2FudGx5IHdvcnNlIHRoYW4gYSBtb3JlIGNvbXBsZXggbW9kZWwuCgpQZXJmb3JtIHRoZSBtb2RlbCBjb21wYXJpc29uIHRlc3QgYmV0d2VlbiAoMSkgdGhlIGZ1bGwgbW9kZWwgdnMgdGhlIG51bGwgbW9kZWwsICgyKSBhICdnZW5kZXItb25seScgbW9kZWwgYW5kIGEgJ2dlbmRlciArIHByb21vdGlvbicgbW9kZWwsIGFuZCAoMykgYmV0d2VlbiB0aGUgZnVsbCBtb2RlbCBhbmQgdGhlICdnZW5kZXIgKyBwcm9tb3Rpb24nIG1vZGVsLgoKYGBge3J9CiN5b3VyIGNvZGUgaGVyZQoKIyAxIAphbm92YShudWxsX21vZGVsLCBmdWxsX21vZGVsKQoKIyAyIApnZW5kZXJfbW9kZWwgPSBsbShkYW1hZ2UgfiBnZW5kZXIsIGRhdGEgPSBmcmF1ZF9kYXRhKSAjIG1ha2UgZ2VuZGVyIG1vZGVsICh3ZSBkbyBub3QgaGF2ZSB0aGlzIHlldCkKYW5vdmEoZ2VuZGVyX21vZGVsLCBnZW5kZXJfcHJvbW90aW9uX21vZGVsKQoKIyAzCmFub3ZhKGdlbmRlcl9wcm9tb3Rpb25fbW9kZWwsIGZ1bGxfbW9kZWwpCgpgYGAKClJhbmsgdGhlIG1vZGVscyBmcm9tIGJlc3QgdG8gd29yc3QgKHVzZSBlcXVhbCByYW5rcyBpZiB0aGVyZSBpcyBubyBzaWduaWZpY2FudCBkaWZmZXJlbmNlKS4KClJhbmtzIGNvbWUgaGVyZToKCjEuIGZ1bGwgbW9kZWwgCjIuIGdlbmRlciBwcm9tb3Rpb24gbW9kZWwKMy4gZ2VuZGVyIG1vZGVsCjQuIG51bGwgbW9kZWwKCi0tLQoKIyMgTG9naXN0aWMgcmVncmVzc2lvbgoKT2Z0ZW4geW91IHdhbnQgdG8gYnVpbGQgYSBtb2RlbCB0byBlaXRoZXIgdW5kZXJzdGFuZCByZWxhdGlvbnNoaXBzIGluIHRoZSBkYXRhLCBtYWtlIHByZWRpY3Rpb25zLCBvciBib3RoLCBhYm91dCBhbiBvdXRjb21lIHZhcmlhYmxlIHRoYXQgaXMgc2NvcmVkIGFzIDAvMSwgcHJlc2VudC9ub3QgcHJlc2VudCwgYXJyZXN0ZWQvbm90IGFycmVzdGVkLCBldGMuIElmIHN1Y2ggYW4gb3V0Y29tZSB2YXJpYWJsZSBoYXMgb25seSB0d28gbGV2ZWxzLCB3ZSBhbHNvIHNwZWFrIG9mIGEgYmluYXJ5IG9yIGRpY2hvdG9tb3VzIG91dGNvbWUuCgpSZWdyZXNzaW9uIG1vZGVscyBjYW4gYmUgYXBwbGllZCBpbiB0aGlzIGNvbnRleHQgdG9vLiBUbyB1bmRlcnN0YW5kIHdoYXQgdGhlIHNwZWNpYWwgaXNzdWUgd2l0aCBiaW5hcnkgb3V0Y29tZSB2YXJpYWJsZXMgaXMsIGxldCdzIGhhdmUgYSBsb29rIGF0IGEgZGF0YXNldC4KCkxvYWQgdGhlIGBhdHRhY2tfZGF0YWAgZGF0YXNldCBmcm9tIGAuL2RhdGFgLiBXZSB1c2UgdGhpcyBkYXRhc2V0IHRvIHJldmlzZSBjb25jZXB0cyBmcm9tIHRoZSBsZWN0dXJlLiBUaGlzIGRhdGFzZXQgcmVwcmVzZW50cyB3aGV0aGVyIG9yIG5vdCBhIHdlYnNpdGUgd2FzIGhhY2tlZCBhbmQgdGhlIG51bWJlciBvZiBhdHRlbXB0ZWQgaGFja2luZyBhdHRhY2tzIENvbHVtbnMgYXJlOgoKLSBoYWNrZWQ6IDAgPSBubywgMSA9IHllcwotIGF0dGVtcHRzCgoKIyMjIFRhc2s6IGxvZ2lzdGljIHJlZ3Jlc3Npb24gLSBsaW5rIGZ1bmN0aW9uIAoKU3VwcG9zZSB5b3Ugd2FudCB0byBtb2RlbCB0aGUgcmVsYXRpb25zaGlwIGJldHdlZW4gYGhhY2tlZGAgYW5kIGBhdHRlbXB0c2AuIElmIHlvdSBsb29rIGF0IHRoZSBwbG90LCB5b3Ugc2VlIHRoYXQgdGhlc2UgZGF0YSBkbyBub3Qgc3RlbSBmcm9tIGEgbm9ybWFsIGRpc3RyaWJ1dGlvbjoKCmBgYHtyfQpsb2FkKCcuL2RhdGEvYXR0YWNrX2RhdGEuUkRhdGEnKQpoaXN0KGF0dGFja19kYXRhJGhhY2tlZCkKYGBgCgpBIGRpc3RyaWJ1dGlvbiBvZiBhIHZhcmlhYmxlICB0aGF0IGNhbiB0YWtlIG9ubHkgMCBhbmQgMSwKCgpXZSBjYW4gYWxzbyBsb29rIGF0IHRoZSAncmF3JyBkYXRhIHRvIGdldCBhIGJldHRlciB1bmRlcnN0YW5kaW5nIG9mIHRoZSByZWxhdGlvbnNoaXAgYmV0d2VlbiB0aGUgdmFyaWFibGVzIGBoYWNrZWRgIGFuZCBgYXR0ZW1wdHNgOgoKYGBge3J9CnBsb3QoYXR0YWNrX2RhdGEkYXR0ZW1wdHMsIGF0dGFja19kYXRhJGhhY2tlZCkKYGBgCgpTbyBsZXQncyBzdGFydCB3aXRoIHdoYXQgd2Uga25vdyBmcm9tIHJlZ3Jlc3Npb24gbW9kZWxsaW5nIGFuZCAnZml0JyBhbiBvcmRpbmFyeSBsaW5lYXIgbW9kZWw6CgpgYGB7cn0Kb3JkaW5hcnlfbW9kZWwgPSBnbG0oaGFja2VkIH4gYXR0ZW1wdHMKICAgICAgICAgICAgICAgICAgICAgLCBkYXRhID0gYXR0YWNrX2RhdGEKICAgICAgICAgICAgICAgICAgICAgLCBmYW1pbHkgPSBnYXVzc2lhbgogICAgICAgICAgICAgICAgICAgICkKc3VtbWFyeShvcmRpbmFyeV9tb2RlbCkKYGBgCgpOb3RlIHRoYXQgYSBHTE0gd2l0aCBmYW1pbHkgImdhdXNzaWFuIiBpcyBpZGVudGljYWwgdG8gYSBub3JtYWwgbGluZWFyIG1vZGVsLiBUaGlzIGlzIGJlY2F1c2UgdGhlIG9yZGluYXJ5IGxpbmVhciBtb2RlbCBhc3N1bWVzIHRoYXQgdGhlIG91dGNvbWUgdmFyaWFibGUgaXMgbm9ybWFsbHkgZGlzdHJpYnV0ZWQgKGkuZS4gZm9sbG93cyBhIEdhdXNzaWFuIGRpc3RyaWJ1dGlvbikuCgpUbyBzZWUgd2hhdCBtaWdodCBiZSBwcm9ibGVtYXRpYywgaGF2ZSBhIGxvb2sgYXQgdGhlIHByZWRpY3RlZCB2YWx1ZXM6CgpgYGB7cn0KcGxvdChvcmRpbmFyeV9tb2RlbCRtb2RlbCRhdHRlbXB0cywgb3JkaW5hcnlfbW9kZWwkZml0dGVkLnZhbHVlcykKYGBgCgpZb3Ugd2lsbCBub3RpY2UgdGhhdCB0aGUgdmFsdWVzIHByZWRpY3RlZCB0aHJvdWdoIHRoZSBtb2RlbCBhcmUgKDEpIG5vdCBvbmx5IDFzIGFuZCAwcyBhbmQgKDIpIGV4Y2VlZCAxIGFuZCBhcmUgZXZlbiBzbWFsbGVyIHRoYW4gMC4gQ2xlYXJseSwgZm9yIGEgZGF0YXNldCB3aGVyZSB0aGUgb3V0Y29tZSB2YXJpYWJsZSBjYW4gb25seSB0YWtlIHRoZSB2YWx1ZSAwIGFuZCAxLCB0aGlzIGlzIGFuIGluYWRlcXVhdGUgd2F5IHRvIG1vZGVsIHRoZSBkYXRhLgoKWW91IGNhbiBhbHNvIGxvb2sgYXQgdGhlIGFjdHVhbCByZWdyZXNzaW9uIGxpbmUgZml0dGVkIHRvIHRoZSByYXcgZGF0YToKCmBgYHtyfQp7cGxvdChhdHRhY2tfZGF0YSRhdHRlbXB0cywgYXR0YWNrX2RhdGEkaGFja2VkKQogIGFibGluZShvcmRpbmFyeV9tb2RlbCl9CmBgYAoKU28gd2UgbmVlZCBhIHNvbHV0aW9uIHRvIHRoYXQgaXNzdWUuCgpMdWNraWx5LCB0aGVyZSBpcyBhIHdheSB0byB0cmFuc2Zvcm0gdGhlIDEvMCBvdXRjb21lIHZhcmlhYmxlIHRvIGEgY29udGludW91cyB2YXJpYWJsZSBzbyB0aGF0IHRoZSBtb2RlbCBjYW4gbWFrZSBwcmVkaWN0aW9ucyBvbiBhIGNvbnRpbnVvdXMgc2NhbGUuCgpBIG5lYXQgd2F5IHRvIGRvIHRoaXMsIGlzIHRoZSBsb2dpdCBmdW5jdGlvbiB0aGF0IHBlcmZvcm1zIHRoZSBmb2xsb3dpbmcgc3RlcHM6CgoxLiBpdCBhc3N1bWVzIHRoYXQgZWFjaCBjYXNlIGhhcyBhIHByb2JhYmlsaXR5IG9mIGJlaW5nIDEgb3IgMAoKTGV0J3MgZG8gdGhpcyBmb3IgYSBzZXF1ZW5jZSBmcm8gMC4wIHRvIDEuMCBpbiBzdGVwcyBvZiAwLjEKCmBgYHtyfQpwcm9iYWJpbGl0aWVzID0gc2VxKGZyb20gPSAwLjAxLCB0byA9IDAuOTksIGJ5PTAuMDEpCmBgYAoKVGhpcyBicmluZ3MgdGhlIHRyYW5zZm9ybWVkIG91dGNvbWUgdmFyaWFibGUgZnJvbSBhIDAsMSBzY2FsZSB0byBhIDA6MSBzY2FsZSAobm90ZTogdGhlIGA6YCByZWFkcyBhcyAidG8iKS4KCjIuIGl0IHRyYW5zZm9ybXMgdGhlc2UgcHJvYmFiaWxpdGllcyB0byB0aGUgb2RkczogYG9kZHMgPSBQLygxLVApYAoKYGBge3J9Cm9kZHMgPSBwcm9iYWJpbGl0aWVzLygxLXByb2JhYmlsaXRpZXMpCmBgYAoKVGhpcyB0cmFuc2Zvcm1zIHRoZSBvdXRjb21lIHZhcmlhYmxlIHRvIGEgc2NhbGUgcmFuZ2luZyBmcm9tIDA6SW5mLgoKYGBge3J9CnBsb3Qob2RkcywgcHJvYmFiaWxpdGllcykKYGBgCgoKMy4gaXQgY2FsY3VsYXRlcyB0aGUgbG9nYXJpdGhtIG9mIHRoZSBvZGRzIChpZiB5b3UgbmVlZCB0byByZWNhcCB0aGUgY29uY2VwdCBvZiBhIGxvZ2FyaXRobSwgcGxlc2UgdGFrZSBhIGxvb2sgYXQgW3RoaXMgdmlkZW8gdHV0b3JpYWxdKGh0dHBzOi8vd3d3LnlvdXR1YmUuY29tL3dhdGNoP3Y9bVFUV3pMcENjVzApKS4gTW9yZSBzcGVjaWZpY2FsbHksIHdlIHRha2UgdGhlIFtuYXR1cmFsIGxvZ2FyaXRobV0oaHR0cHM6Ly9lbi53aWtpcGVkaWEub3JnL3dpa2kvTmF0dXJhbF9sb2dhcml0aG0pLgoKYGBge3J9CmxvZ19vZGRzID0gbG9nKG9kZHMpCmBgYAoKYGBge3J9CnBsb3QobG9nX29kZHMsIHByb2JhYmlsaXRpZXMpCmBgYAoKWW91IGNhbiBzZWUgdGhhdCBub3cgd2UgdHJhbnNmb3JtZWQgdGhlIG91dGNvbWUgdmFyaWFibGUgZnVydGhlciB0byBhIHNjYWxlIHJhbmdpbmcgZnJvbSAtSW5mOkluZi4KCkl0IGlzIHRoaXMgbG9naXQgZnVuY3Rpb24gdGhhdCBpcyB1c2VkIHRvIHRyYW5zZm9ybSB0aGUgb3V0Y29tZSB2YXJpYWJsZSBmcm9tIGEgMSwwIGRpc2NyZXRlIHJhbmdlIHRvIGEgY29udGludW91cyByYW5nZSBmcm9tIC1JbmY6SW5mLgoKWW91IHdpbGwgc2VlIHRoaXMgaW4gYWN0aW9uIGZ1cnRoZXIgYmVsb3cuLi4KCkluIHRoZSBHTE0gZnVuY3Rpb24sIHlvdSBjYW4gc3BlY2lmeSB0aGlzIGJ5IHVzaW5nIHRoZSBgZmFtaWx5ID0gYCBhcmd1bWVudCBhbmQgc2V0dGluZyBpdCB0byAiYmlub21pYWwiIChzaW5jZSBvdXIgb3V0Y29tZSB2YXJpYWJsZSBzdGVtcyBmcm9tIGEgYmlub21pYWwgZGlzdHJpYnV0aW9uKS4KCiMjIyBUYXNrOiBsb2dpc3RpYyByZWdyZXNzaW9uIC0gZml0dGluZyB0aGUgR0xNCgpCdWlsZCBhIGxvZ2lzdGljIHJlZ3Jlc3Npb24gbW9kZWwgdGhhdCBtb2RlbHMgd2hldGhlciBvciBub3QgYSB3ZWJzaXRlIHdhcyBoYWNrZWQgdGhyb3VnaCB0aGUgbnVtYmVyIG9mIGF0dGFja3M6CgpgYGB7cn0KI3lvdXIgY29kZSBoZXJlCmF0dGFja19tb2RlbCA9IGdsbShoYWNrZWQgfiBhdHRlbXB0cwogICAgLCBkYXRhID0gYXR0YWNrX2RhdGEKICAgICwgZmFtaWx5ID0gJ2Jpbm9taWFsJykKc3VtbWFyeShhdHRhY2tfbW9kZWwpCmBgYAoKIyMjIFRhc2s6IGxvZ2lzdGljIHJlZ3Jlc3Npb24gLSBpbnRlcnByZXRpbmcgdGhlIG1vZGVsCgpUYWtlIGEgbG9vayBhdCB0aGUgbW9kZWwgc3VtbWFyeS4gUmVtZW1iZXIgd2hhdCB0aGUgbG9naXQgbW9kZWwgZG9lcz8gSWYgd2UgbW9kZWwgdGhlICJsb2ctb2RkcyIsIHRoZW4gdGhlIGNvZWZmaWNpZW50cyAod2hhdCB3ZSBjYWxsIHRoZSBpbnRlcmNlcHQgYW5kIHNsb3BlIGluIGxpbmVhciByZWdyZXNzaW9uKSBuZWVkIHRvIGJlIGludGVycHJldGVkIGFzIHN1Y2guCgpCdXQgYmVjYXVzZSB0aGUgbG9nLW9kZHMgYXJlIGhhcmQgdG8gaW50ZXJwcmV0LCB3ZSB3YW50IHRvIHRyYW5zZm9ybSB0aGVtIGJhY2sgdG8gdGhlIG1vcmUgaW50ZXJwcmV0YWJsZSBvZGRzLgoKRnJvbSB0aGUgdmlkZW8gYWJvdmUsIHlvdSB3aWxsIGhhdmUgbGVhcm5lZCB0aGF0IHlvdSBjYW4gcmV2ZXJzZSB0aGUgbmF0dXJhbCBsb2dhcml0aG0gYnkgdGFraW5nIFtlXShodHRwczovL2VuLndpa2lwZWRpYS5vcmcvd2lraS9FXyhtYXRoZW1hdGljYWxfY29uc3RhbnQpKSB0byB0aGUgcG93ZXIgb2YgdGhlIGxvZ2FyaXRobS4KCkRvIHRoaXMgdHJhbnNmb3JtYXRpb246CgpgYGB7cn0KI3lvdXIgY29kZSBoZXJlCgpleHAoYXR0YWNrX21vZGVsJGNvZWZmaWNpZW50cykKCmBgYAoKV2hhdCBkb2VzIHRoaXMgeWllbGQgKGkuZS4gaG93IGRvIHlvdSBpbnRlcHJldCB0aGVzZSB2YWx1ZXMpPwoKV2hhdCB3b3VsZCB0aGUgaW50ZXJwcmV0YXRpb24gb2YgdGhlc2UgZmluZGluZ3MgbG9vayBsaWtlIGluIHlvdXIgb3duIHdvcmRzPwoKLS0+IFtodHRwczovL3N0YXRzLmlkcmUudWNsYS5lZHUvb3RoZXIvbXVsdC1wa2cvZmFxL2dlbmVyYWwvZmFxLWhvdy1kby1pLWludGVycHJldC1vZGRzLXJhdGlvcy1pbi1sb2dpc3RpYy1yZWdyZXNzaW9uL10oaHR0cHM6Ly9zdGF0cy5pZHJlLnVjbGEuZWR1L290aGVyL211bHQtcGtnL2ZhcS9nZW5lcmFsL2ZhcS1ob3ctZG8taS1pbnRlcnByZXQtb2Rkcy1yYXRpb3MtaW4tbG9naXN0aWMtcmVncmVzc2lvbi8pCgojIyMgVGFzazogbG9naXN0aWMgcmVncmVzc2lvbiAtIGN1cnZlIGZpdHRpbmcKClNpbWlsYXIgdG8gdGhlICJsaW5lLWZpdHRpbmciIG9mIGxpbmVhciByZWdyZXNzaW9uLCB3ZSBjYW4gYWxzbyBsb29rIGF0IHRoZSBmaXR0ZWQgbW9kZWwgdmlzdWFsbHkuCgpgYGB7cn0KbG9ncmVnID0gZ2xtKGhhY2tlZCB+IGF0dGVtcHRzCiAgICAgICAgICAgICAsIGZhbWlseT1iaW5vbWlhbAogICAgICAgICAgICAgLCBkYXRhID0gYXR0YWNrX2RhdGEpCntwbG90KGF0dGFja19kYXRhJGF0dGVtcHRzLCBhdHRhY2tfZGF0YSRoYWNrZWQpCiAgY3VydmUocHJlZGljdChsb2dyZWcKICAgICAgICAgICAgICAgICwgZGF0YS5mcmFtZShhdHRlbXB0cz14KQogICAgICAgICAgICAgICAgLCB0eXBlPSJyZXNwIikKICAgICAgICAsIGFkZD1UUlVFKX0KYGBgCgoKWW91IGNhbiBzZWUgdGhhdCB0aGUgbW9kZWwgKD0gdGhlIGN1cnZlKSBwcmVkaWN0cyB2YWx1ZXMgZXhjbHVzaXZlbHkgaW4gdGhlIDA6MSByYW5nZS4gSG93ZXZlciwgeW91IGNhbiBhbHNvIHNlZSB0aGF0IHdoaWxlIHRoZSBtYWpvcml0eSBvZiB0aGUgcHJlZGljdGVkIHZhbHVlcyBpcyBlaXRoZXIgMCBvciAxLCBzb21lIHZhbHVlcyBhcmUgaW4gYmV0d2VlbiAoZS5nLiBhcm91bmQgYXR0ZW1wdHMgPT0gNTAwKS4gVGhpcyBpcyB0aGUgcmVhc29uIHdoeSB5b3UgbmVlZCB0aHJlc2hvbGRzIGlmIHlvdSB3YW50IHRvIGFzY2VydGFpbiB0aGUgYWNjdXJhY3kgb2Ygc3VjaCBhIG1vZGVsLiBJbiBZZWFyIDMsIHlvdSB3aWxsIGxlYXJuIGFib3V0IGFwcGxpY2F0aW9ucyBvZiB0aGlzIHRocmVzaG9sZGluZyBmb3IgbG9naXN0aWMgcmVncmVzc2lvbiBpbiBtYWNoaW5lIGxlYXJuaW5nLgoKWW91IGNhbiBzZWUgdGhlIHJlbGF0aW9uc2hpcCBiZXR3ZWVuIGZpdHRlZCB2YWx1ZXMgKGkuZS4gcHJvYmFiaWxpdGllcyBvZiBhIGNhc2UgYmVpbmcgaW4gb24gZW9mIHRoZSB0d28gb3V0Y29tZSBjbGFzc2VzICAtIGhhY2tlZCB2cyBub3QgaGFja2VkIC0gZ2l2ZW4gYSBjZXJ0YWluIG51bWJlciBvZiBhdHRlbXB0cykgYW5kIHRoZSBvYnNlcnZlZCB2YWx1ZXM6CgpgYGB7cn0KcGxvdChsb2dyZWckZml0dGVkLnZhbHVlcywgbG9ncmVnJG1vZGVsJGhhY2tlZCkKYGBgCgoKIyMgRU5ECgotLS0=