SOLUTIONS

Homework week 2: Regression modelling in R

The Global Terrorism Database

We will use a dataset that includes variables of 180,000 terrorist attacks between 1970-2017. Note that we excluded some variables for clarity and did some preprocessing on the data.

Suppose you are given that dataset and you’re asked to inform policy-makers about the relationship between terrorist attack details and the number of victims. This notebook asks you to perform initial descriptive statistics and then build and evaluate models of the data.

You can load the dataset as follows:

Variable codebook:

This command loads a dataframe called gtd in your notebook. You can query that dataframe as usual.

Have a look at the names (columns) in the data.frame:

names(gtd)
[1] "eventid" "iyear"   "imonth"  "iday"    "nperps"  "suicide" "ransom"  "nkill"  
[9] "nwound" 

… and show the first 10 rows:

head(gtd, 10)
         eventid iyear imonth iday nperps suicide ransom nkill nwound
1  9.733093e-313  1970      0    0      7       0      1     0      0
2  9.733144e-313  1970      1    2      3       0      0     0      0
3  9.733144e-313  1970      1    2      1       0      0     0      0
4  9.733144e-313  1970      1    3      1       0      0     0      0
5  9.733147e-313  1970      1    8      1       0      0     0      0
6  9.733148e-313  1970      1   11      1       0      0     1      0
7  9.733150e-313  1970      1   15      5       0      0     0      0
8  9.733152e-313  1970      1   19      3       0      0     0      0
9  9.733152e-313  1970      1   19      2       0      0     0      0
10 9.733153e-313  1970      1   20      1       0      0     1      0

Note that we excluded variables (for clarity) and observations (to avoid missing values), so the actual dimensions of this dataframe are:

dim(gtd)
[1] 9147    9

Understanding the data

Let’s start with understanding the data bit better. You’d want to do this to avoid modelling relationships that are not meaningful.

Task

Look at the frequencies of the number of perpetrators and subset these frequency counts by the suicide and ransom variable.

table(gtd$nperps)

    0     1     2     3     4     5     6     7     8     9    10    11    12    13 
   25  1346  1780  1272   955   530   365   150   154    63   230    25   112    20 
   14    15    16    17    18    19    20    21    22    23    24    25    26    28 
   18   192     5     9    19     4   290     7     5     1    18    72     3     1 
   29    30    33    34    35    36    39    40    42    44    45    47    48    49 
    1   267     1     1    15     8     2   128     1     1     3     1     1     1 
   50    51    55    56    60    68    70    72    75    80    90   100   101   120 
  219     3     1     1    91     1    35     1     5    42     3   211     1     8 
  130   138   140   150   160   180   190   200   204   250   290   300   350   400 
    7     1     2    82     2     3     1   137     1     7     1    64     5    18 
  500   600   700   750  1000  1200  1500  2500  3200 10000 20000 
   20     8     8     1    12     2     2     1     1    36     1 
table(gtd$nperps, gtd$suicide, gtd$ransom)
, ,  = 0

       
           0    1
  0       25    0
  1     1294   31
  2     1731   16
  3     1191   17
  4      868    9
  5      475    9
  6      336    4
  7      133    2
  8      139    1
  9       51    0
  10     215    0
  11      18    0
  12     103    0
  13      20    0
  14      17    0
  15     187    0
  16       5    0
  17       8    0
  18      12    0
  19       4    0
  20     279    0
  21       7    0
  22       5    0
  23       1    0
  24      17    0
  25      70    0
  26       3    0
  28       1    0
  29       1    0
  30     257    2
  33       1    0
  34       1    0
  35      15    0
  36       7    1
  39       2    0
  40     125    0
  42       1    0
  44       1    0
  45       3    0
  47       1    0
  48       1    0
  49       1    0
  50     218    0
  51       3    0
  55       1    0
  56       1    0
  60      90    0
  68       1    0
  70      35    0
  72       1    0
  75       5    0
  80      42    0
  90       3    0
  100    208    0
  101      1    0
  120      8    0
  130      7    0
  138      1    0
  140      2    0
  150     81    1
  160      2    0
  180      3    0
  190      1    0
  200    136    0
  204      1    0
  250      7    0
  290      1    0
  300     63    0
  350      5    0
  400     18    0
  500     20    0
  600      8    0
  700      8    0
  750      1    0
  1000    12    0
  1200     2    0
  1500     2    0
  2500     1    0
  3200     1    0
  10000   36    0
  20000    1    0

, ,  = 1

       
           0    1
  0        0    0
  1       20    1
  2       33    0
  3       64    0
  4       78    0
  5       46    0
  6       25    0
  7       15    0
  8       14    0
  9       12    0
  10      15    0
  11       7    0
  12       9    0
  13       0    0
  14       1    0
  15       5    0
  16       0    0
  17       1    0
  18       7    0
  19       0    0
  20      11    0
  21       0    0
  22       0    0
  23       0    0
  24       1    0
  25       2    0
  26       0    0
  28       0    0
  29       0    0
  30       8    0
  33       0    0
  34       0    0
  35       0    0
  36       0    0
  39       0    0
  40       3    0
  42       0    0
  44       0    0
  45       0    0
  47       0    0
  48       0    0
  49       0    0
  50       1    0
  51       0    0
  55       0    0
  56       0    0
  60       1    0
  68       0    0
  70       0    0
  72       0    0
  75       0    0
  80       0    0
  90       0    0
  100      3    0
  101      0    0
  120      0    0
  130      0    0
  138      0    0
  140      0    0
  150      0    0
  160      0    0
  180      0    0
  190      0    0
  200      1    0
  204      0    0
  250      0    0
  290      0    0
  300      1    0
  350      0    0
  400      0    0
  500      0    0
  600      0    0
  700      0    0
  750      0    0
  1000     0    0
  1200     0    0
  1500     0    0
  2500     0    0
  3200     0    0
  10000    0    0
  20000    0    0

Task

In which year was the number of perpetrators (on average) the highest?

perps_year[which.max(perps_year$nperps),] # 1996
   Group.1   nperps
26    1996 1110.928

Task

What is the most common value of the number of persons killed and wounded?

Hint: ?hist and ?table

table(gtd$nwound)

   0    1    2    3    4    5    6    7    8    9   10   11   12   13   14   15   16 
6512 1145  419  253  127   97   69   66   42   34   38   27   28   12   16   21    9 
  17   18   19   20   21   22   23   24   25   26   27   28   29   30   31   32   33 
  16   11   11   22    6    6    1    2   10    7    4    5    2   14    3    5    1 
  34   35   36   37   38   39   40   41   42   45   48   50   53   54   55   56   60 
   3    1    1    2    4    2    9    2    2    2    5   11    2    1    2    1    6 
  65   68   69   70   72   74   76   78   80   81   85   90   96   98  100  104  106 
   1    2    1    7    1    1    3    1    2    1    1    2    1    1    9    1    1 
 120  121  127  138  150  161  170  192  217  236  650  671 8190 8191 
   1    1    1    1    1    1    1    1    1    1    1    1    1    1 

Task

Display the relationship between the number of killed victims and the number of wounded victims in a figure.

What kind of a relationship do you expect?

Hint: ?plot

Task

What is the mean number of perpetrators when the attack was a suicide attack?

Hint: ?tapply

#your code comes here
tapply(gtd$nperps, gtd$suicide, mean)
        0         1 
65.476306  5.191489 

Simple regression

We’ll start with simple regression (i.e. one predictor variable):

Task

Build a simple regression model that models the number of wounded victims through the number of perpetrators.

summary(lm1)

Call:
lm(formula = gtd$nwound ~ gtd$nperps)

Residuals:
   Min     1Q Median     3Q    Max 
  -3.8   -3.8   -3.8   -2.8 8187.2 

Coefficients:
              Estimate Std. Error t value Pr(>|t|)   
(Intercept)  3.8327993  1.2796622   2.995  0.00275 **
gtd$nperps  -0.0002303  0.0019183  -0.120  0.90445   
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 121.8 on 9145 degrees of freedom
Multiple R-squared:  1.576e-06, Adjusted R-squared:  -0.0001078 
F-statistic: 0.01441 on 1 and 9145 DF,  p-value: 0.9045

Task

How satisfied are you with your model? One way to assess the “model fit” is to calculate the root mean square error - RMSE - (residuals). Calculate that metric and think about the meaning of this fit index. What does it tell you and how satisfied are you with it?

sqrt(mean(lm1$residuals^2))
[1] 121.7937

Task

Plot the fitted values (in green) and the observed values (in blue) to assess the model fit visually.

Multiple regression

Now you might want to use multiple variables in your model:

Task

Build a multiple regression model that models the number of killed victims through the variables suicide and ransom. Include only the two main effects (and let the intercept in the model).

summary(lm2)

Call:
lm(formula = gtd$nkill ~ gtd$suicide + gtd$ransom)

Residuals:
    Min      1Q  Median      3Q     Max 
 -50.11   -2.43   -1.43   -0.51 1333.89 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)   2.4273     0.2385  10.177   <2e-16 ***
gtd$suicide  47.6782     2.3028  20.705   <2e-16 ***
gtd$ransom   -1.9174     1.1566  -1.658   0.0974 .  
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 22.21 on 9144 degrees of freedom
Multiple R-squared:  0.04518,   Adjusted R-squared:  0.04497 
F-statistic: 216.3 on 2 and 9144 DF,  p-value: < 2.2e-16

Task

Have a look at a potential interaction between these two predictor variables. Use the interaction.plot function to look at the joint relationship of these two variables on the number of killed victims.

What does this graph tell you? Can you identify the main effects and (potential) interaction?

Task

Now look at the interaction in a numerical manner.

Hint: ?tapply

tapply(gtd$nkill, list(gtd$suicide, gtd$ransom), mean)
          0         1
0  2.421848 0.6328125
1 50.612903 1.0000000

Task

Suppose you want to expand the model by adding the interaction term to it. Build that model.

summary(lm3)

Call:
lm(formula = gtd$nkill ~ gtd$suicide * gtd$ransom)

Residuals:
    Min      1Q  Median      3Q     Max 
 -50.61   -2.42   -1.42   -0.63 1333.39 

Coefficients:
                       Estimate Std. Error t value Pr(>|t|)    
(Intercept)              2.4218     0.2385  10.155   <2e-16 ***
gtd$suicide             48.1911     2.3148  20.819   <2e-16 ***
gtd$ransom              -1.7890     1.1579  -1.545   0.1224    
gtd$suicide:gtd$ransom -47.8239    22.3531  -2.139   0.0324 *  
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 22.2 on 9143 degrees of freedom
Multiple R-squared:  0.04565,   Adjusted R-squared:  0.04534 
F-statistic: 145.8 on 3 and 9143 DF,  p-value: < 2.2e-16

Task

Based on the RMSE of each of the two models above (2 main effects vs 2 main effects + 1 interaction), which one do you prefer?

sqrt(mean(lm3$residuals^2))
[1] 22.19921

Task

Have a look at the distribution of the nperps and nkill column. Are there some potential outliers in there?

Hint: ?plot

Task

Re-run the best fitting regression model again after exluding the potential outliers. The decision for the “best” model can be made based on the RMSE:

sqrt(mean(lm_clean$residuals^2))
[1] 9.78957

Model selection

Now suppose you want to let the model building process be decided by a stepwise model selection procedure.

Task

Build a “null model” that only contains an intercept.

summary(null_model)

Call:
lm(formula = gtd_clean$nwound ~ 1)

Residuals:
   Min     1Q Median     3Q    Max 
 -2.03  -2.03  -2.03  -1.03 668.97 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)   2.0274     0.1388   14.61   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 13.27 on 9144 degrees of freedom

Task

Build a full model for the number of wounded victims modeled through the number of perpetrators, “suicide” and “ransom”.

summary(lm4)

Call:
lm(formula = gtd_clean$nwound ~ gtd_clean$nperps * gtd_clean$suicide * 
    gtd_clean$ransom)

Residuals:
   Min     1Q Median     3Q    Max 
-32.87  -1.77  -1.77  -0.77 669.23 

Coefficients: (1 not defined because of singularities)
                                                      Estimate Std. Error t value
(Intercept)                                          1.771e+00  1.395e-01  12.701
gtd_clean$nperps                                     4.745e-05  2.035e-04   0.233
gtd_clean$suicide                                    3.120e+01  1.430e+00  21.820
gtd_clean$ransom                                    -1.380e+00  7.303e-01  -1.890
gtd_clean$nperps:gtd_clean$suicide                  -1.044e-01  8.345e-02  -1.251
gtd_clean$nperps:gtd_clean$ransom                    1.684e-02  3.153e-02   0.534
gtd_clean$suicide:gtd_clean$ransom                  -3.151e+01  1.302e+01  -2.421
gtd_clean$nperps:gtd_clean$suicide:gtd_clean$ransom         NA         NA      NA
                                                    Pr(>|t|)    
(Intercept)                                           <2e-16 ***
gtd_clean$nperps                                      0.8157    
gtd_clean$suicide                                     <2e-16 ***
gtd_clean$ransom                                      0.0588 .  
gtd_clean$nperps:gtd_clean$suicide                    0.2111    
gtd_clean$nperps:gtd_clean$ransom                     0.5932    
gtd_clean$suicide:gtd_clean$ransom                    0.0155 *  
gtd_clean$nperps:gtd_clean$suicide:gtd_clean$ransom       NA    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 12.92 on 9138 degrees of freedom
Multiple R-squared:  0.05327,   Adjusted R-squared:  0.05265 
F-statistic:  85.7 on 6 and 9138 DF,  p-value: < 2.2e-16

Task

Determine the best fitting model in a backward model selection procedure.

#your code comes here
step(lm4, direction = 'backward')
Start:  AIC=46807.07
gtd_clean$nwound ~ gtd_clean$nperps * gtd_clean$suicide * gtd_clean$ransom


Step:  AIC=46807.07
gtd_clean$nwound ~ gtd_clean$nperps + gtd_clean$suicide + gtd_clean$ransom + 
    gtd_clean$nperps:gtd_clean$suicide + gtd_clean$nperps:gtd_clean$ransom + 
    gtd_clean$suicide:gtd_clean$ransom

                                     Df Sum of Sq     RSS   AIC
- gtd_clean$nperps:gtd_clean$ransom   1     47.64 1525430 46805
- gtd_clean$nperps:gtd_clean$suicide  1    261.08 1525644 46807
<none>                                            1525382 46807
- gtd_clean$suicide:gtd_clean$ransom  1    978.23 1526361 46811

Step:  AIC=46805.36
gtd_clean$nwound ~ gtd_clean$nperps + gtd_clean$suicide + gtd_clean$ransom + 
    gtd_clean$nperps:gtd_clean$suicide + gtd_clean$suicide:gtd_clean$ransom

                                     Df Sum of Sq     RSS   AIC
- gtd_clean$nperps:gtd_clean$suicide  1    261.09 1525691 46805
<none>                                            1525430 46805
- gtd_clean$suicide:gtd_clean$ransom  1    986.90 1526417 46809

Step:  AIC=46804.92
gtd_clean$nwound ~ gtd_clean$nperps + gtd_clean$suicide + gtd_clean$ransom + 
    gtd_clean$suicide:gtd_clean$ransom

                                     Df Sum of Sq     RSS   AIC
- gtd_clean$nperps                    1      9.11 1525700 46803
<none>                                            1525691 46805
- gtd_clean$suicide:gtd_clean$ransom  1    960.18 1526651 46809

Step:  AIC=46802.98
gtd_clean$nwound ~ gtd_clean$suicide + gtd_clean$ransom + gtd_clean$suicide:gtd_clean$ransom

                                     Df Sum of Sq     RSS   AIC
<none>                                            1525700 46803
- gtd_clean$suicide:gtd_clean$ransom  1    960.02 1526660 46807

Call:
lm(formula = gtd_clean$nwound ~ gtd_clean$suicide + gtd_clean$ransom + 
    gtd_clean$suicide:gtd_clean$ransom)

Coefficients:
                       (Intercept)                   gtd_clean$suicide  
                             1.774                              30.654  
                  gtd_clean$ransom  gtd_clean$suicide:gtd_clean$ransom  
                            -1.233                             -31.196  

Task

Run the model selection again but this time using the forward model selection procedure.

step(lm4, direction = 'forward'
     , scope = list(lower = null_model, upper = lm4))
Start:  AIC=46807.07
gtd_clean$nwound ~ gtd_clean$nperps * gtd_clean$suicide * gtd_clean$ransom


Call:
lm(formula = gtd_clean$nwound ~ gtd_clean$nperps * gtd_clean$suicide * 
    gtd_clean$ransom)

Coefficients:
                                        (Intercept)  
                                          1.771e+00  
                                   gtd_clean$nperps  
                                          4.745e-05  
                                  gtd_clean$suicide  
                                          3.120e+01  
                                   gtd_clean$ransom  
                                         -1.380e+00  
                 gtd_clean$nperps:gtd_clean$suicide  
                                         -1.044e-01  
                  gtd_clean$nperps:gtd_clean$ransom  
                                          1.684e-02  
                 gtd_clean$suicide:gtd_clean$ransom  
                                         -3.151e+01  
gtd_clean$nperps:gtd_clean$suicide:gtd_clean$ransom  
                                                 NA  

Task

Compare the RMSE of the null model, the full model and the best fitting model (if different from the full model).

Display the residuals in a graph using the colours green, black, and blue for the null model, observed values, and the full model, respectively.

Hint: ?points

Did you expect the observations for the null model? See whether you can discover the reason for that relationship in the model outcome (i.e. the coefficients).

END


LS0tCnRpdGxlOiAiU29sdXRpb25zOiBSZWdyZXNzaW9uIG1vZGVsbGluZyBpbiBSLCBQU00yIgphdXRob3I6IEIgS2xlaW5iZXJnLCBJIHZhbiBkZXIgVmVndApzdWJ0aXRsZTogRGVwdCBvZiBTZWN1cml0eSBhbmQgQ3JpbWUgU2NpZW5jZSwgVUNMCm91dHB1dDogaHRtbF9ub3RlYm9vawotLS0KCioqU09MVVRJT05TKioKCkhvbWV3b3JrIHdlZWsgMjogUmVncmVzc2lvbiBtb2RlbGxpbmcgaW4gUgoKIyMgVGhlIEdsb2JhbCBUZXJyb3Jpc20gRGF0YWJhc2UKCldlIHdpbGwgdXNlIGEgZGF0YXNldCB0aGF0IGluY2x1ZGVzIHZhcmlhYmxlcyBvZiBbMTgwLDAwMCB0ZXJyb3Jpc3QgYXR0YWNrcyBiZXR3ZWVuIDE5NzAtMjAxN10oaHR0cHM6Ly93d3cua2FnZ2xlLmNvbS9TVEFSVC1VTUQvZ3RkL3ZlcnNpb24vMykuIE5vdGUgdGhhdCB3ZSBleGNsdWRlZCBzb21lIHZhcmlhYmxlcyBmb3IgY2xhcml0eSBhbmQgZGlkIHNvbWUgcHJlcHJvY2Vzc2luZyBvbiB0aGUgZGF0YS4KClN1cHBvc2UgeW91IGFyZSBnaXZlbiB0aGF0IGRhdGFzZXQgYW5kIHlvdSdyZSBhc2tlZCB0byBpbmZvcm0gcG9saWN5LW1ha2VycyBhYm91dCB0aGUgcmVsYXRpb25zaGlwIGJldHdlZW4gdGVycm9yaXN0IGF0dGFjayBkZXRhaWxzIGFuZCB0aGUgbnVtYmVyIG9mIHZpY3RpbXMuIFRoaXMgbm90ZWJvb2sgYXNrcyB5b3UgdG8gcGVyZm9ybSBpbml0aWFsIGRlc2NyaXB0aXZlIHN0YXRpc3RpY3MgYW5kIHRoZW4gYnVpbGQgYW5kIGV2YWx1YXRlIG1vZGVscyBvZiB0aGUgZGF0YS4KCllvdSBjYW4gbG9hZCB0aGUgZGF0YXNldCBhcyBmb2xsb3dzOgoKYGBge3J9CmxvYWQoJy4vZGF0YS9nbG9iYWxfdGVycm9yaXNtX2RhdGFiYXNlLlJEYXRhJykKYGBgCgpbVmFyaWFibGUgY29kZWJvb2tdKGh0dHBzOi8vd3d3LnN0YXJ0LnVtZC5lZHUvZ3RkL2Rvd25sb2Fkcy9Db2RlYm9vay5wZGYpOgoKLSBldmVudGlkOiB1bmlxdWUgZXZlbnQgaWRlbnRpZmllcgotIGl5ZWFyOiB5ZWFyCi0gaW1vbnRoOiBtb250aAotIGlkYXk6IGRheQotIG5wZXJwczogbnVtYmVyIG9mIHBlcnBldHJhdG9ycwotIHN1aWNpZGU6IHdoZXRoZXIgdGhlIGF0dGFjayB3YXMgYSBzdWljaWRlIGF0dGFjaAotIHJhbnNvbTogd2hldGhlciByYW5zb20gd2FzIGRlbWFuZGVkCi0gbmtpbGw6IG51bWJlciBvZiBraWxsZWQgdmljdGltcwotIG53b3VuZDogbnVtYmVyIG9mIHdvdW5kZWQgdmljdGltcwoKClRoaXMgY29tbWFuZCBsb2FkcyBhIGRhdGFmcmFtZSBjYWxsZWQgYGd0ZGAgaW4geW91ciBub3RlYm9vay4gWW91IGNhbiBxdWVyeSB0aGF0IGRhdGFmcmFtZSAgYXMgdXN1YWwuCgpIYXZlIGEgbG9vayBhdCB0aGUgbmFtZXMgKGNvbHVtbnMpIGluIHRoZSBkYXRhLmZyYW1lOgoKYGBge3J9Cm5hbWVzKGd0ZCkKYGBgCgouLi4gYW5kIHNob3cgdGhlIGZpcnN0IDEwIHJvd3M6CgpgYGB7cn0KaGVhZChndGQsIDEwKQpgYGAKCgpOb3RlIHRoYXQgd2UgZXhjbHVkZWQgdmFyaWFibGVzIChmb3IgY2xhcml0eSkgYW5kIG9ic2VydmF0aW9ucyAodG8gYXZvaWQgbWlzc2luZyB2YWx1ZXMpLCBzbyB0aGUgYWN0dWFsIGRpbWVuc2lvbnMgb2YgdGhpcyBkYXRhZnJhbWUgYXJlOgoKYGBge3J9CmRpbShndGQpCmBgYAoKIyMgVW5kZXJzdGFuZGluZyB0aGUgZGF0YQoKTGV0J3Mgc3RhcnQgd2l0aCB1bmRlcnN0YW5kaW5nIHRoZSBkYXRhIGJpdCBiZXR0ZXIuIFlvdSdkIHdhbnQgdG8gZG8gdGhpcyB0byBhdm9pZCBtb2RlbGxpbmcgcmVsYXRpb25zaGlwcyB0aGF0IGFyZSBub3QgbWVhbmluZ2Z1bC4KCiMjIyBUYXNrCgpMb29rIGF0IHRoZSBmcmVxdWVuY2llcyBvZiB0aGUgbnVtYmVyIG9mIHBlcnBldHJhdG9ycyBhbmQgc3Vic2V0IHRoZXNlIGZyZXF1ZW5jeSBjb3VudHMgYnkgdGhlIHN1aWNpZGUgYW5kIHJhbnNvbSB2YXJpYWJsZS4KCmBgYHtyfQojeW91ciBjb2RlIGNvbWVzIGhlcmUKdGFibGUoZ3RkJG5wZXJwcykKdGFibGUoZ3RkJG5wZXJwcywgZ3RkJHN1aWNpZGUsIGd0ZCRyYW5zb20pCmBgYAoKIyMjIFRhc2sKCkluIHdoaWNoIHllYXIgd2FzIHRoZSBudW1iZXIgb2YgcGVycGV0cmF0b3JzIChvbiBhdmVyYWdlKSB0aGUgaGlnaGVzdD8KCmBgYHtyfQojeW91ciBjb2RlIGNvbWVzIGhlcmUKcGVycHNfeWVhciA9IGFnZ3JlZ2F0ZShndGRbIm5wZXJwcyJdLCBsaXN0KGd0ZCRpeWVhciksIG1lYW4pCnBlcnBzX3llYXJbd2hpY2gubWF4KHBlcnBzX3llYXIkbnBlcnBzKSxdICMgMTk5NgpgYGAKCiMjIyBUYXNrCgpXaGF0IGlzIHRoZSBtb3N0IGNvbW1vbiB2YWx1ZSBvZiB0aGUgbnVtYmVyIG9mIHBlcnNvbnMga2lsbGVkIGFuZCB3b3VuZGVkPwoKSGludDogYD9oaXN0YCBhbmQgYD90YWJsZWAKCmBgYHtyfQojeW91ciBjb2RlIGNvbWVzIGhlcmUKa2lsbF9jb3VudHMgPSBoaXN0KGd0ZCRua2lsbCkKd291bmRfY291bnRzID0gaGlzdChndGQkbndvdW5kKQoKdGFibGUoZ3RkJG5raWxsKQp0YWJsZShndGQkbndvdW5kKQojIDAgbW9zdCBmcmVxdWVudCBmb3IgYm90aApgYGAKIyMjIFRhc2sKCkRpc3BsYXkgdGhlIHJlbGF0aW9uc2hpcCBiZXR3ZWVuIHRoZSBudW1iZXIgb2Yga2lsbGVkIHZpY3RpbXMgYW5kIHRoZSBudW1iZXIgb2Ygd291bmRlZCB2aWN0aW1zIGluIGEgZmlndXJlLgoKV2hhdCBraW5kIG9mIGEgcmVsYXRpb25zaGlwIGRvIHlvdSBleHBlY3Q/CgpIaW50OiBgP3Bsb3RgCgpgYGB7cn0KI3lvdXIgY29kZSBjb21lcyBoZXJlCnBsb3QoZ3RkJG53b3VuZCwgZ3RkJG5raWxsKQpgYGAKCiMjIyBUYXNrCgpXaGF0IGlzIHRoZSBtZWFuIG51bWJlciBvZiBwZXJwZXRyYXRvcnMgd2hlbiB0aGUgYXR0YWNrIHdhcyBhIHN1aWNpZGUgYXR0YWNrPwoKSGludDogYD90YXBwbHlgCgpgYGB7cn0KI3lvdXIgY29kZSBjb21lcyBoZXJlCnRhcHBseShndGQkbnBlcnBzLCBndGQkc3VpY2lkZSwgbWVhbikKYGBgCgojIyBTaW1wbGUgcmVncmVzc2lvbgoKV2UnbGwgc3RhcnQgd2l0aCBzaW1wbGUgcmVncmVzc2lvbiAoaS5lLiBvbmUgcHJlZGljdG9yIHZhcmlhYmxlKToKCgojIyMgVGFzawoKQnVpbGQgYSBzaW1wbGUgcmVncmVzc2lvbiBtb2RlbCB0aGF0IG1vZGVscyB0aGUgbnVtYmVyIG9mIHdvdW5kZWQgdmljdGltcyB0aHJvdWdoIHRoZSBudW1iZXIgb2YgcGVycGV0cmF0b3JzLgoKYGBge3J9CiN5b3VyIGNvZGUgY29tZXMgaGVyZQpsbTEgPSBsbShndGQkbndvdW5kIH4gZ3RkJG5wZXJwcykKc3VtbWFyeShsbTEpCmBgYAoKIyMjIFRhc2sKCkhvdyBzYXRpc2ZpZWQgYXJlIHlvdSB3aXRoIHlvdXIgbW9kZWw/IE9uZSB3YXkgdG8gYXNzZXNzIHRoZSAibW9kZWwgZml0IiBpcyB0byBjYWxjdWxhdGUgdGhlIHJvb3QgbWVhbiBzcXVhcmUgZXJyb3IgLSBSTVNFIC0gKHJlc2lkdWFscykuIENhbGN1bGF0ZSB0aGF0IG1ldHJpYyBhbmQgdGhpbmsgYWJvdXQgdGhlIG1lYW5pbmcgb2YgdGhpcyBmaXQgaW5kZXguIFdoYXQgZG9lcyBpdCB0ZWxsIHlvdSBhbmQgaG93IHNhdGlzZmllZCBhcmUgeW91IHdpdGggaXQ/CgpgYGB7cn0KI3lvdXIgY29kZSBjb21lcyBoZXJlCnNxcnQobWVhbihsbTEkcmVzaWR1YWxzXjIpKQpgYGAKCiMjIyBUYXNrCgpQbG90IHRoZSBmaXR0ZWQgdmFsdWVzIChpbiBncmVlbikgYW5kIHRoZSBvYnNlcnZlZCB2YWx1ZXMgKGluIGJsdWUpIHRvIGFzc2VzcyB0aGUgbW9kZWwgZml0IHZpc3VhbGx5LgoKYGBge3J9CiN5b3VyIGNvZGUgY29tZXMgaGVyZQoKe3Bsb3QoZ3RkJG53b3VuZCwgY29sID0gJ2dyZWVuJywgeWxpbSA9IGMoMCwgMjUwKSkKcG9pbnRzKGxtMSRmaXR0ZWQudmFsdWVzLCBjb2wgPSAnYmx1ZScpfQoKYGBgCgoKIyMgTXVsdGlwbGUgcmVncmVzc2lvbgoKTm93IHlvdSBtaWdodCB3YW50IHRvIHVzZSBtdWx0aXBsZSB2YXJpYWJsZXMgaW4geW91ciBtb2RlbDoKCiMjIyBUYXNrCgpCdWlsZCBhIG11bHRpcGxlIHJlZ3Jlc3Npb24gbW9kZWwgdGhhdCBtb2RlbHMgdGhlIG51bWJlciBvZiBraWxsZWQgdmljdGltcyB0aHJvdWdoIHRoZSB2YXJpYWJsZXMgc3VpY2lkZSBhbmQgcmFuc29tLiBJbmNsdWRlIG9ubHkgdGhlIHR3byBtYWluIGVmZmVjdHMgKGFuZCBsZXQgdGhlIGludGVyY2VwdCBpbiB0aGUgbW9kZWwpLgoKYGBge3J9CiN5b3VyIGNvZGUgY29tZXMgaGVyZQoKbG0yID0gbG0oZ3RkJG5raWxsIH4gZ3RkJHN1aWNpZGUgKyBndGQkcmFuc29tKQpzdW1tYXJ5KGxtMikKYGBgCgoKIyMjIFRhc2sKCkhhdmUgYSBsb29rIGF0IGEgcG90ZW50aWFsIGludGVyYWN0aW9uIGJldHdlZW4gdGhlc2UgdHdvIHByZWRpY3RvciB2YXJpYWJsZXMuIFVzZSB0aGUgYGludGVyYWN0aW9uLnBsb3RgIGZ1bmN0aW9uIHRvIGxvb2sgYXQgdGhlIGpvaW50IHJlbGF0aW9uc2hpcCBvZiB0aGVzZSB0d28gdmFyaWFibGVzIG9uIHRoZSBudW1iZXIgb2Yga2lsbGVkIHZpY3RpbXMuCgpgYGB7cn0KI3lvdXIgY29kZSBjb21lcyBoZXJlCgppbnRlcmFjdGlvbi5wbG90KGd0ZCRzdWljaWRlLCBndGQkcmFuc29tLCBndGQkbmtpbGwpCgpgYGAKCldoYXQgZG9lcyB0aGlzIGdyYXBoIHRlbGwgeW91PyBDYW4geW91IGlkZW50aWZ5IHRoZSBtYWluIGVmZmVjdHMgYW5kIChwb3RlbnRpYWwpIGludGVyYWN0aW9uPwoKIyMjIFRhc2sKCk5vdyBsb29rIGF0IHRoZSBpbnRlcmFjdGlvbiBpbiBhIG51bWVyaWNhbCBtYW5uZXIuCgpIaW50OiBgP3RhcHBseWAKCgpgYGB7cn0KI3lvdXIgY29kZSBjb21lcyBoZXIKCnRhcHBseShndGQkbmtpbGwsIGxpc3QoZ3RkJHN1aWNpZGUsIGd0ZCRyYW5zb20pLCBtZWFuKQoKYGBgCgoKIyMjIFRhc2sKClN1cHBvc2UgeW91IHdhbnQgdG8gZXhwYW5kIHRoZSBtb2RlbCBieSBhZGRpbmcgdGhlIGludGVyYWN0aW9uIHRlcm0gdG8gaXQuIEJ1aWxkIHRoYXQgbW9kZWwuCgoKYGBge3J9CiN5b3VyIGNvZGUgY29tZXMgaGVyZQpsbTMgPSBsbShndGQkbmtpbGwgfiBndGQkc3VpY2lkZSAqIGd0ZCRyYW5zb20pCnN1bW1hcnkobG0zKQpgYGAKCgojIyMgVGFzawoKQmFzZWQgb24gdGhlIFJNU0Ugb2YgZWFjaCBvZiB0aGUgdHdvIG1vZGVscyBhYm92ZSAoMiBtYWluIGVmZmVjdHMgdnMgMiBtYWluIGVmZmVjdHMgKyAxIGludGVyYWN0aW9uKSwgd2hpY2ggb25lIGRvIHlvdSBwcmVmZXI/CgpgYGB7cn0KI3lvdXIgY29kZSBjb21lcyBoZXJlCnNxcnQobWVhbihsbTIkcmVzaWR1YWxzXjIpKQpzcXJ0KG1lYW4obG0zJHJlc2lkdWFsc14yKSkKYGBgCgojIyMgVGFzawoKSGF2ZSBhIGxvb2sgYXQgdGhlIGRpc3RyaWJ1dGlvbiBvZiB0aGUgYG5wZXJwc2AgYW5kIGBua2lsbGAgY29sdW1uLiBBcmUgdGhlcmUgc29tZSBwb3RlbnRpYWwgb3V0bGllcnMgaW4gdGhlcmU/CgpIaW50OiBgP3Bsb3RgCgpgYGB7cn0KcGxvdChndGQkbnBlcnBzKQpgYGAKCgpgYGB7cn0KI3lvdXIgY29kZSBjb21lcyBoZXJlCnBsb3QoZ3RkJG5raWxsKQpgYGAKCiMjIyBUYXNrCgpSZS1ydW4gdGhlIGJlc3QgZml0dGluZyByZWdyZXNzaW9uIG1vZGVsIGFnYWluIGFmdGVyIGV4bHVkaW5nIHRoZSBwb3RlbnRpYWwgb3V0bGllcnMuIFRoZSBkZWNpc2lvbiBmb3IgdGhlICJiZXN0IiBtb2RlbCBjYW4gYmUgbWFkZSBiYXNlZCBvbiB0aGUgUk1TRToKCmBgYHtyfQojeW91ciBjb2RlIGNvbWVzIGhlcmUKZ3RkX2NsZWFuID0gZ3RkW3doaWNoKGd0ZCRucGVycHMgPCA1MDApLF0KZ3RkX2NsZWFuID0gZ3RkW3doaWNoKGd0ZCRua2lsbCA8IDYwMCksXQoKbG1fY2xlYW4gPSBsbShndGRfY2xlYW4kbmtpbGwgfiBndGRfY2xlYW4kc3VpY2lkZSAqIGd0ZF9jbGVhbiRyYW5zb20pCnNxcnQobWVhbihsbV9jbGVhbiRyZXNpZHVhbHNeMikpCmBgYAoKCiMjIE1vZGVsIHNlbGVjdGlvbgoKTm93IHN1cHBvc2UgeW91IHdhbnQgdG8gbGV0IHRoZSBtb2RlbCBidWlsZGluZyBwcm9jZXNzIGJlIGRlY2lkZWQgYnkgYSBzdGVwd2lzZSBtb2RlbCBzZWxlY3Rpb24gcHJvY2VkdXJlLgoKIyMjIFRhc2sKCkJ1aWxkIGEgIm51bGwgbW9kZWwiIHRoYXQgb25seSBjb250YWlucyBhbiBpbnRlcmNlcHQuCgpgYGB7cn0KI3lvdXIgY29kZSBjb21lcyBoZXJlCm51bGxfbW9kZWwgPSBsbShndGRfY2xlYW4kbndvdW5kIH4gMSkKc3VtbWFyeShudWxsX21vZGVsKQpgYGAKCgojIyMgVGFzawoKQnVpbGQgYSBmdWxsIG1vZGVsIGZvciB0aGUgbnVtYmVyIG9mIHdvdW5kZWQgdmljdGltcyBtb2RlbGVkIHRocm91Z2ggdGhlIG51bWJlciBvZiBwZXJwZXRyYXRvcnMsICJzdWljaWRlIiBhbmQgInJhbnNvbSIuCgpgYGB7cn0KI3lvdXIgY29kZSBjb21lcyBoZXJlCmxtNCA9IGxtKGd0ZF9jbGVhbiRud291bmQgfiBndGRfY2xlYW4kbnBlcnBzICogZ3RkX2NsZWFuJHN1aWNpZGUgKiBndGRfY2xlYW4kcmFuc29tKQpzdW1tYXJ5KGxtNCkKYGBgCgoKIyMjIFRhc2sKCkRldGVybWluZSB0aGUgYmVzdCBmaXR0aW5nIG1vZGVsIGluIGEgYmFja3dhcmQgbW9kZWwgc2VsZWN0aW9uIHByb2NlZHVyZS4KCmBgYHtyfQojeW91ciBjb2RlIGNvbWVzIGhlcmUKc3RlcChsbTQsIGRpcmVjdGlvbiA9ICdiYWNrd2FyZCcpCmBgYAoKIyMjIFRhc2sKClJ1biB0aGUgbW9kZWwgc2VsZWN0aW9uIGFnYWluIGJ1dCB0aGlzIHRpbWUgdXNpbmcgdGhlIGZvcndhcmQgbW9kZWwgc2VsZWN0aW9uIHByb2NlZHVyZS4KCmBgYHtyfQojeW91ciBjb2RlIGNvbWVzIGhlcmUKc3RlcChsbTQsIGRpcmVjdGlvbiA9ICdmb3J3YXJkJwogICAgICwgc2NvcGUgPSBsaXN0KGxvd2VyID0gbnVsbF9tb2RlbCwgdXBwZXIgPSBsbTQpKQpgYGAKCgojIyMgVGFzawoKQ29tcGFyZSB0aGUgUk1TRSBvZiB0aGUgbnVsbCBtb2RlbCwgdGhlIGZ1bGwgbW9kZWwgYW5kIHRoZSBiZXN0IGZpdHRpbmcgbW9kZWwgKGlmIGRpZmZlcmVudCBmcm9tIHRoZSBmdWxsIG1vZGVsKS4KCkRpc3BsYXkgdGhlIHJlc2lkdWFscyBpbiBhIGdyYXBoIHVzaW5nIHRoZSBjb2xvdXJzIGdyZWVuLCBibGFjaywgYW5kIGJsdWUgZm9yIHRoZSBudWxsIG1vZGVsLCBvYnNlcnZlZCB2YWx1ZXMsIGFuZCB0aGUgZnVsbCBtb2RlbCwgcmVzcGVjdGl2ZWx5LgoKSGludDogYD9wb2ludHNgCgpgYGB7cn0KI3lvdXIgY29kZSBjb21lcyBoZXJlCnNxcnQobWVhbihudWxsX21vZGVsJHJlc2lkdWFsc14yKSkKc3FydChtZWFuKGxtNCRyZXNpZHVhbHNeMikpCgp7cGxvdChudWxsX21vZGVsJHJlc2lkdWFscywgY29sID0gJ2dyZWVuJywgeWxpbT1jKC0xMDAsIDYwMCkpCnBvaW50cyhsbTQkcmVzaWR1YWxzLCBjb2wgPSAnYmx1ZScpCnBvaW50cyhndGRfY2xlYW4kbndvdW5kLCBjb2wgPSAnYmxhY2snKX0KYGBgCgpEaWQgeW91IGV4cGVjdCB0aGUgb2JzZXJ2YXRpb25zIGZvciB0aGUgbnVsbCBtb2RlbD8gU2VlIHdoZXRoZXIgeW91IGNhbiBkaXNjb3ZlciB0aGUgcmVhc29uIGZvciB0aGF0IHJlbGF0aW9uc2hpcCBpbiB0aGUgbW9kZWwgb3V0Y29tZSAoaS5lLiB0aGUgY29lZmZpY2llbnRzKS4KCiMjIEVORAoKLS0t