Solutions: Text mining in R

Dept of Security and Crime Science, UCL

SOLUTIONS

---

Tutorial 3, Advanced Crime Analysis, BSc Security and Crime Science, UCL

---

Aim of this tutorial

You will use concepts learned in the lectures to:

• use computational linguistics to query text data
• represent text corpora as
• examine the sentiment of text data
• use psycholinguistics to look at additional text variables

Task 1: Preparing the corpus

In this tutorial, you will explore a unique dataset of YouTube transcripts extracted from left-leaning and right-leaning news channels. In the provided dataset, you have the transcripts of 2000 YouTube videos each from FoxNews (a right-leaning US news channel) and from The Young Turks (a left-leaning US news outlet).

Load the original dataframe called media_data from data/media_data.RData.

Make sure you have the following packages installed+loaded into your workspace:

• quanteda
• stringr
• sentimentr
• syuzhet

library(sentimentr)

Attaching package: ‘sentimentr’

The following object is masked from ‘package:syuzhet’:

    get_sentences

Take a look at the data and identify the column that contains the text data:

#your code
summary(media_data)
head(media_data)
tail(media_data)

Now, before you create a corpus from the text column, makle sure that all strings are in the same format. You will see that some are all UPPERCASE. Equally, you can observe that some contain punctuation, while others do not.

Fix this by creating a new column called text_clean that contains the lowercased strings and has the punctuation removed. (hint: take a look at the super useful stringr package and this related SO question)

Now use the text_clean column to create a corpus object called media_corpus from the quanteda package. Remove the orginal text column to avoid excessive data structures and the rename the column text_clean to text (this is important for quanteda to knwo where the text is located):

names(media_data_)

 [1] "channel_vlog_id"      "Filename"             "file_id.x"           
 [4] "file_parent"          "nwords"               "id"                  
 [7] "vlog_id"              "channel_id"           "file_id.y"           
[10] "url"                  "view_count"           "date_posted"         
[13] "landing_url"          "upvotes"              "downvotes"           
[16] "days_until_reference" "view_count_corrected" "pol"                 
[19] "eng_prop"             "ascii"                "channel_name"        
[22] "text_clean"          

Take a look at the summary of the new object media_corpus:

#your code
summary(media_corpus)

Use the summary function (hint: you might want to change the n= argument) on the corpus to create the object corpus_statistics and calculate:

• the total number of tokens in the corpus
• the type/token ratio for each document
• the average type/token ration for both channels separately (hint: tapply)

#the average type/token ration for both channels separately
tapply(corpus_statistics$TTR, corpus_statistics$channel_name, mean)

foxnewschannel  theyoungturks 
     0.4436150      0.3447911 

Task 2: Building a TFIDF representation

Next, build a TFIDF representation from the corpus using the count TF representation and the inverse DF representation. Use the stemmed tokens but leave the stopwords in when you create a DFM.

Step 1: create the DFM

Take a look at the first 10 rows and first ten columns of your DFM.

#your code
media_dfm[1:10, 1:10]

Document-feature matrix of: 10 documents, 10 features (56% sparse).
10 x 10 sparse Matrix of class "dfm"
        features
docs     all right and neo nazi leader richard spencer spoke at
  text1   13    23  76   3    8      1      11      18     1  4
  text2    3     0  41   0    0      0       1       0     0  3
  text3    1     4  21   0    0      1       0       0     0  1
  text4    1     0  25   0    0      0       0       0     0  2
  text5    3     1   9   0    0      0       0       0     0  3
  text6    7    10  48   0    0      0       0       0     0  3
  text7   11    16  40   0    0      1       0       0     0  3
  text8    4     4  14   0    0      0       0       0     0  0
  text9    8     4  46   0    0      0       0       0     0  8
  text10   2     0   3   0    0      0       0       0     0  0

Step 2: Weigh the DFM to a TFIDF representation

Again, take a look at the first 10 rows and first ten columns of your TFIDF-DFM.

#your code
media_tfidf[1:10, 1:10]

Document-feature matrix of: 10 documents, 10 features (56% sparse).
10 x 10 sparse Matrix of class "dfm"
        features
docs            all     right         and      neo     nazi    leader   richard
  text1  0.78787231 2.3894086 0.041283784 5.002685 11.33895 0.7099654 16.751666
  text2  0.18181669 0         0.022271515 0         0       0          1.522879
  text3  0.06060556 0.4155493 0.011407361 0         0       0.7099654  0       
  text4  0.06060556 0         0.013580192 0         0       0          0       
  text5  0.18181669 0.1038873 0.004888869 0         0       0          0       
  text6  0.42423894 1.0388733 0.026073969 0         0       0          0       
  text7  0.66666119 1.6621973 0.021728307 0         0       0.7099654  0       
  text8  0.24242225 0.4155493 0.007604907 0         0       0          0       
  text9  0.48484450 0.4155493 0.024987553 0         0       0          0       
  text10 0.12121113 0         0.001629623 0         0       0          0       
        features
docs      spencer    spoke         at
  text1  42.24217 1.066766 0.17484593
  text2   0       0        0.13113445
  text3   0       0        0.04371148
  text4   0       0        0.08742297
  text5   0       0        0.13113445
  text6   0       0        0.13113445
  text7   0       0        0.13113445
  text8   0       0        0         
  text9   0       0        0.34969186
  text10  0       0        0         

Retrieve the top 10 features (tokens) according to their TFIDF value for both channel_name values. (Hint: look at the groups argument in the topfeatures function).

topfeatures(x = media_tfidf, groups = 'channel_name')

$theyoungturks
      re    gonna      she     yeah     okay    money      tax      her       oh 
3503.312 3273.507 3004.244 2760.814 2441.523 2057.390 1984.010 1865.957 1860.247 
      he 
1837.587 

$foxnewschannel
   presid    tucker     north     laura     korea kavanaugh       fbi      greg 
 1844.192  1678.521  1611.765  1563.885  1486.060  1473.535  1425.031  1361.444 
 investig       she 
 1319.055  1315.552 

Rebuild the TFIDF without stopwords and look at the top features again:

topfeatures(x = media_tfidf_2, groups = 'channel_name')

$theyoungturks
      re    gonna     yeah     okay    money      tax       oh    trump   dollar 
3503.312 3273.507 2760.814 2441.523 2057.390 1984.010 1860.247 1769.254 1742.244 
     guy 
1700.337 

$foxnewschannel
   presid    tucker     north     laura     korea kavanaugh       fbi      greg 
 1844.192  1678.521  1611.765  1563.885  1486.060  1473.535  1425.031  1361.444 
 investig  democrat 
 1319.055  1311.602 

Task 3: Building ngram representations

Now take the above steps a bit further and produce a TF-IDF weighted bi-gram DFM.

Keep in mind that this involves several steps. Once you have created your bi-gram DFM (without the TFIDF weighting), remove those that do not occur in at least 5% of all documents. Stem the tokens.

Step 1: create the bigram DFM and apply the sparsity correction.

bigram_dfm = dfm(x = dfm_without_stopwords
                 , ngrams = 2
                 )

Argument ngrams not used.

What was the original overall sparsity, and how did it change?

bigram_dfm

Document-feature matrix of: 4,000 documents, 861,095 features (99.9% sparse).

bigram_dfm_small

Document-feature matrix of: 4,000 documents, 2,016 features (86.9% sparse).

Step 2: apply TFIDF weighting

What are the top 5 features (per group) before TFIDF weighting, and after TFIDF weighting?

topfeatures(bigram_tfidf, n = 5, groups = 'channel_name')

$theyoungturks
 they_re   you_re     he_s you_know  and_and 
3.817638 3.629969 3.598328 3.523020 3.005292 

$foxnewschannel
  the_president president_trump     north_korea             u_s        kim_jong 
       5.913264        5.472814        5.211625        4.605258        4.135510 

Task 4: Assessing the sentiment of the corpus

Now let’s look at the sentiment of the texts from these news outlets. We’ll start with the sentence-based approach from the sentimentr package. Since only the data from FoxNews was punctuated and hence contained sentences, we’ll focus on these ones only.

Create a new object called foxnews_only that contains only the transcripts of FoxNews:

media_data$channel_name

   [1] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
   [6] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
  [11] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
  [16] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
  [21] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
  [26] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
  [31] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
  [36] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
  [41] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
  [46] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
  [51] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
  [56] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
  [61] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
  [66] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
  [71] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
  [76] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
  [81] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
  [86] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
  [91] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
  [96] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [101] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [106] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [111] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [116] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [121] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [126] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [131] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [136] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [141] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [146] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [151] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [156] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [161] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [166] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [171] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [176] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [181] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [186] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [191] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [196] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [201] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [206] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [211] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [216] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [221] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [226] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [231] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [236] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [241] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [246] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [251] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [256] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [261] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [266] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [271] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [276] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [281] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [286] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [291] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [296] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [301] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [306] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [311] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [316] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [321] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [326] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [331] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [336] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [341] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [346] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [351] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [356] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [361] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [366] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [371] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [376] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [381] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [386] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [391] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [396] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [401] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [406] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [411] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [416] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [421] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [426] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [431] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [436] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [441] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [446] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [451] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [456] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [461] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [466] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [471] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [476] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [481] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [486] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [491] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [496] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [501] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [506] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [511] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [516] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [521] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [526] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [531] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [536] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [541] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [546] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [551] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [556] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [561] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [566] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [571] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [576] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [581] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [586] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [591] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [596] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [601] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [606] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [611] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [616] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [621] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [626] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [631] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [636] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [641] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [646] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [651] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [656] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [661] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [666] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [671] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [676] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [681] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [686] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [691] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [696] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [701] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [706] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [711] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [716] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [721] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [726] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [731] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [736] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [741] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [746] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [751] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [756] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [761] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [766] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [771] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [776] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [781] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [786] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [791] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [796] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [801] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [806] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [811] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [816] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [821] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [826] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [831] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [836] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [841] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [846] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [851] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [856] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [861] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [866] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [871] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [876] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [881] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [886] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [891] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [896] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [901] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [906] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [911] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [916] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [921] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [926] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [931] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [936] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [941] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [946] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [951] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [956] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [961] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [966] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [971] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [976] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [981] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [986] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [991] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [996] theyoungturks theyoungturks theyoungturks theyoungturks theyoungturks
 [ reached getOption("max.print") -- omitted 3000 entries ]
Levels: foxnewschannel theyoungturks

Now use the sentiment function to retrieve the sentiment of each sentence from this sub-corpus and store the results in a variable called foxnews_sentiments (this will take a while):

foxnews_sentiments

        element_id sentence_id word_count   sentiment
     1:          1           1          3  0.43301270
     2:          1           2          1  0.00000000
     3:          1           3          6 -0.20412415
     4:          1           4         18 -0.08249579
     5:          1           5         11 -0.61809826
    ---                                              
108491:       2000          34          9 -0.16666667
108492:       2000          35          6 -0.20412415
108493:       2000          36         13  0.00000000
108494:       2000          37         12  0.21650635
108495:       2000          38          5 -0.17888544

The object foxnews_sentiments now contains a sentiment value for each sentence in this sub-corpus.

Take a look at the distribution of these sentiments using a histogram:

What is the mean/median/min/max sentiment?

summary(foxnews_sentiments$sentiment)

    Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
-2.88871 -0.07217  0.00000  0.02910  0.16499  3.04667 

Task 5: Using the sentiment trajectory approach

Now let’s use the more advanced dynamic approach that can handle valence shifters as well as unpunctuated data.

Step 1: load the local source to access the ncs (= naive context sentiment) function by running the command below:

You now have access to the sentiment trajectory algorithm developed and introduced in this paper.

The main wrapper function is called ncs_full and asks you to specify the following arguments:

• txt_input_col: the column where your text is located
• txt_id_col: an identifier column
• low_pass_filter_size: the degree of smoothing in the Fourier transformation
• transform_values: whether or not to scale the values to -1.00 : +1.0
• normalize_values: whether or not to normalise the values (i.e. mean = 0, sd = 1)
• min_tokens: the minimum number of tokens that a text must contain to be processed
• cluster_lower: the lower size of the context window around the sentiment word
• cluster_upper: the upper size of the context window

Extract the sentiment trajectories of the first 100 FoxNews and the first 100 The Young Turks transcripts and leave all values in their default state (i.e. only specify the txt_input_col and txt_id_col argument). Call the resulting data sentiment_trajectories_foxnews and sentiment_trajectories_tyt. Note that ncs_full assumes that your input data is a data.frame (so use the media_data dataframe). Run the analysis on the cleaned text column. This operation will also take a few minutes.

#your code

fn = foxnews_only[1:100, ]
tyt =  media_data[media_data$channel_name == 'theyoungturks', ][1:100, ]
sentiment_trajectories_foxnews = ncs_full(txt_input_col = fn$text_clean
                                  , txt_id_col = fn$id)
sentiment_trajectories_tyt = ncs_full(txt_input_col = tyt$text_clean
                                  , txt_id_col = tyt$id)

Now take a look at some shapes of the sentiment trajectories. Compare 2 shapes from FoxNews with 2 shapes from The Young Turks by plotting them:

Task 6: Psycholinguistic variables

Finally, let’s explore the psycholinguistics dimension (and additional, deeper linguistic constructs as retrieved through the Linguistic Inquiry and Word Count Software a.k.a. LIWC.

Load the LIWC output for this corpus (LIWC extraction already done) as a csv file from ./data/media_data_liwc.csv and call the resulting object liwc_data.

Look at the first ten rows of this object:

#your code
head(liwc_data, 10)

    X               file   WC Analytic Clout Authentic  Tone  WPS Sixltr   Dic
1   1 theyoungturks_3505 2338    56.11 86.05     15.57 62.27 2338  16.00 86.66
2   2  theyoungturks_312  509    30.84 86.87     14.81  4.19  509  14.54 86.25
3   3 theyoungturks_4214  527    37.27 81.37     50.42  4.54  527  14.61 83.49
4   4 theyoungturks_1986  453    23.83 86.04      6.99 16.32  453  13.25 83.89
5   5 theyoungturks_4241  839    59.75 70.84     21.22 31.89  839  15.85 82.60
6   6 theyoungturks_4568 1372    47.56 76.47     25.93 16.40 1372  17.20 82.73
7   7 theyoungturks_1441 1979    25.75 83.91      2.31 43.24 1979  15.66 82.16
8   8 theyoungturks_2201  800    42.56 78.09     11.76 61.33  800  12.00 83.75
9   9 theyoungturks_1094 2052    69.39 78.37     13.13 39.73 2052  22.47 82.36
10 10 theyoungturks_3579  201    31.21 82.78      2.27 80.01  201  13.43 87.06
   function. pronoun ppron    i   we  you shehe they ipron article  prep auxverb adverb
1      54.79   16.77  8.73 1.97 0.73 2.78  2.18 1.07  8.04    7.14 12.28    7.10   5.77
2      58.55   18.07 10.22 3.73 0.98 0.98  3.14 1.38  7.86    6.68 10.61    4.91   7.07
3      55.60   15.56  9.11 0.76 2.28 0.57  1.33 4.17  6.45    5.69 10.25   10.44   6.07
4      56.73   20.75 11.92 2.43 2.65 0.44  4.64 1.77  8.83    6.18  9.05    5.74   7.06
5      53.16   13.83  6.20 0.83 0.60 1.19  0.36 3.22  7.63    6.56 12.87    9.54   5.36
6      53.64   15.60  8.02 2.19 1.31 2.11  0.29 2.11  7.58    6.92 10.86    7.22   7.43
7      57.71   21.22 12.73 2.53 0.81 2.73  5.66 1.01  8.49    5.71 10.01    8.34   6.22
8      55.12   20.00 10.38 3.00 0.62 2.25  1.50 3.00  9.62    6.50 10.62    8.25   4.62
9      50.05   14.96  7.41 2.14 0.83 1.61  1.66 1.17  7.55    6.48 13.01    5.75   5.02
10     57.71   21.39 11.94 1.99 1.99 2.99  4.98 0.00  9.45    6.47  9.95    8.96   6.47
    conj negate  verb  adj compare interrog number quant affect posemo negemo  anx
1   7.44   0.68 14.24 3.64    2.27     2.31   0.64  2.01   4.79   3.34   1.41 0.43
2  13.75   0.98 12.38 2.95    1.57     3.34   1.18  2.95   4.52   1.18   3.34 0.39
3   7.40   1.52 18.03 2.85    1.14     1.14   1.52  0.95   3.98   0.95   3.04 0.76
4  11.26   0.88 15.23 3.09    1.99     1.77   2.21  1.55   4.19   1.77   2.43 0.00
5   6.91   0.83 13.83 5.24    3.58     2.50   1.43  3.93   5.84   3.10   2.74 1.79
6   7.80   1.09 15.60 4.23    3.21     2.26   2.33  2.33   4.88   2.11   2.77 0.15
7   8.29   1.26 15.41 3.28    1.77     1.77   0.40  1.62   4.55   2.73   1.77 0.20
8   6.00   1.38 17.12 3.50    3.38     2.12   1.00  2.00   5.62   3.75   1.88 0.25
9   6.53   0.63 11.99 4.78    2.44     2.19   1.51  3.02   5.85   3.31   2.53 0.29
10  4.98   1.99 17.41 3.48    2.49     1.49   1.49  1.99   5.97   4.48   1.49 0.00
   anger  sad social family friend female male cogproc insight cause discrep tentat
1   0.56 0.13  12.75   0.34   0.30   0.09 2.44   10.78    2.78  1.45    1.54   2.01
2   1.57 0.20  15.72   0.59   0.20   3.73 1.38    9.43    2.16  2.55    1.38   1.38
3   2.09 0.19  12.52   0.00   0.57   0.00 1.71   11.39    0.95  1.52    1.52   2.85
4   0.88 0.66  14.13   0.00   0.00   3.75 0.88   13.91    4.42  2.21    1.10   3.75
5   0.24 0.24   8.94   0.00   0.12   0.00 0.48   11.44    2.03  3.69    0.95   1.67
6   2.11 0.07  10.20   0.00   0.15   0.15 0.36   12.17    2.62  1.09    1.60   2.84
7   0.61 0.05  14.10   0.00   0.15   0.20 5.81   14.15    2.27  2.02    3.89   3.59
8   0.62 0.25  12.12   0.12   0.12   0.00 1.50   13.38    2.50  2.75    2.88   3.00
9   1.02 0.15  11.35   0.10   0.10   0.58 3.22   11.94    2.78  2.58    1.95   2.73
10  1.49 0.00  11.44   0.00   0.00   0.50 4.98    9.45    1.99  1.00    0.50   0.50
   certain differ percept  see hear feel  bio body health sexual ingest drives
1     1.15   2.74    3.85 1.58 1.92 0.34 0.60 0.38   0.00   0.00   0.04   6.29
2     1.38   1.77    1.77 0.39 1.18 0.20 1.18 0.00   0.39   0.79   0.00   6.29
3     2.28   4.17    3.80 0.95 2.09 0.76 0.38 0.19   0.19   0.00   0.00   7.40
4     0.66   3.09    1.32 0.22 0.88 0.00 0.44 0.00   0.44   0.00   0.00   8.39
5     1.07   3.58    2.15 1.07 0.48 0.48 0.24 0.24   0.00   0.00   0.00   7.39
6     1.75   2.92    2.62 1.46 1.09 0.07 1.09 0.58   0.36   0.00   0.44   6.34
7     1.72   3.84    2.48 0.56 1.52 0.30 0.25 0.15   0.05   0.05   0.00   7.48
8     1.62   2.88    1.50 0.88 0.50 0.12 0.62 0.00   0.00   0.00   0.25   6.50
9     1.17   3.12    1.75 0.73 0.58 0.39 0.88 0.34   0.34   0.15   0.05   7.46
10    2.99   2.49    2.49 1.49 0.00 0.50 1.00 0.50   0.50   0.00   0.00   8.46
   affiliation achieve power reward risk focuspast focuspresent focusfuture relativ
1         1.80    0.86  2.27   0.86 0.94      3.85        10.01        0.68   11.16
2         1.96    0.98  2.16   1.38 0.39      5.11         6.29        0.59   11.59
3         2.66    1.14  3.42   0.57 0.95      2.85        11.39        3.61   18.03
4         2.87    1.32  1.77   1.99 0.66      4.86         7.95        0.88    7.95
5         1.19    1.67  3.69   1.67 0.24      1.91        11.56        1.07   11.08
6         2.41    0.73  2.62   0.66 0.29      4.74         8.82        0.66   11.30
7         1.26    0.96  3.89   1.11 0.66      2.12        10.06        1.26    9.90
8         1.50    1.38  2.12   1.50 0.38      4.88         9.50        1.12    9.88
9         1.75    1.36  2.49   1.61 0.88      2.24         9.21        0.49    9.84
10        2.99    0.00  4.48   1.00 0.00      3.48        13.93        1.00    7.96
   motion space time work leisure home money relig death informal swear netspeak assent
1    1.41  6.63 3.29 2.91    0.56 0.43  1.07  0.13  0.13     1.11  0.04     0.17   0.68
2    1.96  4.72 5.89 0.59    0.20 0.20  0.20  0.20  0.00     0.79  0.00     0.39   0.20
3    2.66  9.68 6.26 0.95    0.19 0.00  0.00  0.00  0.38     0.38  0.00     0.00   0.38
4    0.66  4.42 3.09 1.99    1.32 0.00  0.66  0.22  0.22     1.55  0.00     0.22   0.66
5    2.15  4.53 4.41 3.58    0.24 0.00  2.86  0.00  0.00     0.48  0.00     0.00   0.24
6    1.24  6.12 4.23 2.11    0.36 0.15  1.02  0.15  0.07     1.24  0.22     0.15   0.51
7    1.11  5.76 2.98 3.44    0.40 0.10  0.40  0.10  0.10     1.97  0.10     0.30   1.06
8    1.75  5.50 2.38 3.25    0.38 0.12  0.12  0.00  0.00     1.62  0.00     0.12   0.88
9    1.12  6.38 2.29 3.70    0.88 0.10  1.36  0.05  0.10     1.32  0.05     0.19   0.68
10   1.99  1.49 3.48 3.48    1.00 0.00  1.49  0.00  0.00     2.99  0.50     0.00   1.00
   nonflu filler AllPunc Period Comma Colon SemiC QMark Exclam Dash Quote Apostro
1    0.21   0.00       0      0     0     0     0     0      0    0     0       0
2    0.20   0.00       0      0     0     0     0     0      0    0     0       0
3    0.00   0.00       0      0     0     0     0     0      0    0     0       0
4    0.66   0.00       0      0     0     0     0     0      0    0     0       0
5    0.24   0.00       0      0     0     0     0     0      0    0     0       0
6    0.36   0.00       0      0     0     0     0     0      0    0     0       0
7    0.40   0.10       0      0     0     0     0     0      0    0     0       0
8    0.38   0.25       0      0     0     0     0     0      0    0     0       0
9    0.34   0.05       0      0     0     0     0     0      0    0     0       0
10   1.49   0.00       0      0     0     0     0     0      0    0     0       0
   Parenth OtherP
1        0      0
2        0      0
3        0      0
4        0      0
5        0      0
6        0      0
7        0      0
8        0      0
9        0      0
10       0      0

Now calculate the average of the following variables per group:

• swear words
• focus on the future
• focus on the past
• personal pronouns (ppron)
• amount of analytical language
• references to ‘power’

Note: you will have to create the ‘group’ variable yourself from the file variable.

tapply(liwc_data$power, liwc_data$group, mean)

      FN      TYT 
3.909940 2.942015 

Explore the data further to understand the LIWC.

---

END