Text data 1

Core idea

Text is a unique documentation of human activity.

We are obsessed with documenting.

Text & Crime Science

Text & Crime Science

• hate speech
• police reports
• crimestoppers
• fake reviews
• fear of crime
• cryptofraud

Obtaining text data

Quantifying text data

Example

All I ever wanted was to love women, and in turn to be loved by them back. Their behavior towards me has only earned my hatred, and rightfully so! I am the true victim in all of this. I am the good guy. Humanity struck at me first by condemning me to experience so much suffering. I didn’t ask for this. I didn’t want this. I didn’t start this war. I wasn’t the one who struck first. But I will finish it by striking back. I will punish everyone. And it will be beautiful. Finally, at long last, I can show the world my true worth.

How would you quantify the example?

R examples

#sentences
no_of_sentences = nsentence(er)
no_of_sentences

## text1 
##    13

#words 1
no_of_words_1 = ntoken(er)
no_of_words_1

## text1 
##   123

#words 2
no_of_words_2 = ntype(er)
no_of_words_2

## text1 
##    72

Type-token ratio

ntype(string_a)/ntoken(string_a)

##     text1 
## 0.6363636

ntype(string_b)/ntoken(string_b)

## text1 
##     1

Nuanced meta features

• Characters per word

nchar(er)/ntoken(er)

##    text1 
## 4.317073

• Words per sentence

ntoken(er)/nsentence(er)

##    text1 
## 9.461538

Text representations

DFM with two texts

The Industrial Revolution and its consequences have been a disaster for the human race. They have greatly increased the life-expectancy of those of us who live in “advanced” countries, but they have destabilized society, have made life unfulfilling, have subjected human beings to indignities, have led to widespread psychological suffering (in the Third World to physical suffering as well) and have inflicted severe damage on the natural world. The continued development of technology will worsen the situation.

DFM representation

corpus_tokenised = tokens(mini_corpus)
corpus_dfm = dfm(corpus_tokenised)

knitr::kable(corpus_dfm[, 1:8])

…

knitr::kable(corpus_dfm[, 31:38])

Is this ideal?

What are the most frequent “terms”?

topfeatures(corpus_dfm[1])

##      .      i      ,    the   this     to    and     by     me didn't 
##     12     10      4      4      4      3      3      3      3      3

topfeatures(corpus_dfm[2])

##       the      have         ,        to         .        of       and 
##         7         7         4         3         3         3         2 
##        in suffering     world 
##         2         2         2

Highly recommended: Vsauce on Zipf’s Law

But how to get the important words?

Word importance

Ideally, we want to “reward” words that are:

• important locally
• but not ‘inflated’ globally

Metric for word importance

• Term frequency: occurence/overall words in document

3/ntoken(mini_corpus[1])

##      text1 
## 0.02439024

Term frequency: reward for words that occur often in a document.

Metric for word importance

Problem: some words just occur a lot anyway (e.g. “stopwords”).

Correct for global occurrence:

Document frequency: number of documents with each token.

Combining term frequency and document frequency

• take the local importance

• correct for global occurrences

  x
  and	2
  in	2
  turn	1

TF/DF

#text1: "and"
0.024/2

#text2: "and"
0.022/2

#text1: "turn"
0.008/1

#text2: "turn"
0.000/1

TF-IDF

• Term frequency
• INVERSE document frequency

\(TFIDF = TF/DF\) = \(TFIDF = TF*IDF\), since

\(IDF = 1/DF\)

IDF often modelled as \(IDF = log(\frac{N}{DF})\)

TF-IDF

Note: for the exact formula for the inverse DF refer to the quanteda docs.

knitr::kable(round(dfm_tfidf(corpus_dfm, scheme_tf = 'prop', scheme_df = 'inverse'), 3)[, 10:15])

TF-IDF

• TF: rewards local importance
• IDF: punishes for global occurrence
• TFIDF value as metric for the importance of words per document

There’s more to words

• you can count them [DONE]
• but they also have a function
  • each word has a grammatical function
  • nouns, verbs, pronouns
• called: parts-of-speech

Syntactic dimension

library(qdap)

Part-of-speech tagging

POS depend on POS framework.
Commonly used: Penn Treebank Project

POS types

POS tagging with qdap

er_ = "All I ever wanted was to love women"
pos_tagged = pos(er_)

## 
  |                                                                       
  |                                                                 |   0%
  |                                                                       
  |=================================================================| 100%

pos_tagged$POStagged$POStagged

## [1] all/DT i/FW ever/RB wanted/VBD was/VBD to/TO love/VB women/NNS
## Levels: all/DT i/FW ever/RB wanted/VBD was/VBD to/TO love/VB women/NNS

POS tagging

pos(er, percent = F, progress.bar = F)$POSfreq

##   wrd.cnt CC DT FW IN JJ MD NN NNS PRP PRP$ RB TO VB VBD VBG VBN VBP VBZ
## 1     106  4 10  1 14  8  4 17   1   7    3 10  3  9   6   2   2   3   1
##   WP
## 1  1

pos(ub, percent = F, progress.bar = F)$POSfreq

##   wrd.cnt CC DT IN JJ MD NN NNS PRP PRP$ RB TO VB VBN VBP WP
## 1      77  3  9  8 11  1 15   4   3    1  2  3  2   7   7  1

Considerations in text cleaning

Researcher’s degrees of freedom

• stopword removal
• stemming

Stopword removal

• We know many words are “low in meaning”
• So-called stopwords

You could decide to remove these…

Stopword removal

With stopwords:

Without stopwords

Stemming

• some words originate from the same “stem”
• e.g. “love”, “loved”, “loving”, “lovely”
• but you might want to reduce all these to the stem

Word stems

love_stem = c("love", "loved", "loving", "lovely")

… after stemming

knitr::kable(dfm(love_stem_tok, stem = T))

Our mini corpus

From: (incl. stopwords and without stemming)

… to (without stopwords and stemmed)

RECAP

• levels of text data
• meta features
• syntactic features
• word frequencies
• TFIDF
• parts-pf-speech

Outlook

No tutorial.

Homework: Text data 1 (to come)

Next week: Text data 2

END