Text as Data

• Measurement
  Define what a dictionary score is intended to measure.
• Dictionary construction
  Build and revise a word or phrase list using political texts.
• Scoring
  Turn dictionary matches into scores; explain the denominator and texts with no matches.
• Validation
  Use published evidence and human judgments to assess validity.
• Own project:
  Plan a dictionary measure for your research.

When do MPs describe household economic conditions negatively?

• “Families cannot afford rent or energy bills.”
• Possible comparison: speakers, periods, or government status.
• Read both the household issue and the evaluation: rent identifies the topic; cannot afford describes hardship.

Mentioning inflation does not tell us how an MP evaluates household conditions.

Negative language about a problem can accompany support for a policy.

1. Find relevant sentences: household income, living costs, and affordability.
2. Code the evaluation and its target: how are household conditions described?
3. Validate against human judgments: include dictionary matches and nonmatches.
4. Compare speakers or periods: relevant sentences judged negative / all relevant sentences.

Today we practise two parts separately: counting issue mentions and scoring positive/negative words. We still need to connect each evaluation to its target to answer the opening question.

Example: “Families cannot afford rent or energy bills.”

This counts mentions. To measure negative evaluations, we also need to identify language such as “cannot afford.”

Our lab sample is 5,000 UK House of Commons contributions, 1945–2025.

Keep speaker, date and source IDs when splitting or combining texts.

Each document is one parliamentary speech contribution. Run C01 once in R to load the data and examples; keep this R session open.

# C01: Setup
# First use: uncomment the next line to install.
# install.packages(c("quanteda", "quanteda.textstats"))
library(quanteda)
library(quanteda.textstats)
paths <- c("workflow.R", "W4/classroom/workflow.R",
           "classroom/workflow.R", "../classroom/workflow.R")
found <- paths[file.exists(paths)]
if (length(found) == 0) found <- file.choose()
setwd(dirname(normalizePath(found[1])))
source("workflow.R")

Predict: How many times will prices appear in row A?

# C02: Counts and document length
library(quanteda)
demo <- c(A = "Prices rise. Prices worry voters.",
          B = "Prices fall.")
demo_tokens <- tokens(demo, remove_punct = TRUE)
demo_dfm <- dfm(demo_tokens)
selected <- dfm_match(demo_dfm, c("prices", "rise", "fall"))
as.matrix(selected)
ntoken(demo_tokens)

Compare: Why do the selected row totals differ from document length?

A dictionary maps expressions to predefined categories.

library(quanteda)
dic <- dictionary(list(
  household_costs = c("inflation", "rent", "energy bills"),
  employment = c("unemployment", "job security")
))

dic stores the dictionary for the next example.

Mentioning household costs does not establish economic hardship.

# Run the preceding slide first to create dic.
texts <- c("Rent and energy bills are rising.",
           "We need job security.")
example_tokens <- tokens(texts, remove_punct = FALSE)
matched <- tokens_lookup(example_tokens, dic, capkeys = FALSE)
counts <- dfm(matched)
as.matrix(counts)

rent + energy bills give 2 matches; job security gives 1.

Availability is not evidence of suitability.

Next: Whether adapting or building, decide which expressions belong in the dictionary.

1. Write down what should count, with one example that should not.
2. Read passages that do and do not discuss the issue.
3. List words and phrases that signal the issue.
4. Check false matches and relevant passages the list misses.

For household costs: distinguish energy bills from a parliamentary bill.

Next: use frequency, KWIC, collocations and keyness to find candidate terms and check their meaning.

Predict: Is a frequent word also used in many different speeches?

# C03: Frequency in real speeches (run C01 first)
recap_dfm <- dfm(discovery_tokens)
recap_dfm <- dfm_remove(recap_dfm, stopwords("en"))
# Keep words made of English letters only.
recap_dfm <- dfm_select(recap_dfm, "^[a-z]+$", valuetype = "regex")
freq <- textstat_frequency(recap_dfm)
head(freq, 6)

frequency counts occurrences; docfreq counts speeches with a match.

Try: Change 6 to 12. Which words describe procedure rather than policy?

Code: head(freq, 6)

     feature frequency rank docfreq group
1        hon      6514    1    2826   all
2 government      3190    2    1320   all
3      right      2762    3    1470   all
4     member      2395    4    1129   all
5        can      2391    5    1221   all
6     people      2273    6     995   all

• frequency: total occurrences. hon appears 6,514 times.
• docfreq: speeches containing the word. hon appears in 2,826 speeches.
• rank: order by frequency. group = all: speeches are pooled together.

KWIC = keyword in context. Will security always refer to defence?

# C04: Context in real speeches (run C01 first)
live_contexts <- kwic(discovery_tokens, "security", window = 5)
context_table <- as.data.frame(live_contexts)
columns <- c("docname", "pre", "keyword", "post")
head(context_table[, columns], 2)

In pairs: Classify two matches. Then change the window to 12.

Collocations: adjacent words with a strong statistical association.

# C05: Phrases in real speeches (run C01 first)
coll <- textstat_collocations(discovery_tokens, size = 2, min_count = 20)
row_order <- order(coll$lambda, decreasing = TRUE)
coll <- coll[row_order, ]
head(coll, 6)
first_phrase <- phrase(coll$collocation[1])
phrase_context <- kwic(discovery_tokens, first_phrase, window = 5)
head(phrase_context, 2)

Try: Raise min_count to 50. Are the leading phrases about household costs?

Keyness: which words are distinctive of Conservative versus Labour speeches?

# C06: Relative word use (run C01 and C03 first)
party_dfm <- dfm_subset(recap_dfm,
  party %in% c("Conservative", "Labour"))
party_dfm <- dfm_group(party_dfm, groups = party)
party_dfm <- dfm_trim(party_dfm, min_termfreq = 1)
keys_con <- textstat_keyness(party_dfm, target = "Conservative")
keys_lab <- textstat_keyness(party_dfm, target = "Labour")
head(keys_con, 3)
head(keys_lab, 3)

Compare: Each run uses a different target. Which words stand out for each party?

Use discovery texts to choose terms; keep separate texts to test the finished rules.

Discovery texts → candidate terms → human review → frozen dictionary

Held-out texts → independent human labels → error analysis

• Record the dictionary version and inclusion rules.
• Do not keep tuning on the final evaluation sample.
• Recheck performance when the language or period changes.

A phrase removes one false positive but still misses a relevant passage.

Predict: Which sentence will the phrase dictionary match?

# C07: A complete phrase-lookup example
library(quanteda)
examples <- c("Energy bills are rising.",
              "This bill reforms parliament.",
              "Families cannot afford heating.")
example_tokens <- tokens(examples, remove_punct = FALSE)
cost_dict <- dictionary(list(costs = "energy bills"))
matched <- tokens_lookup(example_tokens, cost_dict)
cost_counts <- dfm(matched)
as.matrix(cost_counts)

Change: Add "cannot afford heating". Which error changes?

Predict: What will bill* retrieve besides bills?

# C08: Wildcards in real speeches (run C01 first)
wild <- kwic(discovery_tokens, "bill*", window = 0)
exact <- kwic(discovery_tokens, c("bill", "bills"),
              valuetype = "fixed", window = 0)
c(wildcard = nrow(wild), exact = nrow(exact))
word_counts <- table(wild$keyword)
word_counts <- sort(word_counts, decreasing = TRUE)
head(word_counts, 8)

Discuss: Does an exact match solve the substantive ambiguity?

wildcard    exact
    1774     1539

These are match counts, not numbers of speeches. Use KWIC to check meaning.

Predict: Will counts and rates identify the same top speeches?

# C09: Rankings on real speeches (run C01 first)
discovery_ids <- docnames(discovery_tokens)
rates <- subset(comparison, doc_id %in% discovery_ids)
rates <- subset(rates, tokens >= 50)
cols <- c("doc_id", "broad", "tokens", "broad_per100")
raw_order <- order(rates$broad, decreasing = TRUE)
rate_order <- order(rates$broad_per100, decreasing = TRUE)
head(rates[raw_order, cols], 3)
head(rates[rate_order, cols], 3)

Compare: What does the 50-token restriction change?

head(..., 3) shows the top three rows; cols selects columns.

Rate = 100 × broad / tokens. broad counts matches to bill*.

20 / 1,349 → 1.48 per 100; 4 / 69 → 5.80 per 100.

More matches does not necessarily mean a higher density.

Let P = positive matches, N = negative matches, L = original token count.

Scoring means turning matches into a number. With P = 3, N = 1 and L = 100: net tone = 2, coverage = 4%. Later, the Proksch et al. case uses a different scoring rule.

# C10: Zero, balance and missing matches
check_texts
check_scores[, c("doc_id", "positive", "negative",
                 "net_per100", "coverage_per100", "balance")]

Compare: Why do unmatched, balanced, and empty differ?

Zero can mean no matches or cancellation; neither proves neutrality.

Predict: Will keeping not make the second sentence negative?

# C11: A deliberate sentiment failure (run C01 first)
negation_tokens <- check_tokens[c("good", "negated")]
as.list(negation_tokens)
negation_dfm <- dfm(negation_tokens)
negation_counts <- dfm_lookup(negation_dfm, toy_sentiment)
as.matrix(negation_counts)
without_stopwords <- tokens_remove(negation_tokens, stopwords("en"))
as.list(without_stopwords)

Explain: Both sentences contain good. Does our word-count rule use not to change its category?

         positive negative
good            1        0
negated         1        0

$good
[1] "policy" "good"

$negated
[1] "policy" "good"

Keep negation words, then add a rule that handles them.

• “Not good” differs from “not only good but excellent.”
• “I reject the claim that this is a failure” contains negative words.
• Quoted criticism may not express the speaker’s own view.
• Long-distance negation can exceed a simple word window.

Rules can improve a baseline without resolving every sentence.

For our dictionaries: do substantive comparisons survive reasonable scoring choices?

Predict: Which version will flag more speeches?

# C12: Dictionary revisions (run C01 first)
policy_entries[, c("version", "category", "pattern")]
dev <- subset(policy_scores,
  doc_id %in% docnames(discovery_tokens))
c(v1 = sum(dev$v1 > 0), v2 = sum(dev$v2 > 0))
energy_context <- kwic(discovery_tokens, "energy", window = 5)
head(energy_context, 3)

Discuss: Scores changed. Do they agree better with human judgments?

What is dev? The table created in C12 for inspecting dictionary rules: one row = one speech, with 4,950 speeches.

v1_per1000 and v2_per1000 are its columns: matches per 1,000 words under each dictionary version.

# C15: Visualizing dictionary sensitivity (run C12 first)
# install.packages("ggplot2")  # run once
library(ggplot2)
ggplot(dev, aes(x = v1_per1000, y = v2_per1000)) +
  geom_point(size = 1, alpha = 0.4, colour = "#314f4f") +
  geom_abline(intercept = 0, slope = 1,
              colour = "firebrick", linetype = "dashed") +
  labs(x = "v1 matches per 1,000 tokens",
       y = "v2 matches per 1,000 tokens") +
  theme_minimal()

Discuss: What can this plot show without human labels?

• Write a codebook with positive, negative, and ambiguous examples.
• Keep coders unaware of dictionary predictions when practical.
• Record disagreement and the adjudication procedure.
• Match the coding unit to the intended use of the score.

Human labels are evidence to scrutinize, not unquestionable truth.

• Include matched and unmatched passages.
• Cover important dates, groups, and document lengths.
• Use an untouched sample for final performance estimates.
• Inspect rare or ambiguous cases separately for diagnosis.

A collection of easy examples cannot establish corpus-wide performance.

Extension: ten illustrative labels, 1 = energy-policy relevant, 0 = not relevant.

# C13: Human agreement, not model accuracy
# install.packages("irr")  # run once
library(irr)
ratings <- data.frame(
  coder_A = c(1, 1, 1, 1, 1, 0, 0, 0, 0, 0),
  coder_B = c(1, 1, 1, 1, 0, 1, 0, 0, 0, 0))
table(ratings$coder_A, ratings$coder_B)
mean(ratings$coder_A == ratings$coder_B)
agreement <- kappa2(ratings, weight = "unweighted")
agreement$value

Discuss: What does kappa (.60) tell us beyond raw agreement (80%)?

The unit is a passage, not a dictionary word.

Before running the code: calculate precision and recall from the preceding table.

# C14: Checking validation arithmetic (run C01 first)
predicted <- validation$predicted
human <- validation$human
TP <- sum(predicted == 1 & human == 1)
FP <- sum(predicted == 1 & human == 0)
FN <- sum(predicted == 0 & human == 1)
precision <- TP / (TP + FP)
recall <- TP / (TP + FN)
F1 <- 2 * TP / (2 * TP + FP + FN)
c(precision = precision, recall = recall, F1 = F1)

Discuss: Could we estimate recall by reading only dictionary matches?

Code: c(precision = precision, recall = recall, F1 = F1)

precision    recall        F1
0.7500000 0.6000000 0.6666667

• Precision = 0.75: Of 8 passages the dictionary flags, 6 are relevant (6 / 8).
• Recall = 0.60: Of 10 relevant passages, the dictionary finds 6 (6 / 10).
• F1 ≈ 0.667: Combines precision and recall: 2 × 6 / (2 × 6 + 2 + 4).

Missed cases matter: reading only matched passages cannot tell us recall.

• Plot scores against human ratings.
• Inspect group means and individual disagreements.
• Check unmatched texts, compressed ranges, and extreme values.
• Compare performance across groups and genres.

Two scores can rise together but still disagree: scores of 1, 2, 3 and 11, 12, 13 have perfect correlation.

One successful comparison does not validate every use.

• Unit: one abstract. Does it contain a policy-relevant statement?
• Develop: start with policy terms; read 400 double-coded abstracts to identify additional terms.
• Refine: keep additions that improve positive-class F1 on the development set.
• Test: evaluate on 200 other human-coded abstracts, kept out of tuning.

Reported average F1: 0.90 development / 0.86 held out.

Instructor’s research-process note: We first tried fine-tuning transformers, but the results were not satisfactory.

About 15% of abstracts were classified as policy relevant.

Only 8% explicitly mentioned policy or policies.

These are descriptive text patterns, not evidence of policy uptake.

Compare the original dictionaries and the augmented version with human judgments.

Do the original and revised dictionaries recover human-rated evaluations?

• Result: positive evaluations separate more clearly than negative ones.
• Use: a tested indicator for comparing evaluative language in political texts.
• Limit: scores cluster near zero and do not have self-evident numerical units.

The study validates a measure. It does not explain why politicians become more positive.

Question: Is more positive opposition language associated with unanimous passage?

1. Collect about 26,000 speeches on 871 German government bills, 2005-2017.
2. Match positive and negative words with translated Lexicoder dictionaries.
3. Aggregate counts by party and bill: log((P + 0.5) / (N + 0.5)).
4. Use bill-level sentiment in logistic regressions of unanimous passage.

Human-coded European Parliament texts provide a separate measurement check.

Compare human and automated tone for the same speaker groups.

• A translated term can have a different political use.
• Translation may produce several possible equivalents.
• Morphology and tokenization affect matches.
• Validate languages separately and examine difficult cases.

Shared category names do not guarantee comparable numerical scales.

Negative words can describe a social problem without attacking another party.

Application: use the same construction, scoring and validation principles.

LIWC = Linguistic Inquiry and Word Count.

• Matches language to predefined dictionary categories.
• Categories include pronouns, emotion and social processes.
• Most category scores are percentages of a text’s words.

Example: 4 category matches in 100 words = 4%.

A category percentage measures word use; interpretation needs validation.

Importing a file does not grant a license to use its dictionary.

Import our ten-word teaching dictionary, not an official LIWC dictionary.

# C16: Custom dictionary (run C01; stay in W4/classroom)
library(quanteda)
own <- dictionary(file = "Dictionaries/teaching_affect.dic", format = "LIWC")
mini <- tokens(c("Good policy.", "A bad crisis."), remove_punct = TRUE)
matched <- tokens_lookup(mini, own, capkeys = FALSE)
hits <- dfm(matched)
counts <- as.matrix(hits)
counts
word_count <- ntoken(mini)
100 * counts / word_count

Discuss: Why divide by all word tokens, not just dictionary matches?

Each column is a party program; each row is a dictionary term. Darker cells indicate larger logged counts; white indicates zero.

Copy the whole block into R. Requires quanteda.

library(quanteda)
text <- c("We may improve public services.",
          "We guarantee better public services.")
dict <- dictionary(list(vague = "may", concrete = "guarantee"))
toks <- tokens(text, remove_punct = TRUE)
counts <- dfm(toks)
hits <- dfm_lookup(counts, dictionary = dict)
as.matrix(hits)

       features
docs    vague concrete
  text1     1        0
  text2     0        1

• text1: may matches the vague category once.
• text2: guarantee matches the concrete category once.
• The values count words matched by our rules.

Try: Add not before guarantee. Will the count change?

Teaching score = 100 × (C − V) / L

C = concrete matches; V = vague matches; L = all word tokens.

This is concrete percentage − vague percentage, in percentage points.

m <- as.matrix(hits)
100 * (m[, "concrete"] - m[, "vague"]) / ntoken(toks)
# text1 text2
#   -20    20

text1: 100 × (0 − 1) / 5 = −20; text2: 100 × (1 − 0) / 5 = 20.

Adapted from Eichorst & Lin (2019): the paper excludes stopwords from the denominator and rescales the difference to 0–1. Our example does neither.

Optional extension: return to finding candidates, now with POS → candidate terms → human review.

# O01: POS candidates (model download on first run)
# install.packages("udpipe")  # run once before class
library(udpipe)
dir.create("udpipe_models", showWarnings = FALSE)
info <- udpipe_download_model("english-ewt",
  model_dir = "udpipe_models", overwrite = FALSE)
model <- udpipe_load_model(info$file_model)
text <- "Housing costs are unaffordable."
annotation <- udpipe_annotate(model, x = text, parser = "none")
ann <- as.data.frame(annotation)
ann[, c("token", "lemma", "upos")]

Discuss: Which tokens would you keep as dictionary candidates?

ann[, c("token", "lemma", "upos")] keeps all rows and selects three columns.

token: observed form · lemma: predicted base form · upos: predicted word class

Discuss: Would you accept Housing → ADJ? Which words are useful dictionary candidates?

Co-occurrence: words appear near each other. Dependency: a parser predicts a grammatical link.

Our O01 used parser = "none"; dependency extraction needs parsing enabled.

Obama’s 2013 inaugural address, included in quanteda. Run O01 first to load model.

library(quanteda)
# install.packages(c("textrank", "igraph"))  # run once
library(textrank)
speech <- corpus_subset(data_corpus_inaugural, Year == 2013)
text <- as.character(speech)
ann <- as.data.frame(udpipe_annotate(model, x = text,
                                  parser = "none"))
keep <- ann$upos %in% c("NOUN", "ADJ")
result <- textrank_keywords(ann$lemma, relevant = keep,
                           ngram_max = 2, sep = " ")
scores <- sort(result$pagerank$vector, decreasing = TRUE)
round(head(scores, 6), 4)

Actual output: Obama 2013, English EWT model

   new   same endure nation common   hard
0.0161 0.0096 0.0095 0.0094 0.0092 0.0083

• Nodes: candidate lemmas. Edges: adjacent noun/adjective pairs.
• PageRank uses pair counts and neighbours’ scores.
• Review: new is broad; some endure forms were tagged as nouns.

pairs <- cooccurrence(ann$lemma,
  relevant = keep)
print(head(pairs, 3))

       term1   term2 cooc
1 individual freedom    2
2        new     job    2
3      other  nation    2

cooc = 2: the pair occurs twice in this speech.

Optional Python demo. POS labels word classes; NER identifies names and their types.

# Install once in a terminal: python -m pip install flair
from flair.data import Sentence
from flair.models import SequenceTagger

tagger = SequenceTagger.load("ner-fast")
sentence = Sentence(
    "Barack Obama spoke at the United Nations in New York.")
tagger.predict(sentence)
for entity in sentence.get_spans("ner"):
    print(entity.text, entity.get_label("ner").value)

For R users: I developed flaiR, an R package that lets you access Flair NLP through R functions.

“Barack Obama spoke at the United Nations in New York.”

Barack Obama PER
United Nations ORG
New York LOC

• NER groups words into a name: United Nations is one entity span.
• Use these spans to find mentions, then inspect their context.
• An organization mention does not tell us whether the speaker supports it.

Seed concept → suggested terms → corpus examples → researcher review

• Embeddings can suggest related expressions.
• LLMs can propose variants or organize candidate lists.
• Antonyms and irrelevant associations can also appear.
• Keep an independent human benchmark.

These tools assist dictionary development; they do not certify validity.

Validate in the language and political setting of the project.

First, label the sentiment expressed in each sentence.

Which sentences need more than word matching?

A hypothetical timing comparison, not a benchmark.

Would faster labels answer our research question? What validation is still needed?

• Corpus boundaries, source, and unit of analysis.
• Dictionary origin, version, language, and modifications.
• Matching, preprocessing, denominator, and zero rules.
• Human validation design, results, and remaining errors.
• Code and access information sufficient to audit the workflow.

Return to our opening question, or choose your own research question:

• I want to measure …, rather than merely count ….
• My unit of analysis is ….
• A plausible false positive is ….
• A plausible false negative is ….
• I would validate the measure using ….

For our opening question: What extra rules would identify negative evaluations of household conditions?

Would a dictionary be a good choice for your project?

• Identify two strengths and two limitations.
• How would you balance speed, transparency, and sensitivity to context?
• What validation evidence would support your choice?

Discuss in pairs for 2 minutes: choose a corpus and explain your decision.

Optional challenge: Add one justified energy-policy expression.

# O02: Your own dictionary (run C01 first)
my_dict <- dictionary(list(energy = c("energy", "gas bills")))
matched <- tokens_lookup(discovery_tokens, my_dict)
my_dfm <- dfm(matched)
my_counts <- as.numeric(my_dfm[, "energy"])
sum(my_counts > 0)
my_context <- kwic(discovery_tokens, "energy", window = 6)
head(my_context, 3)

In pairs: Show one supporting passage and one possible false positive.

• Develop: use 12 teaching sentences to see what v1 and v2 get wrong.
• Freeze: keep the supplied v2 unchanged for the next 12 sentences.
• Test: compare its predictions with the supplied labels, including nonmatches.
• Result: precision 0.80, recall 0.667, F1 0.727.

Illustrative arithmetic, not estimated performance on Parliament.

W5 (October 9): no class. W6 (October 16): human coding and classification.

About 10 minutes of preparation; no model training or upload.

45-60 minutes of core practice.

1. Read the teaching dictionary and calculate sentiment scores.
2. Change an energy-policy rule and inspect its matches.
3. Find false matches and missed cases in development examples.
4. Keep v2 fixed and calculate precision, recall and F1 on test examples.

Lab handout · Lab ZIP