# install.packages("quanteda") # run once before class
library(quanteda)
# Find the lab folder; if asked, select workflow.R inside W4/lab.
paths <- c("W4/lab/workflow.R", "lab/workflow.R",
"../lab/workflow.R", "../workflow.R", "workflow.R")
found <- paths[file.exists(paths)]
if (length(found) == 0) found <- file.choose()
setwd(dirname(normalizePath(found[1])))
source("workflow.R") # load the data and prepared examplesW4 Take-home Lab: LIWC-Format Dictionaries and Validation
Text as Data | National Taiwan University
Before starting
Ungraded take-home practice; no submission. In this lab, we’ll read a dictionary, change one of its rules, and check which sentences it gets right or wrong. Allow 45–60 minutes for sections 1–4, with extra time if you are new to R. The party comparison is optional; the W6 preview takes about 10 additional minutes.
We’ll use the 5,000 parliamentary speech contributions from W2 to look at words in context. We’ll also use short example sentences to check our dictionary matches by hand. You do not need to run W2 first: the setup below loads the data.
Extract the lab ZIP, then open Lab_Session_W4.R in RStudio and read this handout alongside it. Keep workflow.R, Dictionaries/ and Data/ together. Run the whole setup block below once; it finds the lab folder or asks you to choose its workflow.R. Keep the same R session open. Install quanteda once; no LIWC purchase, model download or API is needed. You do not need to render the QMD.
Try the exercises before opening the worked answers. Bring questions to instructor or TA office hours.
Reading the R code
We’ll use quanteda functions and simple R arithmetic. Work through each section from top to bottom in the same R session, checking the output as you go. In RStudio, use Ctrl+Enter (Windows) or Cmd+Enter (Mac) to run a selected line.
| Syntax | Meaning | Example |
|---|---|---|
<- |
Save a result under a name | counts <- dfm(toks) |
$ |
Select one column | test$human |
[rows, columns] |
Select rows or columns | counts[, "positive"] |
==, != |
Equal to, different from | test$v2 != test$human |
sum(x) |
Add values; TRUE counts as 1 | sum(predicted == 1) |
subset(x, condition) |
Keep matching rows | subset(synthetic, split == "development") |
%in% |
Check membership in a list | doc_id %in% discovery_ids |
& |
Both conditions must hold | predicted == 1 & human == 1 |
head(x, 6) |
Display six rows | head(contexts, 6) |
workflow.R loads our texts, dictionaries and prepared examples. Run it through the setup block above; you can leave its functions as they are. We’ll work with the resulting objects in the steps below.
Functions you will reuse
| Task | Ready-made function |
|---|---|
| Build a corpus from a data table | corpus() |
| Split text into tokens | tokens() |
| Define or import a dictionary | dictionary() |
| Match phrases / match word counts | tokens_lookup() / dfm_lookup() |
| Inspect context | kwic() |
| Count words | dfm() and ntoken() |
| Optional: combine speeches by party | dfm_group() |
| Change counts to presence / absence | dfm_weight(..., scheme = "boolean") |
| Display a DFM as a regular table | convert(..., to = "data.frame") |
The five-sentence score table is prepared by setup. You will calculate a simple score and the validation measures below.
LIWC and our teaching dictionary
LIWC means Linguistic Inquiry and Word Count. It counts words in categories such as emotion. In this lab, quanteda reads a small, original teaching dictionary saved in LIWC’s legacy .dic format. It is not the official LIWC dictionary, and our score is not LIWC’s Tone measure.
You only need quanteda. See LIWC’s explanation and quanteda’s dictionary importer for background.
1. Read a dictionary and inspect its matches
10 minutes. Open the supplied ten-word list and see which words it matches. Between the two % lines are the category names; below the second %, each word has a category number.
readLines("Dictionaries/teaching_affect.dic") [1] "%" "1 positive" "2 negative" "%" "good 1"
[6] "excellent 1" "success 1" "successful 1" "improve 1" "bad 2"
[11] "failure 2" "crisis 2" "harm 2" "harmful 2" toy_sentiment <- dictionary(file = "Dictionaries/teaching_affect.dic",
format = "LIWC")
toy_sentimentDictionary object with 2 key entries.
- [positive]:
- excellent, good, improve, success, successful
- [negative]:
- bad, crisis, failure, harm, harmfulLet’s start with two short texts. After importing the dictionary above, run this block:
texts <- c("The policy is good.", "The policy is not good.")
example_tokens <- tokens(texts, remove_punct = TRUE)
example_dfm <- dfm(example_tokens)
example_hits <- dfm_lookup(example_dfm, toy_sentiment)
counts <- convert(example_hits, to = "data.frame")
counts doc_id positive negative
1 text1 1 0
2 text2 1 0Both rows have one positive match. tokens() splits text into tokens; dfm() counts words; dfm_lookup() combines word counts into dictionary categories. The rule does not interpret not.
word_count <- ntoken(example_tokens)
positive <- counts$positive
negative <- counts$negative
net_score <- 100 * (positive - negative) / word_count
net_scoretext1 text2
25 20 Scoring means turning dictionary matches into a number for each text. We subtract negative from positive matches, divide by all word tokens, then multiply by 100. We get 25 and 20: one positive match divided by four and five words. The second score is lower because not adds a word to the denominator. Our rule still counts good as positive.
Now inspect the supplied examples that also include an unmatched text, balanced counts and an empty text. The table uses three summaries, where P = positive matches, N = negative matches and L = all word tokens:
| Column | Calculation | What to read |
|---|---|---|
net_per100 |
100 × (P − N) / L | Positive minus negative matches per 100 words |
coverage_per100 |
100 × (P + N) / L | Percentage of words matched by this small dictionary |
balance |
(P − N) / (P + N) | Among matched words, −1 = all negative; +1 = all positive |
A zero denominator gives NA: the score is undefined. The categories in this teaching dictionary do not overlap, and every entry is one word.
# The setup above created these five named examples.
check_texts good negated
"The policy is good." "The policy is not good."
unmatched balanced
"The minister spoke today." "A good plan with a bad outcome."
empty
"" check_tokens <- tokens(check_texts, remove_punct = TRUE)
check_dfm <- dfm(check_tokens)
check_hits <- dfm_lookup(check_dfm, toy_sentiment)
convert(check_hits, to = "data.frame") doc_id positive negative
1 good 1 0
2 negated 1 0
3 unmatched 0 0
4 balanced 1 1
5 empty 0 0# Prepared by workflow.R using the three formulas in the table above.
check_scores doc_id tokens positive negative net_per100 coverage_per100 balance
1 good 4 1 0 25 25.00000 1
2 negated 5 1 0 20 20.00000 1
3 unmatched 4 0 0 0 0.00000 NA
4 balanced 7 1 1 0 28.57143 0
5 empty 0 0 0 NA NA NA
no_match
1 FALSE
2 FALSE
3 TRUE
4 FALSE
5 TRUECheck: compare good, negated, unmatched, balanced and empty. Which examples have the same net score for different reasons? This table uses the supplied dictionary; it will not update automatically if you change the dictionary later.
Discuss: What would our rule need to do differently to distinguish “good” from “not good”? Keeping not in the tokens is only a first step; we still need a rule that uses it.
2. Build an energy-policy dictionary
15–20 minutes. Now we’ll ask a different question: does a speech discuss energy policy? A relevant speech can be positive, negative or neither. We’ll assign a dictionary prediction of 1 when at least one energy expression matches. To check these predictions, we’ll read the texts ourselves using the rules below.
How to label a speech
Read the whole speech contribution. Judge what it discusses, not whether a dictionary word appears.
- 1 — Relevant: it discusses energy production, supply, prices, affordability or transition, such as electricity bills, heating costs, renewable energy or coal-mine closures. A substantive passage is enough; the speaker does not have to support the policy.
- 0 — Not relevant: it does not discuss those issues. Political
power, an unrelated legislativebill, personalenergyand teargasdo not count. General climate discussion counts only when connected to energy. - Uncertain: leave the label blank, write down why, and discuss it before scoring. Do not treat uncertainty as 0.
For each decision, record a short reason or supporting passage. Apply the same rules to texts with and without dictionary matches.
Try the dictionary
policy_entries <- read.csv("Dictionaries/energy_policy.csv")
head(policy_entries) version category pattern
1 v1 energy energy
2 v1 energy power
3 v1 energy electric*
4 v1 energy gas
5 v1 energy bill*
6 v2 energy energy# Read the v1 patterns, then put them in a dictionary category.
v1_entries <- subset(policy_entries, version == "v1")
energy_v1 <- dictionary(list(energy = v1_entries$pattern))
v2_entries <- subset(policy_entries, version == "v2")
energy_v2 <- dictionary(list(energy = v2_entries$pattern))
energy_v1Dictionary object with 1 key entry.
- [energy]:
- energy, power, electric*, gas, bill*energy_v2Dictionary object with 1 key entry.
- [energy]:
- energy, electric*, gas bill*, renewab*, solar*, wind farm*, fossil fuel*contexts <- kwic(discovery_tokens, pattern = c("power", "gas", "bill*"),
valuetype = "glob", window = 8)
head(contexts, 10)Keyword-in-context with 10 matches.
[speech_0006, 32] opinion. on the question whether any additional | power
[speech_0006, 37] whether any additional power is given by the | bill
[speech_0006, 233] the case. the third object of the | bill
[speech_0006, 254] acquired for public purposes. in effect the | bill
[speech_0008, 24] special act undertakers will be over-ridden by this | bill
[speech_0019, 38] were very widely published and there was a | power
[speech_0019, 165] considered these proposals very carefully. he had | power
[speech_0021, 36] it would facilitate the further progress of the | bill
[speech_0021, 61] to avoid lengthening the committee stage. this | bill
[speech_0021, 426] to remove them. there are in the | bill
| is given by the bill to local authorities
| to local authorities in blighted areas—not war-damaged areas—i
| is to regulate the compensation payable in respect
| provides that where land is compulsorily acquired for
| , particularly as regards special provisions which may
| , which we can be quite certain was
| under the act to submit them to the
| if we were to have a reply to
| is a very drastic measure. the housing
| at present no default powers. if a Dictionaries/energy_policy.csv stores version, category and pattern. V1 includes broad power, gas and bill*; illustrative v2 replaces several broad patterns with terms/phrases. * is a glob wildcard, not a regular expression. This CSV is a quanteda input, not a LIWC import file.
discovery_tokens excludes reserved real-text test IDs. It preserves punctuation and stopwords so phrases do not join across punctuation. A separate punctuation-free stream supplies word-token denominators. A phrase match counts as one expression, not all its constituent words. Read the original speech when KWIC is ambiguous.
To read the first KWIC result in full:
first_id <- contexts$docname[1]
original <- subset(speeches, doc_id == first_id)
original$text[1] "I will try again, but I repudiate the idea that I had any irritation at all: that is a matter of opinion. On the question whether any additional power is given by the Bill to local authorities in blighted areas—not war-damaged areas—I say the answer is, \"Yes.\" To start with, it affords greater expedition in acquisition. It absolves them from personal notifications and allows them to give block notifications of an intention to acquire. All the local authorities, so far as I know, have come to the conclusion that this more expeditious procedure will save them some three months—I am not pinning myself down to an exact period—in the compulsory acquisition of areas for redevelopment in their blighted territory. That is the first point. The second point is that for the period of two years—I assume that two years is the time when the re-examination of the subsidies will take place—there is already provision in the Housing Acts whereby authorities who have exceptional costs for redevelopment may get those exceptional costs made up to the extent of about £15 per house. Thirdly, I intimated that the Chancellor had announced that when the block grant came up for revision account would be taken of all the necessities of the case. The third object of the Bill is to regulate the compensation payable in respect of land or property acquired for public purposes. In effect the Bill provides that where land is compulsorily acquired for public purposes under any enactment within a period of five years from November, 1944, the compensation will be assessed by reference to prices current at 31st March, 1939. The general conception is that the land is projected back into a March, 1939, setting, and the effect will be that any changes in the land or buildings since 1939 will be taken into account in fixing the March, 1939, price. For example, if there had been improvements to a property since March, 1939, 258 they would be taken into account. Owner-occupiers of buildings and agricultural land will be able to qualify for supplementary payments up to a maximum of 30 per cent. over the 1939 value of the buildings or land which are compulsorily acquired from them. The need for this 30 per cent. increase in the case of the owner-occupier is due to the cost of replacement. The owner-occupier must find himself another house in which to live, and the Government, as a result of the compromise last year, fixed upon 30 per cent. as an addition to March, 1939, prices. Again, where improvements have been carried out since March, 1939, to buildings or agricultural land at prices greater than March, 1939, prices the owner is entitled in certain circumstances to a reasonable addition for the cost of these improvements, and should there be any disagreement about the supplementary compensation the dispute goes to arbitration."Change one rule, then rerun
V1 includes power, which can mean political authority. Replace it with wind farm*, then compare the old and revised dictionaries on two teaching sentences:
energy_revised <- dictionary(list(
energy = c("energy", "wind farm*", "electric*", "gas", "bill*")
))
revision_texts <- c(
wind = "The government should approve new wind farms.",
authority = "Parliament must retain the power to question ministers."
)
revision_tokens <- tokens(revision_texts, remove_punct = FALSE)
before_tokens <- tokens_lookup(revision_tokens, energy_v1, capkeys = FALSE)
before_counts <- convert(dfm(before_tokens), to = "data.frame")
after_tokens <- tokens_lookup(revision_tokens, energy_revised, capkeys = FALSE)
after_counts <- convert(dfm(after_tokens), to = "data.frame")
data.frame(sentence = before_counts$doc_id,
before = before_counts$energy,
after = after_counts$energy) sentence before after
1 wind 0 1
2 authority 1 0Read the result: wind changes from 0 to 1; authority changes from 1 to 0. Our change fixes both examples. Next, we’ll check other sentences to see which mistakes remain.
Try: in your own energy_revised, add or remove one entry, or replace a broad word with a phrase. Rerun the matching code after each change. Use kwic(discovery_tokens, phrase("wind farm*")) as a template to inspect your term in real context; read the original speech before deciding. Write down the term you changed, a supporting passage and one possible mistake.
For sections 3–4, we’ll use the supplied energy_v2 so we can compare our results with the worked answers. Your own changes remain in energy_revised; the scores we calculate next will refer to energy_v2.
To count multiword expressions, match the dictionary on the tokens first:
matched_tokens <- tokens_lookup(discovery_tokens, energy_v2, capkeys = FALSE)
match_dfm <- dfm(matched_tokens)
head(match_dfm, 6)Document-feature matrix of: 6 documents, 1 feature (100.00% sparse) and 12 docvars.
features
docs energy
speech_0001 0
speech_0002 0
speech_0003 0
speech_0004 0
speech_0005 0
speech_0006 0Each value counts matched expressions in one speech. The original tokens remain available for context checks. Next, compare dictionary predictions with the labels on our teaching sentences.
3. Diagnose development errors
10 minutes. Let’s look for mistakes before calculating a score. Data/synthetic_validation.csv contains 24 sentences written for this exercise, with labels in the column human. These are separate from the 5,000 parliamentary speeches. We’ll inspect the 12 development sentences now and save the 12 test sentences for section 4.
# Read the labelled teaching examples and generate predictions explicitly.
synthetic <- read.csv("Data/synthetic_validation.csv")
example_corpus <- corpus(synthetic, text_field = "text", docid_field = "doc_id")
example_tokens <- tokens(example_corpus, remove_punct = FALSE)
# Apply v1, then v2. Each matched phrase counts once.
v1_matches <- tokens_lookup(example_tokens, energy_v1, capkeys = FALSE)
v1_dfm <- dfm(v1_matches)
v1_binary <- dfm_weight(v1_dfm, scheme = "boolean")
v1_predictions <- convert(v1_binary, to = "data.frame")
synthetic$v1 <- v1_predictions$energy
v2_matches <- tokens_lookup(example_tokens, energy_v2, capkeys = FALSE)
v2_dfm <- dfm(v2_matches)
v2_binary <- dfm_weight(v2_dfm, scheme = "boolean")
v2_predictions <- convert(v2_binary, to = "data.frame")
synthetic$v2 <- v2_predictions$energydfm_weight(..., scheme = "boolean") turns a nonzero count into 1 and keeps zero as 0. convert(..., to = "data.frame") gives a regular table that is easy to inspect. These predictions come from our dictionary, while human contains the supplied teaching labels. Read development cases first; leave test cases for section 4.
dev <- subset(synthetic, split == "development")
dev[, c("doc_id", "text", "human", "v1", "v2")] doc_id text human v1 v2
1 dev01 We must address rising energy prices. 1 1 1
2 dev02 Electricity supply needs public investment. 1 1 1
3 dev03 Families cannot afford their gas bills. 1 1 1
4 dev04 Subsidies should support solar panels. 1 0 1
5 dev05 The government should approve new wind farms. 1 0 1
6 dev06 We should reduce fossil fuel imports. 1 0 1
7 dev07 Parliament must retain the power to question ministers. 0 1 0
8 dev08 This bill reforms criminal sentencing. 0 1 0
9 dev09 Police used tear gas at the protest. 0 1 0
10 dev10 The minister brought energy to the debate. 0 1 1
11 dev11 Schools need a larger budget. 0 0 0
12 dev12 The committee discussed train timetables. 0 0 0Try: find one false positive and one false negative under v1. Which word caused the false match? Which expression did the dictionary miss? Check whether v2 fixes these cases, then find a mistake that remains.
# Your error analysis here.Freeze means stop changing the rules before checking test results. We’ll keep the dictionary, labeling instructions, tokenization and prediction rule fixed (here, at least one match means 1). For your own project, save this version so you have a record of exactly what you tested.
4. Check a frozen rule on test examples
10–15 minutes. Finish the development discussion first. Use the supplied v2 as a fixed worked example. Predict which errors will lower precision or recall, then run the code.
test <- subset(synthetic, split == "test")
# 0 = not relevant; 1 = relevant. Keep both categories in the table.
predicted <- factor(test$v2, levels = c(0, 1))
human <- factor(test$human, levels = c(0, 1))
table(predicted, human) human
predicted 0 1
0 5 2
1 1 4- Precision = TP / (TP + FP): how many predicted positives are correct?
- Recall = TP / (TP + FN): how many actual positives were found?
- F1 = 2TP / (2TP + FP + FN): how well are precision and recall balanced?
# Compare each dictionary prediction with its teaching label.
TP <- sum(test$v2 == 1 & test$human == 1)
FP <- sum(test$v2 == 1 & test$human == 0)
FN <- sum(test$v2 == 0 & test$human == 1)
TN <- sum(test$v2 == 0 & test$human == 0)
precision <- TP / (TP + FP)
recall <- TP / (TP + FN)
F1 <- 2 * TP / (2 * TP + FP + FN)
c(TP = TP, FP = FP, FN = FN, TN = TN)TP FP FN TN
4 1 2 5 c(precision = precision, recall = recall, F1 = F1)precision recall F1
0.8000000 0.6666667 0.7272727 Read the result: our dictionary flags 5 sentences, and 4 are relevant (precision = 4/5). Of the 6 relevant sentences, we find 4 (recall = 4/6). F1 combines precision and recall; it is not accuracy. If a denominator is zero in your own data, report that measure as NA.
These results describe the 12 test sentences. To evaluate our dictionary on parliamentary speeches, we would need human labels for those speeches.
Discuss: How would we find missed cases if we inspected only dictionary matches? Why might accuracy look high when very few texts discuss energy? If we change v2 after reading the test errors, what would we need for a fresh evaluation? We’ll return to these questions in W6.
Before W6: five texts and one difficult decision
10 minutes; no upload. W5 is a holiday. Before we meet in W6, read these five parliamentary speeches from our development sample:
development_texts <- subset(coding_template, split == "development")
preview <- head(development_texts, 5)
preview[, c("doc_id", "text")] doc_id
1 speech_3484
2 speech_0299
3 speech_3560
4 speech_1238
5 speech_0963
text
1 That is why I have appealed to the Disraeli wing of the Conservative party, which formed a radical, reforming Government who addressed some of the abuses of power of previous Governments. In appealing to Conservative Members to support the amendments, I draw their attention to the congruity between the intent and content of the amendments and the policies set out in their party's manifesto at the last election. In particular, I draw their attention to the statement that strongly supports my compromise amendments, which reads:" Radical changes that alter the whole character of our constitutional balance could unravel what generations of our predecessors have created. To preserve that stability in future--and the freedoms and rights of our citizens--we need to continue a process of evolution, not revolution. Conservatives embrace evolutionary change that solves real problems and improves the way our constitution works." On that basis, I argue for support from Conservative Members and for them to embrace the evolutionary changes set out in the amendments. I now seek to persuade my comrades in the Labour party, or colleagues in new Labour, to support my amendments tonight.
2 The Report of the Resale Price Maintenance Committee described the evidence they received. I do not think that publication of the evidence in full is necessary to enable traders to give the consideration to the Report which was suggested by my right hon. Friend in his statement on 2nd June, in reply to a Question by my hon. Friend the Member for Elland (Mr. Cobb).
3 This petition is signed by more than 500 constituents of mine, who are extremely concerned about firework nuisance. Over the past few winters, there has been a big increase in firework abuse, with injury to pets and farm animals. I wrote to every veterinary surgery and many local action groups-hence the 500 signatures. The petitioners request that the House of Commons urge the Government to conduct an urgent review of our fireworks legislation and to legislate, among other things, To restrict the times of the year fireworks can be bought;To restrict the times of the day fireworks can be set off; and To license all vendors of fireworks to meet strict safety criteria and to give trading standards the power to revoke the licence of any vendor caught selling fireworks to under-age children. To lie upon the Table.
4 Does the right hon. Lady recollect that the major part of her speech was devoted to telling us that the reasons why there was such a long delay in the introduction of the pension increase were administrative reasons, and that subsequently the Secretary of State for Economic Affairs said the reasons were economic reasons? Surely she could be a little more forthcoming, and, indeed, open, and tell the House whether the reasons are administrative or economic?
5 The information asked for is not available for 1959. From January to March, 1960, goods vehicles of over 3 tons unladen weight were involved in 3,087 accidents. Of these 201 were fatal.Read without dictionary predictions. Apply the three labeling rules above, record a short reason for each 0/1 label, and flag uncertainty instead of guessing. Bring one difficult decision, or a hypothetical boundary case if all five are clear. A random batch need not contain any positives.
| W4 preparation | W6 develops |
|---|---|
| Define a category and defend a label | Annotation quality and inter-coder reliability |
| Notice false positives and negatives | Performance evaluation and error analysis |
| Separate development from evaluation | Training/test splits and cross-validation |
| Keep a transparent dictionary baseline | Naive Bayes and SVM document classifiers |
We’ll use these five speeches to discuss how we decide on a label. Read them without looking at dictionary predictions, and keep them in the development set if we use them to revise our rules. We’ll need more labeled texts before training or evaluating a classifier.
Real-text validation in W6
In W6, we’ll work through human coding and validation with real texts. For now, complete the short-sentence exercise and the five-speech preview above. Setup also reserves 50 parliamentary speeches for later testing. We leave those untouched in W4; they are a separate set from the 12 test sentences we just used.
Optional: compare parties
This extension is not needed for the four core tasks.
Show the party comparison
After sections 1–4, you can summarize dictionary matches by party for the full sample. policy_tokens preserves punctuation for phrase matching; toks contains the original word tokens for the denominator:
# quanteda adds counts within each party.
party_matches <- tokens_lookup(policy_tokens, energy_v2, capkeys = FALSE)
party_dfm <- dfm(party_matches)
party_dfm <- dfm_group(party_dfm, groups = party)
# Group original word counts separately to preserve the denominator.
party_words <- dfm(toks)
party_words <- dfm_group(party_words, groups = party)
party_summary <- convert(party_dfm, to = "data.frame")
party_summary$tokens <- ntoken(party_words)
party_summary$per1000 <- 1000 * party_summary$energy / party_summary$tokens
head(party_summary, 6) doc_id energy tokens per1000
1 Alliance Party of Northern Ireland 0 44 0.00000000
2 Communist Party of Great Britain 0 36 0.00000000
3 Conservative 151 447158 0.33768824
4 Democratic Unionist Party 1 11814 0.08464534
5 Green Party of England and Wales 2 217 9.21658986
6 Independent 1 1394 0.71736011dfm_group() performs the grouping; convert() turns the result into a table. We keep the final division explicit to show its denominator. Do not apply dfm_weight(..., "prop") to a matched-only DFM here: it would divide by matched counts instead of all original words.
For each party, we calculate 1,000 * total matches / total original word tokens. We add the counts before dividing, rather than averaging the rates of individual speeches. These rates tell us how often our dictionary matches words in each party’s sampled speeches. To interpret a difference, we also need to consider when the speeches were made, their topics, and how well the dictionary works. A match alone does not tell us whether the speaker supports an energy policy.
POS and TextRank examples are in the separate classroom materials. They are not part of this take-home lab.
Reading connections
Our three required readings are Denny and Spirling (2018), DOI, on preprocessing sensitivity; Rauh (2018), DOI, on sentiment validation in political texts; and Proksch et al. (2019), DOI, on political-language measurement across settings.
For another example of dictionary development and testing, see the supplementary lecture case by Boyle et al. (2025), DOI. That study measures policy relevance in research abstracts; our exercise asks whether a sentence discusses energy.
Free reading: Computational Analysis of Communication, Chapters 10-11. Technical references: LIWC Workbench, dictionary import, token lookup.
Selected activities draw on ESS materials by Martijn Schoonvelde (2026), acknowledging Stefan Muller. The parliamentary sample is reused from W2; the short sentences and small dictionaries are course-designed teaching examples.