# install.packages("quanteda") # run once before class
library(quanteda)
# Find the lab folder; if asked, select workflow.R inside W4/lab.
paths <- c("W4/lab/workflow.R", "lab/workflow.R",
"../lab/workflow.R", "../workflow.R", "workflow.R")
found <- paths[file.exists(paths)]
if (length(found) == 0) found <- file.choose()
setwd(dirname(normalizePath(found[1])))
source("workflow.R") # load the data and prepared examplesW4 Lab: Worked Answers
LIWC-format dictionaries, custom policy rules, and a W6 preview
Let’s work through the results from our lab. For the validation exercise, we’ll use the 24 example sentences and their supplied labels. Compare these outputs with your own, and read the sentences behind any differences. Return to the student handout.
Run the setup block once, then follow sections 1–4 in order. If a file picker opens, choose workflow.R inside the extracted lab. The party comparison is optional.
1. LIWC format and sentiment
toy_sentimentDictionary object with 2 key entries.
- [positive]:
- excellent, good, improve, success, successful
- [negative]:
- bad, crisis, failure, harm, harmfultexts <- c("The policy is good.", "The policy is not good.")
example_tokens <- tokens(texts, remove_punct = TRUE)
example_dfm <- dfm(example_tokens)
example_hits <- dfm_lookup(example_dfm, toy_sentiment)
counts <- convert(example_hits, to = "data.frame")
counts doc_id positive negative
1 text1 1 0
2 text2 1 0word_count <- ntoken(example_tokens)
positive <- counts$positive
negative <- counts$negative
100 * (positive - negative) / word_counttext1 text2
25 20 # The setup above created these five named examples.
check_texts good negated
"The policy is good." "The policy is not good."
unmatched balanced
"The minister spoke today." "A good plan with a bad outcome."
empty
"" check_tokens <- tokens(check_texts, remove_punct = TRUE)
check_dfm <- dfm(check_tokens)
check_hits <- dfm_lookup(check_dfm, toy_sentiment)
convert(check_hits, to = "data.frame") doc_id positive negative
1 good 1 0
2 negated 1 0
3 unmatched 0 0
4 balanced 1 1
5 empty 0 0# Prepared by workflow.R using the three formulas explained in class.
check_scores doc_id tokens positive negative net_per100 coverage_per100 balance
1 good 4 1 0 25 25.00000 1
2 negated 5 1 0 20 20.00000 1
3 unmatched 4 0 0 0 0.00000 NA
4 balanced 7 1 1 0 28.57143 0
5 empty 0 0 0 NA NA NA
no_match
1 FALSE
2 FALSE
3 TRUE
4 FALSE
5 TRUE# TRUE counts as 1; the mean is the fraction without a match.
mean(sentiment$no_match)[1] 0.8336Category 1 is positive; category 2 is negative. The dictionary matches words without interpreting who or what is being evaluated. The prepared check_scores table uses the original dictionary; changing a dictionary requires rerunning the scoring code.
We get 25 net matches per 100 tokens for good and 20 for not good. Our rule counts good as positive in both sentences; adding not increases the denominator. An unmatched text has zero coverage. A text with balanced positive and negative counts has zero net tone but nonzero coverage. An empty text has a zero denominator, so its rates are missing. When we apply the dictionary to the speeches, a text with no matches may still express an opinion in words our list does not include.
Our ten-word teaching dictionary uses LIWC’s .dic file format. We apply it with quanteda and calculate our own score, so the result differs from LIWC’s Tone measure. See LIWC Analysis and quanteda import documentation.
2. Build an energy-policy dictionary
policy_entries <- read.csv("Dictionaries/energy_policy.csv")
v1_entries <- subset(policy_entries, version == "v1")
energy_v1 <- dictionary(list(energy = v1_entries$pattern))
v2_entries <- subset(policy_entries, version == "v2")
energy_v2 <- dictionary(list(energy = v2_entries$pattern))
contexts <- kwic(discovery_tokens, pattern = c("power", "gas", "bill*"),
valuetype = "glob", window = 8)
head(contexts, 8)Keyword-in-context with 8 matches.
[speech_0006, 32] opinion. on the question whether any additional | power
[speech_0006, 37] whether any additional power is given by the | bill
[speech_0006, 233] the case. the third object of the | bill
[speech_0006, 254] acquired for public purposes. in effect the | bill
[speech_0008, 24] special act undertakers will be over-ridden by this | bill
[speech_0019, 38] were very widely published and there was a | power
[speech_0019, 165] considered these proposals very carefully. he had | power
[speech_0021, 36] it would facilitate the further progress of the | bill
| is given by the bill to local authorities
| to local authorities in blighted areas—not war-damaged areas—i
| is to regulate the compensation payable in respect
| provides that where land is compulsorily acquired for
| , particularly as regards special provisions which may
| , which we can be quite certain was
| under the act to submit them to the
| if we were to have a reply to example_context <- kwic(discovery_tokens, pattern = "bill*", window = 8)
# The supplied corpus contains matches to bill*.
first_id <- example_context$docname[1]
original <- subset(speeches, doc_id == first_id)
original$text[1] "I will try again, but I repudiate the idea that I had any irritation at all: that is a matter of opinion. On the question whether any additional power is given by the Bill to local authorities in blighted areas—not war-damaged areas—I say the answer is, \"Yes.\" To start with, it affords greater expedition in acquisition. It absolves them from personal notifications and allows them to give block notifications of an intention to acquire. All the local authorities, so far as I know, have come to the conclusion that this more expeditious procedure will save them some three months—I am not pinning myself down to an exact period—in the compulsory acquisition of areas for redevelopment in their blighted territory. That is the first point. The second point is that for the period of two years—I assume that two years is the time when the re-examination of the subsidies will take place—there is already provision in the Housing Acts whereby authorities who have exceptional costs for redevelopment may get those exceptional costs made up to the extent of about £15 per house. Thirdly, I intimated that the Chancellor had announced that when the block grant came up for revision account would be taken of all the necessities of the case. The third object of the Bill is to regulate the compensation payable in respect of land or property acquired for public purposes. In effect the Bill provides that where land is compulsorily acquired for public purposes under any enactment within a period of five years from November, 1944, the compensation will be assessed by reference to prices current at 31st March, 1939. The general conception is that the land is projected back into a March, 1939, setting, and the effect will be that any changes in the land or buildings since 1939 will be taken into account in fixing the March, 1939, price. For example, if there had been improvements to a property since March, 1939, 258 they would be taken into account. Owner-occupiers of buildings and agricultural land will be able to qualify for supplementary payments up to a maximum of 30 per cent. over the 1939 value of the buildings or land which are compulsorily acquired from them. The need for this 30 per cent. increase in the case of the owner-occupier is due to the cost of replacement. The owner-occupier must find himself another house in which to live, and the Government, as a result of the compromise last year, fixed upon 30 per cent. as an addition to March, 1939, prices. Again, where improvements have been carried out since March, 1939, to buildings or agricultural land at prices greater than March, 1939, prices the owner is entitled in certain circumstances to a reasonable addition for the cost of these improvements, and should there be any disagreement about the supplementary compensation the dispute goes to arbitration."In these contexts, power may refer to political authority, bill* may match legislation, and gas may refer to tear gas. We need to read the passage to decide whether each match concerns energy. If we propose another version, we should be able to point to development examples that explain each change.
Change one rule, then rerun
V1 includes power, which can mean political authority. Replace it with wind farm*, then compare the old and revised dictionaries on two teaching sentences:
energy_revised <- dictionary(list(
energy = c("energy", "wind farm*", "electric*", "gas", "bill*")
))
revision_texts <- c(
wind = "The government should approve new wind farms.",
authority = "Parliament must retain the power to question ministers."
)
revision_tokens <- tokens(revision_texts, remove_punct = FALSE)
before_tokens <- tokens_lookup(revision_tokens, energy_v1, capkeys = FALSE)
before_counts <- convert(dfm(before_tokens), to = "data.frame")
after_tokens <- tokens_lookup(revision_tokens, energy_revised, capkeys = FALSE)
after_counts <- convert(dfm(after_tokens), to = "data.frame")
data.frame(sentence = before_counts$doc_id,
before = before_counts$energy,
after = after_counts$energy) sentence before after
1 wind 0 1
2 authority 1 0Read the result: wind changes from 0 to 1; authority changes from 1 to 0. Our change fixes both examples. Next, we’ll check other sentences to see which mistakes remain.
Your own revision may differ from this example. To explain it, show the entry you changed, a passage that supports the change, and a case it might still get wrong. We can use kwic(discovery_tokens, phrase("wind farm*")) as a starting point for checking a phrase in the speeches.
For sections 3–4, we’ll use the supplied energy_v2 so we can compare our results. Your own changes remain in energy_revised; the scores below refer to energy_v2.
3. Development errors
# Read the labelled teaching examples and generate predictions explicitly.
synthetic <- read.csv("Data/synthetic_validation.csv")
example_corpus <- corpus(synthetic, text_field = "text", docid_field = "doc_id")
example_tokens <- tokens(example_corpus, remove_punct = FALSE)
# Apply v1, then v2. Each matched phrase counts once.
v1_matches <- tokens_lookup(example_tokens, energy_v1, capkeys = FALSE)
v1_dfm <- dfm(v1_matches)
v1_binary <- dfm_weight(v1_dfm, scheme = "boolean")
v1_predictions <- convert(v1_binary, to = "data.frame")
synthetic$v1 <- v1_predictions$energy
v2_matches <- tokens_lookup(example_tokens, energy_v2, capkeys = FALSE)
v2_dfm <- dfm(v2_matches)
v2_binary <- dfm_weight(v2_dfm, scheme = "boolean")
v2_predictions <- convert(v2_binary, to = "data.frame")
synthetic$v2 <- v2_predictions$energy
dev <- subset(synthetic, split == "development")
errors <- subset(dev, v1 != human)
errors[, c("doc_id", "text", "human", "v1", "v2")] doc_id text human v1 v2
4 dev04 Subsidies should support solar panels. 1 0 1
5 dev05 The government should approve new wind farms. 1 0 1
6 dev06 We should reduce fossil fuel imports. 1 0 1
7 dev07 Parliament must retain the power to question ministers. 0 1 0
8 dev08 This bill reforms criminal sentencing. 0 1 0
9 dev09 Police used tear gas at the protest. 0 1 0
10 dev10 The minister brought energy to the debate. 0 1 1The development examples show why we might remove standalone power, gas and bill*, and add expressions about renewable energy, solar panels, wind farms and fossil fuels. V2 fixes several mistakes, but it still matches energy when the sentence describes a person’s enthusiasm. To address that case, we would need a rule that uses the surrounding words, then more examples to check whether it works.
4. Synthetic test results
test <- subset(synthetic, split == "test")
# 0 = not relevant; 1 = relevant. Keep both categories in the table.
predicted <- factor(test$v2, levels = c(0, 1))
human <- factor(test$human, levels = c(0, 1))
table(predicted, human) human
predicted 0 1
0 5 2
1 1 4errors <- subset(test, v2 != human)
errors[, c("doc_id", "text", "human", "v2")] doc_id text human v2
17 test05 The closure of coal mines requires a transition plan. 1 0
18 test06 Heating tariffs must remain affordable. 1 0
22 test10 Her energy transformed the committee. 0 1Calculate the counts and fractions explicitly:
predicted <- test$v2
human <- test$human
TP <- sum(predicted == 1 & human == 1)
FP <- sum(predicted == 1 & human == 0)
FN <- sum(predicted == 0 & human == 1)
TN <- sum(predicted == 0 & human == 0)
precision <- TP / (TP + FP)
recall <- TP / (TP + FN)
F1 <- 2 * TP / (2 * TP + FP + FN)
c(TP = TP, FP = FP, FN = FN, TN = TN)TP FP FN TN
4 1 2 5 c(precision = precision, recall = recall, F1 = F1)precision recall F1
0.8000000 0.6666667 0.7272727 These denominators are nonzero. In other data, report an undefined rate as NA when its denominator is zero.
For v2, we get TP = 4, FP = 1, FN = 2, TN = 5. Precision = 0.80, recall = 0.667, F1 = 0.727. V1 finds the same four relevant sentences but also flags four irrelevant ones. V2 reduces those false positives to one: figurative energy. It still misses the sentences about coal-mine closures and heating tariffs.
These results describe the 12 test sentences. To evaluate our dictionary on parliamentary speeches, we would need human labels for those speeches.
If we read only the matches, we would never find the two missed sentences and could not calculate recall. When relevant texts are rare, predicting 0 for every text can give high accuracy while finding none of them. F1 helps us focus on precision and recall, but we should also report the four counts and explain how we chose the texts. Without predicted positives, precision is undefined; without relevant texts in our labeled sample, recall is undefined.
We could now add coal or heating, but that would use information from the test sentences to change our dictionary. We would need a fresh test set to evaluate that new version.
W6 preview: labels before models
W5 (October 9) is a holiday. W6 (October 16), Human Coding and Document Classification, covers annotation quality, inter-coder reliability, training/test sets and performance; its lab introduces Naive Bayes, SVM and cross-validation.
For the five-speech warm-up, explain how you reached each label. Point to a passage, apply our inclusion/exclusion rules, and note anything you are unsure about. We’ll compare difficult cases and refine the labeling rules in W6. No model training or extra submission is required now.
We’ll use these five speeches for discussion. We’ll need more labeled texts for training and evaluation, with a separate test set that we leave aside while developing the model. Make your own reading-based judgments before looking at dictionary predictions.
Real-text validation in W6
Use the three labeling rules in the handout for the five-speech preview. We’ll work through independent coding and evaluation in W6. The parliamentary speeches have no supplied human labels; we’ll need to collect those judgments before we can check the dictionary’s performance on them.
Optional: compare parties
# quanteda adds counts within each party.
party_matches <- tokens_lookup(policy_tokens, energy_v2, capkeys = FALSE)
party_dfm <- dfm(party_matches)
party_dfm <- dfm_group(party_dfm, groups = party)
# Group original word counts separately to preserve the denominator.
party_words <- dfm(toks)
party_words <- dfm_group(party_words, groups = party)
party_summary <- convert(party_dfm, to = "data.frame")
party_summary$tokens <- ntoken(party_words)
party_summary$per1000 <- 1000 * party_summary$energy / party_summary$tokens
head(party_summary, 6) doc_id energy tokens per1000
1 Alliance Party of Northern Ireland 0 44 0.00000000
2 Communist Party of Great Britain 0 36 0.00000000
3 Conservative 151 447158 0.33768824
4 Democratic Unionist Party 1 11814 0.08464534
5 Green Party of England and Wales 2 217 9.21658986
6 Independent 1 1394 0.71736011We use dfm_group() to add counts within each party, ntoken() to count the original words, and convert() to display the result as a table. For each party, we divide total matches by total original word counts and multiply by 1,000. This lets us compare speeches of different total lengths. To interpret party differences, we still need to check the dates, topics and dictionary matches; the rate alone does not tell us whether a party supports a policy.
R session information
sessionInfo()R version 4.4.1 (2024-06-14)
Platform: aarch64-apple-darwin20
Running under: macOS 26.7.1
Matrix products: default
BLAS: /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/lib/libRblas.0.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/lib/libRlapack.dylib; LAPACK version 3.12.0
locale:
[1] C.UTF-8/C.UTF-8/C.UTF-8/C/C.UTF-8/C.UTF-8
time zone: Asia/Taipei
tzcode source: internal
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] quanteda_4.3.1
loaded via a namespace (and not attached):
[1] digest_0.6.39 fastmatch_1.1-8 fastmap_1.2.0 xfun_0.57
[5] Matrix_1.7-5 lattice_0.22-9 magrittr_2.0.4 knitr_1.51
[9] htmltools_0.5.9 rmarkdown_2.30 lifecycle_1.0.5 cli_3.6.5
[13] stopwords_2.4 grid_4.4.1 compiler_4.4.1 tools_4.4.1
[17] evaluate_1.0.5 Rcpp_1.1.1 yaml_2.3.12 otel_0.2.0
[21] rlang_1.1.7 jsonlite_2.0.0 htmlwidgets_1.6.4 stringi_1.8.7