Text Preparation in R and Python: Collaborative Workflow

Week 2: Text Preparation in R and Python: Collaborative Workflow

• Set up RStudio and Positron for text analysis with R and Python.
• Organize code with GitHub and submit written coursework through shared Overleaf projects.
• Inspect supplied corpora and check project data feasibility: coverage, metadata, and access.
• Clean and tokenize text, build a DFM, and evaluate preprocessing choices.

1. Domain knowledge defines the research problem and relevant evidence.
2. Human judgment guides interpretation and validation.
3. Discovery and testing require distinct stages.
4. The task determines which methods are useful.
5. Validation evaluates whether the measure fits the claim.

Classroom example: measuring expressed preference for coffee.

Found data: document how the texts were produced, preserved and retrieved.

The next step is to examine what the collection process leaves out.

Changing the unit to speaker-day or party-year changes the comparison.

Scanned PDFs may need OCR before they contain usable text.

The tools can work together: lemmas or selected annotations can become features in a quanteda DFM.

Capabilities overlap and depend on the language, model, and task.

One possible application: compare the meanings of policy statements.

The languages can share a project. Each stage follows the research question.

An IDE organizes the work. Object or file exchange connects the languages.

In 2022, RStudio the company became Posit, reflecting its broader support for data science across languages.

The branch matters: analysis can use a matrix, token positions, or other representations.

stringr handles strings. quanteda provides the corpus, tokens, and DFM workflow.

texts <- readLines("data/speeches.txt", encoding = "UTF-8")
length(texts)
head(texts)

• Confirm the source encoding rather than guessing from appearance.
• Inspect accented characters and Chinese text after import.
• Count missing texts and duplicated identifiers.
• Check whether a row is a line, paragraph, or whole document.

A UTF-8 argument does not repair a wrongly identified source encoding.

Text includes [Interruption.], DRAMATICALLY, and !!!.

Which elements are metadata, meaningful language, or recording conventions?

Keep an untouched copy of the raw text.

s <- "Trade [Commons] and jobs [Wales]"

str_remove_all(s, "\\[.*\\]")
# "Trade "

str_remove_all(s, "\\[[^\\[\\]]*\\]") |> str_squish()
# "Trade and jobs"

The first pattern spans from the first [ to the last ]. The second handles separate, non-nested bracket groups.

agenda_raw <- "  Export Trade [Commons] -- vol. 407  "
agenda_clean <- agenda_raw |>
  str_remove_all("\\[[^\\[\\]]*\\]") |>
  str_to_lower() |>
  str_replace_all("[^[:alnum:][:space:]]", " ") |>
  str_squish()
agenda_clean

[1] "export trade vol 407"

This rule preserves numbers and removes punctuation. Another task may need a different rule.

These short texts let us inspect every token and verify every count.

library(quanteda)

corp <- corpus(
  raw,
  docid_field = "doc_id",
  text_field = "text"
)

docnames(corp)
docvars(corp)

Each row supplies one text. The remaining columns become document variables.

Text: “Health services need health workers.”

After normalization: health, services, need, health, workers

A token is an occurrence. A type is a distinct form under the chosen representation.

Both use tokens: units produced by tokenization for analysis or model processing.

One word can become several LLM tokens. R can call either kind of tokenizer; the workflow determines the unit.

toks <- tokens(corp, remove_punct = TRUE) |>
  tokens_tolower()

as.list(toks)[[1]]

[1] "health"   "workers"  "need"     "health"   "services"

Inspect the output when texts contain hyphens, URLs, abbreviations, or non-Latin scripts.

The relevant question is how the choice affects the intended measure.

sw <- setdiff(stopwords("en"), c("not", "no", "nor"))
toks_content <- tokens_remove(toks, pattern = sw)

Retaining not preserves a clue. A unigram model still needs a way to use its scope.

A DFM stores one row per document and one column per feature.

Selected columns from our count DFM. Features can also represent phrases or other categories.

“Workers support government.”
“Government support workers.”

The two sentences have the same lowercase unigram counts.

• Word order indicates who supports whom.
• Negation requires relationships between words.
• Ambiguous words need context.
• Sarcasm may depend on discourse beyond a sentence.

N-grams and syntax retain some additional information.

phrase_toks <- tokens(
  "We support the armed forces.",
  remove_punct = TRUE
) |> tokens_tolower()

tokens_compound(
  phrase_toks,
  pattern = phrase("armed forces")
)

armed_forces becomes a single feature. Detect phrases before removing words that belong inside them.

tokens("Workers do not support the policy.",
       remove_punct = TRUE) |>
  tokens_tolower() |>
  tokens_ngrams(n = 2)

workers_do, do_not, not_support, support_the, the_policy

Generate these before stopword removal if original adjacency matters. Deleting words first can create pairs that were never adjacent.

A frequent or associated word pair is a candidate feature. Inspect its uses before adding it to the measurement rule.

Neither approach guarantees correct semantic grouping. A dictionary built from full words may fail to match stemmed features.

A BoW preprocessing recipe should not automatically become an LLM input recipe.

• All three provide segmented token and predicted upos (POS).
• Dependency parsing adds heads and relation labels.
• lemma is a predicted base form; this Thai model returns _ (unavailable).

Instructor demonstration: interpret the prepared model outputs shown in these slides.

library(udpipe)
# Set model_path to a downloaded Chinese-GSD model.
zh_model <- udpipe_load_model(model_path)
ann_zh <- udpipe_annotate(zh_model, x = "我喜歡咖啡。",
  doc_id = "zh_coffee", parser = "default") |> as.data.frame()
ann_zh[c("token_id", "token", "upos",
         "head_token_id", "dep_rel")]

Chinese: chinese-gsd; English: english-ewt; Thai: R + UDPipe 2 API.

• parser = "default" adds dependency parsing to tokens and POS.
• head_token_id identifies the head; dep_rel labels the relationship. Head 0 marks the sentence root.

# Assume ann_multi contains an annotation table.
a <- subset(ann_multi, upos != "PUNCT")
a <- a[order(a$doc_id, a$sentence_id,
             as.integer(a$token_id)), ]

ja_tokens <- split(a$token[a$doc_id == "ja_01"],
                   a$doc_id[a$doc_id == "ja_01"])
ja_dfm <- dfm(as.tokens(ja_tokens), tolower = FALSE)

Use as.tokens() to preserve the existing segmentation. Build the Chinese DFM from the Chinese documents in the same way.

Code illustration: annotations → tokens → DFM.

x <- dfm(toks_content)
dim(x)
topfeatures(x, 6)

[1] 4 9

    health    workers   services government   supports     policy 
         3          2          2          2          2          2 

Rows retain the document IDs. The full DFM includes features beyond the selected columns shown earlier.

docfreq(x)
x_trim <- dfm_trim(x, min_docfreq = 2,
                  docfreq_type = "count")

This threshold removes not. A rare feature can still carry essential meaning.

The choice changes what repeated words contribute. Model requirements may constrain which representation is appropriate.

For this lesson, use raw counts and a base-10 logarithm:

• N is the number of documents in the reference corpus.
• df(t) counts documents that contain feature t.
• A feature in every document receives IDF = 0 under this formula.

Different normalizations and smoothing rules produce different weights.

For health, N = 4, df = 2, so IDF = log₁₀(2) ≈ 0.3010.

x_weighted <- dfm_tfidf(x, scheme_tf = "count",
                      scheme_df = "inverse", base = 10)

kwic(toks, pattern = "support*", window = 3)

Read the context around a matched term before treating its count as support for a policy.

Wordfish is an unsupervised method for scaling texts on one dimension.

Keep the corpus and research question fixed while comparing a chosen rule.

Download and install: R and Python 3, then RStudio Desktop and Positron.

Try the consoles: 1 + 1 in R; print(1 + 1) in Python. Both return 2.

Open Lab_Session_W2.R in lab/. In the R console, install:

install.packages(c("dplyr", "stringr", "ggplot2",
  "quanteda", "quanteda.textstats", "quanteda.textplots"))

Self-study practice: ungraded; no submission required. Use the supplied sample of 5,000 House of Commons speech contributions (1945–2025).

1. Inspect strings and clean agenda metadata with stringr.
2. Build, subset, and reshape a quanteda corpus.
3. Tokenize, preserve phrases, and inspect KWIC contexts.
4. Build and trim DFMs; explore frequencies, plots, and keyness.
5. Work through the 11 practice exercises in the lab.

• Run the worked examples section by section and inspect each output.
• Bring the exercise number, relevant code, and output or error message.
• Discuss questions before class, if time permits, or ask during TA hours.

Explore one application paper introduced earlier. Start with its research question and a main figure.

1. What is the research question, and what counts as a document?

2. How could word removal, document aggregation, or weighting affect the measure?

3. What evidence supports the authors’ interpretation beyond successful preprocessing?

UDPipe guide, Japanese GSD, Chinese GSD.

The English, Thai, Japanese, and Chinese texts are synthetic teaching examples.