Using APIs

SICSS, 2022

Christopher Barrie

Introduction

  • Why get tweeters?
    • Get network characteristics
    • Get user demographics
    • Linked survey designs

Introduction

Hydrating

tweet_IDs <- readRDS("data/wm_IDs_samp.rds")
head(tweet_IDs)
[1] "823360655835168640" "822727580923154432" "822953780127862784"
[4] "823136522199253120" "821584199949942784" "822868286979342208"
library(academictwitteR)

hydrated_tweets <- hydrate_tweets(tweet_IDs, errors = T,
                                  data_path = "data/hydrated_tweets/")
Batch 1 out of 10 : ids 823360655835168640 to 822916951878103040 
Total  78  tweet(s) can't be retrieved.
Total of  22  out of  1000  tweet(s) retrieved.
Batch 2 out of 10 : ids 823501383282241408 to 822817264915378048 
Total  154  tweet(s) can't be retrieved.
Total of  46  out of  1000  tweet(s) retrieved.
Batch 3 out of 10 : ids 823087701117337600 to 823227815374061440 
Total  225  tweet(s) can't be retrieved.
Total of  75  out of  1000  tweet(s) retrieved.
Batch 4 out of 10 : ids 823045907264499712 to 825034383769767808 
Total  301  tweet(s) can't be retrieved.
Total of  99  out of  1000  tweet(s) retrieved.
Batch 5 out of 10 : ids 822984540494950400 to 822996642618101760 
Error in make_query(url = endpoint_url, params = params, bearer_token = bearer_token,  : 
  something went wrong. Status code: 400

Hydrating

Alternatively:

Use the hydrator tool from DocTheNow

Getting tweeters

library(academictwitteR)
library(dplyr)
library(lubridate)
library(ggplot2)


cjb_ID <- get_user_id("cbarrie")
cjb_ID
   cbarrie 
"95226101"

Getting whom I follow (friends)

userfwing <- get_user_following(cjb_ID)

ids <- userfwing$id

head(ids)
[1] "1451947528333762562" "326311946"           "2458351124"         
[4] "29943199"            "124852105"           "111456333"
id url created_at profile_image_url public_metrics verified username description entities name location protected pinned_tweet_id from_id
104684456 https://t.co/TcDWPSCaeU 2010-01-14T02:29:06.000Z https://pbs.twimg.com/profile_images/1433888388914974720/xRq90qJ9_normal.jpg 8428 FALSE marieeberry Associate Prof at @JosefKorbel at @UofDenver | On sabbatical trying to write some books and snuggle my basset hound and take lots of naps ✌️ 0, 23, https://t.co/TcDWPSCaeU, http://www.marieeberry.com, marieeberry.com Marie Berry Denver, CO FALSE 1500881627584421889 95226101
826168640118546433 https://t.co/RQ5IytdKWo 2017-01-30T20:42:08.000Z https://pbs.twimg.com/profile_images/1411032253292023817/IOs20oR-_normal.jpg 1885 FALSE CodeOceanHQ Where great #computational research is created, organized, and shared. #computationalscience #biotechnology #biopharma #bioinformatics #scientificcomputing 0, 23, https://t.co/RQ5IytdKWo, http://codeocean.com, codeocean.com Code Ocean New York, NY FALSE NA 95226101
1084732873670828032 https://t.co/SeBSGFUwgJ 2019-01-14T08:44:05.000Z https://pbs.twimg.com/profile_images/1328363052614430723/knPCsbnu_normal.jpg 3788 FALSE tonirodon Politòleg. Assistant Professor. Universitat Pompeu Fabra (UPF) @politiquesUPF Research Fellow @LSEGovernment @elpatidescobert 0, 23, https://t.co/SeBSGFUwgJ, http://www.tonirodon.cat, tonirodon.cat Toni Rodon NA FALSE NA 95226101
990274510053150720 2018-04-28T17:00:16.000Z https://pbs.twimg.com/profile_images/1450518618110889989/NdYqz7ji_normal.jpg 344 FALSE sophysteria Disciplinary tourist specializing in intemperate comments, greatest achievements are in the garden... NULL Sophia Woodman NA FALSE NA 95226101
47583917 https://t.co/821wMuhOqv 2009-06-16T09:15:37.000Z https://pbs.twimg.com/profile_images/1137448537657085954/pQ7Pqhfy_normal.jpg 231655 TRUE cathynewman I present Channel 4 News and Times Radio. I love breaking exclusive stories and interviews. Instagram cathynewmanc4 Agent https://t.co/c8gRG5l2cV 0, 23, https://t.co/821wMuhOqv, http://on.fb.me/1yJVG7w, on.fb.me/1yJVG7w Cathy Newman London FALSE NA 95226101
321098699 https://t.co/Pd9gKQPd1j 2011-06-21T01:09:15.000Z https://pbs.twimg.com/profile_images/1055563488876249088/HwO_fnOk_normal.jpg 912 FALSE RyanKennedy7 Associate Professor of Political Science at the University of Houston, director of the Machine-Assisted Human Decision-making (MAHD) Lab. 0, 23, https://t.co/Pd9gKQPd1j, https://ryanpkennedy.weebly.com/, ryanpkennedy.weebly.com Ryan Kennedy Houston, TX FALSE NA 95226101
316738448 https://t.co/oHjLDbJNae 2011-06-13T22:08:02.000Z https://pbs.twimg.com/profile_images/527107106868981760/aDhyDyR9_normal.jpeg 6775 FALSE ylelkes associate prof, UPenn. political communication, pol psych, public opinion. retweet means we're best friends now. 0, 23, https://t.co/oHjLDbJNae, http://ylelkes.com, ylelkes.com Yphtach Lelkes Philadelphia, PA FALSE 1501722494708985858 95226101
1197994613665083394 https://t.co/mgp7qoedxi 2019-11-22T21:45:55.000Z https://pbs.twimg.com/profile_images/1448799306471837705/t9UtVUXv_normal.jpg 635032 TRUE EmmaRaducanu london|toronto|shenyang|bucharest insta:emmaraducanu 0, 23, https://t.co/mgp7qoedxi, https://www.instagram.com/emmaraducanu, instagram.com/emmaraducanu Emma Raducanu London, England FALSE NA 95226101
723667033532018689 https://t.co/jFth1KgOZs 2016-04-23T00:17:20.000Z https://pbs.twimg.com/profile_images/723960899120316416/YVpibxGx_normal.jpg 6421 FALSE JOSS_TheOJ The Journal of Open Source Software. The developer friendly software journal. Tweets by: @KMMoerman and Editorialbot 0, 23, https://t.co/jFth1KgOZs, http://joss.theoj.org/, joss.theoj.org JOSS NA FALSE NA 95226101
534266326 2012-03-23T12:52:41.000Z https://pbs.twimg.com/profile_images/1145065155249803264/l7PswQhy_normal.png 28209 FALSE metrics52 Cofounder @AvelaEducation Professor @MITEcon Director @BlueprintMIT Author https://t.co/d8dHi5zb5r NULL Josh Angrist NA FALSE NA 95226101
20309837 https://t.co/cdvcO13Jmm 2009-02-07T13:47:06.000Z https://pbs.twimg.com/profile_images/1463195870220926985/1hW1BWT9_normal.jpg 22160 FALSE mikarv Associate Professor in Digital Rights & Regulation @UCLLaws. Trying to make data rights reality. Own views, open DMs, RTs not endorsements. 0, 23, https://t.co/cdvcO13Jmm, http://michae.lv, michae.lv Michael Veale #UCUStrike London FALSE 1491468351314485259 95226101
179616555 2010-08-17T18:39:51.000Z https://pbs.twimg.com/profile_images/1202531971936579584/ZSTB4-yJ_normal.jpg 22498 FALSE 1an_Higgins 1 of @gradecricketer. Joke teller for me. Law Graduate for dad. Insta 📸: higgins_ian NULL Ian Higgins Melbourne, Australia FALSE 1317749089493135360 95226101
1213103147993853953 https://t.co/s2Ib00t755 2020-01-03T14:22:44.000Z https://pbs.twimg.com/profile_images/1369978318410383360/pN-FZQzZ_normal.jpg 564 FALSE aies_edinburgh AI Ethics and Society is an Edinburgh-based network of researchers investigating the ethical and social implications of data-driven technologies. 0, 23, https://t.co/s2Ib00t755, https://www.ai-ethics.org/, ai-ethics.org AIES Edinburgh Edinburgh, Scotland FALSE NA 95226101
2815077014 https://t.co/mkzg98j9Ju 2014-09-17T14:58:03.000Z https://pbs.twimg.com/profile_images/512256295542333440/8Jo4w8kV_normal.jpeg 89723 FALSE chrmanning Director, @StanfordAILab. Assoc. Director, @StanfordHAI. Founder, @stanfordnlp. Prof. Computer Science (CS) & Linguistics, @Stanford. 🇦🇺 Do #NLProc & #AI. 👋 0, 23, https://t.co/mkzg98j9Ju, https://nlp.stanford.edu/~manning/, nlp.stanford.edu/~manning/ Christopher Manning Palo Alto FALSE NA 95226101
2837102661 http://t.co/nLDxDlbppw 2014-10-19T12:23:24.000Z https://pbs.twimg.com/profile_images/523812099177185281/oXO-AnDV_normal.jpeg 627 FALSE Mourakiboun_org Association 0, 22, http://t.co/nLDxDlbppw, http://www.mourakiboun.org, mourakiboun.org Mourakiboun Rue 8300, Imm Luxor 2, App BM1 FALSE NA 95226101
857208506994307072 https://t.co/lBK5GVd3tJ 2017-04-26T12:23:29.000Z https://pbs.twimg.com/profile_images/1146394089392418816/6JmL5hON_normal.png 1017 FALSE thmskrr Research Group Leader @EXCInequality | labor markets, automation, occupational change, social hierarchies, political behavior 0, 23, https://t.co/lBK5GVd3tJ, http://thomaskurer.net, thomaskurer.net Thomas Kurer NA FALSE NA 95226101
70155097 2009-08-30T16:31:34.000Z https://pbs.twimg.com/profile_images/1988068134/D_mocratie2_normal.jpg 615930 FALSE HaythemElMekki Journaliste et Chroniqueur chez Radio Mosaïque FM, entre autres. Views are my own. NULL Haythem El Mekki Tunisia FALSE NA 95226101
88943180 https://t.co/ikfgxHag3T 2009-11-10T14:52:45.000Z https://pbs.twimg.com/profile_images/1315062635923214336/c6L7gWR6_normal.jpg 1362 FALSE chainsawriot Chung-hong Chan, PhD / Fellow, MZES, Universität Mannheim / Secretary @ica_cm /Formerly @jmschku / I don’t know my research interests. #rstats #emacs #rcpp 0, 23, https://t.co/ikfgxHag3T, http://chainsawriot.com, chainsawriot.com Chung-hong Chan Mannheim, Germany FALSE 1460936485264310276 95226101
479007283 https://t.co/U5tULK5ibS 2012-01-30T22:25:51.000Z https://pbs.twimg.com/profile_images/1459553456004476932/XSPf04_u_normal.jpg 636 FALSE abwiedemann Assistant Professor of Politics and Int'l Affairs @Princeton | Political Economy and Inequality | Author of "Indebted Societies" (CUP) https://t.co/ZcVIIzpPBx 0, 23, https://t.co/U5tULK5ibS, https://www.abwiedemann.com/, abwiedemann.com Andreas Wiedemann Princeton, NJ FALSE 1415753412352774146 95226101
46625755 https://t.co/CCi7BkD6Jq 2009-06-12T10:36:57.000Z https://pbs.twimg.com/profile_images/1337415098/amelia_normal.jpg 48668 TRUE ameliagentleman NOW IN PAPERBACK! The Windrush Betrayal https://t.co/cllrhl7NMb 0, 23, https://t.co/CCi7BkD6Jq, https://www.theguardian.com/profile/ameliagentleman, theguardian.com/profile/amelia… amelia gentleman London FALSE 1346047340642258947 95226101

User-level inference

Example from here

from m3inference import M3Inference
import pprint
m3 = M3Inference() # see docstring for details
pred = m3.infer('./test/data_resized.jsonl') # also see docstring for details
pprint.pprint(pred)
OrderedDict([('720389270335135745',
              {'age': {'19-29': 0.1546,
                       '30-39': 0.114,
                       '<=18': 0.0481,
                       '>=40': 0.6833},
               'gender': {'female': 0.0066, 'male': 0.9934},
               'org': {'is-org': 0.7508, 'non-org': 0.2492}}),
             ('21447363',
              {'age': {'19-29': 0.0157,
                       '30-39': 0.9837,
                       '<=18': 0.0004,
                       '>=40': 0.0002},
               'gender': {'female': 0.9866, 'male': 0.0134},
               'org': {'is-org': 0.0002, 'non-org': 0.9998}}),
    ...
  ...

User-level inference

devtools::install_github("pablobarbera/twitter_ideology/pkg/tweetscores")
library(tweetscores)

User-level inference

results <- estimateIdeology("cbarrie", ids)
plot(results)

User-level geolocation

userfwing %>%
  group_by(location) %>%
  summarise(count = n()) %>%
  top_n(count)
# A tibble: 12 × 2
   location            count
   <chr>               <int>
 1 Cambridge, MA          15
 2 Edinburgh              22
 3 Edinburgh, Scotland    39
 4 London                 60
 5 London, England        46
 6 New York, NY           24
 7 Oxford                 15
 8 Oxford, England        40
 9 San Francisco, CA      18
10 United Kingdom         13
11 Washington, DC         29
12 <NA>                  384