Data Science for Economists

class: center, middle, inverse, title-slide

# Data Science for Economists
## Lecture 3: Learning to love the shell
### Grant McDermott
### University of Oregon | <a href="https://github.com/uo-ec607">EC 607</a>

---

name: toc

# Table of contents

1. [Prologue](#prologue)

2. [Introduction](#intro)

3. [Bash shell basics](#basics)

4. [Files and directories](#files)

5. [Working with text files](#text)

6. [Redirecting and pipes](#pipes)

7. [Iteration (for loops)](#iteration)

8. [Scripting](#scripting)

9. [User roles and file permissions](#permissions)

10. [Next steps](#steps)

11. [Appendix (Windows users only)](#windows)

---
class: inverse, center, middle
name: prologue

# Prologue

---

# Student presentation: Git GUI

As I said in our previous lecture, I think the Git + RStudio (and/or the shell) workflow will be optimal for most of you.

But here are some nice standalone Git GUIs that you might want to consider:
- [Gitkraken](https://www.gitkraken.com/)
- [SourceTree](https://www.sourcetreeapp.com/)
- [GitHub Desktop](https://desktop.github.com/)
- [Atom](https://github.atom.io/)
- [VS Code](https://code.visualstudio.com/docs/introvideos/versioncontrol)

---

# Checklist

☑ Have you cloned the [course lecture repo](https://github.com/uo-ec607/lectures) to your local machine?

☑ Once that's done, pull to get the latest lecture slides.

☑ Do you have Bash-compatible shell? (**Windows** users: see [here](#windows) before continuing.)

</br>

I'm also going to recommend that you spruce up your GitHub profiles. 
- Add your full name, a profile picture, link to website, etc.
- No-one wants to work with (or hire) a barcode.

</br>

Today's lecture is the last detour before we get back to data analyis with R and RStudio.
- Laying proper foundations with Git and the shell will put us in a strong positon for advanced data science work as the course develops.

---
class: inverse, center, middle
name: intro

# Introduction
<html><div style='float:left'></div><hr color='#EB811B' size=1px width=796px></html>

---

# The Unix philosophy

The shell tools that we're going to be using today have their roots in the [Unix](https://en.wikipedia.org/wiki/Unix) family of operating systems originally developed at Bells Labs in the 1970s.

Besides paying homage, acknowledging the Unix lineage is important because these tools still embody the "[Unix philosophy](https://en.wikipedia.org/wiki/Unix_philosophy)":

> **Do One Thing And Do It Well**

By pairing and chaining well-designed individual components, we can build powerful and much more complex larger systems.
- You can see why the Unix philosophy is also referred to as "minimalist and modular".
  
Again, this philosophy is very clearly expressed in the design and functionality of the Unix shell.

---

# Definitions

Don't be thrown off by terminology: *shell*, *terminal*, *tty*, *command prompt*, etc. 
- These are all basically just different names for the same thing.<sup>1</sup>
- They are all referring to a **command line interface** (CLI).

.footnote[
<sup>1</sup> [Truth be told](https://unix.stackexchange.com/questions/4126/what-is-the-exact-difference-between-a-terminal-a-shell-a-tty-and-a-con), there are some subtle and sometimes important differences, as well as some interesting history behind the names. But we can safely ignore these here.
]

There are many shell variants, but we're going to focus on [**Bash**](https://www.gnu.org/software/bash/) (i.e. **B**ourne **a**gain **sh**ell).
- Included by default on Linux and MacOS.
- Windows users need to install a Bash-compatible shell first (again, see [here](#windows)).

</br>

(For the record, I primarily use [zsh](https://ohmyz.sh/) (i.e. the "Z shell") &mdash; which is now also the [default shell](https://support.apple.com/en-us/HT208050) on MacOS. As a result, my shell might look slightly different to yours during live coding sessions. The commands will stay the same, though.)

---

# Why bother with the shell?

1. Power
  - Both for executing commands and for fixing problems. There are some things you just can't do in an IDE or GUI.
  - It also avoids memory complications associated with certain applications and/or IDEs. We'll get to this issue later in the course.
2. Reproducibility
 - Scripting is reproducible, while clicking is not.
3. Interacting with servers and super computers
  - The shell is often the only game in town for high performance computing. We'll get to this later in the course.
4. Automating workflow and analysis pipelines
  - Easily track and reproduce an entire project (e.g. use a Makefile to combine multiple programs, scripts, etc.)

We're going to focus on 1, 2 and 3 in this course. That's not to say that 4 is unimportant (far from it!), but we just won't have time to cover it. 
- [Here](https://stat545.com/automation-overview.html), [here](https://books.ropensci.org/drake/index.html), and [here](https://web.stanford.edu/~gentzkow/research/CodeAndData.pdf) are great places to start learning about automation on your own.

---

# Things that I use the shell for

- Git
- Renaming and moving files *en masse*
- Finding things on my computer
- Combining and manipulating PDFs
- Installing and updating software
- Scheduling tasks
- Monitoring system resources
- Connecting to cloud environments
- Running analyses ("jobs") on super computers
- etc.

<i>
Personal aside: One of the (many) nice things about being a Linux user is that it demystifies the shell. You end up using the shell for various day-to-day operations and so inevitably become very comfortable using it. Every year I manage to convince at least one student to switch...
- I recommend [Endeavour OS](https://endeavouros.com/), though [Ubuntu](https://ubuntu.com/) and [Elementary](https://elementary.io/) are good places to start too.
</i>

---
class: inverse, center, middle
name: basics

# Bash shell basics
<html><div style='float:left'></div><hr color='#EB811B' size=1px width=796px></html>

---

# First look

Let's open up our Bash shell.

A convenient way to do this is through [RStudio's built-in Terminal](https://support.rstudio.com/hc/en-us/articles/115010737148-Using-the-RStudio-Terminal).
- Hitting `Shift`+`Alt`+`T` will cause a "Terminal" tab to open up in the bottom-left window pane (i.e. next to the "Console" tab).
- This should run Bash by default if it is installed on your system. (Windows users: Once again, see [here](#wsl-rstudio).)

</br>

P.S. Of course, it's always possible to open up the Bash shell directly if you prefer.
- [Linux](https://www.wikihow.com/Open-a-Terminal-Window-in-Ubuntu)
- [Mac](https://www.techwalla.com/articles/how-to-open-terminal-on-a-macbook)
- [Windows](https://www.howtogeek.com/265900/everything-you-can-do-with-windows-10s-new-bash-shell/)

---

# First look (cont.)

You should see something like:

```bash
 username@hostname:~$
```

This is shell-speak for: "Who am I and where am I?"

---
count: false

# First look (cont.)

You should see something like:

```bash
 `username`@hostname:~$ 
```

This is shell-speak for: "Who am I and where am I?"

- `username` denotes a specific user (one of potentially many on this computer).

---
count: false

# First look (cont.)

You should see something like:

```bash
 username`@hostname`:~$ 
```

This is shell-speak for: "Who am I and where am I?"

- `username` denotes a specific user (one of potentially many on this computer).

- `@hostname` denotes the name of the computer or server.

---
count: false

# First look (cont.)

You should see something like:

```bash
 username@hostname`:~`$ 
```

This is shell-speak for: "Who am I and where am I?"

- `username` denotes a specific user (one of potentially many on this computer).

- `@hostname` denotes the name of the computer or server.

- `:~` denotes the directory path (where `~` signifies the user's home directory).

---
count: false

# First look (cont.)

You should see something like:

```bash
 username@hostname:~`$`
```

This is shell-speak for: "Who am I and where am I?"

- `username` denotes a specific user (one of potentially many on this computer).

- `@hostname` denotes the name of the computer or server.

- `:~` denotes the directory path (where `~` signifies the user's home directory).

- `$` denotes the start of the command prompt.
  - We'll get to this later, but for a special "superuser" called root, the dollar sign will change to a `#`.

---

# Useful keyboard shortcuts

- `Tab` completion.

- Use the `↑` (and `↓`) keys to scroll through previous commands.

- `Ctrl`+`→` (and `Ctrl`+`←`) to skip whole words at a time.

- `Ctrl`+`a` moves the cursor to the beginning of the line.

- `Ctrl`+`e` moves the cursor to the end of the line.

- `Ctrl`+`k` deletes everything to the right of the cursor.

- `Ctrl`+`u` deletes everything to the left of the cursor.

- `Ctrl`+`Shift`+`c` to copy and `Ctrl`+`Shift`+`v` to paste.

- `clear` to clear your terminal.

---

# Syntax

All Bash commands have the same basic syntax:

**<center>command option(s) argument(s)</center>**

Examples:

```bash
  $ ls -lh ~/Documents/
  ```
  ```bash
  $ sort -u myfile.txt
  ```

---
count: false

# Syntax

All Bash commands have the same basic syntax:

**<center><span style='background: #ffff88;'>command</span> option(s) argument(s)</center>**

Examples:

```bash
  $ `ls` -lh ~/Documents/
  ```
  ```bash
  $ `sort` -u myfile.txt
  ```

---
count: false

# Syntax

All Bash commands have the same basic syntax:

**<center>command <span style='background: #ffff88;'>option(s)</span> argument(s)</center>**

Examples:

```bash
  $ ls `-lh` ~/Documents/
  ```
  ```bash
  $ sort `-u` myfile.txt
  ```

---
count: false

# Syntax

All Bash commands have the same basic syntax:

**<center>command option(s) <span style='background: #ffff88;'>argument(s)</span></center>**

Examples:

```bash
  $ ls -lh `~/Documents/`
  ```
  ```bash
  $ sort -u `myfile.txt`
  ```

---

# Syntax

All Bash commands have the same basic syntax:

**<center>command option(s) argument(s)</center>**

Examples:

```bash
  $ ls -lh ~/Documents/
  ```
  ```bash
  $ sort -u myfile.txt
  ```

</br>

**commands**
- You don't always need options or arguments. (E.g. `$ ls ~/Documents/` and `$ ls -lh` are both valid commands that will yield output.)
- However, you always need a command.

---

# Syntax (cont.)

**options** (also called **flags**)
- Start with a dash.
- Usually one letter.
- Multiple options can be chained together under a single dash.
  ```bash
  $ ls -l -a -h /var/log ## This works
  $ ls -lah /var/log ## So does this
  ```
- An exception is with (rarer) options requiring two dashes.
  ```bash
  $ ls --group-directories-first --human-readable /var/log
  ```

**arguments**
- Tell the command *what* to operate on.
- Usually a file, path, or a set of files and folders.

---

# Help: man

The `man` command ("manual pages") is your friend if you ever need help.
- Tip: Hit spacebar to scroll down a page at a time, "h" to see the help notes of the `man` command itself and "q" to quit.

```bash
$ man ls
```

```
## LS(1)                            User Commands                           LS(1)
## 
## NAME
##        ls - list directory contents
## 
## SYNOPSIS
##        ls [OPTION]... [FILE]...
## 
## DESCRIPTION
##        List  information  about  the FILEs (the current directory by default).
##        Sort entries alphabetically if none of -cftuvSUX nor --sort  is  speci‐
##        fied.
## 
##        Mandatory  arguments  to  long  options are mandatory for short options
##        too.
## 
##        -a, --all
##               do not ignore entries starting with .
## 
##        -A, --almost-all
##               do not list implied . and ..
## 
##        --author
##               with -l, print the author of each file
## 
##        -b, --escape
##               print C-style escapes for nongraphic characters
## 
##        --block-size=SIZE
##               with  -l,  scale  sizes  by  SIZE  when  printing  them;   e.g.,
##               '--block-size=M'; see SIZE format below
## 
##        -B, --ignore-backups
##               do not list implied entries ending with ~
## 
##        -c     with -lt: sort by, and show, ctime (time of last modification of
##               file status information); with -l: show ctime and sort by  name;
##               otherwise: sort by ctime, newest first
## 
##        -C     list entries by columns
## 
##        --color[=WHEN]
##               colorize  the output; WHEN can be 'always' (default if omitted),
##               'auto', or 'never'; more info below
## 
##        -d, --directory
##               list directories themselves, not their contents
## 
##        -D, --dired
##               generate output designed for Emacs' dired mode
## 
##        -f     do not sort, enable -aU, disable -ls --color
## 
##        -F, --classify
##               append indicator (one of */=>@|) to entries
## 
##        --file-type
##               likewise, except do not append '*'
## 
##        --format=WORD
##               across -x, commas -m, horizontal -x, long -l, single-column  -1,
##               verbose -l, vertical -C
## 
##        --full-time
##               like -l --time-style=full-iso
## 
##        -g     like -l, but do not list owner
## 
##        --group-directories-first
##               group directories before files;
## 
##               can   be  augmented  with  a  --sort  option,  but  any  use  of
##               --sort=none (-U) disables grouping
## 
##        -G, --no-group
##               in a long listing, don't print group names
## 
##        -h, --human-readable
##               with -l and -s, print sizes like 1K 234M 2G etc.
## 
##        --si   likewise, but use powers of 1000 not 1024
## 
##        -H, --dereference-command-line
##               follow symbolic links listed on the command line
## 
##        --dereference-command-line-symlink-to-dir
##               follow each command line symbolic link
## 
##               that points to a directory
## 
##        --hide=PATTERN
##               do not list implied entries matching shell  PATTERN  (overridden
##               by -a or -A)
## 
##        --hyperlink[=WHEN]
##               hyperlink file names; WHEN can be 'always' (default if omitted),
##               'auto', or 'never'
## 
##        --indicator-style=WORD
##               append indicator with style WORD to entry names: none (default),
##               slash (-p), file-type (--file-type), classify (-F)
## 
##        -i, --inode
##               print the index number of each file
## 
##        -I, --ignore=PATTERN
##               do not list implied entries matching shell PATTERN
## 
##        -k, --kibibytes
##               default  to  1024-byte  blocks for disk usage; used only with -s
##               and per directory totals
## 
##        -l     use a long listing format
## 
##        -L, --dereference
##               when showing file information for a symbolic link, show informa‐
##               tion  for  the file the link references rather than for the link
##               itself
## 
##        -m     fill width with a comma separated list of entries
## 
##        -n, --numeric-uid-gid
##               like -l, but list numeric user and group IDs
## 
##        -N, --literal
##               print entry names without quoting
## 
##        -o     like -l, but do not list group information
## 
##        -p, --indicator-style=slash
##               append / indicator to directories
## 
##        -q, --hide-control-chars
##               print ? instead of nongraphic characters
## 
##        --show-control-chars
##               show nongraphic characters as-is (the default, unless program is
##               'ls' and output is a terminal)
## 
##        -Q, --quote-name
##               enclose entry names in double quotes
## 
##        --quoting-style=WORD
##               use  quoting style WORD for entry names: literal, locale, shell,
##               shell-always,  shell-escape,  shell-escape-always,   c,   escape
##               (overrides QUOTING_STYLE environment variable)
## 
##        -r, --reverse
##               reverse order while sorting
## 
##        -R, --recursive
##               list subdirectories recursively
## 
##        -s, --size
##               print the allocated size of each file, in blocks
## 
##        -S     sort by file size, largest first
## 
##        --sort=WORD
##               sort  by  WORD instead of name: none (-U), size (-S), time (-t),
##               version (-v), extension (-X)
## 
##        --time=WORD
##               change the default of  using  modification  times;  access  time
##               (-u): atime, access, use; change time (-c): ctime, status; birth
##               time: birth, creation;
## 
##               with -l, WORD determines which time to show;  with  --sort=time,
##               sort by WORD (newest first)
## 
##        --time-style=TIME_STYLE
##               time/date format with -l; see TIME_STYLE below
## 
##        -t     sort by time, newest first; see --time
## 
##        -T, --tabsize=COLS
##               assume tab stops at each COLS instead of 8
## 
##        -u     with  -lt:  sort by, and show, access time; with -l: show access
##               time and sort by name; otherwise: sort by  access  time,  newest
##               first
## 
##        -U     do not sort; list entries in directory order
## 
##        -v     natural sort of (version) numbers within text
## 
##        -w, --width=COLS
##               set output width to COLS.  0 means no limit
## 
##        -x     list entries by lines instead of by columns
## 
##        -X     sort alphabetically by entry extension
## 
##        -Z, --context
##               print any security context of each file
## 
##        -1     list one file per line.  Avoid '\n' with -q or -b
## 
##        --help display this help and exit
## 
##        --version
##               output version information and exit
## 
##        The  SIZE  argument  is  an  integer and optional unit (example: 10K is
##        10*1024).  Units are K,M,G,T,P,E,Z,Y  (powers  of  1024)  or  KB,MB,...
##        (powers  of 1000).  Binary prefixes can be used, too: KiB=K, MiB=M, and
##        so on.
## 
##        The TIME_STYLE argument can be  full-iso,  long-iso,  iso,  locale,  or
##        +FORMAT.   FORMAT  is  interpreted  like in date(1).  If FORMAT is FOR‐
##        MAT1<newline>FORMAT2, then FORMAT1 applies to non-recent files and FOR‐
##        MAT2  to  recent files.  TIME_STYLE prefixed with 'posix-' takes effect
##        only outside the POSIX locale.  Also the TIME_STYLE  environment  vari‐
##        able sets the default style to use.
## 
##        Using  color  to distinguish file types is disabled both by default and
##        with --color=never.  With --color=auto, ls emits color codes only  when
##        standard  output is connected to a terminal.  The LS_COLORS environment
##        variable can change the settings.  Use the dircolors command to set it.
## 
##    Exit status:
##        0      if OK,
## 
##        1      if minor problems (e.g., cannot access subdirectory),
## 
##        2      if serious trouble (e.g., cannot access command-line argument).
## 
## AUTHOR
##        Written by Richard M. Stallman and David MacKenzie.
## 
## REPORTING BUGS
##        GNU coreutils online help: <https://www.gnu.org/software/coreutils/>
##        Report any translation bugs to <https://translationproject.org/team/>
## 
## COPYRIGHT
##        Copyright © 2020 Free Software Foundation, Inc.   License  GPLv3+:  GNU
##        GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
##        This  is  free  software:  you  are free to change and redistribute it.
##        There is NO WARRANTY, to the extent permitted by law.
## 
## SEE ALSO
##        Full documentation <https://www.gnu.org/software/coreutils/ls>
##        or available locally via: info '(coreutils) ls invocation'
## 
## GNU coreutils 8.32                March 2020                             LS(1)
```

---

# Help: man (cont.)

A useful feature of `man` is quick pattern searching with "/pattern".
- Try this now by running `$ man ls` again and then typing "/human" and hitting the return key. 
- To continue on to the next case, hit `n`. (It won't work here because the pattern "human" only occurs once in the manual file.)

</br>

Again, this and other `man` tricks tricks are detailed in the help pages (hit "h").

---

# Help: cheat

I also like the [cheat](https://github.com/chrisallenlane/cheat) utility, which provides a more readable summary / cheatsheet of various command. You'll need to install it first. (Linux and MacOS only.)

```
$ cheat ls

## # Displays everything in the target directory
## ls path/to/the/target/directory
## 
## # Displays everything including hidden files
## ls -a
## 
## # Displays all files, along with the size (with unit suffixes) and timestamp
## ls -lh 
## 
## # Display files, sorted by size
## ls -S
## 
## # Display directories only
## ls -d */
## 
## # Display directories only, include hidden
## ls -d .*/ */
```

---
class: inverse, center, middle
name: files

# Files and directories
<html><div style='float:left'></div><hr color='#EB811B' size=1px width=796px></html>

---

# Navigation

Key navigation commands:

- `pwd` to print (the current) working directory.

- `cd` to change directory.

```bash
$ pwd
```

```
## /home/grant/Documents/Teaching/EC607/lectures/03-shell
```

You can use absolute paths, but it's better to use relative paths and invoke special symbols for a user's home folder (`~`), current directory (`.`), and parent directory (`..`) as needed.

```bash
$ cd examples ## Move into the "examples" sub-directory of this lecture directory.
$ cd ../.. ## Now go back up two directories.
$ pwd
```

```
## /home/grant/Documents/Teaching/EC607/lectures
```

---

# Navigation (cont.)

Beware of directory names that contain spaces. Say you have a directory called "My Documents". (I'm looking at you, Windows.)

- Why won't `$ cd My Documents` work?

**Answer:** Bash syntax is super pedantic about spaces and ordering. Here it thinks that "My" and "Documents" are separate arguments.

- Small brain: Use quotation marks: `$ cd "My Documents"`.

- Big brain: Use Tab completion to automatically "escape" the space: `$ cd My\ Documents`. 
  
  - Galaxy brain: Don't use spaces in file and folder names.

---

# Navigation (cont.)

---

# Listing files and their properties

We're about to go into more depth about the `ls` command.
- To do this effectively, it will be helpful if we're all working off the same group of files and folders.
- Navigate to the directory containing these lecture notes (i.e. `03-shell`). Now list the contents of the  `examples/` sub-directory with the `-lh` option ("long format", "human readable").

```bash
$ # cd PathWhereYouClonedThisRepo/lectures/03-shell ## change as needed
$ ls -lh examples
```

```
## total 148K
## drwxr-xr-x 2 grant users 4.0K May 25 20:30 ABC
## drwxr-xr-x 2 grant users 4.0K Sep 10  2020 Bad folder name
## -rw-r--r-- 1 grant users  149 Feb 24 12:48 hello.R
## -rwxr-xr-x 1 grant users   37 Jan 14  2019 hello.sh
## drwxr-xr-x 2 grant users 4.0K May 25 20:30 meals
## -rw-r--r-- 1 grant users   32 May 25 20:30 nursery.txt
## -rwxrwxrwx 1 grant users  153 Jan 12  2019 reps.txt
## -rw-r--r-- 1 grant users 120K Jan 12  2019 sonnets.txt
```

---

# Listing files and their properties (cont.)