Statistical Inference
Estimation & Sampling
What is a random variable?
A random variable is the outcome of an experiment.
Examples:
- Roll a die → outcome ∈ {1, 2, 3, 4, 5, 6}
- Flip a coin → outcome ∈ {H, T}
- Pick a random person and measure their height → outcome ∈ ℝ⁺
Each random variable has:
- Outcomes (the “support”) — the universe of possible values
- Probabilities — how likely each outcome is
Deterministic vs. stochastic
- Deterministic: same input always gives the same output
- Stochastic: something we don’t (or can’t) measure causes the outcome to vary
Examples
Deterministic
- Position of a planet
- Weight of a particular quarter
- Height of Mount Everest (?)
Stochastic
- Baseball: hit or not?
- Weight of a randomly sampled quarter
- Subway trip time
The boundary is blurry — a coin flip is deterministic in principle (physics), but stochastic in practice (we can’t control the toss precisely enough).
Population vs. sample
- Population: every object in a well-defined set
- All N. Americans, all internet users, all fair dice, all US adults aged 18–65…
- Sample: a subset that we actually observe
- We never get to see the whole population
The magic of statistics: you can say something precise about the population even though you don’t get to see it all.
Sampling matters
Population: height of everyone in US aged 18–65
Some samples:
- Heights of the three instructors in this room
- Heights of everyone in this room
- All Microsoft employees
- Self-reported heights on OkCupid (people lie by ~1 inch!)
How you sample determines whether you can say anything meaningful.
Rolling dice: Theory
Outcomes: \[\{1, 2, 3, 4, 5, 6\}\]
Probabilities (fair die): \[\left\{\frac{1}{6}, \frac{1}{6}, \frac{1}{6}, \frac{1}{6}, \frac{1}{6}, \frac{1}{6}\right\}\]
This is “Theory land” — we know the full population.
In most real problems, we don’t get to see this!
A convenient toy problem: the die is simple enough that we can actually compute the truth, then see how well our estimates work.
Expected value / population mean
The expected value is the long-run average you’d get if you repeated the experiment forever.
\[\mu = \mathbb{E}[X] = \sum_{i \in \text{outcomes}} x_i \, p(x_i)\]
For our fair die:
\[\mu = 1 \cdot \frac{1}{6} + 2 \cdot \frac{1}{6} + 3 \cdot \frac{1}{6} + 4 \cdot \frac{1}{6} + 5 \cdot \frac{1}{6} + 6 \cdot \frac{1}{6}\]
\[= \frac{1}{6}(1 + 2 + 3 + 4 + 5 + 6) = \frac{21}{6} = 3.5\]
When you see a Greek letter (μ, σ) in statistics, it means the true, unknowable population quantity.
Rolling dice: Reality
In practice, we don’t know the probabilities. We just get a sample:
\[x_1 = 4, \quad x_2 = 3, \quad x_3 = 6, \quad \ldots, \quad x_n = 1\]
Theory (unknown)
- outcomes: {1, …, 6}
- probs: {1/6, …, 1/6}
- μ = 3.5
Reality (what we see)
- sample: \(x_1, x_2, \ldots, x_n\)
- no probabilities given
- μ = ???
Goal: estimate the true unknown mean μ from our sample.
Estimators
An intuitive estimator: the empirical average (sample mean)
\[\bar{x}_N = \frac{1}{N} \sum_{i=1}^{N} x_i\]
This is called a statistic — any function of the data we use to estimate something.
But we chose this! We could have used:
- \(\frac{x_{\max} + x_{\min}}{2}\) (midrange)
- The mode (most frequent value)
- Something else entirely
Why is the sample mean good? And how do we even know if an estimator is “good” without seeing the truth? 🤔
Variance
How much do outcomes vary around the mean?
\[\sigma^2 = \sum_{i \in \text{outcomes}} (x_i - \mu)^2 \, p(x_i)\]
For our fair die (μ = 3.5):
\[\sigma^2 = \frac{1}{6}\left\{(1-3.5)^2 + (2-3.5)^2 + \cdots + (6-3.5)^2\right\}\]
Why square? If we just summed \((x_i - \mu)\), positives and negatives would cancel out. Squaring ensures all deviations count.
Standard deviation
\[\sigma = \sqrt{\text{Var}(X)} = \sqrt{\sum_{i \in \text{outcomes}} (x_i - \mu)^2 \, p(x_i)}\]
Why both?
| Units |
height² (weird!) |
height (natural) |
| Math |
Easier to work with |
Carrying √ is annoying |
| Interpretation |
Less intuitive |
“How much things typically vary” |
We don’t know σ either — we also need to estimate the variance.
How good is an estimator?
Thought experiment (frequentist thinking = “imagine doing it many times”):
Use \(\bar{x}_{100}\) to estimate μ. Sample 100 random people, compute their average height.
Now do it again. And again. And again…
| #1 |
{64, 57, 66.1, …} |
67.3 |
| #2 |
{53, 71, …, 80} |
72.9 |
| #3 |
{68, 62, …, 70} |
54.7 |
| #4 |
{59, 74, …, 65} |
55.1 |
| ⋮ |
|
⋮ |
Each sample gives a different estimate. What does the distribution of these estimates look like?
The sampling distribution
Plot the estimates (not the original data):
\[\bar{x}_{100}^{(1)}, \; \bar{x}_{100}^{(2)}, \; \bar{x}_{100}^{(3)}, \; \ldots\]
They form a bell curve centered around μ — this is the sampling distribution.
“I definitely didn’t understand this thing when I finished my stats class, but this is the key thing in all of stats.”
Two questions about an estimator
1. Is it unbiased?
Is the sampling distribution centered around μ?
. . .
You can prove that \(\mathbb{E}[\bar{x}_N] = \mu\) — without knowing μ!
. . .
2. How much does it vary?
How tight is the sampling distribution?
→ This leads us to the standard error.
Standard error vs. standard deviation
Standard deviation (σ)
How much heights vary in the population.
A fact of nature. Fixed.
“Heights vary ±6 inches in the world”
Standard error (\(\hat{\sigma}_{\bar{x}_N}\))
How much the estimator varies across repeated samples.
Depends on the estimator AND sample size.
More samples → smaller SE → tighter estimate
These sound similar but are very different things.
- σ is about the population
- SE is about your procedure for estimating
What’s next
- We’ll simulate all of this in R
- Sample from populations, compute estimates, build sampling distributions
- “It’s just a bunch of for loops”
- Then: confidence intervals — putting error bars on our estimates
- Then: hypothesis testing — “is this effect real or just noise?”
The people who invented statistics in the 1920s couldn’t do simulation — they had to derive everything by hand. We get to use computers. 🎉