0. Logistical Info
- Section date: 11/8
- Associated lectures: 10/31, 11/2
- Associated pset: Pset 8, due 11/10
- Office hours on 11/8 from 9-11pm at Quincy Dining Hall
- Remember to fill out the attendance form
0.1 Summary + Practice Problem PDFs
Summary + Practice Problems PDF
Practice Problem Solutions PDF
1. Multinomial
We first generalize the notion of Bernoulli trials to many categories; this vocabulary for “categorical trials” is not standard/necessary for the class, just introduced by me to help define the Multinomial.
Consider categorical trials, where the outcome of a trial falls into one of $k$ categories (e.g., the roll of a die has $6$ categories, the flip of a coin has $2$, etc.). Let $\mathbf p \in \mathbb R^k$ be a probability vector (where each entry is in $[0, 1]$ and the entries add up to $p$), where $p_i$ is the probability that the outcome falls into the $i^{\text{th}}$ category. $\newcommand{\Mult}{\mathrm{Mult}}\newcommand{\cov}{\mathrm{Cov}}\newcommand{\Pois}{\mathrm{Pois}}\newcommand{\Bin}{\mathrm{Bin}}$
Multinomial story: Suppose we run $n$ independent and identically distributed (i.i.d.) categorical trials with $k$ categories and probability vector $\mathbf p$. Let $\mathbf X$ (a $k$-dimensional random vector) count the number of trials that fell into each category. Then $\mathbf X$ is distributed Multinomial: $\mathbf X \sim \Mult_k(n, \mathbf p)$.
1.1 Multinomial Properties
- Marginal: For $\mathbf X \sim \Mult_k(n, \mathbf p)$, $X_j \sim \Bin(n, p_j)$.
- Conditioning: For $\mathbf X \sim \Mult_k(n, \mathbf p)$, \begin{align*} (X_2, \ldots, X_n) | X_1 = x_1 &\sim \Mult_{k-1}(n-x_1, \left(\frac{p_2}{1 - p_1}, \ldots, \frac{p_n}{1-p_1} \right) ). \end{align*}
- Lumping: Suppose $\mathbf X \sim \Mult_k(n, \mathbf p)$. Then we can group (lump) categories in any way to get a new Multinomial random variable by adding up the associated probabilities. For example, if $(X_1, X_2, X_3, X_4, X_5) \sim \Mult_5\left( n, (p_1, p_2, p_3, p_4, p_5) \right)$, then some valid examples are \begin{align*} (X_1+X_4, X_2, X_3+X_5) &\sim \Mult_3\left(n, ( p_1+p_4, p_2, p_3+p_5 )\right),\\ (X_1+X_2, X_3, X_4, X_5) &\sim \Mult_4\left(n, ( p_1+p_2, p_3, p_4, p_5 )\right). \end{align*}
- Covariance: For $\mathbf X \sim \Mult_k(n, \mathbf p)$, $\cov(X_i, X_j) = -np_ip_j$.
- Chicken-Egg extension: Suppose $N \sim \Pois(\lambda)$ and $\mathbf X | N = n \sim \Mult_k (n, \mathbf p)$ where $k, \mathbf p$ don’t depend on $n$. Then for $j = 1, 2, \ldots, k$, \begin{align*} X_j &\sim \Pois(\lambda p_j). \end{align*}
2. Multivariate Normal
Suppose $\mathbf X$ is a $k$-dimensional random vector. Then $\mathbf X$ follows Multivariate Normal (MVN) distribution if for any constants $t_1, \ldots, t_k \in \mathbb R$, \begin{align*} t_1 X_1 + \cdots + t_k X_k \end{align*} is Normal (where $0$ is consider to follow a degenerate Normal distribution). The $k=2$ case is called the Bivariate Normal.
2.1 Multivariate Normal Properties
- Uncorrelated MVN implies independence: Suppose $(X, Y)$ is bivariate normal with $\cov(X, Y) = 0$ (i.e., $X$ and $Y$ are uncorrelated). Then $X$ and $Y$ are independent.
More generally, if $\mathbf X$ and $\mathbf Y$ (potentially vectors) are components of the same MVN and $X_i, Y_j$ are uncorrelated for any $i, j$, then $\mathbf X$ and $\mathbf Y$ are independent.
- Independence of sum and difference: Suppose $X \sim \mathcal N(\mu_1, \sigma^2)$ and $Y \sim \mathcal N(\mu_2, \sigma^2)$ are independent. Then $X+Y$ and $X-Y$ are also independent.
- Concatenation: Suppose $\mathbf X = (X_1, \ldots, X_n)$ and $\mathbf Y = (Y_1, \ldots Y_m)$ are both Multivariate Normal with $\mathbf X, \mathbf Y$ independent of each other. Then $(X_1, \ldots, X_n, Y_1, \ldots, Y_m)$ is also Multivariate Normal.
- Subvector: Suppose $(X, Y, Z)$ is Multivariate Normal. Then $(X, Y)$ is also Multivariate Normal. In general, any subvector of a Multivariate Normal still follows a Multivariate Normal distribution.