Week 8: Multinomial, Multivariate Normal

0. Logistical Info

• Section date: 11/8
• Associated lectures: 10/31, 11/2
• Associated pset: Pset 8, due 11/10
• Office hours on 11/8 from 9-11pm at Quincy Dining Hall
• Remember to fill out the attendance form

0.1 Summary + Practice Problem PDFs

Summary + Practice Problems PDF

Practice Problem Solutions PDF

1. Multinomial

We first generalize the notion of Bernoulli trials to many categories; this vocabulary for “categorical trials” is not standard/necessary for the class, just introduced by me to help define the Multinomial.

Consider categorical trials, where the outcome of a trial falls into one of $k$ categories (e.g., the roll of a die has $6$ categories, the flip of a coin has $2$, etc.). Let $\mathbf{p}\in {\mathbb{R}}^k$ be a probability vector (where each entry is in $[0,1]$ and the entries add up to $p$), where $p_i$ is the probability that the outcome falls into the $i^{\text{th}}$ category. $$

Multinomial story: Suppose we run $n$ independent and identically distributed (i.i.d.) categorical trials with $k$ categories and probability vector $\mathbf{p}$. Let $\mathbf{X}$ (a $k$-dimensional random vector) count the number of trials that fell into each category. Then $\mathbf{X}$ is distributed Multinomial: $\mathbf{X}\sim {\mathrm{Mult}}_k(n,\mathbf{p})$.

1.1 Multinomial Properties

• Marginal: For $\mathbf{X}\sim {\mathrm{Mult}}_k(n,\mathbf{p})$, $X_j\sim \mathrm{Bin}(n,p_j)$.
• Conditioning: For $\mathbf{X}\sim {\mathrm{Mult}}_k(n,\mathbf{p})$, $$\begin{array}{rl}(X_2,\dots ,X_n)|X_1=x_1& \sim {\mathrm{Mult}}_{k-1}(n-x_1,(\frac{p_2}{1-p_1},\dots ,\frac{p_n}{1-p_1})).\end{array}$$
• Lumping: Suppose $\mathbf{X}\sim {\mathrm{Mult}}_k(n,\mathbf{p})$. Then we can group (lump) categories in any way to get a new Multinomial random variable by adding up the associated probabilities. For example, if $(X_1,X_2,X_3,X_4,X_5)\sim {\mathrm{Mult}}_5(n,(p_1,p_2,p_3,p_4,p_5))$, then some valid examples are $$\begin{array}{rl}(X_1+X_4,X_2,X_3+X_5)& \sim {\mathrm{Mult}}_3(n,(p_1+p_4,p_2,p_3+p_5)),\\ (X_1+X_2,X_3,X_4,X_5)& \sim {\mathrm{Mult}}_4(n,(p_1+p_2,p_3,p_4,p_5)).\end{array}$$
• Covariance: For $\mathbf{X}\sim {\mathrm{Mult}}_k(n,\mathbf{p})$, $\mathrm{Cov}(X_i,X_j)=-n{p}_i{p}_j$.
• Chicken-Egg extension: Suppose $N\sim \mathrm{Pois}(\lambda )$ and $\mathbf{X}|N=n\sim {\mathrm{Mult}}_k(n,\mathbf{p})$ where $k,\mathbf{p}$ don’t depend on $n$. Then for $j=1,2,\dots ,k$, $$\begin{array}{rl}{X}_j& \sim \mathrm{Pois}(\lambda p_j).\end{array}$$

2. Multivariate Normal

Suppose $\mathbf{X}$ is a $k$-dimensional random vector. Then $\mathbf{X}$ follows Multivariate Normal (MVN) distribution if for any constants $t_1,\dots ,t_k\in \mathbb{R}$, $$\begin{array}{r}{t}_1{X}_1+\cdots +t_k{X}_k\end{array}$$ is Normal (where $0$ is consider to follow a degenerate Normal distribution). The $k=2$ case is called the Bivariate Normal.

2.1 Multivariate Normal Properties

• Uncorrelated MVN implies independence: Suppose $(X,Y)$ is bivariate normal with $\mathrm{Cov}(X,Y)=0$ (i.e., $X$ and $Y$ are uncorrelated). Then $X$ and $Y$ are independent.
  More generally, if $\mathbf{X}$ and $\mathbf{Y}$ (potentially vectors) are components of the same MVN and $X_i,Y_j$ are uncorrelated for any $i,j$, then $\mathbf{X}$ and $\mathbf{Y}$ are independent.

• Independence of sum and difference: Suppose $X\sim \mathcal{N}({\mu }_1,{\sigma }^2)$ and $Y\sim \mathcal{N}({\mu }_2,{\sigma }^2)$ are independent. Then $X+Y$ and $X-Y$ are also independent.
• Concatenation: Suppose $\mathbf{X}=(X_1,\dots ,X_n)$ and $\mathbf{Y}=(Y_1,\dots Y_m)$ are both Multivariate Normal with $\mathbf{X},\mathbf{Y}$ independent of each other. Then $(X_1,\dots ,X_n,Y_1,\dots ,Y_m)$ is also Multivariate Normal.
• Subvector: Suppose $(X,Y,Z)$ is Multivariate Normal. Then $(X,Y)$ is also Multivariate Normal. In general, any subvector of a Multivariate Normal still follows a Multivariate Normal distribution.