UA MATH564 概率论公式与定理总结

Part 1 Elementary Probability

Tip 1: Probability is a measure satisfying monotonicity and σ\sigmaσ-countably additivity
Tip 2: Boole’s Inequality
P(⋃i=1nAi)≤∑i=1nP(Ai),n∈NP(\bigcup_{i=1}^{n} A_i) \le \sum_{i=1}^{n} P(A_i),n \in \mathbb{N}P(i=1⋃nAi)≤i=1∑nP(Ai),n∈N

Tip 3: Poincare’s Formula
Sm=∑1≤i1<i2<⋯<im≤nP(Ai1∩Ai2∩⋯∩Aim)S~m=∑1≤i1<i2<⋯<im≤nP(Ai1∪Ai2∪⋯∪Aim)P(⋃i=1nAi)=∑m=1n(−1)m+1Sm,P(⋂i=1nAi)=∑m=1n(−1)m+1S~mS_m = \sum_{1 \le i_1 < i_2 < \cdots < i_m \le n} P(A_{i_1} \cap A_{i_2} \cap \cdots \cap A_{i_m}) \\ \tilde{S}_m = \sum_{1 \le i_1 < i_2 < \cdots < i_m \le n}P(A_{i_1} \cup A_{i_2} \cup \cdots \cup A_{i_m}) \\ P(\bigcup_{i=1}^{n} A_i) = \sum_{m=1}^n (-1)^{m+1}S_m,\ \ P(\bigcap_{i=1}^{n} A_i) = \sum_{m=1}^n (-1)^{m+1}\tilde{S}_mSm=1≤i1<i2<⋯<im≤n∑P(Ai1∩Ai2∩⋯∩Aim)S~m=1≤i1<i2<⋯<im≤n∑P(Ai1∪Ai2∪⋯∪Aim)P(i=1⋃nAi)=m=1∑n(−1)m+1Sm, P(i=1⋂nAi)=m=1∑n(−1)m+1S~m

Tip 4: Inclusive-exclusive Formula (set form and identical function form)
#⋃i=1nAi=∑i=1n#Ai+(−1)m+1∑1≤i1<⋯<im≤n#⋂j=1mAij+(−1)n+1#⋂i=1nAiI⋃i=1nAi=1−∏i=1n(1−Xi)=∑i=1nXi−∑1≤i1<i2≤nXi1Xi2+⋯+(−1)m+1∑1≤i1<⋯<im≤nXi1⋯Xim⋯+(−1)nX1⋯Xn,Xi=IAi\# \bigcup_{i=1}^n A_i = \sum_{i=1}^n \#A_i +(-1)^{m+1} \sum_{1 \le i_1 < \cdots < i_m \le n}\# \bigcap_{j=1}^m A_{i_j} + (-1)^{n+1}\#\bigcap_{i=1}^n A_{i} \\I_{\bigcup_{i=1}^n A_i} =1 - \prod_{i=1}^n (1-X_i) = \sum_{i=1}^n X_i - \sum_{1 \le i_1 < i_2 \le n} X_{i_1}X_{i_2} + \cdots \\+ (-1)^{m+1}\sum_{1 \le i_1 < \cdots < i_m \le n} X_{i_1}\cdots X_{i_m} \cdots +(-1)^n X_1\cdots X_n, X_i = I_{A_i}#i=1⋃nAi=i=1∑n#Ai+(−1)m+11≤i1<⋯<im≤n∑#j=1⋂mAij+(−1)n+1#i=1⋂nAiI⋃i=1nAi=1−i=1∏n(1−Xi)=i=1∑nXi−1≤i1<i2≤n∑Xi1Xi2+⋯+(−1)m+11≤i1<⋯<im≤n∑Xi1⋯Xim⋯+(−1)nX1⋯Xn,Xi=IAi

Tip 5: The Law of Total Probability
P(A)=∑i=1nP(A∣Ci)P(Ci)P(A) = \sum_{i=1}^n P(A|C_i)P(C_i)P(A)=i=1∑nP(A∣Ci)P(Ci)

Tip 6: Bayesian Formula
P(Ci∣A)=P(A,Ci)P(A)=P(A∣Ci)P(Ci)∑i=1nP(A∣Ci)P(Ci)P(C_i|A) = \frac{P(A,C_i)}{P(A)}= \frac{P(A|C_i)P(C_i)}{\sum_{i=1}^n P(A|C_i)P(C_i)}P(Ci∣A)=P(A)P(A,Ci)=∑i=1nP(A∣Ci)P(Ci)P(A∣Ci)P(Ci)

Tip 7: Binomial Theorem, Negative Binomial Theorem and Multinomial Theorem
(x+y)n=∑k=0nCnkxn−kyk=∑k=0∞Cnkxn−kyk(x+y)−r=1xr(1+yx)−r=∑k=0∞C−rkx−r−kyk=∑k=0∞(−1)kCr+k−1kx−r−kyk(x1+x2+...+xr)n=∑n1+n2+...+nr=nn!n1!n2!...nr!x1n1x2n2...xrnr(x+y)^n = \sum_{k=0}^{n} C_n^k x^{n-k}y^k = \sum_{k=0}^{\infty} C_n^k x^{n-k}y^k \\ (x+y)^{-r} = \frac{1}{x^r}(1+\frac{y}{x})^{-r} = \sum_{k=0}^{\infty} C_{-r}^k x^{-r-k}y^k = \sum_{k=0}^{\infty}(-1)^kC_{r+k-1}^{k} x^{-r-k}y^k \\ (x_1+x_2+...+x_r)^n = \sum_{n_1+n_2 + ... +n_r=n} \frac{n!}{n_1!n_2!...n_r!} x_1^{n_1}x_2^{n_2}...x_r^{n_r} (x+y)n=k=0∑nCnkxn−kyk=k=0∑∞Cnkxn−kyk(x+y)−r=xr1(1+xy)−r=k=0∑∞C−rkx−r−kyk=k=0∑∞(−1)kCr+k−1kx−r−kyk(x1+x2+...+xr)n=n1+n2+...+nr=n∑n1!n2!...nr!n!x1n1x2n2...xrnr

Tip 8: Stirling Number of the Second Kind (number of methods to put nnn objects into kkk urn, and every urn should not be empty)
S2(n,n−1)=Cn2S2(n+1,k)=kS2(n,k)+S2(n,k−1)xn=∑k=1nS2(n,k)(x)kΔm0n=m!S2(n,m),n≥mS_2(n,n-1) = C_n^2 \\ S_2(n+1,k) = kS_2(n,k) + S_2(n,k-1) \\ x^n = \sum_{k=1}^n S_2(n,k) (x)_k \\ \Delta^m 0^n = m! S_2(n,m),n \ge m S2(n,n−1)=Cn2S2(n+1,k)=kS2(n,k)+S2(n,k−1)xn=k=1∑nS2(n,k)(x)kΔm0n=m!S2(n,m),n≥m

Tip 9: Stirling’s Formula
Γ(n+1)=n!≈2πn(ne)n\Gamma(n+1) = n! \approx \sqrt{2\pi n} \left( \frac{n}{e} \right)^nΓ(n+1)=n!≈2πn(en)n

Part 2 Transformation of Random Variable
Method 1: By definition P(Y=g(X)≤y)=P(X≤g−1(y))P(Y = g(X) \le y) = P(X \le g^{-1}(y))P(Y=g(X)≤y)=P(X≤g−1(y)). Applicable when X,Y∈RX,Y \in \mathbb{R}X,Y∈R

Method 2: By Jacobian Determinant fY(y)=fX(h(y))∣Jh(y)∣f_Y(y) = f_X(h(y)) |Jh(y)|fY(y)=fX(h(y))∣Jh(y)∣. Applicable when X,Y∈RnX,Y \in \mathbb{R}^nX,Y∈Rn. Note that Y=g(X),X=h(Y)Y = g(X), \ X = h(Y)Y=g(X), X=h(Y) and JJJ indicates Jacobian determinant. Generally n=2,3n=2,3n=2,3 for hand calculation.

Method 3: In mathematical statistics, sometimes we also use additivity or MGF to find distribution.

Part 3 Moment Generating Function and Prob Inequality
Tip 1: Moment Generating Function and Moment
MX(t)=EetX,MX(n)(0)=EXnM_X(t) = Ee^{tX},\ M^{(n)}_X(0) = EX^nMX(t)=EetX, MX(n)(0)=EXn

Tip 2: Markov Inequality
P(X>x)≤E[g(X)]g(x)P(X >x ) \le \frac{E[g(X)]}{g(x)}P(X>x)≤g(x)E[g(X)]

Tip 3: Chebyshev Inequality
P(∣Y−EY∣>x)≤VarYx2,∀x>0P(|Y-EY|>x) \le \frac{VarY}{x^2},\forall x>0P(∣Y−EY∣>x)≤x2VarY,∀x>0

Tip 4: Chernoff Bound
ln⁡P(X>x)≤inf⁡t∈R{KX(t)−tx},KX(t)=ln⁡MX(t)\ln P(X > x) \le \inf_{t \in \mathbb{R}} \{K_X(t)-tx\},\ K_X(t) = \ln M_X(t)lnP(X>x)≤t∈Rinf{KX(t)−tx}, KX(t)=lnMX(t)

Part 4 Conditional Density and Conditional Expectation
Tip 1: Conditional Density
fX∣Y(x∣y)=f(x,y)f(y)f_{X|Y}(x|y) = \frac{f(x,y)}{f(y)}fX∣Y(x∣y)=f(y)f(x,y)

Tip 2: Law of Total Expectation
E[Y]=E[E[Y∣X]]E[Y] = E[E[Y|X]]E[Y]=E[E[Y∣X]]

Tip 3: Law of Total Variance
Var(Y)=Var(E[Y∣X])+E[Var(Y∣X)]Var(Y) = Var(E[Y|X]) + E[Var(Y|X)]Var(Y)=Var(E[Y∣X])+E[Var(Y∣X)]

Tip 4: Variance of Sum of Two Random Variables
Var(X±Y)=Var(X)+Var(Y)±2Cov(X,Y)Var(X\pm Y) = Var(X) + Var(Y) \pm 2Cov(X,Y)Var(X±Y)=Var(X)+Var(Y)±2Cov(X,Y)

Tip 5: Law of Total Covariance
Cov(Y1,Y2)=E[Cov(Y1,Y2∣X)]+Cov[E(Y1∣X),E(Y2∣X)]Cov(Y_1,Y_2) = E[Cov(Y_1,Y_2|X)] + Cov[E(Y_1|X),E(Y_2|X)]Cov(Y1,Y2)=E[Cov(Y1,Y2∣X)]+Cov[E(Y1∣X),E(Y2∣X)]

Tip 6: Linear Conbination of Random Variables
Var(∑i=1maiXi)=∑i,j=1maiajCov(Xi,Xj)Cov(∑i=1maiXi,∑j=1nbjYj)=∑i=1m∑j=1naibjCov(Xi,Yj)Var \left( \sum_{i=1}^m a_iX_i \right) = \sum_{i,j = 1}^m a_ia_jCov(X_i,X_j) \\ Cov \left( \sum_{i=1}^m a_iX_i,\sum_{j=1}^n b_jY_j \right) = \sum_{i=1}^m \sum_{j=1}^n a_ib_jCov(X_i,Y_j)Var(i=1∑maiXi)=i,j=1∑maiajCov(Xi,Xj)Cov(i=1∑maiXi,j=1∑nbjYj)=i=1∑mj=1∑naibjCov(Xi,Yj)

Part 5 Distribution of Order Statistics
Theorem 1: Distribution of one order statistic
FX(j)=∑k=jnCnk[F(x)]k[1−F(x)]n−kF_{X_{(j)}} = \sum_{k=j}^n C_n^k [F(x)]^k[1-F(x)]^{n-k}FX(j)=k=j∑nCnk[F(x)]k[1−F(x)]n−k

Theorem 2: Density of one order statistic
fX(j)(x)=jCnj[F(x)]j−1[1−F(x)]n−jf(x)f_{X_{(j)}}(x) = jC_n^j [F(x)]^{j-1}[1-F(x)]^{n-j}f(x)fX(j)(x)=jCnj[F(x)]j−1[1−F(x)]n−jf(x)

Theorem 3: Density of one order statistics, for j>ij>ij>i,
fX(i),X(j)(xi,xj)=n!(i−1)!(j−i−1)!(n−j)!f(xi)f(xj)[F(xi)]i−1[F(xj)−F(xi)]j−i−1[1−F(xj)]n−jf_{X_{(i)},X_{(j)}}(x_i,x_j)=\frac{n!}{(i-1)!(j-i-1)!(n-j)!}f(x_{i})f(x_{j})[F(x_i)]^{i-1}[F(x_j)-F(x_i)]^{j-i-1}[1-F(x_j)]^{n-j}fX(i),X(j)(xi,xj)=(i−1)!(j−i−1)!(n−j)!n!f(xi)f(xj)[F(xi)]i−1[F(xj)−F(xi)]j−i−1[1−F(xj)]n−j

Theorem 4: Density of all order statistics
f(x(1),⋯,x(n))=n!f(x1)f(x2)⋯f(xn),x1<x2,⋯<xnf(x_{(1)},\cdots,x_{(n)})=n!f(x_1)f(x_2)\cdots f(x_n),x_1<x_2,\cdots<x_nf(x(1),⋯,x(n))=n!f(x1)f(x2)⋯f(xn),x1<x2,⋯<xn

Part 6 Basic Asymptotic Theory
Tip 1: Convergence mode of random variable sequence:

Convergence almost surely, Xn→Xa.s.X_n \to X\ a.s.Xn→X a.s.
P[lim⁡n→∞Xn=X]=1P[\lim_{n \to \infty}X_n = X]=1P[n→∞limXn=X]=1
Lp(p>0)L_p(p>0)Lp(p>0) Convergence Xn→LpXX_n \to_{L_p} XXn→LpX
lim⁡n→∞E[∣Xn−X∣p]=0\lim_{n\to \infty} E[|X_n-X|^p] = 0n→∞limE[∣Xn−X∣p]=0
If p=1p=1p=1，called convergence in mean; If p=2p=2p=2，called mean square convergence
Convergence in probability, Xn→PXX_n \to_P XXn→PX
∀ϵ>0,P[∣Xn−X∣>ϵ]=0\forall \epsilon>0, P[|X_n-X|>\epsilon]=0∀ϵ>0,P[∣Xn−X∣>ϵ]=0
Convergence in distribution Xn→DXX_n \to_D XXn→DX
For arbitrary bounded continuous function, h:(X,B(X))→(X,B(X))h:(\mathcal{X},\mathcal{B}(\mathcal{X}))\to (\mathcal{X},\mathcal{B}(\mathcal{X}))h:(X,B(X))→(X,B(X))，
lim⁡n→∞Eh(Xn)=Eh(X)\lim_{n \to \infty} Eh(X_n) = Eh(X)n→∞limEh(Xn)=Eh(X)

Tip 2: Relations among different convergence mode

p>qp>qp>q, if Xn→LpXX_n \to_{L_p}XXn→LpX, then Xn→LqXX_n \to_{L_q} XXn→LqX
If Xn→LpXX_n \to_{L_p}XXn→LpX, then Xn→PXX_n \to_P XXn→PX
If Xn→Xa.s.X_n \to X\ a.s.Xn→X a.s., then Xn→PXX_n \to_P XXn→PX
If Xn→Xa.s.X_n \to X\ a.s.Xn→X a.s., then g(Xn)→g(X)a.s.g(X_n) \to g(X)\ a.s.g(Xn)→g(X) a.s., ggg is bounded continuous function
If Xn→DXX_n \to_{D}XXn→DX, then g(Xn)→Dg(X)g(X_n) \to_{D} g(X)g(Xn)→Dg(X)
If Xn→PXX_n \to_{P}XXn→PX, then g(Xn)→Pg(X)g(X_n) \to_{P} g(X)g(Xn)→Pg(X)
Xn→Xa.s.X_n \to X\ a.s.Xn→X a.s., g(x)g(x)g(x) is convex and xp=o(g(x))x^p=o(g(x))xp=o(g(x)). If {Eg(Xn)}\{Eg(X_n)\}{Eg(Xn)} is a bounded sequence, then Xn→LpXX_n\to_{L_p}XXn→LpX
(Slutsky’s Theorem) If Xn→DXX_n \to_D XXn→DX, Yn→PcY_n \to_P cYn→Pc, then XnYn→DcXX_nY_n \to_D cXXnYn→DcX, Xn+Yn→DX+cX_n+Y_n \to_D X+cXn+Yn→DX+c

Tip 3: Weak Law of Large Number, if population variance is bounded
Xˉ→L2EX=μ\bar{X} \to_{L_2} EX = \muXˉ→L2EX=μ

Tip 4: Classical Central Limit Theorem
Zn=Xˉ−μσ/n→DN(0,1)Z_n = \frac{\bar{X}-\mu}{\sigma/\sqrt{n}} \to_D N(0,1)Zn=σ/nXˉ−μ→DN(0,1)

Tip 5: Delta Method
Var[g(Xˉ)]≈[g′(μ)]2Var(Xˉ−μ)=[g′(μ)]2σ2nVar[g(\bar{X})] \approx [g'(\mu)]^2 Var(\bar{X}-\mu) = [g'(\mu)]^2 \frac{\sigma^2}{n}Var[g(Xˉ)]≈[g′(μ)]2Var(Xˉ−μ)=[g′(μ)]2nσ2

Tip 6: Delta Method Refined Central Limit Theorem
Zn=g(Xˉ)−g(μ)[g′(μ)]2σ2n→DN(0,1)Z_n = \frac{g(\bar{X})-g(\mu)}{\sqrt{[g'(\mu)]^2 \frac{\sigma^2}{n}}} \to_D N(0,1)Zn=[g′(μ)]2nσ2g(Xˉ)−g(μ)→DN(0,1)

Part 7 Independence
Tip 1: Independence of Events

If A and B are independent, P(A∩B)=P(A)P(B)P(A\cap B) = P(A)P(B)P(A∩B)=P(A)P(B)
If A, B and C are pairwisely independent, P(A∩B)=P(A)P(B)P(A\cap B) = P(A)P(B)P(A∩B)=P(A)P(B), P(A∩C)=P(A)P(C)P(A\cap C) = P(A)P(C)P(A∩C)=P(A)P(C), P(B∩C)=P(B)P(C)P(B\cap C) = P(B)P(C)P(B∩C)=P(B)P(C)
If A, B and C are independent, then they are pairwisely independent and P(A∩B∩C)=P(A)P(B)P(C)P(A\cap B \cap C) = P(A)P(B)P(C)P(A∩B∩C)=P(A)P(B)P(C)

Tip 2: Independent Random Variables

Independence: fX,Y(x,y)=fX(x)fY(y)f_{X,Y}(x,y) = f_X(x)f_Y(y)fX,Y(x,y)=fX(x)fY(y)
Linearly Independence: Cov(X,Y)=0Cov(X,Y) = 0Cov(X,Y)=0, or E[XY]=E(X)E(Y)E[XY] = E(X)E(Y)E[XY]=E(X)E(Y), Var(X+Y)=Var(X)+Var(Y)Var(X+Y) = Var(X)+Var(Y)Var(X+Y)=Var(X)+Var(Y)
Mean Independence: E[Y∣X]=EYE[Y|X]=EYE[Y∣X]=EY
Independence>Mean Independence>Linearly Independence
MX+Y(t)=MX(t)MY(t)M_{X+Y}(t) = M_X(t)M_Y(t)MX+Y(t)=MX(t)MY(t)
g(X)g(X)g(X) and h(Y)h(Y)h(Y) are independent