UA MATH567 高维统计IV Lipschitz组合9 矩阵函数、半正定序与迹不等式

矩阵函数
半正定序(positive semi-definite order, PSD order)
迹不等式

这一讲的目标是提供一些矩阵分析的工具，因为下一讲我们要尝试导出随机矩阵的Bernstein不等式。

矩阵函数

假设XXX是对称矩阵，则XXX的所有特征值都是实数，我们可以写出XXX的谱分解为
X=∑i=1nλiuiuiTX = \sum_{i=1}^n \lambda_i u_iu_i^TX=i=1∑nλiuiuiT

其中λi\lambda_iλi是特征值，uiu_iui是对应的特征向量，假设fff是一个实函数，则我们定义矩阵函数为
f(X)=∑i=1nf(λi)uiuiTf(X) = \sum_{i=1}^n f(\lambda_i)u_iu_i^Tf(X)=i=1∑nf(λi)uiuiT

例多项式与幂级数
i）称f(X)f(X)f(X)为矩阵多项式如果
f(x)=∑i=0paixi,∀x∈Rf(x) = \sum_{i=0}^p a_ix^i,\forall x \in \mathbb{R}f(x)=i=0∑paixi,∀x∈R

则
f(X)=∑j=1m∑i=0paiλjiuiuiT=∑i=0paiXif(X) =\sum_{j=1}^m \sum_{i=0}^pa_i \lambda_j^i u_iu_i^T = \sum_{i=0}^p a_iX^if(X)=j=1∑mi=0∑paiλjiuiuiT=i=0∑paiXi

ii）称f(X)f(X)f(X)为矩阵的指数函数如果f(x)=exf(x)=e^xf(x)=ex，则
f(X)=∑i=1meλiuiuiT=eXf(X)=\sum_{i=1}^m e^{\lambda_i}u_iu_i^T = e^Xf(X)=i=1∑meλiuiuiT=eX

iii）对一般的解析函数，我们通常不能直接把它的表达式套用到矩阵上，而是只能用它的幂级数来表示，如果
f(x)=∑i=0∞ai(x−x0)if(x) = \sum_{i=0}^{\infty} a_i(x-x_0)^if(x)=i=0∑∞ai(x−x0)i

则
f(X)=∑j=1mf(λj)ujujT=∑i=0∞ai(X−x0In)if(X) = \sum_{j=1}^m f(\lambda_j)u_ju_j^T = \sum_{i=0}^{\infty} a_i(X-x_0I_n)^if(X)=j=1∑mf(λj)ujujT=i=0∑∞ai(X−x0In)i

半正定序(positive semi-definite order, PSD order)

记X≽0X \succcurlyeq 0X≽0，如果XXX是半正定矩阵（也就是λi(X)≥0\lambda_i(X) \ge 0λi(X)≥0），称X≽YX \succcurlyeq YX≽Y如果X−YX-YX−Y是半正定矩阵；这个序关系被称为半正定序，它是一个偏序关系。关于半正定序有下面的结论（下面给出的证明仅供参考）：

Before proof, let me state several results:

∥A∥=s1(A)=∣λ1(A)∣\left\| A \right\|=s_1(A) = |\lambda_1(A)|∥A∥=s1(A)=∣λ1(A)∣, so ∥A∥≤t\left\| A \right\|\le t∥A∥≤t means ∀i\forall i∀i, ∣λi(A)∣≤t|\lambda_i(A)| \le t∣λi(A)∣≤t
According to Courant-Fisher’s min-max theorem, λi(A)=max⁡dimE=imin⁡x∈S(E)⟨Ax,x⟩\lambda_i(A)=\max_{dim E = i}\min_{x \in S(E)}\langle Ax,x\rangleλi(A)=maxdimE=iminx∈S(E)⟨Ax,x⟩. If A⪯BA \preceq BA⪯B, ∀x∈S(E)\forall x \in S(E)∀x∈S(E), ⟨Ax,x⟩≤⟨Bx,x⟩\langle Ax,x\rangle \le \langle Bx,x\rangle⟨Ax,x⟩≤⟨Bx,x⟩. Thus, λi(A)≤λi(B)\lambda_i(A) \le \lambda_i(B)λi(A)≤λi(B)

Part (a)
If ∥X∥≤t\left\| X \right\| \le t∥X∥≤t, ∀i=1,⋯,n\forall i=1,\cdots,n∀i=1,⋯,n, ∣λi(X)∣≤t|\lambda_i(X)| \le t∣λi(X)∣≤t, −t≤λi(X)≤t-t \le \lambda_i(X)\le t−t≤λi(X)≤t. Thus, X+tI⪰0,−X+tI⪰0X+tI\succeq 0,-X+tI \succeq 0X+tI⪰0,−X+tI⪰0, or −tI⪯X⪯tI-tI \preceq X \preceq tI−tI⪯X⪯tI.

If −tI⪯X⪯tI-tI \preceq X \preceq tI−tI⪯X⪯tI, X+tI⪰0,−X+tI⪰0X+tI\succeq 0,-X+tI \succeq 0X+tI⪰0,−X+tI⪰0, by definition, −t≤λi(X)≤t,∀i-t \le \lambda_i(X)\le t,\forall i−t≤λi(X)≤t,∀i, so ∥X∥≤t\left\| X \right\| \le t∥X∥≤t.

Part (b)
If ∥X∥≤K\left\|X \right\| \le K∥X∥≤K, ∀i\forall i∀i, ∣λi(X)∣≤K|\lambda_i(X)| \le K∣λi(X)∣≤K, then f(λi(X))≤g(λi(X)),∀if(\lambda_i(X)) \le g(\lambda_i(X)),\forall if(λi(X))≤g(λi(X)),∀i. So f(X)⪯g(X)f(X) \preceq g(X)f(X)⪯g(X).

Part (c)
If X⪯YX \preceq YX⪯Y, ∀i\forall i∀i, λi(X)≤λi(Y)\lambda_i(X) \le \lambda_i(Y)λi(X)≤λi(Y). Since fff is an increasing function, f(λi(X))≤f(λi(Y))f(\lambda_i(X)) \le f(\lambda_i(Y))f(λi(X))≤f(λi(Y)), so f(X)⪯f(Y)f(X) \preceq f(Y)f(X)⪯f(Y).

Part (d)
X=[1−1−1−1],Y=[−1−1−1−1]X = \left[ \begin{matrix} 1&-1\\-1&-1 \end{matrix}\right],Y=\left[ \begin{matrix} -1&-1\\-1&-1 \end{matrix}\right]X=[1−1−1−1],Y=[−1−1−1−1]λ1(X)=2,λ2(X)=−2,λ1(Y)=0,λ2(Y)=−2λ1(X2)=2,λ2(X2)=0,λ1(Y2)=2,λ2(Y2)=0\lambda_1(X)=\sqrt{2},\lambda_2(X)=-\sqrt{2},\lambda_1(Y)=0,\lambda_2(Y)=-2 \\ \lambda_1(X^2)=2,\lambda_2(X^2)=0,\lambda_1(Y^2)=2,\lambda_2(Y^2)=0λ1(X)=2,λ2(X)=−2,λ1(Y)=0,λ2(Y)=−2λ1(X2)=2,λ2(X2)=0,λ1(Y2)=2,λ2(Y2)=0

Part (e)
If X⪯YX \preceq YX⪯Y, ∀i\forall i∀i, λi(X)≤λi(Y)\lambda_i(X) \le \lambda_i(Y)λi(X)≤λi(Y). Since fff is an increasing function, f(λi(X))≤f(λi(Y))f(\lambda_i(X)) \le f(\lambda_i(Y))f(λi(X))≤f(λi(Y)), so
∑i=1nf(λi(X))≤∑i=1nf(λi(Y))\sum_{i=1}^n f(\lambda_i(X))\le \sum_{i=1}^n f(\lambda_i(Y))i=1∑nf(λi(X))≤i=1∑nf(λi(Y))

Thus, trf(X)⪯trf(Y)trf(X) \preceq trf(Y)trf(X)⪯trf(Y).

Part (f)
If YYY is identity matrix, 0⪯X⪯I0 \preceq X \preceq I0⪯X⪯I, ∀i\forall i∀i, 0≤λi(X)≤10\le \lambda_i(X) \le 10≤λi(X)≤1, and then 1λi(X)≥1\frac{1}{\lambda_i(X)}\ge 1λi(X)1≥1. So X−1⪰IX^{-1}\succeq IX−1⪰I.

For arbitrary invertible YYY, 0⪯(YT)−1/2XY−1/2⪯I0\preceq (Y^T)^{-1/2}XY^{-1/2} \preceq I0⪯(YT)−1/2XY−1/2⪯I, (YT)1/2X−1Y1/2⪰I(Y^T)^{1/2}X^{-1}Y^{1/2}\succeq I(YT)1/2X−1Y1/2⪰I, so X−1⪰Y−1X^{-1}\succeq Y^{-1}X−1⪰Y−1

Part (g)
For any t>0t>0t>0, 0⪯X⪯Y0 \preceq X \preceq Y0⪯X⪯Y, −(X+t)−1⪯−(Y+t)−1-(X+t)^{-1}\preceq -(Y+t)^{-1}−(X+t)−1⪯−(Y+t)−1. For log⁡(X)=∫0∞(1+t)−1−(X+t)−1dt\log(X)=\int_0^{\infty}(1+t)^{-1}-(X+t)^{-1}dtlog(X)=∫0∞(1+t)−1−(X+t)−1dt, log⁡(X)⪯log⁡(Y)\log(X)\preceq \log(Y)log(X)⪯log(Y)

迹不等式

迹不等式在研究随机矩阵的概率不等式时非常有用，最主要的原因就是矩阵的乘法不满足交换律，以Hoeffding不等式的证明为例，对于随机变量，我们有
ex+y=exeye^{x+y} = e^x e^yex+y=exey

于是我们可以把一列随机变量拆分称指数的积或者合并到指数上的和，但对于随机矩阵而言eX+Y=eXeYe^{X+Y}=e^Xe^YeX+Y=eXeY不一定成立，所以我们需要能代替乘法交换律的工具，最容易想到的当然就是矩阵的迹了，因为在迹中做矩阵乘法是可以交换次序的，下面介绍两个常用的迹不等式：

Golden-Thompson不等式 A,BA,BA,B是两个nnn阶对称实矩阵，则
tr(eA+B)≤tr(eAeB)tr(e^{A+B}) \le tr(e^Ae^B)tr(eA+B)≤tr(eAeB)

Lieb不等式 假设HHH是nnn阶对称实矩阵，定义
f(X)=tr(eH+log⁡X)f(X) = tr(e^{H+\log X})f(X)=tr(eH+logX)

则f(X)f(X)f(X)是nnn阶对称实正定矩阵空间（这是一个convex cone）上的concave function，根据Jensen不等式
Ef(X)≤f(EX)Etr(eH+log⁡X)≤tr(eH+log⁡EX)Z=log⁡X,则Etr(eH+Z)≤tr(eH+log⁡EeZ)Ef(X) \le f(EX) \\ Etr(e^{H+\log X}) \le tr(e^{H+\log EX}) \\ Z = \log X,则Etr(e^{H+Z}) \le tr(e^{H+\log Ee^Z})Ef(X)≤f(EX)Etr(eH+logX)≤tr(eH+logEX)Z=logX,则Etr(eH+Z)≤tr(eH+logEeZ)