reduced rank regression model

- The Fixed-X Case
- Classical Multivariate Regression Model
- The Random-X Case
Multivariate Reduced-Rank Regression
- RRR的应用

关于reduced rank regression model的内容可以直接跳到Multivariate Reduced-Rank Regression的部分

Multivariate linear regression is a natural extension of multiple linear regression in that both techniques try to interpret possible linear relationships between certain input and output variables. Multiple regression is concerned with studying to what extent the behavior of a single output variable Y is influenced by a set of r input variables X = (X1, ··· ,Xr) T ^T T .
Multivariate regression has s output variables Y = (Y1, ··· ,Ys) T ^T T , each of whose behavior may be influenced by exactly the same set of inputs
X = (X1, ··· ,Xr) T ^T T .
So, not only are the components of X correlated with each other, but in multivariate regression, the components of Y are also correlated with each other (and with the components of X). In this chapter, we are interested in estimating the regression relationship between Y and X, taking into account the various dependencies between the r-vector X and the s-vector Y and the dependencies within X and within Y.
we describe the multivariate reduced-rank regression model (RRR) (Izenman, 1975), which is an enhancement of the classical multivariate regression model and has recently received research attention in the statistics and econometrics literature. The following reasons explain the popularity of this model: RRR provides a unified approach to many of the diverse classical multivariate statistical techniques; it lends itself quite naturally to analyzing a wide variety of statistical problems involving reduction of dimensionality and the search for structure in multivariate data; and it is relatively simple to program because the regression estimates depend only upon the sample covariance matrices of X and Y and the eigendecomposition of a certain symmetric matrix that generalizes the multiple squared correlation coefficient R2 from multiple regression.

我们需要考虑X是随机和非随机两种情况

The Fixed-X Case

Let Y = (Y1, ··· ,Ys) T ^T T be a random s-vector-valued output variate with
mean vector µ Y µ_Y µY and covariance matrix Σ Y Y Σ_{Y Y} ΣYY , and let X = (X1, ··· ,Xr) T ^T T
be a fixed (nonstochastic) r-vector-valued input variate. The components
of the output vector Y will typically be continuous responses, and the
components of the input vector X may be indicator or “dummy” variables
that are set up by the researcher to identify known groupings of the data
associated with distinct subpopulations or experimental conditions.

Suppose we observe n replications,
( X j T , Y j T ) τ (X_j^T , Y_j^T )^τ (XjT,YjT)τ, j = 1, 2,… ,n,
on the (r + s)-vector ( X τ , Y τ ) τ (X^τ , Y^τ )^τ (Xτ,Yτ)τ . We define an (r × n)-matrix X and an
(s × n)-matrix Y by X = ( X 1 , ⋅ ⋅ ⋅ , X n ) , Y = ( Y 1 , ⋅ ⋅ ⋅ , Y n ) \mathcal X = (X1, ··· , Xn), \mathcal Y = (Y1, ··· , Yn) X=(X1,⋅⋅⋅,Xn),Y=(Y1,⋅⋅⋅,Yn).

Form the mean vectors,
X ˉ = n − 1 ∑ j = 1 n X j \bar{X} = n^{−1}\sum_{j=1}^n Xj Xˉ=n−1∑j=1nXj , Y ˉ = n − 1 ∑ j = 1 n Y j \bar{Y} = n^{−1}\sum_{j=1}^n Yj Yˉ=n−1∑j=1nYj
and let
X = ( X ˉ , ⋅ ⋅ ⋅ , X ˉ ) , Y = ( Y ˉ , ⋅ ⋅ ⋅ , Y ˉ ) \mathcal{X} = (\bar{X} , ··· , \bar{X} ), \mathcal{Y} = (\bar{Y} , ··· , \bar{Y} ) X=(Xˉ,⋅⋅⋅,Xˉ),Y=(Yˉ,⋅⋅⋅,Yˉ)
The centered versions of X and Y are defined by
X c = X − X ˉ = ( X 1 − X ˉ , ⋅ ⋅ ⋅ , X n − X ˉ ) \mathcal X_c=\mathcal X − \bar{\mathcal X}=(X1 − \bar{X} , ··· , Xn − \bar{X} ) Xc=X−Xˉ=(X1−Xˉ,⋅⋅⋅,Xn−Xˉ),
Y c = Y − Y ˉ = ( Y 1 − Y ˉ , ⋅ ⋅ ⋅ , Y n − Y ˉ ) \mathcal Y_c = \mathcal Y − \bar{\mathcal Y} = (Y1 − \bar{Y} , ··· , Yn − \bar{Y} ) Yc=Y−Yˉ=(Y1−Yˉ,⋅⋅⋅,Yn−Yˉ),
respectively.

Classical Multivariate Regression Model

Consider the multivariate linear regression model
Y s × n = µ s × n + Θ s × r X r × n + ϵ s × n Y_{s×n} =µ_{s×n} + Θ_{s×r} X_ {r×n} +\epsilon_ {s×n} Ys×n=µs×n+Θs×rXr×n+ϵs×n,
( θ 11 θ 12 . . . θ 1 n θ 21 θ 22 . . . θ 2 n . . . . . . . . . X s 1 θ s 2 . . . θ s r ) ∗ ( X 11 X 12 . . . X 1 n X 21 X 22 . . . X 2 n . . . . . . . . . X r 1 X r 2 . . . X r n ) \begin{pmatrix} \theta_{11} & \theta_{12} &...& \theta_{1n} \\\theta_{21} & \theta_{22} &...& \theta_{2n}\\ ...&...& &...&\\X_{s1} & \theta_{s2} &...& \theta_{sr} \end{pmatrix}*\begin{pmatrix} X_{11} & X_{12} &...& X_{1n} \\ X_{21} & X_{22} &...& X_{2n}\\ ...&...& &...&\\X_{r1} & X_{r2} &...& X_{rn} \end{pmatrix} ⎝⎜⎜⎛θ11θ21...Xs1θ12θ22...θs2.........θ1nθ2n...θsr⎠⎟⎟⎞∗⎝⎜⎜⎛X11X21...Xr1X12X22...Xr2.........X1nX2n...Xrn⎠⎟⎟⎞

ϵ = ( ϵ 1 , ϵ 2 , ⋅ ⋅ ⋅ , ϵ n ) \epsilon= (\epsilon_1, \epsilon_2, ··· , \epsilon_n) ϵ=(ϵ1,ϵ2,⋅⋅⋅,ϵn) is the (s × n) error matrix whose columns are each random s-vectors with mean 0 and the same unknown nonsingular (s × s) error covariance matrix Σ ϵ ϵ Σ_{ \epsilon \epsilon} Σϵϵ,and pairs of column vectors, ( ϵ j , ϵ k ) ( \epsilon_j , \epsilon_k) (ϵj,ϵk), j = k, are uncorrelated with each other

When the Xs are considered to be fixed in repeated sampling (e.g., in designed experiments), the so-called design matrix X consists of known constants and possibly also observed values of covariates, Θ is a full-rank matrix of unknown fixed effects, and µ = µ 0 1 n τ µ = µ_01^τ_n µ=µ01nτ, where µ 0 µ_0 µ0 is an unknown s-vector of constants.

LS Estimation
If we set µ = Y ˉ − Θ X ˉ \bar{\mathcal Y} - Θ\bar{\mathcal X} Yˉ−ΘXˉ, the model (6.7) reduces to Y c = Θ X c + ϵ \mathcal Y_c = Θ \mathcal X_c + \epsilon Yc=ΘXc+ϵ .

依据

Kronecker products

从上面的分析可知，multivariate regression没什么用
In its basic classical formulation, therefore, we see that multivariate regression is a procedure that has no true multivariate content. That is, there
is no reason to create specialized software to carry out a multivariate regression of Y on X when the same result can more easily be obtained by using existing multiple regression routines. This is one reason why many books on multivariate analysis do not contain a separate chapter on multivariate regression and also why the topics of multiple regression and multivariate regression are so often confused with each other.

The Random-X Case

Multivariate Reduced-Rank Regression

Most applications of reduced-rank regression have been directed toward problems in time series (time domain and frequency domain) and econometrics. This development has led to the introduction of the related topic of cointegration into the econometric literature.

下面给出Reduced-Rank Regression的模型

于是我们将矩阵C分解

为了求得最优解我们需要一个定理

简单来说

上面第二个等式好像中间两个是减号

RRR的应用

摘自：《Modern Multivariate Statistical Techniques》