本文为《Linear algebra and its applications》的读书笔记

Least-Squares problems

Inconsistent systems arise often in applications. When a solution is demanded and none exists, the best one can do is to find an x\boldsymbol xx that makes AxA\boldsymbol xAx as close as possible to b\boldsymbol bb. Think of AxA\boldsymbol xAx as an approximationapproximationapproximation to b\boldsymbol bb. The smaller the distance between b\boldsymbol bb and AxA\boldsymbol xAx, given by ∥b−Ax∥\left\|\boldsymbol b - A\boldsymbol x\right\|∥b−Ax∥, the better the approximation.

The general least-squares problem is to find an x\boldsymbol xx that makes ∥b−Ax∥\left\|\boldsymbol b - A\boldsymbol x\right\|∥b−Ax∥ as small as possible. ∥b−Ax∥\left\|\boldsymbol b - A\boldsymbol x\right\|∥b−Ax∥ is called the least-squares error of this approximation.

The adjective “least-squares” arises from the fact that ∥b−Ax∥\left\|\boldsymbol b - A\boldsymbol x\right\|∥b−Ax∥ is the square root of a sum of squares.

Notice that AxA\boldsymbol xAx will necessarily be in ColAColAColA. So we seek an x\boldsymbol xx that makes AxA\boldsymbol xAx the closest point in ColAColAColA to b\boldsymbol bb. See Figure 1.

Solution of the General Least-Squares Problem

ATAx=ATbA^TA\boldsymbol x=A^T\boldsymbol bATAx=ATb

Apply the Best Approximation Theorem in Section 6.3 to the subspace ColAColAColA. Let
b^=projColAb\hat \boldsymbol b=proj_{ColA}\boldsymbol bb^=projColAbBecause b^\hat\boldsymbol bb^ is in ColAColAColA, the equation Ax=b^A\boldsymbol x =\hat\boldsymbol bAx=b^ is consistent, and there is an x^\hat\boldsymbol xx^ in Rn\mathbb R^nRn such that
Ax^=b^(1)A\hat\boldsymbol x =\hat\boldsymbol b\ \ \ \ \ \ \ \ \ \ (1)Ax^=b^ (1)Since b^\hat\boldsymbol bb^ is the closest point in ColAColAColA to b\boldsymbol bb, a vector x^\hat\boldsymbol xx^ is a least-squares solution of Ax=bA\boldsymbol x =\boldsymbol bAx=b if and only if x^\hat\boldsymbol xx^ satisfies (1). See Figure 2. [There are many solutions of (1) if the equation has free variables.]
Suppose x^\hat\boldsymbol xx^ satisfies Ax^=b^A\hat\boldsymbol x =\hat\boldsymbol bAx^=b^. By the Orthogonal Decomposition Theorem in Section 6.3, b−b^\boldsymbol b -\hat\boldsymbol bb−b^ is orthogonal to ColAColAColA, so b−Ax^\boldsymbol b - A\hat\boldsymbol xb−Ax^ is orthogonal to each column of AAA. If aj\boldsymbol a_jaj is any column of AAA, then aj⋅(b−Ax^)=0\boldsymbol a_j \cdot (\boldsymbol b- A\hat\boldsymbol x)=0aj⋅(b−Ax^)=0, and ajT(b−Ax^)=0\boldsymbol a^T_j(\boldsymbol b - A\hat\boldsymbol x)= 0ajT(b−Ax^)=0. Since each ajT\boldsymbol a^T_jajT is a row of ATA^TAT ,
AT(b−Ax^)=0(2)A^T (\boldsymbol b - A\hat\boldsymbol x)=\boldsymbol 0\ \ \ \ \ \ \ (2)AT(b−Ax^)=0 (2)(This equation also follows from (ColA)⊥=NulAT(Col\ A)^\perp=Nul\ A^T(Col A)⊥=Nul AT.) Thus
ATb−ATAx^=0ATAx^=ATb\begin{aligned}A^T \boldsymbol b - A^TA\hat\boldsymbol x&=\boldsymbol 0\\ A^TA\hat\boldsymbol x&=A^T \boldsymbol b\end{aligned}ATb−ATAx^ATAx^=0=ATbThese calculations show that each least-squares solution of Ax=bA\boldsymbol x =\boldsymbol bAx=b satisfies the equation
The equation represents a system of equations called the normal equations (法方程) for Ax=bA\boldsymbol x =\boldsymbol bAx=b. A solution of (3) is often denoted by x^\hat\boldsymbol xx^.

The next theorem gives useful criteria for determining when there is only one least-squares solution of Ax=bA\boldsymbol x =\boldsymbol bAx=b. (Of course, the orthogonal projection b^\hat\boldsymbol bb^ is always unique.)

Formula (4) for x^\hat \boldsymbol xx^ is useful mainly for theoretical purposes and for hand calculations when ATAA^TAATA is a 2×22\times 22×2 invertible matrix.

PROOF

If Ax=0A\boldsymbol x =\boldsymbol 0Ax=0, then ATAx=0A^TA\boldsymbol x =\boldsymbol 0ATAx=0 ∴NulA⊆NulATA\therefore NulA\subseteq NulA^TA∴NulA⊆NulATA.
If ATAx=0A^TA\boldsymbol x =\boldsymbol 0ATAx=0, then xTATAx=(Ax)TAx=0\boldsymbol x^TA^TA\boldsymbol x =(A\boldsymbol x)^TA\boldsymbol x=\boldsymbol 0xTATAx=(Ax)TAx=0 ∴Ax=0\therefore A\boldsymbol x=\boldsymbol 0∴Ax=0 ∴NulATA⊆NulA\therefore NulA^TA\subseteq NulA∴NulATA⊆NulA. ∴NulA=NulATA∴rankATA=n−dimNulATA=n−dimNulA=rankA\therefore NulA= NulA^TA\\\therefore rankA^TA=n-dimNulA^TA=n-dimNulA=rankA∴NulA=NulATA∴rankATA=n−dimNulATA=n−dimNulA=rankASo when AAA has nnn linearly independent columns, rankATA=rankA=nrankA^TA=rankA=nrankATA=rankA=n, which means ATAA^TAATA is an invertible matrix.

当 AAA 的列正交时，快速找到最小二乘解

The next example shows how to find a least-squares solution of Ax=bA\boldsymbol x =\boldsymbol bAx=b when the columns of AAA are orthogonal. Such matrices often appear in linear regression problems.

EXAMPLE 4

Find a least-squares solution of Ax=bA\boldsymbol x =\boldsymbol bAx=b for

SOLUTION

Because the columns a1\boldsymbol a_1a1 and a2\boldsymbol a_2a2 of AAA are orthogonal, the orthogonal projection of b\boldsymbol bb onto ColAColAColA is given by
Now that b^\hat \boldsymbol bb^ is known, we can solve Ax^=b^A\hat\boldsymbol x=\hat\boldsymbol bAx^=b^. But this is trivial, since we already know what weights to place on the columns of AAA to produce b^\hat\boldsymbol bb^. It is clear from (5) that

In some cases, the normal equations for a least-squares problem can be illillill-conditioned(病态的)conditioned(病态的)conditioned(病态的); that is, small errors in the calculations of the entries of ATAA^TAATA can sometimes cause relatively large errors in the solution x^\hat \boldsymbol xx^.

利用 QR 分解

If the columns of AAA are linearly independent, the least-squares solution can often be computed more reliably through a QRQRQR factorization of AAA (described in Section 6.4).

Chapter 6 (Orthogonality and Least Squares): Least-Squares problems (最小二乘问题)相关推荐

Chapter 6 (Orthogonality and Least Squares): Applications to linear models
目录 Least-Squares Lines (最小二乘直线) The General Linear Model Least-Squares Fitting of Other Curves Multi ...
Chapter 4, FAQ about Master Theorm, exercises and problems
FAQ about the Master theorem Q1: Why in case 1, f(n) must be polynomially smaller than n^log(b,a)? R ...
Coursera自动驾驶课程第13讲：Least Squares
在上一讲<Coursera自动驾驶课程第12讲:Semantic Segmentation>我们学习了深度学习的另一个重要应用:语义分割.至此,本课程的视觉感知模块就介绍完了. 从本讲开始 ...
【Moving Least Squares】【移动最小二乘法】
基于移动最小二乘的图像变形一.背景意义写这篇博文是应为目前为止我看到了好多领域里的经典paper算法都有涉及到移动最小二乘(MLS).可见这个算法非常重要,先来看一下它的相关经典应用: 1.图像变 ...
perfect squares java,Perfect Squares
题目来源一道DP题,求一个数最少由几个平方数组成.我一开始用二维DP, 因为是个完全背包问题,所以需要注意每个数都是可以取多次的. 代码如下: class Solution { public: in ...
最小二乘估计（Least squares estimation）
最小二乘估计(Least squares estimation) 最小二乘估计面向的问题最小二乘估计(Least squares estimation) 加权最小二乘估计(Weighted leas ...
Python Tutorial中英双语对照文档1
本文根据官方文档 http://www.pythondoc.com/pythontutorial3/ 和中文文档 http://www.pythondoc.com/pythontutorial3/ ...
matlab中没有linearmodel,MATLAB线性代数简明教程(Linear Algebra Using MATLAB)
MATLAB线性代数简明教程(Linear Algebra Using MATLAB) 编辑锁定讨论上传视频 <MATLAB线性代数简明教程(Linear Algebra Using MA ...
005_Data Structures_数据结构
Data Structures 数据结构 This chapter describes some things you've learned about already in more detail, ...
INEQUALITY BOOKS
来源:这里 Bất Đẳng Thức Luôn Có Một Sức Cuốn Hút Kinh Khủng, Một Số tài Liệu và Sách Bổ ích Cho Việc Học ...

Chapter 6 (Orthogonality and Least Squares): Least-Squares problems (最小二乘问题)

目录

Least-Squares problems

Solution of the General Least-Squares Problem

ATAx=ATbA^TA\boldsymbol x=A^T\boldsymbol bATAx=ATb

当 AAA 的列正交时，快速找到最小二乘解

利用 QR 分解

Chapter 6 (Orthogonality and Least Squares): Least-Squares problems (最小二乘问题)相关推荐

最新文章

热门文章