Computing Parameters Analytically

Normal Equation

Find the optimum θ\thetaθ without iteration

Minimize J by explicitly taking its derivatives with respect to the θj ’s, and setting them to zero.

Formula:

θ=(XTX)−1XTy\theta={(X^TX)}^{-1}X^Tyθ=(XTX)−1XTy

Octave: pinv (X’X) X’*y

Design matrix (X)

m examples(x(1),y(1)),...(x(m),y(m))(x^{(1)}, y^{(1)}) ,...(x^{(m)} ,y^{(m)})(x(1),y(1)),...(x(m),y(m));n features
x(i)=[x0(i)x1(i)⋅⋅⋅xn(i)]∈Rn+1x^{(i)}=\begin{bmatrix} x_0^{(i)}\\ x_1^{(i)}\\ \cdot\\ \cdot\\ \cdot\\ x_n^{(i)} \end{bmatrix}\in\R^{n+1} x(i)=⎣⎢⎢⎢⎢⎢⎢⎢⎡x0(i)x1(i)⋅⋅⋅xn(i)⎦⎥⎥⎥⎥⎥⎥⎥⎤∈Rn+1

X=[−(x(1))T−−(x(2))T−⋅⋅⋅−(x(m))T−](m×(n+1)−dimensional)X=\begin{bmatrix} -(x^{(1)})^T-\\ -(x^{(2)})^T-\\ \cdot\\ \cdot\\ \cdot\\ -(x^{(m)})^T- \end{bmatrix}(m\times(n+1)-dimensional) X=⎣⎢⎢⎢⎢⎢⎢⎡−(x(1))T−−(x(2))T−⋅⋅⋅−(x(m))T−⎦⎥⎥⎥⎥⎥⎥⎤(m×(n+1)−dimensional)

There is no need to do feature scaling.

Comparison of gradient descent and normal equation:

GradientDesentNormalEquationNeedtochoosealphaNoneedtochoosealphaNeedsmanyiterationsNoneedtoiterateo(kn2)o(n3),needtocalculateinverseofXTXWorkswellwhennislargeSlowifnisverylarge\begin{array}{|c|clr|} \hline Gradient \;Desent&Normal\;Equation\\ \hline Need\;to\;choose\;alpha&No\;need\;to\;choose\;alpha\\ \hline Needs\;many\;iterations&No\;need\;to\;iterate\\ \hline \mathcal{o}(kn^2)&\mathcal{o}(n^3),need\;to\;calculate\;inverse\;of\;X^TX\\ \hline Works\;well\;when\;n\;is\;large&Slow\;if\;n\;is\;very\;large\\ \end{array} GradientDesentNeedtochoosealphaNeedsmanyiterationso(kn2)WorkswellwhennislargeNormalEquationNoneedtochoosealphaNoneedtoiterateo(n3),needtocalculateinverseofXTXSlowifnisverylarge
With the normal equation, computing the inversion has complexity O(n3)\mathcal{O}(n^3)O(n3). So if we have a very large number of features, the normal equation will be slow.

Normal Equation Noninvertibility

If XTXX^TXXTX is noninvertible, the common causes might be having :

Redundant features, where two features are very closely related (i.e. they are linearly dependent)
Too many features (e.g. m ≤ n). In this case, delete some features or use “regularization” (to be explained in a later lesson).

Solutions

Deleting a feature that is linearly dependent with another .(Redundant features)
Deleting one or more features or use regularization when there are too many features( e.g. m≤nm\leq nm≤n).