李航《统计学习方法》SMO算法推导中的思考

1. p.128

图中，从上式到下式的推导不是很明了，困惑在于上式中右边含有α1,α2α1,α2\alpha_1,\alpha_2这样岂不是和左边的α2α2\alpha_2相消？若能相消，上述求偏导的过程中岂不是忽略了v1,v2v1,v2v_1,v_2是α1,α2α1,α2\alpha_1,\alpha_2的函数？答案并非如此，左边的αα\alpha与右边的α2α2\alpha_2相当不同。

2. 定理7.6的证明

即求最优化问题:

minα1,α2W(α1,α2)=12K11α21+12K22α22+y1y2K21α1α2−(α1+α2)+y1α1∑i=3NyiαiKi1+y2α2∑i=3NyiαiKi2s.t.α1y1+α2y2=−∑i=3Nyiαi=ς(1)0≤αi≤C,i=1,2(2)minα1,α2W(α1,α2)=12K11α12+12K22α22+y1y2K21α1α2−(α1+α2)+y1α1∑i=3NyiαiKi1+y2α2∑i=3NyiαiKi2s.t.α1y1+α2y2=−∑i=3Nyiαi=ς(1)0≤αi≤C,i=1,2(2)\min\limits_{\alpha_1,\alpha_2}W(\alpha_1,\alpha_2)=\frac{1}{2}K_{11}\alpha_1^2+\frac{1}{2}K_22\alpha_2^2+y_1y_2K_{21}\alpha_1\alpha_2-(\alpha_1+\alpha_2)+y_1\alpha_1\sum\limits_{i=3}^{N}y_i\alpha_iK_{i1}+y_2\alpha_2\sum\limits_{i=3}^Ny_i\alpha_iK_{i2}\\ \quad s.t. \\ \qquad \alpha_1y_1+\alpha_2y_2=-\sum\limits_{i=3}^Ny_i\alpha_i=\varsigma \qquad(1)\\ \qquad 0\le\alpha_i\le C ,\quad i=1,2 \qquad(2)
沿着约束方向未经剪辑的解为: αnew,unc2=αold2+y2(E1−E2)ηα2new,unc=α2old+y2(E1−E2)η\alpha_2^{new,unc}=\alpha_2^{old}+\frac{y_2(E_1-E_2)}{\eta}
其中， η=K11+K22−2K21=‖Φ(x1)−Φ(x2)‖2Ei=g(xi)−yi,i=1,2g(x)=∑i=1NαiyiK(xi,x)+bη=K11+K22−2K21=‖Φ(x1)−Φ(x2)‖2Ei=g(xi)−yi,i=1,2g(x)=∑i=1NαiyiK(xi,x)+b\eta=K_{11}+K_{22}-2K_{21}=\|\Phi(x_1)-\Phi(x_2)\|^2\\ E_i=g(x_i)-y_i , \quad i=1,2 \\ g(x)=\sum\limits_{i=1}^{N}\alpha_iy_iK(x_i,x)+b
EiEiE_i为函数g(xi)g(xi)g(x_i)对输入xixix_i的预测值与真实输出yiyiy_i之间的差.

证明：

引入记号变量 vi=∑j=3NαjyjK(xi,xj)=g(xi)−∑j=12αjyjK(xi,xj)−b,i=1,2vi=∑j=3NαjyjK(xi,xj)=g(xi)−∑j=12αjyjK(xi,xj)−b,i=1,2v_i=\sum\limits_{j=3}^{N}\alpha_jy_jK(x_i,x_j)=g(x_i)-\sum\limits_{j=1}^{2}\alpha_jy_jK(x_i,x_j)-b, \quad i=1,2
再由α1y1+α2y2=ςα1y1+α2y2=ς\alpha_1y_1+\alpha_2y_2=\varsigma 和 y2i=1yi2=1y_i^2=1可将目标函数W(α1,α2)W(α1,α2)W(\alpha_1,\alpha_2)表示成只含α2α2\alpha_2的函数:

W(α2)=12K11(ς−α2y2)2+12K22α22+y2K21(ς−α2y2)α2−(ς−α2y2)y1−α2+v1(ς−α2y2)+y2v2α2W(α2)=12K11(ς−α2y2)2+12K22α22+y2K21(ς−α2y2)α2−(ς−α2y2)y1−α2+v1(ς−α2y2)+y2v2α2W(\alpha_2)=\frac{1}{2}K_{11}(\varsigma-\alpha_2y_2)^2+\frac{1}{2}K_{22}\alpha_2^2+y_2K_{21}(\varsigma-\alpha_2y_2)\alpha_2-(\varsigma-\alpha_2y_2)y_1-\alpha_2+v_1(\varsigma-\alpha_2y_2)+y_2v_2\alpha_2
对α2α2\alpha_2求偏导可得: ∂W∂α2=K11α2+K22α2−2K21α2−K11ςy2+K21ςy2+y1y2−1−v1y2+v2y2∂W∂α2=K11α2+K22α2−2K21α2−K11ςy2+K21ςy2+y1y2−1−v1y2+v2y2\frac{\partial W}{\partial \alpha_2}=K_{11}\alpha_2+K_{22}\alpha_2-2K_{21}\alpha_2-K_{11}\varsigma y_2+K_{21}\varsigma y_2+y_1y_2-1-v_1y_2+v_2y_2
令其等于0,即可求出上述问题的最优解,得到: (K11+K22−2K21)α2=y2(y2−y1+ςK11−ςK21+v1−v2)(2−1)(K11+K22−2K21)α2=y2(y2−y1+ςK11−ςK21+v1−v2)(2−1)(K_{11}+K_{22}-2K_{21})\alpha_2=y_2(y_2-y_1+\varsigma K_{11}-\varsigma K_{21}+v_1-v_2)\qquad (2-1)
这里，不同与书上，我先求v1−v2v1−v2v_1-v_2,
由viviv_i的定义可知v1−v2v1−v2v_1-v_2为: v1−v2=⟮g(x1)−∑j=12yjαjK1j−b⟯−⟮g(x2)−∑j=12yjαjK2j−b⟯=g(x1)−g(x2)+y1α1K21−y1α1K11+y2α2K22−y2α2K21∵α1y1+α2y2=ς,y21=1∴α1=(ς−y2α2)y1,代入得，=g(x1)−g(x2)+(ς−y2α2)K21−(ς−y2α2)K11−y2α2K21+y2α2K22=g(x1)−g(x2)+ςK21−ςK11+y2α2(K11+K22−2K21)v1−v2=⟮g(x1)−∑j=12yjαjK1j−b⟯−⟮g(x2)−∑j=12yjαjK2j−b⟯=g(x1)−g(x2)+y1α1K21−y1α1K11+y2α2K22−y2α2K21∵α1y1+α2y2=ς,y12=1∴α1=(ς−y2α2)y1,代入得，=g(x1)−g(x2)+(ς−y2α2)K21−(ς−y2α2)K11−y2α2K21+y2α2K22=g(x1)−g(x2)+ςK21−ςK11+y2α2(K11+K22−2K21)v_1-v_2= \lgroup g(x_1)-\sum\limits_{j=1}^{2}y_j\alpha_jK_{1j}-b\rgroup-\lgroup g(x_2)-\sum\limits_{j=1}^{2}y_j\alpha_jK_{2j}-b\rgroup\\ \qquad \quad=g(x_1)-g(x_2)+y_1\alpha_1K_{21}-y_1\alpha_1K_{11}+y_2\alpha_2K_{22}-y_2\alpha_2K_{21}\\ \because \alpha_1y_1+\alpha_2y_2=\varsigma,\quad y_1^2=1\\\therefore \alpha_1=(\varsigma-y_2\alpha_2)y_1 ,代入得，\\\qquad\quad =g(x_1)-g(x_2)+(\varsigma-y_2\alpha_2)K_{21}-(\varsigma-y_2\alpha_2)K_{11}-y_2\alpha_2K_{21}+y_2\alpha_2K_{22}\\\qquad\quad =g(x_1)-g(x_2)+\varsigma K_{21}-\varsigma K_{11}+y_2\alpha_2(K_{11}+K_{22}-2K_{21})
值得注意的是，v1−v2v1−v2v_1-v_2中的α1,α2α1,α2\alpha_1,\alpha_2是没有更新前的αα\alpha,即αoldαold\alpha^{old}
∴v1−v2=g(x1)−g(x2)+ςK21−ςK11+y2αold2(K11+K22−2K21)∴v1−v2=g(x1)−g(x2)+ςK21−ςK11+y2α2old(K11+K22−2K21)\therefore v_1-v_2=g(x_1)-g(x_2)+\varsigma K_{21}-\varsigma K_{11}+y_2\alpha_2^{old}(K_{11}+K_{22}-2K_{21}) 将上式代入到式（2-1）中即可求得αnew,unc2=αold2+y2(E1−E2)ηα2new,unc=α2old+y2(E1−E2)η\alpha_2^{new,unc}=\alpha_2^{old}+\frac{y_2(E_1-E_2)}{\eta}