Summary of Boosted Tree and Factorization Machines

Boosted Tree

Definition:

y^=∑k=1Kfk(x)\widehat y=\sum_{k=1}^{K}f_k(x) y=k=1∑Kfk(x)

In which fk(x)f_k(x)fk(x) is one of K regression trees.

Loss:

Loss=∑i=1nL(yi,y^i)Loss=\sum_{i=1}^{n}L(y_i, \widehat y_i)Loss=i=1∑nL(yi,yi)

Add some regularization:

Loss=∑i=1nL(yi,y^i)+∑k=1KΩ(fk)Loss=\sum_{i=1}^{n}L(y_i,\widehat y_i) + \sum_{k=1}^{K}\Omega(f_k)Loss=i=1∑nL(yi,yi)+k=1∑KΩ(fk)

Additive Training:

y^(1)=0\widehat y^{(1)} = 0y(1)=0

y^(t)=y^(t−1)+ft(x)\widehat y^{(t)} = \widehat y^{(t-1)} + f_t(x)y(t)=y(t−1)+ft(x)

Loss(t)=∑i=1nL(yi,y^i(t))+∑k=1tΩ(fk)Loss^{(t)}=\sum_{i=1}^{n}L(y_i, \widehat y_i^{(t)}) + \sum_{k=1}^{t}\Omega(f_k)Loss(t)=i=1∑nL(yi,yi(t))+k=1∑tΩ(fk)

=∑i=1nL(yi,y^i(t−1)+ft(xi))+∑k=1t−1Ω(fk)+Ω(ft)=\sum_{i=1}^{n}L(y_i, \widehat y_i^{(t-1)}+f_t(x_i))+\sum_{k=1}^{t-1}\Omega(f_k)+\Omega(f_t)=i=1∑nL(yi,yi(t−1)+ft(xi))+k=1∑t−1Ω(fk)+Ω(ft)

=∑i=1nL(yi,y^i(t−1)+ft(xi))+Ω(ft)+C=\sum_{i=1}^{n}L(y_i, \widehat y_i^{(t-1)}+f_t(x_i))+\Omega(f_t)+C=i=1∑nL(yi,yi(t−1)+ft(xi))+Ω(ft)+C

≈∑i=1n[L(yi,y^i(t−1))+ft(xi)∂L∂y^i(t−1)+12ft2(xi)∂L2∂y^i(t−1)]+Ω(ft)+C\approx\sum_{i=1}^{n}[ L(y_i,\widehat y_i^{(t-1)})+f_t(x_i)\frac{\partial L}{\partial \widehat y_i^{(t-1)}}+\frac{1}{2}f_t^{2}(x_i)\frac{\partial L^2}{\partial \widehat y_i^{(t-1)}}]+\Omega(f_t)+C≈i=1∑n[L(yi,yi(t−1))+ft(xi)∂yi(t−1)∂L+21ft2(xi)∂yi(t−1)∂L2]+Ω(ft)+C

=∑i=1n[L(yi,y^i(t−1))+ft(xi)Gi+12ft2(xi)Hi]+Ω(ft)+C=\sum_{i=1}^{n}[ L(y_i,\widehat y_i^{(t-1)})+f_t(x_i)G_i+\frac{1}{2}f_t^{2}(x_i)H_i]+\Omega(f_t)+C=i=1∑n[L(yi,yi(t−1))+ft(xi)Gi+21ft2(xi)Hi]+Ω(ft)+C

=∑i=1n[ft(xi)Gi+12ft2(xi)Hi]+Ω(ft)+C′=\sum_{i=1}^{n}[f_t(x_i)G_i+\frac{1}{2}f_t^{2}(x_i)H_i] + \Omega(f_t) + C'=i=1∑n[ft(xi)Gi+21ft2(xi)Hi]+Ω(ft)+C′

Loss at time t is:

Loss(t)=∑i=1n[ft(xi)Gi+12ft2(xi)Hi]+Ω(ft)+C′Loss^{(t)}=\sum_{i=1}^{n}[f_t(x_i)G_i+\frac{1}{2}f_t^{2}(x_i)H_i] + \Omega(f_t) + C'Loss(t)=i=1∑n[ft(xi)Gi+21ft2(xi)Hi]+Ω(ft)+C′

Use:

ft(x)=wq(x),q:Rd→{1,2,...,M},wi∈Rf_t(x)=w_{q(x)}, q:R^d\rightarrow\{1,2,...,M\}, w_i \in Rft(x)=wq(x),q:Rd→{1,2,...,M},wi∈R

Ω(f)=12λ∑i=1Mwj2+γM\Omega(f)=\frac{1}{2}\lambda\sum_{i=1}^{M}w_j^{2}+\gamma MΩ(f)=21λi=1∑Mwj2+γM

We get:

Loss(t)=∑i=1n[ft(xi)Gi+12ft2(xi)Hi]+Ω(ft)+C′Loss^{(t)}=\sum_{i=1}^{n}[f_t(x_i)G_i+\frac{1}{2}f_t^{2}(x_i)H_i] + \Omega(f_t) + C'Loss(t)=i=1∑n[ft(xi)Gi+21ft2(xi)Hi]+Ω(ft)+C′

=∑i=1n[wq(xi)Gi+12wq(xi)2Hi]+12λ∑j=1Mwj2+γM+C′=\sum_{i=1}^{n}[w_{q(x_i)}G_i+\frac{1}{2} w_{q(x_i)}^2H_i]+\frac{1}{2}\lambda\sum_{j=1}^{M}w_j^{2}+\gamma M+C'=i=1∑n[wq(xi)Gi+21wq(xi)2Hi]+21λj=1∑Mwj2+γM+C′

With Ij={i∣q(xi)=j}I_j=\{i|q(x_i)=j\}Ij={i∣q(xi)=j}:

∑i=1nwq(xi)Gi=∑j=1M[wj∑i∈IjGi]\sum_{i=1}^{n}w_{q(x_i)}G_i=\sum_{j=1}^{M}[w_j\sum_{i\in I_j}^{}G_i]i=1∑nwq(xi)Gi=j=1∑M[wji∈Ij∑Gi]

∑i=1n12wq(xi)2Hi=∑j=1Mwj2∑i∈Ij12Hi\sum_{i=1}^{n}\frac{1}{2}w_{q(x_i)}^2H_i=\sum_{j=1}^{M}w_j^2\sum_{i \in I_j}^{}\frac{1}{2}H_ii=1∑n21wq(xi)2Hi=j=1∑Mwj2i∈Ij∑21Hi

So:

Loss(t)=∑j=1M[wj∑i∈IjGi+wj2∑i∈Ij12Hi+12λwj2]+γM+C′Loss^{(t)}=\sum_{j=1}^{M}[w_j\sum_{i\in I_j}G_i+w_j^2\sum_{i\in I_j}\frac{1}{2}H_i+\frac{1}{2}\lambda w_j^2]+\gamma M + C'Loss(t)=j=1∑M[wji∈Ij∑Gi+wj2i∈Ij∑21Hi+21λwj2]+γM+C′

=∑j=1M[wj∑i∈IjGi+12wj2(λ+∑i∈IjHi)]+γM+C′=\sum_{j=1}^{M}[w_j\sum_{i\in I_j}G_i+\frac{1}{2}w_j^2(\lambda+\sum_{i\in I_j}H_i)]+\gamma M + C'=j=1∑M[wji∈Ij∑Gi+21wj2(λ+i∈Ij∑Hi)]+γM+C′

With Gj′=∑i∈IjGi,Hj′=∑i∈IjHiG_j'=\sum_{i\in I_j}G_i, H_j'=\sum_{i\in I_j}H_iGj′=∑i∈IjGi,Hj′=∑i∈IjHi:

Loss(t)=∑j=1M[wjGj′+12wj2(λ+Hj′)]+γM+C′Loss^{(t)}=\sum_{j=1}^{M}[w_jG_j'+\frac{1}{2}w_j^2(\lambda+H_j')]+\gamma M + C'Loss(t)=j=1∑M[wjGj′+21wj2(λ+Hj′)]+γM+C′

Finally:

wj∗=argmin(wjGj′+12wj2(λ+Hi′))=−Gj′λ+Hi′w_j^*=argmin(w_jG_j'+\frac{1}{2}w_j^2(\lambda+H_i'))=-\frac{G_j'}{\lambda+H_i'}wj∗=argmin(wjGj′+21wj2(λ+Hi′))=−λ+Hi′Gj′

Obj(t)=min(Loss(t))=−12∑j=1MGj′2Hj′+λ+γM+C′Obj^{(t)}=min(Loss^{(t)})=-\frac{1}{2}\sum_{j=1}^{M}\frac{G_j'^2}{H_j'+\lambda}+\gamma M + C'Obj(t)=min(Loss(t))=−21j=1∑MHj′+λGj′2+γM+C′

So for each iteration t of training, greedily seach for a regression tree ft(xi)=wq(xi)f_t(x_i)=w_{q(x_i)}ft(xi)=wq(xi) with wj=−Gj′λ+Hi′w_j=-\frac{G_j'}{\lambda+H_i'}wj=−λ+Hi′Gj′ with minumum Obj(t)Obj^{(t)}Obj(t) and add it to model.

Factorization Machines

y=w0+∑i=1nwixi+∑i=1n−1∑j=i+1n<vi,vj>xixjy=w_0+\sum_{i=1}^{n}w_ix_i+\sum_{i=1}^{n-1}\sum_{j=i+1}^{n}<\boldsymbol{v}_i,\boldsymbol{v}_j>x_ix_jy=w0+i=1∑nwixi+i=1∑n−1j=i+1∑n<vi,vj>xixj

In which

w0∈R,w∈Rn,v∈Rn×kw_0\in R,\boldsymbol{w}\in R^{n},\boldsymbol{v}\in R^{n\times k}w0∈R,w∈Rn,v∈Rn×k

<vi,vj>=∑l=1kvilvjl<\boldsymbol{v}_i,\boldsymbol{v}_j>=\sum_{l=1}^{k}v_{il}v_{jl}<vi,vj>=l=1∑kvilvjl