Good-Turing、Absolute、kneser-ney smooth

Good-Turing smoothing

原文链接

Good-Turing基本思想是：用观察计数较高的N元语法数重新估计概率量的大小，并把它指派给那些具有零计数或者较低计数的N元语法。
公式:
c∗=(c+1)Nc+1Ncc^* = \frac{(c+1)N_{c+1}}{N_c}c∗=Nc(c+1)Nc+1
其中c为某个N-gram出现的频数,NcN_cNc为出现次数为c的N-Gram的词组的个数，c∗为Good−Turing平滑计数c^*为Good-Turing平滑计数c∗为Good−Turing平滑计数
例子:
对于a = [A,A,A,B,B,C,D,E] sum(len(a)) = 7
cA=3c_A = 3cA=3 cC=2c_C = 2cC=2
cB=cD=cE=1c_B = c_D = c_E = 1cB=cD=cE=1
N1N_1N1=4 N2N_2N2=1 N3=1N_3=1N3=1
thus
cA∗=4∗03=0c^*_A = 4*\frac{0}{3} = 0cA∗=4∗30=0
cB∗=3∗11=3c^*_B = 3*\frac{1}{1} = 3cB∗=3∗11=3
cC∗=cD∗=cE∗=2∗14=12c^*_C = c^*_D = c^*_E = 2*\frac{1}{4} = \frac{1}{2}cC∗=cD∗=cE∗=2∗41=21
（注意在他们求概率的时候都需要除以N = 7）
然而这样会导致最高频率的结果最后的计数为0，解决方法如下

最终会出现概率之和不为0的情况，这时候要进行归一化，固定没有见过的结果的概率，将已经见过的概率之和归一化使得共同的概率结果为1
例子可见于L9-NLP-LangModels.pdf第64页

Absolute discounting

一般的interpolation是利用高阶的模型的PMLEP_MLEPMLE乘以一个λ\lambdaλ,而此处是从每个非零计数中减掉一个固定的δ∈(0,1)\delta \in (0,1)δ∈(0,1),一般取δ=0.75\delta = 0.75δ=0.75
在bigram中，公式为
PAD(wi∣wi−1)=c(wi−1,wi)−δc(wi−1)+λ(wi−1)P(w))P_{AD}(w_i|w_{i-1}) = \frac{c(w_{i-1},w_i)-\delta}{c(w_{i-1})}+\lambda(w_{i-1})P(w))PAD(wi∣wi−1)=c(wi−1)c(wi−1,wi)−δ+λ(wi−1)P(w))
其中λ(wi−1)为插值权重系数\lambda(w_{i-1})为插值权重系数λ(wi−1)为插值权重系数
于是完整的公式就是
pabs(wi∣wi−n+1i)=max{c(wi−n+1i)−δ,0}∑wic(wi−n+1i)+(1−λwi−n+1i−1pab(wi∣wi−1i−n+2))p_{abs}(w_i|w^i_{i-n+1}) = \frac{max\{c(w^i_{i-n+1})-\delta,0\}}{\sum_{w_i}c(w^i_{i-n+1})}+(1-\lambda_{w^{i-1}_{i-n+1}}p_{ab}(w_i|w^{i-1}{i-n+2}))pabs(wi∣wi−n+1i)=∑wic(wi−n+1i)max{c(wi−n+1i)−δ,0}+(1−λwi−n+1i−1pab(wi∣wi−1i−n+2))
为了使得结果的总和为1
1−λwi−n+1i−1=δ∑wic(wi−n+1i)N1+(wi−n+1i−1∙)1 - \lambda _ { w _ { i - n + 1 } } ^ { i - 1 } = \frac { \delta } { \sum _ { w _ { i } } c ( w _ { i - n + 1 } ^ { i } ) } N _ { 1 + } ( w _ { i - n + 1 } ^ { i - 1 } \mathbf { \bullet } )1−λwi−n+1i−1=∑wic(wi−n+1i)δN1+(wi−n+1i−1∙)
其中
N1+(wi−n+1i−1∙)=∣{wi:c(wi−n+1i−1wi)>0}∣N _ { 1 + } \left( w _ { i - n + 1 } ^ { i - 1 } \bullet \right) = \left| \left\{ w _ { i } : c \left( w _ { i - n + 1 } ^ { i - 1 } w _ { i } \right) > 0 \right\} \right|N1+(wi−n+1i−1∙)=∣∣{wi:c(wi−n+1i−1wi)>0}∣∣

Kneser-Ney smoothing

bigram下的公式:
pKN(wi∣wi−1)=max⁡(c(wi−1,wi)−δ,0)∑w′c(wi−1,w′)+λwi−1pKN(wi)p _ { K N } \left( w _ { i } | w _ { i - 1 } \right) = \frac { \max \left( c \left( w _ { i - 1 } , w _ { i } \right) - \delta , 0 \right) } { \sum _ { w ^ { \prime } } c \left( w _ { i - 1 } , w ^ { \prime } \right) } + \lambda _ { w _ { i - 1 } } p _ { K N } \left( w _ { i } \right)pKN(wi∣wi−1)=∑w′c(wi−1,w′)max(c(wi−1,wi)−δ,0)+λwi−1pKN(wi)
其中
pKN(wi)=∣{w′:0<c(w′,wi)}∣∣{(w′,w′′):0<c(w′,w′′)}∣p _ { K N } \left( w _ { i } \right) = \frac { \left| \left\{ w ^ { \prime } : 0 < c \left( w ^ { \prime } , w _ { i } \right) \right\} \right| } { \left| \left\{ \left( w ^ { \prime } , w ^ { \prime \prime } \right) : 0 < c \left( w ^ { \prime } , w ^ { \prime \prime } \right) \right\} \right| }pKN(wi)=∣{(w′,w′′):0<c(w′,w′′)}∣∣{w′:0<c(w′,wi)}∣
为的是求解在一个不熟悉的上下文中看见单词wiw_iwi的可能性,这使用wiw_iwi在出现在所有单词的次数和除以所有bigram的和来衡量
减掉一个固定的δ∈(0,1)\delta \in (0,1)δ∈(0,1),一般取δ=0.75\delta = 0.75δ=0.75
λwi−1\lambda_{w_{i-1}}λwi−1是用来平衡使得条件概率pKN(wi∣wi−1)p _ { K N } ( w _ { i } | w _ { i - 1 } )pKN(wi∣wi−1)的总和为1的系数
得出满足条件的λwi−1\lambda_{w_{i-1}}λwi−1结果为
λwi−1=δ∑w′c(wi−1,w′)∣{w′:0<c(wi−1,w′)}∣\lambda _ { w _ { i - 1 } } = \frac { \delta } { \sum _ { w ^ { \prime } } c \left( w _ { i - 1 } , w ^ { \prime } \right) } \left| \left\{ w ^ { \prime } : 0 < c \left( w _ { i - 1 } , w ^ { \prime } \right) \right\} \right|λwi−1=∑w′c(wi−1,w′)δ∣{w′:0<c(wi−1,w′)}∣
可以推广到n-gram
pKN(wi∣wi−n+1i−1)=max⁡(c(wi−n+1i−1,wi)−δ,0)∑w′c(wi−n+1i−1,w′)+δ∣{w′:0<c(wi−n+1i−1,w′)}∣∑wic(wi−n+1i)pKN(wi∣wi−n+2i−1)p _ { K N } \left( w _ { i } | w _ { i - n + 1 } ^ { i - 1 } \right) = \frac { \max \left( c \left( w _ { i - n + 1 } ^ { i - 1 } , w _ { i } \right) - \delta , 0 \right) } { \sum _ { w ^ { \prime } } c \left( w _ { i - n + 1 } ^ { i - 1 } , w ^ { \prime } \right) } + \delta \frac { \left| \left\{ w ^ { \prime } : 0 < c \left( w _ { i - n + 1 } ^ { i - 1 } , w ^ { \prime } \right) \right\} \right| } { \sum _ { w _ { i } } c \left( w _ { i - n + 1 } ^ { i } \right) } p _ { K N } \left( w _ { i } | w _ { i - n + 2 } ^ { i - 1 } \right) pKN(wi∣wi−n+1i−1)=∑w′c(wi−n+1i−1,w′)max(c(wi−n+1i−1,wi)−δ,0)+δ∑wic(wi−n+1i)∣∣{w′:0<c(wi−n+1i−1,w′)}∣∣pKN(wi∣wi−n+2i−1)