【NLP】Words Normalization+PorterStemmer源码解析

Words Normalization

Stemming（词干提取）

词干提取是去除单词的前后缀得到词根的过程

caresses  ->  caress
ponies    ->  poni
ties      ->  ti
caress    ->  caress
cats      ->  catfeed      ->  feed
agreed    ->  agree
disabled  ->  disable

Lemmatisation（词形还原）

词形还原是基于词典，将单词的复杂形态转变成最基础的形态

词形还原不是简单地将前后缀去掉，而是会根据词典将单词进行转换。比如「drove」会转换为「drive」

PorterStemmer源码解析

PorterStemmer是基于Stemming的英文分词工具，源码如下：

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# @Author   : linjie
#!/usr/bin/env python
'''基于Stemming（词干提取，去除单词的前后缀得到词根的过程）的words normalization的实现'''"""Porter Stemming Algorithm
This is the Porter stemming algorithm, ported to Python from the
version coded up in ANSI C by the author. It may be be regarded
as canonical, in that it follows the algorithm presented inPorter, 1980, An algorithm for suffix stripping, Program, Vol. 14,
no. 3, pp 130-137,only differing from it at the points maked --DEPARTURE-- below.See also http://www.tartarus.org/~martin/PorterStemmerThe algorithm as described in the paper could be exactly replicated
by adjusting the points of DEPARTURE, but this is barely necessary,
because (a) the points of DEPARTURE are definitely improvements, and
(b) no encoding of the Porter stemmer I have seen is anything like
as exact as this version, even with the points of DEPARTURE!Vivake Gupta (v@nano.com)Release 1: January 2001Further adjustments by Santiago Bruno (bananabruno@gmail.com)
to allow word input not restricted to one word per line, leading
to:release 2: July 2008
"""import sysclass PorterStemmer:def __init__(self):"""The main part of the stemming algorithm starts here.b is a buffer holding a word to be stemmed. The letters are in b[k0],b[k0+1] ... ending at b[k]. In fact k0 = 0 in this demo program. k isreadjusted downwards as the stemming progresses. Zero termination isnot in fact used in the algorithm.Note that only lower case sequences are stemmed. Forcing to lower caseshould be done before stem(...) is called."""self.b = ""  # buffer for word to be stemmedself.k = 0self.k0 = 0self.j = 0   # j is a general offset into the stringdef cons(self, i):"""cons(i) is TRUE <=> b[i] is a consonant."""if self.b[i] == 'a' or self.b[i] == 'e' or self.b[i] == 'i' or self.b[i] == 'o' or self.b[i] == 'u':return 0if self.b[i] == 'y':if i == self.k0:return 1else:return (not self.cons(i - 1))return 1def m(self):"""m() measures the number of consonant sequences between k0 and j.if c is a consonant sequence and v a vowel sequence, and <..>indicates arbitrary presence,<c><v>       gives 0<c>vc<v>     gives 1<c>vcvc<v>   gives 2<c>vcvcvc<v> gives 3...."""n = 0i = self.k0while 1:if i > self.j:return nif not self.cons(i):breaki = i + 1i = i + 1while 1:while 1:if i > self.j:return nif self.cons(i):breaki = i + 1i = i + 1n = n + 1while 1:if i > self.j:return nif not self.cons(i):breaki = i + 1i = i + 1def vowelinstem(self):"""vowelinstem() is TRUE <=> k0,...j contains a vowel"""for i in range(self.k0, self.j + 1):if not self.cons(i):return 1return 0def doublec(self, j):"""doublec(j) is TRUE <=> j,(j-1) contain a double consonant."""if j < (self.k0 + 1):return 0if (self.b[j] != self.b[j-1]):return 0return self.cons(j)def cvc(self, i):"""cvc(i) is TRUE <=> i-2,i-1,i has the form consonant - vowel - consonantand also if the second c is not w,x or y. this is used when trying torestore an e at the end of a short  e.g.cav(e), lov(e), hop(e), crim(e), butsnow, box, tray."""if i < (self.k0 + 2) or not self.cons(i) or self.cons(i-1) or not self.cons(i-2):return 0ch = self.b[i]if ch == 'w' or ch == 'x' or ch == 'y':return 0return 1def ends(self, s):"""ends(s) is TRUE <=> k0,...k ends with the string s."""length = len(s)if s[length - 1] != self.b[self.k]: # tiny speed-upreturn 0if length > (self.k - self.k0 + 1):return 0if self.b[self.k-length+1:self.k+1] != s:return 0self.j = self.k - lengthreturn 1def setto(self, s):"""setto(s) sets (j+1),...k to the characters in the string s, readjusting k."""length = len(s)self.b = self.b[:self.j+1] + s + self.b[self.j+length+1:]self.k = self.j + lengthdef r(self, s):"""r(s) is used further down."""if self.m() > 0:self.setto(s)def step1ab(self):"""step1ab() gets rid of plurals and -ed or -ing. e.g.caresses  ->  caressponies    ->  ponities      ->  ticaress    ->  caresscats      ->  catfeed      ->  feedagreed    ->  agreedisabled  ->  disablematting   ->  matmating    ->  matemeeting   ->  meetmilling   ->  millmessing   ->  messmeetings  ->  meet"""if self.b[self.k] == 's':if self.ends("sses"):self.k = self.k - 2elif self.ends("ies"):self.setto("i")elif self.b[self.k - 1] != 's':self.k = self.k - 1if self.ends("eed"):if self.m() > 0:self.k = self.k - 1elif (self.ends("ed") or self.ends("ing")) and self.vowelinstem():self.k = self.jif self.ends("at"):   self.setto("ate")elif self.ends("bl"): self.setto("ble")elif self.ends("iz"): self.setto("ize")elif self.doublec(self.k):self.k = self.k - 1ch = self.b[self.k]if ch == 'l' or ch == 's' or ch == 'z':self.k = self.k + 1elif (self.m() == 1 and self.cvc(self.k)):self.setto("e")def step1c(self):"""step1c() turns terminal y to i when there is another vowel in the stem."""if (self.ends("y") and self.vowelinstem()):self.b = self.b[:self.k] + 'i' + self.b[self.k+1:]def step2(self):"""step2() maps double suffices to single ones.so -ization ( = -ize plus -ation) maps to -ize etc. note that thestring before the suffix must give m() > 0."""if self.b[self.k - 1] == 'a':if self.ends("ational"):   self.r("ate")elif self.ends("tional"):  self.r("tion")elif self.b[self.k - 1] == 'c':if self.ends("enci"):      self.r("ence")elif self.ends("anci"):    self.r("ance")elif self.b[self.k - 1] == 'e':if self.ends("izer"):      self.r("ize")elif self.b[self.k - 1] == 'l':if self.ends("bli"):       self.r("ble") # --DEPARTURE--# To match the published algorithm, replace this phrase with#   if self.ends("abli"):      self.r("able")elif self.ends("alli"):    self.r("al")elif self.ends("entli"):   self.r("ent")elif self.ends("eli"):     self.r("e")elif self.ends("ousli"):   self.r("ous")elif self.b[self.k - 1] == 'o':if self.ends("ization"):   self.r("ize")elif self.ends("ation"):   self.r("ate")elif self.ends("ator"):    self.r("ate")elif self.b[self.k - 1] == 's':if self.ends("alism"):     self.r("al")elif self.ends("iveness"): self.r("ive")elif self.ends("fulness"): self.r("ful")elif self.ends("ousness"): self.r("ous")elif self.b[self.k - 1] == 't':if self.ends("aliti"):     self.r("al")elif self.ends("iviti"):   self.r("ive")elif self.ends("biliti"):  self.r("ble")elif self.b[self.k - 1] == 'g': # --DEPARTURE--if self.ends("logi"):      self.r("log")# To match the published algorithm, delete this phrasedef step3(self):"""step3() dels with -ic-, -full, -ness etc. similar strategy to step2."""if self.b[self.k] == 'e':if self.ends("icate"):     self.r("ic")elif self.ends("ative"):   self.r("")elif self.ends("alize"):   self.r("al")elif self.b[self.k] == 'i':if self.ends("iciti"):     self.r("ic")elif self.b[self.k] == 'l':if self.ends("ical"):      self.r("ic")elif self.ends("ful"):     self.r("")elif self.b[self.k] == 's':if self.ends("ness"):      self.r("")def step4(self):"""step4() takes off -ant, -ence etc., in context <c>vcvc<v>."""if self.b[self.k - 1] == 'a':if self.ends("al"): passelse: returnelif self.b[self.k - 1] == 'c':if self.ends("ance"): passelif self.ends("ence"): passelse: returnelif self.b[self.k - 1] == 'e':if self.ends("er"): passelse: returnelif self.b[self.k - 1] == 'i':if self.ends("ic"): passelse: returnelif self.b[self.k - 1] == 'l':if self.ends("able"): passelif self.ends("ible"): passelse: returnelif self.b[self.k - 1] == 'n':if self.ends("ant"): passelif self.ends("ement"): passelif self.ends("ment"): passelif self.ends("ent"): passelse: returnelif self.b[self.k - 1] == 'o':if self.ends("ion") and (self.b[self.j] == 's' or self.b[self.j] == 't'): passelif self.ends("ou"): pass# takes care of -ouselse: returnelif self.b[self.k - 1] == 's':if self.ends("ism"): passelse: returnelif self.b[self.k - 1] == 't':if self.ends("ate"): passelif self.ends("iti"): passelse: returnelif self.b[self.k - 1] == 'u':if self.ends("ous"): passelse: returnelif self.b[self.k - 1] == 'v':if self.ends("ive"): passelse: returnelif self.b[self.k - 1] == 'z':if self.ends("ize"): passelse: returnelse:returnif self.m() > 1:self.k = self.jdef step5(self):"""step5() removes a final -e if m() > 1, and changes -ll to -l ifm() > 1."""self.j = self.kif self.b[self.k] == 'e':a = self.m()if a > 1 or (a == 1 and not self.cvc(self.k-1)):self.k = self.k - 1if self.b[self.k] == 'l' and self.doublec(self.k) and self.m() > 1:self.k = self.k -1def stem(self, p, i, j):"""In stem(p,i,j), p is a char pointer, and the string to be stemmedis from p[i] to p[j] inclusive. Typically i is zero and j is theoffset to the last character of a string, (p[j+1] == '\0'). Thestemmer adjusts the characters p[i] ... p[j] and returns the newend-point of the string, k. Stemming never increases word length, soi <= k <= j. To turn the stemmer into a module, declare 'stem' asextern, and delete the remainder of this file."""# copy the parameters into staticsself.b = pself.k = jself.k0 = iif self.k <= self.k0 + 1:return self.b # --DEPARTURE--# With this line, strings of length 1 or 2 don't go through the# stemming process, although no mention is made of this in the# published algorithm. Remove the line to match the published# algorithm.self.step1ab()self.step1c()self.step2()self.step3()self.step4()self.step5()return self.b[self.k0:self.k+1]if __name__ == '__main__':p = PorterStemmer()output = ''word = 'matting'output += p.stem(word, 0,len(word)-1)print(output)

1、def init(self)

self.b = “” 用于存取原始单词
self.k = 0 用于存取原始单词的长度-1
self.k0 = 0 用于存取起始位置0
self.j = 0 j是字符串的一般偏移量

2、def stem(self, p, i, j)

是该算法的一个总流程函数，其中包含self.xxx的初始化，也包含每个Normalization规则的执行

self.step1ab()
self.step1c()
self.step2()
self.step3()
self.step4()
self.step5()

这六种规则包含了所有Stemming规则，流程走完即可

每个规则其实就是if else的判断，易读。

3、def ends(self, s)

如果返回TRUE，则说明原始单词的最后几位与s匹配了

4、举“matting”例子

word = ‘matting’
此时数据如下：

根据规则，处理matting的规则在step1ab()中的如下代码

运行完后的参数结果如下：

最后返回结果规则是：return self.b[self.k0:self.k+1]，左闭右开
所以最后结果为“mat”

参考

[PorterStemmer源码]：https://tartarus.org/martin/PorterStemmer/python.txt
[词干提取 – Stemming | 词形还原 – Lemmatisation]：https://easyai.tech/ai-definition/stemming-lemmatisation/