用Python計算文本的相似度

因為后期會需要用到這方面的知識，所以先提前准備准備；如何判斷網頁返回內容的相似度？

找到了幾個Python的方法和庫：

還有高大上的“TF-IDF方法”{之前在《數學之美》中看到過，但這里我就不考慮了}

下面主要記錄用不同的Python庫來計算兩段文本之間的相似度(最后要得到的就是一個百分比)：

方法一：difflib12345678910

>>>importdifflib>>>difflib.SequenceMatcher(None,'abcde','abcde').ratio()1.0>>>difflib.SequenceMatcher(None,'abcde','zbcde').ratio()0.80000000000000004>>>difflib.SequenceMatcher(None,'abcde','zyzzy').ratio()0.0

方法二：Levenshtein

import Levenshtein 報錯：ImportError: No module named Levenshtein

於是去：python-Levenshtein 下載源碼進行安裝(在http://www.lfd.uci.edu/~gohlke/pythonlibs/#python-levenshtein其實也有編譯好的exe)，第一次安裝的時候報錯：error: Unable to find vcvarsall.bat ，但其實我是裝了VS2010的，所以執行如下步驟正常安裝：

1.設置環境變量，執行：

SET VS90COMNTOOLS=%VS100COMNTOOLS%

2.再去安裝：

setup.py install

就可以正常，編譯，安裝了。123456789101112131415161718192021222324252627282930313233

$python>>>importLevenshtein>>>help(Levenshtein.ratio)ratio(...)Computesimilarityoftwostrings.ratio(string1,string2)Thesimilarityisanumberbetween0and1,it's usually equal orsomewhat higher than difflib.SequenceMatcher.ratio(), becuase it'sbasedonrealminimaleditdistance.Examples:>>>ratio('Hello world!','Holly grail!')0.58333333333333337>>>ratio('Brian','Jesus')0.0>>>help(Levenshtein.distance)distance(...)ComputeabsoluteLevenshteindistanceoftwostrings.distance(string1,string2)Examples(it's hard to spell Levenshtein correctly):>>> distance('Levenshtein', 'Lenvinsten')4>>> distance('Levenshtein', 'Levensthein')2>>> distance('Levenshtein', 'Levenshten')1>>> distance('Levenshtein', 'Levenshtein')01234567891011121314151617181920212223242526

gitclonegit://github.com/seatgeek/fuzzywuzzy.git fuzzywuzzycdfuzzywuzzypythonsetup.pyinstall>>>fromfuzzywuzzyimportfuzz>>>fromfuzzywuzzyimportprocessSimpleRatio>>>fuzz.ratio("this is a test","this is a test!")96PartialRatio>>>fuzz.partial_ratio("this is a test","this is a test!")100TokenSortRatio>>>fuzz.ratio("fuzzy wuzzy was a bear","wuzzy fuzzy was a bear")90>>>fuzz.token_sort_ratio("fuzzy wuzzy was a bear","wuzzy fuzzy was a bear")100TokenSetRatio>>>fuzz.token_sort_ratio("fuzzy was a bear","fuzzy fuzzy was a bear")84>>>fuzz.token_set_ratio("fuzzy was a bear","fuzzy fuzzy was a bear")100123456789101112131415

importdiff_match_patchtextA="the cat in the red hat"textB="the feline in the blue hat"dmp=diff_match_patch.diff_match_patch()  #create a diff_match_patch objectdiffs=dmp.diff_main(textA,textB)   # All 'diff' jobs start with invoking diff_main()d_value=dmp.diff_levenshtein(diffs)printd_valuemaxLenth=max(len(textA),len(textB))printfloat(d_value)/float(maxLenth)similarity=(1-float(d_value)/float(maxLenth))*100printsimilarity

上面這段代碼的思路也是先計算Levenshtein距離，然后再將其和兩字符串的最大長度相除，得到相似度(不清楚這樣和直接使用Levenshtein擴展有什么區別，畢竟那個直接是用C寫成的，速度可能還要快一些，直接一些)