Word2vec 计算两个文本之间相似度

安装gensim 并且有pyemd，详情见下

 def wmdistance(self, document1, document2):"""Compute the Word Mover's Distance between two documents. When using thiscode, please consider citing the following papers:.. Ofir Pele and Michael Werman, "A linear time histogram metric for improved SIFT matching"... Ofir Pele and Michael Werman, "Fast and robust earth mover's distances"... Matt Kusner et al. "From Word Embeddings To Document Distances".Note that if one of the documents have no words that exist in theWord2Vec vocab, `float('inf')` (i.e. infinity) will be returned.This method only works if `pyemd` is installed (can be installed via pip, but requires a C compiler).Example:>>> # Train word2vec model.>>> model = Word2Vec(sentences)>>> # Some sentences to test.>>> sentence_obama = 'Obama speaks to the media in Illinois'.lower().split()>>> sentence_president = 'The president greets the press in Chicago'.lower().split()>>> # Remove their stopwords.>>> from nltk.corpus import stopwords>>> stopwords = nltk.corpus.stopwords.words('english')>>> sentence_obama = [w for w in sentence_obama if w not in stopwords]>>> sentence_president = [w for w in sentence_president if w not in stopwords]>>> # Compute WMD.>>> distance = model.wmdistance(sentence_obama, sentence_president)"""if not PYEMD_EXT:raise ImportError("Please install pyemd Python package to compute WMD.")# Remove out-of-vocabulary words.len_pre_oov1 = len(document1)len_pre_oov2 = len(document2)document1 = [token for token in document1 if token in self]document2 = [token for token in document2 if token in self]diff1 = len_pre_oov1 - len(document1)diff2 = len_pre_oov2 - len(document2)if diff1 > 0 or diff2 > 0:logger.info('Removed %d and %d OOV words from document 1 and 2 (respectively).',diff1, diff2)if len(document1) == 0 or len(document2) == 0:logger.info('At least one of the documents had no words that were''in the vocabulary. Aborting (returning inf).')return float('inf')dictionary = Dictionary(documents=[document1, document2])vocab_len = len(dictionary)if vocab_len == 1:# Both documents are composed by a single unique tokenreturn 0.0# Sets for faster look-up.docset1 = set(document1)docset2 = set(document2)# Compute distance matrix.distance_matrix = zeros((vocab_len, vocab_len), dtype=double)for i, t1 in dictionary.items():for j, t2 in dictionary.items():if not t1 in docset1 or not t2 in docset2:continue# Compute Euclidean distance between word vectors.distance_matrix[i, j] = sqrt(np_sum((self[t1] - self[t2])**2))if np_sum(distance_matrix) == 0.0:# `emd` gets stuck if the distance matrix contains only zeros.logger.info('The distance matrix is all zeros. Aborting (returning inf).')return float('inf')def nbow(document):d = zeros(vocab_len, dtype=double)nbow = dictionary.doc2bow(document)  # Word frequencies.doc_len = len(document)for idx, freq in nbow:d[idx] = freq / float(doc_len)  # Normalized word frequencies.return d# Compute nBOW representation of documents.d1 = nbow(document1)d2 = nbow(document2)# Compute WMD.return emd(d1, d2, distance_matrix)

Word2vec 计算两个文本之间相似度相关推荐

python 两个word文档之间的相似度_如何用 word2vec 计算两个句子之间的相似度？
现在是2018年7月,在这里总结这个问题下已有的答案,并补充一些2017年以来这方面研究的新进展. 从大类上分,计算句子相似度的方法可以分为两类: 1)无监督的方法,即不使用额外的标注数据,常用的方法 ...
python 句子相似度库_利用python语句的word2vec查找两个句子之间的相似度
我想用word2vectors计算两个句子之间的相似度,我试图得到一个句子向量的向量,这样我就可以计算出一个句子向量的平均值来找到余弦相似度.我试过这个代码,但它不起作用.它给出的输出是带有一的句子向 ...
ITK:计算两个图像之间的均方度量值
ITK:计算两个图像之间的均方度量值内容提要输出结果 C++实现代码内容提要计算两个图像之间的均方根度量. 输出结果来自2张图像的数据通过输出传递: [-10, -10]: 23101.7 ...
如何利用计算机计算天数,如何应用Win10系统电脑中的计算器计算两个日期之间的天数？...
如何应用Win10系统电脑中的计算器计算两个日期之间的天数? 相信很多小伙伴都有在使用windows10系统的电脑,其中的计算器如何才能用来计算两个日期之间的天数呢?方法很简单,下面小编就来为大家介绍 ...
C++/JAVA 计算两篇文章的相似度
C++/JAVA 计算两篇文章的相似度这位少侠,要不要进店瞧瞧? 实验介绍及思路问题描述: 编写程序,计算任意两篇文章的相似度. 基本思路: 利用余弦相似度来计算其相似度. 完整代码 C++ 代码 ...
计算两组标签/关键词相似度算法
原文连接 http://www.zhaochao.net/index.php/2016/02/05/14/ 写作背景标签在互联网行业有大量的应用,给博客打标签,给商品打标签,给新闻打标签.通常每篇文 ...
Python计算向量夹角代码：如何用代码计算两个向量之间的夹角？
Python计算向量夹角代码:如何用代码计算两个向量之间的夹角? 向量夹角是指两个向量之间的夹角,通常用cosine来表示.在机器学习和数据科学领域,计算向量夹角是一个非常基本的概念.Python是一 ...
计算两个日期之间的年数
//问题2:计算两个日期之间的年数 #include <iostream.h> class Date { private :int year, month, day;public :Dat ...
python计算两个向量之间的欧氏距离
python计算两个向量之间的欧氏距离代码: import numpy as np # 向量的值 feature_1 = np.array([1,2,3]) feature_2 = np.array ...

Word2vec 计算两个文本之间相似度

安装gensim 并且有pyemd，详情见下

Word2vec 计算两个文本之间相似度相关推荐

最新文章

热门文章