word2vec加载异常解决：UnicodeDecodeError: ‘utf-8‘ codec can‘t decode bytes in position。。。

原文链接：https://blog.csdn.net/W_Honor/article/details/105037033?utm_medium=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-1.channel_param&depth_1-utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-1.channel_param

NLP领域最出名的python库之一就是gensim，该库包含了常见的word2vec模型，而我们在使用这些模型进行embedding的时候可会出现如下的编码问题：

UnicodeDecodeError: 'utf-8' codec can't decode bytes in position

个错误在提示加载模型时出现了编码错误，即在某一个位置的字节是无法编码的，并会给出相应的的位置。
这个问题在gensim官方github中也有反馈，并且作者统一给出了解决方法：

Answer: The strings (words) stored in your model are not valid utf8. By default, gensim decodes the words using the strict encoding settings, which results in the above exception whenever an invalid utf8 sequence is encountered.
The fix is on your side and it is to either:
a) Store your model using a program that understands unicode and utf8 (such as gensim). Some C and Java word2vec tools are known to truncate the strings at byte boundaries, which can result in cutting a multi-byte utf8 character in half, making it non-valid utf8, leading to this error.
b) Set the unicode_errors flag when running load_word2vec_model, e.g. load_word2vec_model(…, unicode_errors=‘ignore’). Note that this silences the error, but the utf8 problem is still there – invalid utf8 characters will just be ignored in this case.

第一种方法告诉了使用者可以用C和JAVA的相关工具去处文本中的无效编码字符；
第二种方法建议在load_word2vec_model函数中设置参数unicode_errors=‘ignore’，可以暂时忽略这个错误。

def get_word_vectors(model_path, dim, words):wordVec = gensim.models.KeyedVectors.load_word2vec_format(model_path, binary=True, unicode_errors='ignore')wordEmbedding = []for word in words:try:if "<PAD>" == word:wordEmbedding.append(np.zeros(dim))else:vector = wordVec.wv[word]wordEmbedding.append(vector)except:print(word + "不存在于词向量中")wordEmbedding.append(np.random.randn(dim))return np.array(wordEmbedding)

word2vec加载异常解决：UnicodeDecodeError: ‘utf-8‘ codec can‘t decode bytes in position。。。相关推荐

解决python偶尔读文件报错：UnicodeDecodeError: ‘utf-8‘ codec can‘t decode bytes in position 1022-1023: unex....
完整报错为:UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 1022-1023: unexpected end of ...
pycharm运行异常 UnicodeDecodeError: ‘utf-8‘ codec can‘t decode bytes in position 600-601: invalid contin
pycharm运行异常 UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 600-601: invalid contin ...
处理UnicodeDecodeError: ‘XXX' codec can't decode bytes in position...的问题
错误信息: UnicodeDecodeError: 'XXX' codec can't decode bytes in position 2-5: illegal multibyte sequence ...
pandas read_csv ‘utf-8‘ codec can‘t decode bytes in position 1198-1199: invalid continuation byte解决
pandas read_csv 'utf-8' codec can't decode bytes in position 1198-1199: invalid continuation byte解决 ...
成功解决Python中导出图片出现错误SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position
成功解决Python中导出图片出现错误SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position ...
UnicodeDecodeError: ‘utf-8‘ codec can‘t decode bytes in position 5098-5099: invalid continuatio byte
问题描述读取数据集(.csv格式)时遇到如下错误: UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 5098-509 ...
UnicodeDecodeError: ‘utf-8‘ codec can‘t decode bytes in position 708-709: invalid continuation byte
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 708-709: invalid continuation byte
成功解决SyntaxError: (unicode error) ‘unicodeescape‘ codec can‘t decode bytes in position 6-7: malformed
成功解决SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 6-7: malformed ...
成功解决SyntaxError: (unicode error) ‘unicodeescape‘ codec can‘t decode bytes in position 0-1: malformed
成功解决SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 0-1: malformed ...

word2vec加载异常解决：UnicodeDecodeError: ‘utf-8‘ codec can‘t decode bytes in position。。。

word2vec加载异常解决：UnicodeDecodeError: ‘utf-8‘ codec can‘t decode bytes in position。。。相关推荐

最新文章

热门文章