使用gensim训练word2vec

import logging
import randomimport numpy as np
import torchlogging.basicConfig(level=logging.INFO, format='%(asctime)-15s %(levelname)s: %(message)s')# set seed
seed = 666
random.seed(seed)
np.random.seed(seed)
torch.cuda.manual_seed(seed)
torch.manual_seed(seed)# split data to 10 fold
fold_num = 10
data_file = '../data/train_set.csv'
import pandas as pddef all_data2fold(fold_num, num=10000):fold_data = []f = pd.read_csv(data_file, sep='\t', encoding='UTF-8')texts = f['text'].tolist()[:num]labels = f['label'].tolist()[:num]total = len(labels)index = list(range(total))np.random.shuffle(index)all_texts = []all_labels = []for i in index:all_texts.append(texts[i])all_labels.append(labels[i])label2id = {}for i in range(total):label = str(all_labels[i])if label not in label2id:label2id[label] = [i]else:label2id[label].append(i)all_index = [[] for _ in range(fold_num)]for label, data in label2id.items():# print(label, len(data))batch_size = int(len(data) / fold_num)other = len(data) - batch_size * fold_numfor i in range(fold_num):cur_batch_size = batch_size + 1 if i < other else batch_size# print(cur_batch_size)batch_data = [data[i * batch_size + b] for b in range(cur_batch_size)]all_index[i].extend(batch_data)batch_size = int(total / fold_num)other_texts = []other_labels = []other_num = 0start = 0for fold in range(fold_num):num = len(all_index[fold])texts = [all_texts[i] for i in all_index[fold]]labels = [all_labels[i] for i in all_index[fold]]if num > batch_size:fold_texts = texts[:batch_size]other_texts.extend(texts[batch_size:])fold_labels = labels[:batch_size]other_labels.extend(labels[batch_size:])other_num += num - batch_sizeelif num < batch_size:end = start + batch_size - numfold_texts = texts + other_texts[start: end]fold_labels = labels + other_labels[start: end]start = endelse:fold_texts = textsfold_labels = labelsassert batch_size == len(fold_labels)# shuffleindex = list(range(batch_size))np.random.shuffle(index)shuffle_fold_texts = []shuffle_fold_labels = []for i in index:shuffle_fold_texts.append(fold_texts[i])shuffle_fold_labels.append(fold_labels[i])data = {'label': shuffle_fold_labels, 'text': shuffle_fold_texts}fold_data.append(data)logging.info("Fold lens %s", str([len(data['label']) for data in fold_data]))return fold_datafold_data = all_data2fold(10)# build train data for word2vec
fold_id = 9train_texts = []
for i in range(0, fold_id):data = fold_data[i]train_texts.extend(data['text'])logging.info('Total %d docs.' % len(train_texts))logging.info('Start training...')
from gensim.models.word2vec import Word2Vecnum_features = 100     # Word vector dimensionality
num_workers = 8       # Number of threads to run in paralleltrain_texts = list(map(lambda x: list(x.split()), train_texts))
model = Word2Vec(train_texts, workers=num_workers, size=num_features)
model.init_sims(replace=True)# save model
model.save("./word2vec.bin")# load model
model = Word2Vec.load("./word2vec.bin")# convert format
model.wv.save_word2vec_format('./word2vec.txt', binary=False)

自然语言处理实践Task5相关推荐

python documents in chinese_基于 Python 的简单自然语言处理实践
基于 Python 的简单自然语言处理 Twenty News Group 语料集处理 20 Newsgroup 数据集包含了约 20000 篇来自于不同的新闻组的文档,最早由 Ken Lang 搜集 ...
DataWhale第21期组队学习自然语言处理实践（知识图谱）task4— 用户输入-＞知识库的查询语句
参考来源:https://github.com/datawhalechina/team-learning-nlp/blob/master/KnowledgeGraph_Basic/task04.md# ...
r与python自然语言处理_Python自然语言处理实践: 在NLTK中使用斯坦福中文分词器 | 我爱自然语言处理...
斯坦福大学自然语言处理组是世界知名的NLP研究小组,他们提供了一系列开源的Java文本分析工具,包括分词器(Word Segmenter),词性标注工具(Part-Of-Speech Tagger), ...
《自然语言处理实践——聊天机器人技术原理与应用》（王昊奋）简要笔记（全）
第一章概述 1.对话系统分类基于实现方式--检索式.生成式基于功能--问答系统.面向任务的对话系统.闲聊系统.主动推荐系统. 2.框架自动语音识别--自然语言理解--对话管理(Web,社区问答 ...
PyTorch深度学习实战 | 迁移学习与自然语言处理实践
01.文章任务从提供的金融文本中识别出现的未知金融实体,包括金融平台名.企业名.项目名称及产品名称.持有金融牌照的银行.证券.保险.基金等机构.知名的互联网企业如腾讯.淘宝.京东等和训练集中出现的实 ...
自然语言处理实践Task6
基于Bert的文本分类 Bert Pretrain class WhitespaceTokenizer(object):"""WhitespaceTokenizer wi ...
自然语言处理实践Task4
FastText FastText通过Embedding层将单词映射到稠密空间,然后将句子中所有的单词在Embedding空间中进行平均,进而完成分类操作. FastText是一个三层的神经网络,输入 ...
自然语言处理实践Task3
One-hot 这里的One-hot与数据挖掘任务中的操作是一致的,即将每一个单词使用一个离散的向量表示.具体将每个字/词编码一个索引,然后根据索引进行赋值. One-hot表示方法的例子如下: 句子 ...
自然语言处理实践Task2
一.读取训练数据 # sep设置分割符 # nrows设置读取的行数 train_df = pd.read_csv('./data/train_set.csv', sep='\t', nrows=10 ...

自然语言处理实践Task5

使用gensim训练word2vec

自然语言处理实践Task5相关推荐

最新文章

热门文章