Kaggle情绪分类比赛

学习nlp已经有一段日子了，也是反复看一些内容，遗忘真是可怕，想着去Kaggle练练手，也是新手入门Kaggle文本分类，大神们请略过吧。对数据进行了简单的常规处理，用了个2层的bi-lstm的简单模型，之后再改进改进用其他模型试验下效果。代码如下：

import pandas as pd
from keras.layers import Dense,LSTM,Bidirectional,Embedding
from keras.models import Sequential
import keras.preprocessing as preprocessing
from nltk.corpus import stopwords
from keras.utils.np_utils import to_categorical
import nltk
import matplotlib.pyplot as plt
import os
import re
import numpy as np
from sklearn.model_selection import train_test_split
from keras.preprocessing.text import Tokenizer
from nltk import WordNetLemmatizer,word_tokenize
stoplist = stopwords.words('english')
data_train = pd.read_csv(r'D:\Kaggle\train.tsv',sep='\t')
data_test = pd.read_csv(r'D:\Kaggle\test.tsv',sep='\t')
sub = pd.read_csv(r'D:\Kaggle\sampleSubmission.csv')
data_train_X = data_train.Phrase.values
X_test = list(data_test.Phrase.values)
data_train_Y = list(data_train.Sentiment.values)
lemmat = WordNetLemmatizer()
def clean(data):data = [re.sub('[^a-zA-Z0-9]',' ',word) for word in  data] #去非英文字符data_x = []word_set = set()data_emp = []for n,i in enumerate(data):data_word = word_tokenize(i) #分词data_word = [word for word in data_word if word not in stoplist ] #去停用词if data_word == []:data_emp.append(n) #找到停用词空列表的indexelse:data_word = [lemmat.lemmatize(word.lower()) for word in data_word] #去时态语态for j in data_word:word_set.add(j)data_word = ' '.join(data_word)data_x.append(data_word)word_count = len(word_set) #统计词的个数return data_x,data_emp,word_count
X_train,X_val,Y_train,Y_val = train_test_split(data_train_X,data_train_Y,test_size=0.3,random_state=123)#切分数据
X_train,X_emp,X_train_count = clean(X_train)
X_val,X_val_emp,X_val_count = clean(X_val)
Y_train = [Y_train[i] for i in range(len(Y_train)) if i not in X_emp] #去掉空列表的分类项
Y_val = [Y_val[i] for i in range(len(Y_val)) if i not in X_val_emp]
Y_train = to_categorical(Y_train)
Y_val = to_categorical(Y_val)
def word_maxlen(data_x): #统计最长的字符长度len_list = []for i in data_x:i = word_tokenize(i)len_list.append(len(i))maxlen = max(len_list)return maxlen
X_train_maxlen = word_maxlen(X_train)
X_val_maxlen = word_maxlen(X_val)
X_test_maxlen = word_maxlen(X_test)
tokenizer = Tokenizer(num_words=X_train_count) #创建分词器
tokenizer.fit_on_texts(X_train)
X_train = tokenizer.texts_to_sequences(X_train)
X_train = preprocessing.sequence.pad_sequences(X_train,maxlen=X_test_maxlen) #截断和补全
X_val = tokenizer.texts_to_sequences(X_val)
X_val = preprocessing.sequence.pad_sequences(X_val,maxlen=X_test_maxlen)
word_index  = tokenizer.word_index #找回索引
glove_base = r'D:\glove'
glove_dir = os.path.join(glove_base,'glove.6B.100d.txt') #使用预训练好的词向量
emb_vec_dict = {}
with open(glove_dir,'r',encoding='utf8') as f:for line in f:line = line.split()word = line[0]index = np.asarray(line[1:],dtype='float32')emb_vec_dict[word] = index
emd_dim = 100
emb_mat = np.zeros((X_train_count,emd_dim))
for word,i in word_index.items():if i < X_train_count:emb_vec = emb_vec_dict.get(word)if emb_vec is not None:emb_mat[i] = emb_vec
tokenizer = Tokenizer(num_words=X_train_count)
tokenizer.fit_on_texts(X_test)
X_test = tokenizer.texts_to_sequences(X_test)
X_test = preprocessing.sequence.pad_sequences(X_test,maxlen=X_test_maxlen)
model = Sequential()
model.add(Embedding(X_train_count,emd_dim,input_length=X_test_maxlen)) #嵌入到 （X_train_count,emd_dim）的矩阵中
model.add(Bidirectional(LSTM(32,dropout=0.2,recurrent_dropout=0.2,return_sequences=True)))
model.add(Bidirectional(LSTM(32,dropout=0.2,recurrent_dropout=0.2)))
model.add(Dense(5,activation='softmax'))
model.layers[0].set_weights([emb_mat]) #预训练的词嵌入加载到embedding层中
model.layers[0].trainable = False #冻结embedding层
model.compile(optimizer='adam',loss='categorical_crossentropy',metrics=['acc'])
history = model.fit(X_train,Y_train,batch_size=32,epochs=3,validation_data=[X_val,Y_val])

Kaggle情绪分类比赛相关推荐

人脸表情分类与识别：opencv人脸检测+Keras情绪分类
本次讲述的表情分类是识别的分析流程分为: 1.加载pre-model网络与权重: 2.利用opencv的函数进行简单的人脸检测: 3.抠出人脸的图并灰化: 4.表情分类器检测 . 一.表情数据集主要 ...
keras系列︱人脸表情分类与识别：opencv人脸检测+Keras情绪分类（四）
人脸识别热门,表情识别更加.但是表情识别很难,因为人脸的微表情很多,本节介绍一种比较粗线条的表情分类与识别的办法. Keras系列: 1.keras系列︱Sequential与Model模型.kera ...
疫情期间网民情绪识别比赛分享+top1~3解决方案
0.前言 3.4月份的时候参加了datafounction的情绪识别比赛,最终获得了B榜第46名的成绩.虽然不太理想,但还是想记录一下自己的参赛历程,学习一下前排大佬的解决方案,在这里分享一下,希望对 ...
文本分类（一）EWECT微博情绪分类大赛第三名Bert-Last_3embedding_concat最优单模型复现
tensorflow2.0 + transformers EWECT微博情绪分类大赛第三名Bert-Last_3embedding_concat最优单模型复现前言代码部分训练结果总结迭代优化 ...
Kaggle新上比赛：Google AI发起地域包容性图像识别竞赛
(欢迎关注"我爱计算机视觉"公众号,一个有价值有深度的公众号~) 昨天Kaggle新上了一个关于图像识别的比赛,Google AI发起了地域包容性图像识别竞赛(Inclusive ...
Kaggle获奖者自述比赛求生指南：我们如何“穿越”亚马逊热带雨林
本文来自AI新媒体量子位(QbitAI) 大家好,我是思聪 · 格里尔斯,我将向您展示如何从世界上某些竞争最激烈的比赛中拿到金牌.我将面临一个月的比赛挑战,在这些比赛中缺乏正确的求生技巧,你甚至拿不到 ...
人脸表情识别和情绪分类 | Python+TensorFlow（框架）+Keras+PyQt5
人脸表情识别 | Python+Keras+PyQt5 参考学习文章: Keras|基于深度学习的人脸表情识别系统 PyQt5+QtDesigner编写摄像头界面程序(一)--pyqt5.qtdesi ...
【数据竞赛】Kaggle知识点：比赛类型介绍
在Kaggle平台上,经常有好多个比赛可供选择.如下图所示,为正在进行比赛的截图.那么对于入门同学(从没参加过一次Kaggle比赛)如何选择一场适合的比赛呢? 本文将对Kaggle平台上的比赛按照类型 ...
Kaggle新上比赛：胸部X光片肺炎检测
(欢迎关注"我爱计算机视觉"公众号,一个有价值有深度的公众号~) 今天北美放射学会(RSNA®)联合Kaggle发起了医学图像肺炎检测的比赛.医学图像中的肺炎视觉信号,表现为胸部X ...

Kaggle情绪分类比赛

Kaggle情绪分类比赛相关推荐

最新文章

热门文章