《Web安全之机器学习入门》笔记：第十六章 16.3 恶意评论识别（二）

本小节使用NB和RNN两种方法识别恶意评论。

1、数据集获取

（1）正面影评

数据集的位置：../data/movie-review-data/review_polarity/txt_sentoken/pos/

x1,y1=load_files("../data/movie-review-data/review_polarity/txt_sentoken/pos/",0)

（2）负面影评

数据集的位置：../data/movie-review-data/review_polarity/txt_sentoken/neg/

x2,y2=load_files("../data/movie-review-data/review_polarity/txt_sentoken/neg/", 1)

（3）数据获取

def load_one_file(filename):x=""with open(filename) as f:for line in f:x+=linereturn xdef load_files(rootdir,label):list = os.listdir(rootdir)x=[]y=[]for i in range(0, len(list)):path = os.path.join(rootdir, list[i])if os.path.isfile(path):print("Load file %s" % path)y.append(label)x.append(load_one_file(path))return x,ydef load_data():x=[]y=[]x1,y1=load_files("../data/movie-review-data/review_polarity/txt_sentoken/pos/",0)x2,y2=load_files("../data/movie-review-data/review_polarity/txt_sentoken/neg/", 1)x=x1+x2y=y1+y2return x,ydef main(unused_argv):global n_wordsx,y=load_data()x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.4, random_state=0)vp = learn.preprocessing.VocabularyProcessor(max_document_length=MAX_DOCUMENT_LENGTH, min_frequency=1)vp.fit(x)x_train = np.array(list(vp.transform(x_train)))x_test = np.array(list(vp.transform(x_test)))n_words=len(vp.vocabulary_)print('Total words: %d' % n_words)

打印结果

Total words: 28334

2、RNN模型

def do_rnn(trainX, testX, trainY, testY):global n_words# Data preprocessing# Sequence paddingprint("GET n_words embedding %d" % n_words)trainX = pad_sequences(trainX, maxlen=MAX_DOCUMENT_LENGTH, value=0.)testX = pad_sequences(testX, maxlen=MAX_DOCUMENT_LENGTH, value=0.)# Converting labels to binary vectorstrainY = to_categorical(trainY, nb_classes=2)testY = to_categorical(testY, nb_classes=2)# Network buildingnet = tflearn.input_data([None, MAX_DOCUMENT_LENGTH])net = tflearn.embedding(net, input_dim=n_words, output_dim=128)net = tflearn.lstm(net, 128, dropout=0.8)net = tflearn.fully_connected(net, 2, activation='softmax')net = tflearn.regression(net, optimizer='adam', learning_rate=0.001,loss='categorical_crossentropy')# Trainingmodel = tflearn.DNN(net, tensorboard_verbose=3)model.fit(trainX, trainY, validation_set=(testX, testY), show_metric=True,batch_size=32,run_id="maidou")

3、NB模型

def do_NB(x_train, x_test, y_train, y_test):gnb = GaussianNB()y_predict = gnb.fit(x_train, y_train).predict(x_test)score = metrics.accuracy_score(y_test, y_predict)print('NB Accuracy: {0:f}'.format(score))

4、完整代码

import tensorflow as tf
from tensorflow.contrib.learn.python import learn
from sklearn import metrics
from sklearn.model_selection import train_test_split
import numpy as np
from sklearn.naive_bayes import GaussianNB
import os
from sklearn.feature_extraction.text import CountVectorizer
from tensorflow.contrib.layers.python.layers import encoders
from sklearn import svm
import tflearn
from tflearn.data_utils import to_categorical, pad_sequences
from tflearn.datasets import imdbMAX_DOCUMENT_LENGTH = 200
EMBEDDING_SIZE = 50n_words=0def load_one_file(filename):x=""with open(filename) as f:for line in f:x+=linereturn xdef load_files(rootdir,label):list = os.listdir(rootdir)x=[]y=[]for i in range(0, len(list)):path = os.path.join(rootdir, list[i])if os.path.isfile(path):print("Load file %s" % path)y.append(label)x.append(load_one_file(path))return x,ydef load_data():x=[]y=[]x1,y1=load_files("../data/movie-review-data/review_polarity/txt_sentoken/pos/",0)x2,y2=load_files("../data/movie-review-data/review_polarity/txt_sentoken/neg/", 1)x=x1+x2y=y1+y2return x,ydef do_rnn(trainX, testX, trainY, testY):global n_words# Data preprocessing# Sequence paddingprint("GET n_words embedding %d" % n_words)trainX = pad_sequences(trainX, maxlen=MAX_DOCUMENT_LENGTH, value=0.)testX = pad_sequences(testX, maxlen=MAX_DOCUMENT_LENGTH, value=0.)# Converting labels to binary vectorstrainY = to_categorical(trainY, nb_classes=2)testY = to_categorical(testY, nb_classes=2)# Network buildingnet = tflearn.input_data([None, MAX_DOCUMENT_LENGTH])net = tflearn.embedding(net, input_dim=n_words, output_dim=128)net = tflearn.lstm(net, 128, dropout=0.8)net = tflearn.fully_connected(net, 2, activation='softmax')net = tflearn.regression(net, optimizer='adam', learning_rate=0.001,loss='categorical_crossentropy')# Trainingmodel = tflearn.DNN(net, tensorboard_verbose=3)model.fit(trainX, trainY, validation_set=(testX, testY), show_metric=True,batch_size=32,run_id="maidou")def do_NB(x_train, x_test, y_train, y_test):gnb = GaussianNB()y_predict = gnb.fit(x_train, y_train).predict(x_test)score = metrics.accuracy_score(y_test, y_predict)print('NB Accuracy: {0:f}'.format(score))def main(unused_argv):global n_wordsx,y=load_data()x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.4, random_state=0)vp = learn.preprocessing.VocabularyProcessor(max_document_length=MAX_DOCUMENT_LENGTH, min_frequency=1)vp.fit(x)x_train = np.array(list(vp.transform(x_train)))x_test = np.array(list(vp.transform(x_test)))n_words=len(vp.vocabulary_)print('Total words: %d' % n_words)do_NB(x_train, x_test, y_train, y_test)do_rnn(x_train, x_test, y_train, y_test)if __name__ == '__main__':tf.app.run()

5、运行结果

RNN算法结果

......
Training Step: 378  | total loss: 0.08034 | time: 15.892s
| Adam | epoch: 010 | loss: 0.08034 - acc: 0.9874 -- iter: 1152/1200
Training Step: 379  | total loss: 0.07429 | time: 16.334s
| Adam | epoch: 010 | loss: 0.07429 - acc: 0.9887 -- iter: 1184/1200
Training Step: 380  | total loss: 0.07552 | time: 18.796s
| Adam | epoch: 010 | loss: 0.07552 - acc: 0.9867| val_loss: 1.07911 - val_acc: 0.6188 -- iter: 1200/1200
--

NB算法结果

NB Accuracy: 0.493750

如上NB算法准确率仅为49.37%左右，RNN准确率仅61.88%左右。从RNN的训练准确率可达到98%以上，而测试准确率如此不理想，可见泛化性能差了些，还有调优空间

《Web安全之机器学习入门》笔记：第十六章 16.3 恶意评论识别（二）相关推荐

机器学习入门：第十六章 Hopfield神经网络
经过这些年的发展,深度学习慢慢的成为了一种较为流行的算法,从算法的一步步发展来看现在的深度学习算法会觉得深度学习算法不用于已有的神经网络或一些现在流行的算法.虽然深度学习跟神经网络非常的像,但是从原理 ...
深度学习入门笔记（十六）：计算机视觉之边缘检测
欢迎关注WX公众号:[程序员管小亮] 专栏--深度学习入门笔记声明 1)该文章整理自网上的大牛和机器学习专家无私奉献的资料,具体引用的资料请看参考文献. 2)本文仅供学术交流,非商用.所以每一部分具 ...
强化学习（RLAI）读书笔记第十六章Applications and Case Studies（不含alphago）
强化学习(RLAI)读书笔记第十六章Applications and Case Studies(不含alphago) 16.1 TD-Gammon 16.2 Samuel's Checkers Pla ...
[go学习笔记.第十六章.TCP编程] 3.项目-海量用户即时通讯系统-redis介入,用户登录,注册
1.实现功能-完成用户登录在redis手动添加测试用户,并画出示意图以及说明注意事项(后续通过程序注册用户) 如:输入用户名和密码,如果在redis中存在并正确,则登录,否则退出系统,并给出相应提示 ...
强化学习（RLAI）读书笔记第十六章Applications and Case Studies（alphago）
强化学习(RLAI)读书笔记第十六章Applications and Case Studies(alphago) 16.6 Mastering the Game of Go 16.6.1 AlphaG ...
Linux（b站视频兄弟连）自学笔记第十六章——备份与恢复
Linux(b站视频兄弟连)自学笔记第十六章--备份与恢复概述 dump和restore命令概述 dump和restore命令
《Web安全之机器学习入门》笔记：第十六章 16.7 识别异常操作
本小节通过RNN识别异常操作. 1.数据集 def load_user_cmd_new(filename):cmd_list=[]dist=[]with open(filename) as f:i=0 ...
机器学习入门-西瓜书总结笔记第十六章
西瓜书第十六章-强化学习一.任务与奖赏二.K-摇臂赌博机 1.探索与利用 2. ϵ \epsilon ϵ-贪心 3.Softmax 三.有模型学习 1.策略评估 2.策略改进 3.策略迭代与值迭代 ...
[汇编学习笔记][第十六章直接定址表]
第十六章直接定址表 16.1 描述了单元长度的标号格式 code segmenta db 1,2,3,4,5,6,7,8,b dw 0 功能此时标号a,b 不仅代表了内存单元,还代表了内存长度 ...

《Web安全之机器学习入门》笔记：第十六章 16.3 恶意评论识别（二）

《Web安全之机器学习入门》笔记：第十六章 16.3 恶意评论识别（二）相关推荐

最新文章

热门文章