【NLP文本分类】对IMDB电影评论进行情感分析

概述

对imdb中的电影评论进行分类，是一个二分类的问题，这是一种重要且广泛适用的机器学习问题。

数据

imdb的数据包含50000条电影评论。拥有25000条训练数据以及25000条评估数据，有着相同数量的正面与负面评论。

下载imdb数据

imdb中的数据已经被预处理好，为整数序列，每个整数代表着一个特定单词。可用imdb的词典进行翻译。（https://s3.amazonaws.com/text-datasets/imdb.npz）如果不能科学上网，可以在https://pan.baidu.com/s/1pNDbE3VMdYJiiXyaN2roaw 提取码：0wnn下载

读取数据

import tensorflow as tf
from tensorflow import kerasimport numpy as np
mdb = keras.datasets.imdb(train_data, train_labels), (test_data, test_labels) = imdb.load_data('/home/kesci/input/idmb2286/imdb.npz',num_words=10000)

将load_data中的路径改为imdb.npz所在的路径，num_words=15000保留出现频率最高的前10000个词。丢弃罕见单词以保持数据的可管理。

了解数据

在处理数据前，我们需要先了解数据，经过数据的预处理后，每个例子都是整数序列，以整数来表示电影的单词。每个整数对应词典的一个单词。用0和1来确定label的种类。

print("Training entries: {}, labels: {}".format(len(train_data), len(train_labels)))

Training entries: 25000, labels: 25000
我们可以看下第一条评论

print(train_data[0])

[1, 14, 22, 16, 43, 530, 973, 1622, 1385, 65, 458, 4468, 66, 3941, 4, 173, 36, 256, 5, 25, 100, 43, 838, 112, 50, 670, 2, 9, 35, 480, 284, 5, 150, 4, 172, 112, 167, 2, 336, 385, 39, 4, 172, 4536, 1111, 17, 546, 38, 13, 447, 4, 192, 50, 16, 6, 147, 2025, 19, 14, 22, 4, 1920, 4613, 469, 4, 22, 71, 87, 12, 16, 43, 530, 38, 76, 15, 13, 1247, 4, 22, 17, 515, 17, 12, 16, 626, 18, 2, 5, 62, 386, 12, 8, 316, 8, 106, 5, 4, 2223, 5244, 16, 480, 66, 3785, 33, 4, 130, 12, 16, 38, 619, 5, 25, 124, 51, 36, 135, 48, 25, 1415, 33, 6, 22, 12, 215, 28, 77, 52, 5, 14, 407, 16, 82, 10311, 8, 4, 107, 117, 5952, 15, 256, 4, 2, 7, 3766, 5, 723, 36, 71, 43, 530, 476, 26, 400, 317, 46, 7, 4, 12118, 1029, 13, 104, 88, 4, 381, 15, 297, 98, 32, 2071, 56, 26, 141, 6, 194, 7486, 18, 4, 226, 22, 21, 134, 476, 26, 480, 5, 144, 30, 5535, 18, 51, 36, 28, 224, 92, 25, 104, 4, 226, 65, 16, 38, 1334, 88, 12, 16, 283, 5, 16, 4472, 113, 103, 32, 15, 16, 5345, 19, 178, 32]

每条数据的单词数量不同，然而神经网络的输入要求长度必须相同。我们将在下面解决这个问题。

len(train_data[0]),len(train_data[1])

(218,219)

将整数转换为文本

# A dictionary mapping words to an integer index
word_index = imdb.get_word_index()# The first indices are reserved
word_index = {k:(v+3) for k,v in word_index.items()}
word_index["<PAD>"] = 0
word_index["<START>"] = 1
word_index["<UNK>"] = 2  # unknown
word_index["<UNUSED>"] = 3reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])def decode_review(text):return ' '.join([reverse_word_index.get(i, '?') for i in text])

我们可以用decode_review将整数序列转换为文本

decode_review(train_data[20])

"<START> shown in australia as <UNK> this incredibly bad movie is so bad that you become <UNK> and have to watch it to the end just to see if it could get any worse and it does the storyline is so predictable it seems written by a high school <UNK> class the sets are pathetic but marginally better than the <UNK> and the acting is wooden br br the infant <UNK> seems to have been stolen from the props <UNK> of <UNK> <UNK> there didn't seem to be a single original idea in the whole movie br br i found this movie to be so bad that i laughed most of the way through br br malcolm mcdowell should hang his head in shame he obviously needed the money"

准备数据

必须在输入神经网络前转换为张量。

可转换为独热向量
或填充数组，使他们具有相同长度，然后创建一个num_example*max_length的整型张量。可以将此作为神经网络的第一层。
在次我们使用第二种方法

train_data = keras.preprocessing.sequence.pad_sequences(train_data,value=word_index["<PAD>"],padding='post',maxlen=256)test_data = keras.preprocessing.sequence.pad_sequences(test_data,value=word_index["<PAD>"],padding='post',maxlen=256)

将数据集转换为256维，不足的从后面补齐（补零）。

len(train_data[0]), len(train_data[1])

（256，256）
看下处理后的数据

print(train_data[0])

[    1    14    22    16    43   530   973  1622  1385    65   458  446866  3941     4   173    36   256     5    25   100    43   838   11250   670     2     9    35   480   284     5   150     4   172   112167     2   336   385    39     4   172  4536  1111    17   546    3813   447     4   192    50    16     6   147  2025    19    14    224  1920  4613   469     4    22    71    87    12    16    43   53038    76    15    13  1247     4    22    17   515    17    12    16626    18     2     5    62   386    12     8   316     8   106     54  2223  5244    16   480    66  3785    33     4   130    12    1638   619     5    25   124    51    36   135    48    25  1415    336    22    12   215    28    77    52     5    14   407    16    8210311     8     4   107   117  5952    15   256     4     2     7  37665   723    36    71    43   530   476    26   400   317    46     74 12118  1029    13   104    88     4   381    15   297    98    322071    56    26   141     6   194  7486    18     4   226    22    21134   476    26   480     5   144    30  5535    18    51    36    28224    92    25   104     4   226    65    16    38  1334    88    1216   283     5    16  4472   113   103    32    15    16  5345    19178    32     0     0     0     0     0     0     0     0     0     00     0     0     0     0     0     0     0     0     0     0     00     0     0     0     0     0     0     0     0     0     0     00     0     0     0]

创建模型

 input shape is the vocabulary count used for the movie reviews (10,000 words)
vocab_size = 10000model = keras.Sequential()
model.add(keras.layers.Embedding(vocab_size, 16))
model.add(keras.layers.GlobalAveragePooling1D())
#model.add(keras.layers.GlobalMaxPooling1D())
model.add(keras.layers.Dense(16, activation=tf.nn.relu))
model.add(keras.layers.Dropout(0.5))
model.add(keras.layers.Dense(1, activation=tf.nn.sigmoid))model.summary()

损失函数

模型需要一个损失函数和一个用于训练的优化器。由于这是二元分类问题和概率模型输出（具有S形激活的单个单元层），我们将使用binary_crossentropy损失函数。

model.compile(optimizer=tf.train.AdamOptimizer(),loss='binary_crossentropy',metrics=['accuracy'])

建立验证集

10000以前为验证集，一万以后为训练集

x_val = train_data[:10000]
partial_x_train = train_data[10000:]y_val = train_labels[:10000]
partial_y_train = train_labels[10000:]

训练模型

#每一Epochs都进行F1计算
import numpy as np
from keras.callbacks import Callback
from keras.engine.training import Model
from sklearn.metrics import confusion_matrix, f1_score, precision_score, recall_score
class Metrics(Callback):def on_train_begin(self, logs={}):self.val_f1s = []self.val_recalls = []self.val_precisions = []def on_epoch_end(self, epoch, logs={}):val_predict = (np.asarray(self.model.predict(self.validation_data[0]))).round()val_targ = self.validation_data[1]_val_f1 = f1_score(val_targ, val_predict,average='weighted')_val_recall = recall_score(val_targ, val_predict,average='weighted')_val_precision = precision_score(val_targ, val_predict,average='weighted')self.val_f1s.append(_val_f1)self.val_recalls.append(_val_recall)self.val_precisions.append(_val_precision)print( ' — val_f1: %f — val_precision: %f — val_recall %f' %(_val_f1, _val_precision, _val_recall))returnmetrics = Metrics()

from keras.callbacks import EarlyStopping
earlystopping=keras.callbacks.EarlyStopping(monitor='val_acc', patience=8, verbose=0, mode='max')history = model.fit(partial_x_train,partial_y_train,epochs=90,batch_size=512,validation_data=(x_val, y_val),callbacks=[metrics,earlystopping],verbose=1)

测试模型

results = model.evaluate(test_data, test_labels)print(results)

25000/25000 [==============================] - 2s 61us/step
[0.31110355438232423, 0.87736]
我们可以看到损失函数为0.31，准确率为0.87.