As a news addict, I love seeing how politics can garner such emotive responses across social media and wondered if this anecdotal sense of passion could translate into machine learning classification. I found a dataset of Tweets made in reaction to the first Republican Presidential debate in 2016 (here) and wanted to create a three level sentiment classifier that could interpret emotions from the text of the Tweets. This article is part of a suite of methodologies and techniques I put together, for now I will just be focusing on one aspect; the humble Support Vector Machine. As a secondary task, I noticed the dataset was severely imbalanced so wanted to try and upsample the minority classes in an effort to improve the usefulness of the classifier across all labels (this will hopefully help the classifier improve across all categories).

作为一名新闻迷，我喜欢看到政治如何在社交媒体上获得这种情感React，并想知道这种轶事的热情是否可以转化为机器学习分类。 我发现鸣叫的数据集，React在2016年(做的第一共和党总统辩论^ h ERE)，想创建一个三个层次的情感分类，可以从推文的文本解读的情绪。 本文是我汇总的一套方法论和技术的一部分，现在我将只关注一个方面。 卑微的支持向量机。 作为第二项任务，我注意到数据集严重失衡，因此希望尝试对少数类进行上采样，以提高所有标签上分类器的实用性(这有望帮助分类器在所有类别中进行改进)。

数据探索与清理 (Data Exploration & Cleaning)

From looking at the raw figures in the breakdown of the dataset we immediately see an issue in the spread of the Tweets. Negative Tweets are prevalent and are over twice the rate of neutral and positive Tweets combined. This could have ramifications on how well the classifier works in practise. (Most likely, if trained like this, the classifier will be great at understanding a negative tweet but won’t have much practise identifying anything else!)

通过查看数据集细目中的原始数据，我们立即发现Tweets的传播存在问题。 负面推文很普遍，是中性和正面推文相加比率的两倍以上。 这可能会对分类器在实践中的工作效果产生影响。 (最有可能的是，如果经过这样的训练，分类器将非常擅长理解负面推文，但没有太多识别其他东西的练习！)

The first step of ‘cleaning’ the data was to convert all letters to lowercase, then punctuation, numbers and URLs and usernames were removed from the Tweets.

“清理”数据的第一步是将所有字母都转换为小写，然后从推文中删除标点符号，数字，URL和用户名。

Stop-words were removed from the tweets using the ‘NLTK’ stop-words corpus and white space was taken from the tweets, and each word was tokenised so as to represent an individual chunk of data to be considered as such.

使用“ NLTK”停用词语料从推文中删除停用词，并从这些推文中删除空格，并对每个词进行标记，以表示要视为其的单个数据块。

Duplicate tweets were then omitted. A decision was made to remove duplicates after the other pre-processing steps as, due to the nature of Twitter which consists of ‘retweets’, replies to other users or remarksusernames may have the exact same content. This left 9,836 total unique tweets prepared for classification. Negative: 5692, Neutral: 2521 and Positive: 1623. The dataset was split into 80% for training and 20% for testing

重复的推文则被省略。 决定在其他预处理步骤之后删除重复项，因为由于Twitter由“转发”组成的性质，对其他用户的答复或备注用户名可能具有完全相同的内容。 剩下的9,836条独特的tweet可供分类。 负数：5692，中性：2521，正数：1623。该数据集分为80％用于训练和20％用于测试

向量化— TF / IDF(Vectorisation — TF/IDF)

For the purposes of most mathematical modelling performed on text and for the purposes of this experimentation different processes of ‘vectorisation’ were implemented.

出于对文本执行大多数数学建模的目的以及出于本实验的目的，实施了不同的“矢量化”过程。

Text content alone is not capable of being altered and coerced into mathematical space without being transitioned into numbers for the purposes of being read by a machine learning algorithm.

文本内容不能单独更改和强制转换为数学空间，而不能转换为数字，以供机器学习算法读取。

That is why for the purposes of the supervised methods in this project different types of vectorisation were used to convert qualitative data to quantitative data in order for it to be mathematically manipulated. These vectors become embedded features for the models.

这就是为什么出于本项目中受监督方法的目的，使用了不同类型的矢量化将定性数据转换为定量数据，以便对其进行数学处理的原因。 这些向量成为模型的嵌入式功能。

Term Frequency/Inverse Document Frequency (TF/IDF)

术语频率/文档反向频率(TF / IDF)

This was the vectorisation technique used for the Support Vector Machine model. TF/IDF was deployed on the training data with a unigram approach which counts each individual word as a term. ‘Term frequency’ amounts to how frequently a certain word appears in the text, ‘inverse document frequency’ refers to reducing the significance of words which appear most often across all of the text.

这是用于支持向量机模型的向量化技术。 TF / IDF通过会标方法将训练数据部署到训练数据中，该方法将每个单词都算作一个术语。 “词频”等于某个单词在文本中出现的频率，“反向文档频率”是指降低在整个文本中最频繁出现的单词的重要性。

This serves to make words that are seen frequently in a given document but not necessarily all of the documents.

这有助于使单词在给定文档中经常出现，但不一定在所有文档中都可以看到。

数据平衡和采样技术 (Data Balancing & Sampling Techniques)

Data balancing in the form of Random Oversampling, Synthetic Minority Oversampling and a real world unbalanced method were all utilised and compared.

随机均衡，合成少数群体超采样和现实世界中的非均衡方法等形式的数据均衡都得到了利用和比较。

Sampling Techniques

采样技术

It is crucial to take data balancing issues and protocols into consideration as every action should be undertaken to reduce bias and increase true performance but also try and reduce overfitting and have a more nuanced representation of the model’s potential. It is important to consider upsampling techniques as it can make it easier for models to outline its decision boundary. It was decided not to use under sampling techniques as it was felt this would do little to improve performance in this case due to the fact the dataset was quite small initially and was again reduced further due to preprocessing measures and training splits.

考虑到数据平衡问题和协议是至关重要的，因为应该采取一切措施来减少偏差和增加真实性能，同时还要尝试减少过拟合并更细化模型的潜力。 考虑上采样技术非常重要，因为它可以使模型更容易勾勒出决策边界。 决定不使用采样技术，因为在这种情况下，由于实际上数据集非常小，并且由于预处理措施和训练分裂而再次减少，因此感觉无法提高性能。

No sampling

没有采样

Classes may be left unbalanced with models being trained on exactly how the tweets would appear in a real life context. It would be naive to assume that models may not perform as well without upsampling previously. If data in a given domain are naturally severely unbalanced, then training on unbalanced data may produce optimal outputs.

通过在实际环境中准确地显示推文的方式来训练模型，可能会使类失去平衡。 假设模型没有事先进行过采样可能会表现不佳，这是天真的想法。 如果给定域中的数据自然严重不平衡，那么对不平衡数据进行训练可能会产生最佳输出。

Random Over Sampling

随机过采样

Random over-sampling is a process of taking duplicate examples from the two minority classes and adding these to the training set. Examples are chosen at random from the minority classes in the training set and then duplicated and added to the training set where they have the potential to be chosen again.

随机过采样是从两个少数族裔类别中提取重复示例并将其添加到训练集中的过程。 从训练集中的少数几个类别中随机选择示例，然后将其复制并添加到训练集中，以便重新选择它们。

Because the duplicates are exact and there is the potential for duplicate examples to appear multiple times, there is a risk of overfitting the minority classes with this approach and for models implementing this technique to suffer from increased generalisation of data.

由于重复项是精确的，并且重复示例可能多次出现，因此存在使用这种方法过度拟合少数类的风险，并且存在实施该技术的模型可能会遭受数据泛化增加的风险。

For the purposes of this experiment, the minority classes were both up-sampled to the same value as the majority negative class so that each class had 5,692 examples after upsampling was applied.

为了本实验的目的，少数类都被上采样到与多数否定类相同的值，因此在应用上采样后每个类都有5,692个示例。

Synthetic Minority Oversampling Technique SMOTE

综合少数民族过采样技术

Researchers; Chawla, Bowyer, Hall, and Kegelmeyer created this upsampling technique in their paper named for the technique titled “SMOTE: Synthetic Minority Over-sampling Technique.”(read it here!)

研究人员； Chawla，Bowyer，Hall和Kegelmeyer在他们的论文中创建了这种上采样技术，该论文的标题为“ SMOTE：合成少数群体过采样技术”。(在此处阅读！ )

SMOTE is another useful upsampling method. As opposed to random oversampling which creates exact duplicates of data points from the minority class, SMOTE uses a type of data augmentation in order to ‘synthesise’ completely new and unique examples. SMOTE is implemented by choosing instances that are close to each other in the feature space and creating a boundary between these examples and creating a new sample at a certain point along that boundary. This approach tends to be effective, as the new synthetic tweets are closer to other examples in the feature space so are potentially closer in their polarity than the randomly up-sampled examples and because they are not exactly the same examples as in random over sampling, the likelihood of overfitting is reduced.

SMOTE是另一种有用的上采样方法。 与随机过采样会从少数类中创建数据点的确切副本相反，SMOTE使用一种数据扩充来“合成”全新的独特示例。 通过在要素空间中选择彼此靠近的实例并在这些实例之间创建边界并在沿该边界的特定点处创建新样本来实现SMOTE。 这种方法趋于有效，因为新的合成推文在特征空间中更接近于其他示例，因此其极性可能比随机上采样的示例更接近极性，并且由于它们与随机过采样的示例并不完全相同，降低了过度拟合的可能性。

评价 (Evaluation)

Experimental evaluation metrics vary and are usually dependent on the nature of the task being conducted. Some of the typically used evaluation metrics for analytics for analytical procedures include, but are not limited to; accuracy, precision, recall, Mean Squared Error, analysis of the Loss Function, Area Under the Curve, F1-Score. Different models in different domains will result in different results for each metric and suitable ones must be decided upon and must meet evaluation criteria necessary.

实验评估指标各不相同，通常取决于所执行任务的性质。 用于分析过程分析的一些通常使用的评估指标包括但不限于； 精度，精度，召回率，均方误差，损失函数分析，曲线下面积，F1-得分。 不同领域中的不同模型将导致每个指标的结果不同，因此必须确定合适的模型并且必须满足必要的评估标准。

Accuracy

准确性

Accuracy is one of the most frequently measured evaluation metrics in classification tasks and is most often defined as the number of correctly classified labels in proportion to the number of predictions in total.

准确性是分类任务中最经常测量的评估指标之一，最经常定义为与总预测数量成比例的正确分类标签的数量。

F1-Score

F1-分数

The ‘F1 Score’, ‘F-Score’ or the ‘F-Measure’, is a common metric used for the evaluation of Natural Language based tasks. It is often said to be ‘The Harmonic Mean of Precision and Recall’, or conveys the balance between precision and recall.

“ F1分数”，“ F分数”或“ F度量”是用于评估基于自然语言的任务的常用指标。 通常将其称为“精确度和召回率的谐波手段”，或传达精确度和召回率之间的平衡。

The F-Measure expresses the balance between the precision and the recall. As accuracy only gives the percentage of correct results of the model but does not show how adept the model is at finding true positive results, both measures have merit, depending on the need.

F度量表示精度和查全率之间的平衡。 由于准确性仅给出模型正确结果的百分比，而没有显示模型在找到真实阳性结果方面的熟练程度，因此这两种方法都有其优点，具体取决于需求。

Receiver Operating Characteristic (ROC)

接收器工作特性(ROC)

The ROC is a graph which displays the performance of a classification model in terms of its True Positive Rate (TPR) and its False Positive Rate (FPR). TPR is defined as the collective number of true positives output by the model divided by the number of true positives plus the total number of false negatives.

ROC是一个图形，以其真实肯定率(TPR)和错误真实率(FPR)显示分类模型的性能。 TPR定义为模型输出的真实阳性的总数量除以真实阳性的数量加上错误阴性的总数。

Area Under the Curve (AUC)

曲线下面积(AUC)

The AUC statistic is a measure of the dimensional space underneath the ROC curve. This figure gives the aggregated score of model performance across all of the potential classification thresholds. An interpretation of this would be that the model positions a positive random example higher than an example which is randomly negative. The AUC is always a figure between 0 and 1.

AUC统计量是ROC曲线下方维空间的度量。 该图给出了所有潜在分类阈值的模型性能的综合得分。 对此的解释是，模型将正随机示例的位置设置为高于随机负示例的位置。 AUC始终是介于0和1之间的数字。

The ROC metrics is useful as it has invariance to prior class probabilities or class prevalence in the data, along with the AUC. This is important for this study as the classes are severely unequal. The large presence of the negative class indicates the probability of the models correctly classifying a positive tweet randomly is increased.

ROC指标非常有用，因为它与数据中的先前类别概率或类别普遍性以及AUC保持不变。 这对于本研究非常重要，因为班级严重不平等。 否定类别的大量存在表示模型正确分类正推文的可能性增加了。

结果 (Results)

The following is the results obtained from the Support Vector Machine trained with Term Frequency/Inverse Document Frequency vectors with various oversampling techniques to the minority classes.

以下是从支持向量机获得的结果，该支持向量机使用术语频率/文档反向频率向量进行了训练，并采用了多种过采样技术，对少数群体进行了分类。

The figure below shows the results for the Support Vector Machine model, trained on unbalanced training data. The overall accuracy of the model here is 60% but looking at the precision, recall and f1 score for this approach we see how the model has poor performance when categorising the smaller classes here. The model understands the negative class but fails to learn much from the smaller classes as is clear from a quite low 18% and 19% ‘f1’ for the neutral and positive classes.

下图显示了在不平衡训练数据上训练的支持向量机模型的结果。 此处模型的总体准确性为60％，但从这种方法的精度，召回率和f1得分来看，我们看到在对较小的类进行分类时，模型的性能如何较差。 该模型可以理解否定类，但不能从较小的类中学到很多东西，因为对于中性和肯定类，较低的18％和19％的“ f1”很明显。

From the ROC curve and AUC shown below we see a more rounded perspective of the model’s performance. The true positive rate of all three classes was almost the same except for a 1% lower AUC for neutral compared to the other classes. The model’s overall ability at classification is not particularly apparent even though the model has an overall accuracy of 60% true positive rate.

从下面显示的ROC曲线和AUC，我们可以更全面地了解模型的性能。 这三个类别的真实阳性率几乎相同，除了中性的AUC比其他类别低1％。 即使模型的整体准确性为60％真实阳性率，模型的整体分类能力也不是特别明显。

From the confusion matrix shown we see the actual predicted values of the SVM model. The model clearly shows its best performance when classifying the negative class.

从所示的混淆矩阵中，我们可以看到SVM模型的实际预测值。 在对否定类别进行分类时，该模型清楚地显示了其最佳性能。

1086 correct predictions in this class. However, we see only 37 correct predictions of the negative class which is slightly less than 10% of correct predictions here. The neutral class has 55 correct predictions, a slight improvement on the negative class here. It is interesting to note how this model incorrectly labeled a tweet as negative substantially more than any other class. Showing how the model relied heavily on the negative class to influence its decision making.

1086这个班级的正确预测。 但是，我们只看到37个否定类别的正确预测，略小于此处正确预测的10％。 中性阶层有55个正确的预测，在这里，负面类别稍有改善。 有趣的是，该模型如何将一条推文错误地标记为负面信息，而比其他任何类别都多。 说明模型如何严重依赖否定类来影响其决策。

Support Vector Machine TF/IDF Randomly Oversampled Classes

支持向量机TF / IDF随机过采样类

The classification report displayed below shows the results of the SVM TF/IDF model when the random oversampling technique is applied to up-sample the minority classes. It is noteworthy that the overall accuracy for this approach does not differ from the same approach with unbalanced classes but the models performance in correctly classifying the smaller classes does improve slightly, as shown by the improved f1-score on the negative classes.

下面显示的分类报告显示了使用随机过采样技术对少数类进行上采样时，SVM TF / IDF模型的结果。 值得注意的是，此方法的总体准确性与不平衡类的相同方法没有区别，但是正确分类较小类的模型性能确实有所改善，如否定类上改进的f1得分所示。

Shown below is a representation of the ROC curve and AUC figure for the SVM model with random oversampling. The true positive rate of this model across all classes is an improvement of at least 4% across all classes. Not only did the model improve its classification of negative classes by increasing samples of the minority classes but the model improved considerably across all classes. The model is still best at finding the negative class, but it also did not lose any of this knowledge when presented with more diverse training examples.

下面显示的是具有随机过采样的SVM模型的ROC曲线和AUC图。 该模型在所有类别中的真实阳性率在所有类别中至少提高了4％。 该模型不仅通过增加少数群体的样本来改善其否定类别的分类，而且该模型在所有类别中都得到了很大的改善。 该模型仍然最适合查找否定类，但是当提供更多不同的训练示例时，它也不会丢失任何此类知识。

The figure below shows a confusion matrix for the SVM ROS model. It is noteworthy that the classifier is marginally worse at correctly classifying the negative when compared with the unbalanced dataset (1086 versus 977 respectively) but almost doubles its correct classification of the positive class (37 versus 70). The number of correct predictions of the neutral class improved considerably (from 55 to 140). It is also relevant to point out that the total number of incorrectly classified negative examples was reduced significantly.

下图显示了SVM ROS模型的混淆矩阵。 值得注意的是，与不平衡数据集(分别为1086和977)相比，分类器在正确分类负数方面稍差一些，但对正分类的正确分类几乎翻了一番(37对70)。 中性类别的正确预测数大大提高(从55增至140)。 还需要指出的是，错误分类的阴性样本总数大大减少了。

Support Vector Machine TF/IDF SMOTE

支持向量机TF / IDF SMOTE

The figure below shows the classification report of the SVM-TF/IDF with the SMOTE upsampling technique applied. The overall accuracy of the model remains static at 60% however we do see an improved f1 score for the two minority classes when compared to the unbalanced approach but not the randomly up-sampled method.

下图显示了使用SMOTE上采样技术的SVM-TF / IDF的分类报告。 该模型的整体准确性保持60％的静态不变，但是与不平衡方法相比，我们确实看到两个少数族裔的f1得分有所提高，但随机上采样方法却没有。

The figure below displays the ROC curve and AUC number for the SVM with SMOTE. It is clear by comparing the two graphs and metrics that follow, that there is a clear drop in true positive numbers when compared to the ROS approach. This approach is only marginally improved when assessed next to the unbalanced approach. Classification of the negative class had an improvement of 2% when compared to no upsampling and generally the model is between 1% to 2% less capable of classifying any of the classes.

下图显示了具有SMOTE的SVM的ROC曲线和AUC编号。 通过比较下面的两个图表和指标，可以清楚地看到，与ROS方法相比，真实的正数有明显的下降。 仅在不平衡方法旁边进行评估时，才稍微改善此方法。 与没有上采样相比，否定类别的分类提高了2％，通常该模型对任何类别进行分类的能力要低1％至2％。

Finally, the plot below shows the confusion matrix for the SVM with SMOTE. The negative class is still the label that the classifier correctly identities but it is interesting to note how the correct predictions for the neutral class drop almost by half when using this technique compared to ROS (72 versus 140). The classifier here also misclassifies Tweets as negative a lot more using this technique compared to ROS. Rather than diversifying the range of predictions that it makes, the classifier relies heavily on the negative class label in this instance. It is also worth noting how this model not only misclassified the positive class more than the ROS model but makes far less total predictions for this label.

最后，下图显示了带有SMOTE的SVM的混淆矩阵。 否定类别仍然是分类器正确识别的标签，但有趣的是，与ROS相比，使用此技术时，对中性类别的正确预测几乎下降了一半(72对140)。 与ROS相比，此处的分类器还使用此技术将Tweets误分类为负数。 在这种情况下，分类器没有使预测的范围多样化，而是严重依赖于否定类标签。 还值得注意的是，该模型不仅比ROS模型对正分类的错误分类更多，而且对标签的总预测要少得多。

评估与结论(Evaluation & Conclusion)

The Support Vector Machine found it extremely difficult to make correct classifications when trained on imbalanced training data. Not using a parameter tuning technique and just a simple linear approach may have caused issues also.

支持向量机发现，在不平衡的训练数据上进行训练时，很难进行正确的分类。 不使用参数调整技术，而只是简单的线性方法也可能引起问题。

SVM’s are sensitive to imbalanced data and work best with naturally balanced classes. That may have caused decreased performance. It also explains how the unbalanced experimentation yielded less useful results.

SVM对不平衡的数据很敏感，并且在自然平衡的类中效果最佳。 这可能导致性能下降。 它还说明了不平衡的实验如何产生较少有用的结果。

In terms of overall accuracy in a general sense, all three upsampling techniques gave the same accuracy metrics but it is clear intuitively that the best performance was consistently on labelling the negative class.

就一般意义上的总体准确性而言，所有三种上采样技术都给出了相同的准确性指标，但从直觉上可以清楚地看出，最佳结果始终是在标记否定类别上。

Referring to the ‘F1-Measure’ of the classifier, the Randomly upsampled model gave the best results. However it must be remembered that randomly oversampling data creates exact recreations of instances, which has the potential to lead to overfitting.

参照分类器的“ F1-Measure”，随机上采样模型给出了最佳结果。 但是，必须记住，随机过采样的数据会创建实例的精确再现，这有可能导致过拟合。

Jupyter Notebooks with all of the python code to accompany this report can be found on the GitHub repo here! :)

可以在此处的GitHub存储库中找到Jupyter Notebooks及其所有python代码随附于此报告！ :)

Report and code made by Alan Coyne, Freelance Data Scientist based in Dublin, Ireland

由爱尔兰都柏林的自由数据科学家Alan Coyne编写的报告和代码