I recently received a new paper titled“Evaluation of Sentiment Analysis in Finance: From Lexicons to Transformers” published on July 16 2020 in IEEE. The authors, KostadinMishev, Ana Gjorgjevikj, Irena Vodenska, Lubomir T. Chitkushev, and DimitarTrajanov compared more than a hundred sentiment algorithms that were applied on two known financial sentiment datasets and evaluated their effectiveness. Although the purpose of the study was to test the effectiveness of different Natural Language Processing (NLP) models, the findings, in the paper, can tell us much more, about the progress of NLP over the duration of the last decade, especially, to better understand what elements contributed the most to the sentiment prediction task.

我最近收到了一份新论文，题为“金融中的情感分析评估：从Lexicons到变形金刚”，于2020年7月16日在IEEE发表。 作者KostadinMishev，Ana Gjorgjevikj，Irena Vodenska，Lubomir T.Chitkushev和DimitarTrajanov比较了应用于两种已知金融情绪数据集的一百多种情绪算法，并评估了它们的有效性。 尽管研究的目的是测试不同自然语言处理(NLP)模型的有效性，但本文的发现可以告诉我们有关NLP在过去十年中的进展的更多信息，尤其是到更好地了解哪些元素对情绪预测任务的贡献最大。

So let’s start with the definition of the sentiment prediction task. Given a collection of paragraphs, the model classifies each paragraph into one of three possible categories: positive sentiment, negative sentiment, or neutral. The model is then evaluated based on a confusion matrix (3X3) that is constructed from the counts of predicted sentiment versus the ground truths (the true labels of each paragraphs).

因此，让我们从情感预测任务的定义开始。 给定一个段落集合，模型将每个段落分为三个可能的类别之一：积极情绪，消极情绪或中立。 然后，基于混淆矩阵(3X3)对模型进行评估，该混淆矩阵是根据预测的情绪与实际情况(每个段落的真实标签)的计数构建的。

The evaluation metric implemented by the authors is called the Matthews correlation coefficient (MCC) and serves as a measure of the quality of binary (two-class) classifications (Matthews,1975). Although the MCC metric is only applicable for the binary case, the authors do not mention how they applied the MCC function in the multi-class case (3 sentiment classes). Did they use micro-averaging or did they apply the generalized equation for the multi-class case?

作者实施的评估指标称为马修斯相关系数(MCC)，可作为衡量二元(两类)分类质量的指标( 马修斯 ，1975年)。 尽管MCC指标仅适用于二进制情况，但作者没有提及他们在多类情况(3个情感类)中如何应用MCC功能。 他们是使用微平均还是针对多类情况应用了广义方程？

The authors divided the NLP models into five broad categories based on their textual representation: (1) Lexicon-based knowledge, (2) statistical methods, (3) word encoder, (4) Sentence encoder, (5) transformer. Several different models were applied for each category and the performance is reported in a table here.

作者根据其文字表示将NLP模型分为五大类：(1)基于词汇的知识，(2)统计方法，(3)单词编码器，(4)句子编码器，(5)转换器。 每种类别都应用了几种不同的模型，其性能报告在此处的表格中。

The table above demonstrates the progress in sentiment analysis through the years driven by the text representation method. The authors confirm that transformers show superior performances compared to the other evaluated approaches and that the text representation plays the main role as it feeds the semantic meaning of the words and sentences into the models.

上表展示了文本表示法驱动的情感分析在过去几年中的进展。 作者确认，与其他评估方法相比，变形器表现出更出色的性能，并且文本表示法将单词和句子的语义含义馈入模型中，从而发挥了主要作用。

But wait! There are perhaps more conclusions that can be drawn from this experiment regarding the future of NLP. Can we uncover clues about the elements that are still missing to make NLP much more effective in more complex task? What might be the next big breakthrough in order to better represent human language by language models?

可是等等！ 关于NLP的未来，可以从该实验中得出更多结论。 我们是否可以找到有关使NLP在更复杂的任务中更加有效的要素的线索？ 为了通过语言模型更好地表达人类语言，下一个重大突破是什么？

In an effort solve that exact question, I have started to dig further into the models’ outcomes and search for a connection between text representation, model size, and model performance in an attempt to extracting the contribution of model’s size and text representation on the final performance. Based on the authors’ analysis I created the figure below. The Figure below shows the MCC score of each model as a function of the model’s numeric parameters. The colors represent the model main category.

为了解决这个确切的问题，我开始进一步研究模型的结果，并在文本表示，模型大小和模型性能之间寻找联系，以尝试提取模型大小和文本表示对最终结果的贡献。性能。 根据作者的分析，我创建了下图。 下图显示了每个模型的MCC得分与模型的数字参数的关系。 颜色代表模型的主要类别。

From my analysis, it can be seen that the progress of the sentiment prediction task consists of two phases. The first phase is mainly attributed to better text representation while the second phase is due to the introduction of the transformer that can handle huge corpora by increasing network size and administrating millions of parameters.

从我的分析可以看出，情绪预测任务的进度分为两个阶段。 第一阶段主要归因于更好的文本表示，而第二阶段则归因于转换器的引入，该转换器可以通过增加网络规模和管理数百万个参数来处理庞大的语料库。

It is highlight from the above graph that text representation had three major revolutions starting from the early 80s. The first was from lexicon representation to embedding vector representation. The main advantage of embedding vectors is its unsupervised nature, as it does not require any tagging while still capturing meaningful semantic relations between words and benefitting from a model’s generalization capabilities. It’s important to remember that these embedding models, such as word2vec and GloVe, are context-independent. They assign the same pertained vector to the same word regardless of the context around the word. Thus, they cannot handle polysemy or complex semantics in natural languages.

从上图可以看出，文本表示从80年代初开始经历了三大革命。 首先是从词典表示到嵌入矢量表示。 嵌入向量的主要优点是其无监督的性质，因为它不需要任何标记，同时仍可以捕获单词之间有意义的语义关系，并受益于模型的泛化能力。 重要的是要记住，这些嵌入模型(例如word2vec和GloVe)是与上下文无关的。 他们将相同的相关向量分配给同一单词，而不管该单词周围的上下文如何。 因此，他们无法处理自然语言中的多义性或复杂语义。

Then, the context-sensitive word representations introduced towards 2016 with models like ELMo and GPT. These models have vector representations with words that depend on their contexts. ELMo encodes context bidirectionally, while GPT encodes context from left to right. The main contribution of these was their ability to handle polysemy and more complex semantics.

然后，在2016年推出了上下文相关字词表示法，例如ELMo和GPT。 这些模型具有向量表示形式，其词取决于其上下文。 ELMo双向编码上下文，而GPT从左到右编码上下文。 这些的主要贡献是它们处理多义和更复杂语义的能力。

The most recent revolution in NLP is BERT (Bidirectional Encoder Representations from Transformers), which combines bidirectional context encoding and requires minimal architecture changes for a wide range of natural language-processing tasks. The embeddings of the BERT input sequence is the sum of the token embeddings, segment embeddings, and positional embeddings. BERT and the following models are unique in that they can process a batch of sequences, from 1M parameters to the latest models that reached above 500M. From the graph it can be seen that the number of parameters in the model is the main reason for the continuous performance improvement during the last 4 years.

NLP的最新革命是BERT(来自变压器的双向编码器表示)，它结合了双向上下文编码，并且对于各种自然语言处理任务，只需进行最小的体系结构更改即可。 BERT输入序列的嵌入是令牌嵌入，段嵌​​入和位置嵌入的总和。 BERT和以下模型的独特之处在于它们可以处理一系列序列，从1M参数到达到500M以上的最新模型。 从图中可以看出，模型中的参数数量是过去4年中持续改进性能的主要原因。

Although NLP models have come a long way in the recent years and made substantial progress, there is still plenty of room for improvement. According to several studies,1,2 just increasing the network size is not enough, and even today the model is in a state of overparametrization. The next breakthrough will probably come from further progress in text representation, when NLP models will be better able to capture language compositionality (the ability to learn the meaning of a larger piece of text by composing the meaning of its constituents maintaining). A good place to start looking for some ideas about new text representations is in the domain of grammar inference. By learning controlled formal grammar, we can go deeper into our understanding about the elements that should handle compositionally (Solan et al, 2005) with respect to tests, like systematically, substitutivity, productivity, localism etc. (Hupkes et al., 2019; Onnis & Edelman, 2019).

尽管NLP模型在最近几年取得了长足的进步，并取得了长足的进步，但仍有很大的改进空间。 根据数项研究1,2，仅增加网络规模是不够的，即使到今天，该模型仍处于过参数化状态。 下一个突破可能来自文本表示形式的进一步发展，届时NLP模型将能够更好地捕获语言组成性(通过组合其组成成分的含义来学习较大文本的含义的能力)。 在语法推断领域中，是一个开始寻找有关新文本表示形式的想法的好地方。 通过学习受控的形式语法，我们可以对测试应该系统地处理组成部分 (Solan等，2005)的元素有更深入的了解，例如系统地，替代性，生产率，局部性等(Hupkes等，2019; Onnis＆Edelman，2019)。

Biography

传

(1) Hupkes, D., Dankers, V., Mul, M., & Bruni, E. (2019). The compositionality of neural networks: integrating symbolism and connectionism. arXiv preprint arXiv:1908.08351.‏

(1)Hupkes，D.，Dankers，V.，Mul，M.，＆Bruni，E.(2019年)。 神经网络的组成：整合象征主义和连接主义。 arXiv预印本arXiv：1908.08351 。

(2) Kovaleva, O., Romanov, A., Rogers, A., &Rumshisky, A. (2019). Revealing the dark secrets of BERT. arXiv preprint arXiv:1908.08593.‏

(2)Kovaleva，O.，Romanov，A.，Rogers，A.，＆Rumshisky，A.(2019年)。 揭示BERT的黑暗秘密。 arXiv预印本arXiv：1908.08593 。

(3) Onnis, L., & Edelman, S. (2019). Local versus global statistical learning in language.‏

(3)Onnis，L.和Edelman，S.(2019)。 语言的本地统计学习与全球统计学习。

(4) Solan, Z., Horn, D., Ruppin, E., & Edelman, S. (2005). Unsupervised learning of natural languages. Proceedings of the National Academy of Sciences, 102(33), 11629–11634.‏

(4)Solan，Z.，Horn，D.，Ruppin，E.，＆Edelman，S.(2005年)。 自然语言的无监督学习。 美国国家科学院院刊 ， 102 (33)，11629–11634。

(5) Mishev, K., Gjorgjevikj, A., Vodenska, I., Chitkushev, L. T., & Trajanov, D. (2020). Evaluation of Sentiment Analysis in Finance: From Lexicons to Transformers. IEEE Access, 8, 131662–131682.‏

(5)Mishev，K.，Gjorgjevikj，A.，Vodenska，I.，Chitkushev，LT，＆Trajanov，D.(2020年)。 金融中情感分析的评估：从词汇到变形金刚。 IEEE访问 ，8，131662-131682。