bart使用方法

介绍 (Introduction)

BART is a denoising autoencoder for pretraining sequence-to-sequence models. BART is trained by (1) corrupting text with an arbitrary noising function, and (2) learning a model to reconstruct the original text.

BART是一种用于预训练序列到序列模型的去噪自动编码器。通过(1)使用任意噪声功能破坏文本，以及(2)学习模型以重建原始文本来训练BART。

- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension -

-BART：对自然语言生成，翻译和理解进行序列到序列的预训练降噪 -

Don’t worry if that sounds a little complicated; we are going to break it down and see what it all means. To add a little bit of background before we dive into BART, it’s time for the now-customary ode to Transfer Learning with self-supervised models. It’s been said many times over the past couple of years, but Transformers really have achieved incredible success in a wide variety of Natural Language Processing (NLP) tasks.

如果这听起来有点复杂，请不要担心。我们将对其进行分解，并查看其含义。为了在开始使用BART之前增加一些背景知识，现在是时候让传统习俗向自我监督模型转移学习了。在过去的几年中已经有很多次这样的说法，但是Transformers确实在各种自然语言处理(NLP)任务中取得了令人难以置信的成功。

BART uses a standard Transformer architecture (Encoder-Decoder) like the original Transformer model used for neural machine translation but also incorporates some changes from BERT (only uses the encoder) and GPT (only uses the decoder). You can refer to the 2.1 Architecture section of the BART paper for more details.

BART使用标准的Transformer体系结构(编码器-解码器)，就像用于神经机器翻译的原始Transformer模型一样，但是还结合了BERT(仅使用编码器)和GPT(仅使用解码器)的一些更改。您可以参考BART论文的2.1体系结构部分以获取更多详细信息。

培训前的BART (Pre-Training BART)

BART is pre-trained by minimizing the cross-entropy loss between the decoder output and the original sequence.

通过最小化解码器输出和原始序列之间的交叉熵损失来对BART进行预训练。

屏蔽语言建模(MLM) (Masked Language Modeling (MLM))

MLM models such as BERT are pre-trained to predict masked tokens. This process can be broken down as follows:

像BERT这样的MLM模型已经过预训练，可以预测被屏蔽的令牌。此过程可以细分如下：

Replace a random subset of the input with a mask token [MASK]. (Adding noise/corruption)

用掩码标记[MASK]替换输入的随机子集。 (增加噪音/腐败)
The model predicts the original tokens for each of the [MASK] tokens. (Denoising)

该模型预测每个[MASK]的原始令牌令牌。 (去噪)

Importantly, BERT models can “see” the full input sequence (with some tokens replaced with [MASK]) when attempting to predict the original tokens. This makes BERT a bidirectional model, i.e. it can “see” the tokens before and after the masked tokens.

重要的是，当尝试预测原始令牌时，BERT模型可以“看到”完整的输入序列(某些令牌被[MASK]替换)。这使BERT成为双向模型，即它可以“看到”被屏蔽令牌之前和之后的令牌。

This is suited for tasks like classification where you can use information from the full sequence to perform the prediction. However, it is less suited for text generation tasks where the prediction depends only on the previous words.

这适用于诸如分类之类的任务，您可以在其中使用完整序列中的信息来执行预测。但是，它不太适合预测仅依赖于先前单词的文本生成任务。

自回归模型 (Autoregressive Models)

Models used for text generation, such as GPT2, are pre-trained to predict the next token given the previous sequence of tokens. This pre-training objective results in models that are well-suited for text generation, but not for tasks like classification.

在给定先前标记序列的情况下，对用于文本生成的模型(例如GPT2)进行了预训练，以预测下一个标记。这种预训练目标产生的模型非常适合于文本生成，但不适用于分类等任务。

BART序列对序列 (BART Sequence-to-Sequence)

BART has both an encoder (like BERT) and a decoder (like GPT), essentially getting the best of both worlds.

BART同时具有编码器(如BERT)和解码器(如GPT)，从本质上兼顾了两者。

The encoder uses a denoising objective similar to BERT while the decoder attempts to reproduce the original sequence (autoencoder), token by token, using the previous (uncorrupted) tokens and the output from the encoder.

编码器使用类似于BERT的降噪目标，同时解码器尝试使用先前的(未破坏的)令牌和编码器的输出逐个令牌地再现原始序列(自动编码器)。

A significant advantage of this setup is the unlimited flexibility of choosing the corruption scheme; including changing the length of the original input. Or, in fancier terms, the text can be corrupted with an arbitrary noising function.

这种设置的显着优点是选择损坏方案时具有无限的灵活性。包括更改原始输入的长度。或者，更随意地说，文本可以使用任意的杂讯功能破坏。

The corruption schemes used in the paper are summarized below.

本文中使用的损坏方案摘要如下。

Token Masking — A random subset of the input is replaced with [MASK] tokens, like in BERT.令牌掩码-像BERT中一样，将输入的随机子集替换为[MASK]令牌。
Token Deletion — Random tokens are deleted from the input. The model must decide which positions are missing (as the tokens are simply deleted and not replaced with anything else).令牌删除-从输入中删除随机令牌。该模型必须确定缺少哪些位置(因为令牌被简单地删除而不用其他任何东西代替)。
Text Infilling — A number of text spans (length can vary) are each replaced with a single [MASK] token.文本填充-多个文本跨度(长度可以变化)均被单个[MASK]令牌取代。
Sentence Permutation — The input is split based on periods (.), and the sentences are shuffled.句子排列-根据句点(。)分割输入，并随机排列句子。
Document Rotation — A token is chosen at random, and the sequence is rotated so that it starts with the chosen token.文档轮换-随机选择一个令牌，然后旋转序列，使其从所选令牌开始。

The authors note that training BART with text infilling yields the most consistently strong performance across many tasks.

作者指出，对BART进行文本填充训练可以在许多任务中获得最一致的强大性能。

For the task we are interested in, namely paraphrasing, the pre-trained BART model can be fine-tuned directly using the input sequence (original phrase) and the target sequence (paraphrased sentence) as a Sequence-to-Sequence model.

对于我们感兴趣的任务，即释义，可以使用输入序列(原始短语)和目标序列(释义的句子)作为序列转序列模型直接对预训练的BART模型进行微调。

This also works for tasks like summarization and abstractive question answering.

这也适用于诸如摘要和抽象问题解答之类的任务。

建立 (Setup)

We will use the Simple Transformers library, based on the Hugging Face Transformers library, to train the models.

我们将使用基于Hugging Face Transformers库的Simple Transformers库来训练模型。

1. Install Anaconda or Miniconda Package Manager from here.

1.从此处安装Anaconda或Miniconda Package Manager。

2. Create a new virtual environment and install packages.

2.创建一个新的虚拟环境并安装软件包。

conda create -n st python pandas tqdmconda activate st

3. If using CUDA:

3.如果使用CUDA：

conda install pytorch>=1.6 cudatoolkit=10.2 -c pytorch

else:

其他：

conda install pytorch cpuonly -c pytorch

4. Install simpletransformers.

4.安装simpletransformers。

pip install simpletransformers

资料准备 (Data Preparation)

We will be combining three datasets to serve as training data for our BART Paraphrasing Model.

我们将结合三个数据集作为BART释义模型的训练数据。

Google PAWS-Wiki Labeled (Final)

Google PAWS-Wiki标签(最终)
Quora Question Pairs Dataset

Quora问题对数据集
Microsoft Research Paraphrase Corpus (MSRP)

Microsoft研究复述语料库 (MSRP)

The bash script below can be used to easily download and prep the first two datasets, but the MSRP dataset has to be downloaded manually from the link. (Microsoft hasn’t provided a direct link

bart使用方法_使用简单变压器的BART释义相关推荐

jq 给按钮添加方法_手机进行视频编辑的方法，原来如此简单，免费、高效、省心...
相信大家都知道,我们的手机有很多强大的功能,但是视频编辑功能,你会用吗? 一.手机编辑 1.功能介绍相信大家平时都会用到很多视频编辑的方法,但是下面这些,你知道吗,主要有滤镜.裁剪的功能哦! 2.具 ...
检测到目标服务器启用了trace方法_综述：目标检测中的多尺度检测方法
↑ 点击蓝字关注极市平台作者丨SFXiang来源丨AI算法修炼营编辑丨极市平台极市导读本文从降低下采样率与空洞卷积.多尺度训练.优化Anchor尺寸设计.深层和浅层特征融合等多个方面入手,对目标 ...
php7 匿名继承类_PHP7中的匿名类使用方法_后端开发
PHP 微服务集群搭建 - Hyperf_后端开发 Hyperf 是基于 Swoole 4.4+ 实现的高性能.高灵活性的 PHP 协程框架,内置协程服务器及大量常用的组件,性能较传统基于 PHP-F ...
熊猫tv新功能介绍_熊猫简单介绍
熊猫tv新功能介绍 Out of all technologies that is introduced in Data Analysis, Pandas is one of the most pop ...
python计算时间差的方法_如何计算时间差，用Python算法的话
这篇文章主要介绍了python计算时间差的方法,实例分析了Python时间操作的相关模块与技巧,需要的朋友可以参考下本文实例讲述了python计算时间差的方法.分享给大家供大家参考.具体分析如下: ...
计算机word文档无法工作,word无法打开(WORD文档打不开_解决方法_电脑基础知识_IT/计算机_专业资料)...
word无法打开(WORD文档打不开_解决方法_电脑基础知识_IT/计算机_专业资料),哪吒游戏网给大家带来详细的word无法打开(WORD文档打不开_解决方法_电脑基础知识_IT/计算机_专业资料) ...
时间序列预测方法_让我们使用经典方法预测您的时间序列
时间序列预测方法时间序列预测 (Time Series Forecasting) 背景 (Background) We learned various data preparation techni ...
PCI设备的访问方法_桥设备(type1)
PCI设备的访问方法_桥设备(type1) 文章目录 PCI设备的访问方法_桥设备(type1) 参考资料: 一. 硬件结构二. PCI设备类别及配置方法 2.1设备类别 2.2 配置设备时怎么选中 ...
小说下载阅读器_初始简单版
小说下载阅读器_初始简单版相信园子里面的很多人和我一样喜欢阅读小说,下面是这几年用的比较多一点的阅读器,功能相对完整. Windows本地程序/手机 1.http://www.mybook66.co ...

bart使用方法_使用简单变压器的BART释义