假如BERT系论文变成Commit History

最近，我在Twitter上发现了一个有趣的话题，假设有这样一个场景，论文研究在GitHub上发布，而后续论文则会提交与原始论文不同之处。在人工智能机器学习领域，信息过载一直是一个大问题，每个月都有大量新论文发表，这样的通过commit history展示方式或许会给你带来眼前一亮。

下面我们就来蹭蹭大明星BERT的热度，来看看这一场景应用到BERT系论文会是什么样子的？

commit arXiv:1810.04805
Author: Devlin et al.
Date: Thu Oct 11 00:50:01 2018 +0000
Initial Commit: BERT
-Transformer Decoder
+Masked Language Modeling
+Next Sentence Prediction
+WordPiece 30K

commit arXiv:1901.07291
Author: Lample et al.
Date: Sun Nov 10 10:46:37 2019 +0000
Cross-lingual Language Model Pretraining
+Translation Language Modeling(TLM)
+Causal Language Modeling(CLM)

commit arXiv:1906.08237
Author: Yang et al.
Date: Wed Jun 19 17:35:48 2019 +0000
XLNet: Generalized Autoregressive Pretraining for Language Understanding
-Masked Language Modeling
-BERT Transformer
+Permutation Language Modeling
+Transformer-XL
+Two-stream self-attention

commit arXiv:1907.10529
Author: Joshi et al.
Date: Wed Jul 24 15:43:40 2019 +0000
SpanBERT: Improving Pre-training by Representing and Predicting Spans
-Random Token Masking
-Next Sentence Prediction
-Bi-sequence Training
+Continuous Span Masking
+Span-Boundary Objective(SBO)
+Single-Sequence Training

commit arXiv:1907.11692
Author: Liu et al.
Date: Fri Jul 26 17:48:29 2019 +0000
RoBERTa: A Robustly Optimized BERT Pretraining Approach
-Next Sentence Prediction
-Static Masking of Tokens
+Dynamic Masking of Tokens
+Byte Pair Encoding(BPE) 50K
+Large batch size
+CC-NEWS dataset

commit arXiv:1908.10084
Author: Reimers et al.
Date: Tue Aug 27 08:50:17 2019 +0000
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
+Siamese Network Structure
+Finetuning on SNLI and MNLI

commit arXiv:1909.11942
Author: Lan et al.
Date: Thu Sep 26 07:06:13 2019 +0000
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
-Next Sentence Prediction
+Sentence Order Prediction
+Cross-layer Parameter Sharing
+Factorized Embeddings

commit arXiv:1910.01108
Author: Sanh et al.
Date: Wed Oct 2 17:56:28 2019 +0000
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
-Next Sentence Prediction
-Token-Type Embeddings
-[CLS] pooling
+Knowledge Distillation
+Cosine Embedding Loss
+Dynamic Masking

commit arXiv:1911.03894
Author: Martin et al.
Date: Sun Nov 10 10:46:37 2019 +0000
CamemBERT: a Tasty French Language Model
-BERT
-English
+ROBERTA
+French OSCAR dataset(138GB)
+Whole-word Masking(WWM)
+SentencePiece Tokenizer

commit arXiv:1912.05372
Author: Le et al.
Date: Wed Dec 11 14:59:32 2019 +0000
FlauBERT: Unsupervised Language Model Pre-training for French
-BERT
-English
+ROBERTA
+fastBPE
+Stochastic Depth
+French dataset(71GB)
+FLUE(French Language Understanding Evaluation) benchmark

假如BERT系论文变成Commit History相关推荐

LM详解 Bert系论文译读
LM模型详解 ================================================================== BERT: Pre-training of Deep ...
Git Basics - Viewing the Commit History
Git --distributed-even-if-your-workflow-isnt About DocumentationReferenceBookVideosExternal Links Do ...
BERT(一)--论文翻译：BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
转载请注明出处:https://blog.csdn.net/nocml/article/details/124860490 传送门: BERT(一)–论文翻译:BERT: Pre-training o ...
BERT相关论文、文章和代码资源汇总
转自:http://www.52nlp.cn/tag/transformer BERT相关论文.文章和代码资源汇总 4条回复 BERT最近太火,蹭个热点,整理一下相关的资源,包括Paper, 代码和文 ...
【NLP系列】最新BERT相关论文汇总
关注上方"深度学习技术前沿",选择"星标公众号", 资源干货,第一时间送达! 项目资源链接:https://github.com/murufeng/Awesom ...
最新BERT相关论文汇总
原文链接: https://github.com/murufeng/BERT_papers 项目列表展示: 更多项目详细内容,点击阅读原文查看.
NewBeeNLP 年中 | From NewBee To NB
哈喽大家好,这里是NewBeeNLP.今天趁着端午休假,归类梳理了下之前的原创文章,不知道你是从哪篇文章开始关注的呢???? 非常感谢一年多来的喜欢和支持,不管是入门小白还是行业老司机,希望都能在Ne ...
【BERT蒸馏】DistilBERT、Distil-LSTM、TinyBERT、FastBERT（论文+代码）
文章目录 0. 引言 1. FastBERT: a Self-distilling BERT with Adaptive Inference Time 1.1 摘要 1.2 动机 1.3 贡献(适用于 ...
15篇论文全面概览BERT压缩方法
作者 | Mitchell A. Gordon 译者 | 孙薇出品 | AI科技大本营(ID:rgznai100) 模型压缩可减少受训神经网络的冗余--由于几乎没有BERT或者BERT-Large模 ...

假如BERT系论文变成Commit History

假如BERT系论文变成Commit History相关推荐

最新文章

热门文章