手撕Bert代码（torch版）

前言

先从分类说起（run_classifeir.py文件）

前言

BERT（Bidirectional Encoder Representations from Transformers）近期提出之后，作为一个Word2Vec的替代者，其在NLP领域的11个方向大幅刷新了精度，可以说是近年来自残差网络最优突破性的一项技术了。论文的主要特点以下几点：

使用了Transformer 作为算法的主要框架，Trabsformer能更彻底的捕捉语句中的双向关系；
使用了Mask Language Model(MLM) 和 Next Sentence Prediction(NSP) 的多任务训练目标；
使用更强大的机器训练更大规模的数据，使BERT的结果达到了全新的高度，并且Google开源了BERT模型，用户可以直接使用BERT作为Word2Vec的转换矩阵并高效的将其应用到自己的任务中。

ERT的本质上是通过在海量的语料的基础上运行自监督学习方法为单词学习一个好的特征表示，所谓自监督学习是指在没有人工标注的数据上运行的监督学习。在以后特定的NLP任务中，我们可以直接使用BERT的特征表示作为该任务的词嵌入特征。所以BERT提供的是一个供其它任务迁移学习的模型，该模型可以根据任务微调或者固定之后作为特征提取器。

先扔出开源代码链接:pytorch、TensorFlow

附上关于BERT的资料汇总：BERT相关论文、文章和代码资源汇总

先从分类说起（run_classifeir.py文件）

main函数完成了整个分类功能，它首先定义了需要的参数，加载数据，要加载的bert预训练模型，输出路径等。。。

def main():parser = argparse.ArgumentParser()## Required parametersparser.add_argument("--data_dir",default=None,type=str,required=True,help="The input data dir. Should contain the .tsv files (or other data files) for the task.")parser.add_argument("--bert_model", default=None, type=str, required=True,help="Bert pre-trained model selected in the list: bert-base-uncased, ""bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, ""bert-base-multilingual-cased, bert-base-chinese.")parser.add_argument("--task_name",default=None,type=str,required=True,help="The name of the task to train.")parser.add_argument("--output_dir",default=None,type=str,required=True,help="The output directory where the model predictions and checkpoints will be written.")

接下来判断是否使用GPU环境

    if args.local_rank == -1 or args.no_cuda:device = torch.device("cuda" if torch.cuda.is_available() and not args.no_cuda else "cpu")n_gpu = torch.cuda.device_count()else:torch.cuda.set_device(args.local_rank)device = torch.device("cuda", args.local_rank)n_gpu = 1# Initializes the distributed backend which will take care of sychronizing nodes/GPUstorch.distributed.init_process_group(backend='nccl')args.device = device

然后进行数据处理

    processor = processors[task_name]()output_mode = output_modes[task_name]label_list = processor.get_labels()num_labels = len(label_list)

processors方法是来自于run_classifier_dataset_utils.py这个模块的它通过task_name来索引到相应的数据集的处理函数中去，字典中的键值就是对应数据集的处理方法。

processors = {"cola": ColaProcessor,"mnli": MnliProcessor,"mnli-mm": MnliMismatchedProcessor,"mrpc": MrpcProcessor,"sst-2": Sst2Processor,"sts-b": StsbProcessor,"qqp": QqpProcessor,"qnli": QnliProcessor,"rte": RteProcessor,"wnli": WnliProcessor,
}

例如： ColaProcessor，它定义了一个ColaProcessor类（他是继承了它的父类DataProcessor），类中包含了获取训练集的方法（get_train_examples），获取测试集的方法（get_dev_examples），获取标签的方法（get_labels），和创建数据集的方法（_create_examples）。

class ColaProcessor(DataProcessor):"""Processor for the CoLA data set (GLUE version)."""def get_train_examples(self, data_dir):"""See base class."""return self._create_examples(self._read_tsv(os.path.join(data_dir, "train.tsv")), "train")def get_dev_examples(self, data_dir):"""See base class."""return self._create_examples(self._read_tsv(os.path.join(data_dir, "dev.tsv")), "dev")def get_labels(self):"""See base class."""return ["0", "1"]def _create_examples(self, lines, set_type):"""Creates examples for the training and dev sets."""examples = []for (i, line) in enumerate(lines):guid = "%s-%s" % (set_type, i)text_a = line[3]label = line[1]examples.append(InputExample(guid=guid, text_a=text_a, text_b=None, label=label))return examples

接下来看它的父类DataProcessor，它主要是定义了_read_tsv这个方法用于读取tsv文件(当然如果你的数据集格式是 CSV 或者其他格式的文件你也可以重写这个方法，或者添加读取其他文件的方法)。

class DataProcessor(object):"""Base class for data converters for sequence classification data sets."""def get_train_examples(self, data_dir):"""Gets a collection of `InputExample`s for the train set."""raise NotImplementedError()def get_dev_examples(self, data_dir):"""Gets a collection of `InputExample`s for the dev set."""raise NotImplementedError()def get_labels(self):"""Gets the list of labels for this data set."""raise NotImplementedError()@classmethoddef _read_tsv(cls, input_file, quotechar=None):"""Reads a tab separated value file."""with open(input_file, "r", encoding="utf-8") as f:reader = csv.reader(f, delimiter="\t", quotechar=quotechar)lines = []for line in reader:if sys.version_info[0] == 2:line = list(unicode(cell, 'utf-8') for cell in line)lines.append(line)return lines

数据加载完毕，要准备开始训练了，看最后一行代码convert examples to features

    if args.do_train:if args.local_rank in [-1, 0]:tb_writer = SummaryWriter()# Prepare data loadertrain_examples = processor.get_train_examples(args.data_dir)cached_train_features_file = os.path.join(args.data_dir, 'train_{0}_{1}_{2}'.format(list(filter(None, args.bert_model.split('/'))).pop(),str(args.max_seq_length),str(task_name)))try:with open(cached_train_features_file, "rb") as reader:train_features = pickle.load(reader)except:train_features = convert_examples_to_features(train_examples, label_list, args.max_seq_length, tokenizer, output_mode)

convert_examples_to_features这个方法主要是获取token并将token转换到字典中相对应的id

def convert_examples_to_features(examples, label_list, max_seq_length,tokenizer, output_mode):"""Loads a data file into a list of `InputBatch`s."""label_map = {label : i for i, label in enumerate(label_list)}features = []for (ex_index, example) in enumerate(examples):if ex_index % 10000 == 0:logger.info("Writing example %d of %d" % (ex_index, len(examples)))tokens_a = tokenizer.tokenize(example.text_a)tokens_b = Noneif example.text_b:tokens_b = tokenizer.tokenize(example.text_b)# Modifies `tokens_a` and `tokens_b` in place so that the total# length is less than the specified length.# Account for [CLS], [SEP], [SEP] with "- 3"_truncate_seq_pair(tokens_a, tokens_b, max_seq_length - 3)else:# Account for [CLS] and [SEP] with "- 2"if len(tokens_a) > max_seq_length - 2:tokens_a = tokens_a[:(max_seq_length - 2)]# The convention in BERT is:# (a) For sequence pairs:#  tokens:   [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP]#  type_ids: 0   0  0    0    0     0       0 0    1  1  1  1   1 1# (b) For single sequences:#  tokens:   [CLS] the dog is hairy . [SEP]#  type_ids: 0   0   0   0  0     0 0## Where "type_ids" are used to indicate whether this is the first# sequence or the second sequence. The embedding vectors for `type=0` and# `type=1` were learned during pre-training and are added to the wordpiece# embedding vector (and position vector). This is not *strictly* necessary# since the [SEP] token unambiguously separates the sequences, but it makes# it easier for the model to learn the concept of sequences.## For classification tasks, the first vector (corresponding to [CLS]) is# used as as the "sentence vector". Note that this only makes sense because# the entire model is fine-tuned.tokens = ["[CLS]"] + tokens_a + ["[SEP]"]segment_ids = [0] * len(tokens)if tokens_b:tokens += tokens_b + ["[SEP]"]segment_ids += [1] * (len(tokens_b) + 1)input_ids = tokenizer.convert_tokens_to_ids(tokens)# The mask has 1 for real tokens and 0 for padding tokens. Only real# tokens are attended to.input_mask = [1] * len(input_ids)# Zero-pad up to the sequence length.padding = [0] * (max_seq_length - len(input_ids))input_ids += paddinginput_mask += paddingsegment_ids += paddingassert len(input_ids) == max_seq_lengthassert len(input_mask) == max_seq_lengthassert len(segment_ids) == max_seq_lengthif output_mode == "classification":label_id = label_map[example.label]elif output_mode == "regression":label_id = float(example.label)else:raise KeyError(output_mode)if ex_index < 5:logger.info("*** Example ***")logger.info("guid: %s" % (example.guid))logger.info("tokens: %s" % " ".join([str(x) for x in tokens]))logger.info("input_ids: %s" % " ".join([str(x) for x in input_ids]))logger.info("input_mask: %s" % " ".join([str(x) for x in input_mask]))logger.info("segment_ids: %s" % " ".join([str(x) for x in segment_ids]))logger.info("label: %s (id = %d)" % (example.label, label_id))features.append(InputFeatures(input_ids=input_ids,input_mask=input_mask,segment_ids=segment_ids,label_id=label_id))return features

到目前为止数据加载完毕，接下来就要加载bert模型开始训练了（先更新到这，后续部分下次更新~~）