Understanding data storage and ingestion for large-scale deep recommendation model training

不咋好读，不太懂

摘要

问题：Domain-specific accelerator特定领域加速器合并进数据中心尺度的集群，增加大数据集的训练的有效性和吞吐量，data storage and ingestion pipeline (DSI) 和用于存储和预处理训练数据的系统和硬件限制训练能力。DSI系统需要创新。

解决方法：Meta 端到端DSI pipeline，由central data warehouse按照distributed storage和data preprocessing service(DPP)组成，消除data stall。

展示上百个模型是如何训练的，大量数据集是如何存储和read的，在线预处理如何将大量需求放到硬件上的。

1.Introduction

DSA应用于训练大数据集的DNN网络，现有DSA优化模型训练的计算，即反向传播的矩阵运算。

DSI由offline data generation, dataset storage, online preprocessing services组成，存储和发送数据给trainer。DSI的设计影响训练的性能鲜有人关注。

理解DSI需求、unique workload characteristics, systems for industry-scale，deep learning recommendation model training(DLRM).

DSI：1.限制吞吐量，降低DSA利用率。2.消耗大量存储、预处理和训练资源，电力资源。3.模型复杂性和训练DSA增加数据存储和带宽需求。

end to end DSI pipeline保证大范围ML model training at-scale，训练数据由extract-transform-load ETL产生，PB数据存储在centralized data warehouse，data preprocessing service DPP处理大量线上预处理需求

介绍Meta的production-deplyed DSI pipeline，支持DNN 训练。需要存储大量动态改变的数据集，训练需要在线预处理，包括massive compute, network, and memory resourves

主要贡献：

1.介绍了DSI pipeline

2.提供端到端描述 production-deployed DSI pipeline 架构，根据DLRM需求定制

3.展示industry-scale DLRM training workloads特征，包括coordinated training, data generation and storage and online preprocessing

4.outlook

2.recommendation model background

推荐模型训练采用data parallelism和model parallelism。每一个训练工作依赖于data storage and ingestion(DSI) pipeline 提供训练数据，DSI流水线负责产生训练数据，存储和预处理样本

3.meta disaggregated data storage ingestion and training pipeline

最重要的就是这张图

A.data generation and storage

框架输入数据到预测模型，requesting service监控推荐系统的events的输出，避免数据泄露;subsequent streaming and batch extract-load-transform (ETL);subsequent streaming and batch extract-load-transform (ETL) 增加新的raw feature。

1.data generaton:

scribe

2.data storage

store training samples in a data warehouse as partitioned hive table because of hive’s compatibility with both internal systems and open sourve engines including spark and Presto.

two types of features: dense and sparse

B.online preprocessing

overview and requirements

raw bytes从storage中提取, decode into training samples. training sample transform into tensors. new features will be derived. after features are preprocessed, they are batched together into tensors. tensors are loaded into trainers, GPU.

1.scalable preprocessing with DPP

DPP提供在线预处理for training jobs across the datacenter fleet.

DPP control plane: DPP Master recieve a session specification, enable scalable work distribution, split to DPP workers, fault tolerance and auto-scaling, monitor worker health, implement auto-scaling via a controller.

DPP data plane: data workers and clients: for data plane operations of DPP.

Trainers:

4.coordinated training at scale

A.collaborative release process

避免模型版本之间冲突和保留受限制的训练能力

训练三步流程：1.提出想法, exploratory jobs training 几百个小的训练工作. 2.最有前景的ideas训练。 3.release candidates RCs继续训练，在fresh data上评估，最准确的模型应用。

训坏的模型kill然后重新训练

B.Global training demand

模型训练在global fleet of training infrastructure 在全球。

每个模型读不同dataset, cross-region bandwidth is highly-constrained. system and datacenter architects solve this by scheduler and bin-packing. scheduler balances training jobs for each model across regions by requiring each region to contain a copy of all models’ dataset. Bin-packing

C.feature engineering

feature are rapidly changing in production datasets with hundreds of new features added and deprecated each month.

因此需要高效数据存储。

D.summary of key takeaways

Ideas 周期性更新模型生成combo job，导致训练和DSI巅峰；设计全球datacenter，上百个模型进行训练和调度；训练工作不同架构和数据集

【文献阅读未遂】Understanding data storage and ingestion for large-scale deep recommendation model training相关推荐

轨迹压缩文献阅读 TrajStore: An Adaptive Storage System for Very Large Trajectory Data Sets
论文:TrajStore: An adaptive storage system for very large trajectory data sets 概述 TrajStore是一种存储系统,它被设 ...
文献阅读—A detection algorithm for cherry fruits based on the improved YOLO-v4 model
期刊:Neural Computing and Applications 发表时间:2021.05 1.作者想要解决的问题单纯为了提高识别樱桃的精度 (带来了不少麻烦,本人认为这篇文章可借鉴的创新点 ...
论文阅读笔记之Replacing Mobile Camera ISP with a Single Deep Learning Model
这篇文章做了啥根据ccd或者cmos图像传感器获得的图像,要经过一系列的复杂的处理才能得到最终的图像,而且不同的cmos有不同的算法,作者这里的motivation是能不能有一个比较统一的框架来处理 ...
文献阅读总结：网络表示学习/图学习
本文是对网络表示学习/图学习(Network Representation Learning / Graph Learning)领域已读文献的归纳总结,长期更新. 朋友们,我们在github创建了一个 ...
文献阅读：Improving Language Understanding by Generative Pre-Training
文献阅读:Improving Language Understanding by Generative Pre-Training 1. 文章简介 2. 模型介绍 3. 实验考察 1. 训练数据 2. ...
[文献阅读报告]：Take-over Time Prediction for Autonomous Driving in the Real-World: Robust Models, Data Aug
[文献阅读报告]:Take-over Time Prediction for Autonomous Driving in the Real-World: Robust Models, Data Aug ...
[VQA文献阅读] FloodNet: A High Resolution Aerial Imagery Dataset for Post Flood Scene Understanding
背景文章题目:<FloodNet: A High Resolution Aerial Imagery Dataset for Post Flood Scene Understanding> ...
细胞亚器文献阅读之酵母液泡与线粒体的动态互作A Dynamic Interface between Vacuoles and Mitochondria in Yeast
细胞亚器文献阅读之酵母液泡与线粒体的动态互作A Dynamic Interface between Vacuoles and Mitochondria in Yeast 本文和前一篇阅读的文献之间的关 ...
崇志宏【转载】深度学文献阅读路径图---东南大学
深度学习文献众多,技术发展迅速,有系统地阅读文献是掌握深度学习本质的基本方式,如此认真整理的文献不多见,转发给大家共享! 东南大学崇志宏 Deep Learning Papers Reading R ...
四位科研牛人介绍的文献阅读经验
每天保持读至少2-3 篇的文献的习惯.读文献有不同的读法,但最重要的自己总结概括这篇文献到底说了什么,否则就是白读,读的时候好像什么都明白,一合上就什么都不知道,这是读文献的大忌,既浪费时间,最 ...

【文献阅读未遂】Understanding data storage and ingestion for large-scale deep recommendation model training