The following article hopes to (i) explain the first principles behind RDMA, a technology producing an order of magnitude performance increase in scale-limited systems (ii) discuss the shift from a “one-size fits all” approach in the assumptions and governance of distributed systems towards application-specific views, (iii) and demonstrate how to quickly bootstrap a cool technology.

吨他下面的文章希望于(i)解释背后RDMA第一的原则，生产的规模限制的系统级的性能提升一个顺序的技术(二)讨论从移位“一一刀切”的假设方法和面向应用程序特定视图的分布式系统治理(iii)，并演示了如何快速引导很酷的技术。

前言：趋势回顾 (Preface: A Review of Trends)

The last decade observed an acceleration in the volume/diversity of I/O devices at the edge of the network and software applications living therein. The symbiotic relationship between the two magnified this trend. As new application layers required novel forms of computing, new hardware was created to adapt to their demands. Likewise, as new hardware evolved, new markets and experiences increased total time spent in the digital world. A tale of two funnels.

在过去的十年中，网络边缘的I / O设备和其中生活的软件应用程序的数量/多样性正在加速增长。 两者之间的共生关系放大了这一趋势。 由于新的应用程序层需要新颖的计算形式，因此创建了新的硬件以适应其需求。 同样，随着新硬件的发展，新市场和新体验也增加了在数字世界中花费的总时间。 两个漏斗的故事。

By example, on the consumer side, the PC was eclipsed by mobile, wearables, and other interfaces now housing ambient computing resources. This opened markets inside homes, on top of watches, and into virtual reality. On the enterprise side, beyond cost improvements, industrial economies adopted specialized IoT devices, CPU chip-makers added hundreds more cores to machines, and GPUs extended their importance in the application layer. Across these two domains, the rise of the sharing-economy, autonomous, gaming, and social media networks increased society’s reliance on the digital world, and in turn, the reliance on these devices powering it.

例如，在消费者方面，PC被移动，可穿戴设备和现在容纳环境计算资源的其他接口所取代。 这打开了房屋内部，手表顶部和虚拟现实的市场。 在企业方面，除了提高成本外，工业经济体还采用了专门的物联网设备，CPU芯片制造商为机器增加了数百个内核，GPU扩展了它们在应用层中的重要性。 在这两个领域中，共享经济，自治，游戏和社交媒体网络的兴起增加了社会对数字世界的依赖，进而增加了对这些数字设备的依赖。

The next decade will accelerate these trends as devices extend beyond mobile and into AR/VR, neural-control interfaces promote bi-directional flows between physical and virtual signals, and increasingly complex applications shift our digital worlds from two-dimensional planes into three-dimensional experiences (hello, the metaverse).

未来十年将加速这些趋势，因为设备将从移动设备扩展到AR / VR，神经控制接口促进物理和虚拟信号之间的双向流动，并且越来越复杂的应用程序将我们的数字世界从二维平面转变为三维经验(你好，元节)。

As a second-order effect, there has been an exponential increase in the volume of information processed and transferred between devices at the edge of the network and their server counterparts. Many companies have outsourced the management of their services to cloud providers wherein economies of scale, specialization of engineering talent, and optionality has created more efficiency. Accordingly, data centers in the public clouds have rapidly scaled and integrated the advancements discussed previously; they have improved general-purpose resources (faster networking hardware, additional CPU cores, etc.) and added specialized hardware to scale bandwidth and computer power. Nonetheless, many core applications –especially those in distributed systems, such as Paxos – still face scalability challenges due to long-standing principles for how servers should communicate with the external world. With the centralization of compute and networks into data centers, engineers are given an opportunity to redefine the governing principles for how agents isolate themselves and share information. As the world becomes more distributed, these innovations within data transfer will be necessary to power the next generation of digital experiences.

作为二阶效应，网络边缘的设备与服务器对应的设备之间处理和传输的信息量呈指数级增长。 许多公司已将其服务管理外包给云提供商，其中规模经济，工程人才的专业化和可选择性提高了效率。 因此，公共云中的数据中心已Swift扩展并整合了先前讨论的进步； 他们改进了通用资源(更快的网络硬件，附加的CPU内核等)，并添加了专用硬件来扩展带宽和计算机功能。 但是，由于服务器应如何与外部世界通信的长期原则，许多核心应用程序-尤其是分布式系统(如Paxos)中的应用程序仍面临可伸缩性挑战。 通过将计算和网络集中到数据中心，工程师有机会重新定义代理如何隔离自身和共享信息的管理原则。 随着世界变得越来越分散，数据传输中的这些创新对于推动下一代数字体验至关重要。

RDMA简介 (Introduction to RDMA)

Traditional networking protocols, specifically TCP/IP, limit the number of active, concurrent connections due to bottlenecks in (i) OS kernels and (ii) processing the protocol itself. They implement a two-sided model: a sender copies and transfers data over the network to some receiver, who acts upon that information. A Pareto analysis on the TCP stack, shown below, confirms these bottlenecks by revealing where time is spent in handling 64B and 16KB messages (80% overhead from i and ii).

由于(i)OS内核和(ii)处理协议本身的瓶颈，传统的网络协议(特别是TCP / IP)限制了活动的并发连接数。 它们实现了一种双向模型：发送方通过网络将数据复制并传输到某些接收方，接收方根据该信息进行操作。 如下所示，对TCP堆栈的Pareto分析通过揭示在处理64B和16KB消息上花费的时间(i和ii的开销为80％)来确认这些瓶颈。

Remote Direct Memory Access (RDMA) bypasses the kernel and exchanges data without involving operating systems; that is, it treats remote storage as if it were local, enabling one to read and write memory in a remote machine without involving its processor. This removes the overheads that come with traditional networking protocols, as pictured in the comparison below, and helps scale concurrent connections.

远程直接内存访问(RDMA)绕过内核并交换数据，而无需操作系统。 就是说，它将远程存储视为本地存储，从而可以在不涉及其处理器的情况下读写远程计算机中的内存。 如下面的比较所示，这消除了传统网络协议附带的开销，并有助于扩展并发连接。

Fundamentally, RDMA changes prior assumptions on the governance of communication and isolation between machines. In a traditional system, threads within a process on a single machine are able to share memory; however, when other processes attempt to interact with that memory, they are denied access: a concept known as isolation. The notion of a remote machine, or unknown process, accessing memory has serious implications on security. Why allow agents in a malicious world to connect and read/write memory directly to your machine? The potential risks are indeed significant. But, what if you have trusted batches of hardware in a controlled environment? The need for strong isolation and other expensive principles can be relaxed, opening a new set of opportunities and challenges within network communication.

从根本上说，RDMA更改了有关机器之间通信和隔离的治理的先前假设。 在传统的系统中，单台计算机上的进程中的线程能够共享内存。 但是，当其他进程尝试与该内存进行交互时，它们将被拒绝访问：这一概念称为隔离。 远程计算机或未知进程访问内存的概念严重影响了安全性。 为什么要允许恶意世界中的代理直接将内存连接并读取/写入您的计算机？ 潜在的风险确实很大。 但是，如果您在受控环境中信任一批硬件，该怎么办？ 可以放宽对强隔离和其他昂贵原则的需求，从而在网络通信中带来一系列新的机遇和挑战。

Accordingly, data centers have popularized the development and deployment of RDMA. Although previous software-defined technologies, such as DPDK, first introduced kernel bypassing, RDMA is currently the fastest technique as it deploys a dedicated network stack in hardware. From a high-level, the lifecycle of a connection is simple:

因此，数据中心已经普及了RDMA的开发和部署。 尽管先前的软件定义技术(例如DPDK)首先引入了内核绕过，但RDMA是目前最快的技术，因为它在硬件中部署了专用的网络堆栈。 从高层次看，连接的生命周期很简单：

1. Two machines establish a connection and exchange information akin to a handshake. In more detail, the remote OS pins a memory region for RDMA usage, applying protection and other setup but removing itself from the fast data path. The local OS receives a virtual memory address from the remote machine, storing the location and types of operations permitted.两台机器建立连接并交换类似于握手的信息。 更详细地讲，远程OS会为RDMA使用固定一个内存区域，应用保护和其他设置，但会将其自身从快速数据路径中删除。 本地操作系统从远程计算机接收虚拟内存地址，存储允许的操作的位置和类型。
2. The local machine can start invoking commands; these include both one-sided operations (read, write, and atomic) as well as classical two-sided send/receive commands. The two-sided send operation transfers data to a queue of pre-allocated buffers in the remote machine. The two-sided receive operation polls this queue for work. According to previous work, although they are 2x slower in comparison to one-sided operations, they can replace socket-oriented systems while not shifting higher-level application logic; a plug-and-play accelerator, unlike the latter.本地计算机可以开始调用命令； 这些包括单面操作(读取，写入和原子操作)以及经典的双面发送/接收命令。 双向发送操作将数据传输到远程计算机中的预分配缓冲区队列。 双向接收操作轮询此队列以进行工作。 根据以前的工作，尽管它们与单面操作相比要慢2倍，但它们可以替换面向套接字的系统，而不会转移更高级别的应用程序逻辑。 即插即用的加速器，与后者不同。
3. After finishing, scarce memory resources are reclaimed in the RDMA-specific hardware.完成后，在特定于RDMA的硬件中回收稀缺的内存资源。

The advantages of RDMA are especially noticed in state-machine replication (SMR), one of the most ubiquitous and important algorithms in modern computing. Herein, a common distributed consensus protocol is run on a set of servers in order to replicate data both within a single region and across the globe. SMR algorithms are significant for two reasons: latency (represents response time to a client, measured in milliseconds) and fault-tolerance (the ability to continue serving requests in the presence of errors, measured in availability). Data only moves as fast as the speed of light, and thus bringing data closer to users across the world reduces length of travel (i.e., a global cache). Additionally, if a set of servers in Europe shut down, Google and other mission-critical applications need to continue serving requests by re-routing clients to another location; that is, business continuity is critical. Traditionally, SMR leveraged TCP/IP and other socket-oriented networking approaches. Since the consensus protocols (often Paxos/Raft) require leaders to communicate with all replicas, the bottlenecks of TCP/IP in this highly concurrent environment are enlarged. In the experiment below, RDMA proved to only incur a 2µs cost for each outbound message, producing exceptional performance advantages in comparison to state-of-the art implementations.

RDMA的优势在状态机复制(SMR)中尤其明显，它是现代计算中最普遍且最重要的算法之一。 在此，为了在单个区域内以及在全球范围内复制数据，在一组服务器上运行通用的分布式共识协议。 SMR算法之所以重要，有两个原因：延迟(表示对客户端的响应时间，以毫秒为单位)和容错(在存在错误的情况下继续处理请求的能力，以可用性衡量)。 数据的移动速度仅与光速一样快，因此使数据更接近世界各地的用户会减少传输时间(即全局缓存)。 另外，如果欧洲的一组服务器关闭，则Google和其他关键任务应用程序需要通过将客户端重新路由到另一个位置来继续服务请求。 也就是说，业务连续性至关重要。 传统上，SMR利用TCP / IP和其他面向套接字的联网方法。 由于共识协议(通常是Paxos / Raft)要求领导者与所有副本进行通信，因此在这种高度并发的环境中，TCP / IP的瓶颈被扩大了。 在下面的实验中，事实证明，RDMA对于每个出站消息仅产生2µs的成本，与最新的实现方式相比，具有卓越的性能优势。

In his book The Innovator’s Dilemma, Clayton Christenson constructs an important analysis on disruptive technologies: they often start out inferior to prior, more mature infrastructures in performance and/or unit costs. The evolution of RDMA embodied this behavior. And although RDMA continues to permeate into many applications, it still has much improvement in providing more abstractions, higher performance per unit of cost, and a stronger ecosystem of tools.

克莱顿· 克里斯汀森( Clayton Christenson)在他的《创新者的困境》一书中对颠覆性技术进行了重要分析：它们在性能和/或单位成本方面的起步往往不及先前更成熟的基础架构。 RDMA的发展体现了这种行为。 尽管RDMA继续渗透到许多应用程序中，但它在提供更多抽象，单位成本的更高性能以及更强大的工具生态系统方面仍然有很大的改进。

关于RDMA开发的说明： (A Note on RDMA Development:)

Mellanox was a pioneer in developing RDMA, originally producing the most sophisticated hardware and releasing an extensive implementation spec. As NVIDIA moved into cloud computing, gained market power, and expanded free cash flow, they acquired Mellanox in 2019 in a $6.8 billion, all-cash transaction. Although Mellanox’s revenue is marginal in comparison to the rest of NVIDIA’s business, they acquired important patents, talented engineers, and core technologies central to their mission: powering the digital economy with cutting-edge architectures and software libraries. Although their historic competitive advantage has been confined to AI and graphics, GPUs continue to bleed into other application and data layers. This trend follows their focus on expanding research efforts into the entire data center stack. As noted by CEO Jensen Huang when commenting on the Mellanox acquisition, “the data center has become the most important computer in the world, the thing that’s really exciting is that the computer no longer starts and ends at the server. The computer in the future would extend into the network. … And what we want to do is, we want to extend our computing reach from the server, where we are today, out to the entire data center.” Nonetheless, RDMA and the rest of NVIDIA’s product lines represent the need for specialization in an increasingly complex world.

Mellanox是开发RDMA的先驱，最初生产最先进的硬件并发布了广泛的实施规范。 随着NVIDIA进入云计算，获得市场支配力并扩大自由现金流，他们在2019年以68亿美元的全现金交易收购了Mellanox。 尽管Mellanox的收入相对于NVIDIA其他业务而言微不足道，但他们获得了重要的专利，才华横溢的工程师和核心技术，这是其使命的关键：通过尖端的体系结构和软件库为数字经济提供动力。 尽管其历史上的竞争优势仅限于AI和图形，但GPU继续渗入其他应用程序和数据层。 这种趋势是他们致力于将研究工作扩展到整个数据中心堆栈的结果。 正如首席执行官黄仁勋(Jensen Huang)在评论对Mellanox的收购时所说的那样：“数据中心已经成为世界上最重要的计算机，真正令人兴奋的是，计算机不再在服务器上启动和终止。 将来的计算机将扩展到网络中。 …而且我们要做的是，我们希望将计算范围从今天的服务器扩展到整个数据中心。” 尽管如此，RDMA和NVIDIA其他产品线仍代表着在日益复杂的世界中进行专业化的需求。

开始使用 (Get Started)

Hopefully, this Get Started guide will promote experimentation beyond the small community of software engineers operating cloud services. Distributed systems create significant arbitrage, and accordingly, they are often difficult to approach. They present an opportunity for engineers to act as “shapers,” helicoptering up and down the software stack by designing a system at a high-level while optimizing for low-level, first principles. RDMA is especially difficult given its complexity and lack of online tutorials. Thankfully, Claude Barthels Infinity library provides a great starting point by abstracting away details; he packages thousands of lines of code and presents an interface that only requires a few commands to accomplish RDMA operations. This helps newcomers focus on simpler abstractions.

希望，本入门指南将在小型运行云服务的软件工程师社区之外促进实验。 分布式系统会产生大量套利，因此，它们通常很难接近。 它们为工程师提供了一个充当“大人物”的机会，可以通过在较高级别上设计系统，同时针对低级别的第一性原则进行优化来提升和降低软件堆栈。 鉴于RDMA的复杂性和缺乏在线教程，它尤其困难。 值得庆幸的是，Claude Barthels Infinity库通过提取细节为您提供了一个很好的起点。 他打包了数千行代码，并提出了一个界面，该界面只需要几个命令即可完成RDMA操作。 这有助于新手将重点放在更简单的抽象上。

1. Find a cloud provider that houses Linux servers with InfiniBand in their hardware stack (for example, Azure). For academics with access to CloudLab, rent a cluster of “r320” or “c6220” machines查找在其硬件堆栈中装有Linux服务器和InfiniBand的云提供商(例如，Azure)。 对于可以访问CloudLab的学者，请租用“ r320”或“ c6220”计算机集群
2. Install the OFED drivers (if not already present)安装OFED驱动程序(如果尚不存在)

apt-get install libmlx4–1 infiniband-diags ibutils ibverbs-utils rdmacm-utils

Note: IbVerbs forms the necessary software layer to interact with RDMA-specific hardware. Intel explains the library in detail here: https://www.ibm.com/support/knowledgecenter/ssw_aix_72/rdma/rdma_pdf.pdf

注意：IbVerbs形成了与RDMA特定硬件进行交互的必要软件层。 英特尔在此处详细说明了该库： https : //www.ibm.com/support/knowledgecenter/ssw_aix_72/rdma/rdma_pdf.pdf

3. Clone and build the Infinity library

3.克隆并构建Infinity库

git clone https://github.com/claudebarthels/infinity; cd infinity; make library

4. Start developing

4.开始开发

The following pseudocode adapts Barthel’s example for a system with one server and K clients

以下伪代码将Barthel的示例改编为具有一个服务器和K个客户端的系统

Server

服务器

// Create new contextinfinity::core::Context *context = new infinity::core::Context();// Create a queue pairinfinity::queues::QueuePairFactory *qpFactory = new infinity::queues::QueuePairFactory(context);printf("Creating buffers for client to read from and write to\n");    infinity::memory::Buffer *bufferToReadWrite = new infinity::memory::Buffer(RDMA_MEMORY_REGION);infinity::memory::RegionToken *bufferToken = bufferToReadWrite -> createRegionToken();printf("Creating buffers to receive two-sided messages\n");for (int i = 0; i < BUFFER_COUNT; i++) {  infinity::memory::Buffer *bufferToReceive = new infinity::memory::Buffer(context, 128 * sizeof(char));  //must "register" and push our buffers onto the queue!   context -> postReceiveBuffer(bufferToReceive);}   qpFactory -> bindToPort(PORT_NUMBER);for (int i = 0; i < NUM_CLINETS; i++) {  printf("Setting up connection %d (blocking)\n", i);    qp = qpFactory -> acceptIncomingConnection(bufferToken, sizeof(infinity::memory::RegionToken));}//If we want to service two-sided communications std::vector<std::thread> threads;for (int i = 0; i < NUM_THREADS; i++) {  threads.push_back(std::thread(process_queue, context));}void process_queue(infinity::core::Context *context) {  infinity::core::receive_element_t receiveElement;  while(!context->receive(&receiveElement));  processData(receiveElement.buffer);  //reuse buffer by posting back onto the queue  context -> postReceiveBuffer(receiveElement.buffer);}

1. The program initializes a “queue pair factory”, which creates and records “queue pairs” into the context object. These queue pairs represent a connection between two machines

   该程序初始化一个“ 队列对工厂”，该工厂创建“队列对”并将其记录在上下文对象中。 这些队列对表示两台计算机之间的连接

2. We then register two types of buffers: (1) a memory region (i.e., storage layer) that can be accessed by clients through one-sided operations (read/write), denoted by the “region token” (2) a discrete series of buffers that represent unused memory regions, which will be accessed in a queue by clients for two-sided operations. Later on, the clients will push messages into these buffers, and the server can poll the queue in order to process/recycle them.

   然后，我们注册两种类型的缓冲区：(1)可由客户端通过单边操作(读/写)访问的存储区域(即存储层)，用“区域令牌”表示 (2)离散序列代表未使用的内存区域的缓冲区，客户端将在队列中对其进行双向操作。 稍后，客户端将消息推送到这些缓冲区中，服务器可以轮询队列以处理/回收它们。

3. Accept incoming connections for however many clients the system desires接受系统需要的许多客户端的传入连接
4. If we want to handle two-sided operations, the process can offload this work to any number of threads. The queue is already synchronized, and accordingly, we do not need to worry about concurrency issues when polling for work!如果我们要处理双向操作，则该过程可以将这项工作分流到任意数量的线程。 队列已经同步，因此，在轮询工作时我们不必担心并发问题！

Client

客户

infinity::core::Context *context = new infinity::core::Context();infinity::queues::QueuePairFactory *qpFactory = new  infinity::queues::QueuePairFactory(context);//Connect to a remote host– if invalid, the program abortsinfinity::queues::QueuePair *qp = qpFactory -> connectToRemoteHost(SERVER_IP, PORT_NUMBER);//Create and register a local bufferinfinity::memory::Buffer *localBuffer = new infinity::memory::Buffer(context, BUFFER_SIZE);//Store the remote token in which maps to valid memory regionsinfinity::memory::RegionToken *remoteBufferToken = (infinity::memory::RegionToken *) qp -> getUserData();//Read (one-sided) from a remote buffer and wait for completioninfinity::requests::RequestToken requestToken(context);qp -> read(  localbuffer,   0, //locally offset by 0 bytes  remoteBufferToken,   1024, //remote offset by 1024 bytes  128, //read a length of 128 bytes into the buffer  infinity::queues::OperationFlags(),  &requestToken);requestToken.waitUntilCompleted();//Write (one-sided) content of a local buffer to a remote buffer and do not wait for completionqp -> write(localBuffer, remoteBufferToken, &requestToken);requestToken.waitUntilCompleted();infinity::memory::Atomic *atomicOperation = new infinity::memory::Atomic();//Atomic compare and swap operationqp -> compareAndSwap(  remoteBufferToken,   atomicOperation,   0, //compare value  1, //swap value, only propagated if remote value == compare value  infinity::queues::OperationFlags(),   64, //read offset  &requestToken);requestToken.waitUntilCompleted();if (atomicOperation -> getValue() == 1) {  //previously set to 1}// Send (two-sided) content of a local buffer over the queue pair and wait for completionqp -> send(localBuffer, &requestToken);requestToken.waitUntilCompleted();

1. Akin to the server, the program instantiates a queue pair factory / global context类似于服务器，该程序实例化队列对工厂/全局上下文
2. Connects to the remote server连接到远程服务器
3. Creates a local buffer for storing remote reads and propagating writes创建一个本地缓冲区，用于存储远程读取和传播写入
4. Invokes various types of requests, including one-sided reads/writes, two-sided sends, and atomic operations (compare-and-swap) if and only if the remote memory region allows it当且仅当远程内存区域允许时，才调用各种类型的请求，包括单面读/写，双面发送和原子操作(比较和交换)

After modifying the provided makefile for the client/server program above, deploy the programs onto two servers and watch magic unravel. Advanced examples in which focus on concurrency at the client level can be found on my GitHub at: https://github.com/tsalemy/ConcurrentComputing/tree/master/ReplicatedKeyValueStoreRDMA/App

在为上面的客户机/服务器程序修改了提供的makefile之后，将程序部署到两个服务器上并观看解散。 可以在我的GitHub上找到关注客户端级别并发性的高级示例： https : //github.com/tsalemy/ConcurrentComputing/tree/master/ReplicatedKeyValueStoreRDMA/App

*Special thanks to Lior Selve, Sam Royce, and Brandon Santos for their thoughtful feedback

*特别感谢Lior Selve，Sam Royce和Brandon Santos的周到反馈