Neural networks and deep learning are not recent methods. In fact, they are quite old. Perceptrons, the first neural networks, were created in 1958 by Frank Rosenblatt. Even the invention of the ubiquitous building blocks of deep learning architectures happened mostly near the end of the 20th century. For example, convolutional networks were introduced in 1989 in the landmark paper Backpropagation Applied to Handwritten Zip Code Recognition by Yann LeCun et al.

ñeural网络和深厚的学习不是最近的方法。 实际上，它们已经很老了。 感知器是第一个神经网络，由Frank Rosenblatt于1958年创建。 即使是无所不在的深度学习架构构建基块的发明，也大多发生在20世纪末。 例如，卷积网络在1989年由Yann LeCun等人在具有里程碑意义的论文《 反向传播应用于手写邮政编码识别》中引入。

Why did the deep learning revolution had to wait decades?

为什么深度学习革命必须等待数十年？

One major reason was the computational cost. Even the smallest architectures can have dozens of layers and millions of parameters, so repeatedly calculating gradients during is computationally expensive. On large enough datasets, training used to take days or even weeks. Nowadays, you can train a state of the art model in your notebook under a few hours.

主要原因之一是计算成本。 即使是最小的体系结构也可以具有数十个层和数百万个参数，因此在此期间重复计算梯度在计算上是昂贵的。 在足够大的数据集上，培训过去需要几天甚至几周的时间。 如今，您可以在几个小时内在笔记本中训练最先进的模型。

There were three major advances which brought deep learning from a research tool to a method present in almost all areas of our life. These are backpropagation, stochastic gradient descent and GPU computing. In this post, we are going to dive into the latter and see that neural networks are actually embarrassingly parallel algorithms, which can be leveraged to improve computational costs by orders of magnitude.

三项重大进步将深度学习从研究工具带入了我们生活几乎所有领域的方法。 这些是反向传播 ， 随机梯度下降和GPU计算 。 在这篇文章中，我们将深入研究后者，并看到神经网络实际上是令人尴尬的并行算法，可以利用它来将计算成本降低几个数量级。

一大堆线性代数 (A big pile of linear algebra)

Deep neural networks may seem complicated for the first glance. However, if we zoom into them, we can see that its components are pretty simple in most cases. As the always brilliant xkcd puts it, a network is (mostly) a pile of linear algebra.

乍一看，深度神经网络似乎很复杂。 但是，如果我们放大它们，可以看到在大多数情况下其组件非常简单。 正如总是出色的xkcd所说的，网络(主要是一堆线性代数)。

During training, the most commonly used functions are the basic linear algebra operations such as matrix multiplication and addition. The situation is simple: if you call a function a bazillion times, shaving off just the tiniest amount of the time from the function call can compound to a serious amount.

在训练期间，最常用的函数是基本的线性代数运算，例如矩阵乘法和加法。 情况很简单：如果您调用一个函数的次数不计其数，那么节省掉函数调用中最短的时间可能会增加很多。

Using GPU-s not only provide a small improvement here, they supercharge the entire process. To see how it is done, let’s consider activations for instance.

使用GPU-s不仅在这里提供了很小的改进，而且还增强了整个过程。 为了了解它是如何完成的，让我们考虑一下激活。

Suppose that φ is an activation function such as ReLU or Sigmoid. Applied to the output of the previous layer

假设φ是激活函数，例如ReLU或Sigmoid。 应用于上一层的输出

the result is

结果是

(The same goes for multidimensional input such as images.)

(对于多维输入(例如图像)也是如此。)

This requires to loop over the vector and calculate the value for each element. There are two ways to make this computation faster. First, we can calculate each φ(xᵢ) faster. Second, we can calculate the values φ(x₁), φ(x₂), …, φ(xₙ) simultaneously, in parallel. In fact, this is embarrassingly parallel, which means that the computation can be parallelized without any significant additional effort.

这需要遍历向量并计算每个元素的值。 有两种方法可以使此计算更快。 首先，我们可以更快地计算每个φ(xᵢ) 。 其次，我们可以并行地并行计算值φ(x₁ )，φ(x 2 )，…，φ( xₙ ) 。 实际上，这令人尴尬地是并行的 ，这意味着可以并行化计算而无需付出任何额外的努力。

Over the years, doing things faster became much more difficult. A processor’s clock speed used to double almost every year, but this has plateaued recently. Modern processor design has reached a point where packing more transistors into the units has quantum-mechanical barriers.

多年来，更快地做事变得更加困难。 处理器的时钟速度过去几乎每年都翻一番，但是最近一直稳定。 现代处理器设计已经达到了将更多的晶体管封装到单元中的程度，从而产生了量子力学的障碍。

However, calculating the values in parallel does not require faster processors, just more of them. This is how GPUs work, as we are going to see.

但是，并行计算值不需要更快的处理器，只需更多的处理器即可。 我们将看到，这就是GPU的工作方式。

GPU计算原理 (The principles of GPU computing)

Graphics Processing Units, or GPUs in short were developed to create and process images. Since the value of every pixel can be calculated independently of others, it is better to have a lot of weaker processors than a single very strong one doing the calculations sequentially.

开发了图形处理单元(简称为GPU)来创建和处理图像。 由于每个像素的值都可以独立于其他像素进行计算，因此拥有许多较弱的处理器比按顺序执行计算的单个非常强大的处理器要好。

This is the same situation we have for deep learning models. Most operations can be easily decomposed to parts which can be completed independently.

这与深度学习模型的情况相同。 大多数操作可以很容易地分解为可以独立完成的部分。

To give you an analogy, let’s consider a restaurant, which has to produce French fries on a massive scale. To do this, workers must peel, slice and fry the potato. Hiring people to peel the potatoes costs much more than purchasing many more kitchen robots capable to perform this task. Even if the robots are slower, you can buy much more from the budget, so overall the process will be faster.

为了给您一个比喻，让我们考虑一家餐厅，该餐厅必须大规模生产炸薯条。 为此，工人必须将马铃薯去皮，切片和油炸。 雇用人们去皮土豆要比购买更多能够执行此任务的厨房机器人花费更多。 即使机器人速度较慢，您也可以从预算中购买更多东西，因此整个过程将更快。

并行模式 (Modes of parallelism)

When talking about parallel programming, one can classify the computing architectures into four different classes. This was introduced by Michael J. Flynn in 1966 and it is in use ever since.

在谈论并行编程时，可以将计算体系结构分为四个不同的类别。 它是由Michael J. Flynn于1966年推出的，此后一直在使用。

1. Single Instruction, Single Data (SISD)

   小号英格尔我 nstruction，S英格尔d ATA(SISD)

2. Single Instruction, Multiple Data (SIMD)

   小号英格尔我 nstruction， 男 ultiple d ATA(SIMD)

3. Multiple Instructions, Single Data (MISD)

   中号 ultiple 我 nstructions，S英格尔d ATA(MISD)

4. Multiple Instructions, Multiple Data (MIMD)

   中号 ultiple 我 nstructions， 男 ultiple d ATA(MIMD)

A multi-core processor is MIMD, while GPUs are SIMD. Deep learning is a problem for which SIMD is very well suited. When you calculate the activations, the same exact operation needs to be performed, with different data for each call.

多核处理器是MIMD，而GPU是SIMD。 深度学习是SIMD非常适合的问题。 在计算激活数时，需要执行相同的确切操作，并且每个呼叫的数据不同。

延迟与吞吐量 (Latency vs throughput)

To give a more detailed picture on what GPU better than CPU, we need to take a look into latency and throughput. Latency is the time required to complete a single task, while throughput is the number of tasks completed per unit time.

为了更详细地说明哪种GPU比CPU更好，我们需要研究延迟和吞吐量 。 延迟是完成单个任务所需的时间，而吞吐量是每单位时间完成的任务数。

Simply put, a GPU can provide much better throughput, at the cost of latency. For embarrassingly parallel tasks such as matrix computations, this can offer an order of magnitude improvement in performance. However, it is not well suited for complex tasks, such as running an operating system.

简而言之，GPU可以以延迟为代价提供更好的吞吐量。 对于令人尴尬的并行任务(例如矩阵计算)，这可以使性能提高一个数量级。 但是，它不适用于复杂的任务，例如运行操作系统。

CPU, on the other hand, is optimized for latency, not throughput. They can do much more than floating point calculations.

另一方面，CPU针对延迟而不是吞吐量进行了优化。 他们可以做的不仅仅是浮点计算。

通用GPU编程 (General purpose GPU programming)

In practice, general purpose GPU programming was not available for a long time. GPU-s were restricted to do graphics, and if you wanted to leverage their processing power, you needed to learn graphics programming languages such as OpenGL. This was not very practical and the barrier of entry was high.

在实践中，通用GPU编程很长一段时间都不可用。 GPU只能做图形，如果您想利用它们的处理能力，则需要学习诸如OpenGL之类的图形编程语言。 这不是很实际，进入的障碍很高。

This was the case until 2007, when NVIDIA launched the CUDA framework, an extension of C, which provides an API for GPU computing. This significantly flattened the learning curve for users. Fast forward a few years: modern deep learning frameworks use GPUs without us explicitly knowing about it.

直到2007年NVIDIA推出CUDA框架(C的扩展)时，情况一直如此。 这极大地拉平了用户的学习曲线。 快进几年：现代深度学习框架使用GPU，而我们没有明确地了解它。

深度学习的GPU计算 (GPU computing for deep learning)

So, we have talked about how GPU computing can be used for deep learning, but we haven’t seen the effects. The following table shows a benchmark, which was made in 2017. Although it was made three years ago, it still demonstrates the order of magnitude improvement in speed.

因此，我们讨论了如何将GPU计算用于深度学习，但尚未看到效果。 下表显示了基准测试，该基准测试于2017年制定。尽管基准测试是在三年前制定的，但它仍然表明了速度的提高幅度。

现代深度学习框架如何使用GPU (How modern deep learning frameworks use GPUs)

Programming directly in CUDA and writing kernels by yourself is not the easiest thing to do. Thankfully, modern deep learning frameworks such as TensorFlow and PyTorch doesn’t require you to do that. Behind the scenes, the computationally intensive parts are written in CUDA using its deep learning library cuDNN by NVIDIA. These are called from Python, so you don’t need to use them directly at all. Python is really strong in this aspect: it can be combined with C easily, which gives you both the power and the ease of use.

直接在CUDA中编程和自己编写内核并不是一件容易的事。 值得庆幸的是，诸如TensorFlow和PyTorch之类的现代深度学习框架不需要您这样做。 在后台，使用NVIDIA的深度学习库cuDNN在CUDA中编写计算密集型部分。 这些是从Python调用的，因此您根本不需要直接使用它们。 Python在这方面确实很强大：它可以轻松地与C结合使用，这既给您强大的功能，又让其易于使用。

This is similar to how NumPy works behind the scenes: it is blazing fast because its functions are written directly in C.

这类似于NumPy在幕后的工作方式：之所以Swift发展是因为它的功能直接用C编写。

您是否需要构建深度学习平台？ (Do you need to build a deep learning rig?)

If you want to train deep learning models on your own, you have several choices. First, you can build a GPU machine for yourself, however, this can be a significant investment. Thankfully, you don’t need to do that: cloud providers such as Amazon and Google offer remote GPU instances to work on. If you want to access resources for free, check out Google Colab, which offers free access to GPU instances.

如果您想自己训练深度学习模型，则有多种选择。 首先，您可以自己构建一台GPU机器，但这可能是一笔巨大的投资。 值得庆幸的是，您不需要这样做：像Amazon和Google这样的云提供商都提供了可以使用的远程GPU实例。 如果您想免费访问资源，请查看Google Colab ，它提供对GPU实例的免费访问。

结论 (Conclusion)

Deep learning is computationally very intensive. For decades, training neural networks was limited by hardware. Even relatively smaller models had to be trained for days, and training large architectures on huge datasets was impossible.

深度学习在计算上非常密集。 几十年来，训练神经网络一直受到硬件的限制。 即使是相对较小的模型也必须进行几天的训练，并且不可能在庞大的数据集上训练大型架构。

However, with the appearance of general computing GPU programming, deep learning exploded. GPUs excel in parallel programming, and since these algorithms can be parallelized very efficiently, it can accelerate training and inference by several orders of magnitude.

但是，随着通用计算GPU编程的出现，深度学习迅猛发展。 GPU在并行编程方面表现出色，并且由于这些算法可以非常高效地并行化，因此可以将训练和推理速度提高几个数量级。

This has opened the way for rapid growth. Now, even relatively cheap commercially available computers can train state of the art models. Combined with the amazing open source tools such as TensorFlow and PyTorch, people are building awesome things every day. This is truly a great time to be in the field.

这为快速增长开辟了道路。 现在，即使是相对便宜的商用计算机也可以训练最先进的模型。 结合TensorFlow和PyTorch等令人惊叹的开源工具，人们每天都在构建很棒的东西。 这确实是该领域的绝佳时机。