I’ve wanted to get more into deep learning as of late, and while I have decent theoretical experience from classwork and courses, I have very little practical experience. Thus, I’ve decided to fully commit to practicing deep learning with more rigor, and actually build models that do things.

我最近想进一步学习深度学习，尽管我从课堂学习和课程中获得了不错的理论经验，但实践经验却很少。 因此，我决定完全致力于更加严格地实践深度学习，并实际构建可以做事的模型。

The fast.ai course has been fantastic for just that. Jeremy Howard’s focus on practical deep learning is refreshing, reminding me of the advice I used to give friends in college when looking for jobs. Students should commit blocks of time, finish a project, and maybe write about it, to really flesh out the experience of software engineering, end to end. So here I am, implementing the advice by using the fastai library (the company is called fast.ai, the library, fastai), doing a project, and writing about it.

fast.ai课程对此非常了不起。 杰里米·霍华德(Jeremy Howard)对实用深度学习的关注令人耳目一新，使我想起了我以前在大学找朋友时给朋友的建议。 学生应该花一些时间，完成一个项目，或者写一些有关它的文章，以充实地端对端地充实软件工程的经验。 所以我在这里，通过使用fastai库(该公司称为fast.ai，该库称为fastai)来实施建议，执行一个项目，并撰写相关内容。

从传感器数据识别人类活动 (Human Activity Recognition from Sensor Data)

Human movement is an interesting field — it fascinates me, I’m constantly reading about it, and I’m always looking for novel ways to understand it. Luckily for me, UCI has a great repository for machine learning, and specifically a dataset of sensor recordings classified into different human movements.

人类运动是一个有趣的领域，它使我着迷，我一直在阅读它，并且我一直在寻找新颖的方式来理解它。 对我来说幸运的是，UCI有一个很好的机器学习存储库，尤其是分类为不同人类动作的传感器记录数据集 。

I won’t touch too much on the exploration of the data, but to summarize, there are multiple different sensor recordings for the torso, left arm, right arm, left leg, and right leg, in the x, y, and z axes of different movements. Each movement has a 60 second recording split into 5 second segments. There are 19 movements recorded.

我不会过多地探讨数据，但总而言之，在x，y和z轴上，躯干，左臂，右臂，左腿和右腿有多个不同的传感器记录。不同的动作。 每个机芯都有60秒的记录，分为5秒。 记录了19个动作。

论文结果 (Paper Results)

Often Jeremy takes a model that he’s built to solve a problem and demonstrates how it now does better than the paper the problem came from. For my project, there is an accompanying paper that takes the data and compares the performance of models provided by two different libraries in the classification task. The best results of those models that took the tabular data as input is 95.5% (with different validation methods being less accurate, but more realistic). The paper discusses some fine-tuning enhancements that were attempted (or the lack thereof, such as for the artificial neural network’s learning rate and momentum in the PRTools library).

杰里米经常采用他为解决问题而建立的模型，并演示它现在比问题来自的论文做得更好。 对于我的项目， 有一篇附带的论文 ，其中包含数据并比较了分类任务中两个不同库提供的模型的性能。 将表格数据作为输入的那些模型的最佳结果是95.5 ％(使用不同的验证方法的准确性较低，但更现实)。 本文讨论了一些尝试进行的微调增强(或缺少这些增强，例如PRTools库中人工神经网络的学习速度和动量)。

I wanted to see if, using the fastai library, I could get better results. After taking a look at the data, and using bit of intuition, I chose to look at acceleration data only, hoping that that would provide me with enough differentiation to recognize the different movements.

我想看看是否可以使用fastai库获得更好的结果。 在查看了数据并利用直觉之后，我选择仅查看加速度数据，希望这将为我提供足够的区分能力以识别不同的运动。

进攻计划 (Plan of Attack)

I created images from the acceleration data and fed them into a convolutional neural network (CNN) with a pretrained ResNet34 model. I’ll explain these decisions in a bit.

我根据加速度数据创建了图像，并将其输入经过预训练的ResNet34模型的卷积神经网络(CNN)中。 我将稍作解释。

资料准备 (Data Preparation)

Inspired by Jeremy’s confidence in transfer learning, I wanted to try a different route to using the data than just feeding it into a neural network, Transfer learning is the process of using an already pretrained model on a different task, in the hope that this model has enough generalized information to solve the specific task.

受Jeremy对转移学习的信心的启发，我想尝试一种使用数据的方法，而不只是将其输入神经网络。转移学习是在不同任务上使用已经预先训练的模型的过程，希望该模型有足够的通用信息来解决特定任务。

A lot of Jeremy’s initial coursework revolves around using a pretrained ResNet34 (first appeared in this paper) or ResNet50 model for transfer learning. These architectures are specific to image recognition. I’ll explore the architecture of ResNet in a later post; it deserves its own attention. However, just know that ResNet34/50 is pretty good at general image recognition applications.

很多杰里米的最初的课程牯周围使用预训练ResNet34(首先出现于本文 )或ResNet50模型迁移学习。 这些体系结构特定于图像识别。 我将在以后的文章中探讨ResNet的体系结构。 它值得自己注意。 但是，只要知道ResNet34 / 50在一般的图像识别应用程序中是相当不错的。

Of course, I still needed images to use an image recognition model, and all I currently had was tabular data for what’s known as a multivariate time series (multivariate because there are multiple variables being tracked). Here is an example of the data (x axis is time in seconds, y axis is m/s² ):

当然，我仍然需要图像来使用图像识别模型，而我目前所拥有的只是用于多变量时间序列(多变量，因为要跟踪多个变量)的表格数据。 以下是数据示例(x轴是时间，以秒为单位，y轴是m /s²)：

After some research, I discovered a couple of well-established techniques for turning a univariate time series into an image.

经过研究，我发现了一些完善的技术，可以将单变量时间序列转换为图像。

The only image demonstrated for capturing multivariate time series data is the joint recurrence plot, which is a result of taking the Hadamard product of multiple recurrence plots (matrix element-wise multiplication, extremely simple). Now, here’s the gotcha. I tried to use joint recurrence plots from the pyts library, but I couldn’t figure out how to get it to work with my data. It requires an input of (n_samples, n_features, n_timestamps) which I massaged my data to create, but I got just one fairly wimpy plot for a set of recordings, which didn’t feel right. I wish I had saved one of those images for this post, but for now, know that I tried and failed.

展示的用于捕获多元时间序列数据的唯一图像是联合重复图 ，这是采用多个重复图的Hadamard乘积的结果( 矩阵元素逐次乘法，非常简单 )。 现在，这是陷阱。 我试图使用pyts库中的联合递归图，但是我不知道如何使它与我的数据一起使用。 它需要输入(n_samples，n_features，n_timestamps)进行输入，然后通过我的数据进行创建，但是对于一组记录，我只得到了一个相当糟糕的图，感觉不对。 我希望我已经为这篇文章保存了其中一张图片，但是现在，知道我尝试过并失败了。

Instead, I went with a very simple approach. I normalized the acceleration data so it had a minimum of 0 and a maximum of 1. Then, for each body part, for each recording, I turned it into a grayscale pixel, stacked those pixels left to right in time, and top to bottom for each body part. The end result is an image like this:

相反，我采用了非常简单的方法。 我对加速度数据进行了归一化处理，因此最小值为0，最大值为1。然后，对于每个身体部位，对于每次记录，我都将其转换为灰度像素，将这些像素按时间从左到右依次堆叠，从上到下每个身体部位。 最终结果是这样的图像：

Not as pretty as those plots above, but the differences in acceleration are still obvious. Satisfied by the example image I created, I looped through the data and saved all these images to a new folder to feed into the CNN.

虽然不如上面的图漂亮，但是加速度的差异仍然很明显。 对我创建的示例图像感到满意，我遍历了数据并将所有这些图像保存到一个新文件夹中，以输入到CNN中。

快速AI和训练CNN (Fast AI and training the CNN)

Now, the big question was, even though the differences in the image are obvious to the human eye, was it obvious to the eye of a pretrained ResNet34 model that was not trained on small, gray, rectangular, images of no particular object? I was a little skeptical, but it turns out the model did recognize the classes from these images, and pretty well too.

现在，最大的问题是，即使图像上的差异对人眼来说是显而易见的，但对于未经训练的ResNet34模型的眼睛而言，它是否对没有特定物体的小型，灰色，矩形图像进行训练的人的眼睛是否显而易见？ 我有些怀疑，但是事实证明该模型确实可以从这些图像中识别出类，并且也很好。

提取数据 (Pulling in the data)

With the datablock API in the fastai library, preparing the data for the CNN was a breeze. All it took to get the data, split it into training and validation sets, apply transforms, and create what’s known as a databunch for the CNN looked like this (the labels for the image classes are part of the filename):

借助fastai库中的datablock API，为CNN准备数据变得轻而易举。 获取数据，将其分为训练集和验证集，应用转换以及为CNN创建所谓的数据束所需的一切如下所示(图像类的标签是文件名的一部分)：

data = ImageList.from_folder('./images') \                .split_by_rand_pct() \                .label_from_re(r'images/(.*)_p') \                .databunch()

After that, it was also a cinch to quickly explore some of the data

在那之后，快速浏览一些数据也很困难

I want to note that I didn’t normalize the data with imagenet_stats (provided by fastai). Jeremy says in his videos to always normalize the data using the same techniques that the pretrained model was trained with, but I had a difficult time rationalizing that when my input images were already so different. However, I should have at least tried the normalization and seen what happened. I may come back and experiment with that later.

我想指出的是，我没有使用imagenet_stats(由fastai提供)对数据进行规范化。 杰里米(Jeremy)在他的视频中说，总是使用与训练前模型相同的技术对数据进行归一化，但是当我输入的图像已经非常不同时，我很难进行合理化。 但是，我至少应该尝试过标准化，然后看看发生了什么。 我可能会稍后再尝试。

使用预训练模型 (Using the pretrained model)

Applying the pretrained model was ridiculously simple, since the fastai library comes with some models available for use.

由于fastai库随附了一些可用的模型，因此应用预训练的模型非常简单。

learn = cnn_learner(data, models.resnet34, metrics=accuracy)

This is now a model that uses the ResNet34 architecture with pretrained weights, uses my data as input, and uses accuracy as a metric to track during learning. Before training it, I tried to find the best set of learning rates to use. The fastai library has a quick way to examine the learning rates and check the losses to see when the losses start to explode (diverge, really) using lr_find.

现在，这是一个使用ResNet34体系结构和预训练权重的模型，将我的数据用作输入，并使用准确性作为衡量学习过程中的指标。 在进行培训之前，我试图找到最佳的学习率集。 fastai库提供了一种快速的方法来检查学习率并检查损失，以便使用lr_find查看损失何时开始lr_find (实际上是发散)。

Before doing the training, let me answer the question “why do we need to train the network if we are already using a pretrained model?” The pretrained model is good at generalized pattern recognition, but we need it to solve our specific problem. When using a pretrained model for training, most of the layers in the architecture’s weights will be frozen, and won’t be updated. Only the last layer’s weights will be affected by our training, and this is more or less what we want. The initial layers of the architecture often recognize broad, generic things, whereas the deeper layers tend to get more specific in what they recognize. Thus, we want to train the last layers of the model, and touch the initial layers only infrequently.

在进行训练之前，我先回答一个问题：“如果我们已经在使用预先训练的模型，为什么我们需要训练网络？” 预训练模型擅长广义模式识别，但是我们需要它来解决我们的特定问题。 使用预训练的模型进行训练时，体系结构权重中的大多数层将被冻结 ，并且不会被更新。 我们的培训只会影响最后一层的权重，这或多或少就是我们想要的。 体系结构的初始层通常可以识别广泛的通用事物，而较深的层则往往在识别方面更加具体。 因此，我们希望训练模型的最后一层，并且很少接触初始层。

I used the one cycle policy to train the network (The fast.ai course introduced me to the concept).

我使用了单周期策略来训练网络(fast.ai课程向我介绍了这一概念)。

learn.fit_one_cycle(15, slice(1e-4,1e-2))

And the results:

结果：

Loss plots to observe the progression of training:

损失图以观察训练的进展：

learn.recorder.plot_losses()

结果分析 (Results Analysis)

After 13 epochs, I achieved a 98.7% accuracy on identification of the activity by acceleration data alone. The best the paper did, regardless of library and validation technique, was a 95.5% accuracy. To really confirm the results, it would be best if I also attempted the different validation techniques in the paper, but that was beyond the scope of my effort.

13个纪元后，我仅通过加速度数据就可以识别出活动的准确性为98.7％ 。 无论采用何种文库和验证技术，该论文的最佳准确性均为95.5％ 。 为了真正确认结果，最好是我还尝试了本文中的其他验证技术，但这超出了我的努力范围。

Notice that the loss didn’t change much during the last 3 epochs and that the training loss is still less than the validation loss. This indicates that the model I created is not overfitting. General intuition tells us that for the images we are training on, we should always see that the model does well on the training data, and a little less well on the validation data.

请注意，在最近3个时期内损失没有太大变化，并且训练损失仍小于验证损失。 这表明我创建的模型没有过拟合。 一般直觉告诉我们，对于我们正在训练的图像，我们应该始终看到该模型在训练数据上效果良好，而在验证数据上效果较差。

Now, based on the activities the model was trying to classify, I figured that it would struggle the most with the treadmill activity and elevator activity. The confusion matrix above confirms that, with some additional misclassifications in activities like walking in a lot.

现在，基于模型试图分类的活动，我发现它在跑步机活动和电梯活动中的承受力最大。 上面的混淆矩阵证实了这一点，并且在活动中(例如走路很多)还存在一些其他错误分类。

结语 (Wrap Up)

Transfer learning is quite powerful, and I was honestly surprised by how well the model did. However, there are a few things I think aided in the recognition of the features. For example, because the sixty second recordings were split into five second segments, the movement of the pattern through the image could be considered “time invariant.” The model learned to ignore the specific placement of the changes in data and focus more on the appearance of that change, in either its once or cyclic presentation, in the data. In a way this was also a form of “data augmentation” that helped the model avoid overfitting.

转移学习非常强大，而我对模型的表现感到惊讶。 但是，我认为在识别功能方面有一些帮助。 例如，由于60秒的记录被分成了5秒的段，因此图案在图像中的移动可以被认为是“时不变的”。 该模型学会了忽略数据更改的特定位置，而更多地关注数据更改的外观，无论是一次还是周期性呈现。 在某种程度上，这也是一种“数据扩充”形式，可以帮助模型避免过度拟合。

It is worth noting that in this post and in my repo, I didn’t unfreeze the model to train all the layers. I tried training the frozen model and then the unfrozen model with different epoch counts, different learning rates, etc, but the results seemed to suggest that unfreezing the model did little good. I suspect this is because there is a vast difference between my input data and the data ResNet34 was trained on, and training the initial layers on my data didn’t really aid in the understanding of my problem.

值得注意的是，在本文和我的仓库中，我没有解冻模型来训练所有层。 我尝试先训练冻结的模型，然后再训练具有不同历元计数，不同学习率等的未冻结模型，但是结果似乎表明，解冻模型并没有多大用处。 我怀疑这是因为我的输入数据与ResNet34所训练的数据之间存在巨大差异，而训练我的数据的初始层并没有真正帮助理解我的问题。

The fastai library is simple, clean, and intuitive. Often, in data science projects, the common belief is that 80% of the work is data preparation and 20% is actual analysis. With deep learning, that split tends to go 50/50 with how difficult it can be to get the model right; often hours are spent training models that don’t work, and the lack of intuition behind some of the layers makes understanding what to do next difficult. Luckily for me, the fastai library put this project back into the 80/20 split.

fastai库简单，干净且直观。 通常，在数据科学项目中，普遍认为80％的工作是数据准备，而20％是实际分析。 在深度学习中，正确建立模型的难度可能会达到50/50。 通常花费数小时来训练无法使用的模型，并且某些层次背后缺乏直觉，这使得理解下一步的工作变得困难。 对我来说幸运的是，fastai库将该项目放回了80/20分割区。

Everything I just talked about is available, with code, in my github repo, with a nice README and python notebooks.

我刚才谈到的所有内容都可以在我的github repo中通过代码获得，并带有一个不错的README和python笔记本。

I look forward to writing more posts like this, either about papers/subjects I’m studying, or projects I’m working on.

我期待着写更多这样的帖子，关于我正在研究的论文/主题，或者我正在从事的项目。

Thanks for reading! Comments or questions are welcome!

谢谢阅读！ 欢迎发表评论或提问！

[1] https://pyts.readthedocs.io/en/stable/auto_examples/image/plot_gaf.html

[1] https://pyts.readthedocs.io/en/stable/auto_examples/image/plot_gaf.html