Ever since I have started running a couple years ago, I have come to understand the importance of stretching. The problem is, I’m no good at it! When I was introduced to techniques like pose estimation and computer vision in general, I was excited to apply them to this real world problem of yoga form.

自从几年前开始跑步以来，我已经了解了伸展运动的重要性。 问题是，我不擅长！ 当我向我介绍一般的姿势估计和计算机视觉等技术时，我很高兴将其应用于瑜伽世界这一现实世界中的问题。

Pose estimation is a sub-field of computer vision that concerns itself with recognizing individual parts that make up a body (usually a human body). While there are several techniques to achieve this, the approach I use starts by running incoming images through a CNN classifier trained to look for humans. If and when a human body is detected, the pose estimator network looks for the joints and appendages it’s trained on. I can then display the image back to the user with markings on where the computer has judged a body’s parts to be.

姿势估计是计算机视觉的一个子领域，它与识别组成身体(通常是人体)的各个部分有关。 尽管有几种方法可以实现这一点，但我使用的方法是通过经过训练以寻找人类的CNN分类器运行传入图像开始的。 如果并且当检测到人体时，姿势估算器网络会寻找对其进行训练的关节和附肢。 然后，我可以将图像显示给用户，并在计算机判断出人体部位的位置上带有标记。

From the beginning, I knew I wanted my pose analyzer to judge my poses in real time, through my laptop camera. As it turns out, this seriously limited my options for pose estimation models. First I tried the widely used OpenPose from Carnegie Mellon University. As with all computer vision and object detection, there is a trade-off between accuracy and speed. With the larger model size and the puny compute on my personal laptop, this state of the art pose estimator would not serve my purposes. What I finally settled on was a MobileNet for general human detection and a small ResNet model for appendage detection, both of which can be found in Gluon’s Modelzoo. These smaller networks allow my measly computing resources to judge video frames from my webcam in real time (after some finagling). Even by only classifying every other frame of video, the output is not as smooth as I had hoped.

从一开始，我就知道我希望我的姿势分析仪能够通过笔记本电脑摄像头实时判断我的姿势。 事实证明，这严重限制了我对姿势估计模型的选择。 首先，我尝试了卡耐基梅隆大学(Carnegie Mellon University)广泛使用的OpenPose 。 与所有计算机视觉和目标检测一样，在精度和速度之间要进行权衡。 随着更大的模型尺寸和我的便携式笔记本电脑上微不足道的计算，这种最先进的姿势估计器将无法满足我的目的。 我最终选择的是用于普通人类检测的MobileNet和用于附件检测的小型ResNet模型，这两种方法都可以在Gluon的Modelzoo中找到。 这些较小的网络使我微不足道的计算资源可以实时(在查找后)判断来自网络摄像头的视频帧。 即使仅对视频的其他所有帧进行分类，输出也不如我所希望的那样平滑。

At this point my computer could track the appendages of humans in the frame but could not say anything about their yoga form. To accomplish this, I decided to add another network to the image pipeline. Instead of another cumbersome CNN, I thought I could cleverly use a simple ANN trained on the angles between the appendages of the body in question. My hope was that if I could calculate/record the inter-joint angles of bodies in the webcam for various yoga poses, the poses would be linearly separable (albeit in higher dimensions). If this were true, anytime the computer saw a body making a particular combination of angles, it would know which yoga pose they were attempting. In fact, the confidence in the prediction of the pose could even be used as an indicator for the quality of form (if the training data used was from an experience yogi).

在这一点上，我的计算机可以跟踪人体在框架中的附属物，但无法说出他们的瑜伽形式。 为此，我决定在图像管道中添加另一个网络。 我认为我可以聪明地使用一个经过训练的简单的人工神经网络，而不是另一个笨拙的CNN，该人工神经网络在所讨论的身体的附属肢体之间的角度进行训练。 我的希望是，如果我可以针对各种瑜伽姿势计算/记录网络摄像头中各个身体的关节间角度，则这些姿势将是线性可分离的(尽管尺寸较大)。 如果这是真的，那么只要计算机看到一个身体将某个角度组合成特定的角度，它就会知道他们正在尝试哪种瑜伽姿势。 实际上，对姿势预测的信心甚至可以用作形式质量的指标(如果使用的训练数据来自经验瑜伽师)。

This is where Yoga with Uliana, a popular Youtube yogi, comes in. From her 30 Most Common Beginning Yoga Poses video, I selected 15 easily distinguishable poses, ran the video clips through my pose estimator, recorded the angles of each joint for every frame of each clip. Now I had a labeled dataset with several hundred examples to train my ANN on. I used a small Keras model (only two layers), and was able to correctly predict 93% of the poses in my validation set. The best part is, when I added this predictor to my already bloated pipeline, there was no drop in frame rate! Perfect!

这就是流行的Youtube瑜伽士Uliana的Yoga。在她的30个最常见的初学者瑜伽姿势视频中，我选择了15个易于区分的姿势，将视频剪辑通过我的姿势估计器运行，记录了每一帧每个关节的角度每个剪辑。 现在，我有了一个带有数百个示例的标记数据集，以训练我的ANN。 我使用了一个小的Keras模型(只有两层)，并且能够正确地预测我的验证集中93％的姿势。 最好的部分是，当我将此预测变量添加到我已经already肿的管道中时，帧速率没有下降！ 完善！

The complete code is available on my github, all feedback is not only welcomed but desired.

完整的代码可在我的github上找到 ，所有反馈不仅受到欢迎，而且是所希望的。

下一步 (Next Steps)

With the blazing speed at which computer vision is evolving, new pose estimation techniques and models will soon replace the tried and true methods of today. Changing the the networks/replacing them with newer predictors is definitely something I will try going forward.

随着计算机视觉的飞速发展，新的姿态估计技术和模型将很快取代当今尝试和真实的方法。 更改网络/将其替换为较新的预测变量绝对是我将继续尝试的事情。

One limitation of using this software to judge how well a pose is done is that the yoga data was taken from a single source. In the future, to make this project more robust I will train the ANN on several master yogis doing the same pose (ideally both male and female yogis).

使用该软件判断姿势的良好程度的一个限制是瑜伽数据是从单一来源获取的。 将来，为了使该项目更强大，我将在做相同姿势的几位主要瑜伽士(理想情况下是男性和女性瑜伽士)上训练ANN。

Additionally, my ANN is trained on only 15 of Uliana’s yoga poses, somewhat limiting the utility of this project to yoga newbies. One problem is, while I can add new poses to the training data and retrain my ANN, it makes the software pretty inflexible. I’m currently trying to think of ways around this problem, where I wouldn’t have to train a whole new network every time I wanted to add a couple new poses.

此外，我的人工神经网络仅接受了Uliana瑜伽姿势中的15种训练，这在一定程度上限制了该项目对新手瑜伽的实用性。 一个问题是，尽管我可以在训练数据中添加新的姿势并重新训练我的人工神经网络，但它使软件变得非常不灵活。 我目前正在尝试解决该问题的方法，每次我想添加几个新姿势时，我都不必训练整个新网络。

Finally, coming full circle back to my love of running, I want to use what I have learned about pose estimation to analyze my running form!

最后，回到我对跑步的热爱中，我想用我从姿势估计中学到的知识来分析跑步形式！

应用领域 (Applications)

The Covid-19 quarantine has resulted in the surge of the use of home exercise equipment. One such product is the smart mirror, such as the ones made by MIRROR, which streams exercise classes directly to your mirror! This seems like a perfect application for my project, especially if I were to adapt it to recognize other form-sensitive exercises like push-ups or running.

Covid-19隔离区导致家庭健身设备的使用激增。 智能镜子就是其中一种产品，例如MIRROR制造的智能镜子，它可以将运动课程直接流到您的镜子上！ 对于我的项目而言，这似乎是一个完美的应用程序，尤其是当我要使其适应其他形式敏感的练习(例如俯卧撑或跑步)时。