Living in the Pacific Northwest, the intertwined issues of salmon survival and river flow are frequently in the news, and the data we have on our salmon populations is a key piece in the conversation.

生活在西北太平洋地区，鲑鱼生存和河流流量这两个相互交织的问题经常成为新闻，而我们关于鲑鱼种群的数据是谈话中的关键部分。

I wanted to see if deep learning — object detection in particular — was up to the challenge of assisting with the real world problem of counting fish seen passing through a fish ladder.

我想看看深度学习(尤其是对象检测)是否可以应对现实世界中对通过鱼梯的鱼进行计数的难题。

The answer is yes, and there are enough free or open source tools available to accomplish all aspects of the task.

答案是肯定的，并且有足够的免费或开源工具可完成任务的所有方面。

This approach can be used to count other objects as well. If you have an idea and a place to grab a few images, you can use this walkthrough to help you get your own model running.

该方法也可以用于计算其他对象。 如果您有想法并可以获取一些图像，则可以使用本演练来帮助您运行自己的模型。

步骤1：一个有趣的问题 (Step 1: An intriguing issue)

The Pacific Northwest is home to many species of salmon whose lives follow a predictable pattern: hatch in fresh water, migrate to the ocean for the majority of their lives, and then migrate back upstream to their original fresh water hatch sites before they spawn and then die.

西北太平洋地区生活着许多种类的鲑鱼，它们的生活遵循可预测的模式：在淡水中孵化，在大部分生命中移至海洋，然后在产卵前向上游迁移回到其原始淡水孵化场，死。

Salmon aren’t the only ones using the rivers though. Hydroelectric dams dot our rivers for hundreds of miles, potentially preventing the upstream migration. To solve this, our dams are all built with something called a ‘fish ladder.’ These constructions allow fish to jump upstream in short segments much as they would in an openly flowing river.

鲑鱼并不是唯一使用河流的鱼类。 水力发电大坝在我们的河流上遍布数百英里，有可能阻止上游迁移。 为了解决这个问题，我们的水坝全部建有所谓的“鱼梯”。 这些结构使鱼类可以像在一条开放的河流中那样，在短段内向上游跳跃。

Salmon ladders with viewing windows also provide us with the means to witness this migration and to collect data. Data on the number of salmon returning upstream can then be used to determine the length of the fishing season, set limits on fish per day, adjust the flow at the dam, and help us understand how well we’re doing in this balancing act.

带有观察窗的鲑鱼梯也为我们提供了见证迁移和收集数据的方法。 然后，可以将有关返回上游的鲑鱼数量的数据用于确定捕鱼季节的长度，设置每天的鱼类限制，调整水坝的流量，并帮助我们了解我们在这种平衡行动中的状况。

Counting fish is currently performed by trained experts either as real time counts or by reviewing video recordings. Given how labor intensive this is, an assist from an expert tool could provide a nice boost to increase the amount of data taken, the speed at which it is available, or the number of locations where the data is recorded.

目前，计数鱼类是由训练有素的专家进行的，既可以实时计数，也可以通过查看录像来进行。 考虑到这是多么劳动密集的工作，专家工具的帮助可以极大地提高所获取的数据量，可用数据的速度或记录数据的位置数量。

步骤2：图像和预处理 (Step 2: Images and pre-processing)

Whichever project you choose, you’ll need to start with data. Don’t be dissuaded if you can’t start with ideal data; there’s still plenty to learn. I started with web scraped images of vacation photos taken at fish ladders. The images contained a variety of locations, lighting conditions, silhouettes of children, and possible post-production modifications. The images were so difficult to work with that all I could really label was what constituted a “fish”! I ran the rest of this flow with just that and worked out quite a few bugs, so remember that you can start simple. Once I had a sense of what I needed, I found my way to some better images recorded on a video.

无论选择哪个项目，都需要从数据开始。 如果您无法从理想的数据入手，请不要被劝阻； 还有很多东西要学。 我从网上抓取了在鱼梯上拍摄的度假照片的图像开始。 图像包含各种位置，光照条件，儿童剪影以及可能的后期制作修改。 这些图像很难处理，以至于我只能标记出构成“鱼”的东西！ 我仅以此来运行其余的流程，并解决了许多错误，因此请记住，您可以从简单开始。 一旦感觉到需要的东西，我便找到了录制视频中更好图像的方法。

For the images in video format, I used the free VLC Media Player tool to run the video and extract 300+ frames. I selected 317 images including 15 images with no fish at all.

对于视频格式的图像，我使用了免费的VLC Media Player工具来运行视频并提取300多个帧。 我选择了317张图像，其中包括15张完全没有鱼的图像。

To prepare these images for deep learning, I also needed to associate a correctly labeled box with each object to be detected. Trained experts label fish by species, mature vs. juvenile, and hatchery vs. wild.

为了准备这些图像进行深度学习，我还需要将正确标记的框与要检测的每个对象相关联。 受过训练的专家按鱼种，成熟对幼鱼以及孵化场对野生鱼进行标记。

Without fish identification expertise, it was still possible to create 3 classes:

没有鱼类鉴定专业知识，仍然有可能创建3个类别：

• ‘adipose’ for fish having an intact and visible adipose fin“脂肪”用于具有完整可见脂肪鳍的鱼
• ‘no_adipose’ for fish having no adipose fin没有脂肪的鱼的“ no_adipose”
• ‘unknown’ for fish only partially in the viewing window, or whose adipose fin region is obscured by another fish or artifact鱼只在部分观察窗中或鱼类的脂肪鳍区域被另一条鱼或人工制品遮盖，因此“未知”

The free tool labelImg worked well and I had my 317 fish images boxed and labeled in an afternoon.

免费工具labelImg运作良好，一个下午装箱并标记了我的317条鱼图像。

One of the reasons I knew thatI could get a model working with only 300 or so images was the idea of augmentation. Augmentation takes the originals and constructs variations so that the model is exposed to a variety of changes such as lighting and orientation. With classification, this is easily performed with a few python libraries, but for object detection, the label boxes also need to be transformed if the image is flipped or rotated.

我知道可以得到一个仅处理300张左右图像的模型的原因之一就是增强的想法。 增强采用原始图像并构造变体，以便模型可以暴露于各种变化(例如光照和方向)。 使用分类，可以使用几个python库轻松地执行此操作，但是对于对象检测，如果翻转或旋转图像，则还需要转换标签框。

Rather than do these transforms by hand, I leveraged the free tools at Roboflow. I uploaded my images and my label file, selected options to create additional images with random amounts of blur, changes in brightness, and horizontal flip. After this step, I had 951 training images.

我没有手动进行这些转换，而是利用了Roboflow上的免费工具。 我上传了图像和标签文件，并选择了一些选项来创建具有随机模糊量，亮度变化和水平翻转的其他图像。 经过这一步，我得到了951张训练图像。

步骤3：选择并训练模型 (Step 3: Select and train a model)

While it’s possible to build a home-grown object detection model, I opted to start with a known model for my project as a baseline before doing any tailoring. All of this can be done from the model library at Roboflow, and it’s possible to try out more than one.

尽管可以构建自家的对象检测模型，但在进行任何剪裁之前，我选择以项目的已知模型作为基准。 所有这些都可以从Roboflow的模型库中完成 ，并且可以尝试多个。

“You Only Look Once”. YOLO is a popular object detection machine learning model introduced in 2015 by a group of researchers at the University of Washington. Rather than pass an image classifier multiple times over an image to see if there was, say, a dog at the upper left, or maybe at the upper right, this new approach replaced the final layers of an image classifier with additional convolutional layers that allowed it to find all instances in one pass. The immediate improvement in speed was a major leap forward for computer vision and object detection. Since the original paper, the model has been improved several times and a new model built on this earlier foundation was released in June 2020 as YOLO v5. See the repository at https://github.com/ultralytics/yolov5 for more details on the model.

“你只看一次”。 YOLO是由华盛顿大学的一组研究人员于2015年引入的一种流行的对象检测机器学习模型。 这种新方法不是在图像上多次传递图像分类器以查看是否在左上方或右上方有一条狗，而是用附加的卷积层替换了图像分类器的最后一层，从而允许一次找到所有实例。 速度的立即提高是计算机视觉和物体检测的重大飞跃。 自原始论文以来，该模型已进行了多次改进，并且在此较早基础上构建的新模型于2020年6月作为YOLO v5发布。 有关该模型的更多详细信息，请参见https://github.com/ultralytics/yolov5上的存储库。

Given the popularity, speed, and accuracy of YOLO, and the ease of leveraging the tools at Roboflow, trying out the YOLO v5 model was an obvious choice. Starting with a Google Colaboratory template that configured the environment and built the model, I customized this by uploading a new training set experimenting with various epochs and thresholds.

考虑到YOLO的受欢迎程度，速度和准确性，以及在Roboflow上利用工具的便利性 ，尝试YOLO v5模型是一个显而易见的选择。 从配置环境并构建模型的Google Colaboratory模板开始，我通过上传新的训练集来进行自定义，该训练集尝试了各种时期和阈值。

步骤4：探索模型结果 (Step 4: Explore model results)

The results were impressive and informative.

结果令人印象深刻，内容丰富。

Impressive — the YOLO v5 model trained for 500 epochs in about an hour on the 900+ images, ran inference (prediction) on a new image in about 12 msec, and achieved mAP@0.5 of about 70%.

令人印象深刻 – YOLO v5模型在900幅以上的图像上训练了大约一个小时的500个历元，在大约12毫秒内对新图像进行了推断(预测)，并达到了70％的mAP@0.5。

Individually viewing and rating the model’s success on test images would be a bit of a chore, though, so this is where the mAP@0.5 metric is valuable.

但是，单独查看模型并在测试图像上对模型的成功进行评分会比较麻烦，因此这是mAP@0.5指标很有价值的地方。

mAP@0.5

mAP@0.5

This metric contains two pieces. First, ‘mAP’ indicates the mean Average Precision or correctness of each of the 3 labels taking into account all labels. Second, ‘@0.5’ sets a threshold for how much of the predicted fish bounding box overlaps the original annotation. This second part is a key metric in object detection; it prevents the model from getting credit for guessing the correct fish but drawing a box around some other artifact (like a shadow) instead of an actual fish.

该指标包含两部分。 首先，“ mAP”表示考虑了所有标签的3个标签中的每个标签的平均平均精度或正确性。 第二，“ @ 0.5”为预测的鱼边界框与原始注释重叠的阈值进行设置。 第二部分是对象检测中的关键指标。 它会阻止模型因猜测正确的鱼而获得信誉，但会在其他一些人工物体(例如阴影)周围绘制方框而不是实际的鱼。

This model achieved a mAP@0.5 of 70% —but is that good or bad? For some applications, it’s more than enough. In this particular application, there’s more to it than the label assigned to a single image, and ideally the results of a full solution need to be compared to an estimated error in our current fish counting methods.

该模型的mAP@0.5为70％，但这是好是坏？ 对于某些应用程序，这已经绰绰有余。 在这个特定的应用程序中，除了分配给单个图像的标签之外，还有更多其他功能，并且理想情况下，需要将完整解决方案的结果与我们当前的鱼类计数方法中的估计误差进行比较。

Informative — altered lighting conditions, shadows, and crowded conditions can result in both under and over counts. In the image below, the confidence threshold was intentionally set low at 0.4 to expose corner cases or images that provide a challenge for the model. Keep in mind that the goal is to see if we’re ready to use this for a real-life challenge, and that means looking for problems!

信息化 -改变照明条件，阴影和拥挤的条件下可能会导致这两个下和超过计数。 在下图中，将置信度阈值故意设置为0.4低，以暴露对模型提出挑战的极端情况或图像。 请记住，目标是查看我们是否准备好将其用于现实生活中的挑战，这意味着寻找问题！

In the actual application, when the objects are tracked from one frame of video to the next, shadows often move and the label disappears. This next level of challenge will be to address the object tracking from frame to frame so that there is exactly 1 count for each fish, no matter how much time it remains in the viewing window.

在实际应用中，当将对象从视频的一帧跟踪到下一帧时，阴影经常移动并且标签消失。 接下来的挑战将是解决从一个帧到另一个帧的对象跟踪问题，以便每条鱼的精确计数为1，无论它在观察窗口中保留了多少时间。

步骤5：考虑后续步骤 (Step 5: Consider next steps)

Now the most important step — what did I learn?

现在最重要的一步-我学到了什么？

• Images with excellent lighting are required需要具有出色照明的图像
• Viewing window height and width are not critical, but the depth needs to be carefully selected to reduce the number of fish that can obscure other fish查看窗口的高度和宽度不是关键，但是需要仔细选择深度以减少可能使其他鱼遮挡的鱼的数量
• Correct species labels are required for training a model to separate sockeye, chinook, and coho in addition to other species. Not being a fish identification expert, there is the possibility that I mistook a scar for a small adipose fin. Correcting mislabeled images is another way to improve the model.训练模型以分离其他种类的红大马，chinook和coho时，需要正确的物种标签。 如果不是鱼类鉴定专家，我可能会误以为是小脂肪鳍的伤痕。 校正贴错标签的图像是改善模型的另一种方法。
• Salmon swimming upstream in a fish ladder pause to rest for varying amounts of time. In some cases, they will swim slowly and maintain position, and at other times they will slow to the point that they drift backward with the current. This adds an additional level of complexity that will require an advanced system to track objects (fish) from one video frame to the next.在鱼梯上游游的鲑鱼会停下来休息不同的时间。 在某些情况下，它们会缓慢游动并保持姿势，而在另一些情况下，它们会减速到随着电流向后漂移的程度。 这增加了额外的复杂度，这将需要高级系统来跟踪从一个视频帧到下一个视频帧的对象(鱼)。

Most importantly, while the metrics above are from the better images, most of this learning happened on the first set of images, underscoring the point that excellent images are not required to make progress in trying out a project with deep learning and object detection. In other words, not finding ideal images shouldn’t hold you up from getting started!

最重要的是，尽管上述指标来自更好的图像，但大多数学习是在第一组图像上进行的，这突出说明了在尝试进行深度学习和对象检测的项目时，不需要出色的图像即可取得进展。 换句话说，没有找到理想的图像不应阻止您入门！

结论 (Conclusion)

Based on the project so far, I think it’s fair to say that machine learning / AI / deep learning are ready to be taken out of the lab and applied to real world projects.

基于到目前为止的项目，我认为可以说机器学习/人工智能/深度学习已经可以从实验室中带出并应用于实际项目中了。

This is good news — as an indicator species, salmon help us assess the health of our environment. Accurate and timely fish counts are critical to ensuring their survival, and ours.

这是一个好消息-鲑鱼作为指示物种，可以帮助我们评估环境的健康。 准确及时的鱼类计数对于确保它们以及我们的生存至关重要。

Contact me! I’m open to new opportunities and actively looking for my next exciting data science gig.

联络我！ 我乐于接受新的机遇，并积极寻找下一个令人兴奋的数据科学演出。

Link to Repository

链接到存储库

The notebooks used for this project are posted in github. Keep in mind that these are designed to be used on Google Colab and will not run locally. The 3rd notebook contains a pared down version that runs inference only using saved model weights.

用于该项目的笔记本发布在github中。 请记住，这些文件只能在Google Colab上使用，不能在本地运行。 第三个笔记本包含精简版，仅使用保存的模型权重运行推断。

https://github.com/jshaffer94247/Counting-Fish

https://github.com/jshaffer94247/Counting-Fish