足球视频AI(二)——球员与球的目标检测

一、基础概念

1.1 识别目标：

1）固定机位的视频中球员逐帧识别

2）固定机位的视频中球逐帧识别

3）位置换算与记录

1.2 实现思路

1，利用OpenCV的相邻帧差异识别移动物体

2，利用YOLO7的机器学习识别对象，本文主要介绍YOLO7方案

二、代码实现

依赖项：

Nuget Install IVilson.AI.Yolov7net
Nuget Install OpenCvSharp4
Nuget Install OpenCvSharp4.Extensions
Nuget Install OpenCvSharp4.runtime.win
Nuget Install Numpy.Bare

2.1 接口定义

    public interface IDetector<T> :IDisposable{List<T> Detect(Mat mat);}

2.2 OpenCV实现

 public class DetectOpenCV : IDetector<YoloPrediction>{private readonly BackgroundSubtractorGMG fgbg;public DetectOpenCV(){fgbg = BackgroundSubtractorGMG.Create(5);}public event EventHandler<byte[]>? OnMiddelTime;/// <inheritdoc/>public List<YoloPrediction> Detect(Mat mat){Point[][] contours;HierarchyIndex[] hierarchies;List<YoloPrediction> predictions;var topRows = (int)mat.Rows / 8;var midleRows = Convert.ToInt32(mat.Rows / 3.5);predictions = new List<YoloPrediction>();//高斯处理Cv2.GaussianBlur(mat, mat, new Size(13, 13), 0);//二值化var thresh = BackgroundSubtract(mat);OnMiddelTime?.Invoke(null, thresh.ToBytes());//视距最远端处理var k1 = np.array(new[,]{ {1, 1, 1, 1},{1,1,1,1},{1,1,1,1},{0,1,1,0},{0,1,1,0},{1,1,1,1},{1,1,1,1},{1,1,1,1}}, np.uint8);var k1m = InputArray.Create(k1.GetData<int>(), MatType.CV_8U);var k2 = Mat.Ones(25, 15, MatType.CV_8UC1);var up_area = new Mat(thresh, new Rect(0, 0, thresh.Rows, topRows));Cv2.Erode(up_area, up_area, k1m);Cv2.Dilate(up_area, up_area, k2);//视距中间处理k1 = np.array(new[,]{ {1, 1, 1, 1},{1,1,1,1},{0,1,1,0},{0,1,1,0},{0,1,1,0},{1,1,1,1}}, np.uint8);k1m = InputArray.Create(k1.GetData<int>(), MatType.CV_8U);k2 = Mat.Ones(30, 20, MatType.CV_8UC1);var middel_area = new Mat(thresh, new Rect(0, topRows, thresh.Rows, midleRows - topRows));Cv2.Erode(middel_area, middel_area, k1m);Cv2.Dilate(middel_area, middel_area, k2);//视距近景处理var down_area = new Mat(thresh, new Rect(0, midleRows, thresh.Rows, thresh.Height - midleRows));k2 = Mat.Ones(60, 30, MatType.CV_8UC1);Cv2.MorphologyEx(down_area, down_area, MorphTypes.Close, k2);//边缘检测Cv2.FindContours(thresh, out contours, out hierarchies, RetrievalModes.List, ContourApproximationModes.ApproxNone);//检测对象的转换var lable = new YoloLabel() { Id = 0, Kind = YoloLabelKind.Generic, Name = "person" };foreach (var c in contours){var rect = Cv2.BoundingRect(c);if (rect.Width < 20 || rect.Height < 30)continue;if (rect.Y >= midleRows && rect.Height <= 50)continue;if (midleRows >= rect.Y && rect.Y >= topRows && rect.Height <= 35)continue;var prediction = new YoloPrediction(lable);prediction.Rectangle = new System.Drawing.RectangleF(rect.X, rect.Y, rect.Width, rect.Height);predictions.Add(prediction);}return predictions;}/// <summary>/// 球场去背景/// </summary>/// <param name="frame"></param>public Mat BackgroundSubtract(Mat frame){var fg = new Mat();fgbg.Apply(frame, fg);return fg;}public void Dispose(){fgbg?.Dispose();}}

调用目标检测

        [Fact]public void TestCVFindContours(){var detector = new DetectOpenCV();List<YoloPrediction> lst;using (var mat = LoadImages.Load("field_2.jpg")){lst = detector.Detect(mat);}Assert.True(lst.Count > 0);}

2.3 YoloV7实现

https://github.com/WongKinYiu/yolov7

YOLOv7：可训练的免费赠品袋为实时物体检测器设置了新的最先进技术

2.3.1 编译yolov7-tiny.onnx

1, Github 源码下载到本地并安装Python环境。

2, 下载yolov7-tiny.pt

安装Visual Studio Code 开发工具
打开工具->文件->打开目录...  选择yolov7目录->终端->新建终端

将下载的yolov7-tiny.pt 拷贝到yolov7根目录

按yolov7官网的说明,在新建的命令终端执行以下指令

python export.py --weights=yolov7.pt --grid --simplify

查看输出的yolov7-tiny.onnx 文件，拷贝到C#项目中

2.3.2 集成C#

    public class DetectorYolov7 : IDetector<YoloPrediction>{private readonly Yolov7 _yolo;public DetectorYolov7(){_yolo = new Yolov7(Path.Combine(System.AppDomain.CurrentDomain.BaseDirectory, "Assets/yolov7-tiny.onnx"));_yolo.SetupYoloDefaultLabels();}/// <inheritdoc/>public List<YoloPrediction> Detect(Mat mat){var items = _yolo.Predict(mat.ToBitmap());return items;}public void Dispose(){_yolo?.Dispose();}}

调用目标检测

        [Fact]public void TestYoloDetect(){var detector = new DetectorYolov7();List<YoloPrediction> lst;using (var mat = LoadImages.Load("field_2.jpg")){lst = detector.Detect(mat);}Assert.True(lst.Count > 0);}

2.4 标记人或球

        /// <summary>/// 单个对象上画矩形框/// </summary>/// <param name="item"></param>/// <param name="image"></param>protected virtual void PlotBox(YoloPrediction item, Mat image){var x = item.Rectangle.X;var y = item.Rectangle.Y;var width = item.Rectangle.Width;var height = item.Rectangle.Height;var lineSize = (int)Math.Floor((double)image.Cols / 1000);lineSize = lineSize <1 ? 1: lineSize;if (item.Label?.Name?.Equals("sports ball") == true)Cv2.Rectangle(image, new OpenCvSharp.Rect((int)Math.Floor(x), (int)Math.Floor(y), (int)Math.Floor(width), (int)Math.Floor(height)), Scalar.White, lineSize);else if (item.Label?.Name?.Equals("person") == true)Cv2.Rectangle(image, new OpenCvSharp.Rect((int)Math.Floor(x), (int)Math.Floor(y), (int)Math.Floor(width), (int)Math.Floor(height)), Scalar.Yellow, lineSize);elseCv2.Rectangle(image, new OpenCvSharp.Rect((int)Math.Floor(x), (int)Math.Floor(y), (int)Math.Floor(width), (int)Math.Floor(height)), Scalar.Green, lineSize);if (item.Label?.Id > 0)Cv2.PutText(image, $"{item.Label?.Id}", new OpenCvSharp.Point(x, y), HersheyFonts.HersheySimplex, 0.5, Scalar.AliceBlue, lineSize);}

三、总结

实际应用中，Yolov7的目标检测的优势：

1，在广角、侧面、固定机位的镜头下（如足球转播）效果比Opencv明显较好。

2，识别准确率高

劣势：

1，在无人机（高空人物非常小）效果不好，需要训练

2，在广角（鱼眼）效果不好，需要训练

3，在帧处理性能，尤其是CPU设备上处理能力较慢（300毫秒一帧），而GPU能达到30毫秒。

存在不足的地方是，高分辨率、远景的足球视频中，对于足球的识别非常不准确。

因此，后续我们介绍如何自训练Yolov7的模型。