53、RK3588测试视频编解码和 POE OAK Camera编码结合开发
2024-05-24 03:28:25
基本思想：一直想学rk3588的视频编解码，奈何没有设备，最近获得机会，利用空闲时间好好研究一番，正好手中的深度相机oak camera支持视频编码，逐想用软解编码和瑞芯微的mpp硬解码去走一波,本实验使用的poe-rj45接口和usb低电压接口测试
测试数据
硬件：rk3588s开发板oak-d s2深度相机(usb接口)
技术：rk3588s mpp硬解码oak h264编码（最高帧率60fps）yolov7-tiny 单目标检测
硬件频率设置:cpu 频率 408000dmc频率 2112000000npu频率 1000000000目标检测准确的情况下，测试数据如下：解码 总帧率 56-60fps解码+640推理(15ms)总帧率 32fps解码+416推理(7ms) 总帧率 47fps解码+320推理(4ms) 总帧率 48-54fps
首先准备测试视频，截取20s的视频链接: https://pan.baidu.com/s/1_3wff-xizk4G1xBluTtsmQ?pwd=fmc9 提取码: fmc9
ubuntu@ubuntu:~/test_rk3588_mpp$ ffmpeg -ss 00:00:00 -i 1920x1080.mp4 -to 00:00:20 -c:v copy -c:a copy 1920x1080_min.mp4
预先转h264
ubuntu@ubuntu:~/test_rk3588_mpp$ ffmpeg -i 1920x1080_min.mp4 -codec copy -bsf: h264_mp4toannexb -f h264 1920x1080_min.h264一、进入系统
ubuntu@ubuntu:~$ ssh firefly@192.168.71.30
The authenticity of host '192.168.71.30 (192.168.71.30)' can't be established.
ECDSA key fingerprint is SHA256:EABsn4kWnfnqvYNreTaJf6QPvgKnvJPYSFBDsQglD8k.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.71.30' (ECDSA) to the list of known hosts.
firefly@192.168.71.30's password: _____ _ __ _
| ___(_)_ __ ___ / _| |_ _
| |_ | | '__/ _ \ |_| | | | |
| _| | | | | __/ _| | |_| |
|_| |_|_| \___|_| |_|\__, ||___/
Welcome to Ubuntu 20.04.4 LTS (GNU/Linux 5.10.110-rt53 aarch64)* Documentation: http://wiki.t-firefly.com* Management: http://www.t-firefly.comSystem information as of Sun Jun 26 04:18:31 UTC 2022System load: 0.86 0.93 0.67 Up time: 18:42 hours Local users: 2
Memory usage: 34 % of 3464MB IP: 192.168.71.30
Usage of /: 1% of 23G The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.
1)下载源码和测试mpp
测试命令
firefly@firefly:~$ git clone https://github.com/rockchip-linux/mpp.git
firefly@firefly:~$ cd mpp
firefly@firefly:~/mpp$ mkidr build
firefly@firefly:~/mpp/build$ cmake ..&make&sudo make install
firefly@firefly:~/mpp/build/test$ sudo ./mpi_dec_test -t 7 -i 1920x1080_min.h264 -f 4 -h 1080 -w 1920 -o 1920x1080_min.nv12
firefly@firefly:~/mpp/build/test$ ffplay -f rawvideo -video_size 1920*1080 -pixel_format nv12 1920x1080_min.nv12
可以播放
监控日志
firefly@firefly:~$ watch -n 1 tail -f /var/log/syslog
....
Every 1.0s: tail -f /var/log/syslog firefly: Mon Jan 16 07:43:58 2023Jan 16 07:43:56 firefly mpp[12221]: mpi_dec_test: 0x5587aa7160 decode get frame 1896
Jan 16 07:43:57 firefly mpp[12221]: mpi_dec_test: 0x5587aa7160 decode get frame 1897
Jan 16 07:43:57 firefly mpp[12221]: mpi_dec_test: 0x5587aa7160 decode get frame 1898
Jan 16 07:43:57 firefly mpp[12221]: mpi_dec_test: 0x5587aa7160 decode get frame 18992)下载源码和测试（摘自官方测试）ff_media
# 根据操作系统获取源码
git clone https://gitlab.com/firefly-linux/ff_media -b ubuntu20.04# 安装所需环境
apt install gcc g++ make cmake
apt install libasound2-dev
apt install libopencv-dev# 编译
cd ff_media
mkdir build; cd build
cmake ../
make# 运行 demo
## 直接运行 demo 可以查看帮助信息
INFO: usage: Usage: ./demo <Input source> [Options]Options:
-i, --input Input image size
-o, --output Output image size, default same as input
-a, --inputfmt Input image format, default MJPEG
-b, --outputfmt Output image format, default NV12
-c, --count Instance count, default 1
-d, --drmdisplay Drm display, set display plane, set 0 to auto find plane, default disabled
-e, --encodetype Encode encode, set encode type, default disabled
-f, --file Enable save dec output data to file, set filename, default disabled
-p, --port Enable rtsp stream, set push port, depend on encode enabled, default disabled
-m, --enmux Enable save encode data to mp4 file, depend on encode enabled, default disabled
-r, --rotate Image rotation degree, default 00: none1: vertical mirror2: horizontal mirror90: 90 degree180: 180 degree270: 270 degree## 示范：输入是分辨率为 1080p 的 rtsp 摄像头，把解码图像缩放为 720p 输出到显示器上。
./demo 1920x1080.h264 -o 1280x720 -d 0
#打开usb相机
./demo /dev/video12 -i 640x480 -o 640x480 -d 0 测试没问题，速度很快,下一步需要结合官方提供的demo和例子，修改自己可以用的代码
二、先用笔记本测试ffmpeg软解码h264
1)测试软解码在54fps
cmake_minimum_required(VERSION 3.16)
project(untitled1)
find_package(OpenCV REQUIRED)
set(CMAKE_CXX_STANDARD 14)add_executable(untitled1 main.cpp)
target_link_libraries(untitled1${OpenCV_LIBS}-lavformat -lavcodec -lswscale -lavutil -lz)测试代码
#include <stdio.h>
#include <string>
#include <iostream>
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>extern "C"
{#include "libavcodec/avcodec.h"
#include "libavformat/avformat.h"
#include "libswscale/swscale.h"
#include "libavutil/avutil.h"
#include "libavutil/imgutils.h"};
#include <chrono>
#include <iomanip>using namespace std;
using namespace chrono;
int main(int argc, char *argv[]) {const char *path = "/home/ubuntu/test_rk3588_mpp/1920x1080.h264";
// AVDictionary *optional=NULL;
// av_dict_set(&optional,"stimeout","6000000",0);
// av_dict_set(&optional,"rstp_transport","tcp",0);
// av_dict_set(&optional,"buffer_size","1024000",0);
// av_dict_set(&optional,"max_delay","500000",0);AVFormatContext *pFormat = NULL;int ret = avformat_open_input(&pFormat, path, NULL, NULL);if (ret < 0) {perror("avformat_open_input");avformat_free_context(pFormat);return -1;}// av_dict_free(&optional);printf("avformat_open_input successfully\n");ret = avformat_find_stream_info(pFormat, NULL);if (ret < 0) {perror("avformat_find_stream_info\n");return -1;}printf("avformat_find_stream_info successfully\n");int time = pFormat->duration;int mbittime = (time / 100000) / 60;int mminttime = (time / 100000) % 60;printf("video time: %d'm %d's\n", mbittime, mminttime);av_dump_format(pFormat, 0, path, 0);int videoindex = -1;for (int i = 0; i < pFormat->nb_streams; i++) {if (pFormat->streams[i]->codecpar->codec_type == AVMEDIA_TYPE_VIDEO) {videoindex = i;break;}}if (videoindex == -1) {printf("don't find video stream\n");return -1;}AVCodecParameters *codecParameters = pFormat->streams[videoindex]->codecpar;printf("video width %d\n", codecParameters->width);printf("video height %d\n", codecParameters->height);AVCodec *pCodec = avcodec_find_decoder(codecParameters->codec_id);AVCodecContext *pCodecCtx = avcodec_alloc_context3(pCodec);ret = avcodec_open2(pCodecCtx, pCodec, NULL);if (ret < 0) {//打开解码器printf("Could not open codec.\n");return -1;}AVFrame *picture = av_frame_alloc();picture->width = codecParameters->width;picture->height = codecParameters->height;picture->format = AV_PIX_FMT_YUV420P;ret = av_frame_get_buffer(picture, 1);if (ret < 0) {printf("av_frame_get_buffer error\n");return -1;}printf("picture->linesize[0] %d\n", picture->linesize[0]);AVFrame *pFrame = av_frame_alloc();pFrame->width = codecParameters->width;pFrame->height = codecParameters->height;pFrame->format = AV_PIX_FMT_YUV420P;ret = av_frame_get_buffer(pFrame, 1);if (ret < 0) {printf("av_frame_get_buffer error\n");return -1;}AVFrame *pFrameRGB = av_frame_alloc();pFrameRGB->width = codecParameters->width;pFrameRGB->height = codecParameters->height;pFrameRGB->format = AV_PIX_FMT_RGB24;ret = av_frame_get_buffer(pFrameRGB, 1);if (ret < 0) {printf("av_frame_get_buffer error\n");return -1;}int picture_size = av_image_get_buffer_size(AV_PIX_FMT_YUV420P, codecParameters->width, codecParameters->height,1);//计算这个格式的图片，需要多少字节来存储uint8_t *out_buff = (uint8_t *) av_malloc(picture_size * sizeof(uint8_t));av_image_fill_arrays(picture->data, picture->linesize, out_buff, AV_PIX_FMT_YUV420P, codecParameters->width,codecParameters->height, 1);//这个函数 是缓存转换格式，可以不用 以为上面已经设置了AV_PIX_FMT_YUV420PSwsContext *img_convert_ctx = sws_getContext(codecParameters->width, codecParameters->height, AV_PIX_FMT_YUV420P,codecParameters->width, codecParameters->height, AV_PIX_FMT_RGB24, 4,NULL, NULL, NULL);AVPacket *packet = (AVPacket *) av_malloc(sizeof(AVPacket));auto startTime = steady_clock::now();int counter = 0;float fps = 0;while (av_read_frame(pFormat, packet) >= 0) {if (packet->stream_index == videoindex) {ret = avcodec_send_packet(pCodecCtx, packet);if (ret < 0) {printf("avcodec_send_packet error\n");continue;}av_packet_unref(packet);int got_picture = avcodec_receive_frame(pCodecCtx, pFrame);if (got_picture < 0) {printf("avcodec_receive_frame error\n");continue;}sws_scale(img_convert_ctx, pFrame->data, pFrame->linesize, 0,codecParameters->height,pFrameRGB->data, pFrameRGB->linesize);cv::Mat mRGB(cv::Size(codecParameters->width, codecParameters->height), CV_8UC3);mRGB.data = (unsigned char *) pFrameRGB->data[0];cv::Mat mBGR;cv::cvtColor(mRGB, mBGR, cv::COLOR_RGB2BGR);counter++;auto currentTime = steady_clock::now();auto elapsed = duration_cast<duration<float>>(currentTime - startTime);if (elapsed > seconds(1)) {fps = counter / elapsed.count();counter = 0;startTime = currentTime;}std::stringstream fpsStr;fpsStr << "NN fps: " << std::fixed << std::setprecision(2) << fps;cv::putText(mBGR, fpsStr.str(), cv::Point(2, 20), cv::FONT_HERSHEY_TRIPLEX, 0.4, cv::Scalar(0,255,0));cv::imshow("demo", mBGR);cv::waitKey(1);}}av_frame_free(&picture);av_frame_free(&pFrame);av_frame_free(&pFrameRGB);avformat_free_context(pFormat);return 0;
}测试结果
2)修改第一版使用硬件解码h264然后显示，测试帧率
https://github.com/sxj731533730/h264_mpp_yuv
在rk3588 硬件测试mpp解码速度在200fps
测试结果
三、先以ffmpeg软解码去解析oak的视频帧，测试代码，以yolov7-tiny模型为例子
cmakelists.txt
cmake_minimum_required(VERSION 3.16)
project(depthai)
set(CMAKE_CXX_STANDARD 11)
find_package(OpenCV REQUIRED)
#message(STATUS ${OpenCV_INCLUDE_DIRS})
#添加头文件
include_directories(${OpenCV_INCLUDE_DIRS})
include_directories(${CMAKE_SOURCE_DIR}/include)
include_directories(${CMAKE_SOURCE_DIR}/include/utility)
#链接Opencv库
find_package(depthai CONFIG REQUIRED)
add_executable(depthai main.cpp include/utility/utility.cpp)
target_link_libraries(depthai ${OpenCV_LIBS} depthai::opencv -lavformat -lavcodec -lswscale -lavutil -lz)main.cpp
#include <stdio.h>
#include <string>
#include <iostream>
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>extern "C"
{
#include <libavformat/avformat.h>
#include <libavcodec/avcodec.h>
#include <libavutil/imgutils.h>
#include <libswscale/swscale.h>
}#include "utility.hpp"
#include <vector>
#include "depthai/depthai.hpp"static std::atomic<bool> newConfig{false};
using namespace std;
using namespace std::chrono;
using namespace cv;
typedef struct {int width;int height;
} YoloSize;typedef struct {std::string name;int stride;std::vector<YoloSize> anchors;
} YoloLayerData;class BoxInfo {
public:int x1, y1, x2, y2, label, id;float score;
};static inline float sigmoid(float x) {return static_cast<float>(1.f / (1.f + exp(-x)));
}double GetIOU(cv::Rect_<float> bb_test, cv::Rect_<float> bb_gt) {float in = (bb_test & bb_gt).area();float un = bb_test.area() + bb_gt.area() - in;if (un < DBL_EPSILON)return 0;return (double) (in / un);
}std::vector<BoxInfo> decode_infer(vector<float> data_pt, int stride, int net_size, int num_classes,const std::vector<YoloSize> &anchors, float threshold, int idx) {const int s[3] = {20, 40, 80};std::vector<BoxInfo> result;int batchs, channels, height, width, pred_item;batchs = 1;channels = 3;height = s[idx];width = s[idx];pred_item = 6;float data_ptr[batchs * channels * height * width * pred_item];//std::cout << batchs * channels * height * width * pred_item << std::endl << data_pt.size() << std::endl;for (int i = 0; i < data_pt.size(); i++) {data_ptr[i] = data_pt[i];}for (int bi = 0; bi < batchs; bi++) {auto batch_ptr = data_ptr + bi * (channels * height * width * pred_item);for (int ci = 0; ci < channels; ci++) {auto channel_ptr = batch_ptr + ci * (height * width * pred_item);for (int hi = 0; hi < height; hi++) {auto height_ptr = channel_ptr + hi * (width * pred_item);for (int wi = 0; wi < width; wi++) {auto width_ptr = height_ptr + wi * pred_item;auto cls_ptr = width_ptr + 5;auto confidence = sigmoid(width_ptr[4]);for (int cls_id = 0; cls_id < num_classes; cls_id++) {float score = sigmoid(cls_ptr[cls_id]) * confidence;if (score > threshold) {float cx = (sigmoid(width_ptr[0]) * 2.f - 0.5f + wi) * (float) stride;float cy = (sigmoid(width_ptr[1]) * 2.f - 0.5f + hi) * (float) stride;float w = pow(sigmoid(width_ptr[2]) * 2.f, 2) * anchors[ci].width;float h = pow(sigmoid(width_ptr[3]) * 2.f, 2) * anchors[ci].height;BoxInfo box;box.x1 = std::max(0, std::min(net_size, int((cx - w / 2.f))));box.y1 = std::max(0, std::min(net_size, int((cy - h / 2.f))));box.x2 = std::max(0, std::min(net_size, int((cx + w / 2.f))));box.y2 = std::max(0, std::min(net_size, int((cy + h / 2.f))));box.score = score;box.label = cls_id;result.push_back(box);}}}}}}return result;
}void nms(std::vector<BoxInfo> &input_boxes, float NMS_THRESH) {std::sort(input_boxes.begin(), input_boxes.end(), [](BoxInfo a, BoxInfo b) { return a.score > b.score; });std::vector<float> vArea(input_boxes.size());for (int i = 0; i < int(input_boxes.size()); ++i) {vArea[i] = (input_boxes.at(i).x2 - input_boxes.at(i).x1 + 1)* (input_boxes.at(i).y2 - input_boxes.at(i).y1 + 1);}for (int i = 0; i < int(input_boxes.size()); ++i) {for (int j = i + 1; j < int(input_boxes.size());) {float xx1 = std::max(input_boxes[i].x1, input_boxes[j].x1);float yy1 = std::max(input_boxes[i].y1, input_boxes[j].y1);float xx2 = std::min(input_boxes[i].x2, input_boxes[j].x2);float yy2 = std::min(input_boxes[i].y2, input_boxes[j].y2);float w = std::max(float(0), xx2 - xx1 + 1);float h = std::max(float(0), yy2 - yy1 + 1);float inter = w * h;float ovr = inter / (vArea[i] + vArea[j] - inter);if (ovr >= NMS_THRESH) {input_boxes.erase(input_boxes.begin() + j);vArea.erase(vArea.begin() + j);} else {j++;}}}
}void scale_coords(std::vector<BoxInfo> &boxes, int w_from, int h_from, int w_to, int h_to) {float w_ratio = float(w_to) / float(w_from);float h_ratio = float(h_to) / float(h_from);for (auto &box: boxes) {box.x1 *= w_ratio;box.x2 *= w_ratio;box.y1 *= h_ratio;box.y2 *= h_ratio;}return;
}cv::Mat draw_box(cv::Mat &cv_mat, std::vector<BoxInfo> &boxes, const std::vector<std::string> &labels,unsigned char colors[][3]) {for (auto box : boxes) {int width = box.x2 - box.x1;int height = box.y2 - box.y1;cv::Point p = cv::Point(box.x1, box.y1);cv::Rect rect = cv::Rect(box.x1, box.y1, width, height);cv::rectangle(cv_mat, rect, cv::Scalar(colors[box.label][0], colors[box.label][1], colors[box.label][2]));string text = labels[box.label] + ":" + std::to_string(box.score);cv::putText(cv_mat, text, p, cv::FONT_HERSHEY_PLAIN, 1,cv::Scalar(colors[box.label][0], colors[box.label][1], colors[box.label][2]));}return cv_mat;
}int main(int argc, char **argv) {std::vector<YoloLayerData> yolov7_layers{{"324", 32, {{116, 90}, {156, 198}, {373, 326}}},{"304", 16, {{30, 61}, {62, 45}, {59, 119}}},{"output", 8, {{12, 16}, {16, 30}, {33, 23}}},};vector<float> origin_rect_cof;std::vector<YoloLayerData> &layers = yolov7_layers;float threshold = 0.4;float nms_threshold = 0.5;int i = 0;std::vector<std::string> labels = {"cup"};unsigned char colors[][3] = {{0, 255, 0}};float target_width = 640;float target_height = 640;dai::Pipeline pipeline;auto camRgb = pipeline.create<dai::node::ColorCamera>();auto rgbOut = pipeline.create<dai::node::XLinkOut>();rgbOut->setStreamName("rgbOut");//camRgb->video.link(rgbOut->input);camRgb->setPreviewSize(640, 640);camRgb->setBoardSocket(dai::CameraBoardSocket::RGB);camRgb->setResolution(dai::ColorCameraProperties::SensorResolution::THE_1080_P);camRgb->setInterleaved(false);camRgb->setColorOrder(dai::ColorCameraProperties::ColorOrder::RGB);//camRgb->setFps(20);camRgb->setPreviewKeepAspectRatio(false);auto monoLeft = pipeline.create<dai::node::MonoCamera>();auto monoRight = pipeline.create<dai::node::MonoCamera>();auto stereo = pipeline.create<dai::node::StereoDepth>();auto spatialDataCalculator = pipeline.create<dai::node::SpatialLocationCalculator>();auto Encoder = pipeline.create<dai::node::VideoEncoder>();Encoder->setDefaultProfilePreset(camRgb->getVideoSize(), camRgb->getFps(),dai::VideoEncoderProperties::Profile::H265_MAIN);//camRgb->video.link(Encoder->input);//定义//auto cam = pipeline.create<dai::node::XLinkIn>();//camRgb->setStreamName("inFrame");auto net = pipeline.create<dai::node::NeuralNetwork>();dai::OpenVINO::Blob blob("../model_300_300/best_cup.blob");net->setBlob(blob);net->input.setBlocking(false);//基本熟练明白oak的函数使用了 camRgb->preview.link(net->input);camRgb->setFps(20);//camRgb->setVideoSize(1920, 1080);auto xoutDepth = pipeline.create<dai::node::XLinkOut>();auto xoutSpatialData = pipeline.create<dai::node::XLinkOut>();auto xinSpatialCalcConfig = pipeline.create<dai::node::XLinkIn>();xoutDepth->setStreamName("depth");xoutSpatialData->setStreamName("spatialData");xinSpatialCalcConfig->setStreamName("spatialCalcConfig");monoLeft->setResolution(dai::MonoCameraProperties::SensorResolution::THE_400_P);monoLeft->setBoardSocket(dai::CameraBoardSocket::LEFT);monoRight->setResolution(dai::MonoCameraProperties::SensorResolution::THE_400_P);monoRight->setBoardSocket(dai::CameraBoardSocket::RIGHT);stereo->setDefaultProfilePreset(dai::node::StereoDepth::PresetMode::HIGH_ACCURACY);stereo->setLeftRightCheck(true);stereo->setDepthAlign(dai::CameraBoardSocket::RGB);stereo->setExtendedDisparity(true);// LR-check is required for depth alignmentdai::Point2f topLeft(0.4f, 0.4f);dai::Point2f bottomRight(0.6f, 0.6f);dai::SpatialLocationCalculatorConfigData config;config.depthThresholds.lowerThreshold = 100;config.depthThresholds.upperThreshold = 10000;config.roi = dai::Rect(topLeft, bottomRight);spatialDataCalculator->inputConfig.setWaitForMessage(false);spatialDataCalculator->initialConfig.addROI(config);// LinkingmonoLeft->out.link(stereo->left);monoRight->out.link(stereo->right);spatialDataCalculator->passthroughDepth.link(xoutDepth->input);stereo->depth.link(spatialDataCalculator->inputDepth);spatialDataCalculator->out.link(xoutSpatialData->input);xinSpatialCalcConfig->out.link(spatialDataCalculator->inputConfig);//定义输出auto xlinkoutpreviewOut = pipeline.create<dai::node::XLinkOut>();xlinkoutpreviewOut->setStreamName("out264");Encoder->bitstream.link(xlinkoutpreviewOut->input);//定义输出auto xlinkParserOut = pipeline.create<dai::node::XLinkOut>();xlinkParserOut->setStreamName("parseOut");auto xlinkoutOut = pipeline.create<dai::node::XLinkOut>();xlinkoutOut->setStreamName("out");auto xlinkoutpassthroughOut = pipeline.create<dai::node::XLinkOut>();xlinkoutpassthroughOut->setStreamName("passthrough");net->out.link(xlinkParserOut->input);net->passthrough.link(xlinkoutpassthroughOut->input);auto manip = pipeline.create<dai::node::ImageManip>();manip->initialConfig.setResize(target_width, target_height);manip->initialConfig.setFrameType(dai::ImgFrame::Type::NV12);camRgb->preview.link(manip->inputImage);manip->out.link(Encoder->input);//结构推送相机dai::Device device(pipeline,true);//取帧显示// auto inqueue = device.getInputQueue("inFrame");//maxsize 代表缓冲数据auto detqueue = device.getOutputQueue("parseOut", 8, false);//maxsize 代表缓冲数据auto passthrough = device.getOutputQueue("passthrough", 8, false);//maxsize 代表缓冲数据auto depthQueue = device.getOutputQueue("depth", 8, false);auto spatialCalcQueue = device.getOutputQueue("spatialData", 8, false);auto spatialCalcConfigInQueue = device.getInputQueue("spatialCalcConfig");auto videoqueue = device.getOutputQueue("out264", camRgb->getFps(), false);bool printOutputLayersOnce = true;auto color = cv::Scalar(0, 255, 0);auto startTime = steady_clock::now();int counter = 0;float fps = 0;int width = target_width;int height = target_height;AVCodec *pCodec = avcodec_find_decoder(AV_CODEC_ID_H265);AVCodecContext *pCodecCtx = avcodec_alloc_context3(pCodec);int ret = avcodec_open2(pCodecCtx, pCodec, NULL);if (ret < 0) {//打开解码器char buf[1024] = {0};av_strerror(ret, buf, sizeof(buf) - 1);cerr << buf << endl;return -1;}AVFrame *picture = av_frame_alloc();picture->width = width;picture->height = height;picture->format = AV_PIX_FMT_YUV420P;ret = av_frame_get_buffer(picture, 1);if (ret < 0) {char buf[1024] = {0};av_strerror(ret, buf, sizeof(buf) - 1);cerr << buf << endl;return -1;}AVFrame *pFrame = av_frame_alloc();pFrame->width = width;pFrame->height = height;pFrame->format = AV_PIX_FMT_YUV420P;ret = av_frame_get_buffer(pFrame, 1);if (ret < 0) {char buf[1024] = {0};av_strerror(ret, buf, sizeof(buf) - 1);cerr << buf << endl;return -1;}AVFrame *pFrameRGB = av_frame_alloc();pFrameRGB->width = width;pFrameRGB->height = height;pFrameRGB->format = AV_PIX_FMT_RGB24;ret = av_frame_get_buffer(pFrameRGB, 1);if (ret < 0) {char buf[1024] = {0};av_strerror(ret, buf, sizeof(buf) - 1);cerr << buf << endl;return -1;}int picture_size = av_image_get_buffer_size(AV_PIX_FMT_YUV420P, width, height,1);//计算这个格式的图片，需要多少字节来存储uint8_t *out_buff = (uint8_t *) av_malloc(picture_size * sizeof(uint8_t));av_image_fill_arrays(picture->data, picture->linesize, out_buff, AV_PIX_FMT_YUV420P, width,height, 1);//这个函数 是缓存转换格式，可以不用 以为上面已经设置了AV_PIX_FMT_YUV420PSwsContext *img_convert_ctx = sws_getContext(width, height, AV_PIX_FMT_YUV420P,width, height, AV_PIX_FMT_RGB24, 4,NULL, NULL, NULL);AVPacket *packet = av_packet_alloc();// auto startTime = steady_clock::now();
// int counter = 0;
// float fps = 0;while (true) {auto inDepth = depthQueue->get<dai::ImgFrame>();//auto ImgFrame = videoqueue->get<dai::ImgFrame>();// auto ImgFrame = passthrough->get<dai::ImgFrame>();//auto frame = ImgFrame->getCvFrame();//printf("h264\n");//videoFile.write((char *) (h265Packet->getData().data()), h265Packet->getData().size());auto h264Packet = videoqueue->get<dai::ImgFrame>();packet->data = (uint8_t *) h264Packet->getData().data(); //这里填入一个指向完整H264数据帧的指针packet->size = h264Packet->getData().size(); //这个填入H265 数据帧的大小packet->stream_index = 0;cv::Mat frame;ret = avcodec_send_packet(pCodecCtx, packet);//printf("avcodec_send_packet ret=%d \n", ret);if (ret < 0) {char buf[1024] = {0};av_strerror(ret, buf, sizeof(buf) - 1);cerr << buf << endl;continue;}av_packet_unref(packet);while (ret >= 0) {ret = avcodec_receive_frame(pCodecCtx, pFrame);//printf("avcodec_receive_frame ret=%d \n", ret);av_frame_is_writable(pFrame);if (ret == AVERROR(EAGAIN) || ret == AVERROR_EOF || ret < 0) {
// if (ret < 0) {
// char buf[1024] = {0};
// av_strerror(ret, buf, sizeof(buf) - 1);
// cerr << buf << endl;
// continue;
// }break;}sws_scale(img_convert_ctx, pFrame->data, pFrame->linesize, 0,height,pFrameRGB->data, pFrameRGB->linesize);cv::Mat mRGB(cv::Size(width, height), CV_8UC3);mRGB.data = (unsigned char *) pFrameRGB->data[0];cv::Mat mBGR;cv::cvtColor(mRGB, mBGR, cv::COLOR_RGB2BGR);frame = mBGR;}if(frame.empty()){continue;}target_height = frame.rows * 1.0;target_width = frame.cols * 1.0;cv::Mat depthFrameColor;cv::Mat depthFrame = inDepth->getFrame(); // depthFrame values are in millimeterscv::normalize(depthFrame, depthFrameColor, 255, 0, cv::NORM_INF, CV_8UC1);cv::equalizeHist(depthFrameColor, depthFrameColor);cv::applyColorMap(depthFrameColor, depthFrameColor, cv::COLORMAP_HOT);auto inNN = detqueue->get<dai::NNData>();if (printOutputLayersOnce && inNN) {std::cout << "Output layer names: ";for (const auto &ten : inNN->getAllLayerNames()) {std::cout << ten << ", ";}printOutputLayersOnce = false;}std::vector<BoxInfo> result;std::vector<BoxInfo> boxes;auto output = inNN->getLayerFp16(inNN->getAllLayerNames()[2]);boxes = decode_infer(output, layers[2].stride, 640, labels.size(), layers[2].anchors, threshold, 2);result.insert(result.begin(), boxes.begin(), boxes.end());auto output_534 = inNN->getLayerFp16(inNN->getAllLayerNames()[0]);boxes = decode_infer(output_534, layers[1].stride, 640, labels.size(), layers[1].anchors, threshold, 1);result.insert(result.begin(), boxes.begin(), boxes.end());auto output_554 = inNN->getLayerFp16(inNN->getAllLayerNames()[1]);boxes = decode_infer(output_554, layers[0].stride, 640, labels.size(), layers[0].anchors, threshold, 0);result.insert(result.begin(), boxes.begin(), boxes.end());nms(result, nms_threshold);scale_coords(result, 640, 640, frame.cols, frame.rows);cv::Mat frame_show = draw_box(frame, result, labels, colors);counter++;auto currentTime = steady_clock::now();auto elapsed = duration_cast<duration<float>>(currentTime - startTime);if (elapsed > seconds(1)) {fps = counter / elapsed.count();counter = 0;startTime = currentTime;}std::stringstream fpsStr;fpsStr << "NN fps: " << std::fixed << std::setprecision(2) << fps;printf("fps %f\n", fps);cv::putText(frame, fpsStr.str(), cv::Point(2, frame_show.rows - 4), cv::FONT_HERSHEY_TRIPLEX, 0.4,cv::Scalar(0, 255, 0));for (auto &item:result) {topLeft.x = item.x1 * depthFrameColor.cols / target_width / depthFrameColor.cols;topLeft.y = item.y1 * depthFrameColor.rows / target_height / depthFrame.rows;bottomRight.x = (item.x2 * depthFrameColor.cols / target_width) / depthFrameColor.cols;bottomRight.y = (item.y2 * depthFrameColor.rows / target_height) / depthFrameColor.rows;//std::cout<<topLeft.x<<" "<<topLeft.y<<" "<<bottomRight.x<<" "<<bottomRight.y<<" "<<std::endl;config.roi = dai::Rect(topLeft, bottomRight);dai::SpatialLocationCalculatorConfig cfg;cfg.addROI(config);spatialCalcConfigInQueue->send(cfg);auto spatialData = spatialCalcQueue->get<dai::SpatialLocationCalculatorData>()->getSpatialLocations();for (auto depthData : spatialData) {auto roi = depthData.config.roi;roi = roi.denormalize(depthFrameColor.cols, depthFrameColor.rows);auto xmin = (int) roi.topLeft().x;auto ymin = (int) roi.topLeft().y;auto xmax = (int) roi.bottomRight().x;auto ymax = (int) roi.bottomRight().y;auto depthMin = depthData.depthMin;auto depthMax = depthData.depthMax;auto coords = depthData.spatialCoordinates;auto distance = std::sqrt(coords.x * coords.x + coords.y * coords.y + coords.z * coords.z);auto fontType = cv::FONT_HERSHEY_TRIPLEX;std::stringstream depthDistance;depthDistance.precision(2);depthDistance << fixed << static_cast<float>(distance / 1000.0f) << "m";cv::putText(depthFrameColor, depthDistance.str(), cv::Point(item.x1 + 10, item.y1 + 70), fontType, 0.5,color);std::stringstream rgbDistance;rgbDistance.precision(2);rgbDistance << fixed << static_cast<float>(distance / 1000.0f) << "m";cv::putText(frame, rgbDistance.str(), cv::Point(item.x1 + 10, item.y1 + 70), fontType, 0.5, color);cv::rectangle(frame, cv::Rect(cv::Point(item.x1, item.y1), cv::Point(item.x2, item.y2)), color,cv::FONT_HERSHEY_SIMPLEX);std::stringstream rgb_depthX, depthX;rgb_depthX << "X: " << (int) depthData.spatialCoordinates.x << " mm";cv::putText(frame, rgb_depthX.str(), cv::Point(item.x1 + 10, item.y1 + 20), cv::FONT_HERSHEY_TRIPLEX,0.5, color);std::stringstream rgb_depthY, depthY;rgb_depthY << "Y: " << (int) depthData.spatialCoordinates.y << " mm";cv::putText(frame, rgb_depthY.str(), cv::Point(item.x1 + 10, item.y1 + 35), cv::FONT_HERSHEY_TRIPLEX,0.5, color);std::stringstream rgb_depthZ, depthZ;rgb_depthZ << "Z: " << (int) depthData.spatialCoordinates.z << " mm";cv::putText(frame, rgb_depthZ.str(), cv::Point(item.x1 + 10, item.y1 + 50), cv::FONT_HERSHEY_TRIPLEX,0.5, color);cv::rectangle(depthFrameColor, cv::Rect(cv::Point(xmin, ymin), cv::Point(xmax, ymax)), color,cv::FONT_HERSHEY_SIMPLEX);depthX << "X: " << (int) depthData.spatialCoordinates.x << " mm";cv::putText(depthFrameColor, depthX.str(), cv::Point(xmin + 10, ymin + 20), cv::FONT_HERSHEY_TRIPLEX,0.5, color);depthY << "Y: " << (int) depthData.spatialCoordinates.y << " mm";cv::putText(depthFrameColor, depthY.str(), cv::Point(xmin + 10, ymin + 35), cv::FONT_HERSHEY_TRIPLEX,0.5, color);depthZ << "Z: " << (int) depthData.spatialCoordinates.z << " mm";cv::putText(depthFrameColor, depthZ.str(), cv::Point(xmin + 10, ymin + 50), cv::FONT_HERSHEY_TRIPLEX,0.5, color);}// Show the frame}//cv::imshow("depth", depthFrameColor);//cv::imshow("demo", frame_show);//int key = cv::waitKey(1);//if (key == 'q' || key == 'Q') return 0;}return 0;
}这里使用rk3588和oak相结合，存在两个问题，多个usb电流都在1a内，只有一个type-c转usb的电流在2a内，oak的基础电流要求900ma,峰值电流要求在1.5a左右，这就限制了oak的目标检测和推理数据传输，所以需要使用编码方式压缩数据量传输，即使使用usb的线进行数据传输，这个实验使用usb3.0的接口,但是由于电流无法满足要求，只能让oak限制使用2.0的速率dai::Device device(pipeline,true);使用设置帧率20fps进行目标检测，帧率过高容易导致失真，且使用压缩比h264还高效的1/2进行压缩数据量进行传输，在rk3588测试数据
/home/firefly/Downloads/cmake-build-debug/depthai
[2023-02-03 00:38:03.204] [warning] VideoEncoder setDefaultProfilePreset: passing 'width'/ 'height' is deprecated. The size is auto-determined from first frame
[194430101139FA1200] [7.1.1] [1.595] [NeuralNetwork(7)] [warning] Network compiled for 4 shaves, maximum available 12, compiling for 6 shaves likely will yield in better performance
[swscaler @ 0x5592956f50] No accelerated colorspace conversion found from yuv420p to rgb24.
[hevc_rkmpp @ 0x5593179e90] Decoder noticed an info change (640x640), format=0
rga_api version 1.8.1_[4]
Output layer names: 304, 324, output, fps 0.989732
fps 0.989732
fps 0.989732
fps 2.422750
fps 2.422750
fps 2.422750
fps 2.422750pc测试数据 11th Gen Intel® Core™ i5-11260H @ 2.60GHz × 12 Ubuntu 20.04.4 LTS 64-bit Mesa Intel® UHD Graphics (TGL GT1) / Mesa Intel® UHD Graphics (TGL GT1)，去掉帧率限制和usb2.0的接口限制
/home/ubuntu/nanodet/oak_detect_head/cmake-build-debug/depthai
[2023-02-03 08:51:03.664] [warning] VideoEncoder setDefaultProfilePreset: passing 'width'/ 'height' is deprecated. The size is auto-determined from first frame
[19443010F1F6F51200] [3.1] [37.694] [NeuralNetwork(7)] [warning] Network compiled for 4 shaves, maximum available 12, compiling for 6 shaves likely will yield in better performance
Output layer names: 304, 324, output, fps 0.000000
fps 1.764677
fps 6.032386
fps 6.032386
fps 6.032386
fps 6.032386
fps 6.032386
fps 6.032386
fps 6.032386
fps 6.255304
fps 6.2553042)以mpp硬解码去解析oak的视频帧，修改官方的demo代码,先执行
firefly@firefly:~$ sudo cp -r /usr/include/libdrm/* /usr/include/这里提供提供一个单纯的使用mpp解码的oak的编码产生的h264数据https://github.com/sxj731533730/oak_mpp_show.git，真的很快42FPS
测试结果
这就很强了，在开发板的电压和oak峰值限制条件下，使用编码方式降低数据传输量，又用rk的mpp硬件解码加速处理数据，yyds (这里以本地写图片cv.imwrite测试结果,cv.show显示画面好像掉帧率)，进一步修正代码，提速检测
3)修改上述代码使用yolov7-tiny的oak编码－mpp硬件解码－写图片本地，也还是不太快在12fps,看样子限制条件在于oak模型推理速度(模型推理太慢了oak vpu)
参考
Welcome to Core-3588J Manual — Firefly Wiki
Firefly-Linux / ff_media · GitLab
https://github.com/yijiesun/mpp_drm_rga_demo
4. 软件开发相关 — Firefly Wiki
53、RK3588测试视频编解码和 POE OAK Camera编码结合开发相关推荐
- 视频编解码——霍夫曼编码的实现
目录 原理 分析 输入文件 字符出现次数排序功能 .C .H 效果 思考 未完成的代码 .c .h 能够成功运行的代码 原理 视频编解码--熵编码-哈夫曼编码的原理及实现 分析 首先制作一个输入文件, ...
- 如何将ZCU106例程移植到自定义单板上(6)-测试文件编解码
软硬件系统构建好之后,VCU可以正常工作了.可以在VCU软件堆栈的不同层次上对VCU进行编解码,下面介绍一下使用VCU控制软件如何进行编解码. 1 H.264编码 编码指令:ctrlsw_encode ...
- vpu测试_一种普适的手机平台vpu视频编解码性能检测方法
一种普适的手机平台vpu视频编解码性能检测方法 [专利摘要]本发明公开了一种普适手机平台的视频处理单元(VPU)的H.264视频编解码性能检测方法,包括:手机平台利用VPU进行H.264视频编解码的系 ...
- 视频编解码优化以及与AI的实践结合
RealNetworks是流媒体类别的发明者,改变了音频和视频内容在各种设备和全世界消费的方式.在编解码市场的大环境下,RealNetworks推出了新一代的视频压缩算法RMHD,专为下一代的高清流媒 ...
- 北京大学数字视频编解码技术国家工程实验室开源AVS2高清实时编码器
本文转自公众号:PKUVCL.北京大学数字视频编解码技术国家工程实验室视频编码算法研究室 (PKU-VCL) 开源了 AVS2 高清实时编码器 xAVS2(V1.0).原文链接请点击[阅读原文]. 2 ...
- linux视音频解码教程,音视频编解码：NVIDIA Jetson Linux Multimedia API（总结）
音视频编解码:NVIDIA Jetson Linux Multimedia API(总结) 2020年12月02日 | 萬仟网IT编程 | 我要评论 一.官网链接官网链接:https://docs.n ...
- H.264视频编解码的代码移植和优化
基于DSP系统开发的视频编解码系统,国内几乎都是走的移植,优化的路线,并且移植的代码,都是开源的.毕竟花费大量的人力,物力去开发一套自己的代码,并不见得比一些成熟的开源代码效率更高,健壮性更好.更何况 ...
- 解析H．264视频编解码DSP实现与优化
引言 基于互联网的数字视频产业前景看好,而3G的规模部署,也会推动移动视频通信成为现实.但数字化后的视频图像具有数据海量性,给图像的存储和传输造成较大的困难.数字视频产业,是指数字内容中以数字视频形态 ...
- 基于FFMPEG 的跨平台视频编解码研究
第33卷 第11期 2011年11月 武 汉 理 工 大 学 学 报 JOURNALOF WUHANUNIVERSITYOFTECHNOLOGY Vol.33 No.11 췍췍췍췍췍췍췍췍췍췍췍췍췍췍 ...
最新文章
- python3 删除 文件 文件夹
- android-开源项目_我如何摆脱对开源的恐惧，并开始了自己的项目-以及如何做到。...
- 大一计算机实验报告7,计算机一级实验报告7.doc
- BitMap-BitSet(JDK1.8)基本使用入门
- Android之仿ele地图定位效果
- ppsspp java模拟_jar模拟器 免费版|java程序模拟软件
- 华为配置DHCPv6
- EDA技术与VHDL设计
- 在线作图|如何绘制一张气泡图
- 看我是怎样捣鼓Linux的
- 浅谈如何加强企业成本管理
- 第三届全国中医药院校大学生程序设计竞赛题解
- 所谓情商高就是会说话（摘录）
- 搜索引擎Bing必应高级搜索语法
- input输入框短信验证码处理
- 使用GEOquery 下载并读取数据
- 联想Y400安装ubuntu16.04、cuda8.0、cudnn5.1、anaconda2.7、tensorflow1.2、keras、opencv
- 八进制和十六进制表示
- 在Virtual Box中安装Windows7 64位虚拟机系统
- delphi类没有注册