深度学习多框架多平台推理引擎工具
2024-05-14 04:45:54
一种深度学习推理引擎工具，支持多框架、支持多平台推理
项目下载地址：下载地址
支持的计算平台：
- Windows 10 (Visual Studio 2019 x64)
- Linux (x64, armv7, aarch64)
- Android (armeabi-v7a, arm64-v8a)
支持的模型框架：
- TensorFlow Lite
- TensorFlow Lite with delegate (XNNPACK, GPU, EdgeTPU, NNAPI)
- TensorRT (GPU, DLA)
- OpenCV(dnn)
- OpenCV(dnn) with GPU
- OpenVINO with OpenCV (xml+bin)
- ncnn
- ncnn with Vulkan
- MNN (with Vulkan)
- SNPE (Snapdragon Neural Processing Engine SDK (Qualcomm Neural Processing SDK for AI v1.51.0))
- Arm NN
- NNabla
- NNabla with CUDA
下载相关库：
Download prebuilt libraries
- sh third_party/download_prebuilt_libraries.sh
配置编译参数：
Deep learning framework:
- You can enable multiple options althoguh the following example enables just one option
# OpenCV (dnn), OpenVINO cmake .. -DINFERENCE_HELPER_ENABLE_OPENCV=on # Tensorflow Lite cmake .. -DINFERENCE_HELPER_ENABLE_TFLITE=on # Tensorflow Lite (XNNPACK) cmake .. -DINFERENCE_HELPER_ENABLE_TFLITE_DELEGATE_XNNPACK=on # Tensorflow Lite (GPU) cmake .. -DINFERENCE_HELPER_ENABLE_TFLITE_DELEGATE_GPU=on # Tensorflow Lite (EdgeTPU) cmake .. -DINFERENCE_HELPER_ENABLE_TFLITE_DELEGATE_EDGETPU=on # Tensorflow Lite (NNAPI) cmake .. -DINFERENCE_HELPER_ENABLE_TFLITE_DELEGATE_NNAPI=on # TensorRT cmake .. -DINFERENCE_HELPER_ENABLE_TENSORRT=on # ncnn, ncnn + vulkan cmake .. -DINFERENCE_HELPER_ENABLE_NCNN=on # MNN (+ Vulkan) cmake .. -DINFERENCE_HELPER_ENABLE_MNN=on # SNPE cmake .. -DINFERENCE_HELPER_ENABLE_SNPE=on # Arm NN cmake .. -DINFERENCE_HELPER_ENABLE_ARMNN=on # NNabla cmake .. -DINFERENCE_HELPER_ENABLE_NNABLA=on # NNabla with CUDA cmake .. -DINFERENCE_HELPER_ENABLE_NNABLA_CUDA=onEnable/Disable preprocess using OpenCV:
- By disabling this option, InferenceHelper is not dependent on OpenCV
cmake .. -INFERENCE_HELPER_ENABLE_PRE_PROCESS_BY_OPENCV=off
APIs
InferenceHelper
Enumeration
typedef enum {kOpencv,kOpencvGpu,kTensorflowLite,kTensorflowLiteXnnpack,kTensorflowLiteGpu,kTensorflowLiteEdgetpu,kTensorflowLiteNnapi,kTensorrt,kNcnn,kNcnnVulkan,kMnn,kSnpe,kArmnn,kNnabla,kNnablaCuda,
} HelperType;
static InferenceHelper* Create(const HelperType helper_type)
- Create InferenceHelper instance for the selected framework
std::unique_ptr<InferenceHelper> inference_helper(InferenceHelper::Create(InferenceHelper::kTensorflowLite));
static void PreProcessByOpenCV(const InputTensorInfo& input_tensor_info, bool is_nchw, cv::Mat& img_blob)
- Run preprocess (convert image to blob(NCHW or NHWC))
- This is just a helper function. You may not use this function.
- Available when
INFERENCE_HELPER_ENABLE_PRE_PROCESS_BY_OPENCV=on
- Available when
InferenceHelper::PreProcessByOpenCV(input_tensor_info, false, img_blob);
int32_t SetNumThreads(const int32_t num_threads)
- Set the number of threads to be used
- This function needs to be called before initialize
inference_helper->SetNumThreads(4);
int32_t SetCustomOps(const std::vector<std::pair<const char*, const void*>>& custom_ops)
- Set custom ops
- This function needs to be called before initialize
std::vector<std::pair<const char*, const void*>> custom_ops;
custom_ops.push_back(std::pair<const char*, const void*>("Convolution2DTransposeBias", (const void*)mediapipe::tflite_operations::RegisterConvolution2DTransposeBias()));
inference_helper->SetCustomOps(custom_ops);
int32_t Initialize(const std::string& model_filename, std::vector& input_tensor_info_list, std::vector& output_tensor_info_list)
- Initialize inference helper
- Load model
- Set tensor information
std::vector<InputTensorInfo> input_tensor_list;
InputTensorInfo input_tensor_info("input", TensorInfo::TENSOR_TYPE_FP32, false); /* name, data_type, NCHW or NHWC */
input_tensor_info.tensor_dims = { 1, 224, 224, 3 };
input_tensor_info.data_type = InputTensorInfo::kDataTypeImage;
input_tensor_info.data = img_src.data;
input_tensor_info.image_info.width = img_src.cols;
input_tensor_info.image_info.height = img_src.rows;
input_tensor_info.image_info.channel = img_src.channels();
input_tensor_info.image_info.crop_x = 0;
input_tensor_info.image_info.crop_y = 0;
input_tensor_info.image_info.crop_width = img_src.cols;
input_tensor_info.image_info.crop_height = img_src.rows;
input_tensor_info.image_info.is_bgr = false;
input_tensor_info.image_info.swap_color = false;
input_tensor_info.normalize.mean[0] = 0.485f; /* https://github.com/onnx/models/tree/master/vision/classification/mobilenet#preprocessing */
input_tensor_info.normalize.mean[1] = 0.456f;
input_tensor_info.normalize.mean[2] = 0.406f;
input_tensor_info.normalize.norm[0] = 0.229f;
input_tensor_info.normalize.norm[1] = 0.224f;
input_tensor_info.normalize.norm[2] = 0.225f;
input_tensor_list.push_back(input_tensor_info);std::vector<OutputTensorInfo> output_tensor_list;
output_tensor_list.push_back(OutputTensorInfo("MobilenetV2/Predictions/Reshape_1", TensorInfo::TENSOR_TYPE_FP32));inference_helper->initialize("mobilenet_v2_1.0_224.tflite", input_tensor_list, output_tensor_list);
int32_t Finalize(void)
- Finalize inference helper
inference_helper->Finalize();
int32_t PreProcess(const std::vector& input_tensor_info_list)
- Run preprocess
- Call this function before invoke
- Call this function even if the input data is already pre-processed in order to copy data to memory
- Note : Some frameworks don’t support crop, resize. So, it’s better to resize image before calling preProcess.
inference_helper->PreProcess(input_tensor_list);
int32_t Process(std::vector& output_tensor_info_list)
- Run inference
inference_helper->Process(output_tensor_info_list)
TensorInfo (InputTensorInfo, OutputTensorInfo)
Enumeration
enum {kTensorTypeNone,kTensorTypeUint8,kTensorTypeInt8,kTensorTypeFp32,kTensorTypeInt32,kTensorTypeInt64,
};
Properties
std::string name; // [In] Set the name_ of tensor
int32_t id; // [Out] Do not modify (Used in InferenceHelper)
int32_t tensor_type; // [In] The type of tensor (e.g. kTensorTypeFp32)
std::vector<int32_t> tensor_dims; // InputTensorInfo: [In] The dimentions of tensor. (If empty at initialize, the size is updated from model info.)// OutputTensorInfo: [Out] The dimentions of tensor is set from model information
bool is_nchw; // [IN] NCHW or NHWCInputTensorInfo
Enumeration
enum {kDataTypeImage,kDataTypeBlobNhwc, // data_ which already finished preprocess(color conversion, resize, normalize_, etc.)kDataTypeBlobNchw,
};
Properties
void* data; // [In] Set the pointer to image/blob
int32_t data_type; // [In] Set the type of data_ (e.g. kDataTypeImage)struct {int32_t width;int32_t height;int32_t channel;int32_t crop_x;int32_t crop_y;int32_t crop_width;int32_t crop_height;bool is_bgr; // used when channel == 3 (true: BGR, false: RGB)bool swap_color;
} image_info; // [In] used when data_type_ == kDataTypeImagestruct {float mean[3];float norm[3];
} normalize; // [In] used when data_type_ == kDataTypeImage
OutputTensorInfo
Properties
void* data; // [Out] Pointer to the output data_
struct {float scale;uint8_t zero_point;
} quant; // [Out] Parameters for dequantization (convert uint8 to float)
float* GetDataAsFloat()
- Get output data in the form of FP32
- When tensor type is INT8 (quantized), the data is converted to FP32 (dequantized)
const float* val_float = output_tensor_list[0].GetDataAsFloat();
推理库引用：
- tensorflow- https://github.com/tensorflow/tensorflow- Copyright 2019 The TensorFlow Authors- Licensed under the Apache License, Version 2.0- Modification: no- Pre-built binary file is generated from this project- libedgetpu- https://github.com/google-coral/libedgetpu- Copyright 2019 Google LLC- Licensed under the Apache License, Version 2.0- Modification: yes- Pre-built binary file is generated from this project- TensorRT- https://github.com/nvidia/TensorRT- Copyright 2020 NVIDIA Corporation- Licensed under the Apache License, Version 2.0- Modification: yes- Some code are retrieved from this repository- ncnn- https://github.com/Tencent/ncnn- Copyright (C) 2017 THL A29 Limited, a Tencent company. All rights reserved.- Licensed under the BSD 3-Clause License- https://github.com/Tencent/ncnn/blob/master/LICENSE.txt- Modification: no- Pre-built binary file is generated from this project- MNN- https://github.com/alibaba/MNN- Copyright (C) 2018 Alibaba Group Holding Limited- Licensed under the Apache License, Version 2.0- Modification: no- Pre-built binary file is generated from this project- SNPE- https://developer.qualcomm.com/software/qualcomm-neural-processing-sdk- Copyright (c) 2017-2020 Qualcomm Technologies, Inc.- Arm NN- https://github.com/Arm-software/armnn- Copyright (c) 2017 ARM Limited.- NNabla- https://github.com/sony/nnabla- https://github.com/sony/nnabla-ext-cuda- Copyright 2018,2019,2020,2021 Sony Corporation.- Licensed under the Apache License, Version 2.0```深度学习多框架多平台推理引擎工具相关推荐
- 推荐44个最具潜力的顶极深度学习开源框架和平台！！！
关注上方"深度学习技术前沿",选择"星标公众号", 资源干货,第一时间送达! 来源:AI开发者@微信公众号 工欲善其事必先利其器,这也是大部分开发者在日常工作中 ...
- Euler 今日问世！国内首个工业级的图深度学习开源框架，阿里妈妈造
千呼万唤始出来!阿里妈妈正式公布重磅开源项目--图深度学习框架Euler.这是国内首个在核心业务大规模应用后开源的图深度学习框架.此次开源,Euler内置了大量的算法供用户直接使用,相关代码已经可在G ...
- AI学习笔记（九）从零开始训练神经网络、深度学习开源框架
AI学习笔记之从零开始训练神经网络.深度学习开源框架 从零开始训练神经网络 构建网络的基本框架 启动训练网络并测试数据 深度学习开源框架 深度学习框架 组件--张量 组件--基于张量的各种操作 组件- ...
- 深度学习在美团推荐平台排序中的运用
美团作为国内最大的生活服务平台,业务种类涉及食.住.行.玩.乐等领域,致力于让大家吃得更好,活得更好,有数亿用户以及丰富的用户行为.随着业务的飞速发展,美团的用户和商户数在快速增长.在这样的背景下,通 ...
- 【CV实战】年轻人的第一个深度学习CV项目应该是什么样的？（支持13大深度学习开源框架）...
计算机视觉发展至今,许多技术已经非常成熟了,在各行各业落地业务非常多,因此不断的有新同学入行.本次我们就来介绍,对于新手来说,如何做一个最合适的项目.本次讲述一个完整的工业级别图像分类项目的标准流程, ...
- 美团-深度学习在点评推荐平台排序中的运用
团点评作为国内最大的生活服务平台,业务种类涉及食.住.行.玩.乐等领域,致力于让大家吃得更好,活得更好,有数亿用户以及丰富的用户行为.随着业务的飞速发展,美团点评的用户和商户数在快速增长.在这样的背景 ...
- 飞桨深度学习开源框架2.0抢先看：成熟完备的动态图开发模式
百度飞桨于近期宣布,深度学习开源框架2.0抢先版本正式发布,进入2.0时代.其中一项重大升级,就是推出更加成熟完备的命令式编程模式,即通常说的动态图模式.同时在该版本中将默认的开发模式定为动态图模式, ...
- 人工智能深度学习Caffe框架介绍，优秀的深度学习架构
在深度学习领域,Caffe框架是人们无法绕过的一座山.这不仅是因为它无论在结构.性能上,还是在代码质量上,都称得上一款十分出色的开源框架.更重要的是,它将深度学习的每一个细节都原原本本地展现出来,大大 ...
- 【杂谈】超过12个，150页深度学习开源框架指导手册与GitHub项目，初学CV你值得拥有...
之前我们公众号输出了很多深度学习开源框架相关的内容,今天整理成技术手册给大家分享以方便阅读,下面是详细信息. 开源框架背景 现如今开源生态非常完善,深度学习相关的开源框架众多,光是为人熟知的就有caf ...
最新文章
- python使用matplotlib可视化棉签图、棉棒图（stem plot）、展示离散而有序的数据
- 干旱的草原与拉大提琴的牧人
- axure 小程序 lib_小程序定制开发的步骤有哪些？
- Filecoin Gas基础费率大幅回升至2.78 nanoFIL
- matlab人口增长线性回归拟合_matlab中的线性拟合-98529851
- day19异常File类
- 使用 IIS 进行 ASP.NET 2.0 成员/角色管理(2)：实现
- Mimics-基础操作教程-1
- 路由协议-ospf配置
- R语言使用lm函数构建多元回归模型（Multiple Linear Regression）、并根据模型系数写出回归方程、使用fitted函数计算出模型的拟合的y值（响应值）向量
- 计算机音乐谱子十年,十年曲谱钢琴曲_十年 钢琴谱
- java配置jdk和jre_为什么要配置java环境变量？JDK和JRE的区别在哪里？
- 巧用“sfc scannow”命令扫描修复Win8系统
- 燃气热水器出现e4故障是怎么回事？
- 【1066】满足条件的数累加
- 为什么华为 200 万招聘 AI 博士，马斯克却推出脑机接口对抗 AI？
- php微博 百度文库,新浪微博与百度贴吧移动社交功能的竞品分析
- IT品牌类营销软文写法指南，运营人必看干货
- 爱普生6轴机械手原点调整 RC+7,0
- 物流英语与计算机操作,物流英语与计算机模题及正确答案.doc