新手入门保姆级教程，Linux平台和手机端SDK，基于Opencv、MNN、NCNN

端侧部署开源项目：https://github.com/hzpzlz/EasyDeploy

一 MNN编译动态库

环境要求

cmake（建议使用3.10或以上版本）
protobuf（使用3.0或以上版本）
gcc（使用4.9或以上版本）

MNN官方Git：GitHub - alibaba/MNN: MNN is a blazing fast, lightweight deep learning framework, battle-tested by business-critical use cases in Alibaba

目前所用版本：Release 2.1.0 Eager 模式 / CUDA 后端改造 / Winograd Int8 计算实现 · alibaba/MNN · GitHub

1 Linux平台x86_64

编译选项：具体参考MNN官方说明文档->推理框架Linux / macOS编译 · 语雀

cd /path/to/MNN
./schema/generate.shmkdir build
cd build
cmake .. && make -j8
make install  //if failed by permission denied, try "sudo make install"

编译、安装完成之后，一般会在/usr/local/include/MNN目录下生成头文件（后续需要拷贝到自己的工程中）；在/usr/local/lib下生成libMNN.so。可以考虑在~/.bashrc中设置

export LD_LIBRARY_PATH=LD_LIBRARY_PATH:/usr/local/lib

运行可执行文件就不需要再export所需的环境变量了

2 Android平台arm64-v8a

编译选项：具体参考MNN官方说明文档->推理框架Android编译 · 语雀

1 在https://developer.android.com/ndk/downloads下载安装NDK，我这边使用的r19c，附个下载链接吧：https://dl.google.com/android/repository/android-ndk-r19c-linux-x86_64.zip

2 在~/.bashrc中设置环境变量

export NDKROOT=/home/hzp/NDK/android-ndk-r19c

这里的NDKROOT名字可以修改，后续编译会用到

3 编译MNN

cd /path/to/MNN
cd project/android
mkdir build_64
cd build_64
../build_64.sh

编译完成后会在build_64目录下生成Android所需的依赖libMNN.so，这里不需要

make install了，不然会把linux下的覆盖掉，只需要.so文件即可。

二 Opencv编译动态库

Opencv官方Git：GitHub - opencv/opencv: Open Source Computer Vision Library

目前所用版本：Opencv4.5.3，附下载链接 https://github.com/opencv/opencv/archive/refs/tags/4.5.3.zip

1 Linux平台x86_64

简化版本的编译，反正可以支持项目使用了0.0

cd path/to/opencv4.5.3
mkdir build
cd build
cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local ..
make -j8
sudo make install

更加复杂的编译选项可以参照OpenCV: OpenCV configuration options reference

2 Android平台arm64-v8a

该来的总是要来，终究是逃不过，搞个相对复杂些的版本。

cd path/to/opencv4.5.3
vi build_64.sh
//写入以下代码
#!/bin/bash
mkdir build_android
cd build_androidcmake -DCMAKE_BUILD_WITH_INSTALL_RPATH=ON \-DCMAKE_TOOLCHAIN_FILE=${NDKROOT}/build/cmake/android.toolchain.cmake \-DANDROID_NDK=${NDKROOT} \-DANDROID_NATIVE_API_LEVEL=24 \-DANDROID_ABI="arm64-v8a" \-DWITH_CUDA=OFF \-DWITH_MATLAB=OFF \-DANDROID_STL=c++_shared \-DBUILD_ANDROID_PROJECTS=OFF \-DBUILD_ANDROID_EXAMPLES=OFF \-DBUILD_DOCS=OFF \-DBUILD_OPENCV_JAVA=OFF \-DBUILD_PERF_TESTS=OFF \-DBUILD_TESTS=OFF \-DBUILD_SHARED_LIBS=ON \-DCMAKE_INSTALL_PREFIX=/home/hzp/opencv-4.5.3/opencv_android_install/ \../make -j8
make install

最后一行的-DCMAKE_INSTALL_PREFIX=/home/hzp/opencv-4.5.3/opencv_android_install/ \

为安装路径，设置成你想要安装的路径即可，里面的东西后续也会用到。tips：尽量不要设置成/usr/local，会覆盖Linux版本。

三构建自己的项目

终于来到了这个令人激动人心的环节0.0

附上Git上的项目链接，还有很多地方需要完善：GitHub - hzpzlz/EasyDeploy: To reduce deployment costs and speed up project schedules

1 构建CMakeLists.txt

这里简单讲一讲从0开始搭建CMakeLists.txt吧，具体参考Git中的CMakeLists.txt

cmake所需的最低版本和项目名称

cmake_minimum_required(VERSION 3.10.2)
project(segment)

控制项目运行的平台

if(ANDROID)set(ARCH ${ANDROID_ABI})message(STATUS "ANDROID")
elseif(APPLE)message(STATUS "APPLE")
elseif(WIN32)message(STATUS "WIN32")
elseif(UNIX)set(ARCH "x86_64")message(STATUS "UNIX")
else()message(FATAL_ERROR "OTHER")
endif()

设置头文件路径

if(HPC_BACKEND STREQUAL "MNN")set(HPC_INC ${CMAKE_SOURCE_DIR}/dependency/inc/${HPC_BACKEND}/)
elseif(HPC_BACKEND STREQUAL "NCNN")set(HPC_INC ${CMAKE_SOURCE_DIR}/dependency/inc/${HPC_BACKEND}/)
endif()

设置依赖库路径

if(HPC_BACKEND STREQUAL "MNN")set(HPC_LIB ${CMAKE_SOURCE_DIR}/dependency/libs/${ARCH}/${HPC_BACKEND}/libMNN.so)# set(HPC_LIB ${CMAKE_SOURCE_DIR}/dependency/libs/${ARCH}/${HPC_BACKEND})# LINK_DIRECTORIES(${HPC_LIB})
elseif(HPC_BACKEND STREQUAL "NCNN")set(HPC_LIB ${CMAKE_SOURCE_DIR}/dependency/libs/${ARCH}/${HPC_BACKEND}/libNCNN.so)
endif()

设置opencv相关的头文件和依赖库

##opencv 头文件
set(OPENCV_INC ${CMAKE_SOURCE_DIR}/dependency/inc/opencv4/)
##opencv 依赖库 多个.so文件，用FILE
FILE(GLOB_RECURSE OPENCV_LIB ${CMAKE_SOURCE_DIR}/dependency/libs/${ARCH}/opencv2/*.so)

include头文件，这里有两个，分别是MNN和opencv

include_directories(${OPENCV_INC})
include_directories(${HPC_INC})

编写的demo

FILE(GLOB_RECURSE mnist_demo_cpp ${CMAKE_SOURCE_DIR}/src/mnist_demo.cpp)
FILE(GLOB_RECURSE seg_demo_cpp ${CMAKE_SOURCE_DIR}/src/seg_demo.cpp)

生成动态库

#动态库
add_library(seglibSHARED           //SHARED：动态库  STATIC：静态库${seg_demo_cpp}
)
#链接所需的lib
target_link_libraries(seglib${HPC_LIB}${OPENCV_LIB}
)

生成可执行文件

#生成可执行文件
add_executable(segDemo${seg_demo_cpp}
)
#链接所需的lib
target_link_libraries(segDemo${HPC_LIB}${OPENCV_LIB}
)

安装

##install
set(CMAKE_INSTALL_PREFIX ${CMAKE_SOURCE_DIR}/build/install)
set(CMAKE_INSTALL_BINDIR ${CMAKE_INSTALL_PREFIX}/bin/)
set(CMAKE_INSTALL_LIBDIR ${CMAKE_INSTALL_PREFIX}/lib/)
# set(CMAKE_INSTALL_INCLUDEDIR ${CMAKE_INSTALL_PREFIX}/inc/)
set(path_demo_output ${CMAKE_INSTALL_PREFIX}/demo/)install(TARGETS segDemo seglibRUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR}LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR}
)
# install(DIRECTORY ${path_inc_output} DESTINATION ${CMAKE_INSTALL_INCLUDEDIR})
install(FILES ${seg_demo_cpp} DESTINATION ${path_demo_output})

2 放置头文件和依赖库

项目中的头文件和依赖库放置目录

a.头文件：

/dependency/inc目录下存放所需的头文件

MNN：/dependency/inc/MNN目录存放MNN所需头文件，不过因为MNN里面的头文件includ时是#include <MNN/ErrorCode.hpp>这种形式，所以要在/dependency/inc/MNN目录下再新建一个MNN目录，然后执行

cp -r /usr/local/include/MNN/* /dependency/inc/MNN/MNN/

Opencv：也是类似的操作，/dependency/inc目录下新建opencv4，然后执行

/usr/local/include/opencv4/* /dependency/inc/opencv4/

b.依赖库：

/dependency/libs目录下存放所需的依赖

以Linux平台为例，/dependency/libs/x86_64，还是直接截图吧，清晰明了

MNN：

/usr/local/lib/libMNN.so /dependency/libs/x86_64/MNN/

Opencv：

/usr/local/lib/libopencv*.so /dependency/libs/x86_64/opencv2/

Android是类似的，只不过拷贝的是Android对应的编译结果，在此就不赘述了。

四模型转换

使用MNN的工具将onnx转化为mnn所需的模型，编译的时候将编译选项MNN_BUILD_CONVERTER设置当成ON，在build下面就生成MNNConvert工具

执行命令

./MNNConvert -f ONNX --modelFile XXX.onnx --MNNModel XXX.mnn

将onnx转化为mnn模型

五编写demo

一般包括前后处理和推理引擎调用，以分割为例

头文件和命名空间

#include "MNN/Interpreter.hpp"
#include "MNN/MNNDefine.h"
#include "MNN/Tensor.hpp"#include <math.h>
#include <opencv2/opencv.hpp>
#include <iostream>
#include <stdio.h>
using namespace MNN;

读取图像和简单的预处理：

//使用opencv读取图像
cv::Mat raw_image    = cv::imread(image_name.c_str());
int raw_image_height = raw_image.rows;
int raw_image_width  = raw_image.cols;
//将图像resize到模型大小
cv::Mat image;
cv::resize(raw_image, image, cv::Size(MODEL_SIZE, MODEL_SIZE));
printf("input img size: %d %d \n", raw_image_height, raw_image_width);// preprocessing
image.convertTo(image, CV_32FC3, 1.0);
//image = image / 255.0f;
std::cout<<"img channel: "<<image.channels()<<std::endl;

初始化MNN所需参数：

//初始化参数
int precision  = 2;
int power      = 0;
int memory     = 0;
int threads    = 1;
int MODEL_SIZE = 768;   //模型输入大小是768*768

从.mnn加载模型

std::shared_ptr<MNN::Interpreter> net;
net.reset(MNN::Interpreter::createFromFile(model_name.c_str()));
char *version = (char *)net->getModelVersion();
printf("MNN version: %s \n", version);

Init和createSession

MNN::ScheduleConfig config;
config.numThread = threads;
config.type      = static_cast<MNNForwardType>(forward);
MNN::BackendConfig backendConfig;
backendConfig.precision = (MNN::BackendConfig::PrecisionMode)precision;
backendConfig.power = (MNN::BackendConfig::PowerMode) power;
backendConfig.memory = (MNN::BackendConfig::MemoryMode) memory;
config.backendConfig = &backendConfig;auto session = net->createSession(config);
net->releaseModel();

MNN输入

// wrapping input tensor, convert nhwc to nchw
std::vector<int> dims{1,  MODEL_SIZE, MODEL_SIZE, 3};
auto nhwc_Tensor = MNN::Tensor::create<float>(dims, NULL, MNN::Tensor::TENSORFLOW);
auto nhwc_data   = nhwc_Tensor->host<float>();
auto nhwc_size   = nhwc_Tensor->size();::memcpy(nhwc_data, image.data, nhwc_size);std::string input_tensor = "input";   //转onnx时设置的inputtensor
auto inputTensor  = net->getSessionInput(session, nullptr);inputTensor->copyFromHostTensor(nhwc_Tensor);

run

net->runSession(session);

获得MNN输出，使用opencv保存图像

// get output data
std::string output_tensor_name0 = "output"; //转onnx时设置的outputtensorMNN::Tensor *out_tensor  = net->getSessionOutput(session, output_tensor_name0.c_str());MNN::Tensor out_tensor_host(out_tensor, out_tensor->getDimensionType());if  (out_tensor->getDimensionType() == MNN::Tensor::TENSORFLOW) {std::cout<<"using tpye TENSORFLOW"<<std::endl;
}
else if  (out_tensor->getDimensionType() == MNN::Tensor::CAFFE) {std::cout<<"using tpye CAFFE"<<std::endl;
}out_tensor->copyToHostTensor(&out_tensor_host);
std::cout<<"host tensor size: "<<out_tensor_host.size()<<std::endl;auto out_data = out_tensor_host.host<float>();cv::Mat  seg_res(MODEL_SIZE, MODEL_SIZE, CV_32FC1, cv::Scalar(255));
::memcpy(seg_res.data, out_data, MODEL_SIZE * MODEL_SIZE * sizeof(float));cv::imwrite("../assets/seg.jpg", seg_res * 255.0);

一键运行：

#!/bin/bash
####目前仅使用于linux和phone的CPU后端，后续会更新DSP和GPUARCH='linux'  # "linux" "android"mkdir build
cd build
rm * -rf
../build.sh $ARCHmake -j8
make install###for linux
if [ "${ARCH}" = "linux" ]; thenecho "linux ********"./install/bin/segDemo...
...
...
...
...
...

输入：

输出结果：

enjoy！