本文首发于个人博客https://kezunlin.me/post/54e7a3d8/，欢迎阅读最新内容！

tutorial to compile and use pytorch on ubuntu 16.04

PyTorch for Python

install pytorch from anaconda

    conda info --envsconda activate py35

    # newest version# 1.1.0 pytorch/0.3.0 torchvisionconda install pytorch torchvision cudatoolkit=9.0 -c pytorch

    # old version [NOT]# 0.4.1 pytorch/0.2.1 torchvisionconda install pytorch=0.4.1 cuda90 -c pytorch

output

    The following NEW packages will be INSTALLED:

      pytorch            pytorch/linux-64::pytorch-1.1.0-py3.5_cuda9.0.176_cudnn7.5.1_0torchvision        pytorch/linux-64::torchvision-0.3.0-py35_cu9.0.176_1

download from channel pytorch will cost much time!

下载pytorch/linux-64::pytorch-1.1.0-py3.5_cuda9.0.176_cudnn7.5.1_0速度非常慢！

install pytorch from tsinghua

add tsinghua pytorch channels

    conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch/# for legacy win-64conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/peterjc123/conda config --set show_channel_urls yes

使用anaconda官方pytorch源非常慢，用清华源代替。

see tsinghua anaconda

cat ～/.condarc

channels:- https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch/- defaults

install pytorch from tsinghua

    conda create --name torch python==3.7conda activate torch

    conda install -y pytorch torchvisionconda install -y scikit-learn scikit-image pandas matplotlib pillow opencv

The following NEW packages will be INSTALLED:

      pytorch            anaconda/cloud/pytorch/linux-64::pytorch-1.1.0-py3.5_cuda9.0.176_cudnn7.5.1_0torchvision        anaconda/cloud/pytorch/linux-64::torchvision-0.3.0-py35_cu9.0.176_1

test pytorch

    import torchprint(torch.__version__)'1.1.0'

    python -c 'import torch; print(torch.cuda.is_available())'True

pre-trained models

pre-trained model saved to /home/kezunlin/.cache/torch/checkpoints/

    Downloading: "https://download.pytorch.org/models/shufflenetv2_x0.5-f707e7126e.pth" to /home/kezunlin/.cache/torch/checkpoints/shufflenetv2_x0.5-f707e7126e.pth

PyTorch for C

download LibTorch

download from LibTorch

compile from source

compile pytorch

    # method 1git clone --recursive https://github.com/pytorch/pytorchcd pytorch

    # method 2, if you are updating an existing checkoutgit clone https://github.com/pytorch/pytorchcd pytorch git submodule syncgit submodule update --init --recursive

check tags

    git tag -l

    v0.4.0v0.4.1v1.0.0v1.0.1v1.0rc0v1.0rc1v1.1.0

now compile

    git checkout v1.1.0

    # method 1: offical build will generate lots of errors#python setup.py install

     # method 2: normal makemkdir build && cd build && cmake-gui ..

with configs

    BUILD_PYTHON OFF

be sure to use stable version 1.1.0 from here instead of latest version 20190724 (unstable version 1.2.0)

because error will occurs when load models.

for 1.1.0:

      std::shared_ptr<torch::jit::script::Module> module = torch::jit::load("./model.pt");

for latest 1.2.0

      torch::jit::script::Module module = torch::jit::load("./model.pt");

configure output

    ******** Summary ********General:CMake version         : 3.5.1CMake command         : /usr/bin/cmakeSystem                : LinuxC   compiler          : /usr/bin/c  C   compiler id       : GNUC   compiler version  : 5.4.0BLAS                  : MKLCXX flags             :  -fvisibility-inlines-hidden -fopenmp -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-mathBuild type            : ReleaseCompile definitions   : ONNX_ML=1;ONNX_NAMESPACE=onnx_torch;USE_GCC_ATOMICS=1;HAVE_MMAP=1;_FILE_OFFSET_BITS=64;HAVE_SHM_OPEN=1;HAVE_SHM_UNLINK=1;HAVE_MALLOC_USABLE_SIZE=1CMAKE_PREFIX_PATH     : CMAKE_INSTALL_PREFIX  : /usr/local

      TORCH_VERSION         : 1.1.0CAFFE2_VERSION        : 1.1.0BUILD_CAFFE2_MOBILE   : ONBUILD_ATEN_ONLY       : OFFBUILD_BINARY          : OFFBUILD_CUSTOM_PROTOBUF : ONLink local protobuf : ONBUILD_DOCS            : OFFBUILD_PYTHON          : OFFBUILD_CAFFE2_OPS      : ONBUILD_SHARED_LIBS     : ONBUILD_TEST            : OFFINTERN_BUILD_MOBILE   : USE_ASAN              : OFFUSE_CUDA              : ONCUDA static link    : OFFUSE_CUDNN           : ONCUDA version        : 9.2cuDNN version       : 7.1.4CUDA root directory : /usr/local/cudaCUDA library        : /usr/local/cuda/lib64/stubs/libcuda.socudart library      : /usr/local/cuda/lib64/libcudart.socublas library      : /usr/local/cuda/lib64/libcublas.socufft library       : /usr/local/cuda/lib64/libcufft.socurand library      : /usr/local/cuda/lib64/libcurand.socuDNN library       : /usr/local/cuda/lib64/libcudnn.sonvrtc               : /usr/local/cuda/lib64/libnvrtc.soCUDA include path   : /usr/local/cuda/includeNVCC executable     : /usr/local/cuda/bin/nvccCUDA host compiler  : /usr/bin/ccUSE_TENSORRT        : OFFUSE_ROCM              : OFFUSE_EIGEN_FOR_BLAS    : ONUSE_FBGEMM            : OFFUSE_FFMPEG            : OFFUSE_GFLAGS            : OFFUSE_GLOG              : OFFUSE_LEVELDB           : OFFUSE_LITE_PROTO        : OFFUSE_LMDB              : OFFUSE_METAL             : OFFUSE_MKL               : OFFUSE_MKLDNN            : OFFUSE_NCCL              : ONUSE_SYSTEM_NCCL     : OFFUSE_NNPACK            : ONUSE_NUMPY             : ONUSE_OBSERVERS         : ONUSE_OPENCL            : OFFUSE_OPENCV            : OFFUSE_OPENMP            : ONUSE_TBB               : OFFUSE_PROF              : OFFUSE_QNNPACK           : ONUSE_REDIS             : OFFUSE_ROCKSDB           : OFFUSE_ZMQ               : OFFUSE_DISTRIBUTED       : ONUSE_MPI             : ONUSE_GLOO            : ONUSE_GLOO_IBVERBS    : OFFNAMEDTENSOR_ENABLED   : OFFPublic Dependencies  : Threads::ThreadsPrivate Dependencies : qnnpack;nnpack;cpuinfo;/usr/lib/x86_64-linux-gnu/libnuma.so;fp16;/usr/lib/openmpi/lib/libmpi_cxx.so;/usr/lib/openmpi/lib/libmpi.so;gloo;aten_op_header_gen;foxi_loader;rt;gcc_s;gcc;dlConfiguring done

install pytorch

now compile and install

    make -j8sudo make install

output

    Install the project...-- Install configuration: "Release"-- Old export file "/usr/local/share/cmake/Caffe2/Caffe2Targets.cmake" will be replaced.  Removing files [/usr/local/share/cmake/Caffe2/Caffe2Targets-release.cmake].-- Set runtime path of "/usr/local/bin/protoc" to "$ORIGIN"-- Old export file "/usr/local/share/cmake/Gloo/GlooTargets.cmake" will be replaced.  Removing files [/usr/local/share/cmake/Gloo/GlooTargets-release.cmake].-- Set runtime path of "/usr/local/lib/libonnxifi_dummy.so" to "$ORIGIN"-- Set runtime path of "/usr/local/lib/libonnxifi.so" to "$ORIGIN"-- Set runtime path of "/usr/local/lib/libfoxi_dummy.so" to "$ORIGIN"-- Set runtime path of "/usr/local/lib/libfoxi.so" to "$ORIGIN"-- Set runtime path of "/usr/local/lib/libc10.so" to "$ORIGIN"-- Set runtime path of "/usr/local/lib/libc10_cuda.so" to "$ORIGIN:/usr/local/cuda/lib64"-- Set runtime path of "/usr/local/lib/libthnvrtc.so" to "$ORIGIN:/usr/local/cuda/lib64/stubs:/usr/local/cuda/lib64"-- Set runtime path of "/usr/local/lib/libtorch.so" to "$ORIGIN:/usr/local/cuda/lib64:/usr/lib/openmpi/lib"-- Set runtime path of "/usr/local/lib/libcaffe2_detectron_ops_gpu.so" to "$ORIGIN:/usr/local/cuda/lib64"-- Set runtime path of "/usr/local/lib/libcaffe2_observers.so" to "$ORIGIN:/usr/local/cuda/lib64"

pytorch 1.1.0

compile and install will cost more than 2 hours

lib install to /usr/local/lib/libtorch.so

cmake install to /usr/local/share/cmake/Torch

C example

load pytorch model in c see load pytorch model in c

cpp

#include <torch/script.h> // One-stop header.#include <iostream>
#include <memory>int main(int argc, const char* argv[]) {if (argc != 2) {std::cerr << "usage: example-app <path-to-exported-script-module>\n";return -1;}// Deserialize the ScriptModule from a file using torch::jit::load().std::shared_ptr<torch::jit::script::Module> module = torch::jit::load(argv[1]);assert(module != nullptr);std::cout << "ok\n";// Create a vector of inputs.
std::vector<torch::jit::IValue> inputs;
inputs.push_back(torch::ones({1, 3, 224, 224}));// Execute the model and turn its output into a tensor.
at::Tensor output = module->forward(inputs).toTensor();std::cout << output.slice(/*dim=*/1, /*start=*/0, /*end=*/5) << '\n';
}

CMakeLists.txt

cmake_minimum_required(VERSION 3.0 FATAL_ERROR)
project(custom_ops)# /usr/local/share/cmake/Torch
find_package(Torch REQUIRED)
MESSAGE( [Main] " TORCH_INCLUDE_DIRS = ${TORCH_INCLUDE_DIRS}")
MESSAGE( [Main] " TORCH_LIBRARIES = ${TORCH_LIBRARIES}")
include_directories(${TORCH_INCLUDE_DIRS})add_executable(example-app example-app.cpp)
target_link_libraries(example-app "${TORCH_LIBRARIES}")
set_property(TARGET example-app PROPERTY CXX_STANDARD 11)

output

    Found torch: /usr/local/lib/libtorch.so  [Main] TORCH_INCLUDE_DIRS = /usr/local/include;/usr/local/include/torch/csrc/api/include[Main] TORCH_LIBRARIES = torch;torch_library;/usr/local/lib/libc10.so;/usr/local/cuda/lib64/stubs/libcuda.so;/usr/local/cuda/lib64/libnvrtc.so;/usr/local/cuda/lib64/libnvToolsExt.so;/usr/local/cuda/lib64/libcudart.so;/usr/local/lib/libc10_cuda.so[TOLOWER] ALGORITHM_TARGET = algorithm

make

    mkdir build cd build && cmake-gui ..make -j8
> set `Torch_DIR` to `/home/kezunlin/program/libtorch/share/cmake/Torch`
> auto-set `Torch_DIR` to `/usr/local/share/cmake/Torch`

run

    ./example-app model.pt-0.2698 -0.0381  0.4023 -0.3010 -0.0448

errors and solutions

compile errors with libtorch

Build simple c example-cpp using Libtorch fails on arm with undefined reference to c10::Error::Error

@soumith

You might be building libtorch with a compiler that is incompatible with the compiler building your final app.

For example, you built libtorch with gcc 4.9.2 and your final app with gcc 5.1, and the C ABI between both of them is not the same, so you are seeing linker errors like these

issues-linking-with-libtorch-c-11-abi

@christianperone

    if ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU")set(TORCH_CXX_FLAGS "-D_GLIBCXX_USE_CXX11_ABI=0")endif()
> Which forces GCC to use the old C  11 ABI.

@ smth

we have that flag set because we build with gcc 4.9.x, which only has the old ABI.

In GCC 5.1, the ABI for std::string was changed, and binaries compiling with gcc >= 5.1 are not ABI-compatible with binaries build with gcc < 5.1 (like pytorch) unless you set that flag.

resons and solutions

Reasons: LibTorch compiled with GCC-4.9.X (only has the old ABI), and binaries compiling with gcc >= 5.1 are not ABI-compatible
Solution: compile pytorch from source instead of using LibTroch downloaded from the website.

runtime errors with pytorch

errors

    /usr/local/lib/libopencv_imgcodecs.so.3.1.0: undefined reference to `TIFFReadRGBAStrip@LIBTIFF_4.0'

which means opencv link against libtiff 4.0.6

ldd check

    ldd /usr/local/lib/libopencv_imgcodecs.so.3.1.0linux-vdso.so.1 =>  (0x00007ffc92ffc000)libopencv_imgproc.so.3.1 => /usr/local/lib/libopencv_imgproc.so.3.1 (0x00007f32afbca000)libjpeg.so.8 => /usr/local/lib/libjpeg.so.8 (0x00007f32af948000)libpng12.so.0 => /lib/x86_64-linux-gnu/libpng12.so.0 (0x00007f32af723000)libtiff.so.5 => /usr/lib/x86_64-linux-gnu/libtiff.so.5 (0x00007f32af4ae000)

when compile opencv-3.1.0, cmake find /usr/lib/x86_64-linux-gnu/libtiff.so.5

locate libtiff

    locate libtiff.so

    /home/kezunlin/anaconda3/envs/py35/lib/libtiff.so/home/kezunlin/anaconda3/envs/py35/lib/libtiff.so.5/home/kezunlin/anaconda3/envs/py35/lib/libtiff.so.5.4.0/home/kezunlin/anaconda3/lib/libtiff.so/home/kezunlin/anaconda3/lib/libtiff.so.5/home/kezunlin/anaconda3/lib/libtiff.so.5.4.0/home/kezunlin/anaconda3/pkgs/libtiff-4.0.10-h2733197_2/lib/libtiff.so/home/kezunlin/anaconda3/pkgs/libtiff-4.0.10-h2733197_2/lib/libtiff.so.5/home/kezunlin/anaconda3/pkgs/libtiff-4.0.10-h2733197_2/lib/libtiff.so.5.4.0/opt/MATLAB/R2016b/bin/glnxa64/libtiff.so.5/opt/MATLAB/R2016b/bin/glnxa64/libtiff.so.5.0.5/usr/lib/x86_64-linux-gnu/libtiff.so/usr/lib/x86_64-linux-gnu/libtiff.so.5/usr/lib/x86_64-linux-gnu/libtiff.so.5.2.4

It seems that my OpenCV was compiled against libtiff 4, but I have libtiff 5, how to solve this problem?

re-compile opencv-3.1.0 again, new errors occursee here

    CMake Error: The following variables are used in this project, but they are set to NOTFOUND.Please set them or make sure they are set and tested correctly in the CMake files:CUDA_nppi_LIBRARY (ADVANCED)linked by target "opencv_cudev" in directory /home/kezunlin/program/opencv-3.1.0/modules/cudevlinked by target "opencv_cudev" in directory /home/kezunlin/program/opencv-3.1.0/modules/cudevlinked by target "opencv_test_cudev" in directory /home/kezunlin/program/opencv-3.1.0/modules/cudev/test

solutions:

    WITH_CUDA OFFWITH_VTK OFFWITH_TIFF OFFBUILD_PERF_TESTS OFF

for python2, use default /usr/bin/python2.7

for python3, NOT USE anaconda version

编译的过程中，尽量避免使用anaconda目录下的lib

install libwebp

    sudo apt-get -y install libwebp-dev

Reference

pytorch
pytorch github
deep_learning_60min_blitz
pytorch-tutorial
pytorch notebooks
pytorch-beginner
pytorch cppdocs

History

20190626: created.

Copyright

Post author: kezunlin
Post link: https://kezunlin.me/post/54e7a3d8/
Copyright Notice: All articles in this blog are licensed under CC BY-NC-SA 3.0 unless stating additionally.

Ubuntu 16.04上源码编译和安装pytorch教程，并编写C Demo CMakeLists.txt相关推荐

OpenLDAP / ubuntu 18.04 下源码编译及安装说明
一.下载安装包 1.OpenLDAP v2.4.49 链接:https://pan.baidu.com/s/1ezZSgCCUxJV7_ou0v58IHQ 提取码:gmie 2.Berkele ...
ubuntu 16.04下源码安装opencv3.4
源码安装opencv,遇到了一些小波折,这里做个备忘吧. 首先要下载源码,路径: https://github.com/opencv/opencv 下载成功后,在opencv的根目录下执行下面操作: ...
ubuntu 16.04 Nginx源码部署安装
nginx安装 1.安装zlib依赖库:sudo apt-get install zlib1g-dev 2.进入解压相关文件: tar –xzvf openssl-1.0.1.tar.gz tar – ...
Ubuntu 16.04下源码安装Catkin
一.下载源代码 Catkin ~$ sudo apt-get install git ~$ git clone https://github.com/ros/catkin 二.安装依赖项 1.Catk ...
如何在Ubuntu 16.04上使用ProxySQL缓存优化MySQL查询
The author selected the Free Software Foundation to receive a donation as part of the Write for DOna ...
在Ubuntu 16.04.3 LTS 和 Windows 下安装 Go 环境
Ubuntu 16.04.3 LTS 环境 1.安装 sudo apt-get install golang-go 目前go的最新版本是1.9.2,但是在Ubuntu 16.04上采用预编译好的包安装 ...
ubuntu 环境下调试mysql源码_【转】Ubuntu 16.04下 Mysql 5.7.17源码编译与安装
Ubuntu 16.04下 Mysql5.7.17源码编译与安装系统环境一. 系统安装条件 1.cmake MySQL使用cmake跨平台工具预编译源码,用于设置mysql的编译参数. sudo ...
ubuntu安装python_ubuntu18.04下源码编译安装最新版本Python3
原文链接:ubuntu18.04下源码编译安装最新版本Python3 截止到2019年4月9日,Python3最新的版本是3.7.3. 在ubuntu18.04中已经安装的Python3版本是3.6. ...
Ubuntu 16.04上安装SkyEye及测试
说明一下,在Ubuntu 16.04上安装SkyEye方法不是原创,是来自互联网,仅供学习参考. 一.检查支持软件包 gcc, make, vim(optional), ssh, subversion ...

Ubuntu 16.04上源码编译和安装pytorch教程，并编写C Demo CMakeLists.txt

PyTorch for Python

install pytorch from anaconda

install pytorch from tsinghua

add tsinghua pytorch channels

install pytorch from tsinghua

test pytorch

pre-trained models

PyTorch for C

download LibTorch

compile from source

compile pytorch

install pytorch

C example

cpp

CMakeLists.txt

make

run

errors and solutions

compile errors with libtorch

resons and solutions

runtime errors with pytorch

Reference

History

Copyright

Ubuntu 16.04上源码编译和安装pytorch教程，并编写C Demo CMakeLists.txt相关推荐

最新文章

热门文章