Caffe环境搭建指北

《Caffe环境搭建指北》

caffe的全称是Convolutional Architecture for Fast Feature Embedding，是一个清晰高效的深度学习开源框架，是贾扬清大神在2013年下半年利用空闲时间实现的，2013年12月开源的。caffe目前已经不再更新了，相比与pytorch以及TF2.0来说，该框架显得有些过时，但是对于做深度学习终端的开发者，caffe仍旧是很好的选择。不要自己趟这个坑，照着来吧，不会有问题。

Key Words：Caffe环境搭建

Beijing, 2020

作者：RaySue

Code：https://github.com/BVLC/caffe

Agile Pioneer

操作系统　Ubuntu 18.04.1

caffe的各个依赖包之间版本有着依赖关系，以下是本人尝试的可用版本

文章目录

1. 安装Boost v1.56.0
2. 安装GFlags v2.1.2
3. 安装Glog v0.3.3
4. 安装LevelDB v1.20
5. 安装LMDB v0.9.18
6. 安装protobuf v3.6.0
7. 安装HDF5 v5-1.10
8. 安装snappy v1.1.6
9. 安装 BLAS
安装OpenCV
安装CUDA
编译caffe
- 编写载入caffe环境脚本样例
编译caffe
参考

1. 安装Boost v1.56.0

下载地址：https://sourceforge.net/projects/boost/files/boost/1.56.0/

功能介绍

Boost是为C++语言标准库提供扩展的一些C++程序库的总称。Boost库是一个可移植、提供源代码的C++库，作为标准库的后备，是C++标准化进程的开发引擎之一，是为C++语言标准库提供扩展的一些C++程序库的总称。 caffe采用C++语言作为主要开发语言，其中大量的代码依赖于Boost库。

编译命令

注意编译这个的时候，你需要切换到你要用的环境，即which　python是你要用的python路径

./bootstrap.sh --prefix=/install/dir./b2 install

使用conda python3 遇到问题参考
https://blog.csdn.net/renhaofan/article/details/82109105

nm -D xxx.so |grep xxx_Type

2. 安装GFlags v2.1.2

下载地址：https://github.com/gflags/gflags/archive/v2.1.2.zip

功能介绍

GFlags是Google开源的一个处理命令行参数的库，C++开发，可以替代getopt函数。GFlags与getopt函数不同，在GFlags中，标记的定义分散在源代码中，不需要可列举在一个地方。Caffe库采用GFlags库开发caffe的命令行。

编译命令

mkdir buildcd buildexport CXXFLAGS="-fPIC”cmake .. -DBUILD_SHARED_LIBS=ON -DCMAKE_INSTALL_PREFIX=/path/to/installmake VERBOSE=1make make install

3. 安装Glog v0.3.3

下载链接：https://github.com/google/glog/archive/v0.3.3.tar.gz

功能介绍

Glog是一个提供日志功能的库，提供基于C++风格的流日志API，以及各种辅助函数的宏，它的使用方式与C++的stream方式类似。caffe运行时的日志输出依赖于Glog库。

编译命令

./configure --prefix=/install/dirmake -j8make install

报错&解决

报错问题:

src/demangle.cc:170:16: error: ISO C++ forbids comparison between pointer and integer [-fpermissive]if (str == '\0') {^~~~
src/demangle.cc: In function 'bool google::ParseCharClass(google::State*, const char*)':
src/demangle.cc:226:29: error: ISO C++ forbids comparison between pointer and integer [-fpermissive]if (state->mangled_cur == '\0') {

解决方法：

把上面的'\0'改为"\0"

4. 安装LevelDB v1.20

下载地址：https://github.com/google/leveldb/archive/v1.20.zip

功能介绍

LevelDB是Google实现的一个非常高效的Key-Value数据库。它是单进程的服务，性能非常高。是C/C++编程语言的库。caffe支持两种数据库，其中之一为LevelDB。

编译命令

makecp -r include/* install/include/cp out-shared/* install/lib/

5. 安装LMDB v0.9.18

https://github.com/LMDB/lmdb/archive/LMDB_0.9.18.zip

功能介绍
LMDB是一个超级快，超级小的Key-Value数据存储服务，是由OpenLDAP项目的Symas开发的。使用内存映射文件，因此读取数据的性能根内存数据库一样，其大小受限于虚拟地址空间的大小。caffe支持两种数据库，其中之一为LMDB。

编译命令

cd lmdb-xxx/libraries/liblmdb# 修改29行：prefix=/path/to/install
vim Makefilemakemake install

6. 安装protobuf v3.6.0

下载地址：https://github.com/protocolbuffers/protobuf/releases

功能介绍
Google Protocol Buffer（简称ProtoBuf）是一种简便高效的结构化数据存储方式，可以用于结构化数据的串行化，或者说序列化。它很适合作数据存储或RPC数据交换格式。可用于通讯协议、数据存储等领域的语言无关、平台无关、可扩展的序列化结构数据格式。

使用protobuf库首先要自己编写一个.proto文件，定义程序中需要处理的结构化数据，在protobuf中，结构化数据被成为Message。在一个.proto文件中可以定义多个消息类型。用protobuf编译器（protoc.exe）将.proto文件编译成目标语言（python | java | Cpp）会生存对应的.h文件和.cc文件。.proto文件中的每一个消息都有一个对应的类。

caffe使用起来非常简洁，很大程度上是由于caffe采用.proto文件作为用户的输入接口。用户可以直接通过编写.proto文件来定义网络模型和Solver。

编译命令

./autogen.sh./configure --prefix=/install/dirmakemake install# 加入环境变量
export PATH=$PATH:/install/dir/bin# 查看安装是否成功
protoc --version

Python binding安装（这样可以确保caffe和pycaffe对应的protobuf版本一致）

cd $root/pythonpython setup.py buildpython setup.py install

7. 安装HDF5 v5-1.10

下载地址：https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.10

功能介绍
HDF(Hierarchical Data File)是美国国家高级计算应用中心(National Center for Supercomputing Application,NCSA)为了满足各种领域研究需求而研制的一种能高效存储和分发科学数据的新型数据格式。

编译命令

./configure --prefix=/install/dirmakemake install

8. 安装snappy v1.1.6

下载地址：https://github.com/google/snappy/archive/1.1.6.zip

功能介绍

snappy是一个C++库，用来压缩可解压缩的开发包，旨在提供高速压缩和合理的压缩率。snappy比zlib更快，但文件相对要大20%-100%，caffe在数据处理时依赖于snappy库。

编译命令

mkdir build && cd buildcmake .. -DCMAKE_INSTALL_PREFIX=/install/dirmake make install

9. 安装 BLAS

功能介绍

Caffe支持三种AtLas，分别是MKL，AtLas，OpenBlas。其中AtLas，OpenBlas是完全免费的，可以直接通过apt-get方式来安装。OpenBlas的效率接近MKL，比AtLas高很多，所以建议安装OpenBlas。

编译命令

# 可以直接使用apt-get等工具安装
# atlas
sudo apt-get install libatlas-base-dev
# openblas
sudo apt-get install libopenblas-dev

源码安装OpenBlas参考：https://blog.csdn.net/qck15570179927/article/details/103399068

安装OpenCV

参考的教程很多，不再赘述。

安装CUDA

参考https://blog.csdn.net/weixin_42464187/article/details/103307463

编译caffe

编写载入caffe环境脚本样例

# for conda env
export PREFIX=/home/surui/env
export PATH=$PREFIX/bin:$PATH
source activate caffe# caffe python
export PYTHONPATH=/home/surui/Downloads/software/caffe/lib/caffe-master/python:$PYTHONPATH# caffe lib
CAFFE_LIB_PATH=/home/surui/Downloads/software/caffe/lib
# Boost
export BOOST_INCLUDE=$CAFFE_LIB_PATH/boost_1_56_0/install/include
export BOOST_LIBS=$CAFFE_LIB_PATH/boost_1_56_0/install/lib
# GFlags
export GFLAGS_INCLUDE=$CAFFE_LIB_PATH/gflags_v2.1.2/gflags-2.1.2/install/include
export GFLAGS_LIBS=$CAFFE_LIB_PATH/gflags_v2.1.2/gflags-2.1.2/install/lib
# Glog
export GLOG_INCLUDE=$CAFFE_LIB_PATH/glog_v0.3.3/glog-0.3.3/install/include
export GLOG_LIBS=$CAFFE_LIB_PATH/glog_v0.3.3/glog-0.3.3/install/lib
# LevelDB
export LEVELDB_INCLUDE=$CAFFE_LIB_PATH/leveldb_v1.20/leveldb-1.20/install/include
export LEVELDB_LIBS=$CAFFE_LIB_PATH/leveldb_v1.20/leveldb-1.20/install/lib
# LMDB
export LMDB_INCLUDE=$CAFFE_LIB_PATH/LMDB_0.9.18/lmdb-LMDB_0.9.18/libraries/liblmdb/install/include
export LMDB_LIBS=$CAFFE_LIB_PATH/LMDB_0.9.18/lmdb-LMDB_0.9.18/libraries/liblmdb/install/lib
# Protobuf
export PROTOBUF_INCLUDE=$CAFFE_LIB_PATH/protobuf-3.6.0/install/include
export PROTOBUF_LIBS=$CAFFE_LIB_PATH/protobuf-3.6.0/install/lib
export PATH=$CAFFE_LIB_PATH/protobuf-3.6.0/install/bin:$PATH
# HDF5
export HDF5_INCLUDE=$CAFFE_LIB_PATH/hdf5-1.10.0/install/include
export HDF5_LIBS=$CAFFE_LIB_PATH/hdf5-1.10.0/install/lib
# snappy
export SNAPPY_INCLUDE=$CAFFE_LIB_PATH/snappy_1.1.7/snappy-1.1.7/install/include
export SNAPPY_LIBS=$CAFFE_LIB_PATH/snappy_1.1.7/snappy-1.1.7/install/lib
# opencv
export OPENCV_INCLUDE=$CAFFE_LIB_PATH/opencv-3.4.0/install/include
export OPENCV_LIBS=$CAFFE_LIB_PATH/opencv-3.4.0/install/lib# for cuda
export LD_LIBRARY_PATH=$BOOST_LIBS:$GFLAGS_LIBS:$GLOG_LIBS:$LEVELDB_LIBS:$LMDB_LIBS:$PROTOBUF_LIBS:$HDF5_LIBS:$SNAPPY_LIBS:$OPENCV_LIBS:$/home/surui/env/envs/TRT5/lib:$LD_LIBRARY_PATH:/home/surui/Downloads/software/TensorRT-5.1.5.0/lib
export CUDA_INSTALL_DIR=/usr/local/cuda-10.1
export CUDNN_INSTALL_DIR=/home/surui/env/envs/TRT5
export CUDNN_LIBDIR=lib

编译caffe

修改Makefile.config
cp Makefile.config.example Makefile.config
修改以下部分

# uncomment to disable IO dependencies and corresponding data layers
USE_OPENCV := 1
USE_LEVELDB := 1
USE_LMDB := 1
# This code is taken from https://github.com/sh1r0/caffe-android-lib
USE_HDF5 := 1# uncomment to allow MDB_NOLOCK when reading LMDB files (only if necessary)
#       You should not set this flag if you will be reading LMDBs with any
#       possibility of simultaneous read and write
ALLOW_LMDB_NOLOCK := 1# Uncomment if you're using OpenCV 3
OPENCV_VERSION := 3# To customize your choice of compiler, uncomment and set the following.
# N.B. the default for Linux is g++ and the default for OSX is clang++
CUSTOM_CXX := g++# CUDA directory contains bin/ and lib/ directories that we need.
CUDA_DIR := /usr/local/cuda
# On Ubuntu 14.04, if cuda tools are installed via
# "sudo apt-get install nvidia-cuda-toolkit" then use this instead:
# CUDA_DIR := /usr# CUDA architecture setting: going with all of them.
# For CUDA < 6.0, comment the *_50 through *_61 lines for compatibility.
# For CUDA < 8.0, comment the *_60 and *_61 lines for compatibility.
# For CUDA >= 9.0, comment the *_20 and *_21 lines for compatibility.
CUDA_ARCH := -gencode arch=compute_30,code=sm_30 \-gencode arch=compute_35,code=sm_35 \-gencode arch=compute_50,code=sm_50 \-gencode arch=compute_52,code=sm_52 \-gencode arch=compute_60,code=sm_60 \-gencode arch=compute_61,code=sm_61 \-gencode arch=compute_61,code=compute_61.
.
.# NOTE: this is required only if you will compile the python interface.
# We need to be able to find Python.h and numpy/arrayobject.h.
#PYTHON_INCLUDE := /usr/include/python2.7 \
#               /usr/lib/python2.7/dist-packages/numpy/core/include
# Anaconda Python distribution is quite popular. Include path:
# Verify anaconda location, sometimes it's in root.
ANACONDA_HOME := /home/surui/env/envs/caffe
PYTHON_INCLUDE := $(ANACONDA_HOME)/include \$(ANACONDA_HOME)/include/python3.6m \$(ANACONDA_HOME)/lib/python3.6/site-packages/numpy/core/include# Uncomment to use Python 3 (default is Python 2)
#PYTHON_LIBRARIES := boost_python3 python3.5m
PYTHON_LIBRARIES := boost_python3 python3.6m
#PYTHON_INCLUDE := /home/surui/env/envs/caffe/include/python3.6m \
#                  /home/surui/env/envs/caffe/lib/python3.6/site-packages/numpy/core/include# We need to be able to find libpythonX.X.so or .dylib.
PYTHON_LIB := $(ANACONDA_HOME)/lib# Homebrew installs numpy in a non standard path (keg only)
# PYTHON_INCLUDE += $(dir $(shell python -c 'import numpy.core; print(numpy.core.__file__)'))/include
# PYTHON_LIB += $(shell brew --prefix numpy)/lib# Uncomment to support layers written in Python (will link against Python libs)
WITH_PYTHON_LAYER := 1# Whatever else you find you need goes here.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include $(BOOST_INCLUDE) $(GFLAGS_INCLUDE) $(GLOG_INCLUDE) $(LEVELDB_INCLUDE) $(LMDB_INCLUDE) $(PROTOBUF_INCLUDE) $(HDF5_INCLUDE) $(SNAPPY_INCLUDE) $(OPENCV_INCLUDE) $(PROTOBUF_INCLUDE)
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib $(BOOST_LIBS) $(GFLAGS_LIBS) $(GLOG_LIBS) $(LEVELDB_LIBS) $(LMDB_LIBS) $(PROTOBUF_LIBS) $(HDF5_LIBS) $(SNAPPY_LIBS) $(OPENCV_LIBS) $(PROTOBUF_LIBS)

编译命令

make allmake testmake runtestconda install numpyconda install scikit-imagemake pycaffe# 环境变量加入这个
export PYTHONPATH=$caffe_root/python:$PYTHONPATH

错误＆解答

错误 1：
nvcc fatal : Unsupported gpu architecture ‘compute_20’
解答 1：
gpu不支持该架构，所以注释调包含compute_20的行即可

错误 2：
import caffe 报错　
解答 2：
https://blog.csdn.net/jacke121/article/details/79554497

参考

[1] 深度学习－－caffe之经典模型详解与实战
[2] http://caffe.berkeleyvision.org/installation.html
[3] http://caffe.berkeleyvision.org/tutorial/