我的docker随笔26：制作arm平台的python-pandas镜像

需求：
构建 arm （linux_armv7l）平台上用于测试机器训练的 python 镜像，带 numpy、 pandas、sklearn，等。
本文构建所用操作系统为 ubuntu 16.04 64bit（4GB双核），采用容器内安装依赖库的形式，非 Dockerfile，是因为考虑到实际构建中可能会遇到各种问题。

技术总结

在 pc 端运行 arm 镜像容器，使用arm32v7/python，此方式是为了方便制作（也可在 arm 系统上直接制作）。
镜像标签为 slim，其为 Debian 的 buster 版本。容器中无法补齐命令，无法查看以往命令，使用较为麻烦。
安装编译相关工具和库，因为有些 python 库要本地编译（据查，是没有该平台的预编译包）。
安装 numpy 等库。注意，由于官方没有现成的包，需要在本地编译，故会较耗时。
pip 安装会顺带安装相应依赖包。
写程序验证（本文略）。
国内源是为了加快下载速度。编译耗时取决于机器性能。

知识点

在 x86 上运行 arm 容器。
从头开始编译、安装 python 库。
从容器变成镜像。

实验步骤

运行容器

建立 pc 端运行 arm 容器环境：

docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

注：经测试发现，ubuntu 内核需在 4.8 以上。

运行基础镜像：

docker run -itd --name pythonslim arm32v7/python:3.7-slim sh

以下命令中，安装、测试等在容器内进行。与 docker 有关的，在宿主机上进行。本文假定读者能区别出来。

添加源

添加 debian 国内源，文件：

cat > /etc/apt/sources.list <<-EOF
deb http://mirrors.aliyun.com/debian/ buster main non-free contrib
deb-src http://mirrors.aliyun.com/debian/ buster main non-free contrib
deb http://mirrors.aliyun.com/debian-security buster/updates main
deb-src http://mirrors.aliyun.com/debian-security buster/updates main
deb http://mirrors.aliyun.com/debian/ buster-updates main non-free contrib
deb-src http://mirrors.aliyun.com/debian/ buster-updates main non-free contrib
deb http://mirrors.aliyun.com/debian/ buster-backports main non-free contrib
deb-src http://mirrors.aliyun.com/debian/ buster-backports main non-free contrib
EOF

原始内容为：

# deb http://snapshot.debian.org/archive/debian/20200414T000000Z buster main
deb http://deb.debian.org/debian buster main
# deb http://snapshot.debian.org/archive/debian-security/20200414T000000Z buster/updates main
deb http://security.debian.org/debian-security buster/updates main
# deb http://snapshot.debian.org/archive/debian/20200414T000000Z buster-updates main
deb http://deb.debian.org/debian buster-updates main

添加 pip 国内源：

mkdir ~/.pip/
cat > ~/.pip/pip.conf <<-EOF
[global]
index-url = https://pypi.tuna.tsinghua.edu.cn/simple
[install]
trusted-host = https://pypi.tuna.tsinghua.edu.cn
EOF

注1：slim 版本没有 vi 编辑器，故用此法。
注2：也可在 pip 安装时用 -i 临时指定源地址。

安装编译环境

apt-get install gcc g++ gfortran python-dev libopenblas-dev libblas-dev liblapack-dev cython -yapt-get install libfreetype6-dev libpng-dev -y
apt-get install pkg-config -y  # 注：需要此工具找freetype
apt-get install libfontconfig1-dev -y

安装包

pip install numpy==1.18.1 -i https://pypi.tuna.tsinghua.edu.cn/simplepip install pandas==0.23.4 -i https://pypi.tuna.tsinghua.edu.cn/simplepip install scipy==1.4.1 -i https://pypi.tuna.tsinghua.edu.cn/simplepip install Cython -i https://pypi.tuna.tsinghua.edu.cn/simplepip install sklearn -i https://pypi.tuna.tsinghua.edu.cn/simple   注：依赖scipy Cythonpip install six -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install xlrd -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install pyparsing -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install python-dateutil -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install matplotlib==3.2.1 -i https://pypi.tuna.tsinghua.edu.cn/simple  # 注：要freetype，先不安装
pip install pyhht -i https://pypi.tuna.tsinghua.edu.cn/simple  # 注：需要 scipy、matplotlib

注1：可用pip list查看安装的库及其版本。
注2：安装（编译） numpy、pandas、scipy、sklearn 等较耗时，每个包耗时数小时不等（因 slim 容器没有 time 命令，无法知道具体耗时时间）。

查看安装的包

本容器安装的包：

# pip list
Package         Version
--------------- ------------
cycler          0.10.0
Cython          0.29.16
freetype-py     2.1.0.post1
joblib          0.14.1
kiwisolver      1.2.0
matplotlib      3.2.1
numpy           1.18.1
pandas          0.23.4
pip             20.0.2
pyhht           0.1.0
pyparsing       2.4.7
python-dateutil 2.8.1
pytz            2019.3
scikit-learn    0.22.2.post1
scipy           1.4.1
setuptools      46.1.3
six             1.14.0
sklearn         0.0
wheel           0.34.2
xlrd            1.2.0

验证

# python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn import datasets,linear_model

如果没有错误输出，说明安装成功。

制作镜像

查看容器体积：

# du -h --max-depth=1
136M    ./tmp
26M     ./var
2.5M    ./sbin
3.2M    ./bin
220M    ./root
1.5M    ./etc
835M    ./usr
7.1M    ./lib1.2G

将原始的容器保存为镜像：

docker commit pythonslim python-pandas-build:arm

这一步的目的是为了保留编译信息。方便后续制作。

清除不必要的文件：

apt-get autoremove python2   # bzip2 file会被删除
apt-get autoremove gcc g++ gfortran # libgomp1 binutils  binutils-arm-linux-gnueabihf 会被删除apt-get autoremove cython
apt-get autoremove perlapt-get autoremove opensslrm /usr/bin/perl  /usr/bin/perl5.28.1

补回被删除的包：

apt-get install libgomp1 # 注：sklearn依赖此包
apt-get install bzip2

清除缓存：

apt-get clean && rm -rf /var/lib/apt/lists/*rm -rf /root/.cache

经分析，slim 版本的镜像本身就超过 100 MB，加上 python 的几个重要库，经精简后，体积仍近 600 MB。其中占大头的目录是/usr/local/lib/python3.7/site-packages。

将精简后的容器保存为镜像：

docker commit pythonslim python-pandas:arm

打标签，提交(到笔者的阿里云仓库)：

docker tag python-pandas:arm registry.cn-hangzhou.aliyuncs.com/latelee/python-pandas:armdocker push registry.cn-hangzhou.aliyuncs.com/latelee/python-pandas:arm

注：Docker 构建是分层的，不能在python-pandas-build:arm镜像中精简，因为此镜像已超 1 GB，即使删除文件，Docker 镜像亦举减少。因此，需要在pythonslim中精简。

运行：

docker run -itd --name pandas -v $PWD:/work registry.cn-hangzhou.aliyuncs.com/latelee/python-pandas:arm sh

问题及解决

安装了 libfreetype6-dev 后，编译 matplotlib 时还是提示 freetype 版本过低（即找不到库），后添加 pkg-config ，可编译通过。

使用

registry.cn-hangzhou.aliyuncs.com/latelee/python-pandas:arm为公开镜像（仅在当前可访问，后续不保证）。安装软件需要执行apt-get update。

参考

scipy 镜像构建参考：
https://github.com/publysher/docker-alpine-numpy
https://github.com/publysher/docker-alpine-scipy
https://github.com/publysher/docker-alpine-sklearn
https://github.com/amancevice/docker-pandas

scipy 安装指导：
https://docs.scipy.org/doc/scipy-1.1.0/reference/building/linux.html

python 的 alpine 镜像的问题：
https://pythonspeed.com/articles/alpine-docker-python/

scipy 在 alpine 上安装问题：
https://github.com/scipy/scipy/issues/9481
https://github.com/scipy/scipy/issues/9338

附

slim镜像出错信息：

 ERROR: Command errored out with exit status 1:command: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmp5_jvogwacwd: /tmp/pip-install-kyd6kagr/scipyComplete output (137 lines):lapack_opt_info:lapack_mkl_info:customize UnixCCompilerlibraries mkl_rt not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/']NOT AVAILABLEopenblas_lapack_info:customize UnixCCompilercustomize UnixCCompilerlibraries openblas not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/']NOT AVAILABLEopenblas_clapack_info:customize UnixCCompilercustomize UnixCCompilerlibraries openblas,lapack not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/']NOT AVAILABLEatlas_3_10_threads_info:Setting PTATLAS=ATLAScustomize UnixCCompilerlibraries tatlas,tatlas not found in /usr/local/libcustomize UnixCCompilerlibraries lapack_atlas not found in /usr/local/libcustomize UnixCCompilerlibraries tatlas,tatlas not found in /usr/libcustomize UnixCCompilerlibraries lapack_atlas not found in /usr/libcustomize UnixCCompilerlibraries tatlas,tatlas not found in /usr/lib/customize UnixCCompilerlibraries lapack_atlas not found in /usr/lib/<class 'numpy.distutils.system_info.atlas_3_10_threads_info'>NOT AVAILABLEatlas_3_10_info:customize UnixCCompilerlibraries satlas,satlas not found in /usr/local/libcustomize UnixCCompilerlibraries lapack_atlas not found in /usr/local/libcustomize UnixCCompilerlibraries satlas,satlas not found in /usr/libcustomize UnixCCompilerlibraries lapack_atlas not found in /usr/libcustomize UnixCCompilerlibraries satlas,satlas not found in /usr/lib/customize UnixCCompilerlibraries lapack_atlas not found in /usr/lib/<class 'numpy.distutils.system_info.atlas_3_10_info'>NOT AVAILABLEatlas_threads_info:Setting PTATLAS=ATLAScustomize UnixCCompilerlibraries ptf77blas,ptcblas,atlas not found in /usr/local/libcustomize UnixCCompilerlibraries lapack_atlas not found in /usr/local/libcustomize UnixCCompilerlibraries ptf77blas,ptcblas,atlas not found in /usr/libcustomize UnixCCompilerlibraries lapack_atlas not found in /usr/libcustomize UnixCCompilerlibraries ptf77blas,ptcblas,atlas not found in /usr/lib/customize UnixCCompilerlibraries lapack_atlas not found in /usr/lib/<class 'numpy.distutils.system_info.atlas_threads_info'>NOT AVAILABLEatlas_info:customize UnixCCompilerlibraries f77blas,cblas,atlas not found in /usr/local/libcustomize UnixCCompilerlibraries lapack_atlas not found in /usr/local/libcustomize UnixCCompilerlibraries f77blas,cblas,atlas not found in /usr/libcustomize UnixCCompilerlibraries lapack_atlas not found in /usr/libcustomize UnixCCompilerlibraries f77blas,cblas,atlas not found in /usr/lib/customize UnixCCompilerlibraries lapack_atlas not found in /usr/lib/<class 'numpy.distutils.system_info.atlas_info'>NOT AVAILABLElapack_info:customize UnixCCompilerlibraries lapack not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/']NOT AVAILABLElapack_src_info:NOT AVAILABLENOT AVAILABLEsetup.py:420: UserWarning: Unrecognized setuptools command ('dist_info --egg-base /tmp/pip-modern-metadata-n429kcrr'), proceeding with generating Cython sources and expanding templates' '.join(sys.argv[1:])))Running from scipy source directory./tmp/pip-build-env-dwf79396/overlay/lib/python3.7/site-packages/numpy/distutils/system_info.py:624: UserWarning:Atlas (http://math-atlas.sourceforge.net/) libraries not found.Directories to search for the libraries can be specified in thenumpy/distutils/site.cfg file (section [atlas]) or by settingthe ATLAS environment variable.self.calc_info()/tmp/pip-build-env-dwf79396/overlay/lib/python3.7/site-packages/numpy/distutils/system_info.py:624: UserWarning:Lapack (http://www.netlib.org/lapack/) libraries not found.Directories to search for the libraries can be specified in thenumpy/distutils/site.cfg file (section [lapack]) or by settingthe LAPACK environment variable.self.calc_info()/tmp/pip-build-env-dwf79396/overlay/lib/python3.7/site-packages/numpy/distutils/system_info.py:624: UserWarning:Lapack (http://www.netlib.org/lapack/) sources not found.Directories to search for the sources can be specified in thenumpy/distutils/site.cfg file (section [lapack_src]) or by settingthe LAPACK_SRC environment variable.self.calc_info()Traceback (most recent call last):File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 257, in <module>main()File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 240, in mainjson_out['return_val'] = hook(**hook_input['kwargs'])File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 110, in prepare_metadata_for_build_wheelreturn hook(metadata_directory, config_settings)File "/tmp/pip-build-env-dwf79396/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 158, in prepare_metadata_for_build_wheelself.run_setup()File "/tmp/pip-build-env-dwf79396/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 250, in run_setupself).run_setup(setup_script=setup_script)File "/tmp/pip-build-env-dwf79396/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 143, in run_setupexec(compile(code, __file__, 'exec'), locals())File "setup.py", line 540, in <module>setup_package()File "setup.py", line 536, in setup_packagesetup(**metadata)File "/tmp/pip-build-env-dwf79396/overlay/lib/python3.7/site-packages/numpy/distutils/core.py", line 135, in setupconfig = configuration()File "setup.py", line 435, in configurationraise NotFoundError(msg)numpy.distutils.system_info.NotFoundError: No lapack/blas resources found.----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmp5_jvogwa Check the logs for full command output.
WARNING: You are using pip version 19.3.1; however, version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

原因：缺少编译依赖的库。

ModuleNotFoundError: No module named 'Cython'
raise ModuleNotFoundError(message)
ModuleNotFoundError: Please install Cython with a version >= 0.28.5 in order to build a scikit-learn from source.

原因：Cython 未安装。

raise ReadTimeoutError(self._pool, None, "Read timed out.")
pip._vendor.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.

原因：网络原因超时，重试。

src/checkdep_freetype2.c:3:6: error: #error "FreeType version 2.3 or higher is required. You may set the MPLLOCALFREETYPE environment variable to 1 to let Matplotlib download it."#error "FreeType version 2.3 or higher is required. \^~~~~src/checkdep_freetype2.c:10:10: error: #include expects "FILENAME" or <FILENAME>#include FT_FREETYPE_H^~~~~~~~~~~~~src/checkdep_freetype2.c:15:9: note: #pragma message: Compiling with FreeType version FREETYPE_MAJOR.FREETYPE_MINOR.FREETYPE_PATCH.#pragma message("Compiling with FreeType version " \^~~~~~~src/checkdep_freetype2.c:18:4: error: #error "FreeType version 2.3 or higher is required. You may set the MPLLOCALFREETYPE environment variable to 1 to let Matplotlib download it."#error "FreeType version 2.3 or higher is required. \^~~~~error: command 'gcc' failed with exit status 1

原因：安装 freetype。

alpine镜像：

libraries lapack_atlas not found in /usr/local/liblibraries tatlas,tatlas not found in /usr/local/liblibraries lapack_atlas not found in /usr/liblibraries tatlas,tatlas not found in /usr/lib<class 'numpy.distutils.system_info.atlas_3_10_threads_info'>NOT AVAILABLEatlas_3_10_info:libraries lapack_atlas not found in /usr/local/liblibraries satlas,satlas not found in /usr/local/liblibraries lapack_atlas not found in /usr/liblibraries satlas,satlas not found in /usr/lib<class 'numpy.distutils.system_info.atlas_3_10_info'>NOT AVAILABLEatlas_threads_info:Setting PTATLAS=ATLASlibraries lapack_atlas not found in /usr/local/liblibraries ptf77blas,ptcblas,atlas not found in /usr/local/liblibraries lapack_atlas not found in /usr/liblibraries ptf77blas,ptcblas,atlas not found in /usr/lib<class 'numpy.distutils.system_info.atlas_threads_info'>NOT AVAILABLEatlas_info:libraries lapack_atlas not found in /usr/local/liblibraries f77blas,cblas,atlas not found in /usr/local/liblibraries lapack_atlas not found in /usr/liblibraries f77blas,cblas,atlas not found in /usr/lib<class 'numpy.distutils.system_info.atlas_info'>NOT AVAILABLElapack_info:libraries lapack not found in ['/usr/local/lib', '/usr/lib']NOT AVAILABLElapack_src_info:NOT AVAILABLENOT AVAILABLErunning dist_inforunning build_srcbuild_srcbuilding py_modules sourcescreating buildcreating build/src.linux-armv7l-3.5creating build/src.linux-armv7l-3.5/numpycreating build/src.linux-armv7l-3.5/numpy/distutilsbuilding library "npymath" sourcesCould not locate executable gfortranCould not locate executable f95Could not locate executable ifortCould not locate executable ifcCould not locate executable lf95Could not locate executable pgfortranCould not locate executable f90Could not locate executable f77Could not locate executable fortCould not locate executable efortCould not locate executable efcCould not locate executable g77Could not locate executable g95Could not locate executable pathf95Could not locate executable nagfordon't know how to compile Fortran code on platform 'posix'Running from numpy source directory.setup.py:461: UserWarning: Unrecognized setuptools command, proceeding with generating Cython sources and expanding templatesrun_build = parse_setuppy_commands()/tmp/pip-install-lpg1hl36/numpy/numpy/distutils/system_info.py:1896: UserWarning:Optimized (vendor) Blas libraries are not found.Falls back to netlib Blas library which has worse performance.A better performance should be easily gained by switchingBlas library.if self._calc_info(blas):/tmp/pip-install-lpg1hl36/numpy/numpy/distutils/system_info.py:1896: UserWarning:Blas (http://www.netlib.org/blas/) libraries not found.Directories to search for the libraries can be specified in thenumpy/distutils/site.cfg file (section [blas]) or by settingthe BLAS environment variable.if self._calc_info(blas):/tmp/pip-install-lpg1hl36/numpy/numpy/distutils/system_info.py:1896: UserWarning:Blas (http://www.netlib.org/blas/) sources not found.Directories to search for the sources can be specified in thenumpy/distutils/site.cfg file (section [blas_src]) or by settingthe BLAS_SRC environment variable.if self._calc_info(blas):/tmp/pip-install-lpg1hl36/numpy/numpy/distutils/system_info.py:1730: UserWarning:Lapack (http://www.netlib.org/lapack/) libraries not found.Directories to search for the libraries can be specified in thenumpy/distutils/site.cfg file (section [lapack]) or by settingthe LAPACK environment variable.return getattr(self, '_calc_info_{}'.format(name))()/tmp/pip-install-lpg1hl36/numpy/numpy/distutils/system_info.py:1730: UserWarning:Lapack (http://www.netlib.org/lapack/) sources not found.Directories to search for the sources can be specified in thenumpy/distutils/site.cfg file (section [lapack_src]) or by settingthe LAPACK_SRC environment variable.return getattr(self, '_calc_info_{}'.format(name))()/usr/local/lib/python3.5/distutils/dist.py:261: UserWarning: Unknown distribution option: 'define_macros'warnings.warn(msg)Traceback (most recent call last):File "/usr/local/lib/python3.5/site-packages/pip/_vendor/pep517/_in_process.py", line 257, in <module>main()File "/usr/local/lib/python3.5/site-packages/pip/_vendor/pep517/_in_process.py", line 240, in mainjson_out['return_val'] = hook(**hook_input['kwargs'])File "/usr/local/lib/python3.5/site-packages/pip/_vendor/pep517/_in_process.py", line 110, in prepare_metadata_for_build_wheelreturn hook(metadata_directory, config_settings)File "/tmp/pip-build-env-35qyjrc3/overlay/lib/python3.5/site-packages/setuptools/build_meta.py", line 158, in prepare_metadata_for_build_wheelself.run_setup()File "/tmp/pip-build-env-35qyjrc3/overlay/lib/python3.5/site-packages/setuptools/build_meta.py", line 250, in run_setupself).run_setup(setup_script=setup_script)File "/tmp/pip-build-env-35qyjrc3/overlay/lib/python3.5/site-packages/setuptools/build_meta.py", line 143, in run_setupexec(compile(code, __file__, 'exec'), locals())File "setup.py", line 488, in <module>setup_package()File "setup.py", line 480, in setup_packagesetup(**metadata)File "/tmp/pip-install-lpg1hl36/numpy/numpy/distutils/core.py", line 171, in setupreturn old_setup(**new_attr)File "/tmp/pip-build-env-35qyjrc3/overlay/lib/python3.5/site-packages/setuptools/__init__.py", line 144, in setupreturn distutils.core.setup(**attrs)File "/usr/local/lib/python3.5/distutils/core.py", line 148, in setupdist.run_commands()File "/usr/local/lib/python3.5/distutils/dist.py", line 955, in run_commandsself.run_command(cmd)File "/usr/local/lib/python3.5/distutils/dist.py", line 974, in run_commandcmd_obj.run()File "/tmp/pip-build-env-35qyjrc3/overlay/lib/python3.5/site-packages/setuptools/command/dist_info.py", line 31, in runegg_info.run()File "/tmp/pip-install-lpg1hl36/numpy/numpy/distutils/command/egg_info.py", line 26, in runself.run_command("build_src")File "/usr/local/lib/python3.5/distutils/cmd.py", line 313, in run_commandself.distribution.run_command(command)File "/usr/local/lib/python3.5/distutils/dist.py", line 974, in run_commandcmd_obj.run()File "/tmp/pip-install-lpg1hl36/numpy/numpy/distutils/command/build_src.py", line 146, in runself.build_sources()File "/tmp/pip-install-lpg1hl36/numpy/numpy/distutils/command/build_src.py", line 157, in build_sourcesself.build_library_sources(*libname_info)File "/tmp/pip-install-lpg1hl36/numpy/numpy/distutils/command/build_src.py", line 290, in build_library_sourcessources = self.generate_sources(sources, (lib_name, build_info))File "/tmp/pip-install-lpg1hl36/numpy/numpy/distutils/command/build_src.py", line 380, in generate_sourcessource = func(extension, build_dir)File "numpy/core/setup.py", line 661, in get_mathlib_inforaise RuntimeError("Broken toolchain: cannot link a simple C program")RuntimeError: Broken toolchain: cannot link a simple C program----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python /usr/local/lib/python3.5/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpxxww43e_ Check the logs for full command output.
The command '/bin/sh -c pip install $(grep numpy requirements.txt) &&     pip install -r requirements.txt' returned a non-zero code: 1

原因：缺少库，同 slim 版本，本文不采用。