hadoop3源码编译

编译hadoop的原因，官网的hadoop版本是预编译版，由于hadoop需要读写文件，不可能全部用java实现，需要c,c++编译成的动态链接库即.so文件。hadoop编译可以使它支持一些压缩工具。
笔者看了尚硅谷的hadoop教学视频，有意自主动手编译hadoop-3.1.3版本，从官网下载源码解压后发现有start-build-env.sh脚本，该脚本通过dev-support/docker下的Dockerfile文件构建了一个docker镜像，在该镜像构建时安装hadoop编译所需的环境工具。
由于3.1.3版本相对较老，在dev-support/Dockerfile文件中大部分配置不实用，镜像是基于ubuntu:16.04,nodejs版本太老，无法正常安装browser，故需要对其大规模修改。由于ubuntu的社区属性，容易产生软件版本冲突，故这里建议选用Redhat公司支持的linux系列
我的Dockerfie文件如下

FROM centos:7WORKDIR /rootADD jdk-8u291-linux-x64.tar.gz /opt/
# 注意这里jdk放在dev-support/docker文件夹下，注意修改文件名
RUN mv /opt/jdk1.8.0_291 /opt/jdk8
ENV JAVA_HOME /opt/jdk8
ENV CLASSPATH $JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
ENV PATH=$PATH:$JAVA_HOME/binRUN yum update -y && \yum install -y java-1.8.0-openjdk \gcc* \make \snappy* \bzip2* \lzo* \zlib* \openssl* \svn \ncurses* \autocong \automake \libtool \epel-release \*zstd* \gcc-c++ \bats \ShellCheck \python3 \sudo \fuse3 \fuse3-devel \doxygen \git \rsync \patch \vim
######
# Install cmake 3.1.0
######
RUN mkdir -p /opt/cmake && \curl -L -s -S \https://cmake.org/files/v3.20/cmake-3.20.0-linux-x86_64.tar.gz \-o /opt/cmake.tar.gz && \tar xzf /opt/cmake.tar.gz --strip-components 1 -C /opt/cmake
ENV CMAKE_HOME /opt/cmake
ENV PATH "${PATH}:/opt/cmake/bin"######
# Install Google Protobuf 2.5.0
######
RUN mkdir -p /opt/protobuf-src && \curl -L -s -S \https://github.com/google/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz \-o /opt/protobuf.tar.gz && \tar xzf /opt/protobuf.tar.gz --strip-components 1 -C /opt/protobuf-src
RUN cd /opt/protobuf-src && ./configure --prefix=/opt/protobuf && make install
ENV PROTOBUF_HOME /opt/protobuf
ENV PATH "${PATH}:/opt/protobuf/bin"#RUN curl -L -s  https://rpm.nodesource.com/setup_10.x | bash - && \
# RUN yum install -y nodejs && \
#     npm config set registry https://registry.npm.taobao.org
# RUN npm install -g n && \
#     n lts && PATH="$PATH" && \
#     npm install -g bower && \
#     npm install -g ember-cli
RUN yum install -y wget && \mkdir -p /opt/nodejs && \wget -O /opt/nodejs.tar.xz https://npm.taobao.org/mirrors/node/v14.17.5/node-v14.17.5-linux-x64.tar.xz && \tar xf /opt/nodejs.tar.xz --strip-components 1 -C /opt/nodejs && \ln -s /opt/nodejs/bin/npm /usr/local/bin && \ln -s /opt/nodejs/bin/node /usr/local/bin && \npm config set registry https://registry.npm.taobao.org && \npm install -g bower && \npm install -g ember-cliRUN pip3 install pylint==2.6.0 python-dateutil==2.8.1  -i https://pypi.doubanio.com/simpleRUN mkdir -p /opt/maven && \curl -L -s -S \https://dlcdn.apache.org/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz \-o /opt/maven.tar.gz && \tar xzf /opt/maven.tar.gz --strip-components 1 -C /opt/maven
ENV MAVEN_HOME /opt/maven
ENV PATH "${PATH}:/opt/maven/bin"RUN mkdir -p /opt/isa-l-src \&& yum install -y  automake yasm libtool\&& curl -L -s -S \https://github.com/intel/isa-l/archive/v2.29.0.tar.gz \-o /opt/isa-l.tar.gz \&& tar xzf /opt/isa-l.tar.gz --strip-components 1 -C /opt/isa-l-src \&& cd /opt/isa-l-src \&& ./autogen.sh \&& ./configure \&& make "-j$(nproc)" \&& make install \&& cd /root \&& rm -rf /opt/isa-l-src
###
# Avoid out of memory errors in builds
###
ENV MAVEN_OPTS -Xms512m -Xmx3072m
# ENV MAVEN_OPTS -Xms256m -Xmx1536m# Add a welcome message and environment checks.
ADD hadoop_env_checks.sh /root/hadoop_env_checks.sh
RUN chmod 755 /root/hadoop_env_checks.sh
RUN echo '~/hadoop_env_checks.sh' >> /root/.bashrc

我创建了一个source_code 文件夹，将hadoop-3.1.3-src文件夹放入其中。start-build-env.sh文件默认是通过pwd名利把hadoop-3.1.3-src挂载到docker，为了提高该docker容器的复用性（比如之后要编译hive,spark等需要用到类似的环境），在该脚本文件中把默认的 -v "${PWD}:/home/${USER_NAME}/hadoop${V_OPTS:-}" \ 替换成 -v "/xxx/source_code/:/home/${USER_NAME}/source${V_OPTS:-}" \

如上图，说明docker容器构建成功
BUILDING文件中

Building distributions:

Create binary distribution without native code and without documentation:

$ mvn package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true

Create binary distribution with native code and with documentation:

$ mvn package -Pdist,native,docs -DskipTests -Dtar

Create source distribution:

$ mvn package -Psrc -DskipTests

Create source and binary distributions with native code and documentation:

$ mvn package -Pdist,native,docs,src -DskipTests -Dtar

Create a local staging version of the website (in /tmp/hadoop-site)

$ mvn clean site -Preleasedocs; mvn site:stage -DstagingDirectory=/tmp/hadoop-site

Note that the site needs to be built in a second pass after other artifacts.

cd ../source
cd hadoop-3.1.3-src
推荐 mvn package -Pdist,native,docs -DskipTests -Dtar

报错

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce (enforce-banned-dependencies) on project hadoop-client-check-test-invariants: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed. -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :hadoop-client-check-test-invariants

hadoop-client-check-test-invariants 该模块在hadoop各个版本编译过程中经常报错，例如当时最新的版本hadoop-3.3.1 利用提供的脚本无需改变Dockerfile即可成功构建容器，但仍然该模块编译报错
解决方法：该模块属于hadoop-client-modules/，在其pom.xml文件中将该模块注释，不将其编译不会造成太大影响

  <modules><!-- Left as an empty artifact w/dep for compat --><module>hadoop-client</module><!-- Should be used at compile scope for access to IA.Public classes --><module>hadoop-client-api</module><!-- Should be used at runtime scope for remaining classes necessary for hadoop-client-api to function --><module>hadoop-client-runtime</module><!-- Should be used at test scope for those that need access to mini cluster that works with above api and runtime --><module>hadoop-client-minicluster</module><!-- Checks invariants above --><module>hadoop-client-check-invariants</module>
<!--     <module>hadoop-client-check-test-invariants</module> --><!-- Attempt to use the created libraries --><module>hadoop-client-integration-tests</module></modules>

重新执行mvn package -Pdist,native,docs -DskipTests -Dtar

如上图所示编译成功

hadoop3源码编译相关推荐

Hadoop-2.8.0集群搭建、hadoop源码编译和安装、host配置、ssh免密登录、hadoop配置文件中的参数配置参数总结、hadoop集群测试,安装过程中的常见错误
25. 集群搭建 25.1 HADOOP集群搭建 25.1.1集群简介 HADOOP集群具体来说包含两个集群:HDFS集群和YARN集群,两者逻辑上分离,但物理上常在一起 HDFS集群: 负责海量数据 ...
java调用clang编译的so_写Java这么久，JDK源码编译过没？编译JDK源码踩坑纪实
好奇害死羊很多小伙伴们做Java开发,天天写Java代码,肯定离不开Java基础环境:JDK,毕竟我们写好的Java代码也是跑在JVM虚拟机上. 一般来说,我们学Java之前,第一步就是安装JDK环 ...
5单个编译总会编译全部_玩转Android10(五)源码编译开发中常用命令
源码开发编译中,熟练掌握常用命令,可以提高开发工作效率.Android源码中,将相关的命令分为如下几类: 1.初始化源码编译环境初始化编译环境,为后续提供如lunch.make.xxgrep.god ...
mono和monodevelop源码编译安装
之所以用源码编译的方式安装mono和monodevelop,是因为通过yum安装的mono不是最新版本,而且monodevelop不能建 asp.net MVC3的工程. 而且通过源码安装,可以进一步 ...
nginx源码编译、负载均衡及模块的扩展
1.nginx源码编译实验环境: iptables和selinux关闭 redhat6.5 nginx:test1: 172.25.1.11 [root@test1 ~]# ls nginx-1.1 ...
mac通过tree源码编译安装tree
通过tree源码编译安装下载源码:curl -O ftp://mama.indstate.edu/linux/tree/tree-1.6.0.tgz 解压源码:tar xzvf tree-1.6 ...
干货|TensorFlow开发环境搭建（Ubuntu16.04+GPU+TensorFlow源码编译）
点击上方"小白学视觉",选择加"星标"或"置顶" 重磅干货,第一时间送达本文转自|机器学习算法工程师安装平台 1 平台目前Tensor ...
Nginx 源码编译安装
Nginx 源码编译安装环境 Centos7 Nginx1.8.1 下载地址:http://nginx.org/download/ 选择自己想要的版本我这边使用1.8.1,下载地址:http: ...
基本lnmp平台的搭建（源码编译）
lnmp :linux ,nginx ,mysql ,php 系统环境:rhel6.0 selinux and iptables disabled (这里我们都只用源码编译的方式) 1 ste ...

hadoop3源码编译

报错

hadoop3源码编译相关推荐

最新文章

热门文章