hadoop-2.5.1 源码编译

hadoop-2.5.1编译步骤(只需一个节点即可)

环境：	系统 CentOS 6.4 64 位 m1 172.103.55.12 软件： jdk1.7、hadoop- 2.5.1- src.tar.gz、maven3.2.3、protobuf2.5.0

注：需要连接互联网

1. 安装配置JDK（略）

2. 安装基本应用程序 (需要连接互联网网)

yum -y install svn ncurses-devel gcc*
yum -y install lzo-devel zlib-devel autoconf automake libtool cmake openssl –devel

3. 安装 maven

下载 apache-maven-3.2.3-bin.tar.gz
并解压 tar zxvf apache- maven- 3.2.3- bin.tar.gz

配置环境变量
export MAVEN_HOME=/usr/local/program/maven/ apache- maven- 3.2.3
export PATH=$PATH:$MAVEN_HOME/bin
添加之后执行sourc /etc/profile

4. 安装 protobuf（不安装，编译将无法完成）

hadoop使用protocol buffer进行通信，需要下载和安装protobhf-2.5.0.tar.gz；由于官网已经无法下载了，

可以到百度云盘下载http://pan.baidu.com/s/1pJlZubT

编译安装 protobuf
①  cd protobuf - 2.5.0
②  ./configure
③  make
④  make install

检测 protoc –version

5. 编译 hadoop-2.5.1.src

a)     下载  hadoop-2.5.1-src.tat.gz
源码包下载地址  http://apache.claz.org/hadoop/common/hadoop-2.5.1/
红色标志  hadoop-2.5.1- src.tar.gz 为源码包

b) 解压hadoop-2.5.1-src.tar.gz

c) 开始编译

进入hadoop-2.5.1.src 目录：

执行命令 mvn clean package –Pdist,native –DskipTests –Dtar

（或者 mvn package -Pdist,native -DskipTests -Dtar）

此步骤要求逐级能访问公网，并且此步骤比较漫长，请耐心等待。

经过漫长的等待后，终于看到了Building SUCCESS:

[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 4.389 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 3.163 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 6.377 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.562 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 3.136 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 6.757 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 5.506 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [ 6.269 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 4.341 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [03:25 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 23.998 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.160 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [07:52 min]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [11:03 min]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [03:23 min]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 14.324 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.343 s]
[INFO] hadoop-yarn ........................................ SUCCESS [ 0.092 s]
[INFO] hadoop-yarn-api .................................... SUCCESS [02:46 min]
[INFO] hadoop-yarn-common ................................. SUCCESS [02:37 min]
[INFO] hadoop-yarn-server ................................. SUCCESS [ 1.142 s]
[INFO] hadoop-yarn-server-common .......................... SUCCESS [01:02 min]
[INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [02:26 min]
[INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [ 22.258 s]
[INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [ 16.444 s]
[INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [ 51.768 s]
[INFO] hadoop-yarn-server-tests ........................... SUCCESS [ 9.864 s]
[INFO] hadoop-yarn-client ................................. SUCCESS [ 16.057 s]
[INFO] hadoop-yarn-applications ........................... SUCCESS [ 0.162 s]
[INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [ 7.307 s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [ 3.243 s]
[INFO] hadoop-yarn-site ................................... SUCCESS [ 0.099 s]
[INFO] hadoop-yarn-project ................................ SUCCESS [ 18.997 s]
[INFO] hadoop-mapreduce-client ............................ SUCCESS [ 2.306 s]
[INFO] hadoop-mapreduce-client-core ....................... SUCCESS [01:34 min]
[INFO] hadoop-mapreduce-client-common ..................... SUCCESS [01:05 min]
[INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [ 20.109 s]
[INFO] hadoop-mapreduce-client-app ........................ SUCCESS [ 22.691 s]
[INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [ 30.368 s]
[INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [01:30 min]
[INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [ 19.958 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 24.097 s]
[INFO] hadoop-mapreduce ................................... SUCCESS [ 17.757 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [01:59 min]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 48.464 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [ 14.187 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 15.098 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 13.630 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [ 8.080 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [ 6.496 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [ 23.855 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 17.886 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [ 18.728 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 1.060 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 35.296 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 13.476 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.134 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [01:20 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 51:55 min
[INFO] Finished at: 2014-12-09T19:16:49-08:00
[INFO] Final Memory: 142M/350M
[INFO] ------------------------------------------------------------------------

编译成功后， /hadoop-dist/target/hadoop-2.5.1.tar.gz 就是我们需要的文件了；

可能遇到的问题：

报错[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile (default-compile) on project hadoop-hdfs: Fatal error compiling: Error while executing the compiler. InvocationTargetException: Java heap space

分配内存不足,参考如下为maven配置JVM参数: export MAVEN_OPTS=”-Xms256m -Xmx512m -Djava.awt.headless=true”

我直接在profile文件中：export MAVEN_OPTS=-Xmx1024

hadoop-2.5.1 源码编译相关推荐

pip安装wxpython报错traceback_使用源码编译wxpython-基于python2.7
1.前言本文主要讲述在linux环境下进行编译wxpython,在windows下面安装wxpython很简单,只要下载,然后直接执行exe文件,下一步下一步即可安装,在linux下面,则具有很多步 ...
Hadoop-2.8.0集群搭建、hadoop源码编译和安装、host配置、ssh免密登录、hadoop配置文件中的参数配置参数总结、hadoop集群测试,安装过程中的常见错误
25. 集群搭建 25.1 HADOOP集群搭建 25.1.1集群简介 HADOOP集群具体来说包含两个集群:HDFS集群和YARN集群,两者逻辑上分离,但物理上常在一起 HDFS集群: 负责海量数据 ...
hadoop编译java,Hadoop源码编译(2.4.1)
背景 Hadoop 2.4.1预编译版本自带的libhadoop.so是在32位机器上编译上,导致在64位OS上运行时,总出现如下告警: WARN util.NativeCodeLoader: Una ...
大数据之-Hadoop源码编译_源码编译的意义---大数据之hadoop工作笔记0044
然后我们再来看一下hadoop的源码编译, 先看看,可以去官网去下载,hadoop-2.7.2.tar.gz的源码去下载了以后我们需要去编译,源码,为什么? 比如:上面这个源码是32位的,那么如果我 ...
Spark入门教程（二）Spark2.2源码编译及安装配置
Spark2.2源码下载点此进入官网下载地址: 下载Spark spark官网下载截图 //下载好后用传输工具传输到我们的linux上并解压缩 tar -zxvf spark-2.2.0.tgz 源 ...
Apache Ranger源码编译及使用
Ranger源码编译.使用手册 1 Ranger简介 Apache Ranger提供一个集中式安全管理框架,它可以对Hadoop生态的组件如Hive,Hbase进行细粒度的数据访问控制.通过操作Ran ...
Flume 1.7 源码分析（一）源码编译
Flume 1.7 源码分析(一)源码编译 Flume 1.7 源码分析(二)整体架构 Flume 1.7 源码分析(三)程序入口 1 说明 Flume是Cloudera提供的一个高可用的,高可靠的, ...
ambari 2.5.0源码编译安装
参考:https://www.ibm.com/developerworks/cn/opensource/os-cn-bigdata-ambari/index.html Ambari 是什么 Ambar ...
Ubuntu 15.04 安装TensorFlow（源码编译）及测试梵高作画
介绍Google的TensorFlow机器学习开源库,在UbuntuKylin上的安装和和源码编译. 原始官方文档参见:http://www.tensorflow.org. 本电脑配置如下: 3.19 ...
Spark 3.0 发布了，代码拉过来，打个包，跑起来！| 附源码编译
作者 | 敏叔V587 责编 | 徐威龙封图| CSDN 下载于视觉中国 Spark3.0已经发布有一阵子了,官方发布了预览版,带来了一大波更新,对于我们程序员来说,首先当然是代码拉过来,打个包,跑 ...

hadoop-2.5.1 源码编译

hadoop-2.5.1 源码编译相关推荐

最新文章

热门文章