Windows下编译 Hadoop
Windows下编译 Hadoop-2.9.2
系统环境
系统: Windows 10 10.0_x64
maven: Apache Maven 3.6.0
jdk: jdk_1.8.0_201
ProtocolBuffer: portoc-2.5.0
zlib: 1.2.3-lib
OpenSSL: 1_0_2r
cmake: 3.14.3-win64-x64
Cygwin: 2.897_x86_64
Visual Studio: Visual Studio 2010 Professional
hadoop: hadoop-2.9.2
复制代码
Hadoop源码包你们的的编译环境要求
Building on Windows----------------------------------------------------------------------------------
Requirements:* Windows System
* JDK 1.7 or 1.8
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer
* Windows SDK 7.1 or Visual Studio 2010 Professional
* Windows SDK 8.1 (if building CPU rate control for the container executor)
* zlib headers (if building native code bindings for zlib)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)
* Unix command-line tools from GnuWin32: sh, mkdir, rm, cp, tar, gzip. Thesetools must be present on your PATH.
* Python ( for generation of docs using 'mvn site')Unix command-line tools are also included with the Windows Git package which
can be downloaded from http://git-scm.com/downloadsIf using Visual Studio, it must be Visual Studio 2010 Professional (not 2012).
Do not use Visual Studio Express. It does not support compiling for 64-bit,
which is problematic if running a 64-bit system. The Windows SDK 7.1 is free to
download here:http://www.microsoft.com/en-us/download/details.aspx?id=8279The Windows SDK 8.1 is available to download at:http://msdn.microsoft.com/en-us/windows/bg162891.aspxCygwin is neither required nor supported.复制代码
编译必须要设置的环境变量
1.JAVA_HOME必须要设置这个环境变量指向jdk的安装目录
2.Platform这个环境变量也必须设置32位系统为
Platform=Win32
64位系统为Platform=x64
3.ZLIB_HOME为你zlib的安装目录例如
ZLIB_HOME=C:\zlib-1.2.7
编译需要注意的点
1.保证你hadoop源码的路径尽量短,比如放在某个盘的根目录
2.保证你的maven仓库的位置尽量短,比如放在某个盘的根目录(注意设置maven的conf文件)
编译过程中可能出现的问题
- 1.Command execution failed. Cannot run program "msbuild" (in directory "C:\hadoop-2.9.2-src\hadoop-common-project\hadoop-common")
解决方法:将C:\Windows\Microsoft.NET\Framework\v4.0.30319放入PATH环境变量
- 2.(compile-ms-winutils) on project hadoop-common: Command execution failed. Process exited with an error: 1 (Exit value: 1) -> [Help 1]
解决方法:这个问题是由于Visual Studio没有安装或安装的版本不对,换成对应的版本就好了
- 3.(compile-ms-native-dll) on project hadoop-common: Command execution failed. Process exited with an error: 1(Exit value: 1) -> [Help 1]
解决方法:这个问题是由于zlib环境的问题,可能缺少ZLIB_HOME或者你下载的zlib的版本不正确或是你的zlib环境变量你们没有包含zlib.h或缺少unistd.h和getopt.h这两个文件可到GitHub上下载这两个文件.
- 4.(dist) on project hadoop-kms: An Ant BuildException has occured: exec returned: 2
解决方法:这个问题是由于在编译hadoop-kms这个模块的时候下载Tomcat是由于网络问题出现网络断开连接重新编译也会出现这个问题,可以删除源码重新下载对应的源码进行编译(简单粗暴)
编译
下载hadoop源码选择对应的版本,解压源码后放在某个盘的根目录,进入源码的根目录执行mvn package -Pdist,native-win -DskipTests -Dtar
后maven就开始去远程仓库下载需要的依赖这个过程可能需要很久跟带宽速度和墙有很大关系,过程中可能会出现各种问题但是自己都完美解决了,hadoop2.x的版本中我编译了hadoop2.7.7、hadoop-2.8.8、hadoop-2.9.2这三个版本都是编译成功的,但是在编译hadoop3.x版本就会失败,可能3.x对环境的需求相对于2.x有比较大的改变,后续再研究。。。
经过漫长的等待终于出现了久违的画面
[INFO] Reactor Summary for Apache Hadoop Main 2.9.2:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 0.928 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 0.579 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 0.780 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 2.413 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.278 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 1.491 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 3.365 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 3.223 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [ 4.977 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 2.304 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [01:08 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 3.788 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 9.379 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.052 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 23.895 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [ 48.088 s]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [ 25.830 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 25.398 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [ 10.032 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 2.641 s]
[INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [ 13.092 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.053 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [ 0.054 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [ 19.308 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [ 53.206 s]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [ 3.731 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [ 0.056 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 16.246 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 20.362 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [ 2.566 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [ 6.348 s]
[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [ 3.742 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 25.768 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [ 1.535 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [ 6.465 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [ 3.396 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [ 3.146 s]
[INFO] Apache Hadoop YARN Router .......................... SUCCESS [ 3.784 s]
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [ 4.994 s]
[INFO] Apache Hadoop YARN Timeline Service HBase tests .... SUCCESS [ 1.584 s]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [ 0.336 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [ 2.052 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [ 1.535 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [ 0.052 s]
[INFO] Apache Hadoop YARN UI .............................. SUCCESS [ 0.050 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [ 7.322 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [ 0.128 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 32.088 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ 19.070 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [ 2.889 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [ 8.415 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [ 5.543 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [ 4.486 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [ 1.283 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 4.521 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [ 2.765 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 3.277 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 3.957 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [ 1.462 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [ 1.628 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 14.629 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 2.915 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [ 1.689 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [ 1.238 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [ 1.851 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [ 0.052 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 3.016 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 6.852 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 4.349 s]
[INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [ 2.039 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [ 6.595 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 0.961 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 7.219 s]
[INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [ 3.102 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ 5.460 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 14.486 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.075 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 51.638 s]
[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [ 0.581 s]
[INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [ 0.055 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 11:01 min
[INFO] Finished at: 2019-05-08T11:42:52+08:00
[INFO] ------------------------------------------------------------------------C:\Users\Andy\Downloads\hadoop-2.9.2-src>
复制代码
大功告成编译后的hadoop放在hadoop-2.9.2-src\hadoop-dist\target
目录下。
转载于:https://juejin.im/post/5cd2a6606fb9a03238107ece
Windows下编译 Hadoop相关推荐
- Windows下编译Chrome V8
主要还是参考google的官方文档: How to Download and Build V8 Building on Windows 同时也参考了一些其它的中文博客: 脚本引擎小pk:SpiderM ...
- Windows下编译tensorflow-gpu教程
这两个也要看: https://zhuanlan.zhihu.com/p/29029860 https://zhuanlan.zhihu.com/p/34942873 Windows下编译tensor ...
- Windows下编译TensorFlow1.3 C++ library及创建一个简单的TensorFlow C++程序
参考:https://www.cnblogs.com/jliangqiu2016/p/7642471.html Windows下编译TensorFlow1.3 C++ library及创建一个简单的T ...
- windows下编译支持https的curl
先编译好openssl,过程详见:<windows下编译openssl> 编译好zlib,过程详见<windows下编译zlib> 1. 下载curl 官网:https://c ...
- 在Windows下编译FFmpeg详细说明
在Windows下编译FFmpeg详细说明 MinGW:一个可自由使用和自由发布的Windows特定头文件和使用GNC工具集导入库的集合,允许你生成本地的Windows程序而不需要第三方C运行时 Mi ...
- linux下find查找带有指定权限的文件(windows下编译的源代码文件)
find -type f -perm -o=x 查找用户在windows下编译的源代码文件 转载于:https://blog.51cto.com/axlrose/1357610
- lua windows下编译
从Lua5.1开始官方给出的文件只有源代码和makefile文件了,官网给出的bulid方式也是在linux平台,如果只是想找个库使用下可以到这里来下载:http://joedf.ahkscript. ...
- 在windows下编译FFMPEG-最新2009版本
转]在windows下编译FFMPEG-最新2009版本 2010-11-17 18:50 大家可以看到,此篇之前有很多个版本的"在windows下编译FFMPEG",那些都是我在 ...
- fastdfs windows部署_Go在windows下编译Linux可执行文件
欢迎关注我的头条号:Wooola,专注于Java.Golang.微服务架构,致力于每天分享原创文章.快乐编码和开源技术. 前言 最近楼主做了一个滑块验证码登录功能,但有个问题,悲观估计一天大约会产生两 ...
最新文章
- 在Windows应用程序中使用正则表达式
- Windows系统软件推荐
- php的floorval函数,php 取整函数(floor,ceil,round,intval)_PHP教程
- gravity与Layout_gravity的区别
- 帆软报表在已经搭载服务器上开发_史上最全企业数据产品选型对比(含数仓、报表、BI、中台、数据治理)...
- php获取当前域名、主机、URL、端口、参数、网址、路径、代理等
- OpenCV-Python实战(番外篇)——OpenCV、NumPy和Matplotlib直方图比较
- FMEA-MSR 步骤四:失效分析
- 大众点评有多少用户2019_据说2019年度大众点评排名第一
- PS怎样去掉图片上的文字
- 计算机网络技术该考什么证,计算机网络工程师证书
- KY-RTI分布仿真技术:第一章 简介
- ndis拨号软件 linux,[4G模块]华为ME909S-821 NDIS拨号指令流程
- Java毕业设计-医疗信息管理系统
- 阅读小结:MSR:From Captions to Visual Concepts and Back
- Kotlin上的反应式流-SharedFlow和StateFlow
- [简单逆向]某直播APP 收费直播链接获取-AES解密
- 任务栏出现两个重复图标的解决办法
- Android高版本使用DocumentFile读写外置存储的问题
- Python全栈工程师(4:函数)
热门文章
- go split 正则_WEGO使用—华大基因在线GO功能注释
- q函数表格怎么看_会计表格函数玩不会?超实用会计表格函数公式大全,财务人都在用...
- pythonturtle画房子_用python的turtle模块实现给女票画个小心心
- android本地xml文件怎么打开,android 打开本地文件
- http php mysql_apache+php+mysql
- linux批量用户创建,linux 批量用户的创建
- python数列求和-加强版_ES6深入浅出-3 三个点运算 新版字符串-1.函数与对象的语法糖...
- matlab安装教程补充
- 【 FPGA 】UltraFast设计方法学:时序收敛之时序约束基本准则
- 10-GLBP Weighting //2.1.5(GNS3版本,后面都是如此注明)