重新编译hadoop-2.7.2-src的native以支持Snappy解压压缩库
1. 下载所需要的源码包 snappy-1.1.1.tar.gz、protobuf-2.5.0.tar.gz、hadoop-2.7.2-src.tar.gz
[root@sht-sgmhadoopnn-01 ~]# cd /hadoop
[root@sht-sgmhadoopnn-01 hadoop]# mkdir native_snappy
[root@sht-sgmhadoopnn-01 hadoop]# cd native_snappy
[root@sht-sgmhadoopnn-01 native_snappy]# wget http://pkgs.fedoraproject.org/repo/pkgs/snappy/snappy-1.1.1.tar.gz/8887e3b7253b22a31f5486bca3cbc1c2/snappy-1.1.1.tar.gz
--2016-06-21 15:23:35-- http://pkgs.fedoraproject.org/repo/pkgs/snappy/snappy-1.1.1.tar.gz/8887e3b7253b22a31f5486bca3cbc1c2/snappy-1.1.1.tar.gz
Resolving pkgs.fedoraproject.org... 209.132.181.4
Connecting to pkgs.fedoraproject.org|209.132.181.4|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1777992 (1.7M) [application/x-gzip]
Saving to: “snappy-1.1.1.tar.gz”
100%[==========================================================================================================================>] 1,777,992 56.2K/s in 32s
2016-06-21 15:24:08 (54.8 KB/s) - “snappy-1.1.1.tar.gz” saved [1777992/1777992]
You have mail in /var/spool/mail/root
[root@sht-sgmhadoopnn-01 native_snappy]#
###百度云盘下载,然后rz上传
http://pan.baidu.com/share/link?uk=3694838921&shareid=1608967052&third=0&adapt=pc&fr=ftw
[root@sht-sgmhadoopnn-01 native_snappy]# rz
rz waiting to receive.
Starting zmodem transfer. Press Ctrl+C to cancel.
Transferring protobuf-2.5.0.tar.gz...
100% 2345 KB 2345 KB/sec 00:00:01 0 Errors
You have mail in /var/spool/mail/root
[root@sht-sgmhadoopnn-01 native_snappy]#
[root@sht-sgmhadoopnn-01 native_snappy]# wget http://apache.01link.hk/hadoop/common/hadoop-2.7.2/hadoop-2.7.2-src.tar.gz
--2016-06-21 15:37:46-- http://apache.01link.hk/hadoop/common/hadoop-2.7.2/hadoop-2.7.2-src.tar.gz
Resolving apache.01link.hk... 101.78.134.82
Connecting to apache.01link.hk|101.78.134.82|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 18290860 (17M) [application/x-gzip]
Saving to: “hadoop-2.7.2-src.tar.gz”
100%[==========================================================================================================================>] 18,290,860 248K/s in 54s
2016-06-21 15:38:40 (330 KB/s) - “hadoop-2.7.2-src.tar.gz” saved [18290860/18290860]
You have mail in /var/spool/mail/root
[root@sht-sgmhadoopnn-01 native_snappy]#
[root@sht-sgmhadoopnn-01 native_snappy]# ls -l
total 21952
-rw-r--r-- 1 root root 18290860 Jan 26 09:29 hadoop-2.7.2-src.tar.gz
-rw-r--r-- 1 root root 2401901 Jun 21 15:28 protobuf-2.5.0.tar.gz
-rw-r--r-- 1 root root 1777992 Feb 24 2014 snappy-1.1.1.tar.gz
[root@sht-sgmhadoopnn-01 native_snappy]#
2. 准备编译环境
[root@host /]# yum -y install svn
Loaded plugins: fastestmirror, refresh-packagekit, security
Loading mirror speeds from cached hostfile
Setting up Install Process
No package svn available.
[root@sht-sgmhadoopnn-01 yum.repos.d]# lsb_release -a
LSB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch
Distributor ID: CentOS
Description: CentOS release 6.5 (Final)
Release: 6.5
Codename: Final
You have mail in /var/spool/mail/root
[root@sht-sgmhadoopnn-01 yum.repos.d]#
###Centos yum 安装软件 No package XXX available. http://blog.itpub.net/30089851/viewspace-2120628/
[root@sht-sgmhadoopnn-01 yum.repos.d]# yum install svn
[root@sht-sgmhadoopnn-01 yum.repos.d]# yum install autoconf automake libtool cmake
[root@sht-sgmhadoopnn-01 yum.repos.d]# yum install ncurses-devel
[root@sht-sgmhadoopnn-01 yum.repos.d]# yum install openssl-devel
[root@sht-sgmhadoopnn-01 yum.repos.d]# yum install gcc*
3. 编译安装snappy
# 用root用户执行以下命令
[root@sht-sgmhadoopnn-01 native_snappy]#tar -zxvf snappy-1.1.1.tar.gz
[root@sht-sgmhadoopnn-01 native_snappy]# cd snappy-1.1.1/
[root@sht-sgmhadoopnn-01 snappy-1.1.1]# ./configure
[root@sht-sgmhadoopnn-01 snappy-1.1.1]# make
[root@sht-sgmhadoopnn-01 snappy-1.1.1]# make install
4. 查看snappy库文件
[root@sht-sgmhadoopnn-01 snappy-1.1.1]# ls -lh /usr/local/lib |grep snappy
-rw-r--r-- 1 root root 229K Jun 21 15:46 libsnappy.a
-rwxr-xr-x 1 root root 953 Jun 21 15:46 libsnappy.la
lrwxrwxrwx 1 root root 18 Jun 21 15:46 libsnappy.so -> libsnappy.so.1.2.0
lrwxrwxrwx 1 root root 18 Jun 21 15:46 libsnappy.so.1 -> libsnappy.so.1.2.0
-rwxr-xr-x 1 root root 145K Jun 21 15:46 libsnappy.so.1.2.0
[root@sht-sgmhadoopnn-01 snappy-1.1.1]#
5. 编译安装protobuf
# 用root用户执行以下命令
[root@hadoop-01 snappy-1.1.1]# cd ../
You have mail in /var/spool/mail/root
[root@hadoop-01 native_snappy]# tar -zxvf protobuf-2.5.0.tar.gz
[root@hadoop-01 native_snappy]# cd protobuf-2.5.0/
[root@hadoop-01 protobuf-2.5.0]# ./configure
[root@hadoop-01 protobuf-2.5.0]# make
[root@hadoop-01 protobuf-2.5.0]# make install
# 查看protobuf版本以测试是否安装成功
[root@hadoop-01 protobuf-2.5.0]# protoc --version
protoc: error while loading shared libraries: libprotobuf.so.8: cannot open shared object file: No such file or directory
[root@hadoop-01 protobuf-2.5.0]# export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
You have mail in /var/spool/mail/root
[root@hadoop-01 protobuf-2.5.0]# protoc --version
libprotoc 2.5.0
[root@hadoop-01 protobuf-2.5.0]#
6. 编译hadoop native
[root@hadoop-01 native_snappy]# tar -zxvf hadoop-2.7.2-src.tar.gz
[root@hadoop-01 native_snappy]# cd hadoop-2.7.2-src/
[root@hadoop-01 hadoop-2.7.2-src]# mvn clean package -DskipTests -Pdist,native -Dtar -Dsnappy.lib=/usr/local/lib -Dbundle.snappy
.........
.........
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 20:58 min
[INFO] Finished at: 2016-06-21T19:12:51+08:00
[INFO] Final Memory: 179M/521M
[INFO] ------------------------------------------------------------------------
[root@hadoop-01 hadoop-2.7.2-src]#
7.执行成功后，hadoop-dist/target/hadoop-2.7.2.tar.gz即为新生成的二进制安装包。
[root@hadoop-01 hadoop-2.7.2-src]# ll hadoop-dist/target/hadoop-2.7.2.tar.gz
-rw-r--r-- 1 root root 195117479 Jun 21 19:12 hadoop-dist/target/hadoop-2.7.2.tar.gz
[root@hadoop-01 hadoop-2.7.2-src]#
8.重命名为hadoop-2.7.2-snappy.tar.gz
[root@sht-sgmhadoopnn-01 hadoop-2.7.2-src]# cp hadoop-dist/target/hadoop-2.7.2.tar.gz /tmp/hadoop-2.7.2-snappy.tar.gz
重新编译hadoop-2.7.2-src的native以支持Snappy解压压缩库相关推荐
- 【Android 安全】DEX 加密 ( Java 工具开发 | 加密解密算法 API | 编译代理 Application 依赖库 | 解压依赖库 aar 文件 )
文章目录 一.加密解密算法 API 二.编译代理 Application 依赖库 三.解压代理 Application 依赖库 aar 文件 参考博客 : [Android 安全]DEX 加密 ( 常 ...
- android 自编译 img,android源码编译生成ramdisk.img和system.img解压和使用（基于海天雄A9+android2.3.3）...
bin.tar.gz里有simg2img和make_ext4fs两个工具. 一.解压和打包system.img ./simg2img system.img system.img.ext4 生成ext4 ...
- eclipse下编译hadoop源代码(转)
hadoop是一个分布式存储和分布式计算的框架.在日常使用hadoop时,我们会发现hadoop不能完全满足我们的需要,我们可能需要修改hadoop的源代码并重新编译.打包. 下面将详细描述如何从sv ...
- 编译Hadoop源码
背景: 在阅读hadoop源代码过程中会发现部分类或者函数无法找到,这是由于Hadoop2.0使用了Protocol Buffers定义了RPC协议, 而这些Protocol Buffers文件在Ma ...
- 编译hadoop的eclipse插件hadoop-eclipse-plugin-1.2.1.jar
1:下载后hadoop-1.1.2.tar.gz文件,里面包含源代码,并解压到E:\hadoop\hadoop-1.2.1 2:在eclipse导入工程,目录选择:E:\hadoop\hadoop-1 ...
- 通过源码的方式编译hadoop的安装文件
Hadoop2.4.0 重新编译 64 位本地库 原创作者:大鹏鸟 时间:2014-07-28 环境:虚拟机 VirtualBox,操作系统 64 位 CentOS 6.4 下载重新编译需要的软件 ...
- linux 编译hadoop,linux centos 安装编译hadoop2.7.1
因为Apache hadoop 只提供了32位的包,所以需要自己编译64位的包,下面是编译步骤 所需要的包 6:java 1.7+ 下面是具体安装: 1:Java安装 安装之前请卸载系统自带的jdk, ...
- 编译hadoop源码遇到问题 及时解决
最近在研究hadoop,在用maven编译源码的过程中出现了个错误,一直编译过不去: [INFO] -------------------------------------------------- ...
- 使用Ant编译Hadoop eclipse插件
下载Hadoop的release, http://labs.renren.com/apache-mirror/hadoop/common/ 下载ant,http://labs.renren.com/a ...
最新文章
- 学python是什么专业-小白学python是自学好，还是专业老师培训好?
- python语言中文社区-扯个淡：用Python实现中文编程
- css动画-animation各个属性详解(通俗易懂)
- h5微信本地调试 vue_Vue 移动端微信内H5调起支付（利用js sdk）
- GDCM：gdcm::PixelFormat的测试程序
- 计算机网络于20世纪60年代萌芽,计算机网络的发展经过哪几个阶段？
- idea创建web项目的几种方式（没内容，自用）
- Qt4_子类化QTableWidget
- winx修改计算机用户名,怎么修改我的电脑用户名win10
- OpenCV精进之路（零）：core组件——Mat和IplImage访问像素的方法总结
- php5.3 appache phpstudy win7win8win10下 运行速度慢
- C# 用itextsharp把Html转PDF
- 台式计算机可以放到笔记本立马,怎么把台式电脑投影到笔记本电脑
- LeetCode198打家劫社（线性动态规划）
- 融会贯通，并行不悖丨2022年8月《中国数据库行业分析报告》发布！
- 联想YOGA 3笔记本如何U盘启动
- Quantopian自学笔记01
- 全国电信/联通DNS服务器地址列表
- 关注民生民情——华北水利水电大学“情艺”国情社情调查
- Sql Hacks 阅读感悟——数值处理