Linux下部署Hadoop伪分布模式
Hadoop版本为1.2.1
Distribution为Fedora19并使用hadoop账号安装
第一步:配置ssh本地登录证书(虽然为伪分布模式,Hadoop依然会使用SSH进行通信)
[hadoop@promote ~]$ which ssh
/usr/bin/ssh
[hadoop@promote ~]$ which ssh-keygen
/usr/bin/ssh-keygen
[hadoop@promote ~]$ which sshd
/usr/sbin/sshd
[hadoop@promote ~]$ ssh-keygen -t rsa
然后一路回车
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Created directory '/home/hadoop/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Passphrases do not match. Try again.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
2f:a9:60:c7:dc:38:8f:c7:bb:70:de:d4:39:c3:39:87 hadoop@promote.cache-dns.local
The key's randomart image is:
+--[ RSA 2048]----+
| |
| |
| |
| |
| S |
| o o o o + |
| o B.= o E . |
| . o Oo+ = |
| o.=o. |
+-----------------+
最终将在/home/hadoop/.ssh/路径下生成私钥id_rsa和公钥id_rsa.pub
[hadoop@promote .ssh]$ cd /home/hadoop/.ssh/
[hadoop@promote .ssh]$ ls
id_rsa id_rsa.pub
修改sshd服务配置文件:
[hadoop@promote .ssh]$ su root
密码:
[root@promote .ssh]# vi /etc/ssh/sshd_config
启用RSA加密算法验证
RSAAuthentication yes
PubkeyAuthentication yes# The default is to check both .ssh/authorized_keys and .ssh/authorized_keys2
# but this is overridden so installations will only check .ssh/authorized_keys
AuthorizedKeysFile .ssh/authorized_keys
保存并退出,然后重启sshd服务
[root@promote .ssh]# service sshd restart
Redirecting to /bin/systemctl restart sshd.service
然后切换回hadoop用户,将ssh证书公钥拷贝至/home/hadoop/.ssh/authorized_keys文件中
[root@promote .ssh]# su hadoop
[hadoop@promote .ssh]$ cat id_rsa.pub >> authorized_keys
修改~/.ssh/authorized_keys文件的权限为644,~/.ssh文件夹的权限为700,/home/hadoop文件夹的权限为700(权限正确是成功认证的先决条件)
[hadoop@promote .ssh]$ chmod 644 authorized_keys
[hadoop@promote .ssh]$ ssh 192.168.211.129
The authenticity of host 192.168.211.129(192.168.211.129)' can't be established.
RSA key fingerprint is 25:1f:be:72:7b:83:8e:c7:96:b6:71:35:fc:5d:2e:7d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.211.129' (RSA) to the list of known hosts.
Last login: Thu Feb 13 23:42:43 2014
第一次登陆将会将证书内容保存在/home/hadoop/.ssh/known_hosts文件中,以后再次登陆将不需要输入密码
[hadoop@promote .ssh]$ ssh 192.168.211.129
Last login: Thu Feb 13 23:46:04 2014 from 192.168.211.129
至此ssh证书部分配置完成
第二步:安装JDK
[hadoop@promote ~]$ java -version
java version "1.7.0_25"
OpenJDK Runtime Environment (fedora-2.3.10.3.fc19-i386)
OpenJDK Client VM (build 23.7-b01, mixed mode)
将OpenJDK换为Oracle的Java SE
[hadoop@promote .ssh]$ cd ~
[hadoop@promote ~]$ uname -i
i386
在Oracle的官网下载jdk-6u45-linux-i586.bin后上传至服务器,赋予权限并进行安装,最后删除安装包
[hadoop@promote ~]$ chmod u+x jdk-6u45-linux-i586.bin
[hadoop@promote ~]$ ./jdk-6u45-linux-i586.bin
[hadoop@promote ~]$ rm -rf jdk-6u45-linux-i586.bin
出现以下结果说明JDK成功安装:
[hadoop@promote ~]$ /home/hadoop/jdk1.6.0_45/bin/java -version
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
Java HotSpot(TM) Client VM (build 20.45-b01, mixed mode, sharing)
第三步:安装Hadoop
在Hadoop官网下载hadoop-1.2.1.tar.gz并上传至服务器/home/hadoop路径下
[hadoop@promote ~]$ tar -xzf hadoop-1.2.1.tar.gz
[hadoop@promote ~]$ rm -rf hadoop-1.2.1.tar.gz
[hadoop@promote ~]$ cd hadoop-1.2.1/conf/
[hadoop@promote conf]$ vi hadoop-env.sh
将JAVA_HOME指向第二步安装的JDK所在目录
# The java implementation to use. Required.
export JAVA_HOME=/home/hadoop/jdk1.6.0_45
保存并退出
[hadoop@promote ~]$ vi ~/.bash_profile
环境变量PATH后接上Hadoop和JDK的bin目录
......
PATH=$PATH:$HOME/.local/bin:$HOME/bin:/home/hadoop/hadoop-1.2.1/bin:/home/hadoop/jdk1.6.0_45/bin
export PATH
保存并退出,退出登录并重新使用hadoop账号登录,如果出现如下结果,说明环境变量PATH设置成功
[hadoop@promote ~]$ echo $PATH
/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/hadoop/.local/bin:
/home/hadoop/bin:/home/hadoop/hadoop-1.2.1/bin:/home/hadoop/jdk1.6.0_45/bin
第四步:修改Hadoop配置文件
修改core-site.xml(使用IP地址而不是主机名或localhost的好处是不需要修改/etc/hosts):
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?><configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://192.168.211.129:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/hadooptmp</value>
</property>
</configuration>
修改mapred-site.xml:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?><configuration>
<property>
<name>mapred.job.tracker</name>
<value>192.168.211.129:9001</value>
</property>
</configuration>
修改hdfs-site.xml:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?><configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
master中指定的SNN节点和slaves中指定的从节点位置均为本地
[hadoop@promote conf]$ cat masters
192.168.211.129
[hadoop@promote conf]$ cat slaves
192.168.211.129
第五步:初始化HDFS文件系统
特别要注意的是,Hadoop并不识别带“_”的主机名,所以如果你的主机名带有“_”,一定要进行修改,修改方式参照http://blog.csdn.net/a19881029/article/details/20485079
[hadoop@fedora ~]$ hadoop namenode -format
14/03/04 22:13:41 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = fedora/127.0.0.1
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.2.1
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2
-r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG: java = 1.6.0_45
************************************************************/
14/03/04 22:13:42 INFO util.GSet: Computing capacity for map BlocksMap
14/03/04 22:13:42 INFO util.GSet: VM type = 32-bit
14/03/04 22:13:42 INFO util.GSet: 2.0% max memory = 1013645312
14/03/04 22:13:42 INFO util.GSet: capacity = 2^22 = 4194304 entries
14/03/04 22:13:42 INFO util.GSet: recommended=4194304, actual=4194304
14/03/04 22:13:42 INFO namenode.FSNamesystem: fsOwner=hadoop
14/03/04 22:13:42 INFO namenode.FSNamesystem: supergroup=supergroup
14/03/04 22:13:42 INFO namenode.FSNamesystem: isPermissionEnabled=true
14/03/04 22:13:42 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
14/03/04 22:13:42 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
14/03/04 22:13:42 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0
14/03/04 22:13:42 INFO namenode.NameNode: Caching file names occuring more than 10 times
14/03/04 22:13:53 INFO common.Storage: Image file /tmp/hadoop-hadoop/dfs/name/ current/fsimage of size 112 bytes saved in 0 seconds.
14/03/04 22:13:53 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/tmp/hadoop-hadoop/dfs/name/current/edits
14/03/04 22:13:53 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/tmp/hadoop-hadoop/dfs/name/current/edits
14/03/04 22:13:53 INFO common.Storage: Storage directory /tmp/hadoop-hadoop/dfs/name has been successfully formatted.
14/03/04 22:13:53 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at fedora/127.0.0.1
************************************************************/
第六步:启动Hadoop
[hadoop@fedora logs]$ start-all.sh
starting namenode, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-namenode-fedora.out
localhost: starting datanode, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-datanode-fedora.out
localhost: starting secondarynamenode, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-secondarynamenode-fedora.out
starting jobtracker, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-jobtracker-fedora.out
localhost: starting tasktracker, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-tasktracker-fedora.out
hadoop@fedora logs]$ jps
2099 SecondaryNameNode
2184 JobTracker
1976 DataNode
2365 Jps
1877 NameNode
2289 TaskTracker
可以看到所有Hadoop守护进程均已启动
再看看日志文件中有没有报错,如果没有报错,说明Hadoop已经启动成功了
[hadoop@fedora hadoop]$ hadoop dfsadmin -report
Configured Capacity: 39474135040 (36.76 GB)
Present Capacity: 33661652992 (31.35 GB)
DFS Remaining: 33661612032 (31.35 GB)
DFS Used: 40960 (40 KB)
DFS Used%: 0%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0-------------------------------------------------
Datanodes available: 1 (1 total, 0 dead)Name: 192.168.211.129:50010
Decommission Status : Normal
Configured Capacity: 39474135040 (36.76 GB)
DFS Used: 40960 (40 KB)
Non DFS Used: 5812482048 (5.41 GB)
DFS Remaining: 33661612032(31.35 GB)
DFS Used%: 0%
DFS Remaining%: 85.28%
Last contact: Thu Mar 06 09:48:17 CST 2014
尝试执行Map/Reduce任务时也是没问题的
转载于:https://www.cnblogs.com/sean-zou/p/3709998.html
Linux下部署Hadoop伪分布模式相关推荐
- Linux环境搭建Hadoop伪分布模式
Hadoop有三种分布模式:单机模式.伪分布.全分布模式,相比于其他两种,伪分布是最适合初学者开发学习使用的,可以了解Hadoop的运行原理,是最好的选择.接下来,就开始部署环境. 首先要安装好Lin ...
- 大数据2 Hadoop伪分布模式配置部署
为什么80%的码农都做不了架构师?>>> Hadoop伪分布模式配置部署 一.实验环境说明 注意:本实验需要按照上一节单机模式部署后继续进行操作 1. 环境登录 无需密码自动登 ...
- linux下搭建hadoop环境步骤分享
linux下搭建hadoop环境步骤,需要的朋友可以参考下 1.下载hadoop包 wget http://apache.freelamp.com/hadoop/core/stable/hadoop- ...
- Hadoop伪分布模式安装
Hadoop伪分布模式安装 Hadoop的运行模式分为3种: 本地运行模式,伪分布运行模式,完全分布运行模式. (1)本地模式(local mode) 这种运行模式在一台单机上运行,没有HDFS分布式 ...
- hadoop 伪分布模式
学习hadoop有一个星期了,先是看了看书,hadoop权威指南,很不错的入门级教材. 书的内容偏向于基础原理,多读几遍会收获很多. 本人菜鸟一只,为了把自己的学习心得分享给大家,就在此开始记录一下, ...
- Linux下部署多个Tomcat
Linux下部署多个Tomcat 1.环境: 1.1.Centos 5.0 1.2.apache-tomcat-6.0.18 2.需要解决一下几个问题 2.1.不同的tomcat启动和关闭监听不同的端 ...
- linux下部署tomcat没起成功,linux下部署tomcat 上线jpress博客系统
tomcat Tomcat服务器是一个免费的开放源代码的Web应用服务器,属于轻量级应用服务器. tomcat有三个端口 开始部署 安装tomcat 第一步:下载tomcat 安装包 tomcat官网 ...
- Linux 下部署 TFTP 服务
1. TFTP服务简介 TFTP是 Trivial File Transfer Protocol 的缩写,即简单文件传输协议,是一个基于 UDP 协议实现的用于在客户机和服务器之间进行简单文件传输的协 ...
- Linux 部署Hadoop伪分布式集群教程
首先:我们需要下载一些关于Hadoop伪分布式集群需要的工具与tar包 链接: https://pan.baidu.com/s/1oUw1jDCxfghWsnaWauSHKg 提取码:6s5a 接下来 ...
最新文章
- 【Java源码分析】集合框架-Collections工具类-Arrays工具类
- Java内存模型、volatile、原子性、可见性、有序性、happens-before原则
- linux系统编程:自己动手写一个who命令
- 短视频自研还是选择第三方?技术选型前必看的自检清单
- 解决quartz的job无法注入spring对象
- 开源一周岁,MindSpore新特性巨量来袭
- 生成高清缩略图; 添加图片、文字水印; 图片、文字水印透明
- c#利用反射Assembly 对类和成员属性进行操作
- 钉钉微应用怎么进入_海目星激光张荣:激光焊接在锂电池生产应用中越来越多...
- 手机屏幕坏了怎么把数据导出来_手机屏幕失灵怎么办 手机屏幕失灵解决办法【详解】...
- EXCEL表格所有的快捷键
- DBA 小记 — 分库分表、主从、读写分离
- JSch连接SFTP Exception:Algorithm negotiation fail问题解决
- 打开matlab只在任务栏闪一下然后就没反应了解决办法(亲试有效)
- c语言编程学生活动安排表,C语言作业安排表(18学时-周学时2...).doc
- php开发公众号 token验证失败 其中一个原因
- windows下 利用CoreAudio,Libmad,和3D环绕音效原理实现的MP3播放器
- Linux 进程信号
- YUI介绍以及快速入门 Yahoo的JS框架
- Vue项目搭建常用配置文件!
热门文章
- python电影推荐系统的设计与实现_一种电影推荐系统的设计与实现
- c#怎么拟合函数得到参数_吴恩达老师课程笔记系列第32节 -正则化之代价函数(2)...
- 基于Echarts+HTML5可视化数据大屏展示—大数据管理平台中心
- 接口自动化测试_Python自动化测试学习路线之接口自动化测试「模块四」
- oracle字段公式怎么执行,Oracle 在存储过程或函数中执行字符串sql PDF 下载
- 西昌学院计算机,西昌学院
- linux read recv,read()/ recv()成功,但缓冲区未更改且为空
- java查询数据库的方式_Java连接各种数据库方式速查表
- 如何用记事本编写运行java程序?
- SpringBoot项目的 pom.xml第一行报错