Hadoop伪集群环境搭建

结合网上多份文档，不断反复的修正hadoop启动和运行过程中出现的问题，终于把Hadoop2.5.2伪分布式安装起来，跑通了wordcount例子。Hadoop的安装复杂性的体现之一是，Hadoop的安装文档非常多，但是能一个文档走下来的少之又少，尤其是Hadoop不同版本的配置差异非常的大。Hadoop2.5.2于前两天发布，但是它的配置跟2.5.0，2.5.1没有分别。

系统环境: Ubuntu 12.04 LTS x86_32

一、创建用户组和用户

创建用户组，为系统添加一个用户组hadoop

Java代码

sudo addgroup hadoop

创建用户，为系统添加一个用户hadoop

Java代码

useradd -g hadoop hadoop

使用hadoop用户登陆

Java代码

su hadoop

二、使用SSH免密码登录

执行如下命令，生成ssh的密钥，创建的key的路径是/home/hadoop/.ssh/id_rsa

Java代码

ssh-keygen -t rsa -P ""

将 /home/hadoop/.ssh/id_rsa.pub中的内容追加到/home/hadoop/.ssh/authorized_keys中，保存
执行ssh localhost，验证无密码即可登陆

三、禁用IPv6

执行如下命令查看当前IPv6是否禁用，1表示禁用，0表示未禁用，默认是0

Java代码

cat /proc/sys/net/ipv6/conf/all/disable_ipv6

编辑如下文件，添加三行，禁用IPv6

Java代码

sudo vim /etc/sysctl.conf

Java代码

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

重启机器，再次查看IPv6是否禁用

四、安装配置JDK

编辑/etc/profile文件，设置JAVA相关的系统变量

Java代码

export JAVA_HOME=/software/devsoftware/jdk1.7.0_55
export PATH=$JAVA_HOME/bin:$PATH

五、安装配置Hadoop2.5.2

编辑/etc/profile文件，设置Hadoop相关的系统变量

Java代码

export HADOOP_HOME=/home/hadoop/hadoop-2.5.2
export PATH=$HADOOP_HOME/bin:$PATH

执行如下使上面配置的系统变量生效

Java代码

source /etc/profile

将JDK设置到Hadoop的环境脚本/home/hadoop/hadoop-2.5.2/etc/hadoop/hadoop-env.sh中，追加一行

Java代码

export JAVA_HOME=/software/devsoftware/jdk1.7.0_55

六、Hadoop2.5.2配置文件设置

Hadoop2.5.2有四个配置文件需要配置，它们都位于/home/hadoop/hadoop-2.5.2/etc/hadoop目录下。四个文件分别是

core-site.xml
yarn-site.xml
mapred-site.xml
hdfs-site.xml

这写配置文件中有些需要手工创建目录，有些需要根据系统的实际情况，设置hostname，hostname不能是IP或者localhost，需要在/etc/hosts中进行设置。需要补充一点，有几个文档指出，127.0.0.1最好只跟一个hostname(即Hadoop用到的)绑定，把其余的注释掉。这个究竟是否产生影响，没有测，只是按照网上的说法，只保留一个hostname

6.1 core-site.xml配置

Xml代码

<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/data/tmp</value>
<description>A base for other temporary directories.</description>
</property>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hostname:9000</value>
</property>
<property>
<name>hadoop.native.lib</name>
<value>true</value>
<description>Should native hadoop libraries, if present, be used.</description>
</property>
</configuration>

6.2 mapred-site.xml配置

mapred-site.xml文件默认不存在，使用cp命令从mapred-site.xml.template拷贝一份

Java代码

cp mapred-site.xml.template mapred-site.xml

做如下设置，

Xml代码

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>

6.3 yarn-site.xml配置

Xml代码

<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<description>The address of the applications manager interface in the RM.</description>
<name>Yarn.resourcemanager.address</name>
<value>hostname:18040</value>
</property>
<property>
<description>The address of the scheduler interface.</description>
<name>Yarn.resourcemanager.scheduler.address</name>
<value>hostname:18030</value>
</property>
<property>
<description>The address of the RM web application.</description>
<name>Yarn.resourcemanager.webapp.address</name>
<value>hostname:18088</value>
</property>
<property>
<description>The address of the resource tracker interface.</description>
<name>Yarn.resourcemanager.resource-tracker.address</name>
<value>hostname:8025</value>
</property>
</configuration>

6.4 hdfs-site.xml 配置

Xml代码

<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>/home/hadoop/data/hdfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/home/hadoop/data/hdfs/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>

七、Hadoop初始化并启动

格式化Hadoop NameNode

Java代码

hadoop namenode -format

观察日志，如果有输出中包括Storage directory /home/hadoop/data/hdfs/name has been successfully formatted，则表示格式化成功

启动Hadoop

Java代码

/home/hadoop/hadoop-2.5.2/sbin/start-all.sh

使用JDK的jps检查Hadoop状态，如果是如下结果，则表示安装成功

Java代码

10682 DataNode
10463 NameNode
11229 ResourceManager
24647 Jps
11040 SecondaryNameNode
11455 NodeManager

使用netstat -anp|grep java观察Hadoop端口号使用情况

Java代码

tcp 0 0 0.0.0.0:8042 0.0.0.0:* LISTEN 11455/java
tcp 0 0 0.0.0.0:50090 0.0.0.0:* LISTEN 11040/java
tcp 0 0 0.0.0.0:50070 0.0.0.0:* LISTEN 10463/java
tcp 0 0 0.0.0.0:8088 0.0.0.0:* LISTEN 11229/java
tcp 0 0 0.0.0.0:34456 0.0.0.0:* LISTEN 11455/java
tcp 0 0 0.0.0.0:13562 0.0.0.0:* LISTEN 11455/java
tcp 0 0 0.0.0.0:50010 0.0.0.0:* LISTEN 10682/java
tcp 0 0 0.0.0.0:50075 0.0.0.0:* LISTEN 10682/java
tcp 0 0 0.0.0.0:8030 0.0.0.0:* LISTEN 11229/java
tcp 0 0 0.0.0.0:8031 0.0.0.0:* LISTEN 11229/java
tcp 0 0 0.0.0.0:8032 0.0.0.0:* LISTEN 11229/java
tcp 0 0 0.0.0.0:8033 0.0.0.0:* LISTEN 11229/java
tcp 0 0 0.0.0.0:50020 0.0.0.0:* LISTEN 10682/java
tcp 0 0 0.0.0.0:8040 0.0.0.0:* LISTEN 11455/java

浏览NameNode、DataNode信息，可以查看HDFS状态信息

Java代码

http://hostname:50070

浏览ResourceManagered运行状态,可以浏览MapReduce任务的执行情况

Java代码

http://hostname:8088

八、运行Hadoop自带的WordCount实例

创建本地文件用于计算这个文件中的单词数

Java代码

echo "My first hadoop example. Hello Hadoop in input. " > /home/hadoop/input

创建HDFS输入目录，用于将上面的文件写入这个目录

Java代码

hadoop fs -mkdir /user/hadooper

Java代码

hadoop fs -put /home/hadoop/input /user/hadooper

执行Hadoop自带的WordCount例子

Java代码

hadoop jar /home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.2.jar wordcount /user/hadooper/input /user/hadooper/output

MapReduce的过程输出

Java代码

hadoop@hostname:~/hadoop-2.5.2/share/hadoop/mapreduce$ hadoop jar /home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.2.jar wordcount /user/hadooper/input /user/hadooper/output
14/11/23 19:45:04 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
14/11/23 19:45:05 INFO input.FileInputFormat: Total input paths to process : 1
14/11/23 19:45:05 INFO mapreduce.JobSubmitter: number of splits:1
14/11/23 19:45:06 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1416742510596_0001
14/11/23 19:45:06 INFO impl.YarnClientImpl: Submitted application application_1416742510596_0001
14/11/23 19:45:07 INFO mapreduce.Job: The url to track the job: http://hostname:8088/proxy/application_1416742510596_0001/
14/11/23 19:45:07 INFO mapreduce.Job: Running job: job_1416742510596_0001
14/11/23 19:45:18 INFO mapreduce.Job: Job job_1416742510596_0001 running in uber mode : false
14/11/23 19:45:18 INFO mapreduce.Job: map 0% reduce 0%
14/11/23 19:45:26 INFO mapreduce.Job: map 100% reduce 0%
14/11/23 19:45:36 INFO mapreduce.Job: map 100% reduce 100%
14/11/23 19:45:37 INFO mapreduce.Job: Job job_1416742510596_0001 completed successfully
14/11/23 19:45:37 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=102
FILE: Number of bytes written=195793
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=168
HDFS: Number of bytes written=64
HDFS: Number of read operations=6
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=5994
Total time spent by all reduces in occupied slots (ms)=6925
Total time spent by all map tasks (ms)=5994
Total time spent by all reduce tasks (ms)=6925
Total vcore-seconds taken by all map tasks=5994
Total vcore-seconds taken by all reduce tasks=6925
Total megabyte-seconds taken by all map tasks=6137856
Total megabyte-seconds taken by all reduce tasks=7091200
Map-Reduce Framework
Map input records=1
Map output records=8
Map output bytes=80
Map output materialized bytes=102
Input split bytes=119
Combine input records=8
Combine output records=8
Reduce input groups=8
Reduce shuffle bytes=102
Reduce input records=8
Reduce output records=8
Spilled Records=16
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=101
CPU time spent (ms)=2640
Physical memory (bytes) snapshot=422895616
Virtual memory (bytes) snapshot=2055233536
Total committed heap usage (bytes)=308281344
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=49
File Output Format Counters
Bytes Written=64

查看MapReduce的运行结果

Java代码

hadoop@hostname:~/hadoop-2.5.2/share/hadoop/mapreduce$ hadoop fs -cat /user/hadooper/output/part-r-00000
Hadoop 1
Hello 1
My 1
example. 1
first 1
hadoop 1
in 1
input. 1

九、运行Hadoop的PI程序

Java代码

hadoop jar /home/hadoop/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.2.jar pi 10 10

执行结果是3.200000000000000000

十、Hadoop常见问题

1. hadoop不正常推出后，重启后，NameNode将进入Safe Mode，不能提交任务，解决办法：

Java代码

hadoop dfsadmin -safemode leave