Hadoop – 03

Hadoop集群及组件

zookeeper集群图例

#mermaid-svg-IDEVwh1AvQL02BWz .label{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);fill:#333;color:#333}#mermaid-svg-IDEVwh1AvQL02BWz .label text{fill:#333}#mermaid-svg-IDEVwh1AvQL02BWz .node rect,#mermaid-svg-IDEVwh1AvQL02BWz .node circle,#mermaid-svg-IDEVwh1AvQL02BWz .node ellipse,#mermaid-svg-IDEVwh1AvQL02BWz .node polygon,#mermaid-svg-IDEVwh1AvQL02BWz .node path{fill:#ECECFF;stroke:#9370db;stroke-width:1px}#mermaid-svg-IDEVwh1AvQL02BWz .node .label{text-align:center;fill:#333}#mermaid-svg-IDEVwh1AvQL02BWz .node.clickable{cursor:pointer}#mermaid-svg-IDEVwh1AvQL02BWz .arrowheadPath{fill:#333}#mermaid-svg-IDEVwh1AvQL02BWz .edgePath .path{stroke:#333;stroke-width:1.5px}#mermaid-svg-IDEVwh1AvQL02BWz .flowchart-link{stroke:#333;fill:none}#mermaid-svg-IDEVwh1AvQL02BWz .edgeLabel{background-color:#e8e8e8;text-align:center}#mermaid-svg-IDEVwh1AvQL02BWz .edgeLabel rect{opacity:0.9}#mermaid-svg-IDEVwh1AvQL02BWz .edgeLabel span{color:#333}#mermaid-svg-IDEVwh1AvQL02BWz .cluster rect{fill:#ffffde;stroke:#aa3;stroke-width:1px}#mermaid-svg-IDEVwh1AvQL02BWz .cluster text{fill:#333}#mermaid-svg-IDEVwh1AvQL02BWz div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);font-size:12px;background:#ffffde;border:1px solid #aa3;border-radius:2px;pointer-events:none;z-index:100}#mermaid-svg-IDEVwh1AvQL02BWz .actor{stroke:#ccf;fill:#ECECFF}#mermaid-svg-IDEVwh1AvQL02BWz text.actor>tspan{fill:#000;stroke:none}#mermaid-svg-IDEVwh1AvQL02BWz .actor-line{stroke:grey}#mermaid-svg-IDEVwh1AvQL02BWz .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333}#mermaid-svg-IDEVwh1AvQL02BWz .messageLine1{stroke-width:1.5;stroke-dasharray:2, 2;stroke:#333}#mermaid-svg-IDEVwh1AvQL02BWz #arrowhead path{fill:#333;stroke:#333}#mermaid-svg-IDEVwh1AvQL02BWz .sequenceNumber{fill:#fff}#mermaid-svg-IDEVwh1AvQL02BWz #sequencenumber{fill:#333}#mermaid-svg-IDEVwh1AvQL02BWz #crosshead path{fill:#333;stroke:#333}#mermaid-svg-IDEVwh1AvQL02BWz .messageText{fill:#333;stroke:#333}#mermaid-svg-IDEVwh1AvQL02BWz .labelBox{stroke:#ccf;fill:#ECECFF}#mermaid-svg-IDEVwh1AvQL02BWz .labelText,#mermaid-svg-IDEVwh1AvQL02BWz .labelText>tspan{fill:#000;stroke:none}#mermaid-svg-IDEVwh1AvQL02BWz .loopText,#mermaid-svg-IDEVwh1AvQL02BWz .loopText>tspan{fill:#000;stroke:none}#mermaid-svg-IDEVwh1AvQL02BWz .loopLine{stroke-width:2px;stroke-dasharray:2, 2;stroke:#ccf;fill:#ccf}#mermaid-svg-IDEVwh1AvQL02BWz .note{stroke:#aa3;fill:#fff5ad}#mermaid-svg-IDEVwh1AvQL02BWz .noteText,#mermaid-svg-IDEVwh1AvQL02BWz .noteText>tspan{fill:#000;stroke:none}#mermaid-svg-IDEVwh1AvQL02BWz .activation0{fill:#f4f4f4;stroke:#666}#mermaid-svg-IDEVwh1AvQL02BWz .activation1{fill:#f4f4f4;stroke:#666}#mermaid-svg-IDEVwh1AvQL02BWz .activation2{fill:#f4f4f4;stroke:#666}#mermaid-svg-IDEVwh1AvQL02BWz .mermaid-main-font{font-family:"trebuchet ms", verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-IDEVwh1AvQL02BWz .section{stroke:none;opacity:0.2}#mermaid-svg-IDEVwh1AvQL02BWz .section0{fill:rgba(102,102,255,0.49)}#mermaid-svg-IDEVwh1AvQL02BWz .section2{fill:#fff400}#mermaid-svg-IDEVwh1AvQL02BWz .section1,#mermaid-svg-IDEVwh1AvQL02BWz .section3{fill:#fff;opacity:0.2}#mermaid-svg-IDEVwh1AvQL02BWz .sectionTitle0{fill:#333}#mermaid-svg-IDEVwh1AvQL02BWz .sectionTitle1{fill:#333}#mermaid-svg-IDEVwh1AvQL02BWz .sectionTitle2{fill:#333}#mermaid-svg-IDEVwh1AvQL02BWz .sectionTitle3{fill:#333}#mermaid-svg-IDEVwh1AvQL02BWz .sectionTitle{text-anchor:start;font-size:11px;text-height:14px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-IDEVwh1AvQL02BWz .grid .tick{stroke:#d3d3d3;opacity:0.8;shape-rendering:crispEdges}#mermaid-svg-IDEVwh1AvQL02BWz .grid .tick text{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-IDEVwh1AvQL02BWz .grid path{stroke-width:0}#mermaid-svg-IDEVwh1AvQL02BWz .today{fill:none;stroke:red;stroke-width:2px}#mermaid-svg-IDEVwh1AvQL02BWz .task{stroke-width:2}#mermaid-svg-IDEVwh1AvQL02BWz .taskText{text-anchor:middle;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-IDEVwh1AvQL02BWz .taskText:not([font-size]){font-size:11px}#mermaid-svg-IDEVwh1AvQL02BWz .taskTextOutsideRight{fill:#000;text-anchor:start;font-size:11px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-IDEVwh1AvQL02BWz .taskTextOutsideLeft{fill:#000;text-anchor:end;font-size:11px}#mermaid-svg-IDEVwh1AvQL02BWz .task.clickable{cursor:pointer}#mermaid-svg-IDEVwh1AvQL02BWz .taskText.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-IDEVwh1AvQL02BWz .taskTextOutsideLeft.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-IDEVwh1AvQL02BWz .taskTextOutsideRight.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-IDEVwh1AvQL02BWz .taskText0,#mermaid-svg-IDEVwh1AvQL02BWz .taskText1,#mermaid-svg-IDEVwh1AvQL02BWz .taskText2,#mermaid-svg-IDEVwh1AvQL02BWz .taskText3{fill:#fff}#mermaid-svg-IDEVwh1AvQL02BWz .task0,#mermaid-svg-IDEVwh1AvQL02BWz .task1,#mermaid-svg-IDEVwh1AvQL02BWz .task2,#mermaid-svg-IDEVwh1AvQL02BWz .task3{fill:#8a90dd;stroke:#534fbc}#mermaid-svg-IDEVwh1AvQL02BWz .taskTextOutside0,#mermaid-svg-IDEVwh1AvQL02BWz .taskTextOutside2{fill:#000}#mermaid-svg-IDEVwh1AvQL02BWz .taskTextOutside1,#mermaid-svg-IDEVwh1AvQL02BWz .taskTextOutside3{fill:#000}#mermaid-svg-IDEVwh1AvQL02BWz .active0,#mermaid-svg-IDEVwh1AvQL02BWz .active1,#mermaid-svg-IDEVwh1AvQL02BWz .active2,#mermaid-svg-IDEVwh1AvQL02BWz .active3{fill:#bfc7ff;stroke:#534fbc}#mermaid-svg-IDEVwh1AvQL02BWz .activeText0,#mermaid-svg-IDEVwh1AvQL02BWz .activeText1,#mermaid-svg-IDEVwh1AvQL02BWz .activeText2,#mermaid-svg-IDEVwh1AvQL02BWz .activeText3{fill:#000 !important}#mermaid-svg-IDEVwh1AvQL02BWz .done0,#mermaid-svg-IDEVwh1AvQL02BWz .done1,#mermaid-svg-IDEVwh1AvQL02BWz .done2,#mermaid-svg-IDEVwh1AvQL02BWz .done3{stroke:grey;fill:#d3d3d3;stroke-width:2}#mermaid-svg-IDEVwh1AvQL02BWz .doneText0,#mermaid-svg-IDEVwh1AvQL02BWz .doneText1,#mermaid-svg-IDEVwh1AvQL02BWz .doneText2,#mermaid-svg-IDEVwh1AvQL02BWz .doneText3{fill:#000 !important}#mermaid-svg-IDEVwh1AvQL02BWz .crit0,#mermaid-svg-IDEVwh1AvQL02BWz .crit1,#mermaid-svg-IDEVwh1AvQL02BWz .crit2,#mermaid-svg-IDEVwh1AvQL02BWz .crit3{stroke:#f88;fill:red;stroke-width:2}#mermaid-svg-IDEVwh1AvQL02BWz .activeCrit0,#mermaid-svg-IDEVwh1AvQL02BWz .activeCrit1,#mermaid-svg-IDEVwh1AvQL02BWz .activeCrit2,#mermaid-svg-IDEVwh1AvQL02BWz .activeCrit3{stroke:#f88;fill:#bfc7ff;stroke-width:2}#mermaid-svg-IDEVwh1AvQL02BWz .doneCrit0,#mermaid-svg-IDEVwh1AvQL02BWz .doneCrit1,#mermaid-svg-IDEVwh1AvQL02BWz .doneCrit2,#mermaid-svg-IDEVwh1AvQL02BWz .doneCrit3{stroke:#f88;fill:#d3d3d3;stroke-width:2;cursor:pointer;shape-rendering:crispEdges}#mermaid-svg-IDEVwh1AvQL02BWz .milestone{transform:rotate(45deg) scale(0.8, 0.8)}#mermaid-svg-IDEVwh1AvQL02BWz .milestoneText{font-style:italic}#mermaid-svg-IDEVwh1AvQL02BWz .doneCritText0,#mermaid-svg-IDEVwh1AvQL02BWz .doneCritText1,#mermaid-svg-IDEVwh1AvQL02BWz .doneCritText2,#mermaid-svg-IDEVwh1AvQL02BWz .doneCritText3{fill:#000 !important}#mermaid-svg-IDEVwh1AvQL02BWz .activeCritText0,#mermaid-svg-IDEVwh1AvQL02BWz .activeCritText1,#mermaid-svg-IDEVwh1AvQL02BWz .activeCritText2,#mermaid-svg-IDEVwh1AvQL02BWz .activeCritText3{fill:#000 !important}#mermaid-svg-IDEVwh1AvQL02BWz .titleText{text-anchor:middle;font-size:18px;fill:#000;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-IDEVwh1AvQL02BWz g.classGroup text{fill:#9370db;stroke:none;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);font-size:10px}#mermaid-svg-IDEVwh1AvQL02BWz g.classGroup text .title{font-weight:bolder}#mermaid-svg-IDEVwh1AvQL02BWz g.clickable{cursor:pointer}#mermaid-svg-IDEVwh1AvQL02BWz g.classGroup rect{fill:#ECECFF;stroke:#9370db}#mermaid-svg-IDEVwh1AvQL02BWz g.classGroup line{stroke:#9370db;stroke-width:1}#mermaid-svg-IDEVwh1AvQL02BWz .classLabel .box{stroke:none;stroke-width:0;fill:#ECECFF;opacity:0.5}#mermaid-svg-IDEVwh1AvQL02BWz .classLabel .label{fill:#9370db;font-size:10px}#mermaid-svg-IDEVwh1AvQL02BWz .relation{stroke:#9370db;stroke-width:1;fill:none}#mermaid-svg-IDEVwh1AvQL02BWz .dashed-line{stroke-dasharray:3}#mermaid-svg-IDEVwh1AvQL02BWz #compositionStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-IDEVwh1AvQL02BWz #compositionEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-IDEVwh1AvQL02BWz #aggregationStart{fill:#ECECFF;stroke:#9370db;stroke-width:1}#mermaid-svg-IDEVwh1AvQL02BWz #aggregationEnd{fill:#ECECFF;stroke:#9370db;stroke-width:1}#mermaid-svg-IDEVwh1AvQL02BWz #dependencyStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-IDEVwh1AvQL02BWz #dependencyEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-IDEVwh1AvQL02BWz #extensionStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-IDEVwh1AvQL02BWz #extensionEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-IDEVwh1AvQL02BWz .commit-id,#mermaid-svg-IDEVwh1AvQL02BWz .commit-msg,#mermaid-svg-IDEVwh1AvQL02BWz .branch-label{fill:lightgrey;color:lightgrey;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-IDEVwh1AvQL02BWz .pieTitleText{text-anchor:middle;font-size:25px;fill:#000;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-IDEVwh1AvQL02BWz .slice{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-IDEVwh1AvQL02BWz g.stateGroup text{fill:#9370db;stroke:none;font-size:10px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-IDEVwh1AvQL02BWz g.stateGroup text{fill:#9370db;fill:#333;stroke:none;font-size:10px}#mermaid-svg-IDEVwh1AvQL02BWz g.statediagram-cluster .cluster-label text{fill:#333}#mermaid-svg-IDEVwh1AvQL02BWz g.stateGroup .state-title{font-weight:bolder;fill:#000}#mermaid-svg-IDEVwh1AvQL02BWz g.stateGroup rect{fill:#ECECFF;stroke:#9370db}#mermaid-svg-IDEVwh1AvQL02BWz g.stateGroup line{stroke:#9370db;stroke-width:1}#mermaid-svg-IDEVwh1AvQL02BWz .transition{stroke:#9370db;stroke-width:1;fill:none}#mermaid-svg-IDEVwh1AvQL02BWz .stateGroup .composit{fill:white;border-bottom:1px}#mermaid-svg-IDEVwh1AvQL02BWz .stateGroup .alt-composit{fill:#e0e0e0;border-bottom:1px}#mermaid-svg-IDEVwh1AvQL02BWz .state-note{stroke:#aa3;fill:#fff5ad}#mermaid-svg-IDEVwh1AvQL02BWz .state-note text{fill:black;stroke:none;font-size:10px}#mermaid-svg-IDEVwh1AvQL02BWz .stateLabel .box{stroke:none;stroke-width:0;fill:#ECECFF;opacity:0.7}#mermaid-svg-IDEVwh1AvQL02BWz .edgeLabel text{fill:#333}#mermaid-svg-IDEVwh1AvQL02BWz .stateLabel text{fill:#000;font-size:10px;font-weight:bold;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-IDEVwh1AvQL02BWz .node circle.state-start{fill:black;stroke:black}#mermaid-svg-IDEVwh1AvQL02BWz .node circle.state-end{fill:black;stroke:white;stroke-width:1.5}#mermaid-svg-IDEVwh1AvQL02BWz #statediagram-barbEnd{fill:#9370db}#mermaid-svg-IDEVwh1AvQL02BWz .statediagram-cluster rect{fill:#ECECFF;stroke:#9370db;stroke-width:1px}#mermaid-svg-IDEVwh1AvQL02BWz .statediagram-cluster rect.outer{rx:5px;ry:5px}#mermaid-svg-IDEVwh1AvQL02BWz .statediagram-state .divider{stroke:#9370db}#mermaid-svg-IDEVwh1AvQL02BWz .statediagram-state .title-state{rx:5px;ry:5px}#mermaid-svg-IDEVwh1AvQL02BWz .statediagram-cluster.statediagram-cluster .inner{fill:white}#mermaid-svg-IDEVwh1AvQL02BWz .statediagram-cluster.statediagram-cluster-alt .inner{fill:#e0e0e0}#mermaid-svg-IDEVwh1AvQL02BWz .statediagram-cluster .inner{rx:0;ry:0}#mermaid-svg-IDEVwh1AvQL02BWz .statediagram-state rect.basic{rx:5px;ry:5px}#mermaid-svg-IDEVwh1AvQL02BWz .statediagram-state rect.divider{stroke-dasharray:10,10;fill:#efefef}#mermaid-svg-IDEVwh1AvQL02BWz .note-edge{stroke-dasharray:5}#mermaid-svg-IDEVwh1AvQL02BWz .statediagram-note rect{fill:#fff5ad;stroke:#aa3;stroke-width:1px;rx:0;ry:0}:root{--mermaid-font-family: '"trebuchet ms", verdana, arial';--mermaid-font-family: "Comic Sans MS", "Comic Sans", cursive}#mermaid-svg-IDEVwh1AvQL02BWz .error-icon{fill:#522}#mermaid-svg-IDEVwh1AvQL02BWz .error-text{fill:#522;stroke:#522}#mermaid-svg-IDEVwh1AvQL02BWz .edge-thickness-normal{stroke-width:2px}#mermaid-svg-IDEVwh1AvQL02BWz .edge-thickness-thick{stroke-width:3.5px}#mermaid-svg-IDEVwh1AvQL02BWz .edge-pattern-solid{stroke-dasharray:0}#mermaid-svg-IDEVwh1AvQL02BWz .edge-pattern-dashed{stroke-dasharray:3}#mermaid-svg-IDEVwh1AvQL02BWz .edge-pattern-dotted{stroke-dasharray:2}#mermaid-svg-IDEVwh1AvQL02BWz .marker{fill:#333}#mermaid-svg-IDEVwh1AvQL02BWz .marker.cross{stroke:#333}:root { --mermaid-font-family: "trebuchet ms", verdana, arial;}#mermaid-svg-IDEVwh1AvQL02BWz {color: rgba(0, 0, 0, 0.74902);font: normal normal normal normal 16px/26px -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif;}

leader
follower
follower
observer

zookeeper集群

集群安装

1、重启云主机 hadoop1,node-0001,node-0002,node-0003

2、在 hadoop1 上安装配置 zookeeper，并同步给其他主机

拷贝云盘 public/hadoop/zookeeper-3.4.13.tar.gz 到hadoop1

[root@hadoop1 ~]# yum install -y java-1.8.0-openjdk-devel
[root@hadoop1 ~]# tar zxf zookeeper-3.4.13.tar.gz
[root@hadoop1 ~]# mv zookeeper-3.4.13 /usr/local/zookeeper
[root@hadoop1 ~]# cd /usr/local/zookeeper/conf/
[root@hadoop1 conf]# cp zoo_sample.cfg zoo.cfg
[root@hadoop1 conf]# vim zoo.cfg
# 配置文件最后添加
server.1=node-0001:2888:3888
server.2=node-0002:2888:3888
server.3=node-0003:2888:3888
server.4=hadoop1:2888:3888:observer
[root@hadoop1 ~]# for i in node-{0001..0003};dorsync -aXSH --delete /usr/local/zookeeper ${i}:/usr/local/done

所有节点手工启动服务

[root@hadoop1 ~]# mkdir /tmp/zookeeper
[root@hadoop1 ~]# grep -Po "\d+(?==${HOSTNAME})" /usr/local/zookeeper/conf/zoo.cfg >/tmp/zookeeper/myid
[root@hadoop1 ~]# /usr/local/zookeeper/bin/zkServer.sh start
[root@hadoop1 ~]# jps
1001 QuorumPeerMain

当所有节点启动完成以后使用命令验证:

/usr/local/zookeeper/bin/zkServer.sh status

zookeeper集群管理

手册地址

[root@hadoop1 ~]# yum install -y socat
[root@hadoop1 ~]# socat - TCP:node-0001:2181
ruok
imok
[root@hadoop1 bin]# ./zkstats hadoop1 node-{0001..0003}hadoop1 Mode: observernode-0001 Mode: followernode-0002 Mode: leadernode-0003 Mode: follower

kafka集群

kafka安装

1、安装配置 kafka，并同步给其他主机

拷贝云盘 public/hadoop/kafka_2.12-2.1.0.tgz 到 hadoop1

[root@hadoop1 ~]# yum install -y java-1.8.0-openjdk-devel
[root@hadoop1 ~]# tar zxf kafka_2.12-2.1.0.tgz
[root@hadoop1 ~]# mv kafka_2.12-2.1.0 /usr/local/kafka
[root@hadoop1 ~]# for i in node-{0001..0003};dorsync -aXSH --delete /usr/local/kafka ${i}:/usr/local/done

2、修改 node-0001,node-0002,node-0003 配置文件并启动服务

[root@node-0001 ~]# vim /usr/local/kafka/config/server.properties
21   broker.id=1
123  zookeeper.connect=node-0001:2181,node-0002:2181,node-0003:2181
[root@node-0001 ~]# /usr/local/kafka/bin/kafka-server-start.sh -daemon /usr/local/kafka/config/server.properties
[root@node-0001 ~]# jps
1400 Kafka

3、验证（在不同机器上执行）

[root@node-0001 ~]# /usr/local/kafka/bin/kafka-topics.sh --create --partitions 1 --replication-factor 1 --zookeeper localhost:2181 --topic mymsg
#----------------------------------------------------------------------------------------
[root@node-0002 ~]# /usr/local/kafka/bin/kafka-console-producer.sh --broker-list  localhost:9092 --topic mymsg
#----------------------------------------------------------------------------------------
[root@node-0003 ~]# /usr/local/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic mymsg

Hadoop高可用集群

高可用架构图例
#mermaid-svg-04JaD9iQRa0Ctq9b .label{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);fill:#333;color:#333}#mermaid-svg-04JaD9iQRa0Ctq9b .label text{fill:#333}#mermaid-svg-04JaD9iQRa0Ctq9b .node rect,#mermaid-svg-04JaD9iQRa0Ctq9b .node circle,#mermaid-svg-04JaD9iQRa0Ctq9b .node ellipse,#mermaid-svg-04JaD9iQRa0Ctq9b .node polygon,#mermaid-svg-04JaD9iQRa0Ctq9b .node path{fill:#ECECFF;stroke:#9370db;stroke-width:1px}#mermaid-svg-04JaD9iQRa0Ctq9b .node .label{text-align:center;fill:#333}#mermaid-svg-04JaD9iQRa0Ctq9b .node.clickable{cursor:pointer}#mermaid-svg-04JaD9iQRa0Ctq9b .arrowheadPath{fill:#333}#mermaid-svg-04JaD9iQRa0Ctq9b .edgePath .path{stroke:#333;stroke-width:1.5px}#mermaid-svg-04JaD9iQRa0Ctq9b .flowchart-link{stroke:#333;fill:none}#mermaid-svg-04JaD9iQRa0Ctq9b .edgeLabel{background-color:#e8e8e8;text-align:center}#mermaid-svg-04JaD9iQRa0Ctq9b .edgeLabel rect{opacity:0.9}#mermaid-svg-04JaD9iQRa0Ctq9b .edgeLabel span{color:#333}#mermaid-svg-04JaD9iQRa0Ctq9b .cluster rect{fill:#ffffde;stroke:#aa3;stroke-width:1px}#mermaid-svg-04JaD9iQRa0Ctq9b .cluster text{fill:#333}#mermaid-svg-04JaD9iQRa0Ctq9b div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);font-size:12px;background:#ffffde;border:1px solid #aa3;border-radius:2px;pointer-events:none;z-index:100}#mermaid-svg-04JaD9iQRa0Ctq9b .actor{stroke:#ccf;fill:#ECECFF}#mermaid-svg-04JaD9iQRa0Ctq9b text.actor>tspan{fill:#000;stroke:none}#mermaid-svg-04JaD9iQRa0Ctq9b .actor-line{stroke:grey}#mermaid-svg-04JaD9iQRa0Ctq9b .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333}#mermaid-svg-04JaD9iQRa0Ctq9b .messageLine1{stroke-width:1.5;stroke-dasharray:2, 2;stroke:#333}#mermaid-svg-04JaD9iQRa0Ctq9b #arrowhead path{fill:#333;stroke:#333}#mermaid-svg-04JaD9iQRa0Ctq9b .sequenceNumber{fill:#fff}#mermaid-svg-04JaD9iQRa0Ctq9b #sequencenumber{fill:#333}#mermaid-svg-04JaD9iQRa0Ctq9b #crosshead path{fill:#333;stroke:#333}#mermaid-svg-04JaD9iQRa0Ctq9b .messageText{fill:#333;stroke:#333}#mermaid-svg-04JaD9iQRa0Ctq9b .labelBox{stroke:#ccf;fill:#ECECFF}#mermaid-svg-04JaD9iQRa0Ctq9b .labelText,#mermaid-svg-04JaD9iQRa0Ctq9b .labelText>tspan{fill:#000;stroke:none}#mermaid-svg-04JaD9iQRa0Ctq9b .loopText,#mermaid-svg-04JaD9iQRa0Ctq9b .loopText>tspan{fill:#000;stroke:none}#mermaid-svg-04JaD9iQRa0Ctq9b .loopLine{stroke-width:2px;stroke-dasharray:2, 2;stroke:#ccf;fill:#ccf}#mermaid-svg-04JaD9iQRa0Ctq9b .note{stroke:#aa3;fill:#fff5ad}#mermaid-svg-04JaD9iQRa0Ctq9b .noteText,#mermaid-svg-04JaD9iQRa0Ctq9b .noteText>tspan{fill:#000;stroke:none}#mermaid-svg-04JaD9iQRa0Ctq9b .activation0{fill:#f4f4f4;stroke:#666}#mermaid-svg-04JaD9iQRa0Ctq9b .activation1{fill:#f4f4f4;stroke:#666}#mermaid-svg-04JaD9iQRa0Ctq9b .activation2{fill:#f4f4f4;stroke:#666}#mermaid-svg-04JaD9iQRa0Ctq9b .mermaid-main-font{font-family:"trebuchet ms", verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-04JaD9iQRa0Ctq9b .section{stroke:none;opacity:0.2}#mermaid-svg-04JaD9iQRa0Ctq9b .section0{fill:rgba(102,102,255,0.49)}#mermaid-svg-04JaD9iQRa0Ctq9b .section2{fill:#fff400}#mermaid-svg-04JaD9iQRa0Ctq9b .section1,#mermaid-svg-04JaD9iQRa0Ctq9b .section3{fill:#fff;opacity:0.2}#mermaid-svg-04JaD9iQRa0Ctq9b .sectionTitle0{fill:#333}#mermaid-svg-04JaD9iQRa0Ctq9b .sectionTitle1{fill:#333}#mermaid-svg-04JaD9iQRa0Ctq9b .sectionTitle2{fill:#333}#mermaid-svg-04JaD9iQRa0Ctq9b .sectionTitle3{fill:#333}#mermaid-svg-04JaD9iQRa0Ctq9b .sectionTitle{text-anchor:start;font-size:11px;text-height:14px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-04JaD9iQRa0Ctq9b .grid .tick{stroke:#d3d3d3;opacity:0.8;shape-rendering:crispEdges}#mermaid-svg-04JaD9iQRa0Ctq9b .grid .tick text{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-04JaD9iQRa0Ctq9b .grid path{stroke-width:0}#mermaid-svg-04JaD9iQRa0Ctq9b .today{fill:none;stroke:red;stroke-width:2px}#mermaid-svg-04JaD9iQRa0Ctq9b .task{stroke-width:2}#mermaid-svg-04JaD9iQRa0Ctq9b .taskText{text-anchor:middle;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-04JaD9iQRa0Ctq9b .taskText:not([font-size]){font-size:11px}#mermaid-svg-04JaD9iQRa0Ctq9b .taskTextOutsideRight{fill:#000;text-anchor:start;font-size:11px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-04JaD9iQRa0Ctq9b .taskTextOutsideLeft{fill:#000;text-anchor:end;font-size:11px}#mermaid-svg-04JaD9iQRa0Ctq9b .task.clickable{cursor:pointer}#mermaid-svg-04JaD9iQRa0Ctq9b .taskText.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-04JaD9iQRa0Ctq9b .taskTextOutsideLeft.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-04JaD9iQRa0Ctq9b .taskTextOutsideRight.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-04JaD9iQRa0Ctq9b .taskText0,#mermaid-svg-04JaD9iQRa0Ctq9b .taskText1,#mermaid-svg-04JaD9iQRa0Ctq9b .taskText2,#mermaid-svg-04JaD9iQRa0Ctq9b .taskText3{fill:#fff}#mermaid-svg-04JaD9iQRa0Ctq9b .task0,#mermaid-svg-04JaD9iQRa0Ctq9b .task1,#mermaid-svg-04JaD9iQRa0Ctq9b .task2,#mermaid-svg-04JaD9iQRa0Ctq9b .task3{fill:#8a90dd;stroke:#534fbc}#mermaid-svg-04JaD9iQRa0Ctq9b .taskTextOutside0,#mermaid-svg-04JaD9iQRa0Ctq9b .taskTextOutside2{fill:#000}#mermaid-svg-04JaD9iQRa0Ctq9b .taskTextOutside1,#mermaid-svg-04JaD9iQRa0Ctq9b .taskTextOutside3{fill:#000}#mermaid-svg-04JaD9iQRa0Ctq9b .active0,#mermaid-svg-04JaD9iQRa0Ctq9b .active1,#mermaid-svg-04JaD9iQRa0Ctq9b .active2,#mermaid-svg-04JaD9iQRa0Ctq9b .active3{fill:#bfc7ff;stroke:#534fbc}#mermaid-svg-04JaD9iQRa0Ctq9b .activeText0,#mermaid-svg-04JaD9iQRa0Ctq9b .activeText1,#mermaid-svg-04JaD9iQRa0Ctq9b .activeText2,#mermaid-svg-04JaD9iQRa0Ctq9b .activeText3{fill:#000 !important}#mermaid-svg-04JaD9iQRa0Ctq9b .done0,#mermaid-svg-04JaD9iQRa0Ctq9b .done1,#mermaid-svg-04JaD9iQRa0Ctq9b .done2,#mermaid-svg-04JaD9iQRa0Ctq9b .done3{stroke:grey;fill:#d3d3d3;stroke-width:2}#mermaid-svg-04JaD9iQRa0Ctq9b .doneText0,#mermaid-svg-04JaD9iQRa0Ctq9b .doneText1,#mermaid-svg-04JaD9iQRa0Ctq9b .doneText2,#mermaid-svg-04JaD9iQRa0Ctq9b .doneText3{fill:#000 !important}#mermaid-svg-04JaD9iQRa0Ctq9b .crit0,#mermaid-svg-04JaD9iQRa0Ctq9b .crit1,#mermaid-svg-04JaD9iQRa0Ctq9b .crit2,#mermaid-svg-04JaD9iQRa0Ctq9b .crit3{stroke:#f88;fill:red;stroke-width:2}#mermaid-svg-04JaD9iQRa0Ctq9b .activeCrit0,#mermaid-svg-04JaD9iQRa0Ctq9b .activeCrit1,#mermaid-svg-04JaD9iQRa0Ctq9b .activeCrit2,#mermaid-svg-04JaD9iQRa0Ctq9b .activeCrit3{stroke:#f88;fill:#bfc7ff;stroke-width:2}#mermaid-svg-04JaD9iQRa0Ctq9b .doneCrit0,#mermaid-svg-04JaD9iQRa0Ctq9b .doneCrit1,#mermaid-svg-04JaD9iQRa0Ctq9b .doneCrit2,#mermaid-svg-04JaD9iQRa0Ctq9b .doneCrit3{stroke:#f88;fill:#d3d3d3;stroke-width:2;cursor:pointer;shape-rendering:crispEdges}#mermaid-svg-04JaD9iQRa0Ctq9b .milestone{transform:rotate(45deg) scale(0.8, 0.8)}#mermaid-svg-04JaD9iQRa0Ctq9b .milestoneText{font-style:italic}#mermaid-svg-04JaD9iQRa0Ctq9b .doneCritText0,#mermaid-svg-04JaD9iQRa0Ctq9b .doneCritText1,#mermaid-svg-04JaD9iQRa0Ctq9b .doneCritText2,#mermaid-svg-04JaD9iQRa0Ctq9b .doneCritText3{fill:#000 !important}#mermaid-svg-04JaD9iQRa0Ctq9b .activeCritText0,#mermaid-svg-04JaD9iQRa0Ctq9b .activeCritText1,#mermaid-svg-04JaD9iQRa0Ctq9b .activeCritText2,#mermaid-svg-04JaD9iQRa0Ctq9b .activeCritText3{fill:#000 !important}#mermaid-svg-04JaD9iQRa0Ctq9b .titleText{text-anchor:middle;font-size:18px;fill:#000;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-04JaD9iQRa0Ctq9b g.classGroup text{fill:#9370db;stroke:none;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);font-size:10px}#mermaid-svg-04JaD9iQRa0Ctq9b g.classGroup text .title{font-weight:bolder}#mermaid-svg-04JaD9iQRa0Ctq9b g.clickable{cursor:pointer}#mermaid-svg-04JaD9iQRa0Ctq9b g.classGroup rect{fill:#ECECFF;stroke:#9370db}#mermaid-svg-04JaD9iQRa0Ctq9b g.classGroup line{stroke:#9370db;stroke-width:1}#mermaid-svg-04JaD9iQRa0Ctq9b .classLabel .box{stroke:none;stroke-width:0;fill:#ECECFF;opacity:0.5}#mermaid-svg-04JaD9iQRa0Ctq9b .classLabel .label{fill:#9370db;font-size:10px}#mermaid-svg-04JaD9iQRa0Ctq9b .relation{stroke:#9370db;stroke-width:1;fill:none}#mermaid-svg-04JaD9iQRa0Ctq9b .dashed-line{stroke-dasharray:3}#mermaid-svg-04JaD9iQRa0Ctq9b #compositionStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-04JaD9iQRa0Ctq9b #compositionEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-04JaD9iQRa0Ctq9b #aggregationStart{fill:#ECECFF;stroke:#9370db;stroke-width:1}#mermaid-svg-04JaD9iQRa0Ctq9b #aggregationEnd{fill:#ECECFF;stroke:#9370db;stroke-width:1}#mermaid-svg-04JaD9iQRa0Ctq9b #dependencyStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-04JaD9iQRa0Ctq9b #dependencyEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-04JaD9iQRa0Ctq9b #extensionStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-04JaD9iQRa0Ctq9b #extensionEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-04JaD9iQRa0Ctq9b .commit-id,#mermaid-svg-04JaD9iQRa0Ctq9b .commit-msg,#mermaid-svg-04JaD9iQRa0Ctq9b .branch-label{fill:lightgrey;color:lightgrey;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-04JaD9iQRa0Ctq9b .pieTitleText{text-anchor:middle;font-size:25px;fill:#000;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-04JaD9iQRa0Ctq9b .slice{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-04JaD9iQRa0Ctq9b g.stateGroup text{fill:#9370db;stroke:none;font-size:10px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-04JaD9iQRa0Ctq9b g.stateGroup text{fill:#9370db;fill:#333;stroke:none;font-size:10px}#mermaid-svg-04JaD9iQRa0Ctq9b g.statediagram-cluster .cluster-label text{fill:#333}#mermaid-svg-04JaD9iQRa0Ctq9b g.stateGroup .state-title{font-weight:bolder;fill:#000}#mermaid-svg-04JaD9iQRa0Ctq9b g.stateGroup rect{fill:#ECECFF;stroke:#9370db}#mermaid-svg-04JaD9iQRa0Ctq9b g.stateGroup line{stroke:#9370db;stroke-width:1}#mermaid-svg-04JaD9iQRa0Ctq9b .transition{stroke:#9370db;stroke-width:1;fill:none}#mermaid-svg-04JaD9iQRa0Ctq9b .stateGroup .composit{fill:white;border-bottom:1px}#mermaid-svg-04JaD9iQRa0Ctq9b .stateGroup .alt-composit{fill:#e0e0e0;border-bottom:1px}#mermaid-svg-04JaD9iQRa0Ctq9b .state-note{stroke:#aa3;fill:#fff5ad}#mermaid-svg-04JaD9iQRa0Ctq9b .state-note text{fill:black;stroke:none;font-size:10px}#mermaid-svg-04JaD9iQRa0Ctq9b .stateLabel .box{stroke:none;stroke-width:0;fill:#ECECFF;opacity:0.7}#mermaid-svg-04JaD9iQRa0Ctq9b .edgeLabel text{fill:#333}#mermaid-svg-04JaD9iQRa0Ctq9b .stateLabel text{fill:#000;font-size:10px;font-weight:bold;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-04JaD9iQRa0Ctq9b .node circle.state-start{fill:black;stroke:black}#mermaid-svg-04JaD9iQRa0Ctq9b .node circle.state-end{fill:black;stroke:white;stroke-width:1.5}#mermaid-svg-04JaD9iQRa0Ctq9b #statediagram-barbEnd{fill:#9370db}#mermaid-svg-04JaD9iQRa0Ctq9b .statediagram-cluster rect{fill:#ECECFF;stroke:#9370db;stroke-width:1px}#mermaid-svg-04JaD9iQRa0Ctq9b .statediagram-cluster rect.outer{rx:5px;ry:5px}#mermaid-svg-04JaD9iQRa0Ctq9b .statediagram-state .divider{stroke:#9370db}#mermaid-svg-04JaD9iQRa0Ctq9b .statediagram-state .title-state{rx:5px;ry:5px}#mermaid-svg-04JaD9iQRa0Ctq9b .statediagram-cluster.statediagram-cluster .inner{fill:white}#mermaid-svg-04JaD9iQRa0Ctq9b .statediagram-cluster.statediagram-cluster-alt .inner{fill:#e0e0e0}#mermaid-svg-04JaD9iQRa0Ctq9b .statediagram-cluster .inner{rx:0;ry:0}#mermaid-svg-04JaD9iQRa0Ctq9b .statediagram-state rect.basic{rx:5px;ry:5px}#mermaid-svg-04JaD9iQRa0Ctq9b .statediagram-state rect.divider{stroke-dasharray:10,10;fill:#efefef}#mermaid-svg-04JaD9iQRa0Ctq9b .note-edge{stroke-dasharray:5}#mermaid-svg-04JaD9iQRa0Ctq9b .statediagram-note rect{fill:#fff5ad;stroke:#aa3;stroke-width:1px;rx:0;ry:0}:root{--mermaid-font-family: '"trebuchet ms", verdana, arial';--mermaid-font-family: "Comic Sans MS", "Comic Sans", cursive}#mermaid-svg-04JaD9iQRa0Ctq9b .error-icon{fill:#522}#mermaid-svg-04JaD9iQRa0Ctq9b .error-text{fill:#522;stroke:#522}#mermaid-svg-04JaD9iQRa0Ctq9b .edge-thickness-normal{stroke-width:2px}#mermaid-svg-04JaD9iQRa0Ctq9b .edge-thickness-thick{stroke-width:3.5px}#mermaid-svg-04JaD9iQRa0Ctq9b .edge-pattern-solid{stroke-dasharray:0}#mermaid-svg-04JaD9iQRa0Ctq9b .edge-pattern-dashed{stroke-dasharray:3}#mermaid-svg-04JaD9iQRa0Ctq9b .edge-pattern-dotted{stroke-dasharray:2}#mermaid-svg-04JaD9iQRa0Ctq9b .marker{fill:#333}#mermaid-svg-04JaD9iQRa0Ctq9b .marker.cross{stroke:#333}:root { --mermaid-font-family: "trebuchet ms", verdana, arial;}#mermaid-svg-04JaD9iQRa0Ctq9b {color: rgba(0, 0, 0, 0.74902);font: normal normal normal normal 16px/26px -apple-system, "SF UI Text", Arial, "PingFang SC", "Hiragino Sans GB", "Microsoft YaHei", "WenQuanYi Micro Hei", sans-serif;}

hadoop1
<font color=ff0000>NameNode
ResourceManager
DFSZKFailoverController</font>
hadoop2
<font color=ff0000>NameNode
ResourceManager
DFSZKFailoverController</font>
node-0001
<font color=ff0000>DataNode
NodeManager
JournalNode
QuorumPeerMain</font>
node-0002
<font color=ff0000>DataNode
NodeManager
JournalNode
QuorumPeerMain</font>
node-0003
<font color=ff0000>DataNode
NodeManager
JournalNode
QuorumPeerMain</font>

购买云主机

主机名 IP地址 基础配置
hadoop2 192.168.1.56 2cpu, 2G内存
角色与配置
主机名 IP地址 角色服务
hadoop1 192.168.1.50 namenode, resourcemanager, ZKFC
hadoop2 192.168.1.56 namenode, resourcemanager, ZKFC
node-0001 192.168.1.51 datanode, nodemanager, zookeeper, journalnode
node-0002 192.168.1.52 datanode, nodemanager, zookeeper, journalnode
node-0003 192.168.1.53 datanode, nodemanager, zookeeper, journalnode
环境初始化

hadoop1 上执行

[root@hadoop1 ~]# vim /etc/hosts
192.168.1.50    hadoop1
192.168.1.56    hadoop2
192.168.1.51    node-0001
192.168.1.52    node-0002
192.168.1.53    node-0003
[root@hadoop1 ~]# rsync -aXSH --delete /root/.ssh hadoop2:/root/
[root@hadoop1 ~]# for i in hadoop2 node-{0001..0003};dorsync -av /etc/hosts ${i}:/etc/done

hadoop2 上执行

[root@hadoop2 ~]# yum install -y java-1.8.0-openjdk-devel
[root@hadoop2 ~]# vim /etc/ssh/ssh_config
# 60行新添加StrictHostKeyChecking no
集群配置文件

在 hadoop1 上完成以下文件的配置

1、配置 hadoop-env.sh

[root@hadoop1 ~]# vim /usr/local/hadoop/etc/hadoop/hadoop-env.sh
25:  export JAVA_HOME="java-1.8.0-openjdk安装路径"
33:  export HADOOP_CONF_DIR="/usr/local/hadoop/etc/hadoop"

2、配置 slaves

[root@hadoop1 ~]# vim /usr/local/hadoop/etc/hadoop/slaves
node-0001
node-0002
node-0003

3、配置 core-site.xml

[root@hadoop1 ~]# vim /usr/local/hadoop/etc/hadoop/core-site.xml
<configuration><property><name>fs.defaultFS</name><value>hdfs://mycluster</value></property><property><name>hadoop.tmp.dir</name><value>/var/hadoop</value></property><property><name>ha.zookeeper.quorum</name><value>node-0001:2181,node-0002:2181,node-0003:2181</value></property><property><name>hadoop.proxyuser.nfsuser.groups</name><value>*</value></property><property><name>hadoop.proxyuser.nfsuser.hosts</name><value>*</value></property>
</configuration>

4、配置 hdfs-site.xml

[root@hadoop1 ~]# vim /usr/local/hadoop/etc/hadoop/hdfs-site.xml
<configuration><property><name>dfs.nameservices</name><value>mycluster</value></property><property><name>dfs.ha.namenodes.mycluster</name><value>nn1,nn2</value></property><property><name>dfs.namenode.rpc-address.mycluster.nn1</name><value>hadoop1:8020</value></property><property><name>dfs.namenode.rpc-address.mycluster.nn2</name><value>hadoop2:8020</value></property><property><name>dfs.namenode.http-address.mycluster.nn1</name><value>hadoop1:50070</value></property><property><name>dfs.namenode.http-address.mycluster.nn2</name><value>hadoop2:50070</value></property><property><name>dfs.namenode.shared.edits.dir</name><value>qjournal://node-0001:8485;node-0002:8485;node-0003:8485/mycluster</value></property><property><name>dfs.journalnode.edits.dir</name><value>/var/hadoop/journal</value></property><property><name>dfs.client.failover.proxy.provider.mycluster</name><value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value></property><property> <name>dfs.ha.fencing.methods</name><value>sshfence</value></property><property><name>dfs.ha.fencing.ssh.private-key-files</name><value>/root/.ssh/id_rsa</value></property><property><name>dfs.ha.automatic-failover.enabled</name><value>true</value></property><property><name>dfs.replication</name><value>2</value></property><property><name>dfs.hosts.exclude</name><value>/usr/local/hadoop/etc/hadoop/exclude</value></property>
</configuration>

5、配置 mapred-site.xml

[root@hadoop1 ~]# vim /usr/local/hadoop/etc/hadoop/mapred-site.xml
<configuration><property><name>mapreduce.framework.name</name><value>yarn</value></property>
</configuration>

6、配置 yare-site.xml

[root@hadoop1 ~]# vim /usr/local/hadoop/etc/hadoop/yarn-site.xml
<configuration><property><name>yarn.resourcemanager.ha.enabled</name><value>true</value></property><property><name>yarn.resourcemanager.recovery.enabled</name><value>true</value></property><property><name>yarn.resourcemanager.store.class</name><value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value></property><property><name>yarn.resourcemanager.zk-address</name><value>node-0001:2181,node-0002:2181,node-0003:2181</value></property><property><name>yarn.resourcemanager.cluster-id</name><value>yarn-ha</value></property><property><name>yarn.resourcemanager.ha.rm-ids</name><value>rm1,rm2</value></property><property><name>yarn.resourcemanager.hostname.rm1</name><value>hadoop1</value></property><property><name>yarn.resourcemanager.hostname.rm2</name><value>hadoop2</value></property>
<!-- Site specific YARN configuration properties --><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value></property>
</configuration>
初始化启动集群

1、重启机器、在 node-0001，node-0002，node-0003 启动 zookeeper

[root@node-0001 ~]# /usr/local/zookeeper/bin/zkServer.sh start
#----------------------------------------------------------------------------------------
[root@node-0002 ~]# /usr/local/zookeeper/bin/zkServer.sh start
#----------------------------------------------------------------------------------------
[root@node-0003 ~]# /usr/local/zookeeper/bin/zkServer.sh start
#----------------------------------------------------------------------------------------
[root@hadoop1 ~]# zkstats node-{0001..0003}node-0001 Mode: followernode-0002 Mode: leadernode-0003 Mode: follower

2、清空实验数据并同步配置文件（hadoop1 上执行）

[root@hadoop1 ~]# rm -rf /var/hadoop/* /usr/local/hadoop/logs
[root@hadoop1 ~]# for i in hadoop2 node-{0001..0003};dorsync -av /etc/hosts ${i}:/etc/rsync -aXSH --delete /var/hadoop ${i}:/var/rsync -aXSH --delete /usr/local/hadoop ${i}:/usr/local/done

3、在 node-0001，node-0002，node-0003 启动 journalnode 服务

[root@node-0001 ~]# /usr/local/hadoop/sbin/hadoop-daemon.sh start journalnode
[root@node-0001 ~]# jps
1037 JournalNode
#----------------------------------------------------------------------------------------
[root@node-0002 ~]# /usr/local/hadoop/sbin/hadoop-daemon.sh start journalnode
#----------------------------------------------------------------------------------------
[root@node-0003 ~]# /usr/local/hadoop/sbin/hadoop-daemon.sh start journalnode

4、初始化（hadoop1 上执行）

[root@hadoop1 ~]# /usr/local/hadoop/bin/hdfs zkfc -formatZK
[root@hadoop1 ~]# /usr/local/hadoop/bin/hdfs namenode -format
[root@hadoop1 ~]# /usr/local/hadoop/bin/hdfs namenode -initializeSharedEdits
[root@hadoop1 ~]# rsync -aXSH --delete /var/hadoop/dfs hadoop2:/var/hadoop/

5、停止在 node-0001，node-0002，node-0003 上的 journalnode 服务

[root@node-0001 ~]# /usr/local/hadoop/sbin/hadoop-daemon.sh stop journalnode
#----------------------------------------------------------------------------------------
[root@node-0002 ~]# /usr/local/hadoop/sbin/hadoop-daemon.sh stop journalnode
#----------------------------------------------------------------------------------------
[root@node-0003 ~]# /usr/local/hadoop/sbin/hadoop-daemon.sh stop journalnode

6、启动集群

#-------------------- 下面这条命令在 hadoop1 上执行 ----------------------------------------
[root@hadoop1 ~]# /usr/local/hadoop/sbin/start-all.sh
#-------------------- 下面这条命令在 hadoop2 上执行 ----------------------------------------
[root@hadoop2 ~]# /usr/local/hadoop/sbin/yarn-daemon.sh start resourcemanager
验证集群
[root@hadoop1 ~]# /usr/local/hadoop/bin/hdfs haadmin -getServiceState nn1
[root@hadoop1 ~]# /usr/local/hadoop/bin/hdfs haadmin -getServiceState nn2
[root@hadoop1 ~]# /usr/local/hadoop/bin/yarn rmadmin -getServiceState rm1
[root@hadoop1 ~]# /usr/local/hadoop/bin/yarn rmadmin -getServiceState rm2
[root@hadoop1 ~]# /usr/local/hadoop/bin/hdfs dfsadmin -report
[root@hadoop1 ~]# /usr/local/hadoop/bin/yarn node -list

使用高可用集群分析数据实验

[root@hadoop1 ~]# cd /usr/local/hadoop
[root@hadoop1 hadoop]# ./bin/hadoop fs -mkdir /input
[root@hadoop1 hadoop]# ./bin/hadoop fs -put *.txt /input/
[root@hadoop1 hadoop]# ./bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.7.jar wordcount /input /output
[root@hadoop1 hadoop]# ./bin/hadoop fs -cat /output/*

hadoop-集群及组件相关推荐

  1. Hadoop集群高可用及zookeeper+kafka组件搭建

    目录 一.Hadoop集群高可用 1.Zookeeper概述 1)Zookeeper基本概述 3)Zab协议 3)observer 4)zookeeper集群图例 2.zookeeper集群搭建 3. ...

  2. Hadoop集群搭建之Hadoop组件安装

    兵马未动,粮草先行 --汉语成语 系列文章目录 Hadoop集群搭建之Linux系统安装 Hadoop集群搭建之Hadoop组件安装 文章目录 系列文章目录 前言 一.IP和主机名称配置 (一)Had ...

  3. hadoop集群_Ambari搭建hadoop集群

    Ambari搭建hadoop集群 Ambari提供了对Hadoop更加方便快捷的管理功能,主要包含: 通过一步一步的安装向导简化了集群供应. 预先配置好关键的运维指标(metrics),可以直接查看H ...

  4. 【Big Data】HADOOP集群的配置（一）

    Hadoop集群的配置(一) 摘要: hadoop集群配置系列文档,是笔者在实验室真机环境实验后整理而得.以便随后工作所需,做以知识整理,另则与博客园朋友分享实验成果,因为笔者在学习初期,也遇到不少问 ...

  5. Ambari在离线环境中安装Hadoop集群

    2019独角兽企业重金招聘Python工程师标准>>> http://www.hackbase.com/article-973-1.html (WJW)Ambari在离线环境中安装H ...

  6. Hadoop集群搭建（六：HBase的安装配置）

    实验 目的 要求 目的: 1.HBase的高可用完全分布模式的安装和验证 要求: 完成HBase的高可用完全分布模式的安装: HBase的相关服务进程能够正常的启动: HBase控制台能够正常使用: ...

  7. 《Hadoop集群与安全》一2.1　在Hadoop集群中配置操作系统

    本节书摘来自华章出版社<Hadoop集群与安全>一书中的第2章,第2.1节,作者 (美)Danil Zburivsky Sudheesh Narayanan,更多章节内容可以访问云栖社区& ...

  8. Hadoop集群上使用JNI，调用资源文件

    hadoop是基于java的数据计算平台,引入第三方库,例如C语言实现的开发包将会大大增强数据分析的效率和能力. 通常在是用一些工具的时候都要用到一些配置文件.资源文件等.接下来,借一个例子来说明ha ...

  9. hadoop集群硬盘损坏_Hadoop集群(万台规模)的磁盘故障自动处理

    之前的系列文章讲述了无状态服务的自动维修. 后来我们尝试将这套方案应用在有状态的Hadoop集群上,经过几个月的线上运行,也取得了不错的效果,本文将分享这个实践过程. 梳理问题 我们先调研了下Hado ...

  10. 实战CentOS系统部署Hadoop集群服务

    导读 Hadoop是一个由Apache基金会所开发的分布式系统基础架构,Hadoop实现了一个分布式文件系统(Hadoop Distributed File System),简称HDFS.HDFS有高 ...

最新文章

  1. mysql字符集相关问题_MySQL 字符集相关问题
  2. 蓝牙耳机测试用例_移动端测试用例设计总结，这些你得懂！
  3. Angular单元测试框架karma-jasmine里类似ABAP unit框架的setup和class_setup方法
  4. 结对编程作业——毕设导师智能匹配
  5. python 共轭转置_python矩阵运算,转置,逆运算,共轭矩阵实例
  6. simulink方框_4.4设计Simulink框图的界面
  7. 使用jxl来读取Excel中的数据
  8. Django ORM 单表操作
  9. 大疆水弹机器人_现身央视的大疆机甲大师 S1 备受瞩目的教育机器人圆了儿时机甲梦...
  10. pbrt源码中用全主元消去法求矩阵逆的实现
  11. 2、杂项：Bootloader升级方式---擦、写flash在RAM中运行
  12. C# CAD开发 选择集的使用
  13. ccf201703-5 引水入城
  14. 最新版Activit7从原理到项目，工作流精讲上线
  15. Matlab中inv函数的使用
  16. 读书笔记《区块链原理与技术 郑子彬》——区块链共识层
  17. SLCP验厂辅导,SLCP是向最终用户分享已验证的评估数据的关键
  18. log4j日志输出性能优化-缓存、异步
  19. 视频：插入U盘后Win7系统蓝屏，报iusb3xhc.sys
  20. HAVING 子句的力量 和 HAVING 子句又回来了

热门文章

  1. 外观模式 Facade
  2. Syntax error on token(s), misplaced construct(s)
  3. Latex报错 ! Misplaced alignment tab character
  4. Oozie报错Py4JJavaErrornbsp;Secu…
  5. 白话机器学习-长短期记忆网络LSTM
  6. 中央批准！顶尖985大学换帅，开启书记、校长双院士！
  7. video.js API
  8. 一年的第多少周+一个月的第多少天+一周的第多少天
  9. OpenglES2.0 for Android:来做个地球吧
  10. 【DNS系列】什么是正向DNS和反向DNS