Hadoop分布式集群
2024-04-20 10:14:02
文章目录
- 一、环境介绍
- 1、环境组件介绍
- 2、组件功能及作用介绍
- 二、资源下载
- 1、jdk1.8下载
- 2、zookeeper3.4.8下载
- 3、hadoop2.6.5下载
- 三、基础环境搭建
- 1、hadoop用户配置
- 2、免密登录
- 2.1、jdk1.8安装
- 2.2、zookeeper安装
- 2.3、hadoop安装
- 3、确认环境
- 四、Hadoop配置
- 1、hadoop变量文件配置
- 2、hadoop配置文件配置
- 五、启动测试
- 1、初始化集群
- 2、脚本管理
- 3、启动集群测试
- 3.1、脚本管理
- 3.2、namenode测试
- 3.3、resourcemanager状态测试
- 4、页面访问
一、环境介绍
1、环境组件介绍
使用的是三台Linux Centos7系统的机器
| IP地址 | Host | Namenode | Datanode | Zookeeper | DFSZKFailoverController (ZKFC) | Journalnode | Resourcemanager | Nodemanager | JobHistory |
|---|---|---|---|---|---|---|---|---|---|
| 10.20.123.1 | bai1 | √ | √ | √ | √ | √ | √ | ||
| 10.20.123.2 | bai2 | √ | √ | √ | √ | √ | √ | √ | |
| 10.20.123.3 | bai3 | √ | √ | √ | √ | √ |
2、组件功能及作用介绍
- Zookeeper：分布式应用程序协调服务，以Fast Paxos算法为基础，实现同步服务，配置维护和命名服务等分布式应用
- Namenode：管理HDFS，监控Datanode
- DFSZKFailoverController (ZKFC)：监控Namenode的状态，并及时把状态信息写入Zookeeper。当Active状态的Namenode发生故障时，负责故障切换
- Journalnode：存放Namenode的editlog文件（元数据）
- Datanode：存储节点，多副本
- Resourcemanager：负责各个Nodemanager的资源调度
- Nodemanager：管理Datanode的资源
- Jobhistory：记录已经finished的mapreduce运行日志
二、资源下载
1、jdk1.8下载
jdk1.8安装包提取码：LWXB
2、zookeeper3.4.8下载
zookeeper3.4.8安装包提取码：LWXB
3、hadoop2.6.5下载
hadoop2.6.5安装包提取码：LWXB
三、基础环境搭建
1、hadoop用户配置
三台服务器做同样的配置
[root@bai1 ~] vim /etc/hosts
#在hosts中追加解析
10.20.123.1 bai1
10.20.123.2 bai2
10.20.123.3 bai3[root@bai1 ~] useradd hadoop
[root@bai1 ~] passwd hadoop[root@bai1 ~] chmod 640 /etc/sudoers
[root@bai1 ~] vim /etc/sudoers
#添加如下
hadoop ALL=(ALL) ALL
[root@bai1 ~] chmod 440 /etc/sudoers#切换到hadoop用户下将安装包上传至服务器
[root@bai1 ~] su hadoop
[hadoop@bai1 ~]$ ll
-rw-r--r-- 1 hadoop hadoop 199635269 Nov 30 2020 hadoop-2.6.5.tar.gz
-rw-r--r-- 1 hadoop hadoop 181442359 Jun 12 2020 jdk-8u111-linux-x64.tar.gz
-rw-r--r-- 1 hadoop hadoop 22261552 May 10 2021 zookeeper-3.4.8.tar.gz[hadoop@bai1 ~]$ mkdir service
2、免密登录
#传输密钥时的密码为hadoop用户密码
[hadoop@bai1 ~]$ ssh-keygen -t rsa
[hadoop@bai1 ~]$ cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys
[hadoop@bai1 ~]$ ssh-copy-id -i bai2
[hadoop@bai1 ~]$ ssh-copy-id -i bai3[hadoop@bai2 ~]$ ssh-keygen -t rsa
[hadoop@bai2 ~]$ cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys
[hadoop@bai2 ~]$ ssh-copy-id -i bai1
[hadoop@bai2 ~]$ ssh-copy-id -i bai3[hadoop@bai3 ~]$ ssh-keygen -t rsa
[hadoop@bai3 ~]$ cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys
[hadoop@bai3 ~]$ ssh-copy-id -i bai1
[hadoop@bai3 ~]$ ssh-copy-id -i bai2
2.1、jdk1.8安装
[hadoop@bai1 ~]$ tar xzvf jdk-8u111-linux-x64.tar.gz
[hadoop@bai1 ~]$ mv jdk1.8.0_111 service/jdk1.8
[hadoop@bai1 ~]$ vim .bash_profile
#在.bash_profile最后追加
export JAVA_HOME=/home/hadoop/service/jdk1.8
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=$PATH:$HOME/.local/bin:$HOME/bin:${JAVA_HOME}/bin[hadoop@bai1 ~]$ source .bash_profile
[hadoop@bai1 ~]$ java -version
java version "1.8.0_111"
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
2.2、zookeeper安装
[hadoop@bai1 ~]$ tar xzvf zookeeper-3.4.8.tar.gz
[hadoop@bai1 ~]$ mv zookeeper-3.4.8 service/zookeeper
[hadoop@bai1 ~]$ vim service/zookeeper/conf/zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/home/hadoop/service/zookeeper
clientPort=2181
server.1=bai1:2888:3888
server.2=bai2:2888:3888
server.3=bai3:2888:3888[hadoop@bai1 ~]$ mkdir service/zookeeper/data
[hadoop@bai1 ~]$ echo 1 > service/zookeeper/data/myid #按照zoo.cfg中server的顺序echo1,2,3到myid中
[hadoop@bai1 ~]$ vim service/zookeeper/bin/zkEnv.sh
#添加java的家目录
JAVA_HOME="/home/hadoop/service/jdk1.8"
[hadoop@bai1 ~]$ vim .bash_profile
#在刚才添加的JAVA_HOME后追加
export ZK_HOME=/home/hadoop/service/zookeeper
export PATH=$PATH:$HOME/.local/bin:$HOME/bin:${JAVA_HOME}/bin:$ZK_HOME/bin[hadoop@bai1 ~]$ vim service/zookeeper/bin/auto-zk.sh #编写启动脚本
#!/bin/bash
zkbin=$ZK_HOME/bin
serverlist=`cat $zkbin/../conf/zoo.cfg|grep ^server|awk -F= '{print $2}'|awk -F: '{print $1}'`
user=hadoop #这个变量是用我们的hadoop用户去登录for server in ${serverlist[@]}; doecho -e "\nHost [$server]："ssh $user@$server "$zkbin/zkServer.sh $1"
done[hadoop@bai1 ~]$ source .bash_profile
[hadoop@bai1 ~]$ auto-zk.sh start #启动
[hadoop@bai1 ~]$ auto-zk.sh status #查看启动状态Host [bai1]：
ZooKeeper JMX enabled by default
Using config: /home/hadoop/service/zookeeper/bin/../conf/zoo.cfg
Mode: followerHost [bai2]：
ZooKeeper JMX enabled by default
Using config: /home/hadoop/service/zookeeper/bin/../conf/zoo.cfg
Mode: leaderHost [bai3]：
ZooKeeper JMX enabled by default
Using config: /home/hadoop/service/zookeeper/bin/../conf/zoo.cfg
Mode: follower
2.3、hadoop安装
[hadoop@bai1 ~]$ tar xzvf hadoop-2.6.5.tar.gz
[hadoop@bai1 ~]$ mv hadoop-2.6.5 service/hadoop
[hadoop@bai1 ~]$ vim .bash_profile
#在刚才添加的JAVA_HOME和ZK_HOME后追加
export HADOOP_HOME=/home/hadoop/service/hadoop
export PATH=$PATH:$HOME/.local/bin:$HOME/bin:${JAVA_HOME}/bin:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:$ZK_HOME/bin[hadoop@bai1 ~]$ source .bash_profile
[hadoop@bai1 ~]$ hadoop version
Hadoop 2.6.5
Subversion https://github.com/apache/hadoop.git -r e2a9fe0r6t252czf2ebf1454405577650f113497
Compiled by sjlee on 2016-10-02T23:43Z
Compiled with protoc 2.5.0
From source with checksum f05v0qa095a395faa9de2j7ba5j954
This command was run using /home/hadoop/service/hadoop/share/hadoop/common/hadoop-common-2.6.5.jar
3、确认环境
[hadoop@bai1 ~]$ mkdir -p service/hadoop/data/ha/{jn,tmp}
[hadoop@bai1 ~]$ mkdir -p service/hadoop/data/hadoop-yarn/staging/history/{done,done_intermediate}#再次到三台服务器上确认环境没有问题，可以免密登录
[hadoop@bai1 ~]$ java -version
java version "1.8.0_111"
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)[hadoop@bai1 ~]$ auto-zk.sh status #可以收集到信息，并且是启动状态[hadoop@bai1 ~]$ hadoop version
Hadoop 2.6.5
Subversion https://github.com/apache/hadoop.git -r e2a9fe0r6t252czf2ebf1454405577650f113497
Compiled by sjlee on 2016-10-02T23:43Z
Compiled with protoc 2.5.0
From source with checksum f05v0qa095a395faa9de2j7ba5j954
This command was run using /home/hadoop/service/hadoop/share/hadoop/common/hadoop-common-2.6.5.jar
四、Hadoop配置
1、hadoop变量文件配置
~/service/hadoop/etc/hadoop/hadoop-env.sh
export JAVA_HOME=/home/hadoop/service/jdk1.8
~/service/hadoop/etc/hadoop/mapred-env.sh
export JAVA_HOME=/home/hadoop/service/jdk1.8
~/service/hadoop/etc/hadoop/yarn-env.sh
export JAVA_HOME=/home/hadoop/service/jdk1.8
2、hadoop配置文件配置
~/service/hadoop/etc/hadoop/core-site.xml
<configuration><!-- 把两个NameNode的地址组装成一个集群mycluster --><property><name>fs.defaultFS</name><value>hdfs://mycluster</value></property><!-- 指定hadoop运行时产生文件的存储目录 --><property><name>hadoop.tmp.dir</name><value>/home/hadoop/service/hadoop/data/ha/tmp</value></property><!-- 指定ZKFC故障自动切换转移 --><property><name>ha.zookeeper.quorum</name><value>bai1:2181,bai2:2181,bai3:2181</value></property><!-- 开启httpfs --><property><name>hadoop.proxyuser.root.groups</name><value>*</value></property><!-- 开启httpfs --><property><name>hadoop.proxyuser.root.hosts</name><value>*</value></property>
</configuration>
~/service/hadoop/etc/hadoop/hdfs-site.xml
<configuration><!-- 设置dfs副本数，默认3个 --><property><name>dfs.replication</name><value>3</value></property><!-- 完全分布式集群名称 --><property><name>dfs.nameservices</name><value>mycluster</value></property><!-- 集群中NameNode节点都有哪些 --><property><name>dfs.ha.namenodes.mycluster</name><value>nn1,nn2</value></property><!-- nn1的RPC通信地址 --><property><name>dfs.namenode.rpc-address.mycluster.nn1</name><value>bai1:8020</value></property><!-- nn2的RPC通信地址 --><property><name>dfs.namenode.rpc-address.mycluster.nn2</name><value>bai2:8020</value></property><!-- nn1的http通信地址 --><property><name>dfs.namenode.http-address.mycluster.nn1</name><value>bai1:50070</value></property><!-- nn2的http通信地址 --><property><name>dfs.namenode.http-address.mycluster.nn2</name><value>bai2:50070</value></property><!-- 指定NameNode元数据在JournalNode上的存放位置 --><property><name>dfs.namenode.shared.edits.dir</name><value>qjournal://bai1:8485;bai2:8485;bai3:8485/mycluster</value></property><!-- 配置隔离机制，即同一时刻只能有一台服务器对外响应 --><property><name>dfs.ha.fencing.methods</name><value>sshfence</value></property><!-- 使用隔离机制时需要ssh无秘钥登录 --><property><name>dfs.ha.fencing.ssh.private-key-files</name><value>/home/hadoop/.ssh/id_rsa</value></property><!-- 声明journalnode服务器存储目录 --><property><name>dfs.journalnode.edits.dir</name><value>/home/hadoop/service/hadoop/data/ha/jn</value></property><!-- 关闭权限检查 --><property><name>dfs.permissions.enable</name><value>false</value></property><!-- 访问代理类：client，mycluster，active配置失败自动切换实现方式 --><property><name>dfs.client.failover.proxy.provider.mycluster</name><value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value></property><!-- 配置自动故障转移 --><property><name>dfs.ha.automatic-failover.enabled</name><value>true</value></property><!-- 开启Restful服务 --><property><name>dfs.webhdfs.enabled</name><value>true</value></property><!-- https://issues.apache.org/jira/browse/HDFS-9274 --><property><name>dfs.datanode.directoryscan.throttle.limit.ms.per.sec</name><value>1000</value></property><!-- 数据传输线程个数 --><property><name>dfs.datanode.max.transfer.threads</name><value>8192</value></property>
</configuration>
~/service/hadoop/etc/hadoop/mapred-site.xml 没有这个文件的，将mapred-site.xml.template复制成mapred-site.xml
<configuration><!-- 指定mr框架为yarn方式 --><property><name>mapreduce.framework.name</name><value>yarn</value></property><!-- 指定mr历史服务器主机,端口 --><property><name>mapreduce.jobhistory.address</name><value>bai1:10020</value></property><!-- 指定mr历史服务器WebUI主机,端口 --><property><name>mapreduce.jobhistory.webapp.address</name><value>bai1:19888</value></property><!-- 历史服务器的WEB UI上最多显示20000个历史的作业记录信息 --><property><name>mapreduce.jobhistory.joblist.cache.size</name><value>20000</value></property><!-- MR作业在提交时所使用的临时目录 --><property><name>yarn.app.mapreduce.am.staging-dir</name><value>/home/hadoop/service/hadoop/data/hadoop-yarn/staging</value></property><!--配置作业运行日志 --><property><name>mapreduce.jobhistory.done-dir</name><value>${yarn.app.mapreduce.am.staging-dir}/history/done</value></property><!-- MapReduce作业产生的日志存放位置 --><property><name>mapreduce.jobhistory.intermediate-done-dir</name><value>${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate</value></property>
</configuration>
~/service/hadoop/etc/hadoop/yarn-site.xml
<configuration><!-- reducer获取数据的方式 --><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value></property><!--启用resourcemanager ha--><property><name>yarn.resourcemanager.ha.enabled</name><value>true</value></property><!--声明两台resourcemanager的地址--><property><name>yarn.resourcemanager.cluster-id</name><value>rmCluster</value></property><property><name>yarn.resourcemanager.ha.rm-ids</name><value>rm1,rm2</value></property><property><name>yarn.resourcemanager.hostname.rm1</name><value>bai2</value></property><property><name>yarn.resourcemanager.hostname.rm2</name><value>bai3</value></property><!--指定zookeeper集群的地址--><property><name>yarn.resourcemanager.zk-address</name><value>bai1:2181,bai2:2181,bai3:2181</value></property><!--启用自动恢复--><property><name>yarn.resourcemanager.recovery.enabled</name><value>true</value></property><!--指定resourcemanager的状态信息存储在zookeeper集群--><property><name>yarn.resourcemanager.store.class</name><value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value></property>
</configuration>
service/hadoop/etc/hadoop/slaves
bai1
bai2
bai3
配置方面到此基本就结束了
五、启动测试
1、初始化集群
journalnode
#三台机器都要有journalnode，下面两个命令作用一样，选一个执行
[hadoop@bai1 ~]$ hadoop-daemon.sh start journalnode #单独启动一台journalnode
[hadoop@bai1 ~]$ hadoop-daemons.sh start journalnode #在一台上执行，启动集群的journalnode
namenode
执行namenode的命令要分开来
[hadoop@bai1 ~]$ hadoop-daemon.sh start namenode #启动一台主节点#主节点启动成功后在备用节点执行
[hadoop@bai2 ~]$ hdfs namenode -bootstrapStandby
DFSZKFailoverController (ZKFC)
#确保zookeeper集群启动状态
[hadoop@bai1 ~]$ hdfs zkfc -formatZK
到此没有报错的话集群初始化就已经完成了
2、脚本管理
在这里为了方便大家可以使用脚本来管理集群，当然hadoop本身也有一键启动集群的脚本
[hadoop@bai1 ~]$ vim service/hadoop/bin/auto-hdp.sh
#!/bin/bash
sbindir=$HADOOP_HOME/sbin
nodelist=(bai1 bai2 bai3) #改成自己的主机名case $1 in(start)ssh ${nodelist[0]} $sbindir/$1-dfs.shssh ${nodelist[1]} $sbindir/$1-yarn.shssh ${nodelist[2]} $sbindir/yarn-daemon.sh $1 resourcemanagerssh ${nodelist[0]} $sbindir/mr-jobhistory-daemon.sh $1 historyserver;;(stop)ssh ${nodelist[0]} $sbindir/mr-jobhistory-daemon.sh $1 historyserverssh ${nodelist[1]} $sbindir/$1-yarn.shssh ${nodelist[2]} $sbindir/yarn-daemon.sh $1 resourcemanagerssh ${nodelist[0]} $sbindir/$1-dfs.sh;;(status)for node in ${nodelist[@]}; doecho -e "\nHost [$node]："ssh $node "/home/hadoop/service/jdk1.8/bin/jps|sort|grep -v Jps"done;;
esac
3、启动集群测试
3.1、脚本管理
auto-hdp.sh start
bai1: starting namenode, logging to /home/hadoop/service/hadoop/logs/hadoop-hadoop-namenode-bai1.out
bai2: starting namenode, logging to /home/hadoop/service/hadoop/logs/hadoop-hadoop-namenode-bai2.out
bai2: starting datanode, logging to /home/hadoop/service/hadoop/logs/hadoop-hadoop-datanode-bai2.out
bai1: starting datanode, logging to /home/hadoop/service/hadoop/logs/hadoop-hadoop-datanode-bai1.out
bai3: starting datanode, logging to /home/hadoop/service/hadoop/logs/hadoop-hadoop-datanode-bai3.out
Starting journal nodes [bai1 bai2 bai3]
bai3: starting journalnode, logging to /home/hadoop/service/hadoop/logs/hadoop-hadoop-journalnode-bai3.out
bai1: starting journalnode, logging to /home/hadoop/service/hadoop/logs/hadoop-hadoop-journalnode-bai1.out
bai2: starting journalnode, logging to /home/hadoop/service/hadoop/logs/hadoop-hadoop-journalnode-bai2.out
Starting ZK Failover Controllers on NN hosts [bai1 bai2]
bai1: starting zkfc, logging to /home/hadoop/service/hadoop/logs/hadoop-hadoop-zkfc-bai1.out
bai2: starting zkfc, logging to /home/hadoop/service/hadoop/logs/hadoop-hadoop-zkfc-bai2.out
starting yarn daemons
starting resourcemanager, logging to /home/hadoop/service/hadoop/logs/yarn-hadoop-resourcemanager-bai2.out
bai2: starting nodemanager, logging to /home/hadoop/service/hadoop/logs/yarn-hadoop-nodemanager-bai2.out
bai1: starting nodemanager, logging to /home/hadoop/service/hadoop/logs/yarn-hadoop-nodemanager-bai1.out
bai3: starting nodemanager, logging to /home/hadoop/service/hadoop/logs/yarn-hadoop-nodemanager-bai3.out
starting resourcemanager, logging to /home/hadoop/service/hadoop/logs/yarn-hadoop-resourcemanager-bai3.out
starting historyserver, logging to /home/hadoop/service/hadoop/logs/mapred-hadoop-historyserver-bai1.out
auto-hdp.sh status
Host [bai1]：
290172 NameNode
290280 DataNode
290478 JournalNode
290670 DFSZKFailoverController
290777 NodeManager
290943 JobHistoryServerHost [bai2]：
802459 NameNode
802583 DataNode
802752 JournalNode
802947 DFSZKFailoverController
803103 ResourceManager
803236 NodeManagerHost [bai3]：
353827 DataNode
353902 JournalNode
354025 NodeManager
354141 ResourceManager
auto-hdp.sh stop
stopping historyserver
stopping yarn daemons
stopping resourcemanager
bai2: stopping nodemanager
bai1: stopping nodemanager
bai3: stopping nodemanager
no proxyserver to stop
stopping resourcemanager
Stopping namenodes on [bai1 bai2]
bai2: stopping namenode
bai1: stopping namenode
bai2: stopping datanode
bai1: stopping datanode
bai3: stopping datanode
Stopping journal nodes [bai1 bai2 bai3]
bai1: stopping journalnode
bai3: stopping journalnode
bai2: stopping journalnode
Stopping ZK Failover Controllers on NN hosts [bai1 bai2]
3.2、namenode测试
这里我们测试一下集群的高可用性
#之前定义的nn1为bai1，nn2为bai2
[hadoop@bai1 ~]$ hdfs haadmin -getServiceState nn1
active
[hadoop@bai1 ~]$ hdfs haadmin -getServiceState nn2
standby#bai2为备用状态，bai1是存活状态，杀掉bai1，查看bai2
[hadoop@bai1 ~]$ jps
291299 Jps
290172 NameNode
290280 DataNode
290478 JournalNode
290670 DFSZKFailoverController
290777 NodeManager
290943 JobHistoryServer
[hadoop@bai1 ~]$ kill -9 290172#杀掉nn1再来看
[hadoop@bai1 ~]$ hdfs haadmin -getServiceState nn1
21/12/14 10:55:54 INFO ipc.Client: Retrying connect to server: bai1/10.20.123.1:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixe
dSleep(maxRetries=1, sleepTime=1000 MILLISECONDS)Operation failed: Call From bai1/10.20.123.1 to bai1:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused[hadoop@bai1 ~]$ hdfs haadmin -getServiceState nn2
active #看到nn2已经成为存活状态，nn1已经挂掉了，那我们再次启动nn1会是什么结果呢[hadoop@bai1 ~]$ hadoop-daemon.sh start namenode
[hadoop@bai1 ~]$ hdfs haadmin -getServiceState nn1
standby #nn1为备用
[hadoop@bai1 ~]$ hdfs haadmin -getServiceState nn2
active #nn2为主节点
3.3、resourcemanager状态测试
[hadoop@bai1 ~]$ service/hadoop/bin/yarn rmadmin -getServiceState rm1
active #rm1为bai2，存活状态
[hadoop@bai1 ~]$ service/hadoop/bin/yarn rmadmin -getServiceState rm2
standby #rm2为bai3，备用状态
脚本可以正常的使用，将集群启动起来访问几个web页面测试一下
4、页面访问
bai1:50070
bai2:50070
bai1:19888
bai2:8088
Hadoop分布式集群相关推荐
- hadoop分布式集群搭建
hadoop集群搭建前的准备(一定要读):https://blog.51cto.com/14048416/2341450 hadoop分布式集群搭建: 1. 集群规划: 2.具体步骤: (1)上传安装 ...
- 【转】Hadoop分布式集群搭建hadoop2.6+Ubuntu16.04
https://www.cnblogs.com/caiyisen/p/7373512.html 前段时间搭建Hadoop分布式集群,踩了不少坑,网上很多资料都写得不够详细,对于新手来说搭建起来会遇到很 ...
- Hadoop分布式集群搭建hadoop2.6+Ubuntu16.04
前段时间搭建Hadoop分布式集群,踩了不少坑,网上很多资料都写得不够详细,对于新手来说搭建起来会遇到很多问题.以下是自己根据搭建Hadoop分布式集群的经验希望给新手一些帮助.当然,建议先把HDFS ...
- 一、史上最强hadoop分布式集群的搭建
史上最强hadoop分布式集群的搭建 @Author:by Runsen @data:2020-02-23 原创文章,禁止转载 人生最重要的不是所站的位置,而是内心所朝的方向.只要我在每篇博文中写得自 ...
- 搭建hadoop分布式集群
搭建hadoop分布式集群 ~ jdk安装 ip修改 vi /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE=eth0 BOOTPROTO=none H ...
- Hadoop分布式集群的安装与部署实训总结报告
目录 前言 一.Hadoop平台框介绍 1.Hadoop的架构 2.HDFS:遵循主从架构,它具有以下元素. 2.1 名称节点 -Namenode 2.2 数据节点 - Datanode 2.3 块 ...
- Hadoop分布式集群搭建以及案例运行-fs操作
Hadoop分布式集群搭建案例步骤(也可以叫分布式文件系统) 一:创建分布式集群环境 二:设置静态ip以及主机名与映射关系 三:创建用户.配置SSH无密登录 四:子机dn操作 五:配置主机jdk.ha ...
- Hadoop分布式集群搭建完全详细教程
Hadoop分布式集群环境搭建步骤如下 实验环境: 系统:win7 内存:8G(因要开虚拟机,内存建议不低于8G) 硬盘:建议固态 虚拟机:VMware 12 Linux:Centos 7 jdk1. ...
- Hadoop分布式集群搭建完全教程
Hadoop分布式集群环境搭建步骤如下 实验环境: 系统:win7 内存:8G(因要开虚拟机,内存建议不低于8G) 硬盘:建议固态 虚拟机:VMware 12 Linux:Centos 7 jdk1. ...
- Hadoop分布式集群安装在三台Ubuntu系统电脑
前段时间搭建Hadoop分布式集群,踩了不少坑,网上很多资料都写得不够详细,对于新手来说搭建起来会遇到很多问题.以下是自己根据搭建Hadoop分布式集群的经验希望给新手一些帮助.当然,建议先把HDFS ...
最新文章
- Chapter 2 Open Book——8
- sql over的作用及用法
- 解决TextView排版混乱或者自动换行的问题
- PHP cURL可以在单个请求中检索响应标头和正文吗？
- nginx指定配置文件启动
- 【设计模式】策略模式 ( 简介 | 适用场景 | 优缺点 | 代码示例 )
- 使用Myeclipse实现快速调试
- python安装环境配置linux_linux下安装python环境
- Pyinstall打包多个python脚本，包含DLL文件
- Java根据学号提取班级_学生成绩管理系统 1. 能够实现根据以下关键字查询：学生姓名 、学号、班级、课 联合开发网 - pudn.com...
- json web token没有哪个成分_【分享项目】给你看看我们公司的登录认证是怎么做的？！（SpringBoot+Shiro+Token+Redis）...
- 11.14 模拟：总结
- 测试Markdown
- Fedora/CentOS7/RedHat7关闭图形桌面开启文本界面
- python编译成c代码_python如何调用c编译好可执行程序
- 对当今流行的比赛评分系统的简单分类以及《评委计分系统-双屏版》的类别
- 如何利用百度知道推广网站
- Web前端开发 -- HTML基础（2）
- 校招----深信服测试笔经面经
- 计算机软考地址,各省市计算机软考分数查询地址