大数据学习笔记二：Ubuntu/Debian 下安装大数据框架Hadoop

文章目录

安装Java
为Hadoop创建用户
安装Hadoop
配置Hadoop
- 配置环境变量
- 设置配置文件
- 格式化namenode
启动hadoop集群
访问hadoop集群

大数据学习系列文章：大数据-博客专栏

在学习极客时间专栏：《从0开始学大数据》，但专栏中介绍的都是粗略的理论知识，并没有实践。
Hadoop官网：Apache Hadoop

安装Java

Hadoop基于Java，需要先安装Java。
可以参考这篇文章：Linux: debian/ubuntu下安装和使用Java 11

sudo apt install default-jdk

-> # java --version
openjdk 11.0.4 2019-07-16
OpenJDK Runtime Environment (build 11.0.4+11-post-Ubuntu-1ubuntu218.04.3)
OpenJDK 64-Bit Server VM (build 11.0.4+11-post-Ubuntu-1ubuntu218.04.3, mixed mode, sharing)

为Hadoop创建用户

-> # adduser hadoop
Adding user `hadoop' ...
Adding new group `hadoop' (1001) ...
Adding new user `hadoop' (1001) with group `hadoop' ...
Creating home directory `/home/hadoop' ...
Copying files from `/etc/skel' ...
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
Changing the user information for hadoop
Enter the new value, or press ENTER for the defaultroot@ubuntu [16:08:32] [/home/ubuntu]
-> # su - hadoop hadoop@ubuntu:~$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
Generating public/private rsa key pair.
Created directory '/home/hadoop/.ssh'.
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:D7aK2tuZccJJCdNQrRbLqoMiTLTg2qsbzYQ89tOfvDs hadoop@ubuntu
The key's randomart image is:
+---[RSA 2048]----+
|    ....         |
|     o. .        |
|    o..+         |
|oo   o=.         |
|=+o  oo S        |
|.Bo oo o +       |
|=oo+ .= o .      |
|=o+..ooE.        |
|=oo++.=*+        |
+----[SHA256]-----+hadoop@ubuntu:~$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
hadoop@ubuntu:~$ chmod 0600 ~/.ssh/authorized_keys

安装Hadoop

不要切换用户，继续用新创建的hadoop用户
从官网下载：Apache Hadoop Release
我选择3.1.3版本，毕竟自己用，越新越好。

-> % wget http://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-3.1.3/hadoop-3.1.3.tar.gz
--2019-12-08 15:46:32--  http://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-3.1.3/hadoop-3.1.3.tar.gz
Resolving mirrors.tuna.tsinghua.edu.cn (mirrors.tuna.tsinghua.edu.cn)... 101.6.8.193, 2402:f000:1:408:8100::1
Connecting to mirrors.tuna.tsinghua.edu.cn (mirrors.tuna.tsinghua.edu.cn)|101.6.8.193|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 338075860 (322M) [application/octet-stream]
Saving to: ‘hadoop-3.1.3.tar.gz’hadoop-3.1.3.tar.gz              100%[========================================================>] 322.41M  8.15MB/s    in 43s     2019-12-08 15:47:16 (7.50 MB/s) - ‘hadoop-3.1.3.tar.gz’ saved [338075860/338075860]FINISHED --2019-12-08 15:47:16--
Total wall clock time: 43s
Downloaded: 1 files, 322M in 43s (7.50 MB/s)

解压：

-> % tar xzf hadoop-3.1.3.tar.gz
-> % mv hadoop-3.1.3 /opt/hadoop

配置Hadoop

配置环境变量

进行配置，修改~/.bashrc，最后添加：

export HADOOP_HOME=/home/hadoop/hadoop
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

生效：

hadoop@ubuntu:~$ source ~/.bashrc

设置JAVA_HOME环境变量：

nano $HADOOP_HOME/etc/hadoop/hadoop-env.sh

# Technically, the only required environment variable is JAVA_HOME.
# All others are optional.  However, the defaults are probably not
# preferred.  Many sites configure these options outside of Hadoop,
# such as in /etc/profile.d# The java implementation to use. By default, this environment
# variable is REQUIRED on ALL platforms except OS X!
export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-arm64    ##设置为自己的Java路径

设置配置文件

hadoop@ubuntu:~/hadoop$ cd $HADOOP_HOME/etc/hadoop

nano core-site.xml

<configuration>
<property><name>fs.default.name</name><value>hdfs://localhost:9000</value>
</property>
</configuration>

nano hdfs-site.xml

<configuration>
<property><name>dfs.replication</name><value>1</value>
</property><property><name>dfs.name.dir</name><value>file:///home/hadoop/hadoopdata/hdfs/namenode</value>
</property><property><name>dfs.data.dir</name><value>file:///home/hadoop/hadoopdata/hdfs/datanode</value>
</property>
</configuration>

nano mapred-site.xml

<configuration><property><name>mapreduce.framework.name</name><value>yarn</value></property>
</configuration>

nano yarn-site.xml

<configuration><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value></property>
</configuration>

格式化namenode

hadoop@ubuntu:~/hadoop/etc/hadoop$ cd ~
hadoop@ubuntu:~$ hdfs namenode -format
2019-12-08 16:43:26,266 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = ubuntu/192.168.123.165
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 3.1.3
...............

启动hadoop集群

hadoop@ubuntu:~/hadoop$ cd $HADOOP_HOME/sbinhadoop@ubuntu:~/hadoop/sbin$ ./start-dfs.sh
Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [ubuntu]
ubuntu: Warning: Permanently added 'ubuntu,192.168.123.165' (ECDSA) to the list of known hosts.
2019-12-08 16:46:50,578 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicablehadoop@ubuntu:~/hadoop/sbin$ ./start-yarn.sh
Starting resourcemanager
Starting nodemanagers

访问hadoop集群

hadoop@ubuntu:~/hadoop/sbin$ curl 127.0.0.1:9870
<!--Licensed to the Apache Software Foundation (ASF) under one or morecontributor license agreements.  See the NOTICE file distributed withthis work for additional information regarding copyright ownership.The ASF licenses this file to You under the Apache License, Version 2.0(the "License"); you may not use this file except in compliance withthe License.  You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.
-->
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN""http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="REFRESH" content="0;url=dfshealth.html" />
<title>Hadoop Administration</title>
</head>
</html>

参考：

How to Setup Hadoop on Ubuntu 18.04 & 16.04 LTS – TecAdmin

大数据学习笔记二：Ubuntu/Debian 下安装大数据框架Hadoop相关推荐

Django学习笔记-1 在Anaconda下安装和创建Django项目
一.创建虚拟环境 1.打开Anaconda prompt 2.创建虚拟环境 conda create -n mydjango_env 3.查看以及创建的虚拟环境 conda env list 会展示如 ...
phpst安装memcache扩展_在 Ubuntu/Debian 下安装 PHP7.3 教程
介绍最近的 PHP 7.3.0 已经在 2018 年12月6日发布 GA,大家已经可以开始第一时间体验新版本了,这里先放出 PHP7.3 安装的教程以便大家升级. 适用系统: Ubuntu 18. ...
oracle plsql update两个字段_Oracle 学习笔记（Windows 环境下安装 + PL/SQL）
Oracle 安装.PL/SQL 配置使用前言:因更换机械硬盘为 SSD 固态硬盘装了新 Windows 7 系统,需要重新搭建开发环境,把 Oracle 安装过程和 PL/SQL 配置使用做下笔记 ...
ubunntu安装php7.0_在 Ubuntu/Debian 下安装 PHP7.3 教程
介绍最近的 PHP 7.3.0 已经在 2018 年12月6日发布 GA,大家已经可以开始第一时间体验新版本了,这里先放出 PHP7.3 安装的教程以便大家升级. 适用系统: Ubuntu 18. ...
Maven学习笔记(二) ：Maven的安装与配置
在Windows上安装Maven: 1. 首先检查安装JDK 通过命令行运行命令:echo %JAVA_HOME%和java -version,能够查看当前java的安装文件夹及java的版本号 ...
RabbitMq学习笔记001---RabbitMq在Windows下安装配置
rabbitMQ是一个在AMQP协议标准基础上完整的,可服用的企业消息系统.它遵循Mozilla Public License开源协议,采用 Erlang 实现的工业级的消息队列(MQ)服务器,Rab ...
【DSP学习笔记】win10环境下安装CCS5.5
学习DSP,需要安装ccs5,自己装了好几次,目前已经试过如下版本. 6.0以上的只有C55xx系列,并没有C54xx系列.支持C54xx系列的最高版本是CCS5.5.支持win10的版本是从CCS6 ...
大数据学习笔记：Hadoop生态系统
文章目录一.Hadoop是什么二.Hadoop生态系统图三.Hadoop生态圈常用组件 (一)Hadoop (二)HDFS (三)MapReduce (四)Hive (五)Hbase (六)Zo ...
使用Excel分析数据学习笔记之二分类与混淆矩阵
使用Excel分析数据学习笔记之二分类与混淆矩阵混淆矩阵的构成: e.g.1:Bombers and seagulls 案例背景混淆矩阵如何根据混淆矩阵得到ROC曲线? 如何设定最佳阈值(op ...