大数据中一些常用软件

一. 终端仿真软件SecureCRT
二. MySQL的制定化安装部署
三. Hadoop的伪分布式部署
- 1. 介绍：
- 2. 下载解压（安装）Hadoop源码，根据提示准备安装软件：
- 3. 安装支援软件
- 3.编译Hadoop
- 4 部署分布式

一. 终端仿真软件SecureCRT

介绍： secureCRT是一款支持ssh(ssh1,ssh2)的终端仿真程序，简单的说是windows下登陆unix或者linux的一款软件。他也支持telnet和rlogin协议。SecureCRT是一款用于连接运行包括Unix和Windows或者VMS的理想工具。通过使用内涵的VCP命令行程序可以进行加密文件的传输。
使用方法：
1. 下载，安装和注册SecureCrt软件
2. 启动CentOS，并登陆，查看本机IP
```
[root@hadoop000 Desktop]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:0C:29:90:03:0C  inet addr:192.168.137.130  Bcast:192.168.137.255  Mask:255.255.255.0
```
3. 打开SecureCrt, 点击Session Manager下的new session按钮，在弹出对话框中单击下一步。
4. 在弹出对话框的Hostname栏填如本机IP，并在Username栏中输入CentOS系统中登陆的用户名，点击下一步。
5. 在弹出窗口中自定义命名进程名称和描述，单机完成。
6. 双击创建的session名称，在弹出的对话框中输入用户密码，点击ok。
7. 出现如下界面，则配置完成。
8. 现在，就可以在windows上炒作CentOS系统，并让虚拟机在后台运行了。

二. MySQL的制定化安装部署

介绍：
首先，MySQL是一个关系数据库管理系统（Relational Database Management System：RDBMS），指包括相互联系的逻辑组织和存取这些数据的一套程序 (数据库管理系统软件)。关系数据库管理系统就是管理关系数据库，并将数据逻辑组织的系统。其次，MySQL是开源的，意味着免费和可制定化。

MySQL制定化安装

MySQL需要java的支持，先部署java，我们使用jdk-1.8版本, 在官网下载到本地主机后，使用 rz 上传到虚拟机指定目录


#查看是否已经安装java—1.8[root@hadoop001 local]# java -version
java version "1.7.0_45"
OpenJDK Runtime Environment (rhel-2.4.3.3.el6-x86_64 u45-b15)
OpenJDK 64-Bit Server VM (build 24.45-b08, mixed mode)# 务必将java安装在此目录下，如果没有则创建[root@hadoop001 local]# mkdir -p /usr/java
[root@hadoop001 local]# cd /usr/java/
[root@hadoop001 java]#


# 解压（安装）文件后发现两个问题，1. jdk的用户和用户组不是java#2. java的默认路径还是1.7版本的[root@hadoop001 java]# tar -xzvf jdk-8u45-linux-x64.gz
.......
jdk1.8.0_45/lib/ant-javafx.jar
jdk1.8.0_45/LICENSE
[root@hadoop001 java]# ll
total 169216
drwxr-xr-x. 8 uucp  143      4096 Apr 10  2015 jdk1.8.0_45
-rw-r--r--. 1 root root 173271626 May 16 11:25 jdk-8u45-linux-x64.gz
[root@hadoop001 java]# which java
/usr/bin/java# 修改归属和环境变量[root@hadoop001 java]# chown -R root:root jdk1.8.0_45
[root@hadoop001 java]# vi /etc/profile
.....
.....# PATHexport JAVA_HOME=/usr/java/jdk1.8.0_45
export PATH=$JAVA_HOME/bin:$PATH
:wq!
"/etc/profile" 83L, 1883C written
[root@hadoop001 java]# source /etc/profile# 查看归属和变量[root@hadoop001 java]# ll
total 169216
drwxr-xr-x. 8 root root      4096 Apr 10  2015 jdk1.8.0_45
-rw-r--r--. 1 root root 173271626 May 16 11:25 jdk-8u45-linux-x64.gz
[root@hadoop001 java]# which java
/usr/java/jdk1.8.0_45/bin/java
[root@hadoop001 java]# java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

安装制定的MySQL

使用 rz 将MySQL安装源放到 /usr/local下。

解压文件，创建数据库管理员，补全因制定MySQL所缺失的文件

            # 确认安装包存在且软件尚未被安装，然后解压文件并将其目录重命名为mysql[root@hadoop001 local]# lltotal 304512drwxr-xr-x. 2 root root      4096 Sep 23  2011 bindrwxr-xr-x. 2 root root      4096 Sep 23  2011 etcdrwxr-xr-x. 2 root root      4096 Sep 23  2011 gamesdrwxr-xr-x. 2 root root      4096 Sep 23  2011 includedrwxr-xr-x. 3 root root      4096 May 27 12:18 javadrwxr-xr-x. 2 root root      4096 Sep 23  2011 libdrwxr-xr-x. 2 root root      4096 Sep 23  2011 lib64drwxr-xr-x. 2 root root      4096 Sep 23  2011 libexec-rw-r--r--. 1 root root 311771412 May 16 11:38 mysql-5.6.23-linux-glibc2.5-x86_64.tar.gzdrwxr-xr-x. 2 root root      4096 Sep 23  2011 sbindrwxr-xr-x. 5 root root      4096 May 20 19:15 sharedrwxr-xr-x. 2 root root      4096 Sep 23  2011 src[root@hadoop001 local]# ps -ef|grep mysqlroot      4347  2757  0 13:10 pts/0    00:00:00 grep mysql[root@hadoop001 local]# tar -xzvf mysql-5.6.23-linux-glibc2.5-x86_64.tar.gz [root@hadoop001 local]# mv mysql-5.6.23-linux-glibc2.5-x86_64 mysql[root@hadoop001 local]# lltotal 304516drwxr-xr-x.  2 root root      4096 Sep 23  2011 bindrwxr-xr-x.  2 root root      4096 Sep 23  2011 etcdrwxr-xr-x.  2 root root      4096 Sep 23  2011 gamesdrwxr-xr-x.  2 root root      4096 Sep 23  2011 includedrwxr-xr-x.  3 root root      4096 May 27 12:18 javadrwxr-xr-x.  2 root root      4096 Sep 23  2011 libdrwxr-xr-x.  2 root root      4096 Sep 23  2011 lib64drwxr-xr-x.  2 root root      4096 Sep 23  2011 libexecdrwxr-xr-x. 13 root root      4096 May 27 13:12 mysql-rw-r--r--.  1 root root 311771412 May 16 11:38 mysql-5.6.23-linux-glibc2.5-x86_64.tar.gzdrwxr-xr-x.  2 root root      4096 Sep 23  2011 sbindrwxr-xr-x.  5 root root      4096 May 20 19:15 sharedrwxr-xr-x.  2 root root      4096 Sep 23  2011 src# 添加数据管理组dba，gid=101， 添加mysqladmin用户，uid=514#且其主组为dba，附组为root[root@hadoop001 local]# groupadd -g 101 dba[root@hadoop001 local]# useradd -u 514 -g 101 -G root -d /usr/local/mysql mysqladmin[root@hadoop001 local]# id mysqladminuid=514(mysqladmin) gid=101(dba) groups=101(dba),0(root)# 因为没有在/home下创建用户，需要将环境变量等文件复制到该用户制定的家目录下。[root@hadoop001 local]# cp /etc/skel/.* /usr/local/mysql/[root@hadoop001 local]# ll ./mysql/.bash_profile -rw-r--r--. 1 root root 176 May 27 13:18 ./mysql/.bash_profile# 因为是制定版本，需要相应的编辑相应的启动配置文件my.cnf[root@hadoop001 local]# vi /etc/my.cnf [client]port            = 3306socket          = /usr/local/mysql/data/mysql.sock[mysqld]port            = 3306socket          = /usr/local/mysql/data/mysql.sockskip-external-lockingkey_buffer_size = 256Msort_buffer_size = 2Mread_buffer_size = 2Mread_rnd_buffer_size = 4Mquery_cache_size= 32Mmax_allowed_packet = 16Mmyisam_sort_buffer_size=128Mtmp_table_size=32Mtable_open_cache = 512........:wq!# 删除mysqladmin用户下的mysql配置文件，避免/etc/my.cnf损坏#的情况下错误运行mysql[root@hadoop001 mysql]# rm -rf my.cnf

编辑数据库操作权限, 安装制定库

        # 设定 配置文件my.cnf以及mysql用户目录的归属和操作权限[root@hadoop001 local]# ll /etc/my.cnf -rw-r--r--. 1 root root 2218 May 27 13:36 /etc/my.cnf[root@hadoop001 local]# chown mysqladmin:dba /etc/my.cnf & chmod 640 /etc/my.cnf [1] 4426[1]+  Done                    chown mysqladmin:dba /etc/my.cnf[root@hadoop001 local]# ll /etc/my.cnf -rw-r-----. 1 mysqladmin dba 2218 May 27 13:36 /etc/my.cnf[root@hadoop001 local]# chown -R mysqladmin:dba /usr/local/mysql & chmod -R 755 /usr/local/mysql[1] 4463[root@hadoop001 local]# ll /usr/local/mysqltotal 168drwxr-xr-x.  2 mysqladmin dba   4096 May 27 13:12 bin-rwxr-xr-x.  1 mysqladmin dba  17987 Jan 19  2015 COPYINGdrwxr-xr-x.  3 mysqladmin dba   4096 May 27 13:11 data.....#测试一下[root@hadoop001 local]# su - mysqladmin[mysqladmin@hadoop001 ~]$ pwd/usr/local/mysql[mysqladmin@hadoop001 ~]$ mkdir arch#安装自定义的库[mysqladmin@hadoop001 ~]$ scripts/mysql_install_db --user=mysqladmin --basedir=/usr/local/mysql --datadir=/usr/local/mysql/dataInstalling MySQL system tables...2018-05-27 13:47:06

设定开机启动


#按安装提示复制server文件到相应目录[root@hadoop001 mysql]# cp support-files/mysql.server /etc/rc.d/init.d/mysql#设置开机启动[root@hadoop001 mysql]# chmod +x /etc/rc.d/init.d/mysql
[root@hadoop001 mysql]# chkconfig --add mysql
[root@hadoop001 mysql]# chkconfig --level 345 mysql on#安全启动mysql[mysqladmin@hadoop001 ~]$ ./bin/mysqld_safe &
[1] 7533
[mysqladmin@hadoop001 ~]$ 180527 14:37:08 mysqld_safe Logging to '/usr/local/mysql/data/hostname.err'.
180527 14:37:08 mysqld_safe Starting mysqld daemon with databases f#查看运行状态和端口号[mysqladmin@hadoop001 ~]$ service mysql status
MySQL running (8178)                                       [  OK  ]
[mysqladmin@hadoop001 ~]$ netstat -nlp|grep mysql
(Not all processes could be identified, non-owned process infowill not be shown, you would have to be root to see it all.)
tcp        0      0 :::3306                     :::*                        LISTEN      8178/mysqld
unix  2      [ ACC ]     STREAM     LISTENING     54853  8178/mysqld         /usr/local/mysql/data/mysql.sock# 进入mysql[mysqladmin@hadoop001 ~]$ mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.6.23-log MySQL Community Server (GPL)Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.rom /usr/local/mysql/data# 进入mysql查看库，确认安装成功mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| test               |
+--------------------+
4 rows in set (0.01 sec)

三. Hadoop的伪分布式部署

1. 介绍：

狭义的Hadoop是指Hadoop Apache提供的用于分布式计算的软件，主要模块有四个：
- Hadoop Common: 基础模块，用于支持其他模块
- Hadoop Distributed File System (HDFS™): 用于高速读取应用数据的分布式系统
- Hadoop YARN: 一个用于工作安排和资源分配的工作框架。
- Hadoop MapReduce: 基于YARN，特点是用来处理大量的数据。
广义的Hadoop指的是以Hadoop Apache为核心的软件生态圈
如上所提到的， Hadoop是一个分布式计算系统，然而，默认的情况下Hadoop却是单节点的，甚至不是一个独立的进程，而是一个依附于一个独立的java进程。
Hadoop的分布式实现有两种，一种是伪分布，即一台机器中运行多个节点；另一种是集群式，即Hadoop在多台机器中运行。此节介绍如何部署伪分布的Hadoop。
总体上， Hadoop编译有三个步骤：
- 下载Hadoop源码，查看支持其编译的环境需求，环境需要有软件和jar包量部分。
  1. 安装和配置支援软件
  2. 使用安装的maven软件来下载支援的jar包和编译Hadoop
- 之后，Hadoop的伪分布部署分四大块，既四个主要模块的的部署。

2. 下载解压（安装）Hadoop源码，根据提示准备安装软件：

安装前提条件：a. linux操作系统作为推荐的开发和生成平台；b.已安装 java 和 ssh，用于支撑hadoop的源文件的编译启动。
```
    [root@hadoop001 ~]# which java/usr/java/jdk1.8.0_45/bin/java[root@hadoop001 ~]# which ssh/usr/bin/ssh
```

建立一个目录，用于临时存放编译和部署Hadoop的安装文件，然后首先从官网下载Hadoop的源码安装文件，并解压。

    [root@hadoop001 software]# wget http://mirrors.hust.edu.cn/apache/hadoop/common/hadoop-2.8.1/hadoop-2.8.1-src.tar.gz[root@hadoop001 software]# tar -xzvf hadoop-2.8.1-src.tar.gz [root@hadoop001 software]# lltotal 35228drwxr-xr-x. 17 root root     4096 May  7 22:46 hadoop-2.8.1-src-rw-r--r--.  1 root root 36068045 May 15 10:29 hadoop-2.8.1-src.tar.gz[root@hadoop001 software]# ll hadoop-2.8.1-srctotal 220-rw-r--r--.  1 root root 15654 May  5 06:34 BUILDING.txt....-rw-rw-r--.  1 root root 20433 May  7 20:29 pom.xml...

解压后有两个文件需要特别注意，一是讲解编译需求的软件的BUILDING.txt，一是pom.xml 列出支撑Hadoop所需要的jar包。我们首先来看一下编译软件需求：

[root@hadoop001 software]# cd hadoop-2.8.1-src
[root@hadoop001 hadoop-2.8.1-src]# cat BUILDING.txt
....
Requirements:* Windows System
* JDK 1.7+
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer
* Windows SDK 7.1 or Visual Studio 2010 Professional
* Windows SDK 8.1 (if building CPU rate control for the container executor)
* zlib headers (if building native code bindings for zlib)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)
* Unix command-line tools from GnuWin32: sh, mkdir, rm, cp, tar, gzip. Thesetools must be present on your PATH.
* Python ( for generation of docs using 'mvn site')
.....

文件中有的软件不是必须的（if），有的是系统已经自带或自己已经安装了的，结合编译要求和个人需求，通过个人喜好安装必需的软件，我通过windows下载好所有软件，然后上传到虚拟机里，再逐步安装编译Hadoop。其中：.m2.tar 是支援Hadoop运行所需要的jar包，如不单独下载，Hadoop最后编译的过程会相对很久。

3. 安装支援软件

Maven的安装和配置

[root@hadoop001 software]# pwd
/opt/software
[root@hadoop001 software]# ll
total 222576
-rw-r--r--.  1 root root   8617253 May 16 08:44 apache-maven-3.3.9-bin.zip
-rw-r--r--.  1 root root   7546219 May 16 08:44 findbugs-1.3.9.zip
drwxr-xr-x. 17 root root      4096 May  7 22:46 hadoop-2.8.1-src
-rw-r--r--.  1 root root  36068045 May 15 10:29 hadoop-2.8.1-src.tar.gz
-rw-r--r--.  1 root root 173271626 May 16 11:25 jdk-8u45-linux-x64.gz
-rw-r--r--.  1 root root   2401901 May 16 08:44 protobuf-2.5.0.tar.gz
[root@hadoop001 software]# unzip apache-maven-3.3.9-bin.zip
[root@hadoop001 software]# ll -a
total 317044
drwxr-xr-x.  4 root root      4096 May 28 12:57 .
drwxr-xr-x.  4 root root      4096 May 28 11:06 ..
drwxr-xr-x.  6 root root      4096 Nov 10  2015 apache-maven-3.3.9
-rw-r--r--.  1 root root   8617253 May 16 08:44 apache-maven-3.3.9-bin.zip
-rw-r--r--.  1 root root   7546219 May 16 08:44 findbugs-1.3.9.zip
drwxr-xr-x. 17 root root      4096 May  7 22:46 hadoop-2.8.1-src
-rw-r--r--.  1 root root  36068045 May 15 10:29 hadoop-2.8.1-src.tar.gz
-rw-r--r--.  1 root root 173271626 May 16 11:25 jdk-8u45-linux-x64.gz
-rw-r--r--.  1 root root  96721446 May 16 08:46 .m2.tar.gz# 将准备好的jar包安装到家目录下（maven jar库的默认地址）[root@hadoop001 software]# mv .m2.tar.gz ~
[root@hadoop001 ~]# cd
[root@hadoop001 ~]# tar -xzvf .m2.tar.gz
[root@hadoop001 ~]# ll -S -a
total 94696
-rw-r--r--.  1 root root 96721446 May 16 08:46 .m2.tar.gz
......
.....

安装findbugs

[root@hadoop001 ~]# cd -
/opt/software
[root@hadoop001 software]# ll
total 222580
drwxr-xr-x.  6 root root      4096 Nov 10  2015 apache-maven-3.3.9
-rw-r--r--.  1 root root   8617253 May 16 08:44 apache-maven-3.3.9-bin.zip
-rw-r--r--.  1 root root   7546219 May 16 08:44 findbugs-1.3.9.zip
drwxr-xr-x. 17 root root      4096 May  7 22:46 hadoop-2.8.1-src
-rw-r--r--.  1 root root  36068045 May 15 10:29 hadoop-2.8.1-src.tar.gz
-rw-r--r--.  1 root root 173271626 May 16 11:25 jdk-8u45-linux-x64.gz
-rw-r--r--.  1 root root   2401901 May 16 08:44 protobuf-2.5.0.tar.gz
[root@hadoop001 software]# unzip findbugs-1.3.9.zip
Archive:  findbugs-1.3.9.zip

安装Protobuf


# 这个软件的安装包也是一个源码文件，它的编译安装依赖其他一些软件，# 首先安装他们[root@hadoop001 software]# yum install -y gcc gcc-c++ make cmake
...
...
Complete![root@hadoop001 software]# tar -xzvf protobuf-2.5.0.tar.gz
[root@hadoop001 software]# cd protobuf-2.5.0
将文件安装目录设置为/usr/local/protobuf
[root@hadoop001 protobuf-2.5.0]# ./configure --prefix=/usr/local/protobuf
[root@hadoop001 protobuf-2.5.0]# make && make install
....
Libraries have been installed in:/usr/local/protobuf/lib
....

安装BUILDING.txt中提到的其他文件

[root@hadoop001 ~]# others='openssl openssl-devel svn ncurses-devel zlib-devel libtool openssl openssl-devel svn ncurses-devel zlib-devel libtool'
[root@hadoop001 ~]# yum -y install $others
.....
.....

配置环境变量并测试

[root@hadoop001 ~]# vi /etc/profile# /etc/profile....
....# PATHexport JAVA_HOME=/usr/java/jdk1.8.0_45
export PATH=$JAVA_HOME/bin:$PATH
export MAVEN_HOME=/opt/software/apache-maven-3.3.9
export FINDBUGS_HOME=/opt/software/findbugs-1.3.9
export PROTOC_HOME=/usr/local/protobuf
export PATH=$PROTOC_HOME/bin:$FINDBUGS_HOME/bin:$MAVEN_HOME/bin:$PATH
:wq
"/etc/profile" 87L, 2093C written
[root@hadoop001 ~]# source /etc/profile
[root@hadoop001 ~]# which java
/usr/java/jdk1.8.0_45/bin/java
[root@hadoop001 ~]# which mvn
/opt/software/apache-maven-3.3.9/bin/mvn
[root@hadoop001 ~]# findbugs -version
1.3.9
[root@hadoop001 ~]# which protoc
/usr/local/protobuf/bin/protoc
[root@hadoop001 ~]# protoc --version
libprotoc 2.5.0

3.编译Hadoop

#打开BUILDING文件，查看编译选项
[root@hadoop001 software]# cd /opt/software/hadoop-2.8.1-src
[root@hadoop001 hadoop-2.8.1-src]# cat BUILDING.txt
....
Building distributions:Create binary distribution without native code and without documentation:$ mvn package -Pdist -DskipTests -DtarCreate binary distribution with native code and with documentation:$ mvn package -Pdist,native,docs -DskipTests -DtarCreate source distribution:$ mvn package -Psrc -DskipTestsCreate source and binary distributions with native code and documentation:$ mvn package -Pdist,native,docs,src -DskipTests -DtarCreate a local staging version of the website (in /tmp/hadoop-site)$ mvn clean site -Preleasedocs; mvn site:stage -DstagingDirectory=/tmp/hadoop-site---------------------------------------------------------------
# 编译需要时间，等候时可参考下如何部署伪分布
Installing HadoopLook for these HTML files after you build the document by the above commands.* Single Node Setup:hadoop-project-dist/hadoop-common/SingleCluster.html
....
# 确保装的软件归属正确
[root@hadoop001 software]# ll
total 222588
drwxr-xr-x.  6 root root      4096 Nov 10  2015 apache-maven-3.3.9
-rw-r--r--.  1 root root   8617253 May 16 08:44 apache-maven-3.3.9-bin.zip
drwxr-xr-x.  7 root root      4096 Aug 21  2009 findbugs-1.3.9
-rw-r--r--.  1 root root   7546219 May 16 08:44 findbugs-1.3.9.zip
drwxr-xr-x. 17 root root      4096 May  7 22:46 hadoop-2.8.1-src
-rw-r--r--.  1 root root  36068045 May 15 10:29 hadoop-2.8.1-src.tar.gz
-rw-r--r--.  1 root root 173271626 May 16 11:25 jdk-8u45-linux-x64.gz
drwxr-xr-x. 10 root root      4096 May 28 13:21 protobuf-2.5.0
-rw-r--r--.  1 root root   2401901 May 16 08:44 protobuf-2.5.0.tar.gz
[root@hadoop001 software]# ll /usr/local
total 304524
...
drwxr-xr-x.  5 root       root      4096 May 28 13:28 protobuf# 确保pom.xml这个jar包设置文件存在，开始解压
[root@hadoop001 hadoop-2.8.1-src]# ll pom.xml
-rw-rw-r--. 1 root root 20433 May  7 20:29 pom.xml
[root@hadoop001 hadoop-2.8.1-src]#  mvn clean package -Pdist,native, -DskipTests -Dtar
.....
.....
[INFO] Apache Hadoop Client ............................... SUCCESS [  8.561 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  2.054 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  5.613 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [  6.577 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [  3.880 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.040 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 32.162 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 14:02 min
[INFO] Finished at: 2018-05-29T01:17:19-04:00
[INFO] Final Memory: 130M/239M
[INFO] ------------------------------------------------------------------------
[root@hadoop001 hadoop-2.8.1-src]# cp /opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1.tar.gz /opt/software/
[root@hadoop001 software]# tar -xzvf hadoop-2.8.1.tar.gz
[root@hadoop001 software]# cd hadoop-2.8.1
[root@hadoop001 hadoop-2.8.1]# ll
total 148
drwxrwxr-x. 2 chang chang  4096 Jun  2  2017 bin #shell
drwxrwxr-x. 3 chang chang  4096 Jun  2  2017 etc  #配置文件
drwxrwxr-x. 2 chang chang  4096 Jun  2  2017 include
drwxrwxr-x. 3 chang chang  4096 Jun  2  2017 lib #库
drwxrwxr-x. 2 chang chang  4096 Jun  2  2017 libexec
-rw-rw-r--. 1 chang chang 99253 Jun  2  2017 LICENSE.txt
-rw-rw-r--. 1 chang chang 15915 Jun  2  2017 NOTICE.txt
-rw-r--r--. 1 chang chang  1366 Jun  2  2017 README.txt
drwxrwxr-x. 2 chang chang  4096 Jun  2  2017 sbin #启动/关闭Hadoop
drwxrwxr-x. 4 chang chang  4096 Jun  2  2017 share #jar文件

4 部署分布式

修改配置文件

转到配置目录，按照官网提示修改配置文件
[root@hadoop001 hadoop-2.8.1]# pwd
/opt/software/hadoop-2.8.1
[root@hadoop001 hadoop-2.8.1]# vi /etc/profile# /etc/profile....
export HADOOP_HOME=/opt/software/hadoop-2.8.1
export PATH=$HADOOP_HOME/bin:$PROTOC_HOME/bin:$FINDBUGS_HOME/bin:$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH
"/etc/profile" 88L, 2125C written
[root@hadoop001 hadoop-2.8.1]# source /etc/profile
[root@hadoop001 hadoop-2.8.1]# which hadoop
/opt/software/hadoop-2.8.1/bin/hadoop
[root@hadoop001 ~]# cd
[root@hadoop001 ~]# cd /opt/software/hadoop-2.8.1/etc/hadoop
[root@hadoop001 hadoop]# ll
total 156
....
-rw-rw-r--. 1 root root   774 Jun  2  2017 core-site.xml-rw-rw-r--. 1 root root  4666 Jun  2  2017 hadoop-env.sh-rw-rw-r--. 1 root root   775 Jun  2  2017 hdfs-site.xml-rw-rw-r--. 1 root root  4113 Jun  2  2017 mapred-queues.xml.template-rw-rw-r--. 1 root root   690 Jun  2  2017 yarn-site.xml
[root@hadoop001 hadoop]# vi core-site.xml <!-- Put site-specific property overrides in this file. --><configuration><property><name>fs.defaultFS</name><value>hdfs://localhost:9000</value></property>
</configuration>
:wq!
"core-site.xml" 24L, 884C written
[root@hadoop001 hadoop]# vi hdfs-site.xml <!-- Put site-specific property overrides in this file. --><configuration><property><name>dfs.replication</name><value>1</value>   #单节点（伪分布）为1分备份，集群分布则多台备份</property>
</configuration>
:wq
"hdfs-site.xml" 24L, 867C written# 配置JAVA_HOME到Hadoop[root@hadoop001 hadoop]# vi hadoop-env.sh
......# A string representing this instance of hadoop. $USER by default.export HADOOP_IDENT_STRING=$USER
export JAVA_HOME=/usr/java/jdk1.8.0_45/#格式化namenode[root@hadoop001 hadoop]#/opt/software/hadoop-2.8.1
[root@hadoop001 hadoop-2.8.1]# ./bin/hadoop namenode -format
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at hadoop001/192.168.137.138
************************************************************/

作为测试，我们用root用户来登陆下启动下hdfs服务，首先要将localhost要对root授信，在此基础上，进行以下操作

            [root@hadoop001 hadoop-2.8.1]# ./sbin/start-dfs.shStarting namenodes on [hadoop001]hadoop001: starting namenode, logging to /opt/software/hadoop-2.8.1/logs/hadoop-root-namenode-hadoop001.outlocalhost: starting datanode, logging to /opt/software/hadoop-2.8.1/logs/hadoop-root-datanode-hadoop001.outStarting secondary namenodes [0.0.0.0]0.0.0.0: starting secondarynamenode, logging to /opt/software/hadoop-2.8.1/logs/hadoop-root-secondarynamenode-hadoop001.out#显示dfs模块已经启动成功，可以高速接入数据[root@hadoop001 hadoop-2.8.1]# jps22153 Jps21881 DataNode21754 NameNode22042 SecondaryNameNode

生产中，更多的是普通用户的操作，并且不仅是hdfs的分布，这都需要进一步的配置，初步有普通用户授信和YARN和MapReduce的配置，这些将在Hadoop学习篇介绍。