淘宝双11数据分析与预测课程案例:实验环境搭配

1 软件安装清单

安装Hadoop
安装MySQL
安装Hive
安装Sqoop
安装Eclipse
安装ECharts

2 hadoop安装

hadoop在centos的伪分布式安装

3 MySql安装

下载并安装MySQL官方的Yum Repository\
wget -i -c http://dev.mysql.com/get/mysql57-community-release-el7-10.noarch.rpm
使用上面的命令直接安装Yum Repository
yum -y install mysql57-community-release-el7-10.noarch.rpm
安装MySQL服务器
yum -y install mysql-community-server
启动MySQL
systemctl start  mysqld.service
查看MySQL运行状态
systemctl status mysqld.service
关闭Mysql
systemctl stop mysqld.service

4 Hive的安装

Hive下载地址：http://mirror.bit.edu.cn/apache/hive/hive-2.3.4/

    sudo tar -zxvf ./apache-hive-2..4-bin.tar.gz -C /usr/local   # 解压到/usr/local中cd /usr/local/sudo mv apache-hive-2.3.4-bin hive       # 将文件夹名改为hivesudo chown -R hadoop hive            # 修改文件权限  hadoop为当前用户名

vim ~/.bashrc 添加如示内容
添加后
source ~/.bashrc

修改/usr/local/hive/conf下的hive-site.xml；
vim hive-site.xml;

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration><property><name>javax.jdo.option.ConnectionURL</name><value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true</value><description>JDBC connect string for a JDBC metastore</description></property><property><name>javax.jdo.option.ConnectionDriverName</name><value>com.mysql.cj.jdbc.Driver</value><description>Driver class name for a JDBC metastore</description></property><property><name>javax.jdo.option.ConnectionUserName</name><value>hivedb</value>    //一定要和你在mysql创建的用户名相同<description>username to use against metastore database</description></property><property><name>javax.jdo.option.ConnectionPassword</name><value>hivedb</value>一定要和你在mysql创建的用户名密码相同<description>password to use against metastore database</description></property>
</configuration>

4.1 安装mysql

sudo yum install mysql-community-server  //一句话解决
systemctl start mysqld //启动mysql
systemctl status mysqld //查看启动信息
grep "temporary password" /var/log/mysqld.log //查看初始密码
使用初始密码登入后，修改密码
SET PASSWORD=PASSWORD（“新密码”）;
set global validate_password_policy=LOW;  //设置安全策略

注意！不要按厦大的数据库网站来做，hive这里的设置还需要以下步骤：
首先设置密码安全策略，否则你为新用户设置密码是会出错：
set global validate_password_policy=0;
set global validate_password_length=1;
接着创建用户：
create user ‘hivedb’ identified by ‘hivedb’;这里的用户名和密码和hvie-site一定要对应起来；
grant all on . to hive@localhost identified by ‘hivedb’;#将所有数据库的所有表的所有权限赋
flush privileges; #刷新mysql系统权限关系表
退出来，进行hive元数据初始化
schematool -dbType mysql -initSchema
如果初始化失败，进入mysql把hive数据库删掉
下载mysql jdbc:https://dev.mysql.com/downloads/file/?id=480291;

tar -zxvf mysql-connector-java-8.0.13.tar.gz
cp mysql-connector-java-8.0.13/mysql-connector-java-8.0.13.jar /usr/local/hive/lib
start-all.sh #启动hadoop
hive  #启动hive

5: sqoop安装

sqoop下载：http://mirror.bit.edu.cn/apache/sqoop/1.4.7/
下载完成后在下载目录中打开shell；

sudo tar -zxvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz -C /usr/local
cd /usr/local
sudo mv sqoop-1.4.7.bin__hadoop-2.6.0 sqoop
sudo chown -R hadoop sqoop   //hadoop 为当前用户名
cd sqoop/conf/
cp sqoop-env-template.sh sqoop-env.sh
vim sqoop-env.sh
添加下面图片的内容，这些路径都是前几步软件安装的目录

vim ~/.bashrc
添加以下图片内容

source ~/.bashrc 使修改生效
将4.1节的MySQL驱动程序拷贝到$SQOOP_HOME/lib目录下
cp ~/下载/mysql-connector-java-8.0.13/mysql-connector-java-8.0.13.jar /usr/local/sqoop/lib
启动mysql
systemctl start mysqld //启动mysql
sqoop list-databases --connect jdbc:mysql://127.0.0.1:3306/ --username root -P //然后就可以测试sqoop与MySQL之间的连接是否成功：

6: eclipse在Linux上的安装

eclipse在centos的安装

7: spark安装

spark下载地址:http://spark.apache.org/downloads.html

sudo tar -zxf ~/下载/spark-1.6.2-bin-without-hadoop.tgz -C /usr/local/
cd /usr/local
sudo mv ./spark-1.6.2-bin-without-hadoop/ ./spark
sudo chown -R hadoop:hadoop ./spark
cd /usr/local/spark
cp ./conf/spark-env.sh.template ./conf/spark-env.sh
编辑spark-env.sh文件(vim ./conf/spark-env.sh)，在第一行添加以下配置信息:
export SPARK_DIST_CLASSPATH=$(/usr/local/hadoop/bin/hadoop classpath)

安装过程比较繁琐，很简单但需要耐心。