十八、通过Sqoop2-Shell将HDFS中的数据导出到MySQL

一、Sqoop2提供了能够使用REST接口与Sqoop 2服务器通信的命令行shell。客户端可以在两种模式下运行——交互模式和批处理模式。批处理模式目前不支持create、update和clone命令。交互模式支持所有可用的命令。

二、Sqoop2新增了Sqoop中没有的链接器，我们主要是通过链接器的方式来实现数据的create、update和clone，简单说明一下链接器的使用过程并通过一个实例来掌握：使用链接器首先要的是相关链接器的Jar包，譬如你想要将数据从hdfs中传输到mysql，那么你至少需要使用官方提供hdfs数据传输的封装包和jdbc的相关封装包，我们首先需要创建两个链接hdfs-link和jdbc-link，并且配置好这两个链接，然后通过这两个link来创建一个job，并且配置好job，最后start这个job，Java角度来看可以把创建link看作class的对象实例，调用的jar包为class，job看作main函数，start也就是代码执行。

三、将HDFS的数据导出Mysql数据库

1、hadoop配置mapred-site.xml,这个主要是为了Sqoop2能够查看自身的各种status，譬如查看Job是否执行成功

<property><name>mapreduce.jobhistory.address</name><value>master:10020</value>
</property>

$HADOOP_HOME/sbin/mr-jobhistory-daemon.sh start historyserver

2、在Sqoop2客户端设置Sqoop2的Server,12000为默认端口

sqoop:000> set server --host master --port 12000 --webapp sqoop

3、准备HDFS和MYSQL上的数据

HDFS上的需要注意，文件夹下只放一个有数据的文件，其它的需要删除

MYSQL上的表需要注意，我这里的数据是用来进行数据分析的，我的业务的原因，不需要设置主键。
4、启动hadoop、historyserver和sqoop2

$    start-all.sh && ($HADOOP_HOME/sbin/mr-jobhistory-daemon.sh start historyserver) && (sqoop2-server start)

5、查看官方自带的connector

sqoop:000> show connector
+------------------------+---------+------------------------------------------------------------+----------------------+
|          Name          | Version |                           Class                            | Supported Directions |
+------------------------+---------+------------------------------------------------------------+----------------------+
| generic-jdbc-connector | 1.99.7  | org.apache.sqoop.connector.jdbc.GenericJdbcConnector       | FROM/TO              |
| kite-connector         | 1.99.7  | org.apache.sqoop.connector.kite.KiteConnector              | FROM/TO              |
| oracle-jdbc-connector  | 1.99.7  | org.apache.sqoop.connector.jdbc.oracle.OracleJdbcConnector | FROM/TO              |
| hdfs-connector         | 1.99.7  | org.apache.sqoop.connector.hdfs.HdfsConnector              | FROM/TO              |
| ftp-connector          | 1.99.7  | org.apache.sqoop.connector.ftp.FtpConnector                | TO                   |
| kafka-connector        | 1.99.7  | org.apache.sqoop.connector.kafka.KafkaConnector            | TO                   |
| sftp-connector         | 1.99.7  | org.apache.sqoop.connector.sftp.SftpConnector              | TO                   |
+------------------------+---------+------------------------------------------------------------+----------------------+

6、创建hdfs和jdbc的link,我的已经创建，因此使用update来修改

create link --connector hdfs-connector   #创建一个hdfs-connector链接器，必须填Name、URI和Hadoop的Conf目录，其它的根绝实际业务填写，留空直接Enter

sqoop:000> update link --name hdfs-link
Updating link with name hdfs-link
Please update link:
Name: hdfs-linkHDFS clusterURI: hdfs://master:8020
Conf directory: /hadoopeco/hadoop-2.8.5/etc/hadoop
Additional configs::
There are currently 0 values in the map:
entry#
link was successfully updated with status OK

create link --name jdbc-link  #必须填写name、driverclass、connection、username和password，其它的根据实际业务填写，留空Enter

sqoop:000> update link --name jdbc-link
Updating link with name jdbc-link
Please update link:
Name: jdbc-linkDatabase connectionDriver class: com.mysql.jdbc.Driver
Connection String: jdbc:mysql://master:3306/AP_db
Username: root
Password: **********
Fetch Size:
Connection Properties:
There are currently 0 values in the map:
entry# SQL DialectIdentifier enclose:
link was successfully updated with status OK

查看已创建的link

sqoop:000> show link
+-----------+------------------------+---------+
|   Name    |     Connector Name     | Enabled |
+-----------+------------------------+---------+
| hdfs-link | hdfs-connector         | true    |
| jdbc-link | generic-jdbc-connector | true    |
+-----------+------------------------+---------+

7、通过link来创建job

create job -f hdfs-link -t jdbc-link

sqoop:000> update job --name hdfsTojdbc
Updating job with name hdfsTojdbc
Please update job:
Name: hdfsTojdbc    #jobname，必填Input configurationInput directory:  #这个目录是天hdfs下的目录，例/tmp/user/<userid>
Override null value:
Null value: Incremental importIncremental type: 0 : NONE1 : NEW_FILES
Choose: 0           #选0，必填
Last imported date: Database targetSchema name:
Table name: t_c5eac1d455675dfbc99f6c70f7b3971f      #tablename，必填
Column names:
There are currently 0 values in the list:
element#
Staging table:
Clear stage table: 2019-02-15T12:04:49,413 INFO [UpdateThread] org.apache.sqoop.repository.JdbcRepositoryTransaction - Attempting transaction commitThrottling resourcesExtractors:
Loaders: Classpath configurationExtra mapper jars:
There are currently 0 values in the list:
element#
Job was successfully updated with status OK

8、执行Job和查看Job的状态
开始任务

sqoop:000> start job --name "hdfsTojdbc"
Submission details
Job Name: hdfsTojdbc
Server URL: http://localhost:12000/sqoop/
Created by: ljj
Creation date: 2019-02-15 12:27:46 EST
Lastly updated by: ljj
External ID: job_1550244884040_0002http://master:8088/proxy/application_1550244884040_0002/
2019-02-15 12:27:46 EST: BOOTING  - Progress is not available       #未取得进展

查看任务状态，如果没有配置mapreduce.jobhistory.address，就无法查看任务的状态，因为获取不了hdfs的任务状态

sqoop:000> status job -name "hdfsTojdbc"Submission details
Job Name: hdfsTojdbc
Server URL: http://localhost:12000/sqoop/
Created by: ljj
Creation date: 2019-02-15 12:30:30 EST
Lastly updated by: ljj
External ID: job_1550244884040_0002http://master:8088/proxy/application_1550244884040_0002/
2019-02-15 12:31:56 EST: SUCCEEDED
Counters:org.apache.hadoop.mapreduce.FileSystemCounterFILE_LARGE_READ_OPS: 0FILE_WRITE_OPS: 0HDFS_READ_OPS: 59HDFS_BYTES_READ: 47691HDFS_LARGE_READ_OPS: 0FILE_READ_OPS: 0FILE_BYTES_WRITTEN: 3090006FILE_BYTES_READ: 0HDFS_WRITE_OPS: 0HDFS_BYTES_WRITTEN: 0org.apache.hadoop.mapreduce.lib.output.FileOutputFormatCounterBYTES_WRITTEN: 0org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounterBYTES_READ: 0org.apache.hadoop.mapreduce.JobCounterTOTAL_LAUNCHED_MAPS: 9MB_MILLIS_MAPS: 846051328VCORES_MILLIS_MAPS: 826222SLOTS_MILLIS_MAPS: 826222OTHER_LOCAL_MAPS: 9MILLIS_MAPS: 826222org.apache.sqoop.submission.counter.SqoopCountersROWS_READ: 183ROWS_WRITTEN: 183org.apache.hadoop.mapreduce.TaskCounterSPILLED_RECORDS: 0MERGED_MAP_OUTPUTS: 0VIRTUAL_MEMORY_BYTES: 19212435456MAP_INPUT_RECORDS: 0SPLIT_RAW_BYTES: 2202MAP_OUTPUT_RECORDS: 183FAILED_SHUFFLE: 0PHYSICAL_MEMORY_BYTES: 2263040000GC_TIME_MILLIS: 138501CPU_MILLISECONDS: 71780COMMITTED_HEAP_BYTES: 1480065024
Job executed successfully

mysql此时数据插入成功：
mysql默认运行在safe-updates模式下，该模式不允许非主键情况下执行update和delete命令，若需要删除数据，需要接触安全模式

SET SQL_SAFE_UPDATES = 0

事后为了数据库安全务必将0修改回1

四、问题解决

务必配置mapred-site.xml，否则不单止查看不了job的状态，想通过命令“stop job -name jobname”来终止任务时因为获取不了MapReduce的状态而抛出异常