一、将数据从mysql导入 HDFS

sqoop import --connect jdbc:mysql://192.168.76.1:3306/workflow --username root --password a --table project

报错

tool.ImportTool: Encountered IOException running import job: java.io.IOException: No columns to generate for ClassWriter
分析: jdbc驱动问题，注意: mysql版本及jdbc驱动版本

请使用 mysql-connector-java-5.1.46.jar，将它复制到 sqoop/lib下.

日志分析

19/07/31 21:27:17 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `project` AS t LIMIT 1
19/07/31 21:27:17 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `project` AS t LIMIT 1
19/07/31 21:27:17 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/local/hadoop275
Note: /tmp/sqoop-root/compile/904dde1997bf14585ed20790f49bf50f/project.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
19/07/31 21:27:21 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/904dde1997bf14585ed20790f49bf50f/project.jar
19/07/31 21:27:21 WARN manager.MySQLManager: It looks like you are importing from mysql.
19/07/31 21:27:21 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
19/07/31 21:27:21 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
19/07/31 21:27:21 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
19/07/31 21:27:21 INFO mapreduce.ImportJobBase: Beginning import of project
19/07/31 21:27:21 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
19/07/31 21:27:22 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
19/07/31 21:27:22 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.76.144:18040
19/07/31 21:27:34 INFO db.DBInputFormat: Using read commited transaction isolation
19/07/31 21:27:34 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(`id`), MAX(`id`) FROM `project`
19/07/31 21:27:34 INFO db.IntegerSplitter: Split size: 0; Num splits: 4 from: 1 to: 3
19/07/31 21:27:35 INFO mapreduce.JobSubmitter: number of splits:3
19/07/31 21:27:35 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1564545564450_0001
19/07/31 21:27:36 INFO impl.YarnClientImpl: Submitted application application_1564545564450_0001
19/07/31 21:27:36 INFO mapreduce.Job: The url to track the job: http://master:18088/proxy/application_1564545564450_0001/
19/07/31 21:27:36 INFO mapreduce.Job: Running job: job_1564545564450_0001
19/07/31 21:27:50 INFO mapreduce.Job: Job job_1564545564450_0001 running in uber mode : false
19/07/31 21:27:50 INFO mapreduce.Job:  map 0% reduce 0%
19/07/31 21:27:58 INFO mapreduce.Job:  map 33% reduce 0%
19/07/31 21:28:07 INFO mapreduce.Job:  map 67% reduce 0%
19/07/31 21:28:08 INFO mapreduce.Job:  map 100% reduce 0%
19/07/31 21:28:08 INFO mapreduce.Job: Job job_1564545564450_0001 completed successfully
19/07/31 21:28:08 INFO mapreduce.Job: Counters: 31File System CountersFILE: Number of bytes read=0FILE: Number of bytes written=418683FILE: Number of read operations=0FILE: Number of large read operations=0FILE: Number of write operations=0HDFS: Number of bytes read=295HDFS: Number of bytes written=180HDFS: Number of read operations=12HDFS: Number of large read operations=0HDFS: Number of write operations=6Job Counters Killed map tasks=1Launched map tasks=3Other local map tasks=3Total time spent by all maps in occupied slots (ms)=36242Total time spent by all reduces in occupied slots (ms)=0Total time spent by all map tasks (ms)=36242Total vcore-milliseconds taken by all map tasks=36242Total megabyte-milliseconds taken by all map tasks=37111808Map-Reduce FrameworkMap input records=3Map output records=3Input split bytes=295Spilled Records=0Failed Shuffles=0Merged Map outputs=0GC time elapsed (ms)=554CPU time spent (ms)=3130Physical memory (bytes) snapshot=329420800Virtual memory (bytes) snapshot=6247133184Total committed heap usage (bytes)=91422720File Input Format Counters Bytes Read=0File Output Format Counters Bytes Written=180
19/07/31 21:28:08 INFO mapreduce.ImportJobBase: Transferred 180 bytes in 46.5489 seconds (3.8669 bytes/sec)
19/07/31 21:28:08 INFO mapreduce.ImportJobBase: Retrieved 3 records.

测试以上导入是否成功:
hadoop dfs -ls /user/root/project

hadoop dfs -cat /user/root/project/part*

指定hdfs保存路径. /test1 没有用表名作路径
sqoop import --connect jdbc:mysql://192.168.76.1:3306/workflow --username root --password a --table project --target-dir /test1

只导入id , name列 /test2
sqoop import --connect jdbc:mysql://192.168.76.1:3306/workflow --username root --password a --table project --target-dir /test2 --columns 'id,name'

增量导入添加了数据: append
sqoop import --connect jdbc:mysql://192.168.76.1:3306/workflow --username root --password a --table project --target-dir /test3

sqoop import --connect jdbc:mysql://192.168.76.1:3306/workflow --username root --password a --driver com.mysql.jdbc.Driver --target-dir /test3 --direct --table project -m 3 --check-column id --incremental append --last-value 3

二、将MySQL数据库中的表数据导入到Hive中

Sqoop 导入关系型数据到 Hive 的过程是先导入到 hdfs，然后再 load 进入 Hive

sqoop 提供了直接将 mysql 数据导入 Hive 的功能，以下为直接导入

sqoop import --connect jdbc:mysql://192.168.76.1:3306/kuwomusic --username root --password a --table kuwo_music --target-dir /user/test --hive-import --create-hive-table --where 'id<100'

执行成功

查看Hive中数据

Sqoop将MySQL数据导入到HDFS和Hive中相关推荐

python etl工具 sqoop hive_python脚本用sqoop把mysql数据导入hive
转:https://blog.csdn.net/wulantian/article/details/53064123 用python把mysql数据库的数据导入到hive中,该过程主要是通过pytho ...
sqoop把mysql数据导入hive
环境: 软件版本备注 Ubuntu 19.10 sqoop 1.4.7 mysql 8.0.20-0ubuntu0.19.10.1 (Ubuntu) hadoop 3.1.2 hi ...
sqoop将mysql数据导入到hive指定的数据库中
本文作者:合肥工业大学管理学院钱洋 email:1563178220@qq.com 欢迎交流,禁止将本人博客直接复制下来,上传到百度文库等平台. 我们在使用hive时,经常需要建立一些库,以防止总 ...
Java将hive数据导入到hdfs_sqoop数据导入到Hdfs 或者hive
准备一张表需求将 bbs_product 表中的前100条数据导导出来只要id brand_id和 name 这3个字段数据存在 hdfs 目录 /user/xuyou/sqoop/ ...
sqoop mysql hadoop_使用sqoop将mysql数据导入到hadoop
hadoop的安装配置这里就不讲了. Sqoop的安装也很简单. 完成sqoop的安装后,可以这样测试是否可以连接到mysql(注意:mysql的jar包要放到 SQOOP_HOME/lib 下): ...
sqoop把mysql数据导入hbase-完整记录
环境: 软件版本备注 Ubuntu 19.10 sqoop 1.4.7 mysql 8.0.20-0ubuntu0.19.10.1 (Ubuntu) hbase 2.2.4 必须启动 ...
MySQL通过接口导入hive_利用Sqoop将MySQL数据导入Hive中
参考 http://www.cnblogs.com/iPeng0564/p/3215055.html http://www.tuicool.com/articles/j2yayyj http://bl ...
mysql数据导入导出 CSV格式_MySQL中导入导出CSV格式数据
原标题:MySQL中导入导出CSV格式数据预备阅读:MySQL中的主键问题和赋权限问题今天来讲一下在MySQL中导入和导出CSV格式数据的操作. CSV格式数据逗号分隔值(Comma-Separ ...
sqoop实现Mysql、Oracle与hdfs之间数据的互导
通过Sqoop实现Mysql / Oracle 与HDFS / Hbase互导数据 SQOOP的描述 SQOOP是用于对数据进行导入导出的. (1)把MySQL.Oracle等数据库中的数据导入到HD ...

Sqoop将MySQL数据导入到HDFS和Hive中

一、将数据从mysql导入 HDFS

二、将MySQL数据库中的表数据导入到Hive中

Sqoop 导入关系型数据到 Hive 的过程是先导入到 hdfs，然后再 load 进入 Hive

sqoop 提供了直接将 mysql 数据导入 Hive 的功能，以下为直接导入

Sqoop将MySQL数据导入到HDFS和Hive中相关推荐

最新文章

热门文章