sqoop从mysql到hive问题

ClassNotFoundException问题大概会需要在sqoop的lib目录下导入三个包， hive-common-...jar ，commons-logging-...jar, hive-exec-..*.jar ..*为版本编号，jar包从hive的lib目录下copy即可

1、报错信息如下：ERROR tool.ImportTool: Import failed: java.io.IOException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf`

19/09/21 09:28:45 ERROR hive.HiveConfig: Could not load org.apache.hadoop.hive.conf.HiveConf. Make sure HIVE_CONF_DIR is set correctly.
19/09/21 09:28:45 ERROR tool.ImportTool: Import failed: java.io.IOException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConfat org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:50)at org.apache.sqoop.hive.HiveImport.getHiveArgs(HiveImport.java:392)at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:379)at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:337)at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:241)at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:537)at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628)at org.apache.sqoop.Sqoop.run(Sqoop.java:147)at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConfat java.net.URLClassLoader.findClass(URLClassLoader.java:382)at java.lang.ClassLoader.loadClass(ClassLoader.java:424)at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)at java.lang.ClassLoader.loadClass(ClassLoader.java:357)at java.lang.Class.forName0(Native Method)at java.lang.Class.forName(Class.java:264)at org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:44)... 12 more

问题出现的原因是，sqoop安装目录中没有hive相关的包
解决方案：把hive的lib目录下hive-common-1.2.1.jar（hive版本不一致，此包版本可能会不一致，根据自己的来就行）拷贝到sqoop的lib下

[hadoop@hadoop01 lib]$ cp hive-common-1.2.1.jar /home/hadoop/app/sqoop/sqoop1.4.7/lib

2、INFO hive.HiveImport: Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/logging/log4j/spi/LoggerContextFactory

19/09/21 09:30:17 INFO hive.HiveImport: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/logging/log4j/spi/LoggerContextFactory
19/09/21 09:30:17 INFO hive.HiveImport:     at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:637)
19/09/21 09:30:17 INFO hive.HiveImport:     at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
19/09/21 09:30:17 INFO hive.HiveImport:     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
19/09/21 09:30:17 INFO hive.HiveImport:     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
19/09/21 09:30:17 INFO hive.HiveImport:     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
19/09/21 09:30:17 INFO hive.HiveImport:     at java.lang.reflect.Method.invoke(Method.java:498)
19/09/21 09:30:17 INFO hive.HiveImport:     at org.apache.hadoop.util.RunJar.run(RunJar.java:226)
19/09/21 09:30:17 INFO hive.HiveImport:     at org.apache.hadoop.util.RunJar.main(RunJar.java:141)
19/09/21 09:30:17 INFO hive.HiveImport: Caused by: java.lang.ClassNotFoundException: org.apache.logging.log4j.spi.LoggerContextFactory
19/09/21 09:30:17 INFO hive.HiveImport:     at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
19/09/21 09:30:17 INFO hive.HiveImport:     at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
19/09/21 09:30:17 INFO hive.HiveImport:     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
19/09/21 09:30:17 INFO hive.HiveImport:     at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
19/09/21 09:30:17 INFO hive.HiveImport:     ... 8 more
19/09/21 09:30:17 ERROR tool.ImportTool: Import failed: java.io.IOException: Hive exited with status 1at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:384)at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:337)at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:241)at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:537)at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628)at org.apache.sqoop.Sqoop.run(Sqoop.java:147)at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)at org.apache.sqoop.Sqoop.main(Sqoop.java:25

解决方案：把hive的lib目录下commons-logging-1.1.3.jar（hive版本不一致，此包版本可能会不一致，根据自己的来就行）拷贝到sqoop的lib下

[hadoop@hadoop01 lib]$ cp commons-logging-1.1.3.jar /home/hadoop/app/sqoop/sqoop1.4.7/lib

3、Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/hadoop/hive/shims/ShimLoader

19/09/21 09:37:43 INFO hive.HiveImport: Loading uploaded data into Hive
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hive/shims/ShimLoaderat org.apache.hadoop.hive.conf.HiveConf$ConfVars.<clinit>(HiveConf.java:368)at org.apache.hadoop.hive.conf.HiveConf.<clinit>(HiveConf.java:105)at java.lang.Class.forName0(Native Method)at java.lang.Class.forName(Class.java:264)at org.apache.sqoop.hive.HiveConfig.getHiveConf(HiveConfig.java:44)at org.apache.sqoop.hive.HiveImport.getHiveArgs(HiveImport.java:392)at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:379)at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:337)at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:241)at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:537)at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628)at org.apache.sqoop.Sqoop.run(Sqoop.java:147)at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.shims.ShimLoaderat java.net.URLClassLoader.findClass(URLClassLoader.java:382)at java.lang.ClassLoader.loadClass(ClassLoader.java:424)at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)at java.lang.ClassLoader.loadClass(ClassLoader.java:357)... 17 more

解决方案：把hive的lib目录下hive-exec-1.2.1.jar（hive版本不一致，此包版本可能会不一致，根据自己的来就行）拷贝到sqoop的lib下

[hadoop@hadoop01 lib]$ cp hive-exec-1.2.1.jar /home/hadoop/app/sqoop/sqoop1.4.7/lib

之后就可以成功运行了。

sqoop从mysql到hive问题相关推荐

sqoop从mysql导入hive字符集的问题乱码问题
sqoop从mysql导入hive字符集的问题 Mysql都是使用utf8的字符集:所以这个与Mysql的配置无直接关系:但是可以通过修改Mysql上面的表默认字符集来解决,这个是不需要修改Hive配 ...
大数据——sqoop操作mysql和hive导出导入数据
1.sqoop安装 (1)下载CDH版本的sqoop (2)解压并进行环境配置环境变量为: export SQOOP_HOME=/home/sqoop-1.4.6-cdh5.15.1 export ...
sqoop动态分区导入mysql,sqoop 导入数据到hive分区表（外表，内表）指定分区指定数据库指定表...
sqoop 导入数据到hive 1.1.导入数据到hive分区表(内表),指定分区创建hive分区表 –hive-database 指定数据库 –table 指定表 –hive-overwrite ...
sqoop把mysql数据导入hive
环境: 软件版本备注 Ubuntu 19.10 sqoop 1.4.7 mysql 8.0.20-0ubuntu0.19.10.1 (Ubuntu) hadoop 3.1.2 hi ...
sqoop动态分区导入mysql,使用sqoop import从mysql往hive含分区表中导入数据的一些注意事项...
先看下面这条语句,它实现的功能是将特定日期的数据从mysql表中直接导入hive $ sqoop import \ --connect jdbc:mysql://192.168.xx.xx:3306/ ...
用sqoop将mysql的数据导入到hive表中，原理分析
Sqoop 将 Mysql 的数据导入到 Hive 中准备Mysql 数据如图所示,准备一张表,数据随便造一些,当然我这里的数据很简单. 编写命令编写参数文件个人习惯问题,我喜欢把参数写到文件 ...
Mysql和Hive之间通过Sqoop进行数据同步
文章回顾理论大数据框架原理简介大数据发展历程及技术选型实践搭建大数据运行环境之一搭建大数据运行环境之二本地MAC环境配置 CPU数和内存大小查看CPU数 sysctl machdep. ...
【华为云技术分享】使用sqoop导入mysql数据到hive中
Sqoop 是一个数据转储工具,它能够将 hadoop HDFS 中的数据转储到关系型数据库中,也能将关系型数据库中的数据转储到 HDFS 中. Apache Sqoop,是"SQL to ...
巨坑：从Sqoop导入MySQL导入TINYINT(1)类型数据到hive（tinyint），数据为null
问题描述从Sqoop导入MySQL导入TINYINT(1)类型数据到hive(tinyint),数据为null 问题原因 Sqoop在抽取数据到Hive或者HDFS时,会自动将类型为Tinyint( ...

sqoop从mysql到hive问题

1、报错信息如下：ERROR tool.ImportTool: Import failed: java.io.IOException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf`

2、INFO hive.HiveImport: Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/logging/log4j/spi/LoggerContextFactory

3、Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/hadoop/hive/shims/ShimLoader

sqoop从mysql到hive问题相关推荐

最新文章

热门文章