sqoop把mysql数据导入hive

环境:

软件	版本	备注
Ubuntu	19.10
sqoop	1.4.7
mysql	8.0.20-0ubuntu0.19.10.1 (Ubuntu)
hadoop	3.1.2
hive	3.0.0	之所以和hive有关系是因为需要在.bashrc中设置HCAT_HOME
accumulo	2.0.0	需要配合sqoop在.bashrc中设置ACCUMULO_HOMT

准备工作:

①lsof -i:10000确保端口存在占用

②metastore确保打开

③hiveserver2确保打开

数据导入目标:

mysql中的sqoop_hbase中的表格book导入Hive

数据集建立以[1]为准

显示所有数据库:

sqoop list-databases --connect jdbc:mysql://Desktop:3306/ --username appleyuchi --password appleyuchi

显示数据库中所有表格:

sqoop list-tables --connect jdbc:mysql://Desktop:3306/sqoop_hbase --username appleyuchi --password appleyuchi

开始传递数据:

sqoop import --connect jdbc:mysql://Desktop:3306/sqoop_hbase --username appleyuchi --password appleyuchi --table book --hive-import --create-hive-table --fields-terminated-by "\t" -m 5

log:

/home/appleyuchi/bigdata/hadoop-3.1.2/libexec/hadoop-functions.sh: 列 2358: HADOOP_ORG.APACHE.HADOOP.HBASE.UTIL.GETJAVAPROPERTY_USER：無效的變數名稱
/home/appleyuchi/bigdata/hadoop-3.1.2/libexec/hadoop-functions.sh: 列 2453: HADOOP_ORG.APACHE.HADOOP.HBASE.UTIL.GETJAVAPROPERTY_OPTS：無效的變數名稱
/home/appleyuchi/bigdata/hadoop-3.1.2/libexec/hadoop-functions.sh: 列 2358: HADOOP_ORG.APACHE.SQOOP.SQOOP_USER：無效的變數名稱
/home/appleyuchi/bigdata/hadoop-3.1.2/libexec/hadoop-functions.sh: 列 2453: HADOOP_ORG.APACHE.SQOOP.SQOOP_OPTS：無效的變數名稱
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/appleyuchi/bigdata/hadoop-3.1.2/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/appleyuchi/bigdata/apache-hive-3.0.0-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/appleyuchi/bigdata/hbase-2.2.4/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2020-05-28 13:22:47,594 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
Note: /tmp/sqoop-appleyuchi/compile/af856630222c463eee34cb9210be25ff/book.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
2020-05-28 13:22:50,084 WARN manager.MySQLManager: It looks like you are importing from mysql.
2020-05-28 13:22:50,084 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
2020-05-28 13:22:50,084 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
Hive Session ID = 5b44d7c9-8718-4936-996e-5eb4d9f3878aLogging initialized using configuration in jar:file:/home/appleyuchi/bigdata/apache-hive-3.0.0-bin/lib/hive-common-3.0.0.jar!/hive-log4j2.properties Async: true
2020-05-28 13:23:26,576 WARN session.SessionState: METASTORE_FILTER_HOOK will be ignored, since hive.security.authorization.manager is set to instance of HiveAuthorizerFactory.
OK
Time taken: 4.613 seconds
Loading data to table default.book
OK
Time taken: 1.131 seconds

beeline中查看实验结果:

0: jdbc:hive2://Desktop:10000> show tables;
INFO  : Compiling command(queryId=appleyuchi_20200528133356_e09b4c8f-4d75-4c86-810f-a2ca8cb68921): show tables
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Semantic Analysis Completed
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from deserializer)], properties:null)
INFO  : Completed compiling command(queryId=appleyuchi_20200528133356_e09b4c8f-4d75-4c86-810f-a2ca8cb68921); Time taken: 0.123 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=appleyuchi_20200528133356_e09b4c8f-4d75-4c86-810f-a2ca8cb68921): show tables
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=appleyuchi_20200528133356_e09b4c8f-4d75-4c86-810f-a2ca8cb68921); Time taken: 0.028 seconds
INFO  : OK
INFO  : Concurrency mode is disabled, not creating a lock manager
+-----------+
| tab_name  |
+-----------+
| book      |
| train     |
+-----------+
2 rows selected (0.339 seconds)

#--------------------------------------------------------------------------------------------------------------------------------------------------------

main ERROR Could not register mbeans java.security.AccessControlException: access denied ("javax.management.MBeanTrustPermission" "register")

解决办法:

$JAVA_HOME/jre/lib/security/java.policy加入:

grant{

permission javax.management.MBeanTrustPermission "register";

}

References:

[1]sqoop把mysql数据导入hbase-完整记录

sqoop把mysql数据导入hive相关推荐

python etl工具 sqoop hive_python脚本用sqoop把mysql数据导入hive
转:https://blog.csdn.net/wulantian/article/details/53064123 用python把mysql数据库的数据导入到hive中,该过程主要是通过pytho ...
MySQL通过接口导入hive_利用Sqoop将MySQL数据导入Hive中
参考 http://www.cnblogs.com/iPeng0564/p/3215055.html http://www.tuicool.com/articles/j2yayyj http://bl ...
Sqoop将MySQL数据导入到HDFS和Hive中
一.将数据从mysql导入 HDFS sqoop import --connect jdbc:mysql://192.168.76.1:3306/workflow --username root -- ...
sqoop将mysql数据导入到hive指定的数据库中
本文作者:合肥工业大学管理学院钱洋 email:1563178220@qq.com 欢迎交流,禁止将本人博客直接复制下来,上传到百度文库等平台. 我们在使用hive时,经常需要建立一些库,以防止总 ...
使用 Sqoop 将 30W+ MySQL 数据导入 Hive
本实验完成的是,使用 Sqoop 从 MySQL 导出数据到 Hive. 整体步骤分为: 初始化 MySQL 的 30W+ 数据安装配置 Sqoop 在 Hive 中初始化目标表 Sqoop 脚本实 ...
sqoop mysql hadoop_使用sqoop将mysql数据导入到hadoop
hadoop的安装配置这里就不讲了. Sqoop的安装也很简单. 完成sqoop的安装后,可以这样测试是否可以连接到mysql(注意:mysql的jar包要放到 SQOOP_HOME/lib 下): ...
sqoop把mysql数据导入hbase-完整记录
环境: 软件版本备注 Ubuntu 19.10 sqoop 1.4.7 mysql 8.0.20-0ubuntu0.19.10.1 (Ubuntu) hbase 2.2.4 必须启动 ...
使用shell脚本将mysql数据导入HIve中
使用sqoop将us_order表中的数据导入到hive中,hive的库名为exam_ods,表名叫ods_us_order,根据order_date的日期来实现分区导入数据,形成脚本. 解释 #!/ ...
使用sqoop迁移mysql数据到hive
相关参考文献: 1.原文地址:http://www.cnblogs.com/charlist/p/7122198.html 使用Sqoop能够极大简化MySQL数据迁移至Hive之流程,并降低Hado ...

sqoop把mysql数据导入hive

sqoop把mysql数据导入hive相关推荐

最新文章

热门文章