DataX之MySQL-overwrite-HIVE

1、json脚本

{"job": {"setting": {"speed": {"channel": 3},"cipher": {"sfjm": "Y","instance": "AES128"},"errorLimit": {"record": "-1"}},"content": [{"reader": {"name": "mysqlreader","parameter": {"password": "123456","username": "root","connection": [{"jdbcUrl": ["jdbc:mysql://192.168.30.20:3306/test?useUnicode=true&characterEncoding=utf8&useSSL=false&serverTimezone=Asia/Shanghai"],"table": ["test.jkmzmxx_plqd"]}],"column": ["`xm`", "`zjhm`", "`lxdh`", "`city`", "`region`", "`street`", "`community`", "`dz`", "`sjly`", "`lysj`", "`zmyy`", "`zmqzt`", "`nzzt`", "`sffk`", "`jcfkrq`"]}},"writer": {"name": "hdfswriter","parameter": {"password": "123456","username": "hadoop","jdbcUrl": "jdbc:hive2://bigcity1:2181,bigcity2:2181,bigcity3:2181/stg;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2","analyzeSql": "ANALYZE TABLE stg.stg_szsj_jkmzmxx_plqd partition(day_id='20211204') COMPUTE STATISTICS","defaultFS": "hdfs://bigdata:8020","hadoopConfig": {"dfs.client.use.datanode.hostname": "true"},"fileType": "orc","fieldDelimiter": ",","fileName": "stg.stg_szsj_jkmzmxx_plqd","path": "/warehouse/tablespace/managed/hive/stg.db/stg_szsj_jkmzmxx_plqd/day_id=20211204","hdfsUser": "hdfs","writeMode": "overwrite","column": [{"name": "xm","type": "VARCHAR"}, {"name": "zjhm","type": "VARCHAR"}, {"name": "lxdh","type": "VARCHAR"}, {"name": "city","type": "VARCHAR"}, {"name": "region","type": "VARCHAR"}, {"name": "street","type": "VARCHAR"}, {"name": "community","type": "VARCHAR"}, {"name": "dz","type": "VARCHAR"}, {"name": "sjly","type": "VARCHAR"}, {"name": "lysj","type": "TIMESTAMP"}, {"name": "zmyy","type": "VARCHAR"}, {"name": "zmqzt","type": "VARCHAR"}, {"name": "nzzt","type": "VARCHAR"}, {"name": "sffk","type": "VARCHAR"}, {"name": "jcfkrq","type": "TIMESTAMP"}],"originColumn": [{"name": "xm","type": "VARCHAR"}, {"name": "zjhm","type": "VARCHAR"}, {"name": "lxdh","type": "VARCHAR"}, {"name": "city","type": "VARCHAR"}, {"name": "region","type": "VARCHAR"}, {"name": "street","type": "VARCHAR"}, {"name": "community","type": "VARCHAR"}, {"name": "dz","type": "VARCHAR"}, {"name": "sjly","type": "VARCHAR"}, {"name": "lysj","type": "TIMESTAMP"}, {"name": "zmyy","type": "VARCHAR"}, {"name": "zmqzt","type": "VARCHAR"}, {"name": "nzzt","type": "VARCHAR"}, {"name": "sffk","type": "VARCHAR"}, {"name": "jcfkrq","type": "TIMESTAMP"}]}}}]}
}

DataX之MySQL-overwrite-HIVE相关推荐

如何使用 datax 将 mysql 中的数据拉取到 hive ？
需求使用datax将mysql中的数据拉取到hive的ods层步骤首先在mysql中确定好需要拉取的表user_extend, 然后对应在hive中创建好空表,等待拉取这里对应创建的hive表 ...
Mysql和Hive之间通过Sqoop进行数据同步
文章回顾理论大数据框架原理简介大数据发展历程及技术选型实践搭建大数据运行环境之一搭建大数据运行环境之二本地MAC环境配置 CPU数和内存大小查看CPU数 sysctl machdep. ...
mysql到hive数据类型转换
1.类型映射关系 mysql和hive中的数据类型存在差异,在mysql集成数据到hive中这样的场景下,我们希望在hive中的数据是贴源的,所以在hive中希望创建和mysql结构一致的表. mys ...
【hive】mysql到hive数据类型转换
1.类型映射关系 mysql和hive中的数据类型存在差异,在mysql集成数据到hive中这样的场景下,我们希望在hive中的数据是贴源的,所以在hive中希望创建和mysql结构一致的表. mys ...
hadoop读取mysql数据_Pyspark连接mysql、hive、hdfs 实例展示
本文展示pyspark与mysql.hive.hdfs的连接实例.一是为了纪念:终于走过n个坑,自己搭配好本地hadoop环境:一是为了让还不会pyspark的人有个初步的印象,毕竟数据分析师除了会使 ...
sqoop动态分区导入mysql,使用sqoop import从mysql往hive含分区表中导入数据的一些注意事项...
先看下面这条语句,它实现的功能是将特定日期的数据从mysql表中直接导入hive $ sqoop import \ --connect jdbc:mysql://192.168.xx.xx:3306/ ...
数据转移-从MySQL到Hive
转载: https://blog.csdn.net/qq_34100655/article/details/81033022 简介打开Hive 任务描述解决思路技术点简介 Hive是Hadoo ...
ClickHouse数据导入(Flink、Spark、Kafka、MySQL、Hive)
本文分享主要是ClickHouse的数据导入方式,本文主要介绍如何使用Flink.Spark.Kafka.MySQL.Hive将数据导入ClickHouse,具体内容包括: 使用Flink导入数据使 ...
sqoop从mysql导入hive字符集的问题乱码问题
sqoop从mysql导入hive字符集的问题 Mysql都是使用utf8的字符集:所以这个与Mysql的配置无直接关系:但是可以通过修改Mysql上面的表默认字符集来解决,这个是不需要修改Hive配 ...
hive 操作（二）——使用 mysql 作为 hive 的metastore
Hive 基础及安装 Hive 操作(一) hive 操作(二)--使用 mysql 作为 hive 的metastore hive 操作(三)--hive 的数据模型 hive 操作(四) hive ...

DataX之MySQL-overwrite-HIVE

DataX之MySQL-overwrite-HIVE相关推荐

最新文章

热门文章