ETL异构数据源Datax_MySQL同步Oracle(全量)_07
2024-04-29 18:57:54
文章目录
- 1. 清除Oracle数据库中OTBS1表的数据
- 2. 构建json
- 3. 执行脚本
- 4. 同步验证
- 5. 同步分析
- 7. 同步结果
1. 清除Oracle数据库中OTBS1表的数据
Truncate TABLE OTBS1;
2. 构建json
{"core": {"transport": {"channel": {"speed": {"byte": 1048576}}}},"job": {"setting": {"speed": {"channel": 5},"errorLimit": {"record": 0,"percentage": 0.02}},"content": [{"reader": {"name": "mysqlreader","parameter": {"column": ["IDNO","COL1","COL2","COL3","DT","COL5","COL6","COL7","COL8","COL9","COL10"],"splitPk": "IDNO","connection": [{"jdbcUrl": ["jdbc:mysql://127.0.0.1:3306/datax"],"table": ["otbs1"]}],"username": "root","password": "123456"}},"writer": {"name": "oraclewriter","parameter": {"column": ["IDNO","COL1","COL2","COL3","DT","COL5","COL6","COL7","COL8","COL9","COL10"],"connection": [{"jdbcUrl": "jdbc:oracle:thin:@192.xxx.xxx.128xxx1521:orcl","table": ["TEST.OTBS1"]}],"username": "username","password": "password"}}}]}
}
3. 执行脚本
python /app/datax/bin/datax.py /app/datax/job/mysql2oracle
4. 同步验证
5. 同步分析
2021-06-23 14:01:53.357 [job-0] INFO JobContainer - jobContainer starts to do prepare ...
2021-06-23 14:01:53.358 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] do prepare work .
2021-06-23 14:01:53.359 [job-0] INFO JobContainer - DataX Writer.Job [oraclewriter] do prepare work .
2021-06-23 14:01:53.360 [job-0] INFO JobContainer - jobContainer starts to do split ...
2021-06-23 14:01:53.361 [job-0] INFO JobContainer - Job set Channel-Number to 5 channels.
2021-06-23 14:01:53.387 [job-0] INFO SingleTableSplitUtil - split pk [sql=SELECT MIN(IDNO),MAX(IDNO) FROM otbs1] is running...
2021-06-23 14:01:53.450 [job-0] INFO SingleTableSplitUtil - After split(), allQuerySql=[
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from otbs1 where (1 <= IDNO AND IDNO < 41944)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from otbs1 where (41944 <= IDNO AND IDNO < 83887)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from otbs1 where (83887 <= IDNO AND IDNO < 125830)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from otbs1 where (125830 <= IDNO AND IDNO < 167773)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from otbs1 where (167773 <= IDNO AND IDNO < 209716)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from otbs1 where (209716 <= IDNO AND IDNO < 251659)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from otbs1 where (251659 <= IDNO AND IDNO < 293602)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from otbs1 where (293602 <= IDNO AND IDNO < 335545)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from otbs1 where (335545 <= IDNO AND IDNO < 377488)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from otbs1 where (377488 <= IDNO AND IDNO < 419431)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from otbs1 where (419431 <= IDNO AND IDNO < 461374)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from otbs1 where (461374 <= IDNO AND IDNO < 503317)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from otbs1 where (503317 <= IDNO AND IDNO < 545260)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from otbs1 where (545260 <= IDNO AND IDNO < 587203)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from otbs1 where (587203 <= IDNO AND IDNO < 629146)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from otbs1 where (629146 <= IDNO AND IDNO < 671089)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from otbs1 where (671089 <= IDNO AND IDNO < 713032)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from otbs1 where (713032 <= IDNO AND IDNO < 754975)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from otbs1 where (754975 <= IDNO AND IDNO < 796918)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from otbs1 where (796918 <= IDNO AND IDNO < 838861)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from otbs1 where (838861 <= IDNO AND IDNO < 880804)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from otbs1 where (880804 <= IDNO AND IDNO < 922747)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from otbs1 where (922747 <= IDNO AND IDNO < 964690)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from otbs1 where (964690 <= IDNO AND IDNO < 1006633)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from otbs1 where (1006633 <= IDNO AND IDNO <= 1048576)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from otbs1 where IDNO IS NULL
].
2021-06-23 14:01:53.451 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] splits to [26] tasks.
2021-06-23 14:01:53.464 [job-0] INFO JobContainer - DataX Writer.Job [oraclewriter] splits to [26] tasks.
2021-06-23 14:01:53.539 [job-0] INFO JobContainer - jobContainer starts to do schedule ...
2021-06-23 14:01:53.577 [job-0] INFO JobContainer - Scheduler starts [1] taskGroups.
2021-06-23 14:01:53.581 [job-0] INFO JobContainer - Running by standalone Mode.
2021-06-23 14:01:53.634 [taskGroup-0] INFO TaskGroupContainer - taskGroupId=[0] start [5] channels for [26] tasks.
2021-06-23 14:01:53.659 [taskGroup-0] INFO Channel - Channel set byte_speed_limit to 1048576.
2021-06-23 14:01:53.659 [taskGroup-0] INFO Channel - Channel set record_speed_limit to -1, No tps activated.
7. 同步结果
2021-06-23 14:02:13.668 [job-0] INFO JobContainer -
任务启动时刻 : 2021-06-23 14:01:52
任务结束时刻 : 2021-06-23 14:02:13
任务总计耗时 : 21s
任务平均流量 : 3.30MB/s
记录写入速度 : 52428rec/s
读出记录总数 : 1048576
读写失败总数 : 0
ETL异构数据源Datax_MySQL同步Oracle(全量)_07相关推荐
- ETL异构数据源Datax_Oracle同步MySQL(全量)_04
文章目录 一.Oracle同步Mysql 1. 构建json 2. 执行数据同步 3. 查看同步数据 4. 同步数据正确性和准确性 5. 同步日志分析 一.Oracle同步Mysql 1. 构建jso ...
- ETL异构数据源Datax_图形化数据同步_11
数据同步全量增量知识SQL和执行频次不一样,其他的都一样 文章目录 一.Oracle同步Mysql(全量) 1. 添加项目 2. 添加数据源 3. 添加任务 4. 构建json 5. 任务执行 6. ...
- DataX oracle同步mysql(全量和增量)
本篇博客说说DataX如何进行全量和增量数据同步,虽然用演示oracle同步到mysql,但其他数据库之间的同步都差不多 1.DataX介绍 DataX 是一个异构数据源离线同步工具,致力于实现包括关 ...
- ETL异构数据源Datax_工具部署_02
接上一篇:(企业内部) ETL异构数据源Datax_部署前置环境_01 https://gblfy.blog.csdn.net/article/details/118081253 文章目录 一.直接下 ...
- 异构数据源离线同步工具之DataX的安装部署
异构数据源离线同步工具之DataX的安装部署
- ETL异构数据源Datax_日期增量同步_13
文章目录 一.全量同步 1. 增量同步SQL 2. 构建reader 3. 构建writer 4. 字段对应关系映射 5. 构建json 6. 选择同步模板 7. 查询最早时间 8. 修改任务信息 9 ...
- ETL异构数据源Datax_自增ID增量同步_12
文章目录 增量同步方法 1.先同步存量数据 2.再同步增量数据(根据自增ID.日期条件) 前提条件: 1.只针对数据增长,如果老数据被update/delete则无法使用增量同步方式. 基于主键自增I ...
- ETL异构数据源Datax_部署前置环境_01
文章目录 一.概述 1. 是什么? 2. 开源地址 二.简介 2.1. 设计架构 2.2. 框架结构 三.检查环境 3.1. 检验是否安装JDK(1.8以上,推荐1.8) 3.2. 检验是否安装pyt ...
- 【图解】redis主从同步流程——全量同步、部分同步、命令传播
提示:文章写完后,目录可以自动生成,如何生成可参考右边的帮助文档 文章目录 前言 一.全量同步 二.部分同步 三.命令传播 总结 前言 本文主要介绍Redis高可用下的主从同步问题,包括全量同步.部分 ...
最新文章
- java z注释过滤_如何编写一个java程序来过滤所有注释行并只打印java编码行？
- 循环语句until和while
- difference between JSON Model and client model
- Java字符串的十大问题
- VS2008创建dll，并使用dll
- 重庆三峡学院计算机英语期末考试,英语语言学试题B卷及答案(重庆三峡学院)
- 白板机器学习笔记 P22-P27 PCA降维
- Oracle存在gap,发现gap及解决
- PostgreSQL概述
- Abp框架多租户源码解读
- 路孚特：金融行业数据复杂 成本高 难获取 看路孚特如何打破困局
- 广西农业职业技术学院计算机,学科分类与代码-广西农业职业技术学院.doc
- 封装一段html,封装html
- python +高德地图API调用
- STM32F091不识别仿真器的案例
- （PS教程）01-用PS给证件照排版
- 想运营好拼多多店铺，做好产品规划必不可少！
- 小伙Python爬虫并自制新闻网站，太好玩了
- uni-app 实现微信分享
- 区块链主要的核心内容