ETL异构数据源Datax_使用数据分片提升同步速度

文章目录

1. 构建json，添加数据分片
2. Mysql数据清除
3. 数据分片前后对比

1. 构建json，添加数据分片

{"job": {"setting": {"speed": {"channel": 3},"errorLimit": {"record": 0,"percentage": 0.02}},"content": [{"reader": {"name": "oraclereader","parameter": {"column": ["IDNO","COL1","COL2","COL3","DT","COL5","COL6","COL7","COL8","COL9","COL10"],splitPk:"IDNO","connection": [{"jdbcUrl": ["jdbc:oracle:thin:@192.xxx.xxx.xxx:1521:orcl"],"table": ["TEST.OTBS1"]}],"username": "username","password": "password"}},"writer": {"name": "mysqlwriter","parameter": {"column": ["IDNO","COL1","COL2","COL3","DT","COL5","COL6","COL7","COL8","COL9","COL10"],"connection": [{"jdbcUrl": "jdbc:mysql://127.0.0.1:3306/datax?autoReconnect=true&useUnicode=true&characterEncoding=utf8&zeroDateTimeBehavior=CONVERT_TO_NULL&useSSL=false&serverTimezone=CTT&nullCatalogMeansCurrent=true","table": ["otbs1"]}],"username": "root","password": "123456"}}}]}
}

2. Mysql数据清除

清除mysql otbs1表数据

truncate table otbs1;

3. 数据分片前后对比

数据分片前

2021-06-23 12:28:12.390 [job-0] INFO  StandAloneJobContainerCommunicator - Total 1048576 records, 69143488 bytes | Speed 1.65MB/s, 26214 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 26.038s |  All Task WaitReaderTime 8.483s | Percentage 100.00%
2021-06-23 12:28:12.402 [job-0] INFO  JobContainer -
任务启动时刻                    : 2021-06-23 12:27:31
任务结束时刻                    : 2021-06-23 12:28:12
任务总计耗时                    :                 41s
任务平均流量                    :            1.65MB/s
记录写入速度                    :          26214rec/s
读出记录总数                    :             1048576
读写失败总数                    :                   0channel并发3个未生效

数据分片后

2021-06-23 12:59:01.629 [job-0] INFO  JobContainer -
任务启动时刻                    : 2021-06-23 12:58:29
任务结束时刻                    : 2021-06-23 12:59:01
任务总计耗时                    :                 31s
任务平均流量                    :            2.20MB/s
记录写入速度                    :          34952rec/s
读出记录总数                    :             1048576
读写失败总数                    :                   0

速度相比数据分片前提升了10s

同步日志，相比数据分片前做了数据分片处理，并发3个channel处理16个任务。
channel并发3个未生效

2021-06-23 12:58:31.020 [job-0] INFO  JobContainer - jobContainer starts to do prepare ...
2021-06-23 12:58:31.020 [job-0] INFO  JobContainer - DataX Reader.Job [oraclereader] do prepare work .
2021-06-23 12:58:31.020 [job-0] INFO  JobContainer - DataX Writer.Job [mysqlwriter] do prepare work .
2021-06-23 12:58:31.021 [job-0] INFO  JobContainer - jobContainer starts to do split ...
2021-06-23 12:58:31.021 [job-0] INFO  JobContainer - Job set Channel-Number to 3 channels.
2021-06-23 12:58:31.113 [job-0] INFO  SingleTableSplitUtil - split pk [sql=SELECT * FROM ( SELECT IDNO FROM DBTEST.OTBS1 SAMPLE (0.1)  WHERE (IDNO IS NOT NULL)  ORDER BY DBMS_RANDOM.VALUE) WHERE ROWNUM <= 15 ORDER by IDNO ASC] is running...
2021-06-23 12:58:31.389 [job-0] INFO  SingleTableSplitUtil - After split(), allQuerySql=[
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from TEST.OTBS1  where  (42075 <= IDNO AND IDNO < 77408)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from TEST.OTBS1  where  (77408 <= IDNO AND IDNO < 187833)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from TEST.OTBS1  where  (187833 <= IDNO AND IDNO < 263631)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from TEST.OTBS1  where  (263631 <= IDNO AND IDNO < 349253)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from TEST.OTBS1  where  (349253 <= IDNO AND IDNO < 364994)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from TEST.OTBS1  where  (364994 <= IDNO AND IDNO < 434398)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from TEST.OTBS1  where  (434398 <= IDNO AND IDNO < 437250)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from TEST.OTBS1  where  (437250 <= IDNO AND IDNO < 516705)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from TEST.OTBS1  where  (516705 <= IDNO AND IDNO < 555961)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from TEST.OTBS1  where  (555961 <= IDNO AND IDNO < 578695)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from TEST.OTBS1  where  (578695 <= IDNO AND IDNO < 638120)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from TEST.OTBS1  where  (638120 <= IDNO AND IDNO < 655685)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from TEST.OTBS1  where  (655685 <= IDNO AND IDNO < 859873)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from TEST.OTBS1  where  (859873 <= IDNO AND IDNO <= 962533)
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from TEST.OTBS1  where  ((IDNO < 42075) OR (962533 < IDNO))
select IDNO,COL1,COL2,COL3,DT,COL5,COL6,COL7,COL8,COL9,COL10 from TEST.OTBS1  where  IDNO IS NULL
].
2021-06-23 12:58:31.390 [job-0] INFO  JobContainer - DataX Reader.Job [oraclereader] splits to [16] tasks.
2021-06-23 12:58:31.394 [job-0] INFO  JobContainer - DataX Writer.Job [mysqlwriter] splits to [16] tasks.
2021-06-23 12:58:31.431 [job-0] INFO  JobContainer - jobContainer starts to do schedule ...
2021-06-23 12:58:31.460 [job-0] INFO  JobContainer - Scheduler starts [1] taskGroups.
2021-06-23 12:58:31.463 [job-0] INFO  JobContainer - Running by standalone Mode.
2021-06-23 12:58:31.487 [taskGroup-0] INFO  TaskGroupContainer - taskGroupId=[0] start [3] channels for [16] tasks.
2021-06-23 12:58:31.508 [taskGroup-0] INFO  Channel - Channel set byte_speed_limit to -1, No bps activated.
2021-06-23 12:58:31.508 [taskGroup-0] INFO  Channel - Channel set record_speed_limit to -1, No tps activated.

ETL异构数据源Datax_使用数据分片提升同步速度_05相关推荐

ETL异构数据源Datax_工具部署_02
接上一篇:(企业内部) ETL异构数据源Datax_部署前置环境_01 https://gblfy.blog.csdn.net/article/details/118081253 文章目录一.直接下 ...
ETL异构数据源Datax_限速设置_06
文章目录一.提升job内Channel并发有几种配置方式二.配置简述三.案例 3.1. 第一种 3.2. 第二种 3.3. 第三种 3.4. 第四种 3.5. 案例实战 3.6. 总结前言: ...
ETL异构数据源Datax_部署前置环境_01
文章目录一.概述 1. 是什么? 2. 开源地址二.简介 2.1. 设计架构 2.2. 框架结构三.检查环境 3.1. 检验是否安装JDK(1.8以上,推荐1.8) 3.2. 检验是否安装pyt ...
ETL异构数据源Datax_图形化数据同步_11
数据同步全量增量知识SQL和执行频次不一样,其他的都一样文章目录一.Oracle同步Mysql(全量) 1. 添加项目 2. 添加数据源 3. 添加任务 4. 构建json 5. 任务执行 6. ...
ETL异构数据源Datax_数据准备_03
文章目录一.Oracle数据库准备 1. 初始化表结构 2. 生成测试数据二.使用docker创建mysql 2.1. 构建mysql容器 2.1. 创建datax数据库 2.3. 初始化表结构 ...
ETL异构数据源Datax_日期增量同步_13
文章目录一.全量同步 1. 增量同步SQL 2. 构建reader 3. 构建writer 4. 字段对应关系映射 5. 构建json 6. 选择同步模板 7. 查询最早时间 8. 修改任务信息 9 ...
ETL异构数据源Datax_自增ID增量同步_12
文章目录增量同步方法 1.先同步存量数据 2.再同步增量数据(根据自增ID.日期条件) 前提条件: 1.只针对数据增长,如果老数据被update/delete则无法使用增量同步方式. 基于主键自增I ...
ETL异构数据源Datax_使用querySql_08
使用说明当用户配置了这一项之后,DataX系统就会忽略table,column 这些配置型,直接使用这个配置项的内容对数据进行筛选,例如需要进行多表join后同步数据,使用select a,b f ...
异构数据源之间交换数据的工具
DataX:在异构数据源之间交换数据的工具作者: xiuwz 日期: 2012 年 1 月 10 日发表评论 (0) 查看评论 ( 1 votes, average: 5.00 out of 5) ...

ETL异构数据源Datax_使用数据分片提升同步速度_05

文章目录

1. 构建json，添加数据分片

2. Mysql数据清除

3. 数据分片前后对比

ETL异构数据源Datax_使用数据分片提升同步速度_05相关推荐

最新文章

热门文章