binlog解析工具

文章目录

一、工具介绍
- 1.1 工具对比
- 1.2 用途
- 1.3 限制
- 1.4 性能对比
二、工具使用
- 2.1 参数说明
- 2.2 使用案例

一、工具介绍

my2sql是go版MySQL binlog解析工具，通过解析MySQL binlog ，可以生成原始SQL、回滚SQL、去除主键的INSERT SQL等，也可以生成DML统计信息。类似工具有binlog2sql、MyFlash、my2fback等，本工具基于my2fback、binlog_rollback工具二次开发而来。

1.1 工具对比

binlog2sql：Python 编写，用于生成回滚/前滚 SQL 进行数据恢复/补偿
MyFlash：C 语言编写，用于生成反向 binlog 文件（二进制）进行数据恢复
my2sql：Go 语言编写，除了闪回，还提供了前滚和事务分析的功能

1.2 用途

数据快速回滚(闪回)
主从切换后新master丢数据的修复
从binlog生成标准SQL，带来的衍生功能
生成DML统计信息，可以找到哪些表更新的比较频繁
IO高TPS高，查出哪些表在频繁更新
找出某个时间点数据库是否有大事务或者长事务
主从延迟，分析主库执行的SQL语句
除了支持常规数据类型，对大部分工具不支持的数据类型做了支持，比如json、blob、text、emoji等数据类型sql生成

1.3 限制

使用回滚/闪回功能时，binlog格式必须为row,且binlog_row_image=full， DML统计以及大事务分析不受影响
只能回滚DML，不能回滚DDL
支持指定-tl时区来解释binlog中time/datetime字段的内容。开始时间-start-datetime与结束时间-stop-datetime也会使用此指定的时区，但注意此开始与结束时间针对的是binlog event header中保存的unix timestamp。结果中的额外的datetime时间信息都是binlog event header中的unix timestamp
此工具是伪装成从库拉取binlog，需要连接数据库的用户有SELECT, REPLICATION SLAVE, REPLICATION CLIENT权限
MySQL8.0版本需要在配置文件中加入default_authentication_plugin =mysql_native_password，用户密码认证必须是mysql_native_password才能解析

1.4 性能对比

测试项	my2sql	binlog2sql
1.1G binlog生成回滚SQL	1分40秒	65分钟
1.1G binlog生成原始SQL	1分30秒	50分钟
1.1G binlog生成表DML统计信息、以及事务统计信息	40秒	不支持

二、工具使用

2.1 参数说明

-U

优先使用unique key作为where条件，默认false

-add-extraInfo

是否把database/table/datetime/binlogposition…信息以注释的方式加入生成的每条sql前，默认false

datetime=2020-07-16_10:44:09 database=orchestrator table=cluster_domain_name binlog=mysql-bin.011519 startpos=15552 stoppos=15773
UPDATE `orchestrator`.`cluster_domain_name` SET `last_registered`='2020-07-16 10:44:09' WHERE `cluster_name`='192.168.1.1:3306'

-big-trx-row-limit n

找出满足n条sql的事务，默认500条

-databases 、 -tables

库及表条件过滤, 以逗号分隔

-sql
要解析的sql类型，可选参数insert、update、delete，默认全部解析
-doNotAddPrifixDb
默认生成insert into db1.tb1 (x1, x1) values (y1, y1)类sql，也可以生成不带库名的sql
-file-per-table
为每个表生成一个sql文件
-full-columns
生成的sql是否带全列信息，默认false
-ignorePrimaryKeyForInsert
生成的insert语句是否去掉主键，默认false
-output-dir
将生成的结果存放到制定目录
-output-toScreen
将生成的结果打印到屏幕，默认写到文件
-threads
线程数，默认8个
-work-type
2sql：生成原始sql，rollback：生成回滚sql，stats：只统计DML、事务信息

2.2 使用案例

解析标准SQL

[root@zijie releases]# ./my2sql  -user root -password 970125  -work-type 2sql  -start-file /data/mysql/binlog/binlog.000008  -start-datetime "2021-03-01 00:00:00" -stop-datetime "2021-04-01 00:00:00" -output-dir ./
[2021/03/15 10:33:38] [info] binlogsyncer.go:144 create BinlogSyncer with config {1113306 mysql 127.0.0.1 3306 root   utf8 false false <nil> false Local false 0 0s 0s 0 false false 0}
[2021/03/15 10:33:38] [info] binlogsyncer.go:360 begin to sync binlog from position (binlog.000008, 4)
[2021/03/15 10:33:38] [info] stats_process.go:166 start thread to analyze statistics from binlog
[2021/03/15 10:33:38] [info] events.go:208 start thread to write redo/rollback sql into file
[2021/03/15 10:33:38] [info] events.go:58 start thread 1 to generate redo/rollback sql
[2021/03/15 10:33:38] [info] events.go:58 start thread 2 to generate redo/rollback sql
[2021/03/15 10:33:38] [info] repl.go:15 start to get binlog from mysql
[2021/03/15 10:33:38] [info] binlogsyncer.go:777 rotate to (binlog.000008, 4)
[2021/03/15 10:33:43] [info] repl.go:83 deadline exceeded.
[2021/03/15 10:33:43] [info] repl.go:17 finish getting binlog from mysql
[2021/03/15 10:33:43] [info] stats_process.go:266 exit thread to analyze statistics from binlog
[2021/03/15 10:33:43] [info] events.go:183 exit thread 1 to generate redo/rollback sql
[2021/03/15 10:33:43] [info] events.go:183 exit thread 2 to generate redo/rollback sql
[2021/03/15 10:33:43] [info] events.go:272 finish writing redo/forward sql into file
[2021/03/15 10:33:43] [info] events.go:275 exit thread to write redo/rollback sql into file[root@zijie releases]# cat forward.8.sql
INSERT INTO `test`.`t1` (`id`) VALUES (1);
UPDATE `test`.`t1` SET `id`=2 WHERE `id`=1;
DELETE FROM `test`.`t1` WHERE `id`=2;
INSERT INTO `test`.`t1` (`id`,`name`) VALUES (1,'');
UPDATE `test`.`t1` SET `id`=2 WHERE `id`=1;
DELETE FROM `test`.`t1` WHERE `id`=2;
INSERT INTO `test`.`t2` (`id`) VALUES (1);
UPDATE `test`.`t2` SET `id`=2 WHERE `id`=1;
DELETE FROM `test`.`t2` WHERE `id`=2;

解析回滚SQL

[root@zijie releases]# ./my2sql  -user root -password 970125  -work-type rollback  -start-file /data/mysql/binlog/binlog.000008  -start-datetime "2021-03-01 00:00:00" -stop-datetime "2021-04-01 00:00:00" -output-dir ./
[2021/03/15 10:40:30] [info] binlogsyncer.go:144 create BinlogSyncer with config {1113306 mysql 127.0.0.1 3306 root   utf8 false false <nil> false Local false 0 0s 0s 0 false false 0}
[2021/03/15 10:40:30] [info] binlogsyncer.go:360 begin to sync binlog from position (binlog.000008, 4)
[2021/03/15 10:40:30] [info] stats_process.go:166 start thread to analyze statistics from binlog
[2021/03/15 10:40:30] [info] events.go:208 start thread to write redo/rollback sql into file
[2021/03/15 10:40:30] [info] events.go:58 start thread 1 to generate redo/rollback sql
[2021/03/15 10:40:30] [info] events.go:58 start thread 2 to generate redo/rollback sql
[2021/03/15 10:40:30] [info] repl.go:15 start to get binlog from mysql
[2021/03/15 10:40:30] [info] binlogsyncer.go:777 rotate to (binlog.000008, 4)
[2021/03/15 10:40:35] [info] repl.go:83 deadline exceeded.
[2021/03/15 10:40:35] [info] repl.go:17 finish getting binlog from mysql
[2021/03/15 10:40:35] [info] stats_process.go:266 exit thread to analyze statistics from binlog
[2021/03/15 10:40:35] [info] events.go:183 exit thread 1 to generate redo/rollback sql
[2021/03/15 10:40:35] [info] events.go:183 exit thread 2 to generate redo/rollback sql
[2021/03/15 10:40:35] [info] events.go:257 finish writing rollback sql into tmp files, start to revert content order of tmp files
[2021/03/15 10:40:35] [info] rollback_process.go:15 start thread 1 to revert rollback sql files
[2021/03/15 10:40:35] [info] rollback_process.go:41 start to revert tmp file .rollback.8.sql into rollback.8.sql
[2021/03/15 10:40:35] [info] rollback_process.go:156 finish reverting tmp file .rollback.8.sql into rollback.8.sql
[2021/03/15 10:40:35] [info] rollback_process.go:25 exit thread 1 to revert rollback sql files
[2021/03/15 10:40:35] [info] events.go:270 finish reverting content order of tmp files
[2021/03/15 10:40:35] [info] events.go:275 exit thread to write redo/rollback sql into file[root@zijie releases]# cat rollback.8.sql
INSERT INTO `test`.`t2` (`id`) VALUES (2);
UPDATE `test`.`t2` SET `id`=1 WHERE `id`=2;
DELETE FROM `test`.`t2` WHERE `id`=1;
INSERT INTO `test`.`t1` (`id`,`name`) VALUES (2,'');
UPDATE `test`.`t1` SET `id`=1 WHERE `id`=2;
DELETE FROM `test`.`t1` WHERE `id`=1;
INSERT INTO `test`.`t1` (`id`) VALUES (2);
UPDATE `test`.`t1` SET `id`=1 WHERE `id`=2;
DELETE FROM `test`.`t1` WHERE `id`=1;

统计DML以及大事务
统计各个表的DML操作数量，统计一个事务大于500条、时间大于300秒的事务

[root@zijie releases]# ./my2sql  -user root -password 970125  -work-type stats  -start-file /data/mysql/binlog/binlog.000008  -start-datetime "2021-03-01 00:00:00" -stop-datetime "2021-04-01 00:00:00"  -big-trx-row-limit 500 -long-trx-seconds 300   -output-dir ./
[2021/03/15 10:53:18] [info] binlogsyncer.go:144 create BinlogSyncer with config {1113306 mysql 127.0.0.1 3306 root   utf8 false false <nil> false Local false 0 0s 0s 0 false false 0}
[2021/03/15 10:53:18] [info] binlogsyncer.go:360 begin to sync binlog from position (binlog.000008, 4)
[2021/03/15 10:53:18] [info] stats_process.go:166 start thread to analyze statistics from binlog
[2021/03/15 10:53:18] [info] repl.go:15 start to get binlog from mysql
[2021/03/15 10:53:18] [info] binlogsyncer.go:777 rotate to (binlog.000008, 4)
[2021/03/15 10:53:23] [info] repl.go:83 deadline exceeded.
[2021/03/15 10:53:23] [info] repl.go:17 finish getting binlog from mysql
[2021/03/15 10:53:23] [info] stats_process.go:266 exit thread to analyze statistics from binlog[root@zijie releases]# cat biglong_trx.txt
binlog            starttime           stoptime            startpos   stoppos    rows     duration   tables
[root@zijie releases]# cat binlog_status.txt
binlog            starttime           stoptime            startpos   stoppos    inserts  updates  deletes  database        table
binlog.000008     2021-03-14_22:34:38 2021-03-14_22:34:55 783        1534       1        1        1        test            t1
binlog.000008     2021-03-14_22:58:44 2021-03-14_22:58:53 1970       2749       1        1        1        test            t1
binlog.000008     2021-03-14_23:04:13 2021-03-14_23:04:22 3170       3921       1        1        1        test            t2

从某一位置解析出标准SQL，并且持续打印到屏幕

[root@zijie releases]# ./my2sql  -user root -password 970125  -work-type 2sql  -start-file /data/mysql/binlog/binlog.000008  -start-datetime "2021-03-01 00:00:00" -stop-datetime "2021-04-01 00:00:00" -output-toScreen
[2021/03/15 10:55:20] [info] binlogsyncer.go:144 create BinlogSyncer with config {1113306 mysql 127.0.0.1 3306 root   utf8 false false <nil> false Local false 0 0s 0s 0 false false 0}
[2021/03/15 10:55:20] [info] binlogsyncer.go:360 begin to sync binlog from position (binlog.000008, 4)
[2021/03/15 10:55:20] [info] stats_process.go:166 start thread to analyze statistics from binlog
[2021/03/15 10:55:20] [info] events.go:208 start thread to write redo/rollback sql into file
[2021/03/15 10:55:20] [info] events.go:58 start thread 1 to generate redo/rollback sql
[2021/03/15 10:55:20] [info] events.go:58 start thread 2 to generate redo/rollback sql
[2021/03/15 10:55:20] [info] repl.go:15 start to get binlog from mysql
[2021/03/15 10:55:20] [info] binlogsyncer.go:777 rotate to (binlog.000008, 4)
INSERT INTO `test`.`t1` (`id`) VALUES (1)
UPDATE `test`.`t1` SET `id`=2 WHERE `id`=1
DELETE FROM `test`.`t1` WHERE `id`=2
INSERT INTO `test`.`t1` (`id`,`name`) VALUES (1,'')
UPDATE `test`.`t1` SET `id`=2 WHERE `id`=1
DELETE FROM `test`.`t1` WHERE `id`=2
INSERT INTO `test`.`t2` (`id`) VALUES (1)
UPDATE `test`.`t2` SET `id`=2 WHERE `id`=1
DELETE FROM `test`.`t2` WHERE `id`=2

binlog解析工具—my2sql相关推荐

技术分享 | binlog 实用解析工具 my2sql
作者:赵黎明爱可生 MySQL DBA 团队成员,Oracle 10g OCM,MySQL 5.7 OCP,擅长数据库性能问题诊断.事务与锁问题的分析等,负责处理客户 MySQL 及我司自研 DMP ...
log解析工具 px4_详解MySQL Binlog解析工具--binlog2sql，基于表级别的数据恢复
概述最近碰到某个表需要做数据回退,如果回退整个数据库动作又太大,所以用了binlog2sql来实现,这里先介绍一下binlog2sql的相关内容. binlog2sql是一个开源的Python开发的 ...
MySQL Binlog 解析工具 Maxwell 详解
maxwell 简介 Maxwell是一个能实时读取MySQL二进制日志binlog,并生成 JSON 格式的消息,作为生产者发送给 Kafka,Kinesis.RabbitMQ.Redis.Goog ...
mysql binlog查看工具_【使用 Rust 写 Parser】4. 解析 binlog
系列所有文章 PrivateRookie:[使用 Rust 写 Parser]1. 初识 nomzhuanlan.zhihu.com PrivateRookie:[使用 Rust 写 Parser] ...
Java XML解析工具 dom4j介绍及使用实例
Java XML解析工具 dom4j介绍及使用实例 dom4j介绍 dom4j的项目地址:http://sourceforge.net/projects/dom4j/?source=directory ...
介绍Shell脚本的参数解析工具
Argbash是一个代码生成器,它为你的脚本生成一个量身定制的解析库.与其他bash模块的通用代码不同,它生成你的脚本所需的最少代码. Argbash是一个代码生成器,它为你的脚本生成一个量身定制的解 ...
[开源]C#二维码生成解析工具，可添加自定义Logo
原文:[开源]C#二维码生成解析工具,可添加自定义Logo 二维码又称 QR Code,QR 全称 Quick Response,是一个近几年来移动设备上超流行的一种编码方式,它比传统的 Bar Co ...
【Android 逆向】逆向修改游戏应用 ( APK 解析工具 | 解包 -＞分析 -＞重打包 -＞签名流程 )
文章目录一.APK 解析工具二.解包 -> 分析 -> 重打包 -> 签名流程一.APK 解析工具使用 [Android 逆向]使用 Python 编写 APK 批处理分析 ...
mflac文件解析工具_9 个爱不释手的 JSON 工具
JSONLint JSONCompare jtc ijson JSON格式化和验证器 Altova XMLSpy JSON和 XML编辑器 Code Beautify JSON工具 Visual St ...

binlog解析工具—my2sql