http://hi.baidu.com/dba_hui/item/a4b23a60ae1d6882c4d2497c

查看slave复制状态，发现SQL线程停止应用：mysql> show slave status \G;

*************************** 1. row ***************************

Slave_IO_State: Waiting for master to send event

Master_Host: 192.168.9.61

Master_User: repl

Master_Port: 3312

Connect_Retry: 60

Master_Log_File: mysql-bin.000034

Read_Master_Log_Pos: 742553356

Relay_Log_File: relay-log.000004

Relay_Log_Pos: 41832895

Relay_Master_Log_File: mysql-bin.000034

Slave_IO_Running: Yes

Slave_SQL_Running: No

Replicate_Do_DB:

Replicate_Ignore_DB:

Replicate_Do_Table:

Replicate_Ignore_Table:

Replicate_Wild_Do_Table: eoc.%,test.%,mysql.%

Replicate_Wild_Ignore_Table:

Last_Errno: 1032

Last_Error: Could not execute Delete_rows event on table eoc.eoc_course_rank; Can't find record in 'eoc_course_rank', Error_code: 1032; handler error HA_ERR_END_OF_FILE; the event's master log mysql-bin.000034, end_log_pos 684728241

Skip_Counter: 0

Exec_Master_Log_Pos: 684727611

Relay_Log_Space: 257830925

Until_Condition: None

Until_Log_File:

Until_Log_Pos: 0

Master_SSL_Allowed: No

Master_SSL_CA_File:

Master_SSL_CA_Path:

Master_SSL_Cert:

Master_SSL_Cipher:

Master_SSL_Key:

Seconds_Behind_Master: NULL

Master_SSL_Verify_Server_Cert: No

Last_IO_Errno: 0

Last_IO_Error:

Last_SQL_Errno: 1032

Last_SQL_Error: Could not execute Delete_rows event on table eoc.eoc_course_rank; Can't find record in 'eoc_course_rank', Error_code: 1032; handler error HA_ERR_END_OF_FILE; the event's master log mysql-bin.000034, end_log_pos 684728241

1 row in set (0.00 sec)

错误日志如下：

120719 1:01:07 [ERROR] Slave SQL: Could not execute Delete_rows event on table eoc.eoc_course_rank; Can't find record in 'eoc_course_rank', Error_code: 1032; handler error HA_ERR_END_OF_FILE; the event's master log mysql-bin.000034, end_log_pos 684728241, Error_code: 1032

120719 1:01:07 [Warning] Slave: Can't find record in 'eoc_course_rank' Error_code: 1032

120719 1:01:07 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'mysql-bin.000034' position 684727611

从上面的情况可以看出，在slave执行从master复制过来的delete语句因为slave找不到要删除的数据，导致SQL线程报错，停止应用。也就是说master和slave上数据不一致导致的该错误。

遇到这种情况，一般都是找出错误，然后修改slave上的数据，然后启动slave，继续应用。

比如上面，要删除的数据在slave上找不到，我们就想知道是删除哪条数据呢？但是show slave status中又没有明确给出错误的语句，因此需要分析binary log来查看是哪条语句。

binary log的格式是mixed，初步使用mysqlbinlog查看，发现该条语句是RBR，而不是SBR,因此需要使用--base64-output=decode-rows -v -v选项，如下：

使用binary log：

[root@awstats usr]# /usr/local/mysql5157_2/bin/mysqlbinlog --no-defaults --base64-output=decode-rows -v -v mysql-bin.000034 | grep -A 200 '684728241' > zhh.txt

或者

使用relay log：

[root@mysqldb var]# /usr/local/mysql51572/bin/mysqlbinlog --no-defaults --base64-output=decode-rows -v -v relay-log.000004 | grep -A 200 '684728241' > zhh.txt

查看文件，发现以下内容：

#120719 1:01:03 server id 1 end_log_pos 684728241 Delete_rows: table id 16854 flags: STMT_END_F

### DELETE FROM eoc.eoc_course_rank

### WHERE

### @1=2 /* TINYINT meta=0 nullable=0 is_null=0 */

### @2=392 /* SHORTINT meta=0 nullable=0 is_null=0 */

### @3=574 /* INT meta=0 nullable=0 is_null=0 */

### @4='xxx' /* VARSTRING(192) meta=192 nullable=0 is_null=0 */

### @5='yyy' /* VARSTRING(1500) meta=1500 nullable=1 is_null=0 */

### @6='20111202-2df9eef5c0e33cfd643927cb1bb048fa17a40dfa.jpg' /* VARSTRING(600) meta=600 nullable=1 is_null=0 */

### @7=3 /* INT meta=0 nullable=1 is_null=0 */

### @8=256 /* INT meta=0 nullable=1 is_null=0 */

# at 684728241

即出错误的语句为：

delete from eoc.eoc_course_rank

where

第一列等于2

第二列等于392

......

根据该表的主键在slave查询，发现where条件中有一个字段值不同，因此在slave上找不到要删除的数据。

mysql> use eoc;

Reading table information for completion of table and column names

You can turn off this feature to get a quicker startup with -A

Database changed

mysql> SELECT * FROM eoc_course_rank

-> WHERE course_id=574 \G;

*************************** 1. row ***************************

rank_type: 2

rank_id: 385

course_id: 574

course_name: xxx

detail_info: yyy

brief_image: 20111202-2df9eef5c0e33cfd643927cb1bb048fa17a40dfa.jpg

cp_id: 3

grade_id: 256

1 row in set (0.00 sec)

ERROR:

No query specified

mysql> UPDATE eoc_course_rank SET rank_id= 392 WHERE course_id=574;

Query OK, 1 row affected (0.00 sec)

Rows matched: 1 Changed: 1 Warnings: 0

mysql> commit;

Query OK, 0 rows affected (0.00 sec)

mysql> stop slave;

Query OK, 0 rows affected (0.00 sec)

mysql> start slave;

Query OK, 0 rows affected (0.00 sec)

mysql> show slave status \G;

*************************** 1. row ***************************

Slave_IO_State: Waiting for master to send event

Master_Host: 192.168.9.61

Master_User: repl

Master_Port: 3312

Connect_Retry: 60

Master_Log_File: mysql-bin.000034

Read_Master_Log_Pos: 746616004

Relay_Log_File: relay-log.000004

Relay_Log_Pos: 95495411

Relay_Master_Log_File: mysql-bin.000034

Slave_IO_Running: Yes

Slave_SQL_Running: Yes

Replicate_Do_DB:

Replicate_Ignore_DB:

Replicate_Do_Table:

Replicate_Ignore_Table:

Replicate_Wild_Do_Table: eoc.%,test.%,mysql.%

Replicate_Wild_Ignore_Table:

Last_Errno: 0

Last_Error:

Skip_Counter: 0

Exec_Master_Log_Pos: 738390127

Relay_Log_Space: 261893867

Until_Condition: None

Until_Log_File:

Until_Log_Pos: 0

Master_SSL_Allowed: No

Master_SSL_CA_File:

Master_SSL_CA_Path:

Master_SSL_Cert:

Master_SSL_Cipher:

Master_SSL_Key:

Seconds_Behind_Master: 4791

Master_SSL_Verify_Server_Cert: No

Last_IO_Errno: 0

Last_IO_Error:

Last_SQL_Errno: 0

Last_SQL_Error:

1 row in set (0.00 sec)

mysql> show slave status \G;

*************************** 1. row ***************************

Slave_IO_State: Waiting for master to send event

Master_Host: 192.168.9.61

Master_User: repl

Master_Port: 3312

Connect_Retry: 60

Master_Log_File: mysql-bin.000034

Read_Master_Log_Pos: 747488636

Relay_Log_File: relay-log.000005

Relay_Log_Pos: 904068

Relay_Master_Log_File: mysql-bin.000034

Slave_IO_Running: Yes

Slave_SQL_Running: Yes

Replicate_Do_DB:

Replicate_Ignore_DB:

Replicate_Do_Table:

Replicate_Ignore_Table:

Replicate_Wild_Do_Table: eoc.%,test.%,mysql.%

Replicate_Wild_Ignore_Table:

Last_Errno: 0

Last_Error:

Skip_Counter: 0

Exec_Master_Log_Pos: 747488636

Relay_Log_Space: 104594214

Until_Condition: None

Until_Log_File:

Until_Log_Pos: 0

Master_SSL_Allowed: No

Master_SSL_CA_File:

Master_SSL_CA_Path:

Master_SSL_Cert:

Master_SSL_Cipher:

Master_SSL_Key:

Seconds_Behind_Master: 0

Master_SSL_Verify_Server_Cert: No

Last_IO_Errno: 0

Last_IO_Error:

Last_SQL_Errno: 0

Last_SQL_Error:

1 row in set (0.00 sec)

上面说了，产生该错误的根本原因是主从数据不一致。为什么主从数据不一致呢？

从报错的时间点分析，发现该错误数据是一个event执行后产生的，查看该event，发现有存在结果集不确定的语句。修改该event的sql，再次执行该event调用的stored procedure，状态仍然为双YES。

另外有一点不明白：

my.cnf中设置的binary log是mixed格式：在未修改该sql之前，该行数据binary log记录的是RBR格式；修改后记录的是SBR格式。