以前的Blog被新浪封了，以后就写在这里了。

今天一个测试环境的Mysql忽然挂掉，其实Mysql是已经死掉了，但是ps仍能看到。Mysql的版本为5.0.38。

在Mysqld的Log里输出如下信息：

1 121112 9:57:36InnoDB: Assertion failure in thread 1153980752 in file btr0cur.c line 3591

2 InnoDB: Failing assertion: extern_len >=part_len3 InnoDB: We intentionally generate a memory trap.4 InnoDB: Submit a detailed bug report to http://bugs.mysql.com.

5 InnoDB: If you getrepeated assertion failures or crashes, even6 InnoDB: immediately after the mysqld startup, there may be7 InnoDB: corruption inthe InnoDB tablespace. Please refer to8 InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html

9 InnoDB: about forcing recovery.10 121112 9:57:36 - mysqld got signal 11;11 This could be because you hit a bug. It is also possible that thisbinary12 or one of the libraries it was linked against iscorrupt, improperly built,13 or misconfigured. This error can also be caused by malfunctioning hardware.14 We will tryour best to scrape up some info that will hopefully help diagnose15 the problem, but since we have already crashed, something isdefinitely wrong16 and thismay fail.17

18 key_buffer_size=8388600

19 read_buffer_size=131072

20 max_used_connections=14

21 max_connections=800

22 threads_connected=10

23 It ispossible that mysqld could use up to24 key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 1748985K25 bytes of memory26 Hope that's ok; if not, decrease some variables in the equation.

27

28 thd=(nil)29 Attempting backtrace. You can use the following information to find out

30 where mysqld died. If you see no messages after this, something went31 terribly wrong...32 frame pointer isNULL, did you compile with33 -fomit-frame-pointer? Aborting backtrace!

34 The manual page at http://www.mysql.com/doc/en/Crashing.html contains

35 information that should help you find out what is causing the crash.

当时反应就是Kill掉这个Mysqld，尝试重启，但是重启仍然报错，于是尝试使用Mysql5.1和Mysql5.5启动，仍不能解决问题，Mysql仍然是无法启动成功。

这时候在Mysql里设置

innodb_force_recovery=1

但是Mysqld的Log里不打印Force Recovery的信息，觉得不对劲，语句又仔细去观察Mysqld的Log，发现Mysql是在APPLY Redo的时候挂掉，innodb_force_recovery=1是在Apply redo完成之后再进行Force Recovery，所以就没有进入到Force Recovery环节，于是尝试设置innodb_force_recovery=6，跳过Apply Redo，成功启动，赶紧把所有表全部DUMP出来，然后又启动一个新库再倒入进去。正好对这个测试库也升一下级。