mysqld詭異crash

突然收到告警短信，提示有一組服務器MHA已經切換，登錄服務器后查看錯誤日志如下（其中相關insert語句已經處理）：

mysql版本：5.5.24

151221 16:54:26  InnoDB: Assertion failure in thread 139867452008192 in file ha_innodb.cc line 1476
InnoDB: Failing assertion: current <= max_value
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
08:54:26 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=2
max_threads=1000
thread_count=1
connection_count=1
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4366462 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x6ba32000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f356dcade20 thread_stack 0x40000
/usr/local/mysql/bin/mysqld(my_print_stacktrace+0x2e)[0x76cf7e]
/usr/local/mysql/bin/mysqld(handle_fatal_signal+0x386)[0x6566d6]
/lib64/libpthread.so.0[0x305b20f790]
/lib64/libc.so.6(gsignal+0x35)[0x305ae32625]
/lib64/libc.so.6(abort+0x175)[0x305ae33e05]
/usr/local/mysql/bin/mysqld[0x7ceb5a]
/usr/local/mysql/bin/mysqld[0x7da5e0]
/usr/local/mysql/bin/mysqld(_ZN7handler21update_auto_incrementEv+0x209)[0x659909]
/usr/local/mysql/bin/mysqld[0x7d818f]
/usr/local/mysql/bin/mysqld(_ZN7handler12ha_write_rowEPh+0x51)[0x65b961]
/usr/local/mysql/bin/mysqld(_Z12write_recordP3THDP5TABLEP12st_copy_info+0x5d)[0x54a88d]
/usr/local/mysql/bin/mysqld(_Z12mysql_insertP3THDP10TABLE_LISTR4ListI4ItemERS3_IS5_ES6_S6_15enum_duplicatesb+0xa4a)[0x5508ba]
/usr/local/mysql/bin/mysqld(_Z21mysql_execute_commandP3THD+0x24d6)[0x5601f6]
/usr/local/mysql/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0xfe)[0x563bee]
/usr/local/mysql/bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1356)[0x564fc6]
/usr/local/mysql/bin/mysqld(_Z10do_commandP3THD+0xc2)[0x5652f2]
/usr/local/mysql/bin/mysqld(_Z24do_handle_one_connectionP3THD+0xf2)[0x5f0b02]
/usr/local/mysql/bin/mysqld(handle_one_connection+0x4a)[0x5f0bda]
/lib64/libpthread.so.0[0x305b207a51]
/lib64/libc.so.6(clone+0x6d)[0x305aee89ad]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (8e8bcc10): INSERT INTO t1 (name) VALUES ('yayun') Connection ID (thread ID): 130
Status: NOT_KILLED

提示INSERT INTO t1 (name) VALUES ('yayun');導致mysqld crash
這么簡單的語句也會導致mysqld crash？

我們看看表結構：

mysql> show create table t1\G
*************************** 1. row ***************************
       Table: t1
Create Table: CREATE TABLE `t1` (
  `id` tinyint(4) NOT NULL AUTO_INCREMENT,
  `name` char(20) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=128 DEFAULT CHARSET=utf8
1 row in set (0.00 sec)

如果不仔細看還發現不出貓膩，那么仔細看看呢？沒錯，就是id字段，是tinyint類型，tinyint的取值范圍是：帶符號的范圍是-128到127。無符號的范圍是0到255
在建表的時候就指定了AUTO_INCREMENT=128，已經到臨界點了。當再次插入就會導致mysqld進程重啟。再次插入則改值就被重新設置了。我們來看看。

mysql> CREATE TABLE t1 (id  TINYINT not null AUTO_INCREMENT PRIMARY KEY,name char(20)) ENGINE=InnoDB AUTO_INCREMENT=128;
Query OK, 0 rows affected (0.00 sec)

mysql> show create table t1\G
*************************** 1. row ***************************
       Table: t1
Create Table: CREATE TABLE `t1` (
  `id` tinyint(4) NOT NULL AUTO_INCREMENT,
  `name` char(20) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=128 DEFAULT CHARSET=utf8
1 row in set (0.00 sec)

插入一條記錄，馬上導致mysqld進程crash。

mysql>  INSERT INTO t1 (name) VALUES ('yayun');
ERROR 2013 (HY000): Lost connection to MySQL server during query

再次插入就ok了。

mysql> select * from t1;
+----+-------+
| id | name  |
+----+-------+
|  1 | yayun |
|  2 | yayun |
+----+-------+
2 rows in set (0.00 sec)

mysql> show create table t1\G
*************************** 1. row ***************************
       Table: t1
Create Table: CREATE TABLE `t1` (
  `id` tinyint(4) NOT NULL AUTO_INCREMENT,
  `name` char(20) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8
1 row in set (0.00 sec)

 

總結：

總之出現上面的情況也是歷史業務留下來的坑，已經督促開發進行更改。上面這個屬於一個bug，該bug早有人反饋，我又踩一次坑。

參考資料：

http://bugs.mysql.com/bug.php?id=66836