事情要從倆月前的一個坑說起，一台新的測試服務器，新項目一元奪寶用的。
配置aws上的一台雲主機，系統盤8G，一塊300G的雲硬盤。
拿到機器后，另一運維小哥安裝php，nginx，mysql等軟件。 安裝完后，然忘了掛載雲硬盤。 隨着根分區日志越來越多，mysql數據量增大。。。磁盤已滿，mysql自行掛了。。。哭。。
開發找上門來，mysql插不進去了，我連上服務器，ps -ef。。 沒有進程。
嘗試重啟。。失敗。
查看錯誤日志。。如下：
1 161206 08:07:41 mysqld_safe Starting mysqld daemon with databases from /data/mysql 2 2016-12-06 08:07:41 4473 [Note] InnoDB: The InnoDB memory heap is disabled 3 2016-12-06 08:07:41 4473 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 4 2016-12-06 08:07:41 4473 [Note] InnoDB: Compressed tables use zlib 1.2.3 5 2016-12-06 08:07:41 4473 [Note] InnoDB: Using Linux native AIO 6 2016-12-06 08:07:41 4473 [Note] InnoDB: Using CPU crc32 instructions 7 2016-12-06 08:07:41 4473 [Note] InnoDB: Initializing buffer pool, size = 128.0M 8 2016-12-06 08:07:41 4473 [Note] InnoDB: Completed initialization of buffer pool 9 2016-12-06 08:07:41 4473 [Note] InnoDB: Highest supported file format is Barracuda. 10 2016-12-06 08:07:41 4473 [Note] InnoDB: Log scan progressed past the checkpoint lsn 416493596 11 2016-12-06 08:07:41 4473 [Note] InnoDB: Database was not shutdown normally! 12 2016-12-06 08:07:41 4473 [Note] InnoDB: Starting crash recovery. 13 2016-12-06 08:07:41 4473 [Note] InnoDB: Reading tablespace information from the .ibd files... 14 2016-12-06 08:07:41 4473 [Note] InnoDB: Restoring possible half-written data pages 15 2016-12-06 08:07:41 4473 [Note] InnoDB: from the doublewrite buffer... 16 InnoDB: Doing recovery: scanned up to log sequence number 416570218 17 2016-12-06 08:07:41 4473 [Note] InnoDB: Starting an apply batch of log records to the database... 18 InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 19 InnoDB: Apply batch completed 20 InnoDB: Last MySQL binlog file position 0 177880661, file name mysql-bin.000020 21 2016-12-06 08:07:42 4473 [Note] InnoDB: 128 rollback segment(s) are active. 22 2016-12-06 08:07:42 4473 [Note] InnoDB: Waiting for purge to start 23 2016-12-06 08:07:42 4473 [Note] InnoDB: 5.6.14 started; log sequence number 416570218 24 /usr/local/app/mysql-5.6.14/bin/mysqld: Error writing file '/data/mysql/mysql-slow.log' (Errcode: 28 - No space left on device) 25 2016-12-06 08:07:42 4473 [ERROR] Could not use /data/mysql/mysql-slow.log for logging (error 28). Turning logging off for the whole duration of the MySQL server process. To turn it on again: fix the cause, shutdown the MySQL server and restart it. 26 08:07:42 UTC - mysqld got signal 11 ; 27 This could be because you hit a bug. It is also possible that this binary 28 or one of the libraries it was linked against is corrupt, improperly built, 29 or misconfigured. This error can also be caused by malfunctioning hardware. 30 We will try our best to scrape up some info that will hopefully help 31 diagnose the problem, but since we have already crashed, 32 something is definitely wrong and this may fail. 33 34 key_buffer_size=33554432 35 read_buffer_size=33554432 36 max_used_connections=0 37 max_threads=1000 38 thread_count=0 39 connection_count=0 40 It is possible that mysqld could use up to 41 key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 65582393 K bytes of memory 42 Hope that's ok; if not, decrease some variables in the equation. 43 44 Thread pointer: 0x0 45 Attempting backtrace. You can use the following information to find out 46 where mysqld died. If you see no messages after this, something went 47 terribly wrong... 48 stack_bottom = 0 thread_stack 0x40000 49 /usr/local/app/mysql-5.6.14/bin/mysqld(my_print_stacktrace+0x35)[0x8dd875] 50 /usr/local/app/mysql-5.6.14/bin/mysqld(handle_fatal_signal+0x40b)[0x64ddbb] 51 /lib64/libpthread.so.0(+0xf7e0)[0x7f926e2b77e0] 52 The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains 53 information that should help you find out what is causing the crash. 54 161206 08:07:42 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
第25行亮了。No space left on device。找到問題了，df一看，根分區使用率100%。。。剩下/app目錄300G使用率1%在那孤零零掛着。
解決辦法：
移動原有文件到數據分區
# mv /data/mysql /app/
將
datadir = /data/mysql 改為datadir = /app/mysql
重啟正常。