MySQL日志文件太大，導致磁盤空間不足，MySQL無法重啟 [ERROR] Disk is full writing '/data/mysql/logs/mysql-bin

一、定位原因
 首先，先放MySQL的相關配置

　　basedir=/usr/local/mysql
        datadir=/data/mysql/data

        #binlog
        log_slave_updates=1
        binlog_format=row
        log-bin=/data/mysql/logs/mysql-bin
        #日志文件的存放時間是7天
        expire_logs_days=7
        sync_binlog=1
        binlog_cache_size=4M
        max_binlog_size=1024M
        max_binlog_cache_size=256M
        lower_case_table_names=1
        #錯誤日志
        log-error=/data/mysql/logs/error.log

 查看error.log出現如下錯誤日志   

  2019-10-24T08:40:00.186176Z 704591 [ERROR] Disk is full writing '/data/mysql/logs/mysql-bin.000014' (Errcode: 16044192 - No space left on device). Waiting for someone to free space...
  2019-10-24T08:40:00.186193Z 704591 [ERROR] Retry in 60 secs. Message reprinted in 600 secs
  2019-10-24T08:50:00.188449Z 704591 [ERROR] Disk is full writing '/data/mysql/logs/mysql-bin.000014' (Errcode: 16044192 - No space left on device). Waiting for someone to free space...
  2019-10-24T08:50:00.188477Z 704591 [ERROR] Retry in 60 secs. Message reprinted in 600 secs
  2019-10-24T09:00:00.190693Z 704591 [ERROR] Disk is full writing '/data/mysql/logs/mysql-bin.000014' (Errcode: 16044192 - No space left on device). Waiting for someone to free space...
  2019-10-24T09:00:00.190720Z 704591 [ERROR] Retry in 60 secs. Message reprinted in 600 secs
  2019-10-24T09:10:00.192057Z 704591 [ERROR] Disk is full writing '/data/mysql/logs/mysql-bin.000014' (Errcode: 16044192 - No space left on device). Waiting for someone to free space...
  2019-10-24T09:10:00.192083Z 704591 [ERROR] Retry in 60 secs. Message reprinted in 600 secs
  2019-10-24T09:20:00.193426Z 704591 [ERROR] Disk is full writing '/data/mysql/logs/mysql-bin.000014' (Errcode: 16044192 - No space left on device). Waiting for someone to free space...
  2019-10-24T09:20:00.193453Z 704591 [ERROR] Retry in 60 secs. Message reprinted in 600 secs
  2019-10-24T09:25:51.789734Z 704901 [Note] Got packets out of order
  2019-10-24T09:30:00.195120Z 704591 [ERROR] Disk is full writing '/data/mysql/logs/mysql-bin.000014' (Errcode: 16044192 - No space left on device). Waiting for someone to free space...
  2019-10-24T09:30:00.195146Z 704591 [ERROR] Retry in 60 secs. Message reprinted in 600 secs
  2019-10-24T09:40:00.196549Z 704591 [ERROR] Disk is full writing '/data/mysql/logs/mysql-bin.000014' (Errcode: 16044192 - No space left on device). Waiting for someone to free space...
  2019-10-24T09:40:00.196575Z 704591 [ERROR] Retry in 60 secs. Message reprinted in 600 secs

查看mysql-bin

  [root@iZ2zefyvsfkm30mv8ve3ccZ logs]# ll
   total 3469032
   -rw-r----- 1 mysql mysql   39371438 Oct 30 17:27 error.log
   -rw-r----- 1 mysql mysql  431598428 Oct 18 16:47 mysql-bin.000013
   -rw-r----- 1 mysql mysql        234 Oct 30 16:47 mysql-bin.000014
   -rw-r----- 1 mysql mysql  431598428 Oct 30 16:47 mysql-bin.000015
   -rw-r----- 1 mysql mysql         68 Oct 30 16:47 mysql-bin.index
   drwxr-xr-x 2 mysql mysql       4096 Oct 30 17:11 relay-log
   -rw-r----- 1 mysql mysql 3077816504 Oct 30 16:47 slow.log

 查看磁盤空間使用情況
   　　[root@iZ2zefyvsfkm30mv8ve3ccZ logs]# df -h

  Filesystem      Size  Used Avail Use% Mounted on
  /dev/vda1        20G  19G    11M 100% /
  devtmpfs        3.9G     0  3.9G   0% /dev
  tmpfs           3.9G     0  3.9G   0% /dev/shm
  tmpfs           3.9G  600K  3.9G   1% /run
  tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
  tmpfs           783M     0  783M   0% /run/user/0

 

  /dev/vda1 空間占用100%，空間嚴重不足；

  切記不可直接刪除mysql-bin*文件，直接刪除會導致mysql-bin.index文件異常，該MySQL使用主從復制，如果直接刪除，可能無法主從復制

結果：

  日志文件太大，導致磁盤空間不足，MySQL無法自動重啟；

  解決方法；
  　　在安裝未考慮當前服務器是否掛載數據盤，就將MySQL安裝到服務器，而現在磁盤空間僅為40G，才會導致如此問題。
  　　最直接的方法就是：掛載數據盤

二、掛載數據盤
         1.查看當前服務器是否存在未掛載的數據盤，執行"fdisk -l"命令查看。

  [root@iZ2zefyvsfkm30mv8ve3ccZ logs]# fdisk -l

  Disk /dev/vda: 21.5 GB, 21474836480 bytes, 41943040 sectors
  Units = sectors of 1 * 512 = 512 bytes
  Sector size (logical/physical): 512 bytes / 512 bytes
  I/O size (minimum/optimal): 512 bytes / 512 bytes
  Disk label type: dos
  Disk identifier: 0x000a7708

     Device Boot      Start         End      Blocks   Id  System
  /dev/vda1   *        2048    41943039    20970496   83  Linux

  Disk /dev/vdb: 536.9 GB, 536870912000 bytes, 1048576000 sectors
  Units = sectors of 1 * 512 = 512 bytes
  Sector size (logical/physical): 512 bytes / 512 bytes
  I/O size (minimum/optimal): 512 bytes / 512 bytes

 

  存在未掛載數據盤 /dev/vdb,如果不存在可能需要擴容，忘了收了，該服務器是阿里雲的。

2、執行"fdisk  /dev/vdb" 對數據盤進行分區

  [root@iZ2zefyvsfkm30mv8ve3ccZ logs]# fdisk  /dev/vdb
  Command (m for help):m
  Command (m for help):n
  Partition number （1-4）：1
  First cylinder（1-175664， default 1）：1
  Last cylinder or + size or sizeM or + sizeK(1-1775664,default 175664): 回車
  Using default value 175664

  Command (m for help):wq

3、執行"fdisk -l" 查看分區情況

  [root@iZ2zefyvsfkm30mv8ve3ccZ logs]# fdisk -l

  Disk /dev/vda: 21.5 GB, 21474836480 bytes, 41943040 sectors
  Units = sectors of 1 * 512 = 512 bytes
  Sector size (logical/physical): 512 bytes / 512 bytes
  I/O size (minimum/optimal): 512 bytes / 512 bytes
  Disk label type: dos
  Disk identifier: 0x000a7708

     Device Boot      Start         End      Blocks   Id  System
  /dev/vda1   *        2048    41943039    20970496   83  Linux

  Disk /dev/vdb: 536.9 GB, 536870912000 bytes, 1048576000 sectors
  Units = sectors of 1 * 512 = 512 bytes
  Sector size (logical/physical): 512 bytes / 512 bytes
  I/O size (minimum/optimal): 512 bytes / 512 bytes
  Disk label type: dos
  Disk identifier: 0xd471030e

     Device Boot      Start         End      Blocks   Id  System
  /dev/vdb1            2048  1048575999   524286976   83  Linux

  這時候需要注意，有兩種情況，
  第一種，你這個數據盤是新購買的，以前沒有用過（沒有存儲過數據），那么就要對其進行格式化，使用“mkfs.ext3 /dev/vdb1”命令。
  第二種，如果你之前就有數據盤，只是你把系統盤重置了，需要將數據盤再次掛載到重置后的系統上，那么就不要對其格式化（跳過此步驟），直接執行后面的步驟掛載就可以了。
  由於當前數據盤是新的未曾使用過，也就不需要格式化；

4.編輯"/etc/fstab"

  由於的MySQL的datadir和log是在/data/mysql/目錄下，為了后面轉移數據先將其掛載到/data/backup(必須提前創建)目錄下，之后再正式掛載到/dat目錄下。
  在其中添加
   /dev/xvdb1  /data/backup  ext3    defaults    1  1

 

  [root@iZ2zefyvsfkm30mv8ve3ccZ logs]# vim /etc/fstab

  #
  # /etc/fstab
  # Created by anaconda on Wed Dec 12 07:53:08 2018
  #
  # Accessible filesystems, by reference, are maintained under '/dev/disk'
  # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
  #
  UUID=d67c3b17-255b-4687-be04-f29190d37396 /                              ext4    defaults        1 1
  /dev/vdb1                                 　　　　　　　　 /data/backup            ext3    defaults        1 1

 

  也可以使用
   echo '/dev/vdb1 /data/backup   ext3    defaults        1 1 ' >> /etc/fstab

 

5.使用"mount -a"掛載新數據盤，並用'df -h'命令查看

  [root@iZ2zefyvsfkm30mv8ve3ccZ logs]# mount -a
  [root@iZ2zefyvsfkm30mv8ve3ccZ logs]# df -h
  Filesystem      Size  Used Avail Use% Mounted on
  /dev/vda1        20G   19G   11M 100% /
  devtmpfs        3.9G     0  3.9G   0% /dev
  tmpfs           3.9G     0  3.9G   0% /dev/shm
  tmpfs           3.9G  552K  3.9G   1% /run
  tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
  tmpfs           783M     0  783M   0% /run/user/0
  /dev/vdb1       493G    0G  468G   0% /data/backup

  掛載成功，接下來就是轉移數據

三、轉移數據

  1、備份/data/mysql的所有數據
  cd /data/
  tar cvf /data/backup/mysql.tar mysql
  rm -rf mysql

 2.卸載數據盤
  umount -v /data/backup
 3.修改 /etc/fstab,重新掛載到/data/目錄下

 [root@iZ2zefyvsfkm30mv8ve3ccZ logs]# vim /etc/fstab

  #
  # /etc/fstab
  # Created by anaconda on Wed Dec 12 07:53:08 2018
  #
  # Accessible filesystems, by reference, are maintained under '/dev/disk'
  # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
  #
  UUID=d67c3b17-255b-4687-be04-f29190d37396 /                       ext4    defaults        1 1
  /dev/vdb1                                 /data                   ext3    defaults        1 1

 4. mount -a
  或者 mount /dev/vdb1 /data 這只是臨時掛載

  [root@iZ2zefyvsfkm30mv8ve3ccZ logs]# df -h
  Filesystem      Size  Used Avail Use% Mounted on
  /dev/vda1        20G  8.4G   11G  45% /
  devtmpfs        3.9G     0  3.9G   0% /dev
  tmpfs           3.9G     0  3.9G   0% /dev/shm
  tmpfs           3.9G  552K  3.9G   1% /run
  tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
  tmpfs           783M     0  783M   0% /run/user/0
  /dev/vdb1       493G   21G  447G   5% /data

  5. 解壓mysql.tar

  cd /data/
  tar xvf mysql.tar

 6. 重啟mysql

  [root@iZ2zefyvsfkm30mv8ve3ccZ logs]# systemctl restart mysql
  [root@iZ2zefyvsfkm30mv8ve3ccZ logs]# systemctl status mysql
  ● mysql.server.service - LSB: start and stop MySQL
     Loaded: loaded (/etc/rc.d/init.d/mysql.server; bad; vendor preset: disabled)
     Active: active (running) since Wed 2019-10-30 16:47:56 CST; 1h 37min ago
       Docs: man:systemd-sysv-generator(8)
    Process: 4369 ExecStart=/etc/rc.d/init.d/mysql.server start (code=exited, status=0/SUCCESS)
     CGroup: /system.slice/mysql.server.service
             ├─4380 /bin/sh /usr/local/mysql/bin/mysqld_safe --datadir=/data/mysql/data --pid-file=/data/mysql/data/iZ2zefyvsfkm30mv8ve3ccZ.pid
             └─5866 /usr/local/mysql/bin/mysqld --basedir=/usr/local/mysql --datadir=/data/mysql/data --plugin-dir=/usr/local/mysql/lib/plugin --user=mysql --log-error=/data/m...

  Oct 30 16:47:44 iZ2zefyvsfkm30mv8ve3ccZ systemd[1]: Starting LSB: start and stop MySQL...
  Oct 30 16:47:56 iZ2zefyvsfkm30mv8ve3ccZ mysql.server[4369]: Starting MySQL............[  OK  ]
  Oct 30 16:47:56 iZ2zefyvsfkm30mv8ve3ccZ systemd[1]: Started LSB: start and stop MySQL.

   成功。