etcd服務無法寫入數據

偶然聽開發說服務發現是不是出故障了，查看日志發現Error: etcdserver: mvcc: database space exceeded，發現ETCD默認最大空間使用已經達到了2GB，臨時將etcd庫直接刪除重啟這種暴力方式解決了，但是沒有解決根本問題，下次一定還會再出現，所以在本地測試一下增加空間和自動壓縮。

先將etcd最大空間設置為16M啟動

[root@localhost ~]#  etcd --quota-backend-bytes=$((16*1024*1024))

循環寫入數據

[root@localhost ~]# while [ 1 ]; do dd if=/dev/urandom bs=1024 count=1024  | ETCDCTL_API=3 etcdctl put key  || break; done
1024+0 records in
1024+0 records out
1048576 bytes (1.0 MB) copied, 0.010793 s, 97.2 MB/s
OK
1024+0 records in
1024+0 records out
1048576 bytes (1.0 MB) copied, 0.0178137 s, 58.9 MB/s
OK
1024+0 records in
1024+0 records out
1048576 bytes (1.0 MB) copied, 0.0270086 s, 38.8 MB/s
OK
1024+0 records in
1024+0 records out
1048576 bytes (1.0 MB) copied, 0.0208378 s, 50.3 MB/s
OK
1024+0 records in
1024+0 records out
1048576 bytes (1.0 MB) copied, 0.0213647 s, 49.1 MB/s
OK
1024+0 records in
1024+0 records out
1048576 bytes (1.0 MB) copied, 0.0180274 s, 58.2 MB/s
OK
1024+0 records in
1024+0 records out
1048576 bytes (1.0 MB) copied, 0.017903 s, 58.6 MB/s
OK
1024+0 records in
1024+0 records out
1048576 bytes (1.0 MB) copied, 0.00862558 s, 122 MB/s
OK
1024+0 records in
1024+0 records out
1048576 bytes (1.0 MB) copied, 0.0276305 s, 37.9 MB/s
OK
1024+0 records in
1024+0 records out
1048576 bytes (1.0 MB) copied, 0.0169549 s, 61.8 MB/s
OK
1024+0 records in
1024+0 records out
1048576 bytes (1.0 MB) copied, 0.0390035 s, 26.9 MB/s
Error: etcdserver: mvcc: database space exceeded

發現上述空間已滿，查看如下超出16MB限制

[root@localhost ~]# ETCDCTL_API=3 etcdctl --endpoints="http://192.168.32.134:2379" --write-out=table endpoint status
+----------------------------+------------------+---------+---------+-----------+-----------+------------+
|          ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+----------------------------+------------------+---------+---------+-----------+-----------+------------+
| http://192.168.32.134:2379 | 8e9e05c52164694d |   3.3.8 |   19 MB |      true |         2 |         16 |
+----------------------------+------------------+---------+---------+-----------+-----------+------------+

我們再次嘗試往里面寫入數據，發現還是空間已滿

[root@localhost ~]# ETCDCTL_API=3 etcdctl put newkey 123
Error: etcdserver: mvcc: database space exceeded

查看告警被觸發

[root@localhost ~]# ETCDCTL_API=3 etcdctl alarm list
memberID:10276657743932975437 alarm:NOSPACE

刪除多讀的鍵空間將把集群帶回配額限制，因此警告能被接觸:

#獲取當前修訂版本
[root@localhost ~]# rev=$(ETCDCTL_API=3 etcdctl --endpoints=:2379 endpoint status --write-out="json" | egrep -o '"revision":[0-9]*' | egrep -o '[0-9].*')
#壓縮所有舊的修訂版本
[root@localhost ~]# ETCDCTL_API=3 etcdctl compact $rev
compacted revision 1516
#反碎片化過度空間
[root@localhost ~]# ETCDCTL_API=3 etcdctl defrag
Finished defragmenting etcd member[127.0.0.1:2379]
#解除警告
[root@localhost ~]# ETCDCTL_API=3 etcdctl alarm disarm
memberID:10276657743932975437 alarm:NOSPACE
#測試put又允許了
[root@localhost ~]# ETCDCTL_API=3 etcdctl put newkey newvalue
OK

針對上述改進如下：
修改etcd.service文件增加每小時自動壓縮和空間限額為8G

[root@localhost ~]# vim /etc/systemd/system/etcd.service
[Unit]
Description=Etcd
After=network.target
Before=flanneld.service

[Service]
User=root
ExecStart=/usr/local/bin/etcd \
-name etcd1 \
-data-dir /var/lib/etcd \
--advertise-client-urls http://192.168.32.134:2379,http://127.0.0.1:2379 \
--listen-client-urls http://192.168.32.134:2379,http://127.0.0.1:2379 \
--auto-compaction-retention=1 \     #開啟每隔一個小時自動壓縮
--quota-backend-bytes=8388608000    #磁盤空間調整為8GB
Restart=on-failure
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

重啟服務

[root@localhost ~]# systemctl daemon-reload;systemctl restart etcd

ETCD相關操作命令：

#列出所有key
ETCDCTL_API=3 etcdctl get --prefix ""
#基於相同前綴查找
ETCDCTL_API=3 etcdctl get /test/ok --prefix
#添加key
ETCDCTL_API=3 etcdctl put /test/ok 11
#刪除key
ETCDCTL_API=3 etcdctl del /test/ok
#刪除所有/test前綴的節點
ETCDCTL_API=3 etcdctl del /test --prefix
#監聽key
ETCDCTL_API=3 etcdctl wath /test/ok
#監聽子節點
ETCDCTL_API=3 etcdctl /test/ok --prefix
#查看etcd集群成員狀態
ETCDCTL_API=3 etcdctl --write-out=table --endpoints=localhost:2379 member list