0x00 概述
研發反饋skywalking突然無法使用，查看日志發現
2020-10-19 14:25:32,712 - org.apache.skywalking.apm.collector.cache.caffeine.service.ServiceNameCacheCaffeineService -82494825 [grpc-default-executor-149] ERROR [] - No shard available for [get [service_name][type][-87]: routing [null]] org.elasticsearch.action.NoShardAvailableActionException: No shard available for [get [service_name][type][-87]: routing [null]] at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:209) ~[elasticsearch-5.5.0.jar:5.5.0] at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.start(TransportSingleShardAction.java:186) ~[elasticsearch-5.5.0.jar:5.5.0] at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:95) ~[elasticsearch-5.5.0.jar:5.5.0] at org.elasticsearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:59) ~[elasticsearch-5.5.0.jar:5.5.0] at org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:146) ~[elasticsearch-5.5.0.jar:5.5.0] at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.5.0.jar:5.5.0] at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.5.0.jar:5.5.0]
錯誤關鍵字 No shard available for
0x01 磁盤寫滿
經過搜索發現，該類報錯一般是磁盤慢導致的，通過命令df -h發現，存儲數據/data目錄使用量正常，但是es7的安裝目錄/app已經寫滿；
排查發現，安裝目錄內es7的日志文件過多，導致/app目錄被寫滿，刪除部分過期日志后，集群恢復正常；
0x02 總結
除了ES數據存儲目錄需要保持足夠的使用空間，ES自己的運行日志也需要有足夠的磁盤空間，以上兩個任意1個磁盤空間滿了，都會包磁盤不足的錯誤；
注意在es有大量業務的情況下，es自身也會產生大量日志，需要注意磁盤清理或者配置監控。