今天生產環境logstash與elasticsearch出現“cluster_block_exception“報錯的問題,
logstash報錯現象
[2018-12-26T16:18:06,709][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
elasticsearch報錯現象
[2018-12-26T08:00:00,434][WARN ][o.e.x.m.e.l.LocalExporter] unexpected error while indexing monitoring document org.elasticsearch.xpack.monitoring.exporter.ExportException: ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]
經過排查,解決方法
curl -XPUT -H "Content-Type: application/json" http://10.1.3.80:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'
10.1.3.80為es的地址
注:es的硬盤使用率不要超過95%