elasticsearch数据过期删除处理

一、概述

使用elasticsearch收集日志进行处理，时间久了，很老的数据就没用了或者用途不是很大，这个时候就要对过期数据进行清理.这里介绍两种方式清理这种过期的数据。

1、curator

关于版本：

 

安装：

https://www.elastic.co/guide/en/elasticsearch/client/curator/current/installation.html

我使用的是ubuntu系统，所以参考的是https://www.elastic.co/guide/en/elasticsearch/client/curator/current/apt-repository.html

wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
 vim /etc/apt/sources.list.d/curator.list deb [arch=amd64] https://packages.elastic.co/curator/5/debian stable main

sudo apt-get update && sudo apt-get install elasticsearch-curator

 我使用的是elasticsearch-6.5.1，所以安装的是curator5.

安装完成后会生成两个命令：curator、curator_cli,这里我们只先用到curator。

需要创建配置文件：有两个文件一个是config、一个是action

mkdir  {/etc/curator,/data/curator}

config：

# cat config_file.yml client: hosts: - 127.0.0.1 port: 9200 url_prefix: use_ssl: False certficate: client_cert: client_key: ssl_no_validate: False http_auth: timeout: master_only: true logging: loglevel: INFO logfile: "/data/curator/action.log" logformat: default

action:

# cat action_file.yml

--- actions: 1: action: delete_indices description: >- Delete indices older than 7 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly. options: ignore_empty_list: True timeout_override: continue_if_exception: False disable_action: False filters: - filtertype: pattern kind: regex value: '^apm-6.5.1-transaction-|^apm-6.5.1-span-' exclude: - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit_count: 15 exclude: 2: action: delete_indices description: >- Delete indices older than 7 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly. options: ignore_empty_list: True timeout_override: continue_if_exception: False disable_action: False filters: - filtertype: pattern kind: prefix value: loadbalance-api- exclude: - filtertype: age source: name direction: older timestring: '%Y-%m-%d' unit: days unit_count: 20 exclude:

 

---
actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 7 days (based on index name), for logstash-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: regex
      value: 'fluentd-k8s-(2019.02.11|2019.02.12)$'
      exclude: true
    - filtertype: pattern
      kind: prefix
      value: fluentd-k8s-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 15
      exclude:

 

 

 可以设置多个action，每个都以不同的数字分割，使用不同的清理策略，具体可以参考https://www.elastic.co/guide/en/elasticsearch/client/curator/5.6/actions.html

注意自己的index的格式，比如我这里的时间格式有两种：

注意匹配，否则那个action就返回空列表，从而不会删除。

这个历史数据重要的会先落地到hdfs，然后在删除。这个日期根据自己服务器的磁盘和日志的重要性自己规划。重要的比如双11的数据不想删除，想留下来可以写到exclude里面，

或者做一个snapshot备份。接下来设置一个定时任务去删除就好了。

crontab -e *  *  */25 * *  curator --config /etc/curator/config_file.yml  /etc/curator/action_file.yml

 

2、使用脚本删除

 

# cat es-dele-indices.sh #!/bin/bash #delete elasticsearch indices searchIndex=fluentd-k8s elastic_url=127.0.0.1 elastic_port=9200 date2stamp(){ date --utc --date "$1" +%s } dateDiff(){ case $1 in
    -s)  sec=1;     shift;; -m)  sec=60;    shift;; -h)  sec=3600;  shift;; -d)  sec=86400; shift;; *)  sec=86400; shift;; esac dte1=$(date2stamp $1) dte2=$(date2stamp $2) diffSec=$((dte2-dte1)) if ((diffSec < 0)); then abs=-1; else abs=1; fi
  echo $((diffSec/sec*abs)) } for index in $(curl -s "${elastic_url}:${elastic_port}/_cat/indices?v" | grep -E " ${searchIndex}-20[0-9][0-9]\.[0-1][0-9]\.[0-3][0-9]" | awk '{ print $3 }');do
  date=$(echo ${index: -10}|sed 's/\./-/g') cond=$(date +%Y-%m-%d) diff=$(dateDiff -d $date $cond) echo -n "${index} (${diff})"
  if [ $diff -gt 1 ]; then #echo "/ DELETE" curl -XDELETE "${elastic_url}:${elastic_port}/${index}?pretty"
  else
    echo ""
  fi
done