一、日志文件樣式
二、目標
1、備份壓縮.log結尾&&時間樣式為“date +%Y%m%d”的日志文件(如：20170912、20160311等)
2、可指定壓縮范圍（N天前至當天）：如：今天、昨天（date -d "-1 day" +%Y%m%d）至今天、前天至今天
壓縮命名格式為：日期.tar.gz(或：日期.tar.bz2)，壓縮N天范圍內文件后將會生成N個壓縮包
3、可指定壓縮模式（二選一）：tar czf 或 tar cjf
4、可指定刪除范圍：刪除N天前的日志文件，如：今天為20170912，刪除3天前日志，即刪除20170908及其以前的日志文件
5、每周五（不一定周五，可在腳本中指定周幾打包）將壓縮文件打包（打包成功后刪除壓縮文件），並上傳到日志服務器，上傳成功后刪除打包文件
6、默認備份當天日志文件，壓縮模式為“czf”，刪除三天前日志文件
三、腳本內容
#!/bin/bash #author:xiami #date:20170907 #description: compress files of a specified mode and delete logs before a particular date #version:v0.1 strategy_logs_path="/root/apps/myapp/log" date=$(date +%Y%m%d) #------------初始化函數默認參數---------- init_argv(){ #example:init_argv -d 4 -m cjf -r 5 打包壓縮近四天及當天日志，壓縮模式cjf，刪除五天前日志 compress_date=$(date +%Y%m%d) backup_mode="czf" del_days_num=3 backup_path="/root/apps/logs_backup" line=$(ps -ef |grep "ssh-agent"|awk '{if($0!~/grep ssh-agent/)print $0}'|wc -l) while getopts d:m:r: opt do case $opt in d) compress_days="$OPTARG" ;; m) backup_mode="$OPTARG" ;; r) del_days_num="$OPTARG" ;; \?) echo "Usage: `basename $0` [d|m|r]" echo "-d 'Integer' (tar & compress Integer day logfile)" echo "-m 'czf|cjf' (mode)" echo "-r 'Interger' (remove files Integer days ago)" exit 1 ;; esac done if [ ! -d "$backup_path" ];then mkdir -p $backup_path fi # if [ line -lt 1 ];then # echo "Not running ssh-agent" # exit 1 # fi } #------------打包並壓縮日志文件操作函數-------------------- tar_compress_log(){ #參數1：壓縮日期 參數2：壓縮備份模式 local compress_date="$1" local backup_mode="$2" cd "$strategy_logs_path" # ls . -name "*$date*.log"|xargs tar $backup_mode "$date.tar.gz" #壓縮當天日志 if [ "$backup_mode" == 'czf' ];then ls | grep ".*[0-9]\{8\}.*\.log$"|grep ".*"$compress_date".*"|xargs tar $backup_mode "$backup_path/"$1".tar.gz" elif [ "$backup_mode" == 'cjf' ];then ls | grep ".*[0-9]\{8\}.*\.log$"|grep ".*"$compress_date".*"|xargs tar $backup_mode "$backup_path/"$1".tar.bz2" else echo "tar mode error";exit 2 fi } #------------打包並壓縮N天前到當前日期范圍的日志文件---------- compress_range_date(){ #參數1：壓縮備份天數 參數2：壓縮備份模式 local dates="$date" local compress_days="$1" local backup_mode="$2" if [[ ! -z "$compress_days" && "$compress_days" -ne 0 ]];then for i in `seq 1 "$compress_days"`;do dates=$(date -d "-$i day" +%Y%m%d) # n=$(echo $dates|awk '{print NF}') tar_compress_log "$dates" "$backup_mode" done else tar_compress_log "$dates" "$backup_mode" fi } #------------每周五打包當周壓縮文件並在打包成功后刪除單個壓縮文件------------ tar_file(){ weekday=$(date +%u) if [ "$weekday" -eq 5 ];then cd "$backup_path" ls|xargs tar cf "$date-Fri-logs.tar" && rm *.tar.?z* fi } #-----------刪除(del_log_days)天前日志文件(根據mtime)--------- delete_days_log1(){ #參數1：del_days_num local del_days_num="$1" cd "$strategy_logs_path" find . -type f -mtime +$del_days_num |grep ".*[0-9]\{8\}.*\.log"|xargs rm aa -rf #刪除操作，請謹慎 } #----------刪除(del_log_days)天前日志文件（根據log文件名）---- delete_days_log2(){ #參數1：del_days_num local del_days_num="$1" cd "$strategy_logs_path" local num=$(ls |grep ".*[0-9]\{8\}.*\.log$" |grep -o "[0-9]\{8\}"|awk '!day[$0]++'|wc -l) local files=$(ls |grep ".*[0-9]\{8\}.*\.log$" |grep -o "[0-9]\{8\}"|awk '!day[$0]++') if [ ! -z "$num" ];then local field="" for i in `seq 1 "$num"` do field=$(echo $files|awk '{print $v}' v=$i) if [ "$field" -lt $(date -d "-$del_days_num day" +%Y%m%d) ];then ls *$field* |grep '.*[0-9]\{8\}.*\.log$'|xargs rm aa -rf fi done fi } #-----------上傳到日志服務器------------- upload_tgz(){ scp $backup_path/$date.tar.gz 192.168.119.133:/opt/ && rm $backup_path/$date.tar.gz #scp $backup_path/$date*-logs.tar 192.168.119.133:/opt/ && rm $backup_path/$date*-logs.tar } #-----------main------------------------- main(){ init_argv $argv compress_range_date "$compress_days" "$backup_mode" delete_days_log2 $del_days_num tar_file # upload_tgz } #----------執行部分---------------------- argv="" until [ "$#" -eq 0 ];do argv="$argv $1" shift done main $argv
四、運行腳本
日志文件初始數量為圖一所示
4.1 帶參數運行后
4.2 不帶參數運行（默認情況，備份當天日志文件，壓縮模式為“czf”，刪除三天前日志文件）
為了使測試環境一致，首先恢復被刪除日志文件，刪除壓縮文件
不帶參數運行腳本
五、計划任務
略
腳本完善：
1、刪除每天的日志文件，機器中只保留近N天（默認14天）的壓縮文件
2、考慮到安全性，文件上傳改為：遠端日志服務器從機器拉取每天對應的日志文件，而不是機器主動上傳。
3、一次備份多個目錄下日志，目錄列表存放於scan_logdir.inc
完善后腳本：
#!/bin/bash #author:xiami #date:20170907 #description: compress files of a specified mode and delete logs before a particular date, reference: http://www.cnblogs.com/xiami-xm/p/7511087.html #version:v0.2 date=$(date +%Y%m%d) scan_log_dir="$(dirname $0)/scan_logdir.inc" #------------初始化函數默認參數---------- init_argv(){ #example:init_argv -d 4 -m cjf -r 5 打包壓縮近四天及當天日志，壓縮模式cjf，刪除五天前日志 compress_date=$(date +%Y%m%d) backup_mode="cjf" del_days_num=3 backup_path="/root/apps/logs_backup" stay_days=14 line=$(ps -ef |grep "ssh-agent"|awk '{if($0!~/grep ssh-agent/)print $0}'|wc -l) while getopts d:m:r:s: opt do case $opt in d) compress_days="$OPTARG" ;; m) backup_mode="$OPTARG" ;; r) del_days_num="$OPTARG" ;; s) stay_days="$OPTARG" ;; \?) echo "Usage: `basename $0` [d|m|r]" echo "-d 'Integer' (tar & compress Integer day logfile)" echo "-m 'czf|cjf' (mode)" echo "-r 'Interger' (remove files Integer days ago)" exit 1 ;; esac done if [ ! -d "$backup_path" ];then mkdir -p $backup_path fi } #------------（實際操作函數）打包並壓縮日志文件，壓縮成功后立即將文件刪除-------------------- tar_compress_log(){ #參數1：壓縮日期 參數2：壓縮備份模式 local compress_date="$1" local backup_mode="$2" local logs_path="$3" cd "$logs_path" # cd "$strategy_logs_path" # ls . -name "*$date*.log"|xargs tar $backup_mode "$date.tar.gz" #壓縮當天日志 if [ "$backup_mode" == 'czf' ];then ls | grep ".*[0-9]\{8\}.*\.\(log\|csv\)$"|grep ".*"$compress_date".*"|xargs tar $backup_mode $backup_path/dir$a.$compress_date.tar.gz >/dev/null 2>&1 && ls | grep ".*[0-9]\{8\}.*\.\(log\|csv\)$"|grep ".*"$compress_date".*"|xargs rm -rf elif [ "$backup_mode" == 'cjf' ];then ls | grep ".*[0-9]\{8\}.*\.\(log\|csv\)$"|grep ".*"$compress_date".*"|xargs tar $backup_mode $backup_path/dir$a.$compress_date.tar.bz2 >/dev/null 2>&1 && ls | grep ".*[0-9]\{8\}.*\.\(log\|csv\)$"|grep ".*"$compress_date".*"|xargs rm -rf else echo "tar mode error";exit 2 fi } #------------檢查scan_logdir.inc是否有數據------------------- check_scan(){ local line=$(cat "$scan_log_dir" | wc -l) if [ "$line" -eq 0 ];then exit 3; fi } #------------需要備份的日志目錄------------------------------- backup_all_logs_dir(){ check_scan local a=1 while read line;do local logs_path="$line" compress_range_date "$compress_days" "$backup_mode" "$logs_path" # delete_days_log1 "$del_days_num" "$logs_path" delete_days_log2 "$del_days_num" "$logs_path" let a++;echo "-----------------$a----------------------------------" done < "$scan_log_dir" } #------------打包並壓縮N天前到當前日期范圍的日志文件---------- compress_range_date(){ #參數1：壓縮備份天數 參數2：壓縮備份模式 local dates="$date" local compress_days="$1" local backup_mode="$2" local logs_path="$3" # if [[ ! -z "$compress_days" && "$compress_days" -ne 0 ]];then if [ ! -z "$compress_days" ];then for i in `seq 0 "$compress_days"`;do dates=$(date -d "-$i day" +%Y%m%d) # n=$(echo $dates|awk '{print NF}') tar_compress_log "$dates" "$backup_mode" "$logs_path" done else tar_compress_log "$dates" "$backup_mode" "$logs_path" fi } #------------每周五打包當周壓縮文件並在打包成功后刪除單個壓縮文件------------ tar_file(){ weekday=$(date +%u) if [ "$weekday" -eq 5 ];then cd "$backup_path" ls|xargs tar cf "$date-Fri-logs.tar" && rm *.tar.?z* fi } #-----------只保留14天內的壓縮日志文件------------------------------------ stay_tgz_log(){ local stay_days="$stay_days" local rm_date=$(date -d "-$stay_days day" +%Y%m%d) cd $backup_path ls |grep -o "[0-9]\{8\}"|awk '!aa[$0]++'|while read line;do if [ "$line" -le $rm_date ];then ls *$line* |xargs rm -rf fi done } #-----------刪除(del_log_days)天前日志文件(根據mtime)--------- delete_days_log1(){ #參數1：del_days_num local del_days_num="$1" local logs_path="$2" # cd "$strategy_logs_path" cd "$logs_path" find . -type f -mtime +$del_days_num |grep ".*[0-9]\{8\}.*\.\(log\|csv\)$"|xargs rm -rf #刪除操作，請謹慎 } #----------刪除(del_log_days)天前日志文件（根據log文件名）---- delete_days_log2(){ #參數1：del_days_num local del_days_num="$1" local logs_path="$2" # cd "$strategy_logs_path" cd "$logs_path" local num=$(ls |grep ".*[0-9]\{8\}.*\.\(log\|csv\)$" |grep -o "[0-9]\{8\}"|awk '!day[$0]++'|wc -l) local files=$(ls |grep ".*[0-9]\{8\}.*\.\(log\|csv\)$" |grep -o "[0-9]\{8\}"|awk '!day[$0]++') if [ ! -z "$num" ];then local field="" for i in `seq 1 "$num"` do field=$(echo $files|awk '{print $v}' v=$i) if [ "$field" -le $(date -d "-$del_days_num day" +%Y%m%d) ];then ls *$field* |grep '.*[0-9]\{8\}.*\.\(log\|csv\)$'|xargs rm -rf fi done fi } #-----------上傳到日志服務器------------- upload_tgz(){ scp $backup_path/$date.tar.gz 192.168.119.133:/opt/ && rm $backup_path/$date.tar.gz #scp $backup_path/$date*-logs.tar 192.168.119.133:/opt/ && rm $backup_path/$date*-logs.tar } #-----------main------------------------- main(){ init_argv $argv backup_all_logs_dir # compress_range_date "$compress_days" "$backup_mode" # delete_days_log2 $del_days_num stay_tgz_log $stay_days # tar_file # upload_tgz } #----------執行部分---------------------- argv="" until [ "$#" -eq 0 ];do argv="$argv $1" shift done main $argv
日志服務器拉取腳本：
#!/bin/bash date=$(date +%Y%m%d) remote_logs_backup=/root/apps/logs_backup local_logs_backup=/root/apps/local_logs_backup hostsfile="$(dirname $0)/hosts.inc" while read line;do if [ ! -d "$local_logs_backup/$line" ];then mkdir -p "$local_logs_backup/$line" fi scp root@$line:$remote_logs_backup/*$date* $local_logs_backup/$line >/dev/null 2>&1 #&& ssh -nl root $line "rm $remote_logs_backup/* -rf" >/dev/null 2>&1 #ssh登陸或拷貝操作時若文件不存在會報錯，可忽略 done < $hostsfile
[root@xiamihost3 script]# cat hosts.inc 192.168.119.131 #hosts.inc中是目標機器的ip地址 192.168.119.133
基本功能已實現，但腳本還有待改進，例如腳本中使用的grep可以換成egrep，可以少一層轉義。
上傳備份功能還未進行測試，可能會出現錯誤，當然，上傳前肯定得先將本地公鑰放入服務器的authorized_keys文件中，實現ssh無密登陸。
最后想感慨一句，還是python更強大
good good study，day day up