inception+archery SQL審核平台

關閉防火牆和selinux

宿主機安裝mysql，創建archery數據庫，並給所有權限，允許遠程連接到該數據庫

grant all privileges on *.* to 'root'@'%' identified by 'jason_zhang' with grant option;
flush privileges;

 

安裝依賴環境

yum install  ncurses-libs libncurses5-dev ncurses-devel wget  git cmake openssl gcc-c++ zlib zlib-devel openssl-devel -y

注意:centos7 系統自帶的bison的版本過高，在后面測試的時候會報錯！安裝bison-2.5.1

[root@archery tools]# wget http://ftp.gnu.org/gnu/bison/bison-2.5.1.tar.gz
[root@archery tools]# tar -zxvf bison-2.5.1.tar.gz 
[root@archery tools]# cd bison-2.5.1
[root@archery bison-2.5.1]# ./configure &&make &&make install

安裝 Python-3.4.1.tgz

[root@archery tools]#wget https://www.python.org/ftp/python/3.4.1/Python-3.4.1.tgz

[root@archery tools]# tar zxvf Python-3.4.1.tgz 
[root@archery tools]# cd Python-3.4.1
[root@archery Python-3.4.1]# ./configure &&make &&make install

　安裝setuptools

[root@archery tools]# wget --no-check-certificate  https://pypi.python.org/packages/source/s/setuptools/setuptools-19.6.tar.gz#md5=c607dd118eae682c44ed146367a17e26
[root@archery tools]# tar zxvf setuptools-19.6.tar.gz 
[root@archery tools]# cd setuptools-19.6
[root@archery setuptools-19.6]# python3 setup.py build
[root@archery setuptools-19.6]# python3 setup.py install

安裝pip3

[root@archery tools]# wget --no-check-certificate  https://pypi.python.org/packages/source/p/pip/pip-8.0.2.tar.gz#md5=3a73c4188f8dbad6a1e6f6d44d117eeb
[root@archery tools]tar zxvf pip-8.0.2.tar.gz
[root@archery tools]# cd pip-8.0.2
[root@archery pip-8.0.2]# python3 setup.py build
[root@archery pip-8.0.2]# python3 setup.py install

　安裝python3基礎虛擬環境

[root@archery tools]# pip3 install virtualenv
[root@archery tools]# virtualenv venv4archer --python=python3.4
[root@archery tools]# source venv4archer/bin/activate

下載Release v1.3.7　上傳到服務器上、解壓

還需要安裝一些依賴環境

(venv4archer) [root@archery archery-1.3.7]# yum install python36u-devel mysql-devel  -y

(venv4archer) [root@archery archery-1.3.7]# yum install python-devel -y

(venv4archer) [root@archery archery-1.3.7]#  yum install openldap-devel
(venv4archer) [root@archery archery-1.3.7]#pip3 install pyldap

　　

(venv4archer) [root@archery archery-1.3.7]# pip3 install -r requirements.txt

　安裝docker,並啟動服務

yum install -y docker
yum install -y docker-compose
systemctl start docker
systemctl enable docker

 

創建inception和archery配置文件

inception

(venv4archer) [root@archery dockersrc]# mkdir -p /opt/inception

(venv4archer) [root@archery dockersrc]# pwd
/opt/inception
(venv4archer) [root@archery dockersrc]# vim inc.cnf 

[inception]
general_log=1
general_log_file=inception.log
port=6669
socket=/tmp/inc.socket
character-set-client-handshake=0
character-set-server=utf8
inception_remote_system_password=jason_zhang
inception_remote_system_user=root
inception_remote_backup_port=3306
inception_remote_backup_host=192.168.199.177
inception_support_charset=utf8,utf8mb4
inception_osc_on=ON
inception_osc_bin_dir=/usr/bin
                            

archery

mkdir -p /opt/archery/downloads/log

cd /opt/archery

 

(venv4archer) [root@archery archer]# vim settings.py 

(venv4archer) [root@archery archer]# cat settings.py |grep -Ev '^$|^#'
# -*- coding: UTF-8 -*- 
# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
import os
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = 'hfusaf2m4ot#7)fkw#di2bu6(cv0@opwmafx5n#6=3d%x^hpl6'
# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = True
ALLOWED_HOSTS = ['*']
# 解決nginx部署跳轉404
USE_X_FORWARDED_HOST = True
# 請求限制
DATA_UPLOAD_MAX_MEMORY_SIZE = 15728640
# Application definition
INSTALLED_APPS = (
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'django_apscheduler',
    'sql',
    'themis',
)
MIDDLEWARE = (
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
    'django.middleware.security.SecurityMiddleware',
    'common.middleware.check_login_middleware.CheckLoginMiddleware',
    'common.middleware.exception_logging_middleware.ExceptionLoggingMiddleware',
)
ROOT_URLCONF = 'archery.urls'
TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [os.path.join(BASE_DIR, 'common/templates')],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
                'common.utils.global_info.global_info',
            ],
        },
    },
]
WSGI_APPLICATION = 'archery.wsgi.application'
# Internationalization
LANGUAGE_CODE = 'zh-hans'
TIME_ZONE = 'Asia/Shanghai'
USE_I18N = True
USE_TZ = False
# 時間格式化
USE_L10N = False
DATETIME_FORMAT = 'Y-m-d H:i:s'
DATE_FORMAT = 'Y-m-d'
# Static files (CSS, JavaScript, Images)
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'static')
STATICFILES_DIRS = [os.path.join(BASE_DIR, 'common/static'), ]
# 擴展django admin里users字段用到，指定了sql/models.py里的class users
AUTH_USER_MODEL = "sql.users"
AUTH_PASSWORD_VALIDATORS = [
    {
        'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
        'OPTIONS': {
            'min_length': 9,
        }
    },
    {
        'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
    },
]
###############以下部分需要用戶根據自己環境自行修改###################
# 該項目本身的mysql數據庫地址
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'archery',
        'USER': 'root',
        'PASSWORD': 'jason_zhang',
        'HOST': '192.168.199.177',
        'PORT': '3306',
        'OPTIONS': {
            'init_command': "SET sql_mode='STRICT_TRANS_TABLES'",
            'charset': 'utf8mb4'
        },
        'TEST': {
            'NAME': 'test_archery',
            'CHARSET': 'utf8',
        },
    }
}
# themis審核所需mongodb數據庫，賬號角色必須有"anyAction" to "anyResource"權限
MONGODB_DATABASES = {
    "default": {
        "NAME": 'themis',
        "USER": 'root',
        "PASSWORD": 'jason_zhang',
        "HOST": 'mongo',
        "PORT": 27017,
    },
}
# 緩存配置
CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.filebased.FileBasedCache',
        'LOCATION': os.path.join(BASE_DIR, "archery"),
    }
}
# LDAP
ENABLE_LDAP = False
if ENABLE_LDAP:
    import ldap
    from django_auth_ldap.config import LDAPSearch
    AUTHENTICATION_BACKENDS = (
        'django_auth_ldap.backend.LDAPBackend',  # 配置為先使用LDAP認證，如通過認證則不再使用后面的認證方式
        'django.contrib.auth.backends.ModelBackend',  # django系統中手動創建的用戶也可使用，優先級靠后。注意這2行的順序
    )
    AUTH_LDAP_SERVER_URI = "ldap://xxx"
    AUTH_LDAP_USER_DN_TEMPLATE = "cn=%(user)s,ou=xxx,dc=xxx,dc=xxx"
    AUTH_LDAP_ALWAYS_UPDATE_USER = True  # 每次登錄從ldap同步用戶信息
    AUTH_LDAP_USER_ATTR_MAP = {  # key為archery.sql_users字段名，value為ldap中字段名，用戶同步信息
        "username": "cn",
        "display": "displayname",
        "email": "mail"
    }
# LOG配置
LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'verbose': {
            'format': '[%(asctime)s][%(threadName)s:%(thread)d][task_id:%(name)s][%(filename)s:%(lineno)d][%(levelname)s]- %(message)s'
        },
    },
    'handlers': {
        'default': {
            'level': 'DEBUG',
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': 'downloads/log/archery.log',
            'maxBytes': 1024 * 1024 * 100,  # 5 MB
            'backupCount': 5,
            'formatter': 'verbose',
        },
        'console': {
            'level': 'DEBUG',
            'class': 'logging.StreamHandler',
            'formatter': 'verbose'
        }
    },
    'loggers': {
        'default': {  # default日志，存放於log中
            'handlers': ['default'],
            'level': 'DEBUG',
        },
        'django_auth_ldap': {  # django_auth_ldap模塊相關日志
            'handlers': ['default'],
            'level': 'DEBUG',
        },
        'django_apscheduler': {  # django_apscheduler模塊相關日志
            'handlers': ['default'],
            'level': 'DEBUG',
        },
        # 'django.db': {  # 打印SQL語句到console，方便開發
        #     'handlers': ['console'],
        #     'level': 'DEBUG',
        #     'propagate': True,
        # },
        'django.request': {  # 打印請求錯誤堆棧信息到console，方便開發
            'handlers': ['console'],
            'level': 'DEBUG',
            'propagate': True,
        },
    }
}

mongodb

mkdir -p  /opt/mongo/datadir

新建 docker-compose.yml 引導文件

version: '3'

services:
  mongo:
    image: mongo:3.6
    container_name: mongo
    restart: always
    volumes:
      - "/opt/mongo/datadir:/data/db"
    ports:
      - 27017:27017
    environment:
      MONGO_INITDB_ROOT_USERNAME: root
      MONGO_INITDB_ROOT_PASSWORD: jason_zhang

  inception:
    image: registry.cn-hangzhou.aliyuncs.com/lihuanhuan/inception
    container_name: inception
    restart: always
    ports:
      - "6669:6669"
    volumes:
      - "/opt/inception/inc.cnf:/etc/inc.cnf"

  archery:
    image: registry.cn-hangzhou.aliyuncs.com/lihuanhuan/archery:1.3.7
    container_name: archery
    restart: always
    ports:
      - "9123:9123"
    volumes:
      - "/opt/archery/settings.py:/opt/archery/archery/settings.py"
      - "/opt/archery/downloads:/opt/archery/downloads"
    command: ["bash","/opt/archery/src/docker/startup.sh"]
    environment:
      NGINX_PORT: 9123

拉取鏡像　　

(venv4archer) [root@archery opt]# docker-compose -f docker-compose.yml up -d

失敗的話可以刪除后再次操作

docker-compose -f docker-compose.yml kill
docker-compose -f docker-compose.yml rm
docker-compose -f docker-compose.yml up -d

表結構初始化 docker exec -ti archery /bin/bash ##進入archery容器 cd /opt/archery source /opt/venv4archery/bin/activate python3 manage.py makemigrations sql python3 manage.py migrate #創建管理用戶 python3 manage.py createsuperuser #日志查看和問題排查 docker logs archery

登錄： http://192.168.199.177:9123

初次登錄提示密碼為空，重啟archery容器

　　

Docker

archery鏡像：https://dev.aliyun.com/detail.html?spm=5176.1972343.2.2.58c75aaa3iK1Sb&repoId=244140
inception鏡像: https://dev.aliyun.com/detail.html?spm=5176.1972343.2.12.7b475aaaLiCfMf&repoId=142093

docker ps -a
docker rm
docker images
docker rmi
要先查看該image有無使用才能刪除
進入一個容器
docker exec -ti archery /bin/bash

docker inspect   查看容器IP

 測試數據庫能否正常連接，在另外一台服務器上連接宿主機的數據庫

mysql -h 192.168.199.177 -P 3306 -u root -p

測試inception：1、連接mysql  ：  mysql -uroot -h192.168.199.224 -P6669  2、運行inception get variables;  3、能查看到信息就成功了

常見問題：     參考：https://github.com/hhyo/archery/wiki/%E4%BD%BF%E7%94%A8%E8%AF%B4%E6%98%8E#%E9%98%BF%E9%87%8C%E4%BA%91rds%E7%AE%A1%E7%90%86

https://gitee.com/ilanni/archer?tdsourcetag=s_pcqq_aiomsg

1、“SQL上線” sql檢測報錯

被檢測的數據庫密碼帶  *  號

2、回滾失敗

首先檢查

　　1、檢查inception配置文件相關用戶的權限；
　　2、檢查blog_bin是否開啟；log_bin格式，要為ROW ;
　　　　連接mysql
　　　　show global variables like 'log_bin';
　　　　show global variables like '%binlog_format%';
　　　　https://blog.csdn.net/king_kgh/article/details/74800513  （mysql5.7開啟log_bin）
　　3、檢查要執行DML的表是否存在主鍵
　　4、檢查語句執行后有無影響數據庫的數據

3、慢日志查詢

采用percona-toolkit的pt_query_digest收集慢日志，在系統中進行展示，並且支持一鍵獲取優化建議

https://www.cnblogs.com/zishengY/p/6852280.html

安裝（archery 容器）

安裝percona-toolkit，以centos為例

yum -y install http://www.percona.com/downloads/percona-release/redhat/0.1-3/percona-release-0.1-3.noarch.rpm
yum -y install percona-toolkit.x86_64
使用/opt/archery/src/init_sql/mysql_slow_query_review.sql創建慢archery數據庫的日志收集表（下載下來使用第三方客戶端導入archery數據庫）

   將/opt/archery/src/script/analysis_slow_query.sh部署到各個mysql實例，注意修改腳本里面的hostname="${mysql_host}:${mysql_port}"與archery實例信息一致，腳本可以放在任意位置

#!/bin/bash
DIR="$( cd "$( dirname "$0"  )" && pwd  )"
cd $DIR

#配置archery數據庫的連接地址
monitor_db_host="192.168.199.177"
monitor_db_port=3306
monitor_db_user="root"
monitor_db_password="jason_zhang"
monitor_db_database="archery"

#被監控機慢日志位置
slowquery_file="/home/mysql/log_slow.log"          （目錄位置存在）
pt_query_digest="/usr/bin/pt-query-digest"

#被監控機連接信息
hostname="192.168.199.178:3306" # 被監控機連接信息，和archery主庫配置內容保持一致，用於archery做篩選 （被收集的mysql服務器IP）

#獲取上次分析時間，初始化時請刪除last_analysis_time_$hostname文件，可分析全部日志數據
if [ -s last_analysis_time_$hostname ]; then
    last_analysis_time=`cat last_analysis_time_$hostname`
else
    last_analysis_time='1000-01-01 00:00:00'
fi

#收集日志
#RDS需要增加--no-version-check選項
$pt_query_digest \
--user=$monitor_db_user --password=$monitor_db_password --port=$monitor_db_port \
--review h=$monitor_db_host,D=$monitor_db_database,t=mysql_slow_query_review  \
--history h=$monitor_db_host,D=$monitor_db_database,t=mysql_slow_query_review_history  \
--no-report --limit=100% --charset=utf8 \
--since "$last_analysis_time" \
--filter="\$event->{Bytes} = length(\$event->{arg}) and \$event->{hostname}=\"$hostname\"  and \$event->{client}=\$event->{ip} " \
$slowquery_file > /tmp/analysis_slow_query.log

echo `date +"%Y-%m-%d %H:%M:%S"`>last_analysis_time_$hostname

阿里雲RDS

1、確保“系統設置”的阿里雲模塊開啟並正確配置了“系統管理”--“其他項管理”--“全部后台數據”-- “阿里雲認證信息” 配置正確

2、阿里雲配置

重啟一下服務器

數據庫審核 （themis）

https://github.com/CreditEaseDBA/Themis

https://tuteng.gitbooks.io/themis/content/

1、上傳rule.json文件到mongo容器的data文件夾

將archery-1.3.7.zip上傳到宿主機並解壓

docker cp /opt/archery-1.3.7  mongo:/data

2、在mongo容器上執行（賬戶密碼查看archery容器的archery配置文件）

mongoimport -h 127.0.0.1 --port 27017 -d themis -c rule -u root -p jason_zhang --upsert /data/archery-1.3.7/src/script/rule.json --authenticationDatabase admin

 郵件發送

1、系統設置里面配置一個發送郵件的賬戶

2、進入archery容器,修改 /opt/archery/sql/notify.py 的異步調用模塊

# 異步調用
def send_msg(audit_id, msg_type, **kwargs):
    logger.debug('異步發送消息通知，消息audit_id={}，msg_type={}'.format(audit_id, msg_type))
    #p = Thread(target=_send, args=(audit_id, msg_type), kwargs=kwargs)
    #p.start()
    _send(audit_id, msg_type, **kwargs)

 刪除表，直接刪除無法恢復表數據

刪除 test表

delete from test where 1=1;

 drop table test;

 系統升級：https://github.com/jly8866/archer/tree/archer-2.0

　　　　　https://github.com/hhyo/archery/releases/

系統功能說明：  https://github.com/hhyo/archery/wiki/%E4%BD%BF%E7%94%A8%E8%AF%B4%E6%98%8E#%E9%98%BF%E9%87%8C%E4%BA%91rds%E7%AE%A1%E7%90%86

參考：　　

https://www.cnblogs.com/chenjiaxin--007/p/8432795.html

https://blog.csdn.net/xujiamin0022016/article/details/81980393

https://gitee.com/ilanni/archer/tree/archer-2.0/
https://github.com/hhyo/archery
https://github.com/hhyo/archery/wiki/%E9%83%A8%E7%BD%B2
https://gitee.com/ilanni/archer#%E9%98%BF%E9%87%8C%E4%BA%91rds%E7%AE%A1%E7%90%86
https://github.com/hhyo/archery

Archery審核平台集成LDAP認證以及慢日志展示等常見問題解決

https://blog.csdn.net/qq_35209838/article/details/84998460?tdsourcetag=s_pcqq_aiomsg