通過k8s 的定時任務job，我們可以方便的進行定時任務應用的開發，通過minio s3 兼容的cloud native 存儲
我們可以方便的通過http 請求進行數據文件的備份，以下簡單演示下如何進行集成
環境准備
docker 鏡像修改自
https://github.com/Remigius2011/pg-dump,主要是去除了schema 同時添加了gzip 壓縮處理
dockerfile
FROM remigius65/pg-dump
COPY backup.sh /usr/bin/backup.sh
RUN chmod +x /usr/bin/backup.shbackup.sh
主要修改了dump 邏輯，說明備份使用了.pgpass 進行賬戶的處理，省去輸入密碼的問題，mc 客戶端賬戶信息通過
環境變量處理
#!/bin/sh
. /usr/bin/setpwd.sh
export DUMP_FILE="$BACKUP_DIR/$DB_ENV-$(date +"%F-%H%M%S").dump"
if [ ! -d "$BACKUP_DIR" ]; then
echo mkdir -p "$BACKUP_DIR"
mkdir -p "$BACKUP_DIR"
fi
echo "pg_dump -h $PG_HOST -p $PG_PORT -U $PG_USER $PG_DB -f $DUMP_FILE"
pg_dump -h $PG_HOST -p $PG_PORT -U $PG_USER $PG_DB | gzip > $DUMP_FILE
if [ -n "S3_HOST" ]; then
export MC_HOSTS_store="$S3_PROTOCOL://$S3_ACCESS_KEY:$S3_SECRET_KEY@$S3_HOST"
echo "mc cp $DUMP_FILE store/$S3_BUCKET"
mc cp $DUMP_FILE store/$S3_BUCKET
fi
docker-compose 文件
通過docker-compose 運行依賴的服務
version: "3"
services:
postgres:
image: postgres:10.7
ports:
- "5432:5432"
environment:
- "POSTGRES_PASSWORD:dalong"
volumes:
- ./db_data:/var/lib/postgresql/data
backup:
image: dalongrong/pg-dump
environment:
- "PG_HOST=postgres"
- "PG_DB=postgres"
- "PG_PASSWORD=dalong"
- "S3_HOST=s3:9000"
- "S3_ACCESS_KEY=dalongdemo"
- "S3_SECRET_KEY=dalongdemo"
- "S3_PROTOCOL=http"
s3:
image: minio/minio
command: server /export
ports:
- "9000:9000"
volumes:
- ./data:/export
- ./config:/root/.minio
environment:
- "MINIO_ACCESS_KEY=dalongdemo"
- "MINIO_SECRET_KEY=dalongdemo"測試
- 啟動pg && minio
docker-compose up -d s3 postgres- 添加以下測試數據
這個可以根據自己的喜好，自由添加 - 測試數據備份
docker-compose up backup- 測試效果
docker-compose up backup
Starting pg-s3-backup_backup_1 ... done
Attaching to pg-s3-backup_backup_1
backup_1 | pg_dump -h postgres -p 5432 -U postgres postgres -f /pgbackup/prod-2019-03-18-103712.dump
backup_1 | mc cp /pgbackup/prod-2019-03-18-103712.dump store/pgbackup
backup_1 | `/pgbackup/prod-2019-03-18-103712.dump` -> `store/pgbackup/prod-2019-03-18-103712.dump`
backup_1 | Total: 1.03 KB, Transferred: 1.03 KB, Speed: 33.52 KB/s
pg-s3-backup_backup_1 exited with code 0- minio 界面
k8s cronjob
- json 格式文件
{
"kind": "CronJob",
"apiVersion": "batch/v1beta1",
"metadata": {
"name": "pg-backup-job"
},
"spec": {
"schedule": "0 0 1 * *",
"concurrencyPolicy": "Replace",
"suspend": false,
"jobTemplate": {
"metadata": {
"creationTimestamp": null
},
"spec": {
"template": {
"metadata": {
"creationTimestamp": null,
"labels": {
"apprepositories.kubeapps.com/repo-name": "pg-backup-job"
}
},
"spec": {
"containers": [
{
"name": "gitlab-pg-backup",
"image": "dalongrong/pg-dump-gzip",
"env": [
{
"name": "PG_DB",
"value": "postgres"
},
{
"name": "PG_HOST",
"value": "postgres"
},
{
"name": "PG_PASSWORD",
"value": "dalong"
},
{
"name": "PG_PORT",
"value": "5432"
},
{
"name": "S3_ACCESS_KEY",
"value": "dalongdemo"
},
{
"name": "S3_HOST",
"value": "s3:9000"
},
{
"name": "S3_PROTOCOL",
"value": "http"
},
{
"name": "S3_SECRET_KEY",
"value": "dalongdemo"
}
],
"imagePullPolicy": "IfNotPresent"
}
],
"restartPolicy": "OnFailure"
}
}
}
},
"successfulJobsHistoryLimit": 3,
"failedJobsHistoryLimit": 1
}
}- yaml 格式
yaml 格式因為是使用json-> yaml 的工具，格式很難看
kind: CronJob
apiVersion: batch/v1beta1
metadata:
name: gitlab-backup-job
spec:
schedule: "0 0 1 * *"
concurrencyPolicy: Replace
suspend: false
jobTemplate:
metadata:
creationTimestamp: null
spec:
template:
metadata:
creationTimestamp: null
labels:
apprepositories.kubeapps.com/repo-name: gitlab-backup-job
spec:
containers:
- name: gitlab-pg-backup
image: dalongrong/pg-dump-gzip
env:
- name: PG_DB
value: postgres
- name: PG_HOST
value: postgres
- name: PG_PASSWORD
value: dalong
- name: PG_PORT
value: "5432"
- name: S3_ACCESS_KEY
value: dalongdemo
- name: S3_HOST
value: "s3:9000"
- name: S3_PROTOCOL
value: http
- name: S3_SECRET_KEY
value: dalongdemo
imagePullPolicy: IfNotPresent
restartPolicy: OnFailure
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 1- 運行效果
因為k8s 定時備份的不是應用全部通過k8s 部署的，只有備份程序，pg 以及s3 都是外部的
容器日志
說明
結合k8s 的定時任務以及minio 的s3 能力，我們可以方便的制作一個備份以及恢復的方案，而且很簡單
參考資料
https://github.com/Remigius2011/pg-dump
https://github.com/rongfengliang/pg-s3-backup
https://cloud.docker.com/repository/docker/dalongrong/pg-dump-gzip