簡介：
Velero 是一個雲原生的災難恢復和遷移工具，采用 Go 語言編寫，可以安全的備份、恢復和遷移Kubernetes集群資源和持久卷。velero是備份整個 k8s 集群所有的數據，勝過 etcd 備份。
Velero目前包含以下特性：
支持Kubernetes集群數據備份和恢復
支持復制當前Kubernetes集群的資源到其它Kubernetes集群
支持復制生產環境到開發以及測試環境
Velero組件一共分兩部分，分別是服務端和客戶端。
服務端：運行在Kubernetes集群中
客戶端：運行在本地的velero命令行工具，需要在機器上已配置好kubectl及集群kubeconfig
velero使用場景
災備場景：提供備份恢復k8s集群的能力
遷移場景：提供拷貝集群資源到其他集群的能力（復制同步開發，測試，生產環境的集群配置，簡化環境配置）
velero與etcd備份區別
直接備份 Etcd 是將集群的全部資源備份起來，而 Velero 可以對 Kubernetes 集群內對象級別進行備份。
除了對 Kubernetes 集群進行整體備份外，Velero 還可以通過對 Type、Namespace、Label
等對象進行分類備份或者恢復。
github地址：https://github.com/vmware-tanzu/velero
Velero 架構圖
Velero 工作原理
1、本地 Velero 客戶端發送備份指令。
2、Kubernetes 集群內就會創建一個 Backup 對象。
3、BackupController 監測 Backup 對象並開始備份過程。
4、BackupController 會向 API Server 查詢相關數據。
5、BackupController 將查詢到的數據備份到遠端的對象存儲。
支持備份存儲
1、AWS S3 以及兼容 S3 的存儲，例如：Minio
2、Azure BloB 存儲
3、Google Cloud 存儲
4、Aliyun OSS 存儲( https://github.com/AliyunContainerService/velero-plugin)
保障數據一致性
對象存儲的數據是唯一的數據源，也就是說 Kubernetes 集群內的控制器會檢查遠程的 OSS 存儲，發現有備份就會在集群內創建相關 CRD 。如果發現遠端存儲沒有當前集群內的 CRD 所關聯的存儲數據，那么就會刪除當前集群內的 CRD。
部署 Velero 客戶端
1、下載tar包並解壓
https://github.com/vmware-tanzu/velero
2、加入環境變量並賦予權限
[root@master2 ~]# cp velero-v1.4.2-linux-amd64/velero /usr/local/bin/
[root@master2 ~]# chmod +x /usr/local/bin/velero
3、查看 velero velero CLI 版本
[root@master2 ~]# velero
Velero is a tool for managing disaster recovery, specifically for Kubernetes
cluster resources. It provides a simple, configurable, and operationally robust
way to back up your application state and associated data.
If you're familiar with kubectl, Velero supports a similar model, allowing you to
execute commands such as 'velero get backup' and 'velero create schedule'. The same
operations can also be performed as 'velero backup get' and 'velero schedule create'.
Usage:
velero [command]
Available Commands:
backup Work with backups
backup-location Work with backup storage locations
bug Report a Velero bug
client Velero client related commands
completion Output shell completion code for the specified shell (bash or zsh)
create Create velero resources
delete Delete velero resources
describe Describe velero resources
get Get velero resources
help Help about any command
install Install Velero
plugin Work with plugins
restic Work with restic
restore Work with restores
schedule Work with schedules
snapshot-location Work with snapshot locations
version Print the velero version and associated image
Flags:
--add_dir_header If true, adds the file directory to the header
--alsologtostderr log to standard error as well as files
--features stringArray Comma-separated list of features to enable for this Velero process. Combines with values from $HOME/.config/velero/config.json if present
-h, --help help for velero
--kubeconfig string Path to the kubeconfig file to use to talk to the Kubernetes apiserver. If unset, try the environment variable KUBECONFIG, as well as in-cluster configuration
--kubecontext string The context to use to talk to the Kubernetes apiserver. If unset defaults to whatever your current-context is (kubectl config current-context)
--log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0)
--log_dir string If non-empty, write log files in this directory
--log_file string If non-empty, use this log file
--log_file_max_size uint Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
--logtostderr log to standard error instead of files (default true)
-n, --namespace string The namespace in which Velero should operate (default "velero")
--skip_headers If true, avoid header prefixes in the log messages
--skip_log_headers If true, avoid headers when opening log files
--stderrthreshold severity logs at or above this threshold go to stderr (default 2)
-v, --v Level number for the log level verbosity
--vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging
Use "velero [command] --help" for more information about a command.
部署 Velero 服務端
1、查看 00-minio-deployment.yaml 文件中的 MINIO_ACCESS_KEY 和 MINIO_SECRET_KEY 值
cat velero-v1.4.2-linux-amd64/examples/minio/00-minio-deployment.yaml
# Copyright 2017 the Velero contributors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
---
apiVersion: v1
kind: Namespace
metadata:
name: velero
---
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: velero
name: minio
labels:
component: minio
spec:
strategy:
type: Recreate
selector:
matchLabels:
component: minio
template:
metadata:
labels:
component: minio
spec:
volumes:
- name: storage
emptyDir: {}
- name: config
emptyDir: {}
containers:
- name: minio
image: minio/minio:latest
imagePullPolicy: IfNotPresent
args:
- server
- /storage
- --config-dir=/config
env:
- name: MINIO_ACCESS_KEY # 要查找的值
value: "minio"
- name: MINIO_SECRET_KEY # 要查找的值
value: "minio123"
ports:
- containerPort: 9000
volumeMounts:
- name: storage
mountPath: "/storage"
- name: config
mountPath: "/config"
---
apiVersion: v1
kind: Service
metadata:
namespace: velero
name: minio
labels:
component: minio
spec:
# ClusterIP is recommended for production environments.
# Change to NodePort if needed per documentation,
# but only if you run Minio in a test/trial environment, for example with Minikube.
type: NodePort
ports:
- port: 9000
targetPort: 9000
protocol: TCP
selector:
component: minio
---
apiVersion: batch/v1
kind: Job
metadata:
namespace: velero
name: minio-setup
labels:
component: minio
spec:
template:
metadata:
name: minio-setup
spec:
restartPolicy: OnFailure
volumes:
- name: config
emptyDir: {}
containers:
- name: mc
image: minio/mc:latest
imagePullPolicy: IfNotPresent
command:
- /bin/sh
- -c
- "mc --config-dir=/config config host add velero http://minio:9000 minio minio123 && mc --config-dir=/config mb -p velero/velero"
volumeMounts:
- name: config
mountPath: "/config"
2、克隆 velero-plugin 插件項目
git clone https://github.com/AliyunContainerService/velero-plugin
3、進入克隆的插件項目把兩個值添加到 credentials-velero 文件中
[root@master1 ~]# cat /root/velero-plugin/install/credentials-velero
ALIBABA_CLOUD_ACCESS_KEY_ID= minio
ALIBABA_CLOUD_ACCESS_KEY_SECRET= minio123
4、更改 00-minio-deployment.yaml 文件中的 type 類型為 NodePort
[root@master1 ~]# cat /root/velero-v1.4.2-linux-amd64/examples/minio/00-minio-deployment.yaml
# Copyright 2017 the Velero contributors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
---
apiVersion: v1
kind: Namespace
metadata:
name: velero
---
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: velero
name: minio
labels:
component: minio
spec:
strategy:
type: Recreate
selector:
matchLabels:
component: minio
template:
metadata:
labels:
component: minio
spec:
volumes:
- name: storage
emptyDir: {}
- name: config
emptyDir: {}
containers:
- name: minio
image: minio/minio:latest
imagePullPolicy: IfNotPresent
args:
- server
- /storage
- --config-dir=/config
env:
- name: MINIO_ACCESS_KEY
value: "minio"
- name: MINIO_SECRET_KEY
value: "minio123"
ports:
- containerPort: 9000
volumeMounts:
- name: storage
mountPath: "/storage"
- name: config
mountPath: "/config"
---
apiVersion: v1
kind: Service
metadata:
namespace: velero
name: minio
labels:
component: minio
spec:
# ClusterIP is recommended for production environments.
# Change to NodePort if needed per documentation,
# but only if you run Minio in a test/trial environment, for example with Minikube.
type: NodePort # 已經改過了
ports:
- port: 9000
targetPort: 9000
protocol: TCP
selector:
component: minio
---
apiVersion: batch/v1
kind: Job
metadata:
namespace: velero
name: minio-setup
labels:
component: minio
spec:
template:
metadata:
name: minio-setup
spec:
restartPolicy: OnFailure
volumes:
- name: config
emptyDir: {}
containers:
- name: mc
image: minio/mc:latest
imagePullPolicy: IfNotPresent
command:
- /bin/sh
- -c
- "mc --config-dir=/config config host add velero http://minio:9000 minio minio123 && mc --config-dir=/config mb -p velero/velero"
volumeMounts:
- name: config
mountPath: "/config"
5、部署文件
kubectl apply -f /root/velero-v1.4.2-linux-amd64/examples/minio/00-minio-deployment.yaml
velero install \
--provider aws \
--plugins velero/velero-plugin-for-aws:v1.2.0 \
--namespace velero \
--bucket velerobak \
--default-volumes-to-restic \
--use-restic \
--secret-file ./credentials-velero \
--use-volume-snapshots=false \
--backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://{minio_service_ip}:暴露端口號
6、驗證結果
提示這個代表部署成功
[root@master1 ~]# kubectl get pods -n velero
NAME READY STATUS RESTARTS AGE
minio-7b4ff54f67-k77kx 1/1 Running 0 3h58m
minio-setup-nwwr7 0/1 Completed 2 3h58m
restic-m6g9s 1/1 Running 0 3h46m
velero-8dc7498d9-kgrgx 1/1 Running 0 3h46m
7、Velero 在 Kubernetes 集群中創建的 CRD
[root@master1 ~]# kubectl -n velero get crds -l component=velero
NAME CREATED AT
backups.velero.io 2021-10-04T11:44:26Z
backupstoragelocations.velero.io 2021-10-04T11:44:26Z
deletebackuprequests.velero.io 2021-10-04T11:44:26Z
downloadrequests.velero.io 2021-10-04T11:44:26Z
podvolumebackups.velero.io 2021-10-04T11:44:26Z
podvolumerestores.velero.io 2021-10-04T11:44:26Z
resticrepositories.velero.io 2021-10-04T11:44:26Z
restores.velero.io 2021-10-04T11:44:26Z
schedules.velero.io 2021-10-04T11:44:26Z
serverstatusrequests.velero.io 2021-10-04T11:44:26Z
volumesnapshotlocations.velero.io 2021-10-04T11:44:26Z
8、查看 velero 命令及版本
[root@master1 ~]# velero create backup NAME [flags]
Error: accepts 1 arg(s), received 2
Usage:
velero create backup NAME [flags]
Examples:
# create a backup containing all resources
velero backup create backup1
# create a backup including only the nginx namespace
velero backup create nginx-backup --include-namespaces nginx
# create a backup excluding the velero and default namespaces
velero backup create backup2 --exclude-namespaces velero,default
# view the YAML for a backup that doesn't snapshot volumes, without sending it to the server
velero backup create backup3 --snapshot-volumes=false -o yaml
# wait for a backup to complete before returning from the command
velero backup create backup4 --wait
Flags:
--exclude-namespaces stringArray namespaces to exclude from the backup
--exclude-resources stringArray resources to exclude from the backup, formatted as resource.group, such as storageclasses.storage.k8s.io
--from-schedule string create a backup from the template of an existing schedule. Cannot be used with any other filters.
-h, --help help for backup
--include-cluster-resources optionalBool[=true] include cluster-scoped resources in the backup
--include-namespaces stringArray namespaces to include in the backup (use '*' for all namespaces) (default *)
--include-resources stringArray resources to include in the backup, formatted as resource.group, such as storageclasses.storage.k8s.io (use '*' for all resources)
--label-columns stringArray a comma-separated list of labels to be displayed as columns
--labels mapStringString labels to apply to the backup
-o, --output string Output display format. For create commands, display the object but do not send it to the server. Valid formats are 'table', 'json', and 'yaml'. 'table' is not valid for the install command.
-l, --selector labelSelector only back up resources matching this label selector (default <none>)
--show-labels show labels in the last column
--snapshot-volumes optionalBool[=true] take snapshots of PersistentVolumes as part of the backup
--storage-location string location in which to store the backup
--ttl duration how long before the backup can be garbage collected (default 720h0m0s)
--volume-snapshot-locations strings list of locations (at most one per provider) where volume snapshots should be stored
-w, --wait wait for the operation to complete
Global Flags:
--add_dir_header If true, adds the file directory to the header
--alsologtostderr log to standard error as well as files
--features stringArray Comma-separated list of features to enable for this Velero process. Combines with values from $HOME/.config/velero/config.json if present
--kubeconfig string Path to the kubeconfig file to use to talk to the Kubernetes apiserver. If unset, try the environment variable KUBECONFIG, as well as in-cluster configuration
--kubecontext string The context to use to talk to the Kubernetes apiserver. If unset defaults to whatever your current-context is (kubectl config current-context)
--log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0)
--log_dir string If non-empty, write log files in this directory
--log_file string If non-empty, use this log file
--log_file_max_size uint Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
--logtostderr log to standard error instead of files (default true)
-n, --namespace string The namespace in which Velero should operate (default "velero")
--skip_headers If true, avoid header prefixes in the log messages
--skip_log_headers If true, avoid headers when opening log files
--stderrthreshold severity logs at or above this threshold go to stderr (default 2)
-v, --v Level number for the log level verbosity
--vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging
An error occurred: accepts 1 arg(s), received 2
[root@master1 ~]# velero version
Client:
Version: v1.4.2
Git commit: 56a08a4d695d893f0863f697c2f926e27d70c0c5
Server:
Version: v1.4.2