基於k8s手動部署rabbitmq集群

目錄

• 1、RabbitMQ介紹
• 2、RabbitMQ的持久化和鏡像隊列
• 3、RabbitMQ集群在k8s中的部署
  • 3.1 環境介紹
  • 3.2 創建configmap
  • 3.3 創建service
  • 3.4 創建rbac授權
  • 3.5 創建statefulset
  • 3.6 部署檢查

1、RabbitMQ介紹

RabbitMQ是實現了高級消息隊列協議AMQP的開源消息代理軟件（亦稱面向消息的中間件）。RabbitMQ服務器是用Erlang語言編寫的，而集群和故障轉移是構建在開放電信平台框架上的。AMQP：Advanced Message Queue，高級消息隊列協議。它是應用層協議的一個開放標准，為面向消息的中間件設計，基於此協議的客戶端與消息中間件可傳遞消息，並不受產品、開發語言燈條件的限制

AMQP具有如下的特性：

• 可靠性Reliablity：使用了一些機制來保證可靠性，比如持久化、傳輸確認、發布確認
• 靈活的路由Flexible Routing：在消息進入隊列之前，通過Exchange來路由消息。對於典型的路由功能，Rabbit已經提供了一些內置的Exchange來實現。針對更復雜的路由功能，可以將多個Exchange綁定在一起，也通過插件機制實現自己的Exchange
• 消息集群Clustering：多個RabbitMQ服務器可以組成一個集群，形成一個邏輯Broker
• 高可用Highly Avaliable Queues：隊列可以在集群中的機器上進行鏡像，使得在部分節點出問題的情況下隊列仍然可用
• 多種協議Multi-protocol：支持多種消息隊列協議，如STOMP、MQTT等
• 多種語言客戶端Many Clients：幾乎支持所有常用語言，比如Java、.NET、Ruby等
• 管理界面Management UI：提供了易用的用戶界面，使得用戶可以監控和管理消息Broker的許多方面
• 跟蹤機制Tracing：如果消息異常，RabbitMQ提供了消息的跟蹤機制，使用者可以找出發生了什么
• 插件機制Plugin System：提供了許多插件，來從多方面進行擴展，也可以編輯自己的插件

2、RabbitMQ的持久化和鏡像隊列

RabbitMQ持久化分為Exchange、Queue、Message

• Exchange和Queue持久化：指持久化Exchange、Queue元數據，持久化的是自身，服務宕機Exchange和Queue自身就沒有了
• Message持久化：顧名思義就是把每一條消息體持久化，服務宕機，消息不丟失

RabbitMQ的隊列Queue鏡像，指master node在接受到請求后，會同步到其他節點上，以此來保證高可用。在confirm模式下，具體過程如下

clientpublisher發送消息 –> master node接到消息 –> master node將消息持久化到磁盤 –> 將消息異步發送給其他節點 –> master將ack返回給client publisher

3、RabbitMQ集群在k8s中的部署

將RabbitMQ以集群的方式部署在k8s中，前提是RabbitMQ的每個節點都能像傳統方式一樣進行相互的服務發現。因此RabbitMQ在k8s集群中通過rabbitmq_peer_discovery_k8s plugin與k8s apiserver進行交互，獲取各個服務的URL，且RabbitMQ在k8s集群中必須用statefulset和headless service進行匹配

需要注意的是，rabbitmq_peer_discovery_k8s是RabbitMQ官方基於第三方開源項目rabbitmq-autocluster開發，對3.7.X及以上版本提供的Kubernetes下的對等發現插件，可實現rabbitmq集群在k8s中的自動化部署，因此低於3.7.X版本請使用rabbitmq-autocluster

3.1 環境介紹

本文部署的版本是3.8.3

默認部署在default命名空間下，

持久化存儲為storageclass動態存儲，底層為nfs提供

鏡像地址rabbitmq:3.8.3-management

以下yaml參考自官方示例https://github.com/rabbitmq/diy-kubernetes-examples

3.2 創建configmap

01-rabbitmq-configmap.yaml

kind: ConfigMap
apiVersion: v1
metadata:
  name: rabbitmq-cluster-config
  namespace: default
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
data:
    enabled_plugins: |
      [rabbitmq_management,rabbitmq_peer_discovery_k8s].
    rabbitmq.conf: |
      default_user = admin
      default_pass = 123!@#
      ## Cluster formation. See https://www.rabbitmq.com/cluster-formation.html to learn more.
      cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8s
      cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
      ## Should RabbitMQ node name be computed from the pod's hostname or IP address?
      ## IP addresses are not stable, so using [stable] hostnames is recommended when possible.
      ## Set to "hostname" to use pod hostnames.
      ## When this value is changed, so should the variable used to set the RABBITMQ_NODENAME
      ## environment variable.
      cluster_formation.k8s.address_type = hostname
      ## How often should node cleanup checks run?
      cluster_formation.node_cleanup.interval = 30
      ## Set to false if automatic removal of unknown/absent nodes
      ## is desired. This can be dangerous, see
      ##  * https://www.rabbitmq.com/cluster-formation.html#node-health-checks-and-cleanup
      ##  * https://groups.google.com/forum/#!msg/rabbitmq-users/wuOfzEywHXo/k8z_HWIkBgAJ
      cluster_formation.node_cleanup.only_log_warning = true
      cluster_partition_handling = autoheal
      ## See https://www.rabbitmq.com/ha.html#master-migration-data-locality
      queue_master_locator=min-masters
      ## See https://www.rabbitmq.com/access-control.html#loopback-users
      loopback_users.guest = false
      cluster_formation.randomized_startup_delay_range.min = 0
      cluster_formation.randomized_startup_delay_range.max = 2
      # default is rabbitmq-cluster's namespace
      # hostname_suffix
      cluster_formation.k8s.hostname_suffix = .rabbitmq-cluster.default.svc.cluster.local
      # memory
      vm_memory_high_watermark.absolute = 1GB
      # disk
      disk_free_limit.absolute = 2GB

部分參數說明：

• enabled_plugins：聲明開啟的插件名
• default_pass/default_pass：聲明用戶名和密碼（雖然有部分文章記錄可以通過環境變量的方式聲明，但是經測試，針對此版本如果指定了configmap即rabbitmq的配置文件，聲明的環境變量是沒有用的，都需要在配置文件中指定）
• cluster_formation.k8s.address_type：從k8s返回的Pod容器列表中計算對等節點列表，這里只能使用主機名，官方示例中是ip，但是默認情況下在k8s中pod的ip都是不固定的，因此可能導致節點的配置和數據丟失，后面的yaml中會通過引用元數據的方式固定pod的主機名。

更多參數請參考官方文檔，這里就不贅述了

3.3 創建service

02-rabbitmq-service.yaml

kind: Service
apiVersion: v1
metadata:
  labels:
    app: rabbitmq-cluster
  name: rabbitmq-cluster
  namespace: default
spec:
  clusterIP: None
  ports:
  - name: rmqport
    port: 5672
    targetPort: 5672
  selector:
    app: rabbitmq-cluster

---
kind: Service
apiVersion: v1
metadata:
  labels:
    app: rabbitmq-cluster
  name: rabbitmq-cluster-manage
  namespace: default
spec:
  ports:
  - name: http
    port: 15672
    protocol: TCP
    targetPort: 15672
  selector:
    app: rabbitmq-cluster
  type: NodePort

上面定義了兩個Service，一個是rabbitmq的服務端口，一個是管理界面的端口，用戶外部訪問，這里通過NodePort方式進行暴露

3.4 創建rbac授權

前面的介紹中提到了RabbitMQ通過插件與k8s apiserver交互獲得集群中節點相關信息，因此需要對其進行RBAC授權

03-rabbitmq-rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: rabbitmq-cluster
  namespace: default
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: rabbitmq-cluster
  namespace: default
rules:
- apiGroups: [""]
  resources: ["endpoints"]
  verbs: ["get"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: rabbitmq-cluster
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: rabbitmq-cluster
subjects:
- kind: ServiceAccount
  name: rabbitmq-cluster
  namespace: default

3.5 創建statefulset

RabbitMQ在k8s中作為一個有狀態應用進行部署，因此控制器類型為StatefulSet，yaml中還定義了pvc相關內容

04-rabbitmq-cluster-sts.yaml

kind: StatefulSet
apiVersion: apps/v1
metadata:
  labels:
    app: rabbitmq-cluster
  name: rabbitmq-cluster
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: rabbitmq-cluster
  serviceName: rabbitmq-cluster
  template:
    metadata:
      labels:
        app: rabbitmq-cluster
    spec:
      containers:
      - args:
        - -c
        - cp -v /etc/rabbitmq/rabbitmq.conf ${RABBITMQ_CONFIG_FILE}; exec docker-entrypoint.sh
          rabbitmq-server
        command:
        - sh
        env:
        - name: TZ
          value: 'Asia/Shanghai'
        - name: RABBITMQ_ERLANG_COOKIE
          value: 'SWvCP0Hrqv43NG7GybHC95ntCJKoW8UyNFWnBEWG8TY='
        - name: K8S_SERVICE_NAME
          value: rabbitmq-cluster
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: RABBITMQ_USE_LONGNAME
          value: "true"
        - name: RABBITMQ_NODENAME
          value: rabbit@$(POD_NAME).$(K8S_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.local
        - name: RABBITMQ_CONFIG_FILE
          value: /var/lib/rabbitmq/rabbitmq.conf
        image: rabbitmq:3.8.3-management
        imagePullPolicy: IfNotPresent
        livenessProbe:
          exec:
            command:
            - rabbitmq-diagnostics
            - status
          # See https://www.rabbitmq.com/monitoring.html for monitoring frequency recommendations.
          initialDelaySeconds: 60
          periodSeconds: 60
          timeoutSeconds: 15
        name: rabbitmq
        ports:
        - containerPort: 15672
          name: http
          protocol: TCP
        - containerPort: 5672
          name: amqp
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - rabbitmq-diagnostics
            - status
          initialDelaySeconds: 20
          periodSeconds: 60
          timeoutSeconds: 10
        volumeMounts:
        - mountPath: /etc/rabbitmq
          name: config-volume
          readOnly: false
        - mountPath: /var/lib/rabbitmq
          name: rabbitmq-storage
          readOnly: false
        - name: timezone
          mountPath: /etc/localtime
          readOnly: true
      serviceAccountName: rabbitmq-cluster
      terminationGracePeriodSeconds: 30
      volumes:
      - name: config-volume
        configMap:
          items:
          - key: rabbitmq.conf
            path: rabbitmq.conf
          - key: enabled_plugins
            path: enabled_plugins
          name: rabbitmq-cluster-config
      - name: timezone
        hostPath:
          path: /usr/share/zoneinfo/Asia/Shanghai
  volumeClaimTemplates:
  - metadata:
      name: rabbitmq-storage
    spec:
      accessModes:
      - ReadWriteMany
      storageClassName: "managed-nfs-storage"
      resources:
        requests:
          storage: 2Gi

3.6 部署檢查

➜  rabbitmq-cluster pwd
/Users/ssgeek/Documents/k8s-manifests/rabbitmq-cluster
➜  rabbitmq-cluster ls
01-rabbitmq-configmap.yaml   02-rabbitmq-service.yaml     03-rabbitmq-rbac.yaml        04-rabbitmq-cluster-sts.yaml
➜  rabbitmq-cluster kubectl apply -f .
configmap/rabbitmq-cluster-config created
service/rabbitmq-cluster created
service/rabbitmq-cluster-manage created
serviceaccount/rabbitmq-cluster created
role.rbac.authorization.k8s.io/rabbitmq-cluster created
rolebinding.rbac.authorization.k8s.io/rabbitmq-cluster created
statefulset.apps/rabbitmq-cluster created

等待一段時間，查看創建的相關資源

➜  rabbitmq-cluster kubectl get po,sts -l app=rabbitmq-cluster
NAME                     READY   STATUS    RESTARTS   AGE
pod/rabbitmq-cluster-0   1/1     Running   0          3m22s
pod/rabbitmq-cluster-1   1/1     Running   0          2m18s
pod/rabbitmq-cluster-2   1/1     Running   0          2m23s

NAME                                READY   AGE
statefulset.apps/rabbitmq-cluster   3/3     3m25s

查看日志，從日志的最后部分觀察集群建立的狀態

➜  rabbitmq-cluster kubectl logs -f rabbitmq-cluster-0
'/etc/rabbitmq/rabbitmq.conf' -> '/var/lib/rabbitmq/rabbitmq.conf'
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags: list of feature flags found:
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags:   [ ] drop_unroutable_metric
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags:   [ ] empty_basic_get_metric
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags:   [ ] implicit_default_bindings
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags:   [ ] quorum_queue
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags:   [ ] virtual_host_metadata
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags: feature flag states written to disk: yes
2021-02-17 03:30:39.544 [info] <0.269.0> ra: meta data store initialised. 0 record(s) recovered
2021-02-17 03:30:39.547 [info] <0.274.0> WAL: recovering []
2021-02-17 03:31:10.676 [info] <0.313.0> 
 Starting RabbitMQ 3.8.3 on Erlang 22.3.4.1
 Copyright (c) 2007-2020 Pivotal Software, Inc.
 Licensed under the MPL 1.1. Website: https://rabbitmq.com

  ##  ##      RabbitMQ 3.8.3
  ##  ##
  ##########  Copyright (c) 2007-2020 Pivotal Software, Inc.
  ######  ##
  ##########  Licensed under the MPL 1.1. Website: https://rabbitmq.com

  Doc guides: https://rabbitmq.com/documentation.html
  Support:    https://rabbitmq.com/contact.html
  Tutorials:  https://rabbitmq.com/getstarted.html
  Monitoring: https://rabbitmq.com/monitoring.html

  Logs: <stdout>

  Config file(s): /var/lib/rabbitmq/rabbitmq.conf

  Starting broker...2021-02-17 03:31:10.678 [info] <0.313.0> 
 node           : [e-mail]
 home dir       : /var/lib/rabbitmq
 config file(s) : /var/lib/rabbitmq/rabbitmq.conf
 cookie hash    : H+IQL2spD4MDV4jPi7mMAg==
 log(s)         : <stdout>
 database dir   : /var/lib/rabbitmq/mnesia/[e-mail]
2021-02-17 03:31:10.695 [info] <0.313.0> Running boot step pre_boot defined by app rabbit

...省略中間內容

2021-02-17 03:31:13.273 [info] <0.824.0> Statistics database started.
2021-02-17 03:31:13.273 [info] <0.823.0> Starting worker pool 'management_worker_pool' with 3 processes in it
2021-02-17 03:31:13.467 [info] <0.9.0> Server startup complete; 5 plugins started.
 * rabbitmq_peer_discovery_k8s
 * rabbitmq_management
 * rabbitmq_web_dispatch
 * rabbitmq_management_agent
 * rabbitmq_peer_discovery_common
 completed with 5 plugins.
2021-02-17 03:32:30.060 [info] <0.566.0> node '[e-mail]' up
2021-02-17 03:32:31.264 [info] <0.566.0> rabbit on node '[e-mail]' up
2021-02-17 03:33:31.280 [info] <0.566.0> node '[e-mail]' up
2021-02-17 03:33:32.627 [info] <0.566.0> rabbit on node '[e-mail]' up

進入到pod中通過客戶端查看集群狀態

➜  rabbitmq-cluster kubectl exec -it rabbitmq-cluster-0 bash
root@rabbitmq-cluster-0:/# rabbitmqctl cluster_status
Cluster status of node [e-mail] ...
Basics

Cluster name: [e-mail]

Disk Nodes

[e-mail]
[e-mail]
[e-mail]

Running Nodes

[e-mail]
[e-mail]
[e-mail]

Versions

[e-mail]: RabbitMQ 3.8.3 on Erlang 22.3.4.1
[e-mail]: RabbitMQ 3.8.3 on Erlang 22.3.4.1
[e-mail]: RabbitMQ 3.8.3 on Erlang 22.3.4.1

Alarms

(none)

Network Partitions

(none)

Listeners

Node: [e-mail], interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: [e-mail], interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: [e-mail], interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: [e-mail], interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: [e-mail], interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: [e-mail], interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: [e-mail], interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: [e-mail], interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: [e-mail], interface: [::], port: 15672, protocol: http, purpose: HTTP API

Feature flags

Flag: drop_unroutable_metric, state: enabled
Flag: empty_basic_get_metric, state: enabled
Flag: implicit_default_bindings, state: enabled
Flag: quorum_queue, state: enabled
Flag: virtual_host_metadata, state: enabled

通過NodePort訪問管理界面

kubectl get svc -l app=rabbitmq-cluster
NAME                      TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)           AGE
rabbitmq-cluster          ClusterIP   None           <none>        5672/TCP          8m47s
rabbitmq-cluster-manage   NodePort    10.1.239.191   <none>        15672:30888/TCP   8m47s

到這里，在k8s中手動部署一個RabbitMQ集群就完成啦~

參考：
https://www.rabbitmq.com/cluster-formation.html
https://github.com/rabbitmq/diy-kubernetes-examples