k8s 部署 (二) EMQ X 集群

本文部署的 EMQ X Broker 版本為 4.3.1

一、RBAC 鑒權

集群需要使用到 Kubernetes 的 API Server，但是普通 Pod 是沒有權限訪問的，需要授權：

---
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: default
  name: emqx
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: emqx
  namespace: default

rules:
  - apiGroups:
      - ''
    resources:
      - endpoints 
    verbs: 
      - get
      - watch
      - list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: emqx
  namespace: default
roleRef:
  kind: Role
  name: emqx
  apiGroup: rbac.authorization.k8s.io
subjects:
  - kind: ServiceAccount
    name: emqx
    namespace: default

如果沒有授權，會有如下報錯：

Ekka(AutoCluster): Discovery error: {403,"{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"endpoints "emqx-headless" is forbidden: User "system:serviceaccount:default:default" cannot get resource "endpoints" in API group "" in the namespace "default"","reason":"Forbidden","details":{"name":"emqx-headless","kind":"endpoints"},"code":403}"}

二、新增 ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: emqx-cm
data:
  NAME: "emqx"
  CLUSTER__DISCOVERY: "k8s"
  CLUSTER__K8S__ADDRESS_TYPE: "ip"
  CLUSTER__K8S__APISERVER: "https://IP:PORT"
  CLUSTER__K8S__NAMESPACE: "default"
  CLUSTER__K8S__SERVICE_NAME: "emqx-headless"
  CLUSTER__K8S__APP_NAME: "emqx"

默認情況下 EMQ X 使用帶有 EMQX 的前綴的環境變量來覆蓋配置文件中的配置項環境變量名稱到配置文件鍵值名稱映射規則如下:將 EMQX 前綴移除；大寫字符替換成小寫；雙下划線 __ 替換成點 . 詳見：使用環境變量修改配置

• cluster.kubernetes.apiserver 為 kubernetes apiserver 的地址，可以通過 kubectl cluster-info 命令獲取
• cluster.kubernetes.service_name 為 Service 的 name
• cluster.kubernetes.app_name 為 EMQ X Broker 的 node.name 中 @ 符號之前的部分，需要同時將集群中 EMQ X Broker 設置為統一的 node.name 的前綴

三、新增 Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: emqx
  name: emqx
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: emqx
  template:
    metadata:
      labels:
        app: emqx
    spec:
      serviceAccountName: emqx
      containers:
        - envFrom:
          - prefix: EMQX_
            configMapRef: 
              name: emqx-cm            
          image: emqx/emqx:4.3.1
          imagePullPolicy: IfNotPresent  
          livenessProbe:
            exec:
              command:
                - emqx_ctl
                - status
            failureThreshold: 3
            initialDelaySeconds: 60
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1          
          name: emqx
          ports:
            - name: mqtt
              protocol: TCP
              containerPort: 1883
            - name: mqttssl
              protocol: TCP
              containerPort: 8883
            - name: mgmt
              protocol: TCP
              containerPort: 8081
            - name: websocket
              protocol: TCP
              containerPort: 8083
            - name: wss
              protocol: TCP
              containerPort: 8084
            - name: dashboard
              protocol: TCP
              containerPort: 18083  
          resources:
            limits:
              cpu: '2'
              memory: 2Gi
            requests:
              cpu: '1'
              memory: 2Gi
          volumeMounts:
            - mountPath: /etc/localtime
              name: volume-localtime
      volumes:
        - hostPath:
            path: /etc/localtime
            type: ''
          name: volume-localtime

• 1883 MQTT 協議端口
• 8883 MQTT/SSL 端口
• 8083 MQTT/WebSocket 端口
• 8084 MQTT/WebSocket/SSL 端口
• 8081 管理 API 端口
• 18083 Dashboard 端口

四、新增 Service

apiVersion: v1
kind: Service
metadata:
  name: emqx-headless
  labels:
    app: emqx-headless
spec:
  type: ClusterIP
  clusterIP: None
  ports:
    - name: mqtt
      port: 1883
      protocol: TCP
      targetPort: 1883
    - name: mqttssl
      port: 8883
      protocol: TCP
      targetPort: 8883
    - name: mgmt
      port: 8081
      protocol: TCP
      targetPort: 8081
    - name: websocket
      port: 8083
      protocol: TCP
      targetPort: 8083
    - name: wss
      port: 8084
      protocol: TCP
      targetPort: 8084
    - name: dashboard
      port: 18083
      protocol: TCP
      targetPort: 18083      
  selector:
    app: emqx

五、放行 TCP 端口

見：阿里雲 k8s 部署 Spring Cloud Alibaba 微服務實踐 (五) Kubernetes TCP Ingress

參考：

1.從零開始建立 EMQ X MQTT 服務器 的 K8S 集群

2. EMQ X Broker 文檔

​