k8s部署高可用Ingress

部署高可用Ingress

官網地址https://kubernetes.github.io/ingress-nginx/deploy/

獲取ingress的編排文件

wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml

增加節點標簽

kubectl label node k8snode1 ingresscontroller=true
kubectl get nodes --show-labels

附：

• 刪除標簽
  kubectl label node k8snode1 ingresscontroller-
• 更新標簽
  kubectl label node k8snode1 ingresscontroller=false --overwrite

1、修改Deployment為DaemonSet，並注釋掉副本數

kind: DaemonSet
  #replicas: 1

2、啟用hostNetwork網絡，並指定運行節點

hostNetwork暴露ingress-nginx controller的相關業務端口到主機，這樣node節點主機所在網絡的其他主機，都可以通過該端口訪問到此應用程序。

nodeSelector指定之前添加ingresscontroller=true標簽的node

      hostNetwork: true
      nodeSelector:
        ingresscontroller: 'true'

3、修改鏡像地址
registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:0.24.1

4、修改容器端口

          args:
            - /nginx-ingress-controller
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io
            - --http-port=88 #默認80
            - --https-port=4433 #默認443
            
            
          ports:
            - name: http
              containerPort: 88 #更改為88
            - name: https
              containerPort: 4433 #更改為4433

5、增加master節點容忍

      tolerations: #增加容忍，可分配到master節點
      - key: "node-role.kubernetes.io/master"
        operator: "Exists"
        effect: "NoSchedule"

6、mandatory.yaml如下
vim mandatory.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses/status
    verbs:
      - update

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get
      
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

---

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nginx-ingress-controller
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
    spec:
      # wait up to five minutes for the drain of connections
      terminationGracePeriodSeconds: 300
      serviceAccountName: nginx-ingress-serviceaccount
      hostNetwork: true
      nodeSelector:
        ingresscontroller: 'true'
      tolerations: #增加容忍，可分配到master節點
      - key: "node-role.kubernetes.io/master"
        operator: "Exists"
        effect: "NoSchedule"
      containers:
        - name: nginx-ingress-controller
          image: registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:0.24.1
          args:
            - /nginx-ingress-controller
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io
            - --http-port=88 #默認80
            - --https-port=4433 #默認443
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            # www-data -> 33
            runAsUser: 33
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - name: http
              containerPort: 88 #更改為88
            - name: https
              containerPort: 4433 #更改為4433
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          lifecycle:
            preStop:
              exec:
                command:
                  - /wait-shutdown

測試

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-static
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-static
  labels:
    name: nginx-static
spec:
  replicas: 1
  template:
    metadata:
      labels:
       name: nginx-static
    spec:
      containers:
       - name: nginx-static
         image: nginx:latest
         volumeMounts:
          - mountPath: /etc/localtime
            name: vol-localtime
            readOnly: true

         ports:
          - containerPort: 80
      volumes:
         - name: vol-localtime
           hostPath:
            path: /etc/localtime
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-static
  labels:
   name: nginx-static
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
    name: http
  selector:
    name: nginx-static

--- 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: submodule-checker-ingress
spec:
  rules:
  - host: nginx.weave.pub
    http:
      paths:
      - backend:
          serviceName: nginx-static
          servicePort: 80

kubectl create -f nginx-static.yaml

訪問

vim /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

10.x.x.x master1 nginx.test.pub
10.x.x.x master2

[root@master1 ingress]# curl nginx.test.pub
<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx/1.16.1</center>
</body>
</html>

---

1.1 Ingress簡介

1.1.1需求

  coredns是實現pods之間通過域名訪問，如果外部需要訪問service服務，需訪問對應的NodeIP:Port。但是由於NodePort需要指定宿主機端口，一旦服務多起來，多個端口就難以管理。那么，這種情況下，使用Ingress暴露服務更加合適。

1.1.2 Ingress組件說明

  使用Ingress時一般會有三個組件：反向代理負載均衡器、Ingress Controller、Ingress

1、反向代理負載均衡器

  反向代理負載均衡器很簡單，說白了就是 nginx、apache 等中間件，新版k8s已經將Nginx與Ingress Controller合並為一個組件，所以Nginx無需單獨部署，只需要部署Ingress Controller即可。在集群中反向代理負載均衡器可以自由部署，可以使用 Replication Controller、Deployment、DaemonSet 等等方式

2、Ingress Controller

  Ingress Controller 實質上可以理解為是個監視器，Ingress Controller 通過不斷地跟 kubernetes API 打交道，實時的感知后端 service、pod 等變化，比如新增和減少 pod，service 增加與減少等；當得到這些變化信息后，Ingress Controller 再結合下文的 Ingress 生成配置，然后更新反向代理負載均衡器，並刷新其配置，達到服務發現的作用

3、Ingress

  Ingress 簡單理解就是個規則定義；比如說某個域名對應某個 service，即當某個域名的請求進來時轉發給某個 service;這個規則將與 Ingress Controller 結合，然后 Ingress Controller 將其動態寫入到負載均衡器配置中，從而實現整體的服務發現和負載均衡

整體關系如下圖所示：

  從上圖中可以很清晰的看到，實際上請求進來還是被負載均衡器攔截，比如 nginx，然后 Ingress Controller 通過跟 Ingress 交互得知某個域名對應哪個 service，再通過跟 kubernetes API 交互得知 service 地址等信息；綜合以后生成配置文件實時寫入負載均衡器，然后負載均衡器 reload 該規則便可實現服務發現，即動態映射。

3.1.3 Nginx-Ingress工作原理

  ingress controller通過和kubernetes api交互，動態的去感知集群中ingress規則變化；然后讀取它，按照自定義的規則，規則就是寫明了哪個域名對應哪個service，生成一段nginx配置；再寫到nginx-ingress-control的pod里，這個Ingress controller的pod里運行着一個Nginx服務，控制器會把生成的nginx配置寫入/etc/nginx.conf文件中；然后reload一下使配置生效。以此達到域名分配置和動態更新的問題。

說明：基於nginx服務的ingress controller根據不同的開發公司，又分為：

• k8s社區的ingres-nginx（https://github.com/kubernetes/ingress-nginx）
• nginx公司的nginx-ingress（https://github.com/nginxinc/kubernetes-ingress）

3.2.4 Ingress Controller高可用架構

  作為集群流量接入層，Ingress Controller的高可用性顯得尤為重要，高可用性首先要解決的就是單點故障問題，一般常用的是采用多副本部署的方式，我們在Kubernetes集群中部署高可用Ingress Controller接入層同樣采用多節點部署架構，同時由於Ingress作為集群流量接入口，建議采用獨占Ingress節點的方式，以避免業務應用與Ingress服務發生資源爭搶。

  如上述部署架構圖，由多個獨占Ingress實例組成統一接入層承載集群入口流量，同時可依據后端業務流量水平擴縮容Ingress節點。當然如果您前期的集群規模並不大，也可以采用將Ingress服務與業務應用混部的方式，但建議進行資源限制和隔離。