Kubernetes服務發布之Ingress

一、什么是Ingress

Ingress是通過service來關聯pod的，通過ingress controller實現pod的負載均衡，從而實現全局的負載均衡。

二、Ingress的安裝

首先安裝helm管理工具：https://helm.sh/docs/intro/install/

使用helm安裝ingress：https://kubernetes.github.io/ingress-nginx/deploy/#using-helm

2.1 添加helm倉庫

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

2.2 查看並下載helm軟件包 

# helm search repo ingress-nginx
NAME                           CHART VERSION    APP VERSION    DESCRIPTION                                       
ingress-nginx/ingress-nginx    4.0.8            1.0.5          Ingress controller for Kubernetes using NGINX a...

# helm pull ingress-nginx/ingress-nginx
# helm pull ingress-nginx/ingress-nginx --version 3.6.0     # 也可以指定版本下載 

2.3 修改對應的配置

# tar xf ingress-nginx-4.0.8.tgz 
# cd ingress-nginx/
# vim values.yaml

a、ingress-nginx/controller、ingress-nginx/kube-webhook-certgen和defaultbackend-amd64鏡像地址，需要將k8s.gcr.io倉庫中的鏡像同步至公司內網鏡像倉庫
b、將digest注釋
c、hostNetwork設置為 true
d、dnsPolicy設置為 ClusterFirstWithHostNet
Default：繼承Pod所在宿主機的DNS設置 ClusterFirst：優先使用Kubernetes環境的DNS服務（如CoreDNS提供的域名解析服務），將無法解析的域名轉發到從宿主機
ClusterFirstWithHostNet：與ClusterFirst相同，對於以hostNetwork模式運行的Pod，應明確指定使用該策略。
繼承的DNS服務器。
None：忽略Kubernetes環境的DNS配置，通過spec.dnsConfig自定義DNS配置。
e、NodeSelector添加ingress: "true"部署至指定節點
f、類型更改為kind: DaemonSet

# sed -n '186p;12p;13p;14p;18p;19p;598p;599p;600p;604p;605p;721p;722p;723p;727p;59p;83p;284p' values.yaml 
  image:
    registry: registry.cn-hangzhou.aliyuncs.com/lzlx
    image: ingress-nginx-controller
    tag: "v1.0.5"
   # digest: sha256:55a1fcda5b7657c372515fe402c3e39ad93aa59f6e4378e82acd99912fe6028d
  dnsPolicy: ClusterFirstWithHostNet
  hostNetwork: true  # 需要修改dnspolicy為
  kind: DaemonSet
  nodeSelector:
      image:
        registry: registry.cn-hangzhou.aliyuncs.com/lzlx
        image: ingress-nginx-kube-webhook-certgen
        tag: v1.1.1
       # digest: sha256:64d8c73dca984af206adf9d6d7e46aa550362b1d7a01f3a0a91b20cc67868660
  image:
    registry: estry.cn-hangzhou.aliyuncs.com/lzlx
    image: defaultbackend-amd64
    tag: "1.5"

2.4 部署ingress-nginx，將節點打上label ingress=true，然后將節點IP地址加入到clb負載均衡中

kubectl label node k8s-node3 ingress=true
kubectl create ns ingress-nginx
helm install ingress-nginx -n ingress-nginx .

三、創建一個ingress實例

3.1 創建 nginx web服務

# cat nginx.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: mynginx
        image: nginx:1.18.0
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 500m
            memory: 1024Mi
          limits:
            cpu: 1000m
            memory: 2048Mi
# kubectl create -f nginx.yaml

3.2 創建service

# cat nginx-service.yaml 
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  namespace: default
spec:
  type: ClusterIP
  selector:
    app: nginx
  ports:
  - name: http
    port: 80
    targetPort: 80
# kubectl create -f nginx-service.yaml

3.3 創建ingress

官方參考鏈接：https://kubernetes.io/zh/docs/concepts/services-networking/ingress/

# cat nginx-ingress.yaml 
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress.lzlx.com
spec:
  rules:
  - host: ingress.lzlx.com
    http:
      paths:
      - path: /
        pathType: Prefix         # Exact：精確匹配URL路徑，且區分大小寫。 Prefix：基於以/ 分隔的 URL 路徑前綴匹配。匹配區分大小寫，並且對路徑中的元素逐個完成。
        backend:
          service:
            name: nginx-service
            port:
              number: 80
  ingressClassName: nginx        # 如果集群版本 >= 1.19使用新ingressClassName: nginx屬性代替注釋。
# kubectl create -f nginx-ingress.yaml 

3.4 綁定hosts進行訪問，ip地址為k8s-node3部署ingress服務的那台機器。

四、ingress的配置

4.1 Redirect，訪問域名重定向到baidu.com

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress.lzlx.com
  annotations:
    nginx.ingress.kubernetes.io/permanent-redirect: https://www.baidu.com
spec:
  rules:
  - host: ingress.lzlx.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-service
            port:
              number: 80
  ingressClassName: nginx

它會自動在ingress-nginx-controller中增加相關的配置

# kubectl exec -it ingress-nginx-controller-5s4fw -n ingress-nginx -- bash
$ cat /etc/nginx/nginx.conf | grep baidu.com
            return 301 https://www.baidu.com;

4.2 rewrite ，訪問rewrite.lzlx.com/foo/bar/下的任意內容會重定向到/目錄下

# cat nginx-rewrite.yaml 
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: rewrite.lzlx.com
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$1   # 這里指重定向到/，然后加上所有的參數
spec:
  ingressClassName: nginx
  rules:
  - host: rewrite.lzlx.com
    http:
      paths:
      - path: /foo/bar/(.+)
        pathType: Prefix
        backend:
          service:
            name: nginx-service
            port:
              number: 80
# kubectl exec -it nginx-deployment-76447d65c-694gr -- bash
root@nginx-deployment-76447d65c-694gr:/# cat /usr/share/nginx/html/b.txt 
this is html dir

 4.3 ssl 證書配置

https://kubernetes.github.io/ingress-nginx/user-guide/tls/

生成證書，生產環境都是購買的通用證書，使用Default SSL Certificate，可以在ingress前面加個nginx proxy將證書配置到nginx proxy上，然后再反向代理到后端的ingress 80端口即可。

上傳域名證書，將域名證書創建成secret

# kubectl create secret tls m-test.com --key m-test.com.key --cert m-test.com.crt -n default
secret/m-test.com created

# kubectl get secret
NAME                  TYPE                                  DATA   AGE
default-token-zxp2q   kubernetes.io/service-account-token   3      34d
m-test.com            kubernetes.io/tls                     2      6s

創建ingress文件

# cat nginx-lzlx.m-test.com.yaml 
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: lzlx.m-test.com
spec:
  ingressClassName: nginx
  rules:
  - host: lzlx.m-test.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-service
            port:
              number: 80
  tls:
  - hosts:
    - lzlx.m-test.com
    secretName: m-test.com

# kubectl create -f nginx-lzlx.m-test.com.yaml 

創建完成再訪問的時候會自動https跳轉，如果想取消自動跳轉加上下面的參數

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: lzlx.m-test.com
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "false"

4.4 黑白名單配置

Annotations：只對指定的ingress生效

ConfigMap：全局生效

黑名單可以使用ConfigMap去配置，白名單建議使用Annotations去配置。

# cat nginx-lzlx.m-test.com.yaml 
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: lzlx.m-test.com
  annotations:
    nginx.ingress.kubernetes.io/whitelist-source-range: "192.168.0.1/24"  # 允許的白名單IP

黑名單配置

# kubectl edit  cm -n ingress-nginx ingress-nginx-controller -oyaml

apiVersion: v1
data:
  allow-snippet-annotations: "true"
  block-cidrs: 120.22.1.33                   # 要加入黑名單的IP地址

# kubectl delete pod -n ingress-nginx --all  # 重啟ingress pod，生產慎用

4.5 匹配請求頭，如果是移動設備則跳轉到m.baidu.com。server-snippet用來寫比較復雜的語法

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/server-snippet: |
        set $agentflag 0;

        if ($http_user_agent ~* "(Mobile)" ){
          set $agentflag 1;
        }

        if ( $agentflag = 1 ) {
          return 301 https://m.baidu.com;
        }

4.6 金絲雀 canary實現灰度發布

創建一個v2版本的ingress，使它的流量和正常的比為50%。

# cat www.test.com-canary.yaml 
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: www.test.com-v2                                    # 和正常的ingress入口做一個區分
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"             # 開啟灰度發布，必須先啟用canary
    nginx.ingress.kubernetes.io/canary-weight: "50"        # 權重比例，50%的走canary 這個版本
spec:
  ingressClassName: nginx
  rules:
  - host: www.test.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-service-v2
            port:
              number: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service-v2
spec:
  type: ClusterIP
  selector:
    app: nginx-v2
  ports:
  - name: http
    port: 80
    targetPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment-v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-v2
  template:
    metadata:
      labels:
        app: nginx-v2
    spec:
      containers:
      - name: mynginx
        image: nginx:1.18.0
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 500m
            memory: 1024Mi
          limits:
            cpu: 1000m
            memory: 2048Mi
# kubectl create -f www.test.com-canary.yaml 
# kubectl exec -it nginx-deployment-v2-6888df4554-pl7vz -- bash
# echo "v2" > /usr/share/nginx/html/index.html        # 將canary版本的主頁設置為v2，普通版本的設置為v1
# kubectl get ingress
NAME                     CLASS   HOSTS                    ADDRESS         PORTS     AGE
www.test.com-v1          nginx   www.test.com             10.111.38.227   80        2d23h
www.test.com-v2          nginx   www.test.com             10.111.38.227   80        2d22h

# curl www.test.com  # 進行測試
v2
# curl www.test.com
v2
# curl www.test.com
v1
# curl www.test.com
v1

nginx.ingress.kubernetes.io/canary-by-header：用於通知 Ingress 將請求路由到 Canary Ingress 中指定的服務的標頭。當請求頭設置為 時always，它將被路由到金絲雀。當標頭設置為 時never，它永遠不會被路由到金絲雀。對於任何其他值，標頭將被忽略，並按優先級將請求與其他金絲雀規則進行比較。

nginx.ingress.kubernetes.io/canary-by-header-value：要匹配的標頭值，用於通知 Ingress 將請求路由到 Canary Ingress 中指定的服務。當請求頭設置為這個值時，它將被路由到金絲雀。對於任何其他標頭值，標頭將被忽略，並按優先級將請求與其他金絲雀規則進行比較。此注釋必須與 一起使用。注釋是 的擴展，nginx.ingress.kubernetes.io/canary-by-header允許自定義標頭值而不是使用硬編碼值。如果nginx.ingress.kubernetes.io/canary-by-header未定義注釋，則沒有任何影響。

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: www.test.com-v2
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: "user"
    nginx.ingress.kubernetes.io/canary-by-header-value: "leon"

#  curl -H "user: leon" www.test.com
v2
# curl www.test.com
v1

4.7 http登錄認證

官方樣例：https://kubernetes.github.io/ingress-nginx/examples/auth/basic/

# yum install httpd -y      # 安裝htpasswd命令

# htpasswd -c auth leon     # -c 參數創建auth密碼文件 (必須是auth否則會報 ingress auth 503 Service Temporarily Unavailable) 用戶名是leon輸入密碼
New password: 
Re-type new password: 
Adding password for user leon

# kubectl create secret generic auth-wwwtest --from-file=auth     # 使用剛生成的auth-wwwtest文件，創建auth-wwwtest 的secret

# kubectl get secret auth-wwwtest -o yaml
apiVersion: v1
data:
  auth-wwwtest: bHpseDokYXByMSR1N2hYZlliZiR2OEFjYmFQcWdRdFYvUlJlNmpMQ3kxCg==
kind: Secret
metadata:
  - apiVersion: v1
  name: auth-wwwtest
  namespace: default
type: Opaque

# cat www.test.com.yaml 
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: www.test.com-v1
  annotations:
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: auth-wwwtest
    nginx.ingress.kubernetes.io/auth-realm: 'describe auth'

# kubectl apply -f www.test.com.yaml