Ingress-nginx進階

一、Ingress-nginx和Nginx-ingress的區別

Ingress-nginx：kubernetes官方維護的ingress

Nginx-ingress：nginx官方維護的ingress

# Ingress-nginx的官方文檔：
  https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#rewrite

# Nginx-ingress的官方文檔：
  https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/

# Ingress-nginx源碼地址
  https://github.com/kubernetes/ingress-nginx

# Nginx-ingress源碼地址
  Github：https://github.com/nginxinc/kubernetes-ingress/blob/master/docs/nginx-ingress-controllers.md

部署建議：

DaemonSet，找幾台專門的服務器進行配置ingress (如果沒有的足夠的資源，就設置QoS，保證ingress最后刪除的那個策略)
hostNetwork: true  # 這個設置為true

二、常用ingress配置

2.1、創建一個簡單的ingress實例

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-test
  namespace: ratel-test1
spec:
  rules:
  - host: ingress.test.com
    http:
      paths:
      - backend:
          serviceName: ingress-test   # 代理名字為ingress-test
          servicePort: 80             # port為80的svc
        path: /
        
# 創建ingress
kubectl create -f ingress-demo.yaml

2.2、Redirect

apiVersion: v1
items:
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      nginx.ingress.kubernetes.io/permanent-redirect: https://www.baidu.com   # 重定向到想去的url
    name: ingress-test
    namespace: ratel-test1
  spec:
    rules:
    - host: ingress.test.com
      http:
        paths:
        - backend:
            serviceName: ingress-test
            servicePort: 80
          path: /
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

2.3、Rewrite

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
  generation: 4
  name: ingress-test
  namespace: ratel-test1
spec:
  rules:
  - host: rewrite.test.com
    http:
      paths:
      - backend:
          serviceName: ingress-test
          servicePort: 80
        path: /something(/|$)(.*)   #($1)($2)

2.4、https

# 官網：https://kubernetes.github.io/ingress-nginx/user-guide/tls/

# 禁用https強制跳轉
nginx.ingress.kubernetes.io/ssl-redirect: "false"

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "false"   # 禁用https強制跳轉
  generation: 1
  name: test-tls
  namespace: ratel-test1
spec:
  rules:
  - host: test-tls.test.com
    http:
      paths:
      - backend:
          serviceName: ingress-test
          servicePort: 80
        path: /
  tls:
  - hosts:
    - test-tls.test.com
    secretName: ca-cert
設置默認證書：--default-ssl-certificate=default/foo-tls
更改的ingress-controller的啟動參數

2.5、Dashboard自定義證書

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: kubernetes-dashboard
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: normal
                operator: In
                values:
                - "true"
      containers:
      - args:
        - --auto-generate-certificates=false
        - --tls-key-file=server.key
        - --tls-cert-file=server.pem
        - --token-ttl=21600
        - --authentication-mode=basic,token
        - --namespace=kubernetes-dashboard
        image: kubernetesui/dashboard:v2.0.0-rc5
        imagePullPolicy: Always
        lifecycle: {}
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /
            port: 8443
            scheme: HTTPS
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 30
        name: kubernetes-dashboard
        ports:
        - containerPort: 8443
          protocol: TCP
        resources: {}
        securityContext:
          privileged: false
          procMount: Default
          runAsNonRoot: false
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /certs
          name: kubernetes-dashboard-new
        - mountPath: /tmp
          name: tmp-volume
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: kubernetes-dashboard
      serviceAccountName: kubernetes-dashboard
      terminationGracePeriodSeconds: 30
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
        operator: Exists
      volumes:
      - name: kubernetes-dashboard-new
        secret:
          defaultMode: 420
          secretName: kubernetes-dashboard-new
      - emptyDir: {}
        name: tmp-volume

2.7、黑白名單

黑名單：拒絕某段IP訪問

白名單：只允許某段IP訪問

Annotations：只對指定的ingress生效

ConfigMap：全局生效

黑名單可以使用ConfigMap去配置，白名單建議使用Annotations去配置

2.7.1、白名單配置（建議使用Annotations）

# 官網：https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#whitelist-source-range
annotations:
  nginx.ingress.kubernetes.io/whitelist-source-range 10.0.0.0/24,172.10.0.1  # 后面可以跟一個或者多個IP

2.7.2、黑名單設置（建議使用ConfigMap）（這是全局生效的）

# 因為黑名單可能會不定時加上去，防止惡意攻擊的。所以用ConfigMap（熱更新）

# 1、更改ingress-nginx的cm
[root@k8s-master01 ~]#  kubectl edit  cm -n ingress-nginx ingress-nginx-controller -oyaml
apiVersion: v1
data:                          # 加上data
  block-cidrs: 192.168.1.201   # 加上block-cidrs，后面也可以跟多個IP，隔開
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: ingress-nginx
    meta.helm.sh/release-namespace: ingress-nginx
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: 0.43.0
    helm.sh/chart: ingress-nginx-3.20.1
  name: ingress-nginx-controller
  namespace: ingress-nginx

# 2、刪除ingress-nginx的pod，重新加載配置
[root@k8s-master01 ~]# kubectl get po -n ingress-nginx
[root@k8s-master01 ~]# kubectl delete po -n ingress-nginx --all  # 生產一個一個刪，防止配置錯誤都掛了

# 3、在192.168.1.201節點上，訪問ingress代理的域名，然后403表示配置成功

2.7.2、針對某個域名設置黑名單--snippet

官網參考：https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/annotations.md#canary

# 比如ingress代理了www.test.com這個域名，那么想針對這個域名（www.test.com）設置訪問黑名單，就編輯這個ingress即可
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  # 在annotations下面加上這幾行配置，有多個IP可以deny多個
  annotations:
    nginx.ingress.kubernetes.io/server-snippet: |-    
      deny 192.168.1.101;
      deny 192.168.1.102;
      allow all;
# 然后在deny的主機上訪問 www.test.com 就403
[root@k8s-master02 ~]# curl ngdemo.qikqiak.com
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx</center>
</body>
</html>

2.8、匹配請求頭設置

annotations:
    nginx.ingress.kubernetes.io/server-snippet: |
        set $agentflag 0;

        if ($http_user_agent ~* "(iPhone)" ){   # 匹配規則設置（設置匹配為iPhone的手機，則重定向到下面的url）
          set $agentflag 1;
        }

        if ( $agentflag = 1 ) {
          return 301 https://www.baidu.com;     # 重定向到指定的url
        }

2.8、速率限制，其他的也一樣參考官網吧

官網參考：https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/annotations.md#canary

2.9、ingress-nginx基本認證（Basic Auth）

同樣我們還可以在 Ingress Controller 上面配置一些基本的 Auth 認證，比如 Basic Auth，可以用 htpasswd 生成一個密碼文件來驗證身份驗證。

[root@k8s-master01 ~]# htpasswd -c auth foo   # 賬號foo   密碼 123456   
New password:              # 123456
Re-type new password:      # 123456
Adding password for user foo

# 生成一個auth文件
[root@k8s-master01 ~]# ls auth
auth

然后根據上面的 auth 文件創建一個 secret 對象：

[root@k8s-master01 ~]# kubectl create secret generic basic-auth --from-file=auth
secret/basic-auth created
[root@k8s-master01 ~]# kubectl get secret basic-auth -o yaml
apiVersion: v1
data:
  auth: Zm9vOiRhcHIxJGZzREw0b0xmJHNuUUNDTFkxbTE2N1BkNUdEMHIwcC8K
kind: Secret
metadata:
  name: basic-auth
  namespace: default
type: Opaque

然后對上面的 my-nginx 應用創建一個具有 Basic Auth 的 Ingress 對象：

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-with-auth
  annotations:
    # 認證類型
    nginx.ingress.kubernetes.io/auth-type: basic
    # 包含 user/password 定義的 secret 對象名
    nginx.ingress.kubernetes.io/auth-secret: basic-auth
    # 要顯示的帶有適當上下文的消息，說明需要身份驗證的原因
    nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required - foo'
spec:
  rules:
  - host: foo.bar.com
    http:
      paths:
      - path: /
        backend:
          serviceName: my-nginx
          servicePort: 80

直接創建上面的資源對象，然后通過下面的命令或者在瀏覽器中直接打開配置的域名：

➜ curl -v http://k8s.qikqiak.com -H 'Host: foo.bar.com'
* Rebuilt URL to: http://k8s.qikqiak.com/
*   Trying 123.59.188.12...
* TCP_NODELAY set
* Connected to k8s.qikqiak.com (123.59.188.12) port 80 (#0)
> GET / HTTP/1.1
> Host: foo.bar.com
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< Server: openresty/1.15.8.2
< Date: Sun, 08 Dec 2019 06:44:35 GMT
< Content-Type: text/html
< Content-Length: 185
< Connection: keep-alive
< WWW-Authenticate: Basic realm="Authentication Required - foo"
<
<html>
<head><title>401 Authorization Required</title></head>
<body>
<center><h1>401 Authorization Required</h1></center>
<hr><center>openresty/1.15.8.2</center>
</body>
</html>

我們可以看到出現了 401 認證失敗錯誤，然后帶上我們配置的用戶名和密碼進行認證：

➜ curl -v http://k8s.qikqiak.com -H 'Host: foo.bar.com' -u 'foo:foo'
* Rebuilt URL to: http://k8s.qikqiak.com/
*   Trying 123.59.188.12...
* TCP_NODELAY set
* Connected to k8s.qikqiak.com (123.59.188.12) port 80 (#0)
* Server auth using Basic with user 'foo'
> GET / HTTP/1.1
> Host: foo.bar.com
> Authorization: Basic Zm9vOmZvbw==
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: openresty/1.15.8.2
< Date: Sun, 08 Dec 2019 06:46:27 GMT
< Content-Type: text/html
< Content-Length: 612
< Connection: keep-alive
< Vary: Accept-Encoding
< Last-Modified: Tue, 19 Nov 2019 12:50:08 GMT
< ETag: "5dd3e500-264"
< Accept-Ranges: bytes
<
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

可以看到已經認證成功了。當然出來 Basic Auth 這一種簡單的認證方式之外，NGINX Ingress Controller 還支持一些其他高級的認證，比如 OAUTH 認證之類的。

三、灰度發布

3.1、准備2個svc，用於演示

[root@k8s-master01 app]# kubectl get svc
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
my-nginx     ClusterIP   10.104.87.14    <none>        80/TCP    91m
my-nginx1    ClusterIP   10.103.175.67   <none>        80/TCP    2m12s
[root@k8s-master01 app]# curl 10.104.87.14
v1
[root@k8s-master01 app]# curl 10.103.175.67 
v2

3.2、開啟基於ingress的灰度發布

官網參考：https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/annotations.md#canary

# 開啟了灰度發布，才能在同一個ns下創建2個同樣域名
# 先創建一個普通的ingress，通過canary.test.com就可以訪問到svc my-nginx，版本是v1
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-nginx
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: canary.test.com  # 將域名映射到 my-nginx 服務
    http:
      paths:
      - path: /
        backend:
          serviceName: my-nginx  # 將所有請求發送到 my-nginx 服務的 80 端口
          servicePort: 80     

3.2.1、基於權重的流量調度

基於權重：基於權重的流量切分的典型應用場景就是藍綠部署，可通過將權重設置為 0 或 100 來實現。例如，可將 Green 版本設置為主要部分，並將 Blue 版本的入口配置為 Canary。最初，將權重設置為 0，因此不會將流量代理到 Blue 版本。一旦新版本測試和驗證都成功后，即可將 Blue 版本的權重設置為 100，即所有流量從 Green 版本轉向 Blue。

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-nginx1
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: canary.test.com
    http:
      paths:
      - path: /
        backend:
          serviceName: my-nginx1
          servicePort: 80     
# 這個是代理V2的ingress，如果使用相同域名，創建會報錯
[root@k8s-master01 app]# kubectl apply -f  bbb.yaml 
Error from server (BadRequest): error when creating "bbb.yaml": admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: host "canary.test.com" and path "/" is already defined in ingress default/my-nginx

# 基於權重
annotations:
  nginx.ingress.kubernetes.io/canary: "true"        # 要開啟灰度發布機制，首先需要啟用 Canary
  nginx.ingress.kubernetes.io/canary-weight: "30"   # 切30%的流量到v2去，設置100就全部切過去了
  
# 創建后查看ingress，可以看到創建了2個代理相同域名的ingress
[root@k8s-master01 app]# kubectl get ingress
NAME        CLASS    HOSTS             ADDRESS         PORTS   AGE
my-nginx    <none>   canary.test.com   10.101.29.125   80      20m
my-nginx1   <none>   canary.test.com   10.101.29.125   80      3m6s

驗證是否配置成功：

[root@k8s-master02 ~]# for i in $(seq 1 3); do curl -s -H canary.test.com; done
v1
v2
v1

3.2.2、基於Request Header（還有個never、always不進行演示）

基於 Request Header: 基於 Request Header 進行流量切分的典型應用場景即灰度發布或 A/B 測試場景

注意：當 Request Header 設置為 never 或 always 時，請求將不會或一直被發送到 Canary 版本，對於任何其他 Header 值，將忽略 Header，並通過優先級將請求與其他 Canary 規則進行優先級的比較。

# 基於 Request Header
annotations:
  kubernetes.io/ingress.class: nginx 
  nginx.ingress.kubernetes.io/canary: "true"                  # 要開啟灰度發布機制，首先需要啟用 Canary
  nginx.ingress.kubernetes.io/canary-by-header-value: canary  # 基於header的流量切分 value
  nginx.ingress.kubernetes.io/canary-by-header: user          # 基於header的流量切分  key
  nginx.ingress.kubernetes.io/canary-weight: "30"             # 會被忽略，因為配置了 canary-by-headerCanary版本

驗證：

➜ for i in $(seq 1 10); do curl -s -H "canary: never" echo.qikqiak.com | grep "Hostname"; done
Hostname: production-856d5fb99-d6bds
Hostname: production-856d5fb99-d6bds
Hostname: production-856d5fb99-d6bds
Hostname: production-856d5fb99-d6bds
Hostname: production-856d5fb99-d6bds
Hostname: production-856d5fb99-d6bds
Hostname: production-856d5fb99-d6bds
Hostname: production-856d5fb99-d6bds
Hostname: production-856d5fb99-d6bds
Hostname: production-856d5fb99-d6bds

# 流量全部切到V2了，從而實現灰度發布
[root@k8s-master02 ~]# for i in $(seq 1 5); do curl -s -H "user: canary" canary.test.com; done
v2
v2
v2
v2
v2

3.2.3、基於 Cookie

基於 Cookie：與基於 Request Header 的 annotation 用法規則類似。例如在 A/B 測試場景下，需要讓地域為北京的用戶訪問 Canary 版本。那么當 cookie 的 annotation 設置為 nginx.ingress.kubernetes.io/canary-by-cookie: "users_from_Beijing"，此時后台可對登錄的用戶請求進行檢查，如果該用戶訪問源來自北京則設置 cookie users_from_Beijing 的值為 always，這樣就可以確保北京的用戶僅訪問 Canary 版本。

annotations:
  kubernetes.io/ingress.class: nginx 
  nginx.ingress.kubernetes.io/canary: "true"   # 要開啟灰度發布機制，首先需要啟用 Canary
  nginx.ingress.kubernetes.io/canary-by-cookie: "users_from_Beijing"  # 基於 cookie
  nginx.ingress.kubernetes.io/canary-weight: "30"  # 會被忽略，因為配置了 canary-by-cookie

➜ for i in $(seq 1 10); do curl -s -b "users_from_Beijing=always" echo.qikqiak.com | grep "Hostname"; done
Hostname: canary-66cb497b7f-48zx4
Hostname: canary-66cb497b7f-48zx4
Hostname: canary-66cb497b7f-48zx4
Hostname: canary-66cb497b7f-48zx4
Hostname: canary-66cb497b7f-48zx4
Hostname: canary-66cb497b7f-48zx4
Hostname: canary-66cb497b7f-48zx4
Hostname: canary-66cb497b7f-48zx4
Hostname: canary-66cb497b7f-48zx4
Hostname: canary-66cb497b7f-48zx4

四、ingress-nginx監控

https://kubernetes.github.io/ingress-nginx/user-guide/monitoring/
https://github.com/kubernetes/ingress-nginx/tree/master/deploy
https://github.com/kubernetes/ingress-nginx/tree/master/deploy/grafana/dashboards