【K8s教程】使用Helm部署Nginx Ingress控制器（DaemonSet hostNetwork 模式）

參考：
https://cloud.tencent.com/developer/article/1761376
https://kubernetes.github.io/ingress-nginx/
https://github.com/kubernetes/ingress-nginx
https://artifacthub.io/packages/helm/ingress-nginx/ingress-nginx

NodePort 和 LoadBlancer 類型的 Service 可以把應用暴露給外部用戶使用，除此之外，Kubernetes 還為我們提供了一個非常重要的資源對象可以用來暴露服務給外部用戶，那就是 Ingress。對於小規模的應用我們使用 NodePort 或許能夠滿足我們的需求，但是當你的應用越來越多的時候，你就會發現對於 NodePort 的管理就非常麻煩了，這個時候使用 Ingress 就非常方便了，可以避免管理大量的端口。

安裝 NGINX Ingress Controller

生產環境可以使用 HA + LB + DaemonSet hostNetwork 模式。

1.獲取資源清單文件：

# helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
# helm repo update
# helm pull ingress-nginx/ingress-nginx
# tar -xvf ingress-nginx-3.35.0.tgz

2.新建一個名為 values-prod.yaml 的 Values 文件，用來覆蓋 ingress-nginx 默認的 Values 值，對應的數據如下所示：

# cat ingress-nginx/values-prod.yaml 
controller:
  name: controller
  image:
    repository: registry.cn-hangzhou.aliyuncs.com/varden/ingress-nginx-controller
    tag: "v0.48.1"
    digest: 

  dnsPolicy: ClusterFirstWithHostNet
   
  hostNetwork: true

  publishService:  # hostNetwork 模式下設置為false，通過節點IP地址上報ingress status數據
    enabled: false

  kind: DaemonSet

  nodeSelector: 
    role: ingressproxy

  service:  # HostNetwork 模式不需要創建service
    enabled: false

  tolerations:
    - operator: "Exists"

  resources:
    limits:
      cpu: 100m
      memory: 90Mi
    requests:
      cpu: 100m
      memory: 90Mi

  terminationGracePeriodSeconds: 300

  admissionWebhooks:
    patch:
      enabled: true
      image:
        registry: docker.io
        image: jettech/kube-webhook-certgen
        tag: v1.5.1

defaultBackend:
  enabled: true
  name: defaultbackend
  image:
    repository: registry.cn-hangzhou.aliyuncs.com/varden/defaultbackend-amd64
    tag: "1.5"

3.然后使用如下命令安裝 ingress-nginx 應用到 ingress-nginx 的命名空間中：

# kubectl create ns ingress-nginx
# kubectl label nodes k8s-master role=ingressproxy
# helm install ingress-nginx ./ingress-nginx -f ./ingress-nginx/values-prod.yaml --namespace ingress-nginx
NAME: ingress-nginx
LAST DEPLOYED: Wed Aug 11 09:53:41 2021
NAMESPACE: ingress-nginx
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The ingress-nginx controller has been installed.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status by running 'kubectl --namespace ingress-nginx get services -o wide -w ingress-nginx-controller'

An example Ingress that makes use of the controller:

  apiVersion: networking.k8s.io/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: nginx
    name: example
    namespace: foo
  spec:
    rules:
      - host: www.example.com
        http:
          paths:
            - backend:
                serviceName: exampleService
                servicePort: 80
              path: /
    # This section is only required if TLS is to be enabled for the Ingress
    tls:
        - hosts:
            - www.example.com
          secretName: example-tls

If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:

  apiVersion: v1
  kind: Secret
  metadata:
    name: example-tls
    namespace: foo
  data:
    tls.crt: <base64 encoded cert>
    tls.key: <base64 encoded key>
  type: kubernetes.io/tls

更新部署命令：helm upgrade ingress-nginx ./ingress-nginx -f ./ingress-nginx/values-prod.yaml --namespace ingress-nginx

4.部署完成后查看 Pod 的運行狀態：

# kubectl get svc -n ingress-nginx
NAME                                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
ingress-nginx-controller-admission   ClusterIP   10.110.55.35    <none>        443/TCP   5h22m
ingress-nginx-defaultbackend         ClusterIP   10.98.122.133   <none>        80/TCP    5h22m

# kubectl get pods -n ingress-nginx
NAME                                            READY   STATUS    RESTARTS   AGE
ingress-nginx-controller-zv9fd                  1/1     Running   0          57m
ingress-nginx-defaultbackend-7b85879754-4khmd   1/1     Running   0          54m

# POD_NAME=$(kubectl get pods -l app.kubernetes.io/name=ingress-nginx -n ingress-nginx -o jsonpath='{.items[0].metadata.name}')
# kubectl exec -it $POD_NAME -n ingress-nginx -- /nginx-ingress-controller --version
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v0.48.1
  Build:         30809c066cd027079cbb32dccc8a101d6fbffdcb
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.20.1

-------------------------------------------------------------------------------

當看到上面的信息證明 ingress-nginx 部署成功了。

5.創建一個 Ingress 資源並簡單測試

# vim example-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  name: example
  namespace: default
spec:
  rules:
  - host: www.example.com
    http:
      paths:
        - backend:
            service:
              name: nginx
              port:
                number: 80
          path: /
          pathType: Prefix

# kubectl apply -f example-ingress.yaml
ingress.networking.k8s.io/example configured

# kubectl get ingress example -o yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx"},"name":"example","namespace":"default"},"spec":{"rules":[{"host":"www.example.com","http":{"paths":[{"backend":{"service":{"name":"nginx","port":{"number":80}}},"path":"/","pathType":"Prefix"}]}}]}}
    kubernetes.io/ingress.class: nginx
  creationTimestamp: "2021-08-11T02:13:54Z"
  generation: 3
  name: example
  namespace: default
  resourceVersion: "1944004"
  uid: 26279640-a55e-46a8-8b07-9569c513306a
spec:
  rules:
  - host: www.example.com
    http:
      paths:
      - backend:
          service:
            name: nginx
            port:
              number: 80
        path: /
        pathType: Prefix
status:
  loadBalancer:
    ingress:
    - ip: 192.168.100.10

# kubectl describe ingress example 
Name:             example
Namespace:        default
Address:          192.168.100.10
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host             Path  Backends
  ----             ----  --------
  www.example.com  
                   /   nginx:80 (10.244.0.42:80,10.244.0.49:80)
Annotations:       kubernetes.io/ingress.class: nginx
Events:
  Type    Reason  Age                  From                      Message
  ----    ------  ----                 ----                      -------
  Normal  Sync    3m51s (x3 over 73m)  nginx-ingress-controller  Scheduled for sync

# curl -H "Host: www.example.com" http://192.168.100.10
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

下圖顯示了客戶端是如果通過 Ingress Controller 連接到其中一個 Pod 的流程，客戶端首先對 ngdemo.qikqiak.com 執行 DNS 解析，得到 Ingress Controller 所在節點的 IP，然后客戶端向 Ingress Controller 發送 HTTP 請求，然后根據 Ingress 對象里面的描述匹配域名，找到對應的 Service 對象，並獲取關聯的 Endpoints 列表，將客戶端的請求轉發給其中一個 Pod。