Kubernetes 部署 Nginx Ingress Controller 之 nginxinc/kubernetes-ingress

更新：這里用的是 nginxinc/kubernetes-ingress ，還有個 kubernetes/ingress-nginx ，它們的區別見 Differences Between nginxinc/kubernetes-ingress and kubernetes/ingress-nginx Ingress Controllers ，后來我們選用了 kubernetes/ingress-nginx ，詳見博文。

開始天真地以為只要寫一個 ingress 配置文件並部署好就行了。

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: cnblogs-ingress
spec:
  rules:
  - host: q.cnblogs.com
    http:
      paths:
        - backend:
            serviceName: q-web
            servicePort: 80

# kubectl apply -f cnblogs-ingress.yaml
# kubectl get ingress
NAME              HOSTS           ADDRESS   PORTS   AGE
cnblogs-ingress   q.cnblogs.com             80      6h18

但部署后發現所有 node 服務器上沒有任何進程監聽 80 端口，顯然不對。

從 k8s 幫助文檔中知道了答案：

You must have an ingress controller to satisfy an Ingress. Only creating an Ingress resource has no effect.
In order for the Ingress resource to work, the cluster must have an ingress controller running.
Unlike other types of controllers which run as part of the kube-controller-manager binary, Ingress controllers are not started automatically with a cluster. Use this page to choose the ingress controller implementation that best fits your cluster.

原來 k8s 沒有內置 ingress controller ，需要安裝第三方的 ingress controller ，比如 nginx ingress controller ，上面通過 cnblogs-ingress.yaml 只是創建了 ingress 資源。那為什么通過 deployment.yaml 創建了 deployment 資源就能正常部署 pod ？那是因為 kube-controller-manager 中內置了 deployment controller 。

我們選用 nginx ingress controller ，部署操作步驟如下（參考文檔）：
1）從 github 上簽出 kubernetes-ingress 倉庫

$ git clone https://github.com/nginxinc/kubernetes-ingress/
$ cd kubernetes-ingress
$ git checkout v1.6.1 -f
$ cd deployments

2）創建 namespace 與 ServiceAccount ，都叫 nginx-ingress

kubectl apply -f common/ns-and-sa.yaml

3）創建 cluster role 與 cluster role binding

kubectl apply -f rbac/rbac.yaml

4）創建 secret
使用自己的證書文件創建 secret

kubectl create secret tls default-server-secret --cert=path/to/cert.pem --key=path/to/key.pem

或者使用 nginx-ingress 自帶的證書創建 sescret

kubectl apply -f common/default-server-secret.yaml

5）創建 ConfigMap

kubectl apply -f common/nginx-config.yaml 

6）創建 custom resource definitions

kubectl apply -f common/custom-resource-definitions.yaml

7）創建 DaemonSet

kubectl apply -f daemon-set/nginx-ingress.yaml

8）查看 pod 是否部署成功

$ kubectl get pods --namespace=nginx-ingress                                                                                                1 ↵
NAME                  READY   STATUS    RESTARTS   AGE
nginx-ingress-7xdzp   1/1     Running   5          12m
nginx-ingress-rs4th   1/1     Running   0          114s
nginx-ingress-w2fnh   1/1     Running   0          12m
nginx-ingress-z54r6   1/1     Running   5          12m

9）創建監聽 31080 端口的 NodePort 類型的 service

配置文件 nodeport.yaml （去掉了443端口）
注0：nodePort 只能使用 30000-32767 范圍的端口。
注1：去掉了443端口，我們在最前端使用了阿里雲負載均衡，請求都通過 http 轉發。

apiVersion: v1
kind: Service
metadata:
  name: nginx-ingress
  namespace: nginx-ingress
spec:
  type: NodePort 
  ports:
  - nodePort: 31080
     port: 80
     targetPort: 80
     protocol: TCP
     name: http
  selector:
    app: nginx-ingress

部署命令

kubectl apply -f service/nodeport.yaml

10）檢查 nginx-ingress 部署成功

進入 nginx-ingress 容器

kubectl exec -it daemonset/nginx-ingress -n nginx-ingress /bin/bash

查看 nginx 配置

cat /etc/nginx/conf.d/production-cnblogs-ingress.conf

確認 ingress 中添加的轉發規則已被導入

upstream production-cnblogs-ingress-q.cnblogs.com-q-web-80 {
	zone production-cnblogs-ingress-q.cnblogs.com-q-web-80 256k;
	random two least_conn;
	
	server 192.168.107.211:80 max_fails=1 fail_timeout=10s max_conns=0;
	server 192.168.186.72:80 max_fails=1 fail_timeout=10s max_conns=0;	
}

server {
    listen 80;
	server_tokens on;
	server_name q.cnblogs.com;
	
	location / {
		proxy_http_version 1.1;
		proxy_connect_timeout 60s;
		proxy_read_timeout 60s;
		proxy_send_timeout 60s;
		client_max_body_size 1m;
		proxy_set_header Host $host;
		proxy_set_header X-Real-IP $remote_addr;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_set_header X-Forwarded-Host $host;
		proxy_set_header X-Forwarded-Port $server_port;
		proxy_set_header X-Forwarded-Proto $scheme;
		proxy_buffering on;
		
		proxy_pass http://production-cnblogs-ingress-q.cnblogs.com-q-web-80;
	}
}

至此 nginx-ingress 部署成功。

解決轉發 X-Forwarded-Proto 請求頭問題

解決方法：在 ingress 配置文件中添加 nginx.org/redirect-to-https: "true"，詳見博問 K8s Nginx Ingress Controller 轉發 X-Forwarded-Proto 請求頭的問題

apiVersion: extensions/v1beta1 
kind: Ingress
metadata:
  name: cnblogs-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.org/redirect-to-https: "true"

添加 proxy_set_header 配置

通過 ingress 的 nginx.org/location-snippets 注解添加如下的配置：

apiVersion: extensions/v1beta1 
kind: Ingress
metadata:
  name: cnblogs-ingress
  annotations:
    nginx.org/location-snippets: |
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection $connection_upgrade;

七層負載均衡對 nginx-ingress 進行健康檢查問題

詳見博問 阿里雲負載均衡對 K8s Nginx Ingress 的健康檢查問題

基於二級域名自動轉發到 service 的實現方法

詳見博問 K8s Ingress 如何自動根據主機名中的二級域名匹配 service

應用獲取不到客戶端真實 IP 地址的問題

詳見博問 K8s 中 ASP.NET Core 應用獲取不到客戶端真實 IP 地址