在K8S集群中使用NGINX Ingress V1.7

NGINX Ingress控制器有兩個項目一個K8S官網維護（官方項目github地址），另一個是NGINX, Inc.維護的項目（NGINX項目github地址），因為差異很多兩個項目不要弄混，以下示例是用NGINX維護的項目（部署說明資料），我們采用Manifests文件部署方式Installation with Manifests。

K8S的環境已經部署完成，詳情請看安裝Kubernetes V1.18.2

希望可以達到的效果示意圖

如果增加Node節點需要手工修改宿主機的Nginx設置，URL路由使用NGINX Ingress控制，路由信息使用Ingress類型編寫，K8S集群自己管理Ingress、Service和Pod之間的負載均衡。

安裝NGINX, Inc.的NGINX Ingress控制器

mkdir ~/nginx-ingress
cd ~/nginx-ingress

#創建Ingress控制器的命名空間和服務賬號
wget https://raw.githubusercontent.com/nginxinc/kubernetes-ingress/release-1.7/deployments/common/ns-and-sa.yaml
kubectl apply -f ns-and-sa.yaml

#創建給服務賬號綁定集群角色
wget https://raw.githubusercontent.com/nginxinc/kubernetes-ingress/release-1.7/deployments/rbac/rbac.yaml
kubectl apply -f rbac.yaml

#TLS證書（僅用於測試）
wget https://raw.githubusercontent.com/nginxinc/kubernetes-ingress/release-1.7/deployments/common/default-server-secret.yaml
#改上述文件的命名空間和名字可以用於其他命名空間，但也只是用於測試
kubectl apply -f default-server-secret.yaml

#MapConfig
wget https://raw.githubusercontent.com/nginxinc/kubernetes-ingress/release-1.7/deployments/common/nginx-config.yaml
kubectl apply -f nginx-config.yaml

#VirtualServer and VirtualServerRoute and TransportServer資源
wget https://raw.githubusercontent.com/nginxinc/kubernetes-ingress/release-1.7/deployments/common/ts-definition.yaml
kubectl apply -f ts-definition.yaml
wget https://raw.githubusercontent.com/nginxinc/kubernetes-ingress/release-1.7/deployments/common/vs-definition.yaml
kubectl apply -f vs-definition.yaml
wget https://raw.githubusercontent.com/nginxinc/kubernetes-ingress/release-1.7/deployments/common/vsr-definition.yaml
kubectl apply -f vsr-definition.yaml

#GlobalConfiguration
wget https://raw.githubusercontent.com/nginxinc/kubernetes-ingress/release-1.7/deployments/common/gc-definition.yaml
kubectl apply -f gc-definition.yaml
wget https://raw.githubusercontent.com/nginxinc/kubernetes-ingress/release-1.7/deployments/common/global-configuration.yaml
kubectl apply -f global-configuration.yaml

#使用DaemonSet方式部署到集群
wget https://raw.githubusercontent.com/nginxinc/kubernetes-ingress/release-1.7/deployments/daemon-set/nginx-ingress.yaml
kubectl apply -f nginx-ingress.yaml

#創建服務(NodePort)
wget https://raw.githubusercontent.com/nginxinc/kubernetes-ingress/release-1.7/deployments/service/nodeport.yaml
kubectl apply -f nodeport.yaml

#查看結果
kubectl get pods --namespace=nginx-ingress

創建示例Pod、Service、Secret、ingress

示例說明詳情，命名空間我設置為nginx-ingress，雖然NGINX Ingress控制器是全命名空間監控的，但路由（Ingress）、服務（Service）、憑證（Secret）都要在一個命名空間下比較安全。

vi cafe-ingress.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: coffee
  namespace: nginx-ingress
spec:
  replicas: 2
  selector:
    matchLabels:
      app: coffee
  template:
    metadata:
      labels:
        app: coffee
    spec:
      containers:
      - name: coffee
        image: nginxdemos/nginx-hello:plain-text
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: coffee-svc
  namespace: nginx-ingress
spec:
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
    name: http
  selector:
    app: coffee
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tea
  namespace: nginx-ingress
spec:
  replicas: 2
  selector:
    matchLabels:
      app: tea
  template:
    metadata:
      labels:
        app: tea 
    spec:
      containers:
      - name: tea 
        image: nginxdemos/nginx-hello:plain-text
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: tea-svc
  namespace: nginx-ingress
  labels:
spec:
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
    name: http
  selector:
    app: tea
---
apiVersion: v1
kind: Secret
metadata:
  name: cafe-secret
  namespace: nginx-ingress
type: Opaque
data:
  tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURMakNDQWhZQ0NRREFPRjl0THNhWFdqQU5CZ2txaGtpRzl3MEJBUXNGQURCYU1Rc3dDUVlEVlFRR0V3SlYKVXpFTE1Ba0dBMVVFQ0F3Q1EwRXhJVEFmQmdOVkJBb01HRWx1ZEdWeWJtVjBJRmRwWkdkcGRITWdVSFI1SUV4MApaREViTUJrR0ExVUVBd3dTWTJGbVpTNWxlR0Z0Y0d4bExtTnZiU0FnTUI0WERURTRNRGt4TWpFMk1UVXpOVm9YCkRUSXpNRGt4TVRFMk1UVXpOVm93V0RFTE1Ba0dBMVVFQmhNQ1ZWTXhDekFKQmdOVkJBZ01Ba05CTVNFd0h3WUQKVlFRS0RCaEpiblJsY201bGRDQlhhV1JuYVhSeklGQjBlU0JNZEdReEdUQVhCZ05WQkFNTUVHTmhabVV1WlhoaApiWEJzWlM1amIyMHdnZ0VpTUEwR0NTcUdTSWIzRFFFQkFRVUFBNElCRHdBd2dnRUtBb0lCQVFDcDZLbjdzeTgxCnAwanVKL2N5ayt2Q0FtbHNmanRGTTJtdVpOSzBLdGVjcUcyZmpXUWI1NXhRMVlGQTJYT1N3SEFZdlNkd0kyaloKcnVXOHFYWENMMnJiNENaQ0Z4d3BWRUNyY3hkam0zdGVWaVJYVnNZSW1tSkhQUFN5UWdwaW9iczl4N0RsTGM2SQpCQTBaalVPeWwwUHFHOVNKZXhNVjczV0lJYTVyRFZTRjJyNGtTa2JBajREY2o3TFhlRmxWWEgySTVYd1hDcHRDCm42N0pDZzQyZitrOHdnemNSVnA4WFprWldaVmp3cTlSVUtEWG1GQjJZeU4xWEVXZFowZXdSdUtZVUpsc202OTIKc2tPcktRajB2a29QbjQxRUUvK1RhVkVwcUxUUm9VWTNyemc3RGtkemZkQml6Rk8yZHNQTkZ4MkNXMGpYa05MdgpLbzI1Q1pyT2hYQUhBZ01CQUFFd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFLSEZDY3lPalp2b0hzd1VCTWRMClJkSEliMzgzcFdGeW5acS9MdVVvdnNWQTU4QjBDZzdCRWZ5NXZXVlZycTVSSWt2NGxaODFOMjl4MjFkMUpINnIKalNuUXgrRFhDTy9USkVWNWxTQ1VwSUd6RVVZYVVQZ1J5anNNL05VZENKOHVIVmhaSitTNkZBK0NuT0Q5cm4yaQpaQmVQQ0k1ckh3RVh3bm5sOHl3aWozdnZRNXpISXV5QmdsV3IvUXl1aTlmalBwd1dVdlVtNG52NVNNRzl6Q1Y3ClBwdXd2dWF0cWpPMTIwOEJqZkUvY1pISWc4SHc5bXZXOXg5QytJUU1JTURFN2IvZzZPY0s3TEdUTHdsRnh2QTgKN1dqRWVxdW5heUlwaE1oS1JYVmYxTjM0OWVOOThFejM4Zk9USFRQYmRKakZBL1BjQytHeW1lK2lHdDVPUWRGaAp5UkU9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
  tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFBS0NBUUVBcWVpcCs3TXZOYWRJN2lmM01wUHJ3Z0pwYkg0N1JUTnBybVRTdENyWG5LaHRuNDFrCkcrZWNVTldCUU5semtzQndHTDBuY0NObzJhN2x2S2wxd2k5cTIrQW1RaGNjS1ZSQXEzTVhZNXQ3WGxZa1YxYkcKQ0pwaVJ6ejBza0lLWXFHN1BjZXc1UzNPaUFRTkdZMURzcGRENmh2VWlYc1RGZTkxaUNHdWF3MVVoZHErSkVwRwp3SStBM0kreTEzaFpWVng5aU9WOEZ3cWJRcCt1eVFvT05uL3BQTUlNM0VWYWZGMlpHVm1WWThLdlVWQ2cxNWhRCmRtTWpkVnhGbldkSHNFYmltRkNaYkp1dmRySkRxeWtJOUw1S0Q1K05SQlAvazJsUkthaTAwYUZHTjY4NE93NUgKYzMzUVlzeFR0bmJEelJjZGdsdEkxNURTN3lxTnVRbWF6b1Z3QndJREFRQUJBb0lCQVFDUFNkU1luUXRTUHlxbApGZlZGcFRPc29PWVJoZjhzSStpYkZ4SU91UmF1V2VoaEp4ZG01Uk9ScEF6bUNMeUw1VmhqdEptZTIyM2dMcncyCk45OUVqVUtiL1ZPbVp1RHNCYzZvQ0Y2UU5SNThkejhjbk9SVGV3Y290c0pSMXBuMWhobG5SNUhxSkpCSmFzazEKWkVuVVFmY1hackw5NGxvOUpIM0UrVXFqbzFGRnM4eHhFOHdvUEJxalpzVjdwUlVaZ0MzTGh4bndMU0V4eUZvNApjeGI5U09HNU9tQUpvelN0Rm9RMkdKT2VzOHJKNXFmZHZ5dGdnOXhiTGFRTC94MGtwUTYyQm9GTUJEZHFPZVBXCktmUDV6WjYvMDcvdnBqNDh5QTFRMzJQem9idWJzQkxkM0tjbjMyamZtMUU3cHJ0V2wrSmVPRmlPem5CUUZKYk4KNHFQVlJ6NWhBb0dCQU50V3l4aE5DU0x1NFArWGdLeWNrbGpKNkY1NjY4Zk5qNUN6Z0ZScUowOXpuMFRsc05ybwpGVExaY3hEcW5SM0hQWU00MkpFUmgySi9xREZaeW5SUW8zY2czb2VpdlVkQlZHWTgrRkkxVzBxZHViL0w5K3l1CmVkT1pUUTVYbUdHcDZyNmpleHltY0ppbS9Pc0IzWm5ZT3BPcmxEN1NQbUJ2ek5MazRNRjZneGJYQW9HQkFNWk8KMHA2SGJCbWNQMHRqRlhmY0tFNzdJbUxtMHNBRzR1SG9VeDBlUGovMnFyblRuT0JCTkU0TXZnRHVUSnp5K2NhVQprOFJxbWRIQ2JIelRlNmZ6WXEvOWl0OHNaNzdLVk4xcWtiSWN1YytSVHhBOW5OaDFUanNSbmU3NFowajFGQ0xrCmhIY3FIMHJpN1BZU0tIVEU4RnZGQ3haWWRidUI4NENtWmlodnhicFJBb0dBSWJqcWFNWVBUWXVrbENkYTVTNzkKWVNGSjFKelplMUtqYS8vdER3MXpGY2dWQ0thMzFqQXdjaXowZi9sU1JxM0hTMUdHR21lemhQVlRpcUxmZVpxYwpSMGlLYmhnYk9jVlZrSkozSzB5QXlLd1BUdW14S0haNnpJbVpTMGMwYW0rUlk5WUdxNVQ3WXJ6cHpjZnZwaU9VCmZmZTNSeUZUN2NmQ21mb09oREN0enVrQ2dZQjMwb0xDMVJMRk9ycW40M3ZDUzUxemM1em9ZNDR1QnpzcHd3WU4KVHd2UC9FeFdNZjNWSnJEakJDSCtULzZzeXNlUGJKRUltbHpNK0l3eXRGcEFOZmlJWEV0LzQ4WGY2ME54OGdXTQp1SHl4Wlp4L05LdER3MFY4dlgxUE9ucTJBNWVpS2ErOGpSQVJZS0pMWU5kZkR1d29seHZHNmJaaGtQaS80RXRUCjNZMThzUUtCZ0h0S2JrKzdsTkpWZXN3WEU1Y1VHNkVEVXNEZS8yVWE3ZlhwN0ZjanFCRW9hcDFMU3crNlRYcDAKWmdybUtFOEFSek00NytFSkhVdmlpcS9udXBFMTVnMGtKVzNzeWhwVTl6WkxPN2x0QjBLSWtPOVpSY21Vam84UQpjcExsSE1BcWJMSjhXWUdKQ2toaVd4eWFsNmhZVHlXWTRjVmtDMHh0VGwvaFVFOUllTktvCi0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: cafe-ingress
  namespace: nginx-ingress
spec:
  tls:
  - hosts:
    - cafe.example.com
    secretName: cafe-secret
  rules:
  - host: cafe.example.com
    http:
      paths:
      - path: /tea
        backend:
          serviceName: tea-svc
          servicePort: 80
      - path: /coffee
        backend:
          serviceName: coffee-svc
          servicePort: 80

PS：host節點必須有！

kubectl apply -f cafe-ingress.yaml

在集群內部檢查部署效果

kubectl get pod,svc,secret,ingress -n nginx-ingress -o wide

#查看路由信息詳情
kubectl describe ingress cafe-ingress  -n nginx-ingress

在Ingress控制器內部會創建一個Nginx配置文件，這個例子里文件名是nginx-ingress-cafe-ingress.conf，可以使用下列命令查看：

kubectl exec  nginx-ingress-dbxhj -n nginx-ingress -- cat /etc/nginx/conf.d/nginx-ingress-cafe-ingress.conf

該配置內容通過nginx-ingress的配置模版自動創建，默認模塊可參考默認模版。找到nginx-ingress控制器pod的Node節點IP和暴露的端口（我的環境是192.168.122.5），執行下列命令測試效果（nginx-ingress控制器的部署文件里已經聲明了hostPort端口，所以可以直接使用NodeIP和Node端口）：

curl --resolve cafe.example.com:443:192.168.122.5 https://cafe.example.com:443/tea --insecure

兩次請求已經由cafe-ingress進行了路由（/tea），然后tea-svc服務進行了負載均衡，所以兩次是訪問不同的Pod（都是tea的Pod）。

將示例中的HTTPS改為HTTP（只是為了省事）

vi cafe-ingress.yaml

#去掉tls節點
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: cafe-ingress
  namespace: nginx-ingress
spec:
  rules:
  - host: cafe.example.com
    http:
      paths:
      - path: /tea
        backend:
          serviceName: tea-svc
          servicePort: 80
      - path: /coffee
        backend:
          serviceName: coffee-svc
          servicePort: 80

kubectl apply -f cafe-ingress.yaml

#測試一下
curl --resolve cafe.example.com:80:192.168.122.5 http://cafe.example.com/tea

修改宿主機Nginx配置

安裝 Nginx

vi /etc/nginx/conf.d/proxy_svrs

upstream proxy_svrs
{
  server 192.168.122.4:80;
  server 192.168.122.5:80;
}

vi /etc/nginx/conf.d/k8s.conf

server{
    listen 80 default_server;
    server_name _;
    location / {
        include ./conf.d/head.conf;
        proxy_pass http://proxy_svrs;
    }
}

vi /etc/nginx/nginx.conf

include /etc/nginx/conf.d/proxy_svrs; #添加
include /etc/nginx/conf.d/*.conf;

nginx -s reload

測試外部訪問

在本機hosts文件中添加：

192.168.0.5 cafe.example.com

#執行
curl http://cafe.example.com/tea