Kubernetes的負載均衡問題(Nginx Ingress)

Kubernetes關於服務的暴露主要是通過NodePort方式，通過綁定minion主機的某個端口,然后進行pod的請求轉發和負載均衡，但這種方式下缺陷是

• Service可能有很多個，如果每個都綁定一個node主機端口的話，主機需要開放外圍一堆的端口進行服務調用，管理混亂
• 無法應用很多公司要求的防火牆規則

理想的方式是通過一個外部的負載均衡器，綁定固定的端口，比如80,然后根據域名或者服務名向后面的Service ip轉發,Nginx很好的解決了這個需求，但問題是如果有新的服務加入，如何去修改Nginx的配置，並且加載這些配置？ Kubernetes給出的方案就是Ingress,Ingress包含了兩大主件Ingress Controller和Ingress.

• Ingress解決的是新的服務加入后，域名和服務的對應問題，基本上是一個ingress的對象，通過yaml進行創建和更新進行加載。
• Ingress Controller是將Ingress這種變化生成一段Nginx的配置，然后將這個配置通過Kubernetes API寫到Nginx的Pod中，然后reload.

 

具體實現如下:

1.生成一個默認的后端，如果遇到解析不到的URL就轉發到默認后端頁面

[root@k8s-master ingress]# cat default-backend.yaml 
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: default-http-backend
  labels:
    k8s-app: default-http-backend
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        k8s-app: default-http-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: default-http-backend
        # Any image is permissable as long as:
        # 1. It serves a 404 page at /
        # 2. It serves 200 on a /healthz endpoint
        image: gcr.io/google_containers/defaultbackend:1.0
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi
---
apiVersion: v1
kind: Service
metadata:
  name: default-http-backend
  namespace: kube-system
  labels:
    k8s-app: default-http-backend
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    k8s-app: default-http-backend

 

2.部署Ingress Controller

具體文件可以參考官方的

https://github.com/kubernetes/ingress/blob/master/examples/daemonset/nginx/nginx-ingress-daemonset.yaml

這里貼一個我的

[root@k8s-master ingress]# cat nginx-ingress-controller.yaml 
apiVersion: v1
kind: ReplicationController
metadata:
  name: nginx-ingress-lb
  labels:
    name: nginx-ingress-lb
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        name: nginx-ingress-lb
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      terminationGracePeriodSeconds: 60
      hostNetwork: true
      containers:
      - image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.7
        name: nginx-ingress-lb
        readinessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          timeoutSeconds: 1
        ports:
        - containerPort: 80
          hostPort: 80
        - containerPort: 443
          hostPort: 443
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: KUBERNETES_MASTER
            value: http://192.168.0.105:8080
        args:
        - /nginx-ingress-controller
        - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
        - --apiserver-host=http://192.168.0.105:8080

曾經出現的問題是，啟動后pod總是在CrashLoopBack的狀態，通過logs一看發現nginx-ingress-controller的啟動總是去連接apiserver內部集群ip的443端口，導致因為安全問題不讓啟動，后來在args里面加入

- --apiserver-host=http://192.168.0.105:8080

后成功啟動.

 

3.配置ingress

 配置如下

[root@k8s-master ingress]# cat dashboard-weblogic.yaml 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: dashboard-weblogic-ingress
  namespace: kube-system
spec:
  rules:
  - host: helloworld.eric
    http:
      paths:
      - path: /console
        backend:
          serviceName: helloworldsvc 
          servicePort: 7001
      - path: /
        backend:
          serviceName: kubernetes-dashboard
          servicePort: 80

理解如下:

• host指虛擬出來的域名，具體地址(我理解應該是Ingress-controller那台Pod所在的主機的地址)應該加入/etc/hosts中,這樣所有去helloworld.eric的請求都會發到nginx
• path:/console匹配后面的應用路徑
• servicePort主要是定義服務的時候的端口，不是NodePort.
• path:/ 匹配后面dashboard應用的路徑,以前通過訪問master節點8080/ui進入dashboard的，但dashboard其實是部署在minion節點中，實際是通過某個路由語句轉發過去而已，dashboard真實路徑如下:

而yaml文件是

[root@k8s-master ~]# cat kubernetes-dashboard.yaml 
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
# Keep the name in sync with image version and
# gce/coreos/kube-manifests/addons/dashboard counterparts
  name: kubernetes-dashboard-latest
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
        version: latest
        kubernetes.io/cluster-service: "true"
    spec:
      containers:
      - name: kubernetes-dashboard
        image: gcr.io/google_containers/kubernetes-dashboard-amd64:v1.5.1
        resources:
          # keep request = limit to keep this container in guaranteed class
          limits:
            cpu: 100m
            memory: 50Mi
          requests:
            cpu: 100m
            memory: 50Mi
        ports:
        - containerPort: 9090
        args:
         -  --apiserver-host=http://192.168.0.105:8080
        livenessProbe:
          httpGet:
            path: /
            port: 9090
          initialDelaySeconds: 30
          timeoutSeconds: 30
---
kind: Service
metadata:
  name: kubernetes-dashboard
  namespace: kube-system
  labels:
    k8s-app: kubernetes-dashboard
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    k8s-app: kubernetes-dashboard
  ports:
  - port: 80
    targetPort: 9090

所以訪問192.168.51.5:9090端口就會出現dashboard

 

4.測試 

Ok,一切就緒，裝逼開始

訪問http://helloworld.eric/console 

 

訪問http://helloword.eric/    出現dashboard

 

5.配置TLS SSL訪問

TLS的配置相當於WebLogic中證書的配置，配置過程如下

• 證書生成

# 生成 CA 自簽證書
mkdir cert && cd cert
openssl genrsa -out ca-key.pem 2048
openssl req -x509 -new -nodes -key ca-key.pem -days 10000 -out ca.pem -subj "/CN=kube-ca"

# 編輯 openssl 配置
cp /etc/pki/tls/openssl.cnf .
vim openssl.cnf

# 主要修改如下
[req]
req_extensions = v3_req # 這行默認注釋關着的 把注釋刪掉
# 下面配置是新增的
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = helloworld.eric
#DNS.2 = kibana.mritd.me

# 生成證書
openssl genrsa -out ingress-key.pem 2048
openssl req -new -key ingress-key.pem -out ingress.csr -subj "/CN=helloworld.eric" -config openssl.cnf
openssl x509 -req -in ingress.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out ingress.pem -days 365 -extensions v3_req -extfile openssl.cnf

 

需要注意的是DNS需要修改成自己的host名,然后在配置csr證書請求的時候需要將域名或者訪問名帶入subj,比如

-subj "/CN=helloworld.eric" 

• 創建secret

kubectl create secret tls ingress-secret --namespace=kube-system --key cert/ingress-key.pem --cert cert/ingress.pem 

• 修改Ingress文件啟用證書

[root@k8s-master ingress]# cat tls-weblogic.yaml 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: dashboard-weblogic-ingress
  namespace: kube-system
spec:
  tls:
  - hosts:
    - helloworld.eric
    secretName: ingress-secret
  rules:
  - host: helloworld.eric
    http:
      paths:
      - path: /console
        backend:
          serviceName: helloworldsvc 
          servicePort: 7001
      - path: /
        backend:
          serviceName: kubernetes-dashboard
          servicePort: 80

•  測試

 然后訪問helloworld.eric/console,會自動轉到https頁面，同時查看證書並加入授信列表，可見

 

訪問helloworld.eric