二進制部署K8S集群（二十二）addons之安裝部署ingress

目錄

• 1.說明
• 2.業務架構圖
• 3.Ingress訪問流程圖
• 4.架構
• 5.部署traefik
  • 5.1 准備traefik鏡像
  • 5.2 准備traefik資源配置清單目錄
  • 5.3 准備rbac.yaml文件
  • 5.4 准備daemonset.yaml文件
  • 5.5 安裝ingress
• 6.創建nginx資源清單目錄
• 7.創建ingress.yml
• 8.創建svc.yml
• 9.創建deploy.yml
• 10.添加dns解析
• 11.配置7層負載
• 12.應用資源配置清單
• 13.修改html
• 14.WEB訪問

1.說明

對於Kubernetes的service，無論是cluster-ip和nodeport均是四層的負載，集群內的服務如何實現七層的負載均衡，這就需要借助於ingress，ingress控制器實現的方式有很多，比如nginx,contour,haproxy,trafik,lstio。幾種常用的ingress功能對比和選型可以參考這里www.kubernetes.org.cn/5948.html

ingress-nginx是七層的負載均衡器，負責統一管理外部對k8s cluster中的service的請求。主要包含：

• ingress-nginx-controller：要據用戶編寫的ingress規則（創建的Ingress的yaml文件），動態的去更改服務的配置文件，並且reload重載使其生效（是自動化的，通過Lua腳本來實現）；
• ingress資源對象：將Nginx的配置抽像成一個Ingress對象
• ingress是K8S的標准資源類型之一，也是一種核心資源，它其實就是一種基於域名和URL路徑，把用戶的請求轉發至指定Service資源的規則。可以將集群外部的請求流量，轉發至集群內部，從而實現“服務暴露”
• ingress控制器是能夠為Ingress資源監聽某套接字，然后根擾Ingress規則匹配機制路由調度流量的一個組件。
  參考鏈接：https://github.com/nginxinc/kubernetes-ingress

總結用ingress好處：

• 同台服務器不同業務不需要再給每個業務映射端口(Nodeport)，只需要每台機安裝一個ingress，利用ingress反代CluserIP，前端機訪問Ingress固定端口
• 添加新業務只需要再創建一個ingress反代新業務的service，再去前端Nginx反代配置servername里面添加一個域名即可以訪問新業務，通過不同的域名訪問不同的業務，不需要再配反代

2.業務架構圖

3.Ingress訪問流程圖

4.架構

主機	角色	IP	節點
hdss7-21.host.com	ingress，nginx	10.4.7.21	node
hdss7-22.host.com	ingress，nginx	10.4.7.22	node
hdss7-11.host.com	dns，nginx七層反代	10.4.7.11	負載均衡機（proxy），dns服務器
hdss7-12.host.com	nginx七層反代	10.4.7.12	負載均衡機（proxy）
hdss7-200.host.com	資源配置清單	10.4.7.200	運維主機

5.部署traefik

5.1 准備traefik鏡像

hdss7-200機主機上操作：

docker pull traefik:v1.7-alpine
docker tag c36f69007d98 harbor.od.com/k8s/traefik:v1.7
docker push harbor.od.com/k8s/traefik:v1.7

5.2 准備traefik資源配置清單目錄

清單下載地址：https://github.com/traefik/traefik/tree/v1.7/examples/k8s

mkdir -p /data/k8s-yaml/traefik && cd /data/k8s-yaml/traefik

5.3 准備rbac.yaml文件

cat > /data/k8s-yaml/traefik/rbac.yaml <<'eof'
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: traefik-ingress-controller
rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
  name: traefik-ingress-controller
  namespace: kube-system
eof

5.4 准備daemonset.yaml文件

cat > /data/k8s-yaml/traefik/daemonset.yaml <<'eof'
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
  labels:
    k8s-app: traefik-ingress-lb
spec:
  selector:
    matchLabels:
      k8s-app: traefik-ingress-lb
  template:
    metadata:
      labels:
        k8s-app: traefik-ingress-lb
        name: traefik-ingress-lb
    spec:
      serviceAccountName: traefik-ingress-controller
      terminationGracePeriodSeconds: 60
      containers:
      - image: harbor.od.com/k8s/traefik:v1.7
        name: traefik-ingress-lb
        ports:
        - name: http
          containerPort: 80
          hostPort: 81
			  - name: web-admin
          containerPort: 8080
          hostPort: 8081
        securityContext:
          capabilities:
            drop:
            - ALL
            add:
            - NET_BIND_SERVICE
        args:
        - --api
        - --kubernetes
        - --logLevel=INFO
        - --insecureskipverify=true
        - --kubernetes.endpoint=https://10.4.7.10:7443
        - --accesslog
        - --accesslog.filepath=/var/log/traefik_access.log
        - --traefiklog
        - --traefiklog.filepath=/var/log/traefik.log
        - --metrics.prometheus
      imagePullSecrets:
      - name: harbor
eof

hostPort: 81 為ingress的程序80端口映射到宿主機供提供訪問的端口

5.5 安裝ingress

kubectl apply -f http://k8s-yaml.od.com/traefik/rbac.yaml 
kubectl apply -f http://k8s-yaml.od.com/traefik/daemonset.yaml 

6.創建nginx資源清單目錄

mkdir /data/k8s-yaml/nginxtest

7.創建ingress.yml

cat > /data/k8s-yaml/nginxtest/ingress.yml <<'eof'
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-web
  namespace: default
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  - host: nginxtest.od.com
    http:
      paths:
        - path: /
          backend:
            serviceName: nginx-test
            servicePort: 80
eof

主機名為nginxtest.od.com，反代到svc的name為nginx-test，路徑為/，端口80

8.創建svc.yml

cat > /data/k8s-yaml/nginxtest/svc.yml <<'eof'
apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: nginx-test
  name: nginx-test
  namespace: default
spec:
  ports:
    - port: 80
      protocol: TCP
  selector:
    app: nginx-test
  sessionAffinity: None
eof

svc標簽選擇器app: nginx-test，反代pod為app：nginx-test

9.創建deploy.yml

cat > /data/k8s-yaml/nginxtest/deploy.yml <<'eof'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-test
  labels:
    app: nginx-test
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx-test
  template:
    metadata:
      labels:
        app: nginx-test
    spec:
      containers:
      - name: nginx-test
        image: harbor.od.com/public/nginx:v1.7.9
        ports:
        - name: web
          containerPort: 80

10.添加dns解析

hdss7-11.host.com上操作

cat >> /var/named/od.com.zone <<'eof'
nginxtest          A    10.4.7.10
eof
vi /var/named/od.com.zone
2020100504  ; serial # 日期加1
systemctl restart named

11.配置7層負載

在hdss7-11.host.com和hdss7-12.host.com上操作

cat >/etc/nginx/conf.d/nginxtest.com.conf <<'eof'
upstream default_backend_traefik {
    server 10.4.7.21:81    max_fails=3 fail_timeout=10s;
    server 10.4.7.22:81    max_fails=3 fail_timeout=10s;
}
server {
    server_name nginxtest.od.com;
  
    location / {
        proxy_pass http://default_backend_traefik;
        proxy_set_header Host       $http_host;
        proxy_set_header x-forwarded-for $proxy_add_x_forwarded_for;
    }
}
eof
nginx -s reload

12.應用資源配置清單

以下都在hdss7-21.host.com或hdss7-22上操作：

[root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/nginxtest/deploy.yml
[root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/nginxtest/svc.yml
[root@hdss7-22 ~]# kubectl apply -f http://k8s-yaml.od.com/nginxtest/ingress.yml
[root@hdss7-22 ~]# kubectl get ing
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
NAME        CLASS    HOSTS              ADDRESS   PORTS   AGE
nginx-web   <none>   nginxtest.od.com             80      18h
[root@hdss7-22 ~]# kubectl get pods -n kube-system -o wide           
NAME                               READY   STATUS    RESTARTS   AGE   IP           NODE                NOMINATED NODE   READINESS GATES
coredns-57c78bdbcd-lsf5z           1/1     Running   4          30h   172.7.21.3   hdss7-21.host.com   <none>           <none>
traefik-ingress-controller-9n8zb   1/1     Running   0          11h   172.7.21.5   hdss7-21.host.com   <none>           <none>
traefik-ingress-controller-wxnqw   1/1     Running   0          11h   172.7.22.4   hdss7-22.host.com   <none>           <none>

13.修改html

[root@hdss7-22 ~]# kubectl get pods -o wide                          
NAME                          READY   STATUS    RESTARTS   AGE   IP           NODE                NOMINATED NODE   READINESS GATES
nginx-test-558df79dc9-d95rp   1/1     Running   0          9h    172.7.21.2   hdss7-21.host.com   <none>           <none>
nginx-test-558df79dc9-qw2fj   1/1     Running   0          9h    172.7.22.2   hdss7-22.host.com   <none>           <none>
[root@hdss7-22 ~]# kubectl exec -it nginx-test-558df79dc9-d95rp -- /bin/bash
root@nginx-test-558df79dc9-d95rp:/# echo WEB1 > /usr/share/nginx/html/index.html
root@nginx-test-558df79dc9-d95rp:/# exit
exit
[root@hdss7-22 ~]# kubectl exec -it nginx-test-558df79dc9-qw2fj -- /bin/bash     
root@nginx-test-558df79dc9-qw2fj:/# echo WEB2 > /usr/share/nginx/html/index.html

14.WEB訪問