k8s的服務訪問ingress & ingress controller
https://www.cnblogs.com/xzkzzz/p/9577640.html
https://blog.csdn.net/cloudUncle/article/details/82940598
https://blog.51cto.com/wzlinux/2332908
1 Ingress簡介
Service雖然解決了服務發現和負載均衡的問題,但它在使用上還是有一些限制:
- 只支持4層負載均衡,沒有7層功能
- 對外訪問時,NodePort類型需要在外部搭建額外的負載均衡,而LoadBalancer要求kubernetes必須跑在支持的cloud provider上
Ingress 就是一個負載均衡的應用,它和 Service 的不同之處在於,Service 只可以支持 4 層的負載均衡,而 Ingress 是支持 7 層的負載均衡,支持 http 和 https,包括通過主機名的訪問已經路徑訪問的過濾。
那為什么不直接使用 Nginx?這是因為在 K8S 集群中,如果每加入一個服務,我們都在 Nginx 中添加一個配置,其實是一個重復性的體力活,只要是重復性的體力活,我們都應該通過技術將它干掉。
Ingress就可以解決上面的問題,其包含兩個組件Ingress Controller和Ingress:
- Ingress:將Nginx的配置抽象成一個Ingress對象,每添加一個新的服務只需寫一個新的Ingress的yaml文件即可;
- Ingress Controller:將新加入的 Ingress 轉化成 Nginx 的配置文件並使之生效,包含 Contour、F5、HAProxy、Istio、Kong、Nginx、Traefik,官方推薦我們使用 Nginx。
首先我們有一個外部的負載均衡器externalLB把請求調度到一個nodePort類型的Service(ingress-nginx)上,然后nodePort類型的Service(ingress-nginx)又把它調度到內部的叫做ingressController的Pod上,ingressCtroller根據ingress中的定義(虛擬主機還是URL),每一組主機名或者URL都對應后端的Pod資源,並且用Service分組(圖中的<service>site1/<service>site2只是用來做分組用的)
Ingress是授權入站連接到達集群服務的規則集合。
- 從外部流量調度到nodeprot上的service
- 從service調度到ingress-controller
- ingress-controller根據ingress中的定義(虛擬主機或者后端的url)
- 根據虛擬主機名調度到后端的一組pod中
2 Ingress Nginx部署
使用Ingress功能步驟:
- 安裝部署ingressCtroller Pod
- 部署ingress-nginx service
- 部署后端服務
- 部署ingress
從前面的描述我們知道,Ingress可以使用yaml的方式進行創建,從而得知Ingress也是標准的K8S資源,其定義的方式也可以使用kubectl explain ingress進行查看:
2.1 部署ingress controller
https://www.cnblogs.com/dingbin/p/9754993.html
在kubernetes 得github上下載ingress得yaml文件,地址:https://github.com/kubernetes/ingress-nginx/tree/master/deploy
ingress得所需得文件:configmap.yaml ,namespace.yaml,rbac.yaml,tcp-services-configmap.yaml,with-rbac.yaml
同時在ingress得官方文檔中介紹到,需要下載service-nodeport.yaml文件,這個文件得目的是為Ingress Controller 接入外部得流量,如果沒有這個文件,是無法通過
外部訪問得。這個文件其實就是為Ingress Controller 創建一個NodePort 類型得Service,這里我稍微修改了下service-nodeport.yaml,如下:
2.1.1 下載ingress相關yaml文件
git clone https://github.com/kubernetes/ingress-nginx cd ingress-nginx/ cd deploy/static/ ls
[root@k8s-master static]# ls configmap.yaml mandatory.yaml namespace.yaml provider rbac.yaml with-rbac.yaml
2.1.2 創建ingress-nginx名稱空間
[root@k8s-master ingress-nginx]# cat namespace.yaml apiVersion: v1 kind: Namespace metadata: name: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- [root@k8s-master ingress-nginx]# kubectl create -f namespace.yaml namespace/ingress-nginx created [root@k8s-master ingress-nginx]# kubectl get namespaces ingress-nginx NAME STATUS AGE ingress-nginx Active 12s
2.1.3 創建ingress controller的Pod
[root@k8s-master ingress-nginx]# ls configmap.yaml mandatory.yaml namespace.yaml rbac.yaml with-rbac.yaml [root@k8s-master ingress-nginx]# [root@k8s-master ingress-nginx]# kubectl apply -f . configmap/nginx-configuration created configmap/tcp-services created configmap/udp-services created namespace/ingress-nginx unchanged configmap/nginx-configuration unchanged configmap/tcp-services unchanged configmap/udp-services unchanged serviceaccount/nginx-ingress-serviceaccount created clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole created role.rbac.authorization.k8s.io/nginx-ingress-role created rolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding created clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding created deployment.apps/nginx-ingress-controller created namespace/ingress-nginx unchanged serviceaccount/nginx-ingress-serviceaccount unchanged clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole unchanged role.rbac.authorization.k8s.io/nginx-ingress-role unchanged rolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding unchanged clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding unchanged deployment.apps/nginx-ingress-controller unchanged
此處遇到一個問題,新版本的Kubernetes在安裝部署中,需要從k8s.grc.io倉庫拉去所需鏡像文件,但是由於國內防火牆問題導致無法正常拉去。
辛虧docker.io倉庫對google的容器做了mirror,可以通過下列命令拉取相關鏡像:
node節點都要拉取這個鏡像,或者master把鏡像傳輸到node
docker pull mirrorgooglecontainers/defaultbackend-amd64:1.5 docker tag mirrorgooglecontainers/defaultbackend-amd64:1.5 k8s.gcr.io/defaultbackend-amd64:1.5 docker image ls|grep defaultbackend-amd64
2.2 部署ingress-nginx service
通過ingress-controller對外提供服務,現在還需要手動給ingress-controller建立一個servcie,接收集群外部流量
[root@k8s-master ingress-nginx]# ls baremetal configmap.yaml mandatory.yaml namespace.yaml rbac.yaml with-rbac.yaml [root@k8s-master ingress-nginx]# cd baremetal/ [root@k8s-master baremetal]# cat service-nodeport.yaml apiVersion: v1 kind: Service metadata: name: ingress-nginx namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx spec: type: NodePort ports: - name: http port: 80 targetPort: 80 protocol: TCP - name: https port: 443 targetPort: 443 protocol: TCP selector: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- [root@k8s-master baremetal]# kubectl apply -f service-nodeport.yaml service/ingress-nginx created [root@k8s-master baremetal]# kubectl get svc -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx NodePort 10.104.90.61 <none> 80:32078/TCP,443:32229/TCP 6s [root@k8s-master baremetal]#
2.3 部署ingress
2.3.1 編寫ingress的配置清單並創建ingress
#cat ingress-default-backend.yaml apiVersion: extensions/v1beta1 #api版本 kind: Ingress #清單類型 metadata: #元數據 name: ingress-default-backend #ingress名稱 namespace: ingress-nginx #所屬命名空間 annotations: #注解信息 kubernetes.io/ingress.class: "nginx" spec: #規格 rules: #定義后端轉發規則 - host: default.backend.com #通過域名進行轉發 http: paths: - path: #配置訪問路徑 backend: #配置后端服務 serviceName: default-http-backend servicePort: 80
[root@k8s-master ingress]# kubectl apply -f ingress-default-backend.yaml
ingress.extensions/ingress-default-backend created
查看ingress-default-backend的詳細信息:
[root@k8s-master ingress]# kubectl describe ingress -n ingress-nginx ingress-default-backend Name: ingress-default-backend Namespace: ingress-nginx Address: Default backend: default-http-backend:80 (<none>) Rules: Host Path Backends ---- ---- -------- default.backend.com default-http-backend:80 (<none>) Annotations: kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx"},"name":"ingress-default-backend","namespace":"ingress-nginx"},"spec":{"rules":[{"host":"default.backend.com","http":{"paths":[{"backend":{"serviceName":"default-http-backend","servicePort":80},"path":null}]}}]}} kubernetes.io/ingress.class: nginx Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal CREATE 7m18s nginx-ingress-controller Ingress ingress-nginx/ingress-default-backend [root@k8s-master ingress]#
進入nginx-ingress-controller進行查看是否注入了nginx配置
[root@k8s-master ingress]# kubectl exec -n ingress-nginx -ti nginx-ingress-controller-79f6884cf6-v5cdj /bin/bash www-data@nginx-ingress-controller-79f6884cf6-v5cdj:/etc/nginx$ ls fastcgi_params geoip lua mime.types modsecurity modules nginx.conf opentracing.json owasp-modsecurity-crs template www-data@nginx-ingress-controller-79f6884cf6-v5cdj:/etc/nginx$cat nginx.conf …………… ## start server default.backend.com server { server_name default.backend.com ; listen 80 ; listen 443 ssl http2 ; set $proxy_upstream_name "-"; location / { set $namespace "ingress-nginx"; set $ingress_name "ingress-default-backend"; set $service_name "default-http-backend"; set $service_port "{0 80 }"; set $location_path "/"; …………… }
部署的ingress的的確確已經注入到nginx-ingress-controller的nginx配置中了
2.3.2 通過ingress訪問服務
修改本地host文件,我的是內網地址
/etc/hosts #或者windows hosts文件 10.6.76.23 default.backend.com 10.6.76.24 default.backend.com
[root@k8s-master ~]# curl 10.6.76.23:32078/ <html> <head><title>404 Not Found</title></head> <body> <center><h1>404 Not Found</h1></center> <hr><center>openresty/1.15.8.1</center> </body> </html> [root@k8s-master ~]# curl default.backend.com:32078 <html> <head><title>503 Service Temporarily Unavailable</title></head> <body> <center><h1>503 Service Temporarily Unavailable</h1></center> <hr><center>openresty/1.15.8.1</center> </body> </html> [root@k8s-master ~]#
3 創建后端服務-TLS-tomcat
3.1 添加tomcat服務
# cat tomcat-demo.yaml apiVersion: v1 kind: Service metadata: name: tomcat namespace: default spec: selector: app: tomcat release: stable ports: - name: http port: 8080 targetPort: 8080 - name: ajp port: 8009 targetPort: 8009 --- apiVersion: apps/v1 kind: Deployment metadata: name: tomcat-demo namespace: default spec: replicas: 3 selector: matchLabels: app: tomcat release: stable template: metadata: labels: app: tomcat release: stable spec: containers: - name: tomcat image: tomcat:latest ports: - name: http containerPort: 8080 - name: ajp containerPort: 8009
[root@k8s-master ingress]# kubectl create -f tomcat-demo.yaml service/tomcat created deployment.apps/tomcat-demo created [root@k8s-master ingress]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 4d5h tomcat ClusterIP 10.98.93.179 <none> 8080/TCP,8009/TCP 6m24s [root@k8s-master ingress]# kubectl get deployments NAME READY UP-TO-DATE AVAILABLE AGE tomcat-demo 3/3 3 3 6m28s [root@k8s-master ingress]# kubectl get pods NAME READY STATUS RESTARTS AGE tomcat-demo-6b78b44bf7-8hhpn 1/1 Running 0 3m1s tomcat-demo-6b78b44bf7-fgwlk 1/1 Running 0 90s tomcat-demo-6b78b44bf7-wlmc7 1/1 Running 0 22s [root@k8s-master ingress]#
3.2 准備證書並生成secret
[root@master ingress-nginx]# openssl genrsa -out tls.key 2048 Generating RSA private key, 2048 bit long modulus ............................+++ ..................+++ e is 65537 (0x10001) [root@master ingress-nginx]# openssl req -new -x509 -key tls.key -out tls.crt -subj /C=CN/ST=Beijing/L=Beijing/O=DevOps/CN=tomcat.wangxu.com
生成secret
[root@k8s-master ingress]# ls ingress-default-backend.yaml tls tls.key wangxu-deploy.yaml wangxu-svc.yaml ingress-nginx tls.crt tomcat-demo.yaml wangxu-ingress.yaml [root@k8s-master ingress]# kubectl create secret tls tomcat-ingress-secret --cert=tls.crt --key=tls.key secret/tomcat-ingress-secret created [root@k8s-master ingress]# kubectl get secret NAME TYPE DATA AGE default-token-wldf4 kubernetes.io/service-account-token 3 4d5h tomcat-ingress-secret kubernetes.io/tls 2 7s [root@k8s-master ingress]# kubectl describe secret tomcat-ingress-secret Name: tomcat-ingress-secret Namespace: default Labels: <none> Annotations: <none> Type: kubernetes.io/tls Data ==== tls.crt: 1294 bytes tls.key: 1675 bytes
3.3 創建ingress
# cat ingress-tomcat-tls.yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: ingress-tomcat-tls namespace: default annotations: kubernetes.io/ingress.class: "nginx" spec: tls: - hosts: - tomcat.wangxu.com secretName: tomcat-ingress-secret rules: - host: tomcat.zwx.com http: paths: - backend: serviceName: tomcat servicePort: 8080
[root@k8s-master ingress]# kubectl apply -f ingress-tomcat-tls.yaml ingress.extensions/ingress-tomcat-tls created [root@k8s-master ingress]#
3.4 測試Nginx服務
[root@k8s-master ingress]# kubectl get svc -n ingress-nginx -o wide NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR ingress-nginx NodePort 10.104.90.61 <none> 80:32078/TCP,443:32229/TCP 4h11m app.kubernetes.io/name=ingress-nginx,app.kubernetes.io/part-of=ingress-nginx [root@k8s-master ingress]#
本地host解析了
4 myAPP樣例
4.1 創建后端服務myAPP svc和deployment
# cat myapp-deploy.yaml apiVersion: apps/v1 kind: Deployment metadata: name: myapp-wangxu namespace: default spec: replicas: 3 selector: matchLabels: app: myapp release: wangxu template: metadata: labels: app: myapp release: wangxu spec: containers: - name: myapp-wangxu image: ikubernetes/myapp:v2 ports: - name: http containerPort: 80
# cat myapp-svc.yaml apiVersion: v1 kind: Service metadata: name: myapp-wangxu namespace: default spec: selector: app: myapp release: wangxu ports: - name: http port: 80 targetPort: 80
[root@k8s-master ~]# kubectl get svc,pod| grep myapp service/myapp-wangxu ClusterIP 10.111.225.232 <none> 80/TCP 41s pod/myapp-wangxu-79b896cc97-6fdjq 1/1 Running 0 45s pod/myapp-wangxu-79b896cc97-js6rj 1/1 Running 0 45s pod/myapp-wangxu-79b896cc97-msjtt 1/1 Running 0 45s [root@k8s-master ~]#
4.2 部署Ingress Controller如果部署過,跳過
前面部署了就不用了
在kubernetes 得github上下載ingress得yaml文件,地址:https://github.com/kubernetes/ingress-nginx/tree/master/deploy
ingress得所需得文件:configmap.yaml ,namespace.yaml,rbac.yaml,tcp-services-configmap.yaml,with-rbac.yaml
同時在ingress得官方文檔中介紹到,需要下載service-nodeport.yaml文件,這個文件得目的是為Ingress Controller 接入外部得流量,如果沒有這個文件,是無法通過
簡單的說就是執行這6個文件
[root@k8s-master ingress-nginx]# tree . ├── baremetal │ └── service-nodeport.yaml ├── configmap.yaml ├── mandatory.yaml ├── namespace.yaml ├── rbac.yaml └── with-rbac.yaml 1 directory, 6 files [root@k8s-master ingress-nginx]# [root@k8s-master ingress-nginx]# kubectl get svc,pods -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/ingress-nginx NodePort 10.104.90.61 <none> 80:32078/TCP,443:32229/TCP 4h34m NAME READY STATUS RESTARTS AGE pod/nginx-ingress-controller-79f6884cf6-v5cdj 1/1 Running 0 4h57m [root@k8s-master ingress-nginx]#
4.3 創建myapp-ingress服務
# cat ingress-myapp.yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: ingress-myapp namespace: default annotations: kubernetes.io/ingress.class: "nginx" # 這里是說明ingress的類型使用的nginx,一定要說明這點,否則ingress Controller 不知道是配置成那種類型的配置文件 spec: rules: - host: myapp.wangxu.com # 使用的是虛擬主機的來訪問的 http: paths: - path: backend: serviceName: myapp-wangxu # 代理的后端的pod的service,通過這個service來生成nginx的upstrm servicePort: 80
[root@k8s-master ~]# kubectl apply -f ingress-myapp.yaml ingress.extensions/ingress-myapp configured [root@k8s-master ~]# kubectl get ingresses|grep myapp ingress-myapp myapp.wangxu.com 80 12m [root@k8s-master ~]#
4.4 訪問的客戶端的機器配置下域名解析
10.6.76.23 myapp.wangxu.com 10.6.76.24 myapp.wangxu.com
4.5 配置https自簽名證書
[root@k8s-master ~]# openssl genrsa -out tls.key 2048 Generating RSA private key, 2048 bit long modulus .........................................+++ ...............................................................................................+++ e is 65537 (0x10001) [root@k8s-master ~]# openssl req -new -x509 -key tls.key -out tls.crt -subj /C=CN/ST=Beijing/L=Beijing/O=DevOps/CN=myapp.wangxu.com [root@k8s-master ~]# #證書生成好了,然后把證書轉成secret, [root@k8s-master ~]# kubectl create secret tls myapp-ingress-secret --cert=tls.crt --key=tls.key secret/myapp-ingress-secret created [root@k8s-master ~]# kubectl get secrets NAME TYPE DATA AGE default-token-wldf4 kubernetes.io/service-account-token 3 4d6h myapp-ingress-secret kubernetes.io/tls 2 18s
4.6 修改myapp-ingress服務,加入證書
修改下 ingress-myapp.yaml 加入剛剛添加的secret,修改后的文件如下:
# cat ingress-myapp.yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: ingress-myapp namespace: default annotations: kubernetes.io/ingress.class: "nginx" spec: tls: # 添加了tls這一段 - hosts: - myapp.wangxu.com secretName: myapp-ingress-secret # 這里結束 rules: - host: myapp.wangxu.com http: paths: - path: backend: serviceName: myapp-wangxu servicePort: 80
[root@k8s-master ~]# kubectl apply -f ingress-myapp.yaml ingress.extensions/ingress-myapp configured [root@k8s-master ~]#
4.7 HTTPS訪問
[root@k8s-master ~]# kubectl get svc -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx NodePort 10.104.90.61 <none> 80:32078/TCP,443:32229/TCP 4h44m [root@k8s-master ~]#
5 Nginx服務樣例
5.1 部署Ingress Controller
參考上面的
5.2 創建后端服務Nginx svc和deployment
# cat nginx-dep.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-test namespace: default spec: replicas: 3 selector: matchLabels: app: nginx release: test template: metadata: labels: app: nginx release: test spec: containers: - name: nginx-test image: nginx:latest ports: - name: http containerPort: 80
# cat nginx-svc.yaml apiVersion: v1 kind: Service metadata: name: nginx-test namespace: default spec: selector: app: nginx release: test ports: - name: http port: 80 targetPort: 80
[root@k8s-master tls-nginx]# kubectl create -f nginx-dep.yaml deployment.apps/nginx-test created [root@k8s-master tls-nginx]# kubectl create -f nginx-svc.yaml service/nginx-test created [root@k8s-master tls-nginx]# kubectl get pods,svc -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod/nginx-test-5ff8cb55bf-6dtm9 1/1 Running 0 20s 10.254.2.50 k8s-node-2 <none> <none> pod/nginx-test-5ff8cb55bf-gxngh 1/1 Running 0 20s 10.254.1.58 k8s-node-1 <none> <none> pod/nginx-test-5ff8cb55bf-hmr8t 1/1 Running 0 20s 10.254.1.57 k8s-node-1 <none> <none> NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 4d6h <none> service/nginx-test ClusterIP 10.97.235.162 <none> 80/TCP 17s app=nginx,release=test [root@k8s-master tls-nginx]#
5.3 配置https自簽名證書
openssl genrsa -out tls.key 2048 openssl req -new -x509 -key tls.key -out tls.crt -subj /C=CN/ST=Beijing/L=Beijing/O=DevOps/CN=nginx.test.com kubectl create secret tls nginx-ingress-secret --cert=tls.crt --key=tls.key
[root@k8s-master tls-nginx]# kubectl get secrets| grep nginx nginx-ingress-secret kubernetes.io/tls 2 86s [root@k8s-master tls-nginx]#
5.4 創建nginx-ingress服務
# cat nginx-tls.yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: ingress-nginx namespace: default annotations: kubernetes.io/ingress.class: "nginx" spec: tls: # 添加了tls這一段 - hosts: - nginx.test.com secretName: nginx-ingress-secret # 這里結束 rules: - host: nginx.test.com http: paths: - path: backend: serviceName: nginx-test servicePort: 80
[root@k8s-master tls-nginx]# kubectl create -f nginx-tls.yaml ingress.extensions/ingress-myapp created [root@k8s-master tls-nginx]#
5.5 域名自解析和HTTPS訪問
10.6.76.23 nginx.test.com
10.6.76.24 nginx.test.com
[root@k8s-master tls-nginx]# kubectl get ingresses -o wide NAME HOSTS ADDRESS PORTS AGE ingress-myapp nginx.test.com 80, 443 2m26s [root@k8s-master tls-nginx]# kubectl get svc -o wide -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR ingress-nginx NodePort 10.104.90.61 <none> 80:32078/TCP,443:32229/TCP 5h13m app.kubernetes.io/name=ingress-nginx,app.kubernetes.io/part-of=ingress-nginx [root@k8s-master tls-nginx]#