k8s的服务访问ingress & ingress controller
https://www.cnblogs.com/xzkzzz/p/9577640.html
https://blog.csdn.net/cloudUncle/article/details/82940598
https://blog.51cto.com/wzlinux/2332908
1 Ingress简介
Service虽然解决了服务发现和负载均衡的问题，但它在使用上还是有一些限制：
- 只支持4层负载均衡，没有7层功能
- 对外访问时，NodePort类型需要在外部搭建额外的负载均衡，而LoadBalancer要求kubernetes必须跑在支持的cloud provider上
Ingress 就是一个负载均衡的应用，它和 Service 的不同之处在于，Service 只可以支持 4 层的负载均衡，而 Ingress 是支持 7 层的负载均衡，支持 http 和 https，包括通过主机名的访问已经路径访问的过滤。
那为什么不直接使用 Nginx？这是因为在 K8S 集群中，如果每加入一个服务，我们都在 Nginx 中添加一个配置，其实是一个重复性的体力活，只要是重复性的体力活，我们都应该通过技术将它干掉。
Ingress就可以解决上面的问题，其包含两个组件Ingress Controller和Ingress：
- Ingress：将Nginx的配置抽象成一个Ingress对象，每添加一个新的服务只需写一个新的Ingress的yaml文件即可；
- Ingress Controller：将新加入的 Ingress 转化成 Nginx 的配置文件并使之生效，包含 Contour、F5、HAProxy、Istio、Kong、Nginx、Traefik，官方推荐我们使用 Nginx。
首先我们有一个外部的负载均衡器externalLB把请求调度到一个nodePort类型的Service(ingress-nginx)上，然后nodePort类型的Service(ingress-nginx)又把它调度到内部的叫做ingressController的Pod上，ingressCtroller根据ingress中的定义(虚拟主机还是URL)，每一组主机名或者URL都对应后端的Pod资源，并且用Service分组(图中的<service>site1/<service>site2只是用来做分组用的)
Ingress是授权入站连接到达集群服务的规则集合。
- 从外部流量调度到nodeprot上的service
- 从service调度到ingress-controller
- ingress-controller根据ingress中的定义（虚拟主机或者后端的url）
- 根据虚拟主机名调度到后端的一组pod中
2 Ingress Nginx部署
使用Ingress功能步骤：
- 安装部署ingressCtroller Pod
- 部署ingress-nginx service
- 部署后端服务
- 部署ingress
从前面的描述我们知道，Ingress可以使用yaml的方式进行创建，从而得知Ingress也是标准的K8S资源，其定义的方式也可以使用kubectl explain ingress进行查看：
2.1 部署ingress controller
https://www.cnblogs.com/dingbin/p/9754993.html
在kubernetes 得github上下载ingress得yaml文件，地址：https://github.com/kubernetes/ingress-nginx/tree/master/deploy
ingress得所需得文件：configmap.yaml ，namespace.yaml，rbac.yaml，tcp-services-configmap.yaml，with-rbac.yaml
同时在ingress得官方文档中介绍到，需要下载service-nodeport.yaml文件，这个文件得目的是为Ingress Controller 接入外部得流量，如果没有这个文件，是无法通过
外部访问得。这个文件其实就是为Ingress Controller 创建一个NodePort 类型得Service，这里我稍微修改了下service-nodeport.yaml，如下：
2.1.1 下载ingress相关yaml文件
git clone https://github.com/kubernetes/ingress-nginx cd ingress-nginx/ cd deploy/static/ ls
[root@k8s-master static]# ls configmap.yaml mandatory.yaml namespace.yaml provider rbac.yaml with-rbac.yaml
2.1.2 创建ingress-nginx名称空间
[root@k8s-master ingress-nginx]# cat namespace.yaml apiVersion: v1 kind: Namespace metadata: name: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- [root@k8s-master ingress-nginx]# kubectl create -f namespace.yaml namespace/ingress-nginx created [root@k8s-master ingress-nginx]# kubectl get namespaces ingress-nginx NAME STATUS AGE ingress-nginx Active 12s
2.1.3 创建ingress controller的Pod
[root@k8s-master ingress-nginx]# ls configmap.yaml mandatory.yaml namespace.yaml rbac.yaml with-rbac.yaml [root@k8s-master ingress-nginx]# [root@k8s-master ingress-nginx]# kubectl apply -f . configmap/nginx-configuration created configmap/tcp-services created configmap/udp-services created namespace/ingress-nginx unchanged configmap/nginx-configuration unchanged configmap/tcp-services unchanged configmap/udp-services unchanged serviceaccount/nginx-ingress-serviceaccount created clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole created role.rbac.authorization.k8s.io/nginx-ingress-role created rolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding created clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding created deployment.apps/nginx-ingress-controller created namespace/ingress-nginx unchanged serviceaccount/nginx-ingress-serviceaccount unchanged clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole unchanged role.rbac.authorization.k8s.io/nginx-ingress-role unchanged rolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding unchanged clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding unchanged deployment.apps/nginx-ingress-controller unchanged
此处遇到一个问题，新版本的Kubernetes在安装部署中，需要从k8s.grc.io仓库拉去所需镜像文件，但是由于国内防火墙问题导致无法正常拉去。
辛亏docker.io仓库对google的容器做了mirror，可以通过下列命令拉取相关镜像：
node节点都要拉取这个镜像，或者master把镜像传输到node
docker pull mirrorgooglecontainers/defaultbackend-amd64:1.5 docker tag mirrorgooglecontainers/defaultbackend-amd64:1.5 k8s.gcr.io/defaultbackend-amd64:1.5 docker image ls|grep defaultbackend-amd64
2.2 部署ingress-nginx service
通过ingress-controller对外提供服务，现在还需要手动给ingress-controller建立一个servcie，接收集群外部流量
[root@k8s-master ingress-nginx]# ls baremetal configmap.yaml mandatory.yaml namespace.yaml rbac.yaml with-rbac.yaml [root@k8s-master ingress-nginx]# cd baremetal/ [root@k8s-master baremetal]# cat service-nodeport.yaml apiVersion: v1 kind: Service metadata: name: ingress-nginx namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx spec: type: NodePort ports: - name: http port: 80 targetPort: 80 protocol: TCP - name: https port: 443 targetPort: 443 protocol: TCP selector: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- [root@k8s-master baremetal]# kubectl apply -f service-nodeport.yaml service/ingress-nginx created [root@k8s-master baremetal]# kubectl get svc -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx NodePort 10.104.90.61 <none> 80:32078/TCP,443:32229/TCP 6s [root@k8s-master baremetal]#
2.3 部署ingress
2.3.1 编写ingress的配置清单并创建ingress
#cat ingress-default-backend.yaml apiVersion: extensions/v1beta1 #api版本 kind: Ingress #清单类型 metadata: #元数据 name: ingress-default-backend #ingress名称 namespace: ingress-nginx #所属命名空间 annotations: #注解信息 kubernetes.io/ingress.class: "nginx" spec: #规格 rules: #定义后端转发规则 - host: default.backend.com #通过域名进行转发 http: paths: - path: #配置访问路径 backend: #配置后端服务 serviceName: default-http-backend servicePort: 80
[root@k8s-master ingress]# kubectl apply -f ingress-default-backend.yaml
ingress.extensions/ingress-default-backend created
查看ingress-default-backend的详细信息：
[root@k8s-master ingress]# kubectl describe ingress -n ingress-nginx ingress-default-backend Name: ingress-default-backend Namespace: ingress-nginx Address: Default backend: default-http-backend:80 (<none>) Rules: Host Path Backends ---- ---- -------- default.backend.com default-http-backend:80 (<none>) Annotations: kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx"},"name":"ingress-default-backend","namespace":"ingress-nginx"},"spec":{"rules":[{"host":"default.backend.com","http":{"paths":[{"backend":{"serviceName":"default-http-backend","servicePort":80},"path":null}]}}]}} kubernetes.io/ingress.class: nginx Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal CREATE 7m18s nginx-ingress-controller Ingress ingress-nginx/ingress-default-backend [root@k8s-master ingress]#
进入nginx-ingress-controller进行查看是否注入了nginx配置
[root@k8s-master ingress]# kubectl exec -n ingress-nginx -ti nginx-ingress-controller-79f6884cf6-v5cdj /bin/bash www-data@nginx-ingress-controller-79f6884cf6-v5cdj:/etc/nginx$ ls fastcgi_params geoip lua mime.types modsecurity modules nginx.conf opentracing.json owasp-modsecurity-crs template www-data@nginx-ingress-controller-79f6884cf6-v5cdj:/etc/nginx$cat nginx.conf …………… ## start server default.backend.com server { server_name default.backend.com ; listen 80 ; listen 443 ssl http2 ; set $proxy_upstream_name "-"; location / { set $namespace "ingress-nginx"; set $ingress_name "ingress-default-backend"; set $service_name "default-http-backend"; set $service_port "{0 80 }"; set $location_path "/"; …………… }
部署的ingress的的确确已经注入到nginx-ingress-controller的nginx配置中了
2.3.2 通过ingress访问服务
修改本地host文件，我的是内网地址
/etc/hosts #或者windows hosts文件 10.6.76.23 default.backend.com 10.6.76.24 default.backend.com
[root@k8s-master ~]# curl 10.6.76.23:32078/ <html> <head><title>404 Not Found</title></head> <body> <center><h1>404 Not Found</h1></center> <hr><center>openresty/1.15.8.1</center> </body> </html> [root@k8s-master ~]# curl default.backend.com:32078 <html> <head><title>503 Service Temporarily Unavailable</title></head> <body> <center><h1>503 Service Temporarily Unavailable</h1></center> <hr><center>openresty/1.15.8.1</center> </body> </html> [root@k8s-master ~]#
3 创建后端服务-TLS-tomcat
3.1 添加tomcat服务
# cat tomcat-demo.yaml apiVersion: v1 kind: Service metadata: name: tomcat namespace: default spec: selector: app: tomcat release: stable ports: - name: http port: 8080 targetPort: 8080 - name: ajp port: 8009 targetPort: 8009 --- apiVersion: apps/v1 kind: Deployment metadata: name: tomcat-demo namespace: default spec: replicas: 3 selector: matchLabels: app: tomcat release: stable template: metadata: labels: app: tomcat release: stable spec: containers: - name: tomcat image: tomcat:latest ports: - name: http containerPort: 8080 - name: ajp containerPort: 8009
[root@k8s-master ingress]# kubectl create -f tomcat-demo.yaml service/tomcat created deployment.apps/tomcat-demo created [root@k8s-master ingress]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 4d5h tomcat ClusterIP 10.98.93.179 <none> 8080/TCP,8009/TCP 6m24s [root@k8s-master ingress]# kubectl get deployments NAME READY UP-TO-DATE AVAILABLE AGE tomcat-demo 3/3 3 3 6m28s [root@k8s-master ingress]# kubectl get pods NAME READY STATUS RESTARTS AGE tomcat-demo-6b78b44bf7-8hhpn 1/1 Running 0 3m1s tomcat-demo-6b78b44bf7-fgwlk 1/1 Running 0 90s tomcat-demo-6b78b44bf7-wlmc7 1/1 Running 0 22s [root@k8s-master ingress]#
3.2 准备证书并生成secret
[root@master ingress-nginx]# openssl genrsa -out tls.key 2048 Generating RSA private key, 2048 bit long modulus ............................+++ ..................+++ e is 65537 (0x10001) [root@master ingress-nginx]# openssl req -new -x509 -key tls.key -out tls.crt -subj /C=CN/ST=Beijing/L=Beijing/O=DevOps/CN=tomcat.wangxu.com
生成secret
[root@k8s-master ingress]# ls ingress-default-backend.yaml tls tls.key wangxu-deploy.yaml wangxu-svc.yaml ingress-nginx tls.crt tomcat-demo.yaml wangxu-ingress.yaml [root@k8s-master ingress]# kubectl create secret tls tomcat-ingress-secret --cert=tls.crt --key=tls.key secret/tomcat-ingress-secret created [root@k8s-master ingress]# kubectl get secret NAME TYPE DATA AGE default-token-wldf4 kubernetes.io/service-account-token 3 4d5h tomcat-ingress-secret kubernetes.io/tls 2 7s [root@k8s-master ingress]# kubectl describe secret tomcat-ingress-secret Name: tomcat-ingress-secret Namespace: default Labels: <none> Annotations: <none> Type: kubernetes.io/tls Data ==== tls.crt: 1294 bytes tls.key: 1675 bytes
3.3 创建ingress
# cat ingress-tomcat-tls.yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: ingress-tomcat-tls namespace: default annotations: kubernetes.io/ingress.class: "nginx" spec: tls: - hosts: - tomcat.wangxu.com secretName: tomcat-ingress-secret rules: - host: tomcat.zwx.com http: paths: - backend: serviceName: tomcat servicePort: 8080
[root@k8s-master ingress]# kubectl apply -f ingress-tomcat-tls.yaml ingress.extensions/ingress-tomcat-tls created [root@k8s-master ingress]#
3.4 测试Nginx服务
[root@k8s-master ingress]# kubectl get svc -n ingress-nginx -o wide NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR ingress-nginx NodePort 10.104.90.61 <none> 80:32078/TCP,443:32229/TCP 4h11m app.kubernetes.io/name=ingress-nginx,app.kubernetes.io/part-of=ingress-nginx [root@k8s-master ingress]#
本地host解析了
4 myAPP样例
4.1 创建后端服务myAPP svc和deployment
# cat myapp-deploy.yaml apiVersion: apps/v1 kind: Deployment metadata: name: myapp-wangxu namespace: default spec: replicas: 3 selector: matchLabels: app: myapp release: wangxu template: metadata: labels: app: myapp release: wangxu spec: containers: - name: myapp-wangxu image: ikubernetes/myapp:v2 ports: - name: http containerPort: 80
# cat myapp-svc.yaml apiVersion: v1 kind: Service metadata: name: myapp-wangxu namespace: default spec: selector: app: myapp release: wangxu ports: - name: http port: 80 targetPort: 80
[root@k8s-master ~]# kubectl get svc,pod| grep myapp service/myapp-wangxu ClusterIP 10.111.225.232 <none> 80/TCP 41s pod/myapp-wangxu-79b896cc97-6fdjq 1/1 Running 0 45s pod/myapp-wangxu-79b896cc97-js6rj 1/1 Running 0 45s pod/myapp-wangxu-79b896cc97-msjtt 1/1 Running 0 45s [root@k8s-master ~]#
4.2 部署Ingress Controller如果部署过，跳过
前面部署了就不用了
在kubernetes 得github上下载ingress得yaml文件，地址：https://github.com/kubernetes/ingress-nginx/tree/master/deploy
ingress得所需得文件：configmap.yaml ，namespace.yaml，rbac.yaml，tcp-services-configmap.yaml，with-rbac.yaml
同时在ingress得官方文档中介绍到，需要下载service-nodeport.yaml文件，这个文件得目的是为Ingress Controller 接入外部得流量，如果没有这个文件，是无法通过
简单的说就是执行这6个文件
[root@k8s-master ingress-nginx]# tree . ├── baremetal │ └── service-nodeport.yaml ├── configmap.yaml ├── mandatory.yaml ├── namespace.yaml ├── rbac.yaml └── with-rbac.yaml 1 directory, 6 files [root@k8s-master ingress-nginx]# [root@k8s-master ingress-nginx]# kubectl get svc,pods -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/ingress-nginx NodePort 10.104.90.61 <none> 80:32078/TCP,443:32229/TCP 4h34m NAME READY STATUS RESTARTS AGE pod/nginx-ingress-controller-79f6884cf6-v5cdj 1/1 Running 0 4h57m [root@k8s-master ingress-nginx]#
4.3 创建myapp-ingress服务
# cat ingress-myapp.yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: ingress-myapp namespace: default annotations: kubernetes.io/ingress.class: "nginx" # 这里是说明ingress的类型使用的nginx，一定要说明这点，否则ingress Controller 不知道是配置成那种类型的配置文件 spec: rules: - host: myapp.wangxu.com # 使用的是虚拟主机的来访问的 http: paths: - path: backend: serviceName: myapp-wangxu # 代理的后端的pod的service，通过这个service来生成nginx的upstrm servicePort: 80
[root@k8s-master ~]# kubectl apply -f ingress-myapp.yaml ingress.extensions/ingress-myapp configured [root@k8s-master ~]# kubectl get ingresses|grep myapp ingress-myapp myapp.wangxu.com 80 12m [root@k8s-master ~]#
4.4 访问的客户端的机器配置下域名解析
10.6.76.23 myapp.wangxu.com 10.6.76.24 myapp.wangxu.com
4.5 配置https自签名证书
[root@k8s-master ~]# openssl genrsa -out tls.key 2048 Generating RSA private key, 2048 bit long modulus .........................................+++ ...............................................................................................+++ e is 65537 (0x10001) [root@k8s-master ~]# openssl req -new -x509 -key tls.key -out tls.crt -subj /C=CN/ST=Beijing/L=Beijing/O=DevOps/CN=myapp.wangxu.com [root@k8s-master ~]# #证书生成好了，然后把证书转成secret， [root@k8s-master ~]# kubectl create secret tls myapp-ingress-secret --cert=tls.crt --key=tls.key secret/myapp-ingress-secret created [root@k8s-master ~]# kubectl get secrets NAME TYPE DATA AGE default-token-wldf4 kubernetes.io/service-account-token 3 4d6h myapp-ingress-secret kubernetes.io/tls 2 18s
4.6 修改myapp-ingress服务，加入证书
修改下 ingress-myapp.yaml 加入刚刚添加的secret，修改后的文件如下：
# cat ingress-myapp.yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: ingress-myapp namespace: default annotations: kubernetes.io/ingress.class: "nginx" spec: tls: # 添加了tls这一段 - hosts: - myapp.wangxu.com secretName: myapp-ingress-secret # 这里结束 rules: - host: myapp.wangxu.com http: paths: - path: backend: serviceName: myapp-wangxu servicePort: 80
[root@k8s-master ~]# kubectl apply -f ingress-myapp.yaml ingress.extensions/ingress-myapp configured [root@k8s-master ~]#
4.7 HTTPS访问
[root@k8s-master ~]# kubectl get svc -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx NodePort 10.104.90.61 <none> 80:32078/TCP,443:32229/TCP 4h44m [root@k8s-master ~]#
5 Nginx服务样例
5.1 部署Ingress Controller
参考上面的
5.2 创建后端服务Nginx svc和deployment
# cat nginx-dep.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-test namespace: default spec: replicas: 3 selector: matchLabels: app: nginx release: test template: metadata: labels: app: nginx release: test spec: containers: - name: nginx-test image: nginx:latest ports: - name: http containerPort: 80
# cat nginx-svc.yaml apiVersion: v1 kind: Service metadata: name: nginx-test namespace: default spec: selector: app: nginx release: test ports: - name: http port: 80 targetPort: 80
[root@k8s-master tls-nginx]# kubectl create -f nginx-dep.yaml deployment.apps/nginx-test created [root@k8s-master tls-nginx]# kubectl create -f nginx-svc.yaml service/nginx-test created [root@k8s-master tls-nginx]# kubectl get pods,svc -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod/nginx-test-5ff8cb55bf-6dtm9 1/1 Running 0 20s 10.254.2.50 k8s-node-2 <none> <none> pod/nginx-test-5ff8cb55bf-gxngh 1/1 Running 0 20s 10.254.1.58 k8s-node-1 <none> <none> pod/nginx-test-5ff8cb55bf-hmr8t 1/1 Running 0 20s 10.254.1.57 k8s-node-1 <none> <none> NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 4d6h <none> service/nginx-test ClusterIP 10.97.235.162 <none> 80/TCP 17s app=nginx,release=test [root@k8s-master tls-nginx]#
5.3 配置https自签名证书
openssl genrsa -out tls.key 2048 openssl req -new -x509 -key tls.key -out tls.crt -subj /C=CN/ST=Beijing/L=Beijing/O=DevOps/CN=nginx.test.com kubectl create secret tls nginx-ingress-secret --cert=tls.crt --key=tls.key
[root@k8s-master tls-nginx]# kubectl get secrets| grep nginx nginx-ingress-secret kubernetes.io/tls 2 86s [root@k8s-master tls-nginx]#
5.4 创建nginx-ingress服务
# cat nginx-tls.yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: ingress-nginx namespace: default annotations: kubernetes.io/ingress.class: "nginx" spec: tls: # 添加了tls这一段 - hosts: - nginx.test.com secretName: nginx-ingress-secret # 这里结束 rules: - host: nginx.test.com http: paths: - path: backend: serviceName: nginx-test servicePort: 80
[root@k8s-master tls-nginx]# kubectl create -f nginx-tls.yaml ingress.extensions/ingress-myapp created [root@k8s-master tls-nginx]#
5.5 域名自解析和HTTPS访问
10.6.76.23 nginx.test.com
10.6.76.24 nginx.test.com
[root@k8s-master tls-nginx]# kubectl get ingresses -o wide NAME HOSTS ADDRESS PORTS AGE ingress-myapp nginx.test.com 80, 443 2m26s [root@k8s-master tls-nginx]# kubectl get svc -o wide -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR ingress-nginx NodePort 10.104.90.61 <none> 80:32078/TCP,443:32229/TCP 5h13m app.kubernetes.io/name=ingress-nginx,app.kubernetes.io/part-of=ingress-nginx [root@k8s-master tls-nginx]#
