深入理解 Ingress

Ingress为弥补NodePort不足而生

NodePort一些不足：

• 一个端口只能一个服务使用，端口需提前规划

• 只支持4层负载均衡

nginx 动态感知pod ip 的变化，根据变化动态设置nginx 的upstream，并实现负载均衡

ingress controller 动态刷新 pod ip 列表 更新到 nginx 的配置文件

 

Pod与Ingress的关系

通过Service相关联

通过Ingress Controller实现Pod的负载均衡 - 支持TCP/UDP 4层和HTTP 7层

 

Ingress Controller

 

1. 部署Ingress Controller

 

Nginx：官方维护的Ingress Controller

部署文档：https://github.com/kubernetes/ingress-nginx/blob/master/docs/deploy/index.md

注意事项：

• 镜像地址修改成国内的：registry.aliyuncs.com/google_containers/nginx-ingress-controller:0.26.1

• 使用宿主机网络：hostNetwork: true

 

 

wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml

mandatory.yaml 里一共有3个configmap  一个是配置七层的，二个是配置四层的分别对应udp tcp

从apiserver 动态获取 endpoint ，从apiserver访问就必须授权，就需要创建serviceaccount，仅仅获取ip 列表，所以授予的权限 查看即可。

由于国外的镜像往往不能被 很好d 拉取。

更换配置中的默认镜像地址registry.cn-hangzhou.aliyuncs.com/benjamin-learn//nginx-ingress-controller:0.20.0

hostNetwork 字段和 container 一个层级

hostNetwork: true

 

创建ingress-controller 发现pod 起不来

报错提示1：

Error generating self-signed certificate: could not create temp pem file /etc/ingress-co

ntroller/ssl/default-fake-certificate.pem: open /etc/ingress-controller/ssl/default-fake-certificate.pem797363033: permission denied

解决方法：

原因:随着版本提高，安全限制越来越高，对于权限的管理也越来越精细

node节点分别  chmod -R 777 /var/lib/docker  授权任意用户有docker临时文件的任意权限

 

 

完整  ingress-controller的 yaml

mandatory.yaml

apiVersion: v1 kind: Namespace metadata:  name: ingress-nginx  labels:  app.kubernetes.io/name: ingress-nginx  app.kubernetes.io/part-of: ingress-nginx  ---  kind: ConfigMap apiVersion: v1 metadata:  name: nginx-configuration  namespace: ingress-nginx  labels:  app.kubernetes.io/name: ingress-nginx  app.kubernetes.io/part-of: ingress-nginx  --- kind: ConfigMap apiVersion: v1 metadata:  name: tcp-services  namespace: ingress-nginx  labels:  app.kubernetes.io/name: ingress-nginx  app.kubernetes.io/part-of: ingress-nginx  --- kind: ConfigMap apiVersion: v1 metadata:  name: udp-services  namespace: ingress-nginx  labels:  app.kubernetes.io/name: ingress-nginx  app.kubernetes.io/part-of: ingress-nginx  --- apiVersion: v1 kind: ServiceAccount metadata:  name: nginx-ingress-serviceaccount  namespace: ingress-nginx  labels:  app.kubernetes.io/name: ingress-nginx  app.kubernetes.io/part-of: ingress-nginx  --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata:  name: nginx-ingress-clusterrole  labels:  app.kubernetes.io/name: ingress-nginx  app.kubernetes.io/part-of: ingress-nginx rules:  - apiGroups:  - ""  resources:  - configmaps  - endpoints  - nodes  - pods  - secrets  verbs:  - list  - watch  - apiGroups:  - ""  resources:  - nodes  verbs:  - get  - apiGroups:  - ""  resources:  - services  verbs:  - get  - list  - watch  - apiGroups:  - ""  resources:  - events  verbs:  - create  - patch  - apiGroups:  - "extensions"  - "networking.k8s.io"  resources:  - ingresses  verbs:  - get  - list  - watch  - apiGroups:  - "extensions"  - "networking.k8s.io"  resources:  - ingresses/status  verbs:  - update  --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: Role metadata:  name: nginx-ingress-role  namespace: ingress-nginx  labels:  app.kubernetes.io/name: ingress-nginx  app.kubernetes.io/part-of: ingress-nginx rules:  - apiGroups:  - ""  resources:  - configmaps  - pods  - secrets  - namespaces  verbs:  - get  - apiGroups:  - ""  resources:  - configmaps  resourceNames: # Defaults to "<election-id>-<ingress-class>" # Here: "<ingress-controller-leader>-<nginx>" # This has to be adapted if you change either parameter # when launching the nginx-ingress-controller.  - "ingress-controller-leader-nginx"  verbs:  - get  - update  - apiGroups:  - ""  resources:  - configmaps  verbs:  - create  - apiGroups:  - ""  resources:  - endpoints  verbs:  - get --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: RoleBinding metadata:  name: nginx-ingress-role-nisa-binding  namespace: ingress-nginx  labels:  app.kubernetes.io/name: ingress-nginx  app.kubernetes.io/part-of: ingress-nginx roleRef:  apiGroup: rbac.authorization.k8s.io  kind: Role  name: nginx-ingress-role subjects:  - kind: ServiceAccount  name: nginx-ingress-serviceaccount  namespace: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata:  name: nginx-ingress-clusterrole-nisa-binding  labels:  app.kubernetes.io/name: ingress-nginx  app.kubernetes.io/part-of: ingress-nginx roleRef:  apiGroup: rbac.authorization.k8s.io  kind: ClusterRole  name: nginx-ingress-clusterrole subjects:  - kind: ServiceAccount  name: nginx-ingress-serviceaccount  namespace: ingress-nginx --- apiVersion: apps/v1 kind: Deployment metadata:  name: nginx-ingress-controller  namespace: ingress-nginx  labels:  app.kubernetes.io/name: ingress-nginx  app.kubernetes.io/part-of: ingress-nginx spec:  replicas: 1  selector:  matchLabels:  app.kubernetes.io/name: ingress-nginx  app.kubernetes.io/part-of: ingress-nginx  template:  metadata:  labels:  app.kubernetes.io/name: ingress-nginx  app.kubernetes.io/part-of: ingress-nginx  annotations:  prometheus.io/port: "10254"  prometheus.io/scrape: "true"  spec:  hostNetwork: true # wait up to five minutes for the drain of connections  terminationGracePeriodSeconds: 300  serviceAccountName: nginx-ingress-serviceaccount  containers:  - name: nginx-ingress-controller  image: registry.aliyuncs.com/google_containers/nginx-ingress-controller:0.26.1  args:  - /nginx-ingress-controller  - --configmap=$(POD_NAMESPACE)/nginx-configuration  - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services  - --udp-services-configmap=$(POD_NAMESPACE)/udp-services  - --publish-service=$(POD_NAMESPACE)/ingress-nginx  - --annotations-prefix=nginx.ingress.kubernetes.io  securityContext:  allowPrivilegeEscalation: true  capabilities:  drop:  - ALL  add:  - NET_BIND_SERVICE # www-data -> 33  runAsUser: 33  env:  - name: POD_NAME  valueFrom:  fieldRef:  fieldPath: metadata.name  - name: POD_NAMESPACE  valueFrom:  fieldRef:  fieldPath: metadata.namespace  ports:  - name: http  containerPort: 80  - name: https  containerPort: 443  livenessProbe:  failureThreshold: 3  httpGet:  path: /healthz  port: 10254  scheme: HTTP  initialDelaySeconds: 10  periodSeconds: 10  successThreshold: 1  timeoutSeconds: 10  readinessProbe:  failureThreshold: 3  httpGet:  path: /healthz  port: 10254  scheme: HTTP  periodSeconds: 10  successThreshold: 1  timeoutSeconds: 10  lifecycle:  preStop:  exec:  command:  - /wait-shutdown ---

创建ingress-service.yaml

由于nodeport 默认的端口是 30000-32767   所以 80 和443 端口想通过nodeport暴露需要修改kube-apiserver的配置，并重启kube-apiserver服务

apiVersion: v1 kind: Service metadata:  name: ingress-nginx  namespace: ingress-nginx  labels:  app.kubernetes.io/name: ingress-nginx  app.kubernetes.io/part-of: ingress-nginx spec:  type: NodePort  ports:  - name: http  port: 80  targetPort: 80  protocol: TCP  nodePort: 80 # http请求对外映射80端口  - name: https  port: 443  targetPort: 443  protocol: TCP  nodePort: 443 # https请求对外映射443端口  selector:  app.kubernetes.io/name: ingress-nginx  app.kubernetes.io/part-of: ingress-nginx  ---

ingress-nginx 名称空间下的资源 皆为 ready

查看pod 所在节点 443 和 80 端口是否正常监听

符合预期 80 443 端口正常监听

2. 创建Ingress规则

创建ingress规则实际指的就是配置 nginx 的server 模块配置，通过yaml 定义的规则自动刷新到pod中的nginx 的配置中

 

创建一个私有仓库的secret 用于拉取 我的阿里云私有镜像

 

kubectl create secret docker-registry myregistry --docker-server=registry.cn-hangzhou.aliyuncs.com --docker-username=benjamin7788 --docker-password=a7260488

web-deployment.yaml 创建一个3副本的java 镜像，使用我的私有阿里云镜像

增加imagePullSecrets和 containers 一个层级 

imagePullSecrets:

     - name: myregistry

apiVersion: apps/v1 kind: Deployment metadata:  labels:  app: web  name: web spec:  replicas: 3  selector:  matchLabels:  app: web  strategy:  rollingUpdate:  maxSurge: 25%  maxUnavailable: 25%  type: RollingUpdate  template:  metadata:  labels:  app: web  spec:  imagePullSecrets:  - name: myregistry  containers:  - image: registry.cn-hangzhou.aliyuncs.com/benjamin-learn/java-demo:lst  imagePullPolicy: Always  name: java  resources: {}  restartPolicy: Always 

web-service.yaml 将这个deployment 用nodeport 方式暴露

 

apiVersion: v1 kind: Service metadata:  labels:  app: web  name: web  namespace: default spec:  ports:  - port: 80  protocol: TCP  targetPort: 8080  selector:  app: web  type: NodePort

 

ingress.yaml

 

apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata:  name: example-ingress spec:  rules:  - host: example.ctnrs.com  http:  paths:  - path: /  backend:  serviceName: web  servicePort: 80

#生产环境

这个example.ctnrs.com 域名一般是由域名运营商负责解析 其中A记录对应的值为运行ingress-controller的 node的公网ip

#测试环境

本地绑定host文件，模拟访问。

windows集群 win+r  输入drivers  进入 etc目录 修改hosts文件  增加一条记录

192.168.31.65 example.ctnrs.com 

查看域名对应规则

当ingress规则创建查看ingress-controller 容器日志 发现 成功创建了一个ingress规则

进入ingress-controller 容器发现 nginx.conf 自动增加了一个server 模块的配置

成功访问java应用

 

请求路径：

http 请求--->node:80 --->(upstream)pod ip--->容器:8080

 

细节：

我们都知道nginx更新配置都需要reload 方能生效，而nginx-conroller内的nginx 采用 lua脚本将配置更新在内存中，使得不需要reload 也可以动态更新配置。

 

管理员 -->ingress yaml -->master（apiserver）--->service---master(apiserver)<--- ingress controller(pod ip) --->nginx.conf(lua脚本)--->upstream(pod ip) --->容器

 

3.根据urlL路由到多个服务

www.ctnrs.com/a  请求到a组服务器

www.ctnrs.com/b  请求到b组服务器

ingress-url.yaml

 

apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata:  name: example-ingress  annotations:  nginx.ingress.kubernetes.io/rewrite-target: / spec:  rules:  - host: example.ctnrs.com  http:  paths:  - path: /a  backend:  serviceName: web  servicePort: 80   - host: example.ctnrs.com  http:  paths:  - path: /b  backend:  serviceName: web2  servicePort: 80

创建 web2应用

kubectl create deploy web2 --image=nginx kubectl expose deploy web2 --port=80 --target-port=80

 

 

4.基于名称的虚拟主机

ingress-vhost.yaml 

apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata:  name: example-ingress spec:  rules:  - host: example1.ctnrs.com  http:  paths:  - path: /  backend:  serviceName: web  servicePort: 80   - host: example2.ctnrs.com  http:  paths:  - path: /  backend:  serviceName: web2  servicePort: 80

绑定本地host文件 192.168.31.65  example1.ctnrs.com  example1.ctnrs.com

访问example1.ctnrs.com

访问 example2.ctnrs.com

4.ingress 配置https

ingress-https.yaml

 

apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata:  name: example-ingress  annotations:  nginx.ingress.kubernetes.io/force-ssl-redirect: 'true' spec:  tls:  - hosts:  - example.ctnrs.com  secretName: secret-tls  rules:  - host: example.ctnrs.com  http:  paths:  - path: /  backend:  serviceName: web  servicePort: 80

 

访问 example.ctnrs.com 直接跳转到 https://example.ctnrs.com

选择信任后，默认使用的证书是 ingress-controller 内默认的证书

 

生成自签证书

cert.sh

 

cat > ca-config.json <<EOF {  "signing": {  "default": {  "expiry": "87600h"  },  "profiles": {  "kubernetes": {  "expiry": "87600h",  "usages": [  "signing",  "key encipherment",  "server auth",  "client auth"  ]  }  }  } } EOF  cat > ca-csr.json <<EOF {  "CN": "kubernetes",  "key": {  "algo": "rsa",  "size": 2048  },  "names": [  {  "C": "CN",  "L": "Beijing",  "ST": "Beijing"  }  ] } EOF  cfssl gencert -initca ca-csr.json | cfssljson -bare ca -  cat > example.ctnrs.com-csr.json <<EOF {  "CN": "example.ctnrs.com",  "hosts": [],  "key": {  "algo": "rsa",  "size": 2048  },  "names": [  {  "C": "CN",  "L": "BeiJing",  "ST": "BeiJing"  }  ] } EOF  cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes example.ctnrs.com-csr.json | cfssljson -bare  example.ctnrs.com #kubectl create secret tls example-ctnrs-com --cert=example.ctnrs.com.pem --key=example.ctnrs.com-key.pem

执行生成证书脚本 ，生成域名证书 example.ctnrs.com-key.pem  example.ctnrs.com.pem

 

创建tls类型secret 

kubectl create secret tls secret-tls --cert=/root/learn/ssl/example.ctnrs.com.pem --key=/root/learn/ssl/example.ctnrs.com -key.pemsecret/secret-tls created

再次查看证书，已经使用了 自签域名证书。

5.Annotations对Ingress个性化配置

ingress-controller的默认代理超时设置

ingress-annotations.yaml 

 

apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata:  name: example-ingress  annotations:  kubernetes.io/ingress.class: "nginx"  nginx.ingress.kubernetes.io/proxy-connect-timeout: "600"  nginx.ingress.kubernetes.io/proxy-send-timeout: "600"  nginx.ingress.kubernetes.io/proxy-read-timeout: "600"  nginx.ingress.kubernetes.io/proxy-body-size: "10m" spec:  rules:  - host: example.ctnrs.com  http:  paths:  - path: /  backend:  serviceName: web  servicePort: 80

通过注解的方式更改 ingress-controller的默认代理超时时间

 

官方注解详解：https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/annotations.md

 

Ingress Controller 高可用方案

方案一：通过keepalived  设置vip访问 的 主备高可用方案，使用daemonset 控制器在每个节点部署一个 ingress controller  

缺点：由于只有一台ingress controller 提供服务，所以当访问量变大，性能会产生瓶颈。

 

通过loadbalancer 反向代理多个 ingress-controller ，实现对多个ingress controller 的负载均衡，用户请求 lb 的ip 然后转发到 后端 ingress controller

nodeselector + daemonset +lb  完成部署

优点：这种很好的弥补了第一种方案的 不足，多台ingress controller 同时提供服务，可提供更大的 访问量。

缺点：lb 这时又会产生单点，需要对 lb 做高可用，这样比较消耗 资源。

其他主流控制器：

Traefik： HTTP反向代理、负载均衡工具

Istio：服务治理，控制入口流量