k8s集群部署多Ingress Controller，deploy ingress时admission webhook报错

error when creating "demo-nginx-ingress.yaml": admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: host "demo.localdev.me" and path "/" is already defined in ingress

集群中部署多个Ingress Controller

如：

1. 社区版本的ingress-nginx controller
2. nginx版本的nginx-ingress controller
3. nginx版本的nginx-plus-ingress controller

创建服务

kubectl create deployment demo --image=httpd --port=80 kubectl expose deployment demo

kubectl expose deployment demo

Ingress

新建挂载到社区版ingress controll的ingress文件demo-ingress-nginx.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: local-ingress-nginx-1
spec:
  ingressClassName: nginx
  rules:
    - host: demo.localdev.me
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: demo
                port:
                  number: 80

新建挂载到nginx版本ingress controll的ingress文件demo-nginx-ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: local-nginx-ingress-1
spec:
  ingressClassName: nginx-2
  rules:
    - host: demo.localdev.me
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: demo
                port:
                  number: 80

执行：

kubectl apply -f demo-nginx-ingress.yaml

再执行

kubectl apply -f demo-nginx-ingress.yaml

报错信息

Error from server (BadRequest): error when creating "demo-nginx-ingress.yaml": admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: host "demo.localdev.me" and path "/" is already defined in ingress default/local-ingress-nginx-1

原因分析

社区版本0.25.0之后，添加了webhook admission校验，不能部署相同的host和path

这个检查是全局性的，ingress发生增、删、改，都会调用该检查，该检查的本配置在ingress-nginx-controller的部署文件中


当相同的service部署到不同的ingress controller时，该校验不通过，报该错误。

实际上应该只检查本ingress内的，不应该全局检查。

这个问题社区已经修复了，但还未发布，参考：https://github.com/kubernetes/ingress-nginx/pull/8221

解决方法

思路

修改ingress-nginx-admission只检查需要的ingress

1. 修改ingress-nginx-admission， 在社区版ingress的deploy.yaml中

   添加objectSelector

​ objectSelector:
​ matchLabels:
​ ingressname: ingress-nginx

# Source: ingress-nginx/templates/admission-webhooks/validating-webhook.yaml
# before changing this value, check the required kubernetes version
# https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#prerequisites
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.0.15
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.1
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
  name: ingress-nginx-admission
webhooks:
  - name: validate.nginx.ingress.kubernetes.io
    matchPolicy: Equivalent
    objectSelector:
      matchLabels:
        ingressname: ingress-nginx
    rules:
      - apiGroups:
          - networking.k8s.io
        apiVersions:
          - v1
        operations:
          - CREATE
          - UPDATE
        resources:
          - ingresses
    failurePolicy: Fail
    sideEffects: None
    admissionReviewVersions:
      - v1
    clientConfig:
      service:
        namespace: ingress-nginx
        name: ingress-nginx-controller-admission
        path: /networking/v1/ingresses

2. 修改demo-ingress-nginx.yaml文件，添加label，确保和ingress-nginx-admission中一致

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: local-ingress-2
  labels:
    ingressname: ingress-nginx
spec:
  ingressClassName: nginx
  rules:
    - host: demo.localdev.me
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: demo
                port:
                  number: 80

备注

1. 如集群中部署的是多套社区版本ingress nginx controller，为了保证特性不丢失，需部署多套ingress-nginx-admission分别给每一个controller使用
2. 如集群中部署的是nginx版本的controller，不需要部署ingress-nginx-admission，nginx版本的controller的数据合法性检查是由controller内部通过nginx -s进行数据合法性检查，不需要web hook admission