https://juejin.cn/post/6844903927318577159
背景介绍
某些情况下，我们在使用Kubernetes作为业务应用的云平台，想要实现应用的蓝绿部署用来迭代应用版本，用lstio太重太复杂，而且它本身定位于流控和网格治理；Ingress-Nginx在0.21版本引入了Canary功能，可以为网关入口配置多个版本的应用程序，使用annotation来控制多个后端服务的流量分配
Ingress-Nginx-Annotation Canary 功能介绍
如果想启用Canary功能，要先设置
nginx.ingress.kubernetes.io/canary: "true"，然后可以启用以下注释来配置Canary
nginx.ingress.kubernetes.io/canary-weight请求到Canary ingress中指定的服务的请求百分比，值为0-100的整数，根据设置的值来决定大概有百分之多少的流量会分配Canary Ingress中指定的后端s服务nginx.ingress.kubernetes.io/canary-by-header基于request header 的流量切分，适用于灰度发布或者A/B测试，当设定的hearder值为always是，请求流量会被一直分配到Canary入口，当hearder值被设置为never时，请求流量不会分配到Canary入口，对于其他hearder值，将忽略，并通过优先级将请求流量分配到其他规则nginx.ingress.kubernetes.io/canary-by-header-value这个配置要和nginx.ingress.kubernetes.io/canary-by-header一起使用，当请求中的hearder key和value 和nginx.ingress.kubernetes.io/canary-by-headernginx.ingress.kubernetes.io/canary-by-header-value匹配时，请求流量会被分配到Canary Ingress入口，对于其他任何hearder值，将忽略，并通过优先级将请求流量分配到其他规则nginx.ingress.kubernetes.io/canary-by-cookie这个配置是基于cookie的流量切分，也适用于灰度发布或者A/B测试，当cookie值设置为always时，请求流量将被路由到Canary Ingress入口，当cookie值设置为never时，请求流量将不会路由到Canary入口，对于其他值，将忽略，并通过优先级将请求流量分配到其他规则
金丝雀规则按优先顺序进行如下排序：canary-by-header - > canary-by-cookie - > canary-weight
1.基于权重的小规模版本测试
- v1版本编排文件
apiVersion: extensions/v1beta1 kind: Ingress metadata: annotations: kubernetes.io/ingress.class: nginx labels: app: echoserverv1 name: echoserverv1 namespace: echoserver spec: rules: - host: echo.chulinx.com http: paths: - backend: serviceName: echoserverv1 servicePort: 8080 path: / --- kind: Service apiVersion: v1 metadata: name: echoserverv1 namespace: echoserver spec: selector: name: echoserverv1 type: ClusterIP ports: - name: echoserverv1 port: 8080 targetPort: 8080 --- apiVersion: extensions/v1beta1 kind: Deployment metadata: name: echoserverv1 namespace: echoserver labels: name: echoserverv1 spec: template: metadata: labels: name: echoserverv1 spec: containers: - image: mirrorgooglecontainers/echoserver:1.10 name: echoserverv1 ports: - containerPort: 8080 name: echoserverv1 复制代码- 查看v1版本创建的资源
$ [K8sSj] kubectl get pod,service,ingress -n echoserver
NAME READY STATUS RESTARTS AGE
pod/echoserverv1-657b966cb5-7grqs 1/1 Running 0 24h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/echoserverv1 ClusterIP 10.99.68.72 <none> 8080/TCP 24h
NAME HOSTS ADDRESS PORTS AGE
ingress.extensions/echoserverv1 echo.chulinx.com 80 24h
复制代码- 访问v1的服务,可以看到10个请求都是访问到一个pod上也就是v1版本的服务
$ [K8sSj] for i in `seq 10`;do curl -s echo.chulinx.com|grep Hostname;done Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv1-657b966cb5-7grqs 复制代码- 创建v2版本的服务
我们开启canary功能，将v2版本的权重设置为50%,这个百分比并不能精确的将请求平均分配到两个版本的服务，而是在50%上下浮动
apiVersion: extensions/v1beta1 kind: Ingress metadata: annotations: kubernetes.io/ingress.class: nginx nginx.ingress.kubernetes.io/canary: "true" nginx.ingress.kubernetes.io/canary-weight: "50" labels: app: echoserverv2 name: echoserverv2 namespace: echoserver spec: rules: - host: echo.chulinx.com http: paths: - backend: serviceName: echoserverv2 servicePort: 8080 path: / --- kind: Service apiVersion: v1 metadata: name: echoserverv2 namespace: echoserver spec: selector: name: echoserverv2 type: ClusterIP ports: - name: echoserverv2 port: 8080 targetPort: 8080 --- apiVersion: extensions/v1beta1 kind: Deployment metadata: name: echoserverv2 namespace: echoserver labels: name: echoserverv2 spec: template: metadata: labels: name: echoserverv2 spec: containers: - image: mirrorgooglecontainers/echoserver:1.10 name: echoserverv2 ports: - containerPort: 8080 name: echoserverv2 复制代码- 再次查看创建的资源
$ [K8sSj] kubectl get pod,service,ingress -n echoserver
NAME READY STATUS RESTARTS AGE
pod/echoserverv1-657b966cb5-7grqs 1/1 Running 0 24h
pod/echoserverv2-856bb5758-f9tqn 1/1 Running 0 4s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/echoserverv1 ClusterIP 10.99.68.72 <none> 8080/TCP 24h service/echoserverv2 ClusterIP 10.111.103.170 <none> 8080/TCP 4s NAME HOSTS ADDRESS PORTS AGE ingress.extensions/echoserverv1 echo.chulinx.com 80 24h ingress.extensions/echoserverv2 echo.chulinx.com 80 4s 复制代码- 访问测试
可以看到请求有4个落到v2版本，6个落到v1版本，理论上来说，请求说越多，落到v2版本的请求数越接近设置的权重50%
$ [K8sSj] for i in `seq 10`;do curl -s echo.chulinx.com|grep Hostname;done Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv1-657b966cb5-7grqs 复制代码2.基于header的A/B测试
- 更改v2版本的编排文件
增加header
nginx.ingress.kubernetes.io/canary-by-header: "v2"
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/canary: "true" nginx.ingress.kubernetes.io/canary-weight: "50" nginx.ingress.kubernetes.io/canary-by-header: "v2" labels: app: echoserverv2 name: echoserverv2 namespace: echoserver spec: rules: - host: echo.chulinx.com http: paths: - backend: serviceName: echoserverv2 servicePort: 8080 path: / --- kind: Service apiVersion: v1 metadata: name: echoserverv2 namespace: echoserver spec: selector: name: echoserverv2 type: ClusterIP ports: - name: echoserverv2 port: 8080 targetPort: 8080 --- apiVersion: extensions/v1beta1 kind: Deployment metadata: name: echoserverv2 namespace: echoserver labels: name: echoserverv2 spec: template: metadata: labels: name: echoserverv2 spec: containers: - image: mirrorgooglecontainers/echoserver:1.10 name: echoserverv2 ports: - containerPort: 8080 name: echoserverv2 复制代码- 更新访问测试
测试了header 为
v2:alwaysv2:neverv2:true这三个hearder值，可以看到当hearder为v2:always时，流量会全部流入v2，当v2:never时，流量会全部流入v1，当v2:true时，也就是非always/never,流量会按照配置的权重流入对应版本的服务
$ [K8sSj] kubectl apply -f appv2.yml ingress.extensions/echoserverv2 configured service/echoserverv2 unchanged deployment.extensions/echoserverv2 unchanged $ [K8sSj] for i in `seq 10`;do curl -s -H "v2:always" echo.chulinx.com|grep Hostname;done Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn $ [K8sSj] for i in `seq 10`;do curl -s -H "v2:never" echo.chulinx.com|grep Hostname;done Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv1-657b966cb5-7grqs $ [K8sSj] for i in `seq 10`;do curl -s -H "v2:true" echo.chulinx.com|grep Hostname;done Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv2-856bb5758-f9tqn 复制代码- 自定义header-value
apiVersion: extensions/v1beta1 kind: Ingress metadata: annotations: kubernetes.io/ingress.class: nginx nginx.ingress.kubernetes.io/canary: "true" nginx.ingress.kubernetes.io/canary-weight: "50" nginx.ingress.kubernetes.io/canary-by-header: "v2" nginx.ingress.kubernetes.io/canary-by-header-value: "true" labels: app: echoserverv2 name: echoserverv2 namespace: echoserver spec: rules: - host: echo.chulinx.com http: paths: - backend: serviceName: echoserverv2 servicePort: 8080 path: / --- kind: Service apiVersion: v1 metadata: name: echoserverv2 namespace: echoserver spec: selector: name: echoserverv2 type: ClusterIP ports: - name: echoserverv2 port: 8080 targetPort: 8080 --- apiVersion: extensions/v1beta1 kind: Deployment metadata: name: echoserverv2 namespace: echoserver labels: name: echoserverv2 spec: template: metadata: labels: name: echoserverv2 spec: containers: - image: mirrorgooglecontainers/echoserver:1.10 name: echoserverv2 ports: - containerPort: 8080 name: echoserverv2 复制代码- 更新测试
可以看到只有header为
v2:never时，请求流量才会流入v2版本，其他值流量都会按照权重设置流入不通版本的服务
$ [K8sSj] kubectl apply -f appv2.yml ingress.extensions/echoserverv2 configured service/echoserverv2 unchanged deployment.extensions/echoserverv2 unchanged $ [K8sSj] for i in `seq 10`;do curl -s -H "v2:true" echo.chulinx.com|grep Hostname;done Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn $ [K8sSj] for i in `seq 10`;do curl -s -H "v2:always" echo.chulinx.com|grep Hostname;done Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn $ [K8sSj] for i in `seq 10`;do curl -s -H "v2:never" echo.chulinx.com|grep Hostname;done Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv1-657b966cb5-7grqs 复制代码3.基于cookie的流控
cookie其实和header原理大致相同，也是ingress自动cookie值，客户访问如果cookie匹配，流量就会流入与之匹配的后端服务
- 更新v2版本的编排文件
apiVersion: extensions/v1beta1 kind: Ingress metadata: annotations: kubernetes.io/ingress.class: nginx nginx.ingress.kubernetes.io/canary: "true" nginx.ingress.kubernetes.io/canary-weight: "50" nginx.ingress.kubernetes.io/canary-by-header: "v2" nginx.ingress.kubernetes.io/canary-by-header-value: "true" nginx.ingress.kubernetes.io/canary-by-cookie: "user_from_shanghai" labels: app: echoserverv2 name: echoserverv2 namespace: echoserver spec: rules: - host: echo.chulinx.com http: paths: - backend: serviceName: echoserverv2 servicePort: 8080 path: / --- kind: Service apiVersion: v1 metadata: name: echoserverv2 namespace: echoserver spec: selector: name: echoserverv2 type: ClusterIP ports: - name: echoserverv2 port: 8080 targetPort: 8080 --- apiVersion: extensions/v1beta1 kind: Deployment metadata: name: echoserverv2 namespace: echoserver labels: name: echoserverv2 spec: template: metadata: labels: name: echoserverv2 spec: containers: - image: mirrorgooglecontainers/echoserver:1.10 name: echoserverv2 ports: - containerPort: 8080 name: echoserverv2 复制代码- 访问测试
可以看和header的访问效果是一样的，只不过cookie不能自定义value
$ [K8sSj] kubectl apply -f appv2.yml ingress.extensions/echoserverv2 configured service/echoserverv2 unchanged deployment.extensions/echoserverv2 unchanged $ [K8sSj] for i in `seq 10`;do curl -s --cookie "user_from_shanghai" echo.chulinx.com|grep Hostname;done Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn # zlx @ zlxdeMacBook-Pro in ~/Desktop/unicom/k8syml/nginx-ingress-canary-deployment [16:01:52] $ [K8sSj] for i in `seq 10`;do curl -s --cookie "user_from_shanghai:always" echo.chulinx.com|grep Hostname;done Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv1-657b966cb5-7grqs Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn # zlx @ zlxdeMacBook-Pro in ~/Desktop/unicom/k8syml/nginx-ingress-canary-deployment [16:02:25] $ [K8sSj] for i in `seq 10`;do curl -s --cookie "user_from_shanghai=always" echo.chulinx.com|grep Hostname;done Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn Hostname: echoserverv2-856bb5758-f9tqn