k8s - 基于nginx-ingress的灰度发布

本文将介绍如何在kubernetes下使用ingress实现灰度发布、蓝绿发布。

假设当前线上环境我们已经有一套服务app-old对外提供7层服务，此时我们修复了一些问题，需要灰度发布上线一个新的版本app-new，但是我们又不希望简单直接地将所有客户端流量切换到新版本app-new中，而是希望仅仅切换20%的流量到新版本app-new中，待运行一段时间稳定，将所有的流量切换到app-new服务中后，再平滑地下线掉app-old服务。

针对以上多种不同的应用发布需求，K8S Ingress Controller支持了多种流量切分方式：

基于Request Header的流量切分，适用于灰度发布以及AB测试场景
基于Cookie的流量切分，适用于灰度发布以及AB测试场景
基于Query Param的流量切分，适用于灰度发布以及AB测试场景
基于服务权重的流量切分，适用于蓝绿发布场景

以下测试基于服务权重的流量切分，也可以将nginx.ingress.kubernetes.io/canary-weight: "30"改为基于header的流量切分。

准备老版本程序

老版本程序app-old

app-old.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: app-old
spec:
  replicas: 2
  selector:
    matchLabels:
      run: app-old
  template:
    metadata:
      labels:
        run: app-old
    spec:
      containers:
      - image: zouhl/app:v2.1
        imagePullPolicy: Always
        name: app-old
        ports:
        - containerPort: 80
          protocol: TCP
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: app-old
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: app-old
  sessionAffinity: None
  type: NodePor

老版本的ingress

app-v1.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-app
  labels:
    app: my-app
  annotations:
    kubernetes.io/ingress.class: nginx
  namespace: default
spec:
  rules:
    - host: test.192.168.2.20.xip.io
      http:
        paths:
          - backend:
              serviceName: app-old
              servicePort: 80
            path: /

在k8s中创建

1
2


kubectl create -f app-old.yaml
kubectl create -f app-v1.yaml

装备新版本程序

新版本app-new.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: app-new
spec:
  replicas: 2
  selector:
    matchLabels:
      run: app-new
  template:
    metadata:
      labels:
        run: app-new
    spec:
      containers:
      - image: zouhl/app:v2.2
        imagePullPolicy: Always
        name: app-new
        ports:
        - containerPort: 80
          protocol: TCP
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: app-new
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: app-new
  sessionAffinity: None
  type: NodePort

新版本canary-ingress

app-v2-canary.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-app-canary
  labels:
    app: my-app
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "30"
  namespace: default
spec:
  rules:
    - host: test.192.168.2.20.xip.io
      http:
        paths:
          - backend:
              serviceName: app-new
              servicePort: 80
            path: /

新版本ingress yaml

app-v2.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-app
  labels:
    app: my-app
  annotations:
    kubernetes.io/ingress.class: nginx
  namespace: default
spec:
  rules:
    - host: test.192.168.2.20.xip.io
      http:
        paths:
          - backend:
              serviceName: app-new
              servicePort: 80
            path: /

发布流程

1
2
3
4
5
6
7
8
9


$ tree                                              
.
├── app-new.yaml
├── app-old.yaml
├── app-v1.yaml
├── app-v2-canary.yaml
└── app-v2.yaml

app-v1已经发布了，现在灰度发布第二版，权重为30%，nginx.ingress.kubernetes.io/canary-weight: "30"，更多参数参考github

1
2


kubectl create -f app-new.yaml
kubectl create -f app-v2-canary.yaml

检查

$ kubectl get ingresses.extensions   
NAME            HOSTS                       ADDRESS   PORTS   AGE
app-ingress     www.example.com                       80      109m
my-app          test.192.168.2.20.xip.io              80      25m
my-app-canary   test.192.168.2.20.xip.io              80      1s
nginx-test      nginx.192.168.2.20.xip.io             80      3h12m

在后台观察，70% to v1，30% to v2

1
2
3
4
5
6


$ while sleep 0.5; do curl "test.192.168.2.20.xip.io";echo; done
{"v2.2 hostname":"app-new-658dfc9c6b-lbmvr"}
{"v2.2 hostname":"app-new-658dfc9c6b-qhwtg"}
{"v1 hostname":"app-old-64fd44b699-4hvlb"}
{"v1 hostname":"app-old-64fd44b699-zb58f"}

如果一切正常则可以正式发布

# delete the canary ingress
kubectl delete -f app-v2-canary.yaml
# set 100% traffic to v2
kubectl apply -f app-v2.yaml

检查ingress

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


$ kubectl get ingresses.extensions    
NAME          HOSTS                       ADDRESS   PORTS   AGE
app-ingress   www.example.com                       80      109m
my-app        test.192.168.2.20.xip.io              80      25m
nginx-test    nginx.192.168.2.20.xip.io             80      3h13m

$ while sleep 0.5; do curl "test.192.168.2.20.xip.io";echo; done
{"v2.2 hostname":"app-new-658dfc9c6b-lbmvr"}
{"v2.2 hostname":"app-new-658dfc9c6b-qhwtg"}