Istio 流量管理：示例二

Istio的流量管理示例一：Istio IngressGateWay和金丝雀示例。

示例

常见发布类型对比：蓝绿发布、滚动发布、灰度发布、金丝雀发布
示例资源 demo.yaml

site1 ...

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: site-gw
  namespace: default  # 指定为 ingress gateway pod 所在的 namespace
spec:
  selector:
    app: istio-ingressgateway  # ingressgateway 的 labels
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "site.kb.cx"
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: site-svc-vs
spec:
  hosts:
  - "site.kb.cx"  # 对应 Gateway/site-gw
  gateways:
  - default/site-gw  # 指定 Ingress Gateway
  http:
  - name: canary
    match:
    - headers:
        x-canary:
          exact: "true"
    route:
    - destination:
        host: site
        subset: v11
      headers:
        request:
          set:
            User-Agent: Chrome
        response:
          add:
            x-canary: "true"
  - name: default
    headers:
      response:
        add:
          version: "v10"
    route:
    - destination:
        host: site
        subset: v10

---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: site
spec:
  host: site
  subsets:
  - name: v10
    labels:
      app: site
      version: v1.0
  - name: v11
    labels:
      app: site
      version: v1.1
---
apiVersion: v1
kind: Service
metadata:
  name: site
  labels:
    app: site
spec:
  ports:
  - port: 8080
    name: http
  selector:
    app: site
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: site-v10
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: site
      version: v1.0
  template:
    metadata:
      name: site
      labels:
        app: site
        version: v1.0
    spec:
      containers:
      - name: site
        image: busybox
        imagePullPolicy: IfNotPresent
        command: ["/bin/sh", "-c", "echo 'welcome to site v1.0' > /var/www/index.html; httpd -f -p 8080 -h /var/www"]
        ports:
        - name: http
          containerPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: site-v11
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: site
      version: v1.1
  template:
    metadata:
      name: site
      labels:
        app: site
        version: v1.1
    spec:
      containers:
      - name: site
        image: busybox
        imagePullPolicy: IfNotPresent
        command: ["/bin/sh", "-c", "echo 'welcome to site v1.1' > /var/www/index.html; httpd -f -p 8080 -h /var/www"]
        ports:
        - name: http
          containerPort: 8080
source repo / download raw

创建资源

crate-resource ...

root@k8s-master:~# istioctl kube-inject -f demo.yaml | kubectl apply -f -
gateway.networking.istio.io/site-gw created
virtualservice.networking.istio.io/site-svc-vs created
destinationrule.networking.istio.io/site created
service/site created
deployment.apps/site-v10 created
deployment.apps/site-v11 created
root@k8s-master:~# kubectl get all
NAME                            READY   STATUS    RESTARTS   AGE
pod/site-v10-8bc8d8dd6-h2fbl    2/2     Running   0          23s
pod/site-v11-86b97859cf-2qsdd   2/2     Running   0          23s

NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP    25h
service/site         ClusterIP   10.99.233.144   <none>        8080/TCP   23s

NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/site-v10   1/1     1            1           23s
deployment.apps/site-v11   1/1     1            1           23s

NAME                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/site-v10-8bc8d8dd6    1         1         1       23s
replicaset.apps/site-v11-86b97859cf   1         1         1       23s
root@k8s-master:~# kubectl get gateways.networking.istio.io
NAME      AGE
site-gw   32s
root@k8s-master:~# kubectl get virtualservices.networking.istio.io
NAME          GATEWAYS              HOSTS            AGE
site-svc-vs   ["default/site-gw"]   ["site.kb.cx"]   35s
root@k8s-master:~# kubectl get destinationrules.networking.istio.io
NAME   HOST   AGE
site   site   40s

istion-info ...

root@k8s-master:~# kubectl get -n istio-system all
NAME                                        READY   STATUS    RESTARTS   AGE
pod/istio-egressgateway-579dc4df64-dtwms    1/1     Running   0          25h
pod/istio-ingressgateway-679bf9454b-np67f   1/1     Running   0          7m53s
pod/istiod-8675d9c57b-jwswg                 1/1     Running   0          25h

NAME                           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                                                      AGE
service/istio-egressgateway    ClusterIP   10.100.182.135   <none>        80/TCP,443/TCP                                                               25h
service/istio-ingressgateway   NodePort    10.96.242.187    <none>        15021:30455/TCP,80:30999/TCP,443:32324/TCP,31400:32261/TCP,15443:32152/TCP   25h
service/istiod                 ClusterIP   10.102.235.99    <none>        15010/TCP,15012/TCP,443/TCP,15014/TCP                                        25h

NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/istio-egressgateway    1/1     1            1           25h
deployment.apps/istio-ingressgateway   1/1     1            1           25h
deployment.apps/istiod                 1/1     1            1           25h

NAME                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/istio-egressgateway-579dc4df64    1         1         1       25h
replicaset.apps/istio-ingressgateway-679bf9454b   1         1         1       25h
replicaset.apps/istiod-8675d9c57b                 1         1         1       25h

示例

访问普通流量

v1 ...

root@k8s-master:~# curl -vvv -H "Host: site.kb.cx" --resolve site.kb.cx:30999:127.0.0.1 http://site.kb.cx:30999
* Added site.kb.cx:30999:127.0.0.1 to DNS cache
* Hostname site.kb.cx was found in DNS cache
*   Trying 127.0.0.1:30999...
* TCP_NODELAY set
* Connected to site.kb.cx (127.0.0.1) port 30999 (#0)
> GET / HTTP/1.1
> Host: site.kb.cx
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< date: Wed, 31 Aug 2022 12:54:20 GMT
< content-type: text/html
< accept-ranges: bytes
< last-modified: Wed, 31 Aug 2022 12:52:32 GMT
< etag: "630f5990-15"
< content-length: 21
< x-envoy-upstream-service-time: 2
< server: istio-envoy
< version: v10
<
welcome to site v1.0
* Connection #0 to host site.kb.cx left intact

金丝雀流量

canary-v11 ...

root@k8s-master:~# curl -vvv -H "Host: site.kb.cx" -H "x-canary: true"  --resolve site.kb.cx:30999:127.0.0.1 http://site.kb.cx:30999
* Added site.kb.cx:30999:127.0.0.1 to DNS cache
* Hostname site.kb.cx was found in DNS cache
*   Trying 127.0.0.1:30999...
* TCP_NODELAY set
* Connected to site.kb.cx (127.0.0.1) port 30999 (#0)
> GET / HTTP/1.1
> Host: site.kb.cx
> User-Agent: curl/7.68.0
> Accept: */*
> x-canary: true
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< date: Wed, 31 Aug 2022 12:55:05 GMT
< content-type: text/html
< accept-ranges: bytes
< last-modified: Wed, 31 Aug 2022 12:52:15 GMT
< etag: "630f597f-15"
< content-length: 21
< x-envoy-upstream-service-time: 34
< server: istio-envoy
< x-canary: true
<
welcome to site v1.1
* Connection #0 to host site.kb.cx left intact

批量打流

$ for i in {1..100}; do curl  -H "x-canary: true"  --resolve site.kb.cx:30999:127.0.0.1 http://site.kb.cx:30999;done
welcome to site v1.1
...

debug

启用 v11 istio-proxy debug 日志

kubectl exec -it site-v11-86b97859cf-2qsdd -c istio-proxy -- curl -XPOST -s -o /dev/null http://localhost:15000/logging?level=debug

通过 istiogateway 重新调用 canary 接口

root@k8s-master:~# curl -vvv -H "Host: site.kb.cx" -H "x-canary: true"  --resolve site.kb.cx:30999:127.0.0.1 http://site.kb.cx:30999
* Added site.kb.cx:30999:127.0.0.1 to DNS cache
* Hostname site.kb.cx was found in DNS cache
*   Trying 127.0.0.1:30999...
* TCP_NODELAY set
* Connected to site.kb.cx (127.0.0.1) port 30999 (#0)
> GET / HTTP/1.1
> Host: site.kb.cx
> User-Agent: curl/7.68.0
> Accept: */*
> x-canary: true
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< date: Wed, 31 Aug 2022 13:12:39 GMT
< content-type: text/html
< accept-ranges: bytes
< last-modified: Wed, 31 Aug 2022 12:52:15 GMT
< etag: "630f597f-15"
< content-length: 21
< x-envoy-upstream-service-time: 8
< server: istio-envoy
< x-canary: true  # header.response.add.x-canary 参数
<
welcome to site v1.1
* Connection #0 to host site.kb.cx left intact

v11 istio-proxy container 日志

$ kubectl logs -f site-v11-86b97859cf-2qsdd -c istio-proxy
...
2022-08-31T13:12:39.807421Z	debug	envoy router	[C649][S12133591032602623479] router decoding headers:
':authority', 'site.kb.cx'
':path', '/'
':method', 'GET'
':scheme', 'http'
'accept', '*/*'
'x-canary', 'true'
'x-forwarded-for', '10.244.0.0'
'x-forwarded-proto', 'http'
'x-request-id', '8dd6b72f-8c57-95bd-b9e5-913b7fd9b596'
'x-envoy-attempt-count', '1'
'user-agent', 'Chrome'  # header.request.set.user-agent 参数
'x-b3-traceid', 'bda577f15668b7a754e1b294369875b7'
'x-b3-spanid', '54e1b294369875b7'
'x-b3-sampled', '1'
'x-envoy-internal', 'true'
'x-forwarded-client-cert', 'By=spiffe://cluster.local/ns/default/sa/default;Hash=66d4f869b122f686023de5e24747066069eb97fe672434b8e04e09d51f590d83;Subject="";URI=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account'

v11 在调用其他容器会继续待用该参数么？

FaQ

upstream_reset_before_response_started{connection_termination} 问题

[2022-08-31T12:10:59.805Z] "GET / HTTP/1.1" 503 UC upstream_reset_before_response_started{connection_termination} - "-" 0 95 4 - "10.244.0.0" "Chrome" "7be5f158-d2ed-9e6c-bb63-e239609e34f1" "site.kb.cx" "10.244.1.22:8080" outbound|8080|v11|site.default.svc.cluster.local 10.244.1.13:59898 10.244.1.13:8080 10.244.0.0:39013 - canary

排查过程

开启 Istio ingressgateway debug 日志级别：

kubectl -n istio-system exec -it istio-ingressgateway-679bf9454b-7m4hh -- curl -XPOST -s -o /dev/null http://localhost:15000/logging?level=debug

查看 kubectl -n istio-system logs -f istio-ingressgateway-679bf9454b-7m4hh 发现错误：

2022-08-31T12:40:11.890424Z	debug	envoy pool	[C18719] client disconnected, failure reason: TLS error: 268436501:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_EXPIRED 33554464:system library:OPENSSL_internal:Broken pipe

看到是证书过期问题，判断可能因为虚拟机环境，时间不同步导致的，尝试删除 istio-ingressgateway 后重建 pod 修复，命令：

kubectl -n istio-system delete pod/istio-ingressgateway-679bf9454b-7m4hh

示例