k8s（openshift） 部署istio1.1

准備工作：

openshift 默認不允許UID為0的容器運行，要先授權scc以便安裝istio

# oc adm policy add-scc-to-user anyuid -z istio-ingress-service-account -n istio-system
# oc adm policy add-scc-to-user anyuid -z default -n istio-system
# oc adm policy add-scc-to-user anyuid -z prometheus -n istio-system
# oc adm policy add-scc-to-user anyuid -z istio-egressgateway-service-account -n istio-system
# oc adm policy add-scc-to-user anyuid -z istio-citadel-service-account -n istio-system
# oc adm policy add-scc-to-user anyuid -z istio-ingressgateway-service-account -n istio-system
# oc adm policy add-scc-to-user anyuid -z istio-cleanup-old-ca-service-account -n istio-system
# oc adm policy add-scc-to-user anyuid -z istio-mixer-post-install-account -n istio-system
# oc adm policy add-scc-to-user anyuid -z istio-mixer-service-account -n istio-system
# oc adm policy add-scc-to-user anyuid -z istio-pilot-service-account -n istio-system
# oc adm policy add-scc-to-user anyuid -z istio-sidecar-injector-service-account -n istio-system
# oc adm policy add-scc-to-user anyuid -z istio-galley-service-account -n istio-system
# oc adm policy add-scc-to-user anyuid -z istio-security-post-install-account -n istio-system

下載istio包

# curl -L https://git.io/getLatestIstio | ISTIO_VERSION=1.1.6 sh -

下載Helm工具

# wget https://storage.googleapis.com/kubernetes-helm/helm-v2.13.1-linux-amd64.tar.gz

# tar -zvxf helm-v2.13.1-linux-amd64.tar.gz
# cp linux-amd64/* /usr/bin/

 

安裝istio：

初始化，向Kubernetes api-server提交CDR

# kubectl create namespace istio-system
# helm template install/kubernetes/helm/istio-init --name istio-init --namespace istio-system | kubectl apply -f -　　

驗證CDR是否提交成功，數量為53

# kubectl get crds | grep 'istio.io\|certmanager.k8s.io' | wc -l

安裝核心組件

# helm template install/kubernetes/helm/istio --name istio --namespace istio-system | kubectl apply -f -

 

嘗試注入:

istio組件需要privileged權限，否則無法創建Pod

# oc adm policy add-scc-to-user privileged -z default -n dev

openshift注入設置，配置Webhook和證書簽名

# vim /etc/origin/master/master-config.patch
admissionConfig:
  pluginConfig:
    MutatingAdmissionWebhook:
      configuration:
        apiVersion: apiserver.config.k8s.io/v1alpha1
        kubeConfigFile: /dev/null
        kind: WebhookAdmission
    ValidatingAdmissionWebhook:
      configuration:
        apiVersion: apiserver.config.k8s.io/v1alpha1
        kubeConfigFile: /dev/null
        kind: WebhookAdmission

# cd /etc/origin/master/
# cp -p master-config.yaml master-config.yaml.prepatch
# oc ex config patch master-config.yaml.prepatch -p "$(cat master-config.patch)" > master-config.yaml
# master-restart api
# master-restart controllers

 

自動注入（默認配置）：

給namespace綁定注入標簽，即使是手動注入也要綁定標簽

# oc label  namespace dev istio-injection=enabled
# oc get namespace -L istio-injection
NAME                                STATUS    AGE       ISTIO-INJECTION
app-storage                         Active    21h       
default                             Active    21h       
dev                                 Active    5h        enabled

關閉特殊Pod的自動注入，比如OpenShift Builds完全沒必要注入istio

修改istio-system下的ConfigMap istio-sidecar-injector，加入以下內容

apiVersion: v1
kind: ConfigMap
metadata:
  name: istio-sidecar-injector
data:
  config: |-
    policy: enabled
    neverInjectSelector:
      - matchExpressions:
        - {key: openshift.io/build.name, operator: Exists}
      - matchExpressions:
        - {key: openshift.io/deployer-pod-for.name, operator: Exists}
    template: |-
      initContainers:
...

 

手動注入：

修改istio-system下的ConfigMap istio-sidecar-injector，關閉自動注入

policy: disabled

修改需要注入的Deployment配置

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: ignored
spec:
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "true"
    spec:
      containers:
      - name: ignored
        image: tutum/curl
        command: ["/bin/sleep","infinity"]

如果sidecar.istio.io/inject=false  即使policy: enabled 也不會注入

 

排錯：

• Pod無法創建

檢查scc privileged 是否給當前空間的default用戶授權

• 無法創建openshift Deployment 或者 Builds

Error creating deployer pod: pods "nginx-20-deploy" is forbidden: unable to validate against any pod security policy: [spec.initContainers[0].securityContext.securityContext.runAsUser: Invalid value: 0: must be in the ranges: [1000080000, 1000089999] spec.initContainers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed capabilities.add: Invalid value: "NET_ADMIN": capability may not be added spec.containers[1].securityContext.securityContext.runAsUser: Invalid value: 1337: must be in the ranges: [1000080000, 1000089999]]

直接排除這些系統Pod或者授權scc

# oc adm policy add-scc-to-user privileged -z deployer -n dev
# oc adm policy add-scc-to-user privileged -z builder -n dev

• Pod能成功創建但是istio-init容器一直是CrashLoopBackOff

這是因為istio-init容器需要特權模式，需要修改容器模板 istio-system/configmap/istio-sidecar-injector

- name: istio-init
   securityContext:
      privileged: true 

•  istio注入后容器不能訪問外部網絡

這是因為istio默認劫持所有流量，需要把外部網絡地址排除掉，最簡單的方式就是只包含k8s內部網絡

修改istio-system/configmap/istio-sidecar-injector

    - "-i"
    - "[[ annotation .ObjectMeta `traffic.sidecar.istio.io/includeOutboundIPRanges`  "172.30.0.0/16,10.128.0.0/14"  ]]"
    - "-x"
    - "[[ annotation .ObjectMeta `traffic.sidecar.istio.io/excludeOutboundIPRanges`  ""  ]]"