Knative 是谷歌在2018年開源的 Serverless 框架，旨在提供一套簡單易用的 Serverless 方案，把 Serverless 標准化。目前參與 Knative 項目的公司有：Google、Pivotal、IBM、Red Hat、SAP。
本文使用3台阿里雲的ecs來搭建 Knative ，截止文章編寫時， Knative 的版本是v0.13，需要：
- v1.15或更高版本的k8s集群
- 1.3.6版本的Istio（使用Istio做網絡層）
1. 安裝k8s
1.1. 安裝kubeadm
kubeadm 是一個構建k8s集群的工具，它提供的 init 和 join 命令是快速構建k8s集群的最佳實踐。
1.1.1. 設置 hostname
分別在 master 節點和 workloads 節點執行下邊命令設置 hostname 。
# 設置 master 節點的 hostname
hostnamectl --static set-hostname k8s-master
# 設置 node1 節點的 hostname
hostnamectl --static set-hostname k8s-node1
# 設置 node2 節點的 hostname
hostnamectl --static set-hostname k8s-node2
以下 1.1.2~1.1.7 的操作需要在所有節點進行。
1.1.2. 設置 hosts
vi /etc/hosts
#在 /ets/hosts 文件添加， master 和 node 節點的 ip
${master的私網IP} k8s-master
${node1的私網IP} k8s-node1
${node2的私網IP} k8s-node2
1.1.3. 安裝 docker
使用 docker 作為容器運行時，1.15.2版本的 k8s 對應 docker 版本如下：
Kubernetes 1.15.2 --> Docker版本1.13.1、17.03、17.06、17.09、18.06、18.09
這里選擇 18.06.1-ce 版本的 docker 進行安裝
# 安裝 yum 工具包
yum install -y yum-utils
# 添加 yum 源
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# 安裝 18.06.1-ce 版本的 docker
yum install -y docker-ce-18.06.1.ce-3.el7
# 啟動 docker 服務並設置為開機啟動
systemctl start docker && systemctl enable docker
# 驗證是否安裝成功
docker version
1.1.4. 禁用 SELinux
This is required to allow containers to access the host filesystem, which is needed by pod networks for example. You have to do this until SELinux support is improved in the kubelet.
kubelet 暫不支持 SELinux ，所以禁用。
# Set SELinux in permissive mode (effectively disabling it)
setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
1.1.5. 關閉交換內存
當內存不足時， linux 會自動使用 swap 將部分內存數據存放到磁盤中，這樣會使性能下降，為了性能考慮推薦關掉。
swapoff -a
vi /etc/fstab
# 注釋掉 swap 相關行（如果有）
1.1.6. 修改 iptables 相關參數
Some users on RHEL/CentOS 7 have reported issues with traffic being routed incorrectly due to iptables being bypassed. You should ensure net.bridge.bridge-nf-call-iptables is set to 1 in your sysctl config
確保流量被正確路由。
vi /etc/sysctl.conf
# /etc/sysctl.conf 添加如下內容
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
# 使配置生效
sysctl -p
1.1.7. 安裝 kubeadm 、 kubelet 、 kubectl
# 配置 kubernetes.repo的源，官方源國內無法訪問，使用阿里雲的源
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
# 15.2 版本的kubeadm
yum install -y kubeadm-1.15.2 kubelet-1.15.2 kubectl-1.15.2 --disableexcludes=kubernetes
# 啟動kubelet服務
systemctl enable --now kubelet
1.2. 創建k8s集群
1.2.1. master 節點初始化
- 在 master 節點執行初始化命令。
pod-network-cidr：設定 pod 網絡的IP地址網段，不同的網絡插件使用的值有所不同， Calico 使用的是192.168.0.0/16， Canal 和 Flannel 使用的是10.244.0.0/16。image-repository：鏡像拉取地址，使用阿里雲的鏡像倉庫。kubernetes-version：指定k8s版本。
# 初始化命令
kubeadm init --pod-network-cidr=192.168.0.0/16 --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.15.2
# 初始化結束后返回：
[init] Using Kubernetes version: v1.15.2
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [*.*.*.* *.*.*.*]
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master localhost] and IPs [*.*.*.* 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master localhost] and IPs [*.*.*.* 127.0.0.1 ::1]
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 19.508137 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.15" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-master as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 419gig.hr760wuwlr6l8zat
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join ${master的私網IP}:6443 --token ${your token} --discovery-token-ca-cert-hash sha256:${your sha256}
- 確保 kubectl 正常工作。
root用戶執行export KUBECONFIG=/etc/kubernetes/admin.conf。
非root用戶執行如下命令。
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
- 給 kubectl 起個別名 k ，簡化一部分操作。
vi ~/.bashrc
# 在 .bashrc 添加一行，為kubectl起個別名k
alias k='kubectl'
# 刷新使修改生效
source ~/.bashrc
- 安裝網絡插件
執行k get nodes ，發現狀態是NotReady，查看kubelet的日志發現網絡沒有就緒（這一點在 master 節點的初始化日志中有提示You should now deploy a pod network to the cluster.）。
# 查看狀態
k get nodes
# 狀態
NAME STATUS ROLES AGE VERSION
k8s-master NotReady master 5m54s v1.15.2
# 查看kubelet日志
journalctl -f -u kubelet
# 一部分日志
kubelet.go:2169] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
cni.go:213] Unable to update cni config: No networks found in /etc/cni/net.d
這是因為 k8s 本身並不提供網絡功能，但是提供了容器網絡接口 CNI 。有一系列開源的網絡插件實現了 CNI 解決集群的容器聯網問題，比如flannel、calico、canal等等，這里我選擇Calico作為解決方案。
mkdir -p ~/calico/ && cd ~/calico/
wget https://docs.projectcalico.org/v3.8/manifests/calico.yaml
k apply -f ~/calico/calico.yaml
# 查看節點狀態
k get nodes --watch
# 狀態
NAME STATUS ROLES AGE VERSION
k8s-master Ready master 11m v1.15.2
節點狀態已經是 Ready 。但是查看 pod 的狀態，發現 calico-node 存在 READY 0/1 的情況。
k get pod --all-namespaces --watch
# pod的狀態
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-node-xxx 0/1 Running 0 118m
# 查看pod信息
k describe pod -n kube-system calico-node-xxx
# 看到報錯
Readiness probe failed: calico/node is not ready: BIRD is not ready: BGP not established with
查看 calico-node 的詳細信息后出現 Readiness probe failed: calico/node is not ready: BIRD is not ready: BGP not established with 的報錯信息。因為官方的 getting-started 沒有把 IP_AUTODETECTION_METHOD 這個IP檢測方法的參數放入 calico.yaml 中， calico 會使用第一個找到的 network interface （往往是錯誤的interface），導致 calico 把 master 也算進 nodes ，於是 master BGP 啟動失敗，而其他 workers 則啟動成功。
解決方法：
vi ~/calico/calico.yaml
...
# Cluster type to identify the deployment type
- name: CLUSTER_TYPE
value: "k8s,bgp"
# Auto-detect the BGP IP address.
# 新增部分開始
- name: IP_AUTODETECTION_METHOD
value: "interface=eth0"
# value: "interface=eth.*"
# value: "interface=can-reach=www.baidu.com"
# 新增部分結束
- name: IP
value: "autodetect"
...
- master 節點參與 pod 調度
出於安全考慮，默認配置下不會將 pod 調度到 master 節點。如果希望將 k8s-master 也當作 workloads 節點使用，可以執行如下命令：
k taint node k8s-master node-role.kubernetes.io/master-
如果要恢復 Master Only 狀態，執行如下命令：
k taint node k8s-master node-role.kubernetes.io/master=true:NoSchedule
1.2.2. workloads節點加入集群
在 workloads 節點執行 master 節點初始化后返回的kubeadm join ${master的私網IP}:6443 ...，加入 k8s 集群。
kubeadm join ${master的私網IP}:6443 --token ${your token} --discovery-token-ca-cert-hash sha256:${your sha256}
# 加入后命令行顯示
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
- 在 master 節點執行
k get nodes查看機器狀態。
# 查看 nodes
k get nodes
# kubectl get nodes 返回
NAME STATUS ROLES AGE VERSION
k8s-master Ready master 23m v1.15.2
k8s-node1 Ready <none> 3m40s v1.15.2
k8s-node1 Ready <none> 3m43s v1.15.2
2. 安裝istio
2.1. helm安裝
helm是 k8s 的包管理工具，類似 Linux 系統下的包管理器，如 yum。這里使用二進制方式進行安裝。
wget https://get.helm.sh/helm-v2.10.0-linux-amd64.tar.gz
tar -zxvf helm-v2.10.0-linux-amd64.tar.gz
mv linux-amd64/helm /usr/local/bin/helm
# 驗證安裝
helm version
# 創建serviceaccount
k -n kube-system create serviceaccount tiller
# 創建role
kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller
# 初始化
helm init --service-account tiller --tiller-image=registry.cn-hangzhou.aliyuncs.com/google_containers/tiller:v2.10.0 --upgrade --skip-refresh
helm repo update
2.2. istio安裝
cd ~
# 下載istio
export ISTIO_VERSION=1.3.6
curl -L https://git.io/getLatestIstio | sh -
cd istio-${ISTIO_VERSION}
# install the Istio CRDs
for i in install/kubernetes/helm/istio-init/files/crd*yaml; do kubectl apply -f $i; done
# create istio-system namespace
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Namespace
metadata:
name: istio-system
labels:
istio-injection: disabled
EOF
這里使用沒有 sidecar 的方式進行 istio 的安裝。
# 沒有sidecar
helm template --namespace=istio-system --set prometheus.enabled=false --set mixer.enabled=false --set mixer.policy.enabled=false --set mixer.telemetry.enabled=false --set pilot.sidecar=false --set pilot.resources.requests.memory=128Mi --set galley.enabled=false --set global.useMCP=false --set security.enabled=false --set global.disablePolicyChecks=true --set sidecarInjectorWebhook.enabled=false --set global.proxy.autoInject=disabled --set global.omitSidecarInjectorConfigMap=true --set gateways.istio-ingressgateway.autoscaleMin=1 --set gateways.istio-ingressgateway.autoscaleMax=2 --set pilot.traceSampling=100 --set global.mtls.auto=false install/kubernetes/helm/istio > ./istio-lean.yaml
k apply -f ~/istio-${ISTIO_VERSION}/istio-lean.yaml
# 查看狀態
k get pods -n istio-system --watch
3. 安裝Knative
Knative 現在有2個關鍵組件： Serving 和 Eventing （早期的 Building 組件已被 Tekton 替代）。 Knative 所需的鏡像國內無法下載，我已經上傳至docker hub。
3.1. 安裝Serving組件
# 下載所需鏡像
docker pull yizhishi/knative_serving_cmd_autoscaler:v0.13.0
docker pull yizhishi/knative_serving_cmd_autoscaler-hpa:v0.13.0
docker pull yizhishi/knative_serving_cmd_controller:v0.13.0
docker pull yizhishi/knative_serving_cmd_activator:v0.13.0
docker pull yizhishi/knative_serving_cmd_networking_istio:v0.13.0
docker pull yizhishi/knative_serving_cmd_webhook:v0.13.0
docker pull yizhishi/knative_serving_cmd_queue:v0.13.0
# crd
k apply -f https://github.com/knative/serving/releases/download/v0.13.0/serving-crds.yaml
# core components，這個serving-core.yaml里的鏡像地址我已經修改過
k apply -f https://raw.githubusercontent.com/yizhishi/knative-yaml/master/knative-serving/serving-core.yaml
# 這個serving-istio.yaml里的鏡像地址我已經修改過
k apply -f https://raw.githubusercontent.com/yizhishi/knative-yaml/master/knative-serving/serving-istio.yaml
k get pod --all-namespaces -o wide --watch
k get service -n istio-system istio-ingressgateway
3.2. 部署Knative應用
mkdir -p ~/knative-demo/ && cd ~/knative-demo/ && >service.yaml
vi service.yaml
# 將下邊內容寫進service.yaml
apiVersion: serving.knative.dev/v1 # Current version of Knative
kind: Service
metadata:
name: helloworld-go # The name of the app
namespace: default # The namespace the app will use
spec:
template:
spec:
containers:
- image: yizhishi/helloworld-go # The URL to the image of the app
env:
- name: TARGET # The environment variable printed out by the sample app
value: "Go Sample v1"
k apply -f ~/knative-demo/service.yaml -v 7
# 查看knative service
k get ksvc helloworld-go
# 顯示
NAME URL LATESTCREATED LATESTREADY READY REASON
helloworld-go http://helloworld-go.default.example.com helloworld-go-xxxx helloworld-go-xxxx True
使用curl -H "Host: helloworld-go.default.example.com" http://${master的私網IP}:31380訪問服務。
# curl -H "Host: helloworld-go.default.example.com" http://${master的私網IP}:31380
Hello World: Go Sample v1!
通過k get pod --watch，看到 helloworld-go 的 pod 從 Pending -> ContainerCreating -> Running -> Terminating ，可以看到pod 自動被拉起，運行了一段時間后，被終止。
k get pod --watch
NAME READY STATUS RESTARTS AGE
helloworld-go-xxx-deployment-xxx-xxx 0/2 Pending 0 0s
helloworld-go-xxx-deployment-xxx-xxx 0/2 Pending 0 0s
helloworld-go-xxx-deployment-xxx-xxx 0/2 ContainerCreating 0 0s
helloworld-go-xxx-deployment-xxx-xxx 0/2 ContainerCreating 0 2s
helloworld-go-xxx-deployment-xxx-xxx 1/2 Running 0 3s
helloworld-go-xxx-deployment-xxx-xxx 2/2 Running 0 3s
helloworld-go-xxx-deployment-xxx-xxx 2/2 Terminating 0 65s
helloworld-go-xxx-deployment-xxx-xxx 0/2 Terminating 0 86s
helloworld-go-xxx-deployment-xxx-xxx 0/2 Terminating 0 87s
helloworld-go-xxx-deployment-xxx-xxx 0/2 Terminating 0 87s
參考：