使用crio作為runtime后，容器的啟動將不依賴docker相關的組件，容器進程更加簡潔。如下使用crio作為runtime啟動一個nginx的進程信息如下：根進程(1)->conmon->nginx。conmon作用於crio和runc(OCI實現)之間，用於在crio啟動容器后托管容器，更多參見conmon
root 15586 1 0 16:49 ? 00:00:00 /usr/local/bin/conmon --syslog -c a4f089f6b251c6269e2f79c41cec0317f4a65729b6075c77bbf4337206050501 -n k8s_nginx-test_nginx-test-24cjg_default_55bbcfe7-d63c-468b-bbcc-35a8b6c71eb9 root 15609 15586 0 16:49 ? 00:00:00 nginx: master process nginx -g daemon off;
安裝minikube
安裝cri-o(以下步驟來自官方文檔)：
- 安裝依賴庫
yum install -y \ btrfs-progs-devel \ containers-common \ device-mapper-devel \ git \ glib2-devel \ glibc-devel \ glibc-static \ go \ gpgme-devel \ libassuan-devel \ libgpg-error-devel \ libseccomp-devel \ libselinux-devel \ pkgconfig \ runc
- 編譯CRI-O，在編譯CRI-O時可以指定Build-tag。當前的CRI-O需要golang 12.x版本來編譯
git clone https://github.com/cri-o/cri-o # or your fork cd cri-o make sudo make install
- 編譯Conmon
git clone https://github.com/containers/conmon cd conmon make sudo make install
crio的配置文件默認為/etc/crio/crio.conf，可以通過命令crio config --default > /etc/crio/crio.conf來生成默認配置文件。
設置CNI網絡（以下步驟來自官方文檔）
git clone https://github.com/containernetworking/plugins cd plugins git checkout v0.8.1 ./build_linux.sh # or build_windows.sh sudo mkdir -p /opt/cni/bin sudo cp bin/* /opt/cni/bin/
- 編譯完CNI之后，把/opt/cni/bin中的二進制文件拷貝到/etc/crio/crio.conf的crio.network.plugin_dir目錄，默認為/usr/libexec/cni;並將cni配置放到crio.network.network_dir目錄
啟動CRI-O
- 在cri-o的源碼目錄下執行如下步驟，啟動CRI-O
sudo make install.systemd sudo systemctl daemon-reload sudo systemctl enable crio sudo systemctl start crio
使用crio-status命令
- 使用crio-status config可以查看當前crio的配置
安裝CRI-O命令行工具crictl
- crictl用法與docker命令類似，可以參見官方文檔
# go get github.com/kubernetes-sigs/cri-tools/cmd/crictl
# cp /root/go/bin/crictl /usr/local/bin
# crictl --runtime-endpoint unix:///var/run/crio/crio.sock version Version: 0.1.0 RuntimeName: cri-o RuntimeVersion: 1.15.1-dev RuntimeApiVersion: v1alpha1
- crictl默認會讀取/etc/crictl.yaml中的runtime-endpoint配置
# cat /etc/crictl.yaml runtime-endpoint: unix:///var/run/crio/crio.sock image-endpoint: unix:///var/run/crio/crio.sock
啟動minikube並配置使用CRI-O
minikube start --container-runtime=cri-o --vm-driver=none
crictl簡單使用
在啟動minikube之后會啟動相關組件pod，使用circtl ps可以看到相關的容器信息，最后一列為POD ID。更多參見crictl
[root@iZj6cid8uez7g44i1t0k7tZ net.d]# crictl ps CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID b69e8be1ef2b0 gcr.io/k8s-minikube/storage-provisioner@sha256:088daa9fcbccf04c3f415d77d5a6360d2803922190b675cb7fc88a9d2d91985a About an hour ago Running storage-provisioner 0 282d5beebf847 dd57045952649 bf261d157914477ee1a5969d28ec687f3fbfc9fb5a664b22df78e57023b0e03b About an hour ago Running coredns 0 4f7a8f3cac5c4 a9df5247ede0f bf261d157914477ee1a5969d28ec687f3fbfc9fb5a664b22df78e57023b0e03b About an hour ago Running coredns 0 6448effa2f7cd dc1027c8d94c5 c21b0c7400f988db4777858edd13b6d3930d62d7ccf026d2415485a52037f384 About an hour ago Running kube-proxy 0 0436f736f2a4a 25cb103bc2e1e k8s.gcr.io/kube-addon-manager@sha256:3e315022a842d782a28e729720f21091dde21f1efea28868d65ec595ad871616 About an hour ago Running kube-addon-manager 0 85ceee77c5c70 cf7378a82993d 301ddc62b80b16315d3c2653cf3888370394277afb3187614cfa20edc352ca0a About an hour ago Running kube-scheduler 0 baf3c10a81831 60d9bcf7a4b83 06a629a7e51cdcc81a5ed6a3e6650348312f20c954ac52ee489a023628ec9c7d About an hour ago Running kube-controller-manager 0 877a92f202a5f 7a67b324cd8c7 b2756210eeabf84f3221da9959e9483f3919dc2aaab4cd45e7cd072fcbde27ed About an hour ago Running etcd 0 74fe384e1645b 355ba11ac783f b305571ca60a5a7818bda47da122683d75e8a1907475681ee8b1efbd06bff12e About an hour ago Running kube-apiserver 0 d112f1dc64113
使用crictl inspect CONTAINER_ID可以查看容器的詳細信息，使用circtl inspectp POD_ID查看pod的詳細信息。crictl inspect CONTAINER_ID|grep sandboxId出來的值就是該容器對應的pod的POD_ID。
配置使用flannel插件
使用如下方式啟動minikube
minikube start \ --extra-config=controller-manager.allocate-node-cidrs=true \ --extra-config=controller-manager.cluster-cidr=10.233.64.0/18 \ --extra-config=kubelet.network-plugin=cni \ --extra-config=kubelet.pod-cidr=10.233.64.0/18 \ --network-plugin=cni \ --container-runtime=cri-o \ --vm-driver=none
按照官方命令安裝flannel插件，並查看coredns是否正常啟動(cni啟動之前coredns處於pending狀態)。執行下面命令前需要確保/etc/cni/net.d/為空(flannel自動生成)或配置文件正確，否則會有錯誤
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml
如果coredns出現如下錯誤，說明/etc/cni/net.d/中的配置文件版本字段錯誤，可以參考flannel官方配置，將cniVersion字段修改為"0.3.1"，這樣coredns稍后會正常啟動。使用crictl inspectp POD_ID可以看到network設置為minikube啟動參數--extra-config=kubelet.pod-cidr指定的值
cannot convert version ["" "0.1.0" "0.2.0"] to 0.4.0
正常啟動后可以在/run/flannel/subnet.env中看到配置的flannel信息，
# cat /run/flannel/subnet.env FLANNEL_NETWORK=10.244.0.0/16 FLANNEL_SUBNET=10.233.64.1/24 FLANNEL_MTU=1450 FLANNEL_IPMASQ=true
查看本地接口，可以看到flannel接口創建成功，后續新創建的pod將使用flannel網絡
]# ip link 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 link/ether 00:16:3e:04:eb:0e brd ff:ff:ff:ff:ff:ff 3: mybridge: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000 link/ether 0a:be:69:1e:02:70 brd ff:ff:ff:ff:ff:ff 7: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default link/ether 06:66:cd:4f:d2:9a brd ff:ff:ff:ff:ff:ff 8: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 02:02:85:cf:25:dd brd ff:ff:ff:ff:ff:ff 301: veth1b2b30e0@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP mode DEFAULT group default link/ether 8e:47:4b:b8:10:be brd ff:ff:ff:ff:ff:ff link-netnsid 0 302: veth2147d829@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP mode DEFAULT group default link/ether fa:3f:fe:5d:91:82 brd ff:ff:ff:ff:ff:ff link-netnsid 1 303: veth54baeef4@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP mode DEFAULT group default link/ether 9a:99:0f:82:ff:34 brd ff:ff:ff:ff:ff:ff link-netnsid 2
flannel原理如下
flannel支持Vxlan，Host-gw和UDP模式，其中UDP一般用於debug模式，可在kube-system命名空間的configmap kube-flannel-cfg中查看flannel的運行模式。更多參見官方文檔
TIPS:
- crio主要有如下4個配置文件
| File | Description |
|---|---|
| crio.conf(5) | CRI-O Configuration file |
| policy.json(5) | Signature Verification Policy File(s) |
| registries.conf(5) | Registries Configuration file |
| storage.conf(5) | Storage Configuration file |
- minikube啟動出現：sudo: crictl: command not found，解決辦法：將crictl放到/usr/bin目錄，參考該issue。原因是/etc/sudoers的secure_path中沒有該路徑
- minikube啟動出現：[certs] certificate apiserver-kubelet-client not signed by CA certificate ca: crypto/rsa: verification error，解決辦法：rm /var/lib/minikube/certs，參考該issue
參考：