主機：centos 7.9
下載
官網GitHub上下載地址：https://github.com/containerd/containerd/releases
問題：
創建容器后，運行的時候報錯：
# ctr -n k8s.io task start -d busybox
ctr: failed to create shim: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/containerd/io.containerd.runtime.v2.task/k8s.io/busybox/log.json: no such file or directory): runc did not terminate successfully: exit status 127: unknown
通過運行runc命令排查，發現缺少依賴：seccomp_api_get
但是centos7.9系統已經安裝libseccomp-devel了，再次檢查發現，seccomp_api_get對libseccomp版本有要求，至少libseccomp v2.4版本才提供的有seccomp_api_get，安裝的版本是v2.3的，不帶這個 (軟件包 libseccomp-devel-2.3.1-4.el7.x86_64 已安裝並且是最新版本)
# runc
runc: symbol lookup error: runc: undefined symbol: seccomp_api_get
網上下載v2.4b版本的，地址：https://rpmfind.net/linux/rpm2html/search.php?query=libseccomp
centos7.9系統使用的最高版本就是v2.3了
換個其他網站：https://pkgs.org/download/libseccomp
估計只能源碼編譯安裝v2.4版本的了
首先需要安裝 seccomp 依賴：yum install -y libseccomp-devel (版本v2.3，后續使用創建容器的時候有問題)
由於 containerd 需要調用 runc，所以我們也需要先安裝 runc，不過 containerd 提供了一個包含相關依賴的壓縮包 cri-containerd-cni-${VERSION}.${OS}-${ARCH}.tar.gz，可以直接使用這個包來進行安裝。
首先從 release 頁面下載最新版本的壓縮包，當前為 1.5.9 版本：
# wget https://github.com/containerd/containerd/releases/download/v${VERSION}/cri-containerd-cni-${VERSION}-linux-amd64.tar.gz
# tar --no-overwrite-dir -C / -xzf cri-containerd-cni-${VERSION}-linux-amd64.tar.gz
wget https://github.com/containerd/containerd/releases/download/v1.5.9/cri-containerd-cni-1.5.9-linux-amd64.tar.gz
# 不用下載containerd-1.5.9-linux-amd64.tar.gz
查看壓縮包里的內容
# tar -tf containerd-1.5.9-linux-amd64.tar.gz
bin/
bin/ctr
bin/containerd-shim-runc-v2
bin/containerd-shim-runc-v1
bin/containerd-shim
bin/containerd
# 可以通過 tar 的 -t 選項直接看到壓縮包中包含哪些文件：
# tar -tf cri-containerd-cni-1.5.9-linux-amd64.tar.gz
etc/
etc/crictl.yaml
etc/systemd/
etc/systemd/system/
etc/systemd/system/containerd.service
etc/cni/
etc/cni/net.d/
etc/cni/net.d/10-containerd-net.conflist
usr/
usr/local/
usr/local/bin/
usr/local/bin/ctr
usr/local/bin/containerd-shim-runc-v2
usr/local/bin/containerd-shim-runc-v1
usr/local/bin/containerd-stress
usr/local/bin/containerd-shim
usr/local/bin/ctd-decoder
usr/local/bin/containerd
usr/local/bin/crictl
usr/local/bin/critest
usr/local/sbin/
usr/local/sbin/runc
opt/
opt/containerd/
opt/containerd/cluster/
opt/containerd/cluster/version
opt/containerd/cluster/gce/
opt/containerd/cluster/gce/cni.template
opt/containerd/cluster/gce/env
opt/containerd/cluster/gce/cloud-init/
opt/containerd/cluster/gce/cloud-init/node.yaml
opt/containerd/cluster/gce/cloud-init/master.yaml
opt/containerd/cluster/gce/configure.sh
opt/cni/
opt/cni/bin/
opt/cni/bin/bandwidth
opt/cni/bin/host-local
opt/cni/bin/static
opt/cni/bin/portmap
opt/cni/bin/vlan
opt/cni/bin/flannel
opt/cni/bin/tuning
opt/cni/bin/ipvlan
opt/cni/bin/ptp
opt/cni/bin/bridge
opt/cni/bin/host-device
opt/cni/bin/macvlan
opt/cni/bin/vrf
opt/cni/bin/sbr
opt/cni/bin/dhcp
opt/cni/bin/loopback
opt/cni/bin/firewall
安裝
# 直接將壓縮包解壓到系統的各個目錄中：
tar --no-overwrite-dir -C / -xzf cri-containerd-cni-1.5.9-linux-amd64.tar.gz
# 將 /usr/local/bin 和 /usr/local/sbin 追加到 ~/.bashrc 文件的 PATH 環境變量中，然后執行下面的命令使其立即生效：(這一步在該系統中不用操作，默認這倆路徑已經在PATH 環境變量中了)
export PATH=$PATH:/usr/local/bin:/usr/local/sbin
source ~/.bashrc
生成配置文件
# containerd 的默認配置文件為 /etc/containerd/config.toml，我們可以通過如下所示的命令生成一個默認的配置：
mkdir /etc/containerd
containerd config default > /etc/containerd/config.toml
啟動
# containerd 壓縮包中包含一個 etc/systemd/system/containerd.service 的文件，這樣我們就可以通過 systemd 來配置 containerd 作為守護進程運行了，內容如下所示：
cat /etc/systemd/system/containerd.service
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target
[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd
Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=infinity
# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
OOMScoreAdjust=-999
[Install]
WantedBy=multi-user.target
這里有兩個重要的參數：
- Delegate: 這個選項允許 containerd 以及運行時自己管理自己創建容器的 cgroups。如果不設置這個選項，systemd 就會將進程移到自己的 cgroups 中，從而導致 containerd 無法正確獲取容器的資源使用情況。
- KillMode: 這個選項用來處理 containerd 進程被殺死的方式。默認情況下，systemd 會在進程的 cgroup 中查找並殺死 containerd 的所有子進程。KillMode 字段可以設置的值如下。
- control-group（默認值）：當前控制組里面的所有子進程，都會被殺掉
- process：只殺主進程
- mixed：主進程將收到 SIGTERM 信號，子進程收到 SIGKILL 信號
- none：沒有進程會被殺掉，只是執行服務的 stop 命令
要將 KillMode 的值設置為 process，這樣可以確保升級或重啟 containerd 時不殺死現有的容器。（不用設置，默認就是process）
啟動 containerd 了，直接執行下面的命令即可：
systemctl daemon-reload
systemctl start containerd
啟動完成后就可以使用 containerd 的本地 CLI 工具 ctr 了，比如查看版本：
# ctr version
Client:
Version: v1.5.9
Revision: 1407cab509ff0d96baa4f0eb6ff9980270e6e620
Go version: go1.16.12
Server:
Version: v1.5.9
Revision: 1407cab509ff0d96baa4f0eb6ff9980270e6e620
UUID: 91a37754-a44f-4152-96b6-f2f7f96194f6
配置文件說明
首先來查看下上面默認生成的配置文件 /etc/containerd/config.toml：
# cat config.toml
disabled_plugins = []
imports = []
oom_score = 0
plugin_dir = ""
required_plugins = []
root = "/var/lib/containerd"
state = "/run/containerd"
version = 2
[cgroup]
path = ""
[debug]
address = ""
format = ""
gid = 0
level = ""
uid = 0
[grpc]
address = "/run/containerd/containerd.sock"
gid = 0
max_recv_message_size = 16777216
max_send_message_size = 16777216
tcp_address = ""
tcp_tls_cert = ""
tcp_tls_key = ""
uid = 0
[metrics]
address = ""
grpc_histogram = false
[plugins]
[plugins."io.containerd.gc.v1.scheduler"]
deletion_threshold = 0
mutation_threshold = 100
pause_threshold = 0.02
schedule_delay = "0s"
startup_delay = "100ms"
[plugins."io.containerd.grpc.v1.cri"]
disable_apparmor = false
disable_cgroup = false
disable_hugetlb_controller = true
disable_proc_mount = false
disable_tcp_service = true
enable_selinux = false
enable_tls_streaming = false
ignore_image_defined_volumes = false
max_concurrent_downloads = 3
max_container_log_line_size = 16384
netns_mounts_under_state_dir = false
restrict_oom_score_adj = false
sandbox_image = "k8s.gcr.io/pause:3.5"
selinux_category_range = 1024
stats_collect_period = 10
stream_idle_timeout = "4h0m0s"
stream_server_address = "127.0.0.1"
stream_server_port = "0"
systemd_cgroup = false
tolerate_missing_hugetlb_controller = true
unset_seccomp_profile = ""
[plugins."io.containerd.grpc.v1.cri".cni]
bin_dir = "/opt/cni/bin"
conf_dir = "/etc/cni/net.d"
conf_template = ""
max_conf_num = 1
[plugins."io.containerd.grpc.v1.cri".containerd]
default_runtime_name = "runc"
disable_snapshot_annotations = true
discard_unpacked_layers = false
no_pivot = false
snapshotter = "overlayfs"
[plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
base_runtime_spec = ""
container_annotations = []
pod_annotations = []
privileged_without_host_devices = false
runtime_engine = ""
runtime_root = ""
runtime_type = ""
[plugins."io.containerd.grpc.v1.cri".containerd.default_runtime.options]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
base_runtime_spec = ""
container_annotations = []
pod_annotations = []
privileged_without_host_devices = false
runtime_engine = ""
runtime_root = ""
runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
BinaryName = ""
CriuImagePath = ""
CriuPath = ""
CriuWorkPath = ""
IoGid = 0
IoUid = 0
NoNewKeyring = false
NoPivotRoot = false
Root = ""
ShimCgroup = ""
SystemdCgroup = false
[plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
base_runtime_spec = ""
container_annotations = []
pod_annotations = []
privileged_without_host_devices = false
runtime_engine = ""
runtime_root = ""
runtime_type = ""
[plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime.options]
[plugins."io.containerd.grpc.v1.cri".image_decryption]
key_model = "node"
[plugins."io.containerd.grpc.v1.cri".registry]
config_path = ""
[plugins."io.containerd.grpc.v1.cri".registry.auths]
[plugins."io.containerd.grpc.v1.cri".registry.configs]
[plugins."io.containerd.grpc.v1.cri".registry.headers]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
tls_cert_file = ""
tls_key_file = ""
[plugins."io.containerd.internal.v1.opt"]
path = "/opt/containerd"
[plugins."io.containerd.internal.v1.restart"]
interval = "10s"
[plugins."io.containerd.metadata.v1.bolt"]
content_sharing_policy = "shared"
[plugins."io.containerd.monitor.v1.cgroups"]
no_prometheus = false
[plugins."io.containerd.runtime.v1.linux"]
no_shim = false
runtime = "runc"
runtime_root = ""
shim = "containerd-shim"
shim_debug = false
[plugins."io.containerd.runtime.v2.task"]
platforms = ["linux/amd64"]
[plugins."io.containerd.service.v1.diff-service"]
default = ["walking"]
[plugins."io.containerd.snapshotter.v1.aufs"]
root_path = ""
[plugins."io.containerd.snapshotter.v1.btrfs"]
root_path = ""
[plugins."io.containerd.snapshotter.v1.devmapper"]
async_remove = false
base_image_size = ""
pool_name = ""
root_path = ""
[plugins."io.containerd.snapshotter.v1.native"]
root_path = ""
[plugins."io.containerd.snapshotter.v1.overlayfs"]
root_path = ""
[plugins."io.containerd.snapshotter.v1.zfs"]
root_path = ""
[proxy_plugins]
[stream_processors]
[stream_processors."io.containerd.ocicrypt.decoder.v1.tar"]
accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"]
args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
path = "ctd-decoder"
returns = "application/vnd.oci.image.layer.v1.tar"
[stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"]
accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"]
args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
path = "ctd-decoder"
returns = "application/vnd.oci.image.layer.v1.tar+gzip"
[timeouts]
"io.containerd.timeout.shim.cleanup" = "5s"
"io.containerd.timeout.shim.load" = "5s"
"io.containerd.timeout.shim.shutdown" = "3s"
"io.containerd.timeout.task.state" = "2s"
[ttrpc]
address = ""
gid = 0
uid = 0
這個配置文件比較復雜，我們可以將重點放在其中的 plugins 配置上面，仔細觀察我們可以發現每一個頂級配置塊的命名都是 plugins."io.containerd.xxx.vx.xxx" 這種形式，每一個頂級配置塊都表示一個插件，其中 io.containerd.xxx.vx 表示插件的類型，vx 后面的 xxx 表示插件的 ID，我們可以通過 ctr 查看插件列表：
# ctr plugin ls
TYPE ID PLATFORMS STATUS
io.containerd.content.v1 content - ok
io.containerd.snapshotter.v1 aufs linux/amd64 skip
io.containerd.snapshotter.v1 btrfs linux/amd64 skip
io.containerd.snapshotter.v1 devmapper linux/amd64 error
io.containerd.snapshotter.v1 native linux/amd64 ok
io.containerd.snapshotter.v1 overlayfs linux/amd64 ok
io.containerd.snapshotter.v1 zfs linux/amd64 skip
io.containerd.metadata.v1 bolt - ok
io.containerd.differ.v1 walking linux/amd64 ok
io.containerd.gc.v1 scheduler - ok
io.containerd.service.v1 introspection-service - ok
io.containerd.service.v1 containers-service - ok
io.containerd.service.v1 content-service - ok
io.containerd.service.v1 diff-service - ok
io.containerd.service.v1 images-service - ok
io.containerd.service.v1 leases-service - ok
io.containerd.service.v1 namespaces-service - ok
io.containerd.service.v1 snapshots-service - ok
io.containerd.runtime.v1 linux linux/amd64 ok
io.containerd.runtime.v2 task linux/amd64 ok
io.containerd.monitor.v1 cgroups linux/amd64 ok
io.containerd.service.v1 tasks-service - ok
io.containerd.internal.v1 restart - ok
io.containerd.grpc.v1 containers - ok
io.containerd.grpc.v1 content - ok
io.containerd.grpc.v1 diff - ok
io.containerd.grpc.v1 events - ok
io.containerd.grpc.v1 healthcheck - ok
io.containerd.grpc.v1 images - ok
io.containerd.grpc.v1 leases - ok
io.containerd.grpc.v1 namespaces - ok
io.containerd.internal.v1 opt - ok
io.containerd.grpc.v1 snapshots - ok
io.containerd.grpc.v1 tasks - ok
io.containerd.grpc.v1 version - ok
io.containerd.grpc.v1 cri linux/amd64 ok
頂級配置塊下面的子配置塊表示該插件的各種配置，比如 cri 插件下面就分為 containerd、cni 和 registry 的配置，而 containerd 下面又可以配置各種 runtime，還可以配置默認的 runtime。比如現在我們要為鏡像配置一個加速器，那么就需要在 cri 配置塊下面的 registry 配置塊下面進行配置 registry.mirrors：
# 修改之前
[plugins."io.containerd.grpc.v1.cri".registry]
config_path = ""
[plugins."io.containerd.grpc.v1.cri".registry.auths]
[plugins."io.containerd.grpc.v1.cri".registry.configs]
[plugins."io.containerd.grpc.v1.cri".registry.headers]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
# 修改之后
[plugins."io.containerd.grpc.v1.cri".registry]
config_path = ""
[plugins."io.containerd.grpc.v1.cri".registry.auths]
[plugins."io.containerd.grpc.v1.cri".registry.configs]
[plugins."io.containerd.grpc.v1.cri".registry.headers]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://bqr1dr1n.mirror.aliyuncs.com"]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."k8s.gcr.io"]
endpoint = ["https://registry.aliyuncs.com/k8sxio"]
- registry.mirrors."xxx": 表示需要配置 mirror 的鏡像倉庫，例如 registry.mirrors."docker.io" 表示配置 docker.io 的 mirror。
- endpoint: 表示提供 mirror 的鏡像加速服務，比如我們可以注冊一個阿里雲的鏡像服務來作為 docker.io 的 mirror。
另外在默認配置中還有兩個關於存儲的配置路徑：
root = "/var/lib/containerd"
state = "/run/containerd"
其中 root 是用來保存持久化數據，包括 Snapshots, Content, Metadata 以及各種插件的數據，每一個插件都有自己單獨的目錄，Containerd 本身不存儲任何數據，它的所有功能都來自於已加載的插件。
而另外的 state 是用來保存運行時的臨時數據的，包括 sockets、pid、掛載點、運行時狀態以及不需要持久化的插件數據。
替換 containerd 默認的 sand_box 鏡像，編輯 /etc/containerd/config.toml
# sandbox_image = "k8s.gcr.io/pause:3.5"
sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.2"
# 重啟
systemctl restart containerd