暂基于kubespary自动部署的1.9.0-coreos版本，kubelet服务相关配置文件：
文件1：/etc/systemd/system/kubelet.service
/etc/systemd/system/kubelet.service
文件内容为：
[Unit]
Description=Kubernetes Kubelet Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Wants=docker.socket
[Service]
EnvironmentFile=-/etc/kubernetes/kubelet.env
ExecStartPre=-/bin/mkdir -p /var/lib/kubelet/volume-plugins
ExecStart=/usr/local/bin/kubelet \
$KUBE_LOGTOSTDERR \
$KUBE_LOG_LEVEL \
$KUBELET_API_SERVER \
$KUBELET_ADDRESS \
$KUBELET_PORT \
$KUBELET_HOSTNAME \
$KUBE_ALLOW_PRIV \
$KUBELET_ARGS \
$DOCKER_SOCKET \
$KUBELET_NETWORK_PLUGIN \
$KUBELET_VOLUME_PLUGIN \
$KUBELET_CLOUDPROVIDER
Restart=always
RestartSec=10s
[Install]
WantedBy=multi-user.target
文件2：/etc/kubernetes/kubelet.env
/etc/kubernetes/kubelet.env 文件内容： # logging to stderr means we get it in the systemd journal KUBE_LOGTOSTDERR="--logtostderr=true" KUBE_LOG_LEVEL="--v=2" # The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces) KUBELET_ADDRESS="--address=172.28.2.211 --node-ip=172.28.2.211" # The port for the info server to serve on # KUBELET_PORT="--port=10250" # You may leave this blank to use the actual hostname KUBELET_HOSTNAME="--hostname-override=node1" KUBELET_ARGS="--pod-manifest-path=/etc/kubernetes/manifests \ --cadvisor-port=0 \ --pod-infra-container-image=gcr.io/google_containers/pause-amd64:3.0 \ --node-status-update-frequency=10s \ --docker-disable-shared-pid=True \ --client-ca-file=/etc/kubernetes/ssl/ca.pem \ --tls-cert-file=/etc/kubernetes/ssl/node-node1.pem \ --tls-private-key-file=/etc/kubernetes/ssl/node-node1-key.pem \ --anonymous-auth=false \ --cgroup-driver=cgroupfs \ --cgroups-per-qos=True \ --fail-swap-on=True \ --enforce-node-allocatable="" --cluster-dns=10.233.0.3 --cluster-domain=cluster.local --resolv-conf=/etc/resolv.conf --kubeconfig=/etc/kubernetes/node-kubeconfig.yaml --kube-reserved cpu=100m,memory=256M --node-labels=node-role.kubernetes.io/node=true --feature-gates=Initializers=False,PersistentLocalVolumes=False " KUBELET_NETWORK_PLUGIN="--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin" KUBELET_VOLUME_PLUGIN="--volume-plugin-dir=/var/lib/kubelet/volume-plugins" # Should this cluster be allowed to run privileged docker containers KUBE_ALLOW_PRIV="--allow-privileged=true" KUBELET_CLOUDPROVIDER="" PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ~
修改需求：
Enabling Unsafe Sysctls
With the warning above in mind, the cluster admin can allow certain unsafe sysctls for very special situations like e.g. high-performance or real-time application tuning. Unsafe sysctls are enabled on a node-by-node basis with a flag of the kubelet, e.g.:
$ kubelet --experimental-allowed-unsafe-sysctls 'kernel.msg*,net.ipv4.route.min_pmtu' ...
修改需求：
kubelet --experimental-allowed-unsafe-sysctls 'kernel.msg*,kernel.shmmax,kernel.sem,net.ipv4.route.min_pmtu'
修改方法：修改环境变量文件/etc/kubernetes/kubelet.env，修改为
（添加了--experimental-allowed-unsafe-sysctls='kernel.msg*,kernel.shmmax,kernel.sem,net.ipv4.route.min_pmtu'"）# logging to stderr means we get it in the systemd journal
KUBE_LOGTOSTDERR="--logtostderr=true"
KUBE_LOG_LEVEL="--v=2"
# The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)
KUBELET_ADDRESS="--address=172.28.2.211 --node-ip=172.28.2.211"
# The port for the info server to serve on
# KUBELET_PORT="--port=10250"
# You may leave this blank to use the actual hostname
KUBELET_HOSTNAME="--hostname-override=node1"
KUBELET_ARGS="--pod-manifest-path=/etc/kubernetes/manifests \
--cadvisor-port=0 \
--pod-infra-container-image=gcr.io/google_containers/pause-amd64:3.0 \
--node-status-update-frequency=10s \
--docker-disable-shared-pid=True \
--client-ca-file=/etc/kubernetes/ssl/ca.pem \
--tls-cert-file=/etc/kubernetes/ssl/node-node1.pem \
--tls-private-key-file=/etc/kubernetes/ssl/node-node1-key.pem \
--anonymous-auth=false \
--cgroup-driver=cgroupfs \
--cgroups-per-qos=True \
--fail-swap-on=True \
--enforce-node-allocatable="" --cluster-dns=10.233.0.3 --cluster-domain=cluster.local --resolv-conf=/etc/resolv.conf --kubeconfig=/etc/kubernetes/node-kubeconfig.yaml --kube-reserved cpu=100m,memory=256M --node-labels=node-role.kubernetes.io/node=true --feature-gates=Initializers=False,PersistentLocalVolumes=False \
--experimental-allowed-unsafe-sysctls='kernel.msg*,kernel.shmmax,kernel.sem,net.ipv4.route.min_pmtu'"
KUBELET_NETWORK_PLUGIN="--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
KUBELET_VOLUME_PLUGIN="--volume-plugin-dir=/var/lib/kubelet/volume-plugins"
# Should this cluster be allowed to run privileged docker containers
KUBE_ALLOW_PRIV="--allow-privileged=true"
KUBELET_CLOUDPROVIDER=""
PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
~
重启kubelet服务，查看是否修改成功：
systemctl restart kubelet
systemctl status kubelet
ps aux | grep kubelet | grep kernel
root@node1:~# systemctl status kubelet
● kubelet.service - Kubernetes Kubelet Server
Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Active: active (running) since 四 2018-03-08 17:20:38 CST; 2min 51s ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
Process: 14844 ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volume-plugins (code=exited, status=0/SUCCESS)
Main PID: 14851 (kubelet)
Tasks: 17
Memory: 50.4M
CPU: 27.792s
CGroup: /system.slice/kubelet.service
└─14851 /usr/local/bin/kubelet --logtostderr=true --v=2 --address=172.28.2.211 --node-ip=172.28.2.211 -
3月 08 17:23:28 node1 kubelet[14851]: E0308 17:23:28.518287 14851 pod_workers.go:186] Error syncing pod 082ad73d
3月 08 17:23:28 node1 kubelet[14851]: W0308 17:23:28.518432 14851 container.go:393] Failed to create summary rea
3月 08 17:23:28 node1 kubelet[14851]: I0308 17:23:28.887368 14851 kubelet.go:1881] SyncLoop (PLEG): "centos1_def
3月 08 17:23:28 node1 kubelet[14851]: W0308 17:23:28.887505 14851 pod_container_deletor.go:77] Container "84c856
3月 08 17:23:29 node1 kubelet[14851]: I0308 17:23:29.188203 14851 kuberuntime_manager.go:403] No ready sandbox f
3月 08 17:23:29 node1 kubelet[14851]: E0308 17:23:29.624710 14851 remote_runtime.go:92] RunPodSandbox from runti
3月 08 17:23:29 node1 kubelet[14851]: E0308 17:23:29.624792 14851 kuberuntime_sandbox.go:54] CreatePodSandbox fo
3月 08 17:23:29 node1 kubelet[14851]: E0308 17:23:29.624814 14851 kuberuntime_manager.go:647] createPodSandbox f
3月 08 17:23:29 node1 kubelet[14851]: E0308 17:23:29.624923 14851 pod_workers.go:186] Error syncing pod 082ad73d
3月 08 17:23:29 node1 kubelet[14851]: W0308 17:23:29.625543 14851 container.go:393] Failed to create summary rea
root@node1:~# ps aux | grep kubelet | grep kernel
root 14851 12.5 0.7 696144 121368 ? Ssl 17:20 0:24 /usr/local/bin/kubelet --logtostderr=true --v=2 --address=172.28.2.211
--node-ip=172.28.2.211 --hostname-override=node1 --allow-privileged=true --pod-manifest-path=/etc/kubernetes/manifests
--cadvisor-port=0 --pod-infra-container-image=gcr.io/google_containers/pause-amd64:3.0 --node-status-update-frequency=10s
--docker-disable-shared-pid=True --client-ca-file=/etc/kubernetes/ssl/ca.pem --tls-cert-file=/etc/kubernetes/ssl/node-node1.pem
--tls-private-key-file=/etc/kubernetes/ssl/node-node1-key.pem --anonymous-auth=false --cgroup-driver=cgroupfs --cgroups-per-qos=True
--fail-swap-on=True --enforce-node-allocatable= --cluster-dns=10.233.0.3 --cluster-domain=cluster.local --resolv-conf=/etc/resolv.conf
--kubeconfig=/etc/kubernetes/node-kubeconfig.yaml --kube-reserved cpu=100m,memory=256M --node-labels=node-role.kubernetes.io/node=true
--feature-gates=Initializers=False,PersistentLocalVolumes=False
--experimental-allowed-unsafe-sysctls=kernel.msg*,kernel.shmmax,kernel.sem,net.ipv4.route.min_pmtu
--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin
--volume-plugin-dir=/var/lib/kubelet/volume-plugins
如上修改成功。
创建POD，参考页面：K8S实用整理(10)-Kubernetes配置POD内核参数sysctl http://www.cnblogs.com/DaweiJ/articles/8528687.html
Kubelet Configurations We Should Care About
下面是我梳理的，我认为必须关注的flag。
| flag | value |
|---|---|
| --address | 0.0.0.0 |
| --allow-privileged | false |
| --cadvisor-port int32 | 4194 |
| --cgroup-driver string | cgroupfs |
| --cluster-dns stringSlice | 10.0.0.10 //todo |
| --cluster-domain string | caas.vivo.com |
| --cni-bin-dir string | /opt/cni/bin |
| --cni-conf-dir string | /etc/cni/net.d |
| --docker-endpoint string | unix:///var/run/docker.sock |
| --eviction-hard string | memory.available<4Gi,<br/> nodefs.available<20Gi,<br/> imagefs.available<5Gi |
| --eviction-max-pod-grace-period int32 | 30 |
| --eviction-minimum-reclaim string | memory.available=500Mi,<br/> nodefs.available=2Gi,,<br/> imagefs.available=2Gi |
| --eviction-pressure-transition-periodduration | 5m0s |
| --eviction-soft string | memory.available<8Gi,<br/> nodefs.available<100Gi,<br/> imagefs.available<20Gi |
| --eviction-soft-grace-period string | memory.available=30s,<br/> nodefs.available=2m,<br/> imagefs.available=2m |
| --experimental-fail-swap-on | + |
| --experimental-kernel-memcg-notification | + |
| --feature-gates string | AllAlpha=false |
| --file-check-frequency duration | 20s |
| --hairpin-mode string | promiscuous-bridge |
| --healthz-port int32 | 10248 |
| --image-gc-high-threshold int32 | 60 |
| --image-gc-low-threshold int32 | 40 |
| --image-pull-progress-deadline duration | 2m0s |
| --kube-api-qps int32 | 5 |
| --kube-reserved mapStringString | cpu=200m,memory=16G |
| --kubeconfig string | /var/lib/kubelet/kubeconfig |
| --max-pods int32 | 50 |
| --minimum-image-ttl-duration duration | 1h |
| --network-plugin string | cni |
| --pod-infra-container-image string | vivo.registry.com/google_containers/pause-amd64:3.0 |
| --pod-manifest-path string | /var/lib/kubelet/pod_manifest |
| --port int32 | 10250 |
| --protect-kernel-defaults | + |
| --read-only-port int32 | 10255 |
| --require-kubeconfig | + |
| --root-dir string | /var/lib/kubelet |
| --runtime-request-timeout duration | 2m0s |
| --serialize-image-pulls | false |
| --sync-frequency duration | 1m0s |
| --system-reserved mapStringString | cpu=100m,memory=32G |
| --volume-plugin-dir string | /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ |
| --volume-stats-agg-period duration | 1m0s |
下面是我最终梳理的，认为需要真正显示设置的flag，如下：
/usr/bin/kubelet —address=0.0.0.0 --port=10250 --allow-privileged=false --cluster-dns=10.0.0.1 --cluster-domain=caas.vivo.com --max-pods=50 --network-plugin=cni --require-kubeconfig --pod-manifest-path=/etc/kubelet.d/ --pod-infra-container-image=vivo.registry.com/google_containers/pause-amd64:3.0 --eviction-hard=memory.available<4Gi,nodefs.available<20Gi,imagefs.available<5Gi --eviction-max-pod-grace-period=30 --eviction-minimum-reclaim=memory.available=500Mi,nodefs.available=2Gi,imagefs.available=2Gi --eviction-pressure-transition-period=5m0s --eviction-soft=memory.available<8Gi,nodefs.available<100Gi,imagefs.available<20Gi --eviction-soft-grace-period=memory.available=30s,nodefs.available=2m,imagefs.available=2m --experimental-kernel-memcg-notification --experimental-fail-swap-on --system-reserved=cpu=100m,memory=8G --kube-reserved=cpu=200m,memory=16G --hairpin-mode=promiscuous-bridge --image-gc-high-threshold=60 --image-gc-low-threshold=40 --serialize-image-pulls=false --protect-kernel-defaults --feature-gates=AllAlpha=false