k8s, etcd 多節點集群部署問題排查記錄

目錄

文章目錄

• 目錄
• 部署環境
• 1. etcd 集群啟動失敗
• 解決
• 2. etcd 健康狀態檢查失敗
• 解決
• 3. kube-apiserver 啟動失敗
• 解決
• 4. kubelet 啟動失敗
• 解決
• 5. Approved CSR 后獲取 nodes 失敗
• 解決
• 6. 訪問 pod app 失敗
• 解決

部署環境

雙節點 IP 配置：

# cat /etc/hosts 

192.168.1.5 vmnote0 
192.168.1.12 vmnote1

部署文檔：https://jimmysong.io/kubernetes-handbook/practice/install-kubernetes-on-centos.html

1. etcd 集群啟動失敗

etcd 雙節點配置之一：

[root@k8s-master ~]# cat /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/local/bin/etcd \
  --name vmnode0 \
  --cert-file=/etc/kubernetes/ssl/kubernetes.pem \
  --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
  --peer-cert-file=/etc/kubernetes/ssl/kubernetes.pem \
  --peer-key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
  --trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
  --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
  --initial-advertise-peer-urls https://192.168.1.5:2380 \
  --listen-peer-urls https://192.168.1.5:2380 \
  --listen-client-urls https://192.168.1.5:2379,http://127.0.0.1:2379 \
  --advertise-client-urls https://192.168.1.5:2379 \
  --initial-cluster-token etcd-cluster-1 \
  --initial-cluster vmnode0=https://192.168.1.5:2380,vmnode1=https://192.168.1.12:2380 \
  --initial-cluster-state new \
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

etcd 雙節點配置之二：

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/local/bin/etcd \
  --name vmnode1 \
  --cert-file=/etc/kubernetes/ssl/kubernetes.pem \
  --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
  --peer-cert-file=/etc/kubernetes/ssl/kubernetes.pem \
  --peer-key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
  --trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
  --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
  --initial-advertise-peer-urls https://192.168.1.12:2380 \
  --listen-peer-urls https://192.168.1.12:2380 \
  --listen-client-urls https://192.168.1.12:2379,http://127.0.0.1:2379 \
  --advertise-client-urls https://192.168.1.12:2379 \
  --initial-cluster-token etcd-cluster-1 \
  --initial-cluster vmnode0=https://192.168.1.5:2380,vmnode1=https://192.168.1.12:2380 \
  --initial-cluster-state new \
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

問題：

Dec 14 12:30:30 k8s-node2.localdomain etcd[2560]: warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
Dec 14 12:30:30 k8s-node2.localdomain etcd[2560]: member 218d8bfb33a29c6 has already been bootstrapped
Dec 14 12:30:30 k8s-node2.localdomain systemd[1]: etcd.service: main process exited, code=exited, status=1/FAILURE
Dec 14 12:30:30 k8s-node2.localdomain systemd[1]: Failed to start Etcd Server.

解決

主要問題在 member 218d8bfb33a29c6 has already been bootstrapped，原因：

One of the member was bootstrapped via discovery service. You must remove the previous data-dir to clean up the member information. Or the member will ignore the new configuration and start with the old configuration. That is why you see the mismatch.

2. etcd 健康狀態檢查失敗

[root@k8s-master ~]# etcdctl \
>   --ca-file=/etc/kubernetes/ssl/ca.pem \
>   --cert-file=/etc/kubernetes/ssl/kubernetes.pem \
>   --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
>   cluster-health
2018-12-14 13:13:15.280712 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
2018-12-14 13:13:15.281964 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
failed to check the health of member 218d8bfb33a29c6 on https://192.168.1.12:2379: Get https://192.168.1.12:2379/health: net/http: TLS handshake timeout
member 218d8bfb33a29c6 is unreachable: [https://192.168.1.12:2379] are all unreachable
failed to check the health of member 499bc1bc6765950c on https://192.168.1.5:2379: Get https://192.168.1.5:2379/health: net/http: TLS handshake timeout
member 499bc1bc6765950c is unreachable: [https://192.168.1.5:2379] are all unreachable
cluster is unhealthy

解決

主要原因：https://192.168.1.12:2379/health: net/http: TLS handshake timeout，等待超時，一般來說可能是因為設置了 http_proxy/https_proxy 做代理翻牆導致不能訪問自身和內網。 所以配置 no_proxy：

PROXY_HOST=127.0.0.1
export all_proxy=http://$PROXY_HOST:8118
export ftp_proxy=http://$PROXY_HOST:8118
export http_proxy=http://$PROXY_HOST:8118
export https_proxy=http://$PROXY_HOST:8118
export no_proxy='localhost,192.168.1.5,192.168.1.12'

3. kube-apiserver 啟動失敗

[root@k8s-master kubernetes]# systemctl status kube-apiserver
● kube-apiserver.service - Kubernetes API Service
   Loaded: loaded (/usr/lib/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Sat 2018-12-15 03:07:12 UTC; 4s ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes
  Process: 3659 ExecStart=/usr/local/bin/kube-apiserver $KUBE_LOGTOSTDERR $KUBE_LOG_LEVEL $KUBE_ETCD_SERVERS $KUBE_API_ADDRESS $KUBE_API_PORT $KUBELET_PORT $KUBE_ALLOW_PRIV $KUBE_SERVICE_ADDRESSES $KUBE_ADMISSION_CONTROL $KUBE_API_ARGS (code=exited, status=1/FAILURE)
 Main PID: 3659 (code=exited, status=1/FAILURE)

Dec 15 03:07:11 k8s-master.localdomain systemd[1]: kube-apiserver.service: main process exited, code=exited, status=1/FAILURE
Dec 15 03:07:11 k8s-master.localdomain systemd[1]: Failed to start Kubernetes API Service.
Dec 15 03:07:11 k8s-master.localdomain systemd[1]: Unit kube-apiserver.service entered failed state.
Dec 15 03:07:11 k8s-master.localdomain systemd[1]: kube-apiserver.service failed.
Dec 15 03:07:12 k8s-master.localdomain systemd[1]: kube-apiserver.service holdoff time over, scheduling restart.
Dec 15 03:07:12 k8s-master.localdomain systemd[1]: start request repeated too quickly for kube-apiserver.service
Dec 15 03:07:12 k8s-master.localdomain systemd[1]: Failed to start Kubernetes API Service.
Dec 15 03:07:12 k8s-master.localdomain systemd[1]: Unit kube-apiserver.service entered failed state.
Dec 15 03:07:12 k8s-master.localdomain systemd[1]: kube-apiserver.service failed.

看不出什么問題，回過頭來看 service 的 systemd 配置：

[Unit]
Description=Kubernetes API Service
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
After=etcd.service

[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/apiserver
ExecStart=/usr/local/bin/kube-apiserver \
        $KUBE_LOGTOSTDERR \
        $KUBE_LOG_LEVEL \
        $KUBE_ETCD_SERVERS \
        $KUBE_API_ADDRESS \
        $KUBE_API_PORT \
        $KUBELET_PORT \
        $KUBE_ALLOW_PRIV \
        $KUBE_SERVICE_ADDRESSES \
        $KUBE_ADMISSION_CONTROL \
        $KUBE_API_ARGS
Restart=on-failure
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

手動拼裝出 ExecStart 來執行：

[root@k8s-master ~]# /usr/local/bin/kube-apiserver --logtostderr=true --v=0 --advertise-address=192.168.1.5 --bind-address=192.168.1.5 --insecure-bind-address=192.168.1.5 --insecure-port=8080 --insecure-bind-address=127.0.0.1 --etcd-servers=https://192.168.1.5:2379,https://192.168.1.12:2379 --port=8080 --kubelet-port=10250 --allow-privileged=true --service-cluster-ip-range=10.254.0.0/16 --admission-control=ServiceAccount,NamespaceLifecycle,NamespaceExists,LimitRanger,ResourceQuota --authorization-mode=RBAC --runtime-config=rbac.authorization.k8s.io/v1beta1 --kubelet-https=true --experimental-bootstrap-token-auth --token-auth-file=/etc/kubernetes/token.csv --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem --client-ca-file=/etc/kubernetes/ssl/ca.pem --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem --etcd-cafile=/etc/kubernetes/ssl/ca.pem --etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem --etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem --enable-swagger-ui=true --apiserver-count=3 --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-path=/var/lib/audit.log --event-ttl=1h
/usr/local/bin/kube-apiserver: line 30: /lib/lsb/init-functions: No such file or directory

問題出現了：/usr/local/bin/kube-apiserver: line 30: /lib/lsb/init-functions: No such file or directory。

安裝 redhat-lsb 包：

yum install redhat-lsb -y

執行還是錯誤：

[root@k8s-master ~]# /usr/local/bin/kube-apiserver --logtostderr=true --v=0 --advertise-address=192.168.1.5 --bind-address=192.168.1.5 --insecure-bind-address=192.168.1.5 --insecure-port=8080 --insecure-bind-address=127.0.0.1 --etcd-servers=https://192.168.1.5:2379,https://192.168.1.12:2379 --port=8080 --kubelet-port=10250 --allow-privileged=true --service-cluster-ip-range=10.254.0.0/16 --admission-control=ServiceAccount,NamespaceLifecycle,NamespaceExists,LimitRanger,ResourceQuota --authorization-mode=RBAC --runtime-config=rbac.authorization.k8s.io/v1beta1 --kubelet-https=true --experimental-bootstrap-token-auth --token-auth-file=/etc/kubernetes/token.csv --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem --client-ca-file=/etc/kubernetes/ssl/ca.pem --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem --etcd-cafile=/etc/kubernetes/ssl/ca.pem --etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem --etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem --enable-swagger-ui=true --apiserver-count=3 --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-path=/var/lib/audit.log --event-ttl=1h
/opt/bin/kube-apiserver not present or not executable      [FAILED]

還是錯誤，這里進入了一個無限循環的大坑。我在 CentOS 上拉了一個 Ubuntu 的 kube-apiserver 執行文件。所以很多依賴程序並不存在，最后決定重新拉取執行文件。

解決

$ wget https://dl.k8s.io/v1.6.0/kubernetes-server-linux-amd64.tar.gz
$ tar -xzvf kubernetes-server-linux-amd64.tar.gz
$ cd kubernetes
$ tar -xzvf  kubernetes-src.tar.gz

$ cp -r server/bin/{kube-apiserver,kube-controller-manager,kube-scheduler,kubectl,kube-proxy,kubelet} /usr/local/bin/

4. kubelet 啟動失敗

[root@k8s-master opt]# systemctl status kubelet
● kubelet.service - Kubernetes Kubelet Server
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Sat 2018-12-15 08:09:53 UTC; 1s ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes
  Process: 12585 ExecStart=/usr/local/bin/kubelet $KUBE_LOGTOSTDERR $KUBE_LOG_LEVEL $KUBELET_API_SERVER $KUBELET_ADDRESS $KUBELET_PORT $KUBELET_HOSTNAME $KUBE_ALLOW_PRIV $KUBELET_POD_INFRA_CONTAINER $KUBELET_ARGS (code=exited, status=200/CHDIR)
 Main PID: 12585 (code=exited, status=200/CHDIR)

Dec 15 08:09:53 k8s-master.localdomain systemd[1]: Unit kubelet.service entered failed state.
Dec 15 08:09:53 k8s-master.localdomain systemd[1]: kubelet.service failed.
Dec 15 08:09:53 k8s-master.localdomain systemd[1]: kubelet.service holdoff time over, scheduling restart.
Dec 15 08:09:53 k8s-master.localdomain systemd[1]: Stopped Kubernetes Kubelet Server.
Dec 15 08:09:53 k8s-master.localdomain systemd[1]: start request repeated too quickly for kubelet.service
Dec 15 08:09:53 k8s-master.localdomain systemd[1]: Failed to start Kubernetes Kubelet Server.
Dec 15 08:09:53 k8s-master.localdomain systemd[1]: Unit kubelet.service entered failed state.
Dec 15 08:09:53 k8s-master.localdomain systemd[1]: kubelet.service failed.

監控日志輸出：

Dec 15 08:25:30 k8s-master.localdomain systemd[1]: Started Kubernetes Kubelet Server.
Dec 15 08:25:30 k8s-master.localdomain systemd[1]: kubelet.service: main process exited, code=exited, status=200/CHDIR
Dec 15 08:25:30 k8s-master.localdomain systemd[1]: Unit kubelet.service entered failed state.
Dec 15 08:25:30 k8s-master.localdomain systemd[1]: kubelet.service failed.
Dec 15 08:25:30 k8s-master.localdomain systemd[1]: kubelet.service holdoff time over, scheduling restart.
Dec 15 08:25:30 k8s-master.localdomain systemd[1]: Stopped Kubernetes Kubelet Server.
Dec 15 08:25:30 k8s-master.localdomain systemd[1]: Started Kubernetes Kubelet Server.
Dec 15 08:25:30 k8s-master.localdomain systemd[13491]: Failed at step CHDIR spawning /usr/local/bin/kubelet: No such file or directory
Dec 15 08:25:30 k8s-master.localdomain systemd[1]: kubelet.service: main process exited, code=exited, status=200/CHDIR
Dec 15 08:25:30 k8s-master.localdomain systemd[1]: Unit kubelet.service entered failed state.
Dec 15 08:25:30 k8s-master.localdomain systemd[1]: kubelet.service failed.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: kubelet.service holdoff time over, scheduling restart.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: Stopped Kubernetes Kubelet Server.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: Started Kubernetes Kubelet Server.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: kubelet.service: main process exited, code=exited, status=200/CHDIR
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: Unit kubelet.service entered failed state.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: kubelet.service failed.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: kubelet.service holdoff time over, scheduling restart.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: Stopped Kubernetes Kubelet Server.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: Started Kubernetes Kubelet Server.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: kubelet.service: main process exited, code=exited, status=200/CHDIR
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: Unit kubelet.service entered failed state.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: kubelet.service failed.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: kubelet.service holdoff time over, scheduling restart.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: Stopped Kubernetes Kubelet Server.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: Started Kubernetes Kubelet Server.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: kubelet.service: main process exited, code=exited, status=200/CHDIR
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: Unit kubelet.service entered failed state.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: kubelet.service failed.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: kubelet.service holdoff time over, scheduling restart.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: Stopped Kubernetes Kubelet Server.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: start request repeated too quickly for kubelet.service
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: Failed to start Kubernetes Kubelet Server.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: Unit kubelet.service entered failed state.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: kubelet.service failed.

Failed at step CHDIR spawning /usr/local/bin/kubelet: No such file or directory 非常可以，查看這個文件的路徑是否正確。奇怪的是這個文件路徑是正確的，於是繼續看 systemd 配置文件：

[Unit]
Description=Kubernetes Kubelet Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory=/var/lib/kubelet
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/kubelet
ExecStart=/usr/local/bin/kubelet \
            $KUBE_LOGTOSTDERR \
            $KUBE_LOG_LEVEL \
            $KUBELET_API_SERVER \
            $KUBELET_ADDRESS \
            $KUBELET_PORT \
            $KUBELET_HOSTNAME \
            $KUBE_ALLOW_PRIV \
            $KUBELET_POD_INFRA_CONTAINER \
            $KUBELET_ARGS
Restart=on-failure

[Install]
WantedBy=multi-user.target

文件路徑還有 /var/lib/kubelet，查看一番，的確沒有。

解決

mkdir /var/lib/kubelet

5. Approved CSR 后獲取 nodes 失敗

[root@k8s-master ~]# kubectl get csr
NAME        AGE       REQUESTOR           CONDITION
csr-bv37w   19m       kubelet-bootstrap   Approved
csr-bwlxd   1m        kubelet-bootstrap   Approved
csr-f8w38   33m       kubelet-bootstrap   Approved
csr-g8927   47m       kubelet-bootstrap   Approved
csr-h7wph   4m        kubelet-bootstrap   Approved
csr-hpl81   50m       kubelet-bootstrap   Approved
csr-qxxsh   40m       kubelet-bootstrap   Approved
csr-r7vzl   51m       kubelet-bootstrap   Approved
csr-w1ccb   21m       kubelet-bootstrap   Approved
[root@k8s-master ~]# kubectl get nodes
No resources found.

查看日志：

[root@k8s-master ~]# journalctl -xe -u kube* | grep error
Dec 15 08:50:15 k8s-master.localdomain kube-scheduler[14917]: E1215 08:50:15.200937   14917 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: Get http://192.168.1.5:8080/api/v1/namespaces/kube-system/endpoints/kube-scheduler: dial tcp 192.168.1.5:8080: getsockopt: connection refused

kube-controller-manager 訪問 kube-apiserver 的 192.168.1.5:8080 失敗了，那么查看一下端口信息：

[root@k8s-master ~]# netstat -lpntu
...
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      14885/kube-apiserve

8080 端口綁定的 ip 是 127.0.0.1 而不是 192.168.1.5，可能是配置文件有問題？查看一下：

KUBE_API_ADDRESS="--advertise-address=192.168.1.5 --bind-address=192.168.1.5 --insecure-bind-address=192.168.1.5 --insecure-port=8080 --insecure-bind-address=127.0.0.1"

的確綁定到了 127.0.0.1。

解決

修改 /etc/kubernetes/apiserver 配置文件：

KUBE_API_ADDRESS="--advertise-address=192.168.1.5 --bind-address=192.168.1.5 --insecure-bind-address=192.168.1.5 --insecure-port=8080 --insecure-bind-address=192.168.1.5"

重啟，解決。

[root@k8s-master kubernetes]# kubectl get csr
NAME        AGE       REQUESTOR           CONDITION
csr-96lj4   37s       kubelet-bootstrap   Approved,Issued
csr-bv37w   23m       kubelet-bootstrap   Approved,Issued
csr-bwlxd   5m        kubelet-bootstrap   Approved,Issued
csr-dpqgm   37s       kubelet-bootstrap   Approved,Issued
csr-f8w38   37m       kubelet-bootstrap   Approved,Issued
csr-g8927   52m       kubelet-bootstrap   Approved,Issued
csr-h7wph   9m        kubelet-bootstrap   Approved,Issued
csr-hpl81   55m       kubelet-bootstrap   Approved,Issued
csr-qxxsh   44m       kubelet-bootstrap   Approved,Issued
csr-r7vzl   56m       kubelet-bootstrap   Approved,Issued
csr-w1ccb   26m       kubelet-bootstrap   Approved,Issued
[root@k8s-master kubernetes]# kubectl get nodes
NAME           STATUS     AGE       VERSION
192.168.1.12   NotReady   10s       v1.6.0
192.168.1.5    Ready      7s        v1.6.0

6. 訪問 pod app 失敗

root@k8s-master ~]# kubectl run nginx --replicas=2 --labels="run=load-balancer-example" --image=nginx  --port=80
deployment "nginx" created

[root@k8s-master ~]# kubectl expose deployment nginx --type=NodePort --name=example-service
service "example-service" exposed

[root@k8s-master ~]# kubectl get deployment
NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
nginx     2         2         2            0           27s

[root@k8s-master ~]# kubectl describe svc example-service
Name:			example-service
Namespace:		default
Labels:			run=load-balancer-example
Annotations:		<none>
Selector:		run=load-balancer-example
Type:			NodePort
IP:			10.254.109.60
Port:			<unset>	80/TCP
NodePort:		<unset>	30019/TCP
Endpoints:		172.17.0.2:80
Session Affinity:	None
Events:			<none>

[root@k8s-master ~]# curl "10.254.109.60:80"
^C

[root@k8s-master ~]# curl "172.17.0.2:80"
^C

解決

第一反應，可能還是 no_proxy 代理的問題，試一試。

export no_proxy='localhost,192.168.1.5,192.168.1.12,10.254.109.60,172.17.0.2'

再試一次就可以了。

[root@k8s-master kubernetes]# source /etc/profile

[root@k8s-master kubernetes]# curl "10.254.109.60:80"
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

[root@k8s-master ~]# curl "172.17.0.2:80"
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>