kukubeadm 1.6.1 + docker1.2.6 安裝問題

 

 kubeadm init --apiserver-advertise-address=192.168.20.229 --pod-network-cidr=10.244.0.0/16

kubelet: error: failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "cgroupfs" is different from docker cgroup driver: "systemd"

docker相比1.10增加了KernelMemory變量和CgroupDriver變量，KernelMemory變量表示是否設置linux內核內存限制，CgroupDriver變量表示使用哪個Cgroup驅動，有兩種驅動，分別是cgroupfs和systemd，默認使用cgroupfs

 

 

由 systemd 變更成 cgroupfs

############################################

或者  --cgroup-driver=systemd \
kubelet的服務配置文件加上這么一行

 

 

使用kubeadm 安裝 kubernetes1.6.1

 環境准備

master 192.168.20.229
node    192.168.20.223

 

軟件版本：

docker使用 1.12.6

 

查看版本

yum list kubeadm  --showduplicates |sort -r
kubeadm.x86_64                        1.6.1-0                        kubernetes
kubeadm.x86_64                        1.6.0-0                        kubernetes

yum list kubelet  --showduplicates |sort -r
kubelet.x86_64                        1.6.1-0                        kubernetes
kubelet.x86_64                        1.6.0-0                         kubernetes
kubelet.x86_64                        1.5.4-0                         kubernetes

yum list kubectl  --showduplicates |sort -r
kubectl.x86_64                        1.6.1-0                        kubernetes
kubectl.x86_64                        1.6.0-0                         kubernetes
kubectl.x86_64                        1.5.4-0                         kubernetes

yum list kubernets-cni  --showduplicates |sort -r
kubernetes-cni              x86_64              0.5.1-0                    kubernetes

 

系統配置

 根據官方文檔Installing Kubernetes on Linux with kubeadm中的Limitations小節中的內容，對各節點系統做如下設置:

創建/etc/sysctl.d/k8s.conf文件，添加如下內容：

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1

 

初始化集群

kubeadm init --kubernetes-version=v1.6.1 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.20.229

kubeadm init執行成功后輸出下面的信息：

kubeadm init --kubernetes-version=v1.6.1 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.61.41
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[init] Using Kubernetes version: v1.6.1
[init] Using Authorization mode: RBAC
[preflight] Running pre-flight checks
[preflight] Starting the kubelet service
[certificates] Generated CA certificate and key.
[certificates] Generated API server certificate and key.
[certificates] API Server serving cert is signed for DNS names [node0 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.61.41]
[certificates] Generated API server kubelet client certificate and key.
[certificates] Generated service account token signing key and public key.
[certificates] Generated front-proxy CA certificate and key.
[certificates] Generated front-proxy client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[apiclient] Created API client, waiting for the control plane to become ready
[apiclient] All control plane components are healthy after 14.583864 seconds
[apiclient] Waiting for at least one node to register
[apiclient] First node has registered after 6.008990 seconds
[token] Using token: e7986d.e440de5882342711
[apiconfig] Created RBAC rules
[addons] Created essential addon: kube-proxy
[addons] Created essential addon: kube-dns
Your Kubernetes master has initialized successfully!
To start using your cluster, you need to run (as a regular user):
  sudo cp /etc/kubernetes/admin.conf $HOME/
  sudo chown $(id -u):$(id -g) $HOME/admin.conf
  export KUBECONFIG=$HOME/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  http://kubernetes.io/docs/admin/addons/
You can now join any number of machines by running the following on each node
as root:

kubeadm join --token 881f96.aaf02f1f8dc53889 192.168.20.229:6443

 

Master Node初始化完成，使用kubeadm初始化的Kubernetes集群在Master節點上的核心組件：kube-apiserver,kube-scheduler, kube-controller-manager是以靜態Pod的形式運行的。

ls /etc/kubernetes/manifests/
etcd.yaml  kube-apiserver.yaml  kube-controller-manager.yaml  kube-scheduler.yaml

 

在/etc/kubernetes/manifests/目錄里可以看到kube-apiserver,kube-scheduler, kube-controller-manager的定義文件。另外集群持久化存儲etcd也是以單點靜態Pod的形式運行的，對於etcd后邊我們會把它切換成etcd集群。

查看一下kube-apiserver.yaml的內容：

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    .......
    - --insecure-port=0

 

注意到kube-apiserver的選項--insecure-port=0，也就是說kubeadm 1.6.0初始化的集群，kube-apiserver沒有監聽默認的http 8080端口。

所以我們使用kubectl get nodes會報The connection to the server localhost:8080 was refused - did you specify the right host or port?。

 

查看kube-apiserver的監聽端口可以看到只監聽了https的6443端口

netstat -nltp | grep apiserver
tcp6       0      0 :::6443                 :::*                    LISTEN      9831/kube-apiserver

 

為了使用kubectl訪問apiserver，在~/.bash_profile中追加下面的環境變量：

export KUBECONFIG=/etc/kubernetes/admin.conf

source ~/.bash_profile

 

此時kubectl命令在master node上就好用了，查看一下當前機器中的Node：

kubectl get nodes
NAME      STATUS     AGE       VERSION
k8s1       NotReady   3m        v1.6.1

 

安裝Pod Network

接下來安裝flannel network add-on：

kubectl create -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel-rbac.yml
kubectl apply -f  https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
serviceaccount "flannel" created
configmap "kube-flannel-cfg" created
daemonset "kube-flannel-ds" created

 

如果Node有多個網卡的話，參考flannel issues 39701，目前需要在kube-flannel.yml中使用--iface參數指定集群主機內網網卡的名稱，否則可能會出現dns無法解析。需要將kube-flannel.yml下載到本地，flanneld啟動參數加上--iface=<iface-name>

......
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: kube-flannel-ds
......
containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel:v0.7.0-amd64
        command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr", "--iface=eth1" ]
......

 

使用kubectl get pod --all-namespaces -o wide確保所有的Pod都處於Running狀態

kubectl get pod --all-namespaces -o wide

或者

kubectl --kubeconfig=/etc/kubernetes/admin.conf get pod  --all-namespaces -o wide

NAMESPACE     NAME                           READY     STATUS    RESTARTS   AGE       IP               NODE
kube-system   etcd-k8s1                      1/1       Running   0          10m       192.168.20.229   k8s1
kube-system   kube-apiserver-k8s1            1/1       Running   0          10m       192.168.20.229   k8s1
kube-system   kube-controller-manager-k8s1   1/1       Running   0          10m       192.168.20.229   k8s1
kube-system   kube-dns-3913472980-g97bm      3/3       Running   0          10m       10.244.1.2       k8s5
kube-system   kube-flannel-ds-k87tt          2/2       Running   0          2m        192.168.20.233   k8s5
kube-system   kube-flannel-ds-lq62q          2/2       Running   0          2m        192.168.20.229   k8s1
kube-system   kube-proxy-0nrp0               1/1       Running   0          10m       192.168.20.229   k8s1
kube-system   kube-proxy-qcds5               1/1       Running   0          10m       192.168.20.233   k8s5
kube-system   kube-scheduler-k8s1            1/1       Running   0          10m       192.168.20.229   k8s1

 

使master node參與工作負載

使用kubeadm初始化的集群，出於安全考慮Pod不會被調度到Master Node上，也就是說Master Node不參與工作負載。

這里搭建的是測試環境可以使用下面的命令使Master Node參與工作負載：

kubectl taint nodes --all  node-role.kubernetes.io/master-

 

測試DNS

[root@k8s1 ~]# kubectl --kubeconfig=/etc/kubernetes/admin.conf run curl --image=radial/busyboxplus:curl -i --tty
If you don't see a command prompt, try pressing enter.
[ root@curl-57077659-s2l5v:/ ]$ nslookup
BusyBox v1.22.1 (2014-09-13 22:15:30 PDT) multi-call binary.

Usage: nslookup [HOST] [SERVER]

Query the nameserver for the IP address of the given HOST
optionally using a specified DNS server

[ root@curl-57077659-s2l5v:/ ]$ nslookup kube-dns.kube-system
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kube-dns.kube-system
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
[ root@curl-57077659-s2l5v:/ ]$ nslookup kubernetes.default
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.default
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local

 

測試OK后，刪除掉curl這個Pod。

kubectl delete deploy curl

 

向集群中添加節點

kubeadm join --token 881f96.aaf02f1f8dc53889 192.168.20.229:6443

 

查看集群中節點：

[root@k8s1 ~]# kubectl --kubeconfig=/etc/kubernetes/admin.conf get nodes
NAME      STATUS    AGE       VERSION
k8s1      Ready     54m       v1.6.1
k8s5      Ready     54m       v1.6.1

 

 

安裝Dashboard插件

 

wget https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/kubernetes-dashboard.yaml

kubectl create -f kubernetes-dashboard.yaml

 

從http://NodeIp:NodePort訪問dashboard，瀏覽器顯示下面的錯誤

User "system:serviceaccount:kube-system:default" cannot list statefulsets.apps in the namespace "default". (get statefulsets.apps)



這是因為Kubernetes 1.6開始API Server啟用了RBAC授權，當前的kubernetes-dashboard.yaml沒有定義授權的ServiceAccount，所以訪問API Server時被拒絕了。

根據https://github.com/kubernetes/dashboard/issues/1803中的內容臨時授予system:serviceaccount:kube-system:default cluster_admin的角色，臨時解決一下。

創建dashboard-rbac.yaml，定義system:serviceaccount:kube-system:default和ClusterRole cluster-admin綁定：

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: dashboard-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin 
subjects:
- kind: ServiceAccount
  name: default
  namespace: kube-system

kubectl  --kubeconfig=/etc/kubernetes/admin.conf create -f dashboard-rbac.yml

 

在集群中運行Heapster

下面安裝Heapster為集群添加使用統計和監控功能，為Dashboard添加儀表盤。

下載最新的Heapster到集群中的某個Node上

wget https://github.com/kubernetes/heapster/archive/v1.3.0.tar.gz

 

使用InfluxDB做為Heapster的后端存儲，開始部署，中間會pull相關鏡像，包含gcr.io/google_containers/heapster_grafana:v2.6.0-2

tar -zxvf v1.3.0.tar.gz
cd heapster-1.3.0/deploy/kube-config/influxdb 

 

添加了RBAC授權

[root@k8s1 influxdb]# cat heapster-rbac.yaml 
apiVersion: v1
kind: ServiceAccount
metadata:
  name: heapster
  namespace: kube-system

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1alpha1
metadata:
  name: heapster
subjects:
  - kind: ServiceAccount
    name: heapster
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: system:heapster
  apiGroup: rbac.authorization.k8s.io

[root@k8s1 influxdb]# vim heapster-deployment.yaml 

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: heapster
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        task: monitoring
        k8s-app: heapster
    spec:
      serviceAccountName: heapster
      containers:
      - name: heapster
        image: gcr.io/google_containers/heapster-amd64:v1.3.0-beta.1
        imagePullPolicy: IfNotPresent
        command:
        - /heapster
        - --source=kubernetes:https://kubernetes.default
        - --sink=influxdb:http://monitoring-influxdb:8086

 

 

 

 

參考

http://blog.frognew.com/2017/04/kubeadm-install-kubernetes-1.6.html

https://github.com/opsnull/follow-me-install-kubernetes-cluster/blob/master/10-%E9%83%A8%E7%BD%B2Heapster%E6%8F%92%E4%BB%B6.md