二進制方式部署K8S（kubernetes)集群（測試、學習環境）-單主雙從

1. 二進制方式部署（一主多從）

1.1 環境准備

角色	IP	組件
master	10.27.134.250	kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubectl 、kubelet 、kube-proxy
node1	10.27.134.251	docker、etcd、kubelet 、kube-proxy
node2	10.27.134.252	docker、etcd、kubelet 、kube-proxy

 

##版本說明

Linux：CentOS Linux release 7.6.1810 (Core)

Container runtimes : docker-20.10.9

Etcd : v3.5.1

Kubernetes : v1.23.1

簽名工具：

cfssl

cfssljson

cfssl-certinfo

 

1.2 系統初始化（所有節點均需操作）

1.2.1 主機名解析

##修改主機名

[root@jss-k8s-1 ~]# hostnamectl set-hostname master  #master節點

[root@jss-k8s-2 ~]# hostnamectl set-hostname node1   #node1節點

[root@jss-k8s-3 ~]# hostnamectl set-hostname node2   #node2節點

 

##三台服務器的地址和主機名添加到三個節點的/etc/hosts文件中

[root@master ~]# cat >> /etc/hosts << EOF

10.27.134.250 master

10.27.134.251 node1

10.27.134.252 node2

EOF

 

1.2.2 時間同步

##kubernetes要求集群中的節點時間必須精確一直，這里使用chronyd服務從網絡同步時間

# 啟動chronyd服務，設置開機啟動

# systemctl start chronyd && systemctl enable chronyd

 

# 核驗時間

# date

 

1.2.3 禁用iptable和firewalld服務

##kubernetes和docker 在運行的中會產生大量的iptables規則，為了不讓系統規則跟它們混淆，直接關閉系統的規則

##關閉firewalld服務

# systemctl stop firewalld && systemctl disable firewalld

 

##關閉iptables服務

# systemctl stop iptables && systemctl disable iptables

 

1.2.4 禁用selinux

##selinux是linux系統下的一個安全服務，如果不關閉它，在安裝集群中會產生各種各樣的奇葩問題

# 永久關閉

# sed -i 's/enforcing/disabled/' /etc/selinux/config

 

# 臨時關閉

# setenforce 0

 

1.2.5 禁用swap分區

##swap分區指的是虛擬內存分區，它的作用是物理內存使用完，之后將磁盤空間虛擬成內存來使用，啟用swap設備會對系統的性能產生非常負面的影響，因此kubernetes要求每個節點都要禁用swap設備，但是如果因為某些原因確實不能關閉swap分區，就需要在集群安裝過程中通過明確的參數進行配置說明

# 永久關閉

# vim /etc/fstab

 

 

# 臨時關閉

# swapoff -a

 

1.2.6 將橋接的 IPv4流量傳遞到 iptables的鏈上

##加載網橋過濾模塊

# cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf

br_netfilter

EOF

 

##修改內核參數

# cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf

net.bridge.bridge-nf-call-ip6tables = 1

net.bridge.bridge-nf-call-iptables = 1

EOF

 

##配置生效

# sysctl --system

 

##重啟系統，讓配置徹底生效

# reboot

 

1.3 部署二進制包

1.3.1 安裝docker

1.3.1.1 下載二進制包

操作對象：所有節點

##在master主機創建目錄存放二進制軟件包

# mkdir -p /usr/local/k8s

# cd /usr/local/k8s

 

##下載docker包，

# wget https://download.docker.com/linux/static/stable/x86_64/docker-20.10.9.tgz

 

##解壓

# tar zxvf docker-20.10.9.tgz

 

##復制到本地運行目錄

# mv docker/* /usr/bin

 

1.3.1.2 systemd管理docker

操作對象：所有節點

##創建docker.service腳本

# cat > /usr/lib/systemd/system/docker.service << EOF

[Unit]

Description=Docker Application Container Engine

Documentation=https://docs.docker.com

After=network-online.target firewalld.service

Wants=network-online.target

 

[Service]

Type=notify

ExecStart=/usr/bin/dockerd

ExecReload=/bin/kill -s HUP $MAINPID

LimitNOFILE=infinity

LimitNPROC=infinity

LimitCORE=infinity

TimeoutStartSec=0

Delegate=yes

KillMode=process

Restart=on-failure

StartLimitBurst=3

StartLimitInterval=60s

 

[Install]

WantedBy=multi-user.target

EOF

 

1.3.1.3 創建配置文件，配置鏡像加速

操作對象：所有節點

#在所有節點主機創建docker配置文件默認的目錄

# mkdir /etc/docker

 

# cat > /etc/docker/daemon.json<< EOF

{

"registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"],

"exec-opts": ["native.cgroupdriver=systemd"],

"log-driver": "json-file",

  "log-opts": {

    "max-size": "100m"

  },

  "storage-driver": "overlay2"

}

EOF

 

1.3.1.4 啟動docker並設置開機啟動

# systemctl daemon-reload && systemctl start docker && systemctl enable docker

 

1.3.2 部署etcd集群

1.3.2.1 下載cfssl

操作對象：master節點

# cd /usr/local/k8s

# cat > cfssl << EOF

https://pkg.cfssl.org/R1.2/cfssl_linux-amd64

https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64

https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64

EOF

 

# wget -i cfssl

 

#賦予可執行的權限

# chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64

 

# mv cfssl_linux-amd64 /usr/local/bin/cfssl

# mv cfssljson_linux-amd64 /usr/local/bin/cfssljson

# mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo

 

1.3.2.2 創建自簽證書頒發機構（CA）的配置文件

##創建證書生成的工作目錄（后面相關證書的生成都在此目錄下操作）

# mkdir -p ~/TLS/{etcd,k8s}

# cd ~/TLS/etcd

 

## 創建ca-config配置文件

# cat > ca-config.json << EOF

{

  "signing": {

    "default": {

      "expiry": "87600h"

    },

    "profiles": {

      "www": {

         "expiry": "87600h",

         "usages": [

            "signing",

            "key encipherment",

            "server auth",

            "client auth"

        ]

      }

    }

  }

}

EOF

 

##創建ca-csr配置文件

# cat > ca-csr.json << EOF

{

    "CN": "etcd CA",

    "key": {

        "algo": "rsa",

        "size": 2048

    },

    "names": [

        {

            "C": "CN",

            "L": "Beijing",

            "ST": "Beijing"

        }

    ]

}

EOF

 

1.3.2.3 生成證書

# cfssl gencert -initca ca-csr.json | cfssljson -bare ca –

 

##會生成ca.pem和ca-key.pem文件

# ls *pem

 

1.3.2.4 自簽CA簽發etcd HTTPS證書

##創建證書申請文件

# cat > server-csr.json << EOF

{

    "CN": "etcd",

    "hosts": [

    "10.27.134.250",

    "10.27.134.251",

    "10.27.134.252"

    ],

    "key": {

        "algo": "rsa",

        "size": 2048

    },

    "names": [

        {

            "C": "CN",

            "L": "BeiJing",

            "ST": "BeiJing"

        }

    ]

}

EOF

 

##生成HTTPS證書

# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

 

##會生成server.pem和server-key.pem文件

# ls server*pem

 

1.3.2.5 下載ETCD二進制包

# cd /usr/local/k8s

# wget https://github.com/etcd-io/etcd/releases/download/v3.5.1/etcd-v3.5.1-linux-amd64.tar.gz

 

##創建etcd的工作目錄

# mkdir -p /opt/etcd/{bin,cfg,ssl}

 

#解壓

# tar zxvf etcd-v3.5.1-linux-amd64.tar.gz

 

##將二進制文件復制到工作目錄

# cp etcd-v3.5.1-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/

 

1.3.2.6 創建etcd 配置文件

# cat > /opt/etcd/cfg/etcd.conf << EOF

#根據實際情況修改節點地址

#[Member]

ETCD_NAME="etcd-1"

ETCD_DATA_DIR="/var/lib/etcd/default.etcd"

ETCD_LISTEN_PEER_URLS="https://10.27.134.250:2380"

ETCD_LISTEN_CLIENT_URLS="https://10.27.134.250:2379"

#[Clustering]

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.27.134.250:2380"

ETCD_ADVERTISE_CLIENT_URLS="https://10.27.134.250:2379"

ETCD_INITIAL_CLUSTER="etcd-1=https://10.27.134.250:2380,etcd-2=https://10.27.134.251:2380,etcd-3=https://10.27.134.252:2380"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_ENABLE_V2="true" 

EOF

 

##解釋：

ETCD_NAME：節點名稱，集群中唯一

ETCD_DATA_DIR：數據目錄

ETCD_LISTEN_PEER_URLS：集群通信監聽地址

ETCD_LISTEN_CLIENT_URLS：客戶端訪問監聽地址

ETCD_INITIAL_ADVERTISE_PEERURLS：集群通告地址

ETCD_ADVERTISE_CLIENT_URLS：客戶端通告地址

ETCD_INITIAL_CLUSTER：集群節點地址

ETCD_INITIALCLUSTER_TOKEN：集群Token

ETCD_INITIALCLUSTER_STATE：加入集群的當前狀態，new是新集群，existing表示加入已有集群

 

1.3.2.7 systemd 管理etcd

##將之前生成的證書復制到etcd的工作目錄

# cp ~/TLS/etcd/ca*pem ~/TLS/etcd/server*pem /opt/etcd/ssl

 

# cat > /usr/lib/systemd/system/etcd.service << EOF

[Unit]

Description=Etcd Server

After=network.target

After=network-online.target

Wants=network-online.target

 

[Service]

Type=notify

EnvironmentFile=/opt/etcd/cfg/etcd.conf

ExecStart=/opt/etcd/bin/etcd \

--cert-file=/opt/etcd/ssl/server.pem \

--key-file=/opt/etcd/ssl/server-key.pem \

--peer-cert-file=/opt/etcd/ssl/server.pem \

--peer-key-file=/opt/etcd/ssl/server-key.pem \

--trusted-ca-file=/opt/etcd/ssl/ca.pem \

--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \

--logger=zap

Restart=on-failure

LimitNOFILE=65536

 

[Install]

WantedBy=multi-user.target

EOF

 

1.3.2.8 將配置文件、證書和etcd.service文件復制到其他節點

##node1

# scp -r /opt/etcd/ root@node1:/opt/

# scp /usr/lib/systemd/system/etcd.service root@node1:/usr/lib/systemd/system/

 

##node2

# scp -r /opt/etcd/ root@node2:/opt/

# scp /usr/lib/systemd/system/etcd.service root@node2:/usr/lib/systemd/system/

 

1.3.2.9 修改其他節點的etcd配置

##node1：修改etcd.conf 配置文件中的節點名稱和節點地址

# vim /opt/etcd/cfg/etcd.conf

 

##node2：修改etcd.conf 配置文件中的節點名稱和節點地址

# vim /opt/etcd/cfg/etcd.conf

 

1.3.2.10 啟動etcd集群

##啟動etcd服務並設置開機啟動，若在node上啟動失敗，可以先啟動master。

# systemctl daemon-reload && systemctl start etcd && systemctl enable etcd

 

##檢查集群狀態

# ETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://10.27.134.250:2379,https://10.27.134.251:2379,https://10.27.134.252:2379" endpoint health --write-out=table

 

1.3.3 部署Master Node

1.3.3.1 部署kube-apiserver

1.3.3.1.1 生成kube-apiserver證書

（1）創建自簽證書頒發機構（CA）配置

# cd ~/TLS/k8s

##創建ca-config配置文件

# cat > ca-config.json << EOF

{

  "signing": {

    "default": {

      "expiry": "87600h"

    },

    "profiles": {

      "kubernetes": {

         "expiry": "87600h",

         "usages": [

            "signing",

            "key encipherment",

            "server auth",

            "client auth"

        ]

      }

    }

  }

}

EOF

 

##創建ca-csr配置文件

# cat > ca-csr.json << EOF

{

    "CN": "kubernetes",

    "key": {

        "algo": "rsa",

        "size": 2048

    },

    "names": [

        {

            "C": "CN",

            "L": "Beijing",

            "ST": "Beijing",

            "O": "k8s",

            "OU": "System"

        }

    ]

}

EOF

 

（2） 生成證書

# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

#會生成ca.pem和ca-key.pem文件

# ls *pem

 

（3） 使用自簽證書簽發kube-apiserver HTTPS證書

##創建證書申請文件

# cat > server-csr.json << EOF

{

    "CN": "kubernetes",

    "hosts": [

      "10.0.0.1",

      "127.0.0.1",

      "10.27.134.250",

      "10.27.134.251",

      "10.27.134.252",

      "kubernetes",

      "kubernetes.default",

      "kubernetes.default.svc",

      "kubernetes.default.svc.cluster",

      "kubernetes.default.svc.cluster.local"

    ],

    "key": {

        "algo": "rsa",

        "size": 2048

    },

    "names": [

        {

            "C": "CN",

            "L": "BeiJing",

            "ST": "BeiJing",

            "O": "k8s",

            "OU": "System"

        }

    ]

}

EOF

 

（4）生成https證書

# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server

##會生成server.pem和server-key.pem文件

# ls server*pem

 

1.3.3.1.2 下載二進制包

# cd /usr/local/k8s/

 

# wget https://dl.k8s.io/v1.23.1/kubernetes-server-linux-amd64.tar.gz

 

# tar zxvf kubernetes-server-linux-amd64.tar.gz

 

##創建kubernetes工作目錄

# mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}

 

##從軟件包內拷貝組件

# cd /usr/local/k8s/kubernetes/server/bin

 

# cp kube-apiserver kube-scheduler kube-controller-manager /opt/kubernetes/bin

 

# cp kubectl /usr/bin/

 

1.3.3.1.3 生成kube-apiserver的token文件

# echo $(head -c 16 /dev/urandom | od -An -t x | tr -d ' '),kubelet-bootstrap,10001,"system:kubelet-bootstrap" > /opt/kubernetes/cfg/token.csv

 

1.3.3.1.4 創建配置文件

# cat > /opt/kubernetes/cfg/kube-apiserver.conf << EOF

KUBE_APISERVER_OPTS="--logtostderr=false \\

--v=2 \\

--log-dir=/opt/kubernetes/logs \\

--etcd-servers=https://10.27.134.250:2379,https://10.27.134.251:2379,https://10.27.134.252:2379 \\

--bind-address=10.27.134.250 \\

--secure-port=6443 \\

--advertise-address=10.27.134.250 \\

--allow-privileged=true \\

--service-cluster-ip-range=10.0.0.0/24 \\

--enable-admission-plugins=NodeRestriction \\

--authorization-mode=RBAC,Node \\

--enable-bootstrap-token-auth=true \\

--token-auth-file=/opt/kubernetes/cfg/token.csv \\

--service-node-port-range=30000-32767 \\

--kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \\

--kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \\

--tls-cert-file=/opt/kubernetes/ssl/server.pem  \\

--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \\

--client-ca-file=/opt/kubernetes/ssl/ca.pem \\

--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \\

--service-account-issuer=api \\

--service-account-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \\

--etcd-cafile=/opt/etcd/ssl/ca.pem \\

--etcd-certfile=/opt/etcd/ssl/server.pem \\

--etcd-keyfile=/opt/etcd/ssl/server-key.pem \\

--requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem \\

--proxy-client-cert-file=/opt/kubernetes/ssl/server.pem \\

--proxy-client-key-file=/opt/kubernetes/ssl/server-key.pem \\

--requestheader-allowed-names=kubernetes \\

--requestheader-extra-headers-prefix=X-Remote-Extra- \\

--requestheader-group-headers=X-Remote-Group \\

--requestheader-username-headers=X-Remote-User \\

--enable-aggregator-routing=true \\

--audit-log-maxage=30 \\

--audit-log-maxbackup=3 \\

--audit-log-maxsize=100 \\

--audit-log-path=/opt/kubernetes/logs/k8s-audit.log"

EOF

 

–logtostderr：啟用日志

––v：日志等級

–log-dir：日志目錄

–etcd-servers：etcd 集群地址

–bind-address：監聽地址

–secure-port：https 安全端口

–advertise-address：集群通告地址

–allow-privileged：啟用授權

–service-cluster-ip-range：Service 虛擬IP 地址段

–enable-admission-plugins：准入控制模塊

–authorization-mode：認證授權，啟用RBAC 授權和節點自管理

–enable-bootstrap-token-auth：啟用TLS bootstrap 機制

–token-auth-file：bootstrap token 文件

–service-node-port-range：Service nodeport 類型默認分配端口范圍

–kubelet-client-xxx：apiserver 訪問kubelet 客戶端證書

–tls-xxx-file：apiserver https 證書

–etcd-xxxfile：連接Etcd 集群證書

–audit-log-xxx：審計日志

–service-account-issuer=string 指定service account token issuer的標識符;該issuer將在iss聲明中分發Token以使標識符生效;該參數的值為字符串或URL

–service-account-signing-key-file=string含有當前service account token issuer私鑰的文件路徑 ;該issuer將以此私鑰簽名發布的ID token（需開啟TokenRequest特性）

 

1.3.3.1.5 拷貝生成的k8s證書

# cp ~/TLS/k8s/ca*pem ~/TLS/k8s/server*pem /opt/kubernetes/ssl

 

1.3.3.1.6 system管理apiserver

# cat > /usr/lib/systemd/system/kube-apiserver.service << EOF

[Unit]

Description=Kubernetes API Server

Documentation=https://github.com/kubernetes/kubernetes

 

[Service]

EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf

ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS

Restart=on-failure

 

[Install]

WantedBy=multi-user.target

EOF

 

1.3.3.1.7 啟動 kube-apiserver並設置開機啟動

# systemctl daemon-reload && systemctl start kube-apiserver && systemctl enable kube-apiserver

 

1.3.3.2 部署kube-controller-manager

1.3.3.2.1 創建證書申請文件

# cd ~/TLS/k8s

 

# cat > kube-controller-manager-csr.json << EOF

{

  "CN": "system:kube-controller-manager",

  "hosts": [],

  "key": {

    "algo": "rsa",

    "size": 2048

  },

  "names": [

    {

      "C": "CN",

      "L": "BeiJing",

      "ST": "BeiJing",

      "O": "system:masters",

      "OU": "System"

    }

  ]

}

EOF

 

1.3.3.2.2 生成證書

# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager

 

1.3.3.2.3 生成kube-controller-manager.kubeconfig配置文件

# KUBE_CONFIG="/opt/kubernetes/cfg/kube-controller-manager.kubeconfig"

# KUBE_APISERVER="https://10.27.134.250:6443"

 

# kubectl config set-cluster kubernetes \

  --certificate-authority=/opt/kubernetes/ssl/ca.pem \

  --embed-certs=true \

  --server=${KUBE_APISERVER} \

  --kubeconfig=${KUBE_CONFIG}

# kubectl config set-credentials kube-controller-manager \

  --client-certificate=./kube-controller-manager.pem \

  --client-key=./kube-controller-manager-key.pem \

  --embed-certs=true \

  --kubeconfig=${KUBE_CONFIG}

# kubectl config set-context default \

  --cluster=kubernetes \

  --user=kube-controller-manager \

  --kubeconfig=${KUBE_CONFIG}

# kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

 

1.3.3.2.4 創建配置文件

# cat > /opt/kubernetes/cfg/kube-controller-manager.conf << EOF

KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \\

--v=2 \\

--log-dir=/opt/kubernetes/logs \\

--leader-elect=true \\

--kubeconfig=/opt/kubernetes/cfg/kube-controller-manager.kubeconfig \\

--bind-address=127.0.0.1 \\

--allocate-node-cidrs=true \\

--cluster-cidr=10.244.0.0/16 \\

--service-cluster-ip-range=10.0.0.0/24 \\

--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \\

--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem  \\

--root-ca-file=/opt/kubernetes/ssl/ca.pem \\

--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \\

--cluster-signing-duration=87600h0m0s"

EOF

 

##解釋：

–master：通過本地非安全本地端口8080 連接apiserver

–leader-elect：當該組件啟動多個時，自動選舉（HA）

–cluster-signing-cert-file/–cluster-signing-key-file：自動為kubelet 頒發證書的CA，與apiserver 保持一致

1.3.3.2.5 systemd 管理controller-manager

# cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF

[Unit]

Description=Kubernetes Controller Manager

Documentation=https://github.com/kubernetes/Kubernetes

 

[Service]

EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf

ExecStart=/opt/kubernetes/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS

Restart=on-failure

 

[Install]

WantedBy=multi-user.target

EOF

 

1.3.3.2.6 啟動kube-controller-manager並設置開機啟動

# systemctl daemon-reload && systemctl start kube-controller-manager && systemctl enable kube-controller-manager

 

1.3.3.3 部署kube-scheduler

1.3.3.3.1 創建證書申請文件

# cd ~/TLS/k8s

# cat > kube-scheduler-csr.json << EOF

{

  "CN": "system:kube-scheduler",

  "hosts": [],

  "key": {

    "algo": "rsa",

    "size": 2048

  },

  "names": [

    {

      "C": "CN",

      "L": "BeiJing",

      "ST": "BeiJing",

      "O": "system:masters",

      "OU": "System"

    }

  ]

}

EOF

 

1.3.3.3.2 生成證書

# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler

 

1.3.3.3.3 生成kube-scheduler.kubeconfig配置文件

# KUBE_CONFIG="/opt/kubernetes/cfg/kube-scheduler.kubeconfig"

# KUBE_APISERVER="https://10.27.134.250:6443"

 

# kubectl config set-cluster kubernetes \

  --certificate-authority=/opt/kubernetes/ssl/ca.pem \

  --embed-certs=true \

  --server=${KUBE_APISERVER} \

  --kubeconfig=${KUBE_CONFIG}

# kubectl config set-credentials kube-scheduler \

  --client-certificate=./kube-scheduler.pem \

  --client-key=./kube-scheduler-key.pem \

  --embed-certs=true \

  --kubeconfig=${KUBE_CONFIG}

# kubectl config set-context default \

  --cluster=kubernetes \

  --user=kube-scheduler \

  --kubeconfig=${KUBE_CONFIG}

# kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

 

1.3.3.3.4 創建配置文件

# cat > /opt/kubernetes/cfg/kube-scheduler.conf << EOF

KUBE_SCHEDULER_OPTS="--logtostderr=false \\

--v=2 \\

--log-dir=/opt/kubernetes/logs \\

--leader-elect \\

--kubeconfig=/opt/kubernetes/cfg/kube-scheduler.kubeconfig \\

--bind-address=127.0.0.1"

EOF

 

##解釋：

–master：通過本地非安全本地端口8080 連接apiserver。

–leader-elect：當該組件啟動多個時，自動選舉（HA）

1.3.3.3.5 systemd 管理scheduler

# cat > /usr/lib/systemd/system/kube-scheduler.service<< EOF

[Unit]

Description=Kubernetes Scheduler

Documentation=https://github.com/kubernetes/Kubernetes

 

[Service]

EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf

ExecStart=/opt/kubernetes/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS

Restart=on-failure

 

[Install]

WantedBy=multi-user.target

EOF

 

1.3.3.3.6 啟動kube-scheduler並設置開機啟動

# systemctl daemon-reload && systemctl start kube-scheduler && systemctl enable kube-scheduler

 

1.3.3.4 查看集群狀態

1.3.3.4.1 生成kubectl連接集群的證書

##創建證書申請文件

# cat > admin-csr.json <<EOF

{

  "CN": "admin",

  "hosts": [],

  "key": {

    "algo": "rsa",

    "size": 2048

  },

  "names": [

    {

      "C": "CN",

      "L": "BeiJing",

      "ST": "BeiJing",

      "O": "system:masters",

      "OU": "System"

    }

  ]

}

EOF

 

1.3.3.4.2 生成證書

# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

 

1.3.3.4.3 生成kubectl連接集群的配置文件

##創建配置文件位置

# mkdir /root/.kube

 

##創建配置文件

# KUBE_CONFIG="/root/.kube/config"

# KUBE_APISERVER="https://10.27.134.250:6443"

 

# kubectl config set-cluster kubernetes \

  --certificate-authority=/opt/kubernetes/ssl/ca.pem \

  --embed-certs=true \

  --server=${KUBE_APISERVER} \

  --kubeconfig=${KUBE_CONFIG}

# kubectl config set-credentials cluster-admin \

  --client-certificate=./admin.pem \

  --client-key=./admin-key.pem \

  --embed-certs=true \

  --kubeconfig=${KUBE_CONFIG}

# kubectl config set-context default \

  --cluster=kubernetes \

  --user=cluster-admin \

  --kubeconfig=${KUBE_CONFIG}

# kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

 

1.3.3.4.4 查看集群組件狀態

# kubectl get cs

 

1.3.3.5 授權kubelet-bootstrap用戶允許請求證書

# kubectl create clusterrolebinding kubelet-bootstrap \

--clusterrole=system:node-bootstrapper \

--user=kubelet-bootstrap

 

1.3.4 部署Worker Node

##master節點也部署成worker節點，一並參加調度

1.3.4.1 創建工作目錄並拷貝二進制文件

##在worker node中創建目錄

# mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}

 

##進入解壓的kubernetes文件夾中

# cd /usr/local/k8s/kubernetes/server/bin

 

##將kubelet、kube-proxy組件拷貝至本地文件中

# cp kubelet kube-proxy /opt/kubernetes/bin

 

1.3.4.2 部署kubelet

1.3.4.2.1 創建配置文件

# cat > /opt/kubernetes/cfg/kubelet.conf << EOF

KUBELET_OPTS="--logtostderr=false \\

--v=2 \\

--log-dir=/opt/kubernetes/logs \\

--hostname-override=master \\

--network-plugin=cni \\

--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \\

--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \\

--config=/opt/kubernetes/cfg/kubelet-config.yml \\

--cert-dir=/opt/kubernetes/ssl \\

--pod-infra-container-image=lizhenliang/pause-amd64:3.0"

EOF

 

##解釋：

--hostname-override：顯示名稱，集群中唯一

--network-plugin：啟用CNI

--kubeconfig：空路徑，會自動生成，后面用於連接apiserver

--bootstrap-kubeconfig：首次啟動向apiserver申請證書

--config：配置參數文件

--cert-dir：kubelet證書生成目錄

--pod-infra-container-image：管理Pod網絡容器的鏡像

1.3.4.2.2 配置參數文件

# cat > /opt/kubernetes/cfg/kubelet-config.yml << EOF

kind: KubeletConfiguration

apiVersion: kubelet.config.k8s.io/v1beta1

address: 0.0.0.0

port: 10250

readOnlyPort: 10255

cgroupDriver: systemd

clusterDNS:

- 10.0.0.2

clusterDomain: cluster.local

failSwapOn: false

authentication:

  anonymous:

    enabled: false

  webhook:

    cacheTTL: 2m0s

    enabled: true

  x509:

    clientCAFile: /opt/kubernetes/ssl/ca.pem

authorization:

  mode: Webhook

  webhook:

    cacheAuthorizedTTL: 5m0s

    cacheUnauthorizedTTL: 30s

evictionHard:

  imagefs.available: 15%

  memory.available: 100Mi

  nodefs.available: 10%

  nodefs.inodesFree: 5%

maxOpenFiles: 1000000

maxPods: 110

EOF

 

1.3.4.2.3 生成kubelet初次加入集群引導kubeconfig文件

# KUBE_CONFIG="/opt/kubernetes/cfg/bootstrap.kubeconfig"

# KUBE_APISERVER=https://10.27.134.250:6443

 

##TOKEN值與token.csv里保持一致

# TOKEN="1b4c126d51e07e21bb272b144b255c14"

 

##生成 kubelet bootstrap kubeconfig 配置文件

# kubectl config set-cluster kubernetes \

  --certificate-authority=/opt/kubernetes/ssl/ca.pem \

  --embed-certs=true \

  --server=${KUBE_APISERVER} \

  --kubeconfig=${KUBE_CONFIG}

# kubectl config set-credentials "kubelet-bootstrap" \

  --token=${TOKEN} \

  --kubeconfig=${KUBE_CONFIG}

# kubectl config set-context default \

  --cluster=kubernetes \

  --user="kubelet-bootstrap" \

  --kubeconfig=${KUBE_CONFIG}

# kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

 

1.3.4.2.4 systemctl管理kubelet

# cat > /usr/lib/systemd/system/kubelet.service << EOF

[Unit]

Description=Kubernetes Kubelet

After=docker.service

 

[Service]

EnvironmentFile=/opt/kubernetes/cfg/kubelet.conf

ExecStart=/opt/kubernetes/bin/kubelet \$KUBELET_OPTS

Restart=on-failure

LimitNOFILE=65536

 

[Install]

WantedBy=multi-user.target

EOF

 

1.3.4.2.5 啟動kubelet並設置開機啟動

# systemctl daemon-reload && systemctl start kubelet && systemctl enable kubelet

 

##注意：kubernetes官方推薦docker等使用systemd作為Cgroup Driver

##如果kubelet服務狀態異常，查看日志是cgroupDriver不匹配的話

##先查看docker使用的Cgroup Driver

# docker info|grep "Cgroup Driver"

 

# 再修改kubelet的yaml文件中Cgroup Driver的參數

 

1.3.4.3 批准 kubelet 證書申請

##每個節點部署申請后都需被批准才能加入集群

（1）在master節點查看 kubelet 證書請求

# kubectl get csr

 

（2）批准申請

# kubectl certificate approve node-csr-31V7IbsN3zDicNmLUTyIs-s9uv9biTzpkMRE6vPrx7w

 

（3）查看節點

# kubectl get nodes

 

1.3.4.4 部署kube-proxy

1.3.4.4.1 創建證書申請文件

# cd ~/TLS/k8s

# cat > kube-proxy-csr.json << EOF

{

  "CN": "system:kube-proxy",

  "hosts": [],

  "key": {

    "algo": "rsa",

    "size": 2048

  },

  "names": [

    {

      "C": "CN",

      "L": "BeiJing",

      "ST": "BeiJing",

      "O": "k8s",

      "OU": "System"

    }

  ]

}

EOF

 

1.3.4.4.2 生成證書

# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem  -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

 

1.3.4.4.3 創建配置文件

# cat > /opt/kubernetes/cfg/kube-proxy.conf << EOF

KUBE_PROXY_OPTS="--logtostderr=false \\

--v=2 \\

--log-dir=/opt/kubernetes/logs \\

--config=/opt/kubernetes/cfg/kube-proxy-config.yml"

EOF

 

1.3.4.4.4 創建參數文件

cat > /opt/kubernetes/cfg/kube-proxy-config.yml << EOF

kind: KubeProxyConfiguration

apiVersion: kubeproxy.config.k8s.io/v1alpha1

bindAddress: 0.0.0.0

metricsBindAddress: 0.0.0.0:10249

clientConnection:

  kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig

hostnameOverride: master

clusterCIDR: 10.244.0.0/16

EOF

 

1.3.4.4.4 生成kube-proxy.kubeconfig配置文件

# KUBE_CONFIG="/opt/kubernetes/cfg/kube-proxy.kubeconfig"

# KUBE_APISERVER="https://10.27.134.250:6443"

 

# kubectl config set-cluster kubernetes \

  --certificate-authority=/opt/kubernetes/ssl/ca.pem \

  --embed-certs=true \

  --server=${KUBE_APISERVER} \

  --kubeconfig=${KUBE_CONFIG}

# kubectl config set-credentials kube-proxy \

  --client-certificate=./kube-proxy.pem \

  --client-key=./kube-proxy-key.pem \

  --embed-certs=true \

  --kubeconfig=${KUBE_CONFIG}

# kubectl config set-context default \

  --cluster=kubernetes \

  --user=kube-proxy \

  --kubeconfig=${KUBE_CONFIG}

# kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

 

1.3.4.4.5 systemd管理kube-proxy

# cat > /usr/lib/systemd/system/kube-proxy.service<< EOF

[Unit]

Description=Kubernetes Proxy

After=network.target

 

[Service]

EnvironmentFile=/opt/kubernetes/cfg/kube-proxy.conf

ExecStart=/opt/kubernetes/bin/kube-proxy \$KUBE_PROXY_OPTS

Restart=on-failure

LimitNOFILE=65536

 

[Install]

WantedBy=multi-user.target

EOF

 

1.3.4.4.6 啟動kube-proxy並設置開機啟動

# systemctl daemon-reload && systemctl start kube-proxy && systemctl enable kube-proxy

 

1.3.4.5 部署網絡組件（在master節點部署）

1.3.4.5.1 Calico

##Calico是一個純三層的數據中心網絡方案，是目前Kubernetes的主流方案

（1）下載配置文件

# cd /usr/local/k8s

 

# wget https://docs.projectcalico.org/v3.20/manifests/calico.yaml --no-check-certificate

 

（2）應用yaml文件

# kubectl apply -f calico.yaml

 

（3）在apply或create命令無法下載鏡像時通過docker pull 來拉取鏡像

# cat calico.yaml |grep image

 

1.3.4.5.2 查看組件狀態

（1）查看部署結果

# kubectl get pods -n kube-system

 

（2）結果實時查看

# kubectl get pods -n kube-system -w

 

（3）查看節點狀態

# kubectl get node

 

1.3.4.6 授權apiserver訪問kubelet

應用場景：例如kubectl logs

# cd /usr/local/k8s/

# cat > apiserver-to-kubelet-rbac.yaml << EOF

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRole

metadata:

  annotations:

    rbac.authorization.kubernetes.io/autoupdate: "true"

  labels:

    kubernetes.io/bootstrapping: rbac-defaults

  name: system:kube-apiserver-to-kubelet

rules:

  - apiGroups:

      - ""

    resources:

      - nodes/proxy

      - nodes/stats

      - nodes/log

      - nodes/spec

      - nodes/metrics

      - pods/log

    verbs:

      - "*"

---

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRoleBinding

metadata:

  name: system:kube-apiserver

  namespace: ""

roleRef:

  apiGroup: rbac.authorization.k8s.io

  kind: ClusterRole

  name: system:kube-apiserver-to-kubelet

subjects:

  - apiGroup: rbac.authorization.k8s.io

    kind: User

    name: kubernetes

EOF

 

##應用yaml文件

# kubectl apply -f apiserver-to-kubelet-rbac.yaml

 

##查看應用

# kubectl get ClusterRole

 

1.3.5 新增Worker Node

1.3.5.1 拷貝已部署好的Node相關文件到新節點

##在Master節點將Worker Node涉及的文件拷貝到新節點node1、node2

# scp -r /opt/kubernetes root@node1:/opt/

 

# scp -r /usr/lib/systemd/system/{kubelet,kube-proxy}.service root@node1:/usr/lib/systemd/system

 

# scp /opt/kubernetes/ssl/ca.pem root@node1:/opt/kubernetes/ssl

 

1.3.5.2 刪除kubelet證書和kubeconfig文件

# rm -f /opt/kubernetes/cfg/kubelet.kubeconfig

# rm -f /opt/kubernetes/ssl/kubelet*

 

注：這幾個文件是證書申請審批后自動生成的，每個Node不同，必須刪除

否則master節點無法獲得node節點信息

 

1.3.5.3 修改主機名

# vim /opt/kubernetes/cfg/kubelet.conf

--hostname-override=node1

 

# vim /opt/kubernetes/cfg/kube-proxy-config.yml

hostnameOverride: node1

 

1.3.5.4 啟動kubelet、kube-proxy並設置開機啟動

# systemctl daemon-reload && systemctl start kubelet kube-proxy && systemctl enable kubelet kube-proxy

1.3.5.5 在Master上批准新Node kubelet證書申請

##查看證書請求

# kubectl get csr

 

##授權請求

# kubectl certificate approve node-csr-31V7IbsN3zDicNmLUTyIs-s9uv9biTzpkMRE6vPrx7w

 

1.3.5.6 查看Node狀態

# kubectl get node

 

##node2節點及更多節點同上。記得刪除kubelet證書、配置文件、修改主機名！

 

1.3.6 部署dashboard可視化模塊

1.3.6.1 服務部署

##下載yml文件

# wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-rc5/aio/deploy/recommended.yaml

 

##修改yam文件，service改為NodePort類型

 

##創建服務

# kubectl create -f recommended.yaml

 

##查看服務對外暴露的端口號

# kubectl get svc -n kubernetes-dashboard

 

1.3.6.2 訪問測試

##瀏覽器訪問https://NodeIP:Port

 

1.3.6.3 創建ServiceAccount進行登錄

##創建admin.conf配置文件

# cat > /opt/kubernetes/cfg/admin.conf << EOF

kind: ClusterRoleBinding

apiVersion: rbac.authorization.k8s.io/v1beta1

metadata:

  name: admin

  annotations:

    rbac.authorization.kubernetes.io/autoupdate: "true"

roleRef:

  kind: ClusterRole

  name: cluster-admin

  apiGroup: rbac.authorization.k8s.io

subjects:

- kind: ServiceAccount

  name: admin

  namespace: kubernetes-dashboard

 

---

apiVersion: v1

kind: ServiceAccount

metadata:

  name: admin

  namespace: kubernetes-dashboard

EOF

 

##查看命名空間內的加密信息是否有token和名稱

# kubectl get secret -n kubernetes-dashboard

 

##使用該命令拿到token，替換上條命令查詢的名稱default-token-9xlqd，然后粘貼到瀏覽器中進行登錄

# kubectl -n kubernetes-dashboard get secret default-token-9xlqd -o jsonpath={.data.token}|base64 -d

 

##使用token登錄dashboard