kubernetes實踐之一：kubernetes二進制包安裝

本文轉載自查看原文 2019-03-20 18:35 718 kubernetes

kubernetes二進制部署

1、環境規划

軟件	版本
Linux操作系統	CentOS Linux release 7.6.1810 (Core)
Kubernetes	1.9
Docker	18.09.3
etcd	3.3.10

角色	IP	組件	推薦配置
k8s_master etcd01	192.168.1.153	kube-apiserver kube-controller-manager kube-scheduler etcd	CPU 2核+ 2G內存+
k8s_node01 etcd02	192.168.1.154	kubelet kube-proxy docker flannel etcd
k8s_node02 etcd03	192.168.1.155	kubelet kube-proxy docker flannel etcd

2、單Master集群架構

3、系統常規參數配置

3.1 關閉selinux

sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config

setenforce 0

3.2 文件數調整

　　　　　　sed -i '/* soft nproc 4096/d' /etc/security/limits.d/20-nproc.conf

　　　　　　echo '* - nofile 65536' >> /etc/security/limits.conf

　　　　　　echo '* soft nofile 65535' >> /etc/security/limits.conf

　　　　　　echo '* hard nofile 65535' >> /etc/security/limits.conf

　　　　　　echo 'fs.file-max = 65536' >> /etc/sysctl.conf

3.3 防火牆關閉

systemctl disable firewalld.service

systemctl stop firewalld.service

3.4 常用工具安裝及時間同步

yum -y install vim telnet iotop openssh-clients openssh-server ntp net-tools.x86_64 wget

ntpdate time.windows.com

3.5 hosts文件配置(3個節點)

vim /etc/hosts

192.168.1.153 k8s_master

　　　　　　 192.168.1.154 k8s_node01

　　　　　　 192.168.1.155 k8s_node02

3.6 服務器之間免密鑰登錄

ssh-keygen

ssh-copy-id -i ~/.ssh/id_rsa.pub root@192.168.1.154

ssh-copy-id -i ~/.ssh/id_rsa.pub root@192.168.1.155

4、自簽ssl證書

4.1 etcd生成證書

cfssl.sh

etcd-cert.sh

etcd.sh

4.1.1 安裝cfssl工具（cfssl.sh）

　　　　　　cd /home/k8s_install/ssl_etcd

　　　　　　chmod +x cfssl.sh

　　　　　　./cfssl.sh

內容如下：

curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /usr/local/bin/cfssl

　　　　　　curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson

　　　　　　curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /usr/local/bin/cfssl-certinfo

　　　　　　chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson /usr/local/bin/cfssl-certinfo

4.1.2 生成etcd 自簽ca證書（etcd-cert.sh）

　　　　　　chmod +x etcd-cert.sh

　　　　　　./etcd-cert.sh

內容如下：

cat > ca-config.json <<EOF

{

"signing": {

"default": {

"expiry": "87600h"

"profiles": {

"www": {

"expiry": "87600h",

"usages": [

"signing",

"key encipherment",

"server auth",

"client auth"

]

}

EOF

cat > ca-csr.json <<EOF

{

"CN": "etcd CA",

"key": {

"algo": "rsa",

"size": 2048

"names": [

{

"C": "CN",

"L": "Beijing",

"ST": "Beijing"

}

]

}

EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca –

#-----------------------

cat > server-csr.json <<EOF

{

"CN": "etcd",

"hosts": [

"192.168.1.153",

"192.168.1.154",

"192.168.1.155",

"192.168.1.156",

"192.168.1.157"

"key": {

"algo": "rsa",

"size": 2048

"names": [

{

"C": "CN",

"L": "BeiJing",

"ST": "BeiJing"

}

]

}

EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

注意：hosts一定要包含所有節點，可以多部署幾個預留節點以便后續擴容，否則還需要重新生成

4.1.3 etcd二進制包安裝

#存放配置文件，可執行文件，證書文件

mkdir /opt/etcd/{cfg,bin,ssl} -p

#ssl 證書切記復制到/opt/etcd/ssl/

cp {ca,server-key,server}.pem /opt/etcd/ssl/

#部署etcd以及增加etcd服務(etcd.sh)

cd /home/k8s_install/soft/

tar -zxvf etcd-v3.3.10-linux-amd64.tar.gz

cd etcd-v3.3.10-linux-amd64

mv etcd etcdctl /opt/etcd/bin/

cd /home/k8s_install/ssl_etcd

chmod +x etcd.sh

參數說明:1.etcd名稱 2.本機ip 3.其他兩個etcd名稱以及地址

./etcd.sh etcd01 192.168.1.153 etcd02=https://192.168.1.154:2380,etcd03=https://192.168.1.155:2380

執行后會卡住實際是在等待其他兩個節點加入

其他兩個node節點部署etcd:

scp -r /opt/etcd/ k8s_node01:/opt/

　　　　　　scp -r /opt/etcd/ k8s_node02:/opt/

　　　　　　scp /usr/lib/systemd/system/etcd.service k8s_node01:/usr/lib/systemd/system/

　　　　　　scp /usr/lib/systemd/system/etcd.service k8s_node02:/usr/lib/systemd/system/

　　　　　　#修改node節點配置文件(2個節點都需要更改)

　　　　　　ssh k8s_node01

　　　　　　vim /opt/etcd/cfg/etcd

　　　　　　ETCD_NAME以及ip地址

　　　　　　systemctl daemon-reload

　　　　　　systemctl start etcd.service

etcd.sh腳本內容如下：

#!/bin/bash

ETCD_NAME=$1

ETCD_IP=$2

ETCD_CLUSTER=$3

WORK_DIR=/opt/etcd

cat <<EOF >$WORK_DIR/cfg/etcd

#[Member]

ETCD_NAME="${ETCD_NAME}"

ETCD_DATA_DIR="/var/lib/etcd/default.etcd"

ETCD_LISTEN_PEER_URLS="https://${ETCD_IP}:2380"

ETCD_LISTEN_CLIENT_URLS="https://${ETCD_IP}:2379"

#[Clustering]

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://${ETCD_IP}:2380"

ETCD_ADVERTISE_CLIENT_URLS="https://${ETCD_IP}:2379"

ETCD_INITIAL_CLUSTER="etcd01=https://${ETCD_IP}:2380,${ETCD_CLUSTER}"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_INITIAL_CLUSTER_STATE="new"

EOF

cat <<EOF >/usr/lib/systemd/system/etcd.service

[Unit]

Description=Etcd Server

After=network.target

After=network-online.target

Wants=network-online.target

[Service]

Type=notify

EnvironmentFile=${WORK_DIR}/cfg/etcd

ExecStart=${WORK_DIR}/bin/etcd \

--name=\${ETCD_NAME} \

--data-dir=\${ETCD_DATA_DIR} \

--listen-peer-urls=\${ETCD_LISTEN_PEER_URLS} \

--listen-client-urls=\${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \

--advertise-client-urls=\${ETCD_ADVERTISE_CLIENT_URLS} \

--initial-advertise-peer-urls=\${ETCD_INITIAL_ADVERTISE_PEER_URLS} \

--initial-cluster=\${ETCD_INITIAL_CLUSTER} \

--initial-cluster-token=\${ETCD_INITIAL_CLUSTER_TOKEN} \

--initial-cluster-state=new \

--cert-file=${WORK_DIR}/ssl/server.pem \

--key-file=${WORK_DIR}/ssl/server-key.pem \

--peer-cert-file=${WORK_DIR}/ssl/server.pem \

--peer-key-file=${WORK_DIR}/ssl/server-key.pem \

--trusted-ca-file=${WORK_DIR}/ssl/ca.pem \

--peer-trusted-ca-file=${WORK_DIR}/ssl/ca.pem

Restart=on-failure

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

EOF

systemctl daemon-reload

systemctl enable etcd

systemctl restart etcd

4.1.4 查看etcd集群健康情況

cd /opt/etcd/ssl

/opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://192.168.1.153:2379,https://192.168.1.154:2379,https://192.168.1.155:2379" cluster-health

5、安裝Docker(node 節點)

5.1 安裝依賴包

yum install -y yum-utils \ device-mapper-persistent-data \ lvm2

5.2 配置官方源（替換為阿里源）

yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

5.3 更新並安裝Docker-CE

yum makecache fast

yum install docker-ce -y

5.4 配置docker加速器

curl -sSL https://get.daocloud.io/daotools/set_mirror.sh | sh -s http://f1361db2.m.daocloud.io

5.5 啟動docker

systemctl restart docker.service

systemctl enable docker.service

6、部署Flannel網絡

Overlay Network:覆蓋網絡，在基礎網絡上疊加的一種虛擬網絡技術模式，該網絡中的主機通過虛擬鏈路連接起來。 VXLAN:將源數據包封裝到UDP中，並使用基礎網絡的IP/MAC作為外層報文頭進行封裝，然后在以太網上傳輸，到達目的地后由隧道端點解封裝並將數據發送給目標地址。 Flannel:是Overlay網絡的一種，也是將源數據包封裝在另一種網絡包里面進行路由轉發和通信，目前已經支持UDP、VXLAN、AWS VPC和GCE路由等數據轉發方式。多主機容器網絡通信其他主流方案:隧道方案( Weave、OpenvSwitch )，路由方案(Calico)等。

6.1 寫入分配的子網段到etcd，供flanneld使用(master)

/opt/etcd/bin/etcdctl --ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --endpoints=https://192.168.1.153:2379,https://192.168.1.154:2379,https://192.168.1.155:2379 set /coreos.com/network/config '{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}'

6.2 二進制包安裝Flannel(node節點 flannel.sh)

下載地址：https://github.com/coreos/flannel/releases/download/

mkdir /opt/kubernetes/{bin,cfg,ssl} -p

cd /home/k8s_install/flannel_install/

tar -zxvf flannel-v0.10.0-linux-amd64.tar.gz

mv {flanneld,mk-docker-opts.sh} /opt/kubernetes/bin/

cd /home/k8s_install/flannel_install

chmod +x flannel.sh

　　　chmod +x /opt/kubernetes/bin/{flanneld,mk-docker-opts.sh}

./flannel.sh https://192.168.1.153:2379,https://192.168.1.154:2379,https://192.168.1.155:2379

腳本內容如下：

#!/bin/bash

ETCD_ENDPOINTS=${1:-"http://127.0.0.1:2379"}

cat <<EOF >/opt/kubernetes/cfg/flanneld

FLANNEL_OPTIONS="--etcd-endpoints=${ETCD_ENDPOINTS} \

-etcd-cafile=/opt/etcd/ssl/ca.pem \

-etcd-certfile=/opt/etcd/ssl/server.pem \

-etcd-keyfile=/opt/etcd/ssl/server-key.pem"

EOF

cat <<EOF >/usr/lib/systemd/system/flanneld.service

[Unit]

Description=Flanneld overlay address etcd agent

After=network-online.target network.target

Before=docker.service

[Service]

Type=notify

EnvironmentFile=/opt/kubernetes/cfg/flanneld

ExecStart=/opt/kubernetes/bin/flanneld --ip-masq \$FLANNEL_OPTIONS

ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env

Restart=on-failure

[Install]

WantedBy=multi-user.target

EOF

cat <<EOF >/usr/lib/systemd/system/docker.service

[Unit]

Description=Docker Application Container Engine

Documentation=https://docs.docker.com

After=network-online.target firewalld.service

Wants=network-online.target

[Service]

Type=notify

EnvironmentFile=/run/flannel/subnet.env

ExecStart=/usr/bin/dockerd \$DOCKER_NETWORK_OPTIONS

ExecReload=/bin/kill -s HUP \$MAINPID

LimitNOFILE=infinity

LimitNPROC=infinity

LimitCORE=infinity

TimeoutStartSec=0

Delegate=yes

KillMode=process

Restart=on-failure

StartLimitBurst=3

StartLimitInterval=60s

[Install]

WantedBy=multi-user.target

EOF

systemctl daemon-reload

systemctl enable flanneld

systemctl restart flanneld

systemctl restart docker

6.3 查看flannel是否部署成功

7、部署Master組件

mkdir /opt/kubernetes/{bin,cfg,ssl} -p

　　 cd /home/k8s_install/k8s_master_componet

tar -zxvf kubernetes-server-linux-amd64.tar.gz

cp -r kubernetes/server/bin/{kube-scheduler,kube-controller-manager,kube-apiserver} /opt/kubernetes/bin

cp kubernetes/server/bin/kubectl /usr/bin/

7.1 Kube-apiserver部署

chmod +x apiserver.sh

./apiserver.sh 192.168.1.153 https://192.168.1.153:2379,https://192.168.1.154:2379,https://192.168.1.155:2379

　　　　腳本內容：

#!/bin/bash

MASTER_ADDRESS=$1

ETCD_SERVERS=$2

cat <<EOF >/opt/kubernetes/cfg/kube-apiserver

KUBE_APISERVER_OPTS="--logtostderr=true \\

--v=4 \\

--etcd-servers=${ETCD_SERVERS} \\

--bind-address=${MASTER_ADDRESS} \\

--secure-port=6443 \\

--advertise-address=${MASTER_ADDRESS} \\

--allow-privileged=true \\

--service-cluster-ip-range=10.0.0.0/24 \\

--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \\

--authorization-mode=RBAC,Node \\

--kubelet-https=true \\

--enable-bootstrap-token-auth \\

--token-auth-file=/opt/kubernetes/cfg/token.csv \\

--service-node-port-range=30000-50000 \\

--tls-cert-file=/opt/kubernetes/ssl/server.pem \\

--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \\

--client-ca-file=/opt/kubernetes/ssl/ca.pem \\

--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \\

--etcd-cafile=/opt/etcd/ssl/ca.pem \\

--etcd-certfile=/opt/etcd/ssl/server.pem \\

--etcd-keyfile=/opt/etcd/ssl/server-key.pem"

EOF

cat <<EOF >/usr/lib/systemd/system/kube-apiserver.service

[Unit]

Description=Kubernetes API Server

Documentation=https://github.com/kubernetes/kubernetes

[Service]

EnvironmentFile=-/opt/kubernetes/cfg/kube-apiserver

ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS

Restart=on-failure

[Install]

WantedBy=multi-user.target

EOF

systemctl daemon-reload

systemctl enable kube-apiserver

systemctl restart kube-apiserver

　　7.1.1 生成kube-apiserver自簽證書(k8s-cert.sh)

chmod +x k8s-cert.sh

./k8s-cert.sh

cp ca.pem server.pem ca-key.pem server-key.pem /opt/kubernetes/ssl/

腳本內容如下：

cat > ca-config.json <<EOF

{

"signing": {

"default": {

"expiry": "87600h"

"profiles": {

"kubernetes": {

"expiry": "87600h",

"usages": [

"signing",

"key encipherment",

"server auth",

"client auth"

]

}

EOF

cat > ca-csr.json <<EOF

{

"CN": "kubernetes",

"key": {

"algo": "rsa",

"size": 2048

"names": [

{

"C": "CN",

"L": "Beijing",

"ST": "Beijing",

"O": "k8s",

"OU": "System"

}

]

}

EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca –

#-----------------------

cat > server-csr.json <<EOF

{

"CN": "kubernetes",

"hosts": [

"10.0.0.1",

"127.0.0.1",

"192.168.1.153",

"192.168.1.154",

"192.168.1.155",

"192.168.1.156",

"192.168.1.157",

"192.168.1.158",

"192.168.1.159",

"192.168.1.160",

"kubernetes",

"kubernetes.default",

"kubernetes.default.svc",

"kubernetes.default.svc.cluster",

"kubernetes.default.svc.cluster.local"

"key": {

"algo": "rsa",

"size": 2048

"names": [

{

"C": "CN",

"L": "BeiJing",

"ST": "BeiJing",

"O": "k8s",

"OU": "System"

}

]

}

EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server

#-----------------------

cat > admin-csr.json <<EOF

{

"CN": "admin",

"hosts": [],

"key": {

"algo": "rsa",

"size": 2048

"names": [

{

"C": "CN",

"L": "BeiJing",

"ST": "BeiJing",

"O": "system:masters",

"OU": "System"

}

]

}

EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

#-----------------------

cat > kube-proxy-csr.json <<EOF

{

"CN": "system:kube-proxy",

"hosts": [],

"key": {

"algo": "rsa",

"size": 2048

"names": [

{

"C": "CN",

"L": "BeiJing",

"ST": "BeiJing",

"O": "k8s",

"OU": "System"

}

]

}

EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

備注:hosts中盡量把未來可拓展的node加進來，之后就不用在重新生成自簽證書了.

7.1.2 生成token證書(kube_token.sh)

sh kube_token.sh

mv token.csv /opt/kubernetes/cfg/

腳本內容如下(kube_token.sh):

# 創建 TLS Bootstrapping Token

BOOTSTRAP_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ' ')

cat > token.csv <<EOF

${BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,"system:kubelet-bootstrap"

EOF

7.1.3 部署kube-apiserver配置文件及服務

cd /home/k8s_install/k8s_master_componet

chmod +x apiserver.sh

./apiserver.sh 192.168.1.153 https://192.168.1.153:2379,https://192.168.1.154:2379,https://192.168.1.155:2379

腳本內容如下（apiserver.sh）：

#!/bin/bash

MASTER_ADDRESS=$1

ETCD_SERVERS=$2

cat <<EOF >/opt/kubernetes/cfg/kube-apiserver

KUBE_APISERVER_OPTS="--logtostderr=true \\

--v=4 \\

--etcd-servers=${ETCD_SERVERS} \\

--bind-address=${MASTER_ADDRESS} \\

--secure-port=6443 \\

--advertise-address=${MASTER_ADDRESS} \\

--allow-privileged=true \\

--service-cluster-ip-range=10.0.0.0/24 \\

--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \\

--authorization-mode=RBAC,Node \\

--kubelet-https=true \\

--enable-bootstrap-token-auth \\

--token-auth-file=/opt/kubernetes/cfg/token.csv \\

--service-node-port-range=30000-50000 \\

--tls-cert-file=/opt/kubernetes/ssl/server.pem \\

--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \\

--client-ca-file=/opt/kubernetes/ssl/ca.pem \\

--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \\

--etcd-cafile=/opt/etcd/ssl/ca.pem \\

--etcd-certfile=/opt/etcd/ssl/server.pem \\

--etcd-keyfile=/opt/etcd/ssl/server-key.pem"

EOF

cat <<EOF >/usr/lib/systemd/system/kube-apiserver.service

[Unit]

Description=Kubernetes API Server

Documentation=https://github.com/kubernetes/kubernetes

[Service]

EnvironmentFile=-/opt/kubernetes/cfg/kube-apiserver

ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS

Restart=on-failure

[Install]

WantedBy=multi-user.target

EOF

systemctl daemon-reload

systemctl enable kube-apiserver

systemctl restart kube-apiserver

7.2 kube-controller-manager部署

7.2.1 部署controller-manager配置文件及服務

cd /home/k8s_install/k8s_master_componet

　　　　　　 chmod +x controller-manager.sh

./controller-manager.sh 127.0.0.1

腳本內容如下:

#!/bin/bash

MASTER_ADDRESS=$1

cat <<EOF >/opt/kubernetes/cfg/kube-controller-manager

KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=true \\

--v=4 \\

--master=${MASTER_ADDRESS}:8080 \\

--leader-elect=true \\

--address=127.0.0.1 \\

--service-cluster-ip-range=10.0.0.0/24 \\

--cluster-name=kubernetes \\

--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \\

--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \\

--root-ca-file=/opt/kubernetes/ssl/ca.pem \\

--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \\

--experimental-cluster-signing-duration=87600h0m0s"

EOF

cat <<EOF >/usr/lib/systemd/system/kube-controller-manager.service

[Unit]

Description=Kubernetes Controller Manager

Documentation=https://github.com/kubernetes/kubernetes

[Service]

EnvironmentFile=-/opt/kubernetes/cfg/kube-controller-manager

ExecStart=/opt/kubernetes/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS

Restart=on-failure

[Install]

WantedBy=multi-user.target

EOF

systemctl daemon-reload

systemctl enable kube-controller-manager

systemctl restart kube-controller-manager

7.3 Kube-scheduler部署

7.3.1 部署Kube-scheduler配置文件及服務

cd /home/k8s_install/k8s_master_componet

chmod +x scheduler.sh

./scheduler.sh 127.0.0.1

腳本內容如下:

#!/bin/bash

MASTER_ADDRESS=$1

cat <<EOF >/opt/kubernetes/cfg/kube-controller-manager

KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=true \\

--v=4 \\

--master=${MASTER_ADDRESS}:8080 \\

--leader-elect=true \\

--address=127.0.0.1 \\

--service-cluster-ip-range=10.0.0.0/24 \\

--cluster-name=kubernetes \\

--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \\

--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \\

--root-ca-file=/opt/kubernetes/ssl/ca.pem \\

--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \\

--experimental-cluster-signing-duration=87600h0m0s"

EOF

cat <<EOF >/usr/lib/systemd/system/kube-controller-manager.service

[Unit]

Description=Kubernetes Controller Manager

Documentation=https://github.com/kubernetes/kubernetes

[Service]

EnvironmentFile=-/opt/kubernetes/cfg/kube-controller-manager

ExecStart=/opt/kubernetes/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS

Restart=on-failure

[Install]

WantedBy=multi-user.target

EOF

systemctl daemon-reload

systemctl enable kube-controller-manager

systemctl restart kube-controller-manager

7.4 查看Master狀態

kubectl get cs

8、部署Node組件

8.1 將kubelet-bootstrap用戶綁定到系統集群角色（master執行）

kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap

8.2 創建kubeconfig文件(存放連接apiserver認證信息,master執行)

cd /home/k8s_install/k8s_master_componet

chmod +x kubeconfig.sh

#執行之前需要保證BOOTSTRAP_TOKEN正確修改腳本BOOTSTRAP_TOKEN

cat /opt/kubernetes/cfg/token.csv

./kubeconfig.sh 192.168.1.153 /home/k8s_install/k8s_master_componet/

參數說明:param1:apiserver地址 param2:生成證書目錄

腳本說明：

APISERVER=$1

SSL_DIR=$2

BOOTSTRAP_TOKEN=18a5ee2b7525d343e112807ba5f101ae

# 創建kubelet bootstrapping kubeconfig

export KUBE_APISERVER="https://$APISERVER:6443"

# 設置集群參數

kubectl config set-cluster kubernetes \

--certificate-authority=$SSL_DIR/ca.pem \

--embed-certs=true \

--server=${KUBE_APISERVER} \

--kubeconfig=bootstrap.kubeconfig

# 設置客戶端認證參數

kubectl config set-credentials kubelet-bootstrap \

--token=${BOOTSTRAP_TOKEN} \

--kubeconfig=bootstrap.kubeconfig

# 設置上下文參數

kubectl config set-context default \

--cluster=kubernetes \

--user=kubelet-bootstrap \

--kubeconfig=bootstrap.kubeconfig

# 設置默認上下文

kubectl config use-context default --kubeconfig=bootstrap.kubeconfig

#----------------------

# 創建kube-proxy kubeconfig文件

kubectl config set-cluster kubernetes \

--certificate-authority=$SSL_DIR/ca.pem \

--embed-certs=true \

--server=${KUBE_APISERVER} \

--kubeconfig=kube-proxy.kubeconfig

kubectl config set-credentials kube-proxy \

--client-certificate=$SSL_DIR/kube-proxy.pem \

--client-key=$SSL_DIR/kube-proxy-key.pem \

--embed-certs=true \

--kubeconfig=kube-proxy.kubeconfig

kubectl config set-context default \

--cluster=kubernetes \

--user=kube-proxy \

--kubeconfig=kube-proxy.kubeconfig

kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

8.3 復制kubeconfig文件到node節點

scp -r *.kubeconfig k8s_node01:/opt/kubernetes/cfg/

scp -r *.kubeconfig k8s_node02:/opt/kubernetes/cfg/

8.4 部署kubelet組件

#復制包到node節點

scp kubelet kube-proxy k8s_node01:/opt/kubernetes/bin/

cd /home/k8s_install/k8s_node_componet

chmod +x kubelet.sh

./kubelet.sh 192.168.1.154

腳本如下:

#!/bin/bash

NODE_ADDRESS=$1

DNS_SERVER_IP=${2:-"10.0.0.2"}

cat <<EOF >/opt/kubernetes/cfg/kubelet

KUBELET_OPTS="--logtostderr=true \\

--v=4 \\

--hostname-override=${NODE_ADDRESS} \\

--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \\

--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \\

--config=/opt/kubernetes/cfg/kubelet.config \\

--cert-dir=/opt/kubernetes/ssl \\

--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0"

EOF

cat <<EOF >/opt/kubernetes/cfg/kubelet.config

kind: KubeletConfiguration

apiVersion: kubelet.config.k8s.io/v1beta1

address: ${NODE_ADDRESS}

port: 10250

readOnlyPort: 10255

cgroupDriver: cgroupfs

clusterDNS:

- ${DNS_SERVER_IP}

clusterDomain: cluster.local.

failSwapOn: false

authentication:

anonymous:

enabled: true

EOF

cat <<EOF >/usr/lib/systemd/system/kubelet.service

[Unit]

Description=Kubernetes Kubelet

After=docker.service

Requires=docker.service

[Service]

EnvironmentFile=/opt/kubernetes/cfg/kubelet

ExecStart=/opt/kubernetes/bin/kubelet \$KUBELET_OPTS

Restart=on-failure

KillMode=process

[Install]

WantedBy=multi-user.target

EOF

systemctl daemon-reload

systemctl enable kubelet

systemctl restart kubelet

8.5 部署kube-proxy組件

cd /home/k8s_install/k8s_node_componet

chmod +x proxy.sh

./proxy.sh 192.168.1.154

腳本如下：

#!/bin/bash

NODE_ADDRESS=$1

cat <<EOF >/opt/kubernetes/cfg/kube-proxy

KUBE_PROXY_OPTS="--logtostderr=true \\

--v=4 \\

--hostname-override=${NODE_ADDRESS} \\

--cluster-cidr=10.0.0.0/24 \\

--proxy-mode=ipvs \\

--kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig"

EOF

cat <<EOF >/usr/lib/systemd/system/kube-proxy.service

[Unit]

Description=Kubernetes Proxy

After=network.target

[Service]

EnvironmentFile=-/opt/kubernetes/cfg/kube-proxy

ExecStart=/opt/kubernetes/bin/kube-proxy \$KUBE_PROXY_OPTS

Restart=on-failure

[Install]

WantedBy=multi-user.target

EOF

systemctl daemon-reload

systemctl enable kube-proxy

systemctl restart kube-proxy

8.6 master授權node請求簽名

kubectl get csr

kubectl certificate approve node-csr-aCTdx_2qXRajXlHrIlg2uPddhpexw8NRHP5EEtPllps

9、查詢集群狀態

# kubectl get node

# kubectl get componentstatus

10、啟動一個測試示例

# kubectl create deployment nginx --image=nginx

# kubectl get pod -o wide

# kubectl expose deployment nginx --port=88 --target-port=80 --type=NodePort

# kubectl get svc nginx

http://192.168.1.154:40287/

授權:

kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous

11、新增一個node節點步驟

本文以157為例子

11.1 參照三章系統常規參數配置

更新hosts(所有節點)

　　　　 vim /etc/hosts

192.168.1.153 k8s_master

192.168.1.154 k8s_node01

192.168.1.155 k8s_node02

192.168.1.157 k8s_node03

11.2 安裝Docker 參照第五章

11.3 從node01復制文件到node03

scp -r /opt/kubernetes/ k8s_node03:/opt/

scp /usr/lib/systemd/system/{kubelet,kube-proxy}.service k8s_node03:/usr/lib/systemd/system/

11.3.1 替換node節點ip

cd /opt/kubernetes/cfg

需要修改:kubelet,kubelet.config,kube-proxy文件ip

11.3.2 刪除node01認證信息

rm -rf /opt/kubernetes/ssl/*

11.3.3 啟動kubelet,kube-proxy服務

systemctl start kubelet.service

systemctl enable kubelet.service

systemctl start kube-proxy.service

　　　　　　 systemctl enable kube-proxy.service

11.3.4 master授權新加入node

# kubectl get csr

kubectl certificate approve node-csr-tE3KOFUjDDu1VLUZQNmEqwhdpRhU-RBRfWIIbAD3Xj8

12、K8s高可用搭建

環境規划：

角色	IP	組件	推薦配置
k8s_master etcd01	192.168.1.153	kube-apiserver kube-controller-manager kube-scheduler etcd	CPU 2核+ 2G內存+
k8s_master02	192.168.1.157	kube-apiserver kube-controller-manager kube-scheduler etcd
k8s_node01 etcd02	192.168.1.154	kubelet kube-proxy docker flannel etcd
k8s_node02 etcd03	192.168.1.155	kubelet kube-proxy docker flannel etcd
Load Balancer (Master)	192.168.1.158 192.168.1.160 (VIP)	Nginx L4
Load Balancer (Backup)	192.168.1.159	Nginx L4

　　　　　多master集群架構

12.1 高可用LB環境搭建(keepalived+nginx)

nginx安裝：

rpm -Uvh http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm

yum install -y nginx

keepalived安裝：

yum -y install keepalived

nginx配置增加stream模塊：

vim /etc/nginx/nginx.conf

stream {

log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';

access_log /var/log/nginx/k8s-access.log main;

upstream k8s-apiserver {

server 192.168.1.153:6443;

server 192.168.1.157:6443;

}

server {

listen 6443;

proxy_pass k8s-apiserver;

}

service nginx start

keepalived配置：

! Configuration File for keepalived

global_defs {

notification_email {

acassen@firewall.loc

failover@firewall.loc

sysadmin@firewall.loc

}

notification_email_from Alexandre.Cassen@firewall.loc

smtp_server 127.0.0.1

smtp_connect_timeout 30

router_id NGINX_MASTER

}

vrrp_script check_nginx {

script "/etc/nginx/check_nginx.sh"

}

vrrp_instance VI_1 {

state MASTER

interface enp0s8

virtual_router_id 51 # VRRP 路由 ID實例，每個實例是唯一的

priority 100 # 優先級，備服務器設置 90

advert_int 1 # 指定VRRP 心跳包通告間隔時間，默認1秒

authentication {

auth_type PASS

auth_pass 1111

}

virtual_ipaddress {

192.168.1.160/24

}

track_script {

check_nginx

}

keepalived檢查nginx狀態腳本:

vim /etc/nginx/check_nginx.sh

count=$(ps -ef |grep nginx |egrep -cv "grep|$$")

if [ "$count" -eq 0 ];then

/etc/init.d/keepalived stop

12.2 高可用Master02搭建

復制master01文件：

scp -r /opt/kubernetes/ k8s_master02:/opt/

scp -r /opt/etcd/ k8s_master02:/opt/

scp /usr/bin/kubectl k8s_master02:/usr/bin/

scp /usr/lib/systemd/system/kube-* k8s_master02:/usr/lib/systemd/system

修改配置文件(master02):

vim /opt/kubernetes/cfg/kube-apiserver

--bind-address=192.168.1.157 \

--advertise-address=192.168.1.157 \

啟動Master服務：

systemctl start kube-apiserver

systemctl enable kube-apiserver

systemctl start kube-scheduler

systemctl enable kube-scheduler

systemctl start kube-controller-manager

systemctl enable kube-controller-manager

# kubectl get node

NAME STATUS ROLES AGE VERSION

192.168.1.154 Ready <none> 6h15m v1.13.4

192.168.1.155 Ready <none> 6h4m v1.13.4

12.3 Node重新指向Load Balancer

cd /opt/kubernetes/cfg

需要修改bootstrap.kubeconfig，kubelet.kubeconfig，kube-proxy.kubeconfig為VIP 192.168.1.160

重新啟動服務:

systemctl restart kubelet

systemctl restart kube-proxy

12.4 測試高可用部署是否成功：

重新創建kubeconfig文件(存放連接apiserver認證信息,master01執行)

cd /home/k8s_install/k8s_master_componet/

./kubelet_new.sh

會生成一個config文件

腳本如下：

kubectl config set-cluster kubernetes --server=https://192.168.1.160:6443 --embed-certs=true --certificate-authority=ca.pem --kubeconfig=config

kubectl config set-credentials cluster-admin --certificate-authority=ca.pem --embed-certs=true --client-key=admin-key.pem --client-certificate=admin.pem --kubeconfig=config

kubectl config set-context default --cluster=kubernetes --user=cluster-admin --kubeconfig=config

kubectl config use-context default --kubeconfig=config

node01節點執行（把config,kubectl復制到node01節點）：

scp /usr/bin/kubectl k8s_node01:/usr/bin/

scp config k8s_node01:/root/

kubectl --kubeconfig=/root/config get node

NAME STATUS ROLES AGE VERSION

192.168.1.154 Ready <none> 7h7m v1.13.4

192.168.1.155 Ready <none> 6h56m v1.13.4

備注：二進制包及部署腳本下載地址

鏈接:https://pan.baidu.com/s/1wr4y84hakxV8kncQ_ogCGw 密碼:pp6c

由於博客園排版問題有需要word可以自行文檔下載地址：

鏈接:https://pan.baidu.com/s/1b2CHnhAW45fxxf8qW-rEvw 密碼:i1nc

新增HA腳本下載地址：

鏈接:https://pan.baidu.com/s/11lkHP1vslHBDZpR49MUYqQ 密碼:ktew

免責聲明！

本站轉載的文章為個人學習借鑒使用，本站對版權不負任何法律責任。如果侵犯了您的隱私權益，請聯系本站郵箱yoyou2525@163.com刪除。

猜您在找 二進制安裝 kubernetes 二進制安裝Kubernetes 1.20 centos安裝nodejs二進制包二進制包安裝Mysql 二進制安裝kubernetes（六） kube-proxy組件安裝 Centos7 二進制安裝 Kubernetes 1.13 二進制安裝kubernetes（七）部署知識點總結 Linux 下安裝NodeJS （二進制包） 1.二進制部署kubernetes 1.二進制部署kubernetes

kubernetes實踐之一：kubernetes二進制包安裝

kubernetes二進制部署

1、環境規划

2、 單Master集群架構

3、 系統常規參數配置

3.1 關閉selinux

3.2 文件數調整

3.3 防火牆關閉

3.4 常用工具安裝及時間同步

3.5 hosts文件配置(3個節點)

3.6 服務器之間免密鑰登錄

4、 自簽ssl證書

4.1 etcd生成證書

4.1.1 安裝cfssl工具（cfssl.sh）

4.1.2 生成etcd 自簽ca證書（etcd-cert.sh）

4.1.3 etcd二進制包安裝

4.1.4 查看etcd集群健康情況

5、 安裝Docker(node 節點)

5.1 安裝依賴包

5.2 配置官方源（替換為阿里源）

5.3 更新並安裝Docker-CE

5.4 配置docker加速器

5.5 啟動docker

6、部署Flannel網絡

6.1 寫入分配的子網段到etcd，供flanneld使用(master)

6.2 二進制包安裝Flannel(node節點 flannel.sh)

6.3 查看flannel是否部署成功

7、部署Master組件

7.1 Kube-apiserver部署

7.1.2 生成token證書(kube_token.sh)

7.1.3 部署kube-apiserver配置文件及服務

7.2 kube-controller-manager部署

7.2.1 部署controller-manager配置文件及服務

7.3 Kube-scheduler部署

7.3.1 部署Kube-scheduler配置文件及服務

7.4 查看Master狀態

8、部署Node組件

8.1 將kubelet-bootstrap用戶綁定到系統集群角色（master執行）

8.2 創建kubeconfig文件(存放連接apiserver認證信息,master執行)

8.3 復制kubeconfig文件到node節點

8.4 部署kubelet組件

8.5 部署kube-proxy組件

8.6 master授權node請求簽名

9、查詢集群狀態

10、啟動一個測試示例

11、新增一個node節點步驟

11.1 參照三章系統常規參數配置

11.2 安裝Docker 參照第五章

11.3 從node01復制文件到node03

11.3.1 替換node節點ip

11.3.2 刪除node01認證信息

11.3.3 啟動kubelet,kube-proxy服務

11.3.4 master授權新加入node

12、K8s高可用搭建

12.1 高可用LB環境搭建(keepalived+nginx)

12.2 高可用Master02搭建

12.3 Node重新指向Load Balancer

12.4 測試高可用部署是否成功：

免責聲明！

2、單Master集群架構

3、系統常規參數配置

4、自簽ssl證書

5、安裝Docker(node 節點)