K8S--安裝配置--kubeadm安裝方式---成功--K8S--dashboard--
安裝--配置--重置--卸載--20210914
基本流程為：
基礎環境配置--安裝docker--安裝k8s主件--安裝flannel（k8s本身沒包括）--安裝dashboard--完成
========================================================
本文所有鏈接，以后可能用不了，還需要自行去找方法，沒辦法
這個教程里，flannel下載安裝就是在網上找的，多次實驗安裝才成功，網上的教程就不行
整個安裝步驟太多了，感覺還是vmware vsphere套件安裝好，至少是可視化的。
數據庫 etcd，
調度器 scheduler，
集群入口 API Server，
控制器 Controller，
服務代理 kube-proxy
直接管理具體業務容器的 kubelet
flannel 網絡連接插件
===================================================
成功---參考以下教程，這個教程k8s主件安裝是完全OK的。
https://blog.csdn.net/caiyqn/article/details/107801349?utm_medium=distribute.pc_relevant.none-task-blog-2~default~baidujs_title~default-0.no_search_link&spm=1001.2101.3001.4242
https://www.cnblogs.com/guoxiaobo/p/15025308.html 這個也詳細
https://www.cnblogs.com/xiao987334176/p/12696740.html 這個也詳細
===================================================
硬件要求，建議至少2 cpu ,2G,20G：3台機器，
192.168.9.3 master
192.168.9.4 node1
192.168.9.5 node2
不然會安裝報錯，后面就安裝不了了
軟件要求，版本要匹配，節點的時間要一致
===本次安裝==軟件版本=======
centos7.8
docker-ce-18.06.1.ce-3.el7
kubelet-1.18.3 kubeadm-1.18.3 kubectl-1.18.3
檢查所有節點配置
cat /etc/redhat-release
CentOS Linux release 7.7.1908 (Core)
uname -a
Linux master 3.10.0-1062.el7.x86_64 #1 SMP Wed Aug 7 18:08:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
free -h
===docker與k8s的版本對應版本是有要求的，最新的--並不是匹配的=========================
軟件安裝中，版本沖突，就換版本，注意系統提示的版本信息
file /usr/bin/kubectl from install of kubectl-1.22.1-0.x86_64 conflicts with file from package kubernetes-client-1.5.2-0.7.git269f928.el7.x86_64
====安裝結果如下==master上檢查==========================
[root@master ~]# docker -v
Docker version 18.06.1-ce, build e68fc7a
[root@master ~]# yum list installed |grep kub
cri-tools.x86_64 1.13.0-0 @kubernetes
kubeadm.x86_64 1.18.3-0 @kubernetes
kubectl.x86_64 1.18.3-0 @kubernetes
kubelet.x86_64 1.18.3-0 @kubernetes
kubernetes-cni.x86_64 0.8.7-0 @kubernetes
[root@master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready master 4h13m v1.18.3
node1 Ready <none> 4h10m v1.18.3
node2 Ready <none> 4h10m v1.18.3
===安裝開始===============================================
1.===基礎環境配置==所有節點操作=============================
==root用戶操作==所有節點==當然也可不用========================
1.1關閉防火牆==所有節點操作============
systemctl stop firewalld
systemctl disable firewalld
1.2關閉selinux==所有節點操作============
sed -i 's/enforcing/disabled/' /etc/selinux/config
setenforce 0
1.3關閉swap==所有節點操作============
臨時關閉：
swapoff -a
永久關閉：注釋掉/etc/fstab文件中的swap行
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
1.4添加hosts==所有節點操作============
vi /etc/hosts
192.168.9.3 master
192.168.9.4 node1
192.168.9.5 node2
1.5內核修改相關==所有節點操作============
加載內核模塊==所有節點操作============
modprobe ip_vs_rr
modprobe br_netfilter
1.6優化內核參數==所有節點操作============
vi /etc/sysctl.d/kubernetes.conf
---復制吧---------------
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
#由於tcp_tw_recycle與kubernetes的NAT沖突，必須關閉！否則會導致服務不通。4.1x內核已經廢棄這項了
#net.ipv4.tcp_tw_recycle=0
#禁止使用 swap 空間，只有當系統 OOM 時才允許使用它
vm.swappiness=0
#不檢查物理內存是否夠用
vm.overcommit_memory=1
#開啟 OOM
vm.panic_on_oom=0
fs.inotify.max_user_instances=8192
fs.inotify.max_user_watches=1048576
fs.file-max=52706963
fs.nr_open=52706963
#關閉不使用的ipv6協議棧，防止觸發docker BUG.
net.ipv6.conf.all.disable_ipv6=1
net.netfilter.nf_conntrack_max=2310720
--再執行，所有節點----------------------
sysctl -p /etc/sysctl.d/kubernetes.conf
sysctl --system
============================
#必須關閉 tcp_tw_recycle，否則和 NAT 沖突，會導致服務不通；
#關閉 IPV6，防止觸發 docker BUG；
=============================
1.7開啟IP轉發功能==所有節點操作============
echo "1" > /proc/sys/net/ipv4/ip_forward
開啟時間同步==所有節點操作===如果時間正常，可以不配置=========
echo "*/3 * * * * /usr/sbin/ntpdate ntp3.aliyun.com &> /dev/null" > /tmp/crontab
crontab /tmp/crontab
1.8創建免密訪問===master上配置============
ssh-keygen -t rsa
ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.9.4
ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.9.5
1.9修改PATH環境變量==所有節點操作============
echo 'PATH=/opt/k8s/bin:$PATH' >>/root/.bashrc
source /root/.bashrc
1.10安裝依賴包==如果系統已經有了，可以不安裝========
yum install -y epel-release
yum install -y conntrack ntpdate ntp ipvsadm ipset jq iptables curl sysstat libseccomp wget lsof telnet
1.11安裝docker==所有節點操作============
1.12#安裝docker需要的工具==所有節點操作============
yum install -y yum-utils device-mapper-persistent-data lvm2
1.13#添加docker鏡像包==所有節點操作============
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
1.13#安裝並更新docker-ce鏡像包==所有節點操作============
1.14#更新緩存
yum makecache fast
1.15#安裝docker默認安裝最新，可指定版本安裝
yum -y install docker-ce
#查看版本
docker -v
1.16#開啟docker服務==所有節點操作============
systemctl enable docker
systemctl start docker
1.17#附：docker鏡像加速方法==所有節點操作============
#添加指定加速地址，這里指定阿里雲的
vim /etc/docker/daemon.json
---復制吧-----------
{
"registry-mirrors": [
"https://registry.cn-hangzhou.aliyuncs.com"
]
}
------------------
1.18#重新加載服務==所有節點操作============
systemctl daemon-reload
systemctl restart docker
#查看 Registry Mirrors是否變成阿里雲加速地址
docker info
1.19添加kubenetes軟件包==所有節點操作============
#這里直接使用阿里雲的軟件包
vi /etc/yum.repos.d/kubernetes.repo
----復制吧------------------
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
-----------------------
1.20#更新軟件包==所有節點操作============
yum makecache fast
==================================
基礎環境配置完成
==================================
1.21==開始安裝k8s主件==所有節點操作============
#安裝kubelet、kubeadm、kubectl==所有節點操作============（由於這套環境后續還有其他用處，因此指定了版本安裝）
yum install -y kubelet-1.18.3 kubeadm-1.18.3 kubectl-1.18.3
systemctl enable kubelet
systemctl start kubelet
2.0==開始部署k8s集群==Master上執行=================
#只能在matser上面部署
----復制吧---------------
kubeadm init \
--apiserver-advertise-address=192.168.9.3 \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.18.3 \
--service-cidr=10.1.0.0/16 \
--pod-network-cidr=10.244.0.0/16
--------------------
====安裝完成 顯示以下內容，表示安裝成功========================
==安裝過程簡要如下=========
。。省略前面顯示過程。。。。。
Your Kubernetes master has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
2.1#==只在Master上執行，不論是不是用root用戶執行整個的安裝，都要執行，（雖然這里說明是只要，非root用戶執行）
-----復制吧-------
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
-------------------
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of machines by running the following on each node
as root:
2.2#下面這個很重要，是node節點加入集群里的命令，直接執行，就好，2個小時有效，2小時后，再用createa命令生成吧
kubeadm join 192.168.9.3:6443 --token rv7mmx.nkpsf0u426eajiwl --discovery-token-ca-cert-hash sha256:4293225b17a544f912e9218da98dabdaf8b4e5c250a63288dcd4a8867b3ac262
===================================================================
2.3===安裝后，會自動下載k8s的相關docker鏡像，查下有哪些====#只在Master上執行=============================
[root@master ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.aliyuncs.com/google_containers/kube-apiserver v1.22.1 f30469a2491a 2 weeks ago 128MB
registry.aliyuncs.com/google_containers/kube-controller-manager v1.22.1 6e002eb89a88 2 weeks ago 122MB
registry.aliyuncs.com/google_containers/kube-scheduler v1.22.1 aca5ededae9c 2 weeks ago 52.7MB
registry.aliyuncs.com/google_containers/kube-proxy v1.22.1 36c4ebbc9d97 2 weeks ago 104MB
registry.aliyuncs.com/google_containers/etcd 3.5.0-0 004811815584 2 months ago 295MB
registry.aliyuncs.com/google_containers/pause 3.5 ed210e3e4a5b 5 months ago 683kB
registry.aliyuncs.com/google_containers/kube-proxy v1.13.1 fdb321fd30a0 2 years ago 80.2MB
registry.aliyuncs.com/google_containers/kube-controller-manager v1.13.1 26e6f1db2a52 2 years ago 146MB
registry.aliyuncs.com/google_containers/kube-scheduler v1.13.1 ab81d7360408 2 years ago 79.6MB
registry.aliyuncs.com/google_containers/kube-apiserver v1.13.1 40a63db91ef8 2 years ago 181MB
registry.aliyuncs.com/google_containers/coredns 1.2.6 f59dcacceff4 2 years ago 40MB
registry.aliyuncs.com/google_containers/etcd 3.2.24 3cab8e1b9802 2 years ago 220MB
registry.aliyuncs.com/google_containers/pause 3.1 da86e6ba6ca1 3 years ago 742kB
2.4==按照上面提示，操作===#只在Master上執行==========================
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
==下面就可以直接使用kubectl命令了========================================
2.5===查看節點狀態=======================================
kubectl get nodes
NAME STATUS ROLES AGE VERSION
master NotReady master 5m47s v1.18.3
2.6===上面notready,是因為flannel插件沒安裝，k8s節點的專用網絡不通=======================================
#由於github官網原因，這2個鏈接目前不能下載了，只能手動下載flannel.docker安裝包，手動load
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/a70459be0084506e4ec919aa1c114638878db11b/Documentation/kube-flannel.yml
kubectl apply -f https://github.com/caoran/kube-flannel.yml/blob/master/kube-flannel.yml
2.7#只在Master上執行，先上傳到master,再執行=====flannel版本不確定，我只下載了0.12.0，當然也用其它網絡插件，其它自行測試吧，
docker load < flanneld-v0.12.0-amd64.docker
#kube-flannel.yml，文件名可自定，內容只能在網上去復制了，再執行
kubectl apply -f kube-flannel.yml
2.8再查看k8s集群狀態
#查看所有節點是否ready
[root@master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready master 5h7m v1.18.3
node1 Ready <none> 5h5m v1.18.3
node2 Ready <none> 5h5m v1.18.3
kubectl get nodes
2.9#查看flannel節點是否運行正常
kubectl get pod -n kube-system | grep flannel
kube-flannel-ds-amd64-2dd6b 1/1 Running 0 5h5m
kube-flannel-ds-amd64-7dg9q 1/1 Running 0 5h5m
kube-flannel-ds-amd64-swspw 1/1 Running 0 5h5m
2.10#查看kube-system啟動的所有pod
kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-7ff77c879f-d7gjx 1/1 Running 0 5h11m
coredns-7ff77c879f-tpr2t 1/1 Running 0 5h11m
etcd-master 1/1 Running 0 5h11m
kube-apiserver-master 1/1 Running 0 5h11m
kube-controller-manager-master 1/1 Running 0 5h11m
kube-flannel-ds-amd64-2dd6b 1/1 Running 0 5h6m
kube-flannel-ds-amd64-7dg9q 1/1 Running 0 5h6m
kube-flannel-ds-amd64-swspw 1/1 Running 0 5h6m
kube-proxy-52xxt 1/1 Running 0 5h9m
kube-proxy-hnn4b 1/1 Running 0 5h8m
kube-proxy-t7zl8 1/1 Running 0 5h11m
kube-scheduler-master 1/1 Running 0 5h11m
2.11====查看集群配置信息======================================
kubectl -n kube-system get cm kubeadm-config -oyaml
==========================================
2.22//查看節點信息
kubectl get pod -n kube-system
kubectl get pod -n kube-system -w //監視
kubectl get pod -n kube-system -o wide // 詳細信息
========================================================
*****到這里，k8s集群后台主件安裝，基本完成*****************
========================================================
3.0===重建集群==#只在Master上執行===
kubeadm reset
再執行2.0步驟
========================================================
4.0==卸載K8S=====#只在Master上執行=====可保存成腳本，自動跑，更好=================
yum remove -y kubelet kubeadm kubectl
kubeadm reset -f
modprobe -r ipip
lsmod
rm -rf ~/.kube/
rm -rf /etc/kubernetes/
rm -rf /etc/systemd/system/kubelet.service.d
rm -rf /etc/systemd/system/kubelet.service
rm -rf /usr/bin/kube*
rm -rf /etc/cni
rm -rf /opt/cni
rm -rf /var/lib/etcd
rm -rf /var/etcd
====以下為可選安裝==================================================================================
5.0 dashboard安裝---K8S的web管理界面==專業術語叫--儀表盤
===master上執行====查看對應版本的Dashboard======= https://github.com/kubernetes/dashboard/releases
5.11 下載dashboard配置文件
5.12 初始化安裝
5.13 創建admin-user.yaml文件
5.14 查看token
5.15 網頁訪問集群
========================================================
5.11 下載dashboard配置文件
wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.1/aio/deploy/recommended.yaml
上面鏈接失效了，用下面的吧，親測OK,手動復制吧
https://www.cnblogs.com/zjz20/p/12588234.html
5.12初始化安裝
kubectl apply -f recommended.yaml
#本例中，recommended.yaml配置了訪問端口30002，端口可自定，隨便。
檢查dashboard服務狀態
kubectl get service -n kubernetes-dashboard | grep dashboard
5.13 創建admin-user.yaml文件，復制或命令授權，二選其一就行==================
5.1301 創建admin-user.yaml，或者dashboard-rbac.yaml,文件名可自定，
#執行，加載到系統里
kubectl apply -f admin-user.yaml
#本例中，admin-user.yaml配置了用戶名admin-user-toen，並放在kube-system命名空間下，並將cluster-admin角色綁定到admin-user-toen賬戶，
這樣admin-user-toen賬戶就有了管理員的權限。默認情況下，kubeadm創建集群時已經創建了cluster-admin角色，我們直接綁定即可.
#查看secrets
kubectl get secrets -n kube-system | grep admin
5.1302 也可用命令進行授權，復制或命令，二選其一就行===本步驟命令未作測試，自行修改測試========================。
#創建賬號
kubectl create serviceaccount admin-user-toen -n kube-system
#授權
kubectl create clusterrolebinding admin-user-toen-rb --clusterrole=cluster-admin --serviceaccount=kube-system:admin-user-toen
#本例中，admin-user.yaml配置了用戶名admin-user-toen，並放在kube-system命名空間下，並將cluster-admin角色綁定到admin-user-toen賬戶，
這樣admin-user-toen賬戶就有了管理員的權限。默認情況下，kubeadm創建集群時已經創建了cluster-admin角色，我們直接綁定即可.
5.14 查看token
kubectl describe secret -n kube-system $(kubectl get secret -n kube-system |grep admin-user-token| awk '{print $1}')
----------------------------------------------------------------------------------------------------
admin-user-token-token-5mfmr kubernetes.io/service-account-token 3 3h30m
[root@master ~]# kubectl describe secret -n kube-system $(kubectl get secret -n kube-system |grep admin-user-token| awk '{print $1}')
Name: admin-user-token-token-5mfmr
Namespace: kube-system
Labels: <none>
Annotations: kubernetes.io/service-account.name: admin-user-token
kubernetes.io/service-account.uid: 9534f4f9-d503-40b0-8981-125fa932c05c
Type: kubernetes.io/service-account-token
Data
====
ca.crt: 1025 bytes
namespace: 11 bytes
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IngyYnAyajdxLVZUb0puZ0RmY19QMlp1c05YTkdiREhReUJyQzNzQU9ITHMifQ.
eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOi
JrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLXRva
2VuLTVtZm1yIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXItdG9r
ZW4iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI5NTM0ZjRmOS1kNTAzLTQwYjAtODk
4MS0xMjVmYTkzMmMwNWMiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06YWRtaW4tdXNlci10b2tlbiJ9.
A-g2llxoqBzQiD4YFeTkrZ_5Q0b7ayO1CorBLu13qclT4myQpoOazLkCdWiKucr2keHGcZbKaZUjqAgDOesXnZSNcU-fUvpwM1wZCiwBF_
mtLeZ3QIre8qjsOV62mXhVQ6aXxYlB27GFlE8LBQMsx-nJm1zc0RtZRW_3V5sTQKmJZF-DUt-HzbjnAA-WV0A5x_owQZ0RpCA-hqdcXh8J2W_
wZ985lEDJ1EmjNFj1yEiDXjbEWzbCr-7uQ2wanRc3U1t9SZWaXfI3QKc9uL0uTK8cdWO8DEO0kiellwRez5NeIwkqsngTHuayiHVCuL-1davQ1Zea-yhC3KAzYHKqOA
--------------------------------------------------------------------------------------------
5.15 網頁訪問集群---master的IP
https://192.168.9.3:30002
選擇---token登錄
輸入上面查到的token
===dashboard安裝完成======已經累成狗了====================================================
==============================================================================
===以下為擴展安裝===網上教程，自行測試========================================
==============================================================================
一個大神的教程：以下都是他的操作，只能說，世界太復雜了
https://www.cnblogs.com/bigberg/
部署一個簡單的例子
https://www.cnblogs.com/bigberg/p/13469637.html
6.0 部署GlusterFS
https://www.cnblogs.com/bigberg/p/13489592.html
7.0 部署Deployment
https://www.cnblogs.com/bigberg/p/13489676.html
8.0 部署RBAC
https://www.cnblogs.com/bigberg/p/13494546.html
9.0 部署StatefulSet
https://www.cnblogs.com/bigberg/p/13494583.html
10. 部署Init容器
https://www.cnblogs.com/bigberg/p/13500297.html
11.0 部署ConfigMap
https://www.cnblogs.com/bigberg/p/13500428.html
12.0 部署Secret
https://www.cnblogs.com/bigberg/p/13553373.html
13.0 在容器內獲取Pod的信息
https://www.cnblogs.com/bigberg/p/13559228.html
14.0 部署Metrics Server
https://www.cnblogs.com/bigberg/p/13559251.html
15.0 Pod健康檢查機制
https://www.cnblogs.com/bigberg/p/13559308.html
16.0 HPA Pod自動擴縮容
https://www.cnblogs.com/bigberg/p/13559410.html
17.0 Job and CronJob
https://www.cnblogs.com/bigberg/p/13563069.html
18.0 部署Ingress
https://www.cnblogs.com/bigberg/p/13563079.html
19.0 部署EFK日志系統
https://www.cnblogs.com/bigberg/p/13571127.html
20.0 部署SonarQube
https://www.cnblogs.com/bigberg/p/13575685.html
21.0 Elasticsearch + Kibana + Filebeat 日志收集
https://www.cnblogs.com/bigberg/p/13551964.html
22.0 Filebeat定義index
https://www.cnblogs.com/bigberg/p/13551987.html