機器准備
30.128 四層代理服務器 節點上需要安裝rke kubectl helm nginx
128節點需要和其它三個節點建立ssh互信實現免密登錄 rke是通過ssh登錄來遠程操作其它節點的
128主機名設置為cf.rancher.com
30.110 k8s-node1
30.112 k8s-node2
30.113 k8s-node3
110，112，113三個節點通過rke部署一個k8s集群 這個集群只用來運行Rancher Server實現Rancher Server的高可用
rke安裝k8s
安裝 rke up --config=./rancher-cluster.yml
卸載 rke remove --config=./rancher-cluster.yml
nodes: - address: 192.168.30.110 user: admin role: [controlplane,worker,etcd] - address: 192.168.30.129 user: admin role: [controlplane,worker,etcd] - address: 192.168.30.133 user: admin role: [controlplane,worker,etcd] services: etcd: snapshot: true creation: 6h retention: 24h ingress: provider: nginx options: use-forwarded-headers: "true"
mkdir ~/.kube
cp kube_config_rancher-cluster.yml ~/.kube/config
表示k8s集群安裝成功
Helm安裝rancher
helm repo add rancher-stable http://rancher-mirror.oss-cn-beijing.aliyuncs.com/server-charts/stable
kubectl create namespace cattle-system
kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.15.0/cert-manager.crds.yaml
kubectl create namespace cert-manager
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager --namespace cert-manager --version v0.15.0
helm install rancher rancher-stable/rancher --namespace cattle-system --set hostname=cf.rancher.com
nginx中證書對應的域名需要與hostname選項匹配,否則ingress將無法代理訪問Rancher
Nginx配置反向代理訪問RancherUI
在128節點上配置
[admin conf.d]$ nginx -v
nginx version: nginx/1.18.0
1.配置主機名
2.生成自簽名證書
#!/bin/bash -e help () { echo ' ================================================================ ' echo ' --ssl-domain: 生成ssl證書需要的主域名，如不指定則默認為www.rancher.local，如果是ip訪問服務，則可忽略；' echo ' --ssl-trusted-domain: 如果想多個域名訪問，則添加擴展域名（SSL_TRUSTED_DOMAIN）,多個擴展域名用逗號隔開；' echo ' --ssl-size: ssl加密位數，默認2048；' echo ' --ssl-cn: 國家代碼(2個字母的代號),默認CN;' echo ' 使用示例:' echo ' ./create_self-signed-cert.sh --ssl-domain=www.test.com --ssl-trusted-domain=www.test2.com \ ' echo ' --ssl-trusted-ip=1.1.1.1,2.2.2.2,3.3.3.3 --ssl-size=2048 --ssl-date=3650' echo ' ================================================================' } case "$1" in -h|--help) help; exit;; esac if [[ $1 == '' ]];then help; exit; fi CMDOPTS="$*" for OPTS in $CMDOPTS; do key=$(echo ${OPTS} | awk -F"=" '{print $1}' ) value=$(echo ${OPTS} | awk -F"=" '{print $2}' ) case "$key" in --ssl-domain) SSL_DOMAIN=$value ;; --ssl-trusted-ip) SSL_TRUSTED_IP=$value ;; --ssl-trusted-domain) SSL_TRUSTED_DOMAIN=$value ;; --ssl-size) SSL_SIZE=$value ;; --ssl-date) SSL_DATE=$value ;; --ca-date) CA_DATE=$value ;; --ssl-cn) CN=$value ;; esac done # CA相關配置 CA_DATE=${CA_DATE:-3650} CA_KEY=${CA_KEY:-cakey.pem} CA_CERT=${CA_CERT:-cacerts.pem} CA_DOMAIN=cattle-ca # ssl相關配置 SSL_CONFIG=${SSL_CONFIG:-$PWD/openssl.cnf} SSL_DOMAIN=${SSL_DOMAIN:-'www.rancher.local'} SSL_DATE=${SSL_DATE:-3650} SSL_SIZE=${SSL_SIZE:-2048} ## 國家代碼(2個字母的代號),默認CN; CN=${CN:-CN} SSL_KEY=$SSL_DOMAIN.key SSL_CSR=$SSL_DOMAIN.csr SSL_CERT=$SSL_DOMAIN.crt echo -e "\033[32m ---------------------------- \033[0m" echo -e "\033[32m | 生成 SSL Cert | \033[0m" echo -e "\033[32m ---------------------------- \033[0m" if [[ -e ./${CA_KEY} ]]; then echo -e "\033[32m ====> 1. 發現已存在CA私鑰，備份"${CA_KEY}"為"${CA_KEY}"-bak，然后重新創建 \033[0m" mv ${CA_KEY} "${CA_KEY}"-bak openssl genrsa -out ${CA_KEY} ${SSL_SIZE} else echo -e "\033[32m ====> 1. 生成新的CA私鑰 ${CA_KEY} \033[0m" openssl genrsa -out ${CA_KEY} ${SSL_SIZE} fi if [[ -e ./${CA_CERT} ]]; then echo -e "\033[32m ====> 2. 發現已存在CA證書，先備份"${CA_CERT}"為"${CA_CERT}"-bak，然后重新創建 \033[0m" mv ${CA_CERT} "${CA_CERT}"-bak openssl req -x509 -sha256 -new -nodes -key ${CA_KEY} -days ${CA_DATE} -out ${CA_CERT} -subj "/C=${CN}/CN=${CA_DOMAIN}" else echo -e "\033[32m ====> 2. 生成新的CA證書 ${CA_CERT} \033[0m" openssl req -x509 -sha256 -new -nodes -key ${CA_KEY} -days ${CA_DATE} -out ${CA_CERT} -subj "/C=${CN}/CN=${CA_DOMAIN}" fi echo -e "\033[32m ====> 3. 生成Openssl配置文件 ${SSL_CONFIG} \033[0m" cat > ${SSL_CONFIG} <<EOM [req] req_extensions = v3_req distinguished_name = req_distinguished_name [req_distinguished_name] [ v3_req ] basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment extendedKeyUsage = clientAuth, serverAuth EOM if [[ -n ${SSL_TRUSTED_IP} || -n ${SSL_TRUSTED_DOMAIN} ]]; then cat >> ${SSL_CONFIG} <<EOM subjectAltName = @alt_names [alt_names] EOM IFS="," dns=(${SSL_TRUSTED_DOMAIN}) dns+=(${SSL_DOMAIN}) for i in "${!dns[@]}"; do echo DNS.$((i+1)) = ${dns[$i]} >> ${SSL_CONFIG} done if [[ -n ${SSL_TRUSTED_IP} ]]; then ip=(${SSL_TRUSTED_IP}) for i in "${!ip[@]}"; do echo IP.$((i+1)) = ${ip[$i]} >> ${SSL_CONFIG} done fi fi echo -e "\033[32m ====> 4. 生成服務SSL KEY ${SSL_KEY} \033[0m" openssl genrsa -out ${SSL_KEY} ${SSL_SIZE} echo -e "\033[32m ====> 5. 生成服務SSL CSR ${SSL_CSR} \033[0m" openssl req -sha256 -new -key ${SSL_KEY} -out ${SSL_CSR} -subj "/C=${CN}/CN=${SSL_DOMAIN}" -config ${SSL_CONFIG} echo -e "\033[32m ====> 6. 生成服務SSL CERT ${SSL_CERT} \033[0m" openssl x509 -sha256 -req -in ${SSL_CSR} -CA ${CA_CERT} \ -CAkey ${CA_KEY} -CAcreateserial -out ${SSL_CERT} \ -days ${SSL_DATE} -extensions v3_req \ -extfile ${SSL_CONFIG} echo -e "\033[32m ====> 7. 證書制作完成 \033[0m" echo echo -e "\033[32m ====> 8. 以YAML格式輸出結果 \033[0m" echo "----------------------------------------------------------" echo "ca_key: |" cat $CA_KEY | sed 's/^/ /' echo echo "ca_cert: |" cat $CA_CERT | sed 's/^/ /' echo echo "ssl_key: |" cat $SSL_KEY | sed 's/^/ /' echo echo "ssl_csr: |" cat $SSL_CSR | sed 's/^/ /' echo echo "ssl_cert: |" cat $SSL_CERT | sed 's/^/ /' echo echo -e "\033[32m ====> 9. 附加CA證書到Cert文件 \033[0m" cat ${CA_CERT} >> ${SSL_CERT} echo "ssl_cert: |" cat $SSL_CERT | sed 's/^/ /' echo echo -e "\033[32m ====> 10. 重命名服務證書 \033[0m" echo "cp ${SSL_DOMAIN}.key tls.key" cp ${SSL_DOMAIN}.key tls.key echo "cp ${SSL_DOMAIN}.crt tls.crt" cp ${SSL_DOMAIN}.crt tls.crt
./create_self-signed-cert.sh --ssl-domain=cf.rancher.com --ssl-trusted-ip=192.168.30.128 --ssl-size=2048 --ssl-date=3650
3. 配置nginx文件/etc/nginx/conf.d
upstream rancher { server 192.168.30.110:80; server 192.168.30.129:80; server 192.168.30.133:80; } map $http_upgrade $connection_upgrade { default Upgrade; '' close; } server { listen 443 ssl http2; server_name cf.rancher.com; ssl_certificate /home/admin/certs/tls.crt; ssl_certificate_key /home/admin/certs/tls.key; location / { proxy_set_header Host $host; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Port $server_port; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_pass http://rancher; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection $connection_upgrade; # 這里將允許您在 Rancher UI 中打開命令行窗口時，窗口可以保留最多15分鍾。沒有這個參數時，默認值為1分鍾，一分鍾后在Rancher>中的shell會自動關閉。 proxy_read_timeout 900s; proxy_buffering off; } } server { listen 80; server_name cf.rancher.com; return 301 https://$server_name$request_uri; }
4.重新啟動nginx
windows客戶端配置
修改 c:\windows\system32\drivers\etc\hosts文件
打開瀏覽器訪問(只能用域名訪問,不能用nginx服務器的IP訪問)
為Agent Pod添加主機別名(/etc/hosts)
如果沒有內部DNS服務器而是通過添加/etc/hosts主機別名的方式指定的Rancher Server域名,那么不管通過哪種方式(自定義、導入、Host驅動等)創建K8S集群,K8S集群運行起來之后,因為cattle-cluster-agent Pod和cattle-node-agent pod無法通過DNS記錄找到Rancher Server URL,最終導致無法通信
1.在local集群中執行以下命令為Rancher Server容器配置hosts
kubectl --kubeconfig=$kubeconfig -n cattle-system \ patch deployments rancher --patch '{ "spec": { "template": { "spec": { "hostAliases": [ { "hostnames": [ "xxx.cnrancher.com" ], "ip": "192.168.1.100" } ] } } } }'
2.cattle-cluster-agent Pod和cattle-node-agent pod需要在LOCAL集群初始化之后才會部署,所以先通過Rancher Server URL訪問Rancher Web UI進行初始化
3.cattle-cluster-agent pod（local集群和業務集群均執行）
kubectl --kubeconfig=$kubeconfig -n cattle-system \ patch deployments cattle-cluster-agent --patch '{ "spec": { "template": { "spec": { "hostAliases": [ { "hostnames": [ "demo.cnrancher.com" ], "ip": "192.168.1.100" } ] } } } }'
4.cattle-node-agent pod（local集群和業務集群均執行）
kubectl --kubeconfig=$kubeconfig -n cattle-system \ patch daemonsets cattle-node-agent --patch '{ "spec": { "template": { "spec": { "hostAliases": [ { "hostnames": [ "xxx.rancher.com" ], "ip": "192.168.1.100" } ] } } } }'
自簽名證書
1.將CA證書以pem格式復制到名為ca-additional.pem的文件中,並用kubectl在命名空間cattle-system中創建tls-ca-additional密文
kubectl -n cattle-system create secret generic tls-ca-additional --from-file=ca-additional.pem
2.在Helm部署Rancher Server時,添加以下參數
helm install rancher rancher-stable/rancher --namespace cattle-system --set hostname=cf.rancher.com --set additionalTrustedCAs=true
使用自定義證書的完整部署步驟
1.在負載均衡器上執行一鍵生成ssl自簽名證書腳本將自動生成tls.crt、tls.key、cacerts.pem三個文件。文件名稱不能修改。ca文件名稱必須是cacerts.pem
2.使用kubectl在命名空間cattle-system中創建tls-ca secret
kubectl create namespace cattle-system
kubectl -n cattle-system create secret generic tls-ca --from-file=/home/admin/certs/cacerts.pem
3.安裝Rancher server
helm install rancher rancher-stable/rancher \
--namespace cattle-system \
--set hostname=cf.rancher.com \
--set ingress.tls.source=secret \
--set privateCA=true
4.修改pod中的hosts解析
kubectl -n cattle-system \ patch deployments rancher --patch '{ "spec": { "template": { "spec": { "hostAliases": [ { "hostnames": [ "cf.rancher.com" ], "ip": "192.168.30.128" } ] } } } }'
5.最終部署結果
所有的pod都能正常運行起來