Kubernetes集群搭建完畢以后可以將公司項目部署至其中
1.准備工作與注意事項
部署項目情況
1，業務架構及服務（dubbo，spring cloud）
2，第三方服務，例如mysql,redis,zookeeper,eruka,mq
3，服務之間怎么通信？
4，資源消耗：硬件資源，帶寬。
部署項目時用到的k8s資源
1，使用namespace進行不同項目隔離，或者不同環境(test,prod,dev)
2，無狀態應用（deployment）
3，有狀態應用（statefulset，pv，pvc）
4，暴露外部訪問（Service，ingress）
5，secert，configmap
2.准備基礎鏡像並推送至鏡像倉庫
3.部署PHP/Java項目
1，項目構建（java） CI/CD環境這個階段自動完成（代碼拉取->代碼編譯構建->鏡像打包->推送到鏡像倉庫）
2，編寫yaml文件，使用鏡像
kubectl->yaml->鏡像倉庫拉取鏡像->service(集群內部訪問)/Ingress暴露給外網用戶
准備環境，在192.168.1.61上配置NFS服務器共享目錄是/ifs/kubernetes
配置PV動態供給
下載nfs支持的yaml文件 下載地址https://github.com/kubernetes-incubator/external-storage/tree/master/nfs-client/deploy
下載以下三個yaml文件
# cat rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: default
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: nfs-client-provisioner-runner
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: run-nfs-client-provisioner
subjects:
- kind: ServiceAccount
name: nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: default
roleRef:
kind: ClusterRole
name: nfs-client-provisioner-runner
apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: leader-locking-nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: default
rules:
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: leader-locking-nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: default
subjects:
- kind: ServiceAccount
name: nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: default
roleRef:
kind: Role
name: leader-locking-nfs-client-provisioner
apiGroup: rbac.authorization.k8s.io
# cat class.yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: #應用使用哪一個class創建PV name: managed-nfs-storage provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME' parameters: archiveOnDelete: "false"
deployment.yaml需要修改制定的NFS服務器地址
# cat deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nfs-client-provisioner
labels:
app: nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: default
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: nfs-client-provisioner
template:
metadata:
labels:
app: nfs-client-provisioner
spec:
serviceAccountName: nfs-client-provisioner
containers:
- name: nfs-client-provisioner
image: quay.io/external_storage/nfs-client-provisioner:latest
volumeMounts:
- name: nfs-client-root
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
value: fuseim.pri/ifs
- name: NFS_SERVER
value: 192.168.1.61
- name: NFS_PATH
value: /ifs/kubernetes
volumes:
- name: nfs-client-root
nfs:
server: 192.168.1.61
path: /ifs/kubernetes
應用
kubectl apply -f rbac.yaml kubectl apply -f class.yaml kubectl apply -f deployment.yaml
部署Harbor鏡像倉庫部署地址為192.168.1.61
克隆延時代碼
git clone https://github.com/lizhenliang/tomcat-java-demo.git
准備mvn和java環境用於構建war包
yum -y install maven java-1.8.0-openjdk.x86_64
修改配置文件修改數據庫地址和密碼以后構建修改配置文件
src/main/resources/application.yml
mvn clean install -Dmaven.test.skip=true
部署tomcat是無狀態，通過service即ingrss暴露
deployment
service
ingress
部署MySQL是有狀態部署，需要使用PV進行數據存儲以及實現自動供給
statefulset
headless service
pv，pvc（storageclass）
構建好以后創建鏡像
cd tomcat-java-demo #構建鏡像包 docker build -t 192.168.1.61/project/java-demo . #推送至鏡像倉庫 docker push 192.168.1.61/project/java-demo
推送鏡像倉庫前需要登錄docker login 192.168.1.61默認用戶名密碼是admin Harbor12345
以及配置好Harbor信任
# cat /etc/docker/daemon.json
{
"registry-mirrors": ["https://7sl94zzz.mirror.aliyuncs.com"],
"insecure-registries": ["192.168.1.61"]
}
新建jave-demo文件夾用於本次部署
創建一個test命名空間
# cat namespace.yaml apiVersion: v1 kind: Namespace metadata: name: test
kubectl apply -f namespace.yaml
查看已經創建了test的命名空間
# kubectl get ns NAME STATUS AGE default Active 41h ingress-nginx Active 16h kube-public Active 41h kube-system Active 41h test Active 43s
創建create放置在命名空間test下 email隨便指定一個即可registry-pull-secret為指定secret名稱需要配置在拉取鏡像的deployment中
kubectl create secret docker-registry registry-pull-secret --docker-username=admin --docker-password=Harbor12345 --docker-email=admin@123.com -n test
PS：使用以上方式創建的secret無法拉取鏡像需要使用以下方式創建
自定義命名空間為test 需要先創建命名空間test否則無法應用會報錯
# cat registry-pull-secret.yaml apiVersion: v1 kind: Secret metadata: name: registry-pull-secret namespace: test data: .dockerconfigjson: ewoJImF1dGhzIjogewoJCSIxOTIuMTY4LjEuNjEiOiB7CgkJCSJhdXRoIjogIllXUnRhVzQ2U0dGeVltOXlNVEl6TkRVPSIKCQl9Cgl9LAoJIkh0dHBIZWFkZXJzIjogewoJCSJVc2VyLUFnZW50IjogIkRvY2tlci1DbGllbnQvMTkuMDMuNiAobGludXgpIgoJfQp9 type: kubernetes.io/dockerconfigjson # kubectl apply -f registry-pull-secret.yaml
配置文件的秘鑰在已經登錄Harbor的主機使用以下命令獲取
cat /root/.docker/config.json | base64 -w0
查看
# kubectl get secret -n test NAME TYPE DATA AGE default-token-5mnvg kubernetes.io/service-account-token 3 11m registry-pull-secret kubernetes.io/dockerconfigjson 1 50s
創建deployment
# cat deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: tomcat-java-demo
namespace: test
spec:
replicas: 3
selector:
matchLabels:
project: www
run: java-demo
template:
metadata:
labels:
project: www
run: java-demo
spec:
imagePullSecrets:
- name: registry-pull-secret
containers:
- name: tomcat
image: 192.168.1.61/project/java-demo
imagePullPolicy: Always
resources:
requests:
cpu: 0.5
memory: 1Gi
limits:
cpu: 1
memory: 2Gi
livenessProbe:
httpGet:
path: /
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 20
readinessProbe:
httpGet:
path: /
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 20
創建
kubectl apply -f deployment.yaml
查看 命名空間是test
# kubectl get pod -n test NAME READY STATUS RESTARTS AGE tomcat-java-demo-66466d7bc7-27q4v 1/1 Running 0 23m tomcat-java-demo-66466d7bc7-hjdgt 1/1 Running 0 23m tomcat-java-demo-66466d7bc7-x8zm9 1/1 Running 0 23m
使用service發布剛剛運行的應用
# cat service.yaml
apiVersion: v1
kind: Service
metadata:
creationTimestamp: null
name: tomcat-java-demo
namespace: test
spec:
ports:
- port: 80
protocol: TCP
targetPort: 8080
selector:
project: www
run: java-demo
type: NodePort
status:
loadBalancer: {}
PS:注意service的命名空間也需要設置為test否則創建的service無法訪問后端的tomcat
應用
kubectl apply -f service.yaml
查看
# kubectl get svc -n test NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE tomcat-java-demo NodePort 10.0.0.44 <none> 80:40995/TCP 4m52s
沒有部署ingress之前可以使用node ip加隨機的nodeport端口進行訪問
部署ingress 部署ingress之前需要確保已經部署好ingress控制器 查看是否已經部署控制器使用命令
kubectl get pod -n ingress-nginx
# cat ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: simple-fanout-example
namespace: test
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
#定義訪問域名
- host: java.ctnrs.com
http:
paths:
- path: /
backend:
#轉發到哪個server下
serviceName: tomcat-java-demo
servicePort: 80
應用
kubectl apply -f ingress.yaml
查看
# kubectl get ingress -n test NAME HOSTS ADDRESS PORTS AGE simple-fanout-example java.ctnrs.com 80 7s
設置hosts可以通過域名訪問http://java.ctnrs.com/
但是沒有配置MySQL只能看見頁面無法實現功能
部署mysql
# cat mysql.yaml
apiVersion: v1
kind: Service
metadata:
name: mysql
labels:
project: java-demo
run: mysql
spec:
ports:
- port: 3306
name: mysql
#不需要clusterIP
clusterIP: None
selector:
project: java-demo
run: mysql
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: db
spec:
selector:
matchLabels:
project: java-demo
run: mysql
serviceName: "mysql"
template:
metadata:
labels:
project: java-demo
run: mysql
spec:
containers:
- name: mysql
image: mysql:5.7
ports:
- containerPort: 3306
env:
- name: MYSQL_ROOT_PASSWORD
value: "123456"
volumeMounts:
- mountPath: /var/lib/mysql
name: data
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: "managed-nfs-storage"
resources:
requests:
storage: 2Gi
應用
kubectl apply -f mysql.yaml
查看
在對應的nfs服務器會創建一個文件夾用於持久保存MySQL數據
查看pv和pvc
kubectl get pv,pvc
MySQL的連接地址是db-0.mysql.default 可以創建一個busybox測試
kubectl run -it --rm --image=busybox bash
# ping db-0.mysql.default PING db-0.mysql.default (172.17.9.3): 56 data bytes 64 bytes from 172.17.9.3: seq=0 ttl=64 time=0.084 ms 64 bytes from 172.17.9.3: seq=1 ttl=64 time=0.076 ms
修改項目的數據庫配置
cd tomcat-java-demo vim src/main/resources/application.yml
重新打包
mvn clean install -Dmaven.test.skip=true
重新打包docker鏡像
docker build -t 192.168.1.61/project/java-demo:1.0 .
推送至Harbor鏡像倉庫
docker push 192.168.1.61/project/java-demo:1.0
登錄私有鏡像倉庫查看已經有了1.0版本
修改deployment.yaml修改一下版本號
滾動更新
kubectl apply -f deployment.yaml
滾動更新是先現在新的鏡像包啟動以后再刪除原有包
拷貝數據庫腳本至容器
kubectl cp tables_ly_tomcat.sql db-0:/
登錄容器導入數據庫表
#登錄容器 kubectl exec -it db-0 bash #登錄數據庫 mysql -uroot -p123456 #導入表 mysql> source tables_ly_tomcat.sql;
因為連接了數據庫所以登錄web頁面java.ctnrs.com即可實現該程序的簡單功能添加信息
部署PHP項目
本次部署WordPress
下載示例代碼
git clone https://github.com/lizhenliang/php-demo
修改數據庫配置
#cd php-demo vim wp-config.php
查看Dockerfile文件
# cat Dockerfile FROM lizhenliang/nginx-php:latest MAINTAINER www.ctnrs.com ADD . /usr/local/nginx/html
構建提交到私有倉庫
docker build -t 192.168.1.61/project/php-demo:1.0 . docker push 192.168.1.61/project/php-demo:1.0
刪除剛剛部署java的namespace為test的命名空間
kubectl delete -f namespace.yaml
刪除剛剛部署的數據庫MySQL
kubectl delete -f mysql.yaml
查看是否已經刪除干凈
#命名空間test的pod svc ingress都沒有了 # kubectl get pod,svc,ingress -n test No resources found. #數據庫db-0的坡道也沒有了，保留了nfs # kubectl get pod NAME READY STATUS RESTARTS AGE nfs-client-provisioner-7665588bd7-66q7w 1/1 Running 0 69m
創建namespace名字為test
# cat namespace.yaml apiVersion: v1 kind: Namespace metadata: name: test #kubectl apply -f namespace.yaml
查看
# kubectl get ns NAME STATUS AGE default Active 2d17h ingress-nginx Active 41h kube-public Active 2d17h kube-system Active 2d17h test Active 17s
創建Harbor加密認證
# cat registry-pull-secret.yaml apiVersion: v1 kind: Secret metadata: name: registry-pull-secret namespace: test data: .dockerconfigjson: ewoJImF1dGhzIjogewoJCSIxOTIuMTY4LjEuNjEiOiB7CgkJCSJhdXRoIjogIllXUnRhVzQ2U0dGeVltOXlNVEl6TkRVPSIKCQl9Cgl9LAoJIkh0dHBIZWFkZXJzIjogewoJCSJVc2VyLUFnZW50IjogIkRvY2tlci1DbGllbnQvMTkuMDMuNiAobGludXgpIgoJfQp9 type: kubernetes.io/dockerconfigjson # kubectl apply -f registry-pull-secret.yaml
查看
# kubectl get secret -n test NAME TYPE DATA AGE default-token-kklxx kubernetes.io/service-account-token 3 86s registry-pull-secret kubernetes.io/dockerconfigjson 1 9s
創建php deployment
# cat deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: php-demo
namespace: test
spec:
replicas: 3
selector:
matchLabels:
project: www
run: php-demo
template:
metadata:
labels:
project: www
run: php-demo
spec:
imagePullSecrets:
- name: registry-pull-secret
containers:
- name: php
image: 192.168.1.61/project/php-demo:1.0
imagePullPolicy: Always
resources:
requests:
cpu: 0.5
memory: 1Gi
limits:
cpu: 1
memory: 2Gi
livenessProbe:
httpGet:
path: /
port: 80
initialDelaySeconds: 60
timeoutSeconds: 20
readinessProbe:
httpGet:
path: /
port: 80
initialDelaySeconds: 60
timeoutSeconds: 20
# kubectl apply -f deployment.yaml
查看
# kubectl get pod -n test NAME READY STATUS RESTARTS AGE php-demo-5c4b869ffd-2cwzr 0/1 Running 0 15s php-demo-5c4b869ffd-7wt48 0/1 Running 0 15s php-demo-5c4b869ffd-gsm4t 0/1 Running 0 15s
創建service對應剛剛創建的deployment
# cat service.yaml
apiVersion: v1
kind: Service
metadata:
creationTimestamp: null
name: php-demo
namespace: test
spec:
ports:
- port: 80
protocol: TCP
targetPort: 80
selector:
project: www
run: php-demo
type: NodePort
status:
loadBalancer: {}
# kubectl apply -f service.yaml
service/php-demo created
查看
# kubectl get svc -n test NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE php-demo NodePort 10.0.0.14 <none> 80:49142/TCP 14s
創建ingress 創建ingress之前確保已經部署了ingress-nginx
# kubectl get pod -n ingress-nginx NAME READY STATUS RESTARTS AGE nginx-ingress-controller-42gsg 1/1 Running 2 41h nginx-ingress-controller-5r29z 1/1 Running 2 41h
# cat ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: simple-fanout-example
namespace: test
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
#定義訪問域名
- host: php.ctnrs.com
http:
paths:
- path: /
backend:
#轉發到哪個server下
serviceName: php-demo
servicePort: 80
# kubectl apply -f ingress.yaml
ingress.extensions/simple-fanout-example created
查看
# kubectl get ingress -n test NAME HOSTS ADDRESS PORTS AGE simple-fanout-example php.ctnrs.com 80 25s
設置hosts以后即可使用域名php.ctnrs.com訪問