基於華為雲的k8s集群部署項目: nginx+php容器，持久化存儲並公網發布

主機配置規划：

服務器名稱(hostname)	系統版本	配置	內網IP	外網IP(模擬)	說明
JumpSrv	CentOS 7.5	2C/4G/40G	192.168.1.252	124.70.***.***	跳板機，公網發布
registry	CentOS 7.5	2C/4G/40G	192.168.1.100		鏡像倉庫，公網下載的鏡像會上傳至此
master	CentOS 7.5	2C/4G/40G	192.168.1.21		master節點
node-0001	CentOS 7.5	2C/4G/40G	192.168.1.31		node節點
node-0002	CentOS 7.5	2C/4G/40G	192.168.1.32		node節點
node-0003	CentOS 7.5	2C/4G/40G	192.168.1.33		node節點

 

架構圖：

 

簡要說明：

• Ingress的作用是將容器提供的服務發布至集群外訪問，他是對集群中服務的外部訪問進行管理的 API 對象, Ingress 可以提供負載均衡、SSL 和基於名稱的虛擬托管。必須具有 ingress 控制器【例如 ingress-nginx】才能滿足 Ingress 的要求。僅創建 Ingress 資源無效。
• Service的作用和負載均衡的作用及其相似。
• 將nginx容器和php容器同時部署在同一個pod里，解析php動態頁面.(同一個pod里面的容器共享主機名和網絡命名空間)
• 使用hostPath卷分別在node中存放日志文件，使用emptyDir卷存放緩存文件（也是在pode所在的node中存放），使用NFS結合PV,PVC存放網頁頁面信息！
• 本文將不同部分的yaml文件拆分編寫（為了理解和學習）
• 省略制作私有鏡像倉庫的過程，本項目的倉庫列表：

[root@master ingress]# curl http://192.168.1.100:5000/v2/_catalog
{"repositories":["coredns","etcd","flannel","kube-apiserver","kube-controller-manager","kube-proxy","kube-scheduler","metrics-server","myos","nginx-ingress-controller","pause"]}
[root@master ingress]# curl http://192.168.1.100:5000/v2/myos/tags/list
{"name":"myos","tags":["php-fpm","httpd","nginx","v1804"]}
[root@master ingress]# curl http://192.168.1.100:5000/v2/nginx-ingress-controller/tags/list
{"name":"nginx-ingress-controller","tags":["0.30.0"]}

• 如何從公網獲得Ingress的鏡像

docker pull registry.cn-beijing.aliyuncs.com/google_registry/nginx-ingress-controller:0.30.0

或者：
wget https://github.com/kubernetes/ingress-nginx/archive/nginx-0.30.0.tar.gz
tar xf nginx-0.30.0.tar.gz
yaml文件在下載包中的位置：ingress-nginx-nginx-0.30.0/deploy/static/mandatory.yaml

 

步驟：

1, 安裝Ingress插件：

[root@master ingress]# curl http://192.168.1.100:5000/v2/nginx-ingress-controller/tags/list
{"name":"nginx-ingress-controller","tags":["0.30.0"]}
[root@master ~]# vim ingress/mandatory.yaml 
221:  image: 192.168.1.100:5000/nginx-ingress-controller:0.30.0
[root@master ~]# kubectl apply -f ingress/mandatory.yaml 
[root@master ~]# kubectl -n ingress-nginx get pod
NAME                                      READY   STATUS    RESTARTS   AGE
nginx-ingress-controller-fc6766d7-ptppp   1/1     Running   0          47s
[root@master ingress]#

 

2, 在registry,master,node-0001,node-0002,node-0003安裝nfs,並在registry上面nfs共享文件夾:

[root@registry ~]# yum install -y nfs-utils
[root@registry ~]# mkdir -m 777 /var/webroot
[root@registry ~]# vim  /etc/exports
/var/webroot    *(rw)
[root@registry ~]# systemctl enable --now nfs
#---------------------------------所有節點都需要 nfs 軟件包-------------------------
[root@node-0001 ~]# yum install -y nfs-utils
#--------------------------------------------------------------------------------
[root@node-0002 ~]# yum install -y nfs-utils
#--------------------------------------------------------------------------------
[root@node-0003 ~]# yum install -y nfs-utils
#-------------------------------下面在任意其他節點測試------------------------------
[root@master ~]# yum install -y nfs-utils
[root@master ~]# showmount -e 192.168.1.100
Export list for 192.168.1.100:
/var/webroot *

3, 創建PV:

[root@master ~]# vim mypv.yaml 
---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: pv-nfs
spec:
  volumeMode: Filesystem
  capacity:
    storage: 30Gi
  accessModes:
  - ReadWriteOnce
  - ReadOnlyMany
  - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  nfs:
    server: 192.168.1.100
    path: /var/webroot

[root@master ~]# kubectl apply -f mypv.yaml 
persistentvolume/pv-nfs created
[root@master ~]# kubectl get pv
NAME     CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS          AGE
pv-nfs   30Gi       RWO,ROX,RWX    Retain           Available       3s

4, 創建PVC:

[root@master configmap]# vim mypvc.yaml 
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-nfs
spec:
  volumeMode: Filesystem
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 25Gi

[root@master configmap]# kubectl apply -f mypvc.yaml
[root@master configmap]# kubectl get pv
NAME     CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM
pv-nfs   30Gi       RWX            Retain           Bound    default/pvc-nfs
[root@master configmap]# kubectl get pvc
NAME      STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc-nfs   Bound    pv-nfs   30Gi       RWO,ROX,RWX                   27s

5，創建configMap（根據nginx的配置文件創建）

[root@master ~]# vim /var/webconf/nginx.conf 
... ...
        location ~ \.php$ {
            root           html;
            fastcgi_pass   127.0.0.1:9000;
            fastcgi_index  index.php;
            include        fastcgi.conf;
        }
... ...
[root@master ~]# kubectl create configmap nginx-conf --from-file=/var/webconf/nginx.conf 
configmap/nginx-conf created
[root@master ~]# kubectl get configmaps 
NAME         DATA   AGE
nginx-conf   1      8s

6, 編寫yaml文件部署nginx+php容器：

vim webnginx.yaml

---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: webnginx
spec:
  selector:
    matchLabels:
      myapp: nginx
  replicas: 3
  template:
    metadata:
      labels:
        myapp: nginx
    spec:
      volumes:
      - name: nginx-conf
        configMap:
          name: nginx-conf
      - name: cache-data
        emptyDir: {}
      - name: log-data
        hostPath:
          path: /var/weblog
          type: DirectoryOrCreate
      - name: website
        persistentVolumeClaim:
          claimName: pvc-nfs
      containers:
      - name: nginx
        image: 192.168.1.100:5000/myos:nginx
        volumeMounts:
        - name: nginx-conf
          subPath: nginx.conf
          mountPath: /usr/local/nginx/conf/nginx.conf
        - name: cache-data
        emptyDir: {}
      - name: log-data
        hostPath:
          path: /var/weblog
          type: DirectoryOrCreate
      - name: website
        persistentVolumeClaim:
          claimName: pvc-nfs
      containers:
      - name: nginx
        image: 192.168.1.100:5000/myos:nginx
        volumeMounts:
        - name: nginx-conf
          subPath: nginx.conf
          mountPath: /usr/local/nginx/conf/nginx.conf
        - name: cache-data
          mountPath: /var/cache
        - name: log-data
          mountPath: /usr/local/nginx/logs
        - name: website
          mountPath: /usr/local/nginx/html
        ports:
        - protocol: TCP
          containerPort: 80
      - name: php-backend
        image: 192.168.1.100:5000/myos:php-fpm
        volumeMounts:
        - name: website
          mountPath: /usr/local/nginx/html
      restartPolicy: Always

7, 編寫service文件：

vim clusterip.yaml

---
kind: Service
apiVersion: v1
metadata:
  name: myweb
spec:
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
  selector:
    myapp: nginx
  type: ClusterIP

8, 創建ingress資源文件：

vim ingress-example.yaml

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-app
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  backend:
    serviceName: myweb
    servicePort: 80

 9，運行上面的yaml文件：(實際操作中是將這些yaml文件合為一個，此處為了學習理解架構！)

kubectl apply -f webnginx.yaml
kubectl apply -f clusterip.yaml
kubectl apply -f ingress-example.yaml

10, 將運行ingress nginx pod的node (node-0002) 發布至公網(綁定彈性公網IP，或使用 ELB 發布到互聯網即可驗證).

[root@master ingress]# kubectl get ingresses
NAME     HOSTS   ADDRESS        PORTS   AGE
my-app   *       192.168.1.32   80      160m