1、基本概念
對於復雜的應用中間件，需要設置鏡像運行的需求、環境變量，並且需要定制存儲、網絡等設置，最后設計和編寫Deployment、Configmap、Service及Ingress等相關yaml配置文件，再提交給kubernetes進行部署。這些復雜的過程將逐步被Helm應用包管理工具實現。
Helm是一個由CNCF孵化和管理的項目，用於對需要在k8s上部署復雜應用進行定義、安裝和更新。Helm以Chart的方式對應用軟件進行描述，可以方便地創建、版本化、共享和發布復雜的應用軟件。
Chart：一個Helm包，其中包含了運行一個應用所需要的工具和資源定義，還可能包含kubernetes集群中的服務定義，類似於Homebrew中的formula、apt中的dpkg或者yum中的rpm文件。
Release：在K8S集群上運行一個Chart實例。在同一個集群上，一個Chart可以安裝多次，例如有一個MySQL Chart，如果想在服務器上運行兩個數據庫，就可以基於這個Chart安裝兩次。每次安裝都會生成新的Release，會有獨立的Release名稱。
Repository：用於存放和共享Chart的倉庫。
簡單來說，Helm的任務是在倉庫中查找需要的Chart，然后將Chart以Release的形式安裝到K8S集群中。
Harbor基本概念：此篇文章很不錯
2、Helm安裝
Helm由兩個組件組成：
- HelmClinet：客戶端，擁有對Repository、Chart、Release等對象的管理能力。
TillerServer：負責客戶端指令和k8s集群之間的交互，根據Chart定義，生成和管理各種k8s的資源對象。
安裝HelmClient：可以通過二進制文件或腳本方式進行安裝。
下載最新版二進制文件：https://github.com/helm/helm/releases
[root@k8s-master01 ~]# tar xf helm-v2.11.0-linux-amd64.tar.gz [root@k8s-master01 ~]# cp linux-amd64/helm linux-amd64/tiller /usr/local/bin/
[root@k8s-master01 ~]# helm version Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"} Error: could not find tiller # 因為沒有安裝tillerServer所以會報找不到tiller
安裝TillerServer
所有節點下載tiller:v[helm-version]鏡像，helm-version為上面helm的版本2.11.0
docker pull dotbalo/tiller:v2.11.0
yum install socat -y
使用helm init安裝tiller
[root@k8s-master01 ~]# helm init --tiller-image dotbalo/tiller:v2.11.0 Creating /root/.helm Creating /root/.helm/repository Creating /root/.helm/repository/cache Creating /root/.helm/repository/local Creating /root/.helm/plugins Creating /root/.helm/starters Creating /root/.helm/cache/archive Creating /root/.helm/repository/repositories.yaml Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com Adding local repo with URL: http://127.0.0.1:8879/charts $HELM_HOME has been configured at /root/.helm. Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster. Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy. To prevent this, run `helm init` with the --tiller-tls-verify flag. For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation Happy Helming!
再次查看helm version及pod狀態
[root@k8s-master01 ~]# helm version Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"} Server: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
[root@k8s-master01 ~]# kubectl get pod -n kube-system | grep tiller tiller-deploy-5d7c8fcd59-d4djx 1/1 Running 0 49s [root@k8s-master01 ~]# kubectl get pod,svc -n kube-system | grep tiller pod/tiller-deploy-5d7c8fcd59-d4djx 1/1 Running 0 3m service/tiller-deploy ClusterIP 10.106.28.190 <none> 44134/TCP 5m
3、Helm使用
3.1 helm search：搜索可用的Chart
Helm初始化完成之后，默認配置為使用官方的k8s chart倉庫
通過search查找可用的Chart
[root@k8s-master01 ~]# helm search gitlab NAME CHART VERSION APP VERSION DESCRIPTION stable/gitlab-ce 0.2.2 9.4.1 GitLab Community Edition stable/gitlab-ee 0.2.2 9.4.1 GitLab Enterprise Edition [root@k8s-master01 ~]# helm search | more NAME CHART VERSION APP VERSION DESCRIPTION stable/acs-engine-autoscaler 2.2.0 2.1.1 Scales worker n odes within agent pools stable/aerospike 0.1.7 v3.14.1.2 A Helm chart fo r Aerospike in Kubernetes stable/anchore-engine 0.9.0 0.3.0 Anchore contain er analysis and policy evaluation engine s... stable/apm-server 0.1.0 6.2.4 The server rece ives data from the Elastic APM agents and ... stable/ark 1.2.2 0.9.1 A Helm chart fo r ark stable/artifactory 7.3.1 6.1.0 DEPRECATED Univ ersal Repository Manager supporting all ma... stable/artifactory-ha 0.4.1 6.2.0 DEPRECATED Univ ersal Repository Manager supporting all ma... stable/auditbeat 0.3.1 6.4.3 A lightweight s hipper to audit the activities of users an... --More--
查看詳細信息
[root@k8s-master01 ~]# helm search gitlab NAME CHART VERSION APP VERSION DESCRIPTION stable/gitlab-ce 0.2.2 9.4.1 GitLab Community Edition stable/gitlab-ee 0.2.2 9.4.1 GitLab Enterprise Edition [root@k8s-master01 ~]# helm inspect stable/gitlab-ce
3.2 Helm install harbor
使用helm repo remove和add刪除repository和添加aliyun的repository
[root@k8s-master01 harbor-helm]# helm repo list NAME URL aliyun https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts
下載harbor，並checkout到0.3.0分支
git clone https://github.com/goharbor/harbor-helm.git
更改requirement.yaml如下
[root@k8s-master01 harbor-helm]# cat requirements.yaml dependencies: - name: redis version: 1.1.15 repository: https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts #repository: https://kubernetes-charts.storage.googleapis.com
下載依賴
[root@k8s-master01 harbor-helm]# helm dependency update Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "aliyun" chart repository Update Complete. ⎈Happy Helming!⎈ Saving 1 charts Downloading redis from repo https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts Deleting outdated charts
所有節點下載相關鏡像
docker pull goharbor/chartmuseum-photon:v0.7.1-v1.6.0 docker pull goharbor/harbor-adminserver:v1.6.0 docker pull goharbor/harbor-jobservice:v1.6.0 docker pull goharbor/harbor-ui:v1.6.0 docker pull goharbor/harbor-db:v1.6.0 docker pull goharbor/registry-photon:v2.6.2-v1.6.0 docker pull goharbor/chartmuseum-photon:v0.7.1-v1.6.0 docker pull goharbor/clair-photon:v2.0.5-v1.6.0 docker pull goharbor/notary-server-photon:v0.5.1-v1.6.0 docker pull goharbor/notary-signer-photon:v0.5.1-v1.6.0
docker pull bitnami/redis:4.0.8-r2
更改values.yaml所有的storageClass為storageClass: "gluster-heketi"
注意修改values.yaml的redis默認配置，添加port至master
master: port: 6379
注意修改charts/redis-1.1.15.tgz 里面的redis的values.yaml的storageClass也為"gluster-heketi"，usePassword為 false
注意修改charts/redis-1.1.15.tgz 里面的redis下template下的svc的name: {{ template "redis.fullname" . }}-master
注意修改相關存儲空間的大小，比如registry。
安裝harbor
helm install --name harbor-v1 . --wait --timeout 1500 --debug --namespace harbor
如果報forbidden的錯誤，需要創建serveraccount
[root@k8s-master01 harbor-helm]# helm install --name harbor-v1 . --set externalDomain=harbor.xxx.net --wait --timeout 1500 --debug --namespace harbor [debug] Created tunnel using local port: '35557' [debug] SERVER: "127.0.0.1:35557" [debug] Original chart version: "" [debug] CHART PATH: /root/harbor-helm Error: release harbor-v1 failed: namespaces "harbor" is forbidden: User "system:serviceaccount:kube-system:default" cannot get namespaces in the namespace "harbor"
解決：
kubectl create serviceaccount --namespace kube-system tiller kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'
再次部署：
...... ==> v1/Pod(related) NAME READY STATUS RESTARTS AGE harbor-v1-redis-84dffd8574-xzrsh 0/1 Running 0 <invalid> harbor-v1-harbor-adminserver-5b59c684b4-g6cjc 1/1 Running 0 <invalid> harbor-v1-harbor-chartmuseum-699cf6599-q6vfw 1/1 Running 0 <invalid> harbor-v1-harbor-clair-6d9bb84485-2p52v 1/1 Running 0 <invalid> harbor-v1-harbor-jobservice-5c9496775d-sj6mb 1/1 Running 0 <invalid> harbor-v1-harbor-notary-server-5fb65b6866-dnnnk 1/1 Running 0 <invalid> harbor-v1-harbor-notary-signer-5bfcfcd5cf-j774t 1/1 Running 0 <invalid> harbor-v1-harbor-registry-75c9b6b457-pqxj6 1/1 Running 0 <invalid> harbor-v1-harbor-ui-5974bd5549-zl9nj 1/1 Running 0 <invalid> harbor-v1-harbor-database-0 1/1 Running 0 <invalid> ==> v1/Secret NAME AGE harbor-v1-harbor-adminserver <invalid> harbor-v1-harbor-chartmuseum <invalid> harbor-v1-harbor-database <invalid> harbor-v1-harbor-ingress <invalid> harbor-v1-harbor-jobservice <invalid> harbor-v1-harbor-registry <invalid> harbor-v1-harbor-ui <invalid> NOTES: Please wait for several minutes for Harbor deployment to complete. Then you should be able to visit the UI portal at https://core.harbor.domain. For more details, please visit https://github.com/goharbor/harbor. ......
查看pod
[root@k8s-master01 harbor-helm]# kubectl get pod -n harbor NAME READY STATUS RESTARTS AGE harbor-v1-harbor-adminserver-5b59c684b4-g6cjc 1/1 Running 1 2m harbor-v1-harbor-chartmuseum-699cf6599-q6vfw 1/1 Running 0 2m harbor-v1-harbor-clair-6d9bb84485-2p52v 1/1 Running 1 2m harbor-v1-harbor-database-0 1/1 Running 0 2m harbor-v1-harbor-jobservice-5c9496775d-sj6mb 1/1 Running 1 2m harbor-v1-harbor-notary-server-5fb65b6866-dnnnk 1/1 Running 0 2m harbor-v1-harbor-notary-signer-5bfcfcd5cf-j774t 1/1 Running 0 2m harbor-v1-harbor-registry-75c9b6b457-pqxj6 1/1 Running 0 2m harbor-v1-harbor-ui-5974bd5549-zl9nj 1/1 Running 2 2m harbor-v1-redis-84dffd8574-xzrsh 1/1 Running 0 2m
查看service
[root@k8s-master01 harbor-helm]# kubectl get svc -n harbor NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE glusterfs-dynamic-database-data-harbor-v1-harbor-database-0 ClusterIP 10.101.10.82 <none> 1/TCP 2h glusterfs-dynamic-harbor-v1-harbor-chartmuseum ClusterIP 10.97.114.51 <none> 1/TCP 36s glusterfs-dynamic-harbor-v1-harbor-registry ClusterIP 10.98.207.16 <none> 1/TCP 36s glusterfs-dynamic-harbor-v1-redis ClusterIP 10.105.214.102 <none> 1/TCP 31s harbor-v1-harbor-adminserver ClusterIP 10.99.152.38 <none> 80/TCP 3m harbor-v1-harbor-chartmuseum ClusterIP 10.99.237.224 <none> 80/TCP 3m harbor-v1-harbor-clair ClusterIP 10.98.217.176 <none> 6060/TCP 3m harbor-v1-harbor-database ClusterIP 10.111.182.188 <none> 5432/TCP 3m harbor-v1-harbor-jobservice ClusterIP 10.98.202.61 <none> 80/TCP 3m harbor-v1-harbor-notary-server ClusterIP 10.110.72.98 <none> 4443/TCP 3m harbor-v1-harbor-notary-signer ClusterIP 10.106.234.19 <none> 7899/TCP 3m harbor-v1-harbor-registry ClusterIP 10.98.80.141 <none> 5000/TCP 3m harbor-v1-harbor-ui ClusterIP 10.98.240.15 <none> 80/TCP 3m harbor-v1-redis ClusterIP 10.107.234.107 <none> 6379/TCP 3m
查看pv和pvc
[root@k8s-master01 harbor-helm]# kubectl get pv,pvc -n harbor | grep harbor persistentvolume/pvc-080d1242-e990-11e8-8a89-000c293ad492 1Gi RWO Delete Bound harbor/database-data-harbor-v1-harbor-database-0 gluster-heketi 2h persistentvolume/pvc-f573b165-e9a3-11e8-882f-000c293bfe27 8Gi RWO Delete Bound harbor/harbor-v1-redis gluster-heketi 1m persistentvolume/pvc-f575855d-e9a3-11e8-882f-000c293bfe27 5Gi RWO Delete Bound harbor/harbor-v1-harbor-chartmuseum gluster-heketi 1m persistentvolume/pvc-f577371b-e9a3-11e8-882f-000c293bfe27 10Gi RWO Delete Bound harbor/harbor-v1-harbor-registry gluster-heketi 1m persistentvolumeclaim/database-data-harbor-v1-harbor-database-0 Bound pvc-080d1242-e990-11e8-8a89-000c293ad492 1Gi RWO gluster-heketi 2h persistentvolumeclaim/harbor-v1-harbor-chartmuseum Bound pvc-f575855d-e9a3-11e8-882f-000c293bfe27 5Gi RWO gluster-heketi 4m persistentvolumeclaim/harbor-v1-harbor-registry Bound pvc-f577371b-e9a3-11e8-882f-000c293bfe27 10Gi RWO gluster-heketi 4m persistentvolumeclaim/harbor-v1-redis Bound pvc-f573b165-e9a3-11e8-882f-000c293bfe27 8Gi RWO gluster-heketi 4m
查看ingress
[root@k8s-master01 harbor-helm]# vim values.yaml [root@k8s-master01 harbor-helm]# kubectl get ingress -n harbor NAME HOSTS ADDRESS PORTS AGE harbor-v1-harbor-ingress core.harbor.domain,notary.harbor.domain 80, 443 53m
安裝時也可以指定域名：--set externalURL=xxx.com
卸載：helm del --purge harbor-v1
4、Harbor使用
訪問測試，需要解析上述域名core.harbor.domain至k8s任意節點
默認賬號密碼：admin/Harbor12345
創建開發環境倉庫：
5、在k8s中使用harbor
查看harbor自帶證書
[root@k8s-master01 ~]# kubectl get secrets/harbor-v1-harbor-ingress -n harbor -o jsonpath="{.data.ca\.crt}" | base64 --decode -----BEGIN CERTIFICATE----- MIIC9DCCAdygAwIBAgIQffFj8E2+DLnbT3a3XRXlBjANBgkqhkiG9w0BAQsFADAU MRIwEAYDVQQDEwloYXJib3ItY2EwHhcNMTgxMTE2MTYwODA5WhcNMjgxMTEzMTYw ODA5WjAUMRIwEAYDVQQDEwloYXJib3ItY2EwggEiMA0GCSqGSIb3DQEBAQUAA4IB DwAwggEKAoIBAQDw1WP6S3O+7zrhVAAZGcrAEdeQxr0c53eyDGcPL6my/h+FhZ1Y KBvY5CLDVES957u/GtEXFfZr9aQT/PZECcccPcyZvt8NscEAuQONfrQFH/VLCvwm XOcbFDR5BXDJR8nqGT6DVq8a1HUEOxiY39bp/Jz2HrDIfD9IMwEuyh/2IVXYHwD0 deaBpOY1slSylpOYWPFfy9UMfCsd+Jc7UCzRaiP3XWP9HMFKc4JTU8CDRR80s9UM siU8QheVXn/Y9SxKaDfrYjaVUkEfJ6cAZkkDLmM1OzSU73N7I4nmm1SUS99vdSiZ yu/R4oDFMezOkvYGBeDhLmmkK3sqWRh+dNoNAgMBAAGjQjBAMA4GA1UdDwEB/wQE AwICpDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDwYDVR0TAQH/BAUw AwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAJjANauFSPZ+Da6VJSV2lGirpQN+EnrTl u5VJxhQQGr1of4Je7aej6216KI9W5/Q4lDQfVOa/5JO1LFaiWp1AMBOlEm7FNiqx LcLZzEZ4i6sLZ965FdrPGvy5cOeLa6D8Vx4faDCWaVYOkXoi/7oH91IuH6eEh+1H u/Kelp8WEng4vfEcXRKkq4XTO51B1Mg1g7gflxMIoeSpXYSO5qwIL5ZqvoAD9H7J CnQFO2xO3wrLq6TXH5Z7+0GWNghGk0GIOvF/ULHLWpsyhU5asKLK//MvORwQNHzL b5LHG9uYeI+Jf12X4TI9qDaTCstiqM8vk1JPvgtSPJ9M62nRKY4ang== -----END CERTIFICATE-----
創建證書
cat <<EOF > /etc/docker/certs.d/core.harbor.domain/ca.crt -----BEGIN CERTIFICATE----- MIIC9DCCAdygAwIBAgIQffFj8E2+DLnbT3a3XRXlBjANBgkqhkiG9w0BAQsFADAU MRIwEAYDVQQDEwloYXJib3ItY2EwHhcNMTgxMTE2MTYwODA5WhcNMjgxMTEzMTYw ODA5WjAUMRIwEAYDVQQDEwloYXJib3ItY2EwggEiMA0GCSqGSIb3DQEBAQUAA4IB DwAwggEKAoIBAQDw1WP6S3O+7zrhVAAZGcrAEdeQxr0c53eyDGcPL6my/h+FhZ1Y KBvY5CLDVES957u/GtEXFfZr9aQT/PZECcccPcyZvt8NscEAuQONfrQFH/VLCvwm XOcbFDR5BXDJR8nqGT6DVq8a1HUEOxiY39bp/Jz2HrDIfD9IMwEuyh/2IVXYHwD0 deaBpOY1slSylpOYWPFfy9UMfCsd+Jc7UCzRaiP3XWP9HMFKc4JTU8CDRR80s9UM siU8QheVXn/Y9SxKaDfrYjaVUkEfJ6cAZkkDLmM1OzSU73N7I4nmm1SUS99vdSiZ yu/R4oDFMezOkvYGBeDhLmmkK3sqWRh+dNoNAgMBAAGjQjBAMA4GA1UdDwEB/wQE AwICpDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDwYDVR0TAQH/BAUw AwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAJjANauFSPZ+Da6VJSV2lGirpQN+EnrTl u5VJxhQQGr1of4Je7aej6216KI9W5/Q4lDQfVOa/5JO1LFaiWp1AMBOlEm7FNiqx LcLZzEZ4i6sLZ965FdrPGvy5cOeLa6D8Vx4faDCWaVYOkXoi/7oH91IuH6eEh+1H u/Kelp8WEng4vfEcXRKkq4XTO51B1Mg1g7gflxMIoeSpXYSO5qwIL5ZqvoAD9H7J CnQFO2xO3wrLq6TXH5Z7+0GWNghGk0GIOvF/ULHLWpsyhU5asKLK//MvORwQNHzL b5LHG9uYeI+Jf12X4TI9qDaTCstiqM8vk1JPvgtSPJ9M62nRKY4ang== -----END CERTIFICATE----- EOF
重啟docker然后使用docker login 登錄
[root@k8s-master01 ~]# docker login core.harbor.domain Username: admin Password: Login Succeeded
如果報證書不信任錯誤x509: certificate signed by unknown authority
可以添加信任
chmod 644 /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
將上述ca.crt添加到/etc/pki/tls/certs/ca-bundle.crt即可
chmod 444 /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
上次鏡像，隨便找一個鏡像打上tag，然后上傳
[root@k8s-master01 ~]# docker push core.harbor.domain/develop/busybox The push refers to a repository [core.harbor.domain/develop/busybox] 8ac8bfaff55a: Pushed latest: digest: sha256:540f2e917216c5cfdf047b246d6b5883932f13d7b77227f09e03d42021e98941 size: 527
6、總結
部署過程中遇到的問題：
1) 由於某種原因無法訪問https://kubernetes-charts.storage.googleapis.com，也懶得FQ，就使用阿里的helm倉庫。(如果FQ了，就沒有以下問題)
2) 由於換成了阿里雲的倉庫，找不到源requirement.yaml中的redis版本，故修改阿里雲倉庫中有的redis。
3) 使用了GFS動態存儲，持久化了Harbor，需要更改values.yaml和redis里面的values.yaml中的storageClass。
4) 阿里雲倉庫中的redis啟用了密碼驗證，但是harbor chart的配置中未啟用密碼，所以干脆禁用了redis的密碼。
5) 使用Helm部署完Harbor以后，jobservice和harbor-ui的pods不斷重啟，通過日志發現未能解析Redis的service，原因是harbor chart里面配置Redis的service是harbor-v1-redis-master，而通過helm dependency update下載的Redis Chart service配置的harbor-v1-redis，為了方便，直接改了redis的svc.yaml文件。
6) 更改完上述文件以后pods還是處於一直重啟的狀態，且報錯：Failed to load and run worker pool: connect to redis server timeout: dial tcp 10.96.137.238:0: i/o timeout，發現Redis的地址+端口少了端口，最后經查證是harbor chart的values的redis配置port的參數，加上后重新部署即成功。
7) 由於Helm安裝的harbor默認啟用了https，故直接配置證書以提高安全性。
8) 將Harbor安裝到k8s上，原作者推薦的是Helm安裝，詳情見：https://github.com/goharbor/harbor/blob/master/docs/kubernetes_deployment.md，文檔見：https://github.com/goharbor/harbor-helm
9) 個人認為Harbor應該獨立於k8s集群之外使用docker-compose單獨部署，這也是最常見的方式，我目前使用的是此種方式(此文檔為第一次部署harbor到k8s,也為了介紹Helm)，而且便於維護及擴展，以及配置LDAP等都很方便。
10) Helm是非常強大的k8s包管理工具。
11) Harbor集成openLDAP點擊
贊助作者：
