Harbor私有鏡像倉庫（上）

一，Harbor簡介

• Vmware的開源項目https://github.com/VMware/harbor

• Harbor可幫助用戶迅速搭建企業級的注冊服務,它提供了管理圖形界面,基於角色的訪問控制(Role Based Access Control),鏡像遠程復制(同步),AD/LDAP集成,以及審計日志等企業用戶需求的功能,同時還原支持中文,深受中國用戶的喜愛

• 該項目自推出一來,在GitHub獲得了超過3300多個star和900多個forks.

 

1.1 基於角色的訪問控制

用戶與Docker鏡像倉庫通過"項目"進行組織管理,一個用戶可以對多個鏡像倉庫在統一命名空間(projec)里有不同的權限

 

1.2 圖形化用戶界面

用戶可以通過瀏覽器來瀏覽,檢索當前Docker鏡像倉庫,管理項目和命名空間

 

1.3 審計管理

所有針對鏡像倉庫的錯做都可以被記錄追溯,用於審計管理

 

1.4 國際化

基於英文與中文語言進行了本地化.可以增加更多的語言支持.

 

1.5 RESTful API:

提供給管理員對於Harbor更多的操控,使得與其他管理軟件集成變得更容易.

 

1.6 LDAP認證

 

1.7 鏡像復制

基於策略的Docker鏡像復制功能,可在不同的數據中心,不同的運行環境之間同步鏡像,並提供友好的管理界面,大大簡化了實際運維中的鏡像管理工作.

 

1.8 與Clair集成

與Clair集成,添加漏洞掃描功能,CLair是coreos開源的容器漏洞掃描工具,在容器逐漸普及的今天,容器鏡像安全問題日益嚴重,Clair是目前少數的開源安全掃描工具.

 

1.9 Notary簽名工具

Notary是Docker鏡像的簽名工具,用來保證鏡像在pull,push和傳輸工程中的一致性和完整性,避免中間人攻擊,避免非法的鏡像更新和運行.

 

二，為Harbor簽發域名證書

openssl是目前最流行的SSl密碼庫工具,提供了一個通用,功能完備的工具套件,用以支持SSL/TLS協議的實現.官網:https://www.openssl.org/source/

 

環境准備

官方文檔: https://github.com/vmware/harbor/blob/master/docs/configure_https.md

主機名	IP	用途	最小資源配比	最佳資源配比
harbor-master	192.168.200.70	harbor私有鏡像倉庫	2CPU-4GBMEM	4CPU-8GBMEM
harbor-slave	192.168.200.109	harbor從庫	2CPU-4GBMEM	4CPU-8GBMEM

hostname -I

uname -r

cat /etc/redhat-release

 

創建自己的CA證書

mkdir -p /data/ssl

cd /data/ssl

which openssl

openssl req -newkey rsa:4096 -nodes -sha256 -keyout ca.key -x509 -days 365 -out ca.crt

 

1. Generating a 4096 bit RSA private key
2. ............................................................................................++
3. ......++
4. writing new private key to 'ca.key'
5. -----
6. You are about to be asked to enter information that will be incorporated
7. into your certificate request.
8. What you are about to enter is what is called a Distinguished Name or a DN.
9. There are quite a few fields but you can leave some blank
10. For some fields there will be a default value,
11. If you enter '.', the field will be left blank.
12. -----
13. Country Name (2 letter code) [XX]:CN #國家
14. State or Province Name (full name) []:Beijing #地區
15. Locality Name (eg, city) [Default City]:Beijing #城市
16. Organization Name (eg, company) [Default Company Ltd]:yunjisuan #公司名稱
17. Organizational Unit Name (eg, section) []:yunjisuan
18. Common Name (eg, your name or your server's hostname) []:www.yunjisuan.com
19. Email Address []: #回車

 

生成證書簽名請求

openssl req -newkey rsa:4096 -nodes -sha256 -keyout www.yunjisuan.com.key -out www.yunjisuan.com.csr

 

1. Generating a 4096 bit RSA private key
2. ...........................................................................................................................................................................................................++
3. ..................................................................................................................................................................................++
4. writing new private key to 'www.yunjisuan.com.key'
5. -----
6. You are about to be asked to enter information that will be incorporated
7. into your certificate request.
8. What you are about to enter is what is called a Distinguished Name or a DN.
9. There are quite a few fields but you can leave some blank
10. For some fields there will be a default value,
11. If you enter '.', the field will be left blank.
12. -----
13. Country Name (2 letter code) [XX]:CN
14. State or Province Name (full name) []:Beijing
15. Locality Name (eg, city) [Default City]:Beijing
16. Organization Name (eg, company) [Default Company Ltd]:yunjisuan
17. Organizational Unit Name (eg, section) []:yunjisuan
18. Common Name (eg, your name or your server's hostname) []:www.yunjisuan.com
19. Email Address []:
21. Please enter the following 'extra' attributes
22. to be sent with your certificate request
23. A challenge password []:
24. An optional company name []:
25. [root@harbor-master ssl]#

 

生成注冊表主機的證書

openssl x509 -req -days 365 -in www.yunjisuan.com.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out www.yunjisuan.com.crt

 

1. Signature ok
2. subject=/C=CN/ST=Beijing/L=Beijing/O=yunjisuan/OU=yunjisuan/CN=www.yunjisuan.com
3. Getting CA Private Key

 

查看證書情況

ls

 

1. ca.crt ca.srl www.yunjisuan.com.csr
2. ca.key www.yunjisuan.com.crt www.yunjisuan.com.key

 

三，信任自簽發的域名證書

由於CA證書是我們自己簽發的Linux操作系統是不信任的,因此我們需要把證書加入到系統的信任證書里

 

將自簽ca證書添加到系統信任

pwd

cp www.yunjisuan.com.crt /etc/pki/ca-trust/source/anchors/ --->復制到這個下面是讓本機先信任這個證書

 

讓系統ca信任設置立即生效

update-ca-trust enable

update-ca-trust extract

 

四，Harbor1.4版本配置與安裝

 

4.1 安裝docker-ce社區版(master和slave都安)

 

安裝依賴包

yum -y install yum-utils device-mapper-persistent-data lvm2

 

添加docker的CE版本的yum源配置文件

curl https://download.docker.com/linux/centos/docker-ce.repo -o /etc/yum.repos.d/docker-ce.repo

ll /etc/yum.repos.d/docker-ce.repo

 

安裝CE版本的docker

yum -y install docker-ce

systemctl start docker #啟動docker

systemctl enable docker #添加開機啟動

docker version #查看docker版本

which docker

 

看一眼子防火牆的狀態(提前安裝過docker得話需要重啟docker)

sestatus -->防火牆什么的都關上! 過程略

 

4.2 下載並安裝harbor私有倉庫

 

harbor所有包

鏈接：https://pan.baidu.com/s/1MSfSWLBsUrvXv3USv233Cg 
提取碼：zmom

 

創建harbor的證書目錄,並復制

mkdir -p /etc/ssl/harbor

cp /data/ssl/www.yunjisuan.com.key /etc/ssl/harbor/

cp /data/ssl/www.yunjisuan.com.crt /etc/ssl/harbor/

ll /etc/ssl/harbor/

 

創建harbor下載目錄並下載harbor-offline-installer-v1.5.0.tgz

mkdir -p /data/install

cd /data/install

pwd

which wget --->這里沒有wget下載，上文有下載鏈接

wget http://harbor.orientsoft.cn/harbor-v1.5.0/harbor-offline-installer-v1.5.0.tgz

ls

tar xf harbor-offline-installer-v1.5.0.tgz

ls

cd harbor

ll

cp harbor.cfg{,.bak}

 

修改harbor.cfg配置文件

cat -n harbor.cfg | sed -n '7p;11p;23p;24p;68p'

 

1. 7 hostname = reg.mydomain.com #要修改成我們證書的域名
2. 11 ui_url_protocol = http #啟用加密傳輸協議https
3. 23 ssl_cert = /data/cert/server.crt #證書的位置
4. 24 ssl_cert_key = /data/cert/server.key #證書密鑰位置
5. 68 harbor_admin_password = Harbor12345 #默認管理員及密碼

 

修改成如下配置

vim harbor.cfg

cat -n harbor.cfg | sed -n '7p;11p;23p;24p;68p'

 

1. 7 hostname = www.yunjisuan.com
2. 11 ui_url_protocol = https
3. 23 ssl_cert = /etc/ssl/harbor/www.yunjisuan.com.crt
4. 24 ssl_cert_key = /etc/ssl/harbor/www.yunjisuan.com.key
5. 68 harbor_admin_password = Harbor12345

 

安裝命令docker-compose(需要1.21版本)

curl -L https://github.com/docker/compose/releases/download/1.21.2/docker-compose-(uname -m) -o /usr/local/bin/docker-compose --->上文有下載鏈接

cd /usr/local/bin/

ll /usr/local/bin/docker-compose

chmod +x /usr/local/bin/docker-compose

which docker-compose

docker-compose --version

 

安裝harbor私有鏡像倉庫

cd /data/install/harbor

./install.sh --with-notary --with-clair

 

1. #--with-notary啟用鏡像簽名; --with-clair啟用漏洞掃描

 

查看harbor啟動的鏡像

docker ps -a

 

1. CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2. be5fcf3a1930 vmware/harbor-jobservice:v1.5.0 "/harbor/start.sh" About a minute ago Up 50 seconds harbor-jobservice
3. ad2f90bb84c3 vmware/nginx-photon:v1.5.0 "nginx -g 'daemon of…" About a minute ago Up About a minute (healthy) 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:4443->4443/tcp nginx
4. d782fcc8ce01 vmware/notary-server-photon:v0.5.1-v1.5.0 "/bin/server-start.sh" About a minute ago Up About a minute notary-server
5. a8edc1ccca5f vmware/clair-photon:v2.0.1-v1.5.0 "/docker-entrypoint.…" About a minute ago Up 46 seconds (healthy) 6060-6061/tcp clair
6. edb4a9a4c11e vmware/harbor-ui:v1.5.0 "/harbor/start.sh" About a minute ago Up About a minute (healthy) harbor-ui
7. 2db41616f8e3 vmware/notary-signer-photon:v0.5.1-v1.5.0 "/bin/signer-start.sh" About a minute ago Up About a minute notary-signer
8. fa7b3e5f6f24 vmware/postgresql-photon:v1.5.0 "/entrypoint.sh post…" About a minute ago Up About a minute (healthy) 5432/tcp clair-db
9. 1401200b682e vmware/harbor-adminserver:v1.5.0 "/harbor/start.sh" About a minute ago Up About a minute (healthy) harbor-adminserver
10. 0b37da8995e3 vmware/redis-photon:v1.5.0 "docker-entrypoint.s…" About a minute ago Up About a minute 6379/tcp redis
11. aab6d323c577 vmware/harbor-db:v1.5.0 "/usr/local/bin/dock…" About a minute ago Up About a minute (healthy) 3306/tcp harbor-db
12. bfbdb8a623e4 vmware/registry-photon:v2.6.2-v1.5.0 "/entrypoint.sh serv…" About a minute ago Up About a minute (healthy) 5000/tcp registry
13. f29345823b5a vmware/mariadb-photon:v1.5.0 "/usr/local/bin/dock…" About a minute ago Up About a minute 3306/tcp notary-db
14. 3756f90a2271 vmware/harbor-log:v1.5.0 "/bin/sh -c /usr/loc…" About a minute ago Up About a minute (healthy) 127.0.0.1:1514->10514/tcp harbor-log

 

4.3 通過瀏覽器進行訪問測試(需要火狐瀏覽器才能跳過)

https://192.168.200.70

輸入賬號密碼登錄:admin harbor12345

---

 

4.4 做一些安全設置

---

---

 

4.5 本地docker上傳測試

 

先做映射

vim /etc/hosts

cat /etc/hosts

 

用docker登錄(只能域名訪問，IP是加密的不認識)

docker login www.yunjisuan.com

admin Harbor12345

 

也可以面交互登錄

docker login -uadmin -pHarbor12345 www.yunjisuan.com

 

先得把鏡像改名加路徑(就是相當於添加)

docker pull centos

docker images

docker tag centos:latest www.yunjisuan.com/library/centos:v1

docker images

docker push www.yunjisuan.com/library/centos:v1

---

 

刷新頁面，然后查看就有了

---

---

---

 

在slave服務器上訪問並下載私有鏡像倉庫里剛上傳的那個鏡像

 

先映射

echo "192.168.200.70 www.yunjisuan.com" >> /etc/hosts --->映射的是Harbor的那台

cat /etc/hosts

 

然后把主的那個證書復制過去並立即生效

cd /data/ssl

ls

scp www.yunjisuan.com.crt 192.168.200.109:/etc/pki/ca-trust/source/anchors/

---

update-ca-trust enable

update-ca-trust extract

 

重啟docker否則沒有用

systemctl restart docker

docker login -uadmin -pHarbor12345 www.yunjisuan.com

 

下載私有鏡像倉庫里的那個剛剛上傳的鏡像文件

docker pull www.yunjisuan.com/library/centos:v1

docker images

 

五，鏡像管理與安全:漏洞掃描和鏡像簽名

 

5.1 添加docker國內公有鏡像源

vim /etc/docker/daemon.json

cat /etc/docker/daemon.json

 

1. {
2. "registry-mirrors":[ "https://registry.docker-cn.com" ]
3. }

systemctl daemon-reload

systemctl restart docker

 

5.2 重新啟動Harbor私有鏡像倉庫

 

讓harbor修改過的配置立刻生效

 

1. [root@harbor-master harbor]# pwd
2. /data/install/harbor
3. [root@harbor-master harbor]# ./prepare

 

清理所有harbor容器進程

 

1. [root@harbor-master harbor]# docker-compose down

 

后台重新啟動所有harbor容器進程

 

1. [root@harbor-master harbor]# docker-compose up -d
3. **docker-compose命令必須在/data/install/harbor目錄下使用，否則找不到**

 

5.3 FAQ:問題解答

 

5.3.1 windows10最新版本默認拒絕非認證的域名證書

如果啟動harbor采用https加密證書的方式,最新版本window10瀏覽器訪問的化,默認會直接說"站點不安全,拒絕連接" 
那么我們可以采用非https的方式啟動harbor

 

1. [root@harbor-master harbor]# sed -n '11p' /data/install/harbor/harbor.cfg

但是我們要是采用非https加密方式啟動harbor的化,最新版本的docker是登錄不了的,這是因為新版本docker默認是以https方式登錄harbor

為了解決登錄問題,我們需要在/etc/docker/下創建一個daemon.json名字的文件,加入http方式登錄的harbor域名

 

1. [root@harbor-slave ~]# cat /etc/docker/daemon.json
2. {
3. "insecure-registries":[ "www.yunjisuan.com" ]
4. }
5. [root@harbor-slave ~]# systemctl restart docker #需要重啟

然后就能登錄成功了

 

六，harbor鏡像的復制與同步

harbor私有倉庫的主從復制,類似於MySQL,屬於1對多的復制

主機名	IP	用途	最小資源配比	最佳資源配比
harbor-master	192.168.200.70	harbor私有鏡像倉庫	2CPU-4GBMEM	4CPU-8GBMEM
harbor-slave	192.168.200.109	harbor從庫	2CPU-4GBMEM	4CPU-8GBMEM

 

6.1 部署Harbor-Slave

請安裝一個harbor私有倉庫作為harbor的從庫,域名為www2.yunjisuan.com

請看上文master部署步驟

 

主找從所以把證書給主一份

cd /data/ssl

ls

scp www2.yunjisuan.com.crt 192.168.200.70:/etc/pki/ca-trust/source/anchors/

 

然后主得立即生效並重啟dcoker.然后harbor還得重啟啟動進程

cd /data/install/harbor

update-ca-trust enable

update-ca-trust extract

systemctl restart docker

docker-compose down --->必須在harbor目錄里（/data/install/harbor）

./prepare

./install.sh --with-clair

---

 

在主上做映射

vim /etc/hosts

cat /etc/hosts

 

查看網頁主從情況

---

 

6.2 搭建LDNS域名解析服務器

主機名	IP	用途	最小資源配比	最佳資源配比
harbor-master	192.168.200.70	harbor私有鏡像倉庫	2CPU-4GBMEM	4CPU-8GBMEM
harbor-slave	192.168.200.109	harbor從庫	2CPU-4GBMEM	4CPU-8GBMEM
LDNS	192.168.200.110	本地DNS

yum -y install bind bind-chroot bind-utils

cd /etc/

cp named.conf{,.bak}

 

把配置文件修改成如下:

vim named.conf

cat named.conf

 

1. options {
2. listen-on port 53 { 192.168.200.110; }; #監聽本機IP地址
3. // listen-on-v6 port 53 { ::1; }; #注釋掉
4. directory "/var/named";
5. dump-file "/var/named/data/cache_dump.db";
6. statistics-file "/var/named/data/named_stats.txt";
7. memstatistics-file "/var/named/data/named_mem_stats.txt";
8. allow-query { any; }; #any允許任意客戶端
9. forwarders { 192.168.200.2; }; #加一段話,網關的位置
11. recursion yes;
13. dnssec-enable no; #改成no不驗證
14. dnssec-validation no; #改成no不驗證
16. /* Path to ISC DLV key */
17. bindkeys-file "/etc/named.iscdlv.key";
19. managed-keys-directory "/var/named/dynamic";
21. pid-file "/run/named/named.pid";
22. session-keyfile "/run/named/session.key";
23. };
25. logging {
26. channel default_debug {
27. file "data/named.run";
28. severity dynamic;
29. };
30. };
32. zone "." IN {
33. type hint;
34. file "named.ca";
35. };
37. zone "yunjisuan.com" IN { #加一個維護域名
38. type master; #類型
39. file "yunjisuan.com.zone"; #域名文件
40. }; #內容
41. include "/etc/named.rfc1912.zones";
42. include "/etc/named.root.key";

---

 

檢查配置文件是否有錯

named-checkconf /etc/named.conf

 

創建正向解析文件

cd /var/named/

ls

cp -p named.empty yunjisuan.com.zone

 

把yunjisuan.com.zone修改成如下

vim yunjisuan.com.zone

cat yunjisuan.com.zone

 

1. $TTL 1D
2. @ IN SOA yunjisuan.com. root.ns1.yunjisuan.com. (
3. 0 ; serial
4. 1D ; refresh
5. 1H ; retry
6. 1W ; expire
7. 3H ) ; minimum
8. NS ns1.yunjisuan.com.
9. ns1 A 192.168.200.110 #LDNS的IP地址,也就是自己
10. www A 192.168.200.70 #harbor主的ip地址
11. www2 A 192.168.200.109 #harbor從的ip地址

 

測試正向解析文件是否有錯

named-checkzone yunjisuan.com yunjisuan.com.zone

 

啟動域名解析服務

systemctl start named

ss -antup | grep named

 

改一下DNS改成自己本機

vim /etc/resolv.conf

cat /etc/resolv.conf

 

1. #nameserver 192.168.200.2
2. nameserver 192.168.200.110

 

用nslookup域名解析命令測試一下

nslookup www.yunjisuan.com

nslookup www2.yunjisuan.com

nslookup www.baidu.com

 

6.3 建立主從復制

然后開始建立主從復制

---

目前測試連接失敗,沒關系,我們先點擊確定把這個保存一下

 

先清空一下日志

cd /var/log/harbor/

ls

> ui.log

 

然后在點擊一下測試連接之后查看日志

cat ui.log

因此發現,Harbor的主從復制是不找本地的hosts文件的,映射了也沒有用. 它直接找DNS

 

把主的DNS改成LDNS服務器的IP

vim /etc/resolv.conf

cat /etc/resolv.conf

 

1. nameserver 192.168.200.110

 

然后得重啟harbor否則也不行 ，因為它讀到緩存去了。

cd /data/install/harbor

docker-compose down --->需要在harbor目錄下(/data/install/harbor)

./prepare

./install.sh --with-clair

---

 

刷新網頁，在測試連接

 

6.4啟用主從復制

---

---

 

在看從的，就把鏡像復制過去了 一開始沒有任何鏡像的

 

在上傳一個鏡像 看看效果

docker pull hello-world

docker images

docker tag hello-world:latest www.yunjisuan.com/library/hello-world:v1

REPOSITORY TAG IMAGE ID CREATED SIZE 
centos latest 75835a67d134 8 weeks ago 200MB 
www.yunjisuan.com/library/centos v1 75835a67d134 8 weeks ago 200MB 
hello-world latest 4ab4c602aa5e 2 months ago 1.84kB 
www.yunjisuan.com/library/hello-world v1 4ab4c602aa5e 2 months ago 1.84kB

 

上傳

[root@wbq-harbor-master harbor]# docker push www.yunjisuan.com/library/hello-world:v1 
The push refers to repository [www.yunjisuan.com/library/hello-world] 
428c97da766c: Pushed 
v1: digest: sha256:1a6fd470b9ce10849be79e99529a88371dff60c60aab424c077007f6979b4812 size: 524

 

主的已經是2個鏡像了

---

 

在看從的也變成2個鏡像了

這樣主從復制就搭建完成了