docker 總結

目錄

• 1 docker安裝部署
  • 1.1 docker介紹
    • 1.1.1 Linux Namespace 技術
    • 1.1.2 Linux control groups
      • 1.1.2.1 查看系統cgroups
      • 1.1.2.2 查看容器的資源限制
      • 1.1.2.3 docker中run 命令中 cgroups 相關命令
    • 1.1.3 cgroups驗證
      • 1.1.3.1 創建容器的CPU權重控制
      • 1.1.3.2 cpu core控制
      • 1.1.3.3 cpu配額控制參數的混合使用
      • 1.1.3.4 內存配額
      • 1.1.3.5 Block IO的限制
      • 1.1.3.6 bps和iops限制
  • 1.2 docker安裝和基礎命令
    • 1.2.1 在線安裝
    • 1.2.2 離線安裝
  • 1.3 docker鏡像加速
  • 1.4 docker基礎命令
• 2 docker鏡像管理
  • 2.1 鏡像管理命令
  • 2.2 鏡像構建
    • 2.2.1 手動構建
    • 2.2.2 Dockerfile構建
• 3 docker 數據管理
  • 3.1 啟動測試容器驗證
  • 3.2 數據卷的特點
  • 3.3 數據卷的使用場景
  • 3.4 鏡像管理
  • 3.5 鏡像縮容
  • 3.6 多階段構建
• 4 Harbor 倉庫使用
  • 4.1 harbor部署
    • 4.1.1 機器環境
    • 4.1.2 hostname
    • 4.1.3 網絡設置
    • 4.1.4 查看主機名
    • 4.1.5 ip:hostname到每一台機器節點
    • 4.1.6 安裝依賴環境，注意：每一台機器都需要安裝此依賴環境
    • 4.1.7 docker部署
      • 4.1.7.1 安裝docker
      • 4.1.7.2 設置docker daemon文件
      • 4.1.7.3 重啟docker服務
    • 4.1.8 安裝compose
    • 4.1.9 Harbor安裝
      • 4.1.9.1 下載Harbor並配置
      • 4.1.9.2 生成證書
      • 4.1.9.3 安裝
      • 4.1.9.4 更新daemon.json文件
  • 4.2 harbor 上傳和下載鏡像
    • 4.2.1 node節點添加私有倉庫
    • 4.2.2 驗證是否可以登錄
    • 4.2.3 鏡像打tag推送
  • 4.3 實現harbor的高可用
    • 4.3.1 准備運行環境
    • 4.3.2 添加倉庫和復制規則
    • 4.3.3 驗證復制是否成功
    • 4.3.4 配置harbor的代理
      • 4.3.4.1 配置keepalived
      • 4.3.4.2 配置haproxy
      • 4.3.4.3 驗證配置
    • 4.3.5 harbor的配置文件解析

1 docker安裝部署

1.1 docker介紹

首先 Docker 是一個在 2013 年開源的應用程序並且是一個基於 go 語言編寫是一個開源的 PAAS 服務(Platform as a Service，平台即服務的縮寫)，go 語言是由google 開發，docker 公司最早叫 dotCloud 后由於 Docker 開源后大受歡迎就將公司改名為 Docker Inc，總部位於美國加州的舊金山，Docker 是基於 linux 內核實現，Docker 最早采用 LXC 技術(LinuX Container 的簡寫，LXC 是 Linux 原生支持的容器技術，可以提供輕量級的虛擬化，可以說 docker 就是基於 LXC 發展起來的(0.1.5 (2013-04-17)，提供 LXC 的高級封裝，發展標准的配置方法)，而虛擬化 技術 KVM(Kernel-based Virtual Machine) 基於模塊實現，Docker 后改為自己研發並開源的 runc 技術運行容器(1.11.0 (2016-04-13)。

1.1.1 Linux Namespace 技術

namespace 是 Linux 系統的底層概念，在內核層實現，即有一些不同類型的命名空間被部署在核內，各個 docker 容器運行在同一個 docker 主進程並且共用同一個宿主機系統內核，各 docker 容器運行在宿主機的用戶空間，每個容器都要有類似於虛擬機一樣的相互隔離的運行空間，但是容器技術是在一個進程內實現運行指定服務的運行環境，並且還可以保護宿主機內核不受其他進程的干擾和影響，如文件系統空間、網絡空間、進程空間等

隔離類別	功能	系統調用參數	內核版本
MNT Namespace	提供磁盤掛載點和文件系統的隔離能力	CLONE_NEWNS	Linux2.4.19
IPC Namespace(Inter-Process Communication)	提供進程間通信的隔離能力	CLONE_NEWIPC	Linux2.6.19
UTS Namespace(UNIX Timesharing System)	提供主機名隔離能力	CLONE_NEWUTS	Linux2.6.19
PID Namespace(Process Identification)	提供進程隔離能力	CLONE_NEWPID	Linux2.6.24
Net Namespace(network)	提供網絡隔離能力	CLONE_NEWNET	Linux2.6.29
User Namespace(user)	提供用戶隔離能力	CLONE_NEWUSER	Linux3.8

#MNT Namespace 
每個容器有獨立的根文件系統和獨立的用戶空間，以實現在容器里面啟動服務並且使用容器的運行環境。
即一個宿主機是ubuntu的服務器，可以在里面啟動一個centos運行環境的容器並且在容器里面啟動一個Nginx服務，此 Nginx運行時使用的運行環境就是centos系統目錄的運行環境，但是在容器里面是不能訪問宿主機的資源，宿主機是使用了 chroot技術把容器鎖定到一個指定的運行目錄里面
例如 #：containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/`容器ID` -address /var/run/docker/containerd/containerd.sock -containerd-binary /opt/kube/bin/containerd -runtime-root /var/run/docker/runtime-runc


#IPC Namespace
一個容器內的進程間通信，允許一個容器內的不同進程的(內存、緩存等)數據訪問，但是不能誇容器訪問其他容器的數據。

#UTS Namespace
UTS namespace（UNIX Timesharing System 包含了運行內核的名稱、版本、底層體 系結構類型等信息）用於系統標識，其中包含hostname和域名domainname，它使得一個容器擁有屬於自己hostname標識，這個主機名標識獨立於宿主機系統和其上的其他容器。
root@harbor:~# docker exec -it 2b9314b71794 bash
nginx [ / ]$ ls
bin  boot  dev  etc  home  lib  lib64  media  mnt  proc  root  run  sbin  srv  sys  tmp  usr  var
nginx [ / ]$ cat /etc/issue
Welcome to Photon 4.0 (\m) - Kernel \r (\l)
nginx [ / ]$ uname -a  #宿主機的內核
Linux 2b9314b71794 4.15.0-152-generic #159-Ubuntu SMP Fri Jul 16 16:28:09 UTC 2021 x86_64
nginx [ / ]$ hostname  #自己的hostname
2b9314b71794

#PID namespace
Linux 系統中，有一個 PID 為 1 的進程(init/systemd)是其他所有進程的父進程，那么在每個容器內也要有一個父進程來管理其下屬的子進程，那么多個容器的進程通PID namespace進程隔離
root@harbor:~# docker exec -it 2b9314b71794 bash  #查看當前父進程nginx: master process nginx -g
nginx [ / ]$ ps -ef
UID             PID   PPID C STIME TTY          TIME CMD
nginx             1      0 0 48:05 ?        00:00:00 nginx: master process nginx -g daemon off;
nginx             6      1 0 48:05 ?        00:00:00 nginx: worker process 
nginx             7      1 0 48:05 ?        00:00:00 nginx: worker process
nginx           346      0 0 10:09 pts/0    00:00:00 bash
nginx           445    346 0 15:46 pts/0    00:00:00 ps -ef
root@harbor:~# ps -ef |grep docker #找容器id對應的宿主機的目錄
root        379      1  0 07:47 ?        00:00:04 /opt/kube/bin/dockerd
root        549    379  0 07:47 ?        00:00:02 containerd --config /var/run/docker/containerd/containerd.toml --log-level warn
root       1152    379  0 07:48 ?        00:00:00 /opt/kube/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 80 -container-ip 172.18.0.3 -container-port 8080 #docker-proxy 負責容器訪問
root       1164    379  0 07:48 ?        00:00:00 /opt/kube/bin/docker-proxy -proto tcp -host-ip 127.0.0.1 -host-port 1514 -container-ip 172.18.0.9 -container-port 10514
root       1171    549  0 07:48 ?        00:00:00 containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/2b9314b71794481593200549b38493ca5101324a753929a608bbc1fb8c3cb78c -address /var/run/docker/containerd/containerd.sock -containerd-binary /opt/kube/bin/containerd -runtime-root /var/run/docker/runtime-runc
root       1192    549  0 07:48 ?        00:00:00 containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/89e6038bf99fb14a7c586b1317b8eb4e08ec2dcbff4d5055d596a3ac80cfed9c -address /var/run/docker/containerd/containerd.sock -containerd-binary /opt/kube/bin/containerd -runtime-root /var/run/docker/runtime-runc
root       1202    549  0 07:48 ?        00:00:00 containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/5dbda7bfb4f52dc6abd5d16f9f311585426c2d091eda339c3c19d1efdf7896fa -address /var/run/docker/containerd/containerd.sock -containerd-binary /opt/kube/bin/containerd -runtime-root /var/run/docker/runtime-runc
root@harbor:~# ps -ef |grep 1171
root       1171    549  0 07:48 ?        00:00:00 containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/2b9314b71794481593200549b38493ca5101324a753929a608bbc1fb8c3cb78c -address /var/run/docker/containerd/containerd.sock -containerd-binary /opt/kube/bin/containerd -runtime-root /var/run/docker/runtime-runc
10000      1411   1171  0 07:48 ?        00:00:00 nginx: master process nginx -g daemon off;
root      33661  30705  0 08:20 pts/1    00:00:00 grep --color=auto 1171
root@harbor:~# ps -ef |grep 1411
10000      1411   1171  0 07:48 ?        00:00:00 nginx: master process nginx -g daemon off;
10000      1610   1411  0 07:48 ?        00:00:00 nginx: worker process
10000      1611   1411  0 07:48 ?        00:00:00 nginx: worker process

#Net Namespace
每一個容器都類似於虛擬機一樣有自己的網卡、監聽端口、TCP/IP 協議棧等， Docker使用network namespace啟動一個`vethX`接口，這樣你的容器將擁有它自己的橋接ip地址，通常是docker0，而docker0實質就是Linux的虛擬網橋,網橋 是在OSI七層模型的數據鏈路層的網絡設備，通過mac地址對網絡進行划分，並且在不同網絡直接傳遞數據。

#User Namespace
User Namespace允許在各個宿主機的各個容器空間內創建相同的用戶名以及相同的用戶UID和GID，只是會把用戶的作用范圍限制在每個容器內，即A容器和B容器可以有相同的用戶名稱/ID的賬戶，但是此用戶的有效范圍僅是當前容器內，不能訪問另外一個容器內的文件系統，即相互隔離。

1.1.2 Linux control groups

在一個容器，如果不對其做任何資源限制，則宿主機會允許其占用無限大的內存空間，有時候會因為代碼bug程序會一直申請內存，直到把宿主機內存占完，為了避免此類的問題出現，宿主機有必要對容器進行資源分配限制，比如CPU、內存等，Linux Cgroups 的全稱是 Linux Control Groups，它最主要的作用，就是限制一個進程組能夠使用的資源上限，包括 CPU、內存、磁盤、網絡帶寬等等。此外，還能夠對進程進行優先級設置，以及將進程掛起和恢復等操作

1.1.2.1 查看系統cgroups

root@harbor:~/harbor# ll /sys/fs/cgroup/
total 0
drwxr-xr-x 15 root root 380 Sep  5 07:47 ./
drwxr-xr-x  9 root root   0 Sep  5 07:47 ../
dr-xr-xr-x  5 root root   0 Sep  5 07:47 blkio/  #blkio：塊設備 IO 限制。
lrwxrwxrwx  1 root root  11 Sep  5 07:47 cpu -> cpu,cpuacct/  #使用調度程序為 cgroup 任務提供 cpu 的訪問
dr-xr-xr-x  5 root root   0 Sep  5 07:47 cpu,cpuacct/  
lrwxrwxrwx  1 root root  11 Sep  5 07:47 cpuacct -> cpu,cpuacct/ #產生 cgroup 任務的 cpu 資源報告。
dr-xr-xr-x  3 root root   0 Sep  5 07:47 cpuset/ #如果是多核心的 cpu，這個子系統會為 cgroup 任務分配單獨的 cpu 和 內存。
dr-xr-xr-x  5 root root   0 Sep  5 07:47 devices/  #允許或拒絕 cgroup 任務對設備的訪問。
dr-xr-xr-x  3 root root   0 Sep  5 07:47 freezer/  #暫停和恢復 cgroup 任務。
dr-xr-xr-x  3 root root   0 Sep  5 07:47 hugetlb/
dr-xr-xr-x  5 root root   0 Sep  5 07:47 memory/ #設置每個 cgroup 的內存限制以及產生內存資源報告。
lrwxrwxrwx  1 root root  16 Sep  5 07:47 net_cls -> net_cls,net_prio/ #標記每個網絡包以供 cgroup 方便使用。
dr-xr-xr-x  3 root root   0 Sep  5 07:47 net_cls,net_prio/
lrwxrwxrwx  1 root root  16 Sep  5 07:47 net_prio -> net_cls,net_prio/
dr-xr-xr-x  3 root root   0 Sep  5 07:47 perf_event/  #增加了對每 group 的監測跟蹤的能力
dr-xr-xr-x  5 root root   0 Sep  5 07:47 pids/
dr-xr-xr-x  2 root root   0 Sep  5 07:47 rdma/
dr-xr-xr-x  6 root root   0 Sep  5 07:47 systemd/
dr-xr-xr-x  5 root root   0 Sep  5 07:47 unified/

1.1.2.2 查看容器的資源限制

目前 docker 已經幾乎支持了所有的 cgroups 資源，可以限制容器對包括 network，device，cpu 和 memory 在內的資源的使用

默認情況下，Docker 啟動一個容器后，會在 /sys/fs/cgroup 目錄下的各個資源目錄下生成以容器 ID 為名字的目錄（group）

/sys/fs/cgroup/cpu/docker/03dd196f415276375f754d51ce29b418b170bd92d88c5e420d6901c32f93dc14

root@harbor:/var/lib/docker# cd /sys/fs/cgroup/
root@harbor:/sys/fs/cgroup# ls
blkio  cpu,cpuacct  cpuset   freezer  memory   net_cls,net_prio  perf_event  rdma     unified
cpu    cpuacct      devices  hugetlb  net_cls  net_prio          pids        systemd
root@harbor:/sys/fs/cgroup# find ./* -iname 2b9314b71794481593200549b38493ca5101324a753929a608bbc1fb8c3cb78c
./blkio/docker/2b9314b71794481593200549b38493ca5101324a753929a608bbc1fb8c3cb78c
./cpu,cpuacct/docker/2b9314b71794481593200549b38493ca5101324a753929a608bbc1fb8c3cb78c
./cpuset/docker/2b9314b71794481593200549b38493ca5101324a753929a608bbc1fb8c3cb78c
./devices/docker/2b9314b71794481593200549b38493ca5101324a753929a608bbc1fb8c3cb78c
./freezer/docker/2b9314b71794481593200549b38493ca5101324a753929a608bbc1fb8c3cb78c
./hugetlb/docker/2b9314b71794481593200549b38493ca5101324a753929a608bbc1fb8c3cb78c
./memory/docker/2b9314b71794481593200549b38493ca5101324a753929a608bbc1fb8c3cb78c
./net_cls,net_prio/docker/2b9314b71794481593200549b38493ca5101324a753929a608bbc1fb8c3cb78c
./perf_event/docker/2b9314b71794481593200549b38493ca5101324a753929a608bbc1fb8c3cb78c
./pids/docker/2b9314b71794481593200549b38493ca5101324a753929a608bbc1fb8c3cb78c
./systemd/docker/2b9314b71794481593200549b38493ca5101324a753929a608bbc1fb8c3cb78c

1.1.2.3 docker中run 命令中 cgroups 相關命令

block IO:
      --blkio-weight value          Block IO (relative weight), between 10 and 1000
      --blkio-weight-device value   Block IO weight (relative device weight) (default [])
      --cgroup-parent string        Optional parent cgroup for the container
CPU:
      --cpu-percent int             CPU percent (Windows only)
      --cpu-period int              Limit CPU CFS (Completely Fair Scheduler) period
      --cpu-quota int               Limit CPU CFS (Completely Fair Scheduler) quota
  -c, --cpu-shares int              CPU shares (relative weight)
      --cpuset-cpus string          CPUs in which to allow execution (0-3, 0,1)
      --cpuset-mems string          MEMs in which to allow execution (0-3, 0,1)
Device:    
      --device value                Add a host device to the container (default [])
      --device-read-bps value       Limit read rate (bytes per second) from a device (default [])
      --device-read-iops value      Limit read rate (IO per second) from a device (default [])
      --device-write-bps value      Limit write rate (bytes per second) to a device (default [])
      --device-write-iops value     Limit write rate (IO per second) to a device (default [])
Memory:      
      --kernel-memory string        Kernel memory limit
  -m, --memory string               Memory limit
      --memory-reservation string   Memory soft limit
      --memory-swap string          Swap limit equal to memory plus swap: '-1' to enable unlimited swap
      --memory-swappiness int       Tune container memory swappiness (0 to 100) (default -1)

1.1.3 cgroups驗證

1.1.3.1 創建容器的CPU權重控制

默認情況下，每個docker容器的cpu份額都是1024，單獨一個容器的份額是沒有意義的，只有在同時運行多個容器時，容器cpu的加權效果才能體現出現。

例如，兩個容器A、B的cpu份額分別為1000和500，在cpu進行時間片分配的時候，容器A比容器B多一倍的機會獲得cpu的時間片，但是分配的結果取決於當時主機和其他容器的運行狀態，實際上也無法保證容器A一定能夠獲得cpu的時間片。比如容器A的進程一直是空閑的，那么容器B是可以獲取比容器A更多的cpu時間片的，極端情況下，例如主機上只運行的一個容器，即使它的cpu份額只有50，它也可以獨占整個主機的cpu資源

cgroups只在容器分配的資源緊缺時，即需要對容器使用的資源進行限制時，才會生效。因此，無法單純的根據某個容器的份額的cpu份額來確定有多少cpu資源分配給它，可以通過cpu share參數可以設置容器使用cpu的優先級，比如啟動了兩個容器及運行查看cpu的cpu的使用百分比

創建兩個容器，分別制定不同的權重比

#docker run -itd --name cpu1024 --cpu-shares 1024 registry.cn-hangzhou.aliyuncs.com/haozheyu/centos-stress:v1 stress -c 10

#docker run -itd --name cpu512 --cpu-shares 512 registry.cn-hangzhou.aliyuncs.com/haozheyu/centos-stress:v1 stress -c 10

查看%cpu的比例

root@harbor2:~# docker exec -it cpu1024 bash
top - 02:27:46 up  2:39,  0 users,  load average: 19.86, 13.08, 5.86
Tasks:  13 total,  11 running,   2 sleeping,   0 stopped,   0 zombie
%Cpu0  : 99.7 us,  0.3 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  4015804 total,  2211948 free,   261200 used,  1542656 buff/cache
KiB Swap:  2097148 total,  2097148 free,        0 used.  3512012 avail Mem 

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                            
     7 root      20   0    7324    100      0 R  19.9  0.0   0:51.90 stress                             
     8 root      20   0    7324    100      0 R  19.9  0.0   0:51.87 stress                             
     9 root      20   0    7324    100      0 R  19.9  0.0   0:51.84 stress                             
    10 root      20   0    7324    100      0 R  19.9  0.0   0:51.86 stress                             
    11 root      20   0    7324    100      0 R  19.9  0.0   0:51.86 stress                             
    12 root      20   0    7324    100      0 R  19.9  0.0   0:51.90 stress  
    
------------------------------------------------
root@harbor2:~# docker exec -it cpu512 bash
top - 02:28:40 up  2:40,  0 users,  load average: 19.94, 14.25, 6.67
Tasks:  13 total,  11 running,   2 sleeping,   0 stopped,   0 zombie
%Cpu(s):100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  4015804 total,  2210800 free,   262280 used,  1542724 buff/cache
KiB Swap:  2097148 total,  2097148 free,        0 used.  3511016 avail Mem 

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                            
     6 root      20   0    7324     96      0 R  10.0  0.0   1:08.04 stress                             
     7 root      20   0    7324     96      0 R  10.0  0.0   1:08.11 stress                             
     8 root      20   0    7324     96      0 R  10.0  0.0   1:08.02 stress                             
     9 root      20   0    7324     96      0 R  10.0  0.0   1:08.40 stress                             
    10 root      20   0    7324     96      0 R  10.0  0.0   1:07.82 stress                             
    11 root      20   0    7324     96      0 R  10.0  0.0   1:07.89 stress 

分別進入cpu512和cpu1024之后可以看到，%cpu的比例差不多是1：2，符合我們設置的–cpu-shares參數

1.1.3.2 cpu core控制

對於多核cpu的服務器，docker還可以控制容器運行使用那些cpu內核，以及使用–cpuset-cpus參數，這對於具有多cpu服務器尤其有用，可以對需要高性能計算的容器進行性能最優的配置

root@harbor2:~# docker run -itd --name cpu1 --cpuset-cpus 0-1 registry.cn-hangzhou.aliyuncs.com/haozheyu/centos-stress:v1
326cacc9423fefe5f942441f449f8f22c9319ce59d28c18ca6fbb4651d8263c5
root@harbor2:~# docker exec -it cpu1 bash
[root@326cacc9423f /]# cat /sys/fs/cgroup/cpuset/cpuset.cpus
0-1

//通過下列指令可以看到容器中進程與cpu內核的綁定關系，達到綁定cpu內核的目的
root@harbor2:~# docker exec -it cpu1 taskset -c -p 1
pid 1's current affinity list: 0,1    #容器內部的第一個進程號pid為1，被綁定到指定到的cpu上運行

1.1.3.3 cpu配額控制參數的混合使用

docker run -itd --name cpu2 --cpuset-cpus 1 --cpu-shares 512 registry.cn-hangzhou.aliyuncs.com/haozheyu/centos-stress:v1 stress -c 1

docker run -itd --name cpu3 --cpuset-cpus 3 --cpu-shares 1024 registry.cn-hangzhou.aliyuncs.com/haozheyu/centos-stress:v1 stress -c 1

1.1.3.4 內存配額

• 與操作系統類似，容器可使用的內存包括兩部分：物理內存和swap
  容器通過 -m或–memory設置內存的使用限額，例如：-m 300M；通過–memory-swap設置內存+swap的使用限額
• 實例如下，允許容器最多使用200M的內存和300M的swap

docker run -itd --name mem1 –m 300M --memory-swap=300M registry.cn-hangzhou.aliyuncs.com/haozheyu/centos-stress:v1 stress -m 3 --vm-bytes 300M
#會生成3個進程，每個進程占用300M內存
root@harbor2:~# docker exec -it 0504806aa832 bash
#top之后會看到這300M使用的內存在3個線程上來回飄

1.1.3.5 Block IO的限制

默認情況下，所有容器能平等地讀寫磁盤，可以通過設置–blkio-weight參數來改變容器block IO的優先級

docker run -itd --name blokio1 --blkio-weight 600 registry.cn-hangzhou.aliyuncs.com/haozheyu/centos-stress:v1 

docker run -itd --name blokio2 --blkio-weight 300 registry.cn-hangzhou.aliyuncs.com/haozheyu/centos-stress:v1

docker exec -it --name blokio2 bash
cat /sys/fs/cgroup/blkio/blkio.weight
300

1.1.3.6 bps和iops限制

bps是byte per second，每秒讀寫的數據量。iops是io per second, 每秒IO的次數。
可通過以下參數控制容器的bps和iops:

--device-read-bps：限制讀某個設備的bps.
--device-write-bps：限制寫某個設備的bps.
--device-read-iops：限制讀某個設備的iops.
--device-write-iops：限制寫某個設備的iops。

下面的示例是限制容器寫/dev/sda 的速率為5 MB/s

docker run -itd --name bps1 --device-write-bps /dev/sda:5MB registry.cn-hangzhou.aliyuncs.com/haozheyu/centos-stress:v1

docker exec -it bps1 bash
dd if=/dev/zero of=test2 bs=1M count=1024 oflag=direct

1.2 docker安裝和基礎命令

1.2.1 在線安裝

如果你過去安裝過 docker，先刪掉:

sudo apt-get remove docker docker-engine docker.io

首先安裝依賴:

sudo apt-get install apt-transport-https ca-certificates curl gnupg2 software-properties-common

根據你的發行版

信任 Docker 的 GPG 公鑰:

curl -fsSL https://download.docker.com/linux/[ubuntu|debian]/gpg | sudo apt-key add -
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

添加軟件倉庫:

echo 'deb [arch=amd64] https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux/ubuntu bionic stable' >> /etc/apt/sources.list

最后安裝

sudo apt-get update
sudo apt-get install docker-ce

驗證

root@harbor2:~# docker run -it --rm nginx bash
Unable to find image 'nginx:latest' locally
latest: Pulling from library/nginx
a330b6cecb98: Pull complete 
5ef80e6f29b5: Pull complete 
f699b0db74e3: Pull complete 
0f701a34c55e: Pull complete 
3229dce7b89c: Pull complete 
ddb78cb2d047: Pull complete 
Digest: sha256:a05b0cdd4fc1be3b224ba9662ebdf98fe44c09c0c9215b45f84344c12867002e
Status: Downloaded newer image for nginx:latest
root@d8840d0d3a84:/# exit 
exit

1.2.2 離線安裝

cat >> limits.conf << EOF
*             soft    core            unlimited
*             hard    core            unlimited
*	      soft    nproc           1000000
*             hard    nproc           1000000
*             soft    nofile          1000000
*             hard    nofile          1000000
*             soft    memlock         32000
*             hard    memlock         32000
*             soft    msgqueue        8192000
*             hard    msgqueue        8192000
EOF
#如果是非root用戶將*修改成用戶名稱再添加一次
cat >> sysctl.conf << EOF
net.ipv4.ip_forward=1
vm.max_map_count=262144
kernel.pid_max=4194303
fs.file-max=1000000
net.ipv4.tcp_max_tw_buckets=6000
net.netfilter.nf_conntrack_max=2097152
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
vm.swappiness=0
EOF

cat >> containerd.service << EOF
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target
[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/bin/containerd
Type=notify
Delegate=yes
KillMode=process
Restart=always
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=1048576
# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
[Install]
WantedBy=multi-user.target
EOF

cat >> docker.service << EOF
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
# Both the old, and new location are accepted by systemd 229 and up, so using the old location
# to make them work for either version of systemd.
StartLimitBurst=3
# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
# this option work for either version of systemd.
StartLimitInterval=60s
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Comment TasksMax if your systemd version does not support it.
# Only systemd 226 and above support this option.
TasksMax=infinity
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target
EOF

cat >> docker.socket << EOF
[Unit]
Description=Docker Socket for the API
PartOf=docker.service
[Socket]
ListenStream=/var/run/docker.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker
[Install]
WantedBy=sockets.target
EOF


#!/bin/bash
DIR=`pwd`
PACKAGE_NAME="docker-19.03.15.tgz"
DOCKER_FILE=${DIR}/${PACKAGE_NAME}
install_docker(){
  grep "Ubuntu" /etc/issue &> /dev/null
  if [ $? -eq 0 ];then
    /bin/echo  "當前系統是`cat /etc/issue`,即將開始系統初始化、配置docker-compose與安裝docker" && sleep 1
    \cp ${DIR}/limits.conf /etc/security/limits.conf
    \cp ${DIR}/sysctl.conf /etc/sysctl.conf
    
    /bin/tar xvf ${DOCKER_FILE}
    \cp docker/*  /usr/bin 
    \cp containerd.service /lib/systemd/system/containerd.service
    \cp docker.service  /lib/systemd/system/docker.service
    \cp docker.socket /lib/systemd/system/docker.socket
    \cp ${DIR}/docker-compose-Linux-x86_64_1.24.1 /usr/bin/docker-compose
    ulimit -n 1000000 
    /bin/su -c - [運維用戶] "ulimit -n 1000000"
    /bin/echo "docker 安裝完成!" && sleep 1 
    systemctl  enable containerd.service && systemctl  restart containerd.service
    systemctl  enable docker.service && systemctl  restart docker.service
    systemctl  enable docker.socket && systemctl  restart docker.socket 
  fi
};install_docker

1.3 docker鏡像加速

國內下載國外的鏡像有時候會很慢，因此可以更改 docker 配置文件添加一個加速器，可以通過加速器達到加速下載鏡像的目的。

#cat >> /etc/docker/daemon.json << EOF
{
  "registry-mirrors": [
    "https://docker.mirrors.ustc.edu.cn",
    "http://hub-mirror.c.163.com",
    "https://mirror.baidubce.com"
  ],
  "max-concurrent-downloads": 10,
  "log-driver": "json-file",
  "log-level": "warn",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
    },
  "data-root": "/var/lib/docker"
}
EOF

root@harbor2:~# systemctl restart docker
root@harbor2:~# docker info

1.4 docker基礎命令

[root@linux-docker ~]# docker ps
[root@linux-docker ~]# docker ps -a

[root@docker-server1 ~]# docker rm 11445b3a84d3
[root@docker-server1 ~]# docker rm -f 11445b3a84d3

[root@docker-server1 ~]# docker run -P docker.io/nginx
[root@docker-server1 ~]# docker run -p 81:80 --name nginx-test-port1 nginx
[root@docker-server1 ~]# docker run -p 192.168.10.205:83:80/udp --name nginx-test-port4

[root@docker-server1 ~]# docker logs -f nginx-test-port3 #持續查看

[root@docker-server1 ~]# docker port nginx-test-port5

[root@docker-server1 ~]# docker run -t -i --name test-centos2 docker.io/centos /bin/bash

[root@linux-docker opt]# docker run -it --rm --name nginx-delete-test docker.io/nginx

[root@docker-server1 ~]# docker run -d centos /usr/bin/tail -f '/etc/hosts'

[root@docker-server1 ~]# docker stop f821d0cd5a99 
[root@docker-server1 ~]# docker start f821d0cd5a99

[root@docker-server1 ~]# docker exec -it f821d0cd5a99 bash

[root@docker-server1 ~]# docker ps -a |awk '{print $1}' |xargs docker rm -f
[root@docker-server1 ~]# docker ps -a -q |xargs docker kill
[root@docker-server1 ~]# docker rm -f `docker ps -aq -f status=exited`

2 docker鏡像管理

2.1 鏡像管理命令

// 查看本地的所有鏡像
docker images
// 刪除虛懸鏡像
docker image prune
// 刪除指定鏡像
docker rmi 鏡像名或ID

// 給鏡像打上TAG
docker tag 鏡像ID xxx:xxx

// 將鏡像保存到文件，將文件發給其他人，其他人再載入即可
docker save 鏡像ID > xxx.tar
// 將文件載入
docker load < xxx.tar

2.2 鏡像構建

2.2.1 手動構建

root@harbor2:~# docker run --name centos-yum-nginx --rm -it centos:7 bash
#在容器里邊安裝以下測試
    1  yum install -y wget bash-com* epel-rel*
    2  yum install -y nginx
    3  vim /etc/nginx/nginx.conf
    4  yum install -y vim
    5  touch 123.txt
root@harbor2:~# docker commit -a "[e-mail]" -m "nginx yum v1" --change="EXPOSE 80 443" centos-yum-nginx centos-nginx:v1
sha256:020e268ddccd5707e82e7cc593be4e78951503351bbde91e2d72e0ffddad61a1  

root@harbor2:~# docker run -it --rm centos-nginx:v1 bash
[root@020e268ddccd /]# ls
123.trxt           bin  etc   lib    media  opt   root  sbin  sys  usr
anaconda-post.log  dev  home  lib64  mnt    proc  run   srv   tmp  var
[root@020e268ddccd /]# exit

2.2.2 Dockerfile構建

腳本中的常用語法如下：

FROM 鏡像基於另一個鏡像，就是在另一個鏡像的基礎上再執行一些腳本構建出新的鏡像
MAINTAINER xx.xxx [e-mail] 鏡像維護者的信息 
RUN 執行命令
WORKDIR 指定工作目錄，相當於cd，一般在Dockerfile結尾會將工作目錄切到常用的目錄下，這樣在docker exec進入容器時就會默認進入到此目錄下，省去用戶再cd目錄的操作
COPY 復制當前目錄的文件到鏡像中，也可以從另一個鏡像復制文件
ADD 添加當前目錄的文件到鏡像中，如果要添加的文件是*.tar.gz|tgz格式，則會自動解壓到鏡像的指定目錄下
ENV 設置鏡像中的環境變量
CMD 指定鏡像在啟動容器時執行的命令，在Dockerfile中CMD只能出現一次
ENTRYPOINT 指定鏡像在啟動容器時執行的命令或腳本，由於CMD只能出現一次，如果要在容器啟動時執行多條命令就可以用entrypoint代替，可以在鏡像中添加一個.sh文件，在.sh文件中寫多條命令，要注意.sh文件需要有可執行權限，一般會RUN chmod u+x xxx.sh添加可執行權限。這里給出一條命令的寫法["java", "-Djava.security.egd=file:/dev/./urandom", "-jar", "/webapps/api.jar", "--spring.profiles.active=test"]
EXPOSE 說明暴露的端口，寫不寫都可以，但是寫了之后，在docker inspect查看鏡像時可以查看到鏡像會暴露哪些端口
VOLUME 該指令可以指定一個或多個目錄作為容器的數據卷。容器運行時應該盡量保持容器存儲層不發生寫操作，對於數據庫類需要保存動態數據的應用，其數據庫文件應該保存於卷(volume)中。為了防止運行時用戶忘記將動態文件所保存目錄掛載為卷，在 Dockerfile 中，我們可以事先指定某些目錄掛載為匿名卷，這樣在運行時如果用戶不指定掛載，其應用也可以正常運行，不會向容器存儲層寫入大量數據
例如：將 /data 目錄作為容器數據卷目錄 /data 目錄就會在運行時自動掛載為匿名卷，任何向 /data 中寫入的信息都不會記錄進容器存儲層，從而保證了容器存儲層的無狀態化。當然，運行時可以覆蓋這個掛載設置。比如
docker run -d -v mydata:/data xxxx 命令中，就使用了 mydata 這個命名卷掛載到了 /data 這個位置，替代了 Dockerfile 中定義的匿名卷的掛載配置

springboot 鏡像構建

FROM registry.cn-hangzhou.aliyuncs.com/haozheyu/jdk:ora8u201-alpine3.9-glibc2.29


ADD target/x-admin-1.0-SNAPSHOT.jar /x-admin-1.0-SNAPSHOT.jar
EXPOSE 8080

# 指定docker容器啟動時運行jar包
ENTRYPOINT ["java", "-jar","/x-admin-1.0-SNAPSHOT.jar"]
# 指定維護者的名字
MAINTAINER [e-mail]

構建nginx鏡像

Nginx 鏡像的 DockerFile

FROM centos:7

MAINTAINER xxx<[e-mail]>

# 安裝軟件
RUN yum -y update && yum -y install gcc gdb strace gcc-c++ autoconf libjpeg libjpeg-devel libpng libpng-devel freetype freetype-devel libxml2 libxml2-devel zlib zlib-devel glibc glibc-devel glib2 glib2-devel bzip2 bzip2-devel ncurses ncurses-devel curl curl-devel e2fsprogs patch e2fsprogs-devel krb5-devel libidn libidn-devel openldap-devel nss_ldap openldap-clients openldap-servers libevent-devel libevent uuid-devel uuid openssl openssl-devel pcre pcre-devel

# 創建用戶
RUN groupadd www
RUN useradd -g www www -s /bin/false

# 定義Nginx版本號
ENV VERSION 1.14.2

# 下載並解壓文件
RUN mkdir -p /usr/local/src/
ADD http://nginx.org/download/nginx-$VERSION.tar.gz /usr/local/src
RUN tar -xvf /usr/local/src/nginx-$VERSION.tar.gz -C /usr/local/src/

# 創建安裝目錄
ENV NGINX_HOME /usr/local/nginx
RUN mkdir -p $NGINX_HOME
RUN chown -R www:www $NGINX_HOME

# 進入解壓目錄
WORKDIR /usr/local/src/nginx-$VERSION

# 編譯安裝
RUN ./configure \
	--user=www \
	--group=www \
	--prefix=$NGINX_HOME \
	--with-http_ssl_module \
	--with-http_realip_module \
	--with-http_gzip_static_module \
	--with-http_stub_status_module
RUN make
RUN make install

# 備份Nginx的配置文件
RUN mv $NGINX_HOME/conf/nginx.conf $NGINX_HOME/conf/nginx.conf.default

# 設置環境變量
ENV PATH $PATH:$NGINX_HOME/sbin

# 創建WebApp目錄
ENV WEB_APP /usr/share/nginx/html
RUN mkdir -p $WEB_APP

# 設置默認工作目錄
WORKDIR $WEB_APP

# 暴露端口
EXPOSE 80
EXPOSE 443

# 清理壓縮包與解壓文件
RUN rm -rf /usr/local/src/nginx*

CMD $NGINX_HOME/sbin/nginx -g 'daemon off;' -c $NGINX_HOME/conf/nginx.conf

構建 Nginx 鏡像

# 構建Nginx鏡像
# docker build -f docker-file -t centos-nginx:1.14.2 .

Docker-Compose 管理 Nginx 鏡像

version: "3.5"

services:
  nginx:
    image: centos-nginx:1.14.2
    container_name: nginx-1.14.2
    privileged: false
    ports:
      - 80:80
      - 443:443
    volumes:
       - '/container/nginx/wwwroot:/usr/share/nginx/html'
       - '/container/nginx/logs:/usr/local/nginx/logs'
       - '/container/nginx/nginx.conf:/usr/local/nginx/conf/nginx.conf'

# 上面的配置是docker-compose.yml文件的內容，數據卷部分可以根據自己的實際情況進行修改
# 注意： 在/container/nginx/nginx.conf配置文件中，需要手動修改root的路徑為/usr/share/nginx/html

創建並啟動 Nginx 容器

# 創建並啟動容器
# docker-compose up -d

# 查看容器的運行狀態
# docker-compose ps

手動構建tengine

FROM centos:7

MAINTAINER xxx<[e-mail]>

# 安裝軟件
RUN yum -y update && yum -y install vim tree htop tmux net-tools telnet wget curl supervistor autoconf git gcc gcc-c++ pcre pcre-devel zlib zlib-devel openssl openssl-devel

# 創建用戶
RUN groupadd tengine
RUN useradd -g tengine tengine

# 定義Tengine版本號
ENV VERSION 2.2.3

# 下載並解壓文件
RUN mkdir -p /usr/local/src/
ADD http://tengine.taobao.org/download/tengine-$VERSION.tar.gz /usr/local/src
RUN tar -xvf /usr/local/src/tengine-$VERSION.tar.gz -C /usr/local/src/

# 創建安裝目錄
ENV TENGINE_HOME /usr/local/tengine
RUN mkdir -p $TENGINE_HOME

# 進入解壓目錄
WORKDIR /usr/local/src/tengine-$VERSION

# 編譯安裝
RUN ./configure \
	--user=tengine \
	--group=tengine \
	--prefix=$TENGINE_HOME \
	--with-http_ssl_module \
	--with-http_realip_module \
	--with-http_concat_module  \
	--with-http_gzip_static_module \
	--with-http_stub_status_module \
	--with-http_upstream_consistent_hash_module
RUN make
RUN make install

# 備份Tengine的配置文件
RUN mv $TENGINE_HOME/conf/nginx.conf $TENGINE_HOME/conf/nginx.conf.default

# 設置環境變量
ENV PATH $PATH:$TENGINE_HOME/sbin

# 創建WebApp目錄
ENV WEB_APP /usr/share/tengine/html
RUN mkdir -p $WEB_APP

# 設置默認工作目錄
WORKDIR $WEB_APP

# 暴露端口
EXPOSE 80
EXPOSE 443

# 清理壓縮包與解壓文件
RUN rm -rf /usr/local/src/tengine*

CMD $TENGINE_HOME/sbin/nginx -g 'daemon off;' -c $TENGINE_HOME/conf/nginx.conf

構建 Tengine 鏡像

# 構建Tengine鏡像
# docker build -f docker-file -t centos-tengine:2.2.3 .

Docker-Compose 管理 Tengine 鏡像

version: "3.5"

services:
  tengine:
    image: centos-tengine:2.2.3
    container_name: tengine:2.2.3
    restart: always
    privileged: false
    ports:
      - 80:80
      - 443:443
    volumes:
       - '/container/tengine/wwwroot/:/usr/share/tengine/html'
       - '/container/tengine/logs:/usr/local/tengine/logs'
       - '/container/tengine/nginx.conf:/usr/local/tengine/conf/nginx.conf'

# 上面的配置是docker-compose.yml文件的內容，數據卷部分可以根據自己的實際情況進行修改
# 注意： 在/container/tengine/nginx.conf配置文件中需要手動修改root的路徑為/usr/share/tengine/html

創建並啟動 Tengine 容器

# 創建並啟動容器
# docker-compose up -d

# 查看容器的運行狀態
# docker-compose ps

構建tomcat 容器

FROM registry.cn-hangzhou.aliyuncs.com/haozheyu/jdk:ora8u201-alpine3.9-glibc2.29
#env 
ENV TZ "Asia/Shanghai" 
ENV LANG en_US.UTF-8 
ENV TERM xterm 
ENV TOMCAT_MAJOR_VERSION 8 
ENV TOMCAT_MINOR_VERSION 8.5.45 
ENV CATALINA_HOME /apps/tomcat 
ENV APP_DIR ${CATALINA_HOME}/webapps 

#tomcat 
RUN mkdir /apps 
ADD apache-tomcat-8.5.45.tar.gz /apps 
RUN ln -sv /apps/apache-tomcat-8.5.45 /apps/tomcat

docker build -t tomcat-bash:v8.5.45 .

3 docker 數據管理

Docker 的鏡像是分層設計的，鏡像層是只讀的，通過鏡像啟動的容器添加了一層可讀寫的文件系統，用戶寫入的數據都保存在這一層當中。 如果要將寫入到容器的數據永久保存，則需要將容器中的數據保存到宿主機的指定目錄，目前 Docker 的數據類型分為兩種:

一是數據卷(data volume)，數據卷類似於掛載的一塊磁盤，數據容器是將數據保存在一個容器上.

二是數據卷容器(Data volume container), 數據卷容器是將宿主機的目錄掛載至一個專門的數據卷容器，然后讓其他容器通過數據卷容器讀寫宿主機的數據。

# docker inspect 89e6038bf99f #查看指定 PID 的容器信息
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/119f71309ce1be75aab7760dd04bcb7c02ad6b41725c10d3bb6ed590a945359f-init/diff:/var/lib/docker/overlay2/eb1a488d3821fa84d0fbf871891404c5c6cd68046775225a9f652b1591fd9a2c/diff:/var/lib/docker/overlay2/f8bacdf8595bab5330b8a07d83d823722e817f6219808088ce5ab873371e403d/diff",
                "MergedDir": "/var/lib/docker/overlay2/119f71309ce1be75aab7760dd04bcb7c02ad6b41725c10d3bb6ed590a945359f/merged",
                "UpperDir": "/var/lib/docker/overlay2/119f71309ce1be75aab7760dd04bcb7c02ad6b41725c10d3bb6ed590a945359f/diff",
                "WorkDir": "/var/lib/docker/overlay2/119f71309ce1be75aab7760dd04bcb7c02ad6b41725c10d3bb6ed590a945359f/work"
            },
            "Name": "overlay2"
        },
        
Lower Dir：image 鏡像層(鏡像本身，只讀) 
Upper Dir：容器的上層(讀寫) 
Merged Dir：容器的文件系統，使用 Union FS（聯合文件系統）將 lowerdir 和 upper Dir：合並給容器使用。 
Work Dir：容器在 宿主機的工作目錄

數據卷實際上就是宿主機上的目錄或者是文件，可以被直接 mount 到容器當中使用。

3.1 啟動測試容器驗證

啟動倆個終端

docker run -it --rm --name test2 -v /tmp:/tmp 53e7a2d8aadd bash
docker run -it --rm --name test1 -v /tmp:/tmp 53e7a2d8aadd bash
[root@2efd6b2984d1 /]# cd /tmp/
[root@2efd6b2984d1 tmp]# touch mytest2
#在另一個容器
[root@81d6a015cd7c /]# cd /tmp/
#宿主機的tmp同樣也有
[root@81d6a015cd7c tmp]# ls
mytest2
root@harbor2:/tmp# ls
mytest2

3.2 數據卷的特點

1、數據卷是宿主機的目錄或者文件，並且可以在多個容器之間共同使用。
2、在宿主機對數據卷更改數據后會在所有容器里面會立即更新。
3、數據卷的數據可以持久保存，即使刪除使用使用該容器卷的容器也不影響。
4、在容器里面的寫入數據不會影響到鏡像本身。

3.3 數據卷的使用場景

1、日志輸出 
2、靜態 web 頁面 
3、應用配置文件 
4、多容器間目錄或文件共享

3.4 鏡像管理

在容器管理這一節主要想說一下容器占用磁盤空間的問題，容器運行久了，占用的空間會越來越大，當一台服務器上運行多個容器時，可能硬盤突然滿了，此時就需要能及時找到哪個容器占用了過多的空間，以及如何盡快清理出空間

通過docker system df命令查看docker占用的磁盤空間大小

TYPE這一列Images是鏡像的大小，Containers是容器占用的大小，Local Volumes是掛載的數據卷大小

• 如果鏡像很大，可以docker images查看是否有哪些鏡像不需要，可以刪除
• 如果數據卷很大，可能是掛載的目錄被寫入很多文件，如容器中運行軟件產生的日志文件或產生的數據文件

通過docker system df -v查看更詳細的數據，包括每個鏡像占用的空間，每個容器占用的空間等

通過docker ps --size查看運行中的容器占用的空間

如果docker根目錄所在的硬盤確實騰不出空間了，可以掛一個新的硬盤，service docker stop關閉docker服務，將當前docker根目錄下的文件移動到新的硬盤（默認docker的根目錄是/var/lib/docker），修改/etc/docker/daemon.json添加或修改data-root到新的硬盤上，再service docker start啟動docker服務，這樣就完成了docker目錄的遷移

3.5 鏡像縮容

構建的鏡像當然是越小越好，使鏡像中包含的程序剛剛好，這樣構建出來的鏡像才能最小化

docker鏡像是由構建的步驟一層一層疊加出來的，這是linux中自帶的overlay文件系統，如果在第1步添加了一個文件到鏡像中，在第2步再刪除該文件，這個文件的大小仍然會算在鏡像大小中，如果將第1步和第2步合並成一步，那么鏡像大小就不再包含該文件大小了，如下：

RUN 添加文件到鏡像中 \
  && 刪除文件

上面的\是轉義換行符的，&&是連接兩個命令的符號

docker提供了一個命令可以查看鏡像的構建歷史步驟

docker history 鏡像名或ID

root@harbor2:/tmp# docker history c0fa8ada5a84
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
c0fa8ada5a84        10 months ago       /bin/sh -c #(nop)  CMD ["/usr/local/openrest…   0B                  
62bbb0b2a328        10 months ago       /bin/sh -c #(nop)  EXPOSE 80                    0B                  
e228ad0a338b        10 months ago       /bin/sh -c #(nop) WORKDIR /                     0B                  
b109c44fb8bb        10 months ago       /bin/sh -c #(nop) ADD file:c26c0723c8896ccb9…   857B                
4ed6730165c6        10 months ago       /bin/sh -c make && make install                 95.1MB              
de61b50377e0        10 months ago       /bin/sh -c ./configure --add-module=/usr/loc…   65.2MB              
ddbc24e63d21        10 months ago       /bin/sh -c rm -rf ./Makefile                    0B                  
49631835d627        10 months ago       /bin/sh -c #(nop) WORKDIR /usr/local/src/ope…   0B                  
6e16d85978fd        10 months ago       /bin/sh -c unzip nginx-module-vts.zip           2.58MB              
f393685b4b14        10 months ago       /bin/sh -c #(nop) WORKDIR /usr/local/src        0B                  
f6d9e27571d4        10 months ago       /bin/sh -c #(nop) ADD file:9bb161e63fe3864a4…   1.49MB              
1548553d9fc8        10 months ago       /bin/sh -c #(nop) ADD file:3d10850fb8b3e933a…   27.1MB              
e8b57fcbd31d        10 months ago       /bin/sh -c apt-get install unzip make gcc li…   210MB               
bc016c01c294        10 months ago       /bin/sh -c apt-get update                       26.2MB              
d820629463da        10 months ago       /bin/sh -c echo 'deb http://mirrors.aliyun.c…   351B                
5991db817e28        10 months ago       /bin/sh -c echo 'deb http://mirrors.aliyun.c…   260B                
1f911ec5d70c        10 months ago       /bin/sh -c echo 'deb http://mirrors.aliyun.c…   171B                
65e875ff00b0        10 months ago       /bin/sh -c echo 'deb http://mirrors.aliyun.c…   81B                 
90088fba500c        10 months ago       /bin/sh -c #(nop) WORKDIR /etc/apt              0B                  
fab5e942c505        10 months ago       /bin/sh -c #(nop)  CMD ["/bin/bash"]            0B                  
<missing>           10 months ago       /bin/sh -c mkdir -p /run/systemd && echo 'do…   7B                  
<missing>           10 months ago       /bin/sh -c set -xe   && echo '#!/bin/sh' > /…   745B                
<missing>           10 months ago       /bin/sh -c rm -rf /var/lib/apt/lists/*          0B                  
<missing>           10 months ago       /bin/sh -c #(nop) ADD file:513ae777bc4042f84…   126MB

通過這個命令可以查看到指定鏡像在構建時每一步操作所占用的大小

3.6 多階段構建

docker也提供了多階段構建的能力，以構建Java鏡像為例，我們需要maven來將源碼編譯成.jar，在容器運行時只需要jre，這樣maven只用於構建時，如果能將maven從鏡像中排除掉，最終構建出的java鏡像就會小很多

# 先基於maven鏡像來編譯出.jar
FROM maven:3-jdk-8 AS builder
COPY ./src/ /usr/src/java/
WORKDIR /usr/src/java/
RUN mvn clean package

# 再基於openjdk鏡像來運行java，此時就需要從上面鏡像中復制編譯好的.jar文件
FROM openjdk:8u102-jre
COPY --from=builder /usr/src/java/target/xxx.jar /usr/local/jar/xxx.jar
CMD ["java", "-jar", "/usr/local/jar/xxx.jar"]

4 Harbor 倉庫使用

Harbor是一個用於存儲和分發Docker鏡像的企業級Registry服務器，由vmware開源，其通過添加一些企業必需的功能特性，例如安全、標識和管理等，擴展了開源 Docker Distribution。作為一個企業級私有 Registry 服務器，Harbor 提供了更好的性能和安全。提升用戶使用 Registry 構建和運行環境傳輸鏡像的效率。Harbor 支持安裝在多個 Registry 節點的鏡像資源復制，鏡像全部保存在私有 Registry 中，確保數據和知識產權在公司內部網絡中管控，另外，Harbor 也提供了高級的安全特性，諸如用戶管理，訪問控制和活動審計等。

vmware 官方開源服務列表地址：https://vmware.github.io/harbor/cn/，

harbor 官方 github 地址：https://github.com/vmware/harbor

harbor 官方網址：https://goharbor.io/

下載地址：https://github.com/vmware/harbor/releases

安裝文檔：https://goharbor.io/docs/2.3.0/install-config/

4.1 harbor部署

4.1.1 機器環境

節點hostname	host	IP
harbor	reg.local.com	192.168.43.131

4.1.2 hostname

[root@base1 ~]# hostnamectl set-hostname harbor --static

4.1.3 網絡設置

[root@base1 ~]# vi /etc/sysconfig/network-scripts/ifcfg-ens33
BOOTPROTO="static" #dhcp改為static 
ONBOOT="yes" #開機啟用本配置
IPADDR=192.168.43.131 #靜態IP
GATEWAY=192.168.43.1 #默認網關
NETMASK=255.255.255.0 #子網掩碼
DNS1=114.114.114.114 #DNS 配置
DNS2=8.8.8.8 #DNS 配置

$# reboot

4.1.4 查看主機名

hostname

4.1.5 ip:hostname到每一台機器節點

echo "192.168.43.131 reg.local.com" >> /etc/hosts

4.1.6 安裝依賴環境，注意：每一台機器都需要安裝此依賴環境

yum install -y conntrack ntpdate ntp ipvsadm ipset jq iptables curl sysstatlibseccomp wget vim net-tools git iproute lrzsz bash-completion tree bridge-utils unzip bind-utils gcc

4.1.7 docker部署

4.1.7.1 安裝docker

yum install -y yum-utils device-mapper-persistent-data lvm2

#緊接着配置一個穩定的倉庫、倉庫配置會保存到/etc/yum.repos.d/docker-ce.repo文件中
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
#更新Yum安裝的相關Docker軟件包&安裝Docker CE
yum update -y && yum install docker-ce

4.1.7.2 設置docker daemon文件

#創建/etc/docker目錄
mkdir /etc/docker
#更新daemon.json文件
cat > /etc/docker/daemon.json <<EOF
{
"exec-opts":["native.cgroupdriver=systemd"],
"log-driver":"json-file",
"log-opts":{"max-size":"100m"}
}
EOF
#注意：一定注意編碼問題，出現錯誤---查看命令：journalctl -amu docker 即可發現錯誤
#創建，存儲docker配置文件
mkdir -p /etc/systemd/system/docker.service.d

4.1.7.3 重啟docker服務

systemctl daemon-reload && systemctl restart docker && systemctl enable docker

4.1.8 安裝compose

打開github.com官網，在登錄頁面的右上角搜索compose找到docker/compose再找releases，（網址：github.com/docker/comp…

復制自己所需版本下提供的兩條命令，在第一台Docker服務器上依次進行操作：

#在線下載docker-compose ,harbor需要借助docker-compose安裝
#復制官網上的上述命令
curl -L https://github.com/docker/compose/releases/download/1.27.4/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose

#賦予該命令執行權限
chmod u+x /usr/local/bin/docker-compose  
#查看其版本信息
docker-compose -version   
docker-compose version 1.24.1, build 4667896b

4.1.9 Harbor安裝

4.1.9.1 下載Harbor並配置

#下載harbor
wget https://github.com/goharbor/harbor/releases/download/v2.1.2/harbor-offline-installer-v2.1.2.tgz
#將下載的安裝包解壓到指定目錄
tar zxf harbor-offline-installer-v2.1.2.tgz -C /usr/local
#切換至解壓后的目錄中
cd /usr/local/harbor/
#編輯這個配置文件
mv harbor.yml.tmpl harbor.yml

vim harbor.yml

修改harbor.yml配置文件

注意點#TODO

# Configuration file of Harbor

# The IP address or hostname to access admin UI and registry service.
# DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
hostname: reg.local.com|192.168.43.131

# http related config
# http: #TODO
  # port for http, default is 80. If https enabled, this port will redirect to https port
  # port: 80#TODO

# https related config
https:
  # https port for harbor, default is 443
  port: 443
  # The path of cert and key files for nginx
  certificate: /data/cert/reg.local.com.crt#TODO
  private_key: /data/cert/reg.local.com.key#TODO

# # Uncomment following will enable tls communication between all harbor components
# internal_tls:
#   # set enabled to true means internal tls is enabled
#   enabled: true
#   # put your cert and key files on dir
#   dir: /etc/harbor/tls/internal

# Uncomment external_url if you want to enable external proxy
# And when it enabled the hostname will no longer used
# external_url: https://reg.mydomain.com:8433

# The initial password of Harbor admin
# It only works in first time to install harbor
# Remember Change the admin password from UI after launching Harbor.
harbor_admin_password: Harbor12345

# Harbor DB configuration
database:
  # The password for the root user of Harbor DB. Change this before any production use.
  password: root123
  # The maximum number of connections in the idle connection pool. If it <=0, no idle connections are retained.
  max_idle_conns: 50
  # The maximum number of open connections to the database. If it <= 0, then there is no limit on the number of open connections.
  # Note: the default number of connections is 1024 for postgres of harbor.
  max_open_conns: 1000

# The default data volume
data_volume: /data

# Harbor Storage settings by default is using /data dir on local filesystem
# Uncomment storage_service setting If you want to using external storage
# storage_service:
#   # ca_bundle is the path to the custom root ca certificate, which will be injected into the truststore
#   # of registry's and chart repository's containers.  This is usually needed when the user hosts a internal storage with self signed certificate.
#   ca_bundle:

#   # storage backend, default is filesystem, options include filesystem, azure, gcs, s3, swift and oss
#   # for more info about this configuration please refer https://docs.docker.com/registry/configuration/
#   filesystem:
#     maxthreads: 100
#   # set disable to true when you want to disable registry redirect
#   redirect:
#     disabled: false

# Clair configuration
clair:
  # The interval of clair updaters, the unit is hour, set to 0 to disable the updaters.
  updaters_interval: 12

# Trivy configuration
#
# Trivy DB contains vulnerability information from NVD, Red Hat, and many other upstream vulnerability databases.
# It is downloaded by Trivy from the GitHub release page https://github.com/aquasecurity/trivy-db/releases and cached
# in the local file system. In addition, the database contains the update timestamp so Trivy can detect whether it
# should download a newer version from the Internet or use the cached one. Currently, the database is updated every
# 12 hours and published as a new release to GitHub.
trivy:
  # ignoreUnfixed The flag to display only fixed vulnerabilities
  ignore_unfixed: false
  # skipUpdate The flag to enable or disable Trivy DB downloads from GitHub
  #
  # You might want to enable this flag in test or CI/CD environments to avoid GitHub rate limiting issues.
  # If the flag is enabled you have to download the `trivy-offline.tar.gz` archive manually, extract `trivy.db` and
  # `metadata.json` files and mount them in the `/home/scanner/.cache/trivy/db` path.
  skip_update: false
  #
  # insecure The flag to skip verifying registry certificate
  insecure: false
  # github_token The GitHub access token to download Trivy DB
  #
  # Anonymous downloads from GitHub are subject to the limit of 60 requests per hour. Normally such rate limit is enough
  # for production operations. If, for any reason, it's not enough, you could increase the rate limit to 5000
  # requests per hour by specifying the GitHub access token. For more details on GitHub rate limiting please consult
  # https://developer.github.com/v3/#rate-limiting
  #
  # You can create a GitHub token by following the instructions in
  # https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line
  #
  # github_token: xxx

jobservice:
  # Maximum number of job workers in job service
  max_job_workers: 10

notification:
  # Maximum retry count for webhook job
  webhook_job_max_retry: 10

chart:
  # Change the value of absolute_url to enabled can enable absolute url in chart
  absolute_url: disabled

# Log configurations
log:
  # options are debug, info, warning, error, fatal
  level: info
  # configs for logs in local storage
  local:
    # Log files are rotated log_rotate_count times before being removed. If count is 0, old versions are removed rather than rotated.
    rotate_count: 50
    # Log files are rotated only if they grow bigger than log_rotate_size bytes. If size is followed by k, the size is assumed to be in kilobytes.
    # If the M is used, the size is in megabytes, and if G is used, the size is in gigabytes. So size 100, size 100k, size 100M and size 100G
    # are all valid.
    rotate_size: 200M
    # The directory on your host that store log
    location: /var/log/harbor

  # Uncomment following lines to enable external syslog endpoint.
  # external_endpoint:
  #   # protocol used to transmit log to external endpoint, options is tcp or udp
  #   protocol: tcp
  #   # The host of external endpoint
  #   host: localhost
  #   # Port of external endpoint
  #   port: 5140

#This attribute is for migrator to detect the version of the .cfg file, DO NOT MODIFY!
_version: 2.0.0

# Uncomment external_database if using external database.
# external_database:
#   harbor:
#     host: harbor_db_host
#     port: harbor_db_port
#     db_name: harbor_db_name
#     username: harbor_db_username
#     password: harbor_db_password
#     ssl_mode: disable
#     max_idle_conns: 2
#     max_open_conns: 0
#   clair:
#     host: clair_db_host
#     port: clair_db_port
#     db_name: clair_db_name
#     username: clair_db_username
#     password: clair_db_password
#     ssl_mode: disable
#   notary_signer:
#     host: notary_signer_db_host
#     port: notary_signer_db_port
#     db_name: notary_signer_db_name
#     username: notary_signer_db_username
#     password: notary_signer_db_password
#     ssl_mode: disable
#   notary_server:
#     host: notary_server_db_host
#     port: notary_server_db_port
#     db_name: notary_server_db_name
#     username: notary_server_db_username
#     password: notary_server_db_password
#     ssl_mode: disable

# Uncomment external_redis if using external Redis server
# external_redis:
#   # support redis, redis+sentinel
#   # host for redis: <host_redis>:<port_redis>
#   # host for redis+sentinel:
#   #  <host_sentinel1>:<port_sentinel1>,<host_sentinel2>:<port_sentinel2>,<host_sentinel3>:<port_sentinel3>
#   host: redis:6379
#   password:
#   # sentinel_master_set must be set to support redis+sentinel
#   #sentinel_master_set:
#   # db_index 0 is for core, it's unchangeable
#   registry_db_index: 1
#   jobservice_db_index: 2
#   chartmuseum_db_index: 3
#   clair_db_index: 4
#   trivy_db_index: 5
#   idle_timeout_seconds: 30

# Uncomment uaa for trusting the certificate of uaa instance that is hosted via self-signed cert.
# uaa:
#   ca_file: /path/to/ca

# Global proxy
# Config http proxy for components, e.g. http://my.proxy.com:3128
# Components doesn't need to connect to each others via http proxy.
# Remove component from `components` array if want disable proxy
# for it. If you want use proxy for replication, MUST enable proxy
# for core and jobservice, and set `http_proxy` and `https_proxy`.
# Add domain to the `no_proxy` field, when you want disable proxy
# for some special registry.
proxy:
  http_proxy:
  https_proxy:
  no_proxy:
  components:
    - core
    - jobservice
    - clair
    - trivy

4.1.9.2 生成證書

一鍵腳本文件create_cert.sh

#!/bin/bash

# 在該目錄下操作生成證書，正好供harbor.yml使用
mkdir -p /data/cert
cd /data/cert

openssl genrsa -out ca.key 4096
openssl req -x509 -new -nodes -sha512 -days 3650 -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=reg.local.com" -key ca.key -out ca.crt
openssl genrsa -out reg.local.com.key 4096
openssl req -sha512 -new -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=reg.local.com" -key reg.local.com.key -out reg.local.com.csr

cat > v3.ext <<-EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[alt_names]
DNS.1=reg.local.com
DNS.2=harbor
DNS.3=ks-allinone
EOF

openssl x509 -req -sha512 -days 3650 -extfile v3.ext -CA ca.crt -CAkey ca.key -CAcreateserial -in reg.local.com.csr -out reg.local.com.crt
    
openssl x509 -inform PEM -in reg.local.com.crt -out reg.local.com.cert

cp reg.local.com.crt /etc/pki/ca-trust/source/anchors/reg.local.com.crt 
update-ca-trust

執行腳本，生成證書

chmod 755 create_cert.sh
./create_cert.sh

4.1.9.3 安裝

#執行自帶的安裝腳本，安裝完畢，瀏覽器即可訪問
./install.sh
...
[Step 5]: starting Harbor ...
Creating network "harbor_harbor" with the default driver
Creating harbor-log ... done
Creating harbor-db     ... done
Creating registry      ... done
Creating registryctl   ... done
Creating redis         ... done
Creating harbor-portal ... done
Creating harbor-core   ... done
Creating nginx             ... done
Creating harbor-jobservice ... done
✔ ----Harbor has been installed and started successfully.----

4.1.9.4 更新daemon.json文件

cat > /etc/docker/daemon.json <<EOF
{
"exec-opts":["native.cgroupdriver=systemd"],
"log-driver":"json-file","log-opts":{"max-size":"100m"},
"registry-mirrors":["https://pee6w651.mirror.aliyuncs.com"],
"insecure-registries": ["https://reg.local.com"]
}
EOF

#確定80端口正在監聽
netstat -antp | grep 80 

#重啟docker
systemctl daemon-reload && systemctl restart docker

#重啟所有容器
cd /usr/local/harbor
docker-compose stop && docker-compose start
Stopping harbor-jobservice ... done
Stopping nginx             ... done
Stopping harbor-core       ... done
Stopping harbor-portal     ... done
Stopping redis             ... done
Stopping registryctl       ... done
Stopping registry          ... done
Stopping harbor-db         ... done
Stopping harbor-log        ... done
Starting log         ... done
Starting registry    ... done
Starting registryctl ... done
Starting postgresql  ... done
Starting portal      ... done
Starting redis       ... done
Starting core        ... done
Starting jobservice  ... done
Starting proxy       ... done

啟動驗證Harbor（admin/Harbor12345）

4.2 harbor 上傳和下載鏡像

注意：如果我們配置的是 https 的話，本地 docker 就不需要有任何操作就可以訪問 harbor 了

4.2.1 node節點添加私有倉庫

# cat /etc/docker/daemon.json 
{
  "registry-mirrors": [
    "https://docker.mirrors.ustc.edu.cn",
    "http://hub-mirror.c.163.com",
    "https://mirror.baidubce.com"
  ],
  "max-concurrent-downloads": 10,
  "log-driver": "json-file",
  "log-level": "warn",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
    },
  "data-root": "/var/lib/docker",
  "insecure-registries": [
    "xx.xx.xx.xx",
    "xx.xx.xx.xx"
  ]
}

or-------------------------------------------
# cat /etc/systemd/system/multi-user.target.wants/docker.service
Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --insecure-registry=xx.xx.xx.xx

4.2.2 驗證是否可以登錄

# docker login 192.168.43.131 --username=admin --password=Harbor12345
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

4.2.3 鏡像打tag推送

# docker tag centos-nginx:v1 192.168.43.131/library/centos-nginx:v1
# docker push 192.168.43.131/library/centos-nginx:v1

4.3 實現harbor的高可用

Harbor 支持基於策略的 Docker 鏡像復制功能，這類似於 MySQL 的主從同步， 其可以實現不同的數據中心、不同的運行環境之間同步鏡像

4.3.1 准備運行環境

192.168.43.66	harbor1	admin	Harbor12345
192.168.43.67	harbor2	admin	Harbor12345

4.3.2 添加倉庫和復制規則

#192.168.43.66
#1 系統管理-倉庫管理-新建目標
目標名稱：67
目標url：http://192.168.43.67
訪問ID：admin
訪問密碼：Harbor12345
#2 系統管理-復制管理-新建目錄
名稱：push-67
復制模式：push-based
源資源過濾器：全部
目標倉庫：67-http://192.168.43.67
觸發模式：事件驅動

#192.168.43.67
#1 系統管理-倉庫管理-新建目標
目標名稱：66
目標url：http://192.168.43.66
訪問ID：admin
訪問密碼：Harbor12345
#2 系統管理-復制管理-新建目錄
名稱：push-66
復制模式：push-based
源資源過濾器：全部
目標倉庫：66-http://192.168.43.66
觸發模式：事件驅動

4.3.3 驗證復制是否成功

#1 將鏡像push到66的goapp項目下
root@harbor2:~# docker tag registry.cn-hangzhou.aliyuncs.com/haozheyu/centos-stress:v1 192.168.43.66/goapp/centos-stress:11
root@harbor2:~# docker push 192.168.43.66/goapp/centos-stress:11
The push refers to repository [192.168.43.66/goapp/centos-stress]
b2afbca5d0cd: Pushed 
174f56854903: Mounted from goapp/centos 
11: digest: sha256:fac58e4d667483e16f2be89dce8cc8112e5aec8e79ba76d7cd83889c600446a9 size: 741

#2 在67上拉取192.168.43.66/goapp/centos-stress:11
root@harbor:~/harbor# docker pull 192.168.43.67/goapp/centos-stress:11
11: Pulling from goapp/centos-stress
2d473b07cdd5: Pull complete 
97002bbdba0c: Pull complete 
Digest: sha256:fac58e4d667483e16f2be89dce8cc8112e5aec8e79ba76d7cd83889c600446a9
Status: Downloaded newer image for 192.168.43.67/goapp/centos-stress:11

4.3.4 配置harbor的代理

配置使用的VIP：192.168.43.250

4.3.4.1 配置keepalived

#harbor1 harbor2 
apt install -y keepalive haproxy
#harbor1 
cat >> /etc/keepalived/keepalived.conf << EOF
vrrp_instance VI_1 {
    state MASTER
    interface ens33
    virtual_router_id 1
    priority 100
    advert_int 3
    unicast_src_ip 192.168.43.66
    unicast_peer {
    192.168.43.67
    }
    authentication {
    auth_type PASS
    auth_pass 123abc
    }
    virtual_ipaddress {
    192.168.43.250 dev ens33 label ens33:1
    }
}
EOF

#harbor2
cat >> /etc/keepalived/keepalived.conf << EOF
vrrp_instance VI_1 {
    state SLAVE
    interface ens33
    virtual_router_id 1
    priority 100
    advert_int 3
    unicast_src_ip 192.168.43.67
    unicast_peer {
    192.168.43.66
    }
    authentication {
    auth_type PASS
    auth_pass 123abc
    }
    virtual_ipaddress {
    192.168.43.250 dev ens33 label ens33:1
    }
}
EOF

#harbor1 harbo2 啟動服務
root@harbor:~/harbor# systemctl start keepalived.service
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:2d:30:86 brd ff:ff:ff:ff:ff:ff
    inet 192.168.43.66/24 brd 192.168.43.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet `192.168.43.250/32` scope global ens33:1
       valid_lft forever preferred_lft forever
    inet6 240e:418:400:8f6:20c:29ff:fe2d:3086/64 scope global dynamic mngtmpaddr noprefixroute 
       valid_lft 3124sec preferred_lft 3124sec
    inet6 fe80::20c:29ff:fe2d:3086/64 scope link 
       valid_lft forever preferred_lft forever
       
#這里看到VIP已經起來了   

4.3.4.2 配置haproxy

#harbor1 harbor2 追加 /etc/haproxy/haproxy.cfg
listen harbor_80
    bind 192.168.43.250:8180
    mode tcp
    balance source
    server 192.168.43.66 192.168.43.66:80 check inter 2000 fall 3 rise 5
    server 192.168.43.67 192.168.43.67:80 check inter 2000 fall 3 rise 5
#harbor1 harbor2 啟動服務
systemctl start haproxy.service

4.3.4.3 驗證配置

root@harbor:~/harbor# cat /etc/docker/daemon.json 
{
  "registry-mirrors": [
    "https://docker.mirrors.ustc.edu.cn",
    "http://hub-mirror.c.163.com"
  ],
  "max-concurrent-downloads": 10,
  "log-driver": "json-file",
  "log-level": "warn",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
    },
  "data-root": "/var/lib/docker",
    "insecure-registries": [
    "192.168.43.66",
    "192.168.43.67",
    "http://192.168.43.250:8180"
		      ]
}

root@harbor:~/harbor# systemctl restart docker
root@harbor:~/harbor# docker login 192.168.43.250:8180 --username=admin --password=Harbor12345
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

root@harbor:~/harbor# docker pull 192.168.43.250:8180/goapp/centos-stress:11
11: Pulling from goapp/centos-stress
Digest: sha256:fac58e4d667483e16f2be89dce8cc8112e5aec8e79ba76d7cd83889c600446a9
Status: Downloaded newer image for 192.168.43.250:8180/goapp/centos-stress:11
192.168.43.250:8180/goapp/centos-stress:11
root@harbor:~/harbor# docker images
REPOSITORY                                TAG                 IMAGE ID            CREATED             SIZE
192.168.43.250:8180/goapp/centos-stress   11                  dc9ee26d0c43        8 hours ago         377MB
192.168.43.67/goapp/centos-stress         11                  dc9ee26d0c43        8 hours ago         377MB

4.3.5 harbor的配置文件解析

hostname: 192.168.43.66 #ip或域名

http:
  port: 80
https:
  port: 443
  certificate: /your/certificate/path
  private_key: /your/private/key/path
harbor_admin_password: Harbor12345

database:  #pgsql的鏈接配置  
  password: root123
  max_idle_conns: 100
  max_open_conns: 900
  
data_volume: /data  #數據存儲路徑這里可以使用共享存儲，也是高可用的一種實現

# 12 hours and published as a new release to GitHub.
trivy:
  ignore_unfixed: false
  skip_update: false
  #
  # insecure The flag to skip verifying registry certificate
  insecure: false

jobservice:
  # Maximum number of job workers in job service
  max_job_workers: 10

notification:
  # Maximum retry count for webhook job
  webhook_job_max_retry: 10

chart:
  # Change the value of absolute_url to enabled can enable absolute url in chart
  absolute_url: disabled

# Log configurations
log:
  # options are debug, info, warning, error, fatal
  level: info
  # configs for logs in local storage
  local:
    # Log files are rotated log_rotate_count times before being removed. If count is 0, old versions are removed rather than rotated.
    rotate_count: 50
    # Log files are rotated only if they grow bigger than log_rotate_size bytes. If size is followed by k, the size is assumed to be in kilobytes.
    # If the M is used, the size is in megabytes, and if G is used, the size is in gigabytes. So size 100, size 100k, size 100M and size 100G
    # are all valid.
    rotate_size: 200M
    # The directory on your host that store log
    location: /var/log/harbor

  # Uncomment following lines to enable external syslog endpoint.
  # external_endpoint:
  #   # protocol used to transmit log to external endpoint, options is tcp or udp
  #   protocol: tcp
  #   # The host of external endpoint
  #   host: localhost
  #   # Port of external endpoint
  #   port: 5140

#This attribute is for migrator to detect the version of the .cfg file, DO NOT MODIFY!
_version: 2.3.0

# 使用外部數據庫
# Uncomment external_database if using external database.
# external_database:
#   harbor:
#     host: harbor_db_host
#     port: harbor_db_port
#     db_name: harbor_db_name
#     username: harbor_db_username
#     password: harbor_db_password
#     ssl_mode: disable
#     max_idle_conns: 2
#     max_open_conns: 0
#   notary_signer:
#     host: notary_signer_db_host
#     port: notary_signer_db_port
#     db_name: notary_signer_db_name
#     username: notary_signer_db_username
#     password: notary_signer_db_password
#     ssl_mode: disable
#   notary_server:
#     host: notary_server_db_host
#     port: notary_server_db_port
#     db_name: notary_server_db_name
#     username: notary_server_db_username
#     password: notary_server_db_password
#     ssl_mode: disable

# 使用外部redis
# Uncomment external_redis if using external Redis server
# external_redis:
#   # support redis, redis+sentinel
#   # host for redis: <host_redis>:<port_redis>
#   # host for redis+sentinel:
#   #  <host_sentinel1>:<port_sentinel1>,<host_sentinel2>:<port_sentinel2>,<host_sentinel3>:<port_sentinel3>
#   host: redis:6379
#   password:
#   # sentinel_master_set must be set to support redis+sentinel
#   #sentinel_master_set:
#   # db_index 0 is for core, it's unchangeable
#   registry_db_index: 1
#   jobservice_db_index: 2
#   chartmuseum_db_index: 3
#   trivy_db_index: 5
#   idle_timeout_seconds: 30

# Uncomment uaa for trusting the certificate of uaa instance that is hosted via self-signed cert.
# uaa:
#   ca_file: /path/to/ca

# Global proxy
proxy:
  http_proxy:
  https_proxy:
  no_proxy:
  components:
    - core
    - jobservice
    - trivy

# 狀態接口
# metric:
#   enabled: false
#   port: 9090
#   path: /metrics