Harbor倉庫安裝使用cfssl工具生成證書

cfssl工具生成證書搭建Harbor倉庫

　　概要：此處記錄使用cfssl工具生成harbor私有證書，並使用證書搭建Harbor倉庫，此證書使用按照kubernetes時使用的ca證書來按照。

一、下載cfssl相關工具

curl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /usr/local/bin/cfssl
curl https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson
curl https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /usr/local/bin/cfssl-certinfo

二、創建ca證書請求文件ca-csr.json

{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "HangZhou",
      "L": "XS",
      "O": "k8s",
      "OU": "System"
    }
  ],
  "ca": {
    "expiry": "876000h"
  }
}

1、目錄結構如下：

[root@allinone tools]# tree /root/tools/
/root/tools/
├── bin
│   ├── cfssl
│   ├── cfssl-certinfo
│   └── cfssljson
└── ca-csr.json

[root@allinone tools]# ll
total 16 drwxr-xr-x. 2 root root 58 Mar 21 14:06 bin -rw-r--r--. 1 root root 997 Mar 21 14:07 ca-csr.json

三、生成ca證書

1、生成ca證書

[root@allinone tools]# ./bin/cfssl gencert -initca ca-csr.json | ./bin/cfssljson -bare ca
2022/03/21 14:07:43 [INFO] generating a new CA key and certificate from CSR
2022/03/21 14:07:43 [INFO] generate received request
2022/03/21 14:07:43 [INFO] received CSR
2022/03/21 14:07:43 [INFO] generating key: rsa-2048
2022/03/21 14:07:44 [INFO] encoded CSR
2022/03/21 14:07:44 [INFO] signed certificate with serial number 683494463359677094895320224613031266478694844972
[root@allinone tools]# ll
total 16
drwxr-xr-x. 2 root root   58 Mar 21 14:06 bin
-rw-r--r--. 1 root root  997 Mar 21 14:07 ca.csr
-rw-r--r--. 1 root root  243 Mar 21 14:05 ca-csr.json
-rw-------. 1 root root 1675 Mar 21 14:07 ca-key.pem
-rw-r--r--. 1 root root 1350 Mar 21 14:07 ca.pem
[root@allinone tools]# 

2、過期時間查看

[root@allinone tools]# openssl x509 -noout -text -in ca.pem|grep -A 5 Validity
        Validity
            Not Before: Mar 21 06:03:00 2022 GMT
            Not After : Feb 25 06:03:00 2122 GMT
        Subject: C=CN, ST=HangZhou, L=XS, O=k8s, OU=System, CN=kubernetes
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
[root@allinone tools]# 

四、創建Harbor證書

1、創建配置文件ca-config.json，告訴ca生成哪些信息的證書

[root@allinone tools]# cat ca-config.json 
{
  "signing": {
    "default": {
      "expiry": "438000h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "438000h"
      }
    }
  }
}

參數解析：

ca-config.json：可以定義多個 profiles，分別指定不同的參數；后續在簽名證書時使用某個profile；
signing：表示該證書可用於簽名其它證書；生成的 ca.pem 證書中 CA=TRUE；
server auth：表示client可以用該 CA 對server提供的證書進行驗證；
client auth：表示server可以用該CA對client提供的證書進行驗證；
profiles 中的 www 是后面cfssl gencert 命令值profiles 指定的值，要相互對應。

2、創建harbor證書簽名請求文件

[root@allinone tools]# cat harbor-csr.json 
{
  "CN": "harbor",
  "hosts": [
    "127.0.0.1",
    "192.168.158.158",  
    "harbor.myharbor.com"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "HangZhou",
      "L": "XS",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

3、生成harbor證書

[root@allinone tools]# ./bin/cfssl gencert \
> -ca=ca.pem \
> -ca-key=ca-key.pem \
> -config=ca-config.json \
> -profile=kubernetes harbor-csr.json | ./bin/cfssljson -bare harbor
2022/03/21 14:27:24 [INFO] generate received request
2022/03/21 14:27:24 [INFO] received CSR
2022/03/21 14:27:24 [INFO] generating key: rsa-2048
2022/03/21 14:27:24 [INFO] encoded CSR
2022/03/21 14:27:24 [INFO] signed certificate with serial number 101205033901552203007274366412176259589716406834
2022/03/21 14:27:24 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

[root@allinone tools]# ll
total 36
drwxr-xr-x. 2 root root   58 Mar 21 14:06 bin
-rw-r--r--. 1 root root  295 Mar 21 14:22 ca-config.json
-rw-r--r--. 1 root root  997 Mar 21 14:07 ca.csr
-rw-r--r--. 1 root root  243 Mar 21 14:05 ca-csr.json
-rw-------. 1 root root 1675 Mar 21 14:07 ca-key.pem
-rw-r--r--. 1 root root 1350 Mar 21 14:07 ca.pem
-rw-r--r--. 1 root root 1074 Mar 21 14:27 harbor.csr
-rw-r--r--. 1 root root  287 Mar 21 14:25 harbor-csr.json
-rw-------. 1 root root 1679 Mar 21 14:27 harbor-key.pem
-rw-r--r--. 1 root root 1440 Mar 21 14:27 harbor.pem

五、安裝Docker及Docker-compose

1、使用如下腳本安裝docker(二進制安裝)

#!/bin/bash

export DOCKER_VER=18.09.7

function install_docker() {
  # check if a container runtime is already installed
  systemctl status docker|grep Active|grep -q running && { echo "[WARN] docker is already running."; return 0; }
  systemctl status containerd|grep Active|grep -q running && { echo "[ERROR] containerd is running, unsupported."; exit 1; }
  
  if [[ "$REGISTRY_MIRROR" == CN ]];then
    DOCKER_URL="https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux/static/stable/x86_64/docker-${DOCKER_VER}.tgz"
  else
    DOCKER_URL="https://download.docker.com/linux/static/stable/x86_64/docker-${DOCKER_VER}.tgz"
  fi

  mkdir -p /opt/kube/bin /etc/docker /etc/ansible/down
  if [[ -f "/etc/ansible/down/docker-${DOCKER_VER}.tgz" ]];then
    echo "[INFO] docker binaries already existed"
  else
    echo -e "[INFO] \033[33mdownloading docker binaries\033[0m $DOCKER_VER"
    if [[ -e /usr/bin/curl ]];then
      curl -C- -O --retry 3 "$DOCKER_URL" || { echo "[ERROR] downloading docker failed"; exit 1; }
    else
      wget -c "$DOCKER_URL" || { echo "[ERROR] downloading docker failed"; exit 1; }
    fi
    mv ./docker-${DOCKER_VER}.tgz /etc/ansible/down
  fi

  tar zxf /etc/ansible/down/docker-${DOCKER_VER}.tgz -C /etc/ansible/down && \
  mv /etc/ansible/down/docker/* /opt/kube/bin && \
  ln -sf /opt/kube/bin/docker /bin/docker 

  echo "[INFO] generate docker service file"
  cat > /etc/systemd/system/docker.service << EOF
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.io
[Service]
Environment="PATH=/opt/kube/bin:/bin:/sbin:/usr/bin:/usr/sbin"
ExecStart=/opt/kube/bin/dockerd
ExecStartPost=/sbin/iptables -I FORWARD -s 0.0.0.0/0 -j ACCEPT
ExecReload=/bin/kill -s HUP \$MAINPID
Restart=on-failure
RestartSec=5
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
Delegate=yes
KillMode=process
[Install]
WantedBy=multi-user.target
EOF

  # configuration for dockerd
  echo "[INFO] generate docker config file"
  if [[ "$REGISTRY_MIRROR" == CN ]];then
    echo "[INFO] prepare register mirror for $REGISTRY_MIRROR"
    cat > /etc/docker/daemon.json << EOF
{
  "registry-mirrors": [
    "https://dockerhub.azk8s.cn",
    "https://docker.mirrors.ustc.edu.cn",
    "http://hub-mirror.c.163.com"
  ],
  "max-concurrent-downloads": 10,
  "log-driver": "json-file",
  "log-level": "warn",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
    },
  "data-root": "/var/lib/docker"
}
EOF
  else
    echo "[INFO] standard config without registry mirrors"
    cat > /etc/docker/daemon.json << EOF
{
  "max-concurrent-downloads": 10,
  "log-driver": "json-file",
  "log-level": "warn",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
    },
  "data-root": "/var/lib/docker"
}
EOF
  fi

  if [[ -e /etc/centos-release || -e /etc/redhat-release ]]; then
    echo "[INFO] turn off selinux in CentOS/Redhat"
    setenforce 0
    echo "SELINUX=disabled" > /etc/selinux/config
  fi

  echo "[INFO] enable and start docker"
  systemctl enable docker
  systemctl daemon-reload && systemctl restart docker && sleep 8
}

install_docker

2、下載docker-compose

curl -L https://get.daocloud.io/docker/compose/releases/download/1.25.5/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose

3、下載harbor，此處為1.8.6

[root@allinone ~]# tar -xf harbor-offline-installer-v1.8.6.tgz  -C /data/

[root@allinone ~]# cd /data/harbor/

[root@allinone harbor]# docker load -i harbor.v1.8.6.tar.gz

[root@allinone harbor]# vim harbor.yml    #部分配置，其他為默認配置

hostname: 192.168.158.158
http:
  port: 80

https:
  port: 443
  certificate: /root/tools/harbor.pem
  private_key: /root/tools/harbor-key.pem

4、安裝harbor

[root@allinone harbor]# ./install.sh      ##注意，此處安裝步驟需在上面配置文件修改之后，安裝完成之后在修改添加https訪問證書路徑經驗證會無效。

5、登錄驗證

[root@allinone harbor]# docker login 192.168.158.158
Username: admin
Password: 
Error response from daemon: Get https://192.168.158.158/v2/: x509: certificate signed by unknown authority

　　此時出現了報錯，我們需要注意一下2點：

　　a）docker login 的地址應該為我們harbor證書簽名請求文件中host字段的其中一個，不在其中會報錯

　　b）自簽證書，登錄需要使用到私鑰，而登錄是默認回去/etc/docker/certs.d/host路徑下找

1)創建證書路徑

[root@allinone certs.d]# mkdir -p /etc/docker/certs.d/192.168.158.158    #同證書簽名請求文件

[root@allinone certs.d]# mkdir -p /etc/docker/certs.d/harbor.myharbor.com #同證書簽名請求文件

2）拷貝ca證書

[root@allinone certs.d]# cp /root/tools/ca.pem    /etc/docker/certs.d/192.168.158.158/ca.crt

[root@allinone certs.d]# cp /root/tools/ca.pem    /etc/docker/certs.d/harbor.myharbor.com/ca.crt

3）再次登錄

[root@allinone certs.d]# docker login 192.168.158.158
Username: admin
Password: 
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
[root@allinone certs.d]# docker login harbor.myharbor.com   #需配置解析 /etc/hosts中即可
Username: admin
Password: 
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

六、構建鏡像，推送至harbor倉庫

1、通常我們使用時會按照項目創建一個私有庫，不公開

 

 

 2、創建一個用戶，並設置為項目管理員

 

 

 3、使用myadmin用戶登錄

 

 

 4、k8s中創建pod拉取鏡像時，私有倉庫未公開的需要指定憑據secrets才可以拉取

kubectl create secret docker-registry harbor.myharbor --docker-server=harbor.myharbor.com --docker-username=myadmin --docker-password=Myadmin12345 --docker-email=[e-mail]

　　harbor.myharbor: 為拉取鏡像私鑰名

　　harbor.myharbor.com：  為登錄harbor地址

　　--docker-username=myadimin   登錄用戶

　　--docker-password=Myadmin12345  登錄密碼

　　--docker-email=[e-mail]    郵箱          

以上信息須與創建用戶時信息一致。