openshift 4.3 靜態IP離線 baremetal 安裝，包含operator hub

本文轉載自查看原文 2020-04-17 21:32 675

https://github.com/wangzheng422/docker_env/blob/master/redhat/ocp4/4.3/4.3.disconnect.operator.md

openshift 4.3 靜態IP離線 baremetal 安裝，包含operator hub

本文描述ocp4.3在baremental(kvm模擬)上面，靜態ip安裝的方法。包括operator hub步驟。

以下是本次使用的虛擬機配置：
https://docs.google.com/spreadsheets/d/1dwgh72r4ZfxEIFV82e_qe0T4w3CpENG_HqSejB3IUQs/edit#gid=0

本次實驗的簡要說明： https://docs.google.com/presentation/d/1_iy_YWnr6YX1tkwhw46ttCMkEoyPcHXvZvGQBDqJMyI/edit

以下是參考材料

https://blog.openshift.com/openshift-4-2-disconnected-install/

https://blog.openshift.com/openshift-4-bare-metal-install-quickstart/

https://github.com/christianh814/ocp4-upi-helpernode#ocp4-upi-helper-node-playbook

https://github.com/openshift/cluster-samples-operator/blob/master/manifests/image-references

https://github.com/e-minguez/ocp4-upi-bm-pxeless-staticips/blob/master/docs/12-post-installation.md

https://www.openshift.com/blog/deploying-a-upi-environment-for-openshift-4-1-on-vms-and-bare-metal

離線安裝包下載

ocp4.3的離線安裝包下載和3.11不太一樣，按照如下方式准備。另外，由於默認的baremental是需要dhcp, pxe環境的，那么需要准備一個工具機，上面有dhcp, tftp, haproxy等工具，另外為了方便項目現場工作，還准備了ignition文件的修改工具，所以離線安裝包需要一些其他第三方的工具。

https://github.com/wangzheng422/ocp4-upi-helpernode 這個工具，是創建工具機用的。

https://github.com/wangzheng422/filetranspiler 這個工具，是修改ignition文件用的。

打包好的安裝包，在這里下載(internal access only)，目前最新版本是ocp 4.3.5： https://drive.google.com/drive/u/2/folders/1Cu3qg2m4jn9psZxWHFTJFdH6ZW3O2vjR

其中包括如下類型的文件：

ocp4.tgz 這個文件包含了iso等安裝介質，以及各種安裝腳本，全部下載的鏡像列表等。需要復制到宿主機，以及工具機上去。，目前打包的內容，包括了4.3.1, 4.3.5 .
registry.tgz 這個文件也是docker image registry的倉庫打包文件，按照CCN教材編制（https://github.com/wangzheng422/docker_env/blob/master/redhat/ocp4/4.2.ccn.devops.deploy.md）, 以后計划再加入kafka，etcd，maridb，servicemesh，集群日志operator所用的鏡像，所以比較小。但是要運行其他operator的話，需要先補充鏡像。按照這里操作： https://github.com/wangzheng422/docker_env/blob/master/redhat/ocp4/4.3/4.3.add.image.md .
rhel-data.tgz 這個文件是 rhel 7 主機的yum更新源，這么大是因為里面有gpu, epel等其他的東西。這個包主要用於安裝宿主機，工具機，以及作為計算節點的rhel。

在外網雲主機上面准備離線安裝源

https://github.com/wangzheng422/docker_env/blob/master/redhat/ocp4/4.3/4.3.build.dist.md

# on vultr

rm -rf /data/ocp4
mkdir -p /data/ocp4
cd /data/ocp4

wget -O build.dist.sh https://raw.githubusercontent.com/wangzheng422/docker_env/dev/redhat/ocp4/4.3/scripts/build.dist.sh

yum -y install podman docker-distribution pigz skopeo docker buildah jq python3-pip python34

pip3 install yq

# https://blog.csdn.net/ffzhihua/article/details/85237411
wget http://mirror.centos.org/centos/7/os/x86_64/Packages/python-rhsm-certificates-1.19.10-1.el7_4.x86_64.rpm
rpm2cpio python-rhsm-certificates-1.19.10-1.el7_4.x86_64.rpm | cpio -iv --to-stdout ./etc/rhsm/ca/redhat-uep.pem | tee /etc/rhsm/ca/redhat-uep.pem

systemctl start docker

docker login -u ****** -p ******** registry.redhat.io
docker login -u ****** -p ******** registry.access.redhat.com
docker login -u ****** -p ******** registry.connect.redhat.com

podman login -u ****** -p ******** registry.redhat.io
podman login -u ****** -p ******** registry.access.redhat.com
podman login -u ****** -p ******** registry.connect.redhat.com

# to download the pull-secret.json, open following link
# https://cloud.redhat.com/openshift/install/metal/user-provisioned
cat << 'EOF' > /data/pull-secret.json
{"auths":{"cloud.openshift.com":*********************
EOF

bash build.dist.sh

# rm -f /data/ocp4/pull-secret.json

output of mirror of images

Success
Update image:  registry.redhat.ren:5443/ocp4/openshift4:4.3.3
Mirror prefix: registry.redhat.ren:5443/ocp4/openshift4

To use the new mirrored repository to install, add the following section to the install-config.yaml:

imageContentSources:
- mirrors:
  - registry.redhat.ren:5443/ocp4/openshift4
  source: quay.io/openshift-release-dev/ocp-release
- mirrors:
  - registry.redhat.ren:5443/ocp4/openshift4
  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev


To use the new mirrored repository for upgrades, use the following to create an ImageContentSourcePolicy:

apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
  name: example
spec:
  repositoryDigestMirrors:
  - mirrors:
    - registry.redhat.ren:5443/ocp4/openshift4
    source: quay.io/openshift-release-dev/ocp-release
  - mirrors:
    - registry.redhat.ren:5443/ocp4/openshift4
    source: quay.io/openshift-release-dev/ocp-v4.0-art-dev

宿主機准備

本次實驗，是在一個32C， 256G 的主機上面，用很多個虛擬機安裝測試。所以先准備這個宿主機。

hostnamectl set-hostname base.redhat.ren

cat << EOF >>  /etc/hosts
172.29.122.232 yum.redhat.ren
EOF

# 准備yum更新源
mkdir /etc/yum.repos.d.bak
mv /etc/yum.repos.d/* /etc/yum.repos.d.bak
cat << EOF > /etc/yum.repos.d/remote.repo
[remote]
name=RHEL FTP
baseurl=ftp://yum.redhat.ren/data
enabled=1
gpgcheck=0

EOF

yum clean all
yum repolist

yum -y install byobu htop 

# 配置dns服務
yum -y install dnsmasq

cat  > /etc/dnsmasq.d/openshift-cluster.conf << EOF
local=/redhat.ren/
address=/yum.redhat.ren/172.29.122.233
address=/registry.redhat.ren/192.168.7.1
EOF

systemctl restart dnsmasq.service && systemctl enable dnsmasq.service && systemctl status dnsmasq.service

firewall-cmd --permanent --add-service=dns
firewall-cmd --reload

systemctl restart dnsmasq

# on kvm host
# cat << EOF >>  /etc/hosts
# 127.0.0.1 vm.redhat.ren
# EOF

# 配置registry
mkdir /etc/crts/ && cd /etc/crts
openssl req \
   -newkey rsa:2048 -nodes -keyout redhat.ren.key \
   -x509 -days 3650 -out redhat.ren.crt -subj \
   "/C=CN/ST=GD/L=SZ/O=Global Security/OU=IT Department/CN=*.redhat.ren"

cp /etc/crts/redhat.ren.crt /etc/pki/ca-trust/source/anchors/
update-ca-trust extract

cd /data
# tar zxf registry.tgz
yum -y install podman docker-distribution pigz skopeo
pigz -dc registry.tgz | tar xf -
cat << EOF > /etc/docker-distribution/registry/config.yml
version: 0.1
log:
  fields:
    service: registry
storage:
    cache:
        layerinfo: inmemory
    filesystem:
        rootdirectory: /data/registry
    delete:
        enabled: true
http:
    addr: :5443
    tls:
       certificate: /etc/crts/redhat.ren.crt
       key: /etc/crts/redhat.ren.key
EOF
# systemctl restart docker
systemctl stop docker-distribution
systemctl enable docker-distribution
systemctl restart docker-distribution
# podman login registry.redhat.ren -u a -p a

firewall-cmd --permanent --add-port=5443/tcp
firewall-cmd --reload

cat /etc/crts/redhat.ren.crt


# https://github.com/christianh814/ocp4-upi-helpernode/blob/master/docs/quickstart.md

# 准備vnc環境

yum -y install tigervnc-server tigervnc gnome-terminal gnome-session gnome-classic-session gnome-terminal nautilus-open-terminal control-center liberation-mono-fonts google-noto-sans-cjk-fonts google-noto-sans-fonts fonts-tweak-tool

yum install -y    qgnomeplatform   xdg-desktop-portal-gtk   NetworkManager-libreswan-gnome   PackageKit-command-not-found   PackageKit-gtk3-module   abrt-desktop   at-spi2-atk   at-spi2-core   avahi   baobab   caribou   caribou-gtk2-module   caribou-gtk3-module   cheese   compat-cheese314   control-center   dconf   empathy   eog   evince   evince-nautilus   file-roller   file-roller-nautilus   firewall-config   firstboot   fprintd-pam   gdm   gedit   glib-networking   gnome-bluetooth   gnome-boxes   gnome-calculator   gnome-classic-session   gnome-clocks   gnome-color-manager   gnome-contacts   gnome-dictionary   gnome-disk-utility   gnome-font-viewer   gnome-getting-started-docs   gnome-icon-theme   gnome-icon-theme-extras   gnome-icon-theme-symbolic   gnome-initial-setup   gnome-packagekit   gnome-packagekit-updater   gnome-screenshot   gnome-session   gnome-session-xsession   gnome-settings-daemon   gnome-shell   gnome-software   gnome-system-log   gnome-system-monitor   gnome-terminal   gnome-terminal-nautilus   gnome-themes-standard   gnome-tweak-tool   nm-connection-editor   orca   redhat-access-gui   sane-backends-drivers-scanners   seahorse   setroubleshoot   sushi   totem   totem-nautilus   vinagre   vino   xdg-user-dirs-gtk   yelp

yum install -y    cjkuni-uming-fonts   dejavu-sans-fonts   dejavu-sans-mono-fonts   dejavu-serif-fonts   gnu-free-mono-fonts   gnu-free-sans-fonts   gnu-free-serif-fonts   google-crosextra-caladea-fonts   google-crosextra-carlito-fonts   google-noto-emoji-fonts   jomolhari-fonts   khmeros-base-fonts   liberation-mono-fonts   liberation-sans-fonts   liberation-serif-fonts   lklug-fonts   lohit-assamese-fonts   lohit-bengali-fonts   lohit-devanagari-fonts   lohit-gujarati-fonts   lohit-kannada-fonts   lohit-malayalam-fonts   lohit-marathi-fonts   lohit-nepali-fonts   lohit-oriya-fonts   lohit-punjabi-fonts   lohit-tamil-fonts   lohit-telugu-fonts   madan-fonts   nhn-nanum-gothic-fonts   open-sans-fonts   overpass-fonts   paktype-naskh-basic-fonts   paratype-pt-sans-fonts   sil-abyssinica-fonts   sil-nuosu-fonts   sil-padauk-fonts   smc-meera-fonts   stix-fonts   thai-scalable-waree-fonts   ucs-miscfixed-fonts   vlgothic-fonts   wqy-microhei-fonts   wqy-zenhei-fonts

vncpasswd

cat << EOF > ~/.vnc/xstartup
#!/bin/sh
unset SESSION_MANAGER
unset DBUS_SESSION_BUS_ADDRESS
gnome-session &
EOF
chmod +x ~/.vnc/xstartup

vncserver :1 -geometry 1280x800
# 如果你想停掉vnc server，這么做
vncserver -kill :1

firewall-cmd --permanent --add-port=6001/tcp
firewall-cmd --permanent --add-port=5901/tcp
firewall-cmd --reload

# connect vnc at port 5901
# export DISPLAY=:1

# https://www.cyberciti.biz/faq/how-to-install-kvm-on-centos-7-rhel-7-headless-server/

# 配置kvm環境
yum -y install qemu-kvm libvirt libvirt-python libguestfs-tools virt-install virt-viewer virt-manager

systemctl enable libvirtd
systemctl start libvirtd

lsmod | grep -i kvm
brctl show
virsh net-list
virsh net-dumpxml default

# 創建實驗用虛擬網絡

cat << EOF >  /data/virt-net.xml
<network>
  <name>openshift4</name>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='openshift4' stp='on' delay='0'/>
  <domain name='openshift4'/>
  <ip address='192.168.7.1' netmask='255.255.255.0'>
  </ip>
</network>
EOF

virsh net-define --file virt-net.xml
virsh net-autostart openshift4
virsh net-start openshift4

# 創建工具機

mkdir -p /data/kvm

virt-install --name="ocp4-aHelper" --vcpus=2 --ram=4096 \
--disk path=/data/kvm/ocp4-aHelper.qcow2,bus=virtio,size=230 \
--os-variant centos7.0 --network network=openshift4,model=virtio \
--boot menu=on --location /data/rhel-server-7.6-x86_64-dvd.iso \
--initrd-inject helper-ks.cfg --extra-args "inst.ks=file:/helper-ks.cfg" 

virt-viewer --domain-name ocp4-aHelper
virsh start ocp4-aHelper
virsh list --all

yum -y install haproxy
# scp haproxy.cfg to /data/ocp4/haproxy
/bin/cp -f /data/ocp4/haproxy.cfg /etc/haproxy/haproxy.cfg
firewall-cmd --permanent --add-port=9001/tcp
firewall-cmd --permanent --add-port=80/tcp
firewall-cmd --permanent --add-port=443/tcp
firewall-cmd --permanent --add-port=10080/tcp
firewall-cmd --reload
systemctl restart haproxy

工具機准備

以下是在工具機里面，進行的安裝操作。


# in helper node
mkdir /etc/yum.repos.d.bak
mv /etc/yum.repos.d/* /etc/yum.repos.d.bak/
cat << EOF > /etc/yum.repos.d/remote.repo
[remote]
name=RHEL FTP
baseurl=ftp://yum.redhat.ren/data
enabled=1
gpgcheck=0

EOF

yum clean all
yum repolist

yum -y install ansible-2.8.10 git unzip podman python36
# scp ocp4.tgz to /root
cd /root
tar zvxf ocp4.tgz
cd /root/ocp4
unzip ocp4-upi-helpernode-master.zip
# podman load -i fedora.tgz
podman load -i filetranspiler.tgz
# 根據現場環境，修改 ocp4-upi-helpernode-master/vars-static.yaml
cd ocp4-upi-helpernode-master
ansible-playbook -e @vars-static.yaml -e staticips=true tasks/main.yml

# try this:
/usr/local/bin/helpernodecheck

# 定制ignition

# on helper node
cd /root/ocp4
mkdir -p /data
# # export BUILDNUMBER=$(cat release.txt | grep 'Name:' | awk '{print $NF}')
# export BUILDNUMBER=4.2.10
# echo ${BUILDNUMBER}
# # export BUILDNUMBER=4.2.4
# export OCP_RELEASE=${BUILDNUMBER}
# export LOCAL_REG='registry.redhat.ren:5443'
# export LOCAL_REPO='ocp4/openshift4'
# export UPSTREAM_REPO='openshift-release-dev'
# export LOCAL_SECRET_JSON="/data/pull-secret.json"
# export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=${LOCAL_REG}/${LOCAL_REPO}:${OCP_RELEASE}
# export RELEASE_NAME="ocp-release"

cat << EOF > /root/.ssh/config
StrictHostKeyChecking no
UserKnownHostsFile=/dev/null
EOF

cd /root/ocp4
# scp ocp4.tgz to /root
# scp install-config.yaml to /root/ocp4
# 根據現場環境，修改 install-config.yaml
# 至少要修改ssh key， 還有 additionalTrustBundle，這個是鏡像倉庫的csr 
# example file at files/4.2/kvm/

/bin/rm -rf *.ign .openshift_install_state.json auth bootstrap master0 master1 master2 worker0 worker1 worker2

openshift-install create ignition-configs --dir=/root/ocp4

# 以下操作本來是想設置網卡地址，但是實踐發現是不需要的。
# 保留在這里，是因為他可以在安裝的時候注入文件，非常有用。
# mkdir -p bootstrap/etc/sysconfig/network-scripts/
# cat <<EOF > bootstrap/etc/sysconfig/network-scripts/ifcfg-ens3
# DEVICE=ens3
# BOOTPROTO=none
# ONBOOT=yes
# IPADDR=192.168.7.12
# NETMASK=255.255.255.0
# GATEWAY=192.168.7.1
# DNS=192.168.7.11
# DNS1=192.168.7.11
# DNS2=192.168.7.1
# DOMAIN=redhat.ren
# PREFIX=24
# DEFROUTE=yes
# IPV6INIT=no
# EOF
# filetranspiler -i bootstrap.ign -f bootstrap -o bootstrap-static.ign
# /bin/cp -f bootstrap-static.ign /var/www/html/ignition/

/bin/cp -f bootstrap.ign /var/www/html/ignition/bootstrap-static.ign
/bin/cp -f master.ign /var/www/html/ignition/master-0.ign
/bin/cp -f master.ign /var/www/html/ignition/master-1.ign
/bin/cp -f master.ign /var/www/html/ignition/master-2.ign
/bin/cp -f worker.ign /var/www/html/ignition/worker-0.ign
/bin/cp -f worker.ign /var/www/html/ignition/worker-1.ign
/bin/cp -f worker.ign /var/www/html/ignition/worker-2.ign

chmod 644 /var/www/html/ignition/*

回到宿主機

本來，到了這一步，就可以開始安裝了，但是我們知道coreos裝的時候，要手動輸入很長的命令行，實際操作的時候，那是不可能輸入對的，輸入錯一個字符，安裝就失敗，要重啟，重新輸入。。。

為了避免這種繁瑣的操作，參考網上的做法，我們就需要為每個主機定制iso了。

這里面有一個坑，我們是不知道主機的網卡名稱的，只能先用coreos iso安裝啟動一次，進入單用戶模式以后，ip a 來查看以下，才能知道，一般來說，是ens3。

# on kvm host

yum -y install genisoimage libguestfs-tools
systemctl start libvirtd

export NGINX_DIRECTORY=/data/ocp4
export RHCOSVERSION=4.3.0
export VOLID=$(isoinfo -d -i ${NGINX_DIRECTORY}/rhcos-${RHCOSVERSION}-x86_64-installer.iso | awk '/Volume id/ { print $3 }')
TEMPDIR=$(mktemp -d)
echo $VOLID
echo $TEMPDIR

cd ${TEMPDIR}
# Extract the ISO content using guestfish (to avoid sudo mount)
guestfish -a ${NGINX_DIRECTORY}/rhcos-${RHCOSVERSION}-x86_64-installer.iso \
  -m /dev/sda tar-out / - | tar xvf -

# Helper function to modify the config files
modify_cfg(){
  for file in "EFI/redhat/grub.cfg" "isolinux/isolinux.cfg"; do
    # Append the proper image and ignition urls
    sed -e '/coreos.inst=yes/s|$| coreos.inst.install_dev=vda coreos.inst.image_url='"${URL}"'\/install\/'"${BIOSMODE}"'.raw.gz coreos.inst.ignition_url='"${URL}"'\/ignition\/'"${NODE}"'.ign ip='"${IP}"'::'"${GATEWAY}"':'"${NETMASK}"':'"${FQDN}"':'"${NET_INTERFACE}"':none:'"${DNS}"' nameserver='"${DNS}"'|' ${file} > $(pwd)/${NODE}_${file##*/}
    # Boot directly in the installation
    sed -i -e 's/default vesamenu.c32/default linux/g' -e 's/timeout 600/timeout 10/g' $(pwd)/${NODE}_${file##*/}
  done
}

URL="http://192.168.7.11:8080/"
GATEWAY="192.168.7.1"
NETMASK="255.255.255.0"
DNS="192.168.7.11"

# BOOTSTRAP
# TYPE="bootstrap"
NODE="bootstrap-static"
IP="192.168.7.12"
FQDN="bootstrap"
BIOSMODE="bios"
NET_INTERFACE="ens3"
modify_cfg

# MASTERS
# TYPE="master"
# MASTER-0
NODE="master-0"
IP="192.168.7.13"
FQDN="master-0"
BIOSMODE="bios"
NET_INTERFACE="ens3"
modify_cfg

# MASTER-1
NODE="master-1"
IP="192.168.7.14"
FQDN="master-1"
BIOSMODE="bios"
NET_INTERFACE="ens3"
modify_cfg

# MASTER-2
NODE="master-2"
IP="192.168.7.15"
FQDN="master-2"
BIOSMODE="bios"
NET_INTERFACE="ens3"
modify_cfg

# WORKERS
NODE="worker-0"
IP="192.168.7.16"
FQDN="worker-0"
BIOSMODE="bios"
NET_INTERFACE="ens3"
modify_cfg

NODE="worker-1"
IP="192.168.7.17"
FQDN="worker-1"
BIOSMODE="bios"
NET_INTERFACE="ens3"
modify_cfg

NODE="worker-2"
IP="192.168.7.18"
FQDN="worker-2"
BIOSMODE="bios"
NET_INTERFACE="ens3"
modify_cfg

# Generate the images, one per node as the IP configuration is different...
# https://github.com/coreos/coreos-assembler/blob/master/src/cmd-buildextend-installer#L97-L103
for node in master-0 master-1 master-2 worker-0 worker-1 worker-2 bootstrap-static; do
  # Overwrite the grub.cfg and isolinux.cfg files for each node type
  for file in "EFI/redhat/grub.cfg" "isolinux/isolinux.cfg"; do
    /bin/cp -f $(pwd)/${node}_${file##*/} ${file}
  done
  # As regular user!
  genisoimage -verbose -rock -J -joliet-long -volset ${VOLID} \
    -eltorito-boot isolinux/isolinux.bin -eltorito-catalog isolinux/boot.cat \
    -no-emul-boot -boot-load-size 4 -boot-info-table \
    -eltorito-alt-boot -efi-boot images/efiboot.img -no-emul-boot \
    -o ${NGINX_DIRECTORY}/${node}.iso .
done

# Optionally, clean up
cd
rm -Rf ${TEMPDIR}

cd ${NGINX_DIRECTORY}

# finally, we can start install :)
# 你可以一口氣把虛擬機都創建了，然后喝咖啡等着。
# 從這一步開始，到安裝完畢，大概30分鍾。
virt-install --name=ocp4-bootstrap --vcpus=4 --ram=8192 \
--disk path=/data/kvm/ocp4-bootstrap.qcow2,bus=virtio,size=120 \
--os-variant rhel8.0 --network network=openshift4,model=virtio \
--boot menu=on --cdrom ${NGINX_DIRECTORY}/bootstrap-static.iso   

# 想登錄進coreos一探究竟？那么這么做
# ssh core@192.168.7.12 
# journalctl -b -f -u bootkube.service

virt-install --name=ocp4-master0 --vcpus=4 --ram=32768 \
--disk path=/data/kvm/ocp4-master0.qcow2,bus=virtio,size=120 \
--os-variant rhel8.0 --network network=openshift4,model=virtio \
--boot menu=on --cdrom ${NGINX_DIRECTORY}/master-0.iso 

# ssh core@192.168.7.13

virt-install --name=ocp4-master1 --vcpus=4 --ram=32768 \
--disk path=/data/kvm/ocp4-master1.qcow2,bus=virtio,size=120 \
--os-variant rhel8.0 --network network=openshift4,model=virtio \
--boot menu=on --cdrom ${NGINX_DIRECTORY}/master-1.iso 

virt-install --name=ocp4-master2 --vcpus=4 --ram=32768 \
--disk path=/data/kvm/ocp4-master2.qcow2,bus=virtio,size=120 \
--os-variant rhel8.0 --network network=openshift4,model=virtio \
--boot menu=on --cdrom ${NGINX_DIRECTORY}/master-2.iso 

virt-install --name=ocp4-worker0 --vcpus=4 --ram=32768 \
--disk path=/data/kvm/ocp4-worker0.qcow2,bus=virtio,size=120 \
--os-variant rhel8.0 --network network=openshift4,model=virtio \
--boot menu=on --cdrom ${NGINX_DIRECTORY}/worker-0.iso 

virt-install --name=ocp4-worker1 --vcpus=4 --ram=32768 \
--disk path=/data/kvm/ocp4-worker1.qcow2,bus=virtio,size=120 \
--os-variant rhel8.0 --network network=openshift4,model=virtio \
--boot menu=on --cdrom ${NGINX_DIRECTORY}/worker-1.iso 

virt-install --name=ocp4-worker2 --vcpus=4 --ram=32768 \
--disk path=/data/kvm/ocp4-worker2.qcow2,bus=virtio,size=120 \
--os-variant rhel8.0 --network network=openshift4,model=virtio \
--boot menu=on --cdrom ${NGINX_DIRECTORY}/worker-2.iso 

# on workstation
# open http://192.168.7.11:9000/
# to check

# if you want to stop or delete vm, try this
# virsh list --all
# virsh stop ***
# virsh destroy ***
# virsh undefine ***

打開瀏覽器，能看到一個監控頁面，我們就能大致了解安裝的進度。

等了一段時間以后，監控上看大概是這樣

在工具機上面

在bootstrap和裝master階段，用這個命令看進度。

openshift-install wait-for bootstrap-complete --log-level debug

一切正常的話，會看到這個。

有時候證書會過期，驗證方法是登錄 bootstrap, 看看過期時間。如果確定過期，要清除所有的openshift-install生成配置文件的緩存，重新來過。

echo | openssl s_client -connect localhost:6443 | openssl x509 -noout -text | grep Not

一般來說，如果在openshift-install這一步之前，按照文檔，刪除了緩存文件，就不會出現過期的現象。

cd ~/ocp4
export KUBECONFIG=/root/ocp4/auth/kubeconfig
echo "export KUBECONFIG=/root/ocp4/auth/kubeconfig" >> ~/.bashrc
oc get nodes

這個時候，只能看到master，是因為worker的csr沒有批准。如果虛擬機是一口氣創建的，那么多半不會遇到下面的問題。

oc get csr

會發現有很多沒有被批准的

批准之

yum -y install jq
oc get csr -ojson | jq -r '.items[] | select(.status == {} ) | .metadata.name' | xargs oc adm certificate approve

然后worker 節點cpu飆高，之后就能看到worker了。

等一會，會看到這個，就對了。

上面的操作完成以后，就可以完成最后的安裝了

openshift-install wait-for install-complete --log-level debug
# here is the output
# INFO Waiting up to 30m0s for the cluster at https://api.ocp4.redhat.ren:6443 to initialize...
# INFO Waiting up to 10m0s for the openshift-console route to be created...
# INFO Install complete!
# INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/root/ocp4/auth/kubeconfig'
# INFO Access the OpenShift web-console here: https://console-openshift-console.apps.ocp4.redhat.ren
# INFO Login to the console with user: kubeadmin, password: vo382-ajzAs-qaEXY-Tr2g2

我們的工具機是帶nfs的，那么就配置高檔一些的nfs存儲吧，不要用emptydir

bash ocp4-upi-helpernode-master/files/nfs-provisioner-setup.sh

# oc edit configs.imageregistry.operator.openshift.io
# 修改 storage 部分
# storage:
#   pvc:
#     claim:
oc patch configs.imageregistry.operator.openshift.io cluster -p '{"spec":{"managementState": "Managed","storage":{"pvc":{"claim":""}}}}' --type=merge

oc patch configs.imageregistry.operator.openshift.io cluster -p '{"spec":{"managementState": "Removed"}}' --type=merge

oc get clusteroperator image-registry

oc get configs.imageregistry.operator.openshift.io cluster -o yaml

oc get configs.samples.operator.openshift.io/cluster -o yaml

oc patch configs.samples.operator.openshift.io/cluster -p '{"spec":{"managementState": "Managed"}}' --type=merge

oc patch configs.samples.operator.openshift.io/cluster -p '{"spec":{"managementState": "Unmanaged"}}' --type=merge

oc patch configs.samples.operator.openshift.io/cluster -p '{"spec":{"managementState": "Removed"}}' --type=merge

配置一下本地的dns ( 把 *.apps.ocp4.redhat.ren 配置成 192.168.7.11 ) ，指向工具機的haproxy，打開瀏覽器就能訪問管理界面了

Operator Hub 離線安裝

https://docs.openshift.com/container-platform/4.2/operators/olm-restricted-networks.html

https://github.com/operator-framework/operator-registry

https://www.cnblogs.com/ericnie/p/11777384.html?from=timeline&isappinstalled=0

https://access.redhat.com/documentation/en-us/openshift_container_platform/4.2/html-single/images/index

operator hub 准備分2個層次，一個是本文章描述的，制作operator hub的離線資源，並鏡像operator 鏡像。做到這一步，能夠在離線部署的ocp4.2上，看到operator hub，並且能夠部署operator。但是如果要用operator來部署要用的組件，那么operator會再去下載鏡像，這個層次的鏡像，也需要離線部署，但是由於每個operator需要的鏡像都不一樣，也沒有統一的地方進行描述，所以需要各個項目現場，根據需要另外部署，本項目會盡量多的下載需要的鏡像，但是目前無法避免遺漏。

# on helper node， 在工具機上
cd /root/ocp4

# scp /etc/crts/redhat.ren.crt 192.168.7.11:/root/ocp4/
oc project openshift-config
oc create configmap ca.for.registry \
    --from-file=registry.redhat.ren=/root/ocp4/redhat.ren.crt
# 如果你想刪除這個config map，這么做
# oc delete configmap ca.for.registry
oc patch image.config.openshift.io/cluster -p '{"spec":{"additionalTrustedCA":{"name":"ca.for.registry"}}}'  --type=merge
# oc patch image.config.openshift.io/cluster -p '{"spec":{"registrySources":{"insecureRegistries":["registry.redhat.ren"]}}}'  --type=merge
oc get image.config.openshift.io/cluster -o yaml

# 以下這個步驟是官網文檔要做的，實踐中發現，disconnected環境不需要
# oc patch OperatorHub cluster --type json -p '[{"op": "add", "path": "/spec/disableAllDefaultSources", "value": true}]'
# 如果你不小心還是照着官網做了，用如下步驟刪掉
# oc patch OperatorHub cluster --type json  -p '[{"op": "remove", "path": "/spec/disableAllDefaultSources"}]'

oc patch OperatorHub cluster --type json \
    -p '[{"op": "add", "path": "/spec/disableAllDefaultSources", "value": true}]'

oc get OperatorHub cluster -o yaml

# yum -y install python36
# 根據項目現場情況，修改以下腳本參數后運行
# bash image.registries.conf.sh yaml.image.ok.list.uniq

# 由於某些ocp 4.2的更新機制，以下操作會觸發集群更新，
# 集群節點會逐個重啟，集群組件也會逐個重啟，請等待集群重啟完畢。
oc apply -f ./99-worker-zzz-container-registries.yaml -n openshift-config
oc apply -f ./99-master-zzz-container-registries.yaml -n openshift-config

# ！！！正常情況，以下操作不需要！！！
# 以下操作，刪除mirror鏡像信息，也會觸發集群更新操作，請等待集群重啟完畢
oc delete -f ./99-worker-zzz-container-registries.yaml -n openshift-config
oc delete -f ./99-master-zzz-container-registries.yaml -n openshift-config

watch oc get machineconfigpools

watch oc get node

從監控界面，能看到節點在升級，重啟。


# on helper node

# 如果想看到redhat的operator，這樣做
# 鏡像源在 docker.io/wangzheng422/custom-registry-redhat
cat <<EOF > redhat-operator-catalog.yaml
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: redhat-operator-catalog
  namespace: openshift-marketplace
spec:
  displayName: Redhat Operator Catalog
  sourceType: grpc
  image: docker.io/wangzheng422/operator-catalog:redhat-2020-03-23
  publisher: Red Hat
EOF
oc create -f redhat-operator-catalog.yaml

# 如果想看到certified的operator，這樣做
# 鏡像源在 docker.io/wangzheng422/custom-registry-certified
cat <<EOF > certified-operator-catalog.yaml
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: certified-operator-catalog
  namespace: openshift-marketplace
spec:
  displayName: Certified Operator Catalog
  sourceType: grpc
  image: docker.io/wangzheng422/operator-catalog:certified-2020-03-23
  publisher: Certified
EOF
oc create -f certified-operator-catalog.yaml


# 如果想看到community的operator，這樣做
# 鏡像源在 docker.io/wangzheng422/custom-registry-community
cat <<EOF > community-operator-catalog.yaml
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: community-operator-catalog
  namespace: openshift-marketplace
spec:
  displayName: Community Operator Catalog
  sourceType: grpc
  image: docker.io/wangzheng422/operator-catalog:community-2020-03-23
  publisher: Community
EOF
oc create -f community-operator-catalog.yaml

# 想刪除這些離線operator hub，就這樣做。
# oc delete -f *-operator-catalog.yaml
# find . -name "*-operator-catalog.yaml" -exec oc delete -f {} \;

oc get pods -n openshift-marketplace
oc get catalogsource -n openshift-marketplace
oc get packagemanifest -n openshift-marketplace

能看到operator 列表

部署一個operator也能成功

其他鏈接

https://www.cnblogs.com/ericnie/p/11764124.html

免責聲明！

本站轉載的文章為個人學習借鑒使用，本站對版權不負任何法律責任。如果侵犯了您的隱私權益，請聯系本站郵箱yoyou2525@163.com刪除。

猜您在找 OpenShift 4.2 離線安裝補充記錄 OpenShift 4.2 etcd operatorhub離線環境部署 Istio在Openshift 3.11的安裝 openshift 3.11安裝部署安裝WRF-4.3 git hub安裝 Helm安裝Prometheus Operator 什么是openshift 如何從網頁安裝群暉NAS並設置靜態IP VMware下安裝ubuntu，橋接上網配置靜態IP