1.openshift搭建

本文轉載自查看原文 2019-05-09 15:46 618 容器雲

第1章主機規划和所需文件

1.1 主機規划

IP地址	域名	用途
11.11.233.125	master01.song.test.cnpc	容器編排、etcd
11.11.233.126	master02.song.test.cnpc	容器編排、etcd
11.11.233.134	master03.song.test.cnpc	容器編排、etcd
11.11.233.127	node1.song.test.cnpc	Infra
11.11.233.128	node2.song.test.cnpc	Infra
11.11.233.129	node3.song.test.cnpc	Infra
11.11.233.130	node4.song.test.cnpc	容器運行
11.11.233.131	node5.song.test.cnpc	容器運行
11.11.233.132	node6.song.test.cnpc	容器運行
11.11.233.133	ha.song.test.cnpc registry.song.test.cnpc	Haproxy，registry

1.2 主機環境檢測與確認

通過top，free，lsblk等命令檢測各服務器的硬件配置是否符合規划

在registry主機上安裝ansible 並執行一下play

1）網絡配置檢測

檢測各服務器網絡配置是否正確，包含ip地址，網絡連通性，bond配置等。

注：bond的故障模擬測試在機房服務器配置網絡過程中完成。

2）時區檢測

使用date命令檢測各服務器時區是否為CST。

運行ntpq -p或chronyc sources -v查看是否配置NTP。

3）主機名檢測

檢測各服務器的主機名是否符合規划。如果未在安裝期間配置，則后續執行命令修改。

4）檢測所有服務器libvirtd服務是否處於停止狀態

# systemctl stop libvirtd

# systemctl disable libvirtd

# systemctl mask libvirtd

關閉服務之后重啟服務器即可。

5）所有節點關閉firewalld

# systemctl stop firewalld

# systemctl disable firewalld

# systemctl mask firewalld

6）所有節點關閉selinux

# setenforce 0;

# sed -i 's/^SELINUX=.*/SELINUX=permissive/' /etc/selinux/config;

NetworkManager

master，node，haproxy節點的selinux不要關。默認為開啟，不要改

NetworkManager默認開啟，不要關

[master]

11.11.233.125 name=master01

11.11.233.126 name=master02

11.11.233.134 name=master03

[node]

11.11.233.127 name=node1

11.11.233.128 name=node2

11.11.233.129 name=node3

11.11.233.130 name=node4

11.11.233.131 name=node5

11.11.233.132 name=node6

[other]

11.11.233.133 name=ha

[test:children]

master

node

other

[test:vars]

ansible_ssh_user=sysadm

ansible_ssh_pass=Passc0de@tpcpjl

1.3 搭建yum倉庫和docker倉庫（ocp3.6）

OpenShift 3安裝及運行依賴的RPM、Docker鏡像及程序，需要在聯網環境下預先下載。

需要下載的文件列表如下：

名稱

備注

YUM源鏡像

OpenShift安裝所依賴的YUM Repo：

l rhel-7-server-extras-rpms-3.6

l rhel-7-server-ose-3.6-rpms

l rhel-7-fast-datapath-rpms-3.6

Docker鏡像

OpenShift運行所依賴的Docker鏡像

l jenkins-2-rhel7-latest.tar.gz

l logging-deployer-v3.6.tar.gz

l metrics-deployer-v3.6.tar.gz

l ose-haproxy-router-v3.6.173.0.96.tar.gz

l jenkins-slave-maven-rhel7-latest.tar.gz

l logging-elasticsearch-v3.6.tar.gz

l metrics-hawkular-openshift-agent-v3.6.tar.gz

l ose-pod-v3.6.173.0.96.tar.gz

l jenkins-slave-nodejs-rhel7-latest.tar.gz

l logging-fluentd-v3.6.tar.gz

l metrics-heapster-v3.6.tar.gz

l ose-sti-builder-v3.6.173.0.96.tar.gz

l logging-auth-proxy-v3.6.tar.gz

l logging-kibana-v3.6.tar.gz

l ose-deployer-v3.6.173.0.96.tar.gz

l registry-console-v3.6.tar.gz

l logging-curator-v3.6.tar.gz

l metrics-cassandra-v3.6.tar.gz

l ose-docker-registry-v3.6.173.0.96.tar.

[root@ha ~]# tree -L 3 /mnt/

/mnt/

├── registry

│ └── docker

│ └── registry

└── yum

├── rhel-7-fast-datapath-rpms

│ ├── Packages

│ └── repodata

├── rhel-7-server-ansible-2.4-rpms

│ ├── Packages

│ └── repodata

├── rhel-7-server-extras-rpms

│ ├── Packages

│ └── repodata

├── rhel-7-server-ose-3.6-rpms

│ ├── Packages

│ └── repodata

├── rhel-7-server-ose-3.7-rpms

│ ├── Packages

│ └── repodata

├── rhel-7-server-ose-3.8-rpms

│ ├── Packages

│ └── repodata

├── rhel-7-server-ose-3.9-rpms

│ ├── Packages

│ └── repodata

└── rhel-7-server-rpms

├── Packages

└── repodata

配置好httpd和 repo文件

[root@ha ~]# cat /etc/yum.repos.d/redhat7.3.repo

[server-ose-3.9-rpms]

baseurl = http://11.11.233.133/rhel-7-server-ose-3.9-rpms

name = Red Hat OpenShift Container Platform 3.9 RPMs

enabled = 0

gpgcheck = 0

[rhel-7-server-ose-3.6-rpms]

name = rhel-7-server-ose-3.6-rpms

baseurl = http://11.11.233.133/rhel-7-server-ose-3.6-rpms/

gpgcheck = 0

enabled = 1

[rhel-7-server-ose-3.8-rpms]

baseurl = http://11.11.233.133/rhel-7-server-ose-3.8-rpms

name = Red Hat OpenShift Container Platform 3.8 RPMs

enabled = 0

gpgcheck = 0

[rhel-7-server-ose-3.7-rpms]

baseurl = http://11.11.233.133/rhel-7-server-ose-3.7-rpms

name = Red Hat OpenShift Container Platform 3.7 RPMs

enabled = 0

gpgcheck = 0

[rhel-7-server-extras-rpms]

baseurl = http://11.11.233.133/rhel-7-server-extras-rpms

name = Red Hat rhel-7-server-extras-rpms RPMs

enabled = 1

gpgcheck = 0

[rhel-7-fast-datapath-rpms]

baseurl = http://11.11.233.133/rhel-7-fast-datapath-rpms

name = Red Hat rhel-7-fast-datapath-rpms RPMs

enabled = 1

gpgcheck = 0

[rhel-7-server-ansible-2.4-rpms]

baseurl = http://11.11.233.133/rhel-7-server-ansible-2.4-rpms

name = Red Hat rhel-7-server-ansible-2.4-rpms RPMs

enabled = 1

gpgcheck = 0

[rhel-7-server-rpms]

baseurl = http://11.11.233.133/rhel-7-server-rpms

name = Red Hat rhel-7-server-rpms RPMs

enabled = 1

gpgcheck = 0

[root@ha ~]# yum repolist

Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager

This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.

repo id repo name status

rhel-7-fast-datapath-rpms Red Hat rhel-7-fast-datapath-rpms RPMs 38

rhel-7-server-ansible-2.4-rpms Red Hat rhel-7-server-ansible-2.4-rpms RPMs 10

rhel-7-server-extras-rpms Red Hat rhel-7-server-extras-rpms RPMs 141

rhel-7-server-ose-3.6-rpms rhel-7-server-ose-3.6-rpms 483

rhel-7-server-rpms Red Hat rhel-7-server-rpms RPMs

倉庫使用 docker-registry

[root@ha ~]# cat /etc/docker-distribution/registry/config.yml

version: 0.1

log:

fields:

service: registry

storage:

cache:

layerinfo: inmemory

filesystem:

rootdirectory: /mnt/registry

http:

addr: :5000

secret: 95d5b1erc2a905586e790f794514ea38

測試鏡像拉取

v3.6: Pulling from registry.song.test.cnpc:5000/openshift3/logging-curator

9cadd93b16ff: Already exists

4aa565ad8b7a: Already exists

d131575534ed: Pull complete

Digest: sha256:9a0d7cf6532da31f08239cc25e74bad118a828b4dc3a67a8bf442ff6faba140f

Status: Downloaded newer image for registry.song.test.cnpc:5000/openshift3/logging-curator:v3.6

第2章安裝OpenShift預備

2.1 安裝軟件包並配置基礎環境

l 在所有節點上安裝OpenShift需要的軟件包。命令如下：

yum -y install wget git net-tools bind-utils iptables-services bridge-utils bash-completion vim atomic-openshift-excluder atomic-openshift-docker-excluder unzip kexec sos psacct;

yum -y update;

atomic-openshift-excluder unexclude;

l 確認SELinux為permissive狀態。命令如下：

setenforce 0;

sed -i 's/^SELINUX=.*/SELINUX=permissive/' /etc/selinux/config;

l 所有節點關閉firewalld。命令如下：

systemctl disable firewalld;

systemctl stop firewalld;

2.2 配置免密登錄

l 在Master節點上生成SSH所需之秘鑰。命令如下，應答輸入請直接輸入回車。

ssh-keygen;

l 在Master節點上配置Master節點到所有節點的SSH主機互信。命令如下，請根據提示輸入遠程主機Root賬戶密碼。

l 如果root登錄關閉，需要開啟，使用如下命令：sed -i 's/PermitRootLogin no/PermitRootLogin yes/' /etc/ssh/sshd_config

l cat /etc/ssh/sshd_config

2.3 本地DNS服務器創建與配置

每個master和node

# 因為dnsmasq服務會和libvirt服務沖突，所以此處把它干掉

yum remove libvirt -y

ps -ef |grep dnsmasq |grep -v grep |awk '{print $2}' |xargs -i kill -9 {}

systemctl disable libvirtd

systemctl stop libvirtd

2.3.1 添加dnsmasq配置

每個master節點添加wildcard域名指向。命令如下：

cat > /etc/dnsmasq.d/openshift-cluster.conf <<EOF

local=/song.test.cnpc/

address=/.apps.song.test.cnpc/11.11.233.133

EOF

若router為高可用部署，此ip應該為ha主機的ip 11.11.233.133

啟動dnsmasq服務

每個master節點啟動dnsmasq服務。命令如下：

systemctl restart dnsmasq;

systemctl enable dnsmasq;

2.3.2 配置iptables

每個master和node節點修改iptables規則。命令如下：

cp /etc/sysconfig/iptables /etc/sysconfig/iptables.bak.$(date "+%Y%m%d%H%M%S");

sed -i '/.*--dport 22 -j ACCEPT.*/a\-A INPUT -p tcp -m state --state NEW -m tcp --dport 53 -j ACCEPT' /etc/sysconfig/iptables;

sed -i '/.*--dport 22 -j ACCEPT.*/a\-A INPUT -p udp -m state --state NEW -m udp --dport 53 -j ACCEPT' /etc/sysconfig/iptables;

systemctl restart iptables;

systemctl restart NetworkManager;

2.3.3 配置各節點域名解析

配置每個Node節點域名解析。命令如下：

cat > /etc/dnsmasq.d/openshift-cluster-node.conf <<EOF

server=11.11.233.125

server=11.11.233.126

server=11.11.233.134

EOF

此部署方式，若第一個節點down，dns輪詢到第二個節點需等5秒，會導致應用通過dns訪問中斷

三個ip分別為master節點ip

systemctl restart dnsmasq;

systemctl enable dnsmasq;

2.3.4 測試DNS解析

在每個node節點執行

nslookup docker-registry-default.apps.jtdjnet.cnpc

2.4 安裝配置docker

2.4.1 安裝Docker

在所有master、node，registry上安裝Docker。命令如下：

yum -y install docker; #安裝docker

systemctl enable docker;

cp /etc/sysconfig/docker /etc/sysconfig/docker.bak.$(date "+%Y%m%d%H%M%S")

sed -i s/".*OPTIONS=.*"/"OPTIONS='--selinux-enabled --log-driver=journald --insecure-registry 172.30.0.0\/16 --insecure-registry registry.song.test.cnpc:5000'"/g /etc/sysconfig/docker;

sed -i 's/registry.access.redhat.com/registry.song.test.cnpc:5000/g' /etc/sysconfig/docker

echo "BLOCK_REGISTRY='--block-registry public --block-registry registry.access.redhat.com' ">>/etc/sysconfig/docker;

2.4.2 配置docker存儲

POC和測試可跳過。生產環境必須配置

磁盤名稱先fdisk -l 看下，有的環境可能不叫sdb，叫vdb等

cat<<EOF>/etc/sysconfig/docker-storage-setup

DEVS=/dev/sdb

VG=docker-vg

SETUP_LVM_THIN_POOL=yes

EOF

docker-storage-setup

設置完后查看配置更改是否成功

cat /etc/sysconfig/docker-storage

DOCKER_STORAGE_OPTIONS="--storage-driver devicemapper --storage-opt dm.fs=xfs --storage-opt dm.thinpooldev=/dev/mapper/docker-docker-pool --storage-opt dm.use_deferred_removal=true --storage-opt dm.use_deferred_deletion=true "

2.4.3 啟動docker

systemctl restart docker;

docker info;

返回值要有registry.song.test.cnpc:5000和172.30.0.0

第3章 OpenShift 3安裝

l 在registry節點安裝openshift的安裝腳本

yum -y install atomic-openshift-utils

l 登錄Master01節點執行安裝。命令如下：

cat > /etc/ansible/hosts <<EOF

# Create an OSEv3 group that contains the masters, nodes, and etcd groups

[OSEv3:children]

masters

nodes

etcd

# Set variables common for all OSEv3 hosts

[OSEv3:vars]

ansible_ssh_user=root

openshift_deployment_type=openshift-enterprise

# uncomment the following to enable htpasswd authentication; defaults to DenyAllPasswordIdentityProvider

openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]
openshift_master_cluster_method=native
openshift_master_cluster_hostname=master.song.test.cnpc.cnpc
openshift_master_cluster_public_hostname=master.song.test.cnpc.cnpc

openshift_docker_options="--selinux-enabled --insecure-registry 172.30.0.0/16 --log-driver json-file --log-opt max-size=50M --log-opt max-file=3 --insecure-registry registry.song.test.cnpc.cnpc:5000 --add-registry registry.song.test.cnpc.cnpc:5000"
openshift_master_default_subdomain=apps.song.test.cnpc.cnpc

os_sdn_network_plugin_name='redhat/openshift-ovs-multitenant'

openshift_hosted_router_selector='router=router'
openshift_hosted_router_replicas=2
openshift_hosted_registry_selector='infra=infra'

openshift_hosted_logging_deploy=true
openshift_logging_image_prefix=registry.song.test.cnpc.cnpc:5000/openshift3/
openshift_logging_image_version=v3.6
openshift_logging_public_master_url=master.song.test.cnpc.cnpc

openshift_hosted_metrics_deploy=true
openshift_metrics_image_prefix=registry.song.test.cnpc.cnpc:5000/openshift3/
openshift_metrics_image_version=v3.6
openshift_hosted_metrics_public_url=https://hawkular-metrics.apps.song.test.cnpc.cnpc/hawkular/metrics

openshift_cockpit_deployer_prefix=registry.song.test.cnpc.cnpc:5000/openshift3/
openshift_cockpit_deployer_version=v3.6

oreg_url=registry.song.test.cnpc.cnpc:5000/openshift3/ose-${component}:${version}
openshift_examples_modify_imagestreams=true

openshift_enable_service_catalog=false

openshift_disable_check="disk_availability,docker_image_availability,memory_availability,docker_storage,package_version,package_availability"

# host group for masters

[masters]

djmast001.song.test.cnpc.cnpc

djmast002.song.test.cnpc.cnpc

djmast003.song.test.cnpc.cnpc

[lb]

djmlbt001.song.test.cnpc.cnpc

# host group for etcd

[etcd]

djmast001.song.test.cnpc.cnpc

djmast002.song.test.cnpc.cnpc

djmast003.song.test.cnpc.cnpc

# host group for nodes, includes region info

[nodes]

djmast001.song.test.cnpc.cnpc

djmast002.song.test.cnpc.cnpc

djmast003.song.test.cnpc.cnpc

djinft001.song.test.cnpc.cnpc openshift_node_labels="{'region': 'infra', 'zone': 'default', 'router': 'router'}"

djinft002.song.test.cnpc.cnpc openshift_node_labels="{'region': 'infra', 'zone': 'default', 'router': 'router'}"

djinft003.song.test.cnpc.cnpc openshift_node_labels="{'region': 'infra', 'zone': 'default', 'infra': 'infra'}"

djnodt001.song.test.cnpc.cnpc openshift_node_labels="{'region': 'primary', 'zone': 'zone1'}"

djnodt002.song.test.cnpc.cnpc openshift_node_labels="{'region': 'primary', 'zone': 'zone2'}"

djnodt003.song.test.cnpc.cnpc openshift_node_labels="{'region': 'primary', 'zone': 'zone3'}"

djnodt004.song.test.cnpc.cnpc openshift_node_labels="{'region': 'primary', 'zone': 'zone4'}"

djnodt005.song.test.cnpc.cnpc openshift_node_labels="{'region': 'primary', 'zone': 'zone5'}"

EOF

執行安裝

ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/byo/config.yml;

備用卸載命令：

ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/adhoc/uninstall.yml;

備注：在安裝的過程中會出現下面問題，Wait for API to become available，這是在調用API接口時找不到對應文件，就會一直嘗試連接

l 重啟sshd服務，命令如下：systemctl restart sshd

免責聲明！

本站轉載的文章為個人學習借鑒使用，本站對版權不負任何法律責任。如果侵犯了您的隱私權益，請聯系本站郵箱yoyou2525@163.com刪除。

猜您在找 [轉]OpenShift 集群搭建指南 OpenShift實戰(一)：OpenShift安裝 OpenShift實戰(二)：OpenShift節點擴容理解OpenShift（5）：從 Docker Volume 到 OpenShift Persistent Volume OpenShift實戰(五)：OpenShift容器監控Metrics OpenShift實戰(六)：OpenShift日志監控EFK Openshift與Kubernetes的區別 OpenShift上的OpenvSwitch入門 openshift scc解析 Openshift部署流程介紹

1.openshift搭建

第1章 主機規划和所需文件

1.1 主機規划

1.2 主機環境檢測與確認

1.3 搭建yum倉庫和docker倉庫（ocp3.6）

第2章 安裝OpenShift預備

2.1 安裝軟件包並配置基礎環境

2.2 配置免密登錄

2.3 本地DNS服務器創建與配置

2.3.1 添加dnsmasq配置

2.3.2 配置iptables

2.3.3 配置各節點域名解析

2.3.4 測試DNS解析

2.4 安裝配置docker

2.4.1 安裝Docker

2.4.2 配置docker存儲

2.4.3 啟動docker

第3章 OpenShift 3安裝

免責聲明！

第1章主機規划和所需文件

第2章安裝OpenShift預備