===========================================================
2017-6-15 kolla多節點部署Openstack雲主機項目實戰
一、Kolla節點環境搭建
1、安裝docker基礎配置：
[root@vm3 opt]# vim /etc/sysconfig/selinux
disabled
[root@vm3 opt]# setenforce 0
[root@vm3 opt]# systemctl stop firewalld
[root@vm3 opt]# systemctl disable firewalld
# yum install epel-release
# yum install -y python-devel libffi-devel gcc openssl-devel git
# curl -sSL https://get.docker.io | bash //注:這條命令安裝的是最新版的docker，會默認下載docker源
# tee /etc/yum.repos.d/docker.repo << 'EOF'
[dockerrepo]
name=Docker Repository
baseurl=https://yum.dockerproject.org/repo/main/centos/$releasever/
enabled=1
gpgcheck=1
gpgkey=https://yum.dockerproject.org/gpg
EOF
# systemctl status docker.service
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Active: inactive (dead)
# mkdir -p /etc/systemd/system/docker.service.d //新建一個docker后台守護進程配置文檔增加配置
# tee /etc/systemd/system/docker.service.d/kolla.conf <<-'EOF'
[Service]
MountFlags=shared
EOF
保存退出，重新加載配置，重啟docker服務
# systemctl daemon-reload
# systemctl restart docker
[root@vm3 ~]# systemctl enable docker.service
# yum install -y python-docker-py
[root@vm3 ~]# pip install --upgrade pip
# pip install -U docker-py
# yum install -y ansible
# git clone https://github.com/openstack/kolla -b stable/ocata
# git clone https://github.com/openstack/kolla-ansible -b stable/ocata
# pip install kolla/ ##安裝kolla
# pip install kolla-ansible
2、建立私有倉庫
[root@vm3 ~]# docker run -d -v /opt/registry:/var/lib/registry -p 4000:5000 --restart=always --name registry registry:2
## /opt/registry是宿主機的目錄，默認docker的registry是使用5000端口，對於OpenStack來說，有端口沖突，所以改成4000
[root@vm3 network-scripts]# vim /usr/lib/systemd/system/docker.service
ExecStart=/usr/bin/docker daemon --insecure-registry 172.16.1.5:4000
3、Kolla本地鏡像部署 http://www.chenshake.com/kolla-installation/ 陳沙克筆記
kolla鏡像源：http://tarballs.openstack.org/kolla/images/
[root@kolla ~]# docker load -i centos-source-registry-ocata.tar.gz
# mkdir /opt/registry
# tar -xf centos-source-registry-ocata.tar.gz -C /opt/registry/
# docker run -d -v /opt/registry:/var/lib/registry -p 4000:5000 --restart=always --name local_registry registry:latest
# curl http://172.16.1.5:4000/v2/_catalog //可以通過curl來訪問驗證本地Registry是否正常，檢查鏡像解壓到regisrty是否有效
[root@kolla ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
registry latest 9d0c4eabab4d 4 weeks ago 33.2MB
[root@kolla ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c9f4499bb0d8 registry:latest "/entrypoint.sh /e..." 2 minutes ago Up 2 minutes 0.0.0.0:4000->5000/tcp local_registry
如果是在虛擬機里裝kolla，希望可以虛擬機中再啟動雲主機，那么你需要把virt_type=qemu
# egrep -c '(vmx|svm)' /proc/cpuinfo
# mkdir -p /etc/kolla/config/nova //服務器默認就是kvm，無需操作該步驟。
cat << EOF > /etc/kolla/config/nova/nova-compute.conf
[libvirt]
virt_type=qemu
cpu_mode = none
EOF
[root@vm3 ~]# cp -r kolla-ansible/etc/kolla/* /etc/kolla/
[root@vm3 ~]# mkdir -p /etc/ansible/inventory
[root@vm3 ~]# cp -r kolla-ansible/ansible/inventory/* /etc/ansible/inventory ##這一步可以不做
[root@vm3 ~]# kolla-genpwd
# vim /etc/kolla/passwords.yml ##或者使用kolla-genpwd產生隨機密鑰，是登錄Dashboard，admin使用的密碼
keystone_admin_password: admin ##注意有空格
[root@vm3 ~]# vim /etc/kolla/globals.yml
kolla_base_distro: "centos"
kolla_install_type: "source"
kolla_internal_vip_address: "172.16.1.254" ##跟物理網卡在同一網段
network_interface: "eth0" ##eth0作為管理網絡
neutron_external_interface: "eth1" ##eth1不能配置IP，它作為外網網絡
docker_registry: "172.16.1.5:4000"
docker_namespace: "lokolla" ##必須是lokolla，否則就會報錯。
openstack_release: "auto" ##auto可能出錯，改為4.0.2
二、需求規划
vm1：計算節點 eth0:172.16.1.6 /dev/vdb 50GB eth1無IP
vm2：計算節點 172.16.1.8 /dev/vdb 50GB
vm3: kolla的master節點（同樣是控制節點 172.16.1.5 必須滿足雙網卡，至少6GB內存： eth0 172.16.1.0/24管理內部網絡 eth1 無IP作為外部網絡）
前提：vm3的kolla環境按照前面的文檔已經部署成功，三台機器的docker環境具備，最好采用最新版本的docker。
1、docker環境部署，針對控制和計算節點每天均做（#號表示兩台機器都敲命令）。
# yum install epel-release
# yum install -y python-devel libffi-devel gcc openssl-devel git
# curl -sSL https://get.docker.io | bash //注:這條命令安裝的是最新版的docker，會默認下載docker源
# mkdir -p /etc/systemd/system/docker.service.d //新建一個docker后台守護進程配置文檔增加配置
# tee /etc/systemd/system/docker.service.d/kolla.conf <<-'EOF'
[Service]
MountFlags=shared
EOF
（1）控制節點和計算節點的docker服務指定鏡像倉庫為172.16.1.5::4000
# vim /usr/lib/systemd/system/docker.service
ExecStart=/usr/bin/docker daemon --insecure-registry 172.16.1.5:4000
# systemctl daemon-reload
# systemctl restart docker
# systemctl stop NetworkManager
# systemctl disable NetworkManager
# vim /etc/hosts
172.16.1.5 vm3
172.16.1.8 vm2
172.16.1.6 vm1
# egrep -c '(vmx|svm)' /proc/cpuinfo
0
（2）如果返回0，那么在kolla節點，需要修改virt_type為qemu，否則不做任何操作
[root@vm3 ~]# mkdir /etc/kolla/config/nova
[root@vm3 ~]# cat << EOF > /etc/kolla/config/nova/nova-compute.conf
[libvirt]
virt_type = qemu
EOF
（3）三台機器均做免密鑰登錄，vm2、vm3此處不做贅述
[root@vm1 ~]# ssh-keygen
[root@vm1 ~]# ssh-copy-id vm2
[root@vm1 ~]# ssh-copy-id vm3
（4）控制節點和計算節點/dev/vdb作為ceph osd存儲。
# fdisk -l /dev/vdb
Disk /dev/vdb: 53.7 GB, 53687091200 bytes, 104857600 sectors
# parted /dev/vdb -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP 1 -1 //標記此塊磁盤為ceph OSD盤，兩台機器均做
# parted /dev/vdb print
Model: Virtio Block Device (virtblk)
Disk /dev/vdb: 53.7GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 1049kB 53.7GB 53.7GB KOLLA_CEPH_OSD_BOOTSTRAP
（5）koll節點的ceph.conf配置
[root@vm3 ~]# mkdir /etc/kolla/config
[root@vm3 ~]# vim /etc/kolla/config/ceph.conf
[global]
osd pool default size = 2 ##因為有2個ceph節點，所以是2，如果只有一個ceph節點就寫1
osd pool default min size = 2
[root@vm3 ~]# vim /etc/kolla/globals.yml ##不使用swift了，太復雜了，而且和cinder貌似沖突的
enable_ceilometer: "yes"
enable_mongodb: "yes"
enable_ceph: "yes"
enable_ceph_rgw: "yes"
enable_cinder: "yes"
enable_ceph_rgw_keystone: "yes"
glance_backend_ceph: "yes"
enable_freezer: "yes"
enable_neutron_lbaas: "yes"
enable_neutron_fwaas: "yes"
enable_neutron_qos: "yes"
其余都是默認啟動的
[root@vm3 ~]# curl http://172.16.1.5:4000/v2/_catalog ##檢查kolla節點的鏡像是否可用，從而驗證kolla環境部署成功
二、編輯ansible的inventory主機清單文件
[root@vm3 ~]# vim /usr/share/kolla-ansible/ansible/inventory/multinode //看來完全沒有必要去拷貝到ansible的inventory目錄下了。
[control]
localhost ansible_connection=local
[network]
localhost ansible_connection=local
[compute]
vm1
vm2
[monitoring]
localhost ansible_connection=local
[storage]
vm1
vm2
（1）安裝openstack CLI相關命令行，控制和計算節點均做
# yum install python-pip -y
# pip install -U python-openstackclient
# pip install -U python-neutronclient
（2）kolla節點初始化環境（可選），運行prechecks檢查清單
[root@vm3 ~]# kolla-genpwd
[root@vm3 ~]# kolla-ansible prechecks -i /usr/share/kolla-ansible/ansible/inventory/multinode ##檢查清單
[root@vm3 ~]# kolla-ansible deploy -i /usr/share/kolla-ansible/ansible/inventory/multinode
[root@vm3 ~]# cp /etc/kolla/admin-openrc.sh /root
# kolla-ansible post-deploy
# cp /etc/kolla/admin-openrc.sh /root //生成admin-openrc.sh文件，路徑為/etc/kolla/admin-openrc.sh
[root@vm3 ~]# source admin-openrc.sh
[root@vm3 ~]# vim /usr/share/kolla-ansible/init-runonce
## 172.16.1.0/24網段是eth1所在的網段，eth1的IP並沒有進行配置
EXT_NET_CIDR='172.16.1.0/24'
EXT_NET_RANGE='start=172.16.1.100,end=172.16.1.199'
EXT_NET_GATEWAY='172.16.1.1'
[root@vm3 ~]# /usr/share/kolla-ansible/init-runonce //執行初始化
（3）ceph的鏡像格式有qcow2改為raw格式
# yum install qemu-img -y
# qemu-img info cirros-0.3.4-x86_64-disk.img
# qemu-img convert -f qcow2 -O raw cirros-0.3.4-x86_64-disk.img cirros-0.3.4-x86_64-disk.raw
# qemu-img info cirros-0.3.4-x86_64-disk.raw
# openstack image create --disk-format raw --container-format bare --public true --file ./cirros-0.3.4-x86_64-disk.raw
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
三、工作過程報錯：
1、TASK [prechecks : Checking docker-py version] *****************
ImportError: No module named docker
解決：兩台機器手動安裝python-docker-py # yum install python-docker-py
2、TASK [mariadb : Checking free port for MariaDB]
TASK [horizon : Checking free port for Horizon
Timeout when waiting for 172.16.1.6:3306 to stop
解決：初步判斷是ansible連接172.16.1.6控制節點數據庫超時，應該是密碼出錯，因為vm1機器曾經作為zabbix，所以MySQL的root密碼不為空
[root@vm1 ~]# systemctl stop mariadb.service //看來部署機器必須是裸機才行。
[root@vm1 ~]# systemctl stop httpd
3、在kolla-ansible deploy的時候，報出 TASK [common : Starting fluentd container]
Get https://172.16.1.5:4000/v1/_ping: http: server gave HTTP response to HTTPS client
解決：很明顯docker客戶端是https服務，而kolla服務端是http服務
在控制節點和計算節點均做：
# vim /usr/lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd --registry-mirror=http://019a7061.m.daocloud.io --insecure-registry 172.16.1.5:4000
參考文檔：http://www.cnblogs.com/52fhy/p/5998747.html
4、TASK [haproxy : Waiting for virtual IP to appear] ***********************************************************************
fatal: [vm1]: FAILED! => {"changed": false, "elapsed": 301, "failed": true, "msg": "Timeout when waiting for 172.16.1.254:3306"}
未解決：從報錯的意思來看，是VIP不出現。
[root@vm1 docker]# netstat -tunlp|grep 3306
tcp 0 0 172.16.1.253:3306 0.0.0.0:* LISTEN 5709/haproxy
haproxy監聽在3306端口，但是沒有這個172.16.1.253的IP地址
在kolla節點查看global.yml，突然想到控制節點只有單網卡，所以再添加一個網卡試試
結果依舊不行，考慮到單控節點，直接把# kolla_internal_vip_address: "172.16.1.254"注釋掉，結果vm1不可達，登錄到控制台發現
172.16.1.254的虛擬IP已經飄逸到eth1上，但是原有的本身物理IP地址172.16.1.6卻消失了。於是它不能注釋掉
於是交換network_interface和neutron_external_interface的網卡試試
[root@vm3 ~]# egrep -v "^$|^#" /etc/kolla/globals.yml
network_interface: "eth1"
neutron_external_interface: "eth0"
登錄到vm1控制節點：[root@vm1 ~]# ip a
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast state UP qlen 1000
link/ether fa:16:3e:56:49:e7 brd ff:ff:ff:ff:ff:ff
inet 172.16.1.6/24 brd 172.16.1.255 scope global dynamic eth0
4: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether fa:16:3e:c7:a8:17 brd ff:ff:ff:ff:ff:ff
再次執行：[root@vm3 ]# kolla-ansible deploy -i /usr/share/kolla-ansible/ansible/inventory/multinode
5、TASK [common : Copying over fluentd input config files] *****************************************************************
ok: [vm1] => (item=00-global)
failed: [vm1] (item=01-syslog) => {"failed": true, "item": "01-syslog", "msg": "AnsibleUndefinedVariable: 'dict object' has no attribute 'ipv4'"}
解決：# vim /etc/kolla/global.yml
奇怪，當我把network_interface: "eth0"和neutron_external_interface: "eth1"，順序復原之后就OK了。
應該是外部網絡不能使用帶有IP的網卡吧
6、其他機器無法ping通VIP 172.16.1.254？
未解決：這個問題對於單控來說，無所謂，我登錄控制節點的物理網卡IP也行，如果是高可用的，應該涉及到pacemaker或者keepalived
四、常用后期命令
[root@vm3 ~]# openstack service list
[root@vm3 ~]# nova service-list
[root@vm3 ~]# cinder service-list
[root@vm3 ~]# glance image-list
[root@vm3 ~]# openstack network agent list
[root@vm3 ~]# nova list --all-tenants ##查看啟動雲主機
[root@vm3 ~]# openstack image list
[root@vm3 ~]# openstack flavor lis
驗證cephL
[root@vm3 ~]# docker ps|grep ceph
b5c33f207c58 172.16.1.5:4000/lokolla/centos-source-ceph-rgw:4.0.2 "kolla_start" 3 hours ago Up About a minute ceph_rgw
05d64b1be164 172.16.1.5:4000/lokolla/centos-source-ceph-mon:4.0.2 "kolla_start" 3 hours ago Up 3 hours
[root@vm3 ~]# docker exec -it ceph_mon ceph -s
[root@vm3 ~]# docker exec -it ceph_mon rbd ls images