kolla-ansible安裝openstack(Ocata)

基本功能部署

基礎環境

角色	操作系統	硬件配置
Depoly	CentOS 7 Server	磁盤:40GB 內存:8GB 網卡:ens3(內網) ens4(外網)
Sched	CentOS 7 Server	磁盤:40GB 內存:8GB 網卡:ens3(內網) ens4(外網)
Nova	CentOS 7 Server	磁盤:40GB 內存:8GB 網卡:ens3(內網) CPU開啟嵌套虛擬化

網絡配置

主機名	網絡地址	角色
deploy	4.0.0.10/24(內網) 192.168.200.10/24(外網)	Depoly
sched	4.0.0.11/24(內網) 192.168.200.11/24(外網)	Sched
nova	4.0.0.12/24(內網) 192.168.200.12/24(外網)	Nova

Deploy基礎環境配置

安裝PIP

# yum install epel-release
# yum install python-pip
# pip install -U pip

安裝PIP編譯環境

# yum install python-devel libffi-devel gcc openssl-devel

安裝ansible

# pip install -U ansible

安裝docker

# tee /etc/yum.repos.d/docker.repo <<-'EOF'
[dockerrepo]
name=Docker Repository
baseurl=https://yum.dockerproject.org/repo/main/centos/7/
enabled=1
gpgcheck=1
gpgkey=https://yum.dockerproject.org/gpg
EOF
# yum makecache fast
# yum install -y docker-engine-1.12.0

配置docker環境

### 配置鏡像加速
# mkdir -p /etc/docker
# tee /etc/docker/daemon.json <<-'EOF'
{
"registry-mirrors":  ["https://registry.docker-cn.com"]
}
EOF
# systemctl daemon-reload
# systemctl restart docker
# systemctl enable docker

安裝kolla-ansible

# git clone http://git.trystack.cn/openstack/kolla-ansible
# cd kolla-ansible
# git checkout stable/ocata
# pip install .
# cp -r etc/kolla /etc
### 拷貝inventory到當前目錄，也就是后面要執行kolla-ansible命令的目錄
# cp ansible/inventory/* ~/

配置kolla-ansible

### 設置網卡信息
# vim /etc/kolla/globals.yml
kolla_internal_vip_address: "4.0.0.9" ### 選擇一個沒被使用的內網地址
keepalived_virtual_router_id: "9" ### 與kolla_internal_vip_address末尾相同，防止同內網環境其他openstack部署環境干擾
network_interface: "ens3"
neutron_external_interface: "ens4"
openstack_logging_debug: "True"
nova_console: "spice"
# kolla-genpwd
# vim /etc/kolla/passwords.yml
keystone_admin_password: admin

關閉防火牆和SELINUX

# systemctl stop firewalld
# systemctl disable firewalld
# setenforce 0
# vim /etc/selinux/config
SELINUX=disabled

SSH免密碼登入

### Deploy執行
# ssh-keygen -t rsa

### Sched和Nova執行
# scp root@4.0.0.10:~/.ssh/id_rsa.pub ./
# cat id_rsa.pub >> ~/.ssh/authorized_keys
# chmod 600 ~/.ssh/authorized_keys

獲取docker鏡像

• 使用kolla官方鏡像源

# vim /etc/kolla/globals.yml
kolla_install_type: "source"
openstack_release: "ocata"
docker_namespace: "kolla"

• 手動構建鏡像

# git clone https://github.com/openstack/kolla
# cd kolla
# git checkout stable/ocata
# pip install tox
# tox -e genconfig
# cp -r etc/kolla /etc/
# vim /etc/kolla/kolla-build.conf
push = true
namespace = kolla
registry = 4.0.0.10:4000
install_type = source
tag = ocata
# kolla-build --config-file=/etc/kolla/kolla-build.conf

### kolla-ansible配置如下
# vim /etc/kolla/globals.yml
docker_registry: "4.0.0.10:4000"
docker_namespace: "kolla"
openstack_release: "ocata"
kolla_install_type: "source"

• 下載鏡像包

# wget http://tarballs.openstack.org/kolla/images/centos-source-registry-ocata.tar.gz

### kolla-ansible配置如下
# vim /etc/kolla/globals.yml
docker_registry: "4.0.0.10:4000"
docker_namespace: "lokolla"
openstack_release: "4.0.2"
kolla_install_type: "source"

設置本地鏡像源

# docker run -d -v /opt/registry:/var/lib/registry -p 4000:5000 --restart=always --name registry registry:2
### 此處使用下載鏡像包方式
# wget http://tarballs.openstack.org/kolla/images/centos-source-registry-ocata.tar.gz
# tar zxf centos-source-registry-ocata.tar.gz -C /opt/registry/

搭建http代理服務器（可選步驟）

在Deploy節點上搭建http服務器

# yum install squid -y

### 修改配置文件，刪除默認自帶的對IP和端口的限制，然后添加如下規則
# vim /etc/squid/squid.conf
http_access allow all

### 設置開機自啟動
# systemctl restart squid
# systemctl enable squid

### 修改ansible腳本
# vim /usr/share/kolla-ansible/ansible/kolla-host.yml
  environment:
    https_proxy : http://4.0.0.10:3128/
    http_proxy : http://4.0.0.10:3128/

修改Sched/Nova的環境的配置文件

### 設置yum代理
# vim /etc/yum.conf
proxy=http://4.0.0.10:3128
# yum makecache

修改部署文件

### Deploy節點上執行
# vim ~/multinode
[control]
4.0.0.11

[network]
4.0.0.11

[compute]
4.0.0.12

[monitoring]
4.0.0.11

[storage]
4.0.0.12

### 配置Sched、Nova主機基礎環境，安裝必備軟件包
# kolla-ansible -i multinode bootstrap-servers

部署openstack

# kolla-ansible prechecks -i multinode
# kolla-ansible deploy -i multinode

驗證openstack安裝

# kolla-ansible post-deploy -i multinode

OpenStack更新

### 修改鏡像版本
# vim /etc/kolla/globals.yml
openstack_release: "4.0.3"
# kolla-ansible pull -i multinode
# kolla-ansible upgrade -i multinode

環境還原

### 將刪除所有容器和卷
# kolla-ansible destroy -i multinode

可選功能部署

Ceph部署

修改配置文件

# vim multinode
[storage]
4.0.0.12

# vim /etc/kolla/globals.yml
enable_ceph: "yes"
enable_ceph_rgw: "yes"
enable_cinder: "yes"
### 也可以同時配置yes
glance_backend_file: "no"
glance_backend_ceph: "yes"

# mkdir -p /etc/kolla/config
# tee /etc/kolla/config/ceph.conf <<-'EOF'
[global]
osd pool default size = 1
osd pool default min size = 1
osd pool default pg num =  128
osd pool default pgp num =  128
EOF

修改Nova節點

### 添加一塊總線為IDE格式為qcow2的磁盤
# fdisk -l
Disk /dev/sda: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

# parted /dev/sda -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP 1 -1
# parted /dev/sda print
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sda: 10.7GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name                      Flags
 1      1049kB  10.7GB  10.7GB               KOLLA_CEPH_OSD_BOOTSTRAP

問題處理

部署問題

CEPH部署失敗

### 問題描述
TASK [ceph : Fetching Ceph keyrings] ********************************************************************************************************************************************************
 [WARNING]: when statements should not include jinja2 templating delimiters such as {{ }} or {% %}. Found: {{ (ceph_files_json.stdout | from_json).changed }}

fatal: [92.0.0.11]: FAILED! => {"failed": true, "msg": "The conditional check '{{ (ceph_files_json.stdout | from_json).changed }}' failed. The error was: No JSON object could be decoded"}

### 解決辦法

### 在OSD節點執行，獲取掛載的CEPH磁盤
# ds4ft=`mount | grep ceph | awk '{print $1}' | tr -d [:digit:] | sort -u`
### 在OSD節點執行，取消掛載
# echo $ds4ft | tr ' ' '\n' | xargs -i umount {}
### 在OSD節點執行，格式化磁盤
# echo $ds4ft | tr ' ' '\n' | xargs -i mkfs.xfs -f {}
### 在OSD節點執行，重新設置標記
# echo $ds4ft | tr ' ' '\n' | xargs -i  parted {} -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP 1 -1
### 在MON和OSD節點執行，刪除所有CEPH容器和配置文件
# docker ps --filter "label=kolla_version" --format "{{.Names}}" -a | grep -E "ceph|osd" | xargs -i docker rm -f {}
# rm -rf /var/lib/kolla/var/lib/ceph /var/lib/kolla/etc/ceph /etc/kolla/ceph-*
### 在MON節點執行
# docker volume rm ceph_mon_config

Mariadb部署失敗

### 問題描述
TASK: [mariadb | Creating haproxy mysql user] *********************************
......
stdout: localhost | FAILED! => {
    "changed": false,
    "failed": true,
    "msg": "unable to connect to database, check login_user and login_password are correct or ~/.my.cnf has the credentials. Exception message: (1045, \"Access denied for user 'root'@'mick-workstation' (using password: YES)\")"
}
msg: Task failed as maximum retries was encountered

### 解決辦法

# docker rm mariadb
# rm -rf /var/lib/docker/volumes/mariadb/_data/*

等待VIP超時

### 問題描述
獲取不到VIP

### 解決辦法

### 嘗試一: 修改kolla_internal_vip_address或keepalived_virtual_router_id防止VIP被其他kolla部署環境占用
# vim /etc/kolla/globals.yml
kolla_internal_vip_address: "4.0.0.5"
keepalived_virtual_router_id: "5"
# kolla-ansible -i multinode deploy --tags="haproxy"

### 嘗試二: 禁用主機的網絡管理器
# docker rm -f haproxy keepalived
# systemctl stop NetworkManager
# systemctl disable NetworkManager
# kolla-ansible -i multinode deploy --tags="haproxy"

VIP自動消失

### 問題描述
獲取到的VIP會自動消失

### 解決辦法

### 嘗試一: IP是否被占用
# ping 4.0.0.9

### 嘗試二: 判斷keepalived監控的服務是否正常
# docker exec -it keepalived bash
# ./check_alive.sh
### 如果打印的值為0說明keepalived監控的服務正常
# echo $?

### 嘗試三: 參考網上一篇帖子的做法(http://www.cnblogs.com/ayao/p/keepalived-loss-vip.html)，將VIP綁定的網卡的BOOTPROTO=none防止因為dhcp導致的VIP消失問題（雖然問題出現時網卡的配置是static的，應該不是dhcp的問題，但是似乎VIP不會消失了）

Bootstrap-servers執行失敗

### 問題描述
docker-engine安裝失敗

### 解決辦法

### 在節點上手動安裝，下載超時也會導致失敗
# yum clean all
# yum makecache fast
# yum install docker-engine-1.12.0 -y

kolla-ansible執行報錯

### 問題描述
Ansible 2.4.0: No test named 'equalto'

### 解決辦法

# pip install --upgrade Jinja2

kolla-ansible執行報錯

### 問題描述
ansible "msg": "shade is required for this module"

### 解決辦法

# pip install shade

鏡像拉取失敗

### 問題描述
Error response from daemon: Get https://4.0.0.10:4000/v1/_ping: http: server gave HTTP response to HTTPS client

Network timed out while trying to connect to http://4.0.0.10:4000/v1/repositories/lokolla/centos-source-fluentd/images. You may want to check your internet connection or if you are behind a proxy.

### 解決辦法

### 嘗試一: 降低docker版本
# yum remove docker-engine
# yum install docker-engine-1.12.0 -y

### 嘗試二: 修改docker服務配置文件
# vim /etc/systemd/system/docker.service.d/kolla.conf
[Service]
MountFlags=shared
ExecStart=
ExecStart=/usr/bin/docker daemon --insecure-registry 92.0.0.10:4000
# systemctl daemon-reload
# systemctl restart docker
# ps -ef|grep docker
root      1294     1  0 11:36 ?        00:00:01 /usr/bin/dockerd --insecure-registry 92.0.0.10:4000

無法生成/etc/hosts文件

### 問題描述
TASK [baremetal : Generate /etc/hosts for all of the nodes] *********************************************************************************************************************************
fatal: [4.0.0.11]: FAILED! => {"failed": true, "msg": "the field 'args' has an invalid value, which appears to include a variable that is undefined. The error was: 'dict object' has no attribute u'ansible_ens3'\n\nThe error appears to have been in '/usr/share/kolla-ansible/ansible/roles/baremetal/tasks/pre-install.yml': line 40, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Generate /etc/hosts for all of the nodes\n  ^ here\n"}
    to retry, use: --limit @/usr/share/kolla-ansible/ansible/kolla-host.retry

### 解決辦法

- 查詢下是不是有節點安裝python失敗，導致沒有執行Gather facts操作造成失敗
- 查詢下是不是有節點的網卡名不對

使用問題

虛擬機創建失敗

### 問題描述
創建帶volume的虛擬機提示"Block device mapping is invalid"
- 查看nova-compute日志:VolumeNotCreated: Volume f103b3f3-d0ff-4a2b-9a5e-4b7ea5a9abdc did not finish being created even after we waited 3 seconds or 2 attempts. And its status is error.
- 查看cinder-volume日志：Volume group "cinder-volumes" not found

### 解決辦法

### 打印vg列表發現的確沒有"cinder-volumes"，這個卷是要手動建立的
# vgdisplay
# dd if=/dev/zero of=./disk.img count=4096 bs=1MB
### 查詢未被占用的loop設備
# losetup -f
# losetup /dev/loop2 disk.img
# pvcreate /dev/loop2
# vgcreate cinder-volumes /dev/loop2
### 重啟容器，然后查看cinder-volume日志是否成功找到"cinder-volumes"
# docker restart cinder-volumes

虛擬機創建失敗

### 問題描述
創建帶volume的虛擬機提示"No valid host was found"
- 查看nova-compute日志: 'iscsiadm -m node -T iqn.2010-10.org.openstack:volume-bbbccab7-bdd7-4086-8d0e-e14898439131 -p 127.0.0.1:3260' failed

### 解決辦法

### 從日志看來是登入iscsi服務器失敗，IP地址不對
# vim /etc/kolla/cinder-volume/cinder.conf
my_ip = 4.0.0.12
# docker restart cinder-volumes

虛擬機創建失敗

### 問題描述
- nova-compute日志提示
ERROR nova.compute.manager [instance: 3af11e19-b4f8-452a-8f3d-3d659be050bd] libvirtError: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.

- libvirtd日志提示
2017-11-01 18:46:09.362+0000: 30348: error : virDBusCall:1570 : error from service: CanSuspend: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the re
ply, the reply timeout expired, or the network connection was broken.

### 解決辦法

檢查節點狀態，發現selinux開啟了，關閉selinux即可

虛擬機創建失敗

### 問題描述
- nova-compute日志提示
ERROR nova.image.glance [req-3d1ff662-1b82-4cf9-8aba-28644c8ba88b ded6f7f63e6743b4ac88213d5d2df5ce dded963ced3946b585c92d0684277eec - - -] Error writing to /var/lib/nova/instances/_base/bc9f28a296e64237d4205a6ea278c3e51c083ba0.part: 'NoneType' object is not iterable
ERROR nova.compute.manager [instance: 2e6b91e4-96f3-41ae-98d3-4d72f1b2cd37]   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 586, in download
ERROR nova.compute.manager [instance: 2e6b91e4-96f3-41ae-98d3-4d72f1b2cd37]     for chunk in image_chunks:
ERROR nova.compute.manager [instance: 2e6b91e4-96f3-41ae-98d3-4d72f1b2cd37] TypeError: 'NoneType' object is not iterable

- glance_api日志提示
WARNING glance.location [req-27feac8f-9cfb-4891-91fd-913bcb7a9f1f ded6f7f63e6743b4ac88213d5d2df5ce dded963ced3946b585c92d0684277eec - default default] Get image 856dbf39-8821-4906-a68c-e78f77d88b27 data failed: Image /var/lib/glance/images/856dbf39-8821-4906-a68c-e78f77d88b27 not found.
ERROR glance.location [req-27feac8f-9cfb-4891-91fd-913bcb7a9f1f ded6f7f63e6743b4ac88213d5d2df5ce dded963ced3946b585c92d0684277eec - default default] Glance tried all active locations to get data for image 856dbf39-8821-4906-a68c-e78f77d88b27 but all have failed.

### 解決辦法

到glance節點的/var/lib/glance/images目錄下的確沒有任何內容，重啟horizon容器，然后刪除鏡像，然后重新上傳並且重命名

虛擬機創建失敗

### 問題描述
rpc超時

### 解決辦法

rabbitmq集群部署的配置文件使用的是主機名，所以HA環境下各個Contoller主機的主機名不能相同

虛擬機創建失敗

### 問題描述
failed to connect to the hypervisor

### 解決辦法

### 查詢虛擬機的嵌套虛擬化是否開啟
# cat /sys/module/kvm_intel/parameters/nested
N
### 如果為N，關閉虛擬機，設置CPU，開啟嵌套虛擬化
# shutdown now

mariadb容器啟動失敗

### 問題描述
[ERROR] WSREP: failed to open gcomm backend connection: 131: invalid UUID: 00000000 (FATAL)
at gcomm/src/pc.cpp:PC():271

### 解決辦法

# rm -rf /var/lib/docker/volumes/mariadb/_data/gvwstate.dat
# docker restart mariadb

宕機后mariadb集群恢復

### 問題描述
mariadb服務異常

### 解決辦法

########## 停止所有mariadb容器 ##########
# docker stop mariadb
########## 找到最后關閉的mariadb主機，如果不記得就隨機選取一台或者根據/var/lib/docker/volumes/mariadb/_data/grastate.dat的seqno進行選取(越大優先級越高)，然后修改其grastate.dat文件的safe_to_bootstrap參數 ##########
# vim /var/lib/docker/volumes/mariadb/_data/grastate.dat
safe_to_bootstrap: 1
########## 修改mariadb容器啟動命令后啟動容器，查詢日志保證mariadb服務正常啟動 ##########
# vim /etc/kolla/mariadb/config.json
"command": "/usr/bin/mysqld_safe --wsrep-new-cluster",
# docker start mariadb
# tail -200f /var/lib/docker/volumes/kolla_logs/_data/mariadb/mariadb.log
########## 啟動其他節點的mariadb容器 ##########
# docker start mariadb
# tail -200f /var/lib/docker/volumes/kolla_logs/_data/mariadb/mariadb.log
########## 確保集群運行正常后，恢復最初修改的config.json（這樣就保證集群中所有的mariadb容器都是平等的）##########
# vim /etc/kolla/mariadb/config.json
"command": "/usr/bin/mysqld_safer",
# docker stop mariadb
# docker start mariadb
# tail -200f /var/lib/docker/volumes/kolla_logs/_data/mariadb/mariadb.log

Horizon無法訪問

### 問題描述
[Tue May 29 09:58:25.056236 2018] [core:error] [pid 20] [client 192.168.0.1:51277] Script timed out before returning headers: django.wsgi
[Tue May 29 09:58:28.017141 2018] [core:error] [pid 81] [client 192.168.0.105:48074] End of script output before headers: django.wsgi, referer: http://47.98.113.179:8011/

### 解決辦法

### 添加 WSGIApplicationGroup 配置項
# vim /etc/httpd/conf.d/openstack-dashboard.conf
...
WSGISocketPrefix run/wsgi
WSGIApplicationGroup %{GLOBAL}
...

Redis容器異常

### 問題描述
5:M 29 May 17:31:11.072 # Short read or OOM loading DB. Unrecoverable error, aborting now.
5:M 29 May 17:31:11.072 # Internal error in RDB reading function at rdb.c:1428 -> Unexpected EOF reading RDB file

### 解決辦法

# rm -rf /var/lib/docker/volumes/redis/_data/dump.rdb
# docker restart redis

在nova-compute容器執行PIP命令權限不夠

### 問題描述
容器內無法執行pip命令

### 解決辦法

### 在容器內執行
# cat /etc/nova/rootwrap.conf | grep filters_path
filters_path=/etc/nova/rootwrap.d,/usr/share/nova/rootwrap

### 在宿主機上執行
# tee pip.filters << EOF
[Filters]
pip: CommandFilter, pip, root
EOF
# docker cp pip.filters nova_compute:/usr/share/nova/rootwrap

# sudo nova-rootwrap /etc/nova/rootwrap.conf pip install -U pip

ovs命令無法正常執行

### 問題描述
# ovs-appctl ofproto/trace br-tun dl_vlan=3
2017-06-05T08:37:32Z|00001|daemon_unix|WARN|/var/run/openvswitch/ovs-vswitchd.pid: open: No such file or directory
ovs-appctl: cannot read pidfile "/var/run/openvswitch/ovs-vswitchd.pid" (No such file or directory)

### 解決辦法

### 添加啟動參數
# vim /etc/kolla/openvswitch-vswitchd/config.json
--pidfile=/var/run/openvswitch/ovs-vswitchd.pid
# docker restart openvswitch_vswitchd