一：各模塊屬性
| 模塊名稱 | 狀態 | 建議實例數 | 功能 | 負載均衡組件 |
| TiDB | 無狀態 | 2 | 接收SQL請求，處理SQL相關邏輯，並通過PB找到存儲數據的TiKV地址 | LVS、HAProxy、F5 |
| PB | 集群 | 3 奇數個節點,推薦>3 | 整個集群的管理模塊，存儲元信息、對TiKV集群進行調度和負載均衡、分配全局事務ID | Raft |
| TiKV | 集群 | 3 | 負責存儲數據 | Raft |
二：環境要求
2.1：開發測試環境：
| 組件 | CPU | 內存 | 本地存儲 | 網絡 | 實例數量(最低要求) |
|---|---|---|---|---|---|
| TiDB | 8核+ | 16 GB+ | 無特殊要求 | 千兆網卡 | 1（可與 PD 同機器） |
| PD | 4核+ | 8 GB+ | SAS, 200 GB+ | 千兆網卡 | 1（可與 TiDB 同機器） |
| TiKV | 8核+ | 32 GB+ | SSD, 200 GB+ | 千兆網卡 | 3 |
2.2：生產環境：
| 組件 | CPU | 內存 | 硬盤類型 | 網絡 | 實例數量(最低要求) |
|---|---|---|---|---|---|
| TiDB | 16核+ | 32 GB+ | SAS | 萬兆網卡（2塊最佳） | 2 |
| PD | 4核+ | 8 GB+ | SSD | 萬兆網卡（2塊最佳） | 3 |
| TiKV | 16核+ | 32 GB+ | SSD | 萬兆網卡（2塊最佳） | 3 |
| 監控 | 8核+ | 16 GB+ | SAS | 千兆網卡 | 1 |
2.3：端口說明：
| 組件 | 默認端口 | 說明 |
|---|---|---|
| TiDB | 4000 | 應用及 DBA 工具訪問通信端口 |
| TiDB | 10080 | TiDB 狀態信息上報通信端口 |
| TiKV | 20160 | TiKV 通信端口 |
| PD | 2379 | 提供 TiDB 和 PD 通信端口 |
| PD | 2380 | PD 集群節點間通信端口 |
| Pump | 8250 | Pump 通信端口 |
| Drainer | 8249 | Drainer 通信端口 |
| Prometheus | 9090 | Prometheus 服務通信端口 |
| Pushgateway | 9091 | TiDB，TiKV，PD 監控聚合和上報端口 |
| Node_exporter | 9100 | TiDB 集群每個節點的系統信息上報通信端口 |
| Blackbox_exporter | 9115 | Blackbox_exporter 通信端口，用於 TiDB 集群端口監控 |
| Grafana | 3000 | Web 監控服務對外服務和客戶端(瀏覽器)訪問端口 |
| Grafana | 8686 | grafana_collector 通信端口，用於將 Dashboard 導出為 PDF 格式 |
| Kafka_exporter | 9308 | Kafka_exporter 通信端口，用於監控 binlog kafka 集群 |
三：環境部署：
3.1：群架構
xm-tidb-01 192.168.1.1
xm-pd-01 192.168.1.2
xm-tikv-01 192.168.1.3
xm-tikv-02 192.168.1.4
xm-tikv-03 192.168.1.5
3.2：創建ext4盤：（所有節點）
參照：https://www.cnblogs.com/jackyzm/p/10402275.html
vim mount-ext4.sh
#!/bin/sh #https://www.cnblogs.com/jackyzm/p/10402275.html ##縮小home空間到5G mkdir /homebak sleep 1 cp -r /home /homebak sleep 5 umount /home lvremove /dev/mapper/centos-home -y sleep 3 lvcreate -L 5G -n home centos -y sleep 3 mkfs.xfs /dev/mapper/centos-home sleep 15 mount /dev/mapper/centos-home /home ##新建ext4分區 lvcreate -L 20G -n ext4 centos -y sleep 3 mkfs.ext4 /dev/mapper/centos-ext4 sleep 15 lsblk -f
./mount-ext4.sh
vim /etc/fstab 添加
UUID=a45530a9-8b07-4b9e-b78b-d6480e239dea /ext4 ext4 defaults,nodelalloc,noatime 0 2
mkdir /ext4
mount -a
mount -t ext4
3.3：安裝依賴包（在主控機操作xm-tidb-01 192.168.1.1）
yum -y install epel-release git curl sshpass
yum install -y python-pip
升級pip
pip install --upgrade pip
[root@zz-01 /]# pip -V pip 19.0.2 from /usr/lib/python2.7/site-packages/pip (python 2.7)
3.4：創建tidb用戶
useradd -m -d /home/tidb tidb
passwd tidb
3.5：配置tidb用戶sudo免密碼
vim /etc/sudoers
添加 tidb ALL=(ALL) NOPASSWD:ALL 到末尾
用:wq!保持並推出
3.6：切換用戶
su - tidb
3.6：創建tidb用戶ssh key
ssh-keygen -t rsa
[tidb@xm-tidb-01 ~]$ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/home/tidb/.ssh/id_rsa): Created directory '/home/tidb/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/tidb/.ssh/id_rsa. Your public key has been saved in /home/tidb/.ssh/id_rsa.pub. The key fingerprint is: SHA256:7oWdyfPJIGh10V8oJR3FB2BbSFJp9W3ZX4G/8leAl/4 tidb@xm-tidb-01 The key's randomart image is: +---[RSA 2048]----+ | .o**B*.| | ++*.oB| | ..+.ooO| | ..o++o| | S . o..o| | + = o ....| | o + O o..| | . . o = . .E| | . + .| +----[SHA256]-----+
四：在中控機部署TiDB-Ansible
4.1：各版本對應關系
| tidb-ansible 分支 | TiDB 版本 | 備注 |
|---|---|---|
| release-2.0 | 2.0 版本 | 最新 2.0 穩定版本，可用於生產環境。 |
| release-2.1 | 2.1 版本 | 最新 2.1 穩定版本，可用於生產環境（建議）。 |
| master | master 版本 | 包含最新特性，每日更新。 |
以tidb用戶進入/home/tidb
4.2：下載對應tidb-ansible版本：（本例下載2.1版本）
release-2.0版本：git clone -b release-2.0 https://github.com/pingcap/tidb-ansible.git
release-2.1版本：git clone -b release-2.1 https://github.com/pingcap/tidb-ansible.git
master版本： git clone https://github.com/pingcap/tidb-ansible.git
4.3：安裝ansible及其依賴：
cd /home/tidb/tidb-ansible/
確定pip版本為19.0.2以上pip -V
[tidb@xm-tidb-01 tidb-ansible]$ pip -V pip 19.0.2 from /usr/lib/python2.7/site-packages/pip (python 2.7)
sudo pip install -r ./requirements.txt
ansible --version
[tidb@xm-tidb-01 tidb-ansible]$ ansible --version ansible 2.6.13 config file = /home/tidb/tidb-ansible/ansible.cfg configured module search path = [u'/home/tidb/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python2.7/site-packages/ansible executable location = /bin/ansible python version = 2.7.5 (default, Oct 30 2018, 23:45:53) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]
五：部署集群ssh互信及sudo規則：（tidb用戶在中控機上操作）
5.1：添加列表
cd /home/tidb/tidb-ansible/
vim hosts.ini
[servers] 192.168.10.221 192.168.10.222 192.168.10.223 192.168.10.224 192.168.10.225 [all:vars] username = tidb ntp_server = pool.ntp.org
5.2： 執行以下命令，按提示輸入部署目標機器 root 用戶密碼。該步驟將在部署目標機器上創建 tidb 用戶，並配置 sudo 規則，配置中控機與部署目標機器之間的 ssh 互信。
ansible-playbook -i hosts.ini create_users.yml -u root -k
[tidb@xm-tidb-01 tidb-ansible]$ ansible-playbook -i hosts.ini create_users.yml -u root -k SSH password: PLAY [all] ************************************************************************************************* TASK [create user] ***************************************************************************************** changed: [192.168.10.225] changed: [192.168.10.224] changed: [192.168.10.223] changed: [192.168.10.222] ok: [192.168.10.221] TASK [set authorized key] ********************************************************************************** changed: [192.168.10.221] changed: [192.168.10.225] changed: [192.168.10.224] changed: [192.168.10.222] changed: [192.168.10.223] TASK [update sudoers file] ********************************************************************************* changed: [192.168.10.221] changed: [192.168.10.223] changed: [192.168.10.224] changed: [192.168.10.222] changed: [192.168.10.225] PLAY RECAP ************************************************************************************************* 192.168.10.221 : ok=3 changed=2 unreachable=0 failed=0 192.168.10.222 : ok=3 changed=3 unreachable=0 failed=0 192.168.10.223 : ok=3 changed=3 unreachable=0 failed=0 192.168.10.224 : ok=3 changed=3 unreachable=0 failed=0 192.168.10.225 : ok=3 changed=3 unreachable=0 failed=0 Congrats! All goes well. :-)
六：在目標機上安裝NTP服務
6.1：cd /home/tidb/tidb-ansible
6.2：ansible-playbook -i hosts.ini deploy_ntp.yml -u tidb -b
如機器未裝ntp服務，腳本會自動安裝
[tidb@xm-tidb-01 tidb-ansible]$ ansible-playbook -i hosts.ini deploy_ntp.yml -u tidb -b PLAY [all] ************************************************************************************************* TASK [get facts] ******************************************************************************************* ok: [192.168.10.221] ok: [192.168.10.225] ok: [192.168.10.224] ok: [192.168.10.222] ok: [192.168.10.223] TASK [RedHat family Linux distribution - make sure ntp, ntpstat have been installed] *********************** changed: [192.168.10.221] => (item=[u'ntp']) changed: [192.168.10.223] => (item=[u'ntp']) changed: [192.168.10.224] => (item=[u'ntp']) changed: [192.168.10.225] => (item=[u'ntp']) changed: [192.168.10.222] => (item=[u'ntp']) TASK [RedHat family Linux distribution - make sure ntpdate have been installed] **************************** ok: [192.168.10.221] => (item=[u'ntpdate']) ok: [192.168.10.222] => (item=[u'ntpdate']) ok: [192.168.10.223] => (item=[u'ntpdate']) ok: [192.168.10.224] => (item=[u'ntpdate']) ok: [192.168.10.225] => (item=[u'ntpdate']) TASK [Debian family Linux distribution - make sure ntp, ntpstat have been installed] *********************** TASK [Debian family Linux distribution - make sure ntpdate have been installed] **************************** TASK [RedHat family Linux distribution - make sure ntpd service has been stopped] ************************** ok: [192.168.10.221] ok: [192.168.10.225] ok: [192.168.10.222] ok: [192.168.10.223] ok: [192.168.10.224] TASK [Debian family Linux distribution - make sure ntp service has been stopped] *************************** TASK [Adjust Time | start to adjust time with pool.ntp.org] ************************************************ changed: [192.168.10.222] changed: [192.168.10.224] changed: [192.168.10.223] changed: [192.168.10.221] changed: [192.168.10.225] TASK [RedHat family Linux distribution - make sure ntpd service has been started] ************************** changed: [192.168.10.221] changed: [192.168.10.222] changed: [192.168.10.223] changed: [192.168.10.224] changed: [192.168.10.225] TASK [Debian family Linux distribution - Make sure ntp service has been started] *************************** PLAY RECAP ************************************************************************************************* 192.168.10.221 : ok=6 changed=3 unreachable=0 failed=0 192.168.10.222 : ok=6 changed=3 unreachable=0 failed=0 192.168.10.223 : ok=6 changed=3 unreachable=0 failed=0 192.168.10.224 : ok=6 changed=3 unreachable=0 failed=0 192.168.10.225 : ok=6 changed=3 unreachable=0 failed=0 Congrats! All goes well. :-)
七：在部署目標機配置cpufreq調節器模式：
7.1查看調節器模式
cpupower frequency-info --governors
[tidb@xm-tidb-01 tidb-ansible]$ cpupower frequency-info --governors analyzing CPU 0: available cpufreq governors: Not Available
返回 “Not Available”，表示當前系統不支持配置 CPUfreq，跳過該步驟即可。
八：分配機器資源，編輯inventory.ini文件：
8.1：vim /home/tidb/tidb-ansible/inventory.ini
## TiDB Cluster Part [tidb_servers] 192.168.10.221 [tikv_servers] 192.168.10.223 192.168.10.224 192.168.10.225 [pd_servers] 192.168.10.222 [spark_master] [spark_slaves] [lightning_server] [importer_server] ## Monitoring Part # prometheus and pushgateway servers [monitoring_servers] 192.168.10.221 [grafana_servers] 192.168.10.221 # node_exporter and blackbox_exporter servers [monitored_servers] 192.168.10.221 192.168.10.222 192.168.10.223 192.168.10.224 192.168.10.225 [alertmanager_servers] [kafka_exporter_servers] ## Binlog Part [pump_servers] [drainer_servers]
8.2：inventory.ini變量調整：
部署目錄調整：
## Global variables [all:vars] #deploy_dir = /home/tidb/deploy deploy_dir = /ext4/deploy
如為某一服務單獨設置部署目錄，可在配置服務主機列表時配置主機變量，以 TiKV 節點為例，其他服務類推，請務必添加第一列別名，以免服務混布時混淆
TiKV1-1 ansible_host=172.16.10.4 deploy_dir=/data1/deploy
8.3：其他變量調整：True、False首字母要大寫
| 變量 | 含義 |
|---|---|
| cluster_name | 集群名稱，可調整 |
| tidb_version | TiDB 版本，TiDB-Ansible 各分支默認已配置 |
| process_supervision | 進程監管方式，默認為 systemd，可選 supervise |
| timezone | 新安裝 TiDB 集群第一次啟動 bootstrap（初始化）時，將 TiDB 全局默認時區設置為該值。TiDB 使用的時區后續可通過 time_zone 全局變量和 session 變量來修改，參考時區支持。 默認為 Asia/Shanghai，可選值參考 timzone 列表。 |
| enable_firewalld | 開啟防火牆，默認不開啟，如需開啟，請將部署建議-網絡要求 中的端口加入白名單 |
| enable_ntpd | 檢測部署目標機器 NTP 服務，默認為 True，請勿關閉 |
| set_hostname | 根據 IP 修改部署目標機器主機名，默認為 False |
| enable_binlog | 是否部署 pump 並開啟 binlog，默認為 False，依賴 Kafka 集群，參見 zookeeper_addrs 變量 |
| zookeeper_addrs | binlog Kafka 集群的 zookeeper 地址 |
| enable_slow_query_log | TiDB 慢查詢日志記錄到單獨文件({{ deploy_dir }}/log/tidb_slow_query.log)，默認為 False，記錄到 tidb 日志 |
| deploy_without_tidb | KV 模式，不部署 TiDB 服務，僅部署 PD、TiKV 及監控服務，請將 inventory.ini 文件中 tidb_servers 主機組 IP 設置為空。 |
| alertmanager_target | 可選：如果你已單獨部署 alertmanager，可配置該變量，格式：alertmanager_host:alertmanager_port |
| grafana_admin_user | Grafana 管理員帳號用戶名，默認為 admin |
| grafana_admin_password | Grafana 管理員帳號密碼，默認為 admin，用於 Ansible 導入 Dashboard 和創建 API Key，如后期通過 grafana web 修改了密碼，請更新此變量 |
| collect_log_recent_hours | 采集日志時，采集最近幾個小時的日志，默認為 2 小時 |
| enable_bandwidth_limit | 在中控機上從部署目標機器拉取診斷數據時，是否限速，默認為 True，與 collect_bandwidth_limit 變量結合使用 |
| collect_bandwidth_limit | 在中控機上從部署目標機器拉取診斷數據時限速多少，單位: Kbit/s，默認 10000，即 10Mb/s，如果是單機多 TiKV 實例部署方式，需除以單機實例個數 |
九：部署任務：
9.1：確認inventory.ini中ansible_user = tidb
## Connection # ssh via normal user ansible_user = tidb
9.2：測試ssh互信
ansible -i inventory.ini all -m shell -a 'whoami'
[tidb@xm-tidb-01 tidb-ansible]$ ansible -i inventory.ini all -m shell -a 'whoami' 192.168.10.224 | SUCCESS | rc=0 >> tidb 192.168.10.223 | SUCCESS | rc=0 >> tidb 192.168.10.221 | SUCCESS | rc=0 >> tidb 192.168.10.222 | SUCCESS | rc=0 >> tidb 192.168.10.225 | SUCCESS | rc=0 >> tidb
9.3：測試sudo免密
ansible -i inventory.ini all -m shell -a 'whoami' -b
[tidb@xm-tidb-01 tidb-ansible]$ ansible -i inventory.ini all -m shell -a 'whoami' -b 192.168.10.224 | SUCCESS | rc=0 >> root 192.168.10.223 | SUCCESS | rc=0 >> root 192.168.10.221 | SUCCESS | rc=0 >> root 192.168.10.222 | SUCCESS | rc=0 >> root 192.168.10.225 | SUCCESS | rc=0 >> root
9.4：下載TiDB binary到中控機：
ansible-playbook local_prepare.yml
[tidb@xm-tidb-01 tidb-ansible]$ ansible-playbook local_prepare.yml
PLAY [do local preparation] ********************************************************************************
TASK [local : Stop if ansible version is too low, make sure that the Ansible version is Ansible 2.4.2 or later, otherwise a compatibility issue occurs.] ***
ok: [localhost] => {
"changed": false,
"msg": "All assertions passed"
}
TASK [local : create downloads and resources directories] **************************************************
changed: [localhost] => (item=/home/tidb/tidb-ansible/downloads)
changed: [localhost] => (item=/home/tidb/tidb-ansible/resources)
changed: [localhost] => (item=/home/tidb/tidb-ansible/resources/bin)
TASK [local : create cert directory] ***********************************************************************
TASK [local : create packages.yml] *************************************************************************
changed: [localhost]
TASK [local : create specific deployment method packages.yml] **********************************************
changed: [localhost]
TASK [local : include_vars] ********************************************************************************
ok: [localhost]
TASK [local : include_vars] ********************************************************************************
ok: [localhost]
TASK [local : detect outbound network] *********************************************************************
ok: [localhost]
TASK [local : set outbound network fact] *******************************************************************
ok: [localhost]
TASK [local : fail] ****************************************************************************************
TASK [local : detect GFW] **********************************************************************************
ok: [localhost]
TASK [local : set GFW fact] ********************************************************************************
ok: [localhost]
TASK [local : download tidb binary] ************************************************************************
FAILED - RETRYING: download tidb binary (4 retries left).
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/tidb-v2.1.4-linux-amd64.tar.gz', u'version': u'v2.1.4', u'name': u'tidb'})
TASK [local : download common binary] **********************************************************************
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/fio-3.8.tar.gz', u'checksum': u'sha256:15739abde7e74b59ac59df57f129b14fc5cd59e1e2eca2ce37b41f8c289c3d58', u'version': 3.8, u'name': u'fio'})
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/grafana_collector-latest-linux-amd64.tar.gz', u'version': u'latest', u'name': u'grafana_collector'})
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/kafka_exporter-1.1.0.linux-amd64.tar.gz', u'version': u'1.1.0', u'name': u'kafka_exporter'})
TASK [local : download diagnosis tools] ********************************************************************
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/tidb-insight-v0.2.5-1-g99b8fea.tar.gz', u'version': u'v0.2.5-1-g99b8fea', u'name': u'tidb-insight'})
TASK [local : download cfssl binary] ***********************************************************************
TASK [local : download cfssljson binary] *******************************************************************
TASK [local : include_tasks] *******************************************************************************
included: /home/tidb/tidb-ansible/roles/local/tasks/binary_deployment.yml for localhost
TASK [local : download other binary] ***********************************************************************
TASK [local : download other binary under gfw] *************************************************************
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/prometheus-2.2.1.linux-amd64.tar.gz', u'version': u'2.2.1', u'name': u'prometheus'})
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/alertmanager-0.14.0.linux-amd64.tar.gz', u'version': u'0.14.0', u'name': u'alertmanager'})
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/node_exporter-0.15.2.linux-amd64.tar.gz', u'version': u'0.15.2', u'name': u'node_exporter'})
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/pushgateway-0.4.0.linux-amd64.tar.gz', u'version': u'0.4.0', u'name': u'pushgateway'})
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/grafana-4.6.3.linux-x64.tar.gz', u'version': u'4.6.3', u'name': u'grafana'})
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/blackbox_exporter-0.12.0.linux-amd64.tar.gz', u'version': u'0.12.0', u'name': u'blackbox_exporter'})
TASK [local : download TiSpark packages] *******************************************************************
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/spark-2.3.2-bin-hadoop2.7.tgz', u'checksum': u'sha256:6246b20d95c7596a29fb26d5b50a3ae3163a35915bec6c515a8e183383bedc43', u'version': u'2.3.2', u'name': u'spark-2.3.2-bin-hadoop2.7.tgz'})
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/tispark-latest-linux-amd64.tar.gz', u'version': u'latest', u'name': u'tispark-latest.tar.gz'})
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/tispark-sample-data.tar.gz', u'version': u'latest', u'name': u'tispark-sample-data.tar.gz'})
TASK [local : unarchive third party binary] ****************************************************************
changed: [localhost] => (item={u'url': u'https://github.com/prometheus/prometheus/releases/download/v2.2.1/prometheus-2.2.1.linux-amd64.tar.gz', u'version': u'2.2.1', u'name': u'prometheus'})
changed: [localhost] => (item={u'url': u'https://github.com/prometheus/alertmanager/releases/download/v0.14.0/alertmanager-0.14.0.linux-amd64.tar.gz', u'version': u'0.14.0', u'name': u'alertmanager'})
changed: [localhost] => (item={u'url': u'https://github.com/prometheus/node_exporter/releases/download/v0.15.2/node_exporter-0.15.2.linux-amd64.tar.gz', u'version': u'0.15.2', u'name': u'node_exporter'})
changed: [localhost] => (item={u'url': u'https://github.com/prometheus/blackbox_exporter/releases/download/v0.12.0/blackbox_exporter-0.12.0.linux-amd64.tar.gz', u'version': u'0.12.0', u'name': u'blackbox_exporter'})
changed: [localhost] => (item={u'url': u'https://github.com/prometheus/pushgateway/releases/download/v0.4.0/pushgateway-0.4.0.linux-amd64.tar.gz', u'version': u'0.4.0', u'name': u'pushgateway'})
changed: [localhost] => (item={u'url': u'https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-4.6.3.linux-x64.tar.gz', u'version': u'4.6.3', u'name': u'grafana'})
TASK [local : unarchive tispark] ***************************************************************************
changed: [localhost]
TASK [local : unarchive tispark-sample-data] ***************************************************************
changed: [localhost]
TASK [local : cp monitoring binary] ************************************************************************
changed: [localhost] => (item=alertmanager)
changed: [localhost] => (item=prometheus)
changed: [localhost] => (item=node_exporter)
changed: [localhost] => (item=pushgateway)
changed: [localhost] => (item=blackbox_exporter)
TASK [local : cp tispark] **********************************************************************************
changed: [localhost]
TASK [local : cp tispark-sample-data] **********************************************************************
changed: [localhost]
TASK [local : unarchive tidb binary] ***********************************************************************
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/tidb-v2.1.4-linux-amd64.tar.gz', u'version': u'v2.1.4', u'name': u'tidb'})
TASK [local : unarchive common binary] *********************************************************************
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/fio-3.8.tar.gz', u'checksum': u'sha256:15739abde7e74b59ac59df57f129b14fc5cd59e1e2eca2ce37b41f8c289c3d58', u'version': 3.8, u'name': u'fio'})
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/grafana_collector-latest-linux-amd64.tar.gz', u'version': u'latest', u'name': u'grafana_collector'})
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/kafka_exporter-1.1.0.linux-amd64.tar.gz', u'version': u'1.1.0', u'name': u'kafka_exporter'})
TASK [local : cp tidb binary] ******************************************************************************
changed: [localhost] => (item={u'url': u'http://download.pingcap.org/tidb-v2.1.4-linux-amd64.tar.gz', u'version': u'v2.1.4', u'name': u'tidb'})
TASK [local : cp fio binary] *******************************************************************************
changed: [localhost] => (item=fio)
TASK [local : cp grafana_collector binary and fonts] *******************************************************
changed: [localhost]
TASK [local : cp kafka_exporter binary] ********************************************************************
changed: [localhost] => (item=kafka_exporter)
TASK [local : cp daemontools binary] ***********************************************************************
TASK [local : cp tidb-insight tarball] *********************************************************************
changed: [localhost]
TASK [local : clean up download dir] ***********************************************************************
changed: [localhost]
PLAY RECAP *************************************************************************************************
localhost : ok=30 changed=22 unreachable=0 failed=0
Congrats! All goes well. :-)
[tidb@xm-tidb-01 tidb-ansible]$
9.5：初始化系統環境
ansible-playbook bootstrap.yml
報錯：內存不足
TASK [check_system_optional : Preflight check - Check TiDB server's RAM] ***********************************
fatal: [192.168.10.221]: FAILED! => {"changed": false, "msg": "This machine does not have sufficient RAM to
run TiDB, at least 16000 MB."}
NO MORE HOSTS LEFT *****************************************************************************************
to retry, use: --limit @/home/tidb/tidb-ansible/retry_files/bootstrap.retry
PLAY RECAP *************************************************************************************************
192.168.10.221 : ok=29 changed=10 unreachable=0 failed=1
192.168.10.222 : ok=29 changed=10 unreachable=0 failed=0
192.168.10.223 : ok=29 changed=10 unreachable=0 failed=0
192.168.10.224 : ok=29 changed=10 unreachable=0 failed=0
192.168.10.225 : ok=29 changed=10 unreachable=0 failed=0
localhost : ok=1 changed=0 unreachable=0 failed=0
ERROR MESSAGE SUMMARY **************************************************************************************
[192.168.10.221]: Ansible FAILED! => playbook: bootstrap.yml; TASK: check_system_optional : Preflight
check - Check TiDB server's RAM; message: {"changed": false, "msg": "This machine does not have sufficient
RAM to run TiDB, at least 16000 MB."}
Ask for help:
Contact us: support@pingcap.com
It seems that you encounter some problems. You can send an email to the above email address, attached with
the tidb-ansible/inventory.ini and tidb-ansible/log/ansible.log files and the error message, or new issue
on https://github.com/pingcap/tidb-ansible/issues. We'll try our best to help you deploy a TiDB cluster.
Thanks. :-)
[tidb@xm-tidb-01 tidb-ansible]$
修改文件：
vim bootstrap.yml
注銷掉：
- { role: check_system_optional, when: not dev_mode|default(false) }
- { role: machine_benchmark, when: not dev_mode|default(false) }
- name: check system
hosts: all
any_errors_fatal: true
roles:
- check_system_static
# - { role: check_system_optional, when: not dev_mode|default(false) }
- name: tikv_servers machine benchmark
hosts: tikv_servers
gather_facts: false
roles:
# - { role: machine_benchmark, when: not dev_mode|default(false) }
再次運行：
ansible-playbook bootstrap.yml
[tidb@xm-tidb-01 tidb-ansible]$ ansible-playbook bootstrap.yml
PLAY [initializing deployment target] **********************************************************************
TASK [check_config_static : Ensure only one monitoring host exists] ****************************************
TASK [check_config_static : Ensure monitored_servers exists] ***********************************************
TASK [check_config_static : Ensure TiDB host exists] *******************************************************
TASK [check_config_static : Ensure PD host exists] *********************************************************
TASK [check_config_static : Ensure TiKV host exists] *******************************************************
TASK [check_config_static : Check ansible_user variable] ***************************************************
TASK [check_config_static : Ensure timezone variable is set] ***********************************************
TASK [check_config_static : Close old SSH control master processes] ****************************************
ok: [localhost]
PLAY [check node config] ***********************************************************************************
TASK [pre-ansible : disk space check - fail when disk is full] *********************************************
ok: [192.168.10.221]
ok: [192.168.10.222]
ok: [192.168.10.223]
ok: [192.168.10.224]
ok: [192.168.10.225]
TASK [pre-ansible : Get distro name from /etc/os-release] **************************************************
ok: [192.168.10.221]
ok: [192.168.10.222]
ok: [192.168.10.223]
ok: [192.168.10.224]
ok: [192.168.10.225]
TASK [pre-ansible : set distro facts] **********************************************************************
ok: [192.168.10.221]
ok: [192.168.10.222]
ok: [192.168.10.223]
ok: [192.168.10.224]
ok: [192.168.10.225]
TASK [pre-ansible : python check] **************************************************************************
ok: [192.168.10.221]
ok: [192.168.10.222]
ok: [192.168.10.223]
ok: [192.168.10.224]
ok: [192.168.10.225]
TASK [pre-ansible : set has_python facts] ******************************************************************
ok: [192.168.10.221]
ok: [192.168.10.222]
ok: [192.168.10.223]
ok: [192.168.10.224]
ok: [192.168.10.225]
TASK [pre-ansible : set has_python facts] ******************************************************************
TASK [pre-ansible : include_tasks] *************************************************************************
TASK [pre-ansible : include_tasks] *************************************************************************
included: /home/tidb/tidb-ansible/roles/pre-ansible/tasks/root_tasks.yml for 192.168.10.221, 192.168.10.222, 192.168.10.223, 192.168.10.224, 192.168.10.225
TASK [pre-ansible : Debian/Ubuntu - install python] ********************************************************
TASK [pre-ansible : Redhat/CentOS - install python] ********************************************************
TASK [pre-ansible : Redhat/CentOS - Make sure ntp, ntpstat have been installed] ****************************
ok: [192.168.10.221] => (item=[u'ntp'])
ok: [192.168.10.222] => (item=[u'ntp'])
ok: [192.168.10.224] => (item=[u'ntp'])
ok: [192.168.10.223] => (item=[u'ntp'])
ok: [192.168.10.225] => (item=[u'ntp'])
TASK [pre-ansible : Debian/Ubuntu - Make sure ntp, ntpstat have been installed] ****************************
TASK [bootstrap : gather facts] ****************************************************************************
ok: [192.168.10.225]
ok: [192.168.10.222]
ok: [192.168.10.221]
ok: [192.168.10.223]
ok: [192.168.10.224]
TASK [bootstrap : group hosts by distribution] *************************************************************
ok: [192.168.10.221]
ok: [192.168.10.222]
ok: [192.168.10.223]
ok: [192.168.10.224]
ok: [192.168.10.225]
TASK [bootstrap : Set deploy_dir if not presented] *********************************************************
TASK [bootstrap : include_tasks] ***************************************************************************
included: /home/tidb/tidb-ansible/roles/bootstrap/tasks/root_tasks.yml for 192.168.10.221, 192.168.10.222, 192.168.10.223, 192.168.10.224, 192.168.10.225
TASK [bootstrap : setting absent kernel params] ************************************************************
ok: [192.168.10.222] => (item={u'name': u'net.ipv4.tcp_tw_recycle', u'value': 0})
ok: [192.168.10.225] => (item={u'name': u'net.ipv4.tcp_tw_recycle', u'value': 0})
ok: [192.168.10.223] => (item={u'name': u'net.ipv4.tcp_tw_recycle', u'value': 0})
ok: [192.168.10.221] => (item={u'name': u'net.ipv4.tcp_tw_recycle', u'value': 0})
ok: [192.168.10.224] => (item={u'name': u'net.ipv4.tcp_tw_recycle', u'value': 0})
TASK [bootstrap : setting present kernel params] ***********************************************************
ok: [192.168.10.221] => (item={u'name': u'net.core.somaxconn', u'value': 32768})
ok: [192.168.10.223] => (item={u'name': u'net.core.somaxconn', u'value': 32768})
ok: [192.168.10.224] => (item={u'name': u'net.core.somaxconn', u'value': 32768})
ok: [192.168.10.225] => (item={u'name': u'net.core.somaxconn', u'value': 32768})
ok: [192.168.10.222] => (item={u'name': u'net.core.somaxconn', u'value': 32768})
ok: [192.168.10.221] => (item={u'name': u'vm.swappiness', u'value': 0})
ok: [192.168.10.223] => (item={u'name': u'vm.swappiness', u'value': 0})
ok: [192.168.10.225] => (item={u'name': u'vm.swappiness', u'value': 0})
ok: [192.168.10.224] => (item={u'name': u'vm.swappiness', u'value': 0})
ok: [192.168.10.222] => (item={u'name': u'vm.swappiness', u'value': 0})
ok: [192.168.10.221] => (item={u'name': u'net.ipv4.tcp_syncookies', u'value': 0})
ok: [192.168.10.223] => (item={u'name': u'net.ipv4.tcp_syncookies', u'value': 0})
ok: [192.168.10.225] => (item={u'name': u'net.ipv4.tcp_syncookies', u'value': 0})
ok: [192.168.10.224] => (item={u'name': u'net.ipv4.tcp_syncookies', u'value': 0})
ok: [192.168.10.222] => (item={u'name': u'net.ipv4.tcp_syncookies', u'value': 0})
ok: [192.168.10.221] => (item={u'name': u'fs.file-max', u'value': 1000000})
ok: [192.168.10.223] => (item={u'name': u'fs.file-max', u'value': 1000000})
ok: [192.168.10.224] => (item={u'name': u'fs.file-max', u'value': 1000000})
ok: [192.168.10.222] => (item={u'name': u'fs.file-max', u'value': 1000000})
ok: [192.168.10.225] => (item={u'name': u'fs.file-max', u'value': 1000000})
TASK [bootstrap : update /etc/security/limits.conf] ********************************************************
ok: [192.168.10.221]
ok: [192.168.10.224]
ok: [192.168.10.223]
ok: [192.168.10.225]
ok: [192.168.10.222]
TASK [bootstrap : disable swap] ****************************************************************************
TASK [bootstrap : create group] ****************************************************************************
ok: [192.168.10.221]
ok: [192.168.10.222]
ok: [192.168.10.224]
ok: [192.168.10.225]
ok: [192.168.10.223]
TASK [bootstrap : create account] **************************************************************************
ok: [192.168.10.222]
ok: [192.168.10.221]
ok: [192.168.10.224]
ok: [192.168.10.225]
ok: [192.168.10.223]
TASK [bootstrap : create top deploy dir when under root] ***************************************************
ok: [192.168.10.221]
ok: [192.168.10.224]
ok: [192.168.10.223]
ok: [192.168.10.222]
ok: [192.168.10.225]
TASK [bootstrap : create wal_dir deploy dir when under root] ***********************************************
TASK [bootstrap : create raftdb_path deploy dir when under root] *******************************************
TASK [bootstrap : set hostname if hostname is not distinguishable] *****************************************
TASK [bootstrap : set hostname in hosts file] **************************************************************
TASK [bootstrap : determine if firewalld is running] *******************************************************
ok: [192.168.10.223]
ok: [192.168.10.221]
ok: [192.168.10.225]
ok: [192.168.10.224]
ok: [192.168.10.222]
TASK [bootstrap : disable firewalld] ***********************************************************************
TASK [bootstrap : or to enable firewalld] ******************************************************************
TASK [bootstrap : check centos configuration file exists] **************************************************
ok: [192.168.10.222]
ok: [192.168.10.221]
ok: [192.168.10.225]
ok: [192.168.10.224]
ok: [192.168.10.223]
TASK [bootstrap : check debian configuration file exists] **************************************************
ok: [192.168.10.221]
ok: [192.168.10.222]
ok: [192.168.10.223]
ok: [192.168.10.224]
ok: [192.168.10.225]
TASK [bootstrap : modify centos irqbalance configuration file] *********************************************
ok: [192.168.10.221]
ok: [192.168.10.224]
ok: [192.168.10.223]
ok: [192.168.10.225]
ok: [192.168.10.222]
TASK [bootstrap : modify debian irqbalance configuration file] *********************************************
TASK [bootstrap : start irqbalance service] ****************************************************************
ok: [192.168.10.221]
ok: [192.168.10.222]
ok: [192.168.10.224]
ok: [192.168.10.225]
ok: [192.168.10.223]
PLAY [check system] ****************************************************************************************
TASK [check_system_static : Disk space check - Fail task when disk is full] ********************************
ok: [192.168.10.221]
ok: [192.168.10.222]
ok: [192.168.10.223]
ok: [192.168.10.224]
ok: [192.168.10.225]
TASK [check_system_static : get facts] *********************************************************************
ok: [192.168.10.221]
ok: [192.168.10.225]
ok: [192.168.10.224]
ok: [192.168.10.222]
ok: [192.168.10.223]
TASK [check_system_static : Preflight check - Linux OS family and distribution version] ********************
TASK [check_system_static : Deploy check_cpufreq script] ***************************************************
changed: [192.168.10.221]
changed: [192.168.10.224]
changed: [192.168.10.223]
changed: [192.168.10.225]
changed: [192.168.10.222]
TASK [check_system_static : Preflight check - Check CPUfreq governors available in the kernel] *************
changed: [192.168.10.221]
changed: [192.168.10.222]
changed: [192.168.10.223]
changed: [192.168.10.224]
changed: [192.168.10.225]
TASK [check_system_static : Preflight check - Check the currently active governor] *************************
changed: [192.168.10.224]
changed: [192.168.10.223]
changed: [192.168.10.222]
changed: [192.168.10.225]
changed: [192.168.10.221]
TASK [check_system_static : Preflight check - Fail when CPU frequency governor is not set to performance mode] ***
TASK [check_system_static : Clean check_cpufreq script] ****************************************************
changed: [192.168.10.221]
changed: [192.168.10.222]
changed: [192.168.10.223]
changed: [192.168.10.224]
changed: [192.168.10.225]
TASK [check_system_static : Preflight check - Check Linux kernel overcommit_memory parameter] **************
changed: [192.168.10.221]
changed: [192.168.10.223]
changed: [192.168.10.222]
changed: [192.168.10.225]
changed: [192.168.10.224]
TASK [check_system_static : Preflight check - Fail when Linux kernel vm.overcommit_memory parameter is set to 2] ***
PLAY [tikv_servers machine benchmark] **********************************************************************
PLAY [create ops scripts] **********************************************************************************
TASK [ops : create check_tikv.sh script] *******************************************************************
changed: [localhost]
TASK [ops : create pd-ctl.sh script] ***********************************************************************
changed: [localhost]
PLAY RECAP *************************************************************************************************
192.168.10.221 : ok=28 changed=5 unreachable=0 failed=0
192.168.10.222 : ok=28 changed=5 unreachable=0 failed=0
192.168.10.223 : ok=28 changed=5 unreachable=0 failed=0
192.168.10.224 : ok=28 changed=5 unreachable=0 failed=0
192.168.10.225 : ok=28 changed=5 unreachable=0 failed=0
localhost : ok=3 changed=2 unreachable=0 failed=0
Congrats! All goes well. :-)
9.6：部署TiDB集群軟件：
ansible-playbook deploy.yml
過程比較多，請耐心等待
PLAY RECAP ************************************************************************************************* 192.168.10.221 : ok=115 changed=60 unreachable=0 failed=0 192.168.10.222 : ok=52 changed=23 unreachable=0 failed=0 192.168.10.223 : ok=60 changed=24 unreachable=0 failed=0 192.168.10.224 : ok=60 changed=24 unreachable=0 failed=0 192.168.10.225 : ok=63 changed=26 unreachable=0 failed=0 localhost : ok=1 changed=0 unreachable=0 failed=0 Congrats! All goes well. :-) [tidb@xm-tidb-01 tidb-ansible]$
注：Grafana Dashboard 上的 Report 按鈕可用來生成 PDF 文件，此功能依賴 fontconfig 包和英文字體。如需使用該功能，登錄 grafana_servers 機器，用以下命令安裝：
sudo yum install fontconfig open-sans-fonts
9.7：啟動集群：
！！！先切換到tidb用戶！！！
su tidb
cd /home/tidb/tidb-ansible
ansible-playbook start.yml
PLAY RECAP ************************************************************************************************* 192.168.10.221 : ok=33 changed=12 unreachable=0 failed=0 192.168.10.222 : ok=12 changed=3 unreachable=0 failed=0 192.168.10.223 : ok=14 changed=3 unreachable=0 failed=0 192.168.10.224 : ok=14 changed=3 unreachable=0 failed=0 192.168.10.225 : ok=14 changed=3 unreachable=0 failed=0 localhost : ok=1 changed=0 unreachable=0 failed=0 Congrats! All goes well. :-) [tidb@xm-tidb-01 tidb-ansible]$
9.8： 測試集群：
在安裝有mysql的其他服務器做連接測試
mysql -u root -h 192.168.10.221 -P 4000
-u和-h后的參數間不可有空格，否則會報錯
mysql -uroot -pxxx -h 192.168.10.221 -P 4000
[root@zabbix ~]# mysql -u root -h 192.168.10.221 -P 4000 Welcome to the MariaDB monitor. Commands end with ; or \g. Your MySQL connection id is 51 Server version: 5.7.10-TiDB-v2.1.4 MySQL Community Server (Apache License 2.0) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MySQL [(none)]>
http://192.168.10.221:3000/
admin admin
感謝：
官方文檔:
https://pingcap.com/docs-cn/overview/
https://blog.csdn.net/xujiamin0022016/article/details/83507038