背景介紹：
有三台CentOS7服務器安裝了ClickHouse
| HostName | IP | 安裝程序 | 實例1端口 | 實例2端口 |
| centf8118.sharding1.db | 192.168.81.18 | clickhouse-server，clickhouse-client | 9000 | 9002 |
| centf8119.sharding2.db | 192.168.81.19 | clickhouse-server，clickhouse-client | 9000 | 9002 |
| centf8120.sharding3.db | 192.168.81.20 | clickhouse-server，clickhouse-client | 9000 | 9002 |
安裝多實例是為了用三台服務器測試3分片2備份集群。最后的部署如下表：
| 備份1 | 備份2 | |
| 分片1 | 192.168.81.18：9000 | 192.168.81.19：9002 |
| 分片2 | 192.168.81.19：9000 | 192.168.81.20：9002 |
| 分片3 | 192.168.81.20：9000 | 192.168.81.18：9002 |
一：安裝clickhouse
【ClickHouse】1：clickhouse安裝 （CentOS7）
二：新增一個clickhouse實例服務
2.1：將/etc/clickhouse-server/config.xml文件拷貝一份改名
[root@centf8118 clickhouse-server]# cp /etc/clickhouse-server/config.xml /etc/clickhouse-server/config9002.xml
2.2：編輯/etc/clickhouse-server/config9002.xml更改以下內容將兩個服務區分開來
多實例修改的config9002.xml：原來內容 <log>/var/log/clickhouse-server/clickhouse-server.log</log> <errorlog>/var/log/clickhouse-server/clickhouse-server.err.log</errorlog> <http_port>8123</http_port> <tcp_port>9000</tcp_port> <mysql_port>9004</mysql_port> <interserver_http_port>9009</interserver_http_port> <path>/data/clickhouse/</path> <tmp_path>/data/clickhouse/tmp/</tmp_path> <user_files_path>/data/clickhouse/user_files/</user_files_path> <access_control_path>/data/clickhouse/access/</access_control_path> <include_from>/etc/clickhouse-server/metrika.xml</include_from> #集群配置文件 多實例修改的config9002.xml：調整后內容 <log>/var/log/clickhouse-server/clickhouse-server-9002.log</log> <errorlog>/var/log/clickhouse-server/clickhouse-server-9002.err.log</errorlog> <http_port>8124</http_port> <tcp_port>9002</tcp_port> <mysql_port>9005</mysql_port> <interserver_http_port>9010</interserver_http_port> <interserver_http_port>9009</interserver_http_port> <path>/data/clickhouse9002/</path> <tmp_path>/data/clickhouse9002/tmp/</tmp_path> <user_files_path>/data/clickhouse9002/user_files/</user_files_path> <access_control_path>/data/clickhouse9002/access/</access_control_path> <include_from>/etc/clickhouse-server/metrika9002.xml</include_from>
2.3：創建對應的目錄
[root@centf8118 data]# mkdir -p /data/clickhouse9002 [root@centf8118 data]# chown -R clickhouse:clickhouse /data/clickhouse9002
PS: 一定要記得更改目錄的所屬組和用戶為clickhouse。
2.4：增加實例對應的服務啟動腳本
[root@centf8118 init.d]# cp /etc/init.d/clickhouse-server /etc/init.d/clickhouse-server9002 [root@centf8118 init.d]# vim /etc/init.d/clickhouse-server9002 調整內容如下： 調整后內容： CLICKHOUSE_CONFIG=$CLICKHOUSE_CONFDIR/config9002.xml CLICKHOUSE_PIDFILE="$CLICKHOUSE_PIDDIR/$PROGRAM-9002.pid" 調整前內容： CLICKHOUSE_CONFIG=$CLICKHOUSE_CONFDIR/config.xml CLICKHOUSE_PIDFILE="$CLICKHOUSE_PIDDIR/$PROGRAM.pid"
2.5：centf81.18完成上述操作后，在其他兩台服務器做以上完全一樣的操作。
三：集群配置（三分片兩備份)
3.1:六個metrika*.xml共同部分：
<yandex>
<!-- 集群配置 -->
<clickhouse_remote_servers>
<!-- 3分片2備份 -->
<xinchen_3shards_2replicas>
<shard>
<weight>1</weight>
<internal_replication>true</internal_replication>
<replica>
<host>192.168.81.18</host>
<port>9000</port>
</replica>
<replica>
<host>192.168.81.19</host>
<port>9002</port>
</replica>
</shard>
<shard>
<weight>1</weight>
<internal_replication>true</internal_replication>
<replica>
<host>192.168.81.19</host>
<port>9000</port>
</replica>
<replica>
<host>192.168.81.20</host>
<port>9002</port>
</replica>
</shard>
<shard>
<weight>1</weight>
<internal_replication>true</internal_replication>
<replica>
<host>192.168.81.20</host>
<port>9000</port>
</replica>
<replica>
<host>192.168.81.18</host>
<port>9002</port>
</replica>
</shard>
</xinchen_3shards_2replicas>
</clickhouse_remote_servers>
<!-- zookeeper 配置 -->
<zookeeper-servers>
<node index="1">
<host>192.168.81.18</host>
<port>4181</port>
</node>
<node index="2">
<host>192.168.81.19</host>
<port>4181</port>
</node>
<node index="3">
<host>192.168.81.20</host>
<port>4181</port>
</node>
</zookeeper-servers>
<!-- macros配置 -->
<macros>
<!-- <replica>192.168.81.18</replica> -->
<layer>01</layer>
<shard>01</shard>
<replica>cluster01-01-1</replica>
</macros>
<networks>
<ip>::/0</ip>
</networks>
<clickhouse_compression>
<case>
<min_part_size>10000000000</min_part_size>
<min_part_size_ratio>0.01</min_part_size_ratio>
<method>lz4</method>
</case>
</clickhouse_compression>
</yandex>
3.2:metrika*.xml不同部分修改如下：
centf81.18實例1(端口:9000)對應metrika.xml調整： <macros> <!-- <replica>centf81.18</replica> --> <layer>01</layer> <shard>01</shard> <replica>cluster01-01-1</replica> </macros> centf81.18實例2(端口:9002)對應metrika9002.xml調整： <macros> <!-- <replica>centf81.18</replica> --> <layer>01</layer> <shard>03</shard> <replica>cluster01-03-2</replica> </macros> centf81.19實例1(端口:9000)對應metrika.xml調整： <macros> <!-- <replica>centf81.19</replica> --> <layer>01</layer> <shard>02</shard> <replica>cluster01-02-1</replica> </macros> centf81.19實例2(端口:9002)對應metrika9002.xml調整： <macros> <!-- <replica>centf81.19</replica> --> <layer>01</layer> <shard>01</shard> <replica>cluster01-01-2</replica> </macros> centf81.20實例1(端口:9000)對應metrika.xml調整： <macros> <!-- <replica>centf81.20</replica> --> <layer>01</layer> <shard>03</shard> <replica>cluster01-03-1</replica> </macros> centf81.20實例2(端口:9002)對應metrika9002.xml調整： <macros> <!-- <replica>centf81.20</replica> --> <layer>01</layer> <shard>02</shard> <replica>cluster01-02-2</replica> </macros>
說明：這其中的規律顯而易見，這里不再說明。如果還不明白可以參照開頭的兩個表格內容便於理解。
其中layer是雙級分片設置，這里是01；然后是shard表示分片編號；最后是replica是副本標識。
這里使用了cluster{layer}-{shard}-{replica}的表示方式，比如cluster01-02-1表示cluster01集群的02分片下的1號副本，這樣既非常直觀的表示又唯一確定副本。
額外提醒：如果一直是跟着我前面的教程來操作的。在這里操作前，先把之前建立的分區表都刪掉。否則會導致服務啟動失敗。
因為這里macros中的備份名稱參數值改了：原來是01/02/03的，現在改為cluster01-01-1這種形式。這樣會導致之前已建好的分區表找不到對應的備份數據。
<macros>
<!-- <replica>192.168.81.18</replica> -->
<layer>01</layer>
<shard>01</shard>
<replica>cluster01-01-1</replica>
</macros>
3.3:啟動高可用clickhouse集群
[root@centf8118 clickhouse-server]# /etc/init.d/clickhouse-server start
[root@centf8118 clickhouse-server]# /etc/init.d/clickhouse-server9002 start
在三個節點上都執行上面的腳本。
3.4:登錄數據庫查看集群信息
centf8119.sharding2.db :) select * from system.clusters; SELECT * FROM system.clusters ┌─cluster───────────────────────────┬─shard_num─┬─shard_weight─┬─replica_num─┬─host_name───┬─host_address─┬─port─┬─is_local─┬─user────┬─default_database─┬─errors_count─┬─estimated_recovery_time─┐ │ xinchen_3shards_2replicas │ 1 │ 1 │ 1 │192.168.81.18│192.168.81.18 │ 9000 │ 0 │ default │ │ 0 │ 0 │ │ xinchen_3shards_2replicas │ 1 │ 1 │ 2 │192.168.81.19│192.168.81.19 │ 9002 │ 0 │ default │ │ 0 │ 0 │ │ xinchen_3shards_2replicas │ 2 │ 1 │ 1 │192.168.81.19│192.168.81.19 │ 9000 │ 1 │ default │ │ 0 │ 0 │ │ xinchen_3shards_2replicas │ 2 │ 1 │ 2 │192.168.81.20│192.168.81.20 │ 9002 │ 0 │ default │ │ 0 │ 0 │ │ xinchen_3shards_2replicas │ 3 │ 1 │ 1 │192.168.81.20│192.168.81.20 │ 9000 │ 0 │ default │ │ 0 │ 0 │ │ xinchen_3shards_2replicas │ 3 │ 1 │ 2 │192.168.81.18│192.168.81.18 │ 9002 │ 0 │ default │ │ 0 │ 0 │ └───────────────────────────────────┴───────────┴──────────────┴─────────────┴─────────────┴──────────────┴──────┴──────────┴─────────┴──────────────────┴──────────────┴─────────────────────────┘
登錄每一個實例，查看集群信息驗證配置是否正確。都符合自己之前的預設，說明集群配置成功了。
clickhouse-client --host 10.30.81.18 --port 9000 clickhouse-client --host 10.30.81.18 --port 9002 clickhouse-client --host 10.30.81.19 --port 9000 clickhouse-client --host 10.30.81.19 --port 9002 clickhouse-client --host 10.30.81.20 --port 9000 clickhouse-client --host 10.30.81.20 --port 9002
三：集群高可用驗證。
3.1：高可用原理
zookeeper+ReplicatedMergeTree(復制表)+Distributed(分布式表)
3.2：首先創建ReplicatedMergeTree引擎表。
需要在三個節點六個實例中都創建，創建sql如下：
CREATE TABLE test_clusters_ha\ (\ dt Date,\ path String \ )\ ENGINE = ReplicatedMergeTree('/clickhouse/tables/{layer}-{shard}/test_clusters_ha','{replica}',dt, dt, 8192);
解釋：
ENGINE = ReplicatedMergeTree('/clickhouse/tables/{layer}-{shard}/test_clusters_ha','{replica}',dt, dt, 8192);
第一個參數為ZooKeeper中該表的路徑。
第二個參數為ZooKeeper中該表的副本名稱。
這里的{layer}，{shard}和{replica}就是metrika*.xml中的macros標簽對應的值。
3.3：創建分布表。
這里只要一個節點創建就行了，假定在sharding1上的9000實例創建。
CREATE TABLE test_clusters_ha_all AS test_clusters_ha ENGINE = Distributed(xinchen_3shards_2replicas, default, test_clusters_ha, rand());
3.4：插入並查看數據
insert into test_clusters_ha_all values('2020-09-01','path1'); insert into test_clusters_ha_all values('2020-09-02','path2'); insert into test_clusters_ha_all values('2020-09-03','path3'); insert into test_clusters_ha_all values('2020-09-04','path4'); insert into test_clusters_ha_all values('2020-09-05','path5'); insert into test_clusters_ha_all values('2020-09-06','path6'); insert into test_clusters_ha_all values('2020-09-07','path7'); insert into test_clusters_ha_all values('2020-09-08','path8'); insert into test_clusters_ha_all values('2020-09-09','path9');
查看數據結果就不貼出來了。總數據9條，分布在三個9000實例的分片中。對應9002是三個分片的備份副本。
3.5：驗證某個節點宕機現有數據查詢一致性
a.將sharding2上的兩個實例服務全部停止，模擬sharding2節點宕機。
[root@centf8119 ~]# service clickhouse-server stop Stop clickhouse-server service: DONE [root@centf8119 ~]# service clickhouse-server9002 stop Stop clickhouse-server service: DONE
b.先驗證在分布式表中查詢數據總量。
centf8118.sharding1.db :) select count(*) from test_clusters_ha_all; SELECT count(*) FROM test_clusters_ha_all ┌─count()─┐ │ 9 │ └─────────┘ 1 rows in set. Elapsed: 0.010 sec.
結果是單節點宕機數據一致性得到保證。同理如果只是宕機sharding3也是一樣的結果。
c.如果sharding2和sharding3同時宕機會如何呢?
[root@centf8120 ~]# service clickhouse-server stop Stop clickhouse-server service: DONE [root@centf8120 ~]# service clickhouse-server9002 stop Stop clickhouse-server service: DONE
d.停掉sharding3兩個實例后，再查詢分布表：
centf8118.sharding1.db :) select count(*) from test_clusters_ha_all; SELECT count(*) FROM test_clusters_ha_all ↗ Progress: 2.00 rows, 8.21 KB (17.20 rows/s., 70.58 KB/s.) Received exception from server (version 20.6.4): Code: 279. DB::Exception: Received from 192.168.81.18:9000. DB::Exception: All connection tries failed. Log: Code: 32, e.displayText() = DB::Exception: Attempt to read after eof (version 20.6.4.44 (official build)) Code: 210, e.displayText() = DB::NetException: Connection refused (192.168.81.19:9000) (version 20.6.4.44 (official build)) Code: 210, e.displayText() = DB::NetException: Connection refused (192.168.81.20:9002) (version 20.6.4.44 (official build)) Code: 210, e.displayText() = DB::NetException: Connection refused (192.168.81.19:9000) (version 20.6.4.44 (official build)) Code: 210, e.displayText() = DB::NetException: Connection refused (192.168.81.20:9002) (version 20.6.4.44 (official build)) Code: 210, e.displayText() = DB::NetException: Connection refused (192.168.81.19:9000) (version 20.6.4.44 (official build)) : While executing Remote. 0 rows in set. Elapsed: 0.119 sec.
直接報錯，可見在該方案中，sharding1作為主節點不能宕機，sharding2，sharding3只允許一個節點宕機。
e. 然后啟動sharding3的兩個實例。再查看查詢分布表是否正常。（能查到數據）
f.在sharding1分布表中插入9條新的數據。再查看sharding1和sharding3的本地表是否有新增數據。 （有新數據）
sharding1: 9000查看分區表： 總共18條數據
本地表( sharding1:9000 + sharding3:9000 + sharding3:9002 ) = 18條數據 = 分區表數據
g.然后重啟sharding2:9000服務，查看sharding2:9000本地表數據是否和sharding3:9002本地表數據一致？ （是一致的)
h.最后重啟sharding2:9002服務，查看sharding2:9002本地表數據是否和sharding1:9000本地表數據一致？ （是一致的)