Hadoop生態圈-CentOs7.5單機部署ClickHouse
作者：尹正傑
版權聲明：原創作品，謝絕轉載！否則將追究法律責任。
到了新的公司，認識了新的同事，生產環境也得你去適應新的集群環境，我新入職的公司的大數據開發同時並不看好hbase，而是對clickhouse青睞有加，聽他們口頭把ClickHouse吹的那么的出神入化，聽的我是有種迫不及待想要對它一探究竟。
最新的一手資料請參考官網：https://clickhouse.yandex/
一.ClickHouse的獨特功能
1>.真正的面向列的DBMS
因為有些系統可以單獨存儲不同列的值，但由於它們針對其他場景進行了優化，因此無法有效地處理分析查詢。例如HBase，BigTable，Cassandra和HyperTable。在這些系統中，您將獲得大約每秒十萬行的吞吐量，但不會達到每秒數億行。
值得注意的是，ClickHouse是一個數據庫管理系統，而不是一個數據庫。ClickHouse允許在運行時創建表和數據庫，加載數據和運行查詢，而無需重新配置和重新啟動服務器。
2>.適用於在線查詢
低延遲意味着可以在沒有延遲的情況下處理查詢，而無需在加載用戶界面頁面的同一時間提前准備答案。換句話說，在線。
3>.數據復制和數據完整性支持
使用異步多主機復制。寫入任何可用副本后，數據將分發到后台的所有剩余副本。系統在不同的副本上維護相同的數據。在大多數故障后自動執行恢復，在復雜情況下 - 半自動執行。
4>.支持近似計算
ClickHouse提供了各種交易性能准確度的方法：
4.1>.聚合函數，用於近似計算不同值，中位數和分位數的數量。
4.2>.基於數據的一部分（樣本）運行查詢並獲得近似結果。在這種情況下，從磁盤中檢索的數據比例較少。
4.2>.為有限數量的隨機密鑰運行聚合，而不是為所有密鑰運行聚合。在數據中密鑰分發的某些條件下，這在使用較少資源的同時提供了相當准確的結果。
5>.SQL支持
ClickHouse支持基於SQL的聲明性查詢語言，在許多情況下與SQL標准相同。支持的查詢包括GROUP BY，ORDER BY，FROM，IN和JOIN子句中的子查詢以及標量子查詢。不支持從屬子查詢和窗口函數。
以上說明並非我個人對ClickHouse的誇張說法，而是摘自官方文檔的說明，可能官網對其軟件有一定的誇張手法在里面，官網文檔請參考：https://clickhouse.yandex/docs/en/
二.部署ClickHouse
推薦的安裝方法參考：
https://github.com/Altinity/clickhouse-rpm-install。文檔寫的頭頭是道，說下載一個安裝shell腳本，結果你按照它的方法做會跑出一系列的坑，我在這里就不吐槽了，喜歡被虐的小伙伴可以去試試，我生產環境用的操作系統基本上都是CentOs的操作系統，我參考了網上多位網友的部署方法，找到了一種最簡單有效的部署方法，我這里一句把坑給大家踩過了，只要我的按照步驟來部署就能成功！
1>.操作平台介紹
[root@yinzhengjie ~]# ip a | grep global | awk '{print $2}' | awk -F '/' '{print $1}' 10.1.2.105 [root@yinzhengjie ~]# [root@yinzhengjie ~]# hostname yinzhengjie [root@yinzhengjie ~]# [root@yinzhengjie ~]# cat /proc/cpuinfo| grep "processor"| wc -l 14 [root@yinzhengjie ~]# [root@yinzhengjie ~]# cat /proc/cpuinfo| grep "cpu cores"| uniq cpu cores : 7 [root@yinzhengjie ~]# [root@yinzhengjie ~]# cat /proc/cpuinfo| grep "physical id"| sort| uniq| wc -l 2 [root@yinzhengjie ~]# [root@yinzhengjie ~]# cat /proc/cpuinfo | grep name | cut -f2 -d: | uniq -c 14 Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz [root@yinzhengjie ~]# [root@yinzhengjie ~]# free -m total used free shared buff/cache available Mem: 11852 1406 8309 103 2136 9954 Swap: 6015 0 6015 [root@yinzhengjie ~]# [root@yinzhengjie ~]# free -g total used free shared buff/cache available Mem: 11 1 8 0 2 9 Swap: 5 0 5 [root@yinzhengjie ~]# [root@yinzhengjie ~]# [root@yinzhengjie ~]# uname -r 3.10.0-862.el7.x86_64 [root@yinzhengjie ~]# [root@yinzhengjie ~]# uname -m x86_64 [root@yinzhengjie ~]# [root@yinzhengjie ~]# cat /etc/redhat-release CentOS Linux release 7.5.1804 (Core) You have new mail in /var/spool/mail/root [root@yinzhengjie ~]#
2>.
檢查是否支持SSE4.2
[root@yinzhengjie ~]# grep -q sse4_2 /proc/cpuinfo && echo "SSE 4.2 supported" || echo "SSE 4.2 not supported" SSE 4.2 supported [root@yinzhengjie ~]#
3>.下載依賴包
我已經將上面的rpm包下載好了，以及需要的依賴包都下載好啦～大家可以直接下載來用即可。百度雲鏈接:https://pan.baidu.com/s/16gPGxOnHKfRZj2ytOil7qQ 密碼:rs4u
[root@yinzhengjie download]# ll total 42916 -rw-r--r-- 1 root root 43945614 Oct 10 20:10 yinzhengjie-clickhouse.tar.gz [root@yinzhengjie download]# [root@yinzhengjie download]# [root@yinzhengjie download]# tar zxf yinzhengjie-clickhouse.tar.gz -C ./ [root@yinzhengjie download]# [root@yinzhengjie download]# ll total 42920 drwxr-xr-x 2 root root 4096 Oct 10 20:03 clickhouse -rw-r--r-- 1 root root 43945614 Oct 10 20:10 yinzhengjie-clickhouse.tar.gz [root@yinzhengjie download]# [root@yinzhengjie download]# cd clickhouse/ [root@yinzhengjie clickhouse]# [root@yinzhengjie clickhouse]# ll total 43080 -rw-r--r-- 1 root root 3048 Oct 10 17:16 clickhouse-client-1.1.54236-4.el7.x86_64.rpm -rw-r--r-- 1 root root 901472 Oct 10 17:16 clickhouse-compressor-1.1.54236-4.el7.x86_64.rpm -rw-r--r-- 1 root root 9043256 Oct 10 17:16 clickhouse-debuginfo-1.1.54236-4.el7.x86_64.rpm -rw-r--r-- 1 root root 33680608 Oct 10 17:18 clickhouse-server-1.1.54236-4.el7.x86_64.rpm -rw-r--r-- 1 root root 8164 Oct 10 17:16 clickhouse-server-common-1.1.54236-4.el7.x86_64.rpm -rw-r--r-- 1 root root 49036 Oct 10 18:35 libtool-ltdl-2.4.2-21.el7_2.x86_64.rpm -rw-r--r-- 1 root root 421792 Oct 10 18:20 unixODBC-2.3.1-11.el7.x86_64.rpm [root@yinzhengjie clickhouse]#
4>.安裝ClickHouse的依賴及其rpm包
[root@yinzhengjie clickhouse]# ll total 43080 -rw-r--r-- 1 root root 3048 Oct 10 17:16 clickhouse-client-1.1.54236-4.el7.x86_64.rpm -rw-r--r-- 1 root root 901472 Oct 10 17:16 clickhouse-compressor-1.1.54236-4.el7.x86_64.rpm -rw-r--r-- 1 root root 9043256 Oct 10 17:16 clickhouse-debuginfo-1.1.54236-4.el7.x86_64.rpm -rw-r--r-- 1 root root 33680608 Oct 10 17:18 clickhouse-server-1.1.54236-4.el7.x86_64.rpm -rw-r--r-- 1 root root 8164 Oct 10 17:16 clickhouse-server-common-1.1.54236-4.el7.x86_64.rpm -rw-r--r-- 1 root root 49036 Oct 10 18:35 libtool-ltdl-2.4.2-21.el7_2.x86_64.rpm -rw-r--r-- 1 root root 421792 Oct 10 18:20 unixODBC-2.3.1-11.el7.x86_64.rpm [root@yinzhengjie clickhouse]# [root@yinzhengjie clickhouse]# You have new mail in /var/spool/mail/root [root@yinzhengjie clickhouse]# [root@yinzhengjie clickhouse]# yum -y localinstall *.rpm Loaded plugins: fastestmirror Examining clickhouse-client-1.1.54236-4.el7.x86_64.rpm: clickhouse-client-1.1.54236-4.el7.x86_64 Marking clickhouse-client-1.1.54236-4.el7.x86_64.rpm to be installed Examining clickhouse-compressor-1.1.54236-4.el7.x86_64.rpm: clickhouse-compressor-1.1.54236-4.el7.x86_64 Marking clickhouse-compressor-1.1.54236-4.el7.x86_64.rpm to be installed Examining clickhouse-debuginfo-1.1.54236-4.el7.x86_64.rpm: clickhouse-debuginfo-1.1.54236-4.el7.x86_64 Marking clickhouse-debuginfo-1.1.54236-4.el7.x86_64.rpm to be installed Examining clickhouse-server-1.1.54236-4.el7.x86_64.rpm: clickhouse-server-1.1.54236-4.el7.x86_64 Marking clickhouse-server-1.1.54236-4.el7.x86_64.rpm to be installed Examining clickhouse-server-common-1.1.54236-4.el7.x86_64.rpm: clickhouse-server-common-1.1.54236-4.el7.x86_64 Marking clickhouse-server-common-1.1.54236-4.el7.x86_64.rpm to be installed Examining libtool-ltdl-2.4.2-21.el7_2.x86_64.rpm: libtool-ltdl-2.4.2-21.el7_2.x86_64 Marking libtool-ltdl-2.4.2-21.el7_2.x86_64.rpm to be installed Examining unixODBC-2.3.1-11.el7.x86_64.rpm: unixODBC-2.3.1-11.el7.x86_64 Marking unixODBC-2.3.1-11.el7.x86_64.rpm to be installed Resolving Dependencies --> Running transaction check ---> Package clickhouse-client.x86_64 0:1.1.54236-4.el7 will be installed ---> Package clickhouse-compressor.x86_64 0:1.1.54236-4.el7 will be installed --> Processing Dependency: libicudata.so.50()(64bit) for package: clickhouse-compressor-1.1.54236-4.el7.x86_64 Determining fastest mirrors 10gen | 2.5 kB 00:00:00 base | 3.6 kB 00:00:00 centosplus | 3.4 kB 00:00:00 epel | 3.2 kB 00:00:00 extras | 3.4 kB 00:00:00 updates | 3.4 kB 00:00:00 (1/3): epel/x86_64/updateinfo | 948 kB 00:00:00 (2/3): epel/x86_64/primary | 3.6 MB 00:00:01 (3/3): updates/7/x86_64/primary_db | 6.0 MB 00:00:01 epel 12723/12723 --> Processing Dependency: libicui18n.so.50()(64bit) for package: clickhouse-compressor-1.1.54236-4.el7.x86_64 --> Processing Dependency: libicuuc.so.50()(64bit) for package: clickhouse-compressor-1.1.54236-4.el7.x86_64 ---> Package clickhouse-debuginfo.x86_64 0:1.1.54236-4.el7 will be installed ---> Package clickhouse-server.x86_64 0:1.1.54236-4.el7 will be installed ---> Package clickhouse-server-common.x86_64 0:1.1.54236-4.el7 will be installed ---> Package libtool-ltdl.x86_64 0:2.4.2-21.el7_2 will be installed ---> Package unixODBC.x86_64 0:2.3.1-11.el7 will be installed --> Running transaction check ---> Package libicu.x86_64 0:50.1.2-15.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved =================================================================================================================================================================================================================== Package Arch Version Repository Size =================================================================================================================================================================================================================== Installing: clickhouse-client x86_64 1.1.54236-4.el7 /clickhouse-client-1.1.54236-4.el7.x86_64 110 clickhouse-compressor x86_64 1.1.54236-4.el7 /clickhouse-compressor-1.1.54236-4.el7.x86_64 2.9 M clickhouse-debuginfo x86_64 1.1.54236-4.el7 /clickhouse-debuginfo-1.1.54236-4.el7.x86_64 48 M clickhouse-server x86_64 1.1.54236-4.el7 /clickhouse-server-1.1.54236-4.el7.x86_64 133 M clickhouse-server-common x86_64 1.1.54236-4.el7 /clickhouse-server-common-1.1.54236-4.el7.x86_64 16 k libtool-ltdl x86_64 2.4.2-21.el7_2 /libtool-ltdl-2.4.2-21.el7_2.x86_64 66 k unixODBC x86_64 2.3.1-11.el7 /unixODBC-2.3.1-11.el7.x86_64 1.2 M Installing for dependencies: libicu x86_64 50.1.2-15.el7 base 6.9 M Transaction Summary =================================================================================================================================================================================================================== Install 7 Packages (+1 Dependent package) Total size: 192 M Total download size: 6.9 M Installed size: 209 M Downloading packages: libicu-50.1.2-15.el7.x86_64.rpm | 6.9 MB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : libtool-ltdl-2.4.2-21.el7_2.x86_64 1/8 Installing : unixODBC-2.3.1-11.el7.x86_64 2/8 Installing : clickhouse-server-common-1.1.54236-4.el7.x86_64 3/8 Installing : libicu-50.1.2-15.el7.x86_64 4/8 Installing : clickhouse-server-1.1.54236-4.el7.x86_64 5/8 Installing : clickhouse-client-1.1.54236-4.el7.x86_64 6/8 Installing : clickhouse-compressor-1.1.54236-4.el7.x86_64 7/8 Installing : clickhouse-debuginfo-1.1.54236-4.el7.x86_64 8/8 Verifying : clickhouse-debuginfo-1.1.54236-4.el7.x86_64 1/8 Verifying : libtool-ltdl-2.4.2-21.el7_2.x86_64 2/8 Verifying : unixODBC-2.3.1-11.el7.x86_64 3/8 Verifying : libicu-50.1.2-15.el7.x86_64 4/8 Verifying : clickhouse-server-1.1.54236-4.el7.x86_64 5/8 Verifying : clickhouse-client-1.1.54236-4.el7.x86_64 6/8 Verifying : clickhouse-compressor-1.1.54236-4.el7.x86_64 7/8 Verifying : clickhouse-server-common-1.1.54236-4.el7.x86_64 8/8 Installed: clickhouse-client.x86_64 0:1.1.54236-4.el7 clickhouse-compressor.x86_64 0:1.1.54236-4.el7 clickhouse-debuginfo.x86_64 0:1.1.54236-4.el7 clickhouse-server.x86_64 0:1.1.54236-4.el7 clickhouse-server-common.x86_64 0:1.1.54236-4.el7 libtool-ltdl.x86_64 0:2.4.2-21.el7_2 unixODBC.x86_64 0:2.3.1-11.el7 Dependency Installed: libicu.x86_64 0:50.1.2-15.el7 Complete! [root@yinzhengjie clickhouse]#
三.配置clickhouse
1>.備份clickhouse-server配置文件config.xml
[root@yinzhengjie ~]# cp /etc/clickhouse-server/config.xml /etc/clickhouse-server/config.xml.`date +%F` [root@yinzhengjie ~]#
2>.修改clickhouse-server配置文件config.xml（指定clickhouse-server的主機地址）
[root@yinzhengjie ~]# sed -i 's#<listen_host>127.0.0.1</listen_host>#<listen_host>10.1.2.105</listen_host>#' /etc/clickhouse-server/config.xml [root@yinzhengjie ~]#
3>.修改clickhouse-server配置文件config.xml（修改默認的9000端口，我們自定義該服務的端口為9999）
[root@yinzhengjie ~]# sed -i 's#<tcp_port>9000</tcp_port>#<tcp_port>9999</tcp_port>#' /etc/clickhouse-server/config.xml [root@yinzhengjie ~]#
4>.備份clickhouse-server配置文件users.xml
[root@yinzhengjie ~]# cp /etc/clickhouse-server/users.xml /etc/clickhouse-server/users.xml.`date +%F` [root@yinzhengjie ~]#
5>.修改clickhouse-server配置文件users.xml（修改/etc/clickhouse-server/users.xml配置文件的第83行，改地址改為你當前部署的服務器IP地址，不推薦使用主機名！下圖是我修改后的值）
四.clickhouse管理方式
1>.啟動clickhouse
[root@yinzhengjie ~]# clickhouse-server --config-file=/etc/clickhouse-server/config.xml Include not found: clickhouse_remote_servers Include not found: clickhouse_compression 2018.10.10 20:37:20.997594 [ 1 ] <Warning> Application: Logging to console 2018.10.10 20:37:21.000228 [ 1 ] <Information> : Starting daemon with revision 54236 2018.10.10 20:37:21.000337 [ 1 ] <Information> Application: starting up 2018.10.10 20:37:21.001299 [ 1 ] <Debug> Application: Set max number of file descriptors to 4096 (was 1024). 2018.10.10 20:37:21.001317 [ 1 ] <Debug> Application: Initializing DateLUT. 2018.10.10 20:37:21.001326 [ 1 ] <Trace> Application: Initialized DateLUT with time zone `Asia/Shanghai'. 2018.10.10 20:37:21.048483 [ 1 ] <Debug> Application: Configuration parameter 'interserver_http_host' doesn't exist or exists and empty. Will use 'yinzhengjie' as replica host. 2018.10.10 20:37:21.048676 [ 1 ] <Debug> ConfigReloader: Loading config `/etc/clickhouse-server/users.xml' 2018.10.10 20:37:21.053081 [ 1 ] <Warning> ConfigProcessor: Include not found: networks 2018.10.10 20:37:21.053114 [ 1 ] <Warning> ConfigProcessor: Include not found: networks 2018.10.10 20:37:21.054194 [ 1 ] <Information> Application: Loading metadata. 2018.10.10 20:37:21.055243 [ 1 ] <Information> DatabaseOrdinary (default): Total 0 tables. 2018.10.10 20:37:21.055919 [ 1 ] <Debug> Application: Loaded metadata. 2018.10.10 20:37:21.056089 [ 1 ] <Information> DatabaseOrdinary (system): Total 0 tables. 2018.10.10 20:37:21.057059 [ 1 ] <Information> Application: Listening http://[::1]:8123 2018.10.10 20:37:21.057138 [ 1 ] <Information> Application: Listening tcp: [::1]:9999 2018.10.10 20:37:21.057198 [ 1 ] <Information> Application: Listening interserver: [::1]:9009 2018.10.10 20:37:21.057289 [ 1 ] <Information> Application: Listening http://10.1.2.105:8123 2018.10.10 20:37:21.057420 [ 1 ] <Information> Application: Listening tcp: 10.1.2.105:9999 2018.10.10 20:37:21.057486 [ 1 ] <Information> Application: Listening interserver: 10.1.2.105:9009 2018.10.10 20:37:21.057937 [ 1 ] <Information> Application: Ready for connections. 2018.10.10 20:37:23.048780 [ 2 ] <Debug> ConfigReloader: Loading config `/etc/clickhouse-server/config.xml' 2018.10.10 20:37:23.053196 [ 2 ] <Warning> ConfigProcessor: Include not found: clickhouse_remote_servers 2018.10.10 20:37:23.053332 [ 2 ] <Warning> ConfigProcessor: Include not found: clickhouse_compression
2>.查看clickhouse服務是否啟動
[root@yinzhengjie ~]# netstat -untalp | grep 9999 tcp 0 0 10.1.2.105:9999 0.0.0.0:* LISTEN 23259/clickhouse-se tcp6 0 0 ::1:9999 :::* LISTEN 23259/clickhouse-se [root@yinzhengjie ~]#
3>.
使用客戶端鏈接測試
[root@yinzhengjie ~]# clickhouse-client --host=10.1.2.105 --port=9999 ClickHouse client version 1.1.54236. Connecting to 10.1.2.105:9999. Connected to ClickHouse server version 1.1.54236. :) :) select now(); SELECT now() ┌───────────────now()─┐ │ 2018-10-10 20:44:14 │ └─────────────────────┘ 1 rows in set. Elapsed: 0.067 sec. :) :) show tables; SHOW TABLES Ok. 0 rows in set. Elapsed: 0.003 sec. :) :)
4>.關閉clickhouse服務
[root@yinzhengjie ~]# ps -aux|grep clickhouse-server | grep -v grep root 24998 0.1 0.3 319708 43372 pts/1 Sl+ 20:48 0:00 clickhouse-server --config-file=/etc/clickhouse-server/config.xml [root@yinzhengjie ~]# [root@yinzhengjie ~]# kill -9 `ps -aux|grep clickhouse-server | grep -v grep | awk '{print $2}'` [root@yinzhengjie ~]# You have new mail in /var/spool/mail/root [root@yinzhengjie ~]# [root@yinzhengjie ~]# ps -aux|grep clickhouse-server | grep -v grep [root@yinzhengjie ~]# [root@yinzhengjie ~]#
5>.為clickhouse-server添加守護進程
[root@yinzhengjie ~]# nohup clickhouse-server --config-file=/etc/clickhouse-server/config.xml >/var/log/yinzhengjie-clickhouse.log 2>&1 & #會在"/var/log"目錄下生成一個名稱為“yinzhengjie-clickhouse.log”的日志文件 [1] 26619 [root@yinzhengjie ~]# [root@yinzhengjie ~]# tail -10f /var/log/yinzhengjie-clickhouse.log 2018.10.10 20:58:53.981429 [ 1 ] <Information> Application: Listening http://[::1]:8123 2018.10.10 20:58:53.981479 [ 1 ] <Information> Application: Listening tcp: [::1]:9999 2018.10.10 20:58:53.981516 [ 1 ] <Information> Application: Listening interserver: [::1]:9009 2018.10.10 20:58:53.981554 [ 1 ] <Information> Application: Listening http://10.1.2.105:8123 2018.10.10 20:58:53.981590 [ 1 ] <Information> Application: Listening tcp: 10.1.2.105:9999 2018.10.10 20:58:53.981624 [ 1 ] <Information> Application: Listening interserver: 10.1.2.105:9009 2018.10.10 20:58:53.981796 [ 1 ] <Information> Application: Ready for connections. 2018.10.10 20:58:55.973241 [ 2 ] <Debug> ConfigReloader: Loading config `/etc/clickhouse-server/config.xml' 2018.10.10 20:58:55.977391 [ 2 ] <Warning> ConfigProcessor: Include not found: clickhouse_remote_servers 2018.10.10 20:58:55.977455 [ 2 ] <Warning> ConfigProcessor: Include not found: clickhouse_compression 2018.10.10 20:59:09.185410 [ 3 ] <Trace> TCPConnectionFactory: TCP Request. Address: 10.1.2.105:41874 2018.10.10 20:59:09.185660 [ 3 ] <Debug> TCPHandler: Connected ClickHouse client version 1.1.54236, user: default. 2018.10.10 20:59:18.439436 [ 3 ] <Debug> executeQuery: (from 10.1.2.105:41874) select now() 2018.10.10 20:59:18.440437 [ 3 ] <Trace> InterpreterSelectQuery: FetchColumns -> Complete 2018.10.10 20:59:18.440867 [ 3 ] <Debug> executeQuery: Query pipeline: Expression Expression One 2018.10.10 20:59:18.441657 [ 3 ] <Information> executeQuery: Read 1 rows, 1.00 B in 0.002 sec., 510 rows/sec., 510.97 B/sec. 2018.10.10 20:59:18.441764 [ 3 ] <Debug> MemoryTracker: Peak memory usage (for query): 1.00 MiB. 2018.10.10 20:59:18.441804 [ 3 ] <Debug> MemoryTracker: Peak memory usage (for user): 1.00 MiB. 2018.10.10 20:59:18.441849 [ 3 ] <Debug> MemoryTracker: Peak memory usage (total): 1.00 MiB. 2018.10.10 20:59:18.441882 [ 3 ] <Information> TCPHandler: Processed in 0.003 sec.
好啦～分享就到這里了，時間過的咋就這么快呢？一不小心就晚上九點多了，關於ClickHouse集群的部署等我后期研究好了也會分享給大家的，我得趕緊走了，不然趕不上地鐵啦！晚安，兄弟們！