介紹
前面幾個章節對Druid的整體架構做了簡單的說明，本文主要描述如何部署Druid的環境
Imply提供了一套完整的部署方式，包括依賴庫，Druid，圖形化的數據展示頁面，SQL查詢組件等。本文將基於Imply套件進行說明
單機部署
依賴
- Java 8 or better
- Node.js 4.5.x or better
- Linux, Mac OS X, or other Unix-like OS (Windows is not supported)
- At least 4GB of RAM
下載與安裝
- 從https://imply.io/get-started 下載最新版本安裝包
- tar -xzf imply-2.5.14.tar.gz
- cd imply-2.5.14
目錄說明如下：
- bin/ - run scripts for included software.
- conf/ - template configurations for a clustered setup.
- conf-quickstart/* - configurations for the single-machine quickstart.
- dist/ - all included software.
- quickstart/ - files related to the single-machine quickstart.
啟動服務
bin/supervise -c conf/supervise/quickstart.conf
安裝驗證
導入測試數據
安裝包中包含一些測試的數據,可以通過執行預先定義好的數據說明文件進行導入
bin/post-index-task --file quickstart/wikiticker-index.json
可視化控制台
- overlord 控制頁面：http://localhost:8090/console.html.
- druid集群頁面：http://localhost:8081
- 數據可視化頁面：http://localhost:9095
- HTTP POST數據查詢
命令：curl -L -H’Content-Type: application/json’ -XPOST –data-binary @quickstart/wikiticker-top-pages.json http://localhost:8082/druid/v2?pretty
結果：
[ { "timestamp" : "2016-06-27T00:00:11.080Z", "result" : [ { "edits" : 29, "page" : "Copa América Centenario" }, { "edits" : 16, "page" : "User:Cyde/List of candidates for speedy deletion/Subpage" }, .......... { "edits" : 8, "page" : "World Deaf Championships" } ] } ]
集群部署
集群配置的規划需要根據需求來定制，下面以一個開發環境機器搭建為例,描述如何搭建一個有HA特性的Druid集群.
集群部署有以下幾點需要說明
1. 為了保證HA,主節點部署兩台
2. 管理節點與查詢節點可以考慮多核大內存的機器
部署規划
| 角色 | 機器 | 配置 | 集群角色 |
|---|---|---|---|
| 主節點 | 10.5.24.137 | 8C16G | Coordinator,Overlord |
| 主節點 | 10.5.24.138 | 8C16G | Coordinator,Overlord |
| 數據節點，查詢節點 | 10.5.24.139 | 8C16G | Historical, MiddleManager, Tranquility,Broker,Pivot Web |
| 數據節點，查詢節點 | 10.5.24.140 | 8C16G | Historical, MiddleManager, Tranquility,（數據節點，查詢節點）Broker |
部署步驟
公共配置
編輯conf/druid/_common/common.runtime.properties 文件內容
#
# Extensions
#
druid.extensions.directory=dist/druid/extensions
druid.extensions.hadoopDependenciesDir=dist/druid/hadoop-dependencies
druid.extensions.loadList=["druid-parser-route","druid-lookups-cached-global","druid-histogram","druid-datasketches","mysql-metadata-storage","druid-hdfs-storage","druid-kafka-indexing-service"]
#
# Logging
#
# Log all runtime properties on startup. Disable to avoid logging properties on startup:
druid.startup.logging.logProperties=true
#
# Zookeeper
#
druid.zk.service.host=master251
druid.zk.paths.base=/druid
# For MySQL:
druid.metadata.storage.type=mysql
druid.metadata.storage.connector.connectURI=jdbc:mysql://master251:3306/druid
druid.metadata.storage.connector.user=druid
druid.metadata.storage.connector.password=druid
#
# Deep storage
#
# For local disk (only viable in a cluster if this is a network mount):
# For HDFS:
druid.storage.type=hdfs
druid.storage.storageDirectory=hdfs://master251:9000/druid/segments
主節點配置
- 創建配置文件：cp conf/supervise/master-no-zk.conf conf/supervise/master.conf
- 編輯master.conf 內容如下：
:verify bin/verify-java :verify bin/verify-version-check coordinator bin/run-druid coordinator conf !p80 overlord bin/run-druid overlord conf
目前的版本中，mysql-metadata-storage沒有包含在默認的安裝包中，如果使用mysql存儲元數據，需要單獨安裝下對應的擴展，是用下列命令在兩個master節點上對需要用到的擴展進行安裝：
root@druid imply-2.5.14# java -classpath "dist/druid/lib/*" -Ddruid.extensions.directory="dist/druid/extensions" io.druid.cli.Main tools pull-deps -c io.druid.extensions:mysql-metadata-storage:0.12.0 -c io.druid.extensions.contrib:druid-rabbitmq:0.12.0 -h org.apache.hadoop:hadoop-client:2.6.0
==默認mysql-metadata-storage帶的mysql驅動是針對Mysql 5.1的，如果使用Mysql的版本是5.5 或是其他版本，可能會出現”Communications link failure”的錯誤，此時需要更新Mysql的驅動。==
在10.5.24.137/138上啟動master相關服務：nohup bin/supervise -c conf/supervise/master.conf > master.log &
數據節點與查詢節點配置
- 安裝NodeJS:apt-get install nodejs
- 創建配置文件：vim conf/supervise/data-with-query.conf
- 編輯data-with-query.conf 內容如下：
:verify bin/verify-java :verify bin/verify-node :verify bin/verify-version-check broker bin/run-druid broker conf imply-ui bin/run-imply-ui conf historical bin/run-druid historical conf middleManager bin/run-druid middleManager conf # Uncomment to use Tranquility Server #!p95 tranquility-server bin/tranquility server -configFile conf/tranquility/server.json # Uncomment to use Tranquility Kafka #!p95 tranquility-kafka bin/tranquility kafka -configFile conf/tranquility/kafka.json
對於集群模式，pivot的配置文件必須調整為mysql,sqllite會導致無法查看datasource,修改conf/pivot/config.xml文件
settingsLocation: location: mysql uri: 'mysql://root:123456@10.5.24.151:3306/druid' table: 'pivot_state' initialSettings: clusters: - name: druid type: druid host: localhost:8082
在10.5.24.139/140兩台機器上分別執行:nohup bin/supervise -c conf/supervise/data-with-query.conf > data-with-query.log &
驗證
可視化控制台
- overlord 控制頁面：http://10.5.24.138:8090/console.html.
- druid集群頁面：http://10.5.24.138:8081
- 數據可視化頁面：http://10.5.24.139:9095