官方安裝指導：https://dolphinscheduler.apache.org/zh-cn/docs/1.3.4/user_doc/quick-start.html
1節點規划
| 機器 | 服務 | 端口 | group |
|---|---|---|---|
| hadoop01 | master、api、logger | 8787(master)、8888(api) | |
| hadoop02 | master、alert、worker、logger | 8787(master)、7878(worker) | hadoop |
| hadoop03 | worker、logger | 7878(worker) | hadoop |
hadoop 組配置后master分發任務才能根據cpu和內存的負載選擇具體哪個worker執行任務
2 准備工作
1、創建目錄
- 確保三個節點都有目錄
/bigdata/soft、/bigdata/install，且所屬用戶及用戶組如下
[hadoop@hadoop01 ~]$ ll /bigdata/
總用量 0
drwxr-xr-x. 2 hadoop hadoop 6 4月 13 14:21 install
drwxr-xr-x. 2 hadoop hadoop 6 4月 13 14:22 soft
- 若沒有這些目錄，那么如下創建；3個節點都運行如下命令
sudo mkdir -p /bigdata/install
sudo mkdir -p /bigdata/soft
sudo chown -R hadoop:hadoop /bigdata/
ll /bigdata/
2、確保已安裝zookeeper集群
- 啟動zookeeper集群
- 確保三節點上已經安裝了zk
- zk版本要求：ZooKeeper (3.4.6+)
- 若沒有安裝，請先安裝再往下繼續
3、啟動HDFS
- 因為ds的資源存儲在HDFS上
- 所以，hadoop01上運行
start-dfs.sh啟動hdfs
3 開始安裝
第一步：安裝包下載，在hadoop01上執行
[hadoop@hadoop01~]$ cd /bigdata/soft/
[hadoop@hadoop01 soft]$ wget https://mirrors.tuna.tsinghua.edu.cn/apache/incubator/dolphinscheduler/1.3.5/apache-dolphinscheduler-incubating-1.3.5-dolphinscheduler-bin.tar.gz
第二步：解壓壓縮包
[hadoop@hadoop01 soft]$ tar -xzvf apache-dolphinscheduler-incubating-1.3.5-dolphinscheduler-bin.tar.gz -C /bigdata/install/
第三步：重命名
[hadoop@hadoop01 soft]$ cd /bigdata/install/
[hadoop@hadoop01 install]$ mv apache-dolphinscheduler-incubating-1.3.5-dolphinscheduler-bin/ dolphinscheduler-1.3.5
[hadoop@hadoop01 install]$ ll
總用量 0
drwxrwxr-x. 9 hadoop hadoop 156 4月 13 14:35 dolphinscheduler-1.3.5
第四步：建庫建表
此處以hadoop03安裝了mysql為例
hadoop01上
[hadoop@hadoop01 install]$ scp /bigdata/install/dolphinscheduler-1.3.5/sql/dolphinscheduler_mysql.sql hadoop03:/bigdata/soft/
hadoop03上，進入MySQL命令行執行
mysql -uroot -p
set global validate_password_policy=LOW;
set global validate_password_length=6;
CREATE DATABASE IF NOT EXISTS dolphinscheduler DEFAULT CHARSET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL PRIVILEGES ON dolphinscheduler.* TO 'root'@'%' IDENTIFIED BY '123456';
GRANT ALL PRIVILEGES ON dolphinscheduler.* TO 'root'@'localhost' IDENTIFIED BY '123456';
flush privileges;
use dolphinscheduler;
source /kkb/soft/dolphinscheduler_mysql.sql;
mysql> show tables;
+--------------------------------+
| Tables_in_dolphinscheduler |
+--------------------------------+
| QRTZ_BLOB_TRIGGERS |
| QRTZ_CALENDARS |
| QRTZ_CRON_TRIGGERS |
| QRTZ_FIRED_TRIGGERS|
| QRTZ_JOB_DETAILS |
| QRTZ_LOCKS |
| QRTZ_PAUSED_TRIGGER_GRPS |
| QRTZ_SCHEDULER_STATE |
| QRTZ_SIMPLE_TRIGGERS |
| QRTZ_SIMPROP_TRIGGERS |
| QRTZ_TRIGGERS |
| t_ds_access_token |
| t_ds_alert |
| t_ds_alertgroup|
| t_ds_command |
| t_ds_datasource|
| t_ds_error_command |
| t_ds_process_definition|
| t_ds_process_instance |
| t_ds_project |
| t_ds_queue |
| t_ds_relation_datasource_user |
| t_ds_relation_process_instance |
| t_ds_relation_project_user |
| t_ds_relation_resources_user |
| t_ds_relation_udfs_user|
| t_ds_relation_user_alertgroup |
| t_ds_resources |
| t_ds_schedules |
| t_ds_session |
| t_ds_task_instance |
| t_ds_tenant|
| t_ds_udfs |
| t_ds_user |
| t_ds_version |
+--------------------------------+
35 rows in set (0.00 sec)
第五步：修改配置文件
[hadoop@hadoop01 ~]$ cd /bigdata/install/dolphinscheduler-1.3.5/conf/
1、alert.properties
配置告警郵箱相關信息
- 此處已126郵箱為例
- 登錄自己的126郵箱
- 開始配置文件
[hadoop@hadoop01 conf]$ vim alert.properties
#alert type is EMAIL/SMS
alert.type=EMAIL
# mail server configuration
mail.protocol=SMTP
mail.server.host=smtp.126.com
mail.server.port=25
mail.sender=youhy964@126.com
mail.user=youhy964@126.com
mail.passwd=WBNPUGCNZMQQYBUT
# TLS
mail.smtp.starttls.enable=true
# SSL
mail.smtp.ssl.enable=false
mail.smtp.ssl.trust=smtp.126.com
2、application-api.properties
- 修改web ui端口號
[hadoop@hadoop01 conf]$ vim application-api.properties
# server port
server.port=8888
3、common.properties
# 修改如下3個屬性的值
[hadoop@hadoop01 conf]$ vim common.properties
resource.storage.type=HDFS
fs.defaultFS=hdfs://hadoop01:8020
yarn.application.status.address=http://hadoop01:8088/ws/v1/cluster/apps/%s
4、datasource.properties
假設mysql安裝在hadoop03節點
# 修改如下幾個屬性的值
[hadoop@hadoop01 conf]$ vim datasource.properties
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
spring.datasource.url=jdbc:mysql://hadoop03:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8&allowMultiQueries=true
spring.datasource.username=root
spring.datasource.password=123456
5、master.properties
[hadoop@hadoop01 conf]$ vim master.properties
master.listen.port=8787
6、worker.properties
[hadoop@hadoop01 conf]$ vim worker.properties
worker.listen.port=7878
worker.groups=hadoop
7、zookeeper.properties
[hadoop@hadoop01 conf]$ vim zookeeper.properties
zookeeper.quorum=hadoop01:2181,hadoop02:2181,hadoop03:2181
8、env/dolphinscheduler_env.sh
# 修改如下屬性，根據自己的實際情況，配置屬性值 |
[hadoop@hadoop01 conf]$ vim env/dolphinscheduler_env.sh
export HADOOP_HOME=/bigdata/install/hadoop-3.1.4
export HADOOP_CONF_DIR=/bigdata/install/hadoop-3.1.4/etc/hadoop
export SPARK_HOME1=/bigdata/install/spark-2.3.3-bin-hadoop2.7
export JAVA_HOME=/bigdata/install/jdk1.8.0_141
export HIVE_HOME=/bigdata/install/hive-3.1.2
第六步：將hdfs-site.xml、core-site.xml 拷貝至ds的 conf 目錄下，同時 把mysql-connector-java-5.1.48-bin.jar驅動包上傳到ds的lib目錄下
[hadoop@hadoop01 conf]$ cp /bigdata/install/hadoop-3.1.4/etc/hadoop/hdfs-site.xml /bigdata/install/dolphinscheduler-1.3.5/conf
[hadoop@hadoop01 conf]$ cp /bigdata/install/hadoop-3.1.4/etc/hadoop/core-site.xml /bigdata/install/dolphinscheduler-1.3.5/conf
# 將mysql-connector-java-5.1.38.jar上傳到hadoop01的/bigdata/soft目錄
[hadoop@hadoop01 soft]$ cd /bigdata/soft/
[hadoop@hadoop01 soft]$ cp mysql-connector-java-5.1.38.jar /bigdata/install/dolphinscheduler-1.3.5/lib/
第七步：到ds的bin目錄下（1.3.5版本，可以跳過此步）
dos2unix dolphinscheduler-daemon.sh #dos2unix:將DOS格式的文本文件轉換成UNIX格式的(DOS/MAC to UNIX text file format converter)
chmod +x dolphinscheduler-daemon.sh
第八步：scp到hadoop02、hadoop03
[hadoop@hadoop01 bin]$ cd /bigdata/install/
[hadoop@hadoop01 install]$ scp -r dolphinscheduler-1.3.5/ hadoop02:$PWD
[hadoop@hadoop01 install]$ scp -r dolphinscheduler-1.3.5/ hadoop03:$PWD
4 調優配置
生產環境上建議，worker.properties里設置的cpu和內存調一下就可以保護worker不至於掛掉，一般別超過cpu核數的2倍，內存留上1~2G（當然如果比較豪，可以預留更多資源），線程數別超過cp倍。
例如：8c16G機器 worker.exec.threads=20 worker.max.cpuload.avg=16 worker.reserved.memory=1
同理對於master調優配置 8c16G機器 master.properties文件 master.max.cpuload.avg=16 master.reserved.memory=1
5 啟動
在hadoop01 啟動 master、api、logger
[hadoop@hadoop01 ~]$ cd /bigdata/install/dolphinscheduler-1.3.5/bin/
[hadoop@hadoop01 bin]$ ./dolphinscheduler-daemon.sh start master-server
[hadoop@hadoop01 bin]$ ./dolphinscheduler-daemon.sh start api-server
[hadoop@hadoop01 bin]$ ./dolphinscheduler-daemon.sh start logger-server
在hadoop02 啟動 master、alert、worker、logger
[hadoop@hadoop02 ~]$ cd /bigdata/install/dolphinscheduler-1.3.5/bin/
[hadoop@hadoop02 bin]$ ./dolphinscheduler-daemon.sh start master-server
[hadoop@hadoop02 bin]$ ./dolphinscheduler-daemon.sh start alert-server
[hadoop@hadoop02 bin]$ ./dolphinscheduler-daemon.sh start worker-server
[hadoop@hadoop02 bin]$ ./dolphinscheduler-daemon.sh start logger-server
在hadoop03 啟動 worker、logger
[hadoop@hadoop03 ~]$ cd /bigdata/install/dolphinscheduler-1.3.5/bin/
[hadoop@hadoop03 bin]$ ./dolphinscheduler-daemon.sh start worker-server
[hadoop@hadoop03 bin]$ ./dolphinscheduler-daemon.sh start logger-server
目前3節點已經啟動了hadoop集群、zookeeper集群、及ds相關進程；
啟動成功之后，會出現如下進程
[hadoop@hadoop01 bin]$ xcall jps
============= hadoop01 jps =============
3680 ApiApplicationServer
2465 DataNode
3761 LoggerServer
3410 JobHistoryServer
2036 QuorumPeerMain
2294 NameNode
3607 MasterServer
2666 SecondaryNameNode
2892 ResourceManager
3037 NodeManager
4351 Jps
============= hadoop02 jps =============
2320 NodeManager
2880 LoggerServer
2179 DataNode
3446 Jps
3287 AlertServer
2616 MasterServer
2798 WorkerServer
2031 QuorumPeerMain
============= hadoop03 jps =============
2160 DataNode
2289 NodeManager
3009 LoggerServer
1996 QuorumPeerMain
2927 WorkerServer
3103 Jps
- 如何關閉ds集群？
- 關閉個節點的各角色，就是將上邊啟動命令中的start換成stop即可
[hadoop@hadoop01 ~]$ cd /bigdata/install/dolphinscheduler-1.3.5/bin/
[hadoop@hadoop01 bin]$ ./dolphinscheduler-daemon.sh stop master-server
[hadoop@hadoop01 bin]$ ./dolphinscheduler-daemon.sh stop api-server
[hadoop@hadoop01 bin]$ ./dolphinscheduler-daemon.sh stop logger-server
[hadoop@hadoop02 ~]$ cd /bigdata/install/dolphinscheduler-1.3.5/bin/
[hadoop@hadoop02 bin]$ ./dolphinscheduler-daemon.sh stop master-server
[hadoop@hadoop02 bin]$ ./dolphinscheduler-daemon.sh stop alert-server
[hadoop@hadoop02 bin]$ ./dolphinscheduler-daemon.sh stop worker-server
[hadoop@hadoop02 bin]$ ./dolphinscheduler-daemon.sh stop logger-server
[hadoop@hadoop03 ~]$ cd /bigdata/install/dolphinscheduler-1.3.5/bin/
[hadoop@hadoop03 bin]$ ./dolphinscheduler-daemon.sh stop worker-server
[hadoop@hadoop03 bin]$ ./dolphinscheduler-daemon.sh stop logger-server
6 訪問驗證
web頁面
-
訪問地址：http://hadoop01:8888/dolphinscheduler/ui/view/login/index.html
-
默認用戶：admin
-
默認密碼：dolphinscheduler123
