基於yum的方式安裝Cloudera Manager Server（使用Mysql 8.0版本）
作者：尹正傑
版權聲明：原創作品，謝絕轉載！否則將追究法律責任。
一.安裝和配置元數據庫
1>.部署MySQL數據庫
我們應啟動的5台服務器中選出來2台服務做MySQL主從同步。我們把mysql服務安裝在node101.yinzhengjie.org.cn上。至於MySQL的部署方式並不是本片博客的重點，不過可參考我之前的筆記：使用二進制安裝包的方式單機部署MySQL8.0.13。
由於這個元數據庫還是比較重要的，在生產環境使用中，建議使用MySQL的復制機制部署一個主從復制。或者使用Keepalived+MySQL部署一個主主雙活的數據庫，以提升其可用性。
2>.配置元數據庫
創建所需的表和賬號。首先我們修改root的密碼。接着創建scm,rman,hive,oozie,hue,sentry等6套庫和用戶，分別用戶Cloudera Manager，Reports Manager，hive，oozie，hue，sentry存儲數據。
[root@node101 ~]# mysql -uroot -pyinzhengjie mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 13 Server version: 8.0.15 MySQL Community Server - GPL Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> mysql> CREATE DATABASE scm DEFAULT CHARACTER SET UTF8; Query OK, 1 row affected, 1 warning (0.01 sec) mysql> mysql> CREATE USER scm@'%' IDENTIFIED WITH mysql_native_password BY 'yinzhengjie'; Query OK, 0 rows affected (0.01 sec) mysql> mysql> GRANT ALL PRIVILEGES ON scm.* TO scm@'%'; Query OK, 0 rows affected (0.01 sec) mysql>
mysql> CREATE DATABASE rman DEFAULT CHARACTER SET UTF8; Query OK, 1 row affected, 1 warning (0.01 sec) mysql> CREATE USER rman@'%' IDENTIFIED WITH mysql_native_password BY 'yinzhengjie'; Query OK, 0 rows affected (0.00 sec) mysql> GRANT ALL PRIVILEGES ON rman.* TO rman@'%'; Query OK, 0 rows affected (0.00 sec) mysql>
mysql> CREATE DATABASE hive DEFAULT CHARACTER SET UTF8; Query OK, 1 row affected, 1 warning (0.01 sec) mysql> CREATE USER hive@'%' IDENTIFIED WITH mysql_native_password BY 'yinzhengjie'; Query OK, 0 rows affected (0.00 sec) mysql> GRANT ALL PRIVILEGES ON hive.* TO hive@'%'; Query OK, 0 rows affected (0.00 sec) mysql>
mysql> CREATE DATABASE oozie DEFAULT CHARACTER SET UTF8; Query OK, 1 row affected, 1 warning (0.00 sec) mysql> CREATE USER oozie@'%' IDENTIFIED WITH mysql_native_password BY 'yinzhengjie'; Query OK, 0 rows affected (0.01 sec) mysql> GRANT ALL PRIVILEGES ON oozie.* TO oozie@'%'; Query OK, 0 rows affected (0.00 sec) mysql>
mysql> CREATE DATABASE hue DEFAULT CHARACTER SET UTF8; Query OK, 1 row affected, 1 warning (0.00 sec) mysql> CREATE USER hue@'%' IDENTIFIED WITH mysql_native_password BY 'yinzhengjie'; Query OK, 0 rows affected (0.01 sec) mysql> GRANT ALL PRIVILEGES ON hue.* TO hue@'%'; Query OK, 0 rows affected (0.01 sec) mysql>
mysql> CREATE DATABASE sentry DEFAULT CHARACTER SET UTF8; Query OK, 1 row affected, 1 warning (0.00 sec) mysql> CREATE USER sentry@'%' IDENTIFIED WITH mysql_native_password BY 'yinzhengjie'; Query OK, 0 rows affected (0.01 sec) mysql> GRANT ALL PRIVILEGES ON sentry.* TO sentry@'%'; Query OK, 0 rows affected (0.00 sec) mysql>
mysql> FLUSH PRIVILEGES; Query OK, 0 rows affected (0.01 sec) mysql> mysql> SELECT user,host,plugin FROM mysql.user; +------------------+-----------+-----------------------+ | user | host | plugin | +------------------+-----------+-----------------------+ | hive | % | mysql_native_password | | hue | % | mysql_native_password | | oozie | % | mysql_native_password | | rman | % | mysql_native_password | | scm | % | mysql_native_password | | sentry | % | mysql_native_password | | mysql.infoschema | localhost | caching_sha2_password | | mysql.session | localhost | caching_sha2_password | | mysql.sys | localhost | caching_sha2_password | | root | localhost | caching_sha2_password | +------------------+-----------+-----------------------+ 10 rows in set (0.00 sec) mysql>
[root@node101 ~]# mysql -uscm -pyinzhengjie #測試是否可以正常登陸 mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 24 Server version: 8.0.15 MySQL Community Server - GPL Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> quit Bye [root@node101 ~]# [root@node101 ~]# mysql -h node101.yinzhengjie.org.cn -uscm -pyinzhengjie mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 25 Server version: 8.0.15 MySQL Community Server - GPL Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> quit Bye [root@node101 ~]#
二.Cloudera Manager 部署方式
1>.yum方式安裝方式
yum安裝方式一：（下載Cloudera Manager的源，修改配置文件並執行安裝，這種方式是通過網絡的方法安裝，效率較低！）
下載地址：https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.16.1/。
[root@node101 ~]# ll /etc/yum.repos.d/ | grep cloudera-manager.repo [root@node101 ~]# [root@node101 ~]# wget -P /etc/yum.repos.d/ https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/cloudera-manager.repo --2019-02-21 11:47:07-- https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/cloudera-manager.repo Resolving archive.cloudera.com (archive.cloudera.com)... 151.101.108.167 Connecting to archive.cloudera.com (archive.cloudera.com)|151.101.108.167|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 290 [binary/octet-stream] Saving to: ‘/etc/yum.repos.d/cloudera-manager.repo’ 100%[========================================================================================================================================================================>] 290 --.-K/s in 0s 2019-02-21 11:47:07 (48.4 MB/s) - ‘/etc/yum.repos.d/cloudera-manager.repo’ saved [290/290] [root@node101 ~]# [root@node101 ~]# ll /etc/yum.repos.d/ | grep cloudera-manager.repo -rw-r--r-- 1 root root 290 Dec 14 00:32 cloudera-manager.repo [root@node101 ~]# [root@node101 ~]#
[root@node101 ~]# grep baseurl /etc/yum.repos.d/cloudera-manager.repo baseurl=https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5/ [root@node101 ~]# [root@node101 ~]# sed -i s'#baseurl=https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5/#baseurl=https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.16.1/#' /etc/yum.repos.d/cloudera-manager.repo #修改為你想要安裝的CM版本 [root@node101 ~]# [root@node101 ~]# grep baseurl /etc/yum.repos.d/cloudera-manager.repo #我這里改為5.16.1版本 baseurl=https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.16.1/ [root@node101 ~]# [root@node101 ~]# yum -y install cloudera-manager-daemons cloudera-manager-server #
按照上面的方式按照是可以按照成功，但是網速想我這樣慢的小伙伴就不推薦使用這種方式，我們可以提前把rpm包下載下來，然后手動安裝即可。 
yum安裝方式二：（如果你發現下載速度很慢，可以在自己電腦上下載安裝包在上傳到虛擬機內，本片博客采用的就是這種方式！大家也看到了，上面的下載速度太慢了！）
下載地址：https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/。
[root@node101 ~]# ll -lht total 754M -rwxr-xr-x 1 root root 11K Feb 21 12:21 cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm -rwxr-xr-x 1 root root 8.5K Feb 21 12:21 cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm -rwxr-xr-x 1 root root 754M Feb 21 12:21 cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm [root@node101 ~]#
[root@node101 ~]# ll -lht total 754M -rwxr-xr-x 1 root root 11K Feb 21 12:21 cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm -rwxr-xr-x 1 root root 8.5K Feb 21 12:21 cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm -rwxr-xr-x 1 root root 754M Feb 21 12:21 cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm [root@node101 ~]# [root@node101 ~]# yum -y install cloudera-manager-*.rpm Loaded plugins: fastestmirror Examining cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm: cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64 Marking cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm to be installed Examining cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm: cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86_64 Marking cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm to be installed Examining cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm: cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7.x86_64 Marking cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm to be installed Resolving Dependencies --> Running transaction check ---> Package cloudera-manager-daemons.x86_64 0:5.16.1-1.cm5161.p0.1.el7 will be installed ---> Package cloudera-manager-server.x86_64 0:5.16.1-1.cm5161.p0.1.el7 will be installed ---> Package cloudera-manager-server-db-2.x86_64 0:5.16.1-1.cm5161.p0.1.el7 will be installed --> Processing Dependency: postgresql-server >= 8.4 for package: cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7.x86_64 Loading mirror speeds from cached hostfile * base: mirrors.shu.edu.cn * extras: mirror.jdcloud.com * updates: centos.cs.nctu.edu.tw --> Running transaction check ---> Package postgresql-server.x86_64 0:9.2.24-1.el7_5 will be installed --> Processing Dependency: postgresql-libs(x86-64) = 9.2.24-1.el7_5 for package: postgresql-server-9.2.24-1.el7_5.x86_64 --> Processing Dependency: postgresql(x86-64) = 9.2.24-1.el7_5 for package: postgresql-server-9.2.24-1.el7_5.x86_64 --> Processing Dependency: libpq.so.5()(64bit) for package: postgresql-server-9.2.24-1.el7_5.x86_64 --> Running transaction check ---> Package postgresql.x86_64 0:9.2.24-1.el7_5 will be installed ---> Package postgresql-libs.x86_64 0:9.2.24-1.el7_5 will be installed --> Finished Dependency Resolution Dependencies Resolved ====================================================================================================================================================================================================================================================== Package Arch Version Repository Size ====================================================================================================================================================================================================================================================== Installing: cloudera-manager-daemons x86_64 5.16.1-1.cm5161.p0.1.el7 /cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64 935 M cloudera-manager-server x86_64 5.16.1-1.cm5161.p0.1.el7 /cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86_64 13 k cloudera-manager-server-db-2 x86_64 5.16.1-1.cm5161.p0.1.el7 /cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7.x86_64 20 k Installing for dependencies: postgresql x86_64 9.2.24-1.el7_5 base 3.0 M postgresql-libs x86_64 9.2.24-1.el7_5 base 234 k postgresql-server x86_64 9.2.24-1.el7_5 base 3.8 M Transaction Summary ====================================================================================================================================================================================================================================================== Install 3 Packages (+3 Dependent packages) Total size: 942 M Total download size: 7.1 M Installed size: 968 M Downloading packages: postgresql-server-9.2.24-1.el7 FAILED http://mirror.lzu.edu.cn/centos/7.6.1810/os/x86_64/Packages/postgresql-server-9.2.24-1.el7_5.x86_64.rpm: [Errno 14] curl#56 - "Recv failure: Connection reset by peer" ] 0.0 B/s | 0 B --:--:-- ETA Trying other mirror. (1/3): postgresql-9.2.24-1.el7_5.x86_64.rpm | 3.0 MB 00:00:03 (2/3): postgresql-server-9.2.24-1.el7_5.x86_64.rpm | 3.8 MB 00:00:05 (3/3): postgresql-libs-9.2.24-1.el7_5.x86_64.rpm | 234 kB 00:00:07 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Total 962 kB/s | 7.1 MB 00:00:07 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : postgresql-libs-9.2.24-1.el7_5.x86_64 1/6 Installing : postgresql-9.2.24-1.el7_5.x86_64 2/6 Installing : postgresql-server-9.2.24-1.el7_5.x86_64 3/6 Installing : cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64 4/6 Installing : cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86_64 5/6 Installing : cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7.x86_64 6/6 Verifying : cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86_64 1/6 Verifying : cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64 2/6 Verifying : cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7.x86_64 3/6 Verifying : postgresql-server-9.2.24-1.el7_5.x86_64 4/6 Verifying : postgresql-libs-9.2.24-1.el7_5.x86_64 5/6 Verifying : postgresql-9.2.24-1.el7_5.x86_64 6/6 Installed: cloudera-manager-daemons.x86_64 0:5.16.1-1.cm5161.p0.1.el7 cloudera-manager-server.x86_64 0:5.16.1-1.cm5161.p0.1.el7 cloudera-manager-server-db-2.x86_64 0:5.16.1-1.cm5161.p0.1.el7 Dependency Installed: postgresql.x86_64 0:9.2.24-1.el7_5 postgresql-libs.x86_64 0:9.2.24-1.el7_5 postgresql-server.x86_64 0:9.2.24-1.el7_5 Complete! [root@node101 ~]#
2>..離線方式安裝
詳情請參考：離線方式部署Cloudera Manager5.15.1。
三.配置並啟動Cloudera Manager
1>.配置Cloudera Manager
[root@node101 ~]# grep CMF_JAVA_OPTS /etc/default/cloudera-scm-server #Cloudera Manager默認分配的JVM堆內存是2G，某些情況下會出現OOM的情況，我們可以適當將其調大。 export CMF_JAVA_OPTS="-Xmx2G -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp" [root@node101 ~]# [root@node101 ~]# sed -i s'#Xmx2G#Xmx4G#' /etc/default/cloudera-scm-server #我們將Cloudera Manager的堆內存調大到4G。 [root@node101 ~]# [root@node101 ~]# grep CMF_JAVA_OPTS /etc/default/cloudera-scm-server export CMF_JAVA_OPTS="-Xmx4G -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp" [root@node101 ~]# [root@node101 ~]#
2>.初始化數據庫配置
Cloudera已經提供了一個sh腳本用戶配置Cloudera Manager連接元數據庫，官方文檔參考：https://www.cloudera.com/documentation/enterprise/5-14-x/topics/cm_ig_installing_configuring_dbs.html#concept_i2r_m3m_hn。
我們的安裝規划下，元數據庫和Cloudera Manager運行於同一個節點，且已經預先創建了scm庫和用戶，因此，命令很簡單：
[root@node101 ~]# /usr/share/cmf/schema/scm_prepare_database.sh mysql scm scm yinzhengjie JAVA_HOME=/usr/java/jdk1.8 Verifying that we can write to /etc/cloudera-scm-server Creating SCM configuration file in /etc/cloudera-scm-server Executing: /usr/java/jdk1.8/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/usr/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db. Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary. [ main] DbCommandExecutor INFO Successfully connected to database. All done, your SCM database is configured correctly! #注意，如果你看到了這一行提示，說明配置完成，否則需要根據提示解決相應的錯誤！ [root@node101 ~]#
3>.啟動Cloudera Manager
[root@node101 ~]# systemctl status cloudera-scm-server ● cloudera-scm-server.service - LSB: Cloudera SCM Server Loaded: loaded (/etc/rc.d/init.d/cloudera-scm-server; bad; vendor preset: disabled) Active: inactive (dead) Docs: man:systemd-sysv-generator(8) Feb 21 12:29:08 node101.yinzhengjie.org.cn systemd[1]: [/run/systemd/generator.late/cloudera-scm-server.service:13] Failed to add dependency on +cloudera-scm-server-db.service, ignoring: Invalid argument Feb 21 12:29:08 node101.yinzhengjie.org.cn systemd[1]: [/run/systemd/generator.late/cloudera-scm-server.service:13] Failed to add dependency on +cloudera-scm-server-db.service, ignoring: Invalid argument [root@node101 ~]# [root@node101 ~]# systemctl start cloudera-scm-server #啟動Cloudera Manager [root@node101 ~]# [root@node101 ~]# systemctl status cloudera-scm-server ● cloudera-scm-server.service - LSB: Cloudera SCM Server Loaded: loaded (/etc/rc.d/init.d/cloudera-scm-server; bad; vendor preset: disabled) Active: active (exited) since Thu 2019-02-21 12:53:21 CST; 2min 38s ago Docs: man:systemd-sysv-generator(8) Process: 7253 ExecStart=/etc/rc.d/init.d/cloudera-scm-server start (code=exited, status=0/SUCCESS) Feb 21 12:53:16 node101.yinzhengjie.org.cn systemd[1]: Starting LSB: Cloudera SCM Server... Feb 21 12:53:16 node101.yinzhengjie.org.cn su[7277]: PAM unable to dlopen(/usr/lib64/security/pam_fprintd.so): /usr/lib64/security/pam_fprintd.so: cannot open shared object file: No such file or directory Feb 21 12:53:16 node101.yinzhengjie.org.cn su[7277]: PAM adding faulty module: /usr/lib64/security/pam_fprintd.so Feb 21 12:53:16 node101.yinzhengjie.org.cn su[7277]: (to cloudera-scm) root on none Feb 21 12:53:21 node101.yinzhengjie.org.cn cloudera-scm-server[7253]: Starting cloudera-scm-server: [ OK ] Feb 21 12:53:21 node101.yinzhengjie.org.cn systemd[1]: Started LSB: Cloudera SCM Server. [root@node101 ~]#
4>.查看Cloudera Manager的日志
[root@node101 ~]# tail -10f /var/log/cloudera-scm-server/cloudera-scm-server.log #查看Cloudera Manager的服務的日志 2019-02-21 12:54:38,541 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Initializing SearchTemplateManager:2019-02-21T04:54:38.540Z 2019-02-21 12:54:38,673 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Generating entities:2019-02-21T04:54:38.673Z 2019-02-21 12:54:38,721 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Num entities:208 2019-02-21 12:54:38,721 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Generating documents:2019-02-21T04:54:38.721Z 2019-02-21 12:54:38,795 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Num docs:221 2019-02-21 12:54:38,823 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Constructing repo:2019-02-21T04:54:38.823Z 2019-02-21 12:54:39,541 INFO WebServerImpl:org.mortbay.log: jetty-6.1.26.cloudera.4 2019-02-21 12:54:39,631 INFO WebServerImpl:org.mortbay.log: Started SelectChannelConnector@0.0.0.0:7180 #注意，如果出現這一行提示，說明你的服務啟動成功啦！默認端口為7180，下面有WebServerImpl：Started Jetty server提示！ 2019-02-21 12:54:39,631 INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server. 2019-02-21 12:54:39,827 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Finished constructing repo:2019-02-21T04:54:39.827Z
5>.訪問Cloudera Manager 的WebUI界面
打開瀏覽器，推薦使用Chrome並將英語設置為首選語言，訪問：http://node101.yinzhengjie.org.cn:7180/cmf/login
