在先電IAAS平台中,搭建先電bigdata平台


以兩台節點為例組件Hadoop分布式集群,這里采用的系統版本Centos7,如下表所示:

主機名

內存

硬盤

IP地址

角色

master

8192MB

100G

192.168.200.131

Ambari-Server

Ambari-Agent

slave1

4096MB

100G

192.168.200.133

Ambari-Agent

1.1 配置主機名

# master

# hostnamectl set-hostname master

# hostname

master

# slave1

# hostnamectl set-hostname slave1

# hostname

slave1

1.2 修改hosts文件

# master

# vi /etc/hosts

192.168.200.131 master.hadoop master

192.168.200.133 slave1.hadoop

# slave1

# vi /etc/hosts

192.168.200.131 master.hadoop

192.168.200.133 slave1.hadoop slave1

注意:主機名映射采用FQDN格式。

1.3 修改yum

# master

將XianDian-BigData-v2.2.iso掛在到/mnt目錄下,將其中的ambari解壓到/opt目錄下,並在master節點配置ftp服務。 

注意:

因為安裝大數據相關軟件包時,可能會用到相關依賴軟件包,所以需要配置Centos7 Yum源,這里可以采用IAASCentos7 Yum源。

# master & slave1

# cd /etc/yum.repos.d/

# rm -vf *

配置Yum

# vi ambari.repo

[centos7]

baseurl=ftp://192.168.2.10/centos7/

注:具體的yum源根據真實環境配置,本次為實驗室測試環境 )

gpgcheck=0

enabled=1

name=centos

[ambari]

name=ambari

baseurl=ftp://10.0.3.61/ambari

注:具體的yum源根據真實環境配置,本次為實驗室測試環境 )

enabled=1

gpgcheck=0

 

# master

# yum -y install httpd

/mnt/文件夾中HDP-2.6.1.0和HDP-UTILS-1.1.0.21兩個文件夾拷貝到/var/www/html/目錄下。啟動httpd服務。

# systemctl enable httpd.service

# systemctl status httpd.service

 

1.4 配置ntp

# master

# yum -y install ntp

# vi /etc/ntp.conf

注釋或者刪除以下四行

server 0.centos.pool.ntp.org iburst

server 1.centos.pool.ntp.org iburst

server 2.centos.pool.ntp.org iburst

server 3.centos.pool.ntp.org iburst

添加以下兩行

server 127.127.1.0

fudge 127.127.1.0 stratum 10

#systemctl enable ntpd

#systemctl start  ntpd

# slave1

# yum -y install ntpdate

# ntpdate master

# systemctl enable ntpdate

1.5 配置SSH

# master & slave1

檢查2個節點是否可以通過無密鑰相互訪問,如果未配置,則進行SSH無密碼公鑰認證配置。如下:

# yum install openssh-clients

# ssh-keygen

# ssh-copy-id master.hadoop

# ssh-copy-id slave1.hadoop

ssh登錄遠程主機查看是否成功

# ssh master.hadoop

# exit

# ssh slave1.hadoop

# exit

1.6 禁用Transparent Huge Pages

操作系統后台有一個叫做khugepaged的進程,它會一直掃描所有進程占用的內存,在可能的情況下會把4kpage交換為Huge Pages,在這個過程中,對於操作的內存的各種分配活動都需要各種內存鎖,直接影響程序的內存訪問性能,並且,這個過程對於應用是透明的,在應用層面不可控制,對於專門為4k page優化的程序來說,可能會造成隨機的性能下降現象。

# master & slave1

# cat /sys/kernel/mm/transparent_hugepage/enabled

[always] madvise never

# echo never > /sys/kernel/mm/transparent_hugepage/enabled

# echo never > /sys/kernel/mm/transparent_hugepage/defrag

# cat /sys/kernel/mm/transparent_hugepage/enabled

always madvise [never]

重啟后失效,需要再次執行

1.7 安裝配置JDK

# master

# mkdir /usr/jdk64/  

# mount -o loop XianDian-BigData-v2.2.iso /mnt/

# cd /mnt/

# tar -zxvf jdk-8u77-linux-x64.tar.gz -C /usr/jdk64/

# vi /etc/profile

export JAVA_HOME=/usr/jdk64/jdk1.8.0_77

export PATH=$JAVA_HOME/bin:$PATH

# source /etc/profile

#java –version

java version "1.8.0_77"

Java(TM) SE Runtime Environment (build 1.8.0_77-b03)

Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)

# slave1

#mkdir /usr/jdk64/

#scp 172.24.14.106://mnt/jdk-8u77-linux-x64.tar.gz .

# tar -zxvf jdk-8u77-linux-x64.tar.gz -C /usr/jdk64/

# vi /etc/profile

export JAVA_HOME=/usr/jdk64/jdk1.8.0_77

export PATH=$JAVA_HOME/bin:$PATH

# source /etc/profile

#java –version

java version "1.8.0_77"

Java(TM) SE Runtime Environment (build 1.8.0_77-b03)

Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)

配置ambari-server

# master

# yum -y install  ambari-server

2.1 安裝MariaDB數據庫

# master

# yum install  mariadb mariadb-server mysql-connector-java

啟動服務

# systemctl enable mariadb

# systemctl start mariadb

配置MySQL

#mysql_secure_installation

enter確認后設置數據庫root密碼,我們這里設置為“bigdata

Remove anonymous users? [Y/n] y

Disallow root login remotely? [Y/n] n

Remove test database and access to it? [Y/n] y

Reload privilege tables now? [Y/n] y

創建ambari數據庫

# mysql -uroot -pbigdata

MariaDB [(none)]> create database ambari;

MariaDB [(none)]> grant all privileges on ambari.* to 'ambari'@'localhost' identified by 'bigdata';

MariaDB [(none)]> grant all privileges on ambari.* to 'ambari'@'%' identified by 'bigdata';

MariaDB [(none)]> use ambari;

MariaDB [ambari]> source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql

MariaDB [ambari]> quit

 

2.2 安裝配置ambari-server

# master

vi /etc/profile

export buildNumber=2.6.0.0

# ambari-server setup

WARNING: SELinux is set to 'permissive' mode and temporarily disabled.

OK to continue [y/n] (y)?

Customize user account for ambari-server daemon [y/n] (n)? n

Checking JDK...

[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8

[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7

[3] Custom JDK

==============================================================================

Enter choice (1): 3

Path to JAVA_HOME: /usr/jdk64/jdk1.8.0_77

Validating JDK on Ambari Server...done.

Completing setup...

Configuring database...

Enter advanced database configuration [y/n] (n)? y

Configuring database...

====================================================================

Choose one of the following options:

[1] - PostgreSQL (Embedded)

[2] - Oracle

[3] - MySQL

[4] - PostgreSQL

[5] - Microsoft SQL Server (Tech Preview)

[6] - SQL Anywhere

====================================================================

Enter choice (1): 3

Hostname (localhost):

Port (3306):

Database name (ambari):

Username (ambari):

Enter Database Password (bigdata):

Proceed with configuring remote database connection properties [y/n] (y)?

Ambari Server 'setup' completed successfully.

 

# ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar

 

啟動ambari-server服務

# ambari-server start

登陸界面http://192.168.200.131:8080/

登錄用戶名密碼為adminadmin

 

配置ambari-agent

# master & slave1

# yum -y install ambari-agent

# vi /etc/ambari-agent/conf/ambari-agent.ini

[server]

hostname= master.hadoop

# ambari-agent restart

# tail -f /var/log/ambari-agent/ambari-agent.log

INFO 2017-01-12 09:44:20,919 Controller.py:265 - Heartbeat response received (id = 1340)

INFO 2017-01-12 09:44:30,820 Heartbeat.py:78 - Building Heartbeat: {responseId = 1340, timestamp = 1484214270820, commandsInProgress = False, componentsMapped = True}


免責聲明!

本站轉載的文章為個人學習借鑒使用,本站對版權不負任何法律責任。如果侵犯了您的隱私權益,請聯系本站郵箱yoyou2525@163.com刪除。



 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM