首先需要准備的軟件都有：Centos系統、SecureCRT 8.5、VMware Workstation Pro、jdk-8u172-linux-x64.tar.gz基本上這個軟件就是今天的戰場。
首先安裝VMware Workstation 並配置激活SecureCRT，關於這兩個軟件的安裝可以查看這個教程SecureCRT安裝、VMware Workstation 14 中文破解版安裝，VMware14安裝centos7。按照要求可以准備三台虛擬機。
一、 hadoop環境准備（可以先安裝一台虛擬機，在復制兩個，統一配置）
1.1查看當前操作系統環境以及主機映射關系
[root@jiang01 ~]# cat /etc/redhat-release CentOS Linux release 7.7.1908 (Core) [root@jiang01 ~]# uname -r 3.10.0-1062.el7.x86_64 [root@jiang01 ~]# free -m total used free shared buff/cache available Mem: 2827 196 2323 9 307 2475 Swap: 3071 0 3071 [root@jiang01 ~]# hostname -i
1.2 修改hostname,重啟三台機器，主機名稱將永久生效
# 1 使用hostnamectl命令 hostnamectl set-hostname xxx //刪除hostname hostnamectl set-hostname "" hostnamectl set-hostname 主機名 # 2 vi /etc/hostname
1.3 關閉防火牆
停止firewall systemctl stop firewalld.service
禁止firewall開機啟動 systemctl disable firewalld.service
1.4 hosts文件設置，只是為了不直接使用ip，使用主機名方便好記，
[root@jiang01 ~]# vi /etc/hosts [root@jiang01 ~]# cat /etc/hosts | grep jiang 192.168.1.16 jiang01 192.168.1.17 jiang02 192.168.1.17 jiang03
1.5 配置三台機器免密登錄 ，網上有很多種方式來配置，我選擇了其中最簡單的那種。命令依次為
生成私鑰並配置禹本紀的免密登錄：ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
拷貝私鑰 ssh-copy-id 用戶名@主機名
查看免秘是否成功 ssh 主機名 
1.6 配置網絡鏡像源
下載鏡像源
curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
清楚緩存
yum clean all && yum update && yum makecache
1.7 SELINUX關閉
sed -i 's#SELINUX=enforcing#SELINUX=disabled#' /etc/selinux/config
1.8 安裝jdk
[root@jiang01 ~]# mkdir app [root@jiang01 ~]# [root@jiang01 ~]# cd app/ [root@jiang01 app]# [root@jiang01 app]# rz rz waiting to receive. Starting zmodem transfer. Press Ctrl+C to cancel. Transferring installJDK.sh... 100% 593 bytes 593 bytes/sec 00:00:01 0 Errors Transferring jdk-8u172-linux-x64.tar.gz... 100% 186447 KB 1456 KB/sec 00:02:08 0 Errors [root@jiang01 app]# [root@jiang01 app]# [root@jiang01 app]# source ./installJDK.sh /usr/local/app/jdk1.8.0_172 安裝完畢！！！\r下面進行測試 Usage: javac <options> <source files> where possible options include: -g Generate all debugging info -g:none Generate no debugging info -g:{lines,vars,source} Generate only some debugging info -nowarn Generate no warnings -verbose Output messages about what the compiler is doing -deprecation Output source locations where deprecated APIs are used -classpath <path> Specify where to find user class files and annotation processors -cp <path> Specify where to find user class files and annotation processors -sourcepath <path> Specify where to find input source files -bootclasspath <path> Override location of bootstrap class files -extdirs <dirs> Override location of installed extensions -endorseddirs <dirs> Override location of endorsed standards path -proc:{none,only} Control whether annotation processing and/or compilation is done. -processor <class1>[,<class2>,<class3>...] Names of the annotation processors to run; bypasses default discovery process -processorpath <path> Specify where to find annotation processors -parameters Generate metadata for reflection on method parameters -d <directory> Specify where to place generated class files -s <directory> Specify where to place generated source files -h <directory> Specify where to place generated native header files -implicit:{none,class} Specify whether or not to generate class files for implicitly referenced files -encoding <encoding> Specify character encoding used by source files -source <release> Provide source compatibility with specified release -target <release> Generate class files for specific VM version -profile <profile> Check that API used is available in the specified profile -version Version information -help Print a synopsis of standard options -Akey[=value] Options to pass to annotation processors -X Print a synopsis of nonstandard options -J<flag> Pass <flag> directly to the runtime system -Werror Terminate compilation if warnings occur @<filename> Read options and filenames from file java版本為： java version "1.8.0_172" Java(TM) SE Runtime Environment (build 1.8.0_172-b11) Java HotSpot(TM) 64-Bit Server VM (build 25.172-b11, mixed mode) 是否刪除安裝包 1) 是 2) 否 #? 1 腳本執行完畢！！！ [root@jiang01 jdk1.8.0_172]#
java安裝腳本
#!/bin/bash #author :lizhenjiang #email:idea_zhenjiang@163.com mkdir /usr/local/app tar -zxf jdk* -C /usr/local/app cd /usr/local/app/jdk* home=`pwd` echo $home echo "export JAVA_HOME=${home}" >> /etc/profile echo "export PATH=\$PATH:\$JAVA_HOME/bin" >> /etc/profile echo "安裝完畢！！！\r下面進行測試" #配置完成，下面的是測試 source /etc/profile javac echo "java版本為：" java -version echo "是否刪除安裝包" select var in 是 否 do break done if [ var = 是 ] then rm jdk-* echo "安裝包已刪除" fi echo "腳本執行完畢！！！"
1.9 編寫集群管理腳本，這些腳本在其它課程上有很多，都可以拿來使用
批量執行服務腳本命令
View Code
運行效果
View Code
[root@jiang01 jdk1.8.0_172]# xcall.sh "yum -y install rsync" ============= jiang01 : yum -y install rsync ============ Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.aliyun.com * extras: mirrors.aliyun.com * updates: mirrors.aliyun.com Resolving Dependencies --> Running transaction check ---> Package rsync.x86_64 0:3.1.2-6.el7_6.1 will be installed --> Finished Dependency Resolution Dependencies Resolved ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: rsync x86_64 3.1.2-6.el7_6.1 base 404 k Transaction Summary ================================================================================ Install 1 Package Total download size: 404 k Installed size: 815 k Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : rsync-3.1.2-6.el7_6.1.x86_64 1/1 Verifying : rsync-3.1.2-6.el7_6.1.x86_64 1/1 Installed: rsync.x86_64 0:3.1.2-6.el7_6.1 Complete! 命令執行成功 ============= jiang02 : yum -y install rsync ============ Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.aliyun.com * extras: mirrors.aliyun.com * updates: mirrors.aliyun.com Resolving Dependencies --> Running transaction check ---> Package rsync.x86_64 0:3.1.2-6.el7_6.1 will be installed --> Finished Dependency Resolution Dependencies Resolved ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: rsync x86_64 3.1.2-6.el7_6.1 base 404 k Transaction Summary ================================================================================ Install 1 Package Total download size: 404 k Installed size: 815 k Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : rsync-3.1.2-6.el7_6.1.x86_64 1/1 Verifying : rsync-3.1.2-6.el7_6.1.x86_64 1/1 Installed: rsync.x86_64 0:3.1.2-6.el7_6.1 Complete! 命令執行成功 ============= jiang03 : yum -y install rsync ============ Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.aliyun.com * extras: mirrors.aliyun.com * updates: mirrors.aliyun.com Resolving Dependencies --> Running transaction check ---> Package rsync.x86_64 0:3.1.2-6.el7_6.1 will be installed --> Finished Dependency Resolution Dependencies Resolved ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: rsync x86_64 3.1.2-6.el7_6.1 base 404 k Transaction Summary ================================================================================ Install 1 Package Total download size: 404 k Installed size: 815 k Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : rsync-3.1.2-6.el7_6.1.x86_64 1/1 Verifying : rsync-3.1.2-6.el7_6.1.x86_64 1/1 Installed: rsync.x86_64 0:3.1.2-6.el7_6.1 Complete! 命令執行成功 [root@jiang01 jdk1.8.0_172]#
批量同步文件腳本：
#!/bin/bash #author :lizhenjiang #email:idea_zhenjiang@163.com #判斷用戶是否傳參 if [ $# -lt 1 ];then echo "請輸入參數"; exit fi #獲取文件路徑 file=$@ #獲取子路徑 filename=`basename $file` #獲取父路徑 dirpath=`dirname $file` #獲取完整路徑 cd $dirpath fullpath=`pwd -P` #同步文件到DataNode for (( i=2;i<=3;i++ )) do tput setaf 6 echo =========== jiang0${i} : $file =========== tput setaf 9 #遠程執行命令 rsync -lr $filename `whoami`@jiang0${i}:$fullpath #判斷命令是否執行成功 if [ $? == 0 ];then echo "命令執行成功" fi done
[root@jiang01 app]# vi /usr/bin/xrsync.sh [root@jiang01 app]# [root@jiang01 app]# [root@jiang01 app]# chmod +x /usr/bin/xrsync.sh [root@jiang01 app]# [root@jiang01 app]# xcall.sh cat /etc/hosts ============= jiang01 : cat /etc/hosts ============ 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.1.16 jiang01 192.168.1.17 jiang02 192.168.1.18 jiang03 命令執行成功 ============= jiang02 : cat /etc/hosts ============ 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.1.16 jiang01 192.168.1.17 jiang02 192.168.1.18 jiang03 命令執行成功 ============= jiang03 : cat /etc/hosts ============ 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.1.16 jiang01 192.168.1.17 jiang02 192.168.1.18 jiang03 命令執行成功
[root@jiang01 app]# xcall.sh mkdir /usr/local/app/ ============= jiang01 : mkdir /usr/local/app/ ============ mkdir: cannot create directory ‘/usr/local/app/’: File exists ============= jiang02 : mkdir /usr/local/app/ ============ 命令執行成功 ============= jiang03 : mkdir /usr/local/app/ ============ 命令執行成功 [root@jiang01 app]# ll total 4 drwxr-xr-x. 8 10 143 4096 Sep 26 11:02 jdk1.8.0_172 [root@jiang01 app]# xrsync.sh jdk1.8.0_172/ =========== jiang02 : jdk1.8.0_172/ =========== 命令執行成功 =========== jiang03 : jdk1.8.0_172/ =========== 命令執行成功 [root@jiang01 app]# [root@jiang01 app]# [root@jiang01 app]# xrsync.sh /etc/profile =========== jiang02 : /etc/profile =========== 命令執行成功 =========== jiang03 : /etc/profile =========== 命令執行成功 [root@jiang01 app]#
2.0 同步網絡時鍾，集群要保證時鍾的一致性，以免集群出現問題
yum install ntpdate ntpdate us.pool.ntp.org hwclock -w 顯示時區： date -R #安裝ntpdate工具 yum install ntpdate -y #使用ntpdate校時（后面的是ntp服務器） ntpdate pool.ntp.org hwclock -w 定時任務（高可用集群時間同步很重要） /5 * /usr/sbin/ntpdate pool.ntp.org >/dev/null 2>&1 每五分鍾同步一次時間
到現在基礎環境已經准備就緒，已經完全准備好了安裝大數據相關的集群。