【大數據】分布式集群部署

1、集群規划部署

節點名稱	NN1	NN2	DN	RM	NM
hadoop01	NameNode		DataNode		NodeManager
hadoop02		SecondaryNameNode	DataNode	ResourceManager	NodeManager
hadoop03			DataNode		NodeManager

 2、參考單機部署，拷貝安裝目錄至相同目錄，使用ln -s 建立軟連接

 

 

3、修改配置文件參數及sh啟動文件--根據集群規划部署配置

 

slaves：記錄了機器名

*.sh：修改JAVA_HOME

yarn-site.xml 

<configuration>

<!-- Site specific YARN configuration properties -->
          <!-- NodeManager獲取數據的方式是shuffle-->
        <property>
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
        </property>
          <!-- 指定YARN的老大（resourcemanager）的地址 -->
        <property>
                <name>yarn.resourcemanager.hostname</name>
                <value>hadoop02</value>
        </property>
</configuration>

hdfs-site.xml 

<configuration>
        <!-- 指定HDFS保存數據副本數量 --> 
        <property>
                <name>dfs.replication</name>
                <value>3</value>
        </property>
    <property>
        <name>dfs.permissions</name>
        <value>false</value>
        <description>
            If "true", enable permission checking in HDFS.
            If "false", permission checking is turned off,
            but all other behavior is unchanged.
            Switching from one parameter value to the other does not change the mode,
            owner or group of files or directories.
        </description>
    </property>
    <!-- 設置secondname的端口   -->
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>hadoop02:50090</value>
    </property>
</configuration>

mapred-site.xml 

<configuration>
          <!-- 告訴hadoop以后MR運行在yarn上 -->
        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>
</configuration>

 

core-site.xml

<configuration>
        <!-- 用來指定hdfs的老大（NameNode）的地址 -->
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://hadoop01:9000</value>
        </property>
          <!-- 用來指定Hadoop運行時產生文件的存放目錄 -->
        <property>
                <name>hadoop.tmp.dir</name>
                <value>/hadoop/tmp</value>
        </property>
</configuration>

 

4、由於是在單機基礎上升級擴展，需要刪除hadoop.tmp.dir目錄文件，並用root授權 chmod 777 -R /hadoop

5、重新格式化：hdfs namenode -foamate

6、配置拷貝：scp -r /home/hadoop/Soft/hadoop-2.7.6/etc/hadoop hadoop@hadoop03:/home/hadoop/Soft/hadoop-2.7.6/etc/

7、Hadoop01：start-dfs.sh

8、Hadoop02：start-yarn.sh

10、使用jps查看進程

 

 

參考：

https://blog.csdn.net/frank409167848/article/details/80968531

https://www.cnblogs.com/frankdeng/p/9047698.html