前言

本節(jié)內(nèi)容是關(guān)于hadoop集群下yarn服務(wù)的高可用搭建，以及其發(fā)生故障轉(zhuǎn)移的處理，同樣需要依賴zookeeper集群的實(shí)現(xiàn)，實(shí)現(xiàn)該集群搭建時(shí)，我們要預(yù)先保證zookeeper集群是啟動(dòng)狀態(tài)。yarn的高可用同樣依賴zookeeper的臨時(shí)節(jié)點(diǎn)及監(jiān)控，實(shí)現(xiàn)服務(wù)的故障轉(zhuǎn)移。其ResourceManager的節(jié)點(diǎn)任務(wù)同樣存儲(chǔ)于zookeeper集群中，實(shí)現(xiàn)數(shù)據(jù)的共享。

正文

• 集群規(guī)劃

YARN高可用

hadoop101	hadoop102	hadoop103
ResourceManager	ResourceManager	ResourceManager
NodeManager	NodeManager	NodeManager
Zookeeper	Zookeeper	Zookeeper

• ?修改yarn服務(wù)配置文件yarn-site.xml

-?yarn-site.xml

<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>
    <!-- 指定 MR 走 shuffle -->
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <!-- 啟用 resourcemanager ha -->
    <property>
        <name>yarn.resourcemanager.ha.enabled</name>
        <value>true</value>
    </property>
    <!-- 聲明三臺(tái) resourcemanager 的地址 -->
    <property>
        <name>yarn.resourcemanager.cluster-id</name>
        <value>cluster-yarn</value>
    </property>
    <!--指定 resourcemanager 的邏輯列表-->
    <property>
        <name>yarn.resourcemanager.ha.rm-ids</name>
        <value>rm1,rm2,rm3</value>
    </property>
    <!-- ========== rm1 的配置 ========== -->
    <!-- 指定 rm1 的主機(jī)名 -->
    <property>
        <name>yarn.resourcemanager.hostname.rm1</name>
        <value>hadoop101</value>
    </property>
    <!-- 指定 rm1 的 web 端地址 -->
    <property>
        <name>yarn.resourcemanager.webapp.address.rm1</name>
        <value>hadoop101:8088</value>
    </property>
    <!-- 指定 rm1 的內(nèi)部通信地址 -->
    <property>
        <name>yarn.resourcemanager.address.rm1</name>
        <value>hadoop101:8032</value>
    </property>
    <!-- 指定 AM 向 rm1 申請(qǐng)資源的地址 -->
    <property>
        <name>yarn.resourcemanager.scheduler.address.rm1</name>
        <value>hadoop101:8030</value>
    </property>
    <!-- 指定供 NM 連接的地址 -->
    <property>
        <name>yarn.resourcemanager.resource-tracker.address.rm1</name>
        <value>hadoop101:8031</value>
    </property>
    <!-- ========== rm2 的配置 ========== -->
    <!-- 指定 rm2 的主機(jī)名 -->
    <property>
        <name>yarn.resourcemanager.hostname.rm2</name>
        <value>hadoop102</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address.rm2</name>
        <value>hadoop102:8088</value>
    </property>
    <property>
        <name>yarn.resourcemanager.address.rm2</name>
        <value>hadoop102:8032</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address.rm2</name>
        <value>hadoop102:8030</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address.rm2</name>
        <value>hadoop102:8031</value>
    </property>
    <!-- ========== rm3 的配置 ========== -->
    <!-- 指定 rm1 的主機(jī)名 -->
    <property>
        <name>yarn.resourcemanager.hostname.rm3</name>
        <value>hadoop103</value>
    </property>
    <!-- 指定 rm1 的 web 端地址 -->
    <property>
        <name>yarn.resourcemanager.webapp.address.rm3</name>
        <value>hadoop103:8088</value>
    </property>
    <!-- 指定 rm1 的內(nèi)部通信地址 -->
    <property>
        <name>yarn.resourcemanager.address.rm3</name>
        <value>hadoop103:8032</value>
    </property>
    <!-- 指定 AM 向 rm1 申請(qǐng)資源的地址 -->
    <property>
        <name>yarn.resourcemanager.scheduler.address.rm3</name>
        <value>hadoop103:8030</value>
    </property>
    <!-- 指定供 NM 連接的地址 -->
    <property>
        <name>yarn.resourcemanager.resource-tracker.address.rm3</name>
        <value>hadoop103:8031</value>
    </property>
    <!-- 指定 zookeeper 集群的地址 -->
    <property>
        <name>yarn.resourcemanager.zk-address</name>
        <value>hadoop101:2181,hadoop102:2181,hadoop103:2181</value>
    </property>
    <!-- 啟用自動(dòng)恢復(fù) -->
    <property>
        <name>yarn.resourcemanager.recovery.enabled</name>
        <value>true</value>
    </property>
    <!-- 指定 resourcemanager 的狀態(tài)信息存儲(chǔ)在 zookeeper 集群 -->
    <property>
        <name>yarn.resourcemanager.store.class</name>
        <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
    </property>
    <!-- 環(huán)境變量的繼承 -->
    <property>
        <name>yarn.nodemanager.env-whitelist</name>
        <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
    </property>
</configuration>

• 同步配置文件yarn-site.xml?

- 命令：hsync yarn-site.xml

• 啟動(dòng)yarn集群服務(wù)

?-命令：start-yarn.sh

• ?訪問(wèn)yarn集群服務(wù)

-訪問(wèn)yarn集群服務(wù)地址查看yarn服務(wù)狀態(tài)

?- 通過(guò)命令查看

• yarn服務(wù)故障轉(zhuǎn)移

- 將hadoop101節(jié)點(diǎn)下線，查看ResourceManager的節(jié)點(diǎn)切換

- hadoop102切換為了主節(jié)點(diǎn)

結(jié)語(yǔ)

關(guān)于hadoop集群之YARN高可用實(shí)現(xiàn)自動(dòng)故障轉(zhuǎn)移的內(nèi)容到這里就結(jié)束了，我們下期見(jiàn)。。。。。。文章來(lái)源地址http://www.zghlxwxcb.cn/news/detail-629056.html

到了這里，關(guān)于（十三）大數(shù)據(jù)實(shí)戰(zhàn)——hadoop集群之YARN高可用實(shí)現(xiàn)自動(dòng)故障轉(zhuǎn)移的文章就介紹完了。如果您還想了解更多內(nèi)容，請(qǐng)?jiān)谟疑辖撬阉鱐OY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！