上篇簡單說明了多節(jié)點的hadoop節(jié)點怎么安裝，但是沒有細致的分析hadoop相關配置，具體怎么根據(jù)環(huán)境進行配置，接下來我們對這些配置進行講解

一. 配置說明

1. hadoop各進程環(huán)境配置

舉個例子：

配置Namenode使用parallelGC和4GB的Java堆，應該在hadoop-env.sh中添加以下語句:

export HDFS_NAMENODE_OPTS="-XX:+UseParallelGC -Xmx4g"

?
其他有用的配置

• HADOOP_PID_DIR : 各進程pid存儲目錄
• HADOOP_LOG_DIR ：各進程日志目錄。日志文件會自動創(chuàng)建
• HADOOP_HEAPSIZE_MAX：hadoop使用的最大java堆內(nèi)存。默認的，hadoop讓JVM決定使用多少。這個值可以被每個進程所設置的值所覆蓋。例如，設置HADOOP_HEAPSIZE_MAX=1g和hadoop_namende_opts ="-Xmx5g"將為NameNode配置5GB的堆。

注意：

In most cases, you should specify the HADOOP_PID_DIR and HADOOP_LOG_DIR directories such that they can only be written to by the users that are going to run the hadoop daemons. Otherwise there is the potential for a symlink attack.

對于大多數(shù)情況，你需要設置HADOOP_PID_DIR和HADOOP_LOG_DIR，因為他們只允許啟動用戶去操作，這就避免了潛在的symlink attack。

?
ing
在系統(tǒng)級的shell環(huán)境中配置HADOOP_HOME也是傳統(tǒng)的做法。例如，/etc/profile.d中的一個簡單腳本:

  HADOOP_HOME=/path/to/hadoop
  export HADOOP_HOME

?

2. hadoop各進程配置

2.1. etc/hadoop/core-site.xml

?

fs.defaultFS
例如，如果將fs.defaultFS配置為hdfs://namenode:9000，則Hadoop客戶端應用程序?qū)⒛J連接到名為namenode的HDFS名稱節(jié)點，并使用9000端口進行通信。
它簡化了應用程序代碼、支持多集群環(huán)境和方便切換文件系統(tǒng)等功能。

?

2.2. etc/hadoop/hdfs-site.xml

2.2.1. NameNode

?

2.2.2. datanode

?

2.3. etc/hadoop/yarn-site.xml

2.3.1. ResourceManager and NodeManager

?

2.3.2. ResourceManager

?

2.3.3. NodeManager

2.3.4. History Server

2.4. etc/hadoop/mapred-site.xml

2.4.1. MapReduce Applications

2.4.2. MapReduce JobHistory Server

?

3. Monitoring Health of NodeManagers

Hadoop provides a mechanism by which administrators can configure the NodeManager to run an administrator supplied script periodically to determine if a node is healthy or not.

管理員可以提供一個腳本，周期性的檢查nodemanager是否健康。
?

基本邏輯：

Administrators can determine if the node is in a healthy state by performing any checks of their choice in the script. If the script detects the node to be in an unhealthy state, it must print a line to standard output beginning with the string ERROR. The NodeManager spawns the script periodically and checks its output. If the script’s output contains the string ERROR, as described above, the node’s status is reported as unhealthy and the node is black-listed by the ResourceManager. No further tasks will be assigned to this node. However, the NodeManager continues to run the script, so that if the node becomes healthy again, it will be removed from the blacklisted nodes on the ResourceManager automatically. The node’s health along with the output of the script, if it is unhealthy, is available to the administrator in the ResourceManager web interface. The time since the node was healthy is also displayed on the web interface.

管理員通過腳本可以決定節(jié)點是否處于健康狀態(tài)。如果腳本檢測到node處于不健康狀態(tài)時，它需要打印出ERROR為首的日志。
nodemanager周期性的產(chǎn)生腳本，并檢查其輸出。如果腳本輸出ERROR，則node狀態(tài)為不健康狀態(tài)，resourcemanager將其加入黑名單，之后將不會有任務分配到這個節(jié)點。但nodemanager會繼續(xù)運行此腳本，以便當其再次健康時，resourcemanager會將其移出黑名單。
管理員可以在yarn的web觀察到node的健康狀態(tài)。

?

?

The health checker script is not supposed to give ERROR if only some of the local disks become bad. NodeManager has the ability to periodically check the health of the local disks (specifically checks nodemanager-local-dirs and nodemanager-log-dirs) and after reaching the threshold of number of bad directories based on the value set for the config property yarn.nodemanager.disk-health-checker.min-healthy-disks, the whole node is marked unhealthy and this info is sent to resource manager also. The boot disk is either raided or a failure in the boot disk is identified by the health checker script.

當本地磁盤損壞時，不支持health checker輸出ERROR，即不能判定為node壞掉，nodemanager有能力周期性的檢查本地磁盤的好壞，具體的，是nodemanager-local-dirs、nodemanager-log-dirs這兩個目錄。當?shù)竭_yarn.nodemanager.disk-health-checker.min-healthy-disks 的上限時，整個node將被標記為不健康狀態(tài)，并將信息發(fā)送給resourcemanager。

當磁盤被突襲或者啟動失敗時，health checker script將會識別。

?
?

二. 配置示例

1. core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://namenodeip:9000</value>
    </property>

    <property>
        <name>io.file.buffer.size</name>
        <value>131072</value>
    </property>


</configuration>

?

2. hdfs-site.xml

<configuration>
    <!-- ===========namenode===========   -->

    <property>
        <name>dfs.namenode.name.dir</name>
        <value>/opt/data/hdfs/namenode,/opt/data02/hdfs/namenode</value>
        <description>If this is a comma-delimited list of directories then the name table is replicated in all of the
            directories, for redundancy.
            Path on the local filesystem where the NameNode stores the namespace and transactions logs persistently.
            用于保存Namenode的namespace和事務日志的路徑
        </description>

    </property>

    <property>
        <name>dfs.blocksize</name>
        <value>134217728</value>
        <description>HDFS blocksize of 256MB for large file-systems.
            對于大型文件系統(tǒng),使用256MB作為一個塊的大小. 這里是128MB。
        </description>
    </property>

    <property>
        <name>dfs.namenode.handler.count</name>
        <value>4096</value>
        <description>
            More NameNode server threads to handle RPCs from large number of DataNodes.
            更多 NameNode 服務器線程來處理來自大量 DataNode 的 RPC請求。
        </description>
    </property>

    <property>
        <name>dfs.hosts.exclude</name>
        <value>/home/${user.name}/software/hadoop/etc/hadoop/decommission</value>
        <description>If necessary, use these files to control the list of allowable datanodes.
            不允許加入的datanode
        </description>
    </property>

    <!-- ===========namenode===========   -->


    <!-- ===========datanode===========   -->
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>/opt/data/hdfs/data,/opt/data02/hdfs/data</value>
        <description>If necessary, use these files to control the list of allowable datanodes.
            不允許加入的datanode
        </description>
    </property>


    <!-- ===========datanode===========   -->


    <property>
        <name>dfs.datanode.data.dir.perm</name>
        <value>750</value>
    </property>

    <property>
        <name>dfs.block.local-path-access.user</name>
        <value>taiyi,hbase</value>
    </property>

    <property>
        <name>dfs.webhdfs.enabled</name>
        <value>true</value>
    </property>

?

3. yarn-site.xml

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>


<configuration>

    <!-- ResourceManager and NodeManager    -->
    <property>
        <name>yarn.acl.enable</name>
        <value>false</value>
        <description>
            Enable ACLs? Defaults to false.
        </description>
    </property>

    <property>
        <name>yarn.admin.acl</name>
        <value>yarn</value>
        <description>
            ACL to set admins on the cluster.
            ACLs are of for comma-separated-usersspacecomma-separated-groups.
            Defaults to special value of * which means anyone.
            Special value of just space means no one has access.
            設置集群的acl權限 ing
        </description>
    </property>


    <property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
        <description>
            Configuration to enable or disable log aggregation
            日志聚合
        </description>
    </property>


    <!-- ResourceManager and NodeManager    -->


    <!--  Configurations for ResourceManager:   -->
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>resourcemanagerIp:8832</value>
        <description>
            ResourceManager host:port for clients to submit jobs.
            host:port If set, overrides the hostname set in yarn.resourcemanager.hostname.
            用于客戶端提交任務
        </description>
    </property>


    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>resourcemanagerIp:8830</value>
        <description>
            ResourceManager host:port for ApplicationMasters to talk to Scheduler to obtain resources.
        </description>
    </property>


    <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>resourcemanagerIp:8831</value>
        <description>
            ResourceManager host:port for NodeManagers.
            host:port If set, overrides the hostname set in yarn.resourcemanager.hostname.
        </description>
    </property>


    <property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>resourcemanagerIp:8833</value>
        <description>
            ResourceManager host:port for administrative commands.
        </description>
    </property>


    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>resourcemanagerIp:8888</value>
        <description>
            ResourceManager web-ui host:port.
        </description>
    </property>


    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>resourcemanagerHostname</value>
        <description>
            ResourceManager host.
            host Single hostname that can be set in place of setting all yarn.
            resourcemanager*address resources.
            Results in default ports for ResourceManager components.
        </description>
    </property>


    <property>
        <name>yarn.resourcemanager.scheduler.class</name>
        <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
        <description>
            ResourceManager Scheduler class.
            CapacityScheduler (recommended), FairScheduler (also recommended), or FifoScheduler.
            Use a fully qualified class name, e.g.,
            org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.
        </description>
    </property>


    <property>
        <name>yarn.scheduler.minimum-allocation-mb</name>
        <value>2048</value>
        <description>
            Minimum limit of memory to allocate to each container request at the Resource Manager.
            In MBs
            每個container最小啟動資源
        </description>
    </property>
    <property>
        <name>yarn.scheduler.maximum-allocation-mb</name>
        <value>8192</value>
        <description>
            Maximum limit of memory to allocate to each container request at the Resource Manager.
        </description>
    </property>
    <property>
        <name>yarn.scheduler.maximum-allocation-vcores</name>
        <value>8</value>
    </property>
    <property>
        <name>yarn.scheduler.minimum-allocation-vcores</name>
        <value>1</value>
    </property>


    <!--  Configurations for ResourceManager:   -->


    <!--  Configurations for NodeManager:  -->

    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>61440</value>
        <description>
            Resource i.e. available physical memory, in MB, for given NodeManager
            Defines total available resources on the NodeManager to be made available to running containers
        </description>
    </property>


    <property>
        <name>yarn.nodemanager.vmem-pmem-ratio</name>
        <value>4</value>
        <description>
            Maximum ratio by which virtual memory usage of tasks may exceed physical memory
            The virtual memory usage of each task may exceed its physical memory limit by this ratio.
            The total amount of virtual memory used by tasks on the NodeManager may exceed its physical memory usage by
            this ratio.
            任務的虛擬內(nèi)存使用可能超過物理內(nèi)存的最大比例
        </description>
    </property>


    <property>
        <name>yarn.nodemanager.local-dirs</name>
        <value>/data/yarn/nm-local-dir,/data02/yarn/nm-local-dir</value>
    </property>


    <property>
        <name>yarn.nodemanager.log-dirs</name>
        <value>/home/taiyi/hadoop/yarn/userlogs</value>
    </property>

    <property>
        <name>yarn.nodemanager.log.retain-seconds</name>
        <value>10800</value>
        <description>
            Default time (in seconds) to retain log files on the NodeManager
            Only applicable if log-aggregation is disabled.
        </description>
    </property>


    <property>
        <name>yarn.nodemanager.remote-app-log-dir</name>
        <value>/home/taiyi/hadoop/yarn/containerlogs</value>
        <description>
            HDFS directory where the application logs are moved on application completion.
            Need to set appropriate permissions. Only applicable if log-aggregation is enabled.
        </description>
    </property>

    <property>
        <name>yarn.nodemanager.remote-app-log-dir-suffix</name>
        <value>logs</value>
        <description>
            Suffix appended to the remote log dir.
            Logs will be aggregated to ${yarn.nodemanager.remote-app-log-dir}/${user}/${thisParam}
            Only applicable if log-aggregation is enabled.
        </description>
    </property>


    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle,spark_shuffle</value>
        <description>
            Shuffle service that needs to be set for Map Reduce applications.
        </description>
    </property>

    <property>
        <name>yarn.nodemanager.aux-services.spark_shuffle.class</name>
        <value>org.apache.spark.network.yarn.YarnShuffleService</value>
    </property>


    <!--  Configurations for NodeManager:  -->


    <!--Configurations for History Server (Needs to be moved elsewhere): -->
    <property>
        <name>yarn.log-aggregation.retain-seconds</name>
        <value>2592000</value>
        <description>
            How long to keep aggregation logs before deleting them. -1 disables.
            Be careful, set this too small and you will spam the name node.
        </description>
    </property>


    <property>
        <name>yarn.log-aggregation.retain-check-interval-seconds</name>
        <value>3600</value>
        <description>
            Time between checks for aggregated log retention.
            If set to 0 or a negative value then the value is computed
            as one-tenth of the aggregated log retention time.
            Be careful, set this too small and you will spam the name node.
        </description>
    </property>


</configuration>

?

4. mapred-site.xml

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>

    <!-- Configurations for MapReduce Applications:   -->

    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
        <description>
            Execution framework set to Hadoop YARN.
        </description>
    </property>


    <property>
        <name>mapreduce.map.memory.mb</name>
        <value>2048</value>
        <description>
            Larger resource limit for maps.
        </description>
    </property>


    <property>
        <name>mapreduce.map.java.opts</name>
        <value>-Xmx2048m</value>
        <description>
            Larger heap-size for child jvms of maps.
        </description>
    </property>


    <property>
        <name>mapreduce.task.io.sort.mb</name>
        <value>512</value>
        <description>
            Higher memory-limit while sorting data for efficiency.
        </description>
    </property>


    <property>
        <name>mapreduce.task.io.sort.factor</name>
        <value>100</value>
        <description>More streams merged at once while sorting files.
        </description>
    </property>


    <property>
        <name>mapreduce.reduce.shuffle.parallelcopies</name>
        <value>50</value>
        <description>Higher number of parallel copies run
            by reduces to fetch outputs from very large number of maps.
        </description>
    </property>

    <!-- Configurations for MapReduce Applications:   -->

    <!--Configurations for MapReduce JobHistory Server:-->
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>joghistorynodeIp:10020</value>
    </property>


    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>joghistorynodeIp:19888</value>
    </property>


<!--    <property>-->
<!--        <name>mapreduce.jobhistory.intermediate-done-dir</name>-->
<!--        <value>/home/taiyi/yarn/mrhistory/tmp</value>-->
<!--        <description>-->
<!--            Directory where history files are written by MapReduce jobs-->
<!--        </description>-->
<!--    </property>-->


<!--    <property>-->
<!--        <name>mapreduce.jobhistory.done-dir</name>-->
<!--        <value>/home/taiyi/yarn/mrhistory/done</value>-->
<!--        <description>-->
<!--            Directory where history files are managed by the MR JobHistory Server.-->
<!--        </description>-->
<!--    </property>-->

    <!--Configurations for MapReduce JobHistory Server:-->


        <!--  用于執(zhí)行任務時尋找hadoop環(huán)境變量  -->
    <property>
        <name>yarn.app.mapreduce.am.env</name>
        <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
    </property>
    <property>
        <name>mapreduce.map.env</name>
        <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
    </property>
    <property>
        <name>mapreduce.reduce.env</name>
        <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
    </property>
</configuration>

參考：
https://hadoop.apache.org/docs/r3.0.3/hadoop-project-dist/hadoop-common/ClusterSetup.html#Monitoring_Health_of_NodeManagers
https://www.jianshu.com/p/01b51bd1c77a

https://blog.csdn.net/oaimm/article/details/25298691文章來源地址http://www.zghlxwxcb.cn/news/detail-692602.html

到了這里，關于【運維】hadoop 集群安裝（三）hdfs、yarn集群配置、nodemanager健康管理講解的文章就介紹完了。如果您還想了解更多內(nèi)容，請在右上角搜索TOY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關文章，希望大家以后多多支持TOY模板網(wǎng)！