Hadoop系列文章目錄
1、hadoop3.1.4簡單介紹及部署、簡單驗證
2、HDFS操作 - shell客戶端
3、HDFS的使用（讀寫、上傳、下載、遍歷、查找文件、整個目錄拷貝、只拷貝文件、列出文件夾下文件、刪除文件及目錄、獲取文件及文件夾屬性等）-java
4、HDFS-java操作類HDFSUtil及junit測試（HDFS的常見操作以及HA環(huán)境的配置）
5、HDFS API的RESTful風格–WebHDFS
6、HDFS的HttpFS-代理服務
7、大數(shù)據(jù)中常見的文件存儲格式以及hadoop中支持的壓縮算法
8、HDFS內(nèi)存存儲策略支持和“冷熱溫”存儲
9、hadoop高可用HA集群部署及三種方式驗證
10、HDFS小文件解決方案–Archive
11、hadoop環(huán)境下的Sequence File的讀寫與合并
12、HDFS Trash垃圾桶回收介紹與示例
13、HDFS Snapshot快照
14、HDFS 透明加密KMS
15、MapReduce介紹及wordcount
16、MapReduce的基本用法示例-自定義序列化、排序、分區(qū)、分組和topN
17、MapReduce的分區(qū)Partition介紹
18、MapReduce的計數(shù)器與通過MapReduce讀取/寫入數(shù)據(jù)庫示例
19、Join操作map side join 和 reduce side join
20、MapReduce 工作流介紹
21、MapReduce讀寫SequenceFile、MapFile、ORCFile和ParquetFile文件
22、MapReduce使用Gzip壓縮、Snappy壓縮和Lzo壓縮算法寫文件和讀取相應的文件
23、hadoop集群中yarn運行mapreduce的內(nèi)存、CPU分配調(diào)度計算與優(yōu)化
本文介紹hadoop的發(fā)展過程、3.1.4的特性、部署及簡單驗證。
本文前提依賴：免密登錄設置、jdk已經(jīng)安裝、zookeeper部署完成且正常運行。具體參見相關(guān)文章，具體在zookeeper專欄、環(huán)境配置。
本文分為三個部分介紹，即hadoop發(fā)展史、hadoop3.1.4部署及驗證。
一、hadoop發(fā)展史
1、Hadoop介紹
Hadoop是Apache旗下的一個用java語言實現(xiàn)開源軟件框架，是一個開發(fā)和運行處理大規(guī)模數(shù)據(jù)的軟件平臺。允許使用簡單的編程模型在大量計算機集群上對大型數(shù)據(jù)集進行分布式處理。
狹義上說，Hadoop指Apache這款開源框架，它的核心組件有：
- HDFS（分布式文件系統(tǒng)）：解決海量數(shù)據(jù)存儲
- YARN（作業(yè)調(diào)度和集群資源管理的框架）：解決資源任務調(diào)度
- MAPREDUCE（分布式運算編程框架）：解決海量數(shù)據(jù)計算
Hadoop通常是指一個更廣泛的概念——Hadoop生態(tài)圈。
當下的Hadoop已經(jīng)成長為一個龐大的體系，隨著生態(tài)系統(tǒng)的成長，新出現(xiàn)的項目越來越多，其中不乏一些非Apache主管的項目，這些項目對HADOOP是很好的補充或者更高層的抽象。比如：
2、Hadoop發(fā)展簡史
Hadoop是Apache Lucene創(chuàng)始人 Doug Cutting 創(chuàng)建的。最早起源于Nutch，它是Lucene的子項目。
Nutch的設計目標是構(gòu)建一個大型的全網(wǎng)搜索引擎，包括網(wǎng)頁抓取、索引、查詢等功能，但隨著抓取網(wǎng)頁數(shù)量的增加，遇到了嚴重的可擴展性問題：如何解決數(shù)十億網(wǎng)頁的存儲和索引問題。
2003年Google發(fā)表了一篇論文為該問題提供了可行的解決方案。論文中描述的是谷歌的產(chǎn)品架構(gòu)，該架構(gòu)稱為：谷歌分布式文件系統(tǒng)（GFS）,可以解決他們在網(wǎng)頁爬取和索引過程中產(chǎn)生的超大文件的存儲需求。
2004年 Google發(fā)表論文向全世界介紹了谷歌版的MapReduce系統(tǒng)。同時期，以谷歌的論文為基礎，Nutch的開發(fā)人員完成了相應的開源實現(xiàn)HDFS和MAPREDUCE，并從Nutch中剝離成為獨立項目HADOOP，到2008年1月，HADOOP成為Apache頂級項目，迎來了它的快速發(fā)展期。
2006年Google發(fā)表了論文是關(guān)于BigTable的，這促使了后來的Hbase的發(fā)展。因此，Hadoop及其生態(tài)圈的發(fā)展離不開Google的貢獻。
3、Hadoop特性優(yōu)點
- 擴容能力（Scalable）：Hadoop是在可用的計算機集群間分配數(shù)據(jù)并完成計算任務的，這些集群可用方便的擴展到數(shù)以千計的節(jié)點中。
- 成本低（Economical）：Hadoop通過普通廉價的機器組成服務器集群來分發(fā)以及處理數(shù)據(jù)，以至于成本很低。
- 高效率（Efficient）：通過并發(fā)數(shù)據(jù)，Hadoop可以在節(jié)點之間動態(tài)并行的移動數(shù)據(jù)，使得速度非常快。
- 可靠性（Rellable）：能自動維護數(shù)據(jù)的多份復制，并且在任務失敗后能自動地重新部署（redeploy）計算任務。所以Hadoop的按位存儲和處理數(shù)據(jù)的能力值得人們信賴。
Hadoop的歷史版本和發(fā)行版公司
4、Hadoop歷史版本
1.x版本系列：hadoop版本當中的第二代開源版本，主要修復0.x版本的一些bug等，該版本已被淘汰
2.x版本系列：架構(gòu)產(chǎn)生重大變化，引入了yarn平臺等許多新特性,是現(xiàn)在使用的主流版本。
3.x版本系列：對HDFS、MapReduce、YARN都有較大升級，還新增了Ozone key-value存儲。
5、Hadoop發(fā)行版公司
Hadoop發(fā)行版本分為開源社區(qū)版和商業(yè)版。社區(qū)版是指由Apache軟件基金會維護的版本，是官方維護的版本體系。
商業(yè)版Hadoop是指由第三方商業(yè)公司在社區(qū)版Hadoop基礎上進行了一些修改、整合以及各個服務組件兼容性測試而發(fā)行的版本，比較著名的有cloudera的CDH、mapR、hortonWorks等。
1）、免費開源版本Apache
http://hadoop.apache.org/
優(yōu)點：擁有全世界的開源貢獻者，代碼更新迭代版本比較快，
缺點：版本的升級，版本的維護，版本的兼容性，版本的補丁都可能考慮不太周到
Apache所有軟件的下載地址（包括各種歷史版本）：http://archive.apache.org/dist/
2）、免費開源版本HortonWorks：
hortonworks主要是雅虎主導Hadoop開發(fā)的副總裁，帶領(lǐng)二十幾個核心成員成立Hortonworks，核心產(chǎn)品軟件HDP（ambari），HDF免費開源，并且提供一整套的web管理界面，供我們可以通過web界面管理我們的集群狀態(tài)，web管理界面軟件HDF網(wǎng)址（http://ambari.apache.org/），2018年，大數(shù)據(jù)領(lǐng)域的兩大巨頭公司Cloudera和Hortonworks宣布平等合并，Cloudera以股票方式收購Hortonworks，Cloudera股東最終獲得合并公司60%的股份
3）、收費版本
軟件收費版本Cloudera:
https://www.cloudera.com/
cloudera主要是美國一家大數(shù)據(jù)公司在apache開源hadoop的版本上，通過自己公司內(nèi)部的各種補丁，實現(xiàn)版本之間的穩(wěn)定運行，大數(shù)據(jù)生態(tài)圈的各個版本的軟件都提供了對應的版本，解決了版本的升級困難，版本兼容性等各種問題。該版本中是根據(jù)節(jié)點進行收費的，一定數(shù)量節(jié)點也是免費的。
6、Hadoop 3.x的版本架構(gòu)和模型介紹
由于Hadoop 2.0是基于JDK 1.7開發(fā)的，而JDK 1.7在2015年4月已停止更新，使Hadoop社區(qū)基于JDK 1.8重新發(fā)布一個新的Hadoop版本，即hadoop 3.0。
Hadoop 3.0中引入了一些重要的功能和優(yōu)化，包括HDFS 可擦除編碼、多Namenode支持、MR Native Task優(yōu)化、YARN基于cgroup的內(nèi)存和磁盤IO隔離、YARN container resizing等。
Apache hadoop 項目組最新消息，hadoop3.x以后將會調(diào)整方案架構(gòu)，將Mapreduce 基于內(nèi)存+io+磁盤，共同處理數(shù)據(jù)。改變最大的是hdfs,hdfs 通過最近block塊計算，根據(jù)最近計算原則，本地block塊，加入到內(nèi)存，先計算，通過IO，共享內(nèi)存計算區(qū)域，最后快速形成計算結(jié)果，比Spark快10倍。
1）、Hadoop 3.0新特性
Hadoop 3.0在功能和性能方面，對hadoop內(nèi)核進行了多項重大改進，主要包括：
- 1、通用性
1、精簡Hadoop內(nèi)核，包括剔除過期的API和實現(xiàn)，將默認組件實現(xiàn)替換成最高效的實現(xiàn)。
2、Classpath isolation：以防止不同版本jar包沖突
3、Shell腳本重構(gòu)： Hadoop 3.0對Hadoop的管理腳本進行了重構(gòu)，修復了大量bug，增加了新特性。
- 2、HDFS
Hadoop3.x中Hdfs在可靠性和支持能力上作出很大改觀：
1、HDFS支持數(shù)據(jù)的擦除編碼，這使得HDFS在不降低可靠性的前提下，節(jié)省一半存儲空間。
2、多NameNode支持，即支持一個集群中，一個active、多個standby namenode部署方式。注：多ResourceManager特性在hadoop 2.0中已經(jīng)支持。
- 3、HDFS糾刪碼
在Hadoop3.X中，HDFS實現(xiàn)了Erasure Coding這個新功能。Erasure coding糾刪碼技術(shù)簡稱EC，是一種數(shù)據(jù)保護技術(shù).最早用于通信行業(yè)中數(shù)據(jù)傳輸中的數(shù)據(jù)恢復，是一種編碼容錯技術(shù)。
它通過在原始數(shù)據(jù)中加入新的校驗數(shù)據(jù)，使得各個部分的數(shù)據(jù)產(chǎn)生關(guān)聯(lián)性。在一定范圍的數(shù)據(jù)出錯情況下，通過糾刪碼技術(shù)都可以進行恢復。
hadoop-3.0之前，HDFS存儲方式為每一份數(shù)據(jù)存儲3份，這也使得存儲利用率僅為1/3，hadoop-3.0引入糾刪碼技術(shù)(EC技術(shù))，實現(xiàn)1份數(shù)據(jù)+0.5份冗余校驗數(shù)據(jù)存儲方式。與副本相比糾刪碼是一種更節(jié)省空間的數(shù)據(jù)持久化存儲方法。標準編碼(比如Reed-Solomon(10,4))會有1.4 倍的空間開銷；然而HDFS副本則會有3倍的空間開銷。
- 4、支持多個NameNodes
最初的HDFS NameNode high-availability實現(xiàn)僅僅提供了一個active NameNode和一個Standby NameNode；并且通過將編輯日志復制到三個JournalNodes上，這種架構(gòu)能夠容忍系統(tǒng)中的任何一個節(jié)點的失敗。然而，一些部署需要更高的容錯度。我們可以通過這個新特性來實現(xiàn)，其允許用戶運行多個Standby NameNode。比如通過配置三個NameNode和五個JournalNodes，這個系統(tǒng)可以容忍2個節(jié)點的故障，而不是僅僅一個節(jié)點。
- 5、MapReduce
Hadoop3.X中的MapReduce較之前的版本作出以下更改：
1、Tasknative優(yōu)化：為MapReduce增加了C/C++的map output collector實現(xiàn)（包括Spill，Sort和IFile等），通過作業(yè)級別參數(shù)調(diào)整就可切換到該實現(xiàn)上。對于shuffle密集型應用，其性能可提高約30%。
2、MapReduce內(nèi)存參數(shù)自動推斷。在Hadoop 2.0中，為MapReduce作業(yè)設置內(nèi)存參數(shù)非常繁瑣，一旦設置不合理，則會使得內(nèi)存資源浪費嚴重，在Hadoop3.0中避免了這種情況。
Hadoop3.x中的MapReduce添加了Map輸出collector的本地實現(xiàn)，對于shuffle密集型的作業(yè)來說，這將會有30%以上的性能提升。
- 6、默認端口更改
在hadoop3.x之前，多個Hadoop服務的默認端口都屬于Linux的臨時端口范圍（32768-61000）。這就意味著用戶的服務在啟動的時候可能因為和其他應用程序產(chǎn)生端口沖突而無法啟動。
現(xiàn)在這些可能會產(chǎn)生沖突的端口已經(jīng)不再屬于臨時端口的范圍，這些端口的改變會影響NameNode, Secondary NameNode, DataNode以及KMS。與此同時，官方文檔也進行了相應的改變，具體可以參見 HDFS-9427以及HADOOP-12811。
Namenode ports: 50470 --> 9871, 50070 --> 9870, 8020 --> 9820
Secondary NN ports: 50091 --> 9869,50090 --> 9868
Datanode ports: 50020 --> 9867, 50010 --> 9866, 50475 --> 9865, 50075 --> 9864
Kms server ports: 16000 --> 9600 (原先的16000與HMaster端口沖突)
- 7、YARN 資源類型
YARN 資源模型（YARN resource model）已被推廣為支持用戶自定義的可數(shù)資源類型（support user-defined countable resource types），不僅僅支持 CPU 和內(nèi)存。
比如集群管理員可以定義諸如 GPUs、軟件許可證（software licenses）或本地附加存儲器（locally-attached storage）之類的資源。YARN 任務可以根據(jù)這些資源的可用性進行調(diào)度。
二、hadoop3.1.4集群部署
本文部署使用的是Apache 的3.1.4版本。
1、集群簡介
HADOOP集群具體來說包含兩個集群：HDFS集群和YARN集群，兩者邏輯上分離，但物理上常在一起。
- HDFS集群負責海量數(shù)據(jù)的存儲，集群中的角色主要有：NameNode、DataNode、SecondaryNameNode
- YARN集群負責海量數(shù)據(jù)運算時的資源調(diào)度，集群中的角色主要有： ResourceManager、NodeManager
- mapreduce是一個分布式運算編程框架，是應用程序開發(fā)包，由用戶按照編程規(guī)范進行程序開發(fā)，后打包運行在HDFS集群上，并且受到Y(jié)ARN集群的資源調(diào)度管理。
2、集群部署方式
Hadoop部署方式分三種：
- standalone mode（獨立模式）
獨立模式又稱為單機模式，僅1個機器運行1個java進程，主要用于調(diào)試。
- Pseudo-Distributed mode（偽分布式模式）
偽分布模式也是在1個機器上運行HDFS的NameNode和DataNode、YARN的 ResourceManger和NodeManager，但分別啟動單獨的java進程，主要用于調(diào)試。
- Cluster mode（群集模式）
集群模式主要用于生產(chǎn)環(huán)境部署。會使用N臺主機組成一個Hadoop集群。這種部署模式下，主節(jié)點和從節(jié)點會分開部署在不同的機器上。
3、hadoop重新編譯
由于appache給出的hadoop的安裝包沒有提供帶C程序訪問的接口，所以我們在使用本地庫（本地庫可以用來做壓縮，以及支持C程序等等）的時候就會出問題,需要對Hadoop源碼包進行重新編譯。
編譯步驟參考：hadoop 3.1.4在centos7上snappy編譯過程
4、Hadoop集群安裝
集群模式主要用于生產(chǎn)環(huán)境部署，需要多臺主機，并且這些主機之間可以相互訪問，我們在之前搭建好基礎環(huán)境的三臺虛擬機上進行Hadoop的搭建。
以下部署使用的是alanchan用戶，其中為了后續(xù)的配置，需要做免密登錄，具體參考鏈接：linux centos7免密登錄設置
1）、集群規(guī)劃
搭建的是集群模式，以四臺主機為例（四臺機子上均創(chuàng)建了alanchan用戶，且在root組，也創(chuàng)建了usr/local/bigdata文件夾），以下是集群規(guī)劃:
機器設置參考zookeeper的配置，同時該三臺機器上部署了zookeeper
2）、主機名（4臺機器）
該步驟是root用戶操作，如果之前在免密登錄時已經(jīng)設置，則跳過。
vi /etc/hostname
server1
#root用戶做，保存退出，重啟機器reboot
#依次做4臺機器（server2、server3、server4）
#驗證
[root@server1 ~]# hostname
server1
3）、Hosts映射（4臺機器）
該步驟是root用戶操作，如果之前在免密登錄時已經(jīng)設置，則跳過。
vi /etc/hosts
#依次修改四臺機器
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 server41
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.10.41 server1
192.168.10.42 server2
192.168.10.43 server3
192.168.10.44 server4
4）、集群時間同步（4臺機器）
該步驟是root用戶操作，如果之前在免密登錄時已經(jīng)設置，則跳過。
yum -y install ntpdate
ntpdate ntp4.aliyun.com
#驗證命令
date
5）、解壓（在server1上做）
該步驟是alanchan用戶操作。
前提是hadoop-3.1.4-bin-snappy-CentOS7.tar.gz安裝文件已經(jīng)上傳至server1上。
# 創(chuàng)建目錄/usr/local/bigdata
mkdir -p /usr/local/bigdata
# 上傳解壓hadoop-3.1.4-bin-snappy-CentOS7.tar.gz安裝文件
su alanchan
[alanchan@server bigdata]$ cd /usr/local/bigdata/
[alanchan@server bigdata]$ ll
-rw-r--r-- 1 alanchan root 303134111 8月 23 16:49 hadoop-3.1.4-bin-snappy-CentOS7.tar.gz
# 解壓
tar -xvzf hadoop-3.1.4-bin-snappy-CentOS7.tar.gz
[alanchan@server bigdata]$ ll
drwxr-xr-x 9 alanchan root 4096 11月 4 2020 hadoop-3.1.4
-rw-r--r-- 1 alanchan root 303134111 8月 23 16:49 hadoop-3.1.4-bin-snappy-CentOS7.tar.gz
# 在每個節(jié)點中創(chuàng)建用于存放數(shù)據(jù)的data目錄
# NameNode數(shù)據(jù)
mkdir -p /usr/local/bigdata/hadoop-3.1.4/data/namenode
# DataNode數(shù)據(jù)
mkdir -p /usr/local/bigdata/hadoop-3.1.4/data/datanode
解壓后的目錄結(jié)構(gòu)如下：
6）、關(guān)鍵文件說明
- hadoop-env.sh
文件中設置的是Hadoop運行時需要的環(huán)境變量。
JAVA_HOME是必須設置的，即使我們當前的系統(tǒng)中設置了JAVA_HOME，因為Hadoop即使是在本機上執(zhí)行，它也是把當前的執(zhí)行環(huán)境當成遠程服務器。
- core-site.xml
hadoop的核心配置文件，有默認的配置項core-default.xml。
core-default.xml與core-site.xml的功能是一樣的，如果在core-site.xml里沒有配置的屬性，則會自動會獲取core-default.xml里的相同屬性的值。
在該文件中的標簽中添加以下配置
<configuration>
在這里添加配置
</configuration>
- hdfs-site.xml
HDFS的核心配置文件，主要配置HDFS相關(guān)參數(shù)，有默認的配置項hdfs-default.xml。
hdfs-default.xml與hdfs-site.xml的功能是一樣的，如果在hdfs-site.xml里沒有配置的屬性，則會自動會獲取hdfs-default.xml里的相同屬性的值。
- mapred-site.xml
MapReduce的核心配置文件，Hadoop默認文件mapred-default.xml,需要使用該文件復制出來一份mapred-site.xml文件
- yarn-site.xml
YARN的核心配置文件,YARN的相關(guān)參數(shù)在該文件中配置，默認的是yarn-default.xml。
- workers
workers文件里面記錄的是集群主機名。一般有以下兩種作用：
- 配合一鍵啟動腳本如start-dfs.sh、stop-yarn.sh用來進行集群啟動。這時候slaves文件里面的主機標記的就是從節(jié)點角色所在的機器。
- 可以配合hdfs-site.xml里面dfs.hosts屬性形成一種白名單機制。
dfs.hosts指定一個文件，其中包含允許連接到NameNode的主機列表。必須指定文件的完整路徑名,那么所有在workers中的主機才可以加入的集群中。如果值為空，則允許所有主機。
5、修改配置文件
解壓后的配置文件都是空的，形如下面，如果沒有配置，系統(tǒng)會使用其自帶的配置文件，例如core-site.xml會使用core-default.xml。
該步驟是alanchan用戶操作。
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<!-- Do not modify this file directly. Instead, copy entries that you -->
<!-- wish to modify from this file into core-site.xml and change them -->
<!-- there. If core-site.xml does not already exist, create it. -->
<configuration>
</configuration>
Apache Hadoop 3.3.4 – Hadoop Cluster Setup
Configuring Hadoop in Non-Secure Mode
Hadoop’s Java configuration is driven by two types of important configuration files:
Read-only default configuration - core-default.xml, hdfs-default.xml, yarn-default.xml and mapred-default.xml.
Site-specific configuration - etc/hadoop/core-site.xml, etc/hadoop/hdfs-site.xml, etc/hadoop/yarn-site.xml and etc/hadoop/mapred-site.xml.
Additionally, you can control the Hadoop scripts found in the bin/ directory of the distribution, by setting site-specific values via the etc/hadoop/hadoop-env.sh and etc/hadoop/yarn-env.sh.
To configure the Hadoop cluster you will need to configure the environment in which the Hadoop daemons execute as well as the configuration parameters for the Hadoop daemons.
HDFS daemons are NameNode, SecondaryNameNode, and DataNode. YARN daemons are ResourceManager, NodeManager, and WebAppProxy. If MapReduce is to be used, then the MapReduce Job History Server will also be running. For large installations, these are generally running on separate hosts.
1）、配置NameNode（core-site.xml）
源碼位置：hadoop-3.1.4-src\hadoop-common-project\hadoop-common\src\main\resources\core-default.xml
[alanchan@server bigdata]$ cd /usr/local/bigdata/hadoop-3.1.4/etc/hadoop
[alanchan@server hadoop]$ ll
-rw-r--r-- 1 alanchan root 9213 11月 4 2020 capacity-scheduler.xml
-rw-r--r-- 1 alanchan root 1335 11月 4 2020 configuration.xsl
-rw-r--r-- 1 alanchan root 1940 11月 4 2020 container-executor.cfg
-rw-r--r-- 1 alanchan root 774 11月 4 2020 core-site.xml
-rw-r--r-- 1 alanchan root 3999 11月 4 2020 hadoop-env.cmd
-rw-r--r-- 1 alanchan root 15903 11月 4 2020 hadoop-env.sh
-rw-r--r-- 1 alanchan root 3323 11月 4 2020 hadoop-metrics2.properties
-rw-r--r-- 1 alanchan root 11392 11月 4 2020 hadoop-policy.xml
-rw-r--r-- 1 alanchan root 3414 11月 4 2020 hadoop-user-functions.sh.example
-rw-r--r-- 1 alanchan root 775 11月 4 2020 hdfs-site.xml
-rw-r--r-- 1 alanchan root 1484 11月 4 2020 httpfs-env.sh
-rw-r--r-- 1 alanchan root 1657 11月 4 2020 httpfs-log4j.properties
-rw-r--r-- 1 alanchan root 21 11月 4 2020 httpfs-signature.secret
-rw-r--r-- 1 alanchan root 620 11月 4 2020 httpfs-site.xml
-rw-r--r-- 1 alanchan root 3518 11月 4 2020 kms-acls.xml
-rw-r--r-- 1 alanchan root 1351 11月 4 2020 kms-env.sh
-rw-r--r-- 1 alanchan root 1747 11月 4 2020 kms-log4j.properties
-rw-r--r-- 1 alanchan root 682 11月 4 2020 kms-site.xml
-rw-r--r-- 1 alanchan root 14713 11月 4 2020 log4j.properties
-rw-r--r-- 1 alanchan root 951 11月 4 2020 mapred-env.cmd
-rw-r--r-- 1 alanchan root 1764 11月 4 2020 mapred-env.sh
-rw-r--r-- 1 alanchan root 4113 11月 4 2020 mapred-queues.xml.template
-rw-r--r-- 1 alanchan root 758 11月 4 2020 mapred-site.xml
drwxr-xr-x 2 alanchan root 4096 11月 4 2020 shellprofile.d
-rw-r--r-- 1 alanchan root 2316 11月 4 2020 ssl-client.xml.example
-rw-r--r-- 1 alanchan root 2697 11月 4 2020 ssl-server.xml.example
-rw-r--r-- 1 alanchan root 2642 11月 4 2020 user_ec_policies.xml.template
-rw-r--r-- 1 alanchan root 10 11月 4 2020 workers
-rw-r--r-- 1 alanchan root 2250 11月 4 2020 yarn-env.cmd
-rw-r--r-- 1 alanchan root 6272 11月 4 2020 yarn-env.sh
-rw-r--r-- 1 alanchan root 2591 11月 4 2020 yarnservice-log4j.properties
-rw-r--r-- 1 alanchan root 690 11月 4 2020 yarn-site.xml
#修改core-site.xml文件 /usr/local/bigdata/hadoop-3.1.4/etc/hadoop/core-site.xml
# 以下內(nèi)容均為增加部分
<property>
<name>fs.defaultFS</name>
<value>hdfs://server1:8020</value>
<description>配置NameNode的URL</description>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://server1:8020</value>
<description>Deprecated. Use (fs.defaultFS) property instead</description>
</property>
<!-- hadoop本地數(shù)據(jù)存儲目錄 format時自動生成數(shù)據(jù)存儲目錄最好是放在本工程的外面，避免擴容時需要剔除該部分內(nèi)容，本例沒有放在外面-->
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/bigdata/hadoop-3.1.4</value>
</property>
<!-- 在Web UI訪問HDFS使用的用戶名。-->
<property>
<name>hadoop.http.staticuser.user</name>
<value>alanchan</value>
</property>
# 以上內(nèi)容均為增加部分
2）、配置HDFS文件（hdfs-site.xml）
源碼位置：hadoop-3.1.4-src\hadoop-hdfs-project\hadoop-hdfs\src\main\resources\hdfs-default.xml
# 文件路徑：/usr/local/bigdata/hadoop-3.1.4/etc/hadoop/hdfs-site.xml
<!-- 設定SNN運行主機和端口。-->
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>server2:9868</value>
<description>
The secondary namenode http server address and port.
</description>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/usr/local/bigdata/hadoop-3.1.4/data/namenode</value>
<description>NameNode存儲名稱空間和事務日志的本地文件系統(tǒng)上的路徑</description>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/usr/local/bigdata/hadoop-3.1.4/data/datanode</value>
<description>DDataNode存儲名稱空間和事務日志的本地文件系統(tǒng)上的路徑</description>
</property>
一般為了將來能方便的增加黑白名單配置
#在namenode機器的hdfs-site.xml配置文件中需要提前配置dfs.hosts.exclude屬性，該屬性指向的文件就是所謂的黑名單列表，會被namenode排除在集群之外。如果文件內(nèi)容為空，則意味著不禁止任何機器。
#提前配置好的目的是讓namenode啟動的時候就能加載到該屬性，只不過還沒有指定任何機器。否則就需要重啟namenode才能加載，因此這樣的操作我們稱之為具有前瞻性的操作。
<property>
<name>dfs.hosts.exclude</name>
<value>/usr/local/bigdata/hadoop-3.1.4/etc/hadoop/excludes</value>
</property>
#編輯dfs.hosts.exclude屬性指向的excludes文件，添加需要退役的主機名稱。
#注意：如果副本數(shù)是3，服役的節(jié)點小于等于3，是不能退役成功的，需要修改副本數(shù)后才能退役。
<!-- 白名單 -->
<property>
<name>dfs.hosts</name>
<value></value>
<description>Names a file that contains a list of hosts that are permitted to connect to the namenode. The full pathname of the file must be specified. If the value is empty, all hosts are permitted.</description>
</property>
<!-- 黑名單 -->
<property>
<name>dfs.hosts.exclude</name>
<value></value>
<description>Names a file that contains a list of hosts that are not permitted to connect to the namenode. The full pathname of the file must be specified. If the value is empty, no hosts are excluded.</description>
</property>
3）、配置YARN（yarn-site.xml）
源碼位置：hadoop-3.1.4-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-common\src\main\resources\yarn-default.xml
# 文件位置 /usr/local/bigdata/hadoop-3.1.4/etc/hadoop/yarn-site.xml
<property>
<description>為每個容器請求分配的最小內(nèi)存限制資源管理器</description>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>1024</value>
</property>
<property>
<description>為每個容器請求分配的最大內(nèi)存限制資源管理器</description>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>8192</value>
</property>
<property>
<description>虛擬內(nèi)存比例，默認為2.1</description>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>2.1</value>
</property>
<!-- NodeManager上運行的附屬服務。需配置成mapreduce_shuffle,才可運行MR程序 -->
<property>
<description>A comma separated list of services where service name should only
contain a-zA-Z0-9_ and can not start with numbers</description>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
<!--<value>mapreduce_shuffle</value>-->
</property>
<!-- yarn集群主角色RM運行機器 -->
<property>
<description>The hostname of the RM.</description>
<name>yarn.resourcemanager.hostname</name>
<value>server1</value>
</property>
4）、配置MapReduce（mapred-site.xml）
源碼位置：hadoop-3.1.4-src\hadoop-mapreduce-project\hadoop-mapreduce-client\hadoop-mapreduce-client-core\src\main\resources\mapred-default.xml
# 文件路徑： /usr/local/bigdata/hadoop-3.1.4/etc/hadoop/mapred-site.xml
<!-- mr程序默認運行方式。yarn集群模式 local本地模式-->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
<description>The runtime framework for executing MapReduce jobs.
Can be one of local, classic or yarn.執(zhí)行MapReduce的方式：yarn/local
</description>
</property>
<!-- MR App Master環(huán)境變量。-->
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>
<!-- MR MapTask環(huán)境變量。-->
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>
<!-- MR ReduceTask環(huán)境變量。-->
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>
5）、workers文件
# 設置三個datanode
# 文件位置：/usr/local/bigdata/hadoop-3.1.4/etc/hadoop/workers
# 文件內(nèi)容如下：
server2
server3
server4
6）、修改hadoop.env環(huán)境變量
# 設置jdk的安裝目錄，改成實際環(huán)境的目錄
export JAVA_HOME=/usr/java/jdk1.8.0_144
#設置用戶以執(zhí)行對應角色shell命令，改成實際的用戶名
export HDFS_NAMENODE_USER=alanchan
export HDFS_DATANODE_USER=alanchan
export HDFS_SECONDARYNAMENODE_USER=alanchan
export YARN_RESOURCEMANAGER_USER=alanchan
export YARN_NODEMANAGER_USER=alanchan
7）、配置環(huán)境變量
該步驟操作用戶是root。
vi /etc/profile
# 設置內(nèi)容如下：
export HADOOP_HOME=/usr/local/bigdata/hadoop-3.1.4
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
source /etc/profile
8）、分發(fā)配置好的Hadoop安裝文件和環(huán)境變量
cd /usr/local/bigdata
# 復制hadoop，在server1上執(zhí)行
scp -r hadoop-3.1.4 server2:$PWD
scp -r hadoop-3.1.4 server3:$PWD
scp -r hadoop-3.1.4 server4:$PWD
# 復制環(huán)境變量，在server1上執(zhí)行
scp /etc/profile server2:/etc
scp /etc/profile server3:/etc
scp /etc/profile server4:/etc
# 在每個節(jié)點上執(zhí)行，使環(huán)境變量生效
source /etc/profile
#驗證環(huán)境變量是否生效，如果正常識別出來hadoop命令，即生效了。直接在centos中輸入命令
hadoop
# 出現(xiàn)如下信息則表示配置成功
[alanchan@server3 bin]$ hadoop
Usage: hadoop [OPTIONS] SUBCOMMAND [SUBCOMMAND OPTIONS]
or hadoop [OPTIONS] CLASSNAME [CLASSNAME OPTIONS]
where CLASSNAME is a user-provided Java class
OPTIONS is none or any of:
buildpaths attempt to add class files from build tree
--config dir Hadoop config directory
--debug turn on shell script debug mode
--help usage information
hostnames list[,of,host,names] hosts to use in slave mode
hosts filename list of hosts to use in slave mode
loglevel level set the log4j level for this command
workers turn on worker mode
SUBCOMMAND is one of:
Admin Commands:
daemonlog get/set the log level for each daemon
Client Commands:
archive create a Hadoop archive
checknative check native Hadoop and compression libraries availability
classpath prints the class path needed to get the Hadoop jar and the required libraries
conftest validate configuration XML files
credential interact with credential providers
......
9）、格式化HDFS
其實就是初始化hadoop的環(huán)境。使用alanchan用戶，在server1上執(zhí)行。
首次啟動HDFS時，必須對其進行格式化操作(hdfs namenode -format)。
cd /usr/local/bigdata/hadoop-3.1.4/bin
[alanchan@server1 bin]$ ll
總用量 880
-rwxr-xr-x 1 alanchan root 392672 11月 4 2020 container-executor
-rwxr-xr-x 1 alanchan root 8707 11月 4 2020 hadoop
-rwxr-xr-x 1 alanchan root 11265 11月 4 2020 hadoop.cmd
-rwxr-xr-x 1 alanchan root 11026 11月 4 2020 hdfs
-rwxr-xr-x 1 alanchan root 8081 11月 4 2020 hdfs.cmd
-rwxr-xr-x 1 alanchan root 6237 11月 4 2020 mapred
-rwxr-xr-x 1 alanchan root 6311 11月 4 2020 mapred.cmd
-rwxr-xr-x 1 alanchan root 416384 11月 4 2020 test-container-executor
-rwxr-xr-x 1 alanchan root 11888 11月 4 2020 yarn
-rwxr-xr-x 1 alanchan root 12840 11月 4 2020 yarn.cmd
[alanchan@server1 bin]$ hdfs namenode -format
2022-08-25 09:59:36,957 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = server1/192.168.10.41
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 3.1.4
STARTUP_MSG: classpath = /usr/local/bigdata/hadoop-3.1.4/etc/hadoop:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/jsp-api-2.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/guava-27.0-jre.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/commons-configuration2-2.1.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/curator-client-2.13.0.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/slf4j-api-1.7.25.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/asm-5.0.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/kerb-util-1.0.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/jetty-security-9.4.20.v20190813.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/failureaccess-1.0.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/jcip-annotations-1.0-1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/animal-sniffer-annotations-1.17.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/accessors-smart-1.2.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/kerby-asn1-1.0.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/jetty-http-9.4.20.v20190813.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/snappy-java-1.0.5.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/checker-qual-2.5.2.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/woodstox-core-5.0.3.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/httpcore-4.4.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/jaxb-api-2.2.11.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/hadoop-auth-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/kerb-core-1.0.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/jackson-annotations-2.9.10.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/jersey-core-1.19.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/kerby-pkix-1.0.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/kerb-admin-1.0.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/jackson-core-2.9.10.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/log4j-1.2.17.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/jettison-1.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/token-provider-1.0.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/commons-lang-2.6.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/jersey-server-1.19.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/jul-to-slf4j-1.7.25.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/kerb-server-1.0.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/kerb-common-1.0.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/kerby-util-1.0.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/jsch-0.1.55.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/jsr311-api-1.1.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/nimbus-jose-jwt-7.9.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/kerb-identity-1.0.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/jersey-json-1.19.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/kerby-config-1.0.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/htrace-core4-4.1.0-incubating.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/jersey-servlet-1.19.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/jetty-server-9.4.20.v20190813.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/jsr305-3.0.2.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/curator-recipes-2.13.0.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/curator-framework-2.13.0.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/jetty-util-9.4.20.v20190813.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/re2j-1.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/kerb-crypto-1.0.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/j2objc-annotations-1.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/commons-lang3-3.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/json-smart-2.3.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/audience-annotations-0.5.0.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/kerby-xdr-1.0.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/commons-net-3.6.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/commons-codec-1.11.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/gson-2.2.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/httpclient-4.5.2.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/javax.servlet-api-3.1.0.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/stax2-api-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/jetty-webapp-9.4.20.v20190813.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/commons-beanutils-1.9.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/commons-math3-3.1.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/zookeeper-3.4.13.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/jetty-io-9.4.20.v20190813.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/kerb-simplekdc-1.0.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/jackson-databind-2.9.10.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/paranamer-2.3.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/avro-1.7.7.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/hadoop-annotations-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/kerb-client-1.0.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/jetty-servlet-9.4.20.v20190813.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/commons-logging-1.1.3.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/jetty-xml-9.4.20.v20190813.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/commons-compress-1.19.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/metrics-core-3.2.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/error_prone_annotations-2.2.0.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/commons-io-2.5.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/netty-3.10.6.Final.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/lib/commons-collections-3.2.2.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/hadoop-common-3.1.4-tests.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/hadoop-common-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/hadoop-kms-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/common/hadoop-nfs-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/guava-27.0-jre.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/commons-configuration2-2.1.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/curator-client-2.13.0.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/asm-5.0.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/kerb-util-1.0.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/jetty-security-9.4.20.v20190813.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/failureaccess-1.0.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/jaxb-impl-2.2.3-1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/jcip-annotations-1.0-1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/animal-sniffer-annotations-1.17.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/accessors-smart-1.2.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/kerby-asn1-1.0.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/json-simple-1.1.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/jetty-http-9.4.20.v20190813.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/jackson-xc-1.9.13.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/snappy-java-1.0.5.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/checker-qual-2.5.2.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/woodstox-core-5.0.3.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/netty-all-4.1.48.Final.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/httpcore-4.4.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/jaxb-api-2.2.11.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/hadoop-auth-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/jetty-util-ajax-9.4.20.v20190813.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/kerb-core-1.0.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/jackson-annotations-2.9.10.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/jersey-core-1.19.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/kerby-pkix-1.0.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/kerb-admin-1.0.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/jackson-core-2.9.10.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/jettison-1.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/token-provider-1.0.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/jersey-server-1.19.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/kerb-server-1.0.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/kerb-common-1.0.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/kerby-util-1.0.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/jsch-0.1.55.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/jsr311-api-1.1.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/nimbus-jose-jwt-7.9.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/kerb-identity-1.0.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/jersey-json-1.19.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/kerby-config-1.0.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/htrace-core4-4.1.0-incubating.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/jersey-servlet-1.19.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/jetty-server-9.4.20.v20190813.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/jsr305-3.0.2.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/curator-recipes-2.13.0.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/curator-framework-2.13.0.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/jetty-util-9.4.20.v20190813.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/re2j-1.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/kerb-crypto-1.0.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/j2objc-annotations-1.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/commons-lang3-3.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/json-smart-2.3.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/audience-annotations-0.5.0.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/kerby-xdr-1.0.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/commons-net-3.6.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/commons-codec-1.11.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/gson-2.2.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/httpclient-4.5.2.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/javax.servlet-api-3.1.0.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/okhttp-2.7.5.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/stax2-api-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/jetty-webapp-9.4.20.v20190813.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/commons-beanutils-1.9.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/commons-math3-3.1.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/zookeeper-3.4.13.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/okio-1.6.0.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/jetty-io-9.4.20.v20190813.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/kerb-simplekdc-1.0.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/jackson-databind-2.9.10.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/paranamer-2.3.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/avro-1.7.7.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/hadoop-annotations-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/kerb-client-1.0.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/jetty-servlet-9.4.20.v20190813.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/jetty-xml-9.4.20.v20190813.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/commons-compress-1.19.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/error_prone_annotations-2.2.0.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/commons-io-2.5.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/netty-3.10.6.Final.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/jackson-jaxrs-1.9.13.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/lib/commons-collections-3.2.2.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/hadoop-hdfs-client-3.1.4-tests.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/hadoop-hdfs-native-client-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/hadoop-hdfs-httpfs-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/hadoop-hdfs-3.1.4-tests.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/hadoop-hdfs-nfs-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/hadoop-hdfs-rbf-3.1.4-tests.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/hadoop-hdfs-rbf-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/hadoop-hdfs-client-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/hadoop-hdfs-native-client-3.1.4-tests.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/hdfs/hadoop-hdfs-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/mapreduce/lib/junit-4.11.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/mapreduce/hadoop-mapreduce-client-core-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/mapreduce/hadoop-mapreduce-client-app-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/mapreduce/hadoop-mapreduce-client-uploader-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/mapreduce/hadoop-mapreduce-client-common-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.4-tests.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/mapreduce/hadoop-mapreduce-client-nativetask-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/lib/jersey-guice-1.19.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/lib/swagger-annotations-1.5.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/lib/dnsjava-2.1.7.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/lib/mssql-jdbc-6.2.1.jre7.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/lib/fst-2.50.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/lib/jackson-jaxrs-base-2.9.10.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/lib/aopalliance-1.0.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/lib/ehcache-3.3.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/lib/jersey-client-1.19.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/lib/jackson-module-jaxb-annotations-2.9.10.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/lib/objenesis-1.0.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/lib/javax.inject-1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/lib/json-io-2.5.1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/lib/java-util-1.9.0.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/lib/HikariCP-java7-2.4.12.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/lib/jackson-jaxrs-json-provider-2.9.10.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/lib/guice-servlet-4.0.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/lib/metrics-core-3.2.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/lib/guice-4.0.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/lib/snakeyaml-1.16.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/hadoop-yarn-server-nodemanager-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/hadoop-yarn-server-tests-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/hadoop-yarn-server-web-proxy-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/hadoop-yarn-server-common-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/hadoop-yarn-server-router-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/hadoop-yarn-services-api-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/hadoop-yarn-registry-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/hadoop-yarn-client-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/hadoop-yarn-services-core-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/hadoop-yarn-common-3.1.4.jar:/usr/local/bigdata/hadoop-3.1.4/share/hadoop/yarn/hadoop-yarn-api-3.1.4.jar
STARTUP_MSG: build = Unknown -r Unknown; compiled by 'root' on 2020-11-04T09:35Z
STARTUP_MSG: java = 1.8.0_144
************************************************************/
2022-08-25 09:59:36,964 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
2022-08-25 09:59:37,059 INFO namenode.NameNode: createNameNode [-format]
2022-08-25 09:59:37,186 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Formatting using clusterid: CID-ffdc8a28-a92d-45fc-bd34-382c1645e64a
2022-08-25 09:59:37,519 INFO namenode.FSEditLog: Edit logging is async:true
2022-08-25 09:59:37,545 INFO namenode.FSNamesystem: KeyProvider: null
2022-08-25 09:59:37,546 INFO namenode.FSNamesystem: fsLock is fair: true
2022-08-25 09:59:37,546 INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false
2022-08-25 09:59:37,561 INFO namenode.FSNamesystem: fsOwner = alanchan (auth:SIMPLE)
2022-08-25 09:59:37,561 INFO namenode.FSNamesystem: supergroup = supergroup
2022-08-25 09:59:37,561 INFO namenode.FSNamesystem: isPermissionEnabled = true
2022-08-25 09:59:37,566 INFO namenode.FSNamesystem: HA Enabled: false
2022-08-25 09:59:37,611 INFO common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
2022-08-25 09:59:37,621 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit: configured=1000, counted=60, effected=1000
2022-08-25 09:59:37,621 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
2022-08-25 09:59:37,625 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
2022-08-25 09:59:37,626 INFO blockmanagement.BlockManager: The block deletion will start around 2022 八月 25 09:59:37
2022-08-25 09:59:37,627 INFO util.GSet: Computing capacity for map BlocksMap
2022-08-25 09:59:37,627 INFO util.GSet: VM type = 64-bit
2022-08-25 09:59:37,630 INFO util.GSet: 2.0% max memory 3.5 GB = 70.9 MB
2022-08-25 09:59:37,630 INFO util.GSet: capacity = 2^23 = 8388608 entries
2022-08-25 09:59:37,641 INFO blockmanagement.BlockManager: dfs.block.access.token.enable = false
2022-08-25 09:59:37,647 INFO Configuration.deprecation: No unit for dfs.namenode.safemode.extension(30000) assuming MILLISECONDS
2022-08-25 09:59:37,647 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2022-08-25 09:59:37,647 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.min.datanodes = 0
2022-08-25 09:59:37,647 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.extension = 30000
2022-08-25 09:59:37,684 INFO blockmanagement.BlockManager: defaultReplication = 3
2022-08-25 09:59:37,684 INFO blockmanagement.BlockManager: maxReplication = 512
2022-08-25 09:59:37,684 INFO blockmanagement.BlockManager: minReplication = 1
2022-08-25 09:59:37,684 INFO blockmanagement.BlockManager: maxReplicationStreams = 2
2022-08-25 09:59:37,684 INFO blockmanagement.BlockManager: redundancyRecheckInterval = 3000ms
2022-08-25 09:59:37,684 INFO blockmanagement.BlockManager: encryptDataTransfer = false
2022-08-25 09:59:37,684 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000
2022-08-25 09:59:37,729 INFO namenode.FSDirectory: GLOBAL serial map: bits=24 maxEntries=16777215
2022-08-25 09:59:37,745 INFO util.GSet: Computing capacity for map INodeMap
2022-08-25 09:59:37,745 INFO util.GSet: VM type = 64-bit
2022-08-25 09:59:37,745 INFO util.GSet: 1.0% max memory 3.5 GB = 35.4 MB
2022-08-25 09:59:37,745 INFO util.GSet: capacity = 2^22 = 4194304 entries
2022-08-25 09:59:37,747 INFO namenode.FSDirectory: ACLs enabled? false
2022-08-25 09:59:37,747 INFO namenode.FSDirectory: POSIX ACL inheritance enabled? true
2022-08-25 09:59:37,747 INFO namenode.FSDirectory: XAttrs enabled? true
2022-08-25 09:59:37,747 INFO namenode.NameNode: Caching file names occurring more than 10 times
2022-08-25 09:59:37,752 INFO snapshot.SnapshotManager: Loaded config captureOpenFiles: false, skipCaptureAccessTimeOnlyChange: false, snapshotDiffAllowSnapRootDescendant: true, maxSnapshotLimit: 65536
2022-08-25 09:59:37,754 INFO snapshot.SnapshotManager: SkipList is disabled
2022-08-25 09:59:37,758 INFO util.GSet: Computing capacity for map cachedBlocks
2022-08-25 09:59:37,758 INFO util.GSet: VM type = 64-bit
2022-08-25 09:59:37,758 INFO util.GSet: 0.25% max memory 3.5 GB = 8.9 MB
2022-08-25 09:59:37,758 INFO util.GSet: capacity = 2^20 = 1048576 entries
2022-08-25 09:59:37,765 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
2022-08-25 09:59:37,765 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
2022-08-25 09:59:37,765 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
2022-08-25 09:59:37,769 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
2022-08-25 09:59:37,769 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
2022-08-25 09:59:37,770 INFO util.GSet: Computing capacity for map NameNodeRetryCache
2022-08-25 09:59:37,770 INFO util.GSet: VM type = 64-bit
2022-08-25 09:59:37,770 INFO util.GSet: 0.029999999329447746% max memory 3.5 GB = 1.1 MB
2022-08-25 09:59:37,771 INFO util.GSet: capacity = 2^17 = 131072 entries
2022-08-25 09:59:37,793 INFO namenode.FSImage: Allocated new BlockPoolId: BP-74677984-192.168.10.41-1661392777786
2022-08-25 09:59:37,859 INFO common.Storage: Storage directory /usr/local/bigdata/hadoop-3.1.4/dfs/name has been successfully formatted.
2022-08-25 09:59:37,879 INFO namenode.FSImageFormatProtobuf: Saving image file /usr/local/bigdata/hadoop-3.1.4/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
2022-08-25 09:59:37,968 INFO namenode.FSImageFormatProtobuf: Image file /usr/local/bigdata/hadoop-3.1.4/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 395 bytes saved in 0 seconds .
2022-08-25 09:59:37,997 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
2022-08-25 09:59:38,002 INFO namenode.FSImage: FSImageSaver clean checkpoint: txid = 0 when meet shutdown.
2022-08-25 09:59:38,002 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at server1/192.168.10.41
************************************************************/
出現(xiàn)以上信息，則表明初始化成功。
重要參考信息：**
Storage directory /usr/local/bigdata/hadoop-3.1.4/dfs/name has been successfully formatted.
**
10）、啟動Hadoop集群
#1、Hadoop集群啟動關(guān)閉-手動逐個進程啟停
#每臺機器上每次手動啟動關(guān)閉一個角色進程
# HDFS集群
hdfs --daemon start namenode|datanode|secondarynamenode
hdfs --daemon stop namenode|datanode|secondarynamenode
# YARN集群
yarn --daemon start resourcemanager|nodemanager
yarn --daemon stop resourcemanager|nodemanager
#2、 Hadoop集群啟動關(guān)閉-shell腳本一鍵啟停
# 在server1上，使用軟件自帶的shell腳本一鍵啟動
#前提：配置好機器之間的SSH免密登錄和workers文件。
#HDFS集群
start-dfs.sh
stop-dfs.sh
#YARN集群
start-yarn.sh
stop-yarn.sh
#Hadoop集群
start-all.sh
stop-all.sh
#驗證
server1
[alanchan@server1 bin]$ cd ../
[alanchan@server1 hadoop-3.1.4]$ cd sbin
[alanchan@server1 sbin]$ ll
總用量 112
-rwxr-xr-x 1 alanchan root 2756 11月 4 2020 distribute-exclude.sh
drwxr-xr-x 4 alanchan root 4096 11月 4 2020 FederationStateStore
-rwxr-xr-x 1 alanchan root 1983 11月 4 2020 hadoop-daemon.sh
-rwxr-xr-x 1 alanchan root 2522 11月 4 2020 hadoop-daemons.sh
-rwxr-xr-x 1 alanchan root 1542 11月 4 2020 httpfs.sh
-rwxr-xr-x 1 alanchan root 1500 11月 4 2020 kms.sh
-rwxr-xr-x 1 alanchan root 1841 11月 4 2020 mr-jobhistory-daemon.sh
-rwxr-xr-x 1 alanchan root 2086 11月 4 2020 refresh-namenodes.sh
-rwxr-xr-x 1 alanchan root 1779 11月 4 2020 start-all.cmd
-rwxr-xr-x 1 alanchan root 2221 11月 4 2020 start-all.sh
-rwxr-xr-x 1 alanchan root 1880 11月 4 2020 start-balancer.sh
-rwxr-xr-x 1 alanchan root 1401 11月 4 2020 start-dfs.cmd
-rwxr-xr-x 1 alanchan root 5170 11月 4 2020 start-dfs.sh
-rwxr-xr-x 1 alanchan root 1793 11月 4 2020 start-secure-dns.sh
-rwxr-xr-x 1 alanchan root 1571 11月 4 2020 start-yarn.cmd
-rwxr-xr-x 1 alanchan root 3342 11月 4 2020 start-yarn.sh
-rwxr-xr-x 1 alanchan root 1770 11月 4 2020 stop-all.cmd
-rwxr-xr-x 1 alanchan root 2166 11月 4 2020 stop-all.sh
-rwxr-xr-x 1 alanchan root 1783 11月 4 2020 stop-balancer.sh
-rwxr-xr-x 1 alanchan root 1455 11月 4 2020 stop-dfs.cmd
-rwxr-xr-x 1 alanchan root 3898 11月 4 2020 stop-dfs.sh
-rwxr-xr-x 1 alanchan root 1756 11月 4 2020 stop-secure-dns.sh
-rwxr-xr-x 1 alanchan root 1642 11月 4 2020 stop-yarn.cmd
-rwxr-xr-x 1 alanchan root 3083 11月 4 2020 stop-yarn.sh
-rwxr-xr-x 1 alanchan root 1982 11月 4 2020 workers.sh
-rwxr-xr-x 1 alanchan root 1814 11月 4 2020 yarn-daemon.sh
-rwxr-xr-x 1 alanchan root 2328 11月 4 2020 yarn-daemons.sh
[alanchan@server sbin]$ start-all.sh
WARNING: Attempting to start all Apache Hadoop daemons as alanchan in 10 seconds.
WARNING: This is not a recommended production deployment configuration.
WARNING: Use CTRL-C to abort.
Starting namenodes on [server1]
Starting datanodes
Starting secondary namenodes [server2]
2022-08-24 14:46:31,052 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting resourcemanager
Starting nodemanagers
[alanchan@server sbin]$ jps
10818 ResourceManager
10229 NameNode
22171 Jps
server2
[root@localhost bigdata]# jps
5648 DataNode
6067 NodeManager
17931 Jps
5837 SecondaryNameNode
server3
[root@localhost bigdata]# jps
19674 Jps
6923 NodeManager
6590 DataNode
server4
[root@localhost bigdata]# jps
19762 Jps
12408 DataNode
12622 NodeManager
至此，理論上已經(jīng)完成了部署，接下來進行初步的驗證。
11）、web UI登錄驗證
1、驗證namenode
http://192.168.10.41:9870/dfshealth.html#tab-overview
2、驗證datanode
點擊首頁的live datanodes（http://192.168.10.41:9870/dfshealth.html#tab-datanode）
每個datanode鏈接地址可以點擊進去，比如http://server2:9864/datanode.html
3、驗證hadoop集群
http://server1:8088/cluster（本機配置了hosts訪問方式）
http://192.168.10.41:8088/cluster
三、簡單驗證
經(jīng)過以上的步驟算是完成了hadoop集群的部署，下面將進行簡單的功能驗證。
1、shell
#在hadoop集群上任何一臺機子均可執(zhí)行
#創(chuàng)建目錄testhadoopcreate
[alanchan@localhost bigdata]$ hadoop fs -mkdir /testhadoopcreate
#查看根目錄下文件情況
[alanchan@localhost bigdata]$ hadoop fs -ls /
Found 3 items
-rw-r--r-- 3 alanchan supergroup 2204 2022-08-24 06:20 /in
drwxr-xr-x - alanchan supergroup 0 2022-08-24 05:55 /testdata
drwxr-xr-x - alanchan supergroup 0 2022-08-24 07:12 /testhadoopcreate
#上傳文件到指定目錄
[alanchan@localhost bigdata]$ hadoop fs -put /usr/local/bigdata/hadoop-3.1.4/NOTICE.txt /testhadoopcreate
#查看上傳文件情況
[alanchan@localhost bigdata]$ hadoop fs -ls /testhadoopcreate
Found 1 items
-rw-r--r-- 3 alanchan supergroup 21867 2022-08-24 07:15 /testhadoopcreate/NOTICE.txt
#如果出現(xiàn)如下異常（由于本機是安裝好了后，再做編譯，使相應的路徑有變化，會出現(xiàn)這種情況）
[alanchan@server1 sbin]$ hadoop fs /
2022-08-26 10:42:25,170 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
/: Unknown command
#檢查命令
hadoop checknative -a
[alanchan@server1 sbin]$ hadoop checknative -a
2022-08-26 10:43:59,775 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Native library checking:
hadoop: false
zlib: false
zstd : false
snappy: false
lz4: false
bzip2: false
openssl: false
ISA-L: false
PMDK: false
2022-08-26 10:43:59,904 INFO util.ExitUtil: Exiting with status 1: ExitException
https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/NativeLibraries.html#Native_Hadoop_Library
2、web Ui
創(chuàng)建目錄test，并且在test目錄下上傳文件
3、map-reduce
cd /usr/local/bigdata/hadoop-3.1.4/share/hadoop/mapreduce/
#運行圓周率計算的例子
hadoop jar hadoop-mapreduce-examples-3.1.4.jar pi 2 4
[alanchan@server4 root]$ cd /usr/local/bigdata/hadoop-3.1.4/share/hadoop/mapreduce/
[alanchan@server4 mapreduce]$ ll
總用量 5612
-rw-r--r-- 1 alanchan root 622132 8月 25 01:58 hadoop-mapreduce-client-app-3.1.4.jar
-rw-r--r-- 1 alanchan root 803483 8月 25 01:58 hadoop-mapreduce-client-common-3.1.4.jar
-rw-r--r-- 1 alanchan root 1656299 8月 25 01:58 hadoop-mapreduce-client-core-3.1.4.jar
-rw-r--r-- 1 alanchan root 215292 8月 25 01:58 hadoop-mapreduce-client-hs-3.1.4.jar
-rw-r--r-- 1 alanchan root 45332 8月 25 01:58 hadoop-mapreduce-client-hs-plugins-3.1.4.jar
-rw-r--r-- 1 alanchan root 85398 8月 25 01:58 hadoop-mapreduce-client-jobclient-3.1.4.jar
-rw-r--r-- 1 alanchan root 1681928 8月 25 01:58 hadoop-mapreduce-client-jobclient-3.1.4-tests.jar
-rw-r--r-- 1 alanchan root 126143 8月 25 01:58 hadoop-mapreduce-client-nativetask-3.1.4.jar
-rw-r--r-- 1 alanchan root 97140 8月 25 01:58 hadoop-mapreduce-client-shuffle-3.1.4.jar
-rw-r--r-- 1 alanchan root 57692 8月 25 01:58 hadoop-mapreduce-client-uploader-3.1.4.jar
-rw-r--r-- 1 alanchan root 316310 8月 25 01:58 hadoop-mapreduce-examples-3.1.4.jar
drwxr-xr-x 2 alanchan root 4096 8月 25 01:58 jdiff
drwxr-xr-x 2 alanchan root 4096 8月 25 01:58 lib
drwxr-xr-x 2 alanchan root 4096 8月 25 01:58 lib-examples
drwxr-xr-x 2 alanchan root 4096 8月 25 01:58 sources
[alanchan@server4 mapreduce]$ hadoop jar hadoop-mapreduce-examples-3.1.4.jar pi 2 4
Number of Maps = 2
Samples per Map = 4
Wrote input for Map #0
Wrote input for Map #1
Starting Job
2022-08-25 02:15:32,519 INFO client.RMProxy: Connecting to ResourceManager at server1/192.168.10.41:8032
2022-08-25 02:15:32,855 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/alanchan/.staging/job_1661393017372_0001
2022-08-25 02:15:33,000 INFO input.FileInputFormat: Total input files to process : 2
2022-08-25 02:15:33,082 INFO mapreduce.JobSubmitter: number of splits:2
2022-08-25 02:15:33,258 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1661393017372_0001
2022-08-25 02:15:33,259 INFO mapreduce.JobSubmitter: Executing with tokens: []
2022-08-25 02:15:33,400 INFO conf.Configuration: resource-types.xml not found
2022-08-25 02:15:33,400 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2022-08-25 02:15:33,782 INFO impl.YarnClientImpl: Submitted application application_1661393017372_0001
2022-08-25 02:15:33,814 INFO mapreduce.Job: The url to track the job: http://server1:8088/proxy/application_1661393017372_0001/
2022-08-25 02:15:33,815 INFO mapreduce.Job: Running job: job_1661393017372_0001
2022-08-25 02:15:40,888 INFO mapreduce.Job: Job job_1661393017372_0001 running in uber mode : false
2022-08-25 02:15:40,889 INFO mapreduce.Job: map 0% reduce 0%
2022-08-25 02:15:46,935 INFO mapreduce.Job: map 100% reduce 0%
2022-08-25 02:15:52,962 INFO mapreduce.Job: map 100% reduce 100%
2022-08-25 02:15:52,968 INFO mapreduce.Job: Job job_1661393017372_0001 completed successfully
2022-08-25 02:15:53,047 INFO mapreduce.Job: Counters: 53
File System Counters
FILE: Number of bytes read=50
FILE: Number of bytes written=666276
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=530
HDFS: Number of bytes written=215
HDFS: Number of read operations=13
HDFS: Number of large read operations=0
HDFS: Number of write operations=3
Job Counters
Launched map tasks=2
Launched reduce tasks=1
Data-local map tasks=2
Total time spent by all maps in occupied slots (ms)=6316
Total time spent by all reduces in occupied slots (ms)=3252
Total time spent by all map tasks (ms)=6316
Total time spent by all reduce tasks (ms)=3252
Total vcore-milliseconds taken by all map tasks=6316
Total vcore-milliseconds taken by all reduce tasks=3252
Total megabyte-milliseconds taken by all map tasks=6467584
Total megabyte-milliseconds taken by all reduce tasks=3330048
Map-Reduce Framework
Map input records=2
Map output records=4
Map output bytes=36
Map output materialized bytes=56
Input split bytes=294
Combine input records=0
Combine output records=0
Reduce input groups=2
Reduce shuffle bytes=56
Reduce input records=4
Reduce output records=0
Spilled Records=8
Shuffled Maps =2
Failed Shuffles=0
Merged Map outputs=2
GC time elapsed (ms)=145
CPU time spent (ms)=1640
Physical memory (bytes) snapshot=844423168
Virtual memory (bytes) snapshot=8351129600
Total committed heap usage (bytes)=749207552
Peak Map Physical memory (bytes)=342159360
Peak Map Virtual memory (bytes)=2788798464
Peak Reduce Physical memory (bytes)=200597504
Peak Reduce Virtual memory (bytes)=2775109632
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=236
File Output Format Counters
Bytes Written=97
Job Finished in 20.591 seconds
Estimated value of Pi is 3.50000000000000000000
四、hadoop集群的基準測試
1、測試寫入速度
#向HDFS文件系統(tǒng)中寫入數(shù)據(jù),100個文件,每個文件100MB,文件存放到/benchmarks/TestDFSIO中
#Throughput：吞吐量、Average IO rate：平均IO率、IO rate std deviation：IO率標準偏差
hadoop jar /usr/local/bigdata/hadoop-3.1.4/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.4-tests.jar TestDFSIO -write -nrFiles 100 -fileSize 100MB
[alanchan@server4 mapreduce]$ hadoop jar /usr/local/bigdata/hadoop-3.1.4/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.4-tests.jar TestDFSIO -write -nrFiles 100 -fileSize 100MB
2022-08-25 02:54:12,517 INFO fs.TestDFSIO: TestDFSIO.1.8
2022-08-25 02:54:12,518 INFO fs.TestDFSIO: nrFiles = 100
2022-08-25 02:54:12,518 INFO fs.TestDFSIO: nrBytes (MB) = 100.0
2022-08-25 02:54:12,518 INFO fs.TestDFSIO: bufferSize = 1000000
2022-08-25 02:54:12,518 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
2022-08-25 02:54:13,127 INFO fs.TestDFSIO: creating control file: 104857600 bytes, 100 files
2022-08-25 02:54:15,421 INFO fs.TestDFSIO: created control files for: 100 files
2022-08-25 02:54:15,493 INFO client.RMProxy: Connecting to ResourceManager at server1/192.168.10.41:8032
2022-08-25 02:54:15,648 INFO client.RMProxy: Connecting to ResourceManager at server1/192.168.10.41:8032
2022-08-25 02:54:15,834 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/alanchan/.staging/job_1661395853554_0002
2022-08-25 02:54:16,127 INFO mapred.FileInputFormat: Total input files to process : 100
2022-08-25 02:54:16,201 INFO mapreduce.JobSubmitter: number of splits:100
2022-08-25 02:54:16,329 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1661395853554_0002
2022-08-25 02:54:16,331 INFO mapreduce.JobSubmitter: Executing with tokens: []
2022-08-25 02:54:16,467 INFO conf.Configuration: resource-types.xml not found
2022-08-25 02:54:16,467 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2022-08-25 02:54:16,517 INFO impl.YarnClientImpl: Submitted application application_1661395853554_0002
2022-08-25 02:54:16,547 INFO mapreduce.Job: The url to track the job: http://server1:8088/proxy/application_1661395853554_0002/
2022-08-25 02:54:16,548 INFO mapreduce.Job: Running job: job_1661395853554_0002
2022-08-25 02:54:21,619 INFO mapreduce.Job: Job job_1661395853554_0002 running in uber mode : false
2022-08-25 02:54:21,620 INFO mapreduce.Job: map 0% reduce 0%
2022-08-25 02:54:41,466 INFO mapreduce.Job: map 1% reduce 0%
2022-08-25 02:54:42,556 INFO mapreduce.Job: map 4% reduce 0%
2022-08-25 02:54:43,609 INFO mapreduce.Job: map 5% reduce 0%
2022-08-25 02:54:44,694 INFO mapreduce.Job: map 11% reduce 0%
2022-08-25 02:54:45,749 INFO mapreduce.Job: map 15% reduce 0%
2022-08-25 02:56:13,530 INFO mapreduce.Job: map 16% reduce 0%
2022-08-25 02:56:18,956 INFO mapreduce.Job: map 17% reduce 0%
2022-08-25 02:56:33,059 INFO mapreduce.Job: map 18% reduce 0%
2022-08-25 02:56:34,150 INFO mapreduce.Job: map 19% reduce 0%
2022-08-25 02:57:19,554 INFO mapreduce.Job: map 20% reduce 0%
2022-08-25 02:57:25,523 INFO mapreduce.Job: map 21% reduce 0%
2022-08-25 02:57:28,760 INFO mapreduce.Job: map 22% reduce 0%
2022-08-25 02:57:30,919 INFO mapreduce.Job: map 22% reduce 4%
2022-08-25 02:57:32,000 INFO mapreduce.Job: map 24% reduce 4%
2022-08-25 02:57:33,084 INFO mapreduce.Job: map 25% reduce 4%
2022-08-25 02:57:34,167 INFO mapreduce.Job: map 26% reduce 4%
2022-08-25 02:57:37,641 INFO mapreduce.Job: map 27% reduce 8%
2022-08-25 02:57:39,795 INFO mapreduce.Job: map 28% reduce 8%
2022-08-25 02:57:44,378 INFO mapreduce.Job: map 29% reduce 8%
2022-08-25 02:57:45,694 INFO mapreduce.Job: map 30% reduce 8%
2022-08-25 02:57:46,785 INFO mapreduce.Job: map 31% reduce 8%
2022-08-25 02:57:47,880 INFO mapreduce.Job: map 32% reduce 8%
2022-08-25 02:57:50,052 INFO mapreduce.Job: map 34% reduce 8%
2022-08-25 02:57:51,136 INFO mapreduce.Job: map 36% reduce 8%
2022-08-25 02:57:52,215 INFO mapreduce.Job: map 37% reduce 8%
2022-08-25 02:58:53,057 INFO mapreduce.Job: map 38% reduce 8%
2022-08-25 02:59:04,346 INFO mapreduce.Job: map 39% reduce 8%
2022-08-25 02:59:07,803 INFO mapreduce.Job: map 40% reduce 8%
2022-08-25 02:59:08,880 INFO mapreduce.Job: map 40% reduce 9%
2022-08-25 02:59:18,849 INFO mapreduce.Job: map 41% reduce 9%
2022-08-25 02:59:21,029 INFO mapreduce.Job: map 41% reduce 10%
2022-08-25 02:59:22,113 INFO mapreduce.Job: map 42% reduce 10%
2022-08-25 02:59:28,829 INFO mapreduce.Job: map 43% reduce 10%
2022-08-25 02:59:57,449 INFO mapreduce.Job: map 44% reduce 10%
2022-08-25 03:00:02,848 INFO mapreduce.Job: map 44% reduce 11%
2022-08-25 03:00:08,275 INFO mapreduce.Job: map 45% reduce 11%
2022-08-25 03:00:12,603 INFO mapreduce.Job: map 46% reduce 11%
2022-08-25 03:00:14,989 INFO mapreduce.Job: map 47% reduce 12%
2022-08-25 03:00:18,244 INFO mapreduce.Job: map 48% reduce 12%
2022-08-25 03:00:20,414 INFO mapreduce.Job: map 49% reduce 12%
2022-08-25 03:00:21,499 INFO mapreduce.Job: map 49% reduce 13%
2022-08-25 03:00:23,674 INFO mapreduce.Job: map 50% reduce 13%
2022-08-25 03:00:29,091 INFO mapreduce.Job: map 51% reduce 13%
2022-08-25 03:00:30,176 INFO mapreduce.Job: map 52% reduce 13%
2022-08-25 03:00:32,348 INFO mapreduce.Job: map 53% reduce 13%
2022-08-25 03:00:33,437 INFO mapreduce.Job: map 53% reduce 15%
2022-08-25 03:00:35,598 INFO mapreduce.Job: map 54% reduce 15%
2022-08-25 03:00:38,831 INFO mapreduce.Job: map 55% reduce 15%
2022-08-25 03:00:40,993 INFO mapreduce.Job: map 56% reduce 15%
2022-08-25 03:00:43,150 INFO mapreduce.Job: map 57% reduce 15%
2022-08-25 03:00:45,313 INFO mapreduce.Job: map 58% reduce 15%
2022-08-25 03:00:47,458 INFO mapreduce.Job: map 59% reduce 15%
2022-08-25 03:00:54,249 INFO mapreduce.Job: map 60% reduce 15%
2022-08-25 03:00:58,591 INFO mapreduce.Job: map 60% reduce 16%
2022-08-25 03:01:08,314 INFO mapreduce.Job: map 61% reduce 16%
2022-08-25 03:01:52,066 INFO mapreduce.Job: map 62% reduce 16%
2022-08-25 03:02:00,733 INFO mapreduce.Job: map 63% reduce 16%
2022-08-25 03:02:04,201 INFO mapreduce.Job: map 63% reduce 17%
2022-08-25 03:02:09,610 INFO mapreduce.Job: map 64% reduce 17%
2022-08-25 03:02:15,017 INFO mapreduce.Job: map 65% reduce 17%
2022-08-25 03:02:25,848 INFO mapreduce.Job: map 66% reduce 17%
2022-08-25 03:02:28,243 INFO mapreduce.Job: map 66% reduce 18%
2022-08-25 03:02:29,329 INFO mapreduce.Job: map 67% reduce 18%
2022-08-25 03:02:39,335 INFO mapreduce.Job: map 68% reduce 18%
2022-08-25 03:02:40,408 INFO mapreduce.Job: map 68% reduce 19%
2022-08-25 03:02:43,653 INFO mapreduce.Job: map 69% reduce 19%
2022-08-25 03:02:44,726 INFO mapreduce.Job: map 70% reduce 19%
2022-08-25 03:02:50,110 INFO mapreduce.Job: map 71% reduce 19%
2022-08-25 03:02:52,830 INFO mapreduce.Job: map 72% reduce 20%
2022-08-25 03:03:03,013 INFO mapreduce.Job: map 73% reduce 20%
2022-08-25 03:03:04,098 INFO mapreduce.Job: map 74% reduce 20%
2022-08-25 03:03:07,344 INFO mapreduce.Job: map 75% reduce 20%
2022-08-25 03:03:12,764 INFO mapreduce.Job: map 76% reduce 20%
2022-08-25 03:03:17,339 INFO mapreduce.Job: map 76% reduce 21%
2022-08-25 03:03:22,761 INFO mapreduce.Job: map 76% reduce 22%
2022-08-25 03:03:23,839 INFO mapreduce.Job: map 77% reduce 22%
2022-08-25 03:03:26,010 INFO mapreduce.Job: map 78% reduce 22%
2022-08-25 03:03:27,097 INFO mapreduce.Job: map 79% reduce 22%
2022-08-25 03:03:29,266 INFO mapreduce.Job: map 79% reduce 23%
2022-08-25 03:03:34,908 INFO mapreduce.Job: map 80% reduce 23%
2022-08-25 03:03:39,241 INFO mapreduce.Job: map 82% reduce 23%
2022-08-25 03:03:52,923 INFO mapreduce.Job: map 83% reduce 23%
2022-08-25 03:03:57,470 INFO mapreduce.Job: map 84% reduce 23%
2022-08-25 03:03:59,630 INFO mapreduce.Job: map 84% reduce 24%
2022-08-25 03:04:10,447 INFO mapreduce.Job: map 85% reduce 24%
2022-08-25 03:04:20,630 INFO mapreduce.Job: map 86% reduce 24%
2022-08-25 03:04:27,116 INFO mapreduce.Job: map 87% reduce 24%
2022-08-25 03:04:29,281 INFO mapreduce.Job: map 87% reduce 25%
2022-08-25 03:04:35,742 INFO mapreduce.Job: map 88% reduce 25%
2022-08-25 03:04:36,816 INFO mapreduce.Job: map 89% reduce 25%
2022-08-25 03:04:42,448 INFO mapreduce.Job: map 90% reduce 26%
2022-08-25 03:04:44,597 INFO mapreduce.Job: map 91% reduce 26%
2022-08-25 03:04:47,840 INFO mapreduce.Job: map 91% reduce 27%
2022-08-25 03:04:50,001 INFO mapreduce.Job: map 92% reduce 27%
2022-08-25 03:04:52,377 INFO mapreduce.Job: map 93% reduce 27%
2022-08-25 03:04:55,620 INFO mapreduce.Job: map 94% reduce 27%
2022-08-25 03:05:17,872 INFO mapreduce.Job: map 95% reduce 28%
2022-08-25 03:05:29,747 INFO mapreduce.Job: map 96% reduce 29%
2022-08-25 03:05:44,877 INFO mapreduce.Job: map 97% reduce 29%
2022-08-25 03:05:48,361 INFO mapreduce.Job: map 97% reduce 30%
2022-08-25 03:06:06,105 INFO mapreduce.Job: map 98% reduce 31%
2022-08-25 03:06:12,564 INFO mapreduce.Job: map 98% reduce 32%
2022-08-25 03:06:13,585 INFO mapreduce.Job: map 100% reduce 32%
2022-08-25 03:06:15,589 INFO mapreduce.Job: map 100% reduce 100%
2022-08-25 03:06:15,594 INFO mapreduce.Job: Job job_1661395853554_0002 completed successfully
2022-08-25 03:06:15,667 INFO mapreduce.Job: Counters: 53
File System Counters
FILE: Number of bytes read=8600
FILE: Number of bytes written=22446697
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=23480
HDFS: Number of bytes written=10485760081
HDFS: Number of read operations=405
HDFS: Number of large read operations=0
HDFS: Number of write operations=102
Job Counters
Launched map tasks=100
Launched reduce tasks=1
Data-local map tasks=100
Total time spent by all maps in occupied slots (ms)=14377072
Total time spent by all reduces in occupied slots (ms)=537813
Total time spent by all map tasks (ms)=14377072
Total time spent by all reduce tasks (ms)=537813
Total vcore-milliseconds taken by all map tasks=14377072
Total vcore-milliseconds taken by all reduce tasks=537813
Total megabyte-milliseconds taken by all map tasks=14722121728
Total megabyte-milliseconds taken by all reduce tasks=550720512
Map-Reduce Framework
Map input records=100
Map output records=500
Map output bytes=7594
Map output materialized bytes=9194
Input split bytes=12190
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=9194
Reduce input records=500
Reduce output records=5
Spilled Records=1000
Shuffled Maps =100
Failed Shuffles=0
Merged Map outputs=100
GC time elapsed (ms)=12694
CPU time spent (ms)=233920
Physical memory (bytes) snapshot=35032891392
Virtual memory (bytes) snapshot=280810868736
Total committed heap usage (bytes)=41180725248
Peak Map Physical memory (bytes)=389386240
Peak Map Virtual memory (bytes)=2800513024
Peak Reduce Physical memory (bytes)=362102784
Peak Reduce Virtual memory (bytes)=2801356800
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=11290
File Output Format Counters
Bytes Written=81
2022-08-25 03:06:15,705 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
2022-08-25 03:06:15,705 INFO fs.TestDFSIO: Date & time: Thu Aug 25 03:06:15 GMT 2022
2022-08-25 03:06:15,705 INFO fs.TestDFSIO: Number of files: 100
2022-08-25 03:06:15,705 INFO fs.TestDFSIO: Total MBytes processed: 10000
2022-08-25 03:06:15,705 INFO fs.TestDFSIO: Throughput mb/sec: 0.72
2022-08-25 03:06:15,705 INFO fs.TestDFSIO: Average IO rate mb/sec: 0.77
2022-08-25 03:06:15,705 INFO fs.TestDFSIO: IO rate std deviation: 0.21
2022-08-25 03:06:15,705 INFO fs.TestDFSIO: Test exec time sec: 720.25
2022-08-25 03:06:15,705 INFO fs.TestDFSIO:
按照上面的yarn-site.xml配置會出現(xiàn)如下異常(虛擬內(nèi)存不夠)
[2022-08-25 02:37:18.885]Container [pid=20349,containerID=container_1661393017372_0006_01_000007] is running 515447296B beyond the 'VIRTUAL' memory limit. Current usage: 142.5 MB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1661393017372_0006_01_000007 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 20393 20349 20349 20349 (java) 242 8 2654449664 36137 /usr/java/jdk1.8.0_144/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/usr/local/bigdata/hadoop-3.1.4/nm-local-dir/usercache/alanchan/appcache/application_1661393017372_0006/container_1661393017372_0006_01_000007/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/usr/local/bigdata/hadoop-3.1.4/logs/userlogs/application_1661393017372_0006/container_1661393017372_0006_01_000007 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.10.44 44408 attempt_1661393017372_0006_m_000005_0 7
|- 20349 20345 20349 20349 (bash) 0 0 115855360 352 /bin/bash -c /usr/java/jdk1.8.0_144/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/usr/local/bigdata/hadoop-3.1.4/nm-local-dir/usercache/alanchan/appcache/application_1661393017372_0006/container_1661393017372_0006_01_000007/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/usr/local/bigdata/hadoop-3.1.4/logs/userlogs/application_1661393017372_0006/container_1661393017372_0006_01_000007 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.10.44 44408 attempt_1661393017372_0006_m_000005_0 7 1>/usr/local/bigdata/hadoop-3.1.4/logs/userlogs/application_1661393017372_0006/container_1661393017372_0006_01_000007/stdout 2>/usr/local/bigdata/hadoop-3.1.4/logs/userlogs/application_1661393017372_0006/container_1661393017372_0006_01_000007/stderr
[2022-08-25 02:37:18.959]Container killed on request. Exit code is 143
解決辦法：文章來源：http://www.zghlxwxcb.cn/news/detail-471594.html
#修改yarn-site.xml文件，將虛擬內(nèi)存與物理內(nèi)存比例設置為4，增加虛擬內(nèi)存檢查為false，默認是true
<!-- 容器虛擬內(nèi)存與物理內(nèi)存之間的比率。-->
<property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>4</value>
</property>
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
<description>Whether virtual memory limits will be enforced for containers</description>
</property>
設置完成后，先關(guān)閉yarn，然后再啟動
stop-yarn.sh
start-yarn.sh
2、測試讀取速度
#在HDFS文件系統(tǒng)中讀入100個文件,每個文件100M
#Throughput：吞吐量、Average IO rate：平均IO率、IO rate std deviation：IO率標準偏差
hadoop jar /export/server/hadoop-3.1.4/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.4-tests.jar TestDFSIO -read -nrFiles 100 -fileSize 100MB
[alanchan@server4 mapreduce]$ hadoop jar /usr/local/bigdata/hadoop-3.1.4/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.4-tests.jar TestDFSIO -read -nrFiles 100 -fileSize 100MB
2022-08-25 03:55:28,687 INFO fs.TestDFSIO: TestDFSIO.1.8
2022-08-25 03:55:28,689 INFO fs.TestDFSIO: nrFiles = 100
2022-08-25 03:55:28,689 INFO fs.TestDFSIO: nrBytes (MB) = 100.0
2022-08-25 03:55:28,689 INFO fs.TestDFSIO: bufferSize = 1000000
2022-08-25 03:55:28,689 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
2022-08-25 03:55:29,301 INFO fs.TestDFSIO: creating control file: 104857600 bytes, 100 files
2022-08-25 03:55:31,885 INFO fs.TestDFSIO: created control files for: 100 files
2022-08-25 03:55:31,957 INFO client.RMProxy: Connecting to ResourceManager at server1/192.168.10.41:8032
2022-08-25 03:55:32,114 INFO client.RMProxy: Connecting to ResourceManager at server1/192.168.10.41:8032
2022-08-25 03:55:32,369 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/alanchan/.staging/job_1661395853554_0008
2022-08-25 03:55:32,652 INFO mapred.FileInputFormat: Total input files to process : 100
2022-08-25 03:55:32,729 INFO mapreduce.JobSubmitter: number of splits:100
2022-08-25 03:55:32,859 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1661395853554_0008
2022-08-25 03:55:32,861 INFO mapreduce.JobSubmitter: Executing with tokens: []
2022-08-25 03:55:33,001 INFO conf.Configuration: resource-types.xml not found
2022-08-25 03:55:33,001 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2022-08-25 03:55:33,049 INFO impl.YarnClientImpl: Submitted application application_1661395853554_0008
2022-08-25 03:55:33,079 INFO mapreduce.Job: The url to track the job: http://server1:8088/proxy/application_1661395853554_0008/
2022-08-25 03:55:33,080 INFO mapreduce.Job: Running job: job_1661395853554_0008
2022-08-25 03:55:38,165 INFO mapreduce.Job: Job job_1661395853554_0008 running in uber mode : false
2022-08-25 03:55:38,166 INFO mapreduce.Job: map 0% reduce 0%
2022-08-25 03:55:47,245 INFO mapreduce.Job: map 3% reduce 0%
2022-08-25 03:55:48,255 INFO mapreduce.Job: map 6% reduce 0%
2022-08-25 03:55:50,263 INFO mapreduce.Job: map 14% reduce 0%
2022-08-25 03:55:51,270 INFO mapreduce.Job: map 22% reduce 0%
2022-08-25 03:55:55,306 INFO mapreduce.Job: map 24% reduce 0%
2022-08-25 03:55:56,310 INFO mapreduce.Job: map 28% reduce 0%
2022-08-25 03:55:58,322 INFO mapreduce.Job: map 30% reduce 0%
2022-08-25 03:56:00,335 INFO mapreduce.Job: map 35% reduce 0%
2022-08-25 03:56:01,340 INFO mapreduce.Job: map 36% reduce 0%
2022-08-25 03:56:02,351 INFO mapreduce.Job: map 43% reduce 0%
2022-08-25 03:56:03,355 INFO mapreduce.Job: map 45% reduce 0%
2022-08-25 03:56:04,363 INFO mapreduce.Job: map 49% reduce 0%
2022-08-25 03:56:06,369 INFO mapreduce.Job: map 50% reduce 0%
2022-08-25 03:56:07,373 INFO mapreduce.Job: map 51% reduce 0%
2022-08-25 03:56:08,377 INFO mapreduce.Job: map 55% reduce 17%
2022-08-25 03:56:09,381 INFO mapreduce.Job: map 57% reduce 17%
2022-08-25 03:56:11,392 INFO mapreduce.Job: map 60% reduce 17%
2022-08-25 03:56:12,396 INFO mapreduce.Job: map 66% reduce 17%
2022-08-25 03:56:13,401 INFO mapreduce.Job: map 71% reduce 17%
2022-08-25 03:56:14,405 INFO mapreduce.Job: map 71% reduce 24%
2022-08-25 03:56:15,409 INFO mapreduce.Job: map 73% reduce 24%
2022-08-25 03:56:16,414 INFO mapreduce.Job: map 75% reduce 24%
2022-08-25 03:56:17,419 INFO mapreduce.Job: map 77% reduce 24%
2022-08-25 03:56:18,422 INFO mapreduce.Job: map 79% reduce 24%
2022-08-25 03:56:19,427 INFO mapreduce.Job: map 84% reduce 24%
2022-08-25 03:56:20,430 INFO mapreduce.Job: map 84% reduce 28%
2022-08-25 03:56:21,440 INFO mapreduce.Job: map 86% reduce 28%
2022-08-25 03:56:23,446 INFO mapreduce.Job: map 92% reduce 28%
2022-08-25 03:56:24,449 INFO mapreduce.Job: map 99% reduce 28%
2022-08-25 03:56:25,452 INFO mapreduce.Job: map 100% reduce 28%
2022-08-25 03:56:26,455 INFO mapreduce.Job: map 100% reduce 100%
2022-08-25 03:56:26,460 INFO mapreduce.Job: Job job_1661395853554_0008 completed successfully
2022-08-25 03:56:26,539 INFO mapreduce.Job: Counters: 53
File System Counters
FILE: Number of bytes read=8588
FILE: Number of bytes written=22446471
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=10485783480
HDFS: Number of bytes written=84
HDFS: Number of read operations=505
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=100
Launched reduce tasks=1
Data-local map tasks=100
Total time spent by all maps in occupied slots (ms)=823913
Total time spent by all reduces in occupied slots (ms)=35507
Total time spent by all map tasks (ms)=823913
Total time spent by all reduce tasks (ms)=35507
Total vcore-milliseconds taken by all map tasks=823913
Total vcore-milliseconds taken by all reduce tasks=35507
Total megabyte-milliseconds taken by all map tasks=843686912
Total megabyte-milliseconds taken by all reduce tasks=36359168
Map-Reduce Framework
Map input records=100
Map output records=500
Map output bytes=7582
Map output materialized bytes=9182
Input split bytes=12190
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=9182
Reduce input records=500
Reduce output records=5
Spilled Records=1000
Shuffled Maps =100
Failed Shuffles=0
Merged Map outputs=100
GC time elapsed (ms)=11482
CPU time spent (ms)=72370
Physical memory (bytes) snapshot=30965710848
Virtual memory (bytes) snapshot=280591712256
Total committed heap usage (bytes)=20881342464
Peak Map Physical memory (bytes)=353017856
Peak Map Virtual memory (bytes)=2795540480
Peak Reduce Physical memory (bytes)=214671360
Peak Reduce Virtual memory (bytes)=2798829568
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=11290
File Output Format Counters
Bytes Written=84
2022-08-25 03:56:26,576 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
2022-08-25 03:56:26,576 INFO fs.TestDFSIO: Date & time: Thu Aug 25 03:56:26 GMT 2022
2022-08-25 03:56:26,576 INFO fs.TestDFSIO: Number of files: 100
2022-08-25 03:56:26,576 INFO fs.TestDFSIO: Total MBytes processed: 10000
2022-08-25 03:56:26,576 INFO fs.TestDFSIO: Throughput mb/sec: 195.68
2022-08-25 03:56:26,576 INFO fs.TestDFSIO: Average IO rate mb/sec: 243.47
2022-08-25 03:56:26,576 INFO fs.TestDFSIO: IO rate std deviation: 138.98
2022-08-25 03:56:26,576 INFO fs.TestDFSIO: Test exec time sec: 54.65
2022-08-25 03:56:26,576 INFO fs.TestDFSIO:
3、清除測試數(shù)據(jù)
#測試期間，會在HDFS集群上創(chuàng)建 /benchmarks目錄，測試完畢后，我們可以清理該目錄。
hadoop jar /usr/local/bigdata/hadoop-3.1.4/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.4-tests.jar TestDFSIO -clean
[alanchan@server4 mapreduce]$ hadoop jar /usr/local/bigdata/hadoop-3.1.4/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.4-tests.jar TestDFSIO -clean
2022-08-25 03:59:44,676 INFO fs.TestDFSIO: TestDFSIO.1.8
2022-08-25 03:59:44,677 INFO fs.TestDFSIO: nrFiles = 1
2022-08-25 03:59:44,677 INFO fs.TestDFSIO: nrBytes (MB) = 1.0
2022-08-25 03:59:44,677 INFO fs.TestDFSIO: bufferSize = 1000000
2022-08-25 03:59:44,677 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
2022-08-25 03:59:45,287 INFO fs.TestDFSIO: Cleaning up test files
五、常見異常處理
1、瀏覽器HDFS文件系統(tǒng)上傳文件時報Couldn’t upload the file錯誤
F12打開谷歌控制臺，看到報as been blocked by CORS policy: No ‘Access-Control-Allow-Origin’ header is present on the requested resource.錯誤，大概意思是由于跨域訪問CORS policy策略，訪問阻塞了：
有時候是網(wǎng)絡的原因，有時候是用戶配置的原因文章來源地址http://www.zghlxwxcb.cn/news/detail-471594.html
2、在格式化期間可能出現(xiàn)如下異常的解決辦法
以下警告解決
2022-08-25 09:59:37,186 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Formatting using clusterid: CID-ffdc8a28-a92d-45fc-bd34-382c1645e64a
[root@master native]# ldd libhadoop.so.1.0.0
./libhadoop.so.1.0.0: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by ./libhadoop.so.1.0.0)
linux-vdso.so.1 => (0x00007fff31efd000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f654dd25000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f654db07000)
libc.so.6 => /lib64/libc.so.6 (0x00007f654d775000)
/lib64/ld-linux-x86-64.so.2 (0x00007f654e155000)
[root@master sbin]# ldd --version
ldd (GNU libc) 2.12
Copyright (C) 2010 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
由 Roland McGrath 和 Ulrich Drepper 編寫。
[root@master sbin]# strings /lib64/libc.so.6|grep GLIBC
GLIBC_2.2.5
GLIBC_2.2.6
GLIBC_2.3
GLIBC_2.3.2
GLIBC_2.3.3
GLIBC_2.3.4
GLIBC_2.4
GLIBC_2.5
GLIBC_2.6
GLIBC_2.7
GLIBC_2.8
GLIBC_2.9
GLIBC_2.10
GLIBC_2.11
GLIBC_2.12
GLIBC_PRIVATE
# 最大支持到2.12版本，而我們hadoop中的版本是2.14，可以確定基本是這個問題造成的；那接下來的解決辦法就是更新系統(tǒng)的CLIBC版本。
cd /usr/local/tools
wget http://ftp.gnu.org/gnu/glibc/glibc-2.17.tar.gz
[root@master tools]# tar -zxvf glibc-2.17.tar.gz -C /usr/local
[root@master tools]# cd /usr/local/glibc-2.17
[root@master glibc-2.17]#
[root@master glibc-2.17]# mkdir build
[root@master glibc-2.17]# cd build/
[root@master build]#../configure --prefix=/usr --disable-profile --enable-add-ons --with-headers=/usr/include --with-binutils=/usr/bin
[root@master build]#make && make install
[root@master build]# strings /lib64/libc.so.6|grep GLIBC
GLIBC_2.2.5
GLIBC_2.2.6
GLIBC_2.3
GLIBC_2.3.2
GLIBC_2.3.3
GLIBC_2.3.4
GLIBC_2.4
GLIBC_2.5
GLIBC_2.6
GLIBC_2.7
GLIBC_2.8
GLIBC_2.9
GLIBC_2.10
GLIBC_2.11
GLIBC_2.12
GLIBC_2.13
GLIBC_2.14
GLIBC_2.15
GLIBC_2.16
GLIBC_2.17
GLIBC_PRIVATE
#重新啟動
[alanchan@server1 sbin]$ start-all.sh
WARNING: Attempting to start all Apache Hadoop daemons as alanchan in 10 seconds.
WARNING: This is not a recommended production deployment configuration.
WARNING: Use CTRL-C to abort.
Starting namenodes on [server1]
Starting datanodes
Starting secondary namenodes [server2]
Starting resourcemanager
Starting nodemanagers
到了這里，關(guān)于1、hadoop3.1.4簡單介紹及部署、簡單驗證的文章就介紹完了。如果您還想了解更多內(nèi)容，請在右上角搜索TOY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！
