CentOS 搭建 Hadoop3 高可用集群

這篇具有很好參考價值的文章主要介紹了CentOS 搭建 Hadoop3 高可用集群。希望對大家有所幫助。如果存在錯誤或未考慮完全的地方，請大家不吝賜教，您也可以點擊"舉報違法"按鈕提交疑問。

Hadoop FullyDistributed Mode 完全分布式

spark101	spark102	spark103
192.168.171.101	192.168.171.102	192.168.171.103
namenode	namenode
journalnode	journalnode	journalnode
datanode	datanode	datanode
nodemanager	nodemanager	nodemanager
recource manager	recource manager
job history
job log	job log	job log

1. 準備

1.1 升級操作系統(tǒng)和軟件

yum -y update

升級后建議重啟

1.2 安裝常用軟件

yum -y install gcc gcc-c++ autoconf automake cmake make rsync vim man zip unzip net-tools zlib zlib-devel openssl openssl-devel pcre-devel tcpdump lrzsz tar wget openssh-server

1.3 修改主機名

hostnamectl set-hostname spark01

hostnamectl set-hostname spark02

hostnamectl set-hostname spark03

1.4 修改IP地址

vim /etc/sysconfig/network-scripts/ifcfg-ens160

網(wǎng)卡配置文件示例

TYPE="Ethernet"
PROXY_METHOD="none"
BROWSER_ONLY="no"
BOOTPROTO="none"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="ens32"
DEVICE="ens32"
ONBOOT="yes"
IPADDR="192.168.171.101"
PREFIX="24"
GATEWAY="192.168.171.2"
DNS1="192.168.171.2"
IPV6_PRIVACY="no"

1.5 關(guān)閉防火墻

sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/configsetenforce 0
systemctl stop firewalld
systemctl disable firewalld

1.6 修改hosts配置文件

vim /etc/hosts

修改內(nèi)容如下：

192.168.171.101	spark01
192.168.171.102	spark02
192.168.171.103	spark03

1.7 上傳軟件配置環(huán)境變量

在所有主機節(jié)點創(chuàng)建軟件目錄

mkdir -p /opt/soft

以下操作在 hadoop101 主機上完成

進入軟件目錄

cd /opt/soft

下載 JDK

wget https://download.oracle.com/otn/java/jdk/8u391-b13/b291ca3e0c8548b5a51d5a5f50063037/jdk-8u391-linux-x64.tar.gz?AuthParam=1698206552_11c0bb831efdf87adfd187b0e4ccf970

下載 zookeeper

wget https://dlcdn.apache.org/zookeeper/zookeeper-3.8.3/apache-zookeeper-3.8.3-bin.tar.gz

下載 hadoop

wget https://dlcdn.apache.org/hadoop/common/hadoop-3.3.5/hadoop-3.3.5.tar.gz

解壓 JDK 修改名稱

解壓 zookeeper 修改名稱

解壓 hadoop 修改名稱

tar -zxvf jdk-8u391-linux-x64.tar.gz -C /opt/soft/
mv jdk1.8.0_391/ jdk-8
tar -zxvf apache-zookeeper-3.8.3-bin.tar.gz
mv apache-zookeeper-3.8.3-bin zookeeper-3
tar -zxvf hadoop-3.3.5.tar.gz -C /opt/soft/
mv hadoop-3.3.5/ hadoop-3

配置環(huán)境變量

vim /etc/profile.d/my_env.sh

編寫以下內(nèi)容：

export JAVA_HOME=/opt/soft/jdk-8
# export set JAVA_OPTS="--add-opens java.base/java.lang=ALL-UNNAMED"

export ZOOKEEPER_HOME=/opt/soft/zookeeper-3

export HDFS_NAMENODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_ZKFC_USER=root
export HDFS_JOURNALNODE_USER=root
export HADOOP_SHELL_EXECNAME=root

export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root


export HADOOP_HOME=/opt/soft/hadoop-3
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

export PATH=$PATH:$JAVA_HOME/bin:$ZOOKEEPER_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

生成新的環(huán)境變量

注意：分發(fā)軟件和配置文件后在所有主機執(zhí)行該步驟

source /etc/profile

2. zookeeper

2.1 編輯配置文件

cd $ZOOKEEPER_HOME/conf

vim zoo.cfg

# 心跳單位，2s
tickTime=2000
# zookeeper-3初始化的同步超時時間，10個心跳單位，也即20s
initLimit=10
# 普通同步：發(fā)送一個請求并得到響應(yīng)的超時時間，5個心跳單位也即10s
syncLimit=5
# 內(nèi)存快照數(shù)據(jù)的存儲位置
dataDir=/home/zookeeper-3/data
# 事務(wù)日志的存儲位置
dataLogDir=/home/zookeeper-3/datalog
# 當(dāng)前zookeeper-3節(jié)點的端口 
clientPort=2181
# 單個客戶端到集群中單個節(jié)點的并發(fā)連接數(shù)，通過ip判斷是否同一個客戶端，默認60
maxClientCnxns=1000
# 保留7個內(nèi)存快照文件在dataDir中，默認保留3個
autopurge.snapRetainCount=7
# 清除快照的定時任務(wù)，默認1小時，如果設(shè)置為0，標識關(guān)閉清除任務(wù)
autopurge.purgeInterval=1
#允許客戶端連接設(shè)置的最小超時時間，默認2個心跳單位
minSessionTimeout=4000
#允許客戶端連接設(shè)置的最大超時時間，默認是20個心跳單位，也即40s，
maxSessionTimeout=300000
#zookeeper-3 3.5.5啟動默認會把AdminService服務(wù)啟動，這個服務(wù)默認是8080端口
admin.serverPort=9001
#集群地址配置
server.1=spark01:2888:3888
server.2=spark02:2888:3888
server.3=spark03:2888:3888

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/home/zookeeper-3/data
dataLogDir=/home/zookeeper-3/datalog 
clientPort=2181
maxClientCnxns=1000
autopurge.snapRetainCount=7
autopurge.purgeInterval=1
minSessionTimeout=4000
maxSessionTimeout=300000
admin.serverPort=9001
server.1=spark01:2888:3888
server.2=spark02:2888:3888
server.3=spark03:2888:3888

2.2 保存后根據(jù)配置文件創(chuàng)建目錄

在每臺服務(wù)器上執(zhí)行

mkdir -p /home/zookeeper-3/data
mkdir -p /home/zookeeper-3/datalog

2.3 myid

spark01

echo 1 > /home/zookeeper-3/data/myid
more /home/zookeeper-3/data/myid

spark02

echo 2 > /home/zookeeper-3/data/myid
more /home/zookeeper-3/data/myid

spark03

echo 3 > /home/zookeeper-3/data/myid
more /home/zookeeper-3/data/myid

2.4 編寫zookeeper-3開機啟動腳本

在/etc/systemd/system/文件夾下創(chuàng)建一個啟動腳本zookeeper-3.service

注意：在每臺服務(wù)器上編寫

cd /etc/systemd/system
vim zookeeper.service

內(nèi)容如下：

[Unit]
Description=zookeeper
After=syslog.target network.target

[Service]
Type=forking
# 指定zookeeper-3 日志文件路徑，也可以在zkServer.sh 中定義
Environment=ZOO_LOG_DIR=/home/zookeeper-3/datalog
# 指定JDK路徑，也可以在zkServer.sh 中定義
Environment=JAVA_HOME=/opt/soft/jdk-8
ExecStart=/opt/soft/zookeeper-3/bin/zkServer.sh start
ExecStop=/opt/soft/zookeeper-3/bin/zkServer.sh stop
Restart=always
User=root
Group=root

[Install]
WantedBy=multi-user.target

[Unit]
Description=zookeeper
After=syslog.target network.target

[Service]
Type=forking
Environment=ZOO_LOG_DIR=/home/zookeeper-3/datalog
Environment=JAVA_HOME=/opt/soft/jdk-8
ExecStart=/opt/soft/zookeeper-3/bin/zkServer.sh start
ExecStop=/opt/soft/zookeeper-3/bin/zkServer.sh stop
Restart=always
User=root
Group=root

[Install]
WantedBy=multi-user.target

systemctl daemon-reload
# 等所有主機配置好后再執(zhí)行以下命令
systemctl start zookeeper
systemctl enable zookeeper
systemctl status zookeeper

3. hadoop

修改配置文件

cd  $HADOOP_HOME/etc/hadoop

hadoop-env.sh

core-site.xml

hdfs-site.xml

workers

mapred-site.xml

yarn-site.xml

hadoop-env.sh 文件末尾追加

export JAVA_HOME=/opt/soft/jdk-8
# export HADOOP_OPTS="--add-opens java.base/java.lang=ALL-UNNAMED"

export HDFS_NAMENODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_ZKFC_USER=root
export HDFS_JOURNALNODE_USER=root
export HADOOP_SHELL_EXECNAME=root

export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root

core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://lihaozhe</value>
  </property>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/home/hadoop/data</value>
  </property>
  <property>
    <name>ha.zookeeper.quorum</name>
    <value>spark01:2181,spark02:2181,spark03:2181</value>
  </property>
  <property>
    <name>hadoop.http.staticuser.user</name>
    <value>root</value>
  </property>
  <property>
    <name>dfs.permissions.enabled</name>
    <value>false</value>
  </property>
  <property>
    <name>hadoop.proxyuser.root.hosts</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.root.groups</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.root.users</name>
    <value>*</value>
  </property>
</configuration>

hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
  <property>
    <name>dfs.nameservices</name>
    <value>lihaozhe</value>
  </property>
  <property>
    <name>dfs.ha.namenodes.lihaozhe</name>
    <value>nn1,nn2</value>
  </property>
  <property>
    <name>dfs.namenode.rpc-address.lihaozhe.nn1</name>
    <value>spark01:8020</value>
  </property>
  <property>
    <name>dfs.namenode.rpc-address.lihaozhe.nn2</name>
    <value>spark02:8020</value>
  </property>
  <property>
    <name>dfs.namenode.http-address.lihaozhe.nn1</name>
    <value>spark01:9870</value>
  </property>
  <property>
    <name>dfs.namenode.http-address.lihaozhe.nn2</name>
    <value>spark02:9870</value>
  </property>
  <property>
    <name>dfs.namenode.shared.edits.dir</name>
    <value>qjournal://spark01:8485;spark02:8485;spark03:8485/lihaozhe</value>
  </property>
  <property>
    <name>dfs.client.failover.proxy.provider.lihaozhe</name>
    <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
  </property>
  <property>
    <name>dfs.ha.fencing.methods</name>
    <value>sshfence</value>
  </property>
  <property>
    <name>dfs.ha.fencing.ssh.private-key-files</name>
    <value>/root/.ssh/id_rsa</value>
  </property>
  <property>
    <name>dfs.journalnode.edits.dir</name>
    <value>/home/hadoop/journalnode/data</value>
  </property>
  <property>
    <name>dfs.ha.automatic-failover.enabled</name>
    <value>true</value>
  </property>
  <property>
    <name>dfs.safemode.threshold.pct</name>
    <value>1</value>
  </property>
</configuration>

workers

spark01
spark02
spark03

mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
  <property>
    <name>mapreduce.application.classpath</name>
    <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
  </property>
  <!-- yarn歷史服務(wù)端口 -->
  <property>
    <name>mapreduce.jobhistory.address</name>
    <value>spark01:10020</value>
  </property>
  <!-- yarn歷史服務(wù)web訪問端口 -->
  <property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>spark01:19888</value>
  </property>
</configuration>

yarn-site.xml

<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>

  <!-- Site specific YARN configuration properties -->
  <property>
    <name>yarn.resourcemanager.ha.enabled</name>
    <value>true</value>
  </property>
  <property>
    <name>yarn.resourcemanager.cluster-id</name>
    <value>cluster1</value>
  </property>
  <property>
    <name>yarn.resourcemanager.ha.rm-ids</name>
    <value>rm1,rm2</value>
  </property>
  <property>
    <name>yarn.resourcemanager.hostname.rm1</name>
    <value>spark01</value>
  </property>
  <property>
    <name>yarn.resourcemanager.hostname.rm2</name>
    <value>spark02</value>
  </property>
  <property>
    <name>yarn.resourcemanager.webapp.address.rm1</name>
    <value>spark01:8088</value>
  </property>
  <property>
    <name>yarn.resourcemanager.webapp.address.rm2</name>
    <value>spark02:8088</value>
  </property>
  <property>
    <name>yarn.resourcemanager.zk-address</name>
    <value>spark01:2181,spark02:2181,spark03:2181</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
  <property>
    <name>yarn.nodemanager.env-whitelist</name>
    <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
  </property>
  <!-- 是否將對容器實施物理內(nèi)存限制 -->
  <property>
    <name>yarn.nodemanager.pmem-check-enabled</name>
    <value>false</value>
  </property>

  <!-- 是否將對容器實施虛擬內(nèi)存限制。 -->
  <property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
  </property>
  <!-- 開啟日志聚集 -->
  <property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
  </property>

  <!-- 設(shè)置yarn歷史服務(wù)器地址 -->
  <property>
    <name>yarn.log.server.url</name>
    <value>http://spark01:19888/jobhistory/logs</value>
  </property>

  <!-- 保存的時間7天 -->
  <property>
    <name>yarn.log-aggregation.retain-seconds</name>
    <value>604800</value>
  </property>
</configuration>

4. 配置ssh免密鑰登錄

創(chuàng)建本地秘鑰并將公共秘鑰寫入認證文件

ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

ssh-copy-id root@spark01

ssh-copy-id root@spark02

ssh-copy-id root@spark03

ssh root@spark01
exit

ssh root@spark02
exit

ssh root@spark03
exit

5. 分發(fā)軟件和配置文件

scp -r /etc/profile.d root@spark02:/etc
scp -r /etc/profile.d root@spark03:/etc
scp -r /opt/soft/zookeeper-3 root@spark02:/opt/soft
scp -r /opt/soft/zookeeper-3 root@spark03:/opt/soft
scp -r /opt/soft/hadoop-3/etc/hadoop/* root@spark02:/opt/soft/hadoop-3/etc/hadoop/
scp -r /opt/soft/hadoop-3/etc/hadoop/* root@spark03:/opt/soft/hadoop-3/etc/hadoop/

6. 在各服務(wù)器上使環(huán)境變量生效

source /etc/profile

7. 啟動zookeeper

7.1 myid

spark01

echo 1 > /home/zookeeper-3/data/myid
more /home/zookeeper-3/data/myid

spark02

echo 2 > /home/zookeeper-3/data/myid
more /home/zookeeper-3/data/myid

spark03

echo 3 > /home/zookeeper-3/data/myid
more /home/zookeeper-3/data/myid

7.2 啟動服務(wù)

在各節(jié)點執(zhí)行以下命令

systemctl daemon-reload
systemctl start zookeeper
systemctl enable zookeeper
systemctl status zookeeper

7.3 驗證

jps

zkServer.sh status

8. Hadoop初始化

1.	啟動三個zookeeper：zkServer.sh start
2.	啟動三個JournalNode：
	hadoop-daemon.sh start journalnode 或者 hdfs --daemon start journalnode
3.	在其中一個namenode上格式化：hdfs namenode -format
4.	把剛剛格式化之后的元數(shù)據(jù)拷貝到另外一個namenode上
    a)	啟動剛剛格式化的namenode :
    	hadoop-daemon.sh start namenode 或者 hdfs --daemon start namenode
    b)	在沒有格式化的namenode上執(zhí)行：hdfs namenode -bootstrapStandby
    c)	啟動第二個namenode: 
    	hadoop-daemon.sh start namenode 或者 hdfs --daemon start namenode
5.	在其中一個namenode上初始化 hdfs zkfc -formatZK
6.	停止上面節(jié)點：stop-dfs.sh
7.	全面啟動：start-all.sh
8. 啟動resourcemanager節(jié)點 
	yarn-daemon.sh start resourcemanager 或者	start-yarn.sh

http://dl.bintray.com/sequenceiq/sequenceiq-bin/hadoop-native-64-2.5.0.tar

不需要執(zhí)行第 8 步

9. 啟動歷史服務(wù)
mapred --daemon start historyserver
10 11 12 不需要執(zhí)行
10、安全模式

hdfs dfsadmin -safemode enter  
hdfs dfsadmin -safemode leave


11、查看哪些節(jié)點是namenodes并獲取其狀態(tài)
hdfs getconf -namenodes
hdfs haadmin -getServiceState nn1
hdfs haadmin -getServiceState nn2

12、強制切換狀態(tài)
hdfs haadmin -transitionToActive --forcemanual spark01

重點提示：

# 關(guān)機之前 依關(guān)閉服務(wù)
stop-yarn.sh
stop-dfs.sh
# 開機后 依次開啟服務(wù)
start-dfs.sh
start-yarn.sh

或者

# 關(guān)機之前關(guān)閉服務(wù)
stop-all.sh
# 開機后開啟服務(wù)
start-all.sh

#jps 檢查進程正常后開啟胡哦關(guān)閉在再做其它操作

9. 修改windows下hosts文件

C:\Windows\System32\drivers\etc\hosts

追加以下內(nèi)容：

192.168.171.101	hadoop101
192.168.171.102	hadoop102
192.168.171.103	hadoop103

Windows11 注意修改權(quán)限

開始搜索 cmd

找到命令頭提示符以管理身份運行
進入 C:\Windows\System32\drivers\etc 目錄
```
cd drivers/etc
```
去掉 hosts文件只讀屬性
```
attrib -r hosts
```
打開 hosts 配置文件
```
start hosts
```

追加以下內(nèi)容后保存

192.168.171.101	spark01
192.168.171.102	spark02
192.168.171.103	spark03

10. 測試

12.1 瀏覽器訪問hadoop集群

瀏覽器訪問: http://spark01:9870

CentOS 搭建 Hadoop3 高可用集群,Hadoop,大數(shù)據(jù),數(shù)據(jù)分析,centos,linux,運維,大數(shù)據(jù),hadoop

瀏覽器訪問:http://spark01:8088

CentOS 搭建 Hadoop3 高可用集群,Hadoop,大數(shù)據(jù),數(shù)據(jù)分析,centos,linux,運維,大數(shù)據(jù),hadoop

瀏覽器訪問:http://spark01:19888/

CentOS 搭建 Hadoop3 高可用集群,Hadoop,大數(shù)據(jù),數(shù)據(jù)分析,centos,linux,運維,大數(shù)據(jù),hadoop

12.2 測試 hdfs

本地文件系統(tǒng)創(chuàng)建測試文件 wcdata.txt

vim wcdata.txt

Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive
FlinkHBase Flink
Hive StormHive Flink HadoopHBase
HiveHadoop Spark HBase StormHBase
Hadoop Hive FlinkHBase Flink Hive StormHive
Flink HadoopHBase Hive
Spark HBaseHive Flink
Storm Hadoop HBase SparkFlinkHBase
StormHBase Hadoop Hive

在 HDFS 上創(chuàng)建目錄 /wordcount/input

hdfs dfs -mkdir -p /wordcount/input

查看 HDFS 目錄結(jié)構(gòu)

hdfs dfs -ls /

hdfs dfs -ls /wordcount

hdfs dfs -ls /wordcount/input

上傳本地測試文件 wcdata.txt 到 HDFS 上 /wordcount/input

hdfs dfs -put wcdata.txt /wordcount/input

檢查文件是否上傳成功

hdfs dfs -ls /wordcount/input

hdfs dfs -cat /wordcount/input/wcdata.txt

12.2 測試 mapreduce

計算 PI 的值

hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.5.jar pi 10 10

單詞統(tǒng)計

hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.5.jar wordcount /wordcount/input/wcdata.txt /wordcount/result

hdfs dfs -ls /wordcount/result

hdfs dfs -cat /wordcount/result/part-r-00000

11. 元數(shù)據(jù)

hadoop101

cd /home/hadoop_data/dfs/name/current

ls

看到如下內(nèi)容:

edits_0000000000000000001-0000000000000000009  edits_inprogress_0000000000000000299  fsimage_0000000000000000298      VERSION
edits_0000000000000000010-0000000000000000011  fsimage_0000000000000000011           fsimage_0000000000000000298.md5
edits_0000000000000000012-0000000000000000298  fsimage_0000000000000000011.md5       seen_txid

查看fsimage

hdfs oiv -p XML -i fsimage_0000000000000000011

將元數(shù)據(jù)內(nèi)容按照指定格式讀取后寫入到新文件中文章來源地址http://www.zghlxwxcb.cn/news/detail-738052.html

hdfs oiv -p XML -i fsimage_0000000000000000011 -o /opt/soft/fsimage.xml

查看edits

將元數(shù)據(jù)內(nèi)容按照指定格式讀取后寫入到新文件中

hdfs oev -p XML -i edits_inprogress_0000000000000000299  -o /opt/soft/edit.xml

到了這里，關(guān)于CentOS 搭建 Hadoop3 高可用集群的文章就介紹完了。如果您還想了解更多內(nèi)容，請在右上角搜索TOY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！

CentOS 搭建 Hadoop3 高可用集群

Hadoop FullyDistributed Mode 完全分布式

1. 準備

1.1 升級操作系統(tǒng)和軟件

1.2 安裝常用軟件

1.3 修改主機名

1.4 修改IP地址

1.5 關(guān)閉防火墻

1.6 修改hosts配置文件

1.7 上傳軟件配置環(huán)境變量

2. zookeeper

2.1 編輯配置文件

2.2 保存后根據(jù)配置文件創(chuàng)建目錄

2.3 myid

2.4 編寫zookeeper-3開機啟動腳本

3. hadoop

hadoop-env.sh 文件末尾追加

core-site.xml

hdfs-site.xml

workers

mapred-site.xml

yarn-site.xml

4. 配置ssh免密鑰登錄

5. 分發(fā)軟件和配置文件

6. 在各服務(wù)器上使環(huán)境變量生效

7. 啟動zookeeper

7.1 myid

7.2 啟動服務(wù)

7.3 驗證

8. Hadoop初始化

重點提示：

9. 修改windows下hosts文件

10. 測試

12.1 瀏覽器訪問hadoop集群

12.2 測試 hdfs

12.2 測試 mapreduce

11. 元數(shù)據(jù)

相關(guān)文章

覺得文章有用就打賞一下文章作者

支付寶掃一掃打賞

微信掃一掃打賞

支付寶掃一掃領(lǐng)取紅包，優(yōu)惠每天領(lǐng)

二維碼1

二維碼2

支付寶掃一掃領(lǐng)取紅包，優(yōu)惠每天領(lǐng)