實(shí)驗(yàn)一、Flume 組件安裝配置
1、下載和解壓 Flume
可 以 從 官 網(wǎng) 下 載 Flume 組 件 安 裝 包 ， 下 載 地 址 如 下 URL 鏈 接 所 示 https://archive.apache.org/dist/flume/1.6.0/
[root@master ~]# ls
anaconda-ks.cfg ? ? ? ? ? ? ? jdk-8u152-linux-x64.tar.gz
apache-flume-1.6.0-bin.tar.gz mysql
apache-hive-2.0.0-bin.tar.gz ? mysql-connector-java-5.1.46.jar
derby.log ? ? ? ? ? ? ? ? ? ? sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz
hadoop-2.7.1.tar.gz ? ? ? ? ? zookeeper-3.4.8.tar.gz
hbase-1.2.1-bin.tar.gz
#使用 root 用戶解壓 Flume 安裝包到“/usr/local/src”路徑，并修改解壓后文件夾名為 flume。
[root@master ~]# tar xf apache-flume-1.6.0-bin.tar.gz -C /usr/local/src/
[root@master ~]# cd /usr/local/src/
[root@master src]# ls
apache-flume-1.6.0-bin hadoop hbase hive jdk sqoop zookeeper
[root@master src]# mv apache-flume-1.6.0-bin flume
[root@master src]# ls
flume hadoop hbase hive jdk sqoop zookeeper
2、Flume 組件部署
步驟一：使用 root 用戶設(shè)置 Flume 環(huán)境變量，并使環(huán)境變量對(duì)所有用戶生效。
[root@master ~]# vim /etc/profile.d/flume.sh
export FLUME_HOME=/usr/local/src/flume
export PATH=${FLUME_HOME}/bin:$PATH
步驟二：修改 Flume 相應(yīng)配置文件。
首先，切換到 hadoop 用戶，并切換當(dāng)前工作目錄到 Flume 的配置文件夾。
[root@master ~]# chown -R hadoop.hadoop /usr/local/src/
[root@master ~]# su - hadoop
Last login: Fri Apr 14 16:31:48 CST 2023 on pts/1
[hadoop@master ~]$ ls
derby.log input student.java zookeeper.out
[hadoop@master ~]$ cd /usr/local/src/flume/conf/
[hadoop@master conf]$ ls
flume-conf.properties.template flume-env.sh.template
flume-env.ps1.template ? ? ? ? log4j.properties
拷貝 flume-env.sh.template 文件并重命名為 flume-env.sh。
[hadoop@master conf]$ cp flume-env.sh.template flume-env.sh
[hadoop@master conf]$ ls
flume-conf.properties.template flume-env.sh ? ? ? ? ? log4j.properties
flume-env.ps1.template ? ? ? ? flume-env.sh.template
步驟三：修改并配置 flume-env.sh 文件。
刪除 JAVA_HOME 變量前的注釋，修改為 JDK 的安裝路徑。
[hadoop@master conf]$ vi flume-env.sh
export JAVA_HOME=/usr/loocal/src/jdk
#export HBASE_CLASSPATH=/usr/local/src/hadoop/etc/hadoop
使用 flume-ng version 命令驗(yàn)證安裝是否成功，若能夠正常查詢 Flume 組件版本為 1.6.0，則表示安裝成功。
[hadoop@master conf]$ flume-ng version
Error: Could not find or load main class org.apache.flume.tools.GetJavaProperty
Flume 1.6.0
Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
Revision: 2561a23240a71ba20bf288c7c2cda88f443c2080
Compiled by hshreedharan on Mon May 11 11:15:44 PDT 2015
From source with checksum b29e416802ce9ece3269d34233baf43f
3、使用 Flume 發(fā)送和接受信息
通過(guò) Flume 將 Web 服務(wù)器中數(shù)據(jù)傳輸?shù)?HDFS 中。
步驟一：在 Flume 安裝目錄中創(chuàng)建 xxx.conf 文件。
[hadoop@master flume]$ vi xxx.conf
a1.sources=r1
a1.sinks=k1
a1.channels=c1
a1.sources.r1.type=spooldir
a1.sources.r1.spoolDir=/usr/local/src/hadoop/logs
a1.sources.r1.fileHeader=true
a1.sinks.k1.type=hdfs
a1.sinks.k1.hdfs.path=hdfs://master:9000/tmp/flume
a1.sinks.k1.hdfs.rollsize=1048760
a1.sinks.k1.hdfs.rollCount=0
a1.sinks.k1.hdfs.rollInterval=900
a1.sinks.k1.hfds.useLocalTimeStamp=true
a1.channels.c1.type=file
a1.channels.c1.capacity=1000
a1.channels.c1.transactionCapacity=100
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
[hadoop@master flume]$ ls /usr/local/src/hadoop/
bin etc ? ? lib ? ? LICENSE.txt NOTICE.txt sbin ? tmp
dfs include libexec logs ? ? ? ? README.txt share
[hadoop@master flume]$ ls /usr/local/src/hadoop/logs/
hadoop-hadoop-namenode-master.example.com.log
hadoop-hadoop-namenode-master.example.com.out
hadoop-hadoop-namenode-master.example.com.out.1
hadoop-hadoop-namenode-master.example.com.out.2
hadoop-hadoop-namenode-master.example.com.out.3
hadoop-hadoop-namenode-master.example.com.out.4
hadoop-hadoop-namenode-master.example.com.out.5
hadoop-hadoop-secondarynamenode-master.example.com.log
hadoop-hadoop-secondarynamenode-master.example.com.out
hadoop-hadoop-secondarynamenode-master.example.com.out.1
hadoop-hadoop-secondarynamenode-master.example.com.out.2
hadoop-hadoop-secondarynamenode-master.example.com.out.3
hadoop-hadoop-secondarynamenode-master.example.com.out.4
hadoop-hadoop-secondarynamenode-master.example.com.out.5
SecurityAuth-hadoop.audit
yarn-hadoop-resourcemanager-master.example.com.log
yarn-hadoop-resourcemanager-master.example.com.out
yarn-hadoop-resourcemanager-master.example.com.out.1
yarn-hadoop-resourcemanager-master.example.com.out.2
yarn-hadoop-resourcemanager-master.example.com.out.3
yarn-hadoop-resourcemanager-master.example.com.out.4
yarn-hadoop-resourcemanager-master.example.com.out.5
步驟二：使用 flume-ng agent 命令加載 simple-hdfs-flume.conf 配置信息，啟 動(dòng) flume 傳輸數(shù)據(jù)。
[[hadoop@master flume]$ flume-ng agent --conf-file xxx.conf --name a1
Warning: No configuration directory set! Use --conf <dir> to override.
Info: Including Hadoop libraries found via (/usr/local/src/hadoop/bin/hadoop) for HDFS access
Info: Excluding /usr/local/src/hadoop/share/hadoop/common/lib/slf4j-api-1.7.10.jar from classpath
Info: Excluding /usr/local/src/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar from classpath
Info: Including HBASE libraries found via (/usr/local/src/hbase/bin/hbase) for HBASE access
Info: Excluding /usr/local/src/hbase/lib/slf4j-api-1.7.7.jar from classpath
Info: Excluding /usr/local/src/hbase/lib/slf4j-log4j12-1.7.5.jar from classpath
Info: Excluding /usr/local/src/hadoop/share/hadoop/common/lib/slf4j-api-1.7.10.jar from classpath
Info: Excluding /usr/local/src/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar from classpath
Info: Including Hive libraries found via (/usr/local/src/hive) for Hive access
...
23/04/21 16:02:35 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SOURCE, name: r1. src.append.accepted == 0
23/04/21 16:02:35 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SOURCE, name: r1. src.append.received == 0
23/04/21 16:02:35 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SOURCE, name: r1. src.events.accepted == 17
23/04/21 16:02:35 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SOURCE, name: r1. src.events.received == 17
23/04/21 16:02:35 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SOURCE, name: r1. src.open-connection.count == 0
23/04/21 16:02:35 INFO source.SpoolDirectorySource: SpoolDir source r1 stopped. Metrics: SOURCE:r1{src.events.accepted=17, src.open-connection.count=0, src.append.received=0, src.append-batch.received=1, src.append-batch.accepted=1, src.append.accepted=0, src.events.received=17}
ctrl+c 退出 flume 傳輸
#首先得開(kāi)啟所有節(jié)點(diǎn)
[hadoop@master flume]$ start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [master]
master: starting namenode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-namenode-master.example.com.out
192.168.88.201: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-slave2.example.com.out
192.168.88.200: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-slave1.example.com.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-secondarynamenode-master.example.com.out
starting yarn daemons
starting resourcemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-resourcemanager-master.example.com.out
192.168.88.200: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-nodemanager-slave1.example.com.out
192.168.88.201: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-nodemanager-slave2.example.com.out
[hadoop@master flume]$ ss -antl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 192.168.88.101:9000 *:*
LISTEN 0 128 *:50090 *:*
LISTEN 0 128 *:50070 *:*
LISTEN 0 128 *:22 *:*
LISTEN 0 128 ::ffff:192.168.88.101:8030 :::*
LISTEN 0 128 ::ffff:192.168.88.101:8031 :::*
LISTEN 0 128 ::ffff:192.168.88.101:8032 :::*
LISTEN 0 128 ::ffff:192.168.88.101:8033 :::*
LISTEN 0 80 :::3306 :::*
LISTEN 0 128 :::22 :::*
LISTEN 0 128 ::ffff:192.168.88.101:8088 :::*
步驟三：查看 Flume 傳輸?shù)?HDFS 的文件，若能查看到 HDFS 上/tmp/flume 目 錄有傳輸?shù)臄?shù)據(jù)文件，則表示數(shù)據(jù)傳輸成功。
[hadoop@master flume]$ hdfs dfs -ls /
Found 5 items
drwxr-xr-x - hadoop supergroup 0 2023-04-07 15:20 /hbase
drwxr-xr-x - hadoop supergroup 0 2023-03-17 17:33 /input
drwxr-xr-x - hadoop supergroup 0 2023-03-17 18:45 /output
drwx------ - hadoop supergroup 0 2023-04-21 16:02 /tmp
drwxr-xr-x - hadoop supergroup 0 2023-04-14 20:48 /user
[hadoop@master flume]$ hdfs dfs -ls /tmp/flume
Found 72 items
-rw-r--r-- 2 hadoop supergroup 1560 2023-04-21 16:02 /tmp/flume/FlumeData.1682064143693
-rw-r--r-- 2 hadoop supergroup 1398 2023-04-21 16:02 /tmp/flume/FlumeData.1682064143694
-rw-r--r-- 2 hadoop supergroup 1456 2023-04-21 16:02 /tmp/flume/FlumeData.1682064143695
-rw-r--r-- 2 hadoop supergroup 1398 2023-04-21 16:02 /tmp/flume/FlumeData.1682064143696
-rw-r--r-- 2 hadoop supergroup 1403 2023-04-21 16:02 /tmp/flume/FlumeData.1682064143697
-rw-r--r-- 2 hadoop supergroup 1434 2023-04-21 16:02 /tmp/flume/FlumeData.1682064143698
-rw-r--r-- 2 hadoop supergroup 1383 2023-04-21 16:02 /tmp/flume/FlumeData.1682064143699
...
-rw-r--r-- 2 hadoop supergroup 1508 2023-04-21 16:02 /tmp/flume/FlumeData.1682064143760
-rw-r--r-- 2 hadoop supergroup 1361 2023-04-21 16:02 /tmp/flume/FlumeData.1682064143761
-rw-r--r-- 2 hadoop supergroup 1359 2023-04-21 16:02 /tmp/flume/FlumeData.1682064143762
-rw-r--r-- 2 hadoop supergroup 1502 2023-04-21 16:02 /tmp/flume/FlumeData.1682064143763
-rw-r--r-- 2 hadoop supergroup 1399 2023-04-21 16:02 /tmp/flume/FlumeData.1682064143764
實(shí)驗(yàn)二、通過(guò)命令監(jiān)控大數(shù)據(jù)平臺(tái)運(yùn)行狀態(tài)
1、通過(guò)命令查看大數(shù)據(jù)平臺(tái)狀態(tài)
步驟一：查看 Linux 系統(tǒng)的信息（uname -a）
[root@master ~]# uname -a
Linux master.example.com 3.10.0-862.el7.x86_64 #1 SMP Fri Apr 20 16:44:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
結(jié)果顯示，該 Linux 節(jié)點(diǎn)名稱為 master，內(nèi)核發(fā)行號(hào)為 3.10.0-693.el7.x86_64。
步驟二：查看硬盤信息
（1）查看所有分區(qū)（fdisk -l）
[root@master ~]# fdisk -l
Disk /dev/sda: 107.4 GB, 107374182400 bytes, 209715200 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000af885
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 2099199 1048576 83 Linux
/dev/sda2 2099200 209715199 103808000 8e Linux LVM
Disk /dev/mapper/centos-root: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/centos-swap: 8455 MB, 8455716864 bytes, 16515072 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/centos-home: 44.1 GB, 44149243904 bytes, 86228992 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
結(jié)果顯示，硬盤空間為 107.4GB。
（2）查看所有交換分區(qū)（swapon -s）
[root@master ~]# swapon -s
Filename Type Size Used Priority
/dev/dm-1 partition 8257532 0 -1
結(jié)果顯示，交換分區(qū)為 8257532。
（3）查看文件系統(tǒng)占比（df -h）
[root@master ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 50G 4.6G 46G 10% /
devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 1.9G 12M 1.9G 1% /run
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/mapper/centos-home 42G 36M 42G 1% /home
/dev/sda1 1014M 142M 873M 14% /boot
tmpfs 378M 0 378M 0% /run/user/0
結(jié)果顯示，掛載點(diǎn)“/”的容量為 50G，已使用 4.6G。
步驟三：查看網(wǎng)絡(luò) IP 地址（ip a）
[root@master ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:20:cd:03 brd ff:ff:ff:ff:ff:ff
inet 192.168.88.101/24 brd 192.168.88.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet6 fe80::75a3:9da2:ede2:5be7/64 scope link noprefixroute
valid_lft forever preferred_lft forever
inet6 fe80::1c1b:d0e3:f01a:7c11/64 scope link tentative noprefixroute dadfailed
valid_lft forever preferred_lft forever
結(jié)果顯示 ens33 的 IP 地址為 192.168.88.101，子網(wǎng)掩碼為 255.255.255.0；回環(huán)地址 為 127.0.0.1，子網(wǎng)掩碼為 255.0.0.0。
步驟四：查看所有監(jiān)聽(tīng)端口（netstat -lntp）
[root@master ~]# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 908/sshd
tcp6 0 0 :::3306 :::* LISTEN 1053/mysqld
tcp6 0 0 :::22 :::* LISTEN 908/sshd
結(jié)果顯示，在監(jiān)聽(tīng)的端口分別為 22、3306。
步驟五：查看所有已經(jīng)建立的連接（netstat -antp）
[root@master ~]# netstat -antp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 908/sshd
tcp 0 36 192.168.88.101:22 192.168.88.1:61450 ESTABLISHED 1407/sshd: root@pts
tcp6 0 0 :::3306 :::* LISTEN 1053/mysqld
tcp6 0 0 :::22 :::* LISTEN 908/sshd
結(jié)果顯示，已經(jīng)連接上的本地端口分別為 50608、22、9000、8031 等。
步驟六：實(shí)時(shí)顯示進(jìn)程狀態(tài)（top），該命令可以查看進(jìn)程對(duì) CPU、內(nèi)存的占比 等。
步驟七：查看 CPU 信息（ cat /proc/cpuinfo）
[root@master ~]# cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 25
model : 80
model name : AMD Ryzen 7 5800U with Radeon Graphics
stepping : 0
cpu MHz : 1896.438
cache size : 512 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 16
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl tsc_reliable nonstop_tsc extd_apicid eagerfpu pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext retpoline_amd vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero ibpb ibrs arat pku ospke overflow_recov succor
bogomips : 3792.87
TLB size : 2560 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 45 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : AuthenticAMD
cpu family : 25
model : 80
model name : AMD Ryzen 7 5800U with Radeon Graphics
stepping : 0
cpu MHz : 1896.438
cache size : 512 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 16
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl tsc_reliable nonstop_tsc extd_apicid eagerfpu pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext retpoline_amd vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero ibpb ibrs arat pku ospke overflow_recov succor
bogomips : 3792.87
TLB size : 2560 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 45 bits physical, 48 bits virtual
步驟八：查看內(nèi)存信息（ cat /proc/meminfo），該命令可以查看總內(nèi)存、空 閑內(nèi)存等信息。
[root@master ~]# cat /proc/meminfo
MemTotal: 3863564 kB
MemFree: 2971196 kB
MemAvailable: 3301828 kB
Buffers: 2120 kB
Cached: 529608 kB
SwapCached: 0 kB
Active: 626584 kB
Inactive: 130828 kB
Active(anon): 226348 kB
Inactive(anon): 11152 kB
Active(file): 400236 kB
Inactive(file): 119676 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 8257532 kB
SwapFree: 8257532 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 225716 kB
Mapped: 28960 kB
Shmem: 11816 kB
Slab: 59428 kB
SReclaimable: 30680 kB
SUnreclaim: 28748 kB
KernelStack: 4432 kB
PageTables: 3664 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 10189312 kB
Committed_AS: 773684 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 185376 kB
VmallocChunk: 34359310332 kB
HardwareCorrupted: 0 kB
AnonHugePages: 180224 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 73536 kB
DirectMap2M: 3072000 kB
DirectMap1G: 3145728 kB
2、通過(guò)命令查看 Hadoop 狀態(tài)
步驟一：切換到 hadoop 用戶
[root@master ~]# su - hadoop
Last login: Fri Apr 21 15:26:05 CST 2023 on pts/0
Last failed login: Fri Apr 21 16:02:08 CST 2023 from slave1 on ssh:notty
There were 4 failed login attempts since the last successful login.
若當(dāng)前的用戶為 root，請(qǐng)切換到 hadoop 用戶進(jìn)行操作。
步驟二：切換到 Hadoop 的安裝目錄
[hadoop@master ~]$ cd /usr/local/src/hadoop/
步驟三：?jiǎn)?dòng) Hadoop
[hadoop@master ~]$ start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [master]
master: starting namenode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-namenode-master.example.com.out
192.168.88.201: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-slave2.example.com.out
192.168.88.200: starting datanode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-datanode-slave1.example.com.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/local/src/hadoop/logs/hadoop-hadoop-secondarynamenode-master.example.com.out
starting yarn daemons
starting resourcemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-resourcemanager-master.example.com.out
192.168.88.200: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-nodemanager-slave1.example.com.out
192.168.88.201: starting nodemanager, logging to /usr/local/src/hadoop/logs/yarn-hadoop-nodemanager-slave2.example.com.out
步驟四：關(guān)閉 Hadoop
[hadoop@master hadoop]$ stop-all.sh
This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh
Stopping namenodes on [master]
master: stopping namenode
192.168.88.201: stopping datanode
192.168.88.200: stopping datanode
Stopping secondary namenodes [0.0.0.0]
0.0.0.0: stopping secondarynamenode
stopping yarn daemons
stopping resourcemanager
192.168.88.200: stopping nodemanager
192.168.88.201: stopping nodemanager
no proxyserver to stop
實(shí)驗(yàn)三、通過(guò)命令監(jiān)控大數(shù)據(jù)平臺(tái)資源狀態(tài)
1、通過(guò)命令查看 YARN 狀態(tài)
步驟一：確認(rèn)切換到目錄 /usr/local/src/hadoop
[hadoop@master hadoop]$ cd /usr/local/src/hadoop/
步驟二：返回主機(jī)界面在 Master 主機(jī)上執(zhí)行 start-all.sh
#master 節(jié)點(diǎn)啟動(dòng) zookeeper
[hadoop@master hadoop]$ zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /usr/local/src/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
#slave1 節(jié)點(diǎn)啟動(dòng) zookeeper
[root@slave1 ~]# zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /usr/local/src/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
#slave2 節(jié)點(diǎn)啟動(dòng) zookeeper
[root@slave2 ~]# zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /usr/local/src/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
master節(jié)點(diǎn)
[hadoop@master hadoop]$ start-all.sh
步 驟 三 ： 執(zhí) 行 JPS 命 令 ， 發(fā) 現(xiàn) Master 上 有 NodeManager 進(jìn)程和 ResourceManager 進(jìn)程，則 YARN 啟動(dòng)完成。
[hadoop@master hadoop]$ jps
執(zhí)行結(jié)果如下，說(shuō)明 YARN 已啟動(dòng)。
[hadoop@master hadoop]$ jps
2642 QuorumPeerMain
2994 SecondaryNameNode
3154 ResourceManager
3413 Jps
2795 NameNode
2、通過(guò)命令查看 HDFS 狀態(tài)
步驟一：目錄操作
切換到 hadoop 目錄，執(zhí)行 cd /usr/local/src/hadoop 命令
[hadoop@master hadoop]$ cd /usr/local/src/hadoop/
查看 HDFS 目錄
[hadoop@master hadoop]$ ./bin/hdfs dfs -ls /
Found 5 items
drwxr-xr-x - hadoop supergroup 0 2023-04-07 15:20 /hbase
drwxr-xr-x - hadoop supergroup 0 2023-03-17 17:33 /input
drwxr-xr-x - hadoop supergroup 0 2023-03-17 18:45 /output
drwx------ - hadoop supergroup 0 2023-04-21 16:02 /tmp
drwxr-xr-x - hadoop supergroup 0 2023-04-14 20:48 /user
步驟二：查看 HDSF 的報(bào)告，執(zhí)行命令： bin/hdfs dfsadmin -report
[hadoop@master hadoop]$ bin/hdfs dfsadmin -report
Configured Capacity: 107321753600 (99.95 GB)
Present Capacity: 102855995392 (95.79 GB)
DFS Remaining: 102852079616 (95.79 GB)
DFS Used: 3915776 (3.73 MB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
-------------------------------------------------
Live datanodes (2):
Name: 192.168.88.201:50010 (slave2)
Hostname: slave2
Decommission Status : Normal
Configured Capacity: 53660876800 (49.98 GB)
DFS Used: 1957888 (1.87 MB)
Non DFS Used: 2232373248 (2.08 GB)
DFS Remaining: 51426545664 (47.89 GB)
DFS Used%: 0.00%
DFS Remaining%: 95.84%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Sun Apr 23 21:56:46 CST 2023
Name: 192.168.88.200:50010 (slave1)
Hostname: slave1
Decommission Status : Normal
Configured Capacity: 53660876800 (49.98 GB)
DFS Used: 1957888 (1.87 MB)
Non DFS Used: 2233384960 (2.08 GB)
DFS Remaining: 51425533952 (47.89 GB)
DFS Used%: 0.00%
DFS Remaining%: 95.83%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Sun Apr 23 21:56:46 CST 2023
步驟三：查看 HDFS 空間情況，執(zhí)行命令：hdfs dfs -df
[hadoop@master hadoop]$ hdfs dfs -df /
Filesystem Size Used Available Use%
hdfs://master:9000 107321753600 3915776 102852079616 0%
3、通過(guò)命令查看 HBase 狀態(tài)
步驟一：?jiǎn)?dòng)運(yùn)行 HBase
切換到 HBase 安裝目錄/usr/local/src/hbase，命令如下：
[hadoop@master hadoop]$ cd /usr/local/src/hbase/
[hadoop@master hbase]$ hbase version
HBase 1.2.1
Source code repository git://asf-dev/home/busbey/projects/hbase revision=8d8a7107dc4ccbf36a92f64675dc60392f85c015
Compiled by busbey on Wed Mar 30 11:19:21 CDT 2016
From source with checksum f4bb4a14bb4e0b72b46f729dae98a772
結(jié)果顯示 HBase1.2.1，說(shuō)明 HBase 正在運(yùn)行，版本號(hào)為 1.2.1。
如果沒(méi)有啟動(dòng)，則執(zhí)行命令 start-hbase.sh 啟動(dòng) HBase。
[hadoop@master hbase]$ start-hbase.sh
master: starting zookeeper, logging to /usr/local/src/hbase/logs/hbase-hadoop-zookeeper-master.example.com.out
slave1: starting zookeeper, logging to /usr/local/src/hbase/logs/hbase-hadoop-zookeeper-slave1.example.com.out
slave2: starting zookeeper, logging to /usr/local/src/hbase/logs/hbase-hadoop-zookeeper-slave2.example.com.out
starting master, logging to /usr/local/src/hbase/logs/hbase-hadoop-master-master.example.com.out
slave2: starting regionserver, logging to /usr/local/src/hbase/logs/hbase-hadoop-regionserver-slave2.example.com.out
slave1: starting regionserver, logging to /usr/local/src/hbase/logs/hbase-hadoop-regionserver-slave1.example.com.out
步驟二：查看 HBase 版本信息
執(zhí)行命令hbase shell，進(jìn)入HBase命令交互界面
[hadoop@master hbase]$ hbase shell
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/src/hbase/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/src/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 1.2.1, r8d8a7107dc4ccbf36a92f64675dc60392f85c015, Wed Mar 30 11:19:21 CDT 2016
hbase(main):001:0>
輸入 version，查詢 HBase 版本
hbase(main):001:0> version
1.2.1, r8d8a7107dc4ccbf36a92f64675dc60392f85c015, Wed Mar 30 11:19:21 CDT 2016
結(jié)果顯示 HBase 版本為 1.2.1。
步驟三：查詢 HBase 狀態(tài)，在 HBase 命令交互界面，執(zhí)行 status 命令
hbase(main):002:0> status
1 active master, 0 backup masters, 2 servers, 0 dead, 1.0000 average load
查詢結(jié)果顯示，1 臺(tái)活動(dòng) master，0 臺(tái)備份 masters，共 3 臺(tái)服務(wù)主機(jī)，平均加載 時(shí)間為 0.6667 秒。
我們還可以“簡(jiǎn)單”查詢 HBase 的狀態(tài)，執(zhí)行命令 status 'simple'
hbase(main):003:0> status 'simple'
active master: master:16000 1682258321433
0 backup masters
2 live servers
slave1:16020 1682258322908
requestsPerSecond=0.0, numberOfOnlineRegions=2, usedHeapMB=19, maxHeapMB=440, numberOfStores=2, numberOfStorefiles=0, storefileUncompressedSizeMB=0, storefileSizeMB=0, memstoreSizeMB=0, storefileIndexSizeMB=0, readRequestsCount=7, writeRequestsCount=4, rootIndexSizeKB=0 totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0, totalCompactingKVs=0, currentCompactedKVs=0, compactionProgressPct=NaN, coprocessors=[MultiRowMutationEndpoint]
slave2:16020 1682258322896
requestsPerSecond=0.0, numberOfOnlineRegions=0, usedHeapMB=11, maxHeapMB=440, numberOfStores=0, numberOfStorefiles=0, storefileUncompressedSizeMB=0, storefileSizeMB=0, memstoreSizeMB=0, storefileIndexSizeMB=0, readRequestsCount=0, writeRequestsCount=0, rootIndexSizeKB=0, totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0, totalCompactingKVs=0, currentCompactedKVs=0, compactionProgressPct=NaN, coprocessors=[]
0 dead servers
Aggregate load: 0, regions: 2
顯示更多的關(guān)于 Master、Slave1 和 Slave2 主機(jī)的服務(wù)端口、請(qǐng)求時(shí)間等詳細(xì)信息。
如果需要查詢更多關(guān)于 HBase 狀態(tài)，執(zhí)行命令 help 'status'
hbase(main):004:0> help 'status'
Show cluster status. Can be 'summary', 'simple', 'detailed', or 'replication'. The
default is 'summary'. Examples:
hbase> status
hbase> status 'simple'
hbase> status 'summary'
hbase> status 'detailed'
hbase> status 'replication'
hbase> status 'replication', 'source'
hbase> status 'replication', 'sink'
hbase(main):005:0> quit
[hadoop@master hbase]$
結(jié)果顯示出所有關(guān)于 status 的命令。
步驟四 停止 HBase 服務(wù)
停止 HBase 服務(wù)，則執(zhí)行命令 stop-hbase.sh。
[hadoop@master hbase]$ stop-hbase.sh
stopping hbase.................
slave1: no zookeeper to stop because no pid file /tmp/hbase-hadoop-zookeeper.pid
slave2: no zookeeper to stop because no pid file /tmp/hbase-hadoop-zookeeper.pid
master: no zookeeper to stop because no pid file /tmp/hbase-hadoop-zookeeper.pid
沒(méi)有錯(cuò)誤提示，顯示$提示符時(shí)，即停止了 HBase 服務(wù)。
4、通過(guò)命令查看 Hive 狀態(tài)
步驟一：?jiǎn)?dòng) Hive
切換到/usr/local/src/hive 目錄，輸入 hive，回車。
[hadoop@master hbase]$ cd /usr/local/src/hive/
[hadoop@master hive]$ hive
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/src/hive/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/src/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/src/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Logging initialized using configuration in jar:file:/usr/local/src/hive/lib/hive-common-2.0.0.jar!/hive-log4j2.properties
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive>
當(dāng)顯示 hive>時(shí)，表示啟動(dòng)成功，進(jìn)入到了 Hive shell 狀態(tài)。
步驟二：Hive 操作基本命令
注意：Hive 命令行語(yǔ)句后面一定要加分號(hào)。
（1）查看數(shù)據(jù)庫(kù)
hive> show databases;
OK
default
sample
Time taken: 0.973 seconds, Fetched: 2 row(s)
顯示默認(rèn)的數(shù)據(jù)庫(kù) default。
（2）查看 default 數(shù)據(jù)庫(kù)所有表
hive> use default;
OK
Time taken: 0.02 seconds
hive> show tables;
OK
test
Time taken: 2.201 seconds, Fetched: 1 row(s)
顯示 default 數(shù)據(jù)中沒(méi)有任何表。
（3）創(chuàng)建表 stu，表的 id 為整數(shù)型，name 為字符型
hive> create table stu(id int,name string);
OK
Time taken: 0.432 seconds
（4）為表 stu 插入一條信息，id 號(hào)為 001，name 為liuyaling
hive> insert into stu values (001,"liuyaling");
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Query ID = hadoop_20230423222915_a95e9891-fdf5-4739-a63e-fcadecc85e28
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1682258121749_0001, Tracking URL = http://master:8088/proxy/application_1682258121749_0001/
Kill Command = /usr/local/src/hadoop/bin/hadoop job -kill job_1682258121749_0001
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2023-04-23 22:31:36,420 Stage-1 map = 0%, reduce = 0%
2023-04-23 22:31:43,892 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 2.72 sec
MapReduce Total cumulative CPU time: 2 seconds 720 msec
Ended Job = job_1682258121749_0001
Stage-4 is selected by condition resolver.
Stage-3 is filtered out by condition resolver.
Stage-5 is filtered out by condition resolver.
Moving data to: hdfs://master:9000/user/hive/warehouse/stu/.hive-staging_hive_2023-04-23_22-30-32_985_529079703757687911-1/-ext-10000
Loading data to table default.stu
MapReduce Jobs Launched:
Stage-Stage-1: Map: 1 Cumulative CPU: 2.72 sec HDFS Read: 4135 HDFS Write: 79 SUCCESS
Total MapReduce CPU Time Spent: 2 seconds 720 msec
OK
Time taken: 72.401 seconds
按照以上操作，繼續(xù)插入兩條信息：id 和 name 分別為 1002、1003 和 yanhaoxiang、tnt。
（5）插入數(shù)據(jù)后查看表的信息
hive> show tables;
OK
stu
test
values__tmp__table__1
values__tmp__table__2
Time taken: 0.026 seconds, Fetched: 4 row(s)
（6）查看表 stu 結(jié)構(gòu)
hive> desc stu;
OK
id int
name string
Time taken: 0.041 seconds, Fetched: 2 row(s)
（7）查看表 stu 的內(nèi)容
hive> select * from stu;
OK
1 liuyaling
1002 yanhaoxiang
1002 yanhaoxiang
1003 tnt
Time taken: 0.118 seconds, Fetched: 4 row(s)
步驟三：通過(guò) Hive 命令行界面查看文件系統(tǒng)和歷史命令
（1）查看本地文件系統(tǒng)，執(zhí)行命令 ! ls /usr/local/src;
hive> ! ls /usr/local/src;
flume
hadoop
hbase
hive
jdk
sqoop
zookeeper
（2）查看 HDFS 文件系統(tǒng)，執(zhí)行命令 dfs -ls /;
hive> dfs -ls /;
Found 5 items
drwxr-xr-x - hadoop supergroup 0 2023-04-23 21:58 /hbase
drwxr-xr-x - hadoop supergroup 0 2023-03-17 17:33 /input
drwxr-xr-x - hadoop supergroup 0 2023-03-17 18:45 /output
drwx------ - hadoop supergroup 0 2023-04-21 16:02 /tmp
drwxr-xr-x - hadoop supergroup 0 2023-04-14 20:48 /user
hive> exit;
（3）查看在 Hive 中輸入的所有歷史命令
進(jìn)入到當(dāng)前用戶 Hadoop 的目錄/home/hadoop，查看.hivehistory 文件。
[hadoop@master hive]$ cd /home/hadoop/
[hadoop@master ~]$ cat .hivehistory
show databases;
create database sample;
show databases;
use sample;
create table student(number string,name string);
exit
show databases;
exit;
use sample;
select * from student;
sqoop export --connect "jdbc:mysql://master:3306/sample?useUnicode=true&characterEncoding=utf-8" --username root --password Password@123! --table student --input-fields-terminated-by '|' --export-dir /user/hive/warehouse/sample.db/student/*
sqoop export --connect "jdbc:mysql://master:3306/sample?useUnicode=true&characterEncoding=utf-8" --username root --password Password@123! --table student --input-fields-terminated-by '|' --export-dir /user/hive/warehouse/sample.db/student/*;
sqoop export --connect "jdbc:mysql://master3306/sample?useUnicode=true&characterEncoding=utf-8" --username root --password Password@123! --table student --input-fields-terminated-by '|' --export-dir /user/hive/warehouse/sample.db/student/*
sqoop export --connect "jdbc:mysql://master3306/sample?useUnicode=true&characterEncoding=utf-8" --username root --password Password@123! --table student --input-fields-terminated-by '|' --export-dir /user/hive/warehouse/sample.db/student/*;
sqoop export --connect "jdbc:mysql://master3306/sample?useUnicode=true&characterEncoding=utf-8" --username root --password Password@123! --table student --input-fields-terminated-by '|' --export-dir /user/hive/warehouse/sample.db/student/*
exit
show databases;
use sample;
show tables;
use default;
show tables;
quit
'
select*from default.test;
quit;
show databases;
use default;
show tables;
insert into stu values (001,"liuyaling");
create table stu(id int,name string);
insert into stu values (001,"liuyaling");
insert into stu values (1002,"yanhaoxiang");
hive
insert into stu values (1003,"tnt");
quit
exit
exit;
quit;
show tables;
insert into stu values (1002,"yanhaoxiang");
insert into stu values (1003,"tnt");
show tables;
desc stu;
select * from stu;
! ls /usr/local/src;
dfs -ls /;
exit;
結(jié)果顯示，之前在 Hive 命令行界面下運(yùn)行的所有命令（含錯(cuò)誤命令）都顯示了出 來(lái)，有助于維護(hù)、故障排查等工作。
實(shí)驗(yàn)四、通過(guò)命令監(jiān)控大數(shù)據(jù)平臺(tái)服務(wù)狀態(tài)
1、通過(guò)命令查看 ZooKeeper 狀態(tài)
步驟一: 查看 ZooKeeper 狀態(tài)，執(zhí)行命令 zkServer.sh status，結(jié)果顯示如下
[hadoop@master ~]$ zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/local/src/zookeeper/bin/../conf/zoo.cfg
Mode: follower
以上結(jié)果中，Mode:follower 表示為 ZooKeeper 的跟隨者。
步驟二: 查看運(yùn)行進(jìn)程
QuorumPeerMain：QuorumPeerMain 是 ZooKeeper 集群的啟動(dòng)入口類，是用來(lái)加載配 置啟動(dòng) QuorumPeer 線程的。
執(zhí)行命令 jps 以查看進(jìn)程情況。
[hadoop@master ~]$ jps
2642 QuorumPeerMain
2994 SecondaryNameNode
3154 ResourceManager
5400 Jps
2795 NameNode
此時(shí) QuorumPeerMain 進(jìn)程已啟動(dòng)。
步驟四: 在成功啟動(dòng) ZooKeeper 服務(wù)后，輸入命令 zkCli.sh，連接到 ZooKeeper 服務(wù)。
[hadoop@master ~]$ zkCli.sh
Connecting to localhost:2181
2023-04-23 22:39:17,093 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.8--1, built on 02/06/2016 03:18 GMT
2023-04-23 22:39:17,096 [myid:] - INFO [main:Environment@100] - Client environment:host.name=master
2023-04-23 22:39:17,096 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.8.0_152
2023-04-23 22:39:17,097 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2023-04-23 22:39:17,097 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/usr/local/src/jdk/jre
2023-04-23 22:39:17,097 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/usr/local/src/zookeeper/bin/../build/classes:/usr/local/src/zookeeper/bin/../build/lib/*.jar:/usr/local/src/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/local/src/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/usr/local/src/zookeeper/bin/../lib/netty-3.7.0.Final.jar:/usr/local/src/zookeeper/bin/../lib/log4j-1.2.16.jar:/usr/local/src/zookeeper/bin/../lib/jline-0.9.94.jar:/usr/local/src/zookeeper/bin/../zookeeper-3.4.8.jar:/usr/local/src/zookeeper/bin/../src/java/lib/*.jar:/usr/local/src/zookeeper/bin/../conf:/usr/local/src/sqoop/lib:
2023-04-23 22:39:17,097 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2023-04-23 22:39:17,097 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2023-04-23 22:39:17,097 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
2023-04-23 22:39:17,097 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux
2023-04-23 22:39:17,097 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64
2023-04-23 22:39:17,097 [myid:] - INFO [main:Environment@100] - Client environment:os.version=3.10.0-862.el7.x86_64
2023-04-23 22:39:17,097 [myid:] - INFO [main:Environment@100] - Client environment:user.name=hadoop
2023-04-23 22:39:17,097 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/home/hadoop
2023-04-23 22:39:17,097 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/home/hadoop
2023-04-23 22:39:17,099 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@69d0a921
Welcome to ZooKeeper!
2023-04-23 22:39:17,122 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1032] - Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2023-04-23 22:39:17,208 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@876] - Socket connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating session
2023-04-23 22:39:17,223 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1299] - Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x187ae88b8390000, negotiated timeout = 30000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0]
步驟五: 使用 Watch 監(jiān)聽(tīng)/hbase 目錄，一旦/hbase 內(nèi)容有變化，將會(huì)有提 示。打開(kāi)監(jiān)視，執(zhí)行命令 get /hbase 1。
[zk: localhost:2181(CONNECTED) 0] get /hbase 1
cZxid = 0x500000002
ctime = Sun Apr 23 21:58:42 CST 2023
mZxid = 0x500000002
mtime = Sun Apr 23 21:58:42 CST 2023
pZxid = 0x500000062
cversion = 18
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 0
numChildren = 14
[zk: localhost:2181(CONNECTED) 1] quit
Quitting...
2023-04-23 22:40:16,816 [myid:] - INFO [main:ZooKeeper@684] - Session: 0x187ae88b8390001 closed
2023-04-23 22:40:16,817 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@519] - EventThread shut down for session: 0x187ae88b8390001
[hadoop@master ~]$
結(jié)果顯示，當(dāng)執(zhí)行命令 set /hbase value-update 后，數(shù)據(jù)版本由 0 變成 1，說(shuō)明 /hbase 處于監(jiān)控中。
2、通過(guò)命令查看 Sqoop 狀態(tài)
步驟一: 查詢 Sqoop 版本號(hào)，驗(yàn)證 Sqoop 是否啟動(dòng)成功。
首先切換到/usr/local/src/sqoop 目錄，執(zhí)行命令：./bin/sqoop-version
[hadoop@master ~]$ cd /usr/local/src/sqoop/
[hadoop@master sqoop]$ ./bin/sqoop-version
Warning: /usr/local/src/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/local/src/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
23/04/23 22:40:59 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
Sqoop 1.4.7
git commit id 2328971411f57f0cb683dfb79d19d4d19d185dd8
Compiled by maugli on Thu Dec 21 15:59:58 STD 2017
結(jié)果顯示：Sqoop 1.4.7，說(shuō)明 Sqoop 版本號(hào)為 1.4.7，并啟動(dòng)成功。
步驟二: 測(cè)試 Sqoop 是否能夠成功連接數(shù)據(jù)庫(kù)
切換到 Sqoop 的 目 錄 ， 執(zhí) 行 命 令 bin/sqoop list-databases --connect jdbc:mysql://master:3306/ --username root --password Password123$，命令中 “master:3306”為數(shù)據(jù)庫(kù)主機(jī)名和端口。
[hadoop@master sqoop]$ bin/sqoop list-databases --connect jdbc:mysql://master:3306/ --username root --password Password@123!
Warning: /usr/local/src/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/local/src/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
23/04/23 22:42:16 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
23/04/23 22:42:16 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
23/04/23 22:42:16 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
Sun Apr 23 22:42:16 CST 2023 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
information_schema
hive
mysql
performance_schema
sample
sys
結(jié)果顯示，可以連接到 MySQL，并查看到 Master 主機(jī)中 MySQL 的所有庫(kù)實(shí)例，如 information_schema、hive、mysql、performance_schema 和 sys 等數(shù)據(jù)庫(kù)。
步驟三: 執(zhí)行命令 sqoop help，可以看到如下內(nèi)容，代表 Sqoop 啟動(dòng)成功。
[hadoop@master sqoop]$ sqoop help
Warning: /usr/local/src/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/local/src/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
23/04/23 22:42:37 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
usage: sqoop COMMAND [ARGS]
Available commands:
codegen Generate code to interact with database records
create-hive-table Import a table definition into Hive
eval Evaluate a SQL statement and display the results
export Export an HDFS directory to a database table
help List available commands
import Import a table from a database to HDFS
import-all-tables Import tables from a database to HDFS
import-mainframe Import datasets from a mainframe server to HDFS
job Work with saved jobs
list-databases List available databases on a server
list-tables List available tables in a database
merge Merge results of incremental imports
metastore Run a standalone Sqoop metastore
version Display version information
See 'sqoop help COMMAND' for information on a specific command.
結(jié)果顯示了 Sqoop 的常用命令和功能，如下表所示。
| 序號(hào) | 命令 | 功能 |
|---|---|---|
| 1 | import | 將數(shù)據(jù)導(dǎo)入到集群 |
| 2 | export | 將集群數(shù)據(jù)導(dǎo)出 |
| 3 | codegen | 生成與數(shù)據(jù)庫(kù)記錄交互的代碼 |
| 4 | create-hivetable | 創(chuàng)建 Hive 表 |
| 5 | eval | 查看 SQL 執(zhí)行結(jié)果 |
| 6 | import-all-tables | 導(dǎo)入某個(gè)數(shù)據(jù)庫(kù)下所有表到 HDFS 中 |
| 7 | job | 生成一個(gè) job |
| 8 | list-databases | 列出所有數(shù)據(jù)庫(kù)名 |
| 9 | list-tables | 列出某個(gè)數(shù)據(jù)庫(kù)下所有的表 |
| 10 | merge | 將 HDFS 中不同目錄下數(shù)據(jù)合在一起，并存放在指定的目錄 中 |
| 11 | metastore | 記錄 Sqoop job 的元數(shù)據(jù)信息，如果不啟動(dòng) metastore 實(shí) 例，則默認(rèn)的元數(shù)據(jù)存儲(chǔ)目錄為：~/.sqoop |
| 12 | help | 打印 Sqoop 幫助信息 |
| 13 | version | 打印 Sqoop 版本信息 |
3、通過(guò)命令查看 Flume 狀態(tài)
步驟一:檢查 Flume 安裝是否成功，執(zhí)行 flume-ng version 命令，查看 Flume 的版本。
[hadoop@master sqoop]$ cd /usr/local/src/flume/
[hadoop@master flume]$ flume-ng version
Flume 1.6.0
Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
Revision: 2561a23240a71ba20bf288c7c2cda88f443c2080
Compiled by hshreedharan on Mon May 11 11:15:44 PDT 2015
From source with checksum b29e416802ce9ece3269d34233baf43f
[hadoop@master flume]$
步驟二:添加 example.conf 到/usr/local/src/flume
[hadoop@master flume]$ vim /usr/local/src/flume/example.conf
a1.sources=r1
a1.sinks=k1
a1.channels=c1
a1.sources.r1.type=spooldir
a1.sources.r1.spoolDir=/usr/local/src/flume/
a1.sources.r1.fileHeader=true
a1.sinks.k1.type=hdfs
a1.sinks.k1.hdfs.path=hdfs://master:9000/flume
a1.sinks.k1.hdfs.rollsize=1048760
a1.sinks.k1.hdfs.rollCount=0
a1.sinks.k1.hdfs.rollInterval=900
a1.sinks.k1.hdfs.useLocalTimeStamp=true
a1.channels.c1.type=file
a1.channels.c1.capacity=1000
a1.channels.c1.transactionCapacity=100
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
步驟三:啟動(dòng) Flume Agent a1 日志控制臺(tái)文章來(lái)源：http://www.zghlxwxcb.cn/news/detail-423686.html
[hadoop@master flume]$ flume-ng agent --conf-file example.conf --name a1 -Dflume.root.logger=INFO,console
Warning: No configuration directory set! Use --conf <dir> to override.
Info: Including Hadoop libraries found via (/usr/local/src/hadoop/bin/hadoop) for HDFS access
Info: Excluding /usr/local/src/hadoop/share/hadoop/common/lib/slf4j-api-1.7.10.jar from classpath
Info: Excluding /usr/local/src/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar from classpath
Info: Including HBASE libraries found via (/usr/local/src/hbase/bin/hbase) for HBASE access
Info: Excluding /usr/local/src/hbase/lib/slf4j-api-1.7.7.jar from classpath
Info: Excluding /usr/local/src/hbase/lib/slf4j-log4j12-1.7.5.jar from classpathxxxxxxxxxx?[hadoop@master flume]$ flume-ng agent --conf-file example.conf --name a1 -Dflume.root.logger=INFO,consoleWarning: No configuration directory set! Use --conf <dir> to override.Info: Including Hadoop libraries found via (/usr/local/src/hadoop/bin/hadoop) for HDFS accessInfo: Excluding /usr/local/src/hadoop/share/hadoop/common/lib/slf4j-api-1.7.10.jar from classpathInfo: Excluding /usr/local/src/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar from classpathInfo: Including HBASE libraries found via (/usr/local/src/hbase/bin/hbase) for HBASE accessInfo: Excluding /usr/local/src/hbase/lib/slf4j-api-1.7.7.jar from classpathInfo: Excluding /usr/local/src/hbase/lib/slf4j-log4j12-1.7.5.jar from classpath[hadoop@master flume]$ /usr/local/src/flume/bin/flume-ng agent --conf ./conf --conf-file ./example.conf ?--name a1 -Dflume.root.logger=INFO,consoleInfo: Sourcing environment configuration script /usr/local/src/flume/conf/flume-env.shInfo: Including Hadoop libraries found via (/usr/local/src/hadoop/bin/hadoop) for HDFS access.../lib/native:/usr/local/src/hadoop/lib/native org.apache.flume.node.Application --conf-file ./example.conf --name a1/usr/local/src/flume/bin/flume-ng: line 241: /usr/loocal/src/jdk/bin/java: No such file or directory
...
23/04/23 22:52:56 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: k1. sink.connection.failed.count == 0
23/04/23 22:52:56 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: k1. sink.event.drain.attempt == 1918
23/04/23 22:52:56 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: k1. sink.event.drain.sucess == 1918
[hadoop@master flume]$ ^C
步驟四: 查看結(jié)果文章來(lái)源地址http://www.zghlxwxcb.cn/news/detail-423686.html
[hadoop@master flume]$ hdfs dfs -lsr /flume
lsr: DEPRECATED: Please use 'ls -R' instead.
-rw-r--r-- 2 hadoop supergroup 1499 2023-04-23 22:52 /flume/FlumeData.1682261559722
-rw-r--r-- 2 hadoop supergroup 1419 2023-04-23 22:52 /flume/FlumeData.1682261559723
-rw-r--r-- 2 hadoop supergroup 1468 2023-04-23 22:52 /flume/FlumeData.1682261559724
...
-rw-r--r-- 2 hadoop supergroup 1795 2023-04-23 22:52 /flume/FlumeData.1682261559817
-rw-r--r-- 2 hadoop supergroup 1841 2023-04-23 22:52 /flume/FlumeData.1682261559818
-rw-r--r-- 2 hadoop supergroup 1665 2023-04-23 22:52 /flume/FlumeData.1682261559819
-rw-r--r-- 2 hadoop supergroup 1439 2023-04-23 22:52 /flume/FlumeData.1682261559820
?
到了這里，關(guān)于flume組件以及通過(guò)命令監(jiān)控大數(shù)據(jù)平臺(tái)轉(zhuǎn)態(tài)的文章就介紹完了。如果您還想了解更多內(nèi)容，請(qǐng)?jiān)谟疑辖撬阉鱐OY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！
