一、虛擬機(jī)安裝CentOS7并配置共享文件夾
二、CentOS 7 上hadoop偽分布式搭建全流程完整教程
三、本機(jī)使用python操作hdfs搭建及常見問題
四、mapreduce搭建
五、mapper-reducer編程搭建
六、hive數(shù)據(jù)倉(cāng)庫安裝

一、環(huán)境搭建

1.打開虛擬機(jī)系統(tǒng)，打開hadoop

確保網(wǎng)頁可以打開

2.修改本機(jī)hosts文件

ifconfig查看當(dāng)前ip,

打開C:\Windows\System32\drivers\etc\，編輯hosts文件 ，在末尾添加192.168.137.134 hadoop4

若不可以編輯
右鍵屬性->安全->選擇Users那個(gè)->編輯->勾選修改 即可

3.進(jìn)行ping測(cè)試連通

打開我的電腦打開任意文件夾->左上角文件->打開Windows powershell->
進(jìn)行ping測(cè)試

4.安裝hdfs

wIn+r 輸入cmd進(jìn)入執(zhí)行(在任意python或conda環(huán)境安裝皆可)

pip install hdfs -i https://pypi.douban.com/simple/

二、python操作hdfs

1.建立連接

PS D:\software\Bandicam\video> python
Python 3.9.7 (default, Sep 16 2021, 16:59:28) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32

Warning:
This Python interpreter is in a conda environment, but the environment has
not been activated.  Libraries may fail to load.  To activate this environment
please see https://conda.io/activation

Type "help", "copyright", "credits" or "license" for more information.
>>> from hdfs.client import Client
>>> link=Client('http://hadoop4:50070')
>>> link.list('/')
[]

2.創(chuàng)建文件夾

新建文件夾出現(xiàn)錯(cuò)誤

>>> link.makedirs('/test')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "D:\Anaconda3\install_position\lib\site-packages\hdfs\client.py", line 1036, in makedirs
    self._mkdirs(hdfs_path, permission=permission)
  File "D:\Anaconda3\install_position\lib\site-packages\hdfs\client.py", line 118, in api_handler
    raise err
hdfs.util.HdfsError: Permission denied: user=dr.who, access=WRITE, inode="/":huangqifa:supergroup:drwxr-xr-x

解決：
虛擬機(jī)端執(zhí)行hadoop fs -chmod -R 777 /

>>> link.makedirs('/test')
>>> link.list('/')
['test']
>>>

創(chuàng)建文件夾出現(xiàn)安全模式問題：

>>> link.makedirs('/test')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "D:\Anaconda3\install_position\lib\site-packages\hdfs\client.py", line 1036, in makedirs
    self._mkdirs(hdfs_path, permission=permission)
  File "D:\Anaconda3\install_position\lib\site-packages\hdfs\client.py", line 118, in api_handler
    raise err
hdfs.util.HdfsError: Cannot create directory /test. Name node is in safe mode.
Resources are low on NN. Please add or free up more resources then turn off safe mode manually. NOTE:  If you turn off safe mode before adding resources, the NN will immediately return to safe mode. Use "hdfs dfsadmin -safemode leave" to turn safe mode off.

解決：
虛擬機(jī)端執(zhí)行

hdfs dfsadmin -safemode leave

3.上傳文件（需保證jps服務(wù)中datanode namenode均打開）

>>> link.makedirs('/test')
>>> link.list('/')
['test']
>>> link.upload('/test','C:/readme.txt')
'/test/readme.txt'
>>> link.list('/test')
['readme.txt']

4.寫文件

>>> link.write('/test/test01.txt',"hello world")
>>> link.list('/test')
['readme.txt', 'test01.txt']

5.下載文件或文件夾

>>> link.download('/test/test01.txt','D:/')
'D:\\test01.txt'
>>>

三、可能出現(xiàn)的問題

問題描述：物理機(jī)ping不到虛擬機(jī)ip或物理機(jī)192.168…h(huán)adoop網(wǎng)頁打不開
重置虛擬機(jī)網(wǎng)絡(luò)中NAT那個(gè)
解決：
虛擬機(jī)軟件->左上角編輯->虛擬機(jī)網(wǎng)絡(luò)編輯器

文章來源地址http://www.zghlxwxcb.cn/news/detail-800551.html

到了這里，關(guān)于本機(jī)使用python操作hdfs搭建及常見問題的文章就介紹完了。如果您還想了解更多內(nèi)容，請(qǐng)?jiān)谟疑辖撬阉鱐OY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！