大家好，我是邵奈一，一個不務(wù)正業(yè)的程序猿、正兒八經(jīng)的斜杠青年。
1、世人稱我為：被代碼耽誤的詩人、沒天賦的書法家、五音不全的歌手、專業(yè)跑龍?zhí)籽輪T、不合格的運(yùn)動員…
2、這幾年，我整理了很多IT技術(shù)相關(guān)的教程給大家，愛生活、愛分享。
3、如果您覺得文章有用，請收藏，轉(zhuǎn)發(fā)，評論，并關(guān)注我，謝謝！
博客導(dǎo)航跳轉(zhuǎn)（請收藏）：邵奈一的技術(shù)博客導(dǎo)航
| 公眾號 | 微信 | CSDN | 掘金 | 51CTO | 簡書 | 微博 |
0x00 教程內(nèi)容
- 準(zhǔn)備工作
- 編寫代碼
- 運(yùn)行效果
- 實(shí)現(xiàn)說明
說明：本實(shí)現(xiàn)非常地粗糙，僅供參考。通過本教程，可以看到運(yùn)行結(jié)果，并且實(shí)現(xiàn)跟 HDFS 交互的最基本的功能。
0x01 基于Hadoop分布式存儲的網(wǎng)盤系統(tǒng)實(shí)現(xiàn)
1. 準(zhǔn)備工作
（1）需要自行安裝好 Hadoop
（2）啟動好 Hadoop
2. 編寫代碼
注意：這里必須要引入Hadoop的依賴，可以參考教程：Java API實(shí)現(xiàn)HDFS的相關(guān)操作 新建一個Maven項(xiàng)目，并引入依賴。
（1）編寫頁面代碼： MainFram
package com.bigdata.mapreduce.swing;
import com.bigdata.mapreduce.utils.HDFSUtil;
import java.awt.event.ActionEvent;
import java.awt.event.ActionListener;
import java.util.ArrayList;
import java.util.List;
import javax.swing.event.TreeSelectionEvent;
import javax.swing.event.TreeSelectionListener;
import javax.swing.*;
import javax.swing.tree.DefaultMutableTreeNode;
public class MainFram extends JFrame{
private JTree tree=null;
private JButton btnUpload = new JButton("上傳");
private JButton btnDownload = new JButton("下載");
private JButton btnCreateDir = new JButton("創(chuàng)建文件夾");
private JButton btndeleteDirOrFiles = new JButton("刪除文件或者文件夾");
private JButton btnRenameDirOrFiles = new JButton("重命名文件或者文件夾");
private JButton btnMoveDir = new JButton("移動文件/文件夾");
private JButton btnListDir = new JButton("列出文件");
private DefaultMutableTreeNode root=new DefaultMutableTreeNode("我的網(wǎng)盤");
private JPanel jp_center=new JPanel();
// 初始化函數(shù)
private void initTree() {
tree=new JTree(root);
tree.addTreeSelectionListener(new TreeSelectionListener() {
@Override
public void valueChanged(TreeSelectionEvent e) {
tree_ValueChanged(e);
}
});
}
private void tree_ValueChanged(TreeSelectionEvent e) {
// JOptionPane.showMessageDialog(this,
// tree.getSelectionPath().getLastPathComponent().toString());
btnListDir_Clicked();
}
public MainFram() {
JPanel jp=(JPanel)this.getContentPane();
initTree();
JScrollPane jsp_tree=new JScrollPane(tree);
// 創(chuàng)建Panel
JPanel jp_top=new JPanel();
// 添加三個按鈕
jp_top.add(btnUpload);
jp_top.add(btnDownload);
jp_top.add(btnCreateDir);
jp_top.add(btndeleteDirOrFiles);
jp_top.add(btnRenameDirOrFiles);
jp_top.add(btnMoveDir);
// jp_top.add(btnListDir);
JSplitPane splitPane_right=new JSplitPane(JSplitPane.VERTICAL_SPLIT,jp_top,jp_center);
splitPane_right.setDividerLocation(100);
splitPane_right.setDividerSize(1);
JSplitPane splitPane=new JSplitPane(JSplitPane.HORIZONTAL_SPLIT,jsp_tree,splitPane_right);
splitPane.setDividerLocation(300);
splitPane.setDividerSize(2);
jp.add(splitPane);
btnUpload.addActionListener(new ActionListener() {
@Override
public void actionPerformed(ActionEvent e) {
btnUpload_Clicked();
}
});
btnDownload.addActionListener(new ActionListener() {
@Override
public void actionPerformed(ActionEvent e) {
btnDownload_Clicked();
}
});
btnCreateDir.addActionListener(new ActionListener() {
@Override
public void actionPerformed(ActionEvent e) {
btnCreateDir_Clicked();
}
});
btnListDir.addActionListener(new ActionListener() {
@Override
public void actionPerformed(ActionEvent e) {
btnListDir_Clicked();
}
});
btndeleteDirOrFiles.addActionListener(new ActionListener() {
@Override
public void actionPerformed(ActionEvent e) {
btndeleteDirOrFiles_Clicked();
}
});
btnRenameDirOrFiles.addActionListener(new ActionListener() {
@Override
public void actionPerformed(ActionEvent e) {
btnRenameDirOrFiles_Clicked();
}
});
btnMoveDir.addActionListener(new ActionListener() {
@Override
public void actionPerformed(ActionEvent e) {
btnMoveDir_Clicked();
}
});
this.setTitle("我的云盤");
this.setSize(1200, 800);
this.setVisible(true);
this.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
}
private void btnDownload_Clicked() {
System.out.println("download");
String message = "請輸入需要下載的文件，HDFS上的文件：\n";
try {
List<String> hdfsDir = HDFSUtil.listRemoteDirAndFiles("/");
for (int k = 0; k < hdfsDir.size(); k++) {
message += hdfsDir.get(k) + "\n";
}
} catch (Exception e) {
e.printStackTrace();
}
String srcPath = JOptionPane.showInputDialog(null, message);
System.out.println(srcPath);
JFileChooser jf = new JFileChooser();
jf.setFileSelectionMode(JFileChooser.DIRECTORIES_ONLY);
jf.setDialogTitle("請選擇要上傳的文件夾...");
jf.showDialog(null, null);
String destPath = jf.getSelectedFile().getAbsolutePath() + "/";
// String[] name = jf.getSelectedFile().getAbsolutePath().split("/");
// String destPath = "/user";
if (destPath.isEmpty()) {
System.out.println("請選擇本地路徑!");
} else {
try {
HDFSUtil.downloadFromHDFS(srcPath, destPath);
} catch (Exception e1) {
e1.printStackTrace();
}
}
System.out.println("下載成功!");
}
private void btnUpload_Clicked() {
System.out.println("upload");
JFileChooser jf = new JFileChooser();
jf.setFileSelectionMode(JFileChooser.FILES_ONLY);
jf.setDialogTitle("請選擇要上傳的文件...");
jf.showDialog(null, null);
String srcPath = jf.getSelectedFile().getAbsolutePath() + "/";
// String destPath = "/";
if (srcPath.isEmpty()) {
System.out.println("本地文件路徑不能為空!");
} else {
String destPath = JOptionPane.showInputDialog(null, "請輸入需要上傳到HDFS的路徑：");
if (!destPath.isEmpty()) {
try {
HDFSUtil.uploadToHDFS(srcPath, destPath);
} catch (Exception e1) {
e1.printStackTrace();
}
} else {
System.out.println("HDFS路徑不能為空!");
}
}
System.out.println("上傳成功!");
}
private void btnMoveDir_Clicked() {
String message1 = "請輸入需要移動的文件或者文件夾名：\n";
String message2 = "請輸入想要移動后的文件夾名：\n";
try {
List<String> hdfsDir = HDFSUtil.listRemoteDirAndFiles("/");
for (int k = 0; k < hdfsDir.size(); k++) {
message1 += hdfsDir.get(k) + "\n";
message2 += hdfsDir.get(k) + "\n";
}
} catch (Exception e) {
e.printStackTrace();
}
String oldName = JOptionPane.showInputDialog(null, message1);
String newName = JOptionPane.showInputDialog(null, message2);
System.out.println(oldName);
if (oldName.isEmpty() || newName.isEmpty()) {
System.out.println("參數(shù)錯誤，請重試！");
} else {
try {
HDFSUtil.moveDirFromHDFS("/" + oldName, "/" + newName);
} catch (Exception e) {
e.printStackTrace();
}
}
System.out.println("移動文件/文件夾成功!");
}
private void btnRenameDirOrFiles_Clicked() {
String message1 = "請輸入需要重命名的文件或者文件夾名：\n";
String message2 = "請輸入想要重命后的文件或者文件夾名：\n";
try {
List<String> hdfsDir = HDFSUtil.listRemoteDirAndFiles("/");
for (int k = 0; k < hdfsDir.size(); k++) {
message1 += hdfsDir.get(k) + "\n";
message2 += hdfsDir.get(k) + "\n";
}
} catch (Exception e) {
e.printStackTrace();
}
String oldName = JOptionPane.showInputDialog(null, message1);
String newName = JOptionPane.showInputDialog(null, message2);
System.out.println(oldName);
if (oldName.isEmpty() || newName.isEmpty()) {
System.out.println("參數(shù)錯誤，請重試！");
} else {
try {
HDFSUtil.renameFromHDFS("/" + oldName, "/" + newName);
} catch (Exception e) {
e.printStackTrace();
}
}
System.out.println("重命名文件/文件夾成功!");
}
private void btndeleteDirOrFiles_Clicked() {
String message = "請輸入需要刪除的文件或者文件夾名：\n";
try {
List<String> hdfsDir = HDFSUtil.listRemoteDirAndFiles("/");
for (int k = 0; k < hdfsDir.size(); k++) {
message += hdfsDir.get(k) + "\n";
}
} catch (Exception e) {
e.printStackTrace();
}
String deleteName = JOptionPane.showInputDialog(null, message);
System.out.println(deleteName);
if (deleteName.isEmpty()) {
System.out.println("請輸入文件或者文件夾名！");
} else {
try {
HDFSUtil.deleteFromHDFS("/" + deleteName);
} catch (Exception e) {
e.printStackTrace();
}
}
System.out.println("刪除文件/文件夾成功!");
}
private void btnListDir_Clicked() {
int flag = 0;
System.out.println("listDir");
try {
List<String> hdfsDir = HDFSUtil.listRemoteDirAndFiles("/");
// List<String> hdfsDir = HDFSUtil.listRemoteDir("/");
List<String> oneDirList = new ArrayList<>();
DefaultMutableTreeNode oneDir = new DefaultMutableTreeNode();
DefaultMutableTreeNode twoDir = new DefaultMutableTreeNode();
for (int i = 0; i < hdfsDir.size(); i++) {
System.out.println(hdfsDir.get(i));
String[] arr = hdfsDir.get(i).split("/");
int length = arr.length;
// 目前只支持而級目錄
if (length == 2) { // length為2表示只有一個文件或者空文件夾，如：/user
oneDir = new DefaultMutableTreeNode(arr[1]);
root.add(oneDir);
} else if (length == 3) { // length為3表示有兩個文件或者兩個空文件夾，如：/user/hello.txt
// 如果一級目錄里已經(jīng)有這個目錄，則不添加這個目錄
for (int j = 0; j < oneDirList.size(); j++) {
if (oneDirList.get(j).equals(arr[1])) {
flag = 1;
}
}
if (flag == 0) {
oneDir = new DefaultMutableTreeNode(arr[1]);
twoDir = new DefaultMutableTreeNode(arr[2]);
oneDirList.add(arr[1]);
oneDir.add(twoDir);
root.add(oneDir);
} else if (flag == 1) {
twoDir = new DefaultMutableTreeNode(arr[2]);
oneDir.add(twoDir);
}
}
}
System.out.println("列出文件成功!");
} catch (Exception e) {
e.printStackTrace();
}
}
private void btnCreateDir_Clicked() {
System.out.println("createDir");
String remoteDirName = JOptionPane.showInputDialog(null, "請輸入需要創(chuàng)建的文件夾名：");
System.out.println(remoteDirName);
if (remoteDirName.isEmpty()) {
System.out.println("請輸入文件名！");
} else {
try {
HDFSUtil.createDirFromHDFS("/" + remoteDirName);
} catch (Exception e) {
e.printStackTrace();
}
}
System.out.println("新建文件夾成功!");
}
public static void main(String[] args) {
JFrame.setDefaultLookAndFeelDecorated(true);
new MainFram();
}
}
（2）編寫HDFS代碼： HDFSUtil
package com.bigdata.mapreduce.utils;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.List;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.* ;
public class HDFSUtil {
private static String hdfsURL="hdfs://localhost:9000";
private static Configuration conf;
static {
conf = new Configuration();
conf.set("fs.defaultFS", hdfsURL);
}
// 從HDFS上下載文件
public static void downloadFromHDFS(String remoteFile,String localFile) throws Exception {
FileSystem fs = FileSystem.get(conf);
Path remotePath = new Path(remoteFile);
Path localPath = new Path(localFile);
fs.copyToLocalFile(remotePath, localPath);
fs.close();
}
// 上傳文件到HDFS
public static void uploadToHDFS(String localfile,String remotefile) throws Exception {
FileSystem fs = FileSystem.get(conf);
Path remotePath = new Path(remotefile);
Path localPath = new Path(localfile);
fs.copyFromLocalFile(localPath, remotePath);
fs.close();
}
// 創(chuàng)建HDFS文件夾
public static void createDirFromHDFS(String remoteDir) throws Exception {
FileSystem fs = FileSystem.get(conf);
Path remotePath = new Path(remoteDir);
//調(diào)用mkdirs函數(shù)創(chuàng)建目錄
fs.mkdirs(remotePath);
fs.close();
}
// 刪除文件和文件夾
public static void deleteFromHDFS(String remoteDir) throws Exception {
FileSystem fs = FileSystem.get(conf);
Path remotePath = new Path(remoteDir);
//調(diào)用mkdirs函數(shù)創(chuàng)建目錄,true表示循環(huán)遞歸刪除
fs.delete(remotePath, true);
fs.close();
}
// 重命名文件
public static void renameFromHDFS(String oldName, String newName) throws Exception {
FileSystem fs = FileSystem.get(conf);
Path hdfsOldName = new Path(oldName);
Path hdfsNewName = new Path(newName);
fs.rename(hdfsOldName, hdfsNewName);
fs.close();
}
// 查看文件夾
public static void ListHDFSDir(String remoteDir) throws Exception {
FileSystem fs = FileSystem.get(conf);
Path dirPath = new Path(remoteDir);
/*遞歸獲取目錄下的所有文件*/
RemoteIterator<LocatedFileStatus> remoteIterator = fs.listFiles(dirPath, true);
/*輸出每個文件的信息*/
while (remoteIterator.hasNext()) {
FileStatus s = remoteIterator.next();
System.out.println("路徑: " + s.getPath().toString());
System.out.println("權(quán)限: " + s.getPermission().toString());
System.out.println("大小: " + s.getLen());
/*返回的是時(shí)間戳,轉(zhuǎn)化為時(shí)間日期格式*/
Long timeStamp = s.getModificationTime();
SimpleDateFormat format = new SimpleDateFormat( "yyyy-MM-dd HH:mm:ss");
String date = format.format(timeStamp);
System.out.println("時(shí)間: " + date);
System.out.println();
}
fs.close();
}
// 列出文件
public static List<String> listRemoteDirAndFiles(String remoteDir) throws Exception {
List<String> remoteDirList = new ArrayList<>();
FileSystem fs = FileSystem.get(conf);
Path dirPath = new Path(remoteDir);
/*遞歸獲取目錄下的所有文件*/
RemoteIterator<LocatedFileStatus> remoteIterator = fs.listFiles(dirPath, true);
/*輸出每個文件的信息*/
while (remoteIterator.hasNext()) {
FileStatus s = remoteIterator.next();
String myPath = s.getPath().toString().substring(21);
remoteDirList.add(myPath);
}
fs.close();
return remoteDirList;
}
// 列出文件夾
public static List<String> listRemoteDir(String remoteDir) throws Exception {
List<String> remoteDirList = new ArrayList<>();
FileSystem fs = FileSystem.get(conf);
Path dirPath = new Path(remoteDir);
FileStatus[] fileStatus = fs.listStatus(dirPath);
for (FileStatus file : fileStatus) {
if (file.isDirectory()) {
listRemoteDir(file.getPath().toString());
remoteDirList.add(file.getPath().toString().substring(21));
} else {
remoteDirList.add(file.getPath().toString().substring(21));
}
}
fs.close();
return remoteDirList;
}
// 移動文件夾
public static void moveDirFromHDFS(String oldPath, String newPath) throws Exception {
FileSystem fs = FileSystem.get(conf);
Path hdfsOldName = new Path(oldPath);
Path hdfsNewName = new Path(newPath);
fs.rename(hdfsOldName, hdfsNewName);
fs.close();
}
// 測試
// public static void main(String[] args) {
// try {
// 1、列出文件
// List<String> list = HDFSUtil.listRemoteDirAndFiles("/");
// for (int i = 0; i < list.size(); i++) {
// System.out.println(list.get(i));
// }
// 2、重名文件
// HDFSUtil.renameFromHDFS("/start-dfs.cmd", "/start-dfs.sh");
// 3、創(chuàng)建文件夾
// HDFSUtil.createDirFromHDFS("/hive");
// 4、移動文件夾（把/路徑下的hive文件夾，放到/user路徑）
// HDFSUtil.moveDirFromHDFS("/hive", "/user");
// 5、刪除文件或者文件夾
// HDFSUtil.deleteFromHDFS("/user");
// HDFSUtil.deleteFromHDFS("/user/hive/start-dfs.sh");
// } catch (Exception e) {
// e.printStackTrace();
// }
// }
}
3. 運(yùn)行效果
很多的功能都已經(jīng)實(shí)現(xiàn)，雖然不是特別地實(shí)用。
直接運(yùn)行MainFram類就可以看到運(yùn)行效果：
4. 實(shí)現(xiàn)說明
（1）如果需要修改HDFS的鏈接，可以修改代碼的值：
private static String hdfsURL="hdfs://localhost:9000";
（2）目前目錄樹其實(shí)是還不規(guī)范的，只是站在零基礎(chǔ)同學(xué)角度的一個實(shí)現(xiàn)方案，需要自行完善。文章來源：http://www.zghlxwxcb.cn/news/detail-497163.html
0xFF 總結(jié)
- 通過本教程，可以擴(kuò)展大家的思維，其實(shí)我們學(xué)了HDFS的API操作，并不是一無是處的，我們其實(shí)是可以自己寫一個簡單的小工具來使用的。
- 關(guān)注本博主，學(xué)習(xí)更多有趣的知識。
邵奈一 原創(chuàng)不易，如轉(zhuǎn)載請標(biāo)明出處，教育是一生的事業(yè)。文章來源地址http://www.zghlxwxcb.cn/news/detail-497163.html
到了這里，關(guān)于基于Hadoop分布式存儲的網(wǎng)盤系統(tǒng)實(shí)現(xiàn)（簡易粗糙版）的文章就介紹完了。如果您還想了解更多內(nèi)容，請?jiān)谟疑辖撬阉鱐OY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！
