使用Java API上傳數(shù)據(jù)到HDFS

@Test
public void test1() throws Exception {
	//構(gòu)造一個(gè)輸入流，代表要上傳的數(shù)據(jù)
	InputStream input = new FileInputStream("d:\\temp\\hadoop-2.7.3.tar.gz");
				
	//NameNodeProxies
	//NameNode

	//指定NameNode 地址
	Configuration conf = new Configuration();
	conf.set("fs.defaultFS", "hdfs://192.168.157.111:9000");
		
	//創(chuàng)建HDFS的客戶端 DistributedFileSystem
	FileSystem client = FileSystem.get(conf);
		
	//構(gòu)造一個(gè)輸出流，指向HDFS
	OutputStream output = client.create(new Path("/folder1/a.tag.gz"));
		
	//緩沖區(qū)
	byte[] buffer = new byte[1024];
	//長(zhǎng)度
	int len = 0;
	while( (len = input.read(buffer)) > 0 ) {
		//寫到輸出流
		output.write(buffer, 0, len);
	}
		
	output.flush();
	output.close();
	input.close();
}

使用工具類簡(jiǎn)化后的代碼:

@Test
public void test2() throws Exception {
	//構(gòu)造一個(gè)輸入流，代表要上傳的數(shù)據(jù)
	InputStream input = new FileInputStream("d:\\temp\\hadoop-2.7.3.tar.gz");
		
	//指定NameNode 地址
	Configuration conf = new Configuration();
	conf.set("fs.defaultFS", "hdfs://192.168.157.111:9000");
		
	//創(chuàng)建HDFS的客戶端
	FileSystem client = FileSystem.get(conf);
		
	//構(gòu)造一個(gè)輸出流，指向HDFS
	OutputStream output = client.create(new Path("/folder1/b.tag.gz"));
		
	//使用工具類簡(jiǎn)化
	IOUtils.copyBytes(input, output, 1024);
}

HDFS數(shù)據(jù)的下載

@Test
public void test1() throws Exception{
	//構(gòu)造一個(gè)輸出流，指向本地
	OutputStream output = new FileOutputStream("d:\\temp\\x.tar.gz");
		
	//指定NameNode的地址
	Configuration conf = new Configuration();
	conf.set("fs.defaultFS", "hdfs://192.168.157.111:9000");
		
	//創(chuàng)建HDFS的客戶端
	FileSystem client = FileSystem.get(conf);
	//得到一個(gè)輸入流
	InputStream input = client.open(new Path("/folder1/b.tag.gz"));
		
	//構(gòu)造一個(gè)緩沖區(qū)
	byte[] buffer = new byte[1024];
	int len = 0;
	while((len=input.read(buffer)) > 0) {
		output.write(buffer, 0, len);
	}
		
	output.flush();
	output.close();
	input.close();
}

使用工具類簡(jiǎn)化后的代碼:文章來源地址http://www.zghlxwxcb.cn/news/detail-601755.html

@Test
public void test1() throws Exception{
	//構(gòu)造一個(gè)輸出流，指向本地
	OutputStream output = new FileOutputStream("d:\\temp\\x.tar.gz");
		
	//指定NameNode的地址
	Configuration conf = new Configuration();
	conf.set("fs.defaultFS", "hdfs://192.168.157.111:9000");
		
	//創(chuàng)建HDFS的客戶端
	FileSystem client = FileSystem.get(conf);
	//得到一個(gè)輸入流
	InputStream input = client.open(new Path("/folder1/b.tag.gz"));
		
	//構(gòu)造一個(gè)緩沖區(qū)
	byte[] buffer = new byte[1024];
	int len = 0;
	while((len=input.read(buffer)) > 0) {
		output.write(buffer, 0, len);
	}
		
	output.flush();
	output.close();
	input.close();
}

到了這里，關(guān)于【hadoop】使用Java API 上傳下載數(shù)據(jù)的文章就介紹完了。如果您還想了解更多內(nèi)容，請(qǐng)?jiān)谟疑辖撬阉鱐OY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！