How to read files with an offset from Hadoop using Java(如何使用 Java 从 Hadoop 中读取具有偏移量的文件)
问题描述
Problem: I want to read a section of a file from HDFS and return it, such as lines 101-120 from a file of 1000 lines.
I don't want to use seek because I have read that it is expensive.
I have log files which I am using PIG to process down into meaningful sets of data. I've been writing an API to return the data for consumption and display by a front end. Those processed data sets can be large enough that I don't want to read the entire file out of Hadoop in one slurp to save wire time and bandwidth. (Let's say 5 - 10MB)
Currently I am using a BufferedReader to return small summary files which is working fine
ArrayList lines = new ArrayList();
...
for (FileStatus item: items) {
// ignoring files like _SUCCESS
if(item.getPath().getName().startsWith("_")) {
continue;
}
in = fs.open(item.getPath());
BufferedReader br = new BufferedReader(new InputStreamReader(in));
String line;
line = br.readLine();
while (line != null) {
line = line.replaceAll("(\r|\n)", "");
lines.add(line.split(" "));
line = br.readLine();
}
}
I've poked around the interwebs quite a bit as well as Stack but haven't found exactly what I need.
Perhaps this is completely the wrong way to go about doing it and I need a completely separate set of code and different functions to manage this. Open to any suggestions.
Thanks!
As added noted based on research from the below discussions: How does Hadoop process records records split across block boundaries? Hadoop FileSplit Reading
I think SEEK is a best option for reading files with huge volumes. It did not cause any problems to me as the volume of data that i was reading was in the range of 2 - 3GB. I did not encounter any issues till today but we did use file splitting to handle the large data set. below is the code which you can use for reading purpose and test the same.
public class HDFSClientTesting {
/**
* @param args
*/
public static void main(String[] args) {
// TODO Auto-generated method stub
try{
//System.loadLibrary("libhadoop.so");
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
conf.addResource(new Path("core-site.xml"));
String Filename = "/dir/00000027";
long ByteOffset = 3185041;
SequenceFile.Reader rdr = new SequenceFile.Reader(fs, new Path(Filename), conf);
Text key = new Text();
Text value = new Text();
rdr.seek(ByteOffset);
rdr.next(key,value);
//Plain text
JSONObject jso = new JSONObject(value.toString());
String content = jso.getString("body");
System.out.println("
" + content + "
");
File file =new File("test.gz");
file.createNewFile();
}
catch (Exception e ){
throw new RuntimeException(e);
}
finally{
}
}
}
这篇关于如何使用 Java 从 Hadoop 中读取具有偏移量的文件的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持编程学习网!
本文标题为:如何使用 Java 从 Hadoop 中读取具有偏移量的文件
基础教程推荐
- Java ECDSAwithSHA256 签名长度不一致 2022-01-01
- 控制台应用程序中的 Java 键盘输入解析 2022-01-01
- doFilter()是在servlet的工作完成之前还是之后执行的? 2022-01-01
- 在java中使用xpath和selenium解析HTML表格数据 2022-01-01
- JPA惰性列表上的流 2022-01-01
- 如何在相机中应用自定义滤镜 [Surfaceview 预览]. 2022-01-01
- 将 Windows 证书导入 Java 2022-01-01
- 在springboot中如何给mybatis加拦截器 2023-04-29
- Maven:无效的目标版本:10 2022-01-01
- 将 double 转换为 Int,向下舍入 2022-01-01
