The ANT project is happy to announce that our extensions to Hadoop to support splitting of bzip2-compressed files have been accepted to appear in the next Hadoop release (will be 0.21.0).<\/p>\n
Support for compression is important in map\/reduce because it reduces the amount of I\/O, and because important input files (for us, our Internet address censuses<\/a>) are provided in compressed format.<\/p>\n Splitting is important in map\/reduce, because splitting allows many computers to process parts<\/em> of a few big files.\u00a0 Since the whole point of Hadoop and map\/reduce is processing big<\/em> files (for us, 4GB or more) with many<\/em> computers (for us, dozens to hundreds), splitting is really essential<\/em>.<\/p>\n Until now, Hadoop did not support splitting of compressed files.\u00a0 Instead, if input data was compressed, you get at most one computer per file.\u00a0 Some work-arounds were possible, but basically unpleasant, and often requiring that one rewrite all the input data is some other format.<\/p>\n Our extensions (see HADOOP-4012<\/a> and MAPREDUCE-830<\/a>, plus HADOOP-3646<\/a> that went into 0.19.0) support Hadoop execution over bzip2 files with automatic splitting<\/strong>.\u00a0 Getting this done was trickier than one might expect:\u00a0 Hadoop really wants to decide where to split files, yet bzip2 can only support splits at specific locations that are different, and users don’t care about either of these but instead only about their <\/em>record boundaries.\u00a0 Fortunately, we were able to align all of these constraints, and deal with the corner cases that inevitably arise.\u00a0 (What if the bzip2 marker appears in normal data?\u00a0 What happens when markers exactly align, or are off-by-one?)<\/p>\n Abdul Qadeer did this work in 2008, working with Yuri Pradkin and me (John Heidemann), and continued to work with the patch through its getting committed.\u00a0 We especially thank Chris Douglas at Yahoo for shepherding patch through the Hadoop bug tracking system, including helping clean it up and add test cases.\u00a0 And we thank Doug Cutting for initially suggesting bzip2<\/a> as a splittable compression scheme.<\/p>\n