Choosing and Configuring Data Compression

For an overview of compression, see Data Compression.

Guidelines for Choosing a Compression Type

  • GZIP compression uses more CPU resources than Snappy or LZO, but provides a higher compression ratio. GZip is often a good choice for cold data, which is accessed infrequently. Snappy or LZO are a better choice for hot data, which is accessed frequently.
  • BZip2 can also produce more compression than GZip for some types of files, at the cost of some speed when compressing and decompressing. HBase does not support BZip2 compression.
  • Snappy often performs better than LZO. It is worth running tests to see if you detect a significant difference.
  • For MapReduce, if you need your compressed data to be splittable, BZip2 and LZO formats can be split. Snappy and GZip blocks are not splittable, but files with Snappy blocks inside a container file format such as SequenceFile or Avro can be split. Snappy is intended to be used with a container format, like SequenceFiles or Avro data files, rather than being used directly on plain text, for example, since the latter is not splittable and cannot be processed in parallel using MapReduce. Splittability is not relevant to HBase data.
  • For MapReduce, you can compress either the intermediate data, the output, or both. Adjust the parameters you provide for the MapReduce job accordingly. The following examples compress both the intermediate data and the output. MR2 is shown first, followed by MR1.
    • MRv2
      hadoop jar hadoop-examples-.jar sort "-Dmapreduce.compress.map.output=true"
      "-Dmapreduce.map.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec"
      "-Dmapreduce.output.compress=true"
      "-Dmapreduce.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec" -outKey
      org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text input output
    • MRv1
      hadoop jar hadoop-examples-.jar sort "-Dmapred.compress.map.output=true"
      "-Dmapred.map.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec"
      "-Dmapred.output.compress=true"
      "-Dmapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec" -outKey
      org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text input output

Configuring Data Compression

Configuring Data Compression Using Cloudera Manager

To configure support for LZO using Cloudera Manager, you must install the GPL Extras parcel, then configure services to use it. See Installing the GPL Extras Parcel and Configuring Services to Use the GPL Extras Parcel.

Configuring Data Compression Using the Command Line

To configure support for LZO in CDH, see (Optional) Install LZO and Configuring LZO. Snappy support is included in CDH.

To use Snappy in a MapReduce job, see Using Snappy with MapReduce. Use the same method for LZO, with the codec com.hadoop.compression.lzo.LzopCodec instead.