Upgrading Flume

Use the instructions that follow to upgrade Flume.

Upgrading Flume from CDH 4 to CDH 5

Use one of the following sets of instructions to upgrade Flume from CDH 4 to CDH 5, depending on which version of Flume you have been running under CDH 4:

Upgrading Flume 1.x to CDH 5

Use the instructions that follow to upgrade Flume 1.x from CDH 4 to CDH 5. You must remove the CDH 4 version and then install the CDH 5 version.

Step 1: Remove Flume 1.x from your cluster.

  1. Stop the Flume Node processes on each node where they are running:
    $ sudo service flume-ng-agent stop
  2. Uninstall the old Flume components:

    On Red Hat-compatible systems:

    $ sudo yum remove flume-ng agent flume-ng

    On SLES systems:

    $ sudo zypper remove flume-ng agent flume-ng

    On Ubuntu systems:

    $ sudo apt-get remove flume-ng agent flume-ng

Step 2. Install the new version of Flume

Follow the instructions in the rest of this document to install Flume 1.x under CDH 5.

Migrating from Flume 0.9.x under CDH 4 to Flume 1.x under CDH 5

Flume 1.x is a significant departure from Flume 0.9.x in its implementation although many of the original concepts are the same. If you're already familiar with Flume, here are some significant points.

  • You still have sources and sinks and they still do the same thing. They are now connected by channels.
  • Channels are pluggable, and dictate durability. Flume 1.x ships with an in-memory channel for fast, but non-durable event delivery. There are also JDBC-based and file-based channels for durable event delivery.
  • There are no more logical or physical nodes. All physical nodes are "agents," and agents can run zero or more sources and sinks.
  • There is no longer a Master, and no ZooKeeper dependency. At this time, Flume runs with a simple file-based configuration system.
  • Just about everything is a plugin — some end user facing, some for tool and system developers.
  • Thrift and Avro legacy Flume sources are provided to enable sending events from Flume 0.9.4 to Flume 1.x.

You must uninstall Flume 0.9.x and then install Flume 1.x, as follows.

Step 1: Remove Flume 0.9.x from your cluster.

  1. Stop the Flume Node processes on each node where they are running:
    $ sudo service flume-node stop
  2. Stop the Flume Master:
    $ sudo service flume-master stop
  3. Uninstall the old Flume components:

    On Red Hat-compatible systems:

    $ sudo yum remove flume

    On SLES systems:

    $ sudo zypper remove flume

    On Ubuntu systems:

    $ sudo apt-get remove flume

Step 2. Install the new version of Flume

Follow the instructions in the rest of this document to install Flume 1.x from a tarball or packages.

Upgrading Flume from an Earlier CDH 5 release

These instructions assume that you are upgrading Flume as part of an upgrade to the latest CDH 5 release, and have already performed the steps in Upgrading from an Earlier CDH 5 Release to the Latest Release.

To upgrade Flume from an earlier CDH 5 release, install the new version of Flume using one of the methods described below: Installing the Flume RPM or Debian Packages or Installing the Flume Tarball.