This is the documentation for Cloudera Enterprise 5.8.x. Documentation for other versions is available at Cloudera Documentation.

Upgrading Flume

Use the instructions that follow to upgrade Flume.
  Note: Running Services

Use the service command to start, stop, and restart CDH components, rather than running scripts in /etc/init.d directly. The service command creates a predictable environment by setting the current working directory to / and removing most environment variables (passing only LANG and TERM). With /etc/init.d, existing environment variables remain in force and can produce unpredictable results. When you install CDH from packages, service is installed as part of the Linux Standard Base (LSB).

Upgrading Flume from CDH 4 to CDH 5

Use one of the following sets of instructions to upgrade Flume from CDH 4 to CDH 5, depending on which version of Flume you have been running under CDH 4:

Upgrading Flume 1.x to CDH 5

Use the instructions that follow to upgrade Flume 1.x from CDH 4 to CDH 5. You must remove the CDH 4 version and then install the CDH 5 version.

Step 1: Remove Flume 1.x from your cluster.

  1. Stop the Flume host processes on each host where they are running: 
    $ sudo service flume-ng-agent stop
  2. Uninstall the old Flume components:

    On Red Hat-compatible systems:

    $ sudo yum remove flume-ng agent flume-ng

    On SLES systems:

    $ sudo zypper remove flume-ng agent flume-ng

    On Ubuntu systems:

    $ sudo apt-get remove flume-ng agent flume-ng

Step 2. Install the new version of Flume

Follow the instructions in the rest of this document to install Flume 1.x under CDH 5.

Migrating from Flume 0.9.x under CDH 4 to Flume 1.x under CDH 5

Flume 1.x is a significant departure from Flume 0.9.x in its implementation although many of the original concepts are the same. If you're already familiar with Flume, here are some significant points.

  • You still have sources and sinks and they still do the same thing. They are now connected by channels.
  • Channels are pluggable, and dictate durability. Flume 1.x ships with an in-memory channel for fast, but non-durable event delivery. Flume has JDBC-based and file-based channels for durable event delivery.
  • Flume no longer has logical or physical nodes. All physical nodes are "agents," and agents can run zero or more sources and sinks.
  • Flume no longer has a Master, and no ZooKeeper dependency. At this time, Flume runs with a file-based configuration system.
  • Just about everything is a plugin — some are for end-users, others for tool and system developers.
  • Thrift and Avro legacy Flume sources are provided to enable sending events from Flume 0.9.4 to Flume 1.x.

You must uninstall Flume 0.9.x and then install Flume 1.x, as follows.

Step 1: Remove Flume 0.9.x from your cluster.

  1. Stop the Flume host processes on each host where they are running: 
    $ sudo service flume-node stop
  2. Stop the Flume Master: 
    $ sudo service flume-master stop
  3. Uninstall the old Flume components:

    On Red Hat-compatible systems:

    $ sudo yum remove flume

    On SLES systems:

    $ sudo zypper remove flume

    On Ubuntu systems:

    $ sudo apt-get remove flume

Step 2. Install the new version of Flume

Follow the instructions in the rest of this document to install Flume 1.x from a tarball or packages.

Upgrading Flume from an Earlier CDH 5 release

These instructions assume that you are upgrading Flume as part of an upgrade to the latest CDH 5 release, and have already performed the steps in Upgrading from an Earlier CDH 5 Release to the Latest Release.

To upgrade Flume from an earlier CDH 5 release, install the new version of Flume using one of the methods described below: Installing the Flume RPM or Debian Packages or Installing the Flume Tarball.

  Important: Configuration files
  • If you install a newer version of a package that is already on the system, configuration files that you have modified will remain intact.
  • If you uninstall a package, the package manager renames any configuration files you have modified from <file> to <file>.rpmsave. If you then re-install the package (probably to install a new version) the package manager creates a new <file> with applicable defaults. You are responsible for applying any changes captured in the original configuration file to the new configuration file. In the case of Ubuntu and Debian upgrades, you will be prompted if you have made changes to a file for which there is a new version. For details, see Automatic handling of configuration files by dpkg.
Page generated July 8, 2016.