Before You Begin

• Read the CDH 5 Release Notes.
• Read the Cloudera Manager 5 Release Notes.
• Ensure Java 1.7 or 1.8 is installed across the cluster. For installation instructions and recommendations, see Upgrading to Oracle JDK 1.7 in a Cloudera Manager Deployment or Upgrading to Oracle JDK 1.8, and make sure you have read Known Issues and Workarounds in Cloudera Manager 5 before you proceed with the upgrade.
• Ensure that the Cloudera Manager minor version is equal to or greater than the CDH minor version. For example:

  Target CDH Version	Minimum Cloudera Manager Version
  5.0.5	5.0.x
  5.1.4	5.1.x
  5.4.1	5.4.x

• Date partition columns: as of Hive version 13, implemented in CDH 5.2, Hive validates the format of dates in partition columns, if they are stored as dates. A partition column with a date in invalid form can neither be used nor dropped once you upgrade to CDH 5.2 or higher. To avoid this problem, do one of the following:
  • Fix any invalid dates before you upgrade. Hive expects dates in partition columns to be in the form YYYY-MM-DD.
  • Store dates in partition columns as strings or integers.
  You can use the following SQL query to find any partition-column values stored as dates:

  SELECT "DBS"."NAME", "TBLS"."TBL_NAME", "PARTITION_KEY_VALS"."PART_KEY_VAL"
  FROM "PARTITION_KEY_VALS"
    INNER JOIN "PARTITIONS" ON "PARTITION_KEY_VALS"."PART_ID" = "PARTITIONS"."PART_ID"
    INNER JOIN "PARTITION_KEYS" ON "PARTITION_KEYS"."TBL_ID" = "PARTITIONS"."TBL_ID"
    INNER JOIN "TBLS" ON "TBLS"."TBL_ID" = "PARTITIONS"."TBL_ID"
    INNER JOIN "DBS" ON "DBS"."DB_ID" = "TBLS"."DB_ID"
      AND "PARTITION_KEYS"."INTEGER_IDX" ="PARTITION_KEY_VALS"."INTEGER_IDX"
      AND "PARTITION_KEYS"."PKEY_TYPE" = 'date';

• Whenever upgrading Impala, whether in CDH or a standalone parcel or package, check your SQL against the newest reserved words listed in incompatible changes. If upgrading across multiple versions or in case of any problems, check against the full list of Impala keywords.
• Run the Host Inspector and fix every issue.
• If using security, run the The Security Inspector.
• Run hdfs fsck / and hdfs dfsadmin -report and fix every issue.
• Run hbase hbck.
• Review the upgrade procedure and reserve a maintenance window with enough time allotted to perform all steps. For production clusters, Cloudera recommends allocating up to a full day maintenance window to perform the upgrade, depending on the number of hosts, the amount of experience you have with Hadoop and Linux, and the particular hardware you are using.
• To avoid lots of alerts during the upgrade process, you can enable maintenance mode on your cluster before you start the upgrade. This will stop email alerts and SNMP traps from being sent, but will not stop checks and configuration validations from being made. Be sure to exit maintenance mode when you have finished the upgrade to re-enable Cloudera Manager alerts.
• Hue validates CA certificates and needs a truststore. To create one, follow the instructions in Hue as a TLS/SSL Client.

Upgrade Unmanaged Components

For information on upgrading these unmanaged components, see Upgrading Mahout, Upgrading Pig, and Upgrading Whirr.

Stop Cluster Services

1. On the Home > Status tab, click to the right of the cluster name and select Stop.
2. Click Stop in the confirmation screen. The Command Details window shows the progress of stopping services.

   When All services successfully stopped appears, the task is complete and you can close the Command Details window.

Back up the HDFS Metadata on the NameNode

1. Go to the HDFS service.
2. Click the Configuration tab.
3. In the Search field, search for "NameNode Data Directories" and note the value.
4. On the active NameNode host, back up the directory listed in the NameNode Data Directories property. If more than one is listed, make a backup of one directory, since each directory is a complete copy. For example, if the NameNode data directory is /data/dfs/nn, do the following as root:

   # cd /data/dfs/nn
   # tar -cvf /root/nn_backup_data.tar .

   You should see output like this:

   ./
   ./current/
   ./current/fsimage
   ./current/fstime
   ./current/VERSION
   ./current/edits
   ./image/
   ./image/fsimage

   If there is a file with the extension lock in the NameNode data directory, the NameNode most likely is still running. Repeat the steps, starting by shutting down the NameNode role.

Back up Metastore Databases

1. For each affected service:
   1. If not already stopped, stop the service.
   2. Back up the database. See Backing Up Databases.

Upgrade Managed Components

• Use the Cloudera "1-click Install" package. This is the simplest way to upgrade only the Cloudera packages.
  1. Check whether you have the CDH 5 "1-click" repository installed.
     • Red Hat/CentOS-compatible and SLES

       rpm -q CDH 5-repository

       If you are upgrading from CDH 5 Beta 1 or higher, and you used the "1-click" package for the previous CDH 5 release, you should see:

       CDH5-repository-1-0

       In this case, skip to installing the CDH 5 packages. If instead you see:

       package CDH 5-repository is not installed

       proceed with installing the 1-click package.

     • Ubuntu and Debian

       dpkg -l | grep CDH 5-repository

       If the repository is installed, skip to installing the CDH 5 packages; otherwise proceed with installing the "1-click" package.

  2. If the CDH 5 "1-click" repository is not already installed on each host in the cluster, follow the instructions below for that host's operating system.
     • Red Hat compatible
       1. Download and install the "1-click Install" package.
          1. Download the CDH 5 "1-click Install" package (or RPM).

             Click the appropriate RPM and Save File to a directory with write access (for example, your home directory).

             OS Version	Link to CDH 5 RPM
             RHEL/CentOS/Oracle 5	RHEL/CentOS/Oracle 5 link
             RHEL/CentOS/Oracle 6	RHEL/CentOS/Oracle 6 link
             RHEL/CentOS/Oracle 7	RHEL/CentOS/Oracle 7 link

          2. Install the RPM for all RHEL versions:

             $ sudo yum --nogpgcheck localinstall cloudera-cdh-5-0.x86_64.rpm 

       2. (Optionally) add a repository key:
          • Red Hat/CentOS/Oracle 5

            $ sudo rpm --import http://archive.cloudera.com/cdh5/redhat/5/x86_64/cdh/RPM-GPG-KEY-cloudera

          • Red Hat/CentOS/Oracle 6

            $ sudo rpm --import http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/RPM-GPG-KEY-cloudera

     • SLES
       1. Download and install the "1-click Install" package:
          1. Download the CDH 5 "1-click Install" package.

             Download the rpm file, choose Save File, and save it to a directory to which you have write access (for example, your home directory).

          2. Install the RPM:

             $ sudo rpm -i cloudera-cdh-5-0.x86_64.rpm

          3. Update your system package index by running:

             $ sudo zypper refresh

       2. (Optionally) add a repository key:

          $ sudo rpm --import http://archive.cloudera.com/cdh5/sles/11/x86_64/cdh/RPM-GPG-KEY-cloudera  

     • Ubuntu and Debian
       1. Download and install the "1-click Install" package:
          1. Download the CDH 5 "1-click Install" package:

             OS Version	Package Link
             Jessie	Jessie package
             Wheezy	Wheezy package
             Precise	Precise package
             Trusty	Trusty package

          2. Install the package by doing one of the following:
             • Choose Open with in the download window to use the package manager.
             • Choose Save File, save the package to a directory to which you have write access (for example, your home directory), and install it from the command line. For example:

               sudo dpkg -i cdh5-repository_1.0_all.deb

       2. (Optionally) add a repository key:
          • Ubuntu Trusty

            $ curl -s http://archive.cloudera.com/cdh5/ubuntu/trusty/amd64/cdh/archive.key | sudo apt-key add -

          • Ubuntu Precise

            $ curl -s http://archive.cloudera.com/cdh5/ubuntu/precise/amd64/cdh/archive.key | sudo apt-key add -

          • Debian Wheezy

            $ curl -s http://archive.cloudera.com/cdh5/debian/wheezy/amd64/cdh/archive.key | sudo apt-key add -

  3. Install the CDH packages:
     • Red Hat compatible

       $ sudo yum clean all
       $ sudo yum install avro-tools crunch flume-ng hadoop-hdfs-fuse hadoop-httpfs hadoop-kms hbase hbase-solr hive-hbase hive-webhcat hue-beeswax hue-hbase hue-impala hue-pig hue-plugins hue-rdbms hue-search hue-spark hue-sqoop hue-zookeeper impala impala-shell kite llama mahout oozie parquet pig pig-udf-datafu search sentry solr solr-mapreduce spark-python sqoop sqoop2 whirr zookeeper

     • SLES

       $ sudo zypper clean --all
       $ sudo zypper install avro-tools crunch flume-ng hadoop-hdfs-fuse hadoop-httpfs hadoop-kms hbase hbase-solr hive-hbase hive-webhcat hue-beeswax hue-hbase hue-impala hue-pig hue-plugins hue-rdbms hue-search hue-spark hue-sqoop hue-zookeeper impala impala-shell kite llama mahout oozie parquet pig pig-udf-datafu search sentry solr solr-mapreduce spark-python sqoop sqoop2 whirr zookeeper

     • Ubuntu and Debian

       $ sudo apt-get update
       $ sudo apt-get install avro-tools crunch flume-ng hadoop-hdfs-fuse hadoop-httpfs hadoop-kms hbase hbase-solr hive-hbase hive-webhcat hue-beeswax hue-hbase hue-impala hue-pig hue-plugins hue-rdbms hue-search hue-spark hue-sqoop hue-zookeeper impala impala-shell kite llama mahout oozie parquet pig pig-udf-datafu search sentry solr solr-mapreduce spark-python sqoop sqoop2 whirr zookeeper

       Note: Installing these packages will also install all the other CDH packages that are needed for a full CDH 5 installation.
• Use your operating system's package management tools to update all packages to the latest version using standard repositories. This approach works well because it minimizes the amount of configuration required and uses the simplest commands. Be aware that this can take a considerable amount of time if you have not upgraded the system recently. To update all packages on your system, use the following command:

  Operating System	Command
  RHEL	$ sudo yum update
  SLES	$ sudo zypper up
  Ubuntu or Debian	$ sudo apt-get upgrade

Update Symlinks for the Newly Installed Components

$ sudo service cloudera-scm-agent restart

Run the Upgrade Wizard

1. Log into the Cloudera Manager Admin console.
2. From the Home > Status tab, click next to the cluster name and select Upgrade Cluster. The Upgrade Wizard starts.
3. In the Choose Method field, select the Use Packages option.
4. In the Choose CDH Version (Packages) field, specify the CDH version of the packages you have installed on your cluster. Click Continue.
5. Read the notices for steps you must complete before upgrading, click the Yes, I ... checkboxes after completing the steps, and click Continue.
6. Cloudera Manager checks that hosts have the correct software installed. If the packages have not been installed, a warning displays to that effect. Install the packages and click Check Again. When there are no errors, click Continue.
7. The Host Inspector runs and displays the CDH version on the hosts. Click Continue.
8. Choose the type of upgrade and restart:
   • Cloudera Manager upgrade - Cloudera Manager performs all service upgrades and restarts the cluster.
     1. Click Continue. The Command Progress screen displays the result of the commands run by the wizard as it shuts down all services, activates the new parcel, upgrades services as necessary, deploys client configuration files, and restarts services. If any of the steps fails or you click the Abort button the Retry button at the top right is enabled.
        
        You can click Retry to retry the step and continue the wizard or click the Cloudera Manager logo to return to the Home > Status tab and manually perform the failed step and all following steps.
     2. Click Continue. The wizard reports the result of the upgrade.
   • Manual upgrade - Select the Let me upgrade the cluster checkbox. Cloudera Manager configures the cluster to the specified CDH version but performs no upgrades or service restarts. Manually doing the upgrade is difficult and is for advanced users only.
     1. Click Continue. Cloudera Manager displays links to documentation describing the required upgrade steps.
9. Click Finish to return to the Home page.

Perform Manual Upgrade or Recover from Failed Steps

Finalize the HDFS Metadata Upgrade

1. Go to the HDFS service.
2. Click the Instances tab.
3. Click the NameNode instance.
4. Select Actions > Finalize Metadata Upgrade and click Finalize Metadata Upgrade to confirm.