Step 1: Remove Hive

  Warning:

You must make sure no Hive processes are running. If Hive processes are running during the upgrade, the new version will not work correctly.

1. Exit the Hive console and make sure no Hive scripts are running.
2. Stop any HiveServer processes that are running. If HiveServer is running as a daemon, use the following command to stop it:

   $ sudo service hive-server stop

   If HiveServer is running from the command line, stop it with <CTRL>-c.

3. Stop the metastore. If the metastore is running as a daemon, use the following command to stop it:

   $ sudo service hive-metastore stop

   If the metastore is running from the command line, stop it with <CTRL>-c.

4. Remove Hive:

   $ sudo yum remove hive

   To remove Hive on SLES systems:

   $ sudo zypper remove hive

   To remove Hive on Ubuntu and Debian systems:

   $ sudo apt-get remove hive

Step 2: Install the new Hive version on all hosts (Hive servers and clients)

See Installing Hive.

  Important: Configuration files

• If you install a newer version of a package that is already on the system, configuration files that you have modified will remain intact.
• If you uninstall a package, the package manager renames any configuration files you have modified from <file> to <file>.rpmsave. If you then re-install the package (probably to install a new version) the package manager creates a new <file> with applicable defaults. You are responsible for applying any changes captured in the original configuration file to the new configuration file. In the case of Ubuntu and Debian upgrades, you will be prompted if you have made changes to a file for which there is a new version. For details, see Automatic handling of configuration files by dpkg.

Step 3: Configure the Hive Metastore

You must configure the Hive metastore and initialize the service before you can use Hive. See Configuring the Hive Metastore for detailed instructions.

Step 4: Upgrade the Metastore Schema

  Important:

• Cloudera strongly encourages you to make a backup copy of your metastore database before running the upgrade scripts. You will need this backup copy if you run into problems during the upgrade or need to downgrade to a previous version.

• You must upgrade the metastore schema to the version corresponding to the new version of Hive before starting Hive after the upgrade. Failure to do so may result in metastore corruption.

• To run a script, you must first cd to the directory that script is in: that is /usr/lib/hive/scripts/metastore/upgrade/<database>.

As of CDH 5, there are now two ways to do this. You could either use Hive's schematool or use the schema upgrade scripts provided with the Hive package.

Using schematool (Recommended):

The Hive distribution includes an offline tool for Hive metastore schema manipulation called schematool. This tool can be used to initialize the metastore schema for the current Hive version. It can also upgrade the schema from an older version to the current one.

$ schematool -dbType derby -upgradeSchemaFrom 0.10.0
Metastore connection URL:        jdbc:derby:;databaseName=metastore_db;create=true
Metastore Connection Driver :    org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User:       APP
Starting upgrade metastore schema from version 0.10.0 to <new_version>
Upgrade script upgrade-0.10.0-to-0.11.0.derby.sql
Completed upgrade-0.10.0-to-0.11.0.derby.sql
Upgrade script upgrade-0.11.0-to-<new_version>.derby.sql
Completed upgrade-0.11.0-to-<new_version>.derby.sql
schemaTool completed

Possible values for the dbType option are mysql, postgres, derby or oracle. The following table lists the Hive versions corresponding to the older CDH releases.

See Using the Hive Schema Tool for more details on how to use schematool.

Using Schema Upgrade Scripts:

Run the appropriate schema upgrade script(s); they are in /usr/lib/hive/scripts/metastore/upgrade/. Start with the script for your database and Hive version, and run all subsequent scripts.

For example, if you are currently running Hive 0.10 with MySQL, and upgrading to Hive 0.13.1, start with the script for Hive 0.10 to 0.11 for MySQL, then run the script for Hive 0.11 to 0.12 for MySQL, then run the script for Hive 0.12 to 0.13.1.

For more information about upgrading the schema, see the README in /usr/lib/hive/scripts/metastore/upgrade/.

Step 5: Configure HiveServer2

HiveServer2 is an improved version of the original HiveServer (HiveServer1, no longer supported). Some configuration is required before you initialize HiveServer2; see Configuring HiveServer2 for details.

Step 6: Upgrade Scripts for HiveServer2 (if necessary)

If you have been running HiveServer1, you may need to make some minor modifications to your client-side scripts and applications when you upgrade:

• HiveServer1 does not support concurrent connections, so many customers run a dedicated instance of HiveServer1 for each client. These can now be replaced by a single instance of HiveServer2.
• HiveServer2 uses a different connection URL and driver class for the JDBC driver. If you have existing scripts that use JDBC to communicate with HiveServer1, you can modify these scripts to work with HiveServer2 by changing the JDBC driver URL from jdbc:hive://hostname:port to jdbc:hive2://hostname:port, and by changing the JDBC driver class name from org.apache.hive.jdbc.HiveDriver to org.apache.hive.jdbc.HiveDriver.

Step 7: Start the Metastore, HiveServer2, and Beeline

See:

• Starting the Metastore
• Starting HiveServer2
• Using Beeline

Step 8: Upgrade the JDBC driver on the clients

The driver used for CDH 4.x does not work with CDH 5.x. Install the new version, following these instructions.

Step 1: Stop all Hive Processes and Daemons

  Warning:

You must make sure no Hive processes are running. If Hive processes are running during the upgrade, the new version will not work correctly.

1. Stop any HiveServer processes that are running:

   $ sudo service hive-server stop 

2. Stop any HiveServer2 processes that are running:

   $ sudo service hive-server2 stop 

3. Stop the metastore:

   $ sudo service hive-metastore stop 

Step 2: Install the new Hive version on all hosts (Hive servers and clients)

SeeInstalling Hive

Step 3: Verify that the Hive Metastore is Properly Configured

See Configuring the Hive Metastore for detailed instructions.

Step 4: Upgrade the Metastore Schema

  Important:

• Cloudera strongly encourages you to make a backup copy of your metastore database before running the upgrade scripts. You will need this backup copy if you run into problems during the upgrade or need to downgrade to a previous version.

• You must upgrade the metastore schema to the version corresponding to the new version of Hive before starting Hive after the upgrade. Failure to do so may result in metastore corruption.

• To run a script, you must first cd to the directory that script is in: that is /usr/lib/hive/scripts/metastore/upgrade/<database>.

As of CDH 5, there are now two ways to do this. You could either use Hive's schematool or use the schema upgrade scripts provided with the Hive package.

Using schematool (Recommended):

The Hive distribution includes an offline tool for Hive metastore schema manipulation called schematool. This tool can be used to initialize the metastore schema for the current Hive version. It can also upgrade the schema from an older version to the current one.

$ schematool -dbType derby -upgradeSchemaFrom 0.10.0
Metastore connection URL:        jdbc:derby:;databaseName=metastore_db;create=true
Metastore Connection Driver :    org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User:       APP
Starting upgrade metastore schema from version 0.10.0 to <new_version>
Upgrade script upgrade-0.10.0-to-0.11.0.derby.sql
Completed upgrade-0.10.0-to-0.11.0.derby.sql
Upgrade script upgrade-0.11.0-to-<new_version>.derby.sql
Completed upgrade-0.11.0-to-<new_version>.derby.sql
schemaTool completed

Possible values for the dbType option are mysql, postgres, derby or oracle. The following table lists the Hive versions corresponding to the older CDH releases.

See Using the Hive Schema Tool for more details on how to use schematool.

Using Schema Upgrade Scripts:

Run the appropriate schema upgrade script(s); they are in /usr/lib/hive/scripts/metastore/upgrade/. Start with the script for your database and Hive version, and run all subsequent scripts.

For example, if you are currently running Hive 0.10 with MySQL, and upgrading to Hive 0.13.1, start with the script for Hive 0.10 to 0.11 for MySQL, then run the script for Hive 0.11 to 0.12 for MySQL, then run the script for Hive 0.12 to 0.13.1.

For more information about upgrading the schema, see the README in /usr/lib/hive/scripts/metastore/upgrade/.

Step 5: Start the Metastore, HiveServer2, and Beeline

See:

• Starting the Metastore
• Starting, Stopping, and Using HiveServer2

The upgrade is now complete.

Troubleshooting: if you failed to upgrade the metastore

If you failed to upgrade the metastore as instructed above, proceed as follows.

1. Identify the problem.
   The symptoms are as follows:

   • Hive stops accepting queries.
   • In a cluster managed by Cloudera Manager, the Hive Metastore canary fails.
   • An error such as the following appears in the Hive Metastore Server logs:

     Hive Schema version 0.13.0 does not match metastore's schema version 0.12.0 Metastore is not upgraded or corrupt.

2. Resolve the problem.
   If the problem you are having matches the symptoms just described, do the following:

   1. Stop all Hive services; for example:

      $ sudo service hive-server2 stop
      $ sudo service hive-metastore stop

   2. Run the Hive schematool, as instructed here.
      Make sure the value you use for the -upgradeSchemaFrom option matches the version you are currently running (not the new version). For example, if the error message in the log is

      Hive Schema version 0.13.0 does not match metastore's schema version 0.12.0 Metastore is not upgraded or corrupt.

      then the value of -upgradeSchemaFrom must be 0.12.0.
   3. Restart the Hive services you stopped.