This is the documentation for Cloudera Enterprise 5.8.x. Documentation for other versions is available at Cloudera Documentation.

Cloudera Navigator Key Trustee Server High Availability

Key Trustee Server high availability applies to read operations only. If either Key Trustee Server fails, the KeyProvider automatically retries fetching keys from the functioning server. New write operations (for example, creating new encryption keys) are not allowed unless both Key Trustee Servers are operational.

If a Key Trustee Server fails, the following operations are impacted:

  • HDFS Encryption
    • You cannot create new encryption keys for encryption zones.
    • You can write to and read from existing encryption zones, but you cannot create new zones.
  • Cloudera Navigator Encrypt
    • You cannot register new Cloudera Navigator Encrypt clients.
    • You can continue reading and writing encrypted data, including creating new mount points, using existing clients.

Cloudera recommends monitoring both Key Trustee Servers. If a Key Trustee Server fails catastrophically, restore it from backup to a new host with the same hostname and IP address as the failed host. See Backing Up and Restoring Key Trustee Server and Clients for more information. Cloudera does not support PostgreSQL promotion to convert a passive Key Trustee Server to an active Key Trustee Server.

Continue reading:

Configuring Key Trustee Server High Availability Using Cloudera Manager

For new installations, use the Set up HDFS Data At Rest Encryption wizard and follow the instructions in Enabling HDFS Encryption Using the Wizard. When prompted, make sure that the Enable High Availability option is selected.

If you already have a Key Trustee Server service, and want to enable high availability, use the Add Role Instances wizard for the Key Trustee Server service instead to add the Passive Key Trustee Server and Passive Database roles.
  Important: You must assign the Key Trustee Server and Database roles to the same host. Assign the Active Key Trustee Server and Active Database roles to one host, and the Passive Key Trustee Server and Passive Database roles to a separate host.
After completing the Add Role Instances wizard, the Passive Key Trustee Server and Passive Database roles fail to start. Complete the following manual actions to start these roles:
  1. Stop the Key Trustee Server service (Key Trustee Server service > Actions > Stop).
  2. Run the Set Up Key Trustee Server Database command (Key Trustee Server service > Actions > Set Up Key Trustee Server Database).
  3. Run the following command on the Active Key Trustee Server:
    $ sudo rsync -zav --exclude .ssl /var/lib/keytrustee/.keytrustee root@keytrustee02.example.com:/var/lib/keytrustee/.

    Replace keytrustee02.example.com with the hostname of the Passive Key Trustee Server.

  4. Run the following command on the Passive Key Trustee Server:
    $ sudo ktadmin init
  5. Start the Key Trustee Server service (Key Trustee Server service > Actions > Start).
      Important: Starting or restarting the Key Trustee Server service attempts to start the Active Database and Passive Database roles. If the Active Database is not running when the Passive Database attempts to start, the Passive Database fails to start. If this occurs, manually restart the Passive Database role after confirming that the Active Database role is running.
  6. Enable synchronous replication (Key Trustee Server service > Actions > Setup Enable Synchronous Replication in HA mode).
  7. Restart the Key Trustee Server service (Key Trustee Server service > Actions > Restart).

For parcel-based Key Trustee Server releases 5.8 and higher, Cloudera Manager automatically backs up Key Trustee Server (using the ktbackup.sh script) after adding the Key Trustee Server service. It also schedules automatic backups using cron. For package-based installations, you must manually back up Key Trustee Server and configure a cron job.

Cloudera Manager configures cron to run the backup script hourly. The latest ten backups are retained in /var/lib/keytrustee in cleartext. For information about using the backup script and configuring the cron job (including how to encrypt backups), see Backing Up Key Trustee Server and Key Trustee KMS Using the ktbackup.sh Script.

Configuring Key Trustee Server High Availability Using the Command Line

Install and configure a second Key Trustee Server following the instructions in Installing Cloudera Navigator Key Trustee Server.

Once you have installed and configured the second Key Trustee Server, initialize the active Key Trustee Server by running the following commands on the active Key Trustee Server host:
  Important: For Key Trustee Server 5.4.0 and higher, the ktadmin init-master command is deprecated, and should not be used. Use the ktadmin init command instead. If you are using SSH software other than OpenSSH, pre-create the SSH key on the active Key Trustee Server before continuing:
$ sudo -u keytrustee ssh-keygen -t rsa /var/lib/keytrustee/.ssh/id_rsa
$ sudo ktadmin init --external-address keytrustee01.example.com
$ sudo rsync -zav --exclude .ssl /var/lib/keytrustee/.keytrustee root@keytrustee02.example.com:/var/lib/keytrustee/.
$ sudo ktadmin db --bootstrap --port 11381 --pg-rootdir /var/lib/keytrustee/db --slave keytrustee02.example.com
## For RHEL/CentOS 7, use 'sudo systemctl [stop|start] <service_name>' instead of 'sudo service <service_name> [stop|start]' ##
$ sudo service keytrustee-db stop
$ sudo service keytrustee-db start
$ sudo service keytrusteed start

Replace keytrustee01.example.com with the fully qualified domain name (FQDN) of the active Key Trustee Server. Replace keytrustee02.example.com with the FQDN of the passive Key Trustee Server. Replace /var/lib/keytrustee/db with the path to the directory you want to use to store the PostgreSQL database.

To use a different port for the database, modify the ktadmin init and ktadmin db commands as follows:

$ sudo ktadmin init --external-address keytrustee01.example.com --db-connect postgresql://localhost:<port>/keytrustee?host=/tmp
$ sudo ktadmin db --bootstrap --port <port> --pg-rootdir /var/lib/keytrustee/db --slave keytrustee02.example.com
If you use a database directory other than /var/lib/keytrustee/db, create or edit the /etc/sysconfig/keytrustee-db file and add the following:
ARGS="--pg-rootdir /path/to/db"

The ktadmin init command generates a self-signed certificate that the Key Trustee Server uses for HTTPS communication.

Initialize the passive Key Trustee Server by running the following commands on the passive host:
$ sudo ktadmin init-slave --master keytrustee01.example.com --pg-rootdir /var/lib/keytrustee/db --no-import-key --no-start
## For RHEL/CentOS 7, use 'sudo systemctl [stop|start] <service_name>' instead of 'sudo service <service_name> [stop|start]' ##
$ sudo service keytrustee-db start
$ sudo ktadmin init --external-address keytrustee02.example.com
$ sudo service keytrusteed start

Replace keytrustee02.example.com with the fully qualified domain name (FQDN) of the passive Key Trustee Server. Replace keytrustee01.example.com with the FQDN of the active Key Trustee Server. Replace /var/lib/keytrustee/db with the path to the directory you want to use to store the PostgreSQL database.

To use a different port for the database, modify the ktadmin init-slave command as follows:

$ sudo ktadmin init-slave --master keytrustee01.example.com --pg-port <port> --pg-rootdir /var/lib/keytrustee/db --no-import-key --no-start
If you use a database directory other than /var/lib/keytrustee/db, create or edit the /etc/sysconfig/keytrustee-db file and add the following:
ARGS="--pg-rootdir /path/to/db"

The ktadmin init-slave command performs an initial database sync by running the pg_basebackup command. The database directory must be empty for this step to work. For information on performing an incremental backup, see the PostgreSQL documentation.

  Note: The /etc/init.d/postgresql script does not work when the PostgreSQL database is started by Key Trustee Server, and cannot be used to monitor the status of the database. Use /etc/init.d/keytrustee-db instead.

The ktadmin init command generates a self-signed certificate that the Key Trustee Server uses for HTTPS communication. Instructions for using alternate certificates (for example, if you have obtained certificates from a trusted Certificate Authority) are provided later.

Enable Synchronous Replication

Key Trustee Server high availability requires synchronous replication to ensure that all rows in the database are inserted in at least two hosts, which protects against key loss.

To enable synchronous replication, run the following command on the active Key Trustee Server:
$ sudo ktadmin enable-synchronous-replication --pg-rootdir /var/lib/keytrustee/db

If you modified the default database location, replace /var/lib/keytrustee/db with the modified path.

(Optional) Replace Self-Signed Certificates with CA-Signed Certificates

  Important: Because clients connect to Key Trustee Server using its fully qualified domain name (FQDN), certificates must be issued to the FQDN of the Key Trustee Server host. If you are using CA-signed certificates, ensure that the generated certificates use the FQDN, and not the short name.

If you have a CA-signed certificate for Key Trustee Server, see Managing Key Trustee Server Certificates for instructions on how to replace the self-signed certificates.

Recovering a Key Trustee Server

If a Key Trustee Server fails, restore it from backup as soon as possible. If the Key Trustee Server hosts fails completely, make sure that you restore the Key Trustee Server to a new host with the same hostname and IP address as the failed host.

For more information, see Backing Up and Restoring Key Trustee Server and Clients.

Page generated July 8, 2016.