This is the documentation for Cloudera Enterprise 5.8.x. Documentation for other versions is available at Cloudera Documentation.

Configuring the Key Management Server (KMS)

Hadoop Key Management Server (KMS) is a cryptographic key management server based on the Hadoop KeyProvider API. It provides a KeyProvider implementation client that interacts with the KMS using the HTTP REST API. Both the KMS and its client support HTTP SPNEGO Kerberos authentication and TLS/SSL-secured communication. The KMS is a Java-based web application that uses a preconfigured Tomcat server bundled with the Hadoop distribution.

For instructions on securing the KMS, see Securing the Key Management Server (KMS).

Cloudera provides two implementations of the Hadoop KMS:
  • Java KeyStore KMS - The default Hadoop KMS included in CDH that uses a file-based Java KeyStore (JKS) for its backing keystore. For parcel-based installations, no additional action is required to install or upgrade the KMS. For package-based installations, you must install additional packages. For more information, see Installing and Upgrading Java KeyStore KMS. Cloudera strongly recommends not using Java Keystore KMS in production environments.
  • Key Trustee KMS - A custom KMS that uses Cloudera Navigator Key Trustee Server for its backing keystore instead of the file-based Java KeyStore (JKS) used by the default Hadoop KMS. Cloudera strongly recommends using Key Trustee KMS in production environments to improve the security, durability, and scalability of your cryptographic key management. For more information about the architecture and components involved in encrypting data at rest for production environments, see Cloudera Navigator Data Encryption Overview and Data at Rest Encryption Reference Architecture. For instructions on installing and upgrading Key Trustee KMS, see:

Continue reading:

Configuring KMS High Availability

For Key Trustee KMS high availability, see Key Trustee KMS High Availability. Java KeyStore KMS does not support high availability.

Configuring the KMS Using Cloudera Manager

If you are using Cloudera Manager, you can view and edit the KMS configuration by navigating to the following pages, depending on the KMS implementation you are using:
  • Key Trustee KMS service > Configuration
  • Java KeyStore KMS service > Configuration

For more information on using Cloudera Manager to find and change configuration parameters, see Modifying Configuration Properties Using Cloudera Manager.

For instructions about configuring the KMS and its clients using the command line for package-based installations, continue reading:

Configuring the KMS Cache Using Cloudera Manager

By default, the KMS caches keys to reduce the number of interactions with the key provider. You can disable the cache by setting the hadoop.kms.cache.enable property to false.

The cache is only used with the getCurrentKey(), getKeyVersion() and getMetadata() methods.

For the getCurrentKey() method, entries are cached for a maximum of 30000 milliseconds to prevent stale keys.

For the getKeyVersion() method, entries are cached with a default inactivity timeout of 600000 milliseconds (10 minutes).

You can configure the cache and its timeout values by adding the following properties to KMS service > Configuration > Advanced > Key Management Server Proxy Advanced Configuration Snippet (Safety Valve) for kms-site.xml: 
<property>
    <name>hadoop.kms.cache.enable</name>
    <value>true</value>
</property>

<property>
    <name>hadoop.kms.cache.timeout.ms</name>
    <value>600000</value>
</property>

<property>
    <name>hadoop.kms.current.key.cache.timeout.ms</name>
    <value>30000</value>
</property>

See Custom Configuration for more information on adding custom properties using the Advanced Configuration Snippet (Safety Valve) feature.

Configuring the Audit Log Aggregation Interval Using the Command Line

Audit logs are generated for GET_KEY_VERSION, GET_CURRENT_KEY, DECRYPT_EEK, and GENERATE_EEK operations.

Entries are aggregated by user, key, and operation for a configurable interval, after which the number of aggregated operations by the user for a given key is written to the audit log.

The interval is configured in milliseconds by adding the hadoop.kms.aggregation.delay.ms property to KMS service > Configuration > Advanced > Key Management Server Proxy Advanced Configuration Snippet (Safety Valve) for kms-site.xml: 
<property>
    <name>hadoop.kms.aggregation.delay.ms</name>
    <value>10000</value>
</property>

For more information about adding custom properties using the Advanced Configuration Snippet (Safety Valve) feature, see Custom Configuration.

Configuring the Java KeyStore KMS Using the Command Line

  Note: Because Key Trustee KMS is supported only in Cloudera Manager deployments, the following command line instructions apply only to Java KeyStore KMS. For instructions about configuring Key Trustee KMS, see Configuring the KMS Using Cloudera Manager.

For instructions about configuring the Java KeyStore KMS and its clients using the command line for package-based installations, continue reading:

Configuring the Java KeyStore KMS KeyProvider Using the Command Line

  Important:
  • If you use Cloudera Manager, do not use these command-line instructions.
  • This information applies specifically to CDH 5.8.x. If you use a lower version of CDH, see the documentation for that version located at Cloudera Documentation.
Configure the KMS backing KeyProvider properties in the /etc/hadoop-kms/conf/kms-site.xml configuration file: 
<property>
    <name>hadoop.kms.key.provider.uri</name>
    <value>jceks://file@/${user.home}/kms.keystore</value>
</property>

<property>
    <name>hadoop.security.keystore.java-keystore-provider.password-file</name>
    <value>keystore_password_file</value>
</property>

If you do not specify the absolute path to the password file, you must include it in the Hadoop CLASSPATH.

Restart the KMS for configuration changes to take effect. See Starting and Stopping the Java KeyStore KMS Using the Command Line for instructions.

Configuring the Java KeyStore KMS Cache Using the Command Line

  Important:
  • If you use Cloudera Manager, do not use these command-line instructions.
  • This information applies specifically to CDH 5.8.x. If you use a lower version of CDH, see the documentation for that version located at Cloudera Documentation.

By default, the KMS caches keys to reduce the number of interactions with the key provider. You can disable the cache by setting the hadoop.kms.cache.enable property to false.

The cache is only used with the getCurrentKey(), getKeyVersion() and getMetadata() methods.

For the getCurrentKey() method, entries are cached for a maximum of 30000 milliseconds to prevent stale keys.

For the getKeyVersion() method, entries are cached with a default inactivity timeout of 600000 milliseconds (10 minutes).

The cache and its timeout values are configured using the following properties in the /etc/hadoop-kms/conf/kms-site.xml configuration file: 
<property>
    <name>hadoop.kms.cache.enable</name>
    <value>true</value>
</property>

<property>
    <name>hadoop.kms.cache.timeout.ms</name>
    <value>600000</value>
</property>

<property>
    <name>hadoop.kms.current.key.cache.timeout.ms</name>
    <value>30000</value>
</property>

Configuring KMS Clients Using the Command Line

  Important:
  • If you use Cloudera Manager, do not use these command-line instructions.
  • This information applies specifically to CDH 5.8.x. If you use a lower version of CDH, see the documentation for that version located at Cloudera Documentation.

To configure KMS clients, set the hadoop.security.key.provider.path property in core-site.xml or hdfs-site.xml. Specify the value in the format kms://<scheme>@<kms_hosts>:<port>/kms. Replace <scheme> with http or https, depending on whether you have configured TLS. Replace <kms_hosts> with a semicolon-separated list of the KMS hosts. Replace <port> with the port number on which the KMS is running (16000 by default).

For example, for a KMS running on http://localhost:16000/kms, the KeyProvider URI is kms://http@localhost:16000/kms. For high availability KMS (Key Trustee KMS only) running on https://kms01.example.com:16000/kms and https://kms02.example.com:16000/kms, the KeyProvider URI is kms://https@kms01.example.com;kms02.example.com:16000/kms.

See the following for an excerpt from core-site.xml:

  <property>
    <name>hadoop.security.key.provider.path</name>
    <value>kms://https@kms01.example.com;kms02.example.com:16000/kms</value>
  </property>

Starting and Stopping the Java KeyStore KMS Using the Command Line

  Important:
  • If you use Cloudera Manager, do not use these command-line instructions.
  • This information applies specifically to CDH 5.8.x. If you use a lower version of CDH, see the documentation for that version located at Cloudera Documentation.
To start or stop KMS use the kms.sh script. For example, to start the KMS: 
$ sudo /usr/lib/hadoop-kms/sbin/kms.sh start

Running the script without parameters lists all possible parameters.

To use an init script to manage the KMS service, use your package manager to install the hadoop-kms-server package from the CDH repository. For example, for RHEL 6:

$ sudo yum install hadoop-kms-server

After installation, use the service hadoop-kms-server command to manage the KMS service.

Configuring the Audit Log Aggregation Interval Using the Command Line

  Important:
  • If you use Cloudera Manager, do not use these command-line instructions.
  • This information applies specifically to CDH 5.8.x. If you use a lower version of CDH, see the documentation for that version located at Cloudera Documentation.

Audit logs are generated for GET_KEY_VERSION, GET_CURRENT_KEY, DECRYPT_EEK, and GENERATE_EEK operations.

Entries are aggregated by user, key, and operation for a configurable interval, after which the number of aggregated operations by the user for a given key is written to the audit log.

The interval is configured in milliseconds using the hadoop.kms.aggregation.delay.ms property: 
<property>
    <name>hadoop.kms.aggregation.delay.ms</name>
    <value>10000</value>
</property>

Configuring the Embedded Tomcat Server Using the Command Line

  Important:
  • If you use Cloudera Manager, do not use these command-line instructions.
  • This information applies specifically to CDH 5.8.x. If you use a lower version of CDH, see the documentation for that version located at Cloudera Documentation.

You can configure the embedded Tomcat server by using the /etc/hadoop-kms/tomcat-conf/conf/server.xml.conf file.

The following environment variables can be set in KMS /etc/hadoop-kms/conf/kms-env.sh script and can be used to alter the default ports and log directory:

  • KMS_HTTP_PORT
  • KMS_ADMIN_PORT
  • KMS_LOG

Restart the KMS for the configuration changes to take effect.

Page generated July 8, 2016.