Step 1: Configure the JobTrackers, TaskTrackers, and Clients

Changes to existing configuration parameters

New configuration parameters

Make changes and additions similar to the following to mapred-site.xml on each node.

  Note: It is simplest to configure all the parameters on all nodes, even though not all of the parameters will be used on any given node. This also makes for robustness if you later change the roles of the nodes in your cluster.

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

  <property>
    <name>mapred.job.tracker</name>
    <value>logicaljt</value> 
    <!-- host:port string is replaced with a logical name -->
  </property>

  <property>
    <name>mapred.jobtrackers.logicaljt</name>
    <value>jt1,jt2</value>
    <description>Comma-separated list of JobTracker IDs.</description>
  </property>

  <property>
    <name>mapred.jobtracker.rpc-address.logicaljt.jt1</name> 
    <!-- RPC address for jt1 -->
    <value>myjt1.myco.com:8021</value>
  </property>

  <property>
    <name>mapred.jobtracker.rpc-address.logicaljt.jt2</name> 
    <!-- RPC address for jt2 -->
    <value>myjt2.myco.com:8022</value>
  </property>

 <property>
    <name>mapred.job.tracker.http.address.logicaljt.jt1</name> 
    <!-- HTTP bind address for jt1 -->
    <value>0.0.0.0:50030</value>
  </property>

  <property>
    <name>mapred.job.tracker.http.address.logicaljt.jt2</name> 
    <!-- HTTP bind address for jt2 -->
    <value>0.0.0.0:50031</value>
  </property>

  <property>
    <name>mapred.ha.jobtracker.rpc-address.logicaljt.jt1</name> 
    <!-- RPC address for jt1 HA daemon -->
    <value>myjt1.myco.com:8023</value>
  </property>

  <property>
    <name>mapred.ha.jobtracker.rpc-address.logicaljt.jt2</name> 
    <!-- RPC address for jt2 HA daemon -->
    <value>myjt2.myco.com:8024</value>
  </property>


  <property>
    <name>mapred.ha.jobtracker.http-redirect-address.logicaljt.jt1</name> 
    <!-- HTTP redirect address for jt1 -->
    <value>myjt1.myco.com:50030</value>
  </property>

  <property>
    <name>mapred.ha.jobtracker.http-redirect-address.logicaljt.jt2</name> 
    <!-- HTTP redirect address for jt2 -->
    <value>myjt2.myco.com:50031</value>
  </property>

 <property>
    <name>mapred.jobtracker.restart.recover</name>
    <value>true</value>
  </property>

  <property>
    <name>mapred.job.tracker.persist.jobstatus.active</name>
    <value>true</value>
  </property>

 <property>
    <name>mapred.job.tracker.persist.jobstatus.hours</name>
    <value>1</value>
  </property>

  <property>
    <name>mapred.job.tracker.persist.jobstatus.dir</name>
    <value>/jobtracker/jobsInfo</value>
  </property>

  <property>
    <name>mapred.client.failover.proxy.provider.logicaljt</name>
    <value>org.apache.hadoop.mapred.ConfiguredFailoverProxyProvider</value>
  </property>

  <property>
    <name>mapred.client.failover.max.attempts</name>
    <value>15</value>
  </property>

  <property>
    <name>mapred.client.failover.sleep.base.millis</name>
    <value>500</value>
  </property>

  <property>
    <name>mapred.client.failover.sleep.max.millis</name>
    <value>1500</value>  
  </property>

 <property>
    <name>mapred.client.failover.connection.retries</name>
    <value>0</value>  
  </property>

 <property>
    <name>mapred.client.failover.connection.retries.on.timeouts</name>
    <value>0</value>  
  </property>
 <property>
    <name>mapred.ha.fencing.methods</name>
    <value>shell(/bin/true)</value>
 </property>
</configuration>
 

  Note:

In pseudo-distributed mode you need to specify mapred.ha.jobtracker.id for each JobTracker, so that the JobTracker knows its identity.

But in a fully-distributed setup, where the JobTrackers run on different nodes, there is no need to set mapred.ha.jobtracker.id, since the JobTracker can infer the ID from the matching address in one of the mapred.jobtracker.rpc-address.<name>.<id> properties.

Step 2: Start the JobTracker daemons

To start the daemons, run the following command on each JobTracker node:

$ sudo service hadoop-0.20-mapreduce-jobtrackerha start

Step 3: Activate a JobTracker

  Note:

• You must be the mapred user to use mrhaadmin commands.
• If Kerberos is enabled, do not use sudo -u mapred when using the hadoop mrhaadmin command. Instead, you must log in with the mapred Kerberos credentials (the short name must be mapred). See Configuring Hadoop Security in CDH 5 for more information.

Unless automatic failover is configured, both JobTrackers will be in a standby state after the jobtrackerha daemons start up.

If Kerberos is not enabled, use the following commands:

To find out what state each JobTracker is in:

$ sudo -u mapred hadoop mrhaadmin -getServiceState <id>

where <id> is one of the values you configured in the mapred.jobtrackers.<name> property – jt1 or jt2 in our sample mapred-site.xml files.

To transition one of the JobTrackers to active and then verify that it is active:

$ sudo -u mapred hadoop mrhaadmin -transitionToActive <id>
$ sudo -u mapred hadoop mrhaadmin -getServiceState <id>

where <id> is one of the values you configured in the mapred.jobtrackers.<name> property – jt1 or jt2 in our sample mapred-site.xml files.

With Kerberos enabled, log in as the mapred user and use the following commands:

To log in as the mapred user and kinit:

$ sudo su - mapred
$ kinit -kt mapred.keytab mapred/<fully.qualified.domain.name>

To find out what state each JobTracker is in:

$ hadoop mrhaadmin -getServiceState <id>

where <id> is one of the values you configured in the mapred.jobtrackers.<name> property – jt1 or jt2 in our sample mapred-site.xml files.

To transition one of the JobTrackers to active and then verify that it is active:

$ hadoop mrhaadmin -transitionToActive <id>
$ hadoop mrhaadmin -getServiceState <id>

where <id> is one of the values you configured in the mapred.jobtrackers.<name> property – jt1 or jt2 in our sample mapred-site.xml files.

Step 4: Verify that failover is working

Use the following commands, depending whether or not Kerberos is enabled.

If Kerberos is not enabled, use the following commands:

To cause a failover from the currently active to the currently inactive JobTracker:

$ sudo -u mapred hadoop mrhaadmin -failover <id_of_active_JobTracker> <id_of_inactive_JobTracker>

For example, if jt1 is currently active:

$ sudo -u mapred hadoop mrhaadmin -failover jt1 jt2 

To verify the failover:

$ sudo -u mapred hadoop mrhaadmin -getServiceState <id>

For example, if jt2 should now be active:

$ sudo -u mapred hadoop mrhaadmin -getServiceState jt2

With Kerberos enabled, use the following commands:

To log in as the mapred user and kinit:

$ sudo su - mapred
$ kinit -kt mapred.keytab mapred/<fully.qualified.domain.name>

To cause a failover from the currently active to the currently inactive JobTracker:

$ hadoop mrhaadmin -failover <id_of_active_JobTracker> <id_of_inactive_JobTracker>

For example, if jt1 is currently active:

$ hadoop mrhaadmin -failover jt1 jt2 

To verify the failover:

$ hadoop mrhaadmin -getServiceState <id>

For example, if jt2 should now be active:

$ hadoop mrhaadmin -getServiceState jt2

Step 1: Configure a ZooKeeper ensemble (if necessary)

To support automatic failover you need to set up a ZooKeeper ensemble running on three or more nodes, and verify its correct operation by connecting using the ZooKeeper command-line interface (CLI). See the ZooKeeper documentation for instructions on how to set up a ZooKeeper ensemble.

  Note: If you are already using a ZooKeeper ensemble for automatic failover, use the same ensemble for automatic JobTracker failover.

Step 2: Configure the parameters for manual failover

See the instructions for configuring the TaskTrackers and JobTrackers under Configuring and Deploying Manual Failover.

Step 3: Configure failover controller parameters

Use the following additional parameters to configure a failover controller for each JobTracker. The failover controller daemons run on the JobTracker nodes.

New configuration parameters

Add the following configuration information to mapred-site.xml:

  <property>
    <name>mapred.ha.automatic-failover.enabled</name>
    <value>true</value>
  </property>

  <property>
    <name>mapred.ha.zkfc.port</name>
    <value>8018</value> 
    <!-- Pick a different port for each failover controller when running one machine -->
  </property>

Add an entry similar to the following to core-site.xml:

<property>
    <name>ha.zookeeper.quorum</name>
    <value>zk1.example.com:2181,zk2.example.com:2181,zk3.example.com:2181 </value> 
    <!-- ZK ensemble addresses -->
</property>

  Note: If you have already configured automatic failover for HDFS, this property is already properly configured; you use the same ZooKeeper ensemble for HDFS and JobTracker HA.

Step 4: Initialize the HA State in ZooKeeper

After you have configured the failover controllers, the next step is to initialize the required state in ZooKeeper. You can do so by running one of the following commands from one of the JobTracker nodes.

  Note: The ZooKeeper ensemble must be running when you use this command; otherwise it will not work properly.

$ sudo service hadoop-0.20-mapreduce-zkfc init

or

$ sudo -u mapred hadoop mrzkfc -formatZK

This will create a znode in ZooKeeper in which the automatic failover system stores its data.

  Note: If you are running a secure cluster, see also Securing access to ZooKeeper.

Step 5: Enable automatic failover

To enable automatic failover once you have completed the configuration steps, you need only start the jobtrackerha and zkfc daemons.

To start the daemons, run the following commands on each JobTracker node:

$ sudo service hadoop-0.20-mapreduce-zkfc start
$ sudo service hadoop-0.20-mapreduce-jobtrackerha start

One of the JobTrackers will automatically transition to active.

Step 6: Verify automatic failover

After enabling automatic failover, you should test its operation. To do so, first locate the active JobTracker. To find out what state each JobTracker is in, use the following command:

$ sudo -u mapred hadoop mrhaadmin -getServiceState <id>

where <id> is one of the values you configured in the mapred.jobtrackers.<name> property – jt1 or jt2 in our sample mapred-site.xml files.

  Note: You must be the mapred user to use mrhaadmin commands.

Once you have located your active JobTracker, you can cause a failure on that node. For example, you can use kill -9 <pid of JobTracker> to simulate a JVM crash. Or you can power-cycle the machine or its network interface to simulate different kinds of outages. After you trigger the outage you want to test, the other JobTracker should automatically become active within several seconds. The amount of time required to detect a failure and trigger a failover depends on the configuration of ha.zookeeper.session-timeout.ms, but defaults to 5 seconds.

If the test does not succeed, you may have a misconfiguration. Check the logs for the zkfc and jobtrackerha daemons to diagnose the problem.