DataFu is a collection of Apache Pig UDFs (User-Defined Functions) for statistical evaluation. They were developed by LinkedIn and are now open source under an Apache 2.0 license.

A number of usage examples and other information are available at https://github.com/linkedin/datafu.

To Use DataFu in a Parcel-deployed Cluster

If your cluster uses parcels, DataFu is installed for you. You need to register the JAR file prior to use with the following command.

REGISTER /opt/cloudera/parcels/CDH/lib/pig/datafu.jar

To Use DataFu in a Package-deployed Cluster:

1. Install the DataFu package:

   Operating system	Install command
   Red-Hat-compatible	sudo yum install pig-udf-datafu
   SLES	sudo zypper install pig-udf-datafu
   Debian or Ubuntu	sudo apt-get install pig-udf-datafu

   This puts the DataFu JAR file (for example, datafu-0.0.4-cdh5.0.0.jar) in /usr/lib/pig.

2. Register the JAR. Replace the <component_version> string with the current DataFu and CDH version numbers.

   REGISTER /usr/lib/pig/datafu-<DataFu_version>-cdh<CDH_version>.jar

   For example:

   REGISTER /usr/lib/pig/datafu-0.0.4-cdh5.0.0.jar