PFPGrowth Class

This job is an implementation of the Parallel FP Growth algorithm described in the paper "PFP: Parallel FP-Growth for Query Recommendation" by Li et al., 2008.

This algorithm calculates the top-K frequent patterns for each item in the database, only regarding patterns that have the specified minimum support.

The algorithm has three steps: first, it counts how often each item occurs in the input database, filters out the infrequent features, and divides the resulting feature list into groups. Next, it generates group-dependent transactions from the input and runs the FP-Growth algorithm on each group. Finally, the results from each group are aggregated to form the final result.

The number of groups should be carefully selected so that the number of items per group it not too large. Ideally, each group should have 5-10 items at most for a large database.

The input for this job should be a plain text file (or files) where each line represents a transaction containing a space-delimited list of transactions.

This example demonstrates a more complicated Jumbo job, with several stages including more than one stage with file input. It uses scheduling dependencies, group aggregation, partition-based grouping using multiple partitions per task, dynamic partition assignment, and custom progress providers.

Reference