Hadoop – Map-Reduce 中的 Reducer

In Hadoop's Map-Reduce programming model, the Reducer is one of the key components that processes the intermediate output generated by the Mapper to produce the final output. In this article, we will dive into the details of the Reducer and explore its functionality, characteristics, and how it can be implemented in a Hadoop environment.

Characteristics of a Reducer

A Reducer performs three key tasks:

1. Aggregation: It aggregates the data generated by the Mapper by grouping them based on keys.

2. Processing: It processes each group of data generated by the Mapper to produce the final output.

3. Output: It generates the final output of the Map-Reduce job.

The Reducer takes a set of input data for each key generated by the Mapper and applies a user-defined function to produce a single output value for that key. The input data for a key is a list of values generated by the Mapper. The Reducer's job is to process this list of values and produce an aggregated output for the key.

Implementing a Reducer

A Reducer is implemented by extending the Reducer class provided by the Map-Reduce framework. The Reducer class takes four generic types as input arguments:

1. The input key type.

2. The input value type.

3. The output key type.

4. The output value type.

The Reducer's input key type and value type should match that of the Mapper's output key type and value type. The output key type and value type should match the final output key type and value type of the Map-Reduce job.

public class CustomReducer extends Reducer<Text, IntWritable, Text, IntWritable> {

    @Override
    public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
        //Aggregation
        int sum = 0;
        for (IntWritable value : values) {
            sum += value.get();
        }

        //Output
        context.write(key, new IntWritable(sum));
    }
}

The reduce() method is called once for each key generated by the Mapper. The method takes three arguments:

1. The key.

2. An iterable list of values generated by the Mapper for that key.

3. The output collector to write the final output.

The reduce() method performs the aggregation and output tasks described earlier. In the example above, the method aggregates the values by summing them up and outputs the final sum for the key.

Conclusion

The Reducer is a crucial component of the Hadoop Map-Reduce programming model. It performs the aggregation, processing, and output tasks required to produce the final output of the Map-Reduce job. By implementing a Reducer, programmers can leverage the power of Hadoop to process large amounts of data in a scalable and efficient manner.