准备工作

确保已安装Hadoop并配置好环境变量。Hadoop版本建议使用2.7.x或更高版本，Java版本需为JDK 8或11。在IDE中创建Maven项目，添加以下依赖项到pom.xml：

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-client</artifactId>
    <version>2.7.7</version>
</dependency>

编写MapReduce程序

创建一个简单的单词计数程序，包含Mapper、Reducer和主驱动类。

Mapper类
实现org.apache.hadoop.mapreduce.Mapper，将输入文本拆分为单词并标记计数为1：

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

public class WordCountMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
    private final static IntWritable one = new IntWritable(1);
    private Text word = new Text();

    @Override
    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
        String[] words = value.toString().split("\\s+");
        for (String w : words) {
            word.set(w);
            context.write(word, one);
        }
    }
}

Reducer类
实现org.apache.hadoop.mapreduce.Reducer，汇总相同单词的计数：

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

public class WordCountReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
    @Override
    protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
        int sum = 0;
        for (IntWritable val : values) {
            sum += val.get();
        }
        context.write(key, new IntWritable(sum));
    }
}

驱动类
配置并提交MapReduce作业：

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class WordCountDriver {
    public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();
        Job job = Job.getInstance(conf, "word count");
        
        job.setJarByClass(WordCountDriver.class);
        job.setMapperClass(WordCountMapper.class);
        job.setReducerClass(WordCountReducer.class);
        
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);
        
        FileInputFormat.addInputPath(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));
        
        System.exit(job.waitForCompletion(true) ? 0 : 1);
    }
}

打包与运行

1. 使用Maven打包程序为JAR文件：

   mvn clean package
   

2. 上传输入文件到HDFS：

   hadoop fs -mkdir /input
   hadoop fs -put local_input.txt /input
   

3. 提交作业到Hadoop集群：

   hadoop jar target/your-jar-name.jar WordCountDriver /input /output
   

4. 查看输出结果：

   hadoop fs -cat /output/part-r-00000
   

关键注意事项

• 输入输出路径通过命令行参数传递，确保路径在HDFS中存在且无冲突。
• 使用IntWritable和Text等Hadoop序列化类型替代Java原生类型。
• 若在本地模式测试，需将core-site.xml和hdfs-site.xml放入项目的资源目录。