目的

1.通过实验掌握基本的MapReduce编程方法；

2.掌握用MapReduce解决一些常见的数据处理问题。

平台

已经配置完成的Hadoop伪分布式环境。

实验内容和要求

假设HDFS中/user/hadoop/input文件夹下有文件wordfile1.txt和wordfile2.txt。现在需要设计一个词频统计程序，统计input文件夹下所有文件中每个单词的出现次数。

1、使用Eclipse编译运行MapReduce程序（运行过程结果截图）

参考链接：http://dblab.xmu.edu.cn/blog/hadoop-build-project-using-eclipse/

2、使用命令行编译打包运行自己的MapReduce程序（运行过程结果截图）

参考链接：http://dblab.xmu.edu.cn/blog/hadoop-build-project-by-shell/

实验步骤

第一，首先创建一个MapReduce项目

通过以下命令

cp /usr/local/hadoop/etc/hadoop/core-site.xml ~/workspace/WordCount/src

cp /usr/local/hadoop/etc/hadoop/hdfs-site.xml ~/workspace/WordCount/src

cp /usr/local/hadoop/etc/hadoop/log4j.properties ~/workspace/WordCount/src

编写java代码

package org.apache.hadoop.examples;


import java.io.IOException;

import java.util.Iterator;

import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Job;

import org.apache.hadoop.mapreduce.Mapper;

import org.apache.hadoop.mapreduce.Reducer;

import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;

import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

import org.apache.hadoop.util.GenericOptionsParser;

 

public class WordCount {

    public WordCount() {

    }

 

    public static void main(String[] args) throws Exception {

     Configuration conf = new Configuration();

        String[] otherArgs = (new GenericOptionsParser(conf, args)).getRemainingArgs();

        if (otherArgs.length < 2) {

            System.err.println("Usage: wordcount <in> [<in>...] <out>");

            System.exit(2);

        }

        Job job = Job.getInstance(conf, "word count");

        job.setJarByClass(WordCount.class);

        job.setMapperClass(WordCount.TokenizerMapper.class);

        job.setCombinerClass(WordCount.IntSumReducer.class);

        job.setReducerClass(WordCount.IntSumReducer.class);

        job.setOutputKeyClass(Text.class);

        job.setOutputValueClass(IntWritable.class);

        for (int i = 0; i < otherArgs.length - 1; ++i) {

            FileInputFormat.addInputPath(job, new Path(otherArgs[i]));

        }

        FileOutputFormat.setOutputPath(job, new Path(otherArgs[otherArgs.length - 1]));

        System.exit(job.waitForCompletion(true) ? 0 : 1);

    }

 

    public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable> {

        private static final IntWritable one = new IntWritable(1);

        private Text word = new Text();

 

        public TokenizerMapper() {

        }

 

        public void map(Object key, Text value, Mapper<Object, Text, Text, IntWritable>.Context context) throws IOException, InterruptedException {

            StringTokenizer itr = new StringTokenizer(value.toString());

            while (itr.hasMoreTokens()) {

                this.word.set(itr.nextToken());

                context.write(this.word, one);

            }

        }

    }

 

    public static class IntSumReducer extends Reducer<Text, IntWritable, Text, IntWritable> {

        private IntWritable result = new IntWritable();

 

        public IntSumReducer() {

        }

 

        public void reduce(Text key, Iterable<IntWritable> values, Reducer<Text, IntWritable, Text, IntWritable>.Context context) throws IOException, InterruptedException {

            int sum = 0;

            IntWritable val;

            for (Iterator i$ = values.iterator(); i$.hasNext(); sum += val.get()) {

                val = (IntWritable)i$.next();

            }

            this.result.set(sum);

            context.write(key, this.result);

        }

    }

}

运行出现如下结果

第二，打包

右击项目->Export

第二，创建两个文件

上传到hadoop

第四

词频统计结果被写入到HDFS的/user/hadoop/output/目录中，运行结束后查看词频统计结果

./bin/hadoop jar ./myapp/WordCount.jar input output

./bin/hdfs dfs -cat output/*

四、实验中遇到的问题及解决方案

问题一

原因：output文件已存在

解决方案：删除output文件

问题二

Exception in thread “main” org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://localhost:9000/user/hadoop/input

原因：在路径hdfs://localhost:9000/user/hadoop/input下，找不到input文件

解决方案：需要将core-site.xml的如下内容配置正确

问题三

hadoop_eclipse-plugin-2.6.0.jar插件导入失败，导致进入eclipse没有DFS Location

并且也没有Hadoop Map/Reduce

原因：eclipse版本与插件版本不匹配

解决方案：之前用的eclipse是用安装包安装的，无法与插件适配。在虚拟机自带的应用商店下载了ecipse后成功适配