＃yyds干货盘点＃MapReduce初级编程实践

＃yyds干货盘点＃MapReduce初级编程实践

目的

1.通过实验掌握基本的MapReduce编程方法；

2.掌握用MapReduce解决一些常见的数据处理问题。

平台

已经配置完成的Hadoop伪分布式环境。

实验内容和要求

假设HDFS中/user/hadoop/input文件夹下有文件wordfile1.txt和wordfile2.txt。现在需要设计一个词频统计程序，统计input文件夹下所有文件中每个单词的出现次数。

1、使用Eclipse编译运行MapReduce程序（运行过程结果截图）

参考链接：

cp /usr/local/hadoop/etc/hadoop/hdfs-site.xml ~/workspace/WordCount/src

cp /usr/local/hadoop/etc/hadoop/log4j.properties ~/workspace/WordCount/src

编写java代码

package org.apache.hadoop.examples; import java.io.IOException; import java.util.Iterator; import java.util.StringTokenizer; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.hadoop.util.GenericOptionsParser;   public class WordCount {     public WordCount() {     }       public static void main(String[] args) throws Exception {      Configuration conf = new Configuration();         String[] otherArgs = (new GenericOptionsParser(conf, args)).getRemainingArgs();         if (otherArgs.length < 2) {             System.err.println("Usage: wordcount [...] ");             System.exit(2);         }         Job job = Job.getInstance(conf, "word count");         job.setJarByClass(WordCount.class);         job.setMapperClass(WordCount.TokenizerMapper.class);         job.setCombinerClass(WordCount.IntSumReducer.class);         job.setReducerClass(WordCount.IntSumReducer.class);         job.setOutputKeyClass(Text.class);         job.setOutputValueClass(IntWritable.class);         for (int i = 0; i < otherArgs.length - 1; ++i) {             FileInputFormat.addInputPath(job, new Path(otherArgs[i]));         }         FileOutputFormat.setOutputPath(job, new Path(otherArgs[otherArgs.length - 1]));         System.exit(job.waitForCompletion(true) ? 0 : 1);     }       public static class TokenizerMapper extends Mapper {         private static final IntWritable one = new IntWritable(1);         private Text word = new Text();           public TokenizerMapper() {         }           public void map(Object key, Text value, Mapper.Context context) throws IOException, InterruptedException {             StringTokenizer itr = new StringTokenizer(value.toString());             while (itr.hasMoreTokens()) {                 this.word.set(itr.nextToken());                 context.write(this.word, one);             }         }     }       public static class IntSumReducer extends Reducer {         private IntWritable result = new IntWritable();           public IntSumReducer() {         }           public void reduce(Text key, Iterable values, Reducer.Context context) throws IOException, InterruptedException {             int sum = 0;             IntWritable val;             for (Iterator i$ = values.iterator(); i$.hasNext(); sum += val.get()) {                 val = (IntWritable)i$.next();             }             this.result.set(sum);             context.write(key, this.result);         }     } }

运行出现如下结果

第二，打包

右击项目->Export

第二，创建两个文件

上传到hadoop

第四

词频统计结果被写入到HDFS的/user/hadoop/output/目录中，运行结束后查看词频统计结果

./bin/hadoop jar ./myapp/WordCount.jar input output

./bin/hdfs dfs -cat output/*

四、实验中遇到的问题及解决方案

问题一

原因：output文件已存在

解决方案：删除output文件

问题二

Exception in thread "main" org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://localhost:9000/user/hadoop/input

原因：在路径hdfs://localhost:9000/user/hadoop/input下，找不到input文件

解决方案：需要将core-site.xml的如下内容配置正确

问题三

hadoop_eclipse-plugin-2.6.0.jar插件导入失败，导致进入eclipse没有DFS Location

并且也没有Hadoop Map/Reduce

原因：eclipse版本与插件版本不匹配

解决方案：之前用的eclipse是用安装包安装的，无法与插件适配。在虚拟机自带的应用商店下载了ecipse后成功适配

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。