官网链接 https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html
单词计算代码官网链接 https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html#Example:_WordCount_v1.0

一、下载安装hadoop2.7.1

下述两种版本任选其一下载，建议下载第二种，不需要做加入.dll和.exe操作，但是空间大。我都下了，使用时，只需要把环境变量路径变一下即可。

1.版本hadoop2.7.1下载

链接:
https://archive.apache.org/dist/hadoop/core/hadoop-2.7.1/

emsp;解压 D:\Env\hadoop-2.7.1

2.其他版本下载

链接:
https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/
下面以2.10.1为例:

我安装的路径D:\Env\hadoop-2.10.2

把这个路径给复制了。
最后还要添加hadoop.dll和winutils.exe文件到你hadoop的bin目录下
再此处下载https://github.com/steveloughran/winutils
解压之后，找到对应版本的放进去

二、配置环境变量

D:\Env\hadoop-2.10.2

%HADOOP_HOME%\bin

三、IDEA运行单词计算

创建一个Maven项目，并且再pom.xml加入如下依赖:

<properties><java.version>1.8</java.version>
<!--        <hadoop.version>2.7.1</hadoop.version>--><hadoop.version>2.10.2</hadoop.version></properties><dependencies><dependency><groupId>org.apache.hadoop</groupId><artifactId>hadoop-common</artifactId><version>${hadoop.version}</version></dependency><dependency><groupId>org.apache.hadoop</groupId><artifactId>hadoop-hdfs</artifactId><version>${hadoop.version}</version></dependency><dependency><groupId>org.apache.hadoop</groupId><artifactId>hadoop-mapreduce-client-core</artifactId><version>${hadoop.version}</version></dependency><dependency><groupId>org.apache.hadoop</groupId><artifactId>hadoop-client</artifactId><version>${hadoop.version}</version></dependency><dependency><groupId>org.apache.hadoop</groupId><artifactId>hadoop-yarn-api</artifactId><version>${hadoop.version}</version></dependency></dependencies>

创建一个WordCount的类
官网上也有，只需要把如下改成自己本地的输入输出地址即可

测试文件: D:\\test\\a.txt


import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import java.io.IOException;
import java.util.StringTokenizer;public class WordCount {public static class TokenizerMapperextends Mapper<Object, Text,     Text, IntWritable> {private final static IntWritable one = new IntWritable(1);private Text word = new Text();@Overridepublic void map(Object key, Text value, Context context) throws IOException, InterruptedException {StringTokenizer itr = new StringTokenizer(value.toString());while (itr.hasMoreTokens()) {word.set(itr.nextToken());context.write(word, one);}}}public static class IntSumReducerextends Reducer<Text,IntWritable,Text,IntWritable> {private IntWritable result = new IntWritable();@Overridepublic void reduce(Text key, Iterable<IntWritable> values,Context context) throws IOException, InterruptedException {int sum = 0;for (IntWritable val : values) {sum += val.get();}result.set(sum);context.write(key, result);}}public static void main(String[] args) throws Exception {Configuration conf = new Configuration();Job job = Job.getInstance(conf, "word count");job.setJarByClass(WordCount.class);job.setMapperClass(TokenizerMapper.class);job.setCombinerClass(IntSumReducer.class);job.setReducerClass(IntSumReducer.class);job.setOutputKeyClass(Text.class);job.setOutputValueClass(IntWritable.class);FileInputFormat.addInputPath(job, new  Path(  "D:\\test\\a.txt" ));FileOutputFormat.setOutputPath(job, new Path("D:\\test\\b" ));System.exit(job.waitForCompletion(true) ? 0 : 1);}
}

运行代码:

Windows下载Hadoop,本地执行MapReduce相关推荐

Windows系统Hadoop下载安装
Windows系统Hadoop下载安装 Hadoop下载 Hadoop安装 Hadoop环境配置 Hadoop服务启动 Hadoop下载官网下载地址:https://mirrors.tuna.tsi ...
python部署到hadoop上_python实现mapreduce（2）——在hadoop中执行
目的:将python实现mapreduce(1)中的python脚本部署到hadoop上,实现mapreduce. 1. 修改脚本执行权限 [tianyc@TeletekHbase ~]$ which ...
【Python学习系列四】Python程序通过hadoop-streaming提交到Hadoop集群执行MapReduce
场景:将Python程序通过hadoop-streaming提交到Hadoop集群执行. 参考:http://www.michael-noll.com/tutorials/writing-an-had ...
Hadoop系列二：Hadoop单节点伪分布部署并执行mapreduce示例wordcount
HDFS在集群上实现分布式文件系统,MapReduce在集群上实现了分布式计算和任务处理.HDFS在MapReduce任务处理过程中提供了文件操作和存储等支持,MapReduce在HDFS的基础上实现 ...
windows 环境下Eclipse开发MapReduce环境设置
1.Hadoop开发环境简介 1.1 Hadoop集群简介 Java版本:jdk-6u31-linux-i586.bin Linux系统:CentOS6.0 Hadoop版本:hadoop-1.0.0 ...
hadoop系列四:mapreduce的使用(二)
转载请在页首明显处注明作者与出处一:说明此为大数据系列的一些博文,有空的话会陆续更新,包含大数据的一些内容,如hadoop,spark,storm,机器学习等. 当前使用的hadoop版本为2.6 ...
Hadoop中关于MapReduce的编程实例（过滤系统日志）
框架 Apache Hadoop:分布式处理架构,结合了 MapReduce(并行处理).YARN(作业调度)和HDFS(分布式文件系统); 一.下载Hadoop相关文件 1.在Hadoop官网上下 ...
Windows下Hadoop的环境安装[转]
1.下载并安装Cygwin,记得cygwin安装中要把SSH选择上,因为后面Hadoop会用到,不详述cygwin的安装过程.我是安装在D:\cygwin下 2.配置系统环境变量在windows命令 ...
windows上hadoop安装(cygwin等)
http://cjnetwork.iteye.com/blog/1474745 hadoop运行方式 1.本机方式:不做任何配置 2.伪分布式:pseudo distributed 3.全分布式:fu ...

Windows下载Hadoop,本地执行MapReduce