ubuntu下hadoop运行wordcount程序

本机环境

ubuntu 12

hadoop 1.1.2

首先保证hadoop配置成功

1、在Hadoop的解压目录的如下位置可以找到WordCount.java的源文件 src/examples/org/apache/hadoop/examples/WordCount.java

新建一个wordcount的文件夹，将WordCount.java拷贝至dev/wordcount文件夹下

2.编译wordcount.java

3.将生成的class文件打包

4.在wordcount下建立file01 file02两个文件

5.启动hadoop,在hdfs上创建input文件夹，并将两个输入文件上传至input文件夹

root@zcf-K42JZ:/usr/local/hadoop# bin/hadoop dfs -ls
ls: Cannot access .: No such file or directory.
root@zcf-K42JZ:/usr/local/hadoop# bin/hadoop dfs -mkdir input
root@zcf-K42JZ:/usr/local/hadoop# bin/hadoop dfs -ls
Found 1 items
drwxr-xr-x   - root supergroup          0 2014-03-04 17:48 /user/root/input
root@zcf-K42JZ:/usr/local/hadoop# bin/hadoop fs -put /home/zcf/ 桌面/file01 input
root@zcf-K42JZ:/usr/local/hadoop# bin/hadoop fs -put /home/zcf/ 桌面/file02 input
root@zcf-K42JZ:/usr/local/hadoop# bin/hadoop fs -ls input
Found 2 items
-rw-r--r--   1 root supergroup         22 2014-03-04 17:50 /user/root/input/file01
-rw-r--r--   1 root supergroup         28 2014-03-04 17:50 /user/root/input/file02

6.运行wordcount.jar

root@zcf-K42JZ:/usr/local/hadoop# bin/hadoop jar wordcount/wordcount.jar org.apache.hadoop.examples.WordCount input output
14/03/04 17:58:14 INFO input.FileInputFormat: Total input paths to process : 2
14/03/04 17:58:14 INFO util.NativeCodeLoader: Loaded the native-hadoop library
14/03/04 17:58:14 WARN snappy.LoadSnappy: Snappy native library not loaded
14/03/04 17:58:15 INFO mapred.JobClient: Running job: job_201403041744_0001
14/03/04 17:58:16 INFO mapred.JobClient:  map 0% reduce 0%
14/03/04 17:58:21 INFO mapred.JobClient:  map 50% reduce 0%
14/03/04 17:58:22 INFO mapred.JobClient:  map 100% reduce 0%
14/03/04 17:58:29 INFO mapred.JobClient:  map 100% reduce 33%
14/03/04 17:58:31 INFO mapred.JobClient:  map 100% reduce 100%
14/03/04 17:58:32 INFO mapred.JobClient: Job complete: job_201403041744_0001
14/03/04 17:58:32 INFO mapred.JobClient: Counters: 29
14/03/04 17:58:32 INFO mapred.JobClient:   Job Counters
14/03/04 17:58:32 INFO mapred.JobClient:     Launched reduce tasks=1
14/03/04 17:58:32 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=8421
14/03/04 17:58:32 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
14/03/04 17:58:32 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
14/03/04 17:58:32 INFO mapred.JobClient:     Launched map tasks=2
14/03/04 17:58:32 INFO mapred.JobClient:     Data-local map tasks=2
14/03/04 17:58:32 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=9155
14/03/04 17:58:32 INFO mapred.JobClient:   File Output Format Counters
14/03/04 17:58:32 INFO mapred.JobClient:     Bytes Written=41
14/03/04 17:58:32 INFO mapred.JobClient:   FileSystemCounters
14/03/04 17:58:32 INFO mapred.JobClient:     FILE_BYTES_READ=79
14/03/04 17:58:32 INFO mapred.JobClient:     HDFS_BYTES_READ=268
14/03/04 17:58:32 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=152857
14/03/04 17:58:32 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=41
14/03/04 17:58:32 INFO mapred.JobClient:   File Input Format Counters
14/03/04 17:58:32 INFO mapred.JobClient:     Bytes Read=50
14/03/04 17:58:32 INFO mapred.JobClient:   Map-Reduce Framework
14/03/04 17:58:32 INFO mapred.JobClient:     Map output materialized bytes=85
14/03/04 17:58:32 INFO mapred.JobClient:     Map input records=2
14/03/04 17:58:32 INFO mapred.JobClient:     Reduce shuffle bytes=85
14/03/04 17:58:32 INFO mapred.JobClient:     Spilled Records=12
14/03/04 17:58:32 INFO mapred.JobClient:     Map output bytes=82
14/03/04 17:58:32 INFO mapred.JobClient:     CPU time spent (ms)=2840
14/03/04 17:58:32 INFO mapred.JobClient:     Total committed heap usage (bytes)=306511872
14/03/04 17:58:32 INFO mapred.JobClient:     Combine input records=8
14/03/04 17:58:32 INFO mapred.JobClient:     SPLIT_RAW_BYTES=218
14/03/04 17:58:32 INFO mapred.JobClient:     Reduce input records=6
14/03/04 17:58:32 INFO mapred.JobClient:     Reduce input groups=5
14/03/04 17:58:32 INFO mapred.JobClient:     Combine output records=6
14/03/04 17:58:32 INFO mapred.JobClient:     Physical memory (bytes) snapshot=382898176
14/03/04 17:58:32 INFO mapred.JobClient:     Reduce output records=5
14/03/04 17:58:32 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=1164251136
14/03/04 17:58:32 INFO mapred.JobClient:     Map output records=8

7.查看运行结果

root@zcf-K42JZ:/usr/local/hadoop# bin/hadoop fs -ls
Found 2 items
drwxr-xr-x   - root supergroup          0 2014-03-04 17:50 /user/root/input
drwxr-xr-x   - root supergroup          0 2014-03-04 17:58 /user/root/output
root@zcf-K42JZ:/usr/local/hadoop# bin/hadoop fs -ls output
Found 3 items
-rw-r--r--   1 root supergroup          0 2014-03-04 17:58 /user/root/output/_SUCCESS
drwxr-xr-x   - root supergroup          0 2014-03-04 17:58 /user/root/output/_logs
-rw-r--r--   1 root supergroup         41 2014-03-04 17:58 /user/root/output/part-r-00000
root@zcf-K42JZ:/usr/local/hadoop# bin/hadoop fs -cat /output/part-r-00000
cat: File does not exist: /output/part-r-00000
root@zcf-K42JZ:/usr/local/hadoop# bin/hadoop fs -cat output/part-r-00000
Bye 1
Goodbye 1
Hadoop  2
Hello   2
World   2

至此，hadoop下的WordCount实例运行结束，如果还想重新运行一遍，这需把hadoop下的output文件夹删除，因为hadoop为了保证结果的正确性，存在输出的文件夹的话，就会报异常，异常如下

 ERROR security.UserGroupInformation: PriviledgedActionException as:
root cause:org.apache.hadoop.mapred.FileAlreadyExistsException:
Output directory output already exists

删除hdfs上的output文件

root@zcf-K42JZ:/usr/local/hadoop# bin/hadoop fs -rmr output
Deleted hdfs://localhost:9000/user/root/output

ubuntu下hadoop运行wordcount程序相关推荐

Ubantu下hadoop运行第一个例子wordcount过程
Ubantu12.04下hadoop-1.2.1运行第一个例子wordcount过程,分享一下将WordCount.java文件放在Hadoop安装目录下,并在安装目录下创建输入目录input,目录 ...
qt程序在Linux下字体乱了,解决linux/Ubuntu下Qt creater 界面程序在编译运行后无法显示中文或中文乱码问题！...
本文解决的主要是界面程序编译运行后无法显示中文的问题,如果在creater 中无法输入中文,下载个IBus或者搜狗之类的中文输入法即可解决! 首先说乱码问题,这个很好解决: 如果是在linux下打开W ...
Ubuntu下 Hadoop 1.2.1 配置安装
http://blog.csdn.net/wyswlp/article/details/10564847 尝试进入大数据领域开发,从目前最火的Hadoop入手是必须的,本文一步步记录了 Ubuntu ...
Linux系统下Hadoop运行环境搭建
1.安装ssh免密登录命令:ssh-keygen overwrite(覆盖写入)输入y 一路回车将生成的密钥发送到本机地址 ssh-copy-id localhost (若报错命令无法找到则需要安 ...
2、运行WordCount程序
转载:http://www.cnblogs.com/xia520pi/archive/2012/05/16/2504205.html 单词计数是最简单也是最能体现MapReduce思想的程序之一,可以 ...
运行wordcount程序
运行WordCount程序单词计数是最简单也是最能体现MapReduce思想的程序之一,可以称为MapReduce版"Hello World",该程序的完整代码可以在Hadoop ...
Spark在Yarn上运行Wordcount程序
过往记忆专注于Hadoop.Spark.Hive.Flume.Hbase,QQ交流群:138615359 首页 Hadoop Spark Hive Hbase Flume 面试题面试题数据结构算 ...
从安装Kafka服务到运行WordCount程序
之所以写这篇文章,是因为Kafka初学的同学在了解了Kafka的基本原理之后,希望在自己的机器上面运行最简单的wordCount的时候,从开始安装Kafka到找到合适的example源码最后到成功运行 ...
IDEA+Hadoop运行TriangleCount程序
IDEA+Hadoop运行MapReduce程序文章目录 IDEA+Hadoop运行MapReduce程序 IDEA新建项目项目配置新建Java类配置编译器运行程序参考博客 IDEA新建项 ...

ubuntu下hadoop运行wordcount程序

ubuntu下hadoop运行wordcount程序相关推荐

最新文章

热门文章