1.案例描述

实例中给出child-parent（孩子——父母）表，要求输出grandchild-grandparent（孙子——爷奶）表。

样例输入如下所示。

file：

child parent

Tom Lucy

Tom Jack

Jone Lucy

Jone Jack

Lucy Mary

Lucy Ben

Jack Alice

Jack Jesse

Terry Alice

Terry Jesse

Philip Terry

Philip Alma

Mark Terry

Mark Alma

家族树状关系谱：

图1 家族谱

样例输出如下所示。

file：

grandchild grandparent

Tom Alice

Tom Jesse

Jone Alice

Jone Jesse

Tom Mary

Tom Ben

Jone Mary

Jone Ben

Philip Alice

Philip Jesse

Mark Alice

Mark Jesse

2. 案例分析

分析这个实例，显然需要进行单表连接，连接的是左表的parent列和右表的child列，且左表和右表是同一个表。

　　连接结果中除去连接的两列就是所需要的结果——"grandchild--grandparent"表。要用MapReduce解决这个实例，首先应该考虑如何实现表的自连接；其次就是连接列的设置；最后是结果的整理。

考虑到MapReduce的shuffle过程会将相同的key会连接在一起，所以可以将map结果的key设置成待连接的列，然后列中相同的值就自然会连接在一起了。再与最开始的分析联系起来：

　　要连接的是左表的parent列和右表的child列，且左表和右表是同一个表，所以在map阶段将读入数据分割成child和parent之后，会将parent设置成key，child设置成value进行输出，并作为左表；再将同一对child和parent中的child设置成key，parent设置成value进行输出，作为右表。为了区分输出中的左右表，需要在输出的value中再加上左右表的信息，比如在value的String最开始处加上字符1表示左表，加上字符2表示右表。这样在map的结果中就形成了左表和右表，然后在shuffle过程中完成连接。reduce接收到连接的结果，其中每个key的value-list就包含了"grandchild--grandparent"关系。取出每个key的value-list进行解析，将左表中的child放入一个数组，右表中的parent放入一个数组，然后对两个数组求笛卡尔积就是最后的结果了。

3. 程序代码

package hadoop_STjoin;

import java.io.IOException;

import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Job;

import org.apache.hadoop.mapreduce.Mapper;

import org.apache.hadoop.mapreduce.Reducer;

importorg.apache.hadoop.mapreduce.lib.input.FileInputFormat;

importorg.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

import org.apache.hadoop.util.GenericOptionsParser;

public class SingleTableJoin {

publicstatic int times = 0;

* map将输出分割child和parent，然后正序输出一次作为右表，

* 反序输出一次作为左表，需要注意的是在输出的value中必须

* 加上左右表的区别标识。

publicstatic class Map extends Mapper<Object, Text, Text, Text>

{

protectedvoid map(Object key, Text value, Context context)

throwsIOException, InterruptedException {

StringchildName = new String(); //孩子名字

StringparentName = new String(); //父母名字

StringrelationType = new String(); //左右表标志

//解析每一行

StringTokenizeriter = new StringTokenizer(value.toString());

String[]values = new String[2];

inti=0;

while(iter.hasMoreTokens())

{

values[i]= iter.nextToken();

i++;

}

if(i == 2 && values[0].compareTo("child") != 0) {

childName= values[0];

parentName= values[1];

//输出左表

relationType= "1";

context.write(newText(parentName),newText(relationType+"+"+childName+"+"+parentName));

//输出右表

relationType= "2";

context.write(newText(childName), newText(relationType+"+"+childName+"+"+parentName));

}

//reduce

publicstatic class Reduce extends Reducer<Text, Text, Text, Text>

{

protectedvoid reduce(Text key, Iterable<Text> values, Context cont)

throwsIOException, InterruptedException {

if(times == 0) {

cont.write(newText("grandChild"), new Text("grandParen"));

times++;

}

intgrandChildNum = 0;

String[]grandChild = new String[10];

intgrandParentNum = 0;

String[]grandParent = new String[10];

System.out.println("key="+key.toString());

for(Texttemp : values)

{

Stringrecord = temp.toString();

System.out.println("record="+record);

if(record.length() == 0) {

System.out.println("continue;");

continue;

}

StringchildName = new String();

StringparenName = new String();

inti = 2;

while(record.charAt(i) != '+')

{

childName+= record.charAt(i);

i++;

}

i++;

while(i< record.length())

{

parenName+= record.charAt(i);

i++;

}

charrelationType = record.charAt(0);

//左表

if('1' == relationType) {

grandChild[grandChildNum]= childName;

grandChildNum++;

}

if('2' == relationType) {

grandParent[grandParentNum]= parenName;

grandParentNum++;

}

System.out.println("grandChildNum="+grandChildNum+"grandParentNum="+grandParentNum);

//grandchild和grandparent数组求笛卡尔儿积

if(grandChildNum != 0 && grandParentNum != 0) {

for(intm = 0; m < grandChildNum; m++)

{

for(intn = 0; n < grandParentNum; n++)

{

cont.write(newText(grandChild[m]), new Text(grandParent[n]));

}

//main

publicstatic void main(String[] args) throws Exception {

Configurationconf = new Configuration();

String[] otherArgs = newGenericOptionsParser(conf, args).getRemainingArgs();

if(otherArgs.length != 2) {

System.err.println("Usage:Single table join <in> <out>");

System.exit(2);

}

Jobjob = new Job(conf, "single table join");

job.setJarByClass(SingleTableJoin.class);

job.setMapperClass(Map.class);

job.setReducerClass(Reduce.class);

job.setOutputKeyClass(Text.class);

job.setOutputValueClass(Text.class);

FileInputFormat.addInputPath(job,new Path(otherArgs[0]));

FileOutputFormat.setOutputPath(job,new Path(otherArgs[1]));

System.exit(job.waitForCompletion(true)?0 : 1);

}

4.程序启动

root@node1:/usr/local/hadoop/hadoop-2.5.2/myJar# hadoop jar SingleTableJoin.jarhadoop_STjoin.SingleTableJoin /usr/local/hadooptempdata/input/stjoin/usr/local/hadooptempdata/output/stjoin

16/12/28 22:38:22 INFO client.RMProxy: Connectingto ResourceManager at node1/192.168.233.129:8032

16/12/28 22:38:25 INFO input.FileInputFormat: Totalinput paths to process : 1

16/12/28 22:38:25 INFO mapreduce.JobSubmitter:number of splits:1

16/12/28 22:38:25 INFO mapreduce.JobSubmitter:Submitting tokens for job: job_1482934139129_0003

16/12/28 22:38:26 INFO impl.YarnClientImpl:Submitted application application_1482934139129_0003

16/12/28 22:38:26 INFO mapreduce.Job: The url totrack the job: http://node1:8088/proxy/application_1482934139129_0003/

16/12/28 22:38:26 INFO mapreduce.Job: Running job:job_1482934139129_0003

16/12/28 22:39:16 INFO mapreduce.Job: Jobjob_1482934139129_0003 running in uber mode : false

16/12/28 22:39:16 INFO mapreduce.Job: map 0% reduce 0%

16/12/28 22:39:53 INFO mapreduce.Job: map 100% reduce 0%

16/12/28 22:40:34 INFO mapreduce.Job: map 100% reduce 100%

16/12/28 22:40:36 INFO mapreduce.Job: Jobjob_1482934139129_0003 completed successfully

16/12/28 22:40:36 INFO mapreduce.Job: Counters: 49

FileSystem Counters

FILE:Number of bytes read=565

FILE:Number of bytes written=198627

FILE:Number of read operations=0

FILE:Number of large read operations=0

FILE:Number of write operations=0

HDFS:Number of bytes read=406

HDFS:Number of bytes written=148

HDFS:Number of read operations=6

HDFS:Number of large read operations=0

HDFS:Number of write operations=2

JobCounters

Launchedmap tasks=1

Launchedreduce tasks=1

Data-localmap tasks=1

Totaltime spent by all maps in occupied slots (ms)=34937

Totaltime spent by all reduces in occupied slots (ms)=37274

Totaltime spent by all map tasks (ms)=34937

Totaltime spent by all reduce tasks (ms)=37274

Totalvcore-seconds taken by all map tasks=34937

Totalvcore-seconds taken by all reduce tasks=37274

Totalmegabyte-seconds taken by all map tasks=35775488

Totalmegabyte-seconds taken by all reduce tasks=38168576

Map-ReduceFramework

Mapinput records=15

Mapoutput records=28

Mapoutput bytes=503

Mapoutput materialized bytes=565

Inputsplit bytes=124

Combineinput records=0

Combineoutput records=0

Reduceinput groups=12

Reduceshuffle bytes=565

Reduceinput records=28

Reduceoutput records=13

SpilledRecords=56

ShuffledMaps =1

FailedShuffles=0

MergedMap outputs=1

GCtime elapsed (ms)=254

CPUtime spent (ms)=2270

Physicalmemory (bytes) snapshot=298164224

Virtualmemory (bytes) snapshot=3775070208

Totalcommitted heap usage (bytes)=139894784

ShuffleErrors

BAD_ID=0

CONNECTION=0

IO_ERROR=0

WRONG_LENGTH=0

WRONG_MAP=0

WRONG_REDUCE=0

FileInput Format Counters

BytesRead=282

FileOutput Format Counters

BytesWritten=148

5. 输出结果

root@node1:/usr/local/hadoop/hadoop-2.5.2/myJar# hdfs dfs -cat /usr/local/hadooptempdata/output/stjoin/*

grandChild grandParen

Tom Alice

Tom Jesse

Jone Alice

Jone Jesse

Tom Ben

Tom Mary

Jone Ben

Jone Mary

Philip Alice

Philip Jesse

Mark Alice

Mark Jesse

Hadoop案例之单表关联输出祖孙关系相关推荐

Hadoop入门（十六）Mapreduce的单表关联程序
"单表关联"要求从给出的数据中寻找所关心的数据,它是对原始数据所包含信息的挖掘 1 实例描述给出child-parent(孩子--父母)表,要求输出grandchild-gran ...
Spark入门（十七）之单表关联
一.单表关联给出child-parent(孩子--父母)表,要求输出grandchild-grandparent(孙子--祖父母)表二.maven设置 <?xml version=" ...
MapReduce实例----单表关联
1.源数据: Tom Lucy Tom Jack Jone Lucy Jone Jack Lucy Mary Lucy Ben Jack Alice Jack Jesse Terry Alice Te ...
Hadoop 学习之单表连接
我在学习hadoop, 在看陆嘉恒编著的hadoop实战,其中有单表连接的程序,我现在整理一下思路.这个问题是课本上的例子. 给出 child-parent 表, 要求输出 grandchild-g ...
MapReduce编程(五) 单表关联
一.问题描述下面给出一个child-parent的表格,要求挖掘其中的父子辈关系,给出祖孙辈关系的表格. 输入文件内容如下: child parent Steven Lucy Steven Jack ...
MapRedece(单表关联)
源数据:Child--Parent表 Tom Lucy Tom Jack Jone Lucy Jone Jack Lucy Marry Lucy Ben Jack Alice Jack Jesse T ...
hadoop之MapReduce的案例（多表关联）
order_detail.txt item_id item_type sp001 type001 sp002 type002 sp003 type002 iteminfo.tx ...
MapReduce编程系列 — 5：单表关联
1.项目名称: 2.项目数据: chile parent Tom Lucy Tom Jack Jone Lucy Jone Jack Lucy Mary Lucy ...
mysql 单表关联_MySQL 基础之单表、多表联查
使用和不使用not null 的区别: 不使用: 查询时用'name is null' 作为条件 mysql>create table t8( -> id int auto_increme ...

Hadoop案例之单表关联输出祖孙关系

1.案例描述

2. 案例分析

3. 程序代码

4.程序启动

5. 输出结果

Hadoop案例之单表关联输出祖孙关系相关推荐

最新文章

热门文章