Injector Job深入分析

Injector Job的主要功能是根据crawlId在hbase中创建一个表，将将文本中的seed注入表中。
（一）命令执行
1、运行命令

[jediael@master local]$ bin/nutch inject seeds/ -crawlId sourcetest
InjectorJob: starting at 2015-03-10 14:59:19
InjectorJob: Injecting urlDir: seeds
InjectorJob: Using class org.apache.gora.hbase.store.HBaseStore as the Gora storage class.
InjectorJob: total number of urls rejected by filters: 0
InjectorJob: total number of urls injected after normalization and filtering: 1
Injector: finished at 2015-03-10 14:59:26, elapsed: 00:00:06

2、查看表中内容

hbase(main):004:0> scan 'sourcetest_webpage'
ROW                                       COLUMN+CELL                                                                                                           com.163.money:http/                      column=f:fi, timestamp=1425970761871, value=\x00'\x8D\x00                                                            com.163.money:http/                      column=f:ts, timestamp=1425970761871, value=\x00\x00\x01L\x02{\x08_                                                   com.163.money:http/                      column=mk:_injmrk_, timestamp=1425970761871, value=y                                                                 com.163.money:http/                      column=mk:dist, timestamp=1425970761871, value=0                                                                      com.163.money:http/                      column=mtdt:_csh_, timestamp=1425970761871, value=?\x80\x00\x00                                                       com.163.money:http/                      column=s:s, timestamp=1425970761871, value=?\x80\x00\x00
1 row(s) in 0.0430 seconds

3、读取数据库中的内容
由于hbase表使用了字节码表示内容，因此需要通过以下命令来查看具体内容

[jediael@master local]$ bin/nutch readdb  -dump ./test -crawlId sourcetest -content
WebTable dump: starting
WebTable dump: done
[jediael@master local]$ cat test/part-r-00000
http://money.163.com/   key:    com.163.money:http/
baseUrl:        null
status: 0 (null)
fetchTime:      1425970759775
prevFetchTime:  0
fetchInterval:  2592000
retriesSinceFetch:      0
modifiedTime:   0
prevModifiedTime:       0
protocolStatus: (null)
parseStatus:    (null)
title:  null
score:  1.0
marker _injmrk_ :       y
marker dist :   0
reprUrl:        null
metadata _csh_ :        ?锟

（二）源码流程分析
类：org.apache.nutch.crawl.InjectorJob
1、程序入口

public static void main(String[] args) throws Exception {int res = ToolRunner.run(NutchConfiguration.create(), new InjectorJob(),args);System.exit(res);}

2、ToolRunner.run(String[] args)
此步骤主要是调用inject方法，其余均是一些参数合规性的检查

public int run(String[] args) throws Exception {…………inject(new Path(args[0]));…………}

3、inject()方法
nutch均使用 Map<String, Object> run(Map<String, Object> args)来运行具体的job,即其使用Map类参数，并返回Map类参数。

<pre name="code" class="java">public void inject(Path urlDir) throws Exception {run(ToolUtil.toArgMap(Nutch.ARG_SEEDDIR, urlDir));}

4、job的具体配置，并创建hbase中的表格

public Map<String, Object> run(Map<String, Object> args) throws Exception {numJobs = 1;currentJobNum = 0;currentJob = new NutchJob(getConf(), "inject " + input);FileInputFormat.addInputPath(currentJob, input);currentJob.setMapperClass(UrlMapper.class);currentJob.setMapOutputKeyClass(String.class);currentJob.setMapOutputValueClass(WebPage.class);currentJob.setOutputFormatClass(GoraOutputFormat.class);DataStore<String, WebPage> store = StorageUtils.createWebStore(currentJob.getConfiguration(), String.class, WebPage.class);GoraOutputFormat.setOutput(currentJob, store, true);currentJob.setReducerClass(Reducer.class);currentJob.setNumReduceTasks(0);currentJob.waitForCompletion(true);ToolUtil.recordJobStatus(null, currentJob, results);
}

5、mapper方法
由于Injector Job中无reducer，因此只要关注mapper即可。
mapper主要完成以下几项工作：
（1）对文本中的内容进行分析，并提取其中的参数
（2）根据filter过滤url
（3）反转url作为key，创建Webpage对象作为value，然后将之写入表中。

protected void map(LongWritable key, Text value, Context context)throws IOException, InterruptedException {String url = value.toString().trim(); // value is line of textif (url != null && (url.length() == 0 || url.startsWith("#"))) {/* Ignore line that start with # */return;}// if tabs : metadata that could be stored// must be name=value and separated by \tfloat customScore = -1f;int customInterval = interval;Map<String, String> metadata = new TreeMap<String, String>();if (url.indexOf("\t") != -1) {String[] splits = url.split("\t");url = splits[0];for (int s = 1; s < splits.length; s++) {// find separation between name and valueint indexEquals = splits[s].indexOf("=");if (indexEquals == -1) {// skip anything without a =continue;}String metaname = splits[s].substring(0, indexEquals);String metavalue = splits[s].substring(indexEquals + 1);if (metaname.equals(nutchScoreMDName)) {try {customScore = Float.parseFloat(metavalue);} catch (NumberFormatException nfe) {}} else if (metaname.equals(nutchFetchIntervalMDName)) {try {customInterval = Integer.parseInt(metavalue);} catch (NumberFormatException nfe) {}} elsemetadata.put(metaname, metavalue);}}try {url = urlNormalizers.normalize(url, URLNormalizers.SCOPE_INJECT);url = filters.filter(url); // filter the url} catch (Exception e) {LOG.warn("Skipping " + url + ":" + e);url = null;}if (url == null) {context.getCounter("injector", "urls_filtered").increment(1);return;} else { // if it passesString reversedUrl = TableUtil.reverseUrl(url); // collect itWebPage row = WebPage.newBuilder().build();row.setFetchTime(curTime);row.setFetchInterval(customInterval);// now add the metadataIterator<String> keysIter = metadata.keySet().iterator();while (keysIter.hasNext()) {String keymd = keysIter.next();String valuemd = metadata.get(keymd);row.getMetadata().put(new Utf8(keymd),ByteBuffer.wrap(valuemd.getBytes()));}if (customScore != -1)row.setScore(customScore);elserow.setScore(scoreInjected);try {scfilters.injectedScore(url, row);} catch (ScoringFilterException e) {if (LOG.isWarnEnabled()) {LOG.warn("Cannot filter injected score for url " + url+ ", using default (" + e.getMessage() + ")");}}context.getCounter("injector", "urls_injected").increment(1);row.getMarkers().put(DbUpdaterJob.DISTANCE, new Utf8(String.valueOf(0)));Mark.INJECT_MARK.putMark(row, YES_STRING);context.write(reversedUrl, row);}}

（三）重点源码学习

Injector Job深入分析相关推荐

android AIDL IPC深入分析
深入分析AIDL原理博客分类: Android 在上一篇文章(Service使用方式)中,介绍了Android进程间通信(IPC)的使用,并给出了一个示例.但并没有深入分析aidl是怎样可以做到进程 ...
Linux堆内存管理深入分析(上)
Linux堆内存管理深入分析 (上半部) 作者:走位@阿里聚安全 0 前言近年来,漏洞挖掘越来越火,各种漏洞挖掘.利用的分析文章层出不穷.从大方向来看,主要有基于栈溢出的漏洞利用和基于堆溢出的漏 ...
深入分析Parquet列式存储格式
深入分析Parquet列式存储格式 Parquet是面向分析型业务的列式存储格式,由Twitter和Cloudera合作开发,2015年5月从Apache的孵化器里毕业成为Apache顶级项目,最新的 ...
GBDT算法原理深入分析
GBDT算法原理深入分析1 https://www.zybuluo.com/yxd/note/611571 GBDT算法原理深入分析2 https://www.wandouip.com/t5i1874 ...
深入分析 Jdk - 集合容器 Map 与 Set
SegmentFault 不兼容部分 markdown,详情请见深入分析 Jdk - 集合容器 Map 与 Set
PHP远程DoS漏洞深入分析及防护方案
5月14日,国内爆出php远程DoS漏洞,官方编号69364.利用该漏洞构造poc发起链接,很容易导致目标主机cpu的占用率100%,涉及PHP多个版本.绿盟科技威胁响应中心随即启动应急机制, 应急响 ...
angularjs源码笔记(3)--injector
2019独角兽企业重金招聘Python工程师标准>>> 简介 injector是用来做参数自动注入的,例如 function fn ($http, $scope, aService) ...
转帖-MySQL Innodb日志机制深入分析
为什么80%的码农都做不了架构师?>>> MySQL Innodb日志机制深入分析 http://blog.csdn.net/yunhua_lee/article/detail ...
javascript笔记：深入分析javascript里对象的创建（上）续篇
今天回来一看我的博客居然有这么多人推荐真是开心极了,看来大家对我的研究有了认可,写博客的动力越来越大了,而且我发现写javascript在博客园里比较受欢迎,写java的受众似乎少多了,可能博客园里j ...

Injector Job深入分析

Injector Job深入分析相关推荐

最新文章

热门文章