深入学习Heritrix---解析Frontier(链接工厂)
在Heritrix的官方文档上有一个Frontier的例子,虽然很简单,但是它却解释Frontier实现的基本原理.在这里就不讨论,有兴趣的读者可以参考相应文档.但是不得不提它的三个核心方法:
(1)next(int timeout):为处理线程提供一个链接.Heritrix的所有处理线程(ToeThread)都是通过调用该方法获取链接的.
(2)schedule(CandidateURI caURI):调度待处理的链接.
(3)finished(CrawlURI cURI):完成一个已处理的链接.
整体结构如下:
BdbMultipleWorkQueues:
它是对Berkeley DB的简单封装.在内部有一个Berkeley Database,存放所有待处理的链接.
package org.archive.crawler.frontier;
public class BdbMultipleWorkQueues
{
//存放所有待处理的URL的数据库
private Database pendingUrisDB = null;
//由key获取一个链接
public CrawlURI get(DatabaseEntry headKey)
throws DatabaseException {
DatabaseEntry result = new DatabaseEntry();
// From Linda Lee of sleepycat:
// "You want to check the status returned from Cursor.getSearchKeyRange
// to make sure that you have OperationStatus.SUCCESS. In that case,
// you have found a valid data record, and result.getData()
// (called by internally by the binding code, in this case) will be
// non-null. The other possible status return is
// OperationStatus.NOTFOUND, in which case no data record matched
// the criteria. "
//由key获取相应的链接
OperationStatus status = getNextNearestItem(headKey, result);
CrawlURI retVal = null;
if (status != OperationStatus.SUCCESS) {
LOGGER.severe("See '1219854 NPE je-2.0 "
+ "entryToObject'. OperationStatus "
+ " was not SUCCESS: "
+ status
+ ", headKey "
+ BdbWorkQueue.getPrefixClassKey(headKey.getData()));
return null;
}
try {
retVal = (CrawlURI)crawlUriBinding.entryToObject(result);
} catch (RuntimeExceptionWrapper rw) {
LOGGER.log(
Level.SEVERE,
"expected object missing in queue " +
BdbWorkQueue.getPrefixClassKey(headKey.getData()),
rw);
return null;
}
retVal.setHolderKey(headKey);
return retVal;//返回链接
}
//从等处理列表获取一个链接
protected OperationStatus getNextNearestItem(DatabaseEntry headKey,
DatabaseEntry result) throws DatabaseException {
Cursor cursor = null;
OperationStatus status;
try {
//打开游标
cursor = this.pendingUrisDB.openCursor(null, null);
// get cap; headKey at this point should always point to
// a queue-beginning cap entry (zero-length value)
status = cursor.getSearchKey(headKey, result, null);
if(status!=OperationStatus.SUCCESS || result.getData().length > 0) {
// cap missing
throw new DatabaseException("bdb queue cap missing");
}
// get next item (real first item of queue)
status = cursor.getNext(headKey,result,null);
} finally {
if(cursor!=null) {
cursor.close();
}
}
return status;
}
/** *//**
* Put the given CrawlURI in at the appropriate place.
* 添加URL到数据库
* @param curi
* @throws DatabaseException
*/
public void put(CrawlURI curi, boolean overwriteIfPresent)
throws DatabaseException {
DatabaseEntry insertKey = (DatabaseEntry)curi.getHolderKey();
if (insertKey == null) {
insertKey = calculateInsertKey(curi);
curi.setHolderKey(insertKey);
}
DatabaseEntry value = new DatabaseEntry();
crawlUriBinding.objectToEntry(curi, value);
// Output tally on avg. size if level is FINE or greater.
if (LOGGER.isLoggable(Level.FINE)) {
tallyAverageEntrySize(curi, value);
}
OperationStatus status;
if(overwriteIfPresent) {
//添加
status = pendingUrisDB.put(null, insertKey, value);
} else {
status = pendingUrisDB.putNoOverwrite(null, insertKey, value);
}
if(status!=OperationStatus.SUCCESS) {
LOGGER.severe("failed; "+status+ " "+curi);
}
}
}
BdbWorkQueue:
代表一个链接队列,该队列中所有的链接都具有相同的键值.它实际上是通过调用BdbMultipleWorkQueues的get方法从等处理链接数据库中取得一个链接的.
package org.archive.crawler.frontier;
public class BdbWorkQueue extends WorkQueue
implements Comparable, Serializabl
{
//获取一个URL
protected CrawlURI peekItem(final WorkQueueFrontier frontier)
throws IOException {
/**
* 关键:从BdbFrontier中返回pendingUris
*/
final BdbMultipleWorkQueues queues = ((BdbFrontier) frontier)
.getWorkQueues();
DatabaseEntry key = new DatabaseEntry(origin);
CrawlURI curi = null;
int tries = 1;
while(true) {
try {
//获取链接
curi = queues.get(key);
} catch (DatabaseException e) {
LOGGER.log(Level.SEVERE,"peekItem failure; retrying",e);
}
return curi;
}
}
WorkQueueFrontier:
实现了最核心的三个方法.
public CrawlURI next()
throws InterruptedException, EndedException {
while (true) { // loop left only by explicit return or exception
long now = System.currentTimeMillis();
// Do common checks for pause, terminate, bandwidth-hold
preNext(now);
synchronized(readyClassQueues) {
int activationsNeeded = targetSizeForReadyQueues() - readyClassQueues.size();
while(activationsNeeded > 0 && !inactiveQueues.isEmpty()) {
activateInactiveQueue();
activationsNeeded--;
}
}
WorkQueue readyQ = null;
Object key = readyClassQueues.poll(DEFAULT_WAIT,TimeUnit.MILLISECONDS);
if (key != null) {
readyQ = (WorkQueue)this.allQueues.get(key);
}
if (readyQ != null) {
while(true) { // loop left by explicit return or break on empty
CrawlURI curi = null;
synchronized(readyQ) {
/**取出一个URL,最终从子类BdbFrontier的
* pendingUris中取出一个链接
*/
curi = readyQ.peek(this);
if (curi != null) {
// check if curi belongs in different queue
String currentQueueKey = getClassKey(curi);
if (currentQueueKey.equals(curi.getClassKey())) {
// curi was in right queue, emit
noteAboutToEmit(curi, readyQ);
//加入正在处理队列中
inProcessQueues.add(readyQ);
return curi; //返回
}
// URI's assigned queue has changed since it
// was queued (eg because its IP has become
// known). Requeue to new queue.
curi.setClassKey(currentQueueKey);
readyQ.dequeue(this);//出队列
decrementQueuedCount(1);
curi.setHolderKey(null);
// curi will be requeued to true queue after lock
// on readyQ is released, to prevent deadlock
} else {
// readyQ is empty and ready: it's exhausted
// release held status, allowing any subsequent
// enqueues to again put queue in ready
readyQ.clearHeld();
break;
}
}
if(curi!=null) {
// complete the requeuing begun earlier
sendToQueue(curi);
}
}
} else {
// ReadyQ key wasn't in all queues: unexpected
if (key != null) {
logger.severe("Key "+ key +
" in readyClassQueues but not allQueues");
}
}
if(shouldTerminate) {
// skip subsequent steps if already on last legs
throw new EndedException("shouldTerminate is true");
}
if(inProcessQueues.size()==0) {
// Nothing was ready or in progress or imminent to wake; ensure
// any piled-up pending-scheduled URIs are considered
this.alreadyIncluded.requestFlush();
}
}
}
//将URL加入待处理队列
public void schedule(CandidateURI caUri) {
// Canonicalization may set forceFetch flag. See
// #canonicalization(CandidateURI) javadoc for circumstance.
String canon = canonicalize(caUri);
if (caUri.forceFetch()) {
alreadyIncluded.addForce(canon, caUri);
} else {
alreadyIncluded.add(canon, caUri);
}
}
BdbFrontier:
继承了WorkQueueFrontier,是Heritrix唯一个具有实际意义的链接工厂.
package org.archive.crawler.frontier;
public class BdbFrontier extends WorkQueueFrontier implements Serializable
{
/** 所有待抓取的链接*/
protected transient BdbMultipleWorkQueues pendingUris;
//初始化pendingUris,父类为抽象方法
protected void initQueue() throws IOException {
try {
this.pendingUris = createMultipleWorkQueues();
} catch(DatabaseException e) {
throw (IOException)new IOException(e.getMessage()).initCause(e);
}
}
private BdbMultipleWorkQueues createMultipleWorkQueues()
throws DatabaseException {
return new BdbMultipleWorkQueues(this.controller.getBdbEnvironment(),
this.controller.getBdbEnvironment().getClassCatalog(),
this.controller.isCheckpointRecover());
}
protected BdbMultipleWorkQueues getWorkQueues() {
return pendingUris;
}
}
BdbUriUniqFilter:
实际上是一个过滤器,它用来检查一个要进入等待队列的链接是否已经被抓取过.
//添加URL
protected boolean setAdd(CharSequence uri) {
DatabaseEntry key = new DatabaseEntry();
LongBinding.longToEntry(createKey(uri), key);
long started = 0;
OperationStatus status = null;
try {
if (logger.isLoggable(Level.INFO)) {
started = System.currentTimeMillis();
}
//添加到数据库
status = alreadySeen.putNoOverwrite(null, key, ZERO_LENGTH_ENTRY);
if (logger.isLoggable(Level.INFO)) {
aggregatedLookupTime +=
(System.currentTimeMillis() - started);
}
} catch (DatabaseException e) {
logger.severe(e.getMessage());
}
if (status == OperationStatus.SUCCESS) {
count++;
if (logger.isLoggable(Level.INFO)) {
final int logAt = 10000;
if (count > 0 && ((count % logAt) == 0)) {
logger.info("Average lookup " +
(aggregatedLookupTime / logAt) + "ms.");
aggregatedLookupTime = 0;
}
}
}
//如果存在,返回false
if(status == OperationStatus.KEYEXIST) {
return false; // not added
} else {
return true;
}
}
转载于:https://www.cnblogs.com/hustcat/archive/2008/10/23/1318203.html
深入学习Heritrix---解析Frontier(链接工厂)相关推荐
- python学习-列表解析、字典解析
文章目录 扩展链接 列表解析 字典解析 扩展链接 一些基础数据类型的博文链接: python学习-数据类型 列表 python学习-数据类型 字典 python学习-数据类型 元组 python学习- ...
- 用深度学习来解析梦境中出现的物体
这篇文章主要的工作算是机器学习和神经科学的结合工作,需要读者在这两个方向有一定的基础. 另有科普版本,结构更加清晰,不过删减了很多内容.科普版本如下: 用深度学习来解析梦境中出现的物体 - 行为与认知 ...
- LOAM学习-代码解析(三)特征点运动估计 laserOdometry
LOAM学习-代码解析(三)特征点运动估计 laserOdometry 前言 一.初始化 二.去除位移畸变 TransformToStart TransformToEnd 三.去除角度畸变 Plugi ...
- 分布式深度学习DDL解析
分布式深度学习DDL解析 一.概述 给一个庞大的GPU集群,在实际的应用中,现有的大数据调度器会导致长队列延迟和低的性能,该文章提出了Tiresias,即一个GPU集群的调度器,专门适应分布式深度学习 ...
- 深度学习常见问题解析
深度学习常见问题解析 计算机视觉与自动驾驶 今天 一.为什么深层神经网络难以训练? 1.梯度消失.梯度消失是指通过隐藏层从后向前看,梯度会变得越来越小,说明前面层的学习会显著慢于后面层的学习,所以学习 ...
- 【深度学习】解析神经网络中的数值稳定性、模型初始化和分布偏移(Pytorch)
[深度学习]解析神经网络中的数值稳定性.模型初始化和分布偏移 文章目录 1 概述1.1 梯度消失和梯度爆炸1.2 打破对称性 2 参数初始化 3 环境和分布偏移3.1 协变量偏移3.2 标签偏移3.3 ...
- 【深度学习】解析深度学习的集成方法
[深度学习]解析深度学习的集成方法 文章目录 1 前言 2 使用集成模型降低方差 3 如何集成神经网络模型3.1 Varying Training Data3.2 Varying Combinatio ...
- 一个学习Python的好链接
一个学习Python的好链接 找到了一个学习python的好博客~ 原博客地址:https://blog.csdn.net/pipisorry/article/details/37742423
- OCP 11G 051题库解析汇总链接
OCP 11G 051题库解析汇总链接 OCP 051题库 1 : OCP-1Z0-051 第1题 CTAS语句建表注意事项 2 : OCP-1Z0-051 第2题 视图的WITH CHEC ...
- 设计模式学习笔记-2 创建者模式-工厂方法模式
设计模式学习笔记-2 创建者模式-工厂方法模式 工厂模式介绍 工厂模式又称工厂方法模式,是一种创建型设计模式,其在父类中提供一个创建对象的方法,允许子类决定实例化对象的类型. 这种设计模式使Java开 ...
最新文章
- Ubuntu15.04 网站服务器环境搭建,php/html/css等学习环境搭建教程
- Android GIF 编解码
- round--向最近的整数取整
- factor--符号矩阵的因式分解
- 微软AirSim,一个无人机和机器人的模拟器
- cmd进入python环境_python2和python3同时存在,如何CMD中进入不同的环境
- Flask 跨域问题
- Spark 报错 : Error: bad symbolic reference. A signature in SparkContext.class refers to term conf
- 互联网+2.0:技术有多强 梦想才有多近
- Oracle笔记:数据库启动的三个阶段
- 如何使用Windows OVERLAPPED优化你的应用
- (28)XIlinx FPGA 原语简介(FPGA不积跬步101)
- Android Studio的怪错:AndroidManifest.xml unresolve symbol package/connot resolve symbol/Animations
- python机器学习-糖尿病数据挖掘
- php学生成绩管理系统完整源代码,PHP学生成绩管理系统
- python合并两个txt、并且逐条合并_用python实现两个文本合并
- 计算机二级考试谁监考,【考证】你们要的计算机二级考试攻略来了!!!
- 王垠面阿里P9,面跪后与P10赵海平互怼:人性最大的愚蠢,是互相为难
- linux系统ss命令详解,ss命令 - Linux命令大全 | linux教程
- 平板xmind怎么添加父主题_XMind8主题使用教程
热门文章
- jflash合并bin文件及hex文件
- 小米笔记本 镜像_小米笔记本Air 13.3原装出厂WIN10 2004 ISO镜像下载
- python爬虫出来空值_pandas | DataFrame基础运算以及空值填充
- Camtasia Studio2023专业的电脑屏幕录像视频编辑软件
- ailed to send crash report due to a network error: SocketException: OS Error: 信号灯超时时间已到 , errno = 12
- python mock server_Mock Server 入门
- Centos6 安装可视化界面
- 高通Android camera驱动框架
- Xiaojie雷达之路---TI实战笔记---OSAL详解
- Delphi开发Web的MVC框架