【Nutch2.2.1基础教程之3】Nutch2.2.1配置文件

nutch-site.xml

在nutch2.2.1中，有两份配置文件：nutch-default.xml与nutch-site.xml。

其中前者是nutch自带的默认属性，一般情况下不要修改。

如果需要修改默认属性，可以在nutch-site.xml中增加一个同名的属性，并修改其值。nutch-site.xml中的属性值会覆盖nutch-default.xml中的值。

1、db.ignore.external.links

若为true，则只抓取本域名内的网页，忽略外部链接。

可以在 regex-urlfilter.txt中增加过滤器达到同样效果，但如果过滤器过多，如几千个，则会大大影响nutch的性能。

<property><name>db.ignore.external.links</name><value>true</value><description>If true, outlinks leading from a page to external hostswill be ignored. This is an effective way to limit the crawl to includeonly initially injected hosts, without creating complex URLFilters.</description>
</property>

2、fetcher.parse

能否在抓取的同时进行解释：可以，但不建议这样做。

<property><name>fetcher.parse</name><value>false</value><description>If true, fetcher will parse content. NOTE: previous releases woulddefault to true. Since 2.0 this is set to false as a safer default.</description>
</property>

官方解释

N.B. In a parsing fetcher, outlinks are processed in the reduce phase (at least when outlinks are followed). If a fetcher's reducer stalls you may run out of memory or disk space, usually after a very long reduce job. Behaviour typical to this is usually observed in this situation.

In summary, if it is possible, users are advised not to use a parsing fetcher as it is heavy on IO and often leads to the above outcome.

3、db.max.outlinks.per.page

默认情况下，Nutch只抓取某个网页的100个外部链接，导致部分链接无法抓取。若要改变此情况，可以修改此配置项。

<property><name>db.max.outlinks.per.page</name><value>100</value><description>The maximum number of outlinks that we'll process for a page.  If this value is nonnegative (>=0), at most db.max.outlinks.per.page outlinks  will be processed for a page; otherwise, all outlinks will be processed.</description>
</property>

官方说明如下：http://wiki.apache.org/nutch/FAQ/

Nutch doesn't crawl relative URLs? Some pages are not indexed but my regex file and everything else is okay - what is going on?

The crawl tool has a default limitation of 100 outlinks of one page that are being fetched. To overcome this limitation change thedb.max.outlinks.per.page property to a higher value or simply -1 (unlimited).

file: conf/nutch-default.xml

 <property>
   <name>db.max.outlinks.per.page</name>
   <value>-1</value>
   <description>The maximum number of outlinks that we'll process for a page.
   If this value is nonnegative (>=0), at most db.max.outlinks.per.page outlinks
   will be processed for a page; otherwise, all outlinks will be processed.
   </description>
 </property>

see also: http://www.mail-archive.com/nutch-user@lucene.apache.org/msg08665.html

4、file.content.limit http.content.limit ftp.content.limit

默认情况下，nutch只抓取网页的前65536个字节，之后的内容将被丢弃。
但对于某些大型网站，首页的内容远远不止65536个字节，甚至前面65536个字节里面均是一些布局信息，并没有任何的超链接。
因此修改默认值如下：

<property><name>file.content.limit</name><value>-1</value><description>The length limit for downloaded content using the fileprotocol, in bytes. If this value is nonnegative (>=0), content longerthan it will be truncated; otherwise, no truncation at all. Do notconfuse this setting with the http.content.limit setting.</description>
</property><property><name>http.content.limit</name><value>-1</value><description>The length limit for downloaded content using the httpprotocol, in bytes. If this value is nonnegative (>=0), content longerthan it will be truncated; otherwise, no truncation at all. Do notconfuse this setting with the file.content.limit setting.</description>
</property><property><name>ftp.content.limit</name><value>-1</value>   <description>The length limit for downloaded content, in bytes.If this value is nonnegative (>=0), content longer than it will be truncated;otherwise, no truncation at all.Caution: classical ftp RFCs never defines partial transfer and, in fact,some ftp servers out there do not handle client side forced close-down verywell. Our implementation tries its best to handle such situations smoothly.</description>
</property>

【Nutch2.2.1基础教程之3】Nutch2.2.1配置文件相关推荐

【Nutch2.2.1基础教程之2.2】集成Nutch/Hbase/Solr构建搜索引擎之二：内容分析
请先参见"集成Nutch/Hbase/Solr构建搜索引擎之一:安装及运行",搭建测试环境 http://blog.csdn.net/jediael_lu/article/deta ...
【Nutch2.2.1基础教程之2.1】集成Nutch/Hbase/Solr构建搜索引擎之一：安装及运行【单机环境】...
1.下载相关软件,并解压版本号如下: (1)apache-nutch-2.2.1 (2) hbase-0.90.4 (3)solr-4.9.0 并解压至/usr/search 2.Nutch的配置 ...
【Nutch2.2.1基础教程之6】Nutch2.2.1抓取流程
一.抓取流程概述 1.nutch抓取流程当使用crawl命令进行抓取任务时,其基本流程步骤如下: (1)InjectorJob 开始第一个迭代 (2)GeneratorJob (3)FetcherJ ...
【Nutch2.2.1基础教程之1】nutch相关异常
1.在任务一开始运行,注入Url时即出现以下错误. InjectorJob: Injecting urlDir: urls InjectorJob: Using class org.apache.go ...
【Nutch2.2.1基础教程之2.1】集成Nutch/Hbase/Solr构建搜索引擎之一：安装及运行【单机环境】
1.下载相关软件,并解压版本号如下: (1)apache-nutch-2.2.1 (2) hbase-0.90.4 (3)solr-4.9.0 并解压至/usr/search 2.Nutch的配置 ...
pgsql数据库默认配置事务类型_PostgreSQL基础教程之：初始化配置
PostgreSQL基础教程之:初始化配置时间:2020-04-27 来源: PostgreSQL基础教程之:初始化配置一.配置pg_hba.conf 先说明客户端认证配置文件pg_hba.con ...
Linux入门基础教程之Linux下软件安装
Linux入门基础教程之Linux下软件安装一.在线安装: sudo apt-get install 即可安装如果在安装完后无法用Tab键补全命令,可以执行: source ~/.zshrc AP ...
python可以处理多大的数据_科多大数据之Python基础教程之Excel处理库openpyxl详解...
原标题:科多大数据之Python基础教程之Excel处理库openpyxl详解科多大数据小课堂来啦~Python基础教程之Excel处理库openpyxl详解 openpyxl是一个第三方库,可以处 ...
什么是python基础教程-python基础教程之python是什么？概念解析
Python,是一种面向对象的解释型计算机程序设计语言,由荷兰人Guido van Rossum于1989年发明,第一个公开发行版发行于1991年. Python是纯粹的自由软件, 源代码和解释器CP ...

【Nutch2.2.1基础教程之3】Nutch2.2.1配置文件

【Nutch2.2.1基础教程之3】Nutch2.2.1配置文件相关推荐

最新文章

热门文章