网络爬虫的基本思路：有关键字指定的url把所有相关的html页面全抓下来（html即为字符串），然后解析为html文本（通常是正则表达式或者县城的工具包如jsoup），提取微博文本信息，然后把文本信息存储起来。

jar包依赖：

httpclient-4.3.1.jar  -------建立HTTP链接，用于从url获取html

httpcore-4.3.jar

httpmime-4.3.1.jar

httpclient-cache-4.3.1.jar

fluent-hc-4.3.1.jar

fastjson-1.1.41.jar   ------解析json的工具包

jsoup-1.7.3.jar      -------解析xml，html的工具包

dom4j-1.6.1.jar     -------读写xml的工具包

commons-lang-2.1.jar

commons-logging-1.2.jar

commons-codec-1.8.jar

我用的是maven进行管理，这是jar包依赖图：

<dependencies><dependency><groupId>junit</groupId><artifactId>junit</artifactId><version>4.11</version><scope>test</scope></dependency><!-- https://mvnrepository.com/artifact/org.apache.httpcomponents/httpclient --><dependency><groupId>org.apache.httpcomponents</groupId><artifactId>httpclient</artifactId><version>4.5.6</version></dependency><!-- https://mvnrepository.com/artifact/org.apache.httpcomponents/httpcore --><dependency><groupId>org.apache.httpcomponents</groupId><artifactId>httpcore</artifactId><version>4.4.10</version></dependency><!-- https://mvnrepository.com/artifact/org.apache.httpcomponents/httpmime --><dependency><groupId>org.apache.httpcomponents</groupId><artifactId>httpmime</artifactId><version>4.5.6</version></dependency><!-- https://mvnrepository.com/artifact/org.apache.httpcomponents/httpclient-cache --><dependency><groupId>org.apache.httpcomponents</groupId><artifactId>httpclient-cache</artifactId><version>4.5.6</version></dependency><!-- https://mvnrepository.com/artifact/org.apache.httpcomponents/fluent-hc --><dependency><groupId>org.apache.httpcomponents</groupId><artifactId>fluent-hc</artifactId><version>4.5.6</version></dependency><!-- https://mvnrepository.com/artifact/com.alibaba/fastjson --><dependency><groupId>com.alibaba</groupId><artifactId>fastjson</artifactId><version>1.2.51</version></dependency><!-- https://mvnrepository.com/artifact/org.jsoup/jsoup --><dependency><groupId>org.jsoup</groupId><artifactId>jsoup</artifactId><version>1.11.3</version></dependency><!-- https://mvnrepository.com/artifact/dom4j/dom4j --><dependency><groupId>dom4j</groupId><artifactId>dom4j</artifactId><version>1.6.1</version></dependency></dependencies>
复制代码