java爬虫之基于httpclient的简单Demo(二)

转载自 java爬虫之基于httpclient的简单Demo(二)

延续demo1的 java爬虫的2种爬取方式（HTTP||Socket）简单Demo(一)，demo2出炉啦，大家想学爬虫都可以从这个网盘学习哦：https://pan.baidu.com/s/1pJJrcqJ#list/path=%2F

免费课程，非常不错。其实还是主要学习一个httpclient，httpclient全是英文文档，看的我心累啊

package com.simple.crawImpl;  import com.simple.Icrawl.ICrawl;
import com.simple.pojos.CrawlResultPojo;
import com.simple.pojos.UrlPojo;
import org.apache.http.HttpEntity;
import org.apache.http.ParseException;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpUriRequest;
import org.apache.http.client.methods.RequestBuilder;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;  import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.URI;
import java.net.URISyntaxException;
import java.util.HashMap;
import java.util.Map;
import java.util.Map.Entry;  /** * * Created by lewis on 2016/10/16. */
public class HttpClientCrawlerImpl implements ICrawl{  public CloseableHttpClient httpClient = HttpClients.custom().build();           //创建定制HttpClient  @Override  public CrawlResultPojo crawl(UrlPojo urlpojo) {  if(urlpojo==null){  return null;  }  CrawlResultPojo crawlResultPojo = new CrawlResultPojo();                //结果集  CloseableHttpResponse response = null;                                  //HTTP返回的各种信息集合，包含协议http标准，httpcode状态码  BufferedReader br = null;                                               //  try {  HttpGet httpGet = new HttpGet(urlpojo.getUrl());  response = httpClient.execute(httpGet);  HttpEntity entity = response.getEntity();                                       //获取输入流  InputStreamReader isr = new InputStreamReader(entity.getContent(),"utf-8");     //字节流转化为字符流，设置编码  br =new BufferedReader(isr);  String line =null;  StringBuilder context = new StringBuilder();  while((line=br.readLine())!=null){  context.append(line+"\n");  }  crawlResultPojo.setSuccess(true);  crawlResultPojo.setPageContent(context.toString());  return crawlResultPojo;  } catch (IOException e) {  e.printStackTrace();  crawlResultPojo.setSuccess(false);  }finally {  try {  if (br!=null)  br.close();                                                                 //关闭流  if(response!=null)  response.close();  } catch (IOException e) {  e.printStackTrace();  }  }  return crawlResultPojo;  }  /** * 带参数post的urlpojo * */  public CrawlResultPojo crawl4Post(UrlPojo urlPojo){  if(urlPojo==null||urlPojo.getUrl()==null){  return null;  }  CrawlResultPojo crawlResultPojo = new CrawlResultPojo();  BufferedReader br= null;  try {  RequestBuilder rb = RequestBuilder.post().setUri(new URI(urlPojo.getUrl()));  Map<String,Object> parasMap = urlPojo.getParasMap() ;  if(parasMap!=null){  for(Entry<String,Object> entry:parasMap.entrySet()){  rb.addParameter(entry.getKey(),entry.getValue().toString());  }  }  HttpUriRequest httpUriRequest = rb.build();  HttpEntity entity =httpClient.execute(httpUriRequest).getEntity();  InputStreamReader isr=new InputStreamReader(entity.getContent(),"utf-8");  br = new BufferedReader(isr);  String line = null;  StringBuilder stringBuilder = new StringBuilder();  while((line=br.readLine())!=null){  stringBuilder.append(line+"\n");  }  crawlResultPojo.setPageContent(stringBuilder.toString());  crawlResultPojo.setSuccess(true);  return crawlResultPojo;  } catch (URISyntaxException e) {  e.printStackTrace();  } catch (ClientProtocolException e) {  e.printStackTrace();  } catch (IOException e) {  e.printStackTrace();  } finally {  try {  if(br!=null)  br.close();  } catch (IOException e) {  e.printStackTrace();  }  }  crawlResultPojo.setSuccess(false);  return crawlResultPojo;  }  public static void main(String []args){  HttpClientCrawlerImpl httpClientCrawlerImpl = new HttpClientCrawlerImpl();  String url = "http://www.wangdaizhijia.com/front_select-plat";  UrlPojo urlPojo = new UrlPojo(url);  Map<String, Object> parasMap = new HashMap<String, Object>();  int max_page_number = 1000;  parasMap.put("currPage", 30);  parasMap.put("params", "");  parasMap.put("sort", 0);  urlPojo.setParasMap(parasMap);  CrawlResultPojo resultPojo = httpClientCrawlerImpl.crawl4Post(urlPojo);  print(resultPojo);  resultPojo=httpClientCrawlerImpl.crawl(urlPojo);  print(resultPojo);  }  public static void print(Object s){  System.out.println(s);  }  }

java爬虫之基于httpclient的简单Demo(二)相关推荐

java爬虫教程百度云_java视频教程java爬虫实战项目httpclient hbase springmvc solr
资源内容: java视频教程java爬虫实战项目httpclient hbase springmvc solr|____猫了个咪-更多IT精品课程.html|____猫了个咪--it视频论坛.url| ...
使用轻量级JAVA 爬虫Gecco工具抓取新闻DEMO
写在前面最近看到Gecoo爬虫工具,感觉比较简单好用,所有写个DEMO测试一下,抓取网站 http://zj.zjol.com.cn/home.html,主要抓取新闻的标题和发布时间做为抓取测试对象 ...
Java爬虫更新mysql数据库（简单事例）
http://webmagic.io/docs/zh/posts/ch4-basic-page-processor/selectable.html webmagic官网文档解释所需jar包: 以及w ...
基于iOS的简单计算器(二)
接上一篇基于iOS的简单计算器(一)皆为po主iOS课程的实验内容. 新增界面高级计算器:能够进行一些高级的运算比如三角函数.阶乘.对数等. 体质计算器:获取用户的身高体重后,反馈用户的体质指数. ...
Java爬虫之利用Jsoup自制简单的搜索引擎
Jsoup 是一款Java 的HTML解析器,可直接解析某个URL地址.HTML文本内容.它提供了一套非常省力的API,可通过DOM,CSS以及类似于jQuery的操作方法来取出和操作数据.Jso ...
flex java blazeds_Flex与java通过BlazeDS实现通信（简单Demo）
因为对于flex本人也是新手,此前按照网上的各种方法进行试验,都没能实现flex与java的通信.也许是eclipse和tomcat版本的问题.后来经过几天的尝试,终于成功了...现和大家分享一下,希 ...
Java案例：基于TCP的简单聊天程序
文章目录一.如何实现TCP通信二.编写C/S架构聊天程序 (一)编写服务器端程序 - Server.java (二)编写客户端程序 - Client.java (三)测试服务器端与客户端能否通信 ...
Java爬虫（二）-- httpClient模拟Http请求+jsoup页面解析
博客学院下载 GitChat TinyMind 论坛 APP 问答商城 VIP会员活动招聘 ITeye 写博客发Chat 传资源登录注册原 Java爬虫(二)-- httpClient ...
基于Java爬虫的课堂考勤管理系统（毕业设计论文）
开发环境和选用的架构: 本系统中采用的开发环境是Windows10+JDK8+Tomcat8+MySQL5.6 使用IDEA作为开发工具,GIT用作版本控制,Maven管理依赖采用SSM+Sprin ...

java爬虫之基于httpclient的简单Demo(二)

java爬虫之基于httpclient的简单Demo(二)相关推荐

最新文章

热门文章