[Java教程]一个更加简单粗暴的爬虫

0

2017-09-30 16:00:13

今天上午看到一篇文章：一个简单粗暴的爬虫 - 必应今日美图。我也用自己的方式更加简单的实现了这个功能。下面我就贴一下自己的代码和思路。

我就不分析原博的思路了，原博写的很清楚。我用的是htmlunit，从原博的第三步开始，观察三级地址的规律， > 1 import com.gargoylesoftware.htmlunit.FailingHttpStatusCodeException; 2 import com.gargoylesoftware.htmlunit.WebClient; 3 import com.gargoylesoftware.htmlunit.html.HtmlAnchor; 4 import com.gargoylesoftware.htmlunit.html.HtmlPage; 5 import org.apache.commons.io.FileUtils; 6 7 import java.io.File; 8 import java.io.IOException; 9 import java.io.InputStream;10 import java.net.MalformedURLException;11 12 /**13 *

14  * class intro15  * 

16 *

17 * Created by fujun on 2017/9/29.18 */19 public class BingPhoto {20 public static void main(String[] args) {21 WebClient webClient = new WebClient();22 webClient.getOptions().setCssEnabled(false);23 webClient.getOptions().setJavaScriptEnabled(false);24 for (int i = 1; i < 491; i++) {25 String url = "http://bing.plmeizi.com/show/" + i;26 try {27 HtmlPage page = webClient.getPage(url);28 HtmlAnchor a = (HtmlAnchor) page.getElementById("picurl");29 String picUrl = a.getHrefAttribute();30 if (picUrl.contains("jpg")) {31 InputStream inputStream = webClient.getPage(picUrl).getWebResponse().getContentAsStream();32 FileUtils.copyToFile(inputStream, new File("E://bingpic//" + i + ".jpg"));33 System.out.println("SUCCEED:" + url);34 } else {35 System.out.println("FAILED:" + url);36 }37 38 } catch (FailingHttpStatusCodeException e) {39 // TODO Auto-generated catch block40 e.printStackTrace();41 } catch (MalformedURLException e) {42 // TODO Auto-generated catch block43 e.printStackTrace();44 } catch (IOException e) {45 // TODO Auto-generated catch block46 e.printStackTrace();47 } finally {48 webClient.close(); // 关闭客户端，释放内存49 }50 }51 }52 }

注意点：

有些页面是404的，例如 src='/images/loading.gif' data-original="http://images2017.cnblogs.com/blog/1250684/201709/1250684-20170930150215075-698272723.png" alt="">

本文网址：http://www.shaoqun.com/a/327023.html

*特别声明：以上内容来自于网络收集，著作权属原作者所有，如有侵权，请联系我们：admin@shaoqun.com。

0