【技术应用】java通过url爬虫获取公众号文章内容

一、前言
二、解决思路
三、爬虫工具
四、代码实现
- 1.爬取公众号文章
- 2.爬取CSDN文章
五、总结

一、前言

平时在微信或者钉钉发送消息时，会自动把url链接转出卡片信息，提高了消息的可读性，是一个很好的用户体验，这两天我们正在开发的一个应用软件也涉及到这一块的内容，由于前端的限制，只能依赖于后端程序接口辅助实现此功能；

二、解决思路

后端提供转换接口，入参为url链接，后端代码通过url链接获取文章信息（文章标题、部分内容、封面图）回传给前端，供前端实现卡片功能,返回值格式如下：

{"title": "从入门到放弃","content": "java对象生命周期","image_url": "http://..."
}

注：今天不讲url如何转卡片，重点想总结一下java如何通过url获取文章信息，或者说java如何通过url爬取文章信息；

三、爬虫工具

网络爬虫前两年是一个很火的功能，之前一提到爬虫大家可能首先想到的是通过python实现爬虫功能，但是通过java实现爬虫功能的应用相对较少，这也是由于python和java的特点决定的，python做爬虫语法更简单，代码更简洁，java的语法比python严格，而且代码也更复杂；

我们要实现的爬虫功能很简单，所以我们主要讲通过java实现爬虫功能；

首先介绍三种爬虫框架/程序：Phantomjs/Casperjs, HtmlUnit, Selenium，关于这三个框架的内容这里不做详细讲解，需要的可以自行百度，这里只做一个特点对比，也方便我们理解技术选型，特点对比如下：

框架	javaScript engine	cookie	request[received]url	Browser	访问速度,稳定性,可扩展性等
Phantomjs/Casperjs	Base on WebKit	支持	支持	Base on WebKit	访问速度较快，有时程序会crash,支持各种js 框架，缺点:支持的js有限
HtmlUnit	Rhino	支持	支持	Firefox or Internet Explorer	访问速度最快,比较稳定,支持各种js框架,可以由页面类容模拟url请求. 缺点:支持的js有限
Selenium	Most engine	支持	不支持	Most Browsers	访问速度太慢，速度也不稳定，而且带有UI,想跨平台必须使用RemoteWebDriver,优点: 支持大部分浏览器

由上述框架的特点可知，我们只是要实现通过url获取文章的标题、内容和图片属性，不涉及js和界面渲染等相关内容，所以我们选择使用htmlunit；
简单介绍：htmlunit 是一款开源的java 页面分析工具，读取页面后，可以有效的使用htmlunit分析页面上的内容。项目可以模拟浏览器运行，被誉为java浏览器的开源实现。这个没有界面的浏览器，运行速度也是非常迅速的。

为了方便解析爬虫获取的html，我们使用jsoup工具，jsoup是一款Java的html解析工具，主要是对html和xml文件进行解析，使用jsoup能轻松从html/xml获取想要的内容。

四、代码实现

我们今天主要列举两种类型的数据爬取：公众号文章和CSDN文章

首先引入pom.xml依赖：

<dependency><groupId>net.sourceforge.htmlunit</groupId><artifactId>htmlunit</artifactId><version>2.55.0</version></dependency><dependency><groupId>org.jsoup</groupId><artifactId>jsoup</artifactId><version>1.8.3</version></dependency>

1.爬取公众号文章

package com.example.tts.service.impl;import com.example.tts.service.ICrawlingService;
import com.example.tts.utils.ToSpeach;
import com.gargoylesoftware.htmlunit.BrowserVersion;
import com.gargoylesoftware.htmlunit.NicelyResynchronizingAjaxController;
import com.gargoylesoftware.htmlunit.WebClient;
import com.gargoylesoftware.htmlunit.html.HtmlPage;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.select.Elements;
import org.springframework.stereotype.Service;import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.Map;
import java.util.Random;@Service
public class WXCrawlingService implements ICrawlingService {static String[] userAgent = {"Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_8; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50","Mozilla/5.0 (Windows; U; Windows NT 6.1; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50","Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0","Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727; .NET CLR 3.0.30729; .NET CLR 3.5.30729; InfoPath.3; rv:11.0) like Gecko","Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)","Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0)","Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)","Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)","Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0.1) Gecko/20100101 Firefox/4.0.1","Mozilla/5.0 (Windows NT 6.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1","Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; en) Presto/2.8.131 Version/11.11","Opera/9.80 (Windows NT 6.1; U; en) Presto/2.8.131 Version/11.11","Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_0) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11","Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Maxthon 2.0)","Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; TencentTraveler 4.0)","Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)","Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; The World)","Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SE 2.X MetaSr 1.0; SE 2.X MetaSr 1.0; .NET CLR 2.0.50727; SE 2.X MetaSr 1.0)","Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; 360SE)","Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Avant Browser)","Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)","Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5","Mozilla/5.0 (iPod; U; CPU iPhone OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5","Mozilla/5.0 (iPad; U; CPU OS 4_3_3 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5","Mozilla/5.0 (Linux; U; Android 2.3.7; en-us; Nexus One Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1","MQQBrowser/26 Mozilla/5.0 (Linux; U; Android 2.3.7; zh-cn; MB200 Build/GRJ22; CyanogenMod-7) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1","Opera/9.80 (Android 2.3.4; Linux; Opera Mobi/build-1107180945; U; en-GB) Presto/2.8.149 Version/11.10","Mozilla/5.0 (Linux; U; Android 3.0; en-us; Xoom Build/HRI39) AppleWebKit/534.13 (KHTML, like Gecko) Version/4.0 Safari/534.13","Mozilla/5.0 (BlackBerry; U; BlackBerry 9800; en) AppleWebKit/534.1+ (KHTML, like Gecko) Version/6.0.0.337 Mobile Safari/534.1+","Mozilla/5.0 (hp-tablet; Linux; hpwOS/3.0.0; U; en-US) AppleWebKit/534.6 (KHTML, like Gecko) wOSBrowser/233.70 Safari/534.6 TouchPad/1.0","Mozilla/5.0 (SymbianOS/9.4; Series60/5.0 NokiaN97-1/20.0.019; Profile/MIDP-2.1 Configuration/CLDC-1.1) AppleWebKit/525 (KHTML, like Gecko) BrowserNG/7.1.18124","Mozilla/5.0 (compatible; MSIE 9.0; Windows Phone OS 7.5; Trident/5.0; IEMobile/9.0; HTC; Titan)","UCWEB7.0.2.37/28/999","NOKIA5700/ UCWEB7.0.2.37/28/999","Openwave/ UCWEB7.0.2.37/28/999","Mozilla/4.0 (compatible; MSIE 6.0; ) Opera/UCWEB7.0.2.37/28/999","Mozilla/6.0 (iPhone; CPU iPhone OS 8_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/8.0 Mobile/10A5376e Safari/8536.25",};static BrowserVersion browser =new BrowserVersion.BrowserVersionBuilder(BrowserVersion.CHROME).setUserAgent(userAgent[new Random().nextInt(userAgent.length)]).build();Map<String, Integer> proxy = new HashMap();private static final WebClient webClient = new WebClient(browser);ToSpeach toSpeach = new ToSpeach();public static String crawling02(String url) throws NullPointerException {ArrayList<String> texts = new ArrayList();webClient.getOptions().setThrowExceptionOnScriptError(false);webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);webClient.getOptions().setActiveXNative(false);webClient.getOptions().setCssEnabled(false);webClient.getOptions().setJavaScriptEnabled(true);webClient.setAjaxController(new NicelyResynchronizingAjaxController());HtmlPage page = null;try {page = webClient.getPage(url);} catch (Exception e) {e.printStackTrace();} finally {webClient.close();}webClient.waitForBackgroundJavaScript(30000);String pageXml = page.asXml();System.out.println(pageXml);Document document = Jsoup.parse(pageXml);String title = document.title();System.out.println(title);// rich_media_titleElements rich_media_title = document.getElementsByClass("rich_media_title");System.out.println("=================title=====================");//System.out.println(rich_media_title);rich_media_title.forEach(element -> System.out.println(element.getElementsByTag("h1").text()));Elements infoListEle = document.getElementsByClass("rich_media_content");System.out.println("==================content====================");//System.out.println(infoListEle);infoListEle.forEach(element -> System.out.println(element.getElementsByTag("p").text()));System.out.println("================获取封面图=====================");String urlXml = pageXml.substring(pageXml.indexOf("cdn_url_1_1"),pageXml.length());urlXml = urlXml.substring(urlXml.indexOf("\"")+1);String url_01_01 = urlXml.substring(0,urlXml.indexOf("\""));System.out.println(url_01_01);return pageXml;}public static void main(String[] args) {//String url = "https://mp.weixin.qq.com/s/mcUJt29Dq9g4EGH1GgcsGQ";String url = "https://mp.weixin.qq.com/s/s2Txc0xDZ8KTnZk-5vp3JQ";String xml = crawling02(url);// System.out.println(readXMLName02(xml,"title"));}public static String readXMLName02(String s, String name) {if (s.contains(name)) {int start = s.indexOf("<" + name + ">") + name.length() + 2;int end = s.indexOf("</" + name + ">");String substring = s.substring(start, end);return substring;}return null;}@Overridepublic ArrayList<String> crawling(String URL) {return null;}}

公众号文章：

爬取结果：

=================title=====================
Spring中获取bean的八种方式，你get了几种？
==================content====================
大家好，我是锋哥！ Java1234 VIP大特惠(仅限100名额)！... 省略
================获取封面图=====================
https://mmbiz.qpic.cn/mmbiz_jpg/JfTPiahTHJhogG8qc16pF4gePH9FfnnLTia8m58vwDztbQCKcpxaoa44htlfuhVBtA9JicRns25WrofQTqzP2icGxw/0?wx_fmt=jpeg

2.爬取CSDN文章

package com.example.tts.service.impl;import com.example.tts.service.ICrawlingService;
import com.gargoylesoftware.htmlunit.BrowserVersion;
import com.gargoylesoftware.htmlunit.NicelyResynchronizingAjaxController;
import com.gargoylesoftware.htmlunit.WebClient;
import com.gargoylesoftware.htmlunit.html.HtmlPage;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.select.Elements;
import org.springframework.stereotype.Service;import java.util.ArrayList;@Service
public class CSDNCrawlingService implements ICrawlingService {private static WebClient webClient= new WebClient(BrowserVersion.CHROME);public static void crawling02(String URL){webClient.getOptions().setThrowExceptionOnScriptError(false);webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);webClient.getOptions().setActiveXNative(false);webClient.getOptions().setCssEnabled(false);webClient.getOptions().setJavaScriptEnabled(true);webClient.setAjaxController(new NicelyResynchronizingAjaxController());HtmlPage page = null;try {page = webClient.getPage(URL);} catch (Exception e) {e.printStackTrace();}finally {webClient.close();}webClient.waitForBackgroundJavaScript(30000);String pageXml = page.asXml();Document document = Jsoup.parse(pageXml);Elements titleListEle = document.getElementsByTag("title");String title = titleListEle.text();System.out.println("============文章标题=================");System.out.println(title);/*Elements infoListEle = document.getElementById("title").getElementsByTag("li");infoListEle.forEach(element -> {System.out.println(element.getElementsByClass("txt-box").first().getElementsByTag("h3").text());System.out.println(element.getElementsByClass("txt-box").first().getElementsByTag("a").attr("href"));});*/}public static void main(String[] args) {String url = "https://blog.csdn.net/Edward1027/article/details/124859901";crawling02(url);}@Overridepublic ArrayList<String> crawling(String URL) {return null;}
}

文章信息：

爬取结果：

============文章标题=================
CSDN插入表格_码农爱德华的博客-CSDN博客_csdn表格
Disconnected from the target VM, address: '127.0.0.1:58937', transport: 'socket'

五、总结

爬虫内容平时应用比较少，只在这里总结这两种应用，有兴趣的可以多总结，学习如果只是学相对片面，还要结合实际应用。