Java爬虫技术(二)爬取京东iPhone商品信息并生成Json日志

准备

配置maven环境
下载浏览器驱动,并引入;

下载浏览器驱动

前往华为云镜像站下载谷歌浏览器驱动

https://mirrors.huaweicloud.com/home

要下载与自己电脑上谷歌浏览器版本相匹配的;

引入pom.xml依赖

        <dependency><groupId>org.seleniumhq.selenium</groupId><artifactId>selenium-java</artifactId><version>3.141.59</version></dependency><dependency><groupId>org.seleniumhq.selenium</groupId><artifactId>selenium-api</artifactId><version>3.141.59</version></dependency>

开始爬取

目标网址

https://search.jd.com/Search?keyword=iphone%2013&psort=3&wq=iphone%2013&psort=3&pvid=37f665a84459460eb713df08bdcd7799&page=1&s=1&click=0

目标是爬取所有47页的商品的价格,商品名称,商品描述;

第一页网址:

https://search.jd.com/Search?keyword=iphone%2013&psort=3&wq=iphone%2013&psort=3&pvid=37f665a84459460eb713df08bdcd7799&page=1&s=1&click=0

第二页网址:

https://search.jd.com/Search?keyword=iphone%2013&psort=3&wq=iphone%2013&psort=3&pvid=37f665a84459460eb713df08bdcd7799&page=3&s=61&click=0

第三页网址:

https://search.jd.com/Search?keyword=iphone%2013&psort=3&wq=iphone%2013&psort=3&pvid=37f665a84459460eb713df08bdcd7799&page=5&s=121&click=0

经过分析发现规律: page=1 , 3 , 5 为奇数 --2i+1
s =1,60,121 依次加60-- ((i-1)*60)+1

主要代码与操作步骤

定义一个浏览器对象

ChromeDriver webDriver = new ChromeDriver();

利用循环爬取45页的数据

for(int i=1;i<45;i++) {webDriver.get("https://search.jd.com/search?keyword=iphone%2013&wq=iphone%2013&cid3=655&psort=3&page="+((2*i)-1)+"&s="+((i-1)*60)+1+"&click=0");

将页面滑到最底部,因为京东页面若不滑到最下面会导致有些数据加载不出来

((JavascriptExecutor)webDriver).executeScript("window.scrollTo(0,document.body.scrollHeight)");

获取源码

 String pageSource = webDriver.getPageSource();

选中一个单元获取其路径

#J_goodsList > ul > li:nth-child(13) > div

#J_goodsList > ul > li:nth-child > div

查看价格路径

#J_goodsList > ul > li:nth-child(16) > div > div.p-price > strong > i

在其循环体下

 div.p-price > strong > i

同理:店铺名称路径 div.p-shop > span > a
商品描述路径:div.p-name.p-name-type-2 > a > em

还要创建一个product类并建对象:

package com.zygxy.shop;public class Product {private String price;private String shopname;private String shopcontext;public String getPrice() {return price;}public void setPrice(String price) {this.price = price;}public String getShopname() {return shopname;}public void setShopname(String shopname) {this.shopname = shopname;}public String getShopcontext() {return shopcontext;}public void setShopcontext(String shopcontext) {this.shopcontext = shopcontext;}
}

全部代码

package com.zygxy.shop;import com.alibaba.fastjson.JSONObject;
import lombok.extern.slf4j.Slf4j;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import org.openqa.selenium.JavascriptExecutor;
import org.openqa.selenium.chrome.ChromeDriver;import java.io.IOException;
import java.util.Properties;
@Slf4j
public class JD {public static void main(String[] args) throws IOException {Properties properties = new Properties();properties.load(JD.class.getClassLoader().getResourceAsStream("application.properties"));System.out.println(properties.getProperty("chromedriver"));System.setProperty("webdriver.chrome.driver",properties.getProperty("chromedriver"));ChromeDriver webDriver = new ChromeDriver();for (int i=1;i<45;i++){webDriver.get("https://search.jd.com/search?keyword=iphone%2013&psort=3&wq=iphone%2013&psort=3&pvid=aa23e9f58a714e9087f316ab6aa993bd&cid3=655&cid2=653&page="+((2*i)-1)+"&s="+((i-1)*60)+1+"&click=0");((JavascriptExecutor) webDriver).executeScript("window.scrollTo(0,document.body.scrollHeight)");String pageSource = webDriver.getPageSource();Document parse = Jsoup.parse(pageSource);Elements select = parse.select("#J_goodsList > ul > li > div ");for(Element e : select){Product product = new Product();Element price = e.selectFirst(" div.p-price > strong > i");product .setPrice(price.text());Element shop_name = e.selectFirst(" div.p-shop > span > a");product.setShopname(shop_name.text());Element shopcontext  = e.selectFirst(" div.p-name.p-name-type-2 > a > em");product.setShopcontext(shopcontext.text());String json = JSONObject.toJSONString(product);log.info(json);}}}
}

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"><modelVersion>4.0.0</modelVersion><groupId>org.example</groupId><artifactId>NovelParse</artifactId><version>1.0-SNAPSHOT</version><properties><maven.compiler.source>8</maven.compiler.source><maven.compiler.target>8</maven.compiler.target></properties><dependencies><!-- https://mvnrepository.com/artifact/org.jsoup/jsoup --><dependency><groupId>org.jsoup</groupId><artifactId>jsoup</artifactId><version>1.13.1</version></dependency><!-- https://mvnrepository.com/artifact/commons-io/commons-io --><dependency><groupId>commons-io</groupId><artifactId>commons-io</artifactId><version>2.4</version></dependency><!-- https://mvnrepository.com/artifact/org.seleniumhq.selenium/selenium-java --><dependency><groupId>org.seleniumhq.selenium</groupId><artifactId>selenium-java</artifactId><version>3.141.59</version></dependency><!-- https://mvnrepository.com/artifact/org.seleniumhq.selenium/selenium-api --><dependency><groupId>org.seleniumhq.selenium</groupId><artifactId>selenium-api</artifactId><version>3.141.59</version></dependency><!-- https://mvnrepository.com/artifact/com.alibaba/fastjson --><dependency><groupId>com.alibaba</groupId><artifactId>fastjson</artifactId><version>1.2.75</version></dependency><!-- https://mvnrepository.com/artifact/ch.qos.logback/logback-core --><dependency><groupId>ch.qos.logback</groupId><artifactId>logback-core</artifactId><version>1.2.3</version></dependency><!-- https://mvnrepository.com/artifact/ch.qos.logback/logback-classic --><dependency><groupId>ch.qos.logback</groupId><artifactId>logback-classic</artifactId><version>1.2.3</version></dependency><dependency><groupId>org.slf4j</groupId><artifactId>slf4j-api</artifactId><version>1.7.25</version><scope>compile</scope></dependency><!-- https://mvnrepository.com/artifact/org.projectlombok/lombok --><dependency><groupId>org.projectlombok</groupId><artifactId>lombok</artifactId><version>1.18.12</version><scope>provided</scope></dependency></dependencies><build><plugins><!-- 如果已经在Maven的全局配置中，配置了JDK编译的界别，这个插件可以省略 --><!-- <plugin><artifactId>maven-compiler-plugin</artifactId><version>2.3.2</version><configuration><source>1.8</source><target>1.8</target></configuration></plugin> --><!-- 在mvn:package阶段使用maven-assembly-plugin可以将当前项目依赖的Jar中的字节码也打包！默认的打包插件maven-jar-plugin，只会将自己写的代码打包，默认仓库中已经安装了所需的依赖！--><plugin><artifactId>maven-assembly-plugin</artifactId><configuration><descriptorRefs><descriptorRef>jar-with-dependencies</descriptorRef></descriptorRefs><archive><manifest><mainClass>com.zygxy.JD</mainClass></manifest></archive></configuration><executions><execution><id>make-assembly</id><phase>package</phase><goals><goal>single</goal></goals></execution></executions></plugin></plugins></build>
</project>

resources配置

application.properties
本地驱动地址
chromedriver =E:\shixun\pachong\src\main\resources\chromedriver.exe

logback.xml 定义了日志的输出地址和格式

<?xml version="1.0" encoding="UTF-8"?>
<configuration><property name="LOG_HOME" value="d://shoplog" /><appender name="console" class="ch.qos.logback.core.ConsoleAppender"><encoder><pattern>%msg%n</pattern></encoder></appender><appender name="rollingFile" class="ch.qos.logback.core.rolling.RollingFileAppender"><rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"><fileNamePattern>${LOG_HOME}/shop.%d{yyyy-MM-dd}.log</fileNamePattern></rollingPolicy><encoder><pattern>%msg%n</pattern></encoder></appender><!-- 日志输出级别 --><root level="INFO"><appender-ref ref="rollingFile"/><appender-ref ref="console" /></root></configuration>