国庆在家太无聊， 用Java爬了上千张小姐姐照片...
2024-04-21 07:03:52
目的
爬取搜狗图片上千张美女图片并下载到本地
准备工作
爬取地址：https://pic.sogou.com/pics?query=%E7%BE%8E%E5%A5%B3
分析
打开上面的地址，按F12开发者工具 - NetWork - XHR - 页面往下滑动XHR栏出现请求信息如下：
Request URL ：https://pic.sogou.com/napi/pc/searchList?mode=1&start=48&xml_len=48&query=%E7%BE%8E%E5%A5%B3
分析这段请求URL的主要几个参数：
start=48 表示从第48张图片开始检索
xml_len=48 从地48张往后获取48张图片
query=？搜索关键词（例：美女，这里浏览器自动做了转码，不影响我们使用）
点击Respose，找个JSON格式器辅助过去看看。
JSON格式：https://www.bejson.com/
分析Respose返回的信息，可以发现我们想要的图片地址放在 picUrl里，
思路
通过以上分析，不难实现下载方法，思路如下：
1.设置URL请求参数
2.访问URL请求，获取图片地址
3.图片地址存入List
4.遍历List，使用线程池下载到本地
代码
SougouImgProcessor.java 爬取图片类
import com.alibaba.fastjson.JSONObject;
import us.codecraft.webmagic.utils.HttpClientUtils;
import victor.chang.crawler.pipeline.SougouImgPipeline;import java.util.ArrayList;
import java.util.List;/*** A simple PageProcessor.* @author code4crafter@gmail.com <br>* @since 0.1.0*/
public class SougouImgProcessor {private String url;private SougouImgPipeline pipeline;private List<JSONObject> dataList;private List<String> urlList;private String word;public SougouImgProcessor(String url,String word) {this.url = url;this.word = word;this.pipeline = new SougouImgPipeline();this.dataList = new ArrayList<>();this.urlList = new ArrayList<>();}public void process(int idx, int size) {String res = HttpClientUtils.get(String.format(this.url, idx, size, this.word));JSONObject object = JSONObject.parseObject(res);List<JSONObject> items = (List<JSONObject>)((JSONObject)object.get("data")).get("items");for(JSONObject item : items){this.urlList.add(item.getString("picUrl"));}this.dataList.addAll(items);}// 下载public void pipelineData(){// 多线程pipeline.processSync(this.urlList, this.word);}public static void main(String[] args) {String url = "https://pic.sogou.com/napi/pc/searchList?mode=1&start=%s&xml_len=%s&query=%s";SougouImgProcessor processor = new SougouImgProcessor(url,"美女");int start = 0, size = 50, limit = 1000; // 定义爬取开始索引、每次爬取数量、总共爬取数量for(int i=start;i<start+limit;i+=size)processor.process(i, size);processor.pipelineData();}}
SougouImgPipeline.java 图片下载类
import java.io.File;
import java.io.FileOutputStream;
import java.io.InputStream;
import java.net.URL;
import java.net.URLConnection;
import java.util.List;
import java.util.Objects;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;/*** Store results in files.<br>* @author code4crafter@gmail.com <br>* @since 0.1.0*/
public class SougouImgPipeline {private String extension = ".jpg";private String path;private volatile AtomicInteger suc;private volatile AtomicInteger fails;public SougouImgPipeline() {setPath("E:/pipeline/sougou");suc = new AtomicInteger();fails = new AtomicInteger();}public SougouImgPipeline(String path) {setPath(path);suc = new AtomicInteger();fails = new AtomicInteger();}public SougouImgPipeline(String path, String extension) {setPath(path);this.extension = extension;suc = new AtomicInteger();fails = new AtomicInteger();}public void setPath(String path) {this.path = path;}/*** 下载* @param url* @param cate* @throws Exception*/private void downloadImg(String url, String cate, String name) throws Exception {String path = this.path + "/" + cate + "/";File dir = new File(path);if (!dir.exists()) { // 目录不存在则创建目录dir.mkdirs();}String realExt = url.substring(url.lastIndexOf(".")); // 获取扩展名String fileName = name + realExt;fileName = fileName.replace("-", "");String filePath = path + fileName;File img = new File(filePath);if(img.exists()){ // 若文件之前已经下载过，则跳过System.out.println(String.format("文件%s已存在本地目录",fileName));return;}URLConnection con = new URL(url).openConnection();con.setConnectTimeout(5000);con.setReadTimeout(5000);InputStream inputStream = con.getInputStream();byte[] bs = new byte[1024];File file = new File(filePath);FileOutputStream os = new FileOutputStream(file, true);// 开始读取 写入int len;while ((len = inputStream.read(bs)) != -1) {os.write(bs, 0, len);}System.out.println("picUrl: " + url);System.out.println(String.format("正在下载第%s张图片", suc.getAndIncrement()));}/*** 单线程处理** @param data* @param word*/public void process(List<String> data, String word) {long start = System.currentTimeMillis();for (String picUrl : data) {if (picUrl == null)continue;try {downloadImg(picUrl, word, picUrl);} catch (Exception e) {fails.incrementAndGet();}}System.out.println("下载成功: " + suc.get());System.out.println("下载失败: " + fails.get());long end = System.currentTimeMillis();System.out.println("耗时：" + (end - start) / 1000 + "秒");}/*** 多线程处理** @param data* @param word*/public void processSync(List<String> data, String word) {long start = System.currentTimeMillis();int count = 0;ExecutorService executorService = Executors.newCachedThreadPool(); // 创建缓存线程池for (int i=0;i<data.size();i++) {String picUrl = data.get(i);if (picUrl == null)continue;String name = "";if(i<10){name="000"+i;}else if(i<100){name="00"+i;}else if(i<1000){name="0"+i;}String finalName = name;executorService.execute(() -> {try {downloadImg(picUrl, word, finalName);} catch (Exception e) {fails.incrementAndGet();}});count++;}executorService.shutdown();try {if (!executorService.awaitTermination(60, TimeUnit.SECONDS)) {// 超时的时候向线程池中所有的线程发出中断(interrupted)。// executorService.shutdownNow();}System.out.println("AwaitTermination Finished");System.out.println("共有URL: "+data.size());System.out.println("下载成功: " + suc);System.out.println("下载失败: " + fails);File dir = new File(this.path + "/" + word + "/");int len = Objects.requireNonNull(dir.list()).length;System.out.println("当前共有文件： "+len);long end = System.currentTimeMillis();System.out.println("耗时：" + (end - start) / 1000.0 + "秒");} catch (InterruptedException e) {e.printStackTrace();}}/*** 多线程分段处理** @param data* @param word* @param threadNum*/public void processSync2(List<String> data, final String word, int threadNum) {if (data.size() < threadNum) {process(data, word);} else {ExecutorService executorService = Executors.newCachedThreadPool();int num = data.size() / threadNum; //每段要处理的数量for (int i = 0; i < threadNum; i++) {int start = i * num;int end = (i + 1) * num;if (i == threadNum - 1) {end = data.size();}final List<String> cutList = data.subList(start, end);executorService.execute(() -> process(cutList, word));}executorService.shutdown();}}}
HttpClientUtils.java http请求工具类
import org.apache.http.Header;
import org.apache.http.HttpEntity;
import org.apache.http.NameValuePair;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.methods.HttpUriRequest;
import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
import org.apache.http.conn.ssl.TrustStrategy;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.message.BasicNameValuePair;
import org.apache.http.ssl.SSLContextBuilder;
import org.apache.http.util.EntityUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;import javax.net.ssl.HostnameVerifier;
import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLSession;
import java.io.IOException;
import java.io.UnsupportedEncodingException;
import java.security.GeneralSecurityException;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;/*** @author code4crafter@gmail.com* Date: 17/3/27*/
public abstract class HttpClientUtils {public static Map<String, List<String>> convertHeaders(Header[] headers) {Map<String, List<String>> results = new HashMap<String, List<String>>();for (Header header : headers) {List<String> list = results.get(header.getName());if (list == null) {list = new ArrayList<String>();results.put(header.getName(), list);}list.add(header.getValue());}return results;}/*** http的get请求* @param url*/public static String get(String url) {return get(url, "UTF-8");}public static Logger logger = LoggerFactory.getLogger(HttpClientUtils.class);/*** http的get请求* @param url*/public static String get(String url, String charset) {HttpGet httpGet = new HttpGet(url);return executeRequest(httpGet, charset);}/*** http的get请求，增加异步请求头参数* @param url*/public static String ajaxGet(String url) {return ajaxGet(url, "UTF-8");}/*** http的get请求，增加异步请求头参数** @param url*/public static String ajaxGet(String url, String charset) {HttpGet httpGet = new HttpGet(url);httpGet.setHeader("X-Requested-With", "XMLHttpRequest");return executeRequest(httpGet, charset);}/*** @param url* @return*/public static String ajaxGet(CloseableHttpClient httpclient, String url) {HttpGet httpGet = new HttpGet(url);httpGet.setHeader("X-Requested-With", "XMLHttpRequest");return executeRequest(httpclient, httpGet, "UTF-8");}/*** http的post请求，传递map格式参数*/public static String post(String url, Map<String, String> dataMap) {return post(url, dataMap, "UTF-8");}/*** http的post请求，传递map格式参数*/public static String post(String url, Map<String, String> dataMap, String charset) {HttpPost httpPost = new HttpPost(url);try {if (dataMap != null) {List<NameValuePair> nvps = new ArrayList<NameValuePair>();for (Map.Entry<String, String> entry : dataMap.entrySet()) {nvps.add(new BasicNameValuePair(entry.getKey(), entry.getValue()));}UrlEncodedFormEntity formEntity = new UrlEncodedFormEntity(nvps, charset);formEntity.setContentEncoding(charset);httpPost.setEntity(formEntity);}} catch (UnsupportedEncodingException e) {e.printStackTrace();}return executeRequest(httpPost, charset);}/*** http的post请求，增加异步请求头参数，传递map格式参数*/public static String ajaxPost(String url, Map<String, String> dataMap) {return ajaxPost(url, dataMap, "UTF-8");}/*** http的post请求，增加异步请求头参数，传递map格式参数*/public static String ajaxPost(String url, Map<String, String> dataMap, String charset) {HttpPost httpPost = new HttpPost(url);httpPost.setHeader("X-Requested-With", "XMLHttpRequest");try {if (dataMap != null) {List<NameValuePair> nvps = new ArrayList<NameValuePair>();for (Map.Entry<String, String> entry : dataMap.entrySet()) {nvps.add(new BasicNameValuePair(entry.getKey(), entry.getValue()));}UrlEncodedFormEntity formEntity = new UrlEncodedFormEntity(nvps, charset);formEntity.setContentEncoding(charset);httpPost.setEntity(formEntity);}} catch (UnsupportedEncodingException e) {e.printStackTrace();}return executeRequest(httpPost, charset);}/*** http的post请求，增加异步请求头参数，传递json格式参数*/public static String ajaxPostJson(String url, String jsonString) {return ajaxPostJson(url, jsonString, "UTF-8");}/*** http的post请求，增加异步请求头参数，传递json格式参数*/public static String ajaxPostJson(String url, String jsonString, String charset) {HttpPost httpPost = new HttpPost(url);httpPost.setHeader("X-Requested-With", "XMLHttpRequest");StringEntity stringEntity = new StringEntity(jsonString, charset);// 解决中文乱码问题stringEntity.setContentEncoding(charset);stringEntity.setContentType("application/json");httpPost.setEntity(stringEntity);return executeRequest(httpPost, charset);}/*** 执行一个http请求，传递HttpGet或HttpPost参数*/public static String executeRequest(HttpUriRequest httpRequest) {return executeRequest(httpRequest, "UTF-8");}/*** 执行一个http请求，传递HttpGet或HttpPost参数*/public static String executeRequest(HttpUriRequest httpRequest, String charset) {CloseableHttpClient httpclient;if ("https".equals(httpRequest.getURI().getScheme())) {httpclient = createSSLInsecureClient();} else {httpclient = HttpClients.createDefault();}String result = "";try {try {CloseableHttpResponse response = httpclient.execute(httpRequest);HttpEntity entity = null;try {entity = response.getEntity();result = EntityUtils.toString(entity, charset);} finally {EntityUtils.consume(entity);response.close();}} finally {httpclient.close();}} catch (IOException ex) {ex.printStackTrace();}return result;}public static String executeRequest(CloseableHttpClient httpclient, HttpUriRequest httpRequest, String charset) {String result = "";try {try {CloseableHttpResponse response = httpclient.execute(httpRequest);HttpEntity entity = null;try {entity = response.getEntity();result = EntityUtils.toString(entity, charset);} finally {EntityUtils.consume(entity);response.close();}} finally {httpclient.close();}} catch (IOException ex) {ex.printStackTrace();}return result;}/*** 创建 SSL连接*/public static CloseableHttpClient createSSLInsecureClient() {try {SSLContext sslContext = new SSLContextBuilder().loadTrustMaterial(new TrustStrategy() {@Overridepublic boolean isTrusted(X509Certificate[] chain, String authType) throws CertificateException {return true;}}).build();SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(sslContext, new HostnameVerifier() {@Overridepublic boolean verify(String hostname, SSLSession session) {return true;}});return HttpClients.custom().setSSLSocketFactory(sslsf).build();} catch (GeneralSecurityException ex) {throw new RuntimeException(ex);}}
}
运行
由于网络等原因，我们发现并不能全部下载成功，不过可以多次运行尝试，可以实现较高的下载成功率。
666，厉害了。。
来源：blog.csdn.net/qq_35402412/article/details/113627625
国庆在家太无聊， 用Java爬了上千张小姐姐照片...相关推荐
- Python爬虫 | 爬取高质量小姐姐照片
Python爬虫 | 爬取高质量小姐姐照片 1.数据来源分析 2.获取author_id_list和img_id 3.制作detial 4.制作detial_list 5.数据保存 6.批量获取 7. ...
- 爬虫实战--爬取1000位小姐姐照片，刷新你三观颜值！
爬取的网站链接:http://www.mmonly.cc/mmtp/ 主页网址唯一图库直接点到末页,一共有1274页. 右键点击一个图片链接,选择审查元素 这里的img是就封面,如果只抓取封面的话,到 ...
- python爬虫应用实战-如何爬取好看的小姐姐照片？
线程锁 Threading模块为我们提供了一个类,Threading.Lock锁.我们创建该类的对象,在线程函数执行之前,"抢占"该锁,执行完成之后,"释放"该 ...
- 循环爬取图片_Python爬虫爬取小姐姐照片！
哔哩哔哩上有一个相簿板块,里面有很多绘画和摄影,自己动手一张一张保存太慢,于是想到了python爬虫. 今天就用python来写个爬虫,爬取每月月榜上的小姐姐,一劳永逸. 爬虫步骤: 1.在bilib ...
- 2021-08-07 Python-爬虫练手:爬取上千张动漫卡通头像
爬取上千张卡通头像 思路实现 一级页面 二级页面 try-except 保存 完整程序 代码 ualist 记得关闭请求 成品欣赏 这两天在学算法,昨天感觉力扣的头像不好看,刚好有两天没碰爬虫了,就百 ...
- 爬取唯美女生网站上所有小姐姐的照片
爬取唯美女生网站 前几天刚好看到有人在教学爬取唯美女生网站(https://www.vmgirls.com)的图片,大致听了一下,只是介绍了一次爬取某一个小姐姐的照片,因此就尝试做一个爬取全站小姐姐照 ...
- 第一个爬虫项目-爬取唯美小姐姐网站
爬取唯美小姐姐网站 链接: 源代码文件下载地址 展示 源代码 import requests import re import os import time# 伪装 用于可以伪装成浏览器. heade ...
- python爬虫：爬取百度小姐姐照片
自从学会了爬虫,身体状况一天不如一天,营养都跟不上了,教大家爬取百度性感小姐姐的图片,先看一下效果. 项目流程 第一步:准备工作 工欲善其事,必先利其器 pip install requests,该模 ...
- python3APP爬虫--爬取王者荣耀小姐姐cosplay照片（附源码）
文章目录 一．絮叨 1.接上 2.启下 二．代码 三．查看数据 四．总结 一．絮叨 1.接上 自己挖的坑,自己填.上一篇python3APP爬虫–爬取王者荣耀英雄图片(附源码)爬取了王者荣耀英雄图片, ...
- 宅在家太无聊了，我用Python做了一个能作弊的抽奖程序
今年的春节真的是魔幻像一场电影,宅在家里好多天了,有的人在客厅围绕茶几已经走了1万步了,有的人没事在客厅放几个玩具,扔圈圈玩.太无聊了,Python好久不写了,赶紧操练起来,反正闲着也是无聊.年前公司 ...
最新文章
- 做了7年软件工程师，从500多场技术面试中学到了什么？
- WinLaucher启动
- android 系统ui修改器,分享两个效果 - Android 系统 UI 管理
- 以下是ECMAScript 2016、2017和2018中所有新增功能的示例
- java包图标是文件_关于更换.jar文件默认图标
- oracle kepserve,Kepware.KEPServer\KEPServerEX_V5操作简介含opc quick client 连接测试
- 《大数据》2015年第2期“专题”——关于大数据交易核心法律问题 —— 数据所有权的探讨及建议...
- 字节跳动简历冷却期_【字节跳动招聘】简历这样写，才不会被秒拒
- KlayGE C++代码风格指南英文版
- 高斯滤波器是低通还是高通_经典模拟滤波器仍值得研究吗？
- 页面导入样式时，使用link和@import有什么区别？
- 1: 认识WPF和一个小Demon
- python实现一个简单的图片浏览器
- 内屏损坏强制usb调试_反渗透膜工作压力及调试
- 【第三期】电商分布式前沿springboot接口服务之删除和更新-Array-专题视频课程...
- 笔记本计算机无法上无线网络,笔记本电脑连不上无线网络的处理方法
- 104道精选iOS面试题PDF
- filebeat7.7.0相关详细配置预览- Setup ILM
- opengles之展翅飞翔的雄鹰
- 计算机关闭应用窗口的方法,电脑怎么从后台关闭程序应用