Java多线程：示例代码

代码

import java.io.IOException;
import java.io.InputStream;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.*;
import java.util.concurrent.*;
import java.util.regex.Matcher;
import java.util.regex.Pattern;/** 如果设置了自定义域名，将博客域名前缀填写入19行的变量userId中，点击运行*/
public class UrlCrawBroke {static int maxPages = 20; // 填写你的博客查找页数static String userId = "hanquan";// 这里填入主页名称：例如主业为 https://hanquan.blog.csdn.net/ 则填入 hanquan 即可static final String homeUrl = "https://" + userId + ".blog.csdn.net/article/list/";static Set<String> urlSet = new HashSet<>();public static void getUrls() throws IOException, InterruptedException {InputStream is;String pageStr;StringBuilder curUrl = null;for (int i = 1; i < maxPages; i++) {Thread.sleep(500);System.out.println("正在查找第 " + i + " 页中的博客地址");curUrl = new StringBuilder(homeUrl);curUrl.append(i);System.out.println(curUrl);is = doGet(curUrl.toString());pageStr = inputStreamToString(is, "UTF-8");// 一整页的html源码List<String> list = getMatherSubstrs(pageStr, "(?<=href=\")https://hanquan.blog.csdn.net/article/details/[0-9]{8,9}(?=\")");urlSet.addAll(list);System.out.println("加入 " + list.size() + " 个url");}}public static void main(String urlstr[]) throws IOException, InterruptedException {// ----------------------------------------------遍历每一页 获取文章链接----------------------------------------------getUrls();// ---------------------------------------------------打印每个链接---------------------------------------------------System.out.println("打印每个链接");for (String s : urlSet) {System.out.println(s);}System.out.println("打印每个链接完毕");// ---------------------------------------------------多线程访问每个链接---------------------------------------------------ExecutorService executor = Executors.newCachedThreadPool();int threadCount = 5; // 并发线程数量for (int i = 0; i < threadCount; i++) {executor.execute(new MyThread(urlSet));}executor.shutdown();}public static InputStream doGet(String urlstr) throws IOException {URL url = new URL(urlstr);HttpURLConnection conn = (HttpURLConnection) url.openConnection();conn.setRequestProperty("User-Agent","Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36");InputStream inputStream = conn.getInputStream();return inputStream;}public static String inputStreamToString(InputStream is, String charset) throws IOException {byte[] bytes = new byte[1024];int byteLength = 0;StringBuffer sb = new StringBuffer();while ((byteLength = is.read(bytes)) != -1) {sb.append(new String(bytes, 0, byteLength, charset));}return sb.toString();}// 正则匹配public static List<String> getMatherSubstrs(String str, String regex) {List<String> list = new ArrayList<String>();Pattern p = Pattern.compile(regex);Matcher m = p.matcher(str);while (m.find()) {list.add(m.group());}return list;}
}class MyThread implements Runnable {public List<String> urlList;public MyThread(Set<String> urls) {List list = new ArrayList(urls);Collections.shuffle(list);this.urlList = list;}@Overridepublic void run() {int i = 0;for (String s : urlList) {try {doGet(s);System.out.println(Thread.currentThread().getName() + "成功访问第" + (++i) + "个链接,共" + urlList.size() + "个:" + s);} catch (IOException e) {e.printStackTrace();}}}public static InputStream doGet(String urlstr) throws IOException {URL url = new URL(urlstr);HttpURLConnection conn = (HttpURLConnection) url.openConnection();conn.setRequestProperty("User-Agent","Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36");InputStream inputStream = conn.getInputStream();return inputStream;}
}

到此结束啦！

Java多线程：示例代码相关推荐

java多线程示例_Java线程示例
java多线程示例 Welcome to the Java Thread Example. Process and Thread are two basic units of execution. C ...
二维码编解码 Java调用示例代码
二维码编解码支持二维码的生成与识别.二维码生成时,直接返回生成的二维码图片二维码生成复制代码二维码解码复制代码注意,该示例代码仅适用于 www.apishop.net网站下API 使用该产 ...
java多线程死锁代码_java多线程死锁编写高质量代码：改善Java程序的151个建议...
java多线程死锁编写高质量代码:改善Java程序的151个建议第1章 Java开发中的通用方法和准则建议1:不要在常量和变量中出现易混淆的字母建议2:莫让常量蜕变成变量建议3:三元操作符的 ...
验证码短信 API 接入指南：Java 语言示例代码
短信验证码是一种通过短信发送的验证码,通常用于验证用户的身份.保障账户安全和防止恶意攻击等.短信验证码已经成为现代社会中不可或缺的一部分,广泛应用于各种场景,包括用户注册和登录.找回密码等等. 为了满 ...
Java多线程同步代码块
/*多线程的安全问题 1.为什么会出现安全问题? 因为程序在运行时,会出现一个线程在判断条件满足后,具备了执行资格,但没有运行代码后一个线程也判断了条件,也具备了执行资格,后一个线程运行了代码,但这 ...
jpush java_JPush Java SDK 示例代码与简要注释
建议使用JPush 服务端SDK的人都先看下接口文档,了解推送api,具体链接:JPush v3 推送api文档本人下载的是JPush Java 3.2.9版本的SDK源码(下载链接:JPush J ...
java银联接口代码_银联接口测试——详细（JAVA）(示例代码)
1.注册账号 https://open.unionpay.com/ajweb/register?locale=zh_CN 2.▼登录账号 -->帮助中心--> 下载,选择网关支付,下载网管 ...
免费的雅虎天气应用创建， free yahoo weather application create ，Homepage URL , java demo示例代码
(国内网可能访问不了,可能需要梯子) 官网免费雅虎天气应用创建,获取授权key: https://developer.yahoo.com/apps/create/ 如果没账号,注册一个,简单用邮箱就 ...
java多线程学习代码笔记
继承Thead创建线程 public class Threadtest extends Thread{@Overridepublic void run() {for (int i = 0; i < ...
（译）java NIO 示例代码
NIO的学习实例 ******************************************************************************************* ...

Java多线程：示例代码

代码

Java多线程：示例代码相关推荐

最新文章

热门文章