httpClient(java http/https)（二）（layui分页+httpclient+https+post）

前言

因为做毕设，所以需要爬取食品相关数据，而且爬取后要做分页。
思路：
先用Java写爬虫爬取数据，然后用layui做分页。
涉及技术点：
（1）Java爬虫
需要用到httpclient，因为接口是post，所以需要带参数的post请求，又因为协议是https协议，所以需要绕过证书验证。
（2）分页
layui做分页（其实做过了就很简单）

因为这篇是实践篇，所以主要放代码

代码

controller

@RequestMapping(value = "/goods/data/",method = RequestMethod.GET)
public Result<List<GoodsData>> get(HttpServletRequest request){// 通过HttpServletRequest获取 limit 和 pageint limit = Integer.valueOf(request.getParameter("limit"));int page = Integer.valueOf(request.getParameter("page"));String url = "https://sac.nifdc.org.cn/api/goods/data";JSONObject jsonStr = new JSONObject();jsonStr.put("check_flag","uq");jsonStr.put("order_by","time");jsonStr.put("pageNumber",page);jsonStr.put("pageSize",limit);String respStr = HttpClientUtil.doPost(url, jsonStr, "utf-8");//String httpOrgCreateTestRtn = HttpClientUtil.doPost(url, jsonStr, "utf-8");//fastjson 将 String 转换为 jsonObjectJSONObject goodData = JSON.parseObject(respStr);JSONObject data = (JSONObject) goodData.get("data");String msg = (String) goodData.get("msg");int code = (int) goodData.get("code");int count = (int) data.get("total");JSONArray rows = (JSONArray) data.get("rows");List<GoodsData> list = new ArrayList<>();for (int i = 0; i < rows.size(); i++) {JSONObject obj = (JSONObject) rows.get(i);GoodsData goodsData = new GoodsData();goodsData.setId((Integer) obj.get("id"));goodsData.setFoodName((String) obj.get("food_name"));goodsData.setProductionName((String) obj.get("production_name"));goodsData.setFoodMode((String) obj.get("food_mode"));goodsData.setCheckNum((Integer) obj.get("check_num"));goodsData.setQalifiedNum((Integer) obj.get("qualified_num"));goodsData.setUnqualifiedNum((Integer) obj.get("unqualified_num"));list.add(goodsData);}return Result.success(list,count);
}

httpUtil(自己写的工具包，包含SSLClient.java和HttpClientUtil.java）

SSLClient.java

package com.neu.foodchain.common.httpUtil;import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import javax.net.ssl.SSLContext;
import javax.net.ssl.TrustManager;
import javax.net.ssl.X509TrustManager;
import org.apache.http.conn.ClientConnectionManager;
import org.apache.http.conn.scheme.Scheme;
import org.apache.http.conn.scheme.SchemeRegistry;
import org.apache.http.conn.ssl.SSLSocketFactory;
import org.apache.http.impl.client.DefaultHttpClient;public class SSLClient extends DefaultHttpClient {public SSLClient() throws Exception{super();SSLContext ctx = SSLContext.getInstance("TLS");X509TrustManager tm = new X509TrustManager() {@Overridepublic void checkClientTrusted(X509Certificate[] chain,String authType) throws CertificateException {}@Overridepublic void checkServerTrusted(X509Certificate[] chain,String authType) throws CertificateException {}@Overridepublic X509Certificate[] getAcceptedIssuers() {return null;}};ctx.init(null, new TrustManager[]{tm}, null);SSLSocketFactory ssf = new SSLSocketFactory(ctx,SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER);ClientConnectionManager ccm = this.getConnectionManager();SchemeRegistry sr = ccm.getSchemeRegistry();sr.register(new Scheme("https", 443, ssf));}
}

HttpClientUtil.java

package com.neu.foodchain.common.httpUtil;import com.alibaba.fastjson.JSONObject;
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.NameValuePair;
import org.apache.http.client.HttpClient;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.message.BasicNameValuePair;
import org.apache.http.util.EntityUtils;
import java.io.UnsupportedEncodingException;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;/*** 利用HttpClient进行post请求的工具类*/
public class HttpClientUtil {public static String doPost(String url, JSONObject jsonstr, String charset){HttpClient httpClient = null;HttpPost httpPost = null;String result = null;try{httpClient = new SSLClient();httpPost = new HttpPost(url);List<NameValuePair> paramList = new ArrayList<>();if(jsonstr != null && jsonstr.size() > 0){Set<String> keySet = jsonstr.keySet();for(String key : keySet) {paramList.add(new BasicNameValuePair(key, String.valueOf(jsonstr.get(key))));}}UrlEncodedFormEntity formEntity = null;try {formEntity = new UrlEncodedFormEntity(paramList,"utf8");} catch (UnsupportedEncodingException e) {e.printStackTrace();}//(5)设置表单的Entity对象到Post请求中httpPost.setEntity(formEntity);HttpResponse response = httpClient.execute(httpPost);if(response != null){HttpEntity resEntity = response.getEntity();if(resEntity != null){result = EntityUtils.toString(resEntity,charset);}}}catch(Exception ex){ex.printStackTrace();}return result;}
}

layui实现分页

<!DOCTYPE html>
<html class="x-admin-sm"><head><meta charset="UTF-8"><title>食品安全</title><meta name="renderer" content="webkit"><meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"><meta name="viewport" content="width=device-width,user-scalable=yes, minimum-scale=0.4, initial-scale=0.8,target-densitydpi=low-dpi" /><link rel="stylesheet" href="css/font.css"><link rel="stylesheet" href="css/xadmin.css"><script src="lib/layui/layui.js" charset="utf-8"></script><script type="text/javascript" src="js/xadmin.js"></script><!--[if lt IE 9]><script src="https://cdn.staticfile.org/html5shiv/r29/html5.min.js"></script><script src="https://cdn.staticfile.org/respond.js/1.4.2/respond.min.js"></script><![endif]--></head><body><div class="x-nav"><span class="layui-breadcrumb"><a href="">食品安全</a><a><cite>食品安全</cite></a></span><a class="layui-btn layui-btn-small" style="line-height:1.6em;margin-top:3px;float:right" onclick="location.reload()" title="刷新"><i class="layui-icon layui-icon-refresh" style="line-height:30px"></i></a></div><div class="layui-fluid"><div class="layui-row layui-col-space15"><div class="layui-col-md12"><div class="layui-card"><div class="layui-card-body "><div class="layui-form layui-col-space5"><div class="layui-inline layui-show-xs-block"><input type="text" name="pro_code" id="pro_code" placeholder="请输入商品追溯码" autocomplete="off" class="layui-input"></div><div class="layui-inline layui-show-xs-block"><input type="text" name="pro_bar_code" id="pro_bar_code" placeholder="请输入商品条码" autocomplete="off" class="layui-input"></div><div class="layui-inline layui-show-xs-block"><input type="text" name="manu_code" id="manu_code" placeholder="请输入生产批号" autocomplete="off" class="layui-input"></div><div class="layui-inline layui-show-xs-block"><button class="layui-btn" lay-submit="" lay-filter="sreach" id="search"><i class="layui-icon"></i></button></div><a class="layui-btn layui-btn-small" style="line-height:1.6em;margin-top:3px;float:right" onclick="desc()" title="使用说明"><i class="layui-icon layui-icon-tips" style="line-height:30px"></i></a></div></div><div class="layui-card-body "><table id="goodsData" lay-filter="demoEvent"></table></div></div></div></div></div></body><script>layui.use(['table','form', 'layedit', 'laydate','element','jquery','layer'], function() {var form = layui.form,layer = layui.layer,element = layui.element,$ = layui.jquery,laydate = layui.laydate,table = layui.table;var url = "/goods/data/";renderProductTable(url);function renderProductTable(url) {table.render({elem: '#goodsData'//,height: 312,url: url //数据接口,page: true //开启分页,cols: [[ //表头{field: 'id', title: 'ID', width:230, sort: true, fixed: 'left'},{field: 'foodName',width:150, title: '食品名称'},{field: 'productionName',width:150, title: '企业名称'},{field: 'foodMode', title: '包装方式', width:200, sort: true},{field: 'checkNum', title: '抽检数量', width:80},{field: 'qalifiedNum', title: '合格数量', width:80},{field: 'unqualifiedNum', title: '不合格数量', width:80}]]});};});</script><script>var _hmt = _hmt || []; (function() {var hm = document.createElement("script");hm.src = "https://hm.baidu.com/hm.js?b393d153aeb26b46e9431fabaf0f6190";var s = document.getElementsByTagName("script")[0];s.parentNode.insertBefore(hm, s);})();</script></html>

httpClient(java http/https)（二）（layui分页+httpclient+https+post）相关推荐

Java爬虫系列二：使用HttpClient抓取页面HTML
爬虫要想爬取需要的信息,首先第一步就要抓取到页面html内容,然后对html进行分析,获取想要的内容.上一篇随笔<Java爬虫系列一:写在开始前>中提到了HttpClient可以抓取页面内 ...
java爬虫之基于httpclient的简单Demo(二)
转载自 java爬虫之基于httpclient的简单Demo(二) 延续demo1的 java爬虫的2种爬取方式(HTTP||Socket)简单Demo(一),demo2出炉啦,大家想学爬虫都可以从这 ...
Java爬虫（二）-- httpClient模拟Http请求+jsoup页面解析
博客学院下载 GitChat TinyMind 论坛 APP 问答商城 VIP会员活动招聘 ITeye 写博客发Chat 传资源登录注册原 Java爬虫(二)-- httpClient ...
java实现layui分页_基于LayUI分页和LayUI laypage分页的使用示例
本文介绍了LayUI分页,LayUI动态分页,LayUI laypage分页,LayUI laypage刷新当前页,分享给大家,具体如下: 效果图: 一.引用js依赖主要是jquery-1.11. ...
【Java】如何优雅的使用HttpClient
HttpClient 是Apache Jakarta Common 下的子项目,可以用来提供高效的.最新的.功能丰富的支持 HTTP 协议的客户端编程工具包,并且它支持 HTTP 协议最新的版本和建议 ...
超详细HttpClient加载安全证书post请求访问https示例
超详细HttpClient加载安全证书post请求访问https示例 (注:此测试java版本为1.7,经过测试java7以上(不含java7)的版本无需加载安全证书,也无需绕过安全证书就可以通过建立 ...
HttpClient Java客户端
文章目录 HttpClient Java客户端建立使用HttpClient的环境 HttpGet请求 HttpGet带参请求 HttpPost请求 HttpPost带参请求(提交表单数据) Http ...
java爬取验证码图片_JAVA HttpClient实现页面信息抓取(获取图片验证码并传入cookie实现信息获取)...
JAVA HttpClient实现页面信息抓取(获取图片验证码并传入cookie实现信息获取) 发布时间:2018-05-18 16:41, 浏览次数:632 , 标签: JAVA HttpClien ...
java url json字符串_使用HttpClient将URL中的JSON查询字符串发送到Web服务(Java)
我有一个我建立的Web服务...我现在要做的是发送一个简单的请求,其中包含一个从Tapestry Web应用程序到该Web服务的json查询字符串.我四处搜索,大多数人都说使用Apache HttpC ...

httpClient(java http/https)（二）（layui分页+httpclient+https+post）

前言

代码

httpClient(java http/https)（二）（layui分页+httpclient+https+post）相关推荐

最新文章

热门文章