HttpClient

网络爬虫就是用程序帮助我们访问网络上的资源，我们一直以来都是使用HTTP协议访问互联网的网页，网络爬虫需要编写程序，在这里使用同样的HTTP协议访问网页。

这里使用Java的HTTP协议客户端 HttpClient这个技术，来实现抓取网页数据。

GET请求

package cn.itcast.crawler.test;import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;import java.io.IOException;public class HttpGetTest {public static void main(String[] args)  {//创建HttpClient对象CloseableHttpClient httpClient = HttpClients.createDefault();//创建HttpGet对象，设置url访问地址HttpGet httpGet = new HttpGet("http://www.itcast.cn");CloseableHttpResponse response = null;try {//使用HttpClient发起请求，获取responseresponse = httpClient.execute(httpGet);//解析响应if (response.getStatusLine().getStatusCode() == 200) {String content = EntityUtils.toString(response.getEntity(), "utf8");System.out.println(content.length());}} catch (IOException e) {e.printStackTrace();}finally {//关闭responsetry {response.close();} catch (IOException e) {e.printStackTrace();}try {httpClient.close();} catch (IOException e) {e.printStackTrace();}}}
}

GET请求

带参数的GET请求

package cn.itcast.crawler.test;import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.utils.URIBuilder;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;import java.io.IOException;
import java.net.URISyntaxException;public class HttpGetParamTest {public static void main(String[] args) throws Exception {//创建HttpClient对象CloseableHttpClient httpClient = HttpClients.createDefault();//设置请求地址是：http://yun.itheima.com/search?keys=Java//创建URIBuilderURIBuilder uriBuilder = new URIBuilder("http://yun.itheima.com/search");//设置参数uriBuilder.setParameter("keys","Java");//创建HttpGet对象，设置url访问地址HttpGet httpGet = new HttpGet(uriBuilder.build());System.out.println("发起请求的信息："+httpGet);CloseableHttpResponse response = null;try {//使用HttpClient发起请求，获取responseresponse = httpClient.execute(httpGet);//解析响应if (response.getStatusLine().getStatusCode() == 200) {String content = EntityUtils.toString(response.getEntity(), "utf8");System.out.println(content.length());}} catch (IOException e) {e.printStackTrace();}finally {//关闭responsetry {response.close();} catch (IOException e) {e.printStackTrace();}try {httpClient.close();} catch (IOException e) {e.printStackTrace();}}}
}

HttpClient ||GET请求||带参数的GET请求相关推荐

POST请求||带参数的POST请求
POST请求 HttpPostTest.java package cn.itcast.crawler.test;import org.apache.http.client.methods.Closea ...
使用httpclient进行 Http get请求带参数cookie
使用httpclient进行 Http get请求带参数cookie 常用的工具类,所有的包配置如下 <dependency><groupI ...
HttpClient Get请求带参数
/** HttpClient Get请求带参数*/@Testpublic void fun1() throws ClientProtocolException, IOException {// 1.创 ...
java http post 传参数_HttpClient之带参数的post请求
具体代码: package cn.itcast.crawler.test; import org.apache.http.HttpEntity; import org.apache.http.Name ...
python get请求带参数_python_request的安装及模拟json的post请求及带参数的get请求
一.Requests模块安装安装方式一: 执行 pip install -U requests 联网安装requests 安装方式二: 进入https://pypi.org/project/requ ...
requests之get请求带参数示例详解
具体见以下详细步骤获取响应数据获取请求头获取响应头.状态码有两点需要引起注意: a.响应结果一般有三种格式:HTML.json.text b.text方法返回的是字符串格式的内容获取cook ...
带参数的Get请求与不带参数的Get请求
不带参数的Get请求 [{"description": "这是一个不带参数的get请求","request": {"uri&quo ...
【安卓开发】Android初级开发(okhttp3发送带header与带参数的GET请求)
1.首先需要先加入相应依赖 dependencies{implementation 'com.squareup.okhttp3:okhttp:3.13.1' implementation 'com.s ...
React Native ---fetch 之GET请求带参数
网上关于其GET请求的用例全是没有带参数的,带参数的又全部是POST请求下面给出带参数的做法. export function get(url,params){if (params) {let par ...

HttpClient ||GET请求||带参数的GET请求

HttpClient

GET请求

GET请求

带参数的GET请求

HttpClient ||GET请求||带参数的GET请求相关推荐

最新文章

热门文章