使用Gzip加速网页的传输
使用Gzip加速网页的传输 - zhiqiangzhan的专栏 - 博客频道 - CSDN.NET
使用Gzip加速网页的传输
分类:
JAVA2009-10-26 10:33
368人阅读
评论(3)
收藏
举报博学,切问,近思--詹子知(http://blog.csdn.net/zhiqiangzhan)
日前笔者在使用HttpClient在处理大数据请求的时候,在连续发请求的时候经常会出现异常 java.io.IOException: chunked stream ended unexpectedly。使用HttpMethod的abort方法也不能完全避免这种异常的出现,但是对于小数据的请求,这种异常就基本上难得一见了。对于同样的页面请求,如何减少网络的数据传输量呢。众所周知,现在大部分的Web Server都是支持数据的压缩传输的。要知道,一般的网页内容经过压缩,大小可以减少到原来的20%以下,而对于纯英文为网站,网页内容更是可以减少到原来内容的5%以下。而要使Web Server对数据进行压缩传输,只需要在请求头上加入Accept-Encoding:gzip, deflate。
[java] view plaincopyprint?
- public HttpMethod createHttpMethod(String url, String type, NameValuePair[] params, String contentType) {
- HttpMethod method = null;
- if (type.equalsIgnoreCase("POST")) {
- method = new PostMethod(url);
- method.setRequestHeader("Content-Type", contentType);
- if(params != null){
- ((PostMethod) method).setRequestBody(params);
- }
- } else {
- method = new GetMethod(url);
- if(params != null){
- method.setQueryString(params);
- }
- }
- method.setRequestHeader("Accept-Encoding", "gzip, deflate");
- return method;
- }
public HttpMethod createHttpMethod(String url, String type, NameValuePair[] params, String contentType) {<br /> HttpMethod method = null;</p> <p> if (type.equalsIgnoreCase("POST")) {<br /> method = new PostMethod(url);<br /> method.setRequestHeader("Content-Type", contentType);<br /> if(params != null){<br /> ((PostMethod) method).setRequestBody(params);<br /> }<br /> } else {<br /> method = new GetMethod(url);<br /> if(params != null){<br /> method.setQueryString(params);<br /> }<br /> }<br /> method.setRequestHeader("Accept-Encoding", "gzip, deflate");<br /> return method;<br /> }
这个时候,如果你请求的Web Server支持Gzip,返回来的响应便是被压缩后的数据,那么把压缩后的数据解析成原来的网页内容便是客户端要做的事情了。对于当前的主流浏览器,都是支持对压缩数据自动解压的,而在我们的应用程序中,我们只要对象网页流稍作处理,便可以得到原来的网页内容。
[java] view plaincopyprint?
- protected String doSuccess(HttpMethod method) throws IOException {
- InputStream in = method.getResponseBodyAsStream();
- Header contentEncodingHeader = method.getResponseHeader("Content-Encoding");
- if (contentEncodingHeader != null) {
- String contentEncoding = contentEncodingHeader.getValue();
- if (contentEncoding.toLowerCase(Locale.US).indexOf("gzip") != -1) {
- in = new GZIPInputStream(in);
- }
- }
- return decoder.decode(in);
- }
protected String doSuccess(HttpMethod method) throws IOException {<br /> InputStream in = method.getResponseBodyAsStream();<br /> Header contentEncodingHeader = method.getResponseHeader("Content-Encoding");<br /> if (contentEncodingHeader != null) {<br /> String contentEncoding = contentEncodingHeader.getValue();<br /> if (contentEncoding.toLowerCase(Locale.US).indexOf("gzip") != -1) {<br /> in = new GZIPInputStream(in);<br /> }<br /> }<br /> return decoder.decode(in);<br /> }
上一篇文章,我们介绍了如何检查文档输入流的编码,本节我们就可以利用上文的HtmlInputStreamDecoder类来把文档流来解析文档内容。完整的代码如下:
[java] view plaincopyprint?
- import java.io.IOException;
- import java.io.InputStream;
- import java.util.Locale;
- import java.util.zip.GZIPInputStream;
- import org.apache.commons.httpclient.Header;
- import org.apache.commons.httpclient.HttpClient;
- import org.apache.commons.httpclient.HttpException;
- import org.apache.commons.httpclient.HttpMethod;
- import org.apache.commons.httpclient.HttpStatus;
- import org.apache.commons.httpclient.MultiThreadedHttpConnectionManager;
- import org.apache.commons.httpclient.NameValuePair;
- import org.apache.commons.httpclient.URI;
- import org.apache.commons.httpclient.URIException;
- import org.apache.commons.httpclient.methods.GetMethod;
- import org.apache.commons.httpclient.methods.PostMethod;
- import org.apache.log4j.Logger;
- public class HttpRequest {
- private static final Logger LOGGER = Logger.getLogger(HttpRequest.class);
- private HttpClient client;
- private HtmlInputStreamDecoder decoder;
- public HttpRequest() {
- this(new HttpClient(new MultiThreadedHttpConnectionManager()));
- }
- public HttpRequest(HttpClient client) {
- this.client = client;
- this.decoder = new HtmlInputStreamDecoder();
- }
- public String doRequest(HttpMethod method) {
- String html = null;
- try {
- int statusCode = client.executeMethod(method);
- switch (statusCode) {
- case HttpStatus.SC_OK:
- html = doSuccess(method);
- break;
- case HttpStatus.SC_MOVED_PERMANENTLY:
- case HttpStatus.SC_MOVED_TEMPORARILY:
- case HttpStatus.SC_SEE_OTHER:
- case HttpStatus.SC_TEMPORARY_REDIRECT:
- doRedirect(method);
- break;
- default:
- html = doError(method);
- }
- } catch (HttpException e) {
- LOGGER.error("Http error occur while visit the url.", e);
- } catch (IOException e) {
- LOGGER.error("IO error occur while visit the url.", e);
- } finally {
- method.abort();
- method.releaseConnection();
- }
- return html;
- }
- protected String doSuccess(HttpMethod method) throws IOException {
- InputStream in = method.getResponseBodyAsStream();
- Header contentEncodingHeader = method.getResponseHeader("Content-Encoding");
- if (contentEncodingHeader != null) {
- String contentEncoding = contentEncodingHeader.getValue();
- if (contentEncoding.toLowerCase(Locale.US).indexOf("gzip") != -1) {
- in = new GZIPInputStream(in);
- }
- }
- return decoder.decode(in);
- }
- protected String doError(HttpMethod method) {
- LOGGER.error("Error Response: " + method.getStatusLine());
- return method.getStatusText();
- }
- protected void doRedirect(HttpMethod method) throws URIException {
- Header locationHeader = method.getResponseHeader("location");
- if (locationHeader != null) {
- String location = locationHeader.getValue();
- if (location == null) {
- location = "/";
- }
- doRequest(new GetMethod(getRedirectUrl(method.getURI(), location)));
- }
- }
- public HttpMethod createHttpMethod(String url, String type, NameValuePair[] params, String contentType) {
- HttpMethod method = null;
- if (type.equalsIgnoreCase("POST")) {
- method = new PostMethod(url);
- method.setRequestHeader("Content-Type", contentType);
- if(params != null){
- ((PostMethod) method).setRequestBody(params);
- }
- } else {
- method = new GetMethod(url);
- if(params != null){
- method.setQueryString(params);
- }
- }
- method.setRequestHeader("Accept-Encoding", "gzip, deflate");
- return method;
- }
- protected static String getRedirectUrl(URI origin, String location) throws URIException {
- String redirect = null;
- if (location.startsWith("http:")) {
- redirect = location;
- } else if (location.startsWith("/")) {
- origin.setPath(location);
- redirect = origin.getURI();
- } else {
- redirect = origin.getURI().replaceAll("(?<=/)[^/]+$", location);
- }
- return redirect;
- }
- }
import java.io.IOException;<br /> import java.io.InputStream;<br /> import java.util.Locale;<br /> import java.util.zip.GZIPInputStream;</p> <p>import org.apache.commons.httpclient.Header;<br /> import org.apache.commons.httpclient.HttpClient;<br /> import org.apache.commons.httpclient.HttpException;<br /> import org.apache.commons.httpclient.HttpMethod;<br /> import org.apache.commons.httpclient.HttpStatus;<br /> import org.apache.commons.httpclient.MultiThreadedHttpConnectionManager;<br /> import org.apache.commons.httpclient.NameValuePair;<br /> import org.apache.commons.httpclient.URI;<br /> import org.apache.commons.httpclient.URIException;<br /> import org.apache.commons.httpclient.methods.GetMethod;<br /> import org.apache.commons.httpclient.methods.PostMethod;<br /> import org.apache.log4j.Logger;</p> <p>public class HttpRequest {<br /> private static final Logger LOGGER = Logger.getLogger(HttpRequest.class);<br /> private HttpClient client;<br /> private HtmlInputStreamDecoder decoder;</p> <p> public HttpRequest() {<br /> this(new HttpClient(new MultiThreadedHttpConnectionManager()));<br /> }</p> <p> public HttpRequest(HttpClient client) {<br /> this.client = client;<br /> this.decoder = new HtmlInputStreamDecoder();<br /> }</p> <p> public String doRequest(HttpMethod method) {<br /> String html = null;<br /> try {<br /> int statusCode = client.executeMethod(method);<br /> switch (statusCode) {<br /> case HttpStatus.SC_OK:<br /> html = doSuccess(method);<br /> break;<br /> case HttpStatus.SC_MOVED_PERMANENTLY:<br /> case HttpStatus.SC_MOVED_TEMPORARILY:<br /> case HttpStatus.SC_SEE_OTHER:<br /> case HttpStatus.SC_TEMPORARY_REDIRECT:<br /> doRedirect(method);<br /> break;<br /> default:<br /> html = doError(method);<br /> }<br /> } catch (HttpException e) {<br /> LOGGER.error("Http error occur while visit the url.", e);<br /> } catch (IOException e) {<br /> LOGGER.error("IO error occur while visit the url.", e);<br /> } finally {<br /> method.abort();<br /> method.releaseConnection();<br /> }<br /> return html;<br /> }</p> <p> protected String doSuccess(HttpMethod method) throws IOException {<br /> InputStream in = method.getResponseBodyAsStream();<br /> Header contentEncodingHeader = method.getResponseHeader("Content-Encoding");<br /> if (contentEncodingHeader != null) {<br /> String contentEncoding = contentEncodingHeader.getValue();<br /> if (contentEncoding.toLowerCase(Locale.US).indexOf("gzip") != -1) {<br /> in = new GZIPInputStream(in);<br /> }<br /> }<br /> return decoder.decode(in);<br /> }</p> <p> protected String doError(HttpMethod method) {<br /> LOGGER.error("Error Response: " + method.getStatusLine());<br /> return method.getStatusText();<br /> }</p> <p> protected void doRedirect(HttpMethod method) throws URIException {<br /> Header locationHeader = method.getResponseHeader("location");<br /> if (locationHeader != null) {<br /> String location = locationHeader.getValue();<br /> if (location == null) {<br /> location = "/";<br /> }<br /> doRequest(new GetMethod(getRedirectUrl(method.getURI(), location)));<br /> }<br /> }</p> <p> public HttpMethod createHttpMethod(String url, String type, NameValuePair[] params, String contentType) {<br /> HttpMethod method = null;</p> <p> if (type.equalsIgnoreCase("POST")) {<br /> method = new PostMethod(url);<br /> method.setRequestHeader("Content-Type", contentType);<br /> if(params != null){<br /> ((PostMethod) method).setRequestBody(params);<br /> }<br /> } else {<br /> method = new GetMethod(url);<br /> if(params != null){<br /> method.setQueryString(params);<br /> }<br /> }<br /> method.setRequestHeader("Accept-Encoding", "gzip, deflate");<br /> return method;<br /> }</p> <p> protected static String getRedirectUrl(URI origin, String location) throws URIException {<br /> String redirect = null;<br /> if (location.startsWith("http:")) {<br /> redirect = location;<br /> } else if (location.startsWith("/")) {<br /> origin.setPath(location);<br /> redirect = origin.getURI();<br /> } else {<br /> redirect = origin.getURI().replaceAll("(?<=/)[^/]+$", location);<br /> }<br /> return redirect;<br /> }</p> <p>}
代码示例:
[java] view plaincopyprint?
- public static void main(String[] args) {
- HttpRequest request = new HttpRequest();
- HttpMethod method = request.createHttpMethod("http://www.csdn.com", "GET", null, "text/html");
- String html = request.doRequest(method);
- System.out.println(html);
- }
public static void main(String[] args) {</p> <p> HttpRequest request = new HttpRequest();<br /> HttpMethod method = request.createHttpMethod("http://www.csdn.com", "GET", null, "text/html");</p> <p> String html = request.doRequest(method);<br /> System.out.println(html);<br /> }
转载于:https://www.cnblogs.com/lexus/archive/2012/03/02/2376783.html
使用Gzip加速网页的传输相关推荐
- 如何给网页加速,如何加速网页速度?
如何加速网页速度? 提高移动网页加载的速度,可以从服务器的优化.网页的容量.请求响应等方面入手,这些方面优化后必然可以提高加载速度. 1.服务器硬件软件配置要好,网络.读写响应等要做好优化. 2.可以 ...
- WordPress 开启 Gzip 为网页加载提速减少响应时间
2019独角兽企业重金招聘Python工程师标准>>> 大家都晓得,开启Gzip能极大地压缩文本数据的体积.对于使用 WordPress 的博主来说,开启服务器的GZip压缩是一个为 ...
- java gzip delphi_delphi 利用indy解码Gzip的网页
求人不如求已..唉... 直接用Fhttp.Compressor:= TIdCompressorZLibEx.Create(nil); 就OK了.. 另外Delphi2007下.. idhttp. ...
- cdn加速和oss传输加速的区别。
概述 本文主要介绍剑盾云对象存储OSS的传输加速功能和CDN加速OSS资源的区别,以便您根据实际业务进行选择. 详细信息 剑盾云对象存储OSS以海量.安全.低成本.高可靠等特点已经成为用户存储静态资源 ...
- 使用雪碧图Css Sprite精灵 | 加速网页响应速度
什么是CSS Sprite精灵? 是用于前端的一种图片应用技术,通常情况,我们的开发的网页或许有很多张图片,假如在一个页面上有50多张小图片,这意味着浏览器要逐个下载50张图片.Css Sprite它 ...
- linux上传网页文件大小,Apache启用GZIP压缩网页传输方法
首先我们先了解Apache Gzip的相关资料. 一.gzip介绍 Gzip是一种流行的文件压缩算法,现在的应用十分广泛,尤其是在Linux平台.当应用Gzip压缩到一个纯文本文件时,效果是非常明显的 ...
- Apache启用GZIP压缩网页传输
首先我们先了解Apache Gzip的相关资料. 一.gzip介绍 Gzip是一种流行的文件压缩算法,现在的应用十分广泛,尤其是在Linux平台.当应用Gzip压缩到一个纯文本文件时,效果是非常明显的 ...
- 五招加速网页响应时间
1. 使用Yslow概览与测量网站加载时间 在决定什么出问题之前,知道网站的加载时间是第一步.它也能让你知道你是否需要为网站加速进行更改. 在我们开始之前,如果你还没有安装YSlow, 请安装.他是M ...
- 百度网盘解析加速网页版[演示站可用]
源码名称:百度网盘解析加速工具 网页版 源码环境:PHP7+MySQL 源码功能: 通过 curl 获取网盘文件信息,处理后显示在网页中.通过 api 接口以及 SVIP 账号的 Cookie(BDU ...
最新文章
- 人工智能项目的六投三不投
- raft论文 中文 pdf_八篇论文总结BERT+KG:预训练模型与知识图谱相结合的研究进展...
- SPOJ 1812 LCS2 - Longest Common Substring II (后缀自动机)【两种做法】
- GraphPad Prism多数据处理实用技巧
- 【深度学习】CVPR 2021 全部论文链接公布!最新1660篇论文合集!附下载链接
- 常见八种安卓开发报错的方式
- 谷歌云试图抢占SAP软件云市场;企业上云迎来“黄金时代”;IBM和SAP帮助金融机构加快采用云技术……...
- 安卓系统开机过程中logo和动画
- 理解有参构造器和无参构造器的作用
- 找出消费者在使用你的产品的时候,什么时刻觉得“值了”
- PHP获取文件夹内所有文件包括子目录文件的名称或路径
- 日期截取年月_2019年7月18日 期权交易日志——用小黄人教你理解期权平价公式...
- 学习WPF绝佳的去处……WPF教程,WPF入门教程,WPF视频教程
- 基于Windows 7环境的WAPI无线网络应用层控制实现
- jq-ui-multiselect插件的使用
- 术语-PM:PM/项目管理 百科
- 关于计算机网络以下说法哪个正确().,青书学堂: (多选题) 关于计算机网络,以下说法哪个正确?( )(本题4.0分)...
- 帮我写一段描写时间过得很快,但是自己又很不想时间过得那么快的小作文
- 下载学习通上没有下载权限的文件
- Java播放声音文件
热门文章
- Nlite精简后期处理总结
- 新工业化如何实现?今年的信息化百人会中藏着“懂行”密码
- 辽宁大学计算机专业专硕分数线,辽宁大学社会工作硕士历年复试分数线_社会工作考研分数线_复试线 - 希赛网...
- 郑州市信息技术类企业100强
- 折腾大半年,西部数据终于收购了东芝半导体业务
- D1net阅闻:贝恩资本等欲以190亿美元收购东芝半导体
- SpringBoot欢迎页支持——index.xml
- Syllable-Based Acoustic Modeling with CTC-SMBR-LSTM翻译
- mysql routeros_转-RouterOS流量控制方案
- Win11应用更新在哪里 Win11怎么更新应用