url 转码 java_URL 转码遇到的坑
1. 背景
做图片中心时, 需要根据图片URL去下载图片, 发现部分URL浏览器里面能访问,但是代码无法下载
原因: 需要对URL进行转码
2. 使用UrlEncode
一开始直接使用UrlEncode对url转码, 如下
原始链接
https://s3.amazonaws.com/fromfactory.club.image/9f/0f/9fbd598c4681773f6371c27dfb64180f.webp
UrlEncode后
https%3a%2f%2fs3.amazonaws.com%2ffromfactory.club.image%2f9f%2f0f%2f9fbd598c4681773f6371c27dfb64180f.webp
2.1 问题
斜杠,冒号等字符也被UrlEncode进行了转码, 显然这样的转码后是不能下载的
3. 原因分析
编码的意义在于,假如URL的参数中的中文或特殊字符在发送到服务端时,服务端无法解析它的真正意义,会导致服务端不能理解客户端的请求, 此时, 需要对它进行编码
3.1 哪些字符需要编码
RFC3986文档规定,URL中只允许包含英文字母(a-zA-Z)、数字(0-9)、-_.~4个特殊字符以及所有保留字符。
保留字符:Url可以划分成若干个组件,协议、主机、路径等。有一些字符(:/?#[]@)是用作分隔不同组件的。
当组件中的普通数据包含这些特殊字符时,需要对其进行编码。
RFC3986中指定了以下字符为==保留字符:! * ’ ( ) ; : @ & = + $ , / ? # [ ]==
对于图片下载中, 如下都不转码
英文字母 a-zA-Z
数字 0-9
4个特殊字符 _.~、
保留字符 ! * ’ ( ) ; : @ & = + $ , / ? # [ ]==
3.2 UrlEncode不转码字符
UrlEncode中维护了dontNeedEncoding, 但是dontNeedEncoding只包含了
英文字母(a-zA-Z)
数字(0-9)
空格 - _ . *
ab8f20ac3c5a4b0f9beb02bd9be627fb_image.png
如下字符也需要不转码: ! ~ ’ ( ) ; : @ & = + $ , / ? # [ ]==
4. 解决方案
重写UrlEncode添加不转码字符
7be93685d8544476aa88d21c35ec3dd1_image.png
4.1 代码
package com.clubfactory.center.product.util;
import sun.security.action.GetPropertyAction;
import java.io.CharArrayWriter;
import java.io.UnsupportedEncodingException;
import java.nio.charset.Charset;
import java.nio.charset.IllegalCharsetNameException;
import java.nio.charset.UnsupportedCharsetException;
import java.security.AccessController;
import java.util.BitSet;
/**
* Utility class for HTML form encoding. This class contains static methods
* for converting a String to the application/x-www-form-urlencoded
MIME
* format. For more information about HTML form encoding, consult the HTML
* specification.
*
*
* When encoding a String, the following rules apply:
*
*
*
The alphanumeric characters "{@code a}" through
* "{@code z}", "{@code A}" through
* "{@code Z}" and "{@code 0}"
* through "{@code 9}" remain the same.
*
The special characters "{@code .}",
* "{@code -}", "{@code *}", and
* "{@code _}" remain the same.
*
The space character " " is
* converted into a plus sign "{@code +}".
*
All other characters are unsafe and are first converted into
* one or more bytes using some encoding scheme. Then each byte is
* represented by the 3-character string
* "{@code %xy}", where xy is the
* two-digit hexadecimal representation of the byte.
* The recommended encoding scheme to use is UTF-8. However,
* for compatibility reasons, if an encoding is not specified,
* then the default encoding of the platform is used.
*
*
*
* For example using UTF-8 as the encoding scheme the string "The
* string ü@foo-bar" would get converted to
* "The+string+%C3%BC%40foo-bar" because in UTF-8 the character
* ü is encoded as two bytes C3 (hex) and BC (hex), and the
* character @ is encoded as one byte 40 (hex).
*
* @author Herb Jellinek
* @since JDK1.0
*/
public class MyURIEncoder {
static BitSet dontNeedEncoding;
static final int caseDiff = ('a' - 'A');
static String dfltEncName = null;
static {
/* The list of characters that are not encoded has been
* determined as follows:
*
* RFC 2396 states:
* -----
* Data characters that are allowed in a URI but do not have a
* reserved purpose are called unreserved. These include upper
* and lower case letters, decimal digits, and a limited set of
* punctuation marks and symbols.
*
* unreserved = alphanum | mark
*
* mark = "-" | "_" | "." | "!" | "~" | "*" | "'" | "(" | ")"
*
* Unreserved characters can be escaped without changing the
* semantics of the URI, but this should not be done unless the
* URI is being used in a context that does not allow the
* unescaped character to appear.
* -----
*
* It appears that both Netscape and Internet Explorer escape
* all special characters from this list with the exception
* of "-", "_", ".", "*". While it is not clear why they are
* escaping the other characters, perhaps it is safest to
* assume that there might be contexts in which the others
* are unsafe if not escaped. Therefore, we will use the same
* list. It is also noteworthy that this is consistent with
* O'Reilly's "HTML: The Definitive Guide" (page 164).
*
* As a last note, Intenet Explorer does not encode the "@"
* character which is clearly not unreserved according to the
* RFC. We are being consistent with the RFC in this matter,
* as is Netscape.
*
*/
dontNeedEncoding = new BitSet(256);
int i;
for (i = 'a'; i <= 'z'; i++) {
dontNeedEncoding.set(i);
}
for (i = 'A'; i <= 'Z'; i++) {
dontNeedEncoding.set(i);
}
for (i = '0'; i <= '9'; i++) {
dontNeedEncoding.set(i);
}
//dontNeedEncoding.set(' '); /* encoding a space to a + is done * in the encode() method */
dontNeedEncoding.set('-');
dontNeedEncoding.set('_');
dontNeedEncoding.set('.');
dontNeedEncoding.set('*');
//对以下在 URI 中具有特殊含义的 ASCII 标点符号 ;/?:@&=+$,# 不需要转义
dontNeedEncoding.set(';');
dontNeedEncoding.set('/');
dontNeedEncoding.set('?');
dontNeedEncoding.set(':');
dontNeedEncoding.set('@');
dontNeedEncoding.set('&');
dontNeedEncoding.set('=');
dontNeedEncoding.set('+');
dontNeedEncoding.set('$');
dontNeedEncoding.set(',');
dontNeedEncoding.set('#');
dontNeedEncoding.set('!');
dontNeedEncoding.set('\'');
dontNeedEncoding.set('(');
dontNeedEncoding.set(')');
dontNeedEncoding.set('%');
dontNeedEncoding.set('[');
dontNeedEncoding.set(']');
dfltEncName = AccessController.doPrivileged(
new GetPropertyAction("file.encoding")
);
}
/**
* You can't call the constructor.
*/
private MyURIEncoder() { }
/**
* Translates a string into {@code x-www-form-urlencoded}
* format. This method uses the platform's default encoding
* as the encoding scheme to obtain the bytes for unsafe characters.
*
* @param s {@code String} to be translated.
* @deprecated The resulting string may vary depending on the platform's
* default encoding. Instead, use the encode(String,String)
* method to specify the encoding.
* @return the translated {@code String}.
*/
@Deprecated
public static String encode(String s) {
String str = null;
try {
str = encode(s, dfltEncName);
} catch (UnsupportedEncodingException e) {
// The system should always have the platform default
}
return str;
}
/**
* Translates a string into {@code application/x-www-form-urlencoded}
* format using a specific encoding scheme. This method uses the
* supplied encoding scheme to obtain the bytes for unsafe
* characters.
*
* Note: The
* "http://www.w3.org/TR/html40/appendix/notes.html#non-ascii-chars">
* World Wide Web Consortium Recommendation
states that
* UTF-8 should be used. Not doing so may introduce
* incompatibilities.
*
* @param s {@code String} to be translated.
* @param enc The name of a supported
* character
* encoding.
* @return the translated {@code String}.
* @exception UnsupportedEncodingException
* If the named encoding is not supported
* @since 1.4
*/
public static String encode(String s, String enc)
throws UnsupportedEncodingException {
boolean needToChange = false;
StringBuffer out = new StringBuffer(s.length());
Charset charset;
CharArrayWriter charArrayWriter = new CharArrayWriter();
if (enc == null)
throw new NullPointerException("charsetName");
try {
charset = Charset.forName(enc);
} catch (IllegalCharsetNameException e) {
throw new UnsupportedEncodingException(enc);
} catch (UnsupportedCharsetException e) {
throw new UnsupportedEncodingException(enc);
}
for (int i = 0; i < s.length();) {
int c = (int) s.charAt(i);
if (dontNeedEncoding.get(c)) {
out.append((char)c);
i++;
} else {
// convert to external encoding before hex conversion
do {
charArrayWriter.write(c);
/*
* If this character represents the start of a Unicode
* surrogate pair, then pass in two characters. It's not
* clear what should be done if a bytes reserved in the
* surrogate pairs range occurs outside of a legal
* surrogate pair. For now, just treat it as if it were
* any other character.
*/
if (c >= 0xD800 && c <= 0xDBFF) {
/*
System.out.println(Integer.toHexString(c)
+ " is high surrogate");
*/
if ( (i+1) < s.length()) {
int d = (int) s.charAt(i+1);
/*
System.out.println("\tExamining "
+ Integer.toHexString(d));
*/
if (d >= 0xDC00 && d <= 0xDFFF) {
/*
System.out.println("\t"
+ Integer.toHexString(d)
+ " is low surrogate");
*/
charArrayWriter.write(d);
i++;
}
}
}
i++;
} while (i < s.length() && !dontNeedEncoding.get((c = (int) s.charAt(i))));
charArrayWriter.flush();
String str = new String(charArrayWriter.toCharArray());
byte[] ba = str.getBytes(charset);
for (int j = 0; j < ba.length; j++) {
out.append('%');
char ch = Character.forDigit((ba[j] >> 4) & 0xF, 16);
// converting to use uppercase letter as part of
// the hex value if ch is a letter.
if (Character.isLetter(ch)) {
ch -= caseDiff;
}
out.append(ch);
ch = Character.forDigit(ba[j] & 0xF, 16);
if (Character.isLetter(ch)) {
ch -= caseDiff;
}
out.append(ch);
}
charArrayWriter.reset();
needToChange = true;
}
}
return (needToChange? out.toString() : s);
}
}
url 转码 java_URL 转码遇到的坑相关推荐
- 根据url动态生成二维码
根据url动态生成二维码的工具类,没有细研究,先贴出来,有兴趣自己深究.public class QRCodeUtil {public static BufferedImage createImage ...
- 根据url地址生成二维码,微信扫描二维码可直接打开网址
需求:根据url地址生成二维码,微信扫描二维码可直接打开网址 html代码: <input id="text" type="text" value=&qu ...
- 解析网络URL地址的二维码
java解析url二维码 解析网络URL地址的二维码 解析网络URL地址的二维码 @Testvoid contextLoads() throws Exception {StringBuffer sb ...
- HTML简洁大气带进度条的URL跳转页面源码
正文: HML简洁大气带进度条的URL跳转页面源码,这程序没什么多介绍的,主要是可用于网站内链接跳转,或者广告链接跳转啥的,UI非常的美观好看. 程序: lanzou.com/iONLE0511hsd ...
- JAVA 实现扫码二维码登录
最近在做一个扫码登录功能,为此我还在网上搜了一下关于微信的扫描登录的实现方式.当这个功能完成了后,我决定将整个实现思路整理出来,方便自己以后查看也方便其他有类似需求的程序猿些. 要实现扫码登录我们需要 ...
- C(++) Websocket实现扫码二维码登录---GoEasy
最近在做一个扫码登录功能,为此我还在网上搜了一下关于微信的扫描登录的实现方式.当这个功能完成了后,我决定将整个实现思路整理出来,方便自己以后查看也方便其他有类似需求的程序猿些. 要实现扫码登录我们需要 ...
- node.js 实现扫码二维码登录
最近在做一个扫码登录功能,为此我还在网上搜了一下关于微信的扫描登录的实现方式.当这个功能完成了后,我决定将整个实现思路整理出来,方便自己以后查看也方便其他有类似需求的程序猿些. 要实现扫码登录我们需要 ...
- 302状态码_HTTP状态码status code详解
http状态码可以让我很方便的了解到请求的所在状态,所以很有必要总结一下,对今后的学习也是很有帮助的. 什么是HTTP状态码 HTTP状态码的作用是:web服务器用来告诉客户端,发生了什么事. 状态码 ...
- VB.net开发微信、支付宝扫码支付源码
扫码消费机介绍:https://item.taobao.com/item.htm?spm=a1z10.1-c.w4004-21914722028.2.2b826baawDkx32&id=170 ...
- h5 实现扫码二维码及条形码(js多种实现方式)
方式一. 只识别二维码 实现方式一 jsQR 个人预览页面网址只扫码二维码 GitHub jsQR inde.html <!DOCTYPE html> <html><he ...
最新文章
- ElasticSearch + xpack 使用
- 英特尔大地震!解雇首席工程官,7纳米延期,或面临集体诉讼……
- gets函数在使用上要注意什么问题
- 0048-三角形的判断
- 重构手法之重新组织数据【1】
- calce大学电池实验数据怎么看_对比太明显!亲眼见证刀片电池VS三元锂电池针刺实验...
- DSP 6678的NETCP
- linux命令键盘快捷键,如何将Linux命令设置成键盘快捷键
- DLL注入 + VEH 的方式处理异常
- lvgl chart
- Python Gstreamer播放rtsp视频流(海康IPCAM)
- 阿里天池工业蒸汽量预测baseline-数据探索篇
- 一文让你读懂什么是智慧数字经营
- Redis键-值数据库 nosql 数据建模(3)------ 如何存储主从表数据 一对多关系
- postman传图片
- 2022.3.2复盘
- 论计算机的维修策略论文,论计算机的维护维修策略(论文).doc
- 俗话说:十赌九输。因为大多数赌局的背后都藏有阴谋。不过也不尽然,有些赌局背后藏有的是:“阳谋”。
- mui简单的详情页面
- 观察装柜一个月的装柜波动
热门文章
- word流程图怎么做虚线框_word虚线框怎么打 word中目录虚线怎么打?
- 如何理解阿里月饼事件中各方的表现
- 以编程方式打印 XPS 文件
- Arduino 和 HC-SR04 超声波传感器 测距
- linux编译安卓源码,Ubuntu下编译Android源码
- 行为识别---不同模型的帧采样策略
- 年薪50万的程序员_程序员年薪50万 ! 工资是不是太高了?
- 需要在计算机安装msxml版本,win7 Office2010提示安装MSXML版本6.10.1129.0怎么办
- 数学建模:线性规划—投资的收益和风险模型 (Python 求解)
- 【React自制全家桶】九、Redux入手