最近有个需求需要对于获取URL页面进行host绑定并且立即生效,在java里面实现可以用代理服务器来实现:因为在测试环境下可能需要通过绑定来访问测试环境的应用
实现代码如下:

public static String getResponseText(String queryUrl,String host,String ip) { //queryUrl,完整的url,host和ip需要绑定的host和ip
InputStream is = null;
BufferedReader br = null;
StringBuffer res = new StringBuffer();
try {
HttpURLConnection httpUrlConn = null;
URL url = new URL(queryUrl);
if(ip!=null){
String str[] = ip.split("\\.");
byte[] b =new byte[str.length];
for(int i=0,len=str.length;i<len;i++){
b[i] = (byte)(Integer.parseInt(str[i],10));
}
Proxy proxy = new Proxy(Proxy.Type.HTTP,
new InetSocketAddress(InetAddress.getByAddress(b), 80)); //b是绑定的ip,生成proxy代理对象,因为http底层是socket实现,
httpUrlConn = (HttpURLConnection) url
.openConnection(proxy);
}else{
httpUrlConn = (HttpURLConnection) url
.openConnection();
}
httpUrlConn.setRequestMethod("GET");
httpUrlConn.setDoOutput(true);
httpUrlConn.setConnectTimeout(2000);
httpUrlConn.setReadTimeout(2000);
httpUrlConn.setDefaultUseCaches(false);
httpUrlConn.setUseCaches(false);

is = httpUrlConn.getInputStream();

那么底层对于proxy对象到底是怎么处理,底层的socket实现到底怎么样,带着这个疑惑看了下jdk的rt.jar对于这块的处理
httpUrlConn = (HttpURLConnection) url.openConnection(proxy)

java.net.URL类里面的openConnection方法:
public URLConnection openConnection(Proxy proxy){

return handler.openConnection(this, proxy); Handler是sun.net.www.protocol.http.Handler.java类,继承java.net. URLStreamHandler.java类,用来处理http连接请求响应的。
}

Handler的方法:
protected java.net.URLConnection openConnection(URL u, Proxy p)
throws IOException {
return new HttpURLConnection(u, p, this);
}

只是简单的生成sun.net.www.protocl.http.HttpURLConnection对象,并进行初始化
protected HttpURLConnection(URL u, Proxy p, Handler handler) {
super(u);
requests = new MessageHeader(); 请求头信息生成类
responses = new MessageHeader(); 响应头信息解析类
this.handler = handler;
instProxy = p; 代理服务器对象
cookieHandler = (CookieHandler)java.security.AccessController.doPrivileged(
new java.security.PrivilegedAction() {
public Object run() {
return CookieHandler.getDefault();
}
});
cacheHandler = (ResponseCache)java.security.AccessController.doPrivileged(
new java.security.PrivilegedAction() {
public Object run() {
return ResponseCache.getDefault();
}
});
}

最终在httpUrlConn.getInputStream();才进行socket连接,发送http请求,解析http响应信息。具体过程如下:

sun.net.www.protocl.http.HttpURLConnection.java的getInputStream方法:

public synchronized InputStream getInputStream() throws IOException {

...socket连接
connect();
...
ps = (PrintStream)http.getOutputStream(); 获得输出流,打开连接之后已经生成。

if (!streaming()) {
writeRequests(); 输出http请求头信息
}
...
http.parseHTTP(responses, pi, this); 解析响应信息
if(logger.isLoggable(Level.FINEST)) {
logger.fine(responses.toString());
}
inputStream = http.getInputStream(); 获得输入流
}

其中connect()调用方法链:
plainConnect(){
...
Proxy p = null;
if (sel != null) {
URI uri = sun.net.www.ParseUtil.toURI(url);
Iterator<Proxy> it = sel.select(uri).iterator();
while (it.hasNext()) {
p = it.next();
try {
if (!failedOnce) {
http = getNewHttpClient(url, p, connectTimeout);
...
}

getNewHttpClient(){
...
return HttpClient.New(url, p, connectTimeout, useCache);
...
}

下面跟进去最终建立socket连接的代码:
sun.net.www.http.HttpClient.java的openServer()方法建立socket连接:

protected synchronized void openServer() throws IOException {
...
if ((proxy != null) && (proxy.type() == Proxy.Type.HTTP)) {
sun.net.www.URLConnection.setProxiedHost(host);
if (security != null) {
security.checkConnect(host, port);
}
privilegedOpenServer((InetSocketAddress) proxy.address());最终socket连接的是设置的代理服务器的地址,
...
}

private synchronized void privilegedOpenServer(final InetSocketAddress server)
throws IOException
{
try {
java.security.AccessController.doPrivileged(
new java.security.PrivilegedExceptionAction() {
public Object run() throws IOException {
openServer(server.getHostName(), server.getPort()); 注意openserver函数 这里的server的getHostName是设置的代理服务器,(ip或者hostname,如果是host绑定设置的代理服务器的ip,那么这里getHostName出来的就是ip地址,可以去查看InetSocketAddress类的getHostName方法)
return null;
}
});
} catch (java.security.PrivilegedActionException pae) {
throw (IOException) pae.getException();
}
}

public void openServer(String server, int port) throws IOException {
serverSocket = doConnect(server, port); 生成的Socket连接对象
try {
serverOutput = new PrintStream(
new BufferedOutputStream(serverSocket.getOutputStream()),
false, encoding); 生成输出流,
} catch (UnsupportedEncodingException e) {
throw new InternalError(encoding+" encoding not found");
}
serverSocket.setTcpNoDelay(true);
}

protected Socket doConnect (String server, int port)
throws IOException, UnknownHostException {
Socket s;
if (proxy != null) {
if (proxy.type() == Proxy.Type.SOCKS) {
s = (Socket) AccessController.doPrivileged(
new PrivilegedAction() {
public Object run() {
return new Socket(proxy);
}});
} else
s = new Socket(Proxy.NO_PROXY);
} else
s = new Socket();
// Instance specific timeouts do have priority, that means
// connectTimeout & readTimeout (-1 means not set)
// Then global default timeouts
// Then no timeout.
if (connectTimeout >= 0) {
s.connect(new InetSocketAddress(server, port), connectTimeout);
} else {
if (defaultConnectTimeout > 0) {
s.connect(new InetSocketAddress(server, port), defaultConnectTimeout);//连接到代理服务器,看下面Socket类的connect方法代码
} else {
s.connect(new InetSocketAddress(server, port));
}
}
if (readTimeout >= 0)
s.setSoTimeout(readTimeout);
else if (defaultSoTimeout > 0) {
s.setSoTimeout(defaultSoTimeout);
}
return s;
}

上面的new InetSocketAddress(server, port)这里会涉及到java DNS cache的处理,

public InetSocketAddress(String hostname, int port) {
if (port < 0 || port > 0xFFFF) {
throw new IllegalArgumentException("port out of range:" + port);
}
if (hostname == null) {
throw new IllegalArgumentException("hostname can't be null");
}
try {
addr = InetAddress.getByName(hostname); //这里会有java DNS缓存的处理,先从缓存取hostname绑定的ip地址,如果取不到再通过OS的DNS cache机制去取,取不到再从DNS服务器上取。
} catch(UnknownHostException e) {
this.hostname = hostname;
addr = null;
}
this.port = port;
}

当然最终的Socket.java的connect方法
java.net.socket

public void connect(SocketAddress endpoint, int timeout) throws IOException {
if (endpoint == null)

if (timeout < 0)
throw new IllegalArgumentException("connect: timeout can't be negative");

if (isClosed())
throw new SocketException("Socket is closed");

if (!oldImpl && isConnected())
throw new SocketException("already connected");

if (!(endpoint instanceof InetSocketAddress))
throw new IllegalArgumentException("Unsupported address type");

InetSocketAddress epoint = (InetSocketAddress) endpoint;

SecurityManager security = System.getSecurityManager();
if (security != null) {
if (epoint.isUnresolved())
security.checkConnect(epoint.getHostName(),
epoint.getPort());
else
security.checkConnect(epoint.getAddress().getHostAddress(),
epoint.getPort());
}
if (!created)
createImpl(true);
if (!oldImpl)
impl.connect(epoint, timeout);
else if (timeout == 0) {
if (epoint.isUnresolved()) //如果没有设置SocketAddress的ip地址,则用域名去访问
impl.connect(epoint.getAddress().getHostName(),
epoint.getPort());
else
impl.connect(epoint.getAddress(), epoint.getPort()); 最终socket连接的是设置的SocketAddress的ip地址,
} else
throw new UnsupportedOperationException("SocketImpl.connect(addr, timeout)");
connected = true;
/*
* If the socket was not bound before the connect, it is now because
* the kernel will have picked an ephemeral port & a local address
*/
bound = true;
}

我们再看下通过socket来发送HTTP请求的处理代码,也就是sun.net.www.protocl.http.HttpURLConnection.java的getInputStream方法中调用的writeRequests()方法:
private void writeRequests() throws IOException { 这段代码就是封装http请求的头请求信息,通过socket发送出去
/* print all message headers in the MessageHeader
* onto the wire - all the ones we've set and any
* others that have been set
*/
// send any pre-emptive authentication
if (http.usingProxy) {
setPreemptiveProxyAuthentication(requests);
}
if (!setRequests) {

/* We're very particular about the order in which we
* set the request headers here. The order should not
* matter, but some careless CGI programs have been
* written to expect a very particular order of the
* standard headers. To name names, the order in which
* Navigator3.0 sends them. In particular, we make *sure*
* to send Content-type: <> and Content-length:<> second
* to last and last, respectively, in the case of a POST
* request.
*/
if (!failedOnce)
requests.prepend(method + " " + http.getURLFile()+" " +
httpVersion, null);
if (!getUseCaches()) {
requests.setIfNotSet ("Cache-Control", "no-cache");
requests.setIfNotSet ("Pragma", "no-cache");
}
requests.setIfNotSet("User-Agent", userAgent);
int port = url.getPort();
String host = url.getHost();
if (port != -1 && port != url.getDefaultPort()) {
host += ":" + String.valueOf(port);
}
requests.setIfNotSet("Host", host);
requests.setIfNotSet("Accept", acceptString);

/*
* For HTTP/1.1 the default behavior is to keep connections alive.
* However, we may be talking to a 1.0 server so we should set
* keep-alive just in case, except if we have encountered an error
* or if keep alive is disabled via a system property
*/

// Try keep-alive only on first attempt
if (!failedOnce && http.getHttpKeepAliveSet()) {
if (http.usingProxy) {
requests.setIfNotSet("Proxy-Connection", "keep-alive");
} else {
requests.setIfNotSet("Connection", "keep-alive");
}
} else {
/*
* RFC 2616 HTTP/1.1 section 14.10 says:
* HTTP/1.1 applications that do not support persistent
* connections MUST include the "close" connection option
* in every message
*/
requests.setIfNotSet("Connection", "close");
}
// Set modified since if necessary
long modTime = getIfModifiedSince();
if (modTime != 0 ) {
Date date = new Date(modTime);
//use the preferred date format according to RFC 2068(HTTP1.1),
// RFC 822 and RFC 1123
SimpleDateFormat fo =
new SimpleDateFormat ("EEE, dd MMM yyyy HH:mm:ss 'GMT'", Locale.US);
fo.setTimeZone(TimeZone.getTimeZone("GMT"));
requests.setIfNotSet("If-Modified-Since", fo.format(date));
}
// check for preemptive authorization
AuthenticationInfo sauth = AuthenticationInfo.getServerAuth(url);
if (sauth != null && sauth.supportsPreemptiveAuthorization() ) {
// Sets "Authorization"
requests.setIfNotSet(sauth.getHeaderName(), sauth.getHeaderValue(url,method));
currentServerCredentials = sauth;
}

if (!method.equals("PUT") && (poster != null || streaming())) {
requests.setIfNotSet ("Content-type",
"application/x-www-form-urlencoded");
}

if (streaming()) {
if (chunkLength != -1) {
requests.set ("Transfer-Encoding", "chunked");
} else {
requests.set ("Content-Length", String.valueOf(fixedContentLength));
}
} else if (poster != null) {
/* add Content-Length & POST/PUT data */
synchronized (poster) {
/* close it, so no more data can be added */
poster.close();
requests.set("Content-Length",
String.valueOf(poster.size()));
}
}

// get applicable cookies based on the uri and request headers
// add them to the existing request headers
setCookieHeader();

}

再来看看把socket响应信息解析为http的响应信息的代码:
sun.net.www.http.HttpClient.java的parseHTTP方法:
private boolean parseHTTPHeader(MessageHeader responses, ProgressSource pi, HttpURLConnection httpuc)
throws IOException {
/* If "HTTP/*" is found in the beginning, return true. Let
* HttpURLConnection parse the mime header itself.
*
* If this isn't valid HTTP, then we don't try to parse a header
* out of the beginning of the response into the responses,
* and instead just queue up the output stream to it's very beginning.
* This seems most reasonable, and is what the NN browser does.
*/

keepAliveConnections = -1;
keepAliveTimeout = 0;

boolean ret = false;
byte[] b = new byte[8];

try {
int nread = 0;
serverInput.mark(10);
while (nread < 8) {
int r = serverInput.read(b, nread, 8 - nread);
if (r < 0) {
break;
}
nread += r;
}
String keep=null;
ret = b[0] == 'H' && b[1] == 'T'
&& b[2] == 'T' && b[3] == 'P' && b[4] == '/' &&
b[5] == '1' && b[6] == '.';
serverInput.reset();
if (ret) { // is valid HTTP - response started w/ "HTTP/1."
responses.parseHeader(serverInput);

// we've finished parsing http headers
// check if there are any applicable cookies to set (in cache)
if (cookieHandler != null) {
URI uri = ParseUtil.toURI(url);
// NOTE: That cast from Map shouldn't be necessary but
// a bug in javac is triggered under certain circumstances
// So we do put the cast in as a workaround until
// it is resolved.
if (uri != null)
cookieHandler.put(uri, (Map<java.lang.String,java.util.List<java.lang.String>>)responses.getHeaders());
}

/* decide if we're keeping alive:
* This is a bit tricky. There's a spec, but most current
* servers (10/1/96) that support this differ in dialects.
* If the server/client misunderstand each other, the
* protocol should fall back onto HTTP/1.0, no keep-alive.
*/
if (usingProxy) { // not likely a proxy will return this
keep = responses.findValue("Proxy-Connection");
}
if (keep == null) {
keep = responses.findValue("Connection");
}
if (keep != null && keep.toLowerCase().equals("keep-alive")) {
/* some servers, notably Apache1.1, send something like:
* "Keep-Alive: timeout=15, max=1" which we should respect.
*/
HeaderParser p = new HeaderParser(
responses.findValue("Keep-Alive"));
if (p != null) {
/* default should be larger in case of proxy */
keepAliveConnections = p.findInt("max", usingProxy?50:5);
keepAliveTimeout = p.findInt("timeout", usingProxy?60:5);
}
} else if (b[7] != '0') {
/*
* We're talking 1.1 or later. Keep persistent until
* the server says to close.
*/
if (keep != null) {
/*
* The only Connection token we understand is close.
* Paranoia: if there is any Connection header then
* treat as non-persistent.
*/
keepAliveConnections = 1;
} else {
keepAliveConnections = 5;
}
}
……
}

对于java.net包的http,ftp等各种协议的底层实现,可以参考rt.jar下面的几个包的代码:
sun.net.www.protocl下的几个包。

在http client中也可以设置代理:
HostConfiguration conf = new HostConfiguration();
conf.setHost(host);
conf.setProxy(ip, 80);
statusCode = httpclient.executeMethod(conf,getMethod);

httpclient自己也是基于socket封装的http处理的库。底层代理的实现是一样的。

另外一种不设置代理,通过反射修改InetAddress的cache也是ok的。但是这种方法非常不推荐,不要使用,因为对于proxy代理服务器概念了解不清楚,最开始还使用这种方法,
public static void jdkDnsNoCache(final String host, final String ip)
throws SecurityException, NoSuchFieldException,
IllegalArgumentException, IllegalAccessException {
if (StringUtils.isBlank(host)) {
return;
}
final Class clazz = java.net.InetAddress.class;
final Field cacheField = clazz.getDeclaredField("addressCache");
cacheField.setAccessible(true);
final Object o = cacheField.get(clazz);
Class clazz2 = o.getClass();
final Field cacheMapField = clazz2.getDeclaredField("cache");
cacheMapField.setAccessible(true);
final Map cacheMap = (Map) cacheMapField.get(o);
AccessController.doPrivileged(new PrivilegedAction() {
public Object run() {
try {
synchronized (o) {// 同步是必须的,因为o可能会有多个线程同时访问修改。
// cacheMap.clear();//这步比较关键,用于清除原来的缓存
// cacheMap.remove(host);
if (!StringUtils.isBlank(ip)) {
InetAddress inet = InetAddress.getByAddress(host,IPUtil.int2byte(ip));
InetAddress addressstart = InetAddress.getByName(host);
Object cacheEntry = cacheMap.get(host);
cacheMap.put(host,newCacheEntry(inet,cacheEntry));
// cacheMap.put(host,newCacheEntry(newInetAddress(host, ip)));
}else{
cacheMap.remove(host);
}
// System.out.println(getStaticProperty(
// "java.net.InetAddress", "addressCacheInit"));
// System.out.println(invokeStaticMethod("java.net.InetAddress","getCachedAddress",new
// Object[]{host}));
}
} catch (Throwable te) {
throw new RuntimeException(te);
}
return null;
}
});
final Map cacheMapafter = (Map) cacheMapField.get(o);
System.out.println(cacheMapafter);

}

关于java中对于DNS的缓存设置可以参考:
1.在${java_home}/jre/lib/secuiry/java.secuiry文件,修改下面为
networkaddress.cache.negative.ttl=0 DNS解析不成功的缓存时间
networkaddress.cache.ttl=0 DNS解析成功的缓存的时间
2.jvm启动时增加下面两个启动环境变量
-Dsun.net.inetaddr.ttl=0
-Dsun.net.inetaddr.negative.ttl=0

如果在java程序中使用,可以这么设置设置:
java.security.Security.setProperty("networkaddress.cache.ttl" , "0");
java.security.Security.setProperty("networkaddress.cache.negative.ttl" , "0");

还有几篇文档链接可以查看:
http://www.rgagnon.com/javadetails/java-0445.html
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6247501

linux下关于OS DNS设置的几个文件是
/etc/resolve.conf
/etc/nscd.conf
/etc/nsswitch.conf

http://www.linuxfly.org/post/543/
http://linux.die.net/man/5/nscd.conf
http://www.linuxhomenetworking.com/wiki/index.php/Quick_HOWTO_:_Ch18_:_Configuring_DNS
http://linux.die.net/man/5/nscd.conf

转载于:https://www.cnblogs.com/cristin/p/7228576.html

HttpUrlConnection底层实现和关于java host绑定ip即时生效的设置及分析相关推荐

  1. The authenticity of host 'izwz9c8l39itikxj328wwuz (IP地址)' can't be established.

    在使用SecureCRT远程连接阿里云ECS时,启动hadoop: 输入命令:./sbin/start-dfs.sh 报错如下: The authenticity of host 'izwz9c8l3 ...

  2. String/StringBuilder/ToString()底层代码解析( JAVA / C# )

    C# /JAVA: 字符串构建利器StringBuilder区别 前言 名词解释 1.1 示例 案例一: 不同变量赋值( = ) 案例二:相同变量赋值( = ) 案例三:变量追加赋值( += ) 1. ...

  3. java neo4j_Neo4j Java REST绑定–第2部分(批处理)

    java neo4j 在第1部分中 ,我们讨论了使用Java REST绑定建立与Neo4j Server的连接. 现在让我们详细了解事务,批处理以及REST请求的实际情况.确保org.neo4j.re ...

  4. Neo4j Java REST绑定–第2部分(批处理)

    在第1部分中 ,我们讨论了使用Java REST绑定建立与Neo4j Server的连接. 现在让我们详细了解事务,批处理以及REST请求的实际情况.确保org.neo4j.rest.logging_ ...

  5. Github 每日精选:可在Java 中绑定 skia 的 2D 图形库Skija;自动对对联系统seq2seq-couplet

    大家好,我是开源菌!天气转凉,有点不想打字,但依然阻挡不了我给大家安利开源项目的冲动.刚刚瞄了一眼今天的榜单,重新上架的 youtube-dl 插件再次登顶第一宝座,好不威风. 1.Skija:可在J ...

  6. java rabbitmq 绑定_RabbitMQ:交换,队列和绑定 - 谁设置了什么?

    当使用RabbitMQ发送消息时,您基本上有交换,队列和绑定 . 我理解他们的想法以及他们如何相互联系,但我不确定是谁设置了什么 . 基本上,我的应用程序中有三个场景 . 场景1:一个发布者,多个工作 ...

  7. windows Redis绑定ip无效,Redis设置密码无效,Windows Redis 配置不生效, Windows Redis requirepass不生效...

    windows Redis绑定ip无效,Redis设置密码无效,Windows Redis 配置不生效, Windows Redis requirepass不生效 >>>>&g ...

  8. 今天用Java开发主机IP扫描神器,零基础Socket编程详细

    目录 一.开发背景 二.准备工作 三.远程主机 IP 探测 四.核心算法 1.IP地址转化为十进制数 2.十进制数转化为IP地址 五.主机 IP 扫描神器界面 六.各功能代码及IP扫描演示 1.主机扫 ...

  9. oracle arp绑定mac地址,使用ARP命令来绑定IP和MAC地址

    使用ARP命令来绑定IP和MAC地址 前言:我本来没有想过写关于ARP绑定的文章,坦白的说一句,在你理解ARP工作的原理时,这其实比较简单.只是看到最近论坛很多人在问关于绑定IP和MAC地址的问题, ...

最新文章

  1. Python--32 模块 包
  2. 网络测试与分析工具简介
  3. python三引号 内部变量_python在三引号中使用变量
  4. hdoj 2041 超级阶梯
  5. SQL Server检索存储过程的结果集
  6. 轻松的java,轻松进行Java Portlets
  7. 将ESXi加入到vCenter中进行管理
  8. 聚类精确度(Cluster Accuracy)
  9. 引入阿里云字体图标库的方法
  10. html的坐标怎么表示,经纬度怎样表示
  11. 远程唤醒、WOL、Magic_Packet
  12. Java元数据区的概念_JVM的元数据区
  13. CF1132D Stressful Training (binary search)(greedy)
  14. Win10 Ubuntu16.04 时间同步问题
  15. Spring boot 事物管理
  16. 【RK3399 GMAC】Linux Debian9 gmac 主控输出clk ,即output模式
  17. Halcon算子大全
  18. cmake-gui使用教程
  19. 数组,向量和矩阵以及空间的维数
  20. $.each()的理解

热门文章

  1. OpenCV 笔记(02)— 图像显示、保存、腐蚀、模糊、canny 边缘检测(imread、imshow、namedWindow、imwrite)
  2. 德国最受欢迎的程序员技能排行
  3. 【C#】类——里式转换
  4. 对列表去重并保持原来的顺序
  5. PVD与CVD性能比较
  6. 快手推荐系统及 Redis 升级存储
  7. Linux crontab 命令基本说明
  8. logcat 结合 grep 过滤 log 日志信息
  9. This version of Android Studio cannot open this project, please retry with Android Studio 3.5 or new
  10. Android全屏,隐藏状态栏和标题栏