添加https后反向代理gateway报错

2023-02-17 14:19:05.328 [reactor-http-epoll-4] ------ ERROR c..si.gateway.exception.JsonExceptionHandler - [全局异常处理]异常请求路径:/102039999,记录异常信息:not an SSL/TLS record: 485454502f312e3120343030200d0a436f6e74656e742d547970653a20746578742f68746d6c3b636861727365743d7574662d380d0a436f6e74656e742d4c616e67756167653a20656e0d0a436f6e74656e742d4c656e6774683a203433350d0a446174653a204672692c2031372046656220323032332030363a31393a303520474d540d0a436f6e6e656374696f6e3a20636c6f73650d0a0d0a3c21646f63747970652068746d6c3e3c68746d6c206c616e673d22656e223e3c686561643e3c7469746c653e48545450205374617475732034303020e280932042616420526571756573743c2f7469746c653e3c7374796c6520747970653d22746578742f637373223e626f6479207b666f6e742d66616d696c793a5461686f6d612c417269616c2c73616e732d73657269663b7d2068312c2068322c2068332c2062207b636f6c6f723a77686974653b6261636b67726f756e642d636f6c6f723a233532354437363b7d206831207b666f6e742d73697a653a323270783b7d206832207b666f6e742d73697a653a313670783b7d206833207b666f6e742d73697a653a313470783b7d2070207b666f6e742d73697a653a313270783b7d2061207b636f6c6f723a626c61636b3b7d202e6c696e65207b6865696768743a3170783b6261636b67726f756e642d636f6c6f723a233532354437363b626f726465723a6e6f6e653b7d3c2f7374796c653e3c2f686561643e3c626f64793e3c68313e48545450205374617475732034303020e280932042616420526571756573743c2f68313e3c2f626f64793e3c2f68746d6c3eio.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 485454502f312e3120343030200d0a436f6e74656e742d547970653a20746578742f68746d6c3b636861727365743d7574662d380d0a436f6e74656e742d4c616e67756167653a20656e0d0a436f6e74656e742d4c656e6774683a203433350d0a446174653a204672692c2031372046656220323032332030363a33313a323020474d540d0a436f6e6e656374696f6e3a20636c6f73650d0a0d0a3c21646f63747970652068746d6c3e3c68746d6c206c616e673d22656e223e3c686561643e3c7469746c653e48545450205374617475732034303020e280932042616420526571756573743c2f7469746c653e3c7374796c6520747970653d22746578742f637373223e626f6479207b666f6e742d66616d696c793a5461686f6d612c417269616c2c73616e732d73657269663b7d2068312c2068322c2068332c2062207b636f6c6f723a77686974653b6261636b67726f756e642d636f6c6f723a233532354437363b7d206831207b666f6e742d73697a653a323270783b7d206832207b666f6e742d73697a653a313670783b7d206833207b666f6e742d73697a653a313470783b7d2070207b666f6e742d73697a653a313270783b7d2061207b636f6c6f723a626c61636b3b7d202e6c696e65207b6865696768743a3170783b6261636b67726f756e642d636f6c6f723a233532354437363b626f726465723a6e6f6e653b7d3c2f7374796c653e3c2f686561643e3c626f64793e3c68313e48545450205374617475732034303020e280932042616420526571756573743c2f68313e3c2f626f64793e3c2f68746d6c3eat io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1254)Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: Error has been observed at the following site(s):|_ checkpoint ⇢ comsi.gateway.config.CorsConfiguration$$Lambda$635/1122108119 [DefaultWebFilterChain]|_ checkpoint ⇢ org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]|_ checkpoint ⇢ com.alibaba.csp.sentinel.adapter.spring.webflux.SentinelWebFluxFilter [DefaultWebFilterChain]|_ checkpoint ⇢ org.springframework.security.web.server.authorization.AuthorizationWebFilter [DefaultWebFilterChain]|_ checkpoint ⇢ org.springframework.security.web.server.authorization.ExceptionTranslationWebFilter [DefaultWebFilterChain]|_ checkpoint ⇢ org.springframework.security.web.server.authentication.logout.LogoutWebFilter [DefaultWebFilterChain]|_ checkpoint ⇢ org.springframework.security.web.server.savedrequest.ServerRequestCacheWebFilter [DefaultWebFilterChain]|_ checkpoint ⇢ org.springframework.security.web.server.context.SecurityContextServerWebExchangeWebFilter [DefaultWebFilterChain]|_ checkpoint ⇢ org.springframework.security.config.web.server.ServerHttpSecurity$OAuth2ResourceServerSpec$BearerTokenAuthenticationWebFilter [DefaultWebFilterChain]|_ checkpoint ⇢ org.springframework.security.web.server.authentication.AuthenticationWebFilter [DefaultWebFilterChain]|_ checkpoint ⇢ org.springframework.security.web.server.context.ReactorContextWebFilter [DefaultWebFilterChain]|_ checkpoint ⇢ org.springframework.security.web.server.header.HttpHeaderWriterWebFilter [DefaultWebFilterChain]2023-02-17 14:31:20.768 [reactor-http-epoll-1] ------ ERROR c..si.gateway.exception.JsonExceptionHandler - [全局异常处理]异常请求路径:/102039999,记录异常信息:not an SSL/TLS record: 485454502f312e3120343030200d0a436f6e74656e742d547970653a20746578742f68746d6c3b636861727365743d7574662d380d0a436f6e74656e742d4c616e67756167653a20656e0d0a436f6e74656e742d4c656e6774683a203433350d0a446174653a204672692c2031372046656220323032332030363a33313a323020474d540d0a436f6e6e656374696f6e3a20636c6f73650d0a0d0a3c21646f63747970652068746d6c3e3c68746d6c206c616e673d22656e223e3c686561643e3c7469746c653e48545450205374617475732034303020e280932042616420526571756573743c2f7469746c653e3c7374796c6520747970653d22746578742f637373223e626f6479207b666f6e742d66616d696c793a5461686f6d612c417269616c2c73616e732d73657269663b7d2068312c2068322c2068332c2062207b636f6c6f723a77686974653b6261636b67726f756e642d636f6c6f723a233532354437363b7d206831207b666f6e742d73697a653a323270783b7d206832207b666f6e742d73697a653a313670783b7d206833207b666f6e742d73697a653a313470783b7d2070207b666f6e742d73697a653a313270783b7d2061207b636f6c6f723a626c61636b3b7d202e6c696e65207b6865696768743a3170783b6261636b67726f756e642d636f6c6f723a233532354437363b626f726465723a6e6f6e653b7d3c2f7374796c653e3c2f686561643e3c626f64793e3c68313e48545450205374617475732034303020e280932042616420526571756573743c2f68313e3c2f626f64793e3c2f68746d6c3e|_ checkpoint ⇢ org.springframework.security.config.web.server.ServerHttpSecurity$ServerWebExchangeReactorContextWebFilter [DefaultWebFilterChain]|_ checkpoint ⇢ org.springframework.security.web.server.WebFilterChainProxy [DefaultWebFilterChain]|_ checkpoint ⇢ HTTP GET "/102039999?0=*" [ExceptionHandlingWebHandler]Stack trace:at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1254)at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1322)at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)at java.lang.Thread.run(Thread.java:748)

环境

域名及https绑定在openshift的router上，用route的Edge模式，证书终止在router（证书卸载），转发向后端请求是http的。后端接入nginx做反向代理，所有项目的流量都通过这个nginx。再向后转发是两个nginx，作为本项目的流量入口，有反代和静态文件解析功能。再向后就是服务gateway。

问题现象

开始项目流量不经过openshift内的nginx，直接通过项目自己的nginx访问，http/https都可以的，后期经过openshift再转发回来就出现问题，gateway报错io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record。但是http访问没有问题。

问题分析

网上查了很多解决方法，大概意思是前后端http/https协议没匹配上，或者ssl/tls证书没用对。项目上后端gateway没启用ssl。第二种情况不适用。按说请求过了router后就从https转成http了，后端gateway不该报ssl错误啊，压根就不该协商成ssl。感觉第一种情况感觉也不适用。问题卡在这很久。各种查gateway、netty、nginx等配置和日志。最后实在没招，tcpdump抓包，筛选出请求头信息，找到了问题。

...F....GET /103COM10 HTTP/1.1
X-Real-IP: 10.65.40.193
X-Forwarded-For: 10.65.40.193,10.65.40.193
Accept: application/json, text/plain, */*
language: en
Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyX25hbWUiOiJ3YW5nemhwMzJAY25vb2MuY29tLmNuIiwic2NvcGUiOlsiTUVNQkVSX1JFQUQiLCJNRU1CRVJfV1JJVEUiXSwidXNlclR5cGUiOiIxIiwiZXhwIjoxNjc2NjY2MDgzLCJ1c2VyTmFtZSI6IueOi-elieiLuSIsInVzZXJJZCI6Indhbmd6aHAzMkBjbm9vYy5jb20uY24iLCJhdXRob3JpdGllcyI6WyJhMGE3N2ZjOWNjYmIwMDc5NjZlY2U5MGY4NWM5NGU1ZSJdLCJqdGkiOiI4YmE5N2JiZS04ZDAyLTQ1ODItYTkxNS05ODcxZjcxMzBmZWUiLCJjbGllbnRfaWQiOiJjbGllbnQzIn0.NNQtwWGZiT2CRY0S_-BwxT3XCQyETefxZDT2dos0FB0m42jtcSsyjW0a4t4i6yReqJgiPNf2awVbGnPtxtPO2ZcNsihOoMs4kPoVyqFBzCSeJ4Gh284ZLED0kxlqyczzDS68TyKrt2OoKhMcc4PGDyqjsYRR-3HMP7dd0aGDOX2oHZbM9ie5k3poVhT3SbQLAjqF70mN6nggw90UlXpIBDkMYLoq0BzNwHcWybUaGeAZm4ddqDLqZC78iLkCGilHPw1EvcnYiv3NFwZJJDT7Ie5kJ1LjYHPtknlhTUyWR-BIvgMo05FBJtWm3da1r0GNmGTevr3_71iIal5hcjR6zg
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36 QIHU 360ENT
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: cors
Sec-Fetch-Dest: empty
Referer: https://xxx.xxx/
Accept-Encoding: gzip, deflate, br
Accept-Language: zh-CN,zh;q=0.9
Cookie: SESSION=c1f790a3-af8d-44e9-b8f9-8a53faa5b4b0; Admin-Token=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyX25hbWUiOiJ3YW5nemhwMzJAY25vb2MuY29tLmNuIiwic2NvcGUiOlsiTUVNQkVSX1JFQUQiLCJNRU1CRVJfV1JJVEUiXSwidXNlclR5cGUiOiIxIiwiZXhwIjoxNjc2NjY2MDgzLCJ1c2VyTmFtZSI6IueOi-elieiLuSIsInVzZXJJZCI6Indhbmd6aHAzMkBjbm9vYy5jb20uY24iLCJhdXRob3JpdGllcyI6WyJhMGE3N2ZjOWNjYmIwMDc5NjZlY2U5MGY4NWM5NGU1ZSJdLCJqdGkiOiI4YmE5N2JiZS04ZDAyLTQ1ODItYTkxNS05ODcxZjcxMzBmZWUiLCJjbGllbnRfaWQiOiJjbGllbnQzIn0.NNQtwWGZiT2CRY0S_-BwxT3XCQyETefxZDT2dos0FB0m42jtcSsyjW0a4t4i6yReqJgiPNf2awVbGnPtxtPO2ZcNsihOoMs4kPoVyqFBzCSeJ4Gh284ZLED0kxlqyczzDS68TyKrt2OoKhMcc4PGDyqjsYRR-3HMP7dd0aGDOX2oHZbM9ie5k3poVhT3SbQLAjqF70mN6nggw90UlXpIBDkMYLoq0BzNwHcWybUaGeAZm4ddqDLqZC78iLkCGilHPw1EvcnYiv3NFwZJJDT7Ie5kJ1LjYHPtknlhTUyWR-BIvgMo05FBJtWm3da1r0GNmGTevr3_71iIal5hcjR6zg; dps_current_user_token=%7B%22appId%22%3A%22fsscsHw%22%2C%22authorized%22%3Atrue%2C%22isEnableCategory%22%3A0%2C%22role%22%3A%7B%22roleName%22%3A%22%E6%B5%81%E7%A8%8B%E7%AE%A1%E7%90%86%E5%91%98%22%2C%22roleType%22%3A%221%22%7D%2C%22userCode%22%3A%22wangzhp32%40.com.cn%22%2C%22userId%22%3A%22wangzhp32%.com.cn%22%2C%22userName%22%3A%22%E7%8E%8B%E7%A5%89%E8%8B%B9%22%2C%22userType%22%3A%221%22%7D
user_name: xx.com.cn
userName: %E7%8E%8B%E7%A5%89%E8%8B%B9
userName: UTF-8
userId: xx.com.cn
userType: 1
exp: 1676666083
jti: 8ba97bbe-8d02-4582-a915-9871f7130fee
payload: {"user_name":"xxx","userName":"???","userId":"@xx.com.cn","authorities":["a0a77fc9ccbb007966ece90f85c94e5e"],"client_id":"client3","sourceIp":"10.65.40.193","hostStr":"10.72.26.129","scope":["MEMBER_READ","MEMBER_WRITE"],"userType":"1","exp":1676666083,"jti":"8ba97bbe-8d02-4582-a915-9871f7130fee"}
Forwarded: proto=https;host="10.72.26.129:32334";for="10.65.40.193:57706"
X-Forwarded-Proto: http
X-Forwarded-Port: 32334
X-Forwarded-Host: 10.72.26.129:32334
host: 172.16.34.133:7101
content-length: 0

就是这个Forwarded: proto=https;host=“10.72.26.129:32334”;for=“10.65.40.193:57706”

抓包命令

tcpdump -i calib54437dba60@if4 -n   -A 'tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x47455420'  > 9900.log

解决

请求头中有个Forwarded，后边三个值，for、host、proto。其中这个proto=https。我理解应该是这个值传到后端gateway中被netty识别，启用了netty.handler.ssl.SslHandler。导致证书被卸载后的http请求和gateway中的ssl对不上报错！
解决办法比较简单，在nginx的配置中设置请求头，要不把Forwarded去掉，要不就替换。
1.去掉用proxy_hide_header Forwarded;我试了不好使，不知道是不因为nginx少模块原因。相关模块是ngx_http_fastcgi_module、ngx_http_proxy_module。有兴趣可以试试。
2.修改Forwarded。proxy_set_header Forwarded proto=http;

server {listen       80;listen       443 ssl;server_name  xxx.xxxx.xxx ;underscores_in_headers on;ssl_certificate         certs/xxx.crt;ssl_certificate_key     certs/xxx.key;location /hw/ {proxy_pass http://xxx.xxx/; proxy_set_header   Host    $host;proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;proxy_set_header Forwarded proto=http;#proxy_hide_header Forwarded;}error_page   500 502 503 504  /50x.html;location = /50x.html {root   html;}}
}