摘要

本文描述了如何解决一个Druid connection 泄露的问题，因为过程有些曲折，就详细记录下解决步骤与思路，如果对你有所帮助与启发，请留言

#项目结构#
MyBatis+TDDL

问题

https://github.com/FS1360472174/javaweb/issues/58

ERROR [com.alibaba.druid.pool.DruidDataSource] - abandon connection, owner thread: qtp1267032364
-14, connected at : 1515409987672, open stackTrace
at java.lang.Thread.getStackTrace(Thread.java:1556)
at com.alibaba.druid.pool.DruidDataSource.getConnectionDirect(DruidDataSource.java:1068)
at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:994)
at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:984)
at com.taobao.tddl.atom.jdbc.TDataSourceWrapper.getConnectionByTargetDataSource(TDataSourceWrapper.java:322)
at com.taobao.tddl.atom.jdbc.TDataSourceWrapper.getConnection0(TDataSourceWrapper.java:284)
at com.taobao.tddl.atom.jdbc.TDataSourceWrapper.getConnection(TDataSourceWrapper.java:255)
at com.taobao.tddl.atom.jdbc.TDataSourceWrapper.getConnection(TDataSourceWrapper.java:222)
at com.taobao.tddl.atom.AbstractTAtomDataSource.getConnection(AbstractTAtomDataSource.java:27)
at com.taobao.tddl.group.jdbc.DataSourceWrapper.getConnection(DataSourceWrapper.java:120)
at com.taobao.tddl.group.jdbc.TGroupConnection.createNewConnection(TGroupConnection.java:191)
at com.taobao.tddl.group.jdbc.TGroupConnection$1.tryOnDataSource(TGroupConnection.java:453)
at com.taobao.tddl.group.jdbc.TGroupConnection$1.tryOnDataSource(TGroupConnection.java:443)
at com.taobao.tddl.group.dbselector.AbstractDBSelector.tryOnDataSourceHolderWithIndex(AbstractDBSelector.java:19

at com.taobao.tddl.group.dbselector.AbstractDBSelector.tryExecute(AbstractDBSelector.java:315)
at com.taobao.tddl.group.jdbc.TGroupConnection.prepareCall(TGroupConnection.java:492)
at com.taobao.tddl.group.jdbc.TGroupConnection.prepareCall(TGroupConnection.java:520)
at com.taobao.tddl.group.jdbc.TGroupConnection.prepareCall(TGroupConnection.java:74)
at com.taobao.tddl.matrix.jdbc.TConnection.prepareCall(TConnection.java:515)
at com.taobao.tddl.matrix.jdbc.TConnection.prepareCall(TConnection.java:483)
14980,1-8 99%
at com.taobao.tddl.group.jdbc.TGroupConnection.prepareCall(TGroupConnection.java:74)
at com.taobao.tddl.matrix.jdbc.TConnection.prepareCall(TConnection.java:515)
at com.taobao.tddl.matrix.jdbc.TConnection.prepareCall(TConnection.java:483)

在线上排查其他问题时，看到日志中有这个ERROR,而且很频繁。
根据abandon connection,得知这是一个数据库连接池问题，废弃的连接处理。

解决步骤

首先搜了下druid 官网 FAQ
https://github.com/alibaba/druid/wiki/常见问题
发现是druid有对连接泄露进行监控处理

https://github.com/alibaba/druid/wiki/连接泄漏监测
https://github.com/alibaba/druid/issues/872
我这边没有配对监控，直接是从日志中看到的，有相应的线程栈信息，可以方便排查。

public DruidPooledConnection getConnectionDirect(long maxWaitMillis) throws SQLException {int notFullTimeoutRetryCnt = 0;DruidPooledConnection poolableConnection;while(true) {while(true) {try {Connection realConnection = poolableConnection.getConnection();this.discardConnection(realConnection);} else {Connection realConnection = poolableConnection.getConnection();if(realConnection.isClosed()) {this.discardConnection((Connection)null);} else {if(!this.isTestWhileIdle()) {break;}long currentTimeMillis = System.currentTimeMillis();long lastActiveTimeMillis = poolableConnection.getConnectionHolder().getLastActiveTimeMillis();long idleMillis = currentTimeMillis - lastActiveTimeMillis;long timeBetweenEvictionRunsMillis = this.getTimeBetweenEvictionRunsMillis();if(timeBetweenEvictionRunsMillis <= 0L) {timeBetweenEvictionRunsMillis = 60000L;}if(idleMillis < timeBetweenEvictionRunsMillis) {break;}this.discardConnection(realConnection);}}}if(this.isRemoveAbandoned()) {StackTraceElement[] stackTrace = Thread.currentThread().getStackTrace();poolableConnection.setConnectStackTrace(stackTrace);poolableConnection.setConnectedTimeNano();poolableConnection.setTraceEnable(true);Map var21 = this.activeConnections;synchronized(this.activeConnections) {this.activeConnections.put(poolableConnection, PRESENT);}}if(!this.isDefaultAutoCommit()) {poolableConnection.setAutoCommit(false);}return poolableConnection;}

这个连接泄露不会导致OOM，因为druid会去主动detroy这些未关闭的连接，也就是上面日志中的错误信息
现在知道是有数据库连接未关闭，但是代码中并没有去管理数据库连接池，都是交给了Spring去管理的呢，而且不是每个数据库操作都会有问题，而是特定的数据库操作有问题
开启debug日志

2018-01-23 21:08:35,760 DEBUG [org.springframework.data.redis.core.RedisConnectionUtils] - Opening RedisConnection
2018-01-23 21:08:35,761 DEBUG [org.springframework.data.redis.core.RedisConnectionUtils] - Closing Redis Connection
2018-01-23 21:08:35,762 DEBUG [org.mybatis.spring.SqlSessionUtils] - Creating a new SqlSession
2018-01-23 21:08:35,762 DEBUG [org.mybatis.spring.SqlSessionUtils] - SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@5ba28a19] was not registered for synchronization because synchronization is not active
2018-01-23 21:08:35,765 DEBUG [org.springframework.jdbc.datasource.DataSourceUtils] - Fetching JDBC Connection from DataSource
2018-01-23 21:08:35,765 DEBUG [org.mybatis.spring.transaction.SpringManagedTransaction] - JDBC Connection [com.taobao.tddl.matrix.jdbc.TConnection@24f6de0] will not be managed by Spring
2018-01-23 21:08:35,766 DEBUG [com.taobao.tddl.group.jdbc.TGroupConnection] -  [TDDL] dataSourceIndex=GroupIndex [index=0, failRetry=false], tddl version: 5.1.7
2018-01-23 21:08:35,810 DEBUG [org.mybatis.spring.SqlSessionUtils] - Closing non transactional SqlSession [org.apache.ibatis.session.defaults.DefaultSqlSession@5ba28a19]
2018-01-23 21:08:35,810 DEBUG [org.springframework.jdbc.datasource.DataSourceUtils] - Returning JDBC Connection to DataSource

这里接可以看出阿里的开源软件距离工业级还有距离，Spring的可以清楚的看到连接释放与返回，而Druid并没有

因为这个是和特定的SQL有关系，所有看下具体的这个SQL操作
```
 @Options(statementType = StatementType.CALLABLE)@Insert("<script> " +  "</script>")int saveDemo(Demo demo);
```
这个是之前人留下的，这种写法目前在项目组不是很常见了，没有实现代码与SQL语句分离，不是很提倡。
这边的StatementType为Callable,看起来比奇怪，一般CallableStatement由于数据库存储过程的操作，显然这个语句不是这么调用存储过程。不知道前任为什么这么写，所以首先干掉了这个StatementType.CALLABLE参数。

结果果然出错了，调用的时候报错，一个好笑的错误，没有使用auto generate id,却调用了。

这个应该是MySQL 5.7 driver的一个bug，前任为了避免这个问题，使用CallableStatement绕过去了

Caused by: Generated keys not requested. You need to specify Statement.RETURN_GENERATED_KEYS to Statement.executeUpdate() or Connection.prepareStatement().
at com.taobao.tddl.repo.mysql.handler.PutMyHandlerCommon.handle(PutMyHandlerCommon.java:52)
at com.taobao.tddl.executor.AbstractGroupExecutor.executeInner(AbstractGroupExecutor.java:59)
at com.taobao.tddl.executor.AbstractGroupExecutor.execByExecPlanNode(AbstractGroupExecutor.java:40)
at com.taobao.tddl.executor.TopologyExecutor.execByExecPlanNode(TopologyExecutor.java:59)
at com.taobao.tddl.executor.MatrixExecutor.execByExecPlanNode(MatrixExecutor.java:282)

所以先改成xml方式验证下，xml默认是PreparedStatement，然后看下是否还能发生错误

发现没有了，所以问题的原因就是CallableStatement造成的

#探索-druid连接管理#
虽然问题解决了，但是还是不知道druid是如何管理连接的，所以需要深入一点

mybatis
首先项目中用到了Mybatis，Mybatis + jdbc的操作流程如下
一个数据库操作取连接是在具体的sql执行时才去取的
引入 Spring-jdbc
进行transaction 管理

    <bean id="transactionManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager"><property name="dataSource" ref="dataSource"/></bean><tx:annotation-driven transaction-manager="transactionManager"/>

这时候连接池管理交给了SpringManagedTransaction

org.mybatis.spring.transaction.SpringManagedTransaction

    private Statement prepareStatement(StatementHandler handler, Log statementLog) throws SQLException {Connection connection = this.getConnection(statementLog);Statement stmt = handler.prepare(connection, this.transaction.getTimeout());handler.parameterize(stmt);return stmt;}

TDDL

TDDL是因为分库分表引入的，它本身依赖druid来做连接池管理
获取连接

在执行的时候TPreparedStatment,
重新包装了SQL，PreparedStatment
AutoCommitTransaction管理一个连接
getConnection,这时候Connection
获取的是TGroupConnection中createNewConnection

关闭连接
SqlSessionUtils.closeSqlSession
TConnection.close()
TConnectionWrapper.close()
DruidPooledConnection.close()
DruidPooledConnection.syncClose()
DruidPooledConnection.recycle() // 这个方法里面实际关闭了了druid的连接

public DruidPooledConnection getConnectionDirect(long maxWaitMillis) throws SQLException {int notFullTimeoutRetryCnt = 0;DruidPooledConnection poolableConnection;...if(this.isRemoveAbandoned()) {StackTraceElement[] stackTrace = Thread.currentThread().getStackTrace();poolableConnection.setConnectStackTrace(stackTrace);poolableConnection.setConnectedTimeNano();poolableConnection.setTraceEnable(true);Map var21 = this.activeConnections;synchronized(this.activeConnections) {// 将一个活跃connectionthis.activeConnections.put(poolableConnection, PRESENT);}}if(!this.isDefaultAutoCommit()) {poolableConnection.setAutoCommit(false);}return poolableConnection;}

remove abandoned

  public int removeAbandoned() {int removeCount = 0;long currrentNanos = System.nanoTime();List<DruidPooledConnection> abandonedList = new ArrayList();// activeConnections这个类变量存储了未关闭的连接Map var5 = this.activeConnections;synchronized(this.activeConnections) {// 获取这边的值Iterator iter = this.activeConnections.keySet().iterator();while(iter.hasNext()) {DruidPooledConnection pooledConnection = (DruidPooledConnection)iter.next();if(!pooledConnection.isRunning()) {long timeMillis = (currrentNanos - pooledConnection.getConnectedTimeNano()) / 1000000L;if(timeMillis >= this.removeAbandonedTimeoutMillis) {iter.remove();pooledConnection.setTraceEnable(false);abandonedList.add(pooledConnection);}}}}if(abandonedList.size() > 0) {Iterator var14 = abandonedList.iterator();while(true) {DruidPooledConnection pooledConnection;do {while(true) {if(!var14.hasNext()) {return removeCount;}pooledConnection = (DruidPooledConnection)var14.next();synchronized(pooledConnection) {if(!pooledConnection.isDisable()) {break;}}}JdbcUtils.close(pooledConnection);pooledConnection.abandond();++this.removeAbandonedCount;++removeCount;} while(!this.isLogAbandoned());StringBuilder buf = new StringBuilder();buf.append("abandon connection, owner thread: ");LOG.error(buf.toString());}} else {return removeCount;}}

可以看到修正的sql操作是有关闭的，但是CallableStatement没有，所以有未关闭的连接

欢迎加入Java互联网技术交流群 392669336,和方丈交流各类java问题

关注【方丈的寺院】，第一时间收到文章的更新，与方丈一起开始技术修行之路

#参考#
http://blog.csdn.net/luanlouis/article/details/40422941

http://blog.csdn.net/luanlouis/article/details/37671851

https://www.jianshu.com/p/ec40a82cae28

http://blog.csdn.net/u013476542/article/details/53256610

ERROR [com.alibaba.druid.pool.DruidDataSource] - abandon connection相关推荐

ERROR com.alibaba.druid.pool.DruidDataSource - abandon connection, open stackTrace: 已解决
错误日志: 2020-07-13 15:37:58,545 [Druid-ConnectionPool-Destroy-2092318840] ERROR com.alibaba.druid.pool ...
com.alibaba.druid.pool.DruidDataSource - abandon connection, open stackTrace
错误: com.alibaba.druid.pool.DruidDataSource - abandon connection, open stackTrace 原因: 连接池为了防止程序从池里取得连 ...
ERROR c.alibaba.druid.pool.DruidDataSource - init datasource error 运行代码提示数据库连接错误
本地项目代码之前一直运行正常,数据库连接配置切换为生产环境数据库的配置并通过本地Maven工具打包(war包,clean再install)后,再切换回原来的测试库配置,本地再启服务跑代码就一直报错,如 ...
com.alibaba.druid.pool.DruidDataSource : testWhileIdle is true, validationQuery not set
2019-10-13 16:37:56.965 ERROR 8088 --- [on(6)-127.0.0.1] com.alibaba.druid.pool.DruidDataSource : ...
com.alibaba.druid.pool.DruidDataSource : {dataSource-1} closed
错误介绍在运行SpringBoot 整合SpringData Jpa的项目时 ,控制台打印这个错误 ,详细信息如下 15:45:53.858 [main] DEBUG org.springframe ...
canal本地运行异常：class com.alibaba.druid.pool.DruidDataSource cannot be cast to
文章目录 1. 现象 2. 原因 3. 解决方案 1. 现象启动时出现异常 class com.alibaba.druid.pool.DruidDataSource cannot be cast t ...
com.alibaba.druid.pool.DruidDataSource.error解决办法
这个错误主要是复制粘贴来的pom.xml文件版本号与代码或开发工具的版本不兼容,登录https://mvnrepository.com/artifact/mysql/mysql-connector-j ...
Druid数据库连接池超时问题com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 1000, active 10
问题描述: com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 1000, active 10at com.alibab ...
异常：com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 60010, active 20, maxActive 20
一.异常 Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connec ...
wait millis 60010, active 20, maxActive 20 处理 com.alibaba.druid.pool.GetConnectionTimeout
druid连接池连接用完导致 Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JD ...

ERROR [com.alibaba.druid.pool.DruidDataSource] - abandon connection

摘要

问题

解决步骤

ERROR [com.alibaba.druid.pool.DruidDataSource] - abandon connection相关推荐

最新文章

热门文章