MySQL5.7 Attempted to use a closed or broken resource pool 处理过程

1，tomcat启动报错不能获取JDBC连接

mysql版本：5.6.10

tomcat版本：7.0

jdk版本：1.7.0_55

开发人员强调说是以前经历过重启下mysql数据库就好了，不过我不太相信，因为我通过sqlyog客户端连接数据库完全正常的，而且我使用以下生成的kill命令清除了所有数据库连接，然后再重启7100的tomcat，也没有发现任何tomcat发起的connection连接过来，所以我怀疑不是mysql数据库的问题：

SELECT CONCAT('kill ',t.`ID`,';') FROM information_schema.`PROCESSLIST` t WHERE t.`HOST` NOT IN('localhost') AND t.`COMMAND` IN ('Sleep');

tomcat启动详细日志如下：

2016-07-11 19:57:23,378 [main:INFO] com.google.code.yanf4j.core.impl.AbstractController - Controller has been stopped.

2016-07-11 19:57:23,381 [main:ERROR] org.springframework.web.context.ContextLoader - Context initialization failed

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'commonSecurityFilter': Injection of autowired dependencies failed; nested exception is org.springframew

ork.beans.factory.BeanCreationException: Could not autowire field: private com.plocc.auth.common.CommonSecurityMetadataSource com.plocc.auth.common.CommonSecurityFilter.securityMetadataSource

; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'commonSecurityMetadataSource': Invocation of init method failed; nested exception

is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: An SQLException was provoked by the following failure:

com.mchange.v2.resourcepool.ResourcePoolException: Attempted to use a closed or broken resource pool; nested exception is java.sql.SQLException: An SQLException was provoked by the following

failure: com.mchange.v2.resourcepool.ResourcePoolException: Attempted to use a closed or broken resource pool

at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:287)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1106)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)

at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294)

at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225)

at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291)

at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)

at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:585)

at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:913)

at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:464)

at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:384)

at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:283)

at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:111)

at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4206)

at org.apache.catalina.core.StandardContext.start(StandardContext.java:4705)

at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)

at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)

at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)

at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943)

at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778)

at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504)

at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)

at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)

at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)

at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)

at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)

at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)

at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)

at org.apache.catalina.core.StandardService.start(StandardService.java:525)

at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)

at org.apache.catalina.startup.Catalina.start(Catalina.java:595)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)

at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)

Caused by: org.springframework.beans.factory.BeanCreationException: Could not autowire field: private com.plocc.auth.common.CommonSecurityMetadataSource com.plocc.auth.common.CommonSecurityFi

lter.securityMetadataSource; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'commonSecurityMetadataSource': Invocation of init meth

od failed; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: An SQLException was provoke

d by the following failure: com.mchange.v2.resourcepool.ResourcePoolException: Attempted to use a closed or broken resource pool; nested exception is java.sql.SQLException: An SQLException wa

s provoked by the following failure: com.mchange.v2.resourcepool.ResourcePoolException: Attempted to use a closed or broken resource pool

at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:506)

at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:87)

at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:284)

... 37 more

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'commonSecurityMetadataSource': Invocation of init method failed; nested exception is org.spr

ingframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: An SQLException was provoked by the following failure: com.mchang

e.v2.resourcepool.ResourcePoolException: Attempted to use a closed or broken resource pool; nested exception is java.sql.SQLException: An SQLException was provoked by the following failure: c

om.mchange.v2.resourcepool.ResourcePoolException: Attempted to use a closed or broken resource pool

at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:135)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:394)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1448)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)

at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294)

at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225)

at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291)

at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)

at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:848)

at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:790)

at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:707)

at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:478)

... 39 more

Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: An SQLException was provoked by the following f

ailure: com.mchange.v2.resourcepool.ResourcePoolException: Attempted to use a closed or broken resource pool; nested exception is java.sql.SQLException: An SQLException was provoked by the fo

llowing failure: com.mchange.v2.resourcepool.ResourcePoolException: Attempted to use a closed or broken resource pool

at org.springframework.orm.ibatis.SqlMapClientTemplate.execute(SqlMapClientTemplate.java:195)

at org.springframework.orm.ibatis.SqlMapClientTemplate.queryForList(SqlMapClientTemplate.java:293)

at org.springframework.orm.ibatis.SqlMapClientTemplate.queryForList(SqlMapClientTemplate.java:287)

at com.plocc.auth.dao.impl.AuthResourceDAOImpl.selectResourceAll(AuthResourceDAOImpl.java:147)

at com.plocc.auth.dao.impl.AuthResourceDAOImpl$$FastClassByCGLIB$$14a1e188.invoke(<generated>)

at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191)

at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:689)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)

at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:622)

at com.plocc.auth.dao.impl.AuthResourceDAOImpl$$EnhancerByCGLIB$$dad3756c.selectResourceAll(<generated>)

at com.plocc.auth.service.impl.AuthenticationServiceImpl.getAuthResourceAll(AuthenticationServiceImpl.java:456)

at sun.reflect.GeneratedMethodAccessor2222.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318)

at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)

at org.springframework.remoting.support.RemoteInvocationTraceInterceptor.invoke(RemoteInvocationTraceInterceptor.java:77)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)

at com.sun.proxy.$Proxy79.getAuthResourceAll(Unknown Source)

at sun.reflect.GeneratedMethodAccessor2222.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at com.caucho.hessian.server.HessianSkeleton.invoke(HessianSkeleton.java:306)

at com.caucho.hessian.server.HessianSkeleton.invoke(HessianSkeleton.java:221)

at org.springframework.remoting.caucho.HessianExporter.doInvoke(HessianExporter.java:198)

at org.springframework.remoting.caucho.HessianExporter.invoke(HessianExporter.java:118)

at org.springframework.remoting.caucho.HessianServiceExporter.handleRequest(HessianServiceExporter.java:66)

at org.springframework.web.servlet.mvc.HttpRequestHandlerAdapter.handle(HttpRequestHandlerAdapter.java:49)

at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:900)

at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:827)

at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)

at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:643)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)

at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)

at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)

at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)

at java.lang.Thread.run(Thread.java:745)

Caused by: java.sql.SQLException: An SQLException was provoked by the following failure: com.mchange.v2.resourcepool.ResourcePoolException: Attempted to use a closed or broken resource pool

at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)

at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:65)

at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:62)

at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:531)

at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)

at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)

at org.springframework.orm.ibatis.SqlMapClientTemplate.execute(SqlMapClientTemplate.java:182)

... 49 more

Caused by: com.mchange.v2.resourcepool.ResourcePoolException: Attempted to use a closed or broken resource pool

at com.mchange.v2.resourcepool.BasicResourcePool.ensureNotBroken(BasicResourcePool.java:1632)

at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:524)

at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)

at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)

... 52 more

七月 11, 2016 7:57:23 下午 org.apache.catalina.core.StandardContext start

严重: Error listenerStart

七月 11, 2016 7:57:23 下午 org.apache.catalina.core.StandardContext start

严重: Context [/PLATFORM_SSO_Web] startup failed due to previous errors

七月 11, 2016 7:57:23 下午 org.apache.catalina.startup.HostConfig deployDirectory

信息: Deploying web application directory test

七月 11, 2016 7:57:23 下午 org.apache.coyote.http11.Http11Protocol start

信息: Starting Coyote HTTP/1.1 on http-7100

七月 11, 2016 7:57:23 下午 org.apache.jk.common.ChannelSocket init

信息: JK: ajp13 listening on /0.0.0.0:7109

七月 11, 2016 7:57:23 下午 org.apache.jk.server.JkMain start

信息: Jk running ID=0 time=0/29 config=null

七月 11, 2016 7:57:23 下午 org.apache.catalina.startup.Catalina start

信息: Server startup in 3616 ms

2，检查工程spring的数据库配置文件

Jdbc配置文件check后，都正常，有相应的maxIdleTime等设置，c3p0正常，如下：

vim ./webapps/PLATFORM_SSO_Web/WEB-INF/classes/META-INF/spring/db/jdbc/applicationContext_jdbc.xml

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE beans PUBLIC "-//SPRING//DTD BEAN//EN" "http://www.springframework.org/dtd/spring-beans.dtd">

<beans>

<bean id="dataSource"

class="com.mchange.v2.c3p0.ComboPooledDataSource"

destroy-method="close">

<value>${sso.datasource.driverClassName}</value>

</property>

<value>${sso.datasource.url}</value>

</property>

<value>${sso.datasource.username}</value>

</property>

<value>${sso.datasource.password}</value>

</property>

<value>${sso.datasource.minPoolSize}</value>

</property>

<value>${sso.datasource.maxPoolSize}</value>

</property>

<value>${sso.datasource.initialPoolSize}</value>

</property>

<value>${sso.datasource.maxIdleTime}</value>

</property>

<value>${sso.datasource.acquireIncrement}</value>

</property>

<!--JDBC的标准参数，用以控制数据源内加载的PreparedStatements数量。但由于预缓存的statements

属于单个connection而不是整个连接池。所以设置这个参数需要考虑到多方面的因素。

如果maxStatements与maxStatementsPerConnection均为0，则缓存被关闭。Default: 0-->

<value>${sso.datasource.maxStatements}</value>

</property>

<value>${sso.datasource.idleConnectionTestPeriod}</value>

</property>

<value>${sso.datasource.acquireRetryAttempts}</value>

</property>

<!--获取连接失败将会引起所有等待连接池来获取连接的线程抛出异常。但是数据源仍有效

保留，并在下次调用getConnection()的时候继续尝试获取连接。如果设为true，那么在尝试

获取连接失败后该数据源将申明已断开并永久关闭。Default: false-->

<value>${sso.datasource.breakAfterAcquireFailure}</value>

</property>

<!--因性能消耗大请只在需要的时候使用它。如果设为true那么在每个connection提交的

时候都将校验其有效性。建议使用idleConnectionTestPeriod或automaticTestTable

等方法来提升连接测试的性能。Default: false -->

<value>${sso.datasource.testConnectionOnCheckout}</value>

</property>

</bean>

</bean>

</beans>

3，查看工程参数配置文件

查看工程启动的参数配置文件，check里面的数据库配置等等，一切正常使用，如下所示：

[tomcat@dev_121_21 apache-tomcat-6.0.37_7100]$ more ./webapps/PLATFORM_SSO_Web/WEB-INF/classes/system.properties |grep -v "#"

system.id=1000

system.env=default

properties.hessian.serviceUrl=http://bs.powerlong.com/PLATFORM_PROPERTIES_Service/remoting/PropertiesService

sso.server.login.url=http://bs.powerlong.com/PLATFORM_SSO_Web/login.htm

sso.domain=

sso.restriction.policy.error.number=5

sso.exception.error.page=

mobile.verify.interface.url=

auth.server.failed.url=http://bs.powerlong.com/PLATFORM_AUTH_Web/auth/failed.htm

sso.datasource.url=jdbc:mysql://zitandbtest.powerlong.com:3306/platform_db?useUnicode=true&characterEncoding=utf-8&autoReconnect=true&preferredTestQuery=select now()

sso.datasource.driverClassName=com.mysql.jdbc.Driver

sso.datasource.username=platform_user

sso.datasource.password=ysh1124

sso.datasource.maxPoolSize=120

sso.datasource.minPoolSize=5

sso.datasource.initialPoolSize=10

sso.datasource.maxIdleTime=10

sso.datasource.acquireIncrement=5

sso.datasource.maxStatements=0

sso.datasource.idleConnectionTestPeriod=10

sso.datasource.acquireRetryAttempts=30

sso.datasource.breakAfterAcquireFailure=true

sso.datasource.testConnectionOnCheckout=true

url.static=http://bsstatic.powerlong.com

url.static.https=http://bsstatic.powerlong.com

url.www=http://bs.powerlong.com/PLATFORM_MANAGER_Web/manager/index.htm

url.address=http://bsstatic.powerlong.com

base_path = http://bs.powerlong.com/PLATFORM_MANAGER_Web

auth_web_path = http://bs.powerlong.com/PLATFORM_AUTH_Web/

[tomcat@dev_121_21 apache-tomcat-6.0.37_7100]$

4，最终的解决办法

（1）重启7100的tomcat后，仍然一模一样的错误。

（2） Kill掉mysql5.7上的所有session线程后重启tomcat，仍然报一模一样的错误。

（3）重启mysql数据库后再重启tomcat，仍然报一模一样的错误。

上午是好的，下午出问题了，重启tomcat和mysql都不能解决，那么问题就在别的地方了？那么问题在哪里？再次检查参数文件里面，有

properties.hessian.serviceUrl=http://bs.powerlong.com/PLATFORM_PROPERTIES_Service/remoting/PropertiesService

这个工程是外部工程，在7200的tomcat上面，难道是外部的工程引起的？

然后ps –eaf|grep 7200; 发现有3个启动的7200进程，那么就是7200启动失败而且卡住了，强行使用kill -9 pid清除7200进程，然后重启7200tomcat成功后，再次启动7100的tomcat进程，这次成功了，没有报错了。

此次问题是由于服务器上的7200端口的tomcat异常，启动了3个，而7100的tomcat需要先去7200上面获取service接口，去通过调用7200的service接口来初始化数据库连接，所以7200异常了，那么7100启动获取资源建立数据库连接池就会报错了。

5，补充后续遇到的处理方案

（1）INNODB_TRX表有事务一直没有被释放

（2）因为模块化，web项目和service项目是分开的，service项目没有启动起来，所以导致web项目启动获取不了资源报错，这个时候需要先启动web对应的service项目，然后再启动web项目即可。