项目场景：

在迁移CDH集群的元数据库后，重新启动集群，发现启动失败。

APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!

问题描述

在启动cloudera-scm-server 时，日志如下

2022-08-25 17:24:22,222 INFO main:com.cloudera.server.cmf.Main: ================================================================================
2022-08-25 17:24:22,230 INFO main:com.cloudera.server.cmf.Main: Starting SCM Server. JVM Args: [-Dlog4j.configuration=file:/etc/cloudera-scm-server/log4j.properties, -Dfile.encoding=UTF-8, -Dcmf.root.logger=INFO,LOGFILE, -Dcmf.log.dir=/var/log/cloudera-scm-server, -Dcmf.log.file=cloudera-scm-server.log, -Dcmf.jetty.threshhold=WARN, -Dcmf.schema.dir=/opt/cloudera/cm/schema, -Djava.awt.headless=true, -Djava.net.preferIPv4Stack=true, -Dpython.home=/opt/cloudera/cm/python, -XX:+UseConcMarkSweepGC, -XX:+UseParNewGC, -XX:+HeapDumpOnOutOfMemoryError, -Xmx2G, -XX:MaxPermSize=256m, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/tmp, -XX:OnOutOfMemoryError=kill -9 %p], Args: [], Version: 6.2.1 (#1426065 built by jenkins on 20190911-0956 git: 94625ae139b266e8cdad073d744766545f7f0472)
2022-08-25 17:24:22,373 INFO main:org.springframework.context.support.ClassPathXmlApplicationContext: Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@11758f2a: startup date [Thu Aug 25 17:24:22 CST 2022]; root of context hierarchy
2022-08-25 17:24:22,432 INFO main:org.springframework.beans.factory.xml.XmlBeanDefinitionReader: Loading XML bean definitions from class path resource [webapp/WEB-INF/spring/beanRefFactory.xml]
2022-08-25 17:24:22,808 INFO main:org.springframework.context.support.GenericApplicationContext: Refreshing org.springframework.context.support.GenericApplicationContext@671a5887: startup date [Thu Aug 25 17:24:22 CST 2022]; root of context hierarchy
2022-08-25 17:24:22,880 INFO main:com.cloudera.enterprise.CommonMain: Reading database properties from /etc/cloudera-scm-server/db.properties
2022-08-25 17:24:22,883 INFO main:com.cloudera.enterprise.CommonMain: Statistics not enabled, c3p0 JMX disabled
2022-08-25 17:24:24,702 INFO main:com.mchange.v2.log.MLog: MLog clients using log4j logging.
2022-08-25 17:24:24,762 INFO main:com.mchange.v2.c3p0.C3P0Registry: Initializing c3p0-0.9.2.1 [built 20-March-2013 10:47:27 +0000; debug? true; trace: 10]
2022-08-25 17:24:24,852 INFO main:com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource: Initializing c3p0 pool... com.mchange.v2.c3p0.PoolBackedDataSource@3d216406 [ connectionPoolDataSource -> com.mchange.v2.c3p0.WrapperConnectionPoolDataSource@b45fca09 [ acquireIncrement -> 3, acquireRetryAttempts -> 5, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 20000, connectionCustomizerClassName -> null, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, debugUnreturnedConnectionStackTraces -> false, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> 1bqvrwraq1mwrx23bgnttf|f288c14, idleConnectionTestPeriod -> 300, initialPoolSize -> 5, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 0, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 50, maxStatements -> 2500, maxStatementsPerConnection -> 0, minPoolSize -> 5, nestedDataSource -> com.mchange.v2.c3p0.DriverManagerDataSource@d045e305 [ description -> null, driverClass -> null, factoryClassLocation -> null, identityToken -> 1bqvrwraq1mwrx23bgnttf|4d654825, jdbcUrl -> jdbc:mysql://172.21.85.147:60001/yptdevscm?useUnicode=true&characterEncoding=UTF-8, properties -> {user=******, password=******} ], preferredTestQuery -> null, propertyCycle -> 0, statementCacheNumDeferredCloseThreads -> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 0, usesTraditionalReflectiveProxies -> false; userOverrides: {} ], dataSourceName -> null, factoryClassLocation -> null, identityToken -> 1bqvrwraq1mwrx23bgnttf|53b8afea, numHelperThreads -> 3 ]
2022-08-25 17:24:44,852 WARN C3P0PooledConnectionPoolManager[identityToken->1bqvrwraq1mwrx23bgnttf|53b8afea]-AdminTaskTimer:com.mchange.v2.async.ThreadPoolAsynchronousRunner: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@125fa98c -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
2022-08-25 17:24:44,854 WARN C3P0PooledConnectionPoolManager[identityToken->1bqvrwraq1mwrx23bgnttf|53b8afea]-AdminTaskTimer:com.mchange.v2.async.ThreadPoolAsynchronousRunner: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@125fa98c -- APPARENT DEADLOCK!!! Complete Status:Managed Threads: 3Active Threads: 3Active Tasks:com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@5da54f8con thread: C3P0PooledConnectionPoolManager[identityToken->1bqvrwraq1mwrx23bgnttf|53b8afea]-HelperThread-#2com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@32d99e7don thread: C3P0PooledConnectionPoolManager[identityToken->1bqvrwraq1mwrx23bgnttf|53b8afea]-HelperThread-#1Pending Tasks:com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@4a245f24com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@3d8ab0c3
Pool thread stack traces:Thread[C3P0PooledConnectionPoolManager[identityToken->1bqvrwraq1mwrx23bgnttf|53b8afea]-HelperThread-#2,5,main]java.net.PlainSocketImpl.socketConnect(Native Method)java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)java.net.Socket.connect(Socket.java:589)java.net.Socket.connect(Socket.java:538)java.net.Socket.<init>(Socket.java:434)java.net.Socket.<init>(Socket.java:244)com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:258)com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:306)com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2504)com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2541)com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2323)com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:832)com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)java.lang.reflect.Constructor.newInstance(Constructor.java:423)com.mysql.jdbc.Util.handleNewInstance(Util.java:408)com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:417)com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:344)com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:146)com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:195)com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:184)com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:200)com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1086)com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1073)com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:44)

原因分析：

盯着 APPARENT DEADLOCK!!! 的报错找了好久，差点要改C3P0的jar包了，

最后猛然发现，是当前机器和远程数据库的网络没有打通。。cloudera-scm-server 的日志里面也体现不出来网络不通的信息。。

迁移数据库后，环境并没有检查很仔细

解决方案：

环境问题真的很让人头疼

APPARENT DEADLOCK Creating emergency threads for unassigned pending tasks相关推荐

c3p0 APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks
2018-01-04 15:02:03,319 ---com.mchange.v2.async.ThreadPoolAsynchronousRunner: com.mchange.v2.async.T ...
C3P0连接池问题，APPARENT DEADLOCK!!! Creating emergency..... [问题点数：20分，结帖人lovekong]...
采用c3p0连接池,每次调试程序,第一次访问时(Tomcat服务器重启后再访问)都会出现以下错误,然后连接库需要很长时间,最终是可以连上的,之后再访问就没问题了,请高手们会诊一下,希望能帮小弟解决此问 ...
警告: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@21eb3f -- APPARENT DEADLOCK!!
警告: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@21eb3f -- APPARENT DEADLOCK!! ...
记录一个c3po连接池APPARENT DEADLOCK解决方法
在使用c3p0数据池时,如果连接池参数配置不当,可能会引发一个意想不到的问题. 数据库:Oracle c3p0版本:0.9.5.2 数据源配置: application.xml <bean id ...
oracle怎么deadlock,APPARENT DEADLOCK!!! 错误解决过程
项目环境数据库:oracle 中间件:WAS 2. 问题描述 2015.05.21(是个好日子),早上发现was后台报错如下: [15-5-21 7:31:01:171 CST] 00000027 ...
c3p0存在严重bug “APPARENT DEADLOCK“的问题
开发时用了Spring JdbcTemplate + c3p0组合,做性能测试时出现严重的性能问题.40的并发访问,开始时候系统正常,几分钟后,应用就不能访问了.停止40并发的压力,1分钟左右又可以正 ...
C3P0错误APPARENT DEADLOCK!!!解决
在使用C3P0的过程中出现了好多错误.最长见就是死锁,占用资源比较大. 在日志中出现大量如下日志 16:18:01.294 ERROR org.hibernate.util.JDBCException ...
C3P0访问数据库死锁，com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@128e59b3 -- APPARENT
错误信息: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@128e59b3 -- APPARENT DEADLO ...
java deadlock oracle_APPARENT DEADLOCK!!! - C3P0连接池DeadLock机制分析
1 问题近期,刚上线不久的生产系统的数据库连接池 C3P0 (版本为0.9.5.2)突然报出 APPARENT DEADLOCK!!! 错误. 1.1 错误日志错误日志如下. com.mchang ...

APPARENT DEADLOCK Creating emergency threads for unassigned pending tasks

项目场景：

问题描述

原因分析：

解决方案：

APPARENT DEADLOCK Creating emergency threads for unassigned pending tasks相关推荐

最新文章

热门文章