场景

业务一切正常，突然收到一堆告警，发现全是
java.util.concurrent.RejectedExecutionException异常报错。
具体看了下代码，里面的执行逻辑也不难，没有外部依赖都是内存多线程cpu类型计算的逻辑。
下面是报错的线程池状态变化，由上到下，按照时间增长，最后达到饱和。

java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 99, active threads = 75, queued tasks = 5000, completed tasks = 1595259]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 101, active threads = 74, queued tasks = 4999, completed tasks = 1595298]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 90, active threads = 70, queued tasks = 4998, completed tasks = 1595354]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 93, active threads = 75, queued tasks = 5000, completed tasks = 1595378]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 85, active threads = 72, queued tasks = 5000, completed tasks = 1595443]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 83, active threads = 70, queued tasks = 5000, completed tasks = 1595447]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 90, active threads = 81, queued tasks = 5000, completed tasks = 1595476]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 85, active threads = 74, queued tasks = 4994, completed tasks = 1595574]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 91, active threads = 80, queued tasks = 4999, completed tasks = 1595678]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 95, active threads = 85, queued tasks = 5000, completed tasks = 1595696]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 95, active threads = 87, queued tasks = 5000, completed tasks = 1595721]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 93, active threads = 85, queued tasks = 4999, completed tasks = 1595727]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 94, active threads = 85, queued tasks = 4998, completed tasks = 1595737]
...
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 109, active threads = 96, queued tasks = 4999, completed tasks = 1596150]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 113, active threads = 102, queued tasks = 4998, completed tasks = 1596191]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 120, active threads = 109, queued tasks = 5000, completed tasks = 1596355]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 120, active threads = 110, queued tasks = 5000, completed tasks = 1596517]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 119, active threads = 110, queued tasks = 4999, completed tasks = 1596519]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 113, active threads = 105, queued tasks = 5000, completed tasks = 1596553]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 122, active threads = 109, queued tasks = 4999, completed tasks = 1596602]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 120, active threads = 120, queued tasks = 5000, completed tasks = 1597061]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 120, active threads = 120, queued tasks = 5000, completed tasks = 1597061]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 120, active threads = 120, queued tasks = 5000, completed tasks = 1597061]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 120, active threads = 120, queued tasks = 5000, completed tasks = 1597061]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 120, active threads = 120, queued tasks = 5000, completed tasks = 1597061]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 120, active threads = 120, queued tasks = 5000, completed tasks = 1597061]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 120, active threads = 120, queued tasks = 5000, completed tasks = 1597061]
java.util.concurrent.ThreadPoolExecutor@57b5ea0f[Running, pool size = 120, active threads = 120, queued tasks = 5000, completed tasks = 1597061]

因为cpu4核，所以coresize = 40,maxsize = 120，都正常。

    private static BlockingQueue<Runnable> executorQueue = new LinkedBlockingQueue<>(5000);private static ExecutorService executorService = new ThreadPoolExecutor(Runtime.getRuntime().availableProcessors() * 10,Runtime.getRuntime().availableProcessors() * 30,60, TimeUnit.SECONDS, executorQueue);

通过对ThreadPoolExecutor类分析，引发java.util.concurrent.RejectedExecutionException主要有两种原因：

  1. 线程池显示的调用了shutdown()之后，再向线程池提交任务的时候，如果你配置的拒绝策略是ThreadPoolExecutor.AbortPolicy的话，这个异常就被会抛出来。
  2. 当你的排队策略为有界队列，并且配置的拒绝策略是ThreadPoolExecutor.AbortPolicy，当线程池的线程数量已经达到了maximumPoolSize的时候，你再向它提交任务，就会抛出ThreadPoolExecutor.AbortPolicy异常。

排查

因为全机房、只有一个节点报错，而其他机器都正常，在这个一定保证正确的前提下。
1、看了线上的请求量、并没有突增，所以排除外部因素
2、看了代码逻辑，逻辑内部并没有打印报错日志，说明不是线程执行耗时导致后面的其他线程排队
3、排除线程池是否提前关闭。并没有，因为手动没有显示关闭，另外看日志也知道里面的线程数还在变化，所以不存在关闭的说法。

考虑是否是机器本身的原因，后面经过排查，看到那个点线上的cpu使用率突然升高、系统负载突然飙升，网卡流出、流入报文数目、tcp连接数也突然为0。具体干了什么，已经让运营去看了。至少说明我们代码没有问题。

如下图，可以感受下：

解决

线上第一时间出现这个问题，没想2秒，看了下报错代码逻辑，然后重启了这个节点实例（当时重启完后，观察又没问题了，带着疑惑然后才去排查）

java.util.concurrent.RejectedExecutionException异常相关推荐

  1. java.util.concurrent.RejectedExecutionException

    2019独角兽企业重金招聘Python工程师标准>>> 遇到java.util.concurrent.RejectedExecutionException 目前看来,最主要有2种原因 ...

  2. 1]解决java.util.concurrent.RejectedExecutionException

    今天学习了java的并发,线程池,同一时间执行一个操作. 报错:java.util.concurrent.RejectedExecutionException,排查发现是等待队列设小了,导致 拒绝策略 ...

  3. 关于java.util.concurrent.RejectedExecutionException: event executor terminated

    多线程报了个java.util.concurrent.RejectedExecutionException: event executor terminated 线程池的拒绝策略 ThreadPool ...

  4. 【ruoyi】java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoo

    前言 ruoyi 4.6.0 jdk1.8 错误 11:48:16.879 [http-nio-9031-exec-25] INFO c.r.f.s.r.UserRealm - [doGetAuthe ...

  5. java.util.concurrent.RejectedExecutionException: event executor terminated 错误分析

    java.util.concurrent.RejectedExecutionException: event executor terminated 错误分析

  6. 已解决java.util.concurrent.ExecutionException异常的正确解决方法，亲测有效！！！

    已解决java.util.concurrent.ExecutionException异常的正确解决方法,亲测有效!!! 文章目录 报错问题 解决方法 福利 报错问题 粉丝群里面的一个小伙伴敲代码时发生 ...

  7. 线程池java.util.concurrent.ThreadPoolExecutor总结

    http://uule.iteye.com/blog/1123185 线程池还具有提高系统性能的优点,因为创建线程和清除线程的开销比较大. 有两种不同类型的线程池:一是固定线程数量的线程池:二是可变数 ...

  8. JDK1.5中的线程池(java.util.concurrent.ThreadPoolExecutor

    http://www.diybl.com/course/3_program/java/javajs/200797/70003.html 在多线程大师Doug Lea的贡献下,在JDK1.5中加入了许多 ...

  9. JDK1.5中的线程池(java.util.concurrent.ThreadPoolExecut

    JDK1.5中的线程池(java.util.concurrent.ThreadPoolExecut 在多线程大师Doug Lea的贡献下,在JDK1.5中加入了许多对并发特性的支持,例如:线程池. 一 ...

  10. 线程池(java.util.concurrent.ThreadPoolExecutor)的使用

    如果大家觉得这个类还不能完全满足自己的要求的话,其实可以照搬这个源码,然后适当改动一下来适合自己的需求,也不失为一种捷径的.呵呵,本人最近在自己做的一个项目中,就来了这一手.虽然不是多好,主要是为了满 ...

最新文章

  1. 资源下载 | 历年 AAAI 最佳论文（since 1996）
  2. 【预告】新房装修网络规划网络布局学习笔记，家庭网络需求文章内容及结构...
  3. django-oscar的默认地点打开后总是united kingdom
  4. P2626 斐波那契数列（升级版）
  5. BUUCTF-- Linux Labs 1---SSH远程登陆
  6. 深度学习中的信息论——交叉熵
  7. 《高性能MySQL》读书笔记-mysql数据类型和应用
  8. 云服务器 ECS ＞ 快照 ＞ 快照概述
  9. Vulnhub-HackInOS
  10. js 正则表达式总结
  11. 计算机凭证打印格式设置,金蝶软件套打如何设置用KP-J103凭证纸打印凭证
  12. QQ淘宝自动转链转发机器人MkStone京东淘宝转链助手
  13. 100行代码搞定Python做OCR识别身份证，文字各种字体！
  14. 电脑蓝屏的几种状况处理
  15. English Summary~July
  16. 超好玩的硬币游戏，你会玩吗？
  17. 数字计算机模拟人脑,人造突触问世 计算机模拟人脑不是梦
  18. 促进目标实现的激励技巧
  19. 开关电源拓扑结构详解
  20. Actor 分布式并行计算模型: The Actor Model for Concurrent Computation

热门文章

  1. Linux时间同步（Fri Nov 16 12:12:13 Local time zone must be set--see zic manual page 2018）解决办法
  2. 新机器部署linux系统（非虚拟机）
  3. Linux(四)：什么是Bash、什么是shell？
  4. AndroidStudio报错Transform output file D:\android\RfidDemo\app\libs\RFID_lib.jar does not exist.
  5. 1.5 极限的存在准则和两个重要极限
  6. ada：世界上第一位程序员
  7. 为自己的博客系统写的一篇自我介绍
  8. c语言编程绘制空间螺旋线,在UG中绘制变螺距螺旋线的方法
  9. 一年之计在于春丨看， 农田里的科技范儿
  10. flutter 轮播组件 Swiper