chaos-monkey-spring-boot小试牛刀

序

本文主要介绍下chaos-monkey-spring-boot

Chaos Monkey for Spring Boot

chaos-monkey-spring-boot是专门为Spring Boot打造的Chaos Monkey

主要有有如下几个方面的Assaults

Latency Assault
Exception Assault
AppKiller Assault

实例

maven

<dependency><groupId>de.codecentric</groupId><artifactId>chaos-monkey-spring-boot</artifactId><version>1.0.1</version>
</dependency>

配置

chaos.monkey.assaults.level=5
chaos.monkey.assaults.latencyRangeStart=10000
chaos.monkey.assaults.latencyRangeEnd=15000
chaos.monkey.assaults.latencyActive=true
chaos.monkey.assaults.exceptionsActive=true
chaos.monkey.assaults.killApplicationActive=true
chaos.monkey.watcher.controller=true
chaos.monkey.watcher.restController=true
chaos.monkey.watcher.service=true
chaos.monkey.watcher.repository=true

运行

2018-04-20 22:50:02.475  INFO 2861 --- [           main] d.c.s.b.c.monkey.component.ChaosMonkey   : _____ _                       __  __             _/ ____| |                     |  \/  |           | || |    | |__   __ _  ___  ___  | \  / | ___  _ __ | | _____ _   _| |    | '_ \ / _` |/ _ \/ __| | |\/| |/ _ \| '_ \| |/ / _ | | | || |____| | | | (_| | (_) \__ \ | |  | | (_) | | | |   |  __| |_| |\_____|_| |_|\__,_|\___/|___/ |_|  |_|\___/|_| |_|_|\_\___|\__, |__/ |_ready to do evil!                                         |___/:: Chaos Monkey for Spring Boot                                    ::

使用-Dspring.profiles.active=chaos-monkey

Assaluts输出

Latency Assault

2018-04-20 22:37:17.373  INFO 2827 --- [nio-8080-exec-9] d.c.s.b.c.monkey.component.ChaosMonkey   : Chaos Monkey - timeout

Exception Assault

2018-04-20 22:37:01.379 ERROR 2827 --- [nio-8080-exec-5] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.RuntimeException: Chaos Monkey - RuntimeException] with root causejava.lang.RuntimeException: Chaos Monkey - RuntimeExceptionat de.codecentric.spring.boot.chaos.monkey.component.ChaosMonkey.generateChaosException(ChaosMonkey.java:76) ~[chaos-monkey-spring-boot-1.0.1.jar:na]at de.codecentric.spring.boot.chaos.monkey.component.ChaosMonkey.callChaosMonkey(ChaosMonkey.java:45) ~[chaos-monkey-spring-boot-1.0.1.jar:na]at de.codecentric.spring.boot.chaos.monkey.watcher.SpringRestControllerAspect.intercept(SpringRestControllerAspect.java:37) ~[chaos-monkey-spring-boot-1.0.1.jar:na]at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_71]at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_71]at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_71]at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_71]at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644) ~[spring-aop-5.0.5.RELEASE.jar:5.0.5.RELEASE]at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633) ~[spring-aop-5.0.5.RELEASE.jar:5.0.5.RELEASE]at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70) ~[spring-aop-5.0.5.RELEASE.jar:5.0.5.RELEASE]at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) ~[spring-aop-5.0.5.RELEASE.jar:5.0.5.RELEASE]at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-5.0.5.RELEASE.jar:5.0.5.RELEASE]at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) ~[spring-aop-5.0.5.RELEASE.jar:5.0.5.RELEASE]at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689) ~[spring-aop-5.0.5.RELEASE.jar:5.0.5.RELEASE]at com.example.controller.HelloController$$EnhancerBySpringCGLIB$$1390f753.hello(<generated>) ~[classes/:na]at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_71]at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_71]at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_71]at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_71]at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209) ~[spring-web-5.0.5.RELEASE.jar:5.0.5.RELEASE]at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136) ~[spring-web-5.0.5.RELEASE.jar:5.0.5.RELEASE]at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102) ~[spring-webmvc-5.0.5.RELEASE.jar:5.0.5.RELEASE]at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:877) ~[spring-webmvc-5.0.5.RELEASE.jar:5.0.5.RELEASE]at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:783) ~[spring-webmvc-5.0.5.RELEASE.jar:5.0.5.RELEASE]at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.0.5.RELEASE.jar:5.0.5.RELEASE]at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991) ~[spring-webmvc-5.0.5.RELEASE.jar:5.0.5.RELEASE]at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925) ~[spring-webmvc-5.0.5.RELEASE.jar:5.0.5.RELEASE]at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:974) ~[spring-webmvc-5.0.5.RELEASE.jar:5.0.5.RELEASE]at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:866) ~[spring-webmvc-5.0.5.RELEASE.jar:5.0.5.RELEASE]at javax.servlet.http.HttpServlet.service(HttpServlet.java:635) ~[tomcat-embed-core-8.5.29.jar:8.5.29]at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:851) ~[spring-webmvc-5.0.5.RELEASE.jar:5.0.5.RELEASE]at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) ~[tomcat-embed-core-8.5.29.jar:8.5.29]at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-8.5.29.jar:8.5.29]at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.29.jar:8.5.29]at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) ~[tomcat-embed-websocket-8.5.29.jar:8.5.29]at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.29.jar:8.5.29]at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.29.jar:8.5.29]at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:158) ~[spring-boot-actuator-2.0.1.RELEASE.jar:2.0.1.RELEASE]at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:126) ~[spring-boot-actuator-2.0.1.RELEASE.jar:2.0.1.RELEASE]at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:111) ~[spring-boot-actuator-2.0.1.RELEASE.jar:2.0.1.RELEASE]at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.5.RELEASE.jar:5.0.5.RELEASE]at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.29.jar:8.5.29]at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.29.jar:8.5.29]at org.springframework.boot.actuate.web.trace.servlet.HttpTraceFilter.doFilterInternal(HttpTraceFilter.java:84) ~[spring-boot-actuator-2.0.1.RELEASE.jar:2.0.1.RELEASE]at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.5.RELEASE.jar:5.0.5.RELEASE]at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.29.jar:8.5.29]at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.29.jar:8.5.29]at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-5.0.5.RELEASE.jar:5.0.5.RELEASE]at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.5.RELEASE.jar:5.0.5.RELEASE]at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.29.jar:8.5.29]at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.29.jar:8.5.29]at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109) ~[spring-web-5.0.5.RELEASE.jar:5.0.5.RELEASE]at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.5.RELEASE.jar:5.0.5.RELEASE]at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.29.jar:8.5.29]at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.29.jar:8.5.29]at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) ~[spring-web-5.0.5.RELEASE.jar:5.0.5.RELEASE]at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.5.RELEASE.jar:5.0.5.RELEASE]at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.29.jar:8.5.29]at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.29.jar:8.5.29]at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) ~[spring-web-5.0.5.RELEASE.jar:5.0.5.RELEASE]at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.5.RELEASE.jar:5.0.5.RELEASE]at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-8.5.29.jar:8.5.29]at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-8.5.29.jar:8.5.29]at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) ~[tomcat-embed-core-8.5.29.jar:8.5.29]at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-8.5.29.jar:8.5.29]at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496) [tomcat-embed-core-8.5.29.jar:8.5.29]at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.29.jar:8.5.29]at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [tomcat-embed-core-8.5.29.jar:8.5.29]at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.29.jar:8.5.29]at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [tomcat-embed-core-8.5.29.jar:8.5.29]at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) [tomcat-embed-core-8.5.29.jar:8.5.29]at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.29.jar:8.5.29]at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790) [tomcat-embed-core-8.5.29.jar:8.5.29]at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459) [tomcat-embed-core-8.5.29.jar:8.5.29]at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.29.jar:8.5.29]at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_71]at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_71]at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.29.jar:8.5.29]at java.lang.Thread.run(Thread.java:745) [na:1.8.0_71]

AppKiller Assault

2018-04-20 22:51:08.271  INFO 2861 --- [nio-8080-exec-3] d.c.s.b.c.monkey.component.ChaosMonkey   : Chaos Monkey - I am killing your Application!
2018-04-20 22:51:08.272  INFO 2861 --- [nio-8080-exec-3] ConfigServletWebServerApplicationContext : Closing org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@2b6856dd: startup date [Fri Apr 20 22:49:58 CST 2018]; root of context hierarchy
2018-04-20 22:51:08.275  INFO 2861 --- [nio-8080-exec-3] o.s.j.e.a.AnnotationMBeanExporter        : Unregistering JMX-exposed beans on shutdown

源码解析

ChaosMonkeyConfiguration

chaos-monkey-spring-boot-1.0.1-sources.jar!/de/codecentric/spring/boot/chaos/monkey/configuration/ChaosMonkeyConfiguration.java

@Configuration
@Profile("chaos-monkey")
@EnableConfigurationProperties({AssaultProperties.class, WatcherProperties.class})
@Import(EndpointConfiguration.class)
public class ChaosMonkeyConfiguration {private static final Logger LOGGER = LoggerFactory.getLogger(ChaosMonkey.class);private final WatcherProperties watcherProperties;private final AssaultProperties assaultProperties;public ChaosMonkeyConfiguration(WatcherProperties watcherProperties, AssaultProperties assaultProperties) {this.watcherProperties = watcherProperties;this.assaultProperties = assaultProperties;try {String chaosLogo = StreamUtils.copyToString(new ClassPathResource("chaos-logo.txt").getInputStream(), Charset.defaultCharset());LOGGER.info(chaosLogo);} catch (IOException e) {LOGGER.info("Chaos Monkey - ready to do evil");}}@Beanpublic ChaosMonkeySettings settings() {return new ChaosMonkeySettings(assaultProperties, watcherProperties);}@Beanpublic ChaosMonkey chaosMonkey() {return new ChaosMonkey(assaultProperties);}@Bean@Conditional(AttackControllerCondition.class)public SpringControllerAspect controllerAspect() {return new SpringControllerAspect(chaosMonkey());}@Bean@Conditional(AttackRestControllerCondition.class)public SpringRestControllerAspect restControllerAspect() {return new SpringRestControllerAspect(chaosMonkey());}@Bean@Conditional(AttackServiceCondition.class)public SpringServiceAspect serviceAspect() {return new SpringServiceAspect(chaosMonkey());}}

这里加载了配置，然后实例化了ChaosMonkey，以及根据条件实例化Controller，RestController，Service注解的aspect，其原理就是根据aspect进行拦截处理，产生相应的chaos

ChaosMonkey

chaos-monkey-spring-boot-1.0.1-sources.jar!/de/codecentric/spring/boot/chaos/monkey/component/ChaosMonkey.java

    public void callChaosMonkey() {if (isTrouble()) {int exceptionRand = assaultProperties.getExceptionRandom();if (assaultProperties.isLatencyActive() && assaultProperties.isExceptionsActive()) {// Timeout or Exception?if (exceptionRand < 7) {generateLatency();} else {generateChaosException();}} else if (assaultProperties.isLatencyActive()) {generateLatency();} else if (assaultProperties.isExceptionsActive()) {generateChaosException();} else if (assaultProperties.isKillApplicationActive()) {killTheBossApp();}}}

ChaosMonkey主要提供了generateLatency、generateChaosException、killTheBossApp这几个方法来分别实现Latency Assault、Exception Assault、AppKiller Assault

Aspect实例

chaos-monkey-spring-boot-1.0.1-sources.jar!/de/codecentric/spring/boot/chaos/monkey/watcher/SpringRestControllerAspect.java

@Aspect
public class SpringRestControllerAspect {private static final Logger LOGGER = LoggerFactory.getLogger(SpringRestControllerAspect.class);private final ChaosMonkey chaosMonkey;public SpringRestControllerAspect(ChaosMonkey chaosMonkey) {this.chaosMonkey = chaosMonkey;}@Pointcut("within(@org.springframework.web.bind.annotation.RestController *)")public void classAnnotatedWithControllerPointcut() {}@Pointcut("execution(* *.*(..))")public void allPublicMethodPointcut() {}@Around("classAnnotatedWithControllerPointcut() && allPublicMethodPointcut()")public Object intercept(ProceedingJoinPoint pjp) throws Throwable {LOGGER.debug(LOGGER.isDebugEnabled() ? "RestController class and public method detected: " + pjp.getSignature() : null);chaosMonkey.callChaosMonkey();return pjp.proceed();}}

可以看到这里使用@Around拦截，然后调用chaosMonkey.callChaosMonkey()总的入口方法，由它判断是否需要产生chaos

小结

chaos-monkey-spring-boot是个好东东，非常适合用来进行故障演练，暴露服务间调用的问题，好提升系统的健壮性、故障自动恢复能力等。

doc

Chaos工程
Twilio的混沌工程实践
chaos-monkey-spring-boot
PRINCIPLES OF CHAOS ENGINEERING
Chaos Engineering的历史、原则以及实践