JVM SandBox源码解析（一）：启动时初始化、启动时加载模块、ModuleHttpServlet进行Http路由

前言

上篇JVM SandBox实现原理详解文章中，主要解析了JVM SandBox的核心实现原理，并且对SandBoxClassLoader和ModuleClassLoader做了源码解析，也解释了在用户自定义模块中为何能使用部分sandbox-core中的类。本文主要对JVM SandBox的核心功能进行源码解析，主要包含以下几部分内容：

启动时初始化
启动时加载模块
ModuleHttpServlet进行Http路由
增强目标类
模块刷新
模块卸载

1、启动时初始化

1）、脚本分析

Attach方式启动：sh sandbox.sh -p pid

启动脚本sandbox.sh如下：

# the sandbox main function
function main() {while getopts "hp:vFfRu:a:A:d:m:I:P:ClSn:X" ARG; docase ${ARG} inh)usageexit;;p) TARGET_JVM_PID=${OPTARG} ;;v) OP_VERSION=1 ;;l) OP_MODULE_LIST=1 ;;R) OP_MODULE_RESET=1 ;;F) OP_MODULE_FORCE_FLUSH=1 ;;f) OP_MODULE_FLUSH=1 ;;u)OP_MODULE_UNLOAD=1ARG_MODULE_UNLOAD=${OPTARG};;a)OP_MODULE_ACTIVE=1ARG_MODULE_ACTIVE=${OPTARG};;A)OP_MODULE_FROZEN=1ARG_MODULE_FROZEN=${OPTARG};;d)OP_DEBUG=1ARG_DEBUG=${OPTARG};;m)OP_MODULE_DETAIL=1ARG_MODULE_DETAIL=${OPTARG};;I) TARGET_SERVER_IP=${OPTARG} ;;P) TARGET_SERVER_PORT=${OPTARG} ;;C) OP_CONNECT_ONLY=1 ;;S) OP_SHUTDOWN=1 ;;n)OP_NAMESPACE=1ARG_NAMESPACE=${OPTARG};;X) set -x ;;?)usageexit_on_err 1;;esacdonereset_for_envcheck_permission# reset IP[ -z "${TARGET_SERVER_IP}" ] && TARGET_SERVER_IP="${DEFAULT_TARGET_SERVER_IP}"# reset PORT[ -z "${TARGET_SERVER_PORT}" ] && TARGET_SERVER_PORT=0# reset NAMESPACE[[ ${OP_NAMESPACE} ]] &&TARGET_NAMESPACE=${ARG_NAMESPACE}[[ -z ${TARGET_NAMESPACE} ]] &&TARGET_NAMESPACE=${DEFAULT_NAMESPACE}if [[ ${OP_CONNECT_ONLY} ]]; then[[ 0 -eq ${TARGET_SERVER_PORT} ]] &&exit_on_err 1 "server appoint PORT (-P) was missing"SANDBOX_SERVER_NETWORK="${TARGET_SERVER_IP};${TARGET_SERVER_PORT}"else# -p was missing[[ -z ${TARGET_JVM_PID} ]] &&exit_on_err 1 "PID (-p) was missing."attach_jvmfi...省略代码...# defaultsandbox_curl "sandbox-info/version"exit}

在执行sandbox.sh的时候，先执行reset_for_env方法，再执行attach_jvm方法

reset_for_env方法：

# reset sandbox work environment
# reset some options for env
reset_for_env() {# 使用默认环境变量JAVA_HOME# use the env JAVA_HOME for default[[ -n "${JAVA_HOME}" ]] &&SANDBOX_JAVA_HOME="${JAVA_HOME}"# 或者通过TARGET_JVM_PID查找 设置sandbox环境变量# use the target JVM for SANDBOX_JAVA_HOME[[ -z "${SANDBOX_JAVA_HOME}" ]] &&SANDBOX_JAVA_HOME="$(lsof -p "${TARGET_JVM_PID}" |grep "/bin/java" |awk '{print $9}' |xargs ls -l |awk '{if($1~/^l/){print $11}else{print $9}}' |xargs ls -l |awk '{if($1~/^l/){print $11}else{print $9}}' |sed 's/\/bin\/java//g')"# 若${JAVA_HOME}/lib/tools.jar存在,则通过-Xbootclasspath/a这个配置,将它加入classpath末尾,为执行attach_jvm方法做准备# append toos.jar to JVM_OPT[[ -f "${SANDBOX_JAVA_HOME}"/lib/tools.jar ]] &&SANDBOX_JVM_OPS="${SANDBOX_JVM_OPS} -Xbootclasspath/a:${SANDBOX_JAVA_HOME}/lib/tools.jar"#fix for windows  shell $HOME diff with user.hometest -n "${USERPROFILE}" -a -z "$(cat "${SANDBOX_TOKEN_FILE}")" && SANDBOX_TOKEN_FILE=${USERPROFILE}/.sandbox.token}

attach_jvm方法：

# attach sandbox to target JVM
# return : attach jvm local info
function attach_jvm() {# got an tokenlocal tokentoken="$(date | head | cksum | sed 's/ //g')"# 通过java -jar命令启动sandbox-core.jar并传递参数:TARGET_JVM_PID、sandbox-agent.jar、启动要用到的数据信息# attach target jvm"${SANDBOX_JAVA_HOME}/bin/java" \${SANDBOX_JVM_OPS} \-jar "${SANDBOX_LIB_DIR}/sandbox-core.jar" \"${TARGET_JVM_PID}" \"${SANDBOX_LIB_DIR}/sandbox-agent.jar" \"home=${SANDBOX_HOME_DIR};token=${token};server.ip=${TARGET_SERVER_IP};server.port=${TARGET_SERVER_PORT};namespace=${TARGET_NAMESPACE}" ||exit_on_err 1 "attach JVM ${TARGET_JVM_PID} fail."# get network from attach resultSANDBOX_SERVER_NETWORK=$(grep "${token}" "${SANDBOX_TOKEN_FILE}" | grep "${TARGET_NAMESPACE}" | tail -1 | awk -F ";" '{print $3";"$4}')[[ -z ${SANDBOX_SERVER_NETWORK} ]] &&exit_on_err 1 "attach JVM ${TARGET_JVM_PID} fail, attach lose response."}

2）、sandbox-core

sandbox-core的pom文件中，通过mainClass指定了这个主函数，java -jar sandbox-core.jar命令会执行这个函数

            <plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-assembly-plugin</artifactId><executions><execution><goals><goal>attached</goal></goals><phase>package</phase><configuration><descriptorRefs><descriptorRef>jar-with-dependencies</descriptorRef></descriptorRefs><archive><manifest><mainClass>com.alibaba.jvm.sandbox.core.CoreLauncher</mainClass></manifest></archive></configuration></execution></executions></plugin>

CoreLauncher：

public class CoreLauncher {/*** 内核启动程序** @param args 参数*             [0] : PID*             [1] : agent.jar's value*             [2] : token*/public static void main(String[] args) {try {// check argsif (args.length != 3|| StringUtils.isBlank(args[0])|| StringUtils.isBlank(args[1])|| StringUtils.isBlank(args[2])) {throw new IllegalArgumentException("illegal args");}new CoreLauncher(args[0], args[1], args[2]);} catch (Throwable t) {t.printStackTrace(System.err);System.err.println("sandbox load jvm failed : " + getCauseMessage(t));System.exit(-1);}}public CoreLauncher(final String targetJvmPid,final String agentJarPath,final String token) throws Exception {// 加载agentattachAgent(targetJvmPid, agentJarPath, token);}// 加载Agentprivate void attachAgent(final String targetJvmPid,final String agentJarPath,final String cfg) throws Exception {VirtualMachine vmObj = null;try {//attach目标pidvmObj = VirtualMachine.attach(targetJvmPid);if (vmObj != null) {//加载sandbox-agent.jarvmObj.loadAgent(agentJarPath, cfg);}} finally {if (null != vmObj) {vmObj.detach();}}}

3）、sandbox-agent

sandbox-agent的pom文件中，配置了Premain-Class和Agent-Class两个参数，并且都指向AgentLauncher这个类

            <plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-assembly-plugin</artifactId><executions><execution><goals><goal>attached</goal></goals><phase>package</phase><configuration><descriptorRefs><descriptorRef>jar-with-dependencies</descriptorRef></descriptorRefs><archive><manifestEntries><Premain-Class>com.alibaba.jvm.sandbox.agent.AgentLauncher</Premain-Class><Agent-Class>com.alibaba.jvm.sandbox.agent.AgentLauncher</Agent-Class><Can-Redefine-Classes>true</Can-Redefine-Classes><Can-Retransform-Classes>true</Can-Retransform-Classes></manifestEntries></archive></configuration></execution></executions></plugin>

通过Attach方式执行会调用AgentLauncher的agentmain(String featureString, Instrumentation inst)方法，Agent方式启动会调用AgentLauncher的premain(String featureString, Instrumentation inst)方法

public class AgentLauncher {/*** 启动加载** @param featureString 启动参数*                      [namespace,prop]* @param inst          inst*/public static void premain(String featureString, Instrumentation inst) {LAUNCH_MODE = LAUNCH_MODE_AGENT;install(toFeatureMap(featureString), inst);}  /*** 动态加载** @param featureString 启动参数*                      [namespace,token,ip,port,prop]* @param inst          inst*/public static void agentmain(String featureString, Instrumentation inst) {LAUNCH_MODE = LAUNCH_MODE_ATTACH;final Map<String, String> featureMap = toFeatureMap(featureString);writeAttachResult(getNamespace(featureMap),getToken(featureMap),install(featureMap, inst));}

两种启动方式，都会通过toFeatureMap(final String featureString)方法解析启动传入的参数featureString，然后调用install(final Map<String, String> featureMap, final Instrumentation inst)方法

4）、AgentLauncher的`install()`方法

public class AgentLauncher {/*** 在当前JVM安装jvm-sandbox** @param featureMap 启动参数配置* @param inst       inst* @return 服务器IP:PORT*/private static synchronized InetSocketAddress install(final Map<String, String> featureMap,final Instrumentation inst) {final String namespace = getNamespace(featureMap);final String propertiesFilePath = getPropertiesFilePath(featureMap);final String coreFeatureString = toFeatureString(featureMap);try {final String home = getSandboxHome(featureMap);// 将Spy注入到BootstrapClassLoaderinst.appendToBootstrapClassLoaderSearch(new JarFile(new File(getSandboxSpyJarPath(home)// SANDBOX_SPY_JAR_PATH)));// 构造SandboxClassLoader,尽量减少Sandbox对现有工程的侵蚀final ClassLoader sandboxClassLoader = loadOrDefineClassLoader(namespace,getSandboxCoreJarPath(home)// SANDBOX_CORE_JAR_PATH);// CoreConfigure类定义final Class<?> classOfConfigure = sandboxClassLoader.loadClass(CLASS_OF_CORE_CONFIGURE);// 反序列化成CoreConfigure类实例final Object objectOfCoreConfigure = classOfConfigure.getMethod("toConfigure", String.class, String.class).invoke(null, coreFeatureString, propertiesFilePath);// CoreServer类定义final Class<?> classOfProxyServer = sandboxClassLoader.loadClass(CLASS_OF_PROXY_CORE_SERVER);// 获取CoreServer单例,真正被实例化的是JettyCoreServerfinal Object objectOfProxyServer = classOfProxyServer.getMethod("getInstance").invoke(null);// CoreServer.isBind()final boolean isBind = (Boolean) classOfProxyServer.getMethod("isBind").invoke(objectOfProxyServer);// 调用JettyCoreServer的bind方法开始进入启动HttpServer流程if (!isBind) {try {classOfProxyServer.getMethod("bind", classOfConfigure, Instrumentation.class).invoke(objectOfProxyServer, objectOfCoreConfigure, inst);} catch (Throwable t) {classOfProxyServer.getMethod("destroy").invoke(objectOfProxyServer);throw t;}}// 返回服务器绑定的地址return (InetSocketAddress) classOfProxyServer.getMethod("getLocal").invoke(objectOfProxyServer);} catch (Throwable cause) {throw new RuntimeException("sandbox attach failed.", cause);}}

install()方法核心流程如下：

BootstrapClassLoader加载sandbox-spy.jar
构造SandboxClassLoader
实例化sandbox-core.jar中的CoreConfigure内核启动配置类
实例化sandbox-core.jar中的ProxyCoreServer，这里真正被实例化的是JettyCoreServer
调用JettyCoreServer的bind()方法开始进入启动HttpServer流程

5）、JettyCoreServer的`bind()`方法

public class JettyCoreServer implements CoreServer {@Overridepublic synchronized void bind(final CoreConfigure cfg, final Instrumentation inst) throws IOException {this.cfg = cfg;try {initializer.initProcess(new Initializer.Processor() {@Overridepublic void process() throws Throwable {// 初始化logbackLogbackUtils.init(cfg.getNamespace(),cfg.getCfgLibPath() + File.separator + "sandbox-logback.xml");logger.info("initializing server. cfg={}", cfg);// 实例化JvmSandboxjvmSandbox = new JvmSandbox(cfg, inst);// 初始化HttpServerinitHttpServer();// 初始化ContextHandlerinitJettyContextHandler();// 启动HttpServerhttpServer.start();}});// 初始化加载所有的模块try {jvmSandbox.getCoreModuleManager().reset();} catch (Throwable cause) {logger.warn("reset occur error when initializing.", cause);}final InetSocketAddress local = getLocal();logger.info("initialized server. actual bind to {}:{}",local.getHostName(),local.getPort());} catch (Throwable cause) {// 这里会抛出到目标应用层,所以在这里留下错误信息logger.warn("initialize server failed.", cause);// 对外抛出到目标应用中throw new IOException("server bind failed.", cause);}logger.info("{} bind success.", this);}

bind()方法核心流程如下：

初始化logback
实例化JvmSandbox
初始化HttpServer和ContextHandler
启动HttpServer
初始化加载所有的模块

实例化JvmSandbox：

public class JvmSandbox {public JvmSandbox(final CoreConfigure cfg,final Instrumentation inst) {// 获取事件处理类实例EventListenerHandler.getSingleton();this.cfg = cfg;// 初始化模块管理实例this.coreModuleManager = SandboxProtector.instance.protectProxy(CoreModuleManager.class, new DefaultCoreModuleManager(cfg,inst,new DefaultCoreLoadedClassDataSource(inst, cfg.isEnableUnsafe()),new DefaultProviderManager(cfg)));// 初始化Spy类init();}

初始化HttpServer和ContextHandler：

public class JettyCoreServer implements CoreServer {private void initHttpServer() {final String serverIp = cfg.getServerIp();final int serverPort = cfg.getServerPort();// 如果IP:PORT已经被占用,则无法继续被绑定// 这里说明下为什么要这么无聊加个这个判断,让Jetty的Server.bind()抛出异常不是更好么？// 比较郁闷的是,如果这个端口的绑定是"SO_REUSEADDR"端口可重用的模式,那么这个server是能正常启动,但无法正常工作的// 所以这里必须先主动检查一次端口占用情况,当然了,这里也会存在一定的并发问题,BUT,我认为这种概率事件我可以选择暂时忽略if (isPortInUsing(serverIp, serverPort)) {throw new IllegalStateException(format("address[%s:%s] already in using, server bind failed.",serverIp,serverPort));}httpServer = new Server(new InetSocketAddress(serverIp, serverPort));QueuedThreadPool qtp = new QueuedThreadPool();// jetty线程设置为daemon,防止应用启动失败进程无法正常退出qtp.setDaemon(true);qtp.setName("sandbox-jetty-qtp-" + qtp.hashCode());httpServer.setThreadPool(qtp);}/** 初始化Jetty's ContextHandler*/private void initJettyContextHandler() {final String namespace = cfg.getNamespace();final ServletContextHandler context = new ServletContextHandler(NO_SESSIONS);final String contextPath = "/sandbox/" + namespace;context.setContextPath(contextPath);context.setClassLoader(getClass().getClassLoader());// web-socket-servletfinal String wsPathSpec = "/module/websocket/*";logger.info("initializing ws-http-handler. path={}", contextPath + wsPathSpec);//noinspection deprecationcontext.addServlet(new ServletHolder(new WebSocketAcceptorServlet(jvmSandbox.getCoreModuleManager())),wsPathSpec);// module-http-servletfinal String pathSpec = "/module/http/*";logger.info("initializing http-handler. path={}", contextPath + pathSpec);context.addServlet(new ServletHolder(new ModuleHttpServlet(cfg, jvmSandbox.getCoreModuleManager())),pathSpec);httpServer.setHandler(context);}

启动了一个Jetty服务之后，后续对模块的加载、卸载、激活、冻结等命令操作都会通过Http请求的方式进行

2、启动时加载模块

JettyCoreServer的bind(final CoreConfigure cfg, final Instrumentation inst)方法会调用jvmSandbox.getCoreModuleManager().reset()初始化加载所有的模块，实际上调用了DefaultCoreModuleManager的reset()方法

public class DefaultCoreModuleManager implements CoreModuleManager {@Overridepublic synchronized CoreModuleManager reset() throws ModuleException {logger.info("resetting all loaded modules:{}", loadedModuleBOMap.keySet());// 强制卸载所有模块unloadAll();// 加载所有模块for (final File moduleLibDir : moduleLibDirArray) {// 用户模块加载目录,加载用户模块目录下的所有模块// 对模块访问权限进行校验if (moduleLibDir.exists() && moduleLibDir.canRead()) {// 初始化模块目录加载器 传入模块lib目录和加载模式(默认加载模式就是attach)new ModuleLibLoader(moduleLibDir, cfg.getLaunchMode()).load(new InnerModuleJarLoadCallback(),new InnerModuleLoadCallback());} else {logger.warn("module-lib not access, ignore flush load this lib. path={}", moduleLibDir);}}return this;}

class ModuleLibLoader {/*** 加载Module** @param mjCb 模块文件加载回调* @param mCb  模块加载回掉*/void load(final ModuleJarLoadCallback mjCb,final ModuleJarLoader.ModuleLoadCallback mCb) {// 开始逐条加载for (final File moduleJarFile : listModuleJarFileInLib()) {try {// 模块文件加载回调mjCb.onLoad(moduleJarFile);// 模块加载new ModuleJarLoader(moduleJarFile, mode).load(mCb);} catch (Throwable cause) {logger.warn("loading module-jar occur error! module-jar={};", moduleJarFile, cause);}}}

1）、模块文件加载回调

public class DefaultCoreModuleManager implements CoreModuleManager {/*** 用户模块文件加载回调*/final private class InnerModuleJarLoadCallback implements ModuleJarLoadCallback {@Overridepublic void onLoad(File moduleJarFile) throws Throwable {providerManager.loading(moduleJarFile);}}

最终会通过模块Jar文件加载链ModuleJarLoadingChain去加载文件。目前来看实现类都是空的，没有起到什么作用

public interface ModuleJarLoadingChain {/*** 加载模块Jar文件 <br>* <p>* 1. 可以在这个实现中对目标期待加载的模块Jar文件进行解密,签名验证等操作<br>* 2. 如果判定加载失败,可以通过抛出异常的形式中断加载,sandbox将会跳过此模块Jar文件的加载<br>* 3. 整个模块文件的加载为一个链式的加载过程<br>* </p>** @param moduleJarFile 期待被加载模块Jar文件* @throws Throwable 模块文件加载异常*/void loading(File moduleJarFile) throws Throwable;}

2）、模块加载过程

class ModuleJarLoader {void load(final ModuleLoadCallback mCb) throws IOException {boolean hasModuleLoadedSuccessFlag = false;ModuleJarClassLoader moduleJarClassLoader = null;logger.info("prepare loading module-jar={};", moduleJarFile);try {// 构造模块类加载器ModuleJarClassLoadermoduleJarClassLoader = new ModuleJarClassLoader(moduleJarFile);// 将当前线程的类加载器从SandboxClassLoader设置成ModuleJarClassLoaderfinal ClassLoader preTCL = Thread.currentThread().getContextClassLoader();Thread.currentThread().setContextClassLoader(moduleJarClassLoader);try {// 加载模块hasModuleLoadedSuccessFlag = loadingModules(moduleJarClassLoader, mCb);} finally {// 将当前线程的类加载器从ModuleJarClassLoader设置成SandboxClassLoaderThread.currentThread().setContextClassLoader(preTCL);}} finally {if (!hasModuleLoadedSuccessFlag&& null != moduleJarClassLoader) {logger.warn("loading module-jar completed, but NONE module loaded, will be close ModuleJarClassLoader. module-jar={};", moduleJarFile);moduleJarClassLoader.closeIfPossible();}}}

ModuleJarLoader的load()方法核心流程如下：

构造模块类加载器ModuleJarClassLoader
将当前线程的类加载器从SandboxClassLoader设置成ModuleJarClassLoader
加载模块
将当前线程的类加载器从ModuleJarClassLoader设置成SandboxClassLoader

class ModuleJarLoader {private boolean loadingModules(final ModuleJarClassLoader moduleClassLoader,final ModuleLoadCallback mCb) {final Set<String> loadedModuleUniqueIds = new LinkedHashSet<String>();// ServiceLoader遍历META-INF/services目录下com.alibaba.jvm.sandbox.api.Module文件中的所有类,并实例化返回(SPI)final ServiceLoader<Module> moduleServiceLoader = ServiceLoader.load(Module.class, moduleClassLoader);final Iterator<Module> moduleIt = moduleServiceLoader.iterator();while (moduleIt.hasNext()) {final Module module;try {module = moduleIt.next();} catch (Throwable cause) {logger.warn("loading module instance failed: instance occur error, will be ignored. module-jar={}", moduleJarFile, cause);continue;}final Class<?> classOfModule = module.getClass();// 判断模块是否实现了@Information标记if (!classOfModule.isAnnotationPresent(Information.class)) {logger.warn("loading module instance failed: not implements @Information, will be ignored. class={};module-jar={};",classOfModule,moduleJarFile);continue;}final Information info = classOfModule.getAnnotation(Information.class);final String uniqueId = info.id();// 判断模块ID是否合法if (StringUtils.isBlank(uniqueId)) {logger.warn("loading module instance failed: @Information.id is missing, will be ignored. class={};module-jar={};",classOfModule,moduleJarFile);continue;}// 判断模块要求的启动模式和容器的启动模式是否匹配if (!ArrayUtils.contains(info.mode(), mode)) {logger.warn("loading module instance failed: launch-mode is not match module required, will be ignored. module={};launch-mode={};required-mode={};class={};module-jar={};",uniqueId,mode,StringUtils.join(info.mode(), ","),classOfModule,moduleJarFile);continue;}try {if (null != mCb) {// 模块加载回调mCb.onLoad(uniqueId, classOfModule, module, moduleJarFile, moduleClassLoader);}} catch (Throwable cause) {logger.warn("loading module instance failed: MODULE-LOADER-PROVIDER denied, will be ignored. module={};class={};module-jar={};",uniqueId,classOfModule,moduleJarFile,cause);continue;}loadedModuleUniqueIds.add(uniqueId);}logger.info("loaded module-jar completed, loaded {} module in module-jar={}, modules={}",loadedModuleUniqueIds.size(),moduleJarFile,loadedModuleUniqueIds);return !loadedModuleUniqueIds.isEmpty();}

ModuleJarLoader的loadingModules()方法核心流程如下：

ServiceLoader遍历META-INF/services目录下com.alibaba.jvm.sandbox.api.Module文件中的所有类，并实例化返回（SPI）
触发模块加载回调

以sandbox-mgr-module为例，源码中并没有META-INF/services这个目录，但是每个Module类都标注了@MetaInfServices(Module.class)注解，标注了该注解打包后会自动帮我们生成META-INF/services下的文件

调用ModuleLoadCallback的onLoad()方法触发模块加载回调，代码如下：

public class DefaultCoreModuleManager implements CoreModuleManager {/*** 用户模块加载回调*/final private class InnerModuleLoadCallback implements ModuleJarLoader.ModuleLoadCallback {@Overridepublic void onLoad(final String uniqueId,final Class moduleClass,final Module module,final File moduleJarFile,final ModuleJarClassLoader moduleClassLoader) throws Throwable {// 如果之前已经加载过了相同ID的模块,则放弃当前模块的加载if (loadedModuleBOMap.containsKey(uniqueId)) {final CoreModule existedCoreModule = get(uniqueId);logger.info("IMLCB: module already loaded, ignore load this module. expected:module={};class={};loader={}|existed:class={};loader={};",uniqueId,moduleClass, moduleClassLoader,existedCoreModule.getModule().getClass().getName(),existedCoreModule.getLoader());return;}// 需要经过ModuleLoadingChain的过滤providerManager.loading(uniqueId,moduleClass,module,moduleJarFile,moduleClassLoader);// 之前没有加载过,这里进行加载logger.info("IMLCB: found new module, prepare to load. module={};class={};loader={};",uniqueId,moduleClass,moduleClassLoader);// 这里进行真正的模块加载load(uniqueId, module, moduleJarFile, moduleClassLoader);}}

真正的模块加载调用load()方法，代码如下：

public class DefaultCoreModuleManager implements CoreModuleManager {/*** 加载并注册模块* <p>1. 如果模块已经存在则返回已经加载过的模块</p>* <p>2. 如果模块不存在,则进行常规加载</p>* <p>3. 如果模块初始化失败,则抛出异常</p>** @param uniqueId          模块ID* @param module            模块对象* @param moduleJarFile     模块所在JAR文件* @param moduleClassLoader 负责加载模块的ClassLoader* @throws ModuleException 加载模块失败*/private synchronized void load(final String uniqueId,final Module module,final File moduleJarFile,final ModuleJarClassLoader moduleClassLoader) throws ModuleException {if (loadedModuleBOMap.containsKey(uniqueId)) {logger.debug("module already loaded. module={};", uniqueId);return;}logger.info("loading module, module={};class={};module-jar={};",uniqueId,module.getClass().getName(),moduleJarFile);// 实例化模块信息final CoreModule coreModule = new CoreModule(uniqueId, moduleJarFile, moduleClassLoader, module);// 注入@Resource资源injectResourceOnLoadIfNecessary(coreModule);callAndFireModuleLifeCycle(coreModule, MODULE_LOAD);// 设置为已经加载coreModule.markLoaded(true);// 如果模块标记了加载时自动激活,则需要在加载完成之后激活模块markActiveOnLoadIfNecessary(coreModule);// 注册到模块列表中loadedModuleBOMap.put(uniqueId, coreModule);// 通知生命周期,模块加载完成callAndFireModuleLifeCycle(coreModule, MODULE_LOAD_COMPLETED);}

DefaultCoreModuleManager的load()方法核心流程如下：

实例化模块信息
注入@Resource资源
如果模块标记了加载时自动激活，则需要在加载完成之后激活模块
注册到模块列表中
通知生命周期，模块加载完成

3、ModuleHttpServlet进行Http路由

ModuleHttpServlet是在启动时，JettyCoreServer类中通过initJettyContextHandler()方法加入到Jetty上下文的

public class JettyCoreServer implements CoreServer {private void initHttpServer() {final String serverIp = cfg.getServerIp();final int serverPort = cfg.getServerPort();// 如果IP:PORT已经被占用,则无法继续被绑定// 这里说明下为什么要这么无聊加个这个判断,让Jetty的Server.bind()抛出异常不是更好么？// 比较郁闷的是,如果这个端口的绑定是"SO_REUSEADDR"端口可重用的模式,那么这个server是能正常启动,但无法正常工作的// 所以这里必须先主动检查一次端口占用情况,当然了,这里也会存在一定的并发问题,BUT,我认为这种概率事件我可以选择暂时忽略if (isPortInUsing(serverIp, serverPort)) {throw new IllegalStateException(format("address[%s:%s] already in using, server bind failed.",serverIp,serverPort));}httpServer = new Server(new InetSocketAddress(serverIp, serverPort));QueuedThreadPool qtp = new QueuedThreadPool();// jetty线程设置为daemon,防止应用启动失败进程无法正常退出qtp.setDaemon(true);qtp.setName("sandbox-jetty-qtp-" + qtp.hashCode());httpServer.setThreadPool(qtp);}/** 初始化Jetty's ContextHandler*/private void initJettyContextHandler() {final String namespace = cfg.getNamespace();final ServletContextHandler context = new ServletContextHandler(NO_SESSIONS);final String contextPath = "/sandbox/" + namespace;context.setContextPath(contextPath);context.setClassLoader(getClass().getClassLoader());// web-socket-servletfinal String wsPathSpec = "/module/websocket/*";logger.info("initializing ws-http-handler. path={}", contextPath + wsPathSpec);//noinspection deprecationcontext.addServlet(new ServletHolder(new WebSocketAcceptorServlet(jvmSandbox.getCoreModuleManager())),wsPathSpec);// module-http-servletfinal String pathSpec = "/module/http/*";logger.info("initializing http-handler. path={}", contextPath + pathSpec);context.addServlet(new ServletHolder(new ModuleHttpServlet(cfg, jvmSandbox.getCoreModuleManager())),pathSpec);httpServer.setHandler(context);}

ModuleHttpServlet中所有的Http请求都会进入doMethod()方法：

public class ModuleHttpServlet extends HttpServlet {   @Overrideprotected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {resp.setCharacterEncoding(cfg.getServerCharset().name());doMethod(req, resp, Http.Method.GET);}@Overrideprotected void doPost(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {resp.setCharacterEncoding(cfg.getServerCharset().name());doMethod(req, resp, Http.Method.POST);}private void doMethod(final HttpServletRequest req,final HttpServletResponse resp,final Http.Method expectHttpMethod) throws ServletException, IOException {// 获取请求路径final String path = req.getPathInfo();// 解析请求路径,获取模块IDfinal String uniqueId = parseUniqueId(path);if (StringUtils.isBlank(uniqueId)) {logger.warn("path={} is not matched any module.", path);resp.sendError(HttpServletResponse.SC_NOT_FOUND);return;}// 根据模块ID获取模块final CoreModule coreModule = coreModuleManager.get(uniqueId);if (null == coreModule) {logger.warn("path={} is matched module {}, but not existed.", path, uniqueId);resp.sendError(HttpServletResponse.SC_NOT_FOUND);return;}// 查找标注@Command的方法,匹配对应的方法final Method method = matchingModuleMethod(path,expectHttpMethod,uniqueId,coreModule.getModule().getClass());if (null == method) {logger.warn("path={} is not matched any method in module {}",path,uniqueId);resp.sendError(HttpServletResponse.SC_NOT_FOUND);return;} else {logger.debug("path={} is matched method {} in module {}", path, method.getName(), uniqueId);}// 自动释放I/O资源final List<Closeable> autoCloseResources = coreModule.append(new ReleaseResource<List<Closeable>>(new ArrayList<Closeable>()) {@Overridepublic void release() {final List<Closeable> closeables = get();if (CollectionUtils.isEmpty(closeables)) {return;}for (final Closeable closeable : get()) {if (closeable instanceof Flushable) {try {((Flushable) closeable).flush();} catch (Exception cause) {logger.warn("path={} flush I/O occur error!", path, cause);}}IOUtils.closeQuietly(closeable);}}});// 生成方法调用参数final Object[] parameterObjectArray = generateParameterObjectArray(autoCloseResources, method, req, resp);final boolean isAccessible = method.isAccessible();// 使用ModuleJarClassLoader invoke目标模块的方法final ClassLoader oriThreadContextClassLoader = Thread.currentThread().getContextClassLoader();try {method.setAccessible(true);Thread.currentThread().setContextClassLoader(coreModule.getLoader());method.invoke(coreModule.getModule(), parameterObjectArray);logger.debug("path={} invoke module {} method {} success.", path, uniqueId, method.getName());} catch (IllegalAccessException iae) {logger.warn("path={} invoke module {} method {} occur access denied.", path, uniqueId, method.getName(), iae);throw new ServletException(iae);} catch (InvocationTargetException ite) {logger.warn("path={} invoke module {} method {} occur error.", path, uniqueId, method.getName(), ite.getTargetException());final Throwable targetCause = ite.getTargetException();if (targetCause instanceof ServletException) {throw (ServletException) targetCause;}if (targetCause instanceof IOException) {throw (IOException) targetCause;}throw new ServletException(targetCause);} finally {Thread.currentThread().setContextClassLoader(oriThreadContextClassLoader);method.setAccessible(isAccessible);coreModule.release(autoCloseResources);}}

ModuleHttpServlet的doMethod()方法核心流程如下：

获取请求路径
解析请求路径，获取模块ID
根据模块ID获取模块
查找标注@Command的方法，匹配对应的方法
生成方法调用参数
使用ModuleJarClassLoader invoke目标模块的方法