问题描述

线上右键告警，报错 OutOfMemoryError 拉取出日志，发现同样的问题：

2018-11-13 10:23:59,151 [] ERROR filter.ExceptionFilter - [DUBBO] Got unchecked and undeclared exception which called by 10.122.2.132. service: com.yuantu.virtual.service.UserAccountDubboService, method: getUserByCard, exception: java.lang.OutOfMemoryError: PermGen space, dubbo version: 2.5.3, current host: 10.122.2.109
java.lang.OutOfMemoryError: PermGen space
at sun.misc.Unsafe.defineClass(Native Method)
at sun.reflect.ClassDefiner.defineClass(ClassDefiner.java:63)
at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:399)
at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:396)
at java.security.AccessController.doPrivileged(Native Method)
at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:395)
at sun.reflect.MethodAccessorGenerator.generateConstructor(MethodAccessorGenerator.java:94)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:48)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at java.lang.reflect.Proxy.newInstance(Proxy.java:764)
at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:755)
at sun.reflect.annotation.AnnotationParser.annotationForMap(AnnotationParser.java:300)
at sun.reflect.annotation.AnnotationParser.parseAnnotation2(AnnotationParser.java:290)
at sun.reflect.annotation.AnnotationParser.parseAnnotation(AnnotationParser.java:223)
at sun.reflect.annotation.AnnotationParser.parseParameterAnnotations2(AnnotationParser.java:181)
at sun.reflect.annotation.AnnotationParser.parseParameterAnnotations(AnnotationParser.java:158)
at java.lang.reflect.Method.getParameterAnnotations(Method.java:772)
at org.apache.ibatis.binding.MapperMethod.getParamNameFromAnnotation(MapperMethod.java:229)
at org.apache.ibatis.binding.MapperMethod.setupMethodSignature(MapperMethod.java:221)
at org.apache.ibatis.binding.MapperMethod.(MapperMethod.java:70)
at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:39)
at com.sun.proxy.$Proxy101.getUserCardBySeqAndType(Unknown Source)
at com.yuantu.virtual.service.impl.UserCardServiceImpl.getUserCardBySeqAndType(UserCardServiceImpl.java:154)
at com.yuantu.virtual.service.impl.UserCardServiceImpl$$FastClassByCGLIB$$a6794f1f.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:698)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
at com.yuantu.common.profiler.ProfilerApsect.invoke(ProfilerApsect.java:36)
at sun.reflect.GeneratedMethodAccessor168.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

排查

检查tomcat配置：

JAVA_OPTS="-server
-Xmx5000m -Xms5000m -Xmn1000m
-Xss1024k -XX:SurvivorRatio=20 -XX:MaxTenuringThreshold=3
-XX:TargetSurvivorRatio=50 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75 -XX:+CMSClassUnloadingEnabled
-XX:+UseCMSInitiatingOccupancyOnly -XX:ParallelGCThreads=4 -XX:ConcGCThreads=4
-XX:+CMSParallelRemarkEnabled -XX:+CMSScavengeBeforeRemark
-XX:+ExplicitGCInvokesConcurrent -XX:+UseTLAB -XX:TLABSize=1024K
-Dsun.rmi.dgc.client.gcInterval=86400000 -Dsun.rmi.dgc.server.gcInterval=86400000
-Djava.awt.headless=true -Dclient.encoding.override=UTF-8
-Dfile.encoding=UTF-8 -Duser.language=zh -Duser.region=CN "

发现并没有设置自动Dump的参数

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/export/home/tomcat/domains/server2/oom.hprof

到zabbix上查看内存并没有明显波动；
分析日志，从java.lang.OutOfMemoryError: PermGen space可看出是永久代内存溢出； 检查发现并没有配置永久代大小；默认大小为64m；

解决

设置永久代大小配置:

-XX:PermSize=512m -XX:MaxPermSize=512m