使用mvn插件surefire 执行工程单元测试出现OOM的解决分析

遇到surefire问题，最好是去官方文档看资料，看说明，因为网上关于surefire的使用经验几乎没有的。

官方说明地址：http://maven.apache.org/plugins/maven-surefire-plugin/examples/system-properties.html

在这篇文章的最末尾，可以看到<argLine>….</argLine>的配置，于是在单元测试的工程中加了插件的配置，如下所示：

这里的配置还有一个provider配置的问题，默认情况下surefire会根据工程中junit的版本来选择provider等，具体的解释可以看：http://maven.apache.org/plugins/maven-surefire-plugin/examples/junit.html

这个网页内容如下：

Using JUnit

Configuring JUnit

To get started with JUnit, you need to add the required version of JUnit to your project:

<dependencies>[...]<dependency><groupId>junit</groupId><artifactId>junit</artifactId><version>4.11</version><scope>test</scope></dependency>[...]
</dependencies>

This is the only step that is required to get started - you can now create tests in your test source directory (e.g., src/test/java).

Different Generations of JUnit support

Surefire supports three different generations of JUnit: JUnit 3.8.x, JUnit 4.x (serial provider) and JUnit 4.7 (junit-core provider with parallel support). The provider is selected based on the JUnit version in your project and the configuration parameters (for parallel).

Upgrade Check for JUnit 4.x

As of Surefire version 2.7, the algorithm for choosing which tests to run has changed. From 2.7 and on, only valid JUnit tests are run for all versions of JUnit, where older versions of the plugin would also run invalid tests that satisfied the naming convention.

When upgrading from a Surefire version prior to 2.7, the build can be run with the flag -Dsurefire.junit4.upgradecheck. This will perform a check and notify you of any invalid tests that will not be run with this version of Surefire (and the build fails). This is only meant to be used as a tool when upgrading to check that all expected tests will be run. It is a transitional feature that will be removed in a future version of surefire.

Provider Selection

If nothing is configured, Surefire detects which JUnit version to use by the following algorithm:

if the JUnit version in the project >= 4.7 and the parallel attribute has ANY valueuse junit47 provider
if JUnit >= 4.0 is presentuse junit4 provider
elseuse junit3.8.1

Please note that the "else" part of this algorithm is also a FAQ response:

You depend on the appropriate version of JUnit being present in the project dependencies, or Surefire may choose the wrong provider. If, for instance, one of your dependencies pulls in JUnit 3.8.1 you risk that surefire chooses the 3.8.1 provider, which will not support annotations or any of the 4.x features.

Use mvn dependency:tree, POM dependency ordering and/or exclusion of transitive dependencies to fix this problem.

Manually Specifying a Provider

You can also manually force a specific provider by adding it as a dependency to Surefire itself:

<plugins>
[...]<plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-surefire-plugin</artifactId><version>2.20</version><dependencies><dependency><groupId>org.apache.maven.surefire</groupId><artifactId>surefire-junit47</artifactId><version>2.20</version></dependency></dependencies></plugin>
[...]
</plugins>

When using this technique there is no check that the proper test-frameworks are present on your project's classpath. Failing to add the proper test-frameworks will result in a build failure.

Running Tests in Parallel

From JUnit 4.7 onwards you can run your tests in parallel. To do this, you must set the parallel parameter, and may change the threadCount or useUnlimitedThreads attribute. For example:

<plugins>[...]<plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-surefire-plugin</artifactId><version>2.20</version><configuration><parallel>methods</parallel><threadCount>10</threadCount></configuration></plugin>[...]
</plugins>

If your tests specify any value for the parallel attribute and your project uses JUnit 4.7+, your request will be routed to the concurrent JUnit provider, which uses the JUnit JUnitCore test runner.

This is particularly useful for slow tests that can have high concurrency.

As of Surefire 2.7, no additional dependencies are needed to use the full set of options with parallel. As of Surefire 2.16, new thread-count attributes are introduced, namely threadCountSuites, threadCountClasses and threadCountMethods. Additionally, the new attributes parallelTestsTimeoutInSeconds and parallelTestsTimeoutForcedInSeconds are used to shut down the parallel execution after an elapsed timeout, and the attribute parallel specifies new values.

See also Fork Options and Parallel Test Execution.

Using Custom Listeners and Reporters

The junit4 and junit47 providers provide support for attaching custom RunListeners to your tests.

You can configure multiple custom listeners like this:

<dependencies>
[...]<dependency><groupId>your-junit-listener-artifact-groupid</groupId><artifactId>your-junit-listener-artifact-artifactid</artifactId><version>your-junit-listener-artifact-version</version><scope>test</scope></dependency>
[...]
</dependencies>
[...]
<plugins>
[...]<plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-surefire-plugin</artifactId><version>2.20</version><configuration><properties><property><name>listener</name><value>com.mycompany.MyResultListener,com.mycompany.MyResultListener2</value></property></properties></configuration></plugin>
[...]
</plugins>

For more information on JUnit, see the JUnit web site.

You can implement JUnit listener interface org.junit.runner.notification.RunListener in a separate test artifact your-junit-listener-artifact with scope=test, or in project test source code src/test/java. You can filter test artifacts by the parameter dependenciesToScan to load its classes in current ClassLoader of surefire-junit* providers.

Since JUnit 4.12 thread safe listener class should be annotated by org.junit.runner.notification.RunListener.ThreadSafe which avoids unnecessary locks in JUnit.

Using a Security Manager (JUnit3 only)

As long as forkCount is not 0 and you use JUnit3, you can run your tests with a Java security manager enabled. The class name of the security manager must be sent as a system property variable to the JUnit3 provider.

JUnit4 uses mechanisms internally that are not compatible with the tested security managers and thus this means of configuring a security manager with JUnit4 is not supported by Surefire.

<plugins>
[...]<plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-surefire-plugin</artifactId><version>2.20</version><configuration><systemPropertyVariables><surefire.security.manager>java.lang.SecurityManager</surefire.security.manager></systemPropertyVariables></configuration></plugin>
[...]
</plugins>

Using a Security Manager (All providers)

Alternatively you can define a policy file that allows all providers to run with Surefire and configure it using the argLine parameter and two system properties:

<plugins>
[...]<plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-surefire-plugin</artifactId><version>2.20</version><configuration><argLine>-Djava.security.manager -Djava.security.policy=${basedir}/src/test/resources/java.policy</argLine></configuration></plugin>
[...]
</plugins>

The disadvantage of this solution is that the policy changes will affect the tests too, which make the security environment less realistic.

Using JUnit Categories

JUnit 4.8 introduced the notion of Categories. You can use JUnit categories by using the groups parameter. As long as the JUnit version in the project is 4.8 or higher, the presence of the "groups" parameter will automatically make Surefire select the junit47 provider, which supports groups.

<plugins>
[...]<plugin><artifactId>maven-surefire-plugin</artifactId><version>2.11</version><configuration><groups>com.mycompany.SlowTests</groups></configuration></plugin>
[...]
</plugins>

This will execute only those tests annotated with the @Category(com.mycompany.SlowTests.class) annotation and those tests annotated with @Category(com.mycompany.SlowerTests.class) if class/interface SlowerTests is subclass of SlowTests:

    public interface SlowTests{}public interface SlowerTests extends SlowTests{}

    public class AppTest {@Test@Category(com.mycompany.SlowTests.class)public void testSlow() {System.out.println("slow");}@Test@Category(com.mycompany.SlowerTests.class)public void testSlower() {System.out.println("slower");}@Test@Category(com.cmycompany.FastTests.class)public void testSlow() {System.out.println("fast");}}

The @Category annotation can also be applied at class-level.

For more information on JUnit, see the JUnit web site.

Since version 2.18.1 and JUnit 4.12, the @Category annotation type is automatically inherited from superclasses, see @java.lang.annotation.Inherited. Make sure that test class inheritance still makes sense together with @Category annotation of the JUnit 4.12 or higher appeared in superclas

关于surefire集成TestNG的官方说明：http://maven.apache.org/surefire/maven-surefire-plugin/examples/testng.html 内容如下：

Using TestNG

Unsupported TestNG versions

- TestNG 5.14.3: Bad formatted pom.xml. - TestNG 5.14.4 and 5.14.5: TestNG is using a missing dependency (org.testng:guice:2.0). Excluding it, may break some features.

Configuring TestNG

To get started with TestNG, include the following dependency in your project (replacing the version with the one you wish to use):

<dependencies>[...]<dependency><groupId>org.testng</groupId><artifactId>testng</artifactId><version>6.9.8</version><scope>test</scope></dependency>[...]
</dependencies>

If you are using an older version of TestNG (<= 5.11), the dependency would instead look like this:

<dependencies>[...]<dependency><groupId>org.testng</groupId><artifactId>testng</artifactId><version>5.11</version><scope>test</scope><classifier>jdk15</classifier></dependency>[...]
</dependencies>

Note: if you are using JDK 1.4 Javadoc annotations for your TestNG tests, replace the classifier jdk15 with jdk14 above.

This is the only step that is required to get started - you can now create tests in your test source directory (e.g., src/test/java). As long as they are named in accordance with the defaults such as *Test.java they will be run by Surefire as TestNG tests.

If you'd like to use a different naming scheme, you can change the includes parameter, as discussed in the Inclusions and Exclusions of Tests example.

Using Suite XML Files

Another alternative is to use TestNG suite XML files. This allows flexible configuration of the tests to be run. These files are created in the normal way, and then added to the Surefire Plugin configuration:

<plugins>[...]<plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-surefire-plugin</artifactId><version>2.20</version><configuration><suiteXmlFiles><suiteXmlFile>testng.xml</suiteXmlFile></suiteXmlFiles></configuration></plugin>[...]
</plugins>

This configuration will override the includes and excludes patterns and run all tests in the suite files.

Specifying Test Parameters

Your TestNG test can accept parameters with the @Parameters annotation. You can also pass parameters from Maven into your TestNG test, by specifying them as system properties, like this:

<plugins>[...]<plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-surefire-plugin</artifactId><version>2.20</version><configuration><systemPropertyVariables><propertyName>firefox</propertyName></systemPropertyVariables></configuration></plugin>[...]
</plugins>

For more information about setting system properties in Surefire tests, see System Properties.

Using Groups

TestNG allows you to group your tests. You can then execute one or more specific groups. To do this with Surefire, use the groups parameter, for example:

<plugins>[...]<plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-surefire-plugin</artifactId><version>2.20</version><configuration><groups>functest,perftest</groups></configuration></plugin>[...]
</plugins>

Likewise, the excludedGroups parameter can be used to run all but a certain set of groups.

Running Tests in Parallel

TestNG allows you to run your tests in parallel, including JUnit tests. To do this, you must set the parallel parameter, and may change the threadCount parameter if the default of 5 is not sufficient. For example:

</plugins>[...]<plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-surefire-plugin</artifactId><version>2.20</version><configuration><parallel>methods</parallel><threadCount>10</threadCount></configuration></plugin>[...]
</plugins>

This is particularly useful for slow tests that can have high concurrency, or to quickly and roughly assess the independence and thread safety of your tests and code.

TestNG 5.10 and plugin version 2.19 or higher allows you to run methods in parallel test using data provider.

</plugins>[...]<plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-surefire-plugin</artifactId><version>2.20</version><configuration><properties><property><name>parallel</name><value>methods</value></property><property><name>dataproviderthreadcount</name><value>30</value></property></properties></configuration></plugin>[...]
</plugins>

TestNG 6.9.8 (JRE 1.7) and plugin version 2.19 or higher allows you to run suites in parallel.

</plugins>[...]<plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-surefire-plugin</artifactId><version>2.20</version><configuration><suiteXmlFiles><file>src/test/resources/testng1.xml</file><file>src/test/resources/testng2.xml</file></suiteXmlFiles><properties><property><name>suitethreadpoolsize</name><value>2</value></property></properties></configuration></plugin>[...]
</plugins>

See also Fork Options and Parallel Test Execution.

Using Custom Listeners and Reporters

TestNG provides support for attaching custom listeners, reporters, annotation transformers and method interceptors to your tests. By default, TestNG attaches a few basic listeners to generate HTML and XML reports.

Unsupported versions: - TestNG 5.14.1 and 5.14.2: Due to an internal TestNG issue, listeners and reporters are not working with TestNG. Please upgrade TestNG to version 5.14.9 or higher. Note: It may be fixed in a future surefire version.

You can configure multiple custom listeners like this:

<dependencies>
[...]<dependency><groupId>your-testng-listener-artifact-groupid</groupId><artifactId>your-testng-listener-artifact-artifactid</artifactId><version>your-testng-listener-artifact-version</version><scope>test</scope></dependency>
[...]
</dependencies>
[...]
</plugins>[...]<plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-surefire-plugin</artifactId><version>2.20</version><configuration><properties><property><name>usedefaultlisteners</name><value>false</value> <!-- disabling default listeners is optional --></property><property><name>listener</name><value>com.mycompany.MyResultListener,com.mycompany.MyAnnotationTransformer,com.mycompany.MyMethodInterceptor</value></property><property><name>reporter</name><value>listenReport.Reporter</value></property></properties></configuration></plugin>[...]
</plugins>

For more information on TestNG, see the TestNG web site.

You can implement TestNG listener interface org.testng.ITestListener in a separate test artifact your-testng-listener-artifact with scope=test, or in project test source code src/test/java. You can filter test artifacts by the parameter dependenciesToScan to load its classes in current ClassLoader of surefire-testng provider. The TestNG reporter class should implement org.testng.IReporter.

The level of verbosity

Since the plugin version 2.19 or higher the verbosity level can be configured in provider property surefire.testng.verbose. The verbosity level is between 0 and 10 where 10 is the most detailed. You can specify -1 and this will put TestNG in debug mode (no longer slicing off stack traces and all). The default level is 0.

</plugins>[...]<plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-surefire-plugin</artifactId><version>2.20</version><configuration>[...]<properties><property><name>surefire.testng.verbose</name><value>10</value></property></properties>[...]</configuration></plugin>[...]
</plugins>

Customizing TestNG Object Factory

Since the plugin version 2.19 and TestNG 5.7 or higher you can customize TestNG object factory by implementing org.testng.IObjectFactory and binding the class name to the key objectfactory in provider properties:

</plugins>[...]<plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-surefire-plugin</artifactId><version>2.20</version><configuration>[...]<properties><property><name>objectfactory</name><value>testng.objectfactory.TestNGCustomObjectFactory</value></property></properties>[...]</configuration></plugin>[...]
</plugins>

Customizing TestNG Test Runner Factory

Since the plugin version 2.19 and TestNG 5.9 or higher you can customize TestNG runner factory by implementing org.testng.ITestRunnerFactory and binding the class name to the key testrunfactory in provider properties:

</plugins>[...]<plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-surefire-plugin</artifactId><version>2.20</version><configuration>[...]<properties><property><name>testrunfactory</name><value>testng.testrunnerfactory.TestNGCustomTestRunnerFactory</value></property></properties>[...]</configuration></plugin>[...]
</plugins>

Running 'testnames' in test tag

Only tests defined in a test tag matching one of these names will be run. In this example two tests run out of 7; namely InstallTest and ATest. The test a-t3 does not match any test in suite.xml.

</plugins>[...]<plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-surefire-plugin</artifactId><version>2.20</version><configuration>[...]<suiteXmlFiles><file>src/test/resources/suite.xml</file></suiteXmlFiles><properties><property><name>testnames</name><value>a-t1,a-t3</value></property></properties>[...]</configuration></plugin>[...]
</plugins>

The test suite 'suite.xml' :

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="Component Tests" verbose="2" annotations="JDK"><test name="a-t1" preserve-order="true" ><classes><class name="server.InstallTest" /><class name="server.ATest" /></classes></test><test name="a-t2" preserve-order="true" ><classes><class name="server.SCHTest" /><class name="server.PRGTest" /><class name="server.SIBBTest" /><class name="server.EDNTest" /><class name="server.PPVTest" /></classes></test>
</suite>

Running TestNG and JUnit Tests

TestNG 6.5.1 and higher provides support to run TestNG and JUnit 4.x in current Maven project. All you need is to introduce TestNG and JUnit dependency in POM.

<dependencies>[...]<dependency><groupId>org.testng</groupId><artifactId>testng</artifactId><version>6.9.4</version><scope>test</scope></dependency><dependency><groupId>junit</groupId><artifactId>junit</artifactId><version>4.10</version><scope>test</scope></dependency>[...]
</dependencies>

You may want to run two providers, e.g. surefire-junit47 and surefire-testng, and avoid running JUnit tests within surefire-testng provider by setting property junit=false.

<plugins>[...]<plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-surefire-plugin</artifactId><version>2.20</version><configuration>[...]<properties><property><name>junit</name><value>false</value></property></properties><threadCount>1</threadCount>[...]</configuration><dependencies><dependency><groupId>org.apache.maven.surefire</groupId><artifactId>surefire-junit47</artifactId><version>2.20</version></dependency><dependency><groupId>org.apache.maven.surefire</groupId><artifactId>surefire-testng</artifactId><version>2.20</version></dependency></dependencies></plugin>[...]
</plugins>

关于FAQ介绍 http://maven.apache.org/surefire/maven-surefire-plugin/faq.html#dumpfiles

Frequently Asked Questions

What is the difference between maven-failsafe-plugin and maven-surefire-plugin?
How can I reuse my test code in other modules?
Surefire fails with the message "The forked VM terminated without properly saying goodbye".
Crashed Surefire or Failsafe plugin must indicate crashed tests
How can I run GWT tests?
How do I use properties set by other plugins in argLine?
How maven-failsafe-plugin allows me to configure the jar file or classes to use?
How to dump fatal errors and stack trace of plugin runtime if it fails?

What is the difference between maven-failsafe-plugin and maven-surefire-plugin?

maven-surefire-plugin is designed for running unit tests and if any of the tests fail then it will fail the build immediately.

maven-failsafe-plugin is designed for running integration tests, and decouples failing the build if there are test failures from actually running the tests.

[top]

How can I reuse my test code in other modules?

Visit this link for your reference, | Attaching tests

[top]

Surefire fails with the message "The forked VM terminated without properly saying goodbye".

Surefire does not support tests or any referenced libraries calling System.exit() at any time. If they do so, they are incompatible with Surefire and you should probably file an issue with the library/vendor. Alternatively the forked VM could also have crashed for a number of reasons. Look for the classical "hs_err*" files indicating VM crashes or examine the Maven log output when the tests execute. Some "extraordinary" output from crashing processes may be dumped to the console/log. If this happens on a CI environment and only after it runs for some time, there is a fair chance your test suite is leaking some kind of OS-level resource that makes things worse at every run. Regular OS-level monitoring tools may give you some indication.

[top]

Crashed Surefire or Failsafe plugin must indicate crashed tests

After a forked JVM has crashed the console of forked JVM prints Crashed tests: and lists the last test which has crashed. In the console log you can find the message The forked VM terminated without properly saying goodbye.

[top]

How can I run GWT tests?

There is a specific gwt-maven-plugin at codehaus, but if you want to run with Surefire, you need the following settings:

Use the following configuration:

<useManifestOnlyJar>false</useManifestOnlyJar>

<forkCount>1</forkCount> Try reuseForks=true and if it doesn't work, fall back to reuseForks=false

[top]

How do I use properties set by other plugins in argLine?

Maven does property replacement for

${...}

values in pom.xml before any plugin is run. So Surefire would never see the place-holders in its argLine property.

Since the Version 2.17 using an alternate syntax for these properties,

@{...}

allows late replacement of properties when the plugin is executed, so properties that have been modified by other plugins will be picked up correctly.

[top]

How maven-failsafe-plugin allows me to configure the jar file or classes to use?

By default maven-failsafe-plugin uses project artifact file in test classpath if packaging is set to "jar" in pom.xml. This can be modified and for instance set to main project classes if you use configuration parameter "classesDirectory". This would mean that you set value "${project.build.outputDirectory}" for the parameter "classesDirectory" in the configuration of plugin.

[top]

How to dump fatal errors and stack trace of plugin runtime if it fails?

By default maven-failsafe-plugin and maven-surefire-plugin dumps fatal errors in dump files and these are located in target/failsafe-reports and target/surefire-reports. Names of dump files are formatted as follows:

[date]-jvmRun[N].dump

[date]-jvmRun[N].dumpstream

[date].dumpstream Forked JVM process and plugin process communicate via std/out. If this channel is corrupted, for a whatever reason, the dump of the corrupted stream appears in *.dumpstream.

在这里甚至发现了一个forkMode的参数，原来运行mvn test也不是一定要启动另外一个fork进程的，通过forkMode可以修改这个配置，forkMode默认值是once，这个配置会启动一个新的jvm fork进程用来跑单元测试，把forkMode参数设置成never，单元测试跑的时候就会由maven的主进程来跑了，也就是说这时候再通过MAVEN_OPTS来进行设置jvm参数也是可以行得通的了。