I am working on a large multi-module project and we are using sonar (now named SonarQube) for quality monitoring. SonarQube offers a lot of interesting information without extra configuration (especially if you are already using maven).
In this project, we aim to make the build fail fast, so we have three levels of testing. In each module we have a set of unit tests (white box testing which tests the individual class with mocked out interactions). Each module also has integration tests to verify that the individual pieces fit together. These integration tests use the database, spring context, drools rules and activiti flows but still test directly (direct java calls on the spring services). At a higher level, we have integration tests which verify that our REST interface works properly and that the various modules work together properly.
Unfortunately though, our sonar instance only counted the unit test coverage and not the integration test coverage. The aim is to get information like this:
Let’s first see some snippets from the original configuration. The running of the unit tests is done using surefire (test classes named *Test) and integration tests are done using the failesafe plugin (test classes named *IT).
<plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>2.14.1</version> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-failsafe-plugin</artifactId> <version>2.16</version> </plugin> |
The REST integration tests run against a Jetty instance using the jetty plugin.
<plugin> <groupId>org.mortbay.jetty</groupId> <artifactId>jetty-maven-plugin</artifactId> <version>8.1.11.v20130520</version> <configuration> <stopKey>foo</stopKey> <stopPort>12346</stopPort> <webApp> <contextPath>/districtcenter-service</contextPath> </webApp> <connectors> <connector implementation="org.eclipse.jetty.server.nio.SelectChannelConnector"> <port>12345</port> </connector> </connectors> </configuration> <executions> <execution> <id>start-jetty</id> <phase>pre-integration-test</phase> <goals> <goal>start</goal> </goals> <configuration> <scanIntervalSeconds>0</scanIntervalSeconds> <daemon>true</daemon> </configuration> </execution> <execution> <id>stop-jetty</id> <phase>post-integration-test</phase> <goals> <goal>stop</goal> </goals> </execution> </executions> <dependencies> <dependency> <groupId>postgresql</groupId> <artifactId>postgresql</artifactId> <version>${postgresql.version}</version> </dependency> </dependencies> </plugin> |
To make sure that sonar can report integration test coverage, we need to make some changes. For starters, we need to define some properties for the sonar plugin. These need to be added in the properties section of your pom.
<sonar.language>java</sonar.language> <sonar.java.coveragePlugin>jacoco</sonar.java.coveragePlugin> <sonar.dynamicAnalysis>reuseReports</sonar.dynamicAnalysis> <sonar.jacoco.reportPath>${project.basedir}/target/jacoco.exec</sonar.jacoco.reportPath> <sonar.jacoco.itReportPath>${project.basedir}/../target/itjacoco.exec</sonar.jacoco.itReportPath> |
Here we tell sonar that this is a Java project and that jacoco should be used to determine the coverage. We also tell sonar to reuse the existing reports in the build. This speeds up your build in Jenkins. Using reuseReports prevents the “mvn sonar:sonar” from re-running the tests and just reading the results from the previous run.
The last two settings define the locations where sonar can find the coverage information which is built using jacoco.
To allow jacoco to gather the coverage information, you have to install an agent when starting the java process. A practical way to determine the commandline parameter to use this agent, you can use the jacoco maven plugin.
<build> <pluginManagement> <plugin> <groupId>org.jacoco</groupId> <artifactId>jacoco-maven-plugin</artifactId> <version>0.5.3.201107060350</version> </plugin> </pluginManagement> </build> |
This plugin can be used to set an “argLine” property which can be used when starting a JVM. As the jacoco instrumentation increases the build time a little, I put this in a profile to assure it is not automatically included. In Jenkins, “-Dcoverage” is added in the maven “Goals and options” part of the “Build” configuration.
<profiles> <profile> <id>coverage</id> <activation> <property> <name>coverage</name> </property> </activation> <build> <plugins> <plugin> <groupId>org.jacoco</groupId> <artifactId>jacoco-maven-plugin</artifactId> <configuration> <includes>my.root.package.*</includes> <destFile>${sonar.jacoco.reportPath}</destFile> <dataFile>${sonar.jacoco.reportPath}</dataFile> </configuration> <executions> <execution> <id>pre-junit-test</id> <goals> <goal>prepare-agent</goal> </goals> </execution> <execution> <id>post-test</id> <goals> <goal>report</goal> </goals> </execution> </executions> </plugin> </plugins> </build> </profile> <profile> <id>nocoverage</id> <activation> <property> <name>!coverage</name> </property> </activation> <properties> <argLine></argLine> <!-- empty default --> </properties> </profile> </profiles> |
The jacoco plugin builds the “argLine” property in the prepare-agen goal and produces reports in the target/site directory in the report goal. The includes property indicates the classes for which coverage should be reported. This needs to be changed to match your project root package.
To assure that the argLine property is also defined when no coverage is needed, I also had to add the nocoverage profile. This is automatically enabled when you do not specify -Dcoverage on your maven command line.
The test runners now need to be changed to include the jacoco agent as stored in the argLine property.
<plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>2.14.1</version> <configuration> <argLine>${argLine}</argLine> </configuration> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-failsafe-plugin</artifactId> <version>2.16</version> <configuration> <argLine>${argLine}</argLine> </configuration> </plugin> |
The only change is the configuration of the argLine. This specifies the arguments to pass to the JVM when forking to run the tests. You can also add heap configurations if needed.
Now the module specific integration tests also have their coverage reported and this is included under the unit test coverage in sonar. The integration test coverage will now also be reported (if you have that widget enabled on your dashboard), but it will always report 0.0% coverage.
The 0.0% in sonar is there because the sonar.jacoco.itReportPath property is set. However, we have not used the property to store coverage data.
In the module which does the integration tests using REST calls, the coverage data needs to be put in the itReportPath. I have redefined the coverage profile to do this. In this profile I also redefined the failsafe plugin not to include the argLine parameter. The coverage data needs to be captured in the jetty instance which runs the was, not in the test code.
<profile> <id>coverage</id> <activation> <property> <name>coverage</name> </property> </activation> <build> <plugins> <plugin> <!-- redo configuration from parent - no coverage in client --> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-failsafe-plugin</artifactId> <version>2.16</version> <configuration> <argLine>-Xmx1024m -Xms256m -XX:MaxPermSize=256m</argLine> </configuration> </plugin> <plugin> <!-- redo configuration from parent - use itReportPath --> <groupId>org.jacoco</groupId> <artifactId>jacoco-maven-plugin</artifactId> <configuration> <includes>be.vlaanderen.awv.dc.*</includes> <destFile>${sonar.jacoco.itReportPath}</destFile> <dataFile>${sonar.jacoco.itReportPath}</dataFile> </configuration> <executions> <execution> <id>pre-test</id> <goals> <goal>prepare-agent</goal> </goals> </execution> <execution> <id>post-test</id> <goals> <goal>report</goal> </goals> </execution> </executions> </plugin> </plugins> </build> </profile> |
Last but not least, we need to change the jetty configuration to capture the coverage data. Unfortunately, this seems impossible using the jetty maven plugin. This plugin normally uses the JVM in which maven is running. You can use a forked JVM using the run-forked goal, but I did not find a way to wait for the deploy to finish initializing.
You can fix this by using the cargo plugin to start jetty instead (or another servlet engine or application server).
<plugin> <groupId>org.codehaus.cargo</groupId> <artifactId>cargo-maven2-plugin</artifactId> <version>1.4.5</version> <executions> <execution> <id>start-jetty</id> <phase>pre-integration-test</phase> <goals><goal>start</goal></goals> <configuration> <configuration> <properties> <cargo.jvmargs>${argLine}</cargo.jvmargs> </properties> </configuration> </configuration> </execution> <execution> <id>stop-jetty</id> <phase>post-integration-test</phase> <goals><goal>stop</goal></goals> </execution> </executions> <configuration> <container> <containerId>jetty8x</containerId> <!--<type>embedded</type>--> <log>${basedir}\target\cargo.log</log> <output>${basedir}\target\jetty.log</output> <dependencies> <dependency> <groupId>postgresql</groupId> <artifactId>postgresql</artifactId> </dependency> </dependencies> </container> <configuration> <properties> <cargo.servlet.port>12345</cargo.servlet.port> <cargo.logging>high</cargo.logging> <cargo.jvmargs>${argLine} -Denv=test</cargo.jvmargs> </properties> </configuration> <deployables> <deployable> <pingURL>http://localhost:12345/myapp/testurl</pingURL> <pingTimeout>600000</pingTimeout> <!-- 5 min, Jenkins is really slow --> <properties> <context>myapp</context> </properties> </deployable> </deployables> </configuration> </plugin> |
It is important to set the correct context and pingURL to allow cargo to know then you war has deployed. As the REST tests are run inside a war module, we have no need to explicitly set the artifact to deploy.
This is it. The configuration as shown above combines all module specific test in the “unit test coverage” section in sonar and puts the REST tests under the “integration test coverage” heading.
If you prefer to have only real unit tests under the “unit test coverage” section and all integration tests together, then you change the configuration of the jacaco plugin.
<plugin> <groupId>org.jacoco</groupId> <artifactId>jacoco-maven-plugin</artifactId> <configuration> <includes>be.vlaanderen.awv.dc.*</includes> <destFile>${sonar.jacoco.reportPath}</destFile> <dataFile>${sonar.jacoco.reportPath}</dataFile> </configuration> <executions> <execution> <id>pre-junit-test</id> <goals> <goal>prepare-agent</goal> </goals> </execution> <execution> <id>pre-integration-test</id> <phase>package</phase> <goals> <goal>prepare-agent</goal> </goals> <configuration> <includes>be.vlaanderen.awv.dc.*</includes> <destFile>${sonar.jacoco.itReportPath}</destFile> </configuration> </execution> <execution> <id>post-test</id> <goals> <goal>report</goal> </goals> </execution> </executions> </plugin> |
The package phase is just before pre-integration-test, so this assures that the argLine property is redefined between the test and integration-test phases, making sure that the failsafe plugin uses the itReportPath argLine.
Hi,
Very useful info. Can you pls. share the final pom so that it will be easy to see everything as a whole.
I think I have shown everything in the code sections above. Are you missing something?
Thanks for posting this. Did certainly help me.
Great post, it did get me started on the configuration
I found it a bit confusing though since it’s a completely different approach: our integration tests are written next to our unit tests and we don’t require a Jetty to do the integration tests (we use Spring).
I finally finished the configh with the help of this post: http://itestfirst.wordpress.com/2013/04/25/sonar-unit-and-integration-test-coverage-with-maven/ and this post http://sourceforge.net/p/eclemma/discussion/614870/thread/00d851e8 (in case anyone else is having troubles)