Maven Testing with Surefire, Failsafe, Integration Tests, Reports, and Quality Gates: Diagnostics, Failure Modes, Security, and Performance

1. Diagnostic sequence: preserve evidence before changing flags

Testing failures tempt teams to add -DskipTests, retries, or ignore flags immediately. That destroys the evidence needed to understand the gate. Use the same production diagnostic sequence established earlier:

1. Preserve concise Maven output and existing reports.
2. Confirm wrapper, Maven, JDK, and plugin identities.
3. Inspect effective POM and active plugin executions.
4. Inspect selected test names and lifecycle phase.
5. Inspect target/surefire-reports and target/failsafe-reports.
6. Determine whether the failure is test code, plugin configuration, forked JVM, dependency/classpath, or infrastructure.
7. Apply the least destructive correction.
8. Re-run in the isolated lab repository and verify counts plus exit status.

mkdir -p evidence
./mvnw -v > evidence/toolchain.txt
./mvnw help:effective-pom -Doutput=evidence/effective-pom.xml
find target/surefire-reports target/failsafe-reports -maxdepth 1 -type f -print 2>/dev/null | sort > evidence/report-files.txt || true

2. Failure mode: an integration test runs under Surefire

Suppose a test starts a local server and is intended for the integration stage, but the class is named ServerTest. Surefire sees the *Test convention and runs it during test. Symptoms include unexpectedly slow unit jobs, failures before packaging, and no Failsafe report for that class.

Repair: rename it to the intentional Failsafe convention such as ServerIT, or define explicit includes with a documented reason. Verify the class disappears from Surefire reports and appears in Failsafe reports under verify.

3. Intentionally broken example: Failsafe executes but verify is omitted

This POM is syntactically valid but operationally misleading. It binds only failsafe:integration-test. Failsafe is specifically designed not to fail the build immediately at that goal. Without failsafe:verify, a failed integration test can be recorded without becoming the final Maven failure.

<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>dev.academy.testing</groupId>
  <artifactId>testing-lab</artifactId>
  <version>1.0.0</version>
  <properties>
    <maven.compiler.release>17</maven.compiler.release>
    <junit.version>6.1.3</junit.version>
  </properties>
  <dependencyManagement><dependencies>
    <dependency><groupId>org.junit</groupId><artifactId>junit-bom</artifactId><version>${junit.version}</version><type>pom</type><scope>import</scope></dependency>
  </dependencies></dependencyManagement>
  <dependencies>
    <dependency><groupId>org.junit.jupiter</groupId><artifactId>junit-jupiter</artifactId><scope>test</scope></dependency>
  </dependencies>
  <build><plugins>
    <plugin><artifactId>maven-compiler-plugin</artifactId><version>3.15.0</version></plugin>
    <plugin><artifactId>maven-surefire-plugin</artifactId><version>3.5.6</version></plugin>
    <plugin>
      <artifactId>maven-failsafe-plugin</artifactId><version>3.5.6</version>
      <executions><execution><id>broken-it-only</id><goals>
        <goal>integration-test</goal>
        <!-- BROKEN ON PURPOSE: verify is omitted. -->
      </goals></execution></executions>
    </plugin>
  </plugins></build>
</project>

4. Failure mode: tests are silently skipped

Look for the exact skip mechanism. -DskipTests skips execution; -DskipITs targets Failsafe integration tests; -Dmaven.test.skip=true can skip test compilation as well. A POM property may also default one of these flags to true. Always inspect the command line and effective POM before assuming a missing report means discovery failure.

5. Failure mode: reports are missing or overwritten

Report directories are generated state. A CI uploader pointed at the wrong path can report “no tests” even when Maven ran them. Custom reportsDirectory values, multiple executions sharing the same location, or cleanup between build and upload can also destroy evidence. Keep Surefire and Failsafe destinations distinct unless there is an explicit, collision-safe reason to combine them.

6. Failure mode: retries mask a persistent defect

rerunFailingTestsCount can rerun failing tests for supported providers. This may help identify nondeterminism, but it changes the evidence. A test that fails first and passes later is still a flaky signal. If the same test requires retries on most runs, retries are not a fix; investigate timing, shared state, ordering, network dependencies, or concurrency.

7. Forked JVM and environment diagnosis

Surefire/Failsafe may fork JVMs. The JDK running Maven and the JVM used for test forks are related but not conceptually identical controls. If a fork crashes, inspect Maven output plus any JVM crash logs and test process resource usage. Do not respond by broadening filesystem/network permissions or running untrusted tests with production credentials.

8. Security boundary

Tests execute project code plus test dependencies/plugins. They can read environment variables, open sockets, write files, and invoke external systems. Run untrusted branches with disposable credentials and constrained infrastructure. Never add real repository passwords, signing keys, CI tokens, or production endpoints to a test merely to make an integration stage pass.

Knowledge check

9. Summary and bridge

False-green Maven testing usually comes from a mismatch among runner selection, lifecycle reach, skip state, report collection, and failure propagation. Lesson 5 turns those failure modes into a checkpoint where you must prove test counts and deliberately reproduce the missing-verify trap.

Official references and version notes

Version-sensitive statements were checked against Apache Maven and JUnit primary documentation on 2026-08-23. The mandatory path pins Maven 3.9.16 via Maven Wrapper 3.3.4, JDK 21 to run Maven, Java 17 as the release target, JUnit 6.1.3, Maven Surefire Plugin 3.5.6, Maven Failsafe Plugin 3.5.6, Help Plugin 3.5.2, and Compiler Plugin 3.15.0.

Apache's current Surefire site advertises 3.6.0-M1. Because the version itself is a milestone identifier, these lessons treat it as version-sensitive preview/milestone material and keep the hands-on path on the final 3.5.6 line. Re-check before adopting a newer line in production.

• Maven Surefire Plugin — Introduction
• Maven Failsafe Plugin — Introduction
• Maven Failsafe — Integration Test Goal
• Maven Failsafe — Verify Goal
• Maven Surefire — Skipping Tests
• Maven Failsafe — Skipping Tests
• Maven Failsafe — Running a Single Test
• JUnit 6.1.3 User Guide — Maven support
• JUnit 6.1.3 Release Notes
• Maven Help Plugin 3.5.2
• Maven Compiler Plugin 3.15.0
• Apache Maven Wrapper
• Maven 3.9.16 Release Notes