Testing and Quality Pipelines: JUnit, Coverage, SonarQube, Selenium, JMeter, and Quality Gates: Concepts, Architecture, and Mental Model

1. The problem: one green stage can hide several failed quality states

A test command can exit zero while Jenkins ingests no report. A report can be parsed successfully while its threshold fails. Sonar analysis can be submitted successfully while the external quality gate later fails. Selenium can prove one browser flow while saying nothing about concurrency. JMeter can produce a valid result file while the target environment is the wrong one. Those are different states with different owners.

The correct question is not simply “did the quality stage pass?” It is: what executed, which evidence was produced, which system ingested it, which policy evaluated it, and which exact source/build/environment did that evidence describe?

2. Mental model: execution → evidence → ingestion → policy → result

flowchart TD
  A["Source SHA + test inputs"] --> B["Agent tool execution"]
  B --> C{"Tool exit code"}
  B --> D["Raw reports: JUnit / coverage / JTL / browser artifacts"]
  D --> E["Jenkins publisher or external analyzer"]
  E --> F["Parsed metrics / external analysis ID"]
  F --> G{"Threshold / quality gate"}
  G -->|pass| H["Policy allows next state"]
  G -->|fail| I["Block / unstable / fail"]
  C --> J["Execution evidence"]
  H --> K["Retained build evidence"]
  I --> K
  K --> L["PR / release / deployment decision"]
  

The arrows matter. A publisher cannot ingest a report that was never created; a policy decision is not the same event as tool execution; and downstream delivery should consume the policy outcome tied to one immutable source/build identity.

3. Quality-pipeline vocabulary

4. JUnit: report ingestion is a separate state

The Jenkins JUnit plugin consumes JUnit-style XML and builds test history. A test runner may already have failed before Jenkins sees XML. Conversely, a misconfigured glob can leave Jenkins with no report even if the tests ran.

post {
  always {
    junit testResults: 'reports/junit.xml',
          allowEmptyResults: false,
          skipPublishingChecks: true
  }
}

Publishing checks is a separate SCM integration. This chapter disables it in the mandatory local lab because Chapter 34 owns external checks/status feedback.

5. Coverage: measurement and Jenkins policy are different

coverage.py, JaCoCo or another coverage engine measures executed code. Jenkins Coverage parses the report and can evaluate quality gates. The plugin documentation explicitly separates those roles: the plugin visualizes/records reports; the coverage tool must generate them first.

recordCoverage(
  tools: [[parser: 'COBERTURA', pattern: 'reports/coverage.xml']],
  id: 'python-coverage',
  name: 'Python Coverage',
  sourceCodeRetention: 'MODIFIED',
  qualityGates: [
    [threshold: 80.0, metric: 'LINE', baseline: 'PROJECT', unstable: true]
  ]
)

A parsed 76% report and a missing report are different incidents. So are “whole project line coverage below 80%” and “tests failed before a report was written.” Preserve those distinctions.

6. SonarQube: submission success is not quality-gate success

The Jenkins SonarQube integration attaches the submitted analysis task to the Pipeline context. The external SonarQube server computes the gate asynchronously. The documented waitForQualityGate flow uses a webhook and can wait without occupying a build node when the Pipeline is structured with agent none around that stage.

stage('Analyze') {
  agent { label 'quality-lab' }
  steps {
    withSonarQubeEnv('sonarqube-lab') {
      sh './run-sonar-analysis.sh'
    }
  }
}
stage('Quality Gate') {
  agent none
  steps {
    timeout(time: 15, unit: 'MINUTES') {
      waitForQualityGate abortPipeline: true
    }
  }
}

The webhook must target the correct Jenkins endpoint, and a webhook secret can authenticate payloads. A scanner exit of zero means analysis submission succeeded; the gate can still become failed later.

7. Selenium and JMeter: external evidence producers

Selenium WebDriver drives a browser and is useful for user-visible behavior; it does not replace unit tests or prove load capacity. JMeter executes a load/performance test plan and emits sample results; it does not prove correctness of business logic. Jenkins can orchestrate both and retain/ingest their evidence.

8. State ledger: what to inspect before changing anything

9. Read-only preflight

set -euo pipefail
printf 'source=%s\n' "$(git rev-parse HEAD)"
java -version 2>&1 | head -n 1
python3 --version
python3 -m pytest --version || true
python3 -m coverage --version || true
printf 'workspace=%s\n' "$PWD"
find reports -maxdepth 1 -type f -printf '%f %s bytes\n' 2>/dev/null | sort || true

On Jenkins, also record the job/build URL and agent label. Do not begin by deleting reports or rerunning tests: first preserve the initial state that explains the incident.

10. Common misconceptions

• “JUnit runs tests.” No—the test runner runs them; the Jenkins JUnit plugin ingests XML.
• “Coverage plugin computes coverage.” No—it records/evaluates reports created by coverage tools.
• “Sonar scanner returned zero, so the gate passed.” Submission and gate completion are separate states.
• “Retrying makes a flaky test reliable.” Retrying can hide instability; preserve the first failure and diagnose the nondeterminism.
• “Selenium/JMeter should target production for realism.” Mandatory labs must use controlled disposable targets; load tests especially can be disruptive.
• “A green build is a quality certificate.” It proves only the policies and evidence actually configured for that build.

11. Micro-lab: classify five observations

For each observation, name the layer before proposing a fix:

1. pytest exits 1 and writes JUnit XML containing one failed test.
2. pytest exits 0 but Jenkins says no test report files were found.
3. Coverage XML parses and Jenkins marks the build unstable at 78% against an 80% gate.
4. Sonar analysis is submitted but the Pipeline times out waiting for a webhook.
5. JMeter writes a valid JTL, but the configured target is a production URL.

The layers are respectively test execution, report ingestion/path, metric policy, external gate/webhook, and environment/safety governance.

Knowledge check

Answer before revealing the explanation.

Official references and version notes

Test-report schemas, Jenkins plugins, SonarQube integration and browser/load-test tooling evolve. Prefer current primary documentation.

• Jenkins LTS changelog
• Jenkins Java Support Policy
• Jenkins Security Advisories
• JUnit plugin
• Coverage plugin
• SonarQube Scanner plugin
• SonarQube Server 2026.1 — Jenkins analysis integration
• SonarQube Server 2026.1 — waitForQualityGate / pipeline pause
• Selenium documentation
• Apache JMeter User Manual
• Jenkins Performance plugin
• pytest
• coverage.py