Testing and Quality Pipelines: JUnit, Coverage, SonarQube, Selenium, JMeter, and Quality Gates: Diagnostics, Failure Modes, Security, and Performance

1. Evidence-first diagnostic sequence

1. Preserve job full name, build number/URL/cause, queue ID and source SHA.
2. Record controller core/Java and relevant plugin versions before upgrade/restart.
3. Confirm Jenkinsfile/shared-library revision and parameter values.
4. Confirm agent, label, executor, workspace and tool/browser/JMeter versions.
5. Record exact command and first exit code.
6. List raw report paths/sizes before cleanup.
7. Inspect publisher/ingestion result separately.
8. Inspect threshold/gate configuration and external task/status separately.
9. Confirm browser/load target environment and authorization.
10. Apply the narrowest fix; rerun only the smallest safe scope.

2. Failure map

3. Broken example 1: ignored test exit code

# BROKEN: the shell is green even when pytest fails.
python -m pytest -q --junitxml=reports/junit.xml || true
echo "quality checks passed"

The report may still show failed tests, but any downstream logic relying only on shell status sees success. Repair by capturing the real status, publishing evidence, then explicitly applying policy.

set +e
python -m pytest -q --junitxml=reports/junit.xml
rc=$?
set -e
printf '%s\n' "$rc" > reports/test-exit-code.txt
test -s reports/junit.xml
exit "$rc"

4. Broken example 2: wrong JUnit path

Runner output:
  generated: build/test-results/junit.xml

Jenkins configuration:
  junit testResults: 'reports/junit.xml', allowEmptyResults: false

Jenkins evidence:
  ERROR: No test report files were found. Configuration error?

Do not switch allowEmptyResults to true. First inspect the workspace, compare producer path with consumer glob, and fix the contract.

5. Broken example 3: coverage threshold versus report failure

Two messages may look similar but require different fixes:

6. Broken example 4: gate wait holds or times out

Preserve the Sonar analysis task ID. A timeout can come from missing/misrouted webhook, wrong Jenkins public URL, blocked network path, webhook secret mismatch or external task failure. It is not automatically a “quality gate failure.”

stage('Quality Gate') {
  agent none
  steps {
    timeout(time: 15, unit: 'MINUTES') {
      waitForQualityGate abortPipeline: true
    }
  }
}

If the gate itself is FAILED/ERROR, inspect gate conditions. If Jenkins never receives completion, inspect integration/webhook state. Do not “fix” a webhook incident by increasing the timeout indefinitely.

7. Broken example 5: waiting while occupying an executor

// Avoid a broad top-level agent solely to wait on an external service.
pipeline {
  agent any
  stages {
    stage('Analysis') { steps { /* ... */ } }
    stage('Quality Gate') {
      steps { timeout(time: 30, unit: 'MINUTES') { waitForQualityGate abortPipeline: true } }
    }
  }
}

A stage-level agent none under a Pipeline designed with agent none at the top prevents a scarce quality agent from sitting idle during an asynchronous gate.

8. Broken example 6: browser/load test points at production

# BROKEN for a course lab
TARGET_URL="https://production.example.invalid"
jmeter -n -t ci/load.jmx -JbaseUrl="$TARGET_URL" -l reports/jmeter.jtl

Repair by enforcing an allowlist or exact disposable host identity before any traffic is sent.

set -euo pipefail
TARGET_HOST="${TARGET_HOST:-127.0.0.1}"
case "$TARGET_HOST" in
  127.0.0.1|localhost) ;;
  *) echo "refusing uncontrolled load target: $TARGET_HOST" >&2; exit 64;;
esac
jmeter -n -t ci/local-smoke.jmx -Jlab.host="$TARGET_HOST" -Jlab.port=8765 -l reports/jmeter.jtl

9. Broken example 7: retry hides deterministic failure

// BROKEN as a default quality policy
retry(3) {
  sh 'python -m pytest -q --junitxml=reports/junit.xml'
}

On a deterministic assertion failure, every retry wastes time and may obscure the first result. If a specific browser startup transient is known, retry only that narrow setup operation and keep attempt-numbered logs.

10. Security-sensitive evidence

• Sonar tokens, webhook secrets and SCM check credentials belong in scoped credential stores, not command-line echoes or archived reports.
• Browser screenshots can contain user data or internal URLs; sanitize synthetic labs and review production artifacts before broad publication.
• JMeter JMX files can contain credentials or endpoint secrets; parameterize them through safe bindings and never commit real values.
• XML/HTML report parsers are attack surfaces; keep plugins current and do not install deprecated publishers merely because an old tutorial uses them.
• Untrusted fork code must not inherit privileged internal quality agents or tokens.

11. Performance diagnosis: measure the quality pipeline itself

Change one hypothesis at a time. More executors, more parallel browser sessions and larger heap are not universal fixes.

12. Worked incident: “quality stage failed”

Build #142 failed. The first evidence shows:

• pytest exit 0
• JUnit XML exists and Jenkins parsed 42/42 passing tests
• Coverage XML parsed at 84%
• Coverage gate threshold is 80%
• Sonar analysis task AX-lab-993 completed successfully
• waitForQualityGate timed out; Sonar server UI shows gate OK

The causal layer is not tests or coverage or Sonar quality policy. Investigate webhook delivery/context. Preserve task ID, Jenkins URL and webhook event before changing the timeout or rerunning analysis.

13. Repair checklist

1. State the failing layer in one sentence.
2. Name the exact evidence proving that classification.
3. Apply the smallest reversible correction.
4. Do not delete or overwrite first-failure reports.
5. Rerun only the scope required to validate the correction.
6. Compare before/after source/build/tool/environment identity.
7. If policy changed, review it as configuration—not as a troubleshooting hack.

Knowledge check

Answer before revealing the explanation.

Official references and version notes

Test-report schemas, Jenkins plugins, SonarQube integration and browser/load-test tooling evolve. Prefer current primary documentation.

• Jenkins LTS changelog
• Jenkins Java Support Policy
• Jenkins Security Advisories
• JUnit plugin
• Coverage plugin
• SonarQube Scanner plugin
• SonarQube Server 2026.1 — Jenkins analysis integration
• SonarQube Server 2026.1 — waitForQualityGate / pipeline pause
• Selenium documentation
• Apache JMeter User Manual
• Jenkins Performance plugin
• pytest
• coverage.py