Checkpoint Lab — Coverage, Test Execution Data, Duplication, and External Analyzer Reports

1. Checkpoint scenario

Build one clean local evidence chain: create a tiny Python project, run real tests, generate coverage and execution reports, deliberately point the scanner at the wrong coverage path, preserve the failure, correct only that path, and prove the corrected persisted measure. Also capture duplication and test-execution state and document how external issues would be added without confusing them with native rules.

The checkpoint succeeds only if the evidence shows why the measure changed. A green number without provenance is insufficient.

2. Exact assumptions and resource preflight

curl -fsS "$SONAR_HOST_URL/api/server/version"
sonar-scanner --version
python --version
python -m pytest --version
python -m coverage --version
git rev-parse --is-shallow-repository
git rev-parse HEAD

3. Predictions before touching the import path

At least the first four predictions must be written before the broken analysis.

4. Execute the wrong-path → corrected-path experiment

1. Create and commit the Lesson 2 fixture.
2. Run pytest/pytest-cov and preserve both XML files plus producer logs/hashes.
3. Run scanner with -Dsonar.python.coverage.reportPaths=reports/does-not-exist.xml.
4. Preserve the broken scanner log and report-task.txt.
5. Wait for/inspect the broken CE task; capture measures.
6. Verify the source SHA and report hashes did not change.
7. Run scanner using the correct project property reports/coverage.xml.
8. Preserve corrected scanner log, task ID, CE task, and measures.
9. Compare producer coverage percentage/line data with Sonar coverage, lines_to_cover, and uncovered_lines.
10. Inspect imported tests, failures/errors/skips/time and duplication metrics without claiming they came from the coverage report.

5. Verification checklist

• Broken and corrected analyses use the exact same Git revision.
• coverage.xml and junit.xml hashes are identical across both scans.
• Broken scanner log records the unresolved/missing coverage report path.
• Correct scanner log records successful Python coverage report processing.
• Both scans have preserved, distinct ceTaskId values.
• Correct CE task reaches SUCCESS before interpreting measures.
• Coverage changes without adding a coverage exclusion or source exclusion.
• No gate/profile/New Code definition changes were used to make the result look better.
• Test-execution measures are traced to JUnit/xUnit XML, not Cobertura XML.
• Duplication measures are traced to indexed source/CPD, not an external report.

6. Required evidence packet

evidence/
  assumptions.md
  server-version.txt
  scanner-version.txt
  python-tool-versions.txt
  revision.txt
  producer-command.txt
  pytest-output.txt
  coverage.xml
  junit.xml
  report-sha256.txt
  sonar-project.properties.snapshot
  broken-scan.log
  broken-report-task.txt
  broken-ce-task.json
  broken-measures.json
  correct-scan.log
  correct-report-task.txt
  correct-ce-task.json
  correct-measures.json
  import-delta.md
  duplication-notes.md
  external-issue-boundary.md
  credential-record.md
  cleanup.txt

Do not copy actual token values, cookies, unrelated environment dumps, proprietary source, or production reports into the packet.

7. Write the causal conclusion

The test producer generated Cobertura XML and xUnit XML successfully for revision X. The first SonarScanner run used a non-existent coverage path; its sensor log preserved the import failure while the source revision and report bytes remained unchanged. The second run changed only the coverage import path to the scanner-visible report and produced a new CE task whose persisted coverage matched the producer evidence. No source, coverage exclusion, gate, profile, or project-key change was used to alter the result. Test execution remained a separate imported evidence family, and duplication remained Sonar-computed CPD evidence.

That is stronger than “coverage is fixed.” It identifies the owning layer and controlled variable.

8. External analyzer boundary note

Document one of these:

• Not exercised: no external analyzer was needed; SARIF/generic paths were reviewed only.
• Optional SARIF exercised in separate project: preserve SARIF 2.1.0 bytes, producer/rule IDs, scanner import log, imported issue, and confirm it does not become a native profile rule.

Commercial features are not required. If your organization later uses paid analyzers/CI/platform integrations, reproduce the same provenance contract rather than substituting screenshots for artifacts.

9. Guarded cleanup and rollback

1. Copy .scannerwork/report-task.txt and all producer/scanner evidence before deletion.
2. Revoke the disposable project-analysis token and optional Browse token.
3. Delete the optional SARIF project only after its evidence is preserved.
4. Delete academy-sq-ch14 from SonarQube only if its exact key is confirmed to be the disposable lab key.
5. Remove the local fixture directories only after verifying the evidence packet exists elsewhere.
6. Do not use bulk project deletion, database edits, search-index deletion, or global cleanup commands.

10. What this adds to a governed production model

Chapter 14 adds evidence provenance to the operating model. Coverage and external findings are trustworthy only when the producer, version, report format, report bytes, path mapping, source revision, scanner import, asynchronous task, and resulting project state can be reconstructed.

This prepares Chapter 15, where security findings and review workflows require even stricter distinctions between analyzer evidence, vulnerability claims, Security Hotspots, taint/dataflow findings, human review, and accepted residual risk.

Knowledge check

Official references and version notes

• Test coverage overview — SonarQube consumes coverage reports generated by external build/test tools; it does not execute tests.
• Test coverage parameters — language-specific and generic coverage import keys including sonar.python.coverage.reportPaths and sonar.coverageReportPaths.
• Test execution parameters — execution-report keys including sonar.testExecutionReportPaths and sonar.python.xunit.reportPath; execution reports are branch-only.
• Generic test data — generic coverage and execution XML formats when a producer has no native integration.
• About external issues — supported external-analyzer integrations and how imported results participate in analysis.
• External analyzer reports — analyzer-specific report paths including Python Pylint/Bandit/Flake8/Mypy/Ruff integrations.
• Generic external-issue format — sonar.externalIssuesReportPaths and ownership semantics for third-party rules.
• SARIF reports — SARIF 2.1.0 requirements and sonar.sarifReportPaths.
• Scanner-only parameters — external report paths, report-task.txt, quality-gate wait behavior, and path resolution.
• Metric definitions — coverage/test metrics and current duplication/CPD thresholds and keys.
• SonarQube downloads — current Community Build release identity.
• SonarScanner CLI 8.1.0.6389 — standalone scanner baseline used in the local lab.