CI/CD Integration with GitHub Actions, GitLab CI, and Jenkins: Diagnostics, Failure Modes, and Production Practices

1. Diagnostic sequence

1. Preserve first-failure artifacts. Copy or retain console output, output.xml, log/report/xUnit, runtime manifest, and job identity before rerunning.
2. Confirm versions. Python, Robot, external libraries, Pabot if used, relevant action/plugin/runner versions.
3. Confirm execution contract. Commit/SHA, working directory, exact command, selected tests/tags/profile, environment name, synthetic input IDs.
4. Validate parse/import graph. Resource/library paths, ${CURDIR}/${EXECDIR}, package installation, current directory.
5. Inspect variable and secret boundaries. Names/presence, not values; check accidental provider-to-Robot mapping.
6. Inspect external state. Target, fixture ownership, cleanup, correlation IDs.
7. Inspect timing/concurrency/container state only if relevant. Pabot workers, CI matrix jobs, ports, shared accounts, service DNS, runner capacity.
8. Apply the least destructive correction. Rerun the smallest controlled slice, keeping the original evidence immutable.

2. Common failure modes and what they actually mean

3. Intentionally broken example: false green + missing evidence

# DO NOT COPY — intentionally broken GitHub Actions fragment
- name: Run Robot
  run: python tools/ci_run.py || true

- name: Upload results
  uses: actions/upload-artifact@v7
  with:
    name: robot-results
    path: artifacts/robot/

There are two independent defects. First, || true discards the Robot failure code, so the delivery gate can become green. Second, the upload step has no if: always(); because the first defect masks failure it happens to run, but if installation or another earlier step fails, the artifact step can still be skipped. The apparently “working” pipeline is therefore doubly misleading.

Repair

- name: Run Robot
  run: python tools/ci_run.py

- name: Upload results
  if: always()
  uses: actions/upload-artifact@v7
  with:
    name: robot-results-${{ github.run_id }}
    path: artifacts/robot/
    if-no-files-found: warn

The Robot step can fail naturally. GitHub still evaluates the final artifact step because of always(). The job remains failed; evidence remains available. If no files exist because failure occurred before Robot started, the upload warning does not replace the original cause.

4. Working-directory and import mismatches

CI often starts from repository root, but a job-level working-directory, Jenkins dir(...), container workdir, or wrapper cwd can change path resolution. Do not “fix” this with a global PYTHONPATH export until you know which import is intended to be package-based and which resource path should be relative to the importing file.

# Minimal evidence, safe to print
python -c "import os,sys; print('cwd=', os.getcwd()); print('python=', sys.executable)"
python -m pip freeze
python -m robot --version

# Then run the smallest suite/file that reproduces the import problem.
python -m robot --dryrun --outputdir artifacts/dryrun suites/ci_contract.robot

5. Secret failures: masking is not a data-flow policy

Provider secret stores reduce accidental exposure, but they cannot guarantee safety after a secret is transformed, concatenated, written into a test fixture, included in an HTTP body, or returned by a library. Keep credentials out of command-line arguments where process listings or logs may expose them; inject through approved environment/credential mechanisms and pass them only to the component that needs them.

For diagnostic evidence, log credential_present=true, credential alias/name, or a non-secret correlation identifier—not the token. Use fake values for this chapter's labs.

6. CI retry is not a flaky-test strategy

A provider-level “retry job” button can be useful after infrastructure incidents, but an automatic retry can hide nondeterminism by replacing the first failure with a later pass. Preserve the first run's Robot output and job identity. If retry is part of an approved design, make it bounded and report both attempts.

Likewise, Robot's Wait Until Keyword Succeeds and rerun/merge workflows solve specific problems; they should not be added globally to make a gate greener.

7. Performance diagnosis by layer

8. Production practices

• Pin Robot and external test dependencies in a reviewed manifest or lock file.
• Record runtime versions and the exact repository commit with every run.
• Keep target allowlists and environment promotion explicit; never use production as a troubleshooting sandbox.
• Name artifacts with immutable run/job/shard identity so later stages cannot overwrite first-failure evidence.
• Use least-privilege provider permissions and credentials; fork/untrusted contribution workflows need stricter secret and checkout boundaries.
• Make cleanup idempotent and scoped to resources owned by the current test run.
• Measure before adding caches, Pabot, matrices, containers, or retries.

Knowledge check

Summary and bridge

A reliable CI investigation begins with the failed run, not with a rerun. Preserve evidence, prove versions and command identity, isolate provider/workspace/dependency/SUT/concurrency layers, and repair the smallest broken contract. Lesson 5 turns those practices into a checkpoint where a forced failure must close the gate and still leave a complete evidence packet.

Current primary references

• Robot Framework 7.4.2 User Guide — execution, return codes, output files, xUnit, selection and result semantics.
• Robot Framework releases — current stable/prerelease status.
• Pabot 5.2.2 and Pabot documentation — optional parallel execution.
• actions/checkout, actions/setup-python, and actions/upload-artifact — current GitHub Actions examples.
• GitLab CI/CD YAML reference and unit test reports.
• Jenkins recording tests and artifacts, Pipeline syntax, and the JUnit step.