Chapter 09Lesson 05~65 minutes

Repository Trust, Updates, Pinning, and Patch Workflows

Govern Linux software updates through authenticated repositories, explicit trust scope, candidate policy, controlled rollout, rollback preparation, and auditable patch evidence.

Repository trustPatchingChange governance

Learning objectives

By the end of this lesson

  • Explain what repository signatures protect and what they do not prove.
  • Identify repository definitions, signing-key scope, candidate priorities, exclusions, pins, and package holds.
  • Design a patch workflow from advisory intake through staging, canary rollout, verification, and rollback.
  • Distinguish package pinning and version locking from a complete vulnerability-management strategy.
  • Create a read-only patch-readiness report across Debian- and RPM-family systems.

1. Repository trust begins with explicit scope

Package managers use cryptographic signatures and digests to detect unauthorized modification of repository metadata or package archives. This establishes a chain from configured trust material to downloaded artifacts. It does not prove that the publisher is competent, that upstream code is safe, that a signing key was never compromised, or that the package is appropriate for your environment.

From repository trust to production rollout
flowchart TD
  K["Approved signing key and repository scope"] --> M["Authenticated metadata"]
  M --> C["Candidate versions and advisories"]
  C --> P["Policy: pins, excludes, holds, maintenance rules"]
  P --> S["Staging and transaction simulation"]
  S --> B["Backup, snapshot, and rollback readiness"]
  B --> R["Canary rollout"]
  R --> V["Health, security, and compatibility verification"]
  V --> F["Fleet rollout and evidence"]

Trust keys should be bound to the repository that requires them, not treated as universal authorization for every source. On APT systems, modern source definitions can use Signed-By. RPM repository definitions commonly specify gpgcheck, key locations, and sometimes repository-metadata checks.

2. Inspect repository and key configuration

Before adding a repository, verify its official documentation through an independent channel, understand the supported distributions and suites, record the key fingerprint, and define an owner and removal process. Avoid convenience instructions that disable signature checks or pipe remote scripts directly into a privileged shell.

printf '=== APT source trust declarations ===\n'
grep -RhsE '^[[:space:]]*(deb |Types:|URIs:|Suites:|Components:|Signed-By:)' \
  /etc/apt/sources.list /etc/apt/sources.list.d 2>/dev/null || true

printf '\n=== repository keyrings ===\n'
find /etc/apt/keyrings /usr/share/keyrings -maxdepth 1 -type f \
  -printf '%p\n' 2>/dev/null | sort || true

printf '\n=== RPM repository trust declarations ===\n'
grep -RhsE '^\[|^name=|^enabled=|^gpgcheck=|^repo_gpgcheck=|^gpgkey=|^baseurl=|^metalink=|^mirrorlist=' \
  /etc/yum.repos.d 2>/dev/null || true
Never solve a signature error by disabling verification

Investigate system time, repository suite changes, expired metadata, key rotation, mirror integrity, proxy interference, and the official recovery procedure. Verification failures are safety signals.

3. Pinning, exclusions, and holds alter candidate selection

APT pinning assigns priorities to versions or repository releases. DNF-family systems support exclusions and version-lock mechanisms whose exact commands depend on version and installed plugins. Holds prevent selected package upgrades. These controls are useful for compatibility incidents, staged migrations, and temporary rollback—but they create security and support debt.

printf '=== Debian-family policy ===\n'
if command -v apt-cache >/dev/null 2>&1; then
  apt-cache policy
  apt-mark showhold 2>/dev/null || true
  find /etc/apt/preferences /etc/apt/preferences.d -maxdepth 1 -type f \
    -print -exec sed -n '1,160p' {} \; 2>/dev/null || true
fi

printf '\n=== RPM-family exclusions and locks ===\n'
grep -RhsE '^[[:space:]]*(exclude|excludepkgs|includepkgs)=' \
  /etc/dnf /etc/yum.repos.d 2>/dev/null || true

if command -v dnf5 >/dev/null 2>&1; then
  dnf5 versionlock list 2>/dev/null || true
elif command -v dnf >/dev/null 2>&1; then
  dnf versionlock list 2>/dev/null || true
fi

Every lock or hold should have a reason, owner, ticket, affected hosts, risk statement, review date, and exit condition. Otherwise temporary exceptions become invisible permanent exposure.

4. A patch is an operational change, not only a package command

01Intake and classify

Map advisories and vulnerabilities to installed packages, exploitability, exposure, business criticality, and compensating controls.

02Resolve candidates

Refresh metadata, identify exact fixed versions and repositories, and detect holds or pins that block them.

03Simulate and test

Review the complete transaction in a representative environment, including service restarts, dependency transitions, migrations, and disk requirements.

04Prepare recovery

Verify backups, snapshots, database rollback, cached packages, console access, and documented abort conditions.

05Canary and observe

Patch a limited cohort, reboot when required, verify health and security, and watch a defined observation window.

06Roll out and prove

Expand in controlled waves, preserve logs, confirm installed versions, close exceptions, and record residual risk.

Not every security update requires the same speed, and not every reboot can wait. Build service tiers and emergency procedures before the incident.

5. Collect update evidence without applying changes

Read-only or simulation commands support planning. Exact advisory commands vary by distribution and installed plugins, so make automation capability-aware and preserve raw output.

if command -v apt-get >/dev/null 2>&1; then
  printf '=== APT upgrade simulation ===\n'
  apt-get --simulate dist-upgrade
  printf '\n=== APT holds ===\n'
  apt-mark showhold
  printf '\n=== upgradable packages ===\n'
  apt list --upgradable 2>/dev/null || true
elif command -v dnf5 >/dev/null 2>&1; then
  printf '=== DNF5 updates ===\n'
  dnf5 check-upgrade || status=$?
  printf 'status=%s\n' "${status:-0}"
  dnf5 advisory list --available 2>/dev/null || true
elif command -v dnf >/dev/null 2>&1; then
  printf '=== DNF updates ===\n'
  dnf check-update || status=$?
  printf 'status=%s\n' "${status:-0}"
  dnf updateinfo list available 2>/dev/null || true
fi

After patching, verify package versions, service health, listening sockets, logs, application checks, monitoring, and whether deleted-but-open libraries require process restarts. Kernel, libc, cryptographic libraries, and service runtimes often need explicit restart or reboot decisions.

6. Rollback is a prepared capability

Package downgrades are not universally safe. Configuration migrations, database schemas, generated state, kernel interfaces, and application data may move forward irreversibly. A reliable rollback plan can involve snapshots, image replacement, database restore, cached package sets, blue/green deployment, or redeployment from a known artifact.

Host snapshot

Fast state restoration

Useful for lab and some VM workloads, but coordinate application-consistent data and external systems.

Image replacement

Immutable recovery

Redeploy a previously validated image rather than reversing many in-place package operations.

Package rollback

Limited and package-specific

Requires available older artifacts and compatibility with current configuration and data.

Service failover

Protect availability

Shift traffic to a healthy cohort while repairing or replacing the failed wave.

7. Hands-on lab: build a patch-readiness report

This lab does not refresh metadata or apply updates. It records current source policy, holds or exclusions, installed kernel, reboot hints, and a transaction preview using existing local metadata.

lab="$HOME/devops-academy/linux/chapter09/lesson05"
mkdir -p "$lab"
report="$lab/patch-readiness.txt"

{
  printf 'captured_at=%s\n' "$(date --iso-8601=seconds)"
  printf '\n=== operating system and kernel ===\n'
  cat /etc/os-release
  uname -a

  if command -v apt-get >/dev/null 2>&1; then
    printf '\n=== APT sources and policy ===\n'
    grep -RhsE '^[[:space:]]*(deb |Types:|URIs:|Suites:|Components:|Signed-By:)' \
      /etc/apt/sources.list /etc/apt/sources.list.d 2>/dev/null || true
    apt-mark showhold
    printf '\n=== APT transaction simulation ===\n'
    apt-get --simulate dist-upgrade
    printf '\n=== reboot hint ===\n'
    [ -e /var/run/reboot-required ] && cat /var/run/reboot-required || printf 'not present\n'
  elif command -v dnf5 >/dev/null 2>&1; then
    printf '\n=== DNF5 repositories ===\n'
    dnf5 repolist
    printf '\n=== available updates ===\n'
    dnf5 check-upgrade || true
    printf '\n=== version locks ===\n'
    dnf5 versionlock list 2>/dev/null || true
  elif command -v dnf >/dev/null 2>&1; then
    printf '\n=== DNF repositories ===\n'
    dnf repolist
    printf '\n=== available updates ===\n'
    dnf check-update || true
    printf '\n=== version locks ===\n'
    dnf versionlock list 2>/dev/null || true
  fi

  printf '\n=== recently installed kernel packages ===\n'
  if command -v dpkg-query >/dev/null 2>&1; then
    dpkg-query -W -f='${binary:Package}\t${Version}\n' 'linux-image*' 2>/dev/null | tail -20 || true
  elif command -v rpm >/dev/null 2>&1; then
    rpm -q kernel --last 2>/dev/null | head -20 || true
  fi
} > "$report"

less "$report"

Verification checklist

8. Common mistakes

Importing a key without binding its scope

A broadly trusted key can authorize packages from unintended sources. Bind trust to specific repository definitions.

Holding a vulnerable package indefinitely

Compatibility controls need owners, deadlines, compensating controls, and a tested exit path.

Patching the entire fleet first

Canaries and waves reduce blast radius and reveal environment-specific failures.

Calling downgrade a rollback plan

Data and configuration migrations may not reverse. Prepare recovery at the service and data layers.

9. Knowledge check

Question 1. What do repository signatures primarily protect?

Question 2. Why must holds and version locks be reviewed?

Question 3. Why use a canary rollout?

10. Summary

Trusted package operations combine cryptographic verification, narrow repository scope, candidate policy, transaction review, tested recovery, and measured rollout. Pins, exclusions, and holds are temporary controls—not substitutes for patch governance. Effective patching proves what changed, why it changed, whether services remain healthy, and how residual risk is closed.

Next chapter

Disks, Partitions, Block Devices, and lsblk

Chapter 10 will move below software packages into storage devices, partitions, filesystems, mounts, capacity, LVM, RAID, swap, and recovery evidence.

11. Further reading

  • Debian apt-secure(8), apt_preferences(5), and Debian Reference package-policy guidance.
  • DNF5 repository, advisory, and version-lock documentation for the installed release.
  • RPM signature and verification documentation, plus the distribution's security advisory portal.

Keep the academy open

Support free, practical DevOps education.

Every lesson is designed to remain readable in a browser, downloadable from GitHub, and usable without a paid learning platform. Contributions help expand and maintain the curriculum.

Ethereum / ERC-20
0x716c4Ab160C4B66F31a28AE2448BfF68fc3a2ef0 Send only Ethereum/ERC-20 compatible assets to this address.