Infrastructure Automation and Immutable-Server Patterns

1. Automation models solve different lifecycle problems

Imperative automation describes steps. Desired-state tools compare observed state with a declared target and apply changes until they converge. Image builders create a versioned machine artifact before deployment. Orchestrators place and replace instances at runtime. Mature platforms combine these models instead of forcing one tool to handle every phase.

flowchart TD
  S["Versioned source, policy, and dependencies"] --> B["Build machine or container image"]
  B --> T["Static, security, boot, and integration tests"]
  T --> P["Publish digest and provenance"]
  P --> D["Progressive deployment"]
  D --> V["Health, telemetry, and policy validation"]
  V --> R{"Healthy?"}
  R -- yes --> C["Promote and record baseline"]
  R -- no --> X["Rollback or replace"]
  C --> O["Observe drift and new requirements"]
  O --> S

“Immutable” means infrastructure is replaced from a known artifact rather than manually repaired in place. Emergency access may still exist, but any durable fix must return to source, image, tests, and deployment automation.

2. Idempotence makes retries predictable

An operation is idempotent when applying it repeatedly produces the same intended state. The mathematical property is:

# Fragile: duplicates a line on every run.
# echo 'net.ipv4.ip_forward=1' >> /etc/sysctl.conf

# Better shell pattern: manage a dedicated file atomically.
set -Eeuo pipefail
value='net.ipv4.ip_forward = 1'
target=/etc/sysctl.d/60-devops-academy.conf
tmp=$(mktemp)
trap 'rm -f "$tmp"' EXIT
printf '%s
' "$value" > "$tmp"

if ! sudo cmp -s "$tmp" "$target"; then
  sudo install -m 0644 -o root -g root "$tmp" "$target"
  sudo sysctl --system
fi

sysctl -n net.ipv4.ip_forward

Prefer tool modules that understand resource semantics over arbitrary shell commands. When shell is required, model detection, change, validation, and rollback explicitly.

3. Desired state needs ownership and conflict rules

Two automation systems managing the same file, service, firewall chain, package, or cloud resource can continuously undo each other. Assign one owner per resource and define interfaces between layers: base image, bootstrap, configuration, application deployment, and runtime policy.

# Illustrative Ansible task: declarative package and service state.
- name: Install and run chrony
  hosts: linux_servers
  become: true
  tasks:
    - name: Install package
      ansible.builtin.package:
        name: chrony
        state: present

    - name: Enable and start service
      ansible.builtin.service:
        name: chronyd
        state: started
        enabled: true

    - name: Verify synchronization state
      ansible.builtin.command: chronyc tracking
      changed_when: false
      register: chrony_tracking

Check mode and diffs are planning aids, not proof that every module or external API is side-effect free. Test the exact automation version, inventory, variables, privilege path, and target image.

4. Machine images turn provisioning into a tested build artifact

An image pipeline starts from an identified base, applies packages and configuration, removes transient credentials and caches, runs tests, records an SBOM or inventory, and publishes an immutable artifact identifier. Cloud-init or first-boot logic should remain minimal and deterministic: identity, final environment-specific configuration, and registration—not a second uncontrolled image build.

# Capture a host-image manifest before publication.
out=/tmp/image-manifest
rm -rf "$out"
mkdir -m 0700 "$out"

cp /etc/os-release "$out/os-release"
uname -a > "$out/kernel.txt"
systemctl list-unit-files --state=enabled --no-pager > "$out/enabled-units.txt"
sysctl -a 2>/dev/null | sort > "$out/sysctl.txt"
ss -lntup > "$out/listeners.txt"

if command -v dpkg-query >/dev/null; then
  dpkg-query -W -f='${binary:Package}	${Version}	${Architecture}
'     | sort > "$out/packages.tsv"
elif command -v rpm >/dev/null; then
  rpm -qa --qf '%{NAME}	%{EPOCHNUM}:%{VERSION}-%{RELEASE}	%{ARCH}
'     | sort > "$out/packages.tsv"
fi

find "$out" -type f -exec chmod 0600 {} +
sha256sum "$out"/* > "$out/SHA256SUMS"

Do not bake long-lived secrets, host keys, machine identities, cloud instance credentials, or stale registration tokens into a reusable image. Generate or retrieve them through an approved first-boot identity flow.

5. Replaceable compute requires explicit state separation

Application data, queues, databases, audit evidence, certificates, host identity, and secrets have different durability and replication requirements. An immutable instance should be disposable only after every required state path is identified and recoverable elsewhere.

Snapshots can accelerate recovery but do not replace logical backups, cross-failure-domain copies, or restore exercises. Treat every state service as a dependency with its own recovery objectives.

6. Progressive rollout bounds the blast radius

Deploy a new artifact to a small representative ring, validate infrastructure and application health, then increase exposure. Health gates should cover boot, network, identity, storage, service readiness, user transactions, errors, latency, saturation, and security controls. A rollback must be tested with current schemas and state transitions.

Example rollout rings:
  0. image boot and integration test
  1. one non-production instance
  2. one production canary with bounded traffic
  3. one availability zone or small percentage
  4. regional expansion
  5. fleet completion

Promotion evidence:
  artifact digest, configuration version, timestamps, health results,
  error/latency comparison, operator approval, rollback readiness

7. Drift is evidence that reality diverged from the declared baseline

Drift can be authorized emergency change, automation failure, manual modification, package update, compromised state, or expected runtime data. Detect it at the appropriate layer: cloud resources, image digest, packages, services, files, sysctl, firewall, users, listeners, and application configuration.

baseline="$HOME/devops-academy/linux/chapter19/lesson05/baseline"
current="$HOME/devops-academy/linux/chapter19/lesson05/current"
mkdir -p "$current"

systemctl list-unit-files --state=enabled --no-pager > "$current/enabled-units.txt"
ss -lntup > "$current/listeners.txt"
sysctl -a 2>/dev/null | sort > "$current/sysctl.txt"

if [[ -d $baseline ]]; then
  diff -ruN "$baseline" "$current" || true
else
  printf 'No approved baseline exists at %s
' "$baseline" >&2
fi

sha256sum "$current"/* > "$current/SHA256SUMS"

Do not automatically “fix” unknown drift before preserving evidence. Classify the change, determine authorization and impact, and decide whether to rebuild, converge, isolate, or investigate.

8. Bootstrap should establish identity and hand off quickly

First-boot automation runs at a sensitive moment with broad privilege and incomplete observability. Keep it short, retry-safe, and independently logged. Confirm time, network, DNS, instance identity, metadata-service controls, package repository trust, and configuration retrieval before starting application workloads.

# Minimal cloud-init illustration; provider and distribution support varies.
#cloud-config
package_update: false
write_files:
  - path: /etc/devops-academy/image-id
    permissions: '0644'
    owner: root:root
    content: 'linux-web-2026-08-05.1'
runcmd:
  - [systemctl, daemon-reload]
  - [systemctl, enable, --now, application.service]
final_message: 'Bootstrap complete after $UPTIME seconds'

# Keep secrets out of user-data when the platform exposes it broadly.
# Fetch short-lived credentials through workload identity instead.

Forward bootstrap logs and expose a clear completion signal. An instance that never completed bootstrap should not enter load balancing or receive production work.

9. Immutable replacement changes the patch workflow

Instead of patching every long-lived host in place, rebuild the base image with updated packages, run tests, and replace instances progressively. Emergency in-place patching may still be necessary, but it creates drift that must be reconciled by a subsequent image release.

Image-based patch workflow:
  advisory -> applicability -> base-image update -> rebuild -> scan/test
  -> publish digest -> canary replacement -> validation -> rollout
  -> old-image retirement -> evidence and baseline update

Emergency exception:
  authorize -> patch exposed hosts -> validate -> record drift
  -> rebuild canonical image -> replace exception hosts -> close exception

Retain enough previous artifacts and configuration to roll back, but retire images with known critical exposure according to policy. Rollback availability is not a reason to keep vulnerable images deployable indefinitely.

10. Measure convergence and replacement as fleet operations

A fleet can be “green” while many nodes run old artifacts or have never reported. Track expected inventory, last-seen time, artifact version, configuration version, policy state, health, and replacement eligibility.

Also monitor rollout duration, failed replacements, rollback rate, image age, drift count, emergency changes, bootstrap failure, capacity headroom, and recovery test age.

11. Hands-on lab: create and compare an immutable host manifest

The lab captures two snapshots of selected host state and verifies that a no-change second capture is identical after normalizing timestamps. It does not modify the system.

set -Eeuo pipefail
lab="$HOME/devops-academy/linux/chapter19/lesson05/lab"
rm -rf "$lab"
mkdir -p "$lab/a" "$lab/b"

capture() {
  local out=$1
  uname -a > "$out/kernel.txt"
  systemctl list-unit-files --state=enabled --no-pager > "$out/enabled-units.txt"
  sysctl -a 2>/dev/null | sort > "$out/sysctl.txt"
  ss -lntup > "$out/listeners.txt"
  sha256sum "$out"/*.txt > "$out/SHA256SUMS"
}

capture "$lab/a"
sleep 1
capture "$lab/b"

diff -ru "$lab/a" "$lab/b" > "$lab/diff.txt" || true
{
  printf 'captured_utc=%s
' "$(date -u --iso-8601=seconds)"
  printf 'host=%s
' "$(hostname)"
  printf 'difference_lines=%s
' "$(wc -l < "$lab/diff.txt")"
} > "$lab/MANIFEST"

printf 'Review %s; transient listeners may legitimately differ.
' "$lab"

12. Replacement runbook and failure gates

1. Resolve the approved image digest, configuration version, and target ring.
2. Verify capacity, state dependencies, backups, identity, network, and rollback artifact.
3. Create replacements without terminating the existing healthy capacity.
4. Validate boot, bootstrap, policy, observability, service readiness, and synthetic transactions.
5. Shift traffic gradually and compare errors, latency, saturation, and business metrics.
6. Drain and terminate old instances only after state and session requirements are satisfied.
7. Preserve evidence, update inventory, and retire superseded vulnerable artifacts.

13. Knowledge check

14. Summary

• Imperative, desired-state, image-build, and orchestration tools address different lifecycle stages.
• Idempotence, ownership, validation, and rollback make automation safe to retry.
• Immutable servers replace instances from tested artifacts and reconcile emergency drift back into source.
• Replaceable compute depends on explicit separation and recovery of durable state, identity, secrets, and evidence.
• Progressive rollout, health gates, fleet convergence, and drift review bound operational risk.

15. Further reading

• Ansible playbooks
• HashiCorp Packer documentation
• cloud-init documentation
• Google SRE Workbook — Canarying Releases
• NIST SP 800-204D — DevSecOps CI/CD security