Idempotency, Convergence, Retries, Checkpoints, and Recovery

1. Production automation is a state transition problem

An imperative command says what to do: “write this file” or “start this service.” Production automation needs a stronger contract: what state should exist after the run? A desired state is that postcondition expressed explicitly. Convergence means repeated executions move the system toward that state and then become quiet once it is reached.

An idempotent operation can be applied repeatedly without creating additional unintended change after the desired result already exists. This does not mean “the command always succeeds.” It means the observable end state remains correct across repetitions.

2. Use the observe → compare → change → verify loop

A production mutation should normally begin with observation. Compare current state with desired state. Change only when necessary. Then observe again and prove the postcondition. The final verification matters because a command returning without an exception does not prove the system reached the intended state.

function Ensure-TextFile {
    [CmdletBinding(SupportsShouldProcess)]
    param(
        [Parameter(Mandatory)][string]$Path,
        [Parameter(Mandatory)][string]$Content
    )

    $current = if (Test-Path -LiteralPath $Path -PathType Leaf) {
        Get-Content -LiteralPath $Path -Raw
    }
    if ($current -eq $Content) {
        return [pscustomobject]@{ Path=$Path; Changed=$false; Verified=$true }
    }

    if ($PSCmdlet.ShouldProcess($Path, 'Converge file content')) {
        $parent = Split-Path -Parent $Path
        if ($parent -and -not (Test-Path -LiteralPath $parent)) {
            New-Item -ItemType Directory -Path $parent -Force | Out-Null
        }
        Set-Content -LiteralPath $Path -Value $Content -NoNewline -Encoding utf8
    }

    $verified = (Test-Path -LiteralPath $Path) -and
        ((Get-Content -LiteralPath $Path -Raw) -eq $Content)
    [pscustomobject]@{ Path=$Path; Changed=$true; Verified=$verified }
}

3. Create-if-missing and update-if-different are different decisions

Many “idempotent” scripts hide two decisions inside one -Force command. Separate them. Missing state may require creation; existing but drifted state may require an update; already-correct state requires no mutation. That distinction gives better logs, tests, approvals, and rollback metadata.

$desired = @{ Mode='staging'; FeatureX=$true } | ConvertTo-Json
$result = Ensure-TextFile -Path ./lab/appsettings.json -Content $desired -WhatIf
$result
# Remove -WhatIf only after reviewing the target and desired content.

4. Verification is part of the command contract

Treat postcondition verification as a separate step rather than trusting the mutation command. For files, verify content or a cryptographic hash. For an API, read the resource back. For a service, query the service state. For a deployment, verify health at the target—not just the CI step status.

Verification should return structured evidence that downstream code can evaluate. Avoid “green text means success” as the only signal.

$expectedHash = (Get-FileHash -LiteralPath ./lab/appsettings.json -Algorithm SHA256).Hash
$evidence = [pscustomobject]@{
    Target = './lab/appsettings.json'
    Exists = Test-Path -LiteralPath ./lab/appsettings.json
    Sha256 = $expectedHash
    CheckedAtUtc = [datetime]::UtcNow
}
$evidence

5. Retry only failures that can plausibly heal

A transient failure may disappear without changing the request: a connection reset, temporary rate limit, or short service restart. A permanent failure needs different input, permissions, configuration, or code. Retrying “access denied” twenty times is not resilience; it delays the useful error.

Classify the failure first. Then bound attempts and total wait time. Include enough context in the final error to explain what was attempted.

6. Bounded backoff and jitter prevent retry storms

Backoff increases the delay between attempts. Jitter adds a small random component so many workers do not retry at the same instant. Keep limits explicit and make the retry predicate narrow.

function Invoke-WithRetry {
    param(
        [Parameter(Mandatory)][scriptblock]$Operation,
        [Parameter(Mandatory)][scriptblock]$IsTransient,
        [ValidateRange(1,10)][int]$MaxAttempts = 4
    )
    for ($attempt = 1; $attempt -le $MaxAttempts; $attempt++) {
        try { return & $Operation }
        catch {
            if ($attempt -ge $MaxAttempts -or -not (& $IsTransient $_)) { throw }
            $baseMs = [math]::Min(4000, 250 * [math]::Pow(2, $attempt - 1))
            $jitterMs = Get-Random -Minimum 0 -Maximum 200
            Start-Sleep -Milliseconds ([int]($baseMs + $jitterMs))
        }
    }
}

7. A checkpoint is resumable evidence, not magic rollback

A checkpoint records completed work so a later run can decide what to repeat. A checkpoint can become stale, corrupt, or inconsistent with the real system. Therefore, resume logic must revalidate important postconditions rather than blindly trusting the state file.

Write checkpoint files atomically where possible: write a temporary file in the same directory, flush/close it, then replace the destination. If multiple workers can update the same checkpoint, you also need a concurrency strategy such as a lock, single writer, or transactional external store.

function Write-Checkpoint {
    param([string]$Path, [object]$State)
    $directory = Split-Path -Parent $Path
    if ($directory) { New-Item -ItemType Directory -Path $directory -Force | Out-Null }
    $temp = "$Path.$([guid]::NewGuid().ToString('N')).tmp"
    $State | ConvertTo-Json -Depth 8 | Set-Content -LiteralPath $temp -Encoding utf8
    Move-Item -LiteralPath $temp -Destination $Path -Force
}

8. Rollback, compensation, cleanup, and restore are not synonyms

Some changes are naturally reversible; others are not. Deleting a queue message, sending an email, rotating a credential, or publishing an external release may not have a true inverse. A compensating action reduces harm or restores service without pretending history never happened.

Document the recovery boundary before the change: what snapshot or prior version exists, which actions are reversible, which require manual approval, and which merely support forward repair.

9. Lab — converge a local configuration safely and run it twice

This lab changes only a disposable directory under your current location. It creates a desired JSON document, applies it through an idempotent function, records evidence, runs again, and verifies that the second run reports no change.

$root = Join-Path $PWD 'chapter20-idempotency-lab'
Remove-Item -LiteralPath $root -Recurse -Force -ErrorAction SilentlyContinue
New-Item -ItemType Directory -Path $root | Out-Null

$path = Join-Path $root 'config/app.json'
$desired = [ordered]@{ environment='lab'; port=8080; enabled=$true } |
    ConvertTo-Json

$first  = Ensure-TextFile -Path $path -Content $desired
$second = Ensure-TextFile -Path $path -Content $desired

[pscustomobject]@{
    FirstChanged  = $first.Changed
    SecondChanged = $second.Changed
    Verified      = $second.Verified
    Sha256        = (Get-FileHash -LiteralPath $path -Algorithm SHA256).Hash
}

10. Verification checklist

Use this checklist before calling an operation production-grade.

• The current state is observed before mutation.
• No change occurs when desired state already exists.
• Postconditions are verified independently.
• Retries are bounded and limited to transient failures.
• Checkpoint data is validated before resume.
• Recovery boundaries are documented honestly.
• The lab directory can be removed safely with Remove-Item -LiteralPath ./chapter20-idempotency-lab -Recurse -Force.

11. Common production mistakes

Common failures include treating -Force as idempotency, retrying authorization failures, writing shared checkpoint files without concurrency control, assuming a successful command proves the target state, and promising rollback for operations that only support compensation or forward repair.

12. Knowledge check

13. Summary and bridge to DSC

Production-safe imperative automation already looks declarative: state is observed, compared, changed only if needed, and verified. The next lesson formalizes that model with modern Microsoft DSC and shows where PowerShell-based DSC resources still fit without pretending legacy PowerShell DSC is the current architecture.

14. Authoritative references

Microsoft Learn — Everything about ShouldProcess
Microsoft Learn — Desired State Configuration overview