Production Logging, Metrics, Events, and Operational Runbooks

1. Logs become useful when they form a contract

A production log should answer who/what/where/when/outcome questions without requiring a human to reconstruct state from colored text. Define stable fields before choosing a destination. Useful baseline fields include UTC timestamp, severity, target, action, duration, outcome, correlation ID, and a non-secret error identifier.

A correlation ID ties events from one logical execution together. Generate it once near the orchestration boundary and pass it to adapters rather than generating unrelated IDs in every helper.

2. JSON Lines keeps events both appendable and machine-readable

JSON Lines stores one JSON object per line. It is simple to append, stream, ship, grep, or ingest into a log platform. Avoid one giant JSON array that must be rewritten after every event.

function Write-JsonEvent {
    param(
        [string]$Path,
        [ValidateSet('Debug','Info','Warning','Error')][string]$Severity,
        [string]$Action,
        [string]$Target,
        [string]$Outcome,
        [string]$CorrelationId,
        [long]$DurationMs = 0,
        [string]$ErrorId
    )
    $event = [ordered]@{
        timestampUtc = [datetime]::UtcNow.ToString('o')
        severity = $Severity
        correlationId = $CorrelationId
        action = $Action
        target = $Target
        durationMs = $DurationMs
        outcome = $Outcome
        errorId = $ErrorId
    }
    Add-Content -LiteralPath $Path -Value ($event | ConvertTo-Json -Compress) -Encoding utf8
}

3. Observability must not become a secret exfiltration channel

Do not log authorization headers, bearer tokens, passwords, connection strings, private keys, raw credential objects, or entire environment-variable dictionaries. Prefer an allowlist of safe fields over a blacklist of secret-looking names.

If diagnostics need identity context, log a non-secret identifier such as credential source name, tenant ID, account alias, or secret version—not the secret value.

function Get-RedactedHeaders {
    param([hashtable]$Headers)
    $copy = [ordered]@{}
    foreach ($key in $Headers.Keys) {
        $copy[$key] = if ($key -match '^(Authorization|X-Api-Key)$') { '<redacted>' } else { $Headers[$key] }
    }
    $copy
}

4. Metrics summarize behavior across executions

A metric is a numeric observation intended for aggregation: count, duration, size, rate, or gauge. Scripts can emit simple counters and timings; external systems such as Prometheus, Application Insights, CloudWatch, or a log analytics platform can store, aggregate, visualize, and alert on them.

Do not turn every PowerShell process into its own monitoring backend. Produce stable metric data and hand it to the platform designed for retention and querying.

$metrics = [pscustomobject]@{
    StepsTotal = 4
    StepsSucceeded = 3
    StepsFailed = 1
    DurationMs = 842
    ArtifactBytes = 12840
}
$metrics | ConvertTo-Json

5. Measure each step, not just total runtime

Per-step duration distinguishes “the script is slow” from “DNS resolution took 8 seconds” or “artifact hashing dominated runtime.” A stopwatch also avoids fragile subtraction around human-formatted timestamps.

$sw = [System.Diagnostics.Stopwatch]::StartNew()
try {
    # perform one bounded operation
    Start-Sleep -Milliseconds 120
    $outcome = 'Succeeded'
} catch {
    $outcome = 'Failed'
    throw
} finally {
    $sw.Stop()
    "durationMs=$($sw.ElapsedMilliseconds) outcome=$outcome"
}

6. End with a stable execution summary object

Logs contain event detail; the top-level summary answers whether the run succeeded and where evidence lives. CI, schedulers, and human operators can consume the same object differently.

$summary = [pscustomobject]@{
    SchemaVersion = 1
    CorrelationId = $correlationId
    StartedAtUtc = $started
    FinishedAtUtc = [datetime]::UtcNow
    Success = ($failures.Count -eq 0)
    FailureCount = $failures.Count
    LogPath = $logPath
    ReportPath = $reportPath
}
$summary

7. Retention and access control are part of logging design

Logs can contain hostnames, paths, user identifiers, resource IDs, failure details, and deployment metadata even after secrets are redacted. Define who may read them, where they are stored, how long they are retained, and how sensitive evidence is deleted. Retention should meet operational and policy needs without becoming indefinite data accumulation.

8. A runbook is the operational interface to automation

A runbook tells an operator how to execute and recover a workflow safely. Minimum sections should cover purpose, prerequisites, supported platforms, inputs, secret sources, dry-run/preview, normal invocation, expected output, verification, known failures, recovery/rollback, escalation, and evidence locations.

A good runbook is tested during normal operations. A recovery procedure first read during an outage is an untested procedure.

9. Lab — build an incident-friendly multi-step report

The following lab performs harmless local observations: platform information, temporary-file creation, hashing, and cleanup planning. Each step emits structured evidence with one correlation ID.

$root = Join-Path $PWD 'chapter20-observability-lab'
New-Item -ItemType Directory -Path $root -Force | Out-Null
$logPath = Join-Path $root 'events.jsonl'
$reportPath = Join-Path $root 'summary.json'
$correlationId = [guid]::NewGuid().ToString()
$steps = [System.Collections.Generic.List[object]]::new()

foreach ($step in @('Platform','CreateArtifact','HashArtifact')) {
    $sw = [Diagnostics.Stopwatch]::StartNew()
    try {
        switch ($step) {
            'Platform' { $value = [Runtime.InteropServices.RuntimeInformation]::OSDescription }
            'CreateArtifact' { 'training' | Set-Content (Join-Path $root 'artifact.txt'); $value='created' }
            'HashArtifact' { $value=(Get-FileHash (Join-Path $root 'artifact.txt') -Algorithm SHA256).Hash }
        }
        $outcome='Succeeded'; $errorId=$null
    } catch {
        $outcome='Failed'; $errorId=$_.FullyQualifiedErrorId; $value=$null
    } finally {
        $sw.Stop()
    }
    Write-JsonEvent -Path $logPath -Severity $(if($outcome -eq 'Succeeded'){'Info'}else{'Error'}) -Action $step -Target $root -Outcome $outcome -CorrelationId $correlationId -DurationMs $sw.ElapsedMilliseconds -ErrorId $errorId
    $steps.Add([pscustomobject]@{ Step=$step; Outcome=$outcome; DurationMs=$sw.ElapsedMilliseconds; Value=$value })
}

$summary=[pscustomobject]@{ SchemaVersion=1; CorrelationId=$correlationId; Success=(-not ($steps.Outcome -contains 'Failed')); Steps=$steps; LogPath=$logPath }
$summary | ConvertTo-Json -Depth 6 | Set-Content $reportPath -Encoding utf8
$summary

10. Runbook template for the lab

Purpose: collect local diagnostic evidence. Prerequisites: PowerShell 7.x and write access to the current directory. Dry run: not necessary because the lab writes only a disposable directory. Verification: inspect events.jsonl and summary.json. Recovery: delete the lab directory. Escalation: preserve the correlation ID and error identifier when handing evidence to another team.

11. Observability review checklist

Before shipping automation, verify that logs are structured, UTC-based, correlated, secret-safe, bounded in retention, and accompanied by a stable summary plus an operator runbook.

• Every run has one correlation ID.
• Every step records target, action, duration, and outcome.
• Errors include identifiers without secret payloads.
• Machine-readable summary output exists.
• Metrics can be exported without parsing console decoration.
• Runbook covers verification and failure recovery.

12. Knowledge check

13. Summary and architecture bridge

Observability makes automation supportable, but maintainability also depends on project structure, dependency contracts, versioning, and reproducible deployment. Lesson 4 turns the course’s functions/modules/tests/configuration into a production project layout.

14. Authoritative references

Microsoft Learn — Write-Information
Microsoft Learn — Start-Transcript
Microsoft Learn — exceptions and diagnostics