Cancellation, Timeouts, Ordering, Synchronization, and Partial Failure

1. Concurrency without stop conditions is incomplete

A parallel script that works when every task succeeds quickly is only a demo. Production concurrency must define how much work may run, how long it may run, how cancellation works, how results identify their input, and what happens when only some tasks fail.

2. Timeouts turn infinite waiting into an explicit failure

ForEach-Object -Parallel -TimeoutSeconds sets a timeout for all input processing. Current documentation says that after the timeout, running scripts are stopped and remaining inputs are ignored. The default value 0 disables this timeout. It cannot be combined with -AsJob.

try {
    1..6 | ForEach-Object -Parallel {
        Start-Sleep -Seconds $_
        [pscustomobject]@{ Item=$_; Completed=$true }
    } -ThrottleLimit 2 -TimeoutSeconds 3 -ErrorAction Stop
} catch {
    Write-Warning "Parallel operation ended: $($_.Exception.Message)"
}

3. Global timeout and per-task deadline are different

A fleet tool often needs a deadline per target, not only one global deadline. When the underlying API supports a timeout, pass it directly. If you wrap an operation, return a timeout status associated with that target rather than silently abandoning identity.

$targets = 1..5 | ForEach-Object { [pscustomobject]@{ Index=$_-1; Target="node-$_"; DelayMs=100*$_ } }
$records = $targets | ForEach-Object -Parallel {
    $sw=[Diagnostics.Stopwatch]::StartNew()
    try {
        if ($_.DelayMs -gt 350) { throw [TimeoutException]::new('simulated target deadline') }
        Start-Sleep -Milliseconds $_.DelayMs
        [pscustomobject]@{Index=$_.Index;Target=$_.Target;Success=$true;DurationMs=$sw.ElapsedMilliseconds;Error=$null}
    } catch {
        [pscustomobject]@{Index=$_.Index;Target=$_.Target;Success=$false;DurationMs=$sw.ElapsedMilliseconds;Error=$_.Exception.Message}
    }
} -ThrottleLimit 2
$records | Sort-Object Index

4. Completion order is not business order

Parallel output order reflects scheduling/completion, not necessarily input order. If order matters, attach an input sequence number before parallel work and sort after collection. Do not infer target identity from array position after the fact.

$work = 0..5 | ForEach-Object { [pscustomobject]@{Index=$_;Value=$_+1} }
$out = $work | ForEach-Object -Parallel {
    Start-Sleep -Milliseconds (60*(7-$_.Value))
    [pscustomobject]@{Index=$_.Index;Value=$_.Value;Square=$_.Value*$_.Value}
} -ThrottleLimit 3
'Completion order:'; $out
'Restored input order:'; $out | Sort-Object Index

5. Partial failure is a normal batch outcome

Do not throw away successful results because one target failed, and do not report the whole fleet successful because most targets worked. Model each attempt as a record and compute the batch summary from those records.

$result = 1..6 | ForEach-Object -Parallel {
    try {
        if ($_ -in 2,5) { throw "simulated failure for $_" }
        [pscustomobject]@{Target="node-$_";Success=$true;Result="ok-$_";Error=$null}
    } catch {
        [pscustomobject]@{Target="node-$_";Success=$false;Result=$null;Error=$_.Exception.Message}
    }
} -ThrottleLimit 3
[pscustomobject]@{
    Total=@($result).Count
    Succeeded=@($result | Where-Object Success).Count
    Failed=@($result | Where-Object { -not $_.Success }).Count
}
$result

6. Retry only a classified transient failure, and bound it

A retry is not a substitute for error handling. Retry conditions should be narrow, attempts finite, and the operation safe to repeat. Permanent validation/authorization errors should fail immediately.

function Invoke-SimulatedWorker {
    param([int]$Id,[int]$MaxAttempts=3)
    for($attempt=1;$attempt -le $MaxAttempts;$attempt++){
        try {
            if($Id -eq 3 -and $attempt -lt 2){ throw [IO.IOException]::new('simulated transient I/O') }
            return [pscustomobject]@{Id=$Id;Success=$true;Attempts=$attempt;Error=$null}
        } catch [IO.IOException] {
            if($attempt -eq $MaxAttempts){ return [pscustomobject]@{Id=$Id;Success=$false;Attempts=$attempt;Error=$_.Exception.Message} }
            Start-Sleep -Milliseconds (100*$attempt)
        }
    }
}

7. Shared mutable state creates synchronization problems

When two threads read and modify the same object concurrently, operations can interleave. “Increment a counter” is conceptually read + add + write, not automatically atomic. Prefer independent worker output and aggregate afterward. If shared mutation is unavoidable, use thread-safe primitives/collections designed for it.

# Preferred: each worker emits an independent result; parent aggregates later.
$values = 1..20 | ForEach-Object -Parallel { [pscustomobject]@{Input=$_;Value=$_*2} } -ThrottleLimit 4
$sum = ($values | Measure-Object Value -Sum).Sum
$sum

# Shared state should use concurrency-aware types only when genuinely necessary.
$seen=[Collections.Concurrent.ConcurrentDictionary[int,bool]]::new()

8. Cancellation should be cooperative when possible

Force-stopping a pipeline can leave an external operation mid-flight. Prefer APIs that accept cancellation/timeout inputs and code that checks stop conditions between bounded units of work. Always combine cancellation with idempotent side effects and cleanup in finally.

$deadline=[datetime]::UtcNow.AddSeconds(2)
foreach($step in 1..100){
    if([datetime]::UtcNow -ge $deadline){
        [pscustomobject]@{Stopped=$true;Reason='DeadlineExceeded';LastStep=$step-1}
        break
    }
    Start-Sleep -Milliseconds 50
}

9. Lab — bounded worker-pool style fleet exercise

The worker input includes an index and target. Each parallel worker returns exactly one record. The parent restores order and summarizes partial failure. Maximum concurrency is an explicit parameter.

$maxConcurrency=3
$targets = 0..8 | ForEach-Object { [pscustomobject]@{Index=$_;Target="lab-node-$($_+1)"} }
$results = $targets | ForEach-Object -Parallel {
    $sw=[Diagnostics.Stopwatch]::StartNew()
    try {
        Start-Sleep -Milliseconds (80 + 20*$_.Index)
        if($_.Index -in 3,7){ throw [IO.IOException]::new('simulated transient/unreachable target') }
        [pscustomobject]@{Index=$_.Index;Target=$_.Target;Success=$true;DurationMs=$sw.ElapsedMilliseconds;Result='healthy';Error=$null}
    } catch {
        [pscustomobject]@{Index=$_.Index;Target=$_.Target;Success=$false;DurationMs=$sw.ElapsedMilliseconds;Result=$null;Error=$_.Exception.Message}
    }
} -ThrottleLimit $maxConcurrency
$ordered=$results | Sort-Object Index
$ordered
$ordered | Group-Object Success | Select-Object Name,Count

10. Throttle is a resource budget, not a speed dial

The right throttle depends on CPU cores, memory, downstream rate limits, connection pools, remote capacity, and task shape. A waiting network workload might benefit from more concurrency than a CPU-bound workload; both can still overwhelm the downstream service if the limit is chosen carelessly.

11. Common production concurrency mistakes

12. Verification checklist

• You distinguish global timeout from per-operation deadlines.
• You preserve target identity and explicit ordering.
• You model partial success/failure per target.
• You bound retries and only retry classified transient failures.
• You avoid shared mutable state by default.
• You treat throttle as a resource budget.

13. Knowledge check

14. Summary and next bridge

Production concurrency needs bounded resources, deadlines, identity, partial-failure records, safe retry boundaries, and deliberate synchronization. The final lesson asks the more fundamental question: after all this concurrency machinery, did the workload actually become meaningfully faster or more efficient?

15. Authoritative references

• Microsoft Learn — ForEach-Object
• Microsoft Learn — about_Thread_Jobs
• Microsoft Learn — Parallel execution