Capstone: Build and Operate a Hardened DevOps Server

1. Capstone mission and safety boundary

Build and operate one small Linux DevOps server that can host an internal status application, receive controlled deployments, expose only approved network services, produce operational telemetry, and recover from common failures. Use a disposable virtual machine or cloud instance—not an irreplaceable personal or production system.

flowchart TD
  G["Versioned source and configuration"] --> B["Provision and validate host"]
  B --> H["Identity, SSH, firewall, updates, time"]
  H --> D["Deploy signed or checksummed artifact"]
  D --> S["Hardened systemd service"]
  S --> O["Logs, metrics, health, alerts"]
  O --> R["Backup and restore evidence"]
  R --> I["Failure injection and incident runbook"]
  I --> C["Acceptance report and improvement backlog"]

The capstone is complete only when another operator can reconstruct the intended state, validate it, restore service, and understand remaining risks from the repository and evidence bundle.

2. Define requirements before provisioning

Write the service objective, approved users, ports, data classes, dependencies, recovery targets, maintenance window, update owner, logging retention, and evidence location. A secure server has fewer unexplained components, not merely more hardening switches.

3. Create a versioned capstone repository

Keep desired configuration, scripts, unit files, checks, runbooks, and evidence schemas in version control. Do not commit private keys, passwords, API tokens, unencrypted backups, or production packet captures.

linux-capstone/
├── README.md
├── inventory/
│   ├── host.example.env
│   └── ports.md
├── provision/
│   ├── 10-packages.sh
│   ├── 20-accounts.sh
│   ├── 30-firewall.nft
│   └── 40-service.sh
├── service/
│   ├── academy-status.service
│   ├── academy-status.env.example
│   └── releases/
├── checks/
│   ├── validate-host.sh
│   ├── health-check.sh
│   └── collect-evidence.sh
├── backup/
│   ├── backup.sh
│   ├── restore-test.sh
│   └── retention.md
├── runbooks/
│   ├── deployment.md
│   ├── incident.md
│   ├── rollback.md
│   └── disaster-recovery.md
└── evidence/
    └── .gitkeep

Scripts should be idempotent or clearly state one-time behavior. Every privileged step must print its target, validate prerequisites, fail safely, and support review before execution.

4. Establish platform, patch, time, and inventory baselines

Use a supported distribution image from a trusted source. Update package metadata and apply the approved update policy before exposing services. Reboot when the kernel or critical libraries require it, then record the new boot and package state.

#!/usr/bin/env bash
set -Eeuo pipefail
umask 027
out=${1:-evidence/baseline}
mkdir -p "$out"

{
  date --iso-8601=seconds
  hostnamectl
  uname -a
  cat /etc/os-release
  uptime
  timedatectl
} > "$out/platform.txt"

{
  dpkg-query -W -f='${binary:Package}	${Version}
' 2>/dev/null || true
  rpm -qa --qf '%{NAME}	%{VERSION}-%{RELEASE}.%{ARCH}
' 2>/dev/null || true
} | sort > "$out/packages.tsv"

systemctl --failed --no-pager > "$out/failed-units.txt"
ss -lntup > "$out/listeners.txt"
findmnt --real > "$out/mounts.txt"
df -hT > "$out/capacity.txt"
find "$out" -type f -print0 | sort -z | xargs -0 sha256sum > "$out/SHA256SUMS"

Record the image ID or build provenance, configuration revision, package snapshot, kernel, reboot time, and enabled services. “Latest” is not reproducible evidence.

5. Build administrative and service identities around least privilege

Use named administrator accounts with individual SSH keys and sudo authorization. Run the application under a dedicated non-login service account or DynamicUser= when durable ownership requirements permit. Keep CI/deployment identities separate from runtime identities.

# Examples: review distribution conventions before applying.
sudo useradd --create-home --shell /bin/bash opsadmin
sudo useradd --system --home-dir /var/lib/academy-status   --create-home --shell /usr/sbin/nologin academy-status

sudo install -d -m 0700 -o opsadmin -g opsadmin /home/opsadmin/.ssh
sudo install -m 0600 -o opsadmin -g opsadmin authorized_keys   /home/opsadmin/.ssh/authorized_keys

# Validate sudoers syntax before activation.
sudo visudo -cf /etc/sudoers
sudo visudo -cf /etc/sudoers.d/opsadmin 2>/dev/null || true

# Confirm identity and account policy.
getent passwd opsadmin academy-status
sudo -l -U opsadmin
sudo -u academy-status -- id

Provision a tested break-glass path with monitored use and expiration. Do not depend on the same network, identity provider, DNS, and SSH path for both normal and emergency access.

6. Harden SSH without locking out recovery

Inspect the effective configuration, create a drop-in where supported, validate with sshd -t, keep an existing session open, and test a second connection before closing access. Requirements differ for automation, bastions, certificates, and compliance.

# /etc/ssh/sshd_config.d/60-capstone.conf
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
MaxAuthTries 3
LoginGraceTime 30
AllowUsers opsadmin
X11Forwarding no
AllowAgentForwarding no
AllowTcpForwarding no
ClientAliveInterval 300
ClientAliveCountMax 2

sudo sshd -t
sudo sshd -T | sort > evidence/sshd-effective.txt
sudo systemctl reload ssh.service 2>/dev/null || sudo systemctl reload sshd.service

# From a separate terminal, test before closing the original session.
ssh -o BatchMode=yes -o ConnectTimeout=5 opsadmin@SERVER 'id; hostname; date --iso-8601=seconds' 

7. Define the firewall as an allowlist with atomic validation

Permit established traffic, loopback, SSH from the approved administration network, and the application port from the approved client network. Confirm IPv4 and IPv6 requirements. Apply changes with console access and a timed rollback mechanism where possible.

#!/usr/sbin/nft -f
flush ruleset

table inet capstone {
  chain input {
    type filter hook input priority 0; policy drop;
    ct state invalid drop
    ct state established,related accept
    iifname "lo" accept
    ip protocol icmp accept
    ip6 nexthdr ipv6-icmp accept

    ip saddr 192.0.2.0/24 tcp dport 22 ct state new accept
    ip saddr 198.51.100.0/24 tcp dport 8080 ct state new accept

    counter log prefix "capstone-drop " limit rate 5/second drop
  }

  chain forward { type filter hook forward priority 0; policy drop; }
  chain output  { type filter hook output priority 0; policy accept; }
}

sudo nft --check --file provision/30-firewall.nft
sudo nft --file provision/30-firewall.nft
sudo nft list ruleset > evidence/nftables-effective.txt
ss -lntup > evidence/listeners-after-firewall.txt

# Test from an approved remote source and from a denied test source.
# Use console access to recover if policy differs from expectations.

A host firewall is one layer. Also document cloud security groups, hypervisor policy, load balancers, Kubernetes policy, and upstream firewalls when present.

8. Build a versioned application artifact and release layout

The capstone service is a small Python HTTP status endpoint using only the standard library. In a real system, pin dependencies, produce provenance, scan the artifact, and deploy by immutable digest.

#!/usr/bin/env python3
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json, os, time

START = time.time()
VERSION = os.environ.get("ACADEMY_VERSION", "unknown")

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path not in ("/", "/healthz"):
            self.send_error(404); return
        body = json.dumps({
            "status": "ok",
            "version": VERSION,
            "uptime_seconds": round(time.time() - START, 3)
        }).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers(); self.wfile.write(body)
    def log_message(self, fmt, *args):
        print(json.dumps({"client": self.client_address[0], "message": fmt % args}), flush=True)

ThreadingHTTPServer(("127.0.0.1", 8080), Handler).serve_forever()

release=2026.08.05.1
install -d "service/releases/$release"
install -m 0755 academy_status.py "service/releases/$release/academy_status.py"
sha256sum "service/releases/$release/academy_status.py"   > "service/releases/$release/SHA256SUMS"
ln -sfn "$release" service/releases/current
readlink -f service/releases/current
sha256sum -c "service/releases/$release/SHA256SUMS"

On the server, use a root-owned release directory and atomically update a symlink only after validation. The service user should read and execute the artifact but not modify releases or the unit file.

9. Run the service under a hardened systemd unit

Start with protections that match the workload, then verify effective settings and application behavior. Some directives depend on kernel and systemd versions.

[Unit]
Description=DevOps Academy capstone status service
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=60
StartLimitBurst=5

[Service]
Type=simple
User=academy-status
Group=academy-status
EnvironmentFile=/etc/academy-status.env
ExecStart=/opt/academy-status/current/academy_status.py
Restart=on-failure
RestartSec=3
TimeoutStopSec=15
KillSignal=SIGTERM

NoNewPrivileges=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectSystem=strict
ProtectHome=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
ProtectClock=yes
RestrictSUIDSGID=yes
LockPersonality=yes
MemoryDenyWriteExecute=yes
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
CapabilityBoundingSet=
AmbientCapabilities=
SystemCallArchitectures=native

CPUQuota=50%
MemoryMax=256M
TasksMax=64
LimitNOFILE=4096

StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

sudo systemd-analyze verify /etc/systemd/system/academy-status.service
sudo systemctl daemon-reload
sudo systemctl enable --now academy-status.service
sudo systemctl status academy-status.service --no-pager
sudo systemctl show academy-status.service   -p User -p Group -p MainPID -p ControlGroup   -p MemoryMax -p TasksMax -p CPUQuotaPerSecUSec
sudo systemd-analyze security academy-status.service --no-pager
curl --fail --silent --show-error http://127.0.0.1:8080/healthz
journalctl -u academy-status.service -n 30 --no-pager

systemd-analyze security is a heuristic review of sandbox settings, not proof that the application is secure. Combine it with code review, dependency controls, network policy, runtime tests, and monitoring.

10. Deploy through validate, canary, switch, verify, and rollback gates

A deployment changes one versioned artifact while preserving the previous release. Validate checksums, interpreter/runtime, unit syntax, disk capacity, ownership, and health before exposing traffic.

#!/usr/bin/env bash
set -Eeuo pipefail
new=${1:?release version required}
root=/opt/academy-status/releases
unit=academy-status.service
previous=$(readlink -f /opt/academy-status/current || true)

[[ -x "$root/$new/academy_status.py" ]]
sha256sum -c "$root/$new/SHA256SUMS"
python3 -m py_compile "$root/$new/academy_status.py"
systemd-analyze verify /etc/systemd/system/$unit

test -n "$previous" && printf 'Previous release: %s
' "$previous"
ln -sfn "$root/$new" /opt/academy-status/current
systemctl restart "$unit"

for attempt in {1..20}; do
  if curl --fail --silent --max-time 2 http://127.0.0.1:8080/healthz; then
    systemctl is-active --quiet "$unit"
    exit 0
  fi
  sleep 1
done

printf 'Health failed; rolling back
' >&2
[[ -n "$previous" ]] && ln -sfn "$previous" /opt/academy-status/current
systemctl restart "$unit"
exit 1

Record release ID, artifact hash, configuration revision, operator, start/end times, health results, journal interval, and rollback status. For multiple servers, deploy progressively and compare cohort metrics.

11. Monitor user work, service state, resources, and evidence freshness

A minimal check should fail distinctly for unit state, listener, HTTP health, version mismatch, capacity, backup age, clock synchronization, and failed security controls. Monitoring must run from both the host and an external vantage point.

#!/usr/bin/env bash
set -Eeuo pipefail
fail=0
check() { if "$@"; then printf 'OK   %q
' "$*"; else printf 'FAIL %q
' "$*"; fail=1; fi; }

check systemctl is-active --quiet academy-status.service
check ss -lnt 'sport = :8080'
check curl --fail --silent --max-time 3 http://127.0.0.1:8080/healthz
check timedatectl show -p NTPSynchronized --value
check test "$(df --output=pcent / | tail -1 | tr -dc '0-9')" -lt 85
check test "$(df --output=ipcent / | tail -1 | tr -dc '0-9')" -lt 85
check test -f /var/lib/academy-backup/LATEST.sha256

systemctl --failed --no-pager
journalctl -u academy-status.service --since '-15 min' -p warning --no-pager
exit "$fail"

Convert checks into metrics and alerts with thresholds, durations, ownership, and runbook links. Avoid alerting on every transient failure; alert when action is required and the operator can verify impact.

12. Back up durable state and prove restoration

Identify durable application configuration, data, certificates, and evidence. Exclude replaceable release artifacts when they can be reproduced from trusted storage. Encrypt off-host backups and separate backup credentials from the server.

#!/usr/bin/env bash
set -Eeuo pipefail
umask 077
src=/var/lib/academy-status
dst=/var/lib/academy-backup
stamp=$(date -u +%Y%m%dT%H%M%SZ)
mkdir -p "$dst"

tar --numeric-owner --xattrs --acls -czf "$dst/state-$stamp.tar.gz" -C "$src" .
sha256sum "$dst/state-$stamp.tar.gz" > "$dst/state-$stamp.tar.gz.sha256"
ln -sfn "state-$stamp.tar.gz.sha256" "$dst/LATEST.sha256"

# Restore test into an isolated directory, never over live data.
testdir=$(mktemp -d)
trap 'rm -rf "$testdir"' EXIT
sha256sum -c "$dst/state-$stamp.tar.gz.sha256"
tar -xzf "$dst/state-$stamp.tar.gz" -C "$testdir"
find "$testdir" -maxdepth 3 -printf '%m %u:%g %p
' | sort | head -n 100
# Add application-specific semantic checks here.

Track successful backup time, recoverable point, restore-test age, duration, object count, checksum result, and off-host replication. A backup job exit code is not a restore test.

13. Run controlled failure-injection exercises

Each exercise must define scope, start condition, abort threshold, expected alert, diagnosis path, rollback, evidence, and cleanup. Perform only in the disposable capstone environment.

Do not inject kernel panic, destructive filesystem corruption, credential theft, or uncontrolled network attacks. Simulate dangerous conditions through isolated fixtures and documented expected evidence.

14. Operate one complete incident from alert to prevention

1. Receive the alert and state impact, scope, start time, and service objective.
2. Stabilize by draining or stopping the unsafe component while preserving evidence.
3. Capture baseline, failed-unit, journal, process, socket, route, storage, policy, and cgroup evidence.
4. Write at least three hypotheses and select one discriminating test.
5. Apply one bounded repair with rollback and record its result.
6. Verify the health endpoint, external transaction, logs, latency, capacity, and security controls.
7. Restore temporary controls, update the runbook, and assign prevention work.
8. Publish a concise incident report separating root cause, contributing factors, detection gaps, and actions.

Incident report sections
1. Executive impact and timeline
2. Detection and response sequence
3. Technical root cause and evidence
4. Contributing conditions and why defenses did not prevent impact
5. Mitigation, repair, verification, and rollback status
6. Data/security assessment
7. What worked and what increased recovery time
8. Corrective actions with owner, priority, due date, and verification method
9. Evidence references and retention classification

15. Acceptance tests and scoring rubric

Score each domain from 0 to 3: absent, partial/manual, repeatable, or repeatable with independent verification. A passing capstone requires no zero in a critical domain and a documented plan for every residual risk.

16. Produce the final evidence bundle

The bundle should allow review without exposing secrets. Include hashes, software versions, effective configuration, test results, screenshots only when necessary, incident records, and restore evidence.

set -Eeuo pipefail
bundle="capstone-evidence-$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -m 0700 "$bundle"

cp -a evidence/. "$bundle/" 2>/dev/null || true
cp README.md "$bundle/"
cp -a runbooks "$bundle/"
git rev-parse HEAD > "$bundle/repository-commit.txt"
git status --short > "$bundle/repository-status.txt"

# Add redacted effective configuration and test outputs, not secrets.
find "$bundle" -type f -print0 | sort -z | xargs -0 sha256sum > "$bundle/SHA256SUMS"
tar -czf "$bundle.tar.gz" "$bundle"
sha256sum "$bundle.tar.gz" > "$bundle.tar.gz.sha256"
printf 'Review bundle and remove sensitive data before sharing: %s.tar.gz
' "$bundle"

Keep the original restricted bundle and a separately redacted review copy. Record who approved release of the review copy and which fields were removed.

17. Close the capstone as an operating system, not a one-time build

Confirm no test listeners, packet captures, debug modes, permissive policies, temporary firewall rules, emergency accounts, old credentials, failed units, or large fixtures remain. Schedule update, backup, restore, certificate, capacity, and access reviews.

18. Knowledge check

19. Course completion summary

• A production Linux server is an operated system of identity, network, service, storage, security, observability, automation, and recovery boundaries.
• Versioned desired state and validated artifacts make changes reviewable and repeatable.
• systemd, cgroups, permissions, firewall policy, and logging enforce and expose runtime behavior.
• Backups, restore exercises, incident runbooks, and failure injection convert assumptions into evidence.
• The Linux troubleshooting method—stabilize, preserve, hypothesize, test, repair, verify, prevent—connects every chapter of the course.

20. Further reading

• systemd service execution environment
• systemd-analyze verification and security review
• OpenSSH sshd_config manual
• nftables project documentation
• NIST SP 800-123 — General Server Security