Self-Managed Administration: Configuration, Email, Object Storage, Backups, Restore, and Maintenance: Guided Hands-On Workflow and Core Operations

1. Scenario and preflight

You are the operator of fictional gitlab.example.invalid. The mandatory exercise does not install GitLab. Instead it models the exact artifacts a recovery runbook must track: version/type, configuration inventory, encryption-secret custody, repository state, database metadata, uploads/artifacts, and an OCI digest marker. This lets every learner practice the recovery logic without risking a server.

2. Build a deterministic source fixture

set -eu
rm -rf ch30-admin-lab
mkdir -p ch30-admin-lab/source/{repo,data,uploads,artifacts,registry} ch30-admin-lab/evidence
cd ch30-admin-lab/source
printf 'gitlab_version=19.3.x-ee\ninstallation=fixture\nrepo_storage=default\n' > platform.env
printf 'SECRET_MATERIAL_STORED_SEPARATELY=true\n' > secrets-custody.txt
printf 'commit=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\nproject=platform-lab/sample\n' > repo/identity.txt
printf '{"project_id":30,"namespace":"platform-lab","issue_count":2}\n' > data/database.json
printf 'avatar-fixture\n' > uploads/avatar.txt
printf 'ci-artifact-fixture\n' > artifacts/build.txt
printf 'image=registry.example.invalid/platform-lab/sample@sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\n' > registry/image.txt
find repo data uploads artifacts registry -type f -print0 | sort -z | xargs -0 sha256sum > ../evidence/source.sha256
cat ../evidence/source.sha256

3. Record version, topology, and backup scope before execution

A usable recovery record says what is inside the application backup, what is external, and who owns each prerequisite. It does not record secret values.

cat > ../evidence/recovery-inventory.json <<'JSON'
{
  "source": "gitlab.example.invalid",
  "gitlab_version": "19.3.x-ee",
  "installation_method": "fixture",
  "repository_storages": ["default"],
  "application_backup": ["database", "repositories", "uploads", "artifacts", "packages", "registry-marker"],
  "separate_backups": ["configuration", "encryption-secrets", "tls-keys", "external-object-storage"],
  "rpo_minutes": 60,
  "rto_minutes": 180
}
JSON
jq . ../evidence/recovery-inventory.json

4. Optional live inspection first

If you already own an isolated Linux-package instance, collect only read-only evidence before any reconfigure/backup action. Record exact version/type, service state, storage names, and health. Do not copy raw gitlab.rb or gitlab-secrets.json into the lab evidence folder.

# OPTIONAL isolated Linux-package instance.
sudo gitlab-ctl status
sudo gitlab-rake gitlab:env:info
sudo gitlab-rake gitlab:check SANITIZE=true
curl -fsS http://127.0.0.1/-/readiness?all=1
# Inspect backup configuration without printing secret files.
sudo test -r /etc/gitlab/gitlab.rb
sudo test -r /etc/gitlab/gitlab-secrets.json

5. Model SMTP safely

Use a fake local relay such as MailHog/Mailpit only if you already have one in the disposable lab. Never use a real mailbox password. Current Linux-package configuration supports encrypted SMTP credentials; in production, prefer that mechanism instead of committing plaintext credentials.

# SAFE CONFIGURATION FIXTURE ONLY — not a production password.
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = 'mail.example.invalid'
gitlab_rails['smtp_port'] = 1025
gitlab_rails['smtp_tls'] = false
gitlab_rails['smtp_enable_starttls_auto'] = false
# username/password intentionally omitted from this file

After a real disposable reconfigure, use the documented Rails-console Notify.test_email(...).deliver_now against the fake receiver and record only success/failure plus timestamp—not SMTP credentials or message contents containing sensitive data.

6. Model object-storage identity, not credentials

The important evidence is provider/endpoint class, object type, bucket, and backup owner. A wrong bucket that authenticates successfully is still a recovery failure. The fixture below intentionally records names only.

cat > ../evidence/object-storage-map.tsv <<'EOF'
object_type	bucket	backup_owner
artifacts	gitlab-lab-artifacts	platform-backup
uploads	gitlab-lab-uploads	platform-backup
packages	gitlab-lab-packages	platform-backup
registry	gitlab-lab-registry	registry-backup
EOF
column -t -s $'\t' ../evidence/object-storage-map.tsv

7. Create the synthetic application backup and integrity manifest

cd ..
tar -czf evidence/application-backup.tar.gz source/repo source/data source/uploads source/artifacts source/registry
sha256sum evidence/application-backup.tar.gz > evidence/application-backup.tar.gz.sha256
# Simulate separately protected config/secrets metadata without storing the secret itself.
printf 'configuration_backup_present=true\nsecrets_backup_present=true\nobject_storage_backup_present=true\n' > evidence/prerequisites.txt
sha256sum evidence/recovery-inventory.json evidence/object-storage-map.tsv evidence/prerequisites.txt > evidence/control-files.sha256
sha256sum -c evidence/application-backup.tar.gz.sha256
sha256sum -c evidence/control-files.sha256

On an optional Linux-package lab the analogous application command is sudo gitlab-backup create; configuration can be separately archived with sudo gitlab-ctl backup-etc. External object storage still needs its own backup according to the storage provider/method.

8. Restore into a separate target and verify identities

Do not overwrite the source fixture. A separate target makes recovery evidence meaningful and catches assumptions about paths and prerequisites.

mkdir -p restore-target
# Verify archive before extraction.
sha256sum -c evidence/application-backup.tar.gz.sha256
tar -xzf evidence/application-backup.tar.gz -C restore-target
# Compare source and restored representative state.
diff -u source/repo/identity.txt restore-target/source/repo/identity.txt
diff -u source/data/database.json restore-target/source/data/database.json
diff -u source/uploads/avatar.txt restore-target/source/uploads/avatar.txt
diff -u source/artifacts/build.txt restore-target/source/artifacts/build.txt
diff -u source/registry/image.txt restore-target/source/registry/image.txt
find restore-target/source -type f -print0 | sort -z | xargs -0 sha256sum > evidence/restored.sha256
# Normalize paths and compare content hashes.
sed 's#  restore-target/source/#  #' evidence/restored.sha256 > evidence/restored.normalized.sha256
diff -u evidence/source.sha256 evidence/restored.normalized.sha256

9. Optional real restore checklist—do not improvise

For an isolated Linux-package restore target, install the exact backup version and CE/EE type, run reconfigure at least once, restore the original gitlab-secrets.json and matching configuration/storage names, place the backup in the configured backup directory, stop Puma and Sidekiq as documented, then run gitlab-backup restore BACKUP=<id>. Afterward start/reconfigure as required and run the documented checks.

# OPTIONAL / DESTRUCTIVE: disposable separate target only.
# Preconditions already proven: exact version/type, secrets/config restored, fresh target.
sudo gitlab-ctl stop puma
sudo gitlab-ctl stop sidekiq
sudo gitlab-ctl status
# sudo gitlab-backup restore BACKUP=<verified-backup-id>
# After documented restore steps:
sudo gitlab-ctl start
sudo gitlab-rake gitlab:check SANITIZE=true
sudo gitlab-rake gitlab:doctor:secrets
sudo gitlab-rake gitlab:artifacts:check
sudo gitlab-rake gitlab:lfs:check
sudo gitlab-rake gitlab:uploads:check

10. Challenge: choose the correct surface

Classify each item before acting: a missing gitlab-secrets.json is a secret-custody problem, not a repository problem; an empty artifact bucket is an object-storage recovery problem, not a PostgreSQL problem; a version mismatch is a target-build problem, not a reason to force restore; a failed fake SMTP test is a mail configuration/transport problem, not proof that Sidekiq is generally unhealthy.

11. Cleanup and evidence retention

# Keep only sanitized evidence needed to assess the drill.
rm -rf source restore-target
# Prove no obvious secret material was retained.
if grep -RniE '(smtp_password|aws_secret_access_key|PRIVATE-TOKEN|BEGIN .*PRIVATE KEY)' evidence; then
  echo 'FAIL: sensitive-looking material found in evidence' >&2; exit 1
fi
find evidence -maxdepth 1 -type f -print | sort

Knowledge check

12. Lesson summary and bridge

You have now executed the recovery logic without relying on a valuable server. Lesson 3 turns the workflow into architecture decisions: hosted responsibility, storage design, RPO/RTO, and the operational cost of externalized services.

Primary sources and version notes

These lessons were finalized against current official GitLab documentation on 2026-08-22 and GitLab 19.3. Self-Managed commands and file locations depend on installation method. Re-check the documentation for the exact version, topology, package/chart, and storage architecture before production administration or recovery.

• GitLab 19.3 release
• Administer GitLab
• Configure GitLab
• Back up GitLab
• Restore GitLab
• Linux package backup configuration
• Docker backup
• Helm chart backup and restore
• Object storage
• SMTP settings
• Encrypted configuration
• Health check
• Maintenance Mode
• Maintenance Rake tasks
• Integrity check Rake tasks
• Repository checks