Debugging Bash with set Options and shellcheck

1. Debug by failure stage, not by guesswork

A shell script can fail while being parsed, during expansion, while resolving a command, inside the command, because input has an unexpected shape, or because the environment differs. Start by reproducing the failure with known inputs and recording the exact interpreter, arguments, working directory, environment assumptions, status, standard output, and standard error.

flowchart TD
  R["Reproduce with fixed input"] --> N{"bash -n passes?"}
  N -- no --> P["Repair parser or quoting structure"]
  N -- yes --> L["Run ShellCheck and inspect warnings"]
  L --> T["Trace a minimal scope with xtrace"]
  T --> S["Inspect variables, arguments, types, and statuses"]
  S --> H["Form one hypothesis"]
  H --> C["Make one controlled change"]
  C --> V["Run regression case and verify status/output"]
  V --> D{"Resolved?"}
  D -- no --> T
  D -- yes --> E["Preserve test and evidence"]

Do not begin by enabling every option or tracing an entire production job that handles secrets. Narrow the failing scope and collect only the evidence required to test a hypothesis.

2. Parse before executing

bash -n script.sh reads commands and reports syntax errors without executing normal commands. It catches missing terminators, malformed compound commands, and many quoting problems, but it cannot prove that expansions, paths, data, or external commands will behave correctly.

bash -n deploy.sh
printf 'syntax_status=%d\n' "$?"

# Parse every tracked shell file before a commit.
while IFS= read -r -d '' script; do
  if ! bash -n "$script"; then
    printf 'Syntax failure: %s\n' "$script" >&2
    exit 1
  fi
done < <(find . -type f -name '*.sh' -print0)

# Display how Bash reads a command name and function.
type -a printf
command -V shellcheck 2>/dev/null || true
declare -f main 2>/dev/null || true

Run the intended interpreter. A script that passes bash -n may fail under sh, and a POSIX shell script should not be validated only with Bash extensions enabled.

3. Xtrace reveals expanded commands—but can reveal secrets

set -x prints commands after expansion and before execution. Improve trace usefulness with PS4, which can include source file, line number, function name, and process identity. Direct trace output to a controlled descriptor with BASH_XTRACEFD so it does not mix with normal standard error.

debug_log=${DEBUG_LOG:-./bash-trace.log}
exec 9> "$debug_log"
export BASH_XTRACEFD=9
export PS4='+ ${BASH_SOURCE##*/}:${LINENO}:${FUNCNAME[0]:-main}: '

trace_demo() {
  local service=$1
  set -x
  printf 'checking service=%s\n' "$service"
  command -v systemctl >/dev/null 2>&1
  set +x
}

trace_demo 'ssh.service'
exec 9>&-
printf 'trace written to %s\n' "$debug_log"

4. Inspect shell state with representations, not ambiguous text

declare -p shows variable attributes and values in a reusable representation. printf '%q' exposes whitespace and special characters. caller, FUNCNAME, BASH_SOURCE, and LINENO help locate call paths. Capture status immediately when it matters.

diagnose_value() {
  local label=$1 value=$2
  printf '%s quoted=%q length=%d\n' "$label" "$value" "${#value}"
}

config_path=$'/tmp/config\tblue.conf'
declare -p config_path
diagnose_value config_path "$config_path"

show_stack() {
  local depth=0
  while caller "$depth"; do
    (( depth += 1 ))
  done
}

outer() { inner; }
inner() { show_stack; }
outer

if grep -q '^missing=' /etc/os-release; then
  grep_status=0
else
  grep_status=$?
fi
printf 'grep_status=%d\n' "$grep_status"

5. set options change semantics; they are not a magic strict mode

set -u reports many expansions of unset parameters. set -o pipefail makes a pipeline fail when any component fails, rather than using only the last command’s status. set -e requests exit after certain unhandled nonzero statuses, but its exceptions depend on syntactic context such as conditions, lists, negation, and pipelines. Functions and command substitutions add further surprises.

# Inspect current option state.
set -o
shopt -p

# Apply options intentionally inside a controlled script or subshell.
(
  set -u
  set -o pipefail

  value=${OPTIONAL_VALUE:-default}
  printf 'value=%s\n' "$value"

  # The pipeline now reports grep failure instead of tee success.
  printf '%s\n' alpha beta | grep gamma | tee result.txt
)
status=$?
printf 'subshell_status=%d\n' "$status"

# Prefer explicit handling around expected failures.
if ! output=$(some_command 2>&1); then
  status=$?
  printf 'some_command failed: %s\n' "$output" >&2
  # Note: because ! reverses status, capture inside a different structure
  # when the original numeric status is required.
fi

The final example demonstrates why status handling needs care: ! reverses the status visible inside the branch. When the original value matters, use an if output=$(command); then ... else status=$?; ... fi form. Chapter 15 develops disciplined option, trap, cleanup, and failure patterns.

6. ShellCheck provides static analysis, not proof

ShellCheck parses shell code and reports suspicious quoting, test syntax, unused values, unsafe loops, portability conflicts, and many shell-specific mistakes. Run it with the intended shell declared by a shebang or configuration. Read the diagnostic explanation before suppressing it.

# Analyze one script and include optional style findings.
shellcheck --severity=style script.sh

# Emit machine-readable results for CI integration.
shellcheck --format=json scripts/*.sh > shellcheck.json

# Analyze every NUL-delimited shell file safely.
while IFS= read -r -d '' script; do
  printf 'checking %s\n' "$script"
  shellcheck "$script"
done < <(find scripts -type f -name '*.sh' -print0)

# A narrow, documented directive applies to the next complete command.
# shellcheck disable=SC1091  # generated path is provisioned by deployment
# source /opt/acme/runtime.env

A clean ShellCheck run does not verify external command behavior, concurrency, permissions, network state, destructive intent, or business logic. Pair static analysis with syntax checks and executable tests.

7. Hands-on lab: repair and regression-test a fragile script

The first script is syntactically valid but mishandles spaces, glob characters, missing arguments, and pipeline failure. Preserve it as evidence, then create a corrected version and a small regression harness.

lab="$HOME/devops-academy/linux/chapter14/lesson05"
rm -rf "$lab"
mkdir -p "$lab/input directory"
cd "$lab"
printf 'ERROR first\nINFO second\n' > 'input directory/app one.log'
printf 'INFO only\n' > 'input directory/*.literal.log'

cat > fragile.sh <<'SCRIPT'
#!/usr/bin/env bash
root=$1
for file in $(find $root -type f -name '*.log'); do
  grep ERROR $file | wc -l
  echo $file
 done
SCRIPT

cat > robust.sh <<'SCRIPT'
#!/usr/bin/env bash

main() {
  if (( $# != 1 )); then
    printf 'Usage: %s DIRECTORY\n' "${0##*/}" >&2
    return 2
  fi

  local root=$1 file count failures=0
  if [[ ! -d $root ]]; then
    printf 'Not a directory: %s\n' "$root" >&2
    return 3
  fi

  while IFS= read -r -d '' file; do
    if count=$(grep -c 'ERROR' -- "$file"); then
      :
    else
      status=$?
      if (( status == 1 )); then
        count=0
      else
        printf 'grep failed for %s with status %d\n' "$file" "$status" >&2
        (( failures += 1 ))
        continue
      fi
    fi
    printf 'file=%q errors=%d\n' "$file" "$count"
  done < <(find "$root" -type f -name '*.log' -print0)

  (( failures == 0 ))
}

main "$@"
SCRIPT

cat > test-robust.sh <<'SCRIPT'
#!/usr/bin/env bash

output=$(./robust.sh 'input directory')
status=$?

if (( status != 0 )); then
  printf 'FAIL: robust.sh returned %d\n' "$status" >&2
  exit 1
fi

if [[ $output != *"errors=1"* || $output != *"errors=0"* ]]; then
  printf 'FAIL: unexpected output\n%s\n' "$output" >&2
  exit 1
fi

if ./robust.sh >/dev/null 2>&1; then
  printf 'FAIL: missing argument should be rejected\n' >&2
  exit 1
fi

printf 'PASS\n'
SCRIPT

chmod u+x fragile.sh robust.sh test-robust.sh
bash -n fragile.sh robust.sh test-robust.sh
shellcheck fragile.sh || true
shellcheck robust.sh test-robust.sh
./test-robust.sh

8. Common debugging mistakes

9. Knowledge check

10. Summary

Bash debugging is strongest when it follows evidence from parser to expansion, command execution, data shape, and environment. Parse with the intended interpreter, analyze statically, trace a narrow scope, display values unambiguously, understand option semantics, and preserve a regression case. Options and linters are tools within that method—not substitutes for explicit error design.

11. Further reading

• GNU Bash Reference Manual — The set builtin.
• GNU Bash Reference Manual — diagnostic variables including BASH_SOURCE and FUNCNAME.
• ShellCheck project and online analyzer.
• ShellCheck diagnostic explanations and directives.