JSON and YAML Workflows with jq and yq

1. JSON and YAML are data models, not line-oriented text

grep, sed, and awk remain valuable, but they do not understand nested objects, arrays, quoting, YAML document boundaries, anchors, or JSON string escaping. Structured processors parse the document, apply an expression to the data model, and serialize the result. This creates a reviewable boundary between shell arguments and configuration structure.

flowchart TD
  S["Shell arguments and environment"] --> V["Validate scalar inputs"]
  V --> A["Pass values with --arg, --argjson, or strenv"]
  J["JSON or YAML documents"] --> P["Parse into structured values"]
  A --> F["Apply jq or yq expression"]
  P --> F
  F --> C{"Required fields and policy valid?"}
  C -- no --> E["Nonzero status and diagnostics"]
  C -- yes --> O["Serialize deterministic output"]
  O --> W["Atomic write or pipeline consumption"]

2. Query and construct JSON with jq

A jq filter consumes JSON values and produces zero or more JSON values. Use -r for raw strings, -c for compact JSON, -e when false or null should become nonzero status, and -n to construct data without input.

# Read required fields and fail when the expression is false or null.
jq -e '.service.name and (.service.replicas | type == "number")' manifest.json

# Produce one raw value for shell consumption.
service=$(jq -er '.service.name' manifest.json)

# Select and reshape records.
jq -c '.deployments[]
  | select(.enabled == true)
  | {name, image, replicas: (.replicas // 1)}' inventory.json

# Construct JSON safely from shell values.
jq -n \
  --arg name "$service" \
  --arg image "$image" \
  --argjson replicas "$replicas" \
  '{service:{name:$name,image:$image,replicas:$replicas}}' 

--arg encodes a shell string as a JSON string. --argjson parses its value as JSON and should be used only after validating that the shell input represents the intended JSON type. Do not concatenate user values into the jq program.

3. Arrays, reductions, and slurp mode support operational summaries

Use map, group_by, sort_by, add, and reduce to calculate summaries without converting structured values into text and reparsing them.

# Count deployments by environment.
jq '.deployments
  | group_by(.environment)
  | map({environment: .[0].environment, count: length})' inventory.json

# Sum requested replicas, defaulting missing values to one.
jq '[.deployments[] | (.replicas // 1)] | add // 0' inventory.json

# Slurp newline-delimited JSON into an array.
jq -s 'sort_by(.time) | {events: ., count: length}' events.ndjson

# Read NUL-free JSON safely into Bash records.
while IFS= read -r record; do
  name=$(jq -r '.name' <<<"$record")
  printf 'deployment=%s\n' "$name"
done < <(jq -c '.deployments[]' inventory.json)

Large documents and repeated subprocesses can become expensive. Prefer one jq program that performs the complete transformation when practical, and stream records only when the data size or workflow requires it.

4. Query and update YAML with yq v4

yq eval applies an expression to documents in sequence; eval-all loads all documents and files before evaluating once, which is useful for merges. -i updates a file in place, while -e makes false, null, or no-match results fail.

# Read and validate values.
yq -e '.service.name and (.service.replicas | type == "!!int")' service.yaml
yq -r '.service.name' service.yaml

# Pass environment values as strings rather than expression source.
SERVICE_IMAGE='registry.example/app:2026.08' \
  yq '.service.image = strenv(SERVICE_IMAGE)' service.yaml

# Update in place only after creating a backup and validating input.
cp -- service.yaml service.yaml.bak
REPLICAS=4 yq -i '.service.replicas = env(REPLICAS)' service.yaml

# Convert YAML to JSON for a jq pipeline.
yq -o=json '.' service.yaml | jq -e '.service.replicas >= 1' 

In-place editing is convenient but should be paired with source control, backup, schema validation, or atomic replacement. YAML serializers may normalize formatting, quoting, comments, anchors, or key style; test the exact tool version against the repository’s conventions.

5. Merge policy must be explicit

A merge operator needs a declared precedence and array policy. Replacing a scalar is straightforward; combining arrays may mean append, replace, deduplicate, or merge by a key. Do not assume the default matches the deployment model.

# base.yaml is loaded first; overlay.yaml takes precedence.
yq eval-all '
  select(fileIndex == 0) * select(fileIndex == 1)
' base.yaml overlay.yaml > rendered.yaml

# Validate the rendered result before replacing an existing artifact.
yq -e '
  .service.name and
  (.service.replicas >= 1) and
  (.service.image | test("^[^:]+:[^:]+$"))
' rendered.yaml >/dev/null

# Atomic replacement on the same filesystem.
tmp=$(mktemp -- rendered.yaml.tmp.XXXXXXXX)
yq eval-all 'select(fileIndex == 0) * select(fileIndex == 1)' \
  base.yaml overlay.yaml > "$tmp"
mv -f -- "$tmp" rendered.yaml

Dynamic eval features in data tools should receive only trusted expressions. Ordinary variable values belong in data variables such as --arg or strenv, not in generated program text.

6. Hands-on lab: validate and render a service manifest

The lab creates a base YAML document and an environment overlay, renders them with yq, validates the result, and emits a compact JSON deployment record with jq.

lab="$HOME/devops-academy/linux/chapter15/lesson04"
rm -rf "$lab"
mkdir -p "$lab"
cd "$lab"

command -v jq >/dev/null || { printf 'jq is required\n' >&2; exit 69; }
command -v yq >/dev/null || { printf 'yq v4 is required\n' >&2; exit 69; }
yq --version

cat > base.yaml <<'YAML'
service:
  name: payments-api
  image: registry.example/payments:2026.08
  replicas: 1
  labels:
    owner: platform
    tier: backend
YAML

cat > production.yaml <<'YAML'
service:
  replicas: 4
  labels:
    environment: production
YAML

yq eval-all 'select(fileIndex == 0) * select(fileIndex == 1)' \
  base.yaml production.yaml > rendered.yaml

yq -e '
  .service.name and
  (.service.replicas | type == "!!int") and
  (.service.replicas >= 1) and
  .service.image and
  .service.labels.environment
' rendered.yaml >/dev/null

yq -o=json '.service' rendered.yaml |
  jq -c '{name, image, replicas, environment: .labels.environment}' \
  > deployment.json

jq -e '.name == "payments-api" and .replicas == 4' deployment.json
cat rendered.yaml
cat deployment.json
printf 'PASS\n' 

7. Common structured-data mistakes

8. Knowledge check

9. Summary

Structured configuration should remain structured throughout the workflow. Parse with jq or the pinned yq implementation, pass shell values as data, use exit-status modes for validation, define merge precedence, and serialize outputs rather than constructing them manually. In-place mutation requires the same backup, validation, and atomicity discipline as any other configuration change.

10. Further reading

• jq Manual.
• jq Tutorial.
• yq v4 — Evaluate.
• yq v4 — Evaluate All.