Build a Troubleshooting Runbook from Symptom to Evidence to Hypothesis to Verified Remediation

AtlasMart wants a troubleshooting document that an on-call engineer can use at 03:00 without guessing. A useful runbook is not a list of commands. It is a state machine: symptom → evidence → hypotheses → discriminating test → one remediation → verification → rollback/escalation. This lesson applies that sequence to slow query, lock wait, memory pressure and cold-cache/I/O symptoms.

Learning outcomes

Reproducible lab baseline

1. Runbook skeleton

2. Drill A — broad query / cardinality symptom

Symptom: p99 rises for one analytical endpoint while lock wait remains low. Evidence: broad PROFILE has much larger intermediate rows/DB hits than returned rows. Hypothesis: query shape ignores explicit observed relationships. Remediation: use the relationship-shaped query, not a heap change.

CYPHER 25
PROFILE
MATCH (c:ObsCustomer), (i:ObsItem)
WHERE i.segment = c.segment
RETURN c.region AS region, count(*) AS pairs
ORDER BY pairs DESC;

CYPHER 25
PROFILE
MATCH (c:ObsCustomer)-[:OBS_VIEWED]->(i:ObsItem)
WHERE i.active
RETURN c.region AS region,
       count(*) AS views,
       round(avg(i.price), 2) AS avgPrice
ORDER BY views DESC;

3. Drill B — lock contention symptom

Symptom: writes queue even though CPU is not saturated. Evidence: SHOW TRANSACTIONS shows waiting state, rising wait time and lock dependency on INV-HOT-18. Hypothesis: hot shared entity serializes reservations. Remediation options: shorten transaction, partition invariant if business semantics permit, remodel to per-SKU/event state, or reduce concurrent access. Do not simply enlarge the pool.

from neo4j import GraphDatabase
from threading import Thread, Event
import time

URI = "bolt://127.0.0.1:7687"
AUTH = ("neo4j", "atlasmart-course-2026")
driver = GraphDatabase.driver(URI, auth=AUTH)
first_has_lock = Event()

def writer_a():
    with driver.session(database="neo4j") as session:
        tx = session.begin_transaction()
        tx.run("MATCH (x:ObsInventory {obsId:$id}) "
               "SET x.reserved = x.reserved + 1, x.version = x.version + 1",
               id="INV-HOT-18").consume()
        first_has_lock.set()
        time.sleep(5)  # intentionally hold the write lock in this disposable lab
        tx.commit()

def writer_b():
    first_has_lock.wait()
    with driver.session(database="neo4j") as session:
        tx = session.begin_transaction()
        tx.run("MATCH (x:ObsInventory {obsId:$id}) "
               "SET x.reserved = x.reserved + 1, x.version = x.version + 1",
               id="INV-HOT-18").consume()
        tx.commit()

a=Thread(target=writer_a); b=Thread(target=writer_b)
a.start(); b.start()
# While A sleeps, use a third cypher-shell/session with SHOW TRANSACTIONS.
a.join(); b.join(); driver.close()

CYPHER 25
SHOW TRANSACTIONS
YIELD transactionId, username, database, currentQuery, currentQueryStatus,
      elapsedTime, waitTime, currentQueryElapsedTime, currentQueryWaitTime,
      activeLockCount, resourceInformation,
      currentQueryAllocatedBytes, allocatedDirectBytes, estimatedUsedHeapMemory,
      pageHits, pageFaults, currentQueryPageHits, currentQueryPageFaults,
      indexes, runtime, statusDetails
RETURN *
ORDER BY currentQueryElapsedTime DESC;

4. Drill C — query-memory symptom

Symptom: one analytical request allocates heavily or reaches transaction memory limits. Evidence: current query memory rises while page-fault evidence is not dominant. Hypothesis: unnecessary collection materialization. Remediation: stream aggregation or bound/chunk the workload before changing global transaction limits.

CYPHER 25
// Disposable lab only. Start smaller; increase gradually while observing transaction memory.
UNWIND range(1,50000) AS n
WITH collect({n:n, bucket:n % 100}) AS rows
RETURN size(rows) AS materializedRows,
       reduce(s=0, x IN rows | s + x.n) AS checksum;

CYPHER 25
UNWIND range(1,50000) AS n
RETURN count(*) AS rowCount, sum(n) AS checksum;

5. Drill D — cold-start page-cache / I/O symptom

On the disposable course container only, a planned restart naturally makes the page cache cold. Run the same read workload after restart and again after warmup; record page-fault/block-I/O and latency evolution. Do not clear production OS/page caches to manufacture this state.

docker restart atlasmart-neo4j
# Wait for normal readiness using your existing course health check.
# Run the same bounded read benchmark three times and record:
# request p50/p95/p99, SHOW TRANSACTIONS page hits/faults while active,
# and docker stats Block I/O / CPU / memory snapshots.

6. Alert thresholds come from baselines and headroom

7. Escalation package

If the incident survives local diagnosis, preserve enough evidence for another engineer or vendor support to reproduce the reasoning without leaking secrets.

8. Cleanup and proof of reset

CYPHER 25
MATCH (n)
WHERE n:ObsCustomer OR n:ObsItem OR n:ObsInventory
DETACH DELETE n;
DROP CONSTRAINT obs_customer_id IF EXISTS;
DROP CONSTRAINT obs_item_id IF EXISTS;
DROP CONSTRAINT obs_inventory_id IF EXISTS;

CYPHER 25
MATCH (n)
WHERE n:ObsCustomer OR n:ObsItem OR n:ObsInventory
RETURN count(n) AS remainingObsNodes;
SHOW CONSTRAINTS
YIELD name
WHERE name STARTS WITH 'obs_'
RETURN name;

9. Final runbook acceptance

Production judgment

Summary and next step

Chapter 18 turns observability into an operational method rather than a dashboard catalog. You can now separate query work, locks, memory, page cache, host resources, result/client time and capacity growth with evidence. Chapter 19 uses that discipline when Neo4j data is split across multiple and composite databases, where domain boundaries add new routing, security, resource and operational consequences.

Authoritative references

• Current Neo4j versions — Current Neo4j 2026.07.1 and 5.26.30 LTS release snapshot.
• Monitoring overview — Current monitoring surfaces: logs, metrics, query/transaction management, connections, jobs, and reports.
• Metrics — Enterprise metrics architecture and supported export surfaces.
• Essential metrics — Server, Neo4j, cluster, and workload signals that require correlation rather than single-metric tuning.
• Metrics reference — Current page-cache, JVM, CPU, file-descriptor, transaction and store metric names.
• Logging — Current neo4j/debug/http/gc/query/security log behavior and edition boundaries.
• Show and terminate transactions — Current SHOW TRANSACTIONS fields for locks, wait time, memory, page hits/faults, runtime, indexes, and status.
• Manage queries — Current query visibility through transaction management rather than legacy dbms.listQueries().
• Memory configuration — Heap, page cache, native memory, transaction limits, and memory-recommendation guidance.
• Disks, RAM and other tips — Page-cache warmup, disk behavior, and operating-system considerations.
• Performance — Performance topics spanning memory, disks, statistics, execution plans, Bolt threads, and filesystem tuning.
• Show configuration settings — Current SHOW SETTINGS syntax and self-managed server scope.
• Transaction logs — Transaction-log location, rotation, retention, pruning, and capacity implications.
• Query tuning and PROFILE — Execution-plan evidence for rows, database hits, memory, access paths, and operator shape.
• Python driver performance — Driver-side timing, database selection, result consumption, and application/database boundary considerations.