Build Dashboards, Baselines, SLO Alerts, and Incident Timelines

Learning outcomes

Observability is operationally useful only when it changes how a team detects and explains user impact. A dashboard full of counters is not an observability strategy. ServiceHub needs explicit service-level objectives (SLOs), known normal ranges under labeled load, alerts that combine user impact with leading indicators, and an incident timeline that preserves database, OS, configuration, deployment, and topology events.

1. Start with service questions and golden signals

Add durability/topology signals when the service depends on them: replica freshness, Galera primary-component health, backup age, filesystem capacity, and binary-log retention can be leading indicators even before end-user latency breaks.

DROP DATABASE IF EXISTS servicehub17;CREATE DATABASE servicehub17 CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;USE servicehub17;CREATE TABLE tickets (  ticket_id BIGINT PRIMARY KEY AUTO_INCREMENT,  customer_id BIGINT NOT NULL,  status ENUM('open','waiting','closed') NOT NULL,  priority TINYINT NOT NULL,  opened_at DATETIME(6) NOT NULL,  closed_at DATETIME(6) NULL,  INDEX ix_status_opened(status, opened_at),  INDEX ix_customer(customer_id)) ENGINE=InnoDB;CREATE TABLE ticket_events (  event_id BIGINT PRIMARY KEY AUTO_INCREMENT,  ticket_id BIGINT NOT NULL,  event_type VARCHAR(40) NOT NULL,  event_at DATETIME(6) NOT NULL,  payload VARCHAR(500) NULL,  INDEX ix_ticket_time(ticket_id, event_at)) ENGINE=InnoDB;INSERT INTO tickets(customer_id,status,priority,opened_at,closed_at)WITH RECURSIVE seq AS (  SELECT 1 AS n  UNION ALL SELECT n+1 FROM seq WHERE n < 500)SELECT 1000 + (n % 80),       CASE WHEN n % 7 = 0 THEN 'closed' WHEN n % 3 = 0 THEN 'waiting' ELSE 'open' END,       1 + (n % 5),       NOW(6) - INTERVAL n MINUTE,       CASE WHEN n % 7 = 0 THEN NOW(6) - INTERVAL (n-2) MINUTE ELSE NULL ENDFROM seq;INSERT INTO ticket_events(ticket_id,event_type,event_at,payload)SELECT ticket_id,       CASE WHEN ticket_id % 4=0 THEN 'comment' ELSE 'status_change' END,       opened_at + INTERVAL 30 SECOND,       RPAD('x', 120, 'x')FROM tickets;

2. Build a free local snapshot table with context

The simplest dashboard backend can be SQL itself. The goal is not to replace a production telemetry system; it is to teach the data model: timestamp, server identity, workload label, observation-window context, and raw numeric metrics.

USE servicehub17;CREATE TABLE obs_snapshots (  sampled_at DATETIME(6) NOT NULL,  workload_label VARCHAR(80) NOT NULL,  metric_name VARCHAR(128) NOT NULL,  metric_value DECIMAL(30,6) NOT NULL,  PRIMARY KEY(sampled_at, workload_label, metric_name)) ENGINE=InnoDB;

INSERT INTO servicehub17.obs_snapshots(sampled_at,workload_label,metric_name,metric_value)SELECT NOW(6),'baseline',VARIABLE_NAME,CAST(VARIABLE_VALUE AS DECIMAL(30,6))FROM performance_schema.global_statusWHERE VARIABLE_NAME IN (  'Questions','Threads_connected','Threads_running',  'Created_tmp_tables','Created_tmp_disk_tables',  'Innodb_buffer_pool_reads','Innodb_buffer_pool_read_requests',  'Innodb_os_log_written');SELECT * FROM servicehub17.obs_snapshotsORDER BY sampled_at, metric_name;

This table intentionally stores raw values. Rates require at least two samples and awareness of counter resets. Production collectors normally add host, server UUID/role, version, deployment ID, and restart/uptime context.

3. Baselines: label the workload or the numbers are misleading

A baseline is not “yesterday’s average.” Record known operating modes: idle, normal weekday, batch import, monthly close, cache-cold restart, failover catch-up. Compare like with like.

SELECT COUNT(*) FROM servicehub17.tickets WHERE status='open';SELECT customer_id, COUNT(*) AS cFROM servicehub17.ticketsGROUP BY customer_idORDER BY c DESCLIMIT 20;INSERT INTO servicehub17.obs_snapshots(sampled_at,workload_label,metric_name,metric_value)SELECT NOW(6),'normal_read_mix',VARIABLE_NAME,CAST(VARIABLE_VALUE AS DECIMAL(30,6))FROM performance_schema.global_statusWHERE VARIABLE_NAME IN (  'Questions','Threads_running','Created_tmp_disk_tables',  'Innodb_buffer_pool_reads');

With repeated samples, calculate medians/percentiles or simple range bands externally. Do not invent a universal “Threads_running > 20 is bad” rule. A 64-core analytics host and a 2-core VM have different saturation points.

4. Alert design: user impact plus leading indicators

An alert should answer: what service is at risk, how urgent is it, who owns the response, and what evidence should the responder inspect first?

Use multi-window alerts where possible: a short fast-burn alert catches acute outages, while a longer slow-burn alert catches gradual regressions. Exact windows/thresholds must come from the service SLO and measured baseline.

5. Incident timeline: correlate changes, not screenshots

During an incident, create one ordered narrative. Include deployment/configuration changes, MariaDB restart/version, log messages, metric changes, query digests, replica/Galera state transitions, OS saturation, and operator actions. Time synchronization matters: mismatched host time zones can make causality look reversed.

CREATE TABLE servicehub17.incident_events (  event_time DATETIME(6) NOT NULL,  source VARCHAR(40) NOT NULL,  event_type VARCHAR(60) NOT NULL,  detail VARCHAR(1000) NOT NULL,  PRIMARY KEY(event_time, source, event_type)) ENGINE=InnoDB;INSERT INTO servicehub17.incident_events VALUES(NOW(6),'deploy','release','servicehub-api release 2026.08.20.3'),(NOW(6) + INTERVAL 1 SECOND,'db','observation','statement digest latency increased'),(NOW(6) + INTERVAL 2 SECOND,'os','observation','disk write latency sample elevated'),(NOW(6) + INTERVAL 3 SECOND,'operator','action','captured EXPLAIN and disabled suspect feature flag');SELECT * FROM servicehub17.incident_events ORDER BY event_time;

In a real incident, timestamps come from authoritative sources rather than manually fabricated entries. The lab demonstrates the schema and ordering discipline. Preserve evidence before resetting Performance Schema or rotating/deleting logs.

6. Wrong approach: dashboard by threshold folklore

A dashboard that turns every metric red above a copied internet threshold creates noise and teaches responders to ignore alerts. Another common mistake is a green dashboard built only from server availability while users are timing out.

7. Optional Prometheus/Grafana extension—clearly outside the mandatory lab

Prometheus, Grafana, exporters, OpenTelemetry collectors, and managed monitoring platforms can automate storage and visualization. They are useful extensions, but this chapter’s mandatory lab remains local/free and collector-neutral. If you deploy an exporter, verify its MariaDB permissions, query cost, scrape interval, cardinality, TLS/authentication, version support, and whether it exposes sensitive labels or SQL text.

8. Production judgment, chapter synthesis, and cleanup

Chapter 17 has built an evidence ladder: targeted Performance Schema instrumentation → sys convenience views → bounded logs → cross-layer database/OS correlation → service-level baselines and incident timelines. The next chapter turns those observations into controlled performance engineering. Tuning should begin only after you can measure the workload and verify whether a change improved the intended user outcome.

DROP DATABASE IF EXISTS servicehub17;