Chapter 22 · Observability and Diagnostics: AWR, ASH, ADDM, Wait Events, and Tracing

Wait Interface, Time Model Statistics, DB Time, DB CPU, and Bottleneck Classification

Measure DB time, DB CPU, non-idle waits and throughput as interval deltas; classify bottlenecks from correlated evidence rather than blaming the largest lifetime wait counter.

Advanced120–140 minutesTime-model/wait delta + throughput labDB time is concurrency-weighted, not wall timeNo AWR/ASH requiredLast reviewed: August 2026

Learning outcomes

During one five-minute outage, the largest cumulative wait in the instance is db file sequential read. The team buys faster storage, but the next outage is unchanged because the current problem was actually row-lock contention. Oracle's wait interface and time model turn “slow” into measured interval accounting. The important unit is the same incident window and the same workload volume.

01

Define wait event, wait class, DB time, DB CPU and nested time-model statistics.

02

Capture V$SYS_TIME_MODEL/V$SYSTEM_EVENT/V$SYSSTAT snapshots and compute deltas over one bounded interval.

03

Explain why DB time can exceed wall-clock time and why DB CPU is only one component of DB time.

04

Correlate non-idle wait seconds with commits/executions/logical reads instead of using cumulative rank alone.

05

Classify CPU, I/O, commit, concurrency/lock and application/network hypotheses without claiming a wait-name diagnosis is causality.

Generation-time baseline, licensing, scope, and evidence boundary

Mandatory examples target Oracle AI Database Free 26ai, reviewed against RU 23.26.3, SQL Developer 26.2, and SQLcl 26.2.1. Free is limited to 2 foreground CPU cores, 2 GB combined SGA/PGA memory, 12 GB user data, and one installation per logical environment; Oracle provides no Release Update patches or Support service requests for Free. The course CDB/PDB baseline is FREE/FREEPDB1. Core dynamic performance views, DBMS_XPLAN runtime plans, DBMS_MONITOR SQL trace and TKPROF are used as the pack-independent diagnostic lane. Current 26ai licensing includes Oracle Diagnostics Pack and Oracle Tuning Pack in Free, but each is an extra-cost pack on EE/EE-ES. AWR, ADDM and V$ACTIVE_SESSION_HISTORY belong to Diagnostics Pack; Real-Time SQL/PLSQL Monitoring belongs to Tuning Pack and also requires Diagnostics Pack. CONTROL_MANAGEMENT_PACK_ACCESS defaults to DIAGNOSTIC+TUNING on both Free and Enterprise Edition, so a value of DIAGNOSTIC+TUNING proves functional enablement—not that an EE/EE-ES customer purchased the packs. Never query AWR/ASH/ADDM/SQL Monitor merely because the views exist on an EE/EE-ES system whose entitlement is unknown.

1. Wait events describe where a foreground session cannot continue

A wait event names a reason a session yields/waits, such as a single-block read, log-file sync, row-lock enqueue or client message. A wait class groups events into categories such as User I/O, Commit, Concurrency, Application and Network. Idle waits must be separated from workload delays.

2. DB time is foreground database-call time

DB time is the sum of time foreground sessions spend executing database calls. It roughly consists of DB CPU plus non-idle foreground wait time. Because many sessions run concurrently, 10 sessions each spending one second inside the database can produce roughly 10 DB seconds in one wall-clock second. DB time is therefore a workload-consumption measure, not elapsed clock time.

sql · current cumulative time model
SELECT stat_name,valueFROM v$sys_time_modelWHERE stat_name IN (  'DB time',  'DB CPU',  'sql execute elapsed time',  'parse time elapsed',  'PL/SQL execution elapsed time')ORDER BY stat_name;

VALUE is in microseconds. Time-model statistics overlap/nest—sql execute elapsed time is part of DB time—so do not sum every row into a total.

3. Build a pack-independent snapshot table

sql · diagnostic admin setup
BEGIN  EXECUTE IMMEDIATE 'DROP TABLE sh22_metric_snap PURGE';EXCEPTION WHEN OTHERS THEN  IF SQLCODE != -942 THEN RAISE; END IF;END;/CREATE TABLE sh22_metric_snap (  snap_label   VARCHAR2(8) NOT NULL,  captured_at  TIMESTAMP WITH TIME ZONE NOT NULL,  metric_type  VARCHAR2(12) NOT NULL,  metric_name  VARCHAR2(128) NOT NULL,  metric_value NUMBER NOT NULL);
sql · capture T1
INSERT INTO sh22_metric_snapSELECT 'T1',SYSTIMESTAMP,'TIME',stat_name,valueFROM v$sys_time_modelWHERE stat_name IN ('DB time','DB CPU');INSERT INTO sh22_metric_snapSELECT 'T1',SYSTIMESTAMP,'WAIT',event,time_waited_microFROM v$system_eventWHERE wait_class <> 'Idle';INSERT INTO sh22_metric_snapSELECT 'T1',SYSTIMESTAMP,'STAT',name,valueFROM v$sysstatWHERE name IN (  'user commits',  'execute count',  'session logical reads',  'physical reads');COMMIT;

4. Generate a small repeatable ServiceHub workload

sql · workload table as SERVICEHUB_OWNER
BEGIN  EXECUTE IMMEDIATE 'DROP TABLE sh22_time_work PURGE';EXCEPTION WHEN OTHERS THEN  IF SQLCODE != -942 THEN RAISE; END IF;END;/CREATE TABLE sh22_time_work ASSELECT  LEVEL AS id,  MOD(LEVEL,1000) AS technician_id,  MOD(LEVEL*17,10000)/10 AS amount,  RPAD('servicehub',80,'x') AS payloadFROM dualCONNECT BY LEVEL <= 100000;CREATE INDEX sh22_time_tech_ixON sh22_time_work(technician_id);BEGIN  DBMS_STATS.GATHER_TABLE_STATS(USER,'SH22_TIME_WORK',cascade=>TRUE);END;/SELECT /*+ gather_plan_statistics */       SUM(amount)FROM sh22_time_workWHERE technician_id BETWEEN 100 AND 700;SELECT *FROM TABLE(  DBMS_XPLAN.DISPLAY_CURSOR(    NULL,NULL,'ALLSTATS LAST +PREDICATE'  ));

The point is to create observable database work, not a performance benchmark. Run the query several times if the interval is otherwise too quiet.

5. Capture T2 and calculate deltas

sql · capture T2
INSERT INTO sh22_metric_snapSELECT 'T2',SYSTIMESTAMP,'TIME',stat_name,valueFROM v$sys_time_modelWHERE stat_name IN ('DB time','DB CPU');INSERT INTO sh22_metric_snapSELECT 'T2',SYSTIMESTAMP,'WAIT',event,time_waited_microFROM v$system_eventWHERE wait_class <> 'Idle';INSERT INTO sh22_metric_snapSELECT 'T2',SYSTIMESTAMP,'STAT',name,valueFROM v$sysstatWHERE name IN (  'user commits',  'execute count',  'session logical reads',  'physical reads');COMMIT;
sql · top interval deltas
SELECT *FROM (  SELECT    b.metric_type,    b.metric_name,    b.metric_value-a.metric_value AS delta_value  FROM sh22_metric_snap a  JOIN sh22_metric_snap b    ON b.metric_type=a.metric_type   AND b.metric_name=a.metric_name  WHERE a.snap_label='T1'    AND b.snap_label='T2'    AND b.metric_value>=a.metric_value  ORDER BY    CASE b.metric_type WHEN 'WAIT' THEN 1 WHEN 'TIME' THEN 2 ELSE 3 END,    delta_value DESC)FETCH FIRST 25 ROWS ONLY;

For TIME/WAIT, divide the microsecond delta by 1,000,000 for seconds. For throughput counters, keep the native count and divide by elapsed seconds only if the two snapshot timestamps define a clean interval.

6. Add a controlled concurrency wait

Reuse Lesson 1's two-session row-lock test for 10–20 seconds, then take another T2. The interval should now include enq: TX - row lock contention. That correlation—known injected blocker plus session evidence plus wait delta—is far stronger than a lifetime wait ranking.

sql · current non-idle wait deltas by class/event
SELECT  wait_class,  event,  total_waits,  time_waited_microFROM v$system_eventWHERE wait_class <> 'Idle'ORDER BY time_waited_micro DESCFETCH FIRST 20 ROWS ONLY;

7. Bottleneck classification is a hypothesis table

Evidence pattern Likely next question
DB CPU dominates DB time + OS CPU saturated Which SQL/parse/PLSQL consumes CPU? Is CPU capacity actually constrained?
User I/O waits rise with physical reads Which SQL/files/objects and access paths cause reads? Is storage service time abnormal?
Commit waits rise with commit rate Redo/log writer/storage/commit batching behavior?
Application/Concurrency enqueue waits Who blocks whom? Which transaction/object/business operation?
Network waits rise Database-to-client idle/chatty fetch behavior or actual network constraint?

A wait class names where time accumulated; it is not automatically the root cause. For example, User I/O can be caused by a bad plan reading too many blocks rather than slow storage.

8. Deliberately wrong: “DB CPU = 70 seconds in a 60-second interval means the clock is wrong”

DB CPU/DB time sum across concurrently active foreground sessions. On a multi-CPU/concurrent workload they can exceed wall elapsed time. The correct comparison is capacity/concurrency-aware: DB CPU seconds per wall second, OS CPU utilization/run queue, and throughput during the same interval.

9. Cleanup

sql · cleanup
DROP TABLE sh22_metric_snap PURGE;DROP TABLE sh22_time_work PURGE;

10. Production judgment

Always timestamp the interval and capture throughput with waits/time. Compare deltas to a known-good window at similar business load before changing storage, CPU, parameters or plans. Use wait names as navigational evidence, not causal proof. Preserve application/SLO/OS context alongside database counters.

The entire lesson is pack-independent and Free-compatible. No restart or COMPATIBLE change is required. Lesson 3 automates this interval-capture idea through AWR—but only after an explicit Diagnostics Pack entitlement check.

Check your understanding

  1. What does DB time measure?
  2. Can DB time exceed wall-clock elapsed time?
  3. Why must V$SYSTEM_EVENT be sampled twice?
  4. Does a User I/O wait prove the storage device is slow?
  5. Why capture throughput in the same interval as waits?
Review the answers

It sums foreground time spent executing database calls, including DB CPU and non-idle waits.

Yes. Concurrent sessions contribute time simultaneously.

Its counters are cumulative; deltas isolate the incident window.

No. Excess reads from an inefficient plan can create User I/O time even with healthy storage.

Wait time only has meaning relative to how much useful work the database completed.

Authoritative references

Keep knowledge open

Help the academy stay free and grow.

If these tutorials save you time, a small donation supports new lessons, technical review, diagrams, examples, and long-term maintenance.

ETHEthereum / ERC-20 only
0x716c4Ab160C4B66F31a28AE2448BfF68fc3a2ef0

Send only Ethereum or ERC-20 compatible assets to this address.