Chapter 18 · Columnar Storage, Compression, Encoding, Vectorized Execution, and Analytical Scan Economics
Column Encodings, Dictionary/Run-Length/Delta-Like Compression Concepts and Data Distribution
Connect dictionary, run-length, delta-like, and general compression choices to actual column distributions, and prove why encoding effectiveness is a property of data plus implementation rather than a universal codec ranking.
Learning outcomes
Explain the central warehouse mechanism in “Column Encodings, Dictionary/Run-Length/Delta-Like Compression Concepts and Data Distribution” and connect it to AtlasMart’s declared grain and governed metrics.
Keep logical correctness, history, and control totals unchanged while evaluating the lesson’s physical or operational choice.
Run or interpret the deterministic local evidence and distinguish what it proves from engine-, cache-, scale-, or cloud-dependent behavior.
Diagnose the controlled failure, repair it safely, and state the production checks required before adopting the pattern.
The Chapter 15–17 canonical current-state controls remain 9 paid lines, 7 orders, 11 units, 740 USD GMV, 450 USD cost, and 290 USD gross profit. Chapter 18 expands those nine seed lines deterministically to 216,000 benchmark rows only to make storage mechanisms observable. Benchmark rows are never reported as new production facts.
Runtime: mandatory path uses Python 3 standard library; generation evidence below used Python 3.13.5 and SQLite 3.46.1. Storage: local filesystem only. Source: synthetic AtlasMart sales facts. Time: dates are UTC business dates; no time-zone conversion is hidden in the benchmark. Grain: one current paid order line. Security: synthetic identifiers only; no secrets or PII. History: the benchmark is a deterministic physical scale expansion, not a new business history. Cache: OS cache is not flushed; timing probes use two warm-ups and seven measured runs. Cost: no paid service is required.
1. Encoding is a representation decision driven by distribution
AtlasMart stores repeated channels (web,
mobile, sales), repeated business
dates, increasing identifiers, and monetary integers. An
encoding changes how logical values are
represented before or alongside general-purpose compression.
Compression reduces physical bytes while
preserving values. These are related but not interchangeable:
dictionary IDs, run lengths, or deltas may expose regularity
that a compressor can exploit, while a high-cardinality column
may gain little.
2. Dictionary, run-length, and delta-like mechanisms
| Mechanism | Good fit | AtlasMart example | Boundary |
|---|---|---|---|
| Dictionary | Repeated values from a relatively small domain | channel/product/status codes | A huge dictionary can erase savings and add decode work. |
| Run-length | Long adjacent runs of the same encoded value | date values after date-oriented clustering | Shuffling the same values destroys runs without changing cardinality. |
| Delta-like | Ordered numeric sequences with small differences | dates or monotonically increasing IDs | Large/irregular jumps need wider deltas or fallback encoding. |
| General compression | Repeated byte patterns across encoded payload | gzip in the mandatory lab | Codec ratio and CPU cost depend on implementation/data. |
3. Observable sizing from the 216,000-row fixture
The harness compares simple didactic encodings before gzip. If a channel code were stored as an ordinary 32-bit integer, 216,000 values require 864,000 bytes. A one-byte dictionary code plus the tiny fixture dictionary requires 216,017 bytes. The sorted business-date column also occupies 864,000 bytes as plain 32-bit values; 184 value runs fit the didactic RLE model in 1,472 bytes, while a first-value + 16-bit-delta representation uses 432,002 bytes.
Those numbers teach mechanism, not Parquet’s exact on-disk format. Real formats add page headers, null/definition levels, indexes, dictionaries, checksums, metadata, and codec-specific blocks.
4. Controlled failure: rank codecs without looking at data
“RLE is best for dates” is wrong if rows are randomly distributed. “Dictionary is best for strings” is wrong if nearly every string is unique or the implementation falls back after dictionary growth. A warehouse should collect cardinality, run-length, null, range, and value-width distributions and let the format/engine choose or validate encodings from evidence.
The clustered and shuffled Chapter 18 stores contain exactly the same logical dates. Their cardinality is identical, yet the clustered layout produces strong min/max locality and long date runs while the shuffled layout does not.
5. Hands-on: reproduce encoding-size arithmetic
rows = 216_000channels = [0, 1, 0, 2, 1, 0, 2, 1, 0] * 24_000# Plain 32-bit integer representation versus one-byte dictionary codes.plain_channel = rows * 4dictionary_payload = rows * 1 + 17 # 17 bytes for the tiny fixture dictionary# The full Chapter 18 harness sorts business dates and observes 184 runs.plain_date = rows * 4rle_date = 184 * (4 + 4) # value + run length# Didactic delta-like representation: first int32, then signed int16 deltas.delta_like = 4 + (rows - 1) * 2print(plain_channel, dictionary_payload)print(plain_date, rle_date, delta_like)assert (plain_channel, dictionary_payload) == (864000, 216017)assert (plain_date, rle_date, delta_like) == (864000, 1472, 432002)
Expected output is 864000 216017 and
864000 1472 432002. This is explicit calculation,
not an invented compression percentage. A different real codec
is free to produce different physical bytes.
6. Encoding, CPU, and updates are a three-way tradeoff
Smaller encoded payloads reduce I/O and often improve cache residency, but decode work consumes CPU. A compressed analytical segment that is rewritten in batches can tolerate heavier encoding than a structure updated row-by-row. Production choice therefore includes ingestion throughput, compaction/rewrite behavior, backfill cost, memory pressure, and whether the engine can execute on encoded values without fully materializing them.
7. Data quality and compatibility boundaries
Encoding cannot repair malformed units, wrong currencies, or
semantic code drift. Chapter 12–13 contracts still decide
whether mobile and MOBILE are
equivalent before encoding. For compatibility, test actual
readers: the Parquet specification defines encodings, but
implementation support can lag newer encodings. A migration must
be readable by every intended consumer before old files are
retired.
8. Production judgment and bridge
Prefer encodings that fit measured distributions and supported readers, then verify size, decode CPU, and replayability. Keep raw/governed inputs so encoded files are rebuildable. Lesson 3 moves from “how values occupy bytes” to “which chunks the query can avoid reading at all.”
Knowledge check
Check your understanding
- What does dictionary encoding replace logical values with?
- Why can shuffling hurt RLE without changing cardinality?
- Why is smaller storage not automatically lower query CPU?
- What must be verified before enabling a newer format encoding?
- Can encoding fix semantic code drift?
Review the answers
1. Compact identifiers that reference a dictionary of distinct values.
2. It breaks adjacent repeated runs.
3. Decode work may increase even when fewer bytes are read.
4. Reader/writer compatibility across the actual consumer ecosystem.
5. No; semantic normalization belongs to governed transforms/data quality.
Authoritative references
- Apache Parquet — ConceptsOfficial terminology for row groups, column chunks, pages, and the units at which I/O and encoding occur.
- Apache Parquet — File FormatOfficial layout showing column chunks organized inside row groups and metadata used to locate relevant chunks.
- Apache Parquet — EncodingsOfficial definitions of plain, dictionary, run-length/bit-packed, and delta encodings. Parquet is a reference, not a prerequisite for the mandatory lab.
- DuckDB — Execution FormatCurrent official example of a vectorized analytical engine. This course uses the page only as a non-prerequisite reference for the execution concept.
- SQLite — EXPLAIN QUERY PLANOfficial documentation for the local row-store plan evidence. The plan text is diagnostic output, not a stable application API.
- Python — gzipStandard-library compression used by the dependency-free local storage harness.
- Python — structStandard-library binary packing used for fixed-width didactic column chunks.
- Python — sqlite3Standard-library SQLite interface used for the local row-oriented table and query-plan evidence.
- Kimball Group — Dimensional Modeling TechniquesBackground for keeping dimensional grain and metric semantics stable while changing physical storage and access paths.