Encoding or Compression: Why not both?

Data compression and data encoding get thrown around as if they meant the same thing. And to be fair, if you go by the textbook definition of Shannon Entropy, they actually do. Both change the byte representation of input data to achieve a size reduction. But if you leave the text book behind and apply them in a system that stores and processes a lot of data, encoding and compression are two different tools that solve two different problems.

First, let’s have a look at both to see what justifies a distinction in practice.

Encoding: Lightweight and Data-Aware

Encoding schemes are narrow specialists. Each one is built around a specific, well-understood pattern in the data and therefore needs a good understanding of it. It can, e.g., detect a set of arbitrary values with few distinct ones, numbers clustered in a small range, or long runs of identical values. Because the scheme knows exactly what kind of redundancy it’s looking for, applying and reversing it is cheap, often just a handful of instructions per value. Cheap enough that many of these schemes can be vectorized with SIMD.

That narrowness has a second, more important consequence: because the transform is so simple and structured, you can often operate directly on the encoded representation without ever decompressing it. A filter can run as an integer comparison instead of a string comparison, or skip a whole block by comparing its bounds, all without materializing a single decoded value. That’s not a nice-to-have side effect, it’s the entire point. In contrast to compression, an encoding is not only used to shrink data, but to make data operations faster. It does not aim for the smallest size, but for the representation best for data processing.

As encodings only apply to one specific property of the data, you need a big toolbox to use them meaningfully. The ones we implement in CedarDB include dictionary encoding, single-value encoding, frame-of-reference (FOR), and truncation (dropping unused high-order bytes of an integer)¹, and we pick the one that best fits the data whenever we transform a set of cooled values to our analytics-optimized layout. Why picking the right one matters becomes clear when looking at the schemes in detail.

Dictionary Encoding

A widely used and easy to understand encoding is dictionary encoding, which we already covered in our post on string compression². Dictionary encoding replaces every value with a small fixed-width integer key that points into a table of the distinct values actually present in the column. This does not only reduce the size of the data, as each distinct string is only stored once and each occurrence replaced by a 1 to 3 byte integer. It also allows for comparisons of string values based on their integer keys, so a costly string comparison turns into a cheap integer comparison instead.

An Example of Dictionary Encoding.

High-Level Overview of Dictionary Encoding.

If you not only assign any key to strings, but do so in string-sorted order, you can even answer inequality comparisons or sort entire string arrays by their integer keys. While this adds additional overhead during encoding, it only needs to sort the unique strings and will pay off quickly. However, this is not change friendly as a new value in the middle will invalidate all keys behind it, requiring a re-encoding of the entire block, so this is only worth doing for truly cold data.

Frame-of-Reference Encoding

While dictionary encoding is great when a column has few distinct values, it starts falling apart on something like a timestamp or order ID column where every value can be different. Luckily, a large number of distinct values does not automatically mean more entropy, and we can utilize a different pattern in the data instead. While the domain of timestamps is huge and spans most of Earth’s past and future history, the values you see in practice are often much closer together. Frame-of-reference (FOR) encoding targets exactly this pattern: values that are almost all distinct, but are clustered tightly relative to each other. Think of a column that stores timestamps scattered throughout the last year. Without encoding, each one needs 8 bytes to store its timestamp as an absolute value. However, the differences between them and their minimum are small enough to fit in far fewer bytes.

An Example of Frame-Of-Reference Encoding.

High-Level Overview of FOR Encoding.

For FOR encoding, one can store the minimum value once in the column’s header and then every value as a fixed-width delta from that minimum, so a value that would cost 8 bytes raw might cost only 2 or 3 bytes encoded. CedarDB actually goes a level further and subdivides a column into smaller sets of about a thousand values each, each with its own local minimum and byte width. This helps prevent a handful of outliers from forcing the entire column into a wider representation and instead keeps their impact localized. Furhter, if the data is at least roughly sorted by the encoded value, it can allow for using even smaller deltas. Because the reference value is stored once and the deltas are fixed-width, decoding is just “add the minimum back,” which vectorizes trivially, and a range filter can check a block’s min/max header before touching a single value, skipping the block entirely if it can’t possibly match.

Each encoding relies on one specific property of the data distribution, and, as we’ve seen on the timestamp example, an encoding that is fitting for one distribution might not work at all for a different one.

In contrast, compression algorithms like zstd, LZ4, or gzip couldn’t care less what your data means. They don’t know if they’re looking at a column of timestamps, a paragraph of English text, or a JPEG. They operate on raw bytes and find redundancy statistically, through techniques like LZ77-style back-references and entropy coding, rather than through knowledge of a specific data type’s structure. That generality is the whole selling point. A single compressor works on anything because it isn’t restricted to one narrow pattern. Instead, it can often find and eliminate redundancy that a type-specific encoding leaves on the table entirely, such as cross-value patterns, repeated substrings and skewed byte-value distributions.

The cost of that generality shows up at read time. A general-purpose compressor produces an opaque block of bytes. You cannot do binary search on the data, there is no per-value random access and no comparing two values without decoding both of them first. To read anything out of a compressed block, you decompress the whole block back to its original bytes and then operate on that. That’s a perfectly fine trade-off when you’re reading a file end to end, but it’s a costly when a query only needs to check one predicate against a hundred out of a million values in that block.

Trade-Offs

Talk is cheap, so we ran the numbers. We compared the encoding schemes above against zstd on synthetic data in standalone C++ experiments, then compared the impact of encoding and compression on CedarDB against the ClickBench dataset.

Standalone: Dictionary vs. zstd on a String Column

We generated a column of 300 distinct URL paths (16-45 bytes each, sharing locale prefixes and category words), duplicated with a Zipfian skew across 2 million rows, and compared plain dictionary encoding against zstd, including nesting both by zstd-compressing the dictionary’s integer ID array.

Representation	Size	Ratio vs. raw
Raw (length-prefixed strings)	54.0 MB	1.00x
Dictionary encoding	4.0 MB	13.47x
zstd (level 3)	5.7 MB	9.45x
zstd (level 19)	2.8 MB	19.64x
Dictionary + zstd on the ID array (layered)	2.2 MB	24.77x

Zstd alone beats plain dictionary encoding on ratio on one of its highest levels, because it’s finding byte-level redundancy that a fixed-width integer substitution can’t. But layering zstd on top of the dictionary’s already-narrow ID array beats even zstd-19-on-raw, because a column of small fixed-width integers is denser, more regular input than variable-length text. The key idea here is to treat them as separate tools: encode first, then compress the (now much smaller and more regular) result.

The size alone undersells the real story. Let’s look at a simple equality filter (WHERE path = '...'), matching ~300k rows:

Method	Time
Dictionary-encoded	0.4 ms
Zstd-compressed	64.1 ms

168x. Not because zstd’s decompressor is slow in absolute terms, it decompresses the whole 54 MB column in about 40 ms, but because a filter against zstd-compressed data has no choice but to fully materialize the column before it can compare a single value. In contrast, the dictionary-encoded filter never needs to leave the encoded representation.

Standalone: Frame-of-Reference vs. zstd on a Numeric Column

For frame-of-reference, we’ll use 5 million int64 values simulating a timestamp column. The values overall trend upwards, with a slight jitter applied for randomness. Despite both, all values remain within a 500k-wide range. This results in about 19 bits of entropy out of the 64 bits used for each value.

Representation	Size	Ratio vs. raw
Raw `int64` array	40.0 MB	1.00x
Frame-of-reference	20.0 MB	2.00x
zstd (level 3)	10.4 MB	3.84x
zstd (level 19)	8.9 MB	4.52x
FOR + zstd on the deltas (layered)	9.5 MB	4.20x

Let’s again look at a simple filter query (value BETWEEN ...) matching ~1% of rows:

Method	Time
Frame-of-reference	3.2 ms
Zstd-compressed	56.5 ms

17.8x. Smaller than for dictionary encoding, but the trend is the same: an encoding you can filter in place beats one you have to fully unpack first, regardless of who wins on raw bytes.

Compressing data with zstd isn’t free either. Compressing either column at zstd level 19 took 32.6 seconds for the string column and 19.1 seconds for the numeric one, several orders of magnitude slower than encoding, which stayed below 30ms for both.

The Real Thing: CedarDB on ClickBench

While synthetic benchmarks are nice to drill down on specifics, what matters is the effect in practice. We’ll use ClickBench’s hits table, a ~100M-row table of semi-real web analytics events, and compare encoding and compression in CedarDB. We load the data twice, once without zstd compression and once with. The cedardb_compression_infos system view gives us an insight on how each column is compressed and encoded. All four columns below get the identical lightweight encoding regardless of whether zstd runs afterward, so what changes is purely compression_ratio, the extra layer zstd adds on top:

Column	Chosen Encoding	Encoding Ratio	Compression Ratio	Total Ratio
`watchid`	uncompressed	1.00x	1.00x	1.00x
`isrefresh`	truncate	2.00x	15.6x	31.2x
`useragentminor`	sorted-string dictionary	15.7x	4.80x	75.4x
`counterid`	FOR/truncate/dictionary	4.95x	63.0x	312.0x

For watchid, a near-unique 64-bit ID, neither layer finds anything to work with, and neither encoding nor compression is used. Some data just has high entropy. isrefresh, a heavily skewed flag column, sits at the other end: encoding already achieves a 2x reduction by truncating to a narrower type, but zstd’s entropy coding finds another 15.6x on top of that. counterid is the most interesting, as it chooses three different encodings for different sections of the column. And it shows a clear advantage of using a second-level compressor such as zstd. While dictionary encoding, e.g., can reduce the larger counter to a small fixed-width ID, it can’t do anything about runs of that same ID repeating across many consecutive rows. Encoding can only apply one method at a time to allow efficient operations on the encoded data. Zstd, however, can exploit this additional redundancy, which is where its extra 63x on top of the encoding’s own 4.95x comes from. Across the whole table, that combination shrinks hits from the 21.4 GiB encoded size (compression=none) to 7.88 GiB (compression=zstd), a 2.72x reduction. And the encoding size is already significantly smaller than the 75.56 GiB CSV, which means compression and encoding combined can achieve a 9.59x total size reduction.

And the beauty of it is that compression has no impact on the hot path of data processing, as once the data is in memory, the representation is identical whether data was compressed on disk or not. Compression’s only performance touchpoint is loading cold, non-buffer-resident data from disk. That cost depends on a lot of external factors, such as the number of cores, the throughput of the disk used, and the compression rate. Zstd’s own published benchmarks put single-core decompression at roughly 1.5-2 GB/s, so unpacking even a large batch of touched pages costs single-digit milliseconds. Overall the impact can range from a small penalty for machines with few cores but fast SSDs to a performance improvement for big machines with slow disks. Combined with the cost savings for less storage consumption, there is little downside to using zstd on top of an encoding if it leads to a significant reduction in storage size. For CedarDB, we employ zstd compression if it will further reduce the encoded data on disk by at least 20%.

The Short Version

	Encoding	Compression
Data-type aware	Yes	No
Typical ratio	Good, pattern-specific	Better, general
Cost to apply	Very low	Low to very high
Queryable without decode	Yes	No
Best fit	Hot, query-touched data	Cold storage, transfer

Why Not Both?

Given all that, the trade-off isn’t really a trade-off. Encoding and compression aren’t competing for the same job. They’re complementing each other.

How we handle things here at CedarDB is to always keep data encoded, compress only for storage. Every column always gets the lightweight encoding applied wherever we can, because that’s what the query engine operates on directly. With encoding, a reduction in size is just one of the benefits, as it allows more data to be resident in memory, but it is not the only one. Zstd is layered on top of that encoded representation purely for on-disk storage, and only when it’s actually worth it. That is, if it significantly reduces the size on disk.

So what can you take away? Encode always, compress only where it earns its CPU cost, and don’t see the two as competitors. Instead, let the two layers do the part they’re actually good at.

Want to see how well your data compresses in a modern database system? Give CedarDB a try.

Start Now

Appendix

The code below allows you to reproduce the two microbenchmarks.

Dictionary vs. zstd on a String Column

Show Code

dict_vs_zstd.cpp

// dict_vs_zstd.cpp
//
// Self-contained experiment comparing dictionary encoding against
// zstd compression on a synthetic categorical string column of website urls
//
// Build:
//   g++ -O2 -std=c++20 dict_vs_zstd.cpp -o dict_vs_zstd -lzstd
//
// Run:
//   ./dict_vs_zstd
//
#include <zstd.h>

#include <algorithm>
#include <chrono>
#include <cmath>
#include <cstdint>
#include <cstdio>
#include <cstring>
#include <random>
#include <string>
#include <string_view>
#include <vector>

using Clock = std::chrono::steady_clock;
static double ms_since(Clock::time_point t0) {
    return std::chrono::duration<double, std::milli>(Clock::now() - t0).count();
}

// A single global RNG seeded with a fixed value so the whole experiment
static std::mt19937 rng(42);

// ---------------------------------------------------------------------------
// Step 1: build ~300 distinct URL-path strings with realistic shared
// structure (locale prefix + category words + fixed suffix), lengths roughly
// in the 10-60 byte range.
// ---------------------------------------------------------------------------
static std::vector<std::string> build_distinct_values(size_t target_count) {
    // Locale prefixes: shared substrings across many rows (like "/us/", "/de/").
    static const std::vector<std::string> locales = {
        "us", "uk", "de", "fr", "jp", "cn", "in", "br", "au", "ca"
    };
    // Category words of varying length so concatenations span ~10-60 bytes
    static const std::vector<std::string> categories = {
        "tv", "phones", "books", "toys", "music", "tools", "office", "health",
        "beauty", "games", "home", "kids", "shoes", "sports",
        "outdoor-camping-gear", "kitchen-and-dining", "automotive-parts",
        "womens-fashion-clothing", "mens-fashion-clothing",
        "electronics-and-computers", "garden-and-patio-furniture",
        "baby-and-toddler-supplies", "pet-supplies-and-accessories",
        "movies-and-tv-shows", "software-and-video-games",
        "arts-crafts-and-sewing", "musical-instruments", "office-products",
        "industrial-and-scientific", "collectibles-and-fine-art"
    };
    // Occasionally vary the trailing "page" so not every path ends the same way
    static const std::vector<std::string> suffixes = {
        "index.html", "landing.html", "page.html"
    };

    std::vector<std::string> values;
    values.reserve(target_count);
    std::vector<std::string> shuffled_locales = locales;
    std::vector<std::string> shuffled_categories = categories;

    // Deterministically enumerate locale x category combinations (10 x 30 = 300)
    std::uniform_int_distribution<size_t> suffix_pick(0, suffixes.size() - 1);
    for (const auto& loc : locales) {
        for (const auto& cat : categories) {
            if (values.size() >= target_count) break;
            std::string s = "/" + loc + "/" + cat + "/" + suffixes[suffix_pick(rng)];
            values.push_back(std::move(s));
        }
    }
    return values;
}

// ---------------------------------------------------------------------------
// Step 2: Zipf sampler over `n` ranks (rank 0 = most frequent)
// ---------------------------------------------------------------------------
struct ZipfSampler {
    std::vector<double> cumulative; // cumulative probability per rank
    std::uniform_real_distribution<double> unif{0.0, 1.0};

    explicit ZipfSampler(size_t n, double s = 1.0) {
        std::vector<double> weights(n);
        double sum = 0.0;
        for (size_t r = 0; r < n; ++r) {
            weights[r] = 1.0 / std::pow(static_cast<double>(r + 1), s);
            sum += weights[r];
        }
        cumulative.resize(n);
        double running = 0.0;
        for (size_t r = 0; r < n; ++r) {
            running += weights[r] / sum;
            cumulative[r] = running;
        }
    }

    size_t sample(std::mt19937& gen) {
        double u = unif(gen);
        auto it = std::lower_bound(cumulative.begin(), cumulative.end(), u);
        return static_cast<size_t>(it - cumulative.begin());
    }
};

// A no-op sink to prevent the optimizer from eliding work whose result we never use
static volatile uint64_t g_sink = 0;

int main() {
    constexpr size_t kDistinctValues = 300;
    constexpr size_t kNumRows = 2'000'000;
    // Take average of 5 runs
    constexpr int kRepeats = 5;

    // ---- Build dictionary values ----
    std::vector<std::string> values = build_distinct_values(kDistinctValues);
    printf("Built %zu distinct values (target %zu)\n", values.size(), kDistinctValues);
    size_t min_len = 1e9, max_len = 0;
    for (auto& v : values) { min_len = std::min(min_len, v.size()); max_len = std::max(max_len, v.size()); }
    printf("Value length range: %zu - %zu bytes\n", min_len, max_len);

    // ---- Sorted dictionary ----
    std::vector<std::string> dict_sorted = values;
    std::sort(dict_sorted.begin(), dict_sorted.end());

    // Map from insertion-order index -> sorted-dictionary ID
    std::vector<uint16_t> orig_to_sorted_id(values.size());
    for (size_t i = 0; i < values.size(); ++i) {
        auto it = std::lower_bound(dict_sorted.begin(), dict_sorted.end(), values[i]);
        orig_to_sorted_id[i] = static_cast<uint16_t>(it - dict_sorted.begin());
    }

    // ---- Sample 2M row assignments with a Zipf skew over insertion order ----
    ZipfSampler zipf(kDistinctValues, /*s=*/1.0);
    std::vector<uint16_t> row_ids(kNumRows);
    for (size_t i = 0; i < kNumRows; ++i) {
        size_t orig_idx = zipf.sample(rng);
        row_ids[i] = orig_to_sorted_id[orig_idx];
    }

    // =========================================================================
    // Representation 1
    // RAW - concatenated strings, each with a uint16_t length prefix
    // =========================================================================
    // Precompute per-sorted-ID string_views to avoid repeated hashing/lookup.
    std::vector<std::string_view> id_to_str(dict_sorted.size());
    for (size_t i = 0; i < dict_sorted.size(); ++i) id_to_str[i] = dict_sorted[i];

    size_t raw_size = 0;
    for (uint16_t id : row_ids) raw_size += 2 + id_to_str[id].size();

    std::vector<char> raw_blob;
    raw_blob.reserve(raw_size);
    for (uint16_t id : row_ids) {
        std::string_view s = id_to_str[id];
        uint16_t len = static_cast<uint16_t>(s.size());
        raw_blob.insert(raw_blob.end(), reinterpret_cast<char*>(&len), reinterpret_cast<char*>(&len) + 2);
        raw_blob.insert(raw_blob.end(), s.begin(), s.end());
    }
    printf("Raw blob built: %zu bytes\n", raw_blob.size());

    // =========================================================================
    // Representation 2
    // DICTIONARY ENCODING
    // =========================================================================
    std::vector<char> dict_blob;
    std::vector<uint32_t> dict_offsets;
    dict_offsets.reserve(dict_sorted.size() + 1);
    {
        uint32_t off = 0;
        for (auto& s : dict_sorted) {
            dict_offsets.push_back(off);
            dict_blob.insert(dict_blob.end(), s.begin(), s.end());
            off += static_cast<uint32_t>(s.size());
        }
        dict_offsets.push_back(off); // end offset
    }
    size_t dict_blob_size = dict_blob.size();
    size_t dict_offsets_size = dict_offsets.size() * sizeof(uint32_t);
    size_t id_array_size = row_ids.size() * sizeof(uint16_t); // = kNumRows * 2

    size_t dict_encoding_total = dict_blob_size + dict_offsets_size + id_array_size;

    // Timed: (re)build the ID array from scratch
    double build_dict_encoding_ms = 0;
    {
        std::vector<uint16_t> tmp(kNumRows);
        for (int r = 0; r < kRepeats; ++r) {
            auto t0 = Clock::now();
            for (size_t i = 0; i < kNumRows; ++i) {
                tmp[i] = row_ids[i];
            }
            build_dict_encoding_ms += ms_since(t0);
        }
        build_dict_encoding_ms /= kRepeats;
        g_sink += tmp[0];
    }

    // =========================================================================
    // Representation 3: ZSTD on raw blob, level 3 and level 19
    // =========================================================================
    size_t bound = ZSTD_compressBound(raw_blob.size());
    std::vector<char> zstd_raw_l3(bound), zstd_raw_l19(bound);

    double zstd_raw_l3_encode_ms = 0;
    size_t zstd_raw_l3_size = 0;
    for (int r = 0; r < kRepeats; ++r) {
        auto t0 = Clock::now();
        zstd_raw_l3_size = ZSTD_compress(zstd_raw_l3.data(), bound, raw_blob.data(), raw_blob.size(), 3);
        zstd_raw_l3_encode_ms += ms_since(t0);
    }
    zstd_raw_l3_encode_ms /= kRepeats;

    double zstd_raw_l19_encode_ms = 0;
    size_t zstd_raw_l19_size = 0;
    {
        // Level 19 is slow; a single measurement is enough (not repeated 5x).
        auto t0 = Clock::now();
        zstd_raw_l19_size = ZSTD_compress(zstd_raw_l19.data(), bound, raw_blob.data(), raw_blob.size(), 19);
        zstd_raw_l19_encode_ms = ms_since(t0);
    }
    zstd_raw_l3.resize(zstd_raw_l3_size);
    zstd_raw_l19.resize(zstd_raw_l19_size);

    // Full decompression of the zstd(raw) stream back into row strings.
    double zstd_raw_decode_ms = 0;
    std::vector<char> decompressed(raw_blob.size());
    for (int r = 0; r < kRepeats; ++r) {
        auto t0 = Clock::now();
        size_t rc = ZSTD_decompress(decompressed.data(), decompressed.size(), zstd_raw_l3.data(), zstd_raw_l3.size());
        std::vector<std::string_view> rows;
        rows.reserve(kNumRows);
        size_t pos = 0;
        while (pos + 2 <= rc) {
            uint16_t len; std::memcpy(&len, decompressed.data() + pos, 2);
            pos += 2;
            rows.emplace_back(decompressed.data() + pos, len);
            pos += len;
        }
        zstd_raw_decode_ms += ms_since(t0);
        g_sink += rows.size();
    }
    zstd_raw_decode_ms /= kRepeats;

    // =========================================================================
    // Representation 4
    // ZSTD on top of the dictionary encoding
    // =========================================================================
    std::vector<char> id_bytes(id_array_size);
    std::memcpy(id_bytes.data(), row_ids.data(), id_array_size);
    size_t id_bound = ZSTD_compressBound(id_bytes.size());
    std::vector<char> zstd_ids_l3(id_bound);
    size_t zstd_ids_l3_size = ZSTD_compress(zstd_ids_l3.data(), id_bound, id_bytes.data(), id_bytes.size(), 3);
    zstd_ids_l3.resize(zstd_ids_l3_size);
    size_t layered_total = dict_blob_size + dict_offsets_size + zstd_ids_l3_size;

    // Decode dictionary encoding back to strings
    double dict_decode_ms = 0;
    for (int r = 0; r < kRepeats; ++r) {
        auto t0 = Clock::now();
        std::vector<std::string_view> rows;
        rows.reserve(kNumRows);
        for (size_t i = 0; i < kNumRows; ++i) {
            uint16_t id = row_ids[i];
            uint32_t start = dict_offsets[id];
            uint32_t end = dict_offsets[id + 1];
            rows.emplace_back(dict_blob.data() + start, end - start);
        }
        dict_decode_ms += ms_since(t0);
        g_sink += rows.size();
    }
    dict_decode_ms /= kRepeats;

    // =========================================================================
    // Simulated equality-filter query: "find all rows equal to X"
    // =========================================================================
    std::string X = dict_sorted[orig_to_sorted_id[0]]; // sorted-ID of insertion-order rank 0
    uint16_t X_id = orig_to_sorted_id[0];

    // (a) Dictionary-encoded path
    double dict_filter_ms = 0;
    size_t dict_matches = 0;
    for (int r = 0; r < kRepeats; ++r) {
        auto t0 = Clock::now();
        auto it = std::lower_bound(dict_sorted.begin(), dict_sorted.end(), X);
        uint16_t target_id = static_cast<uint16_t>(it - dict_sorted.begin());
        (void)X_id;
        size_t count = 0;
        for (size_t i = 0; i < kNumRows; ++i) {
            count += (row_ids[i] == target_id) ? 1 : 0;
        }
        dict_filter_ms += ms_since(t0);
        dict_matches = count;
    }
    dict_filter_ms /= kRepeats;

    // (b) Zstd-compressed path
    double zstd_filter_ms = 0;
    size_t zstd_matches = 0;
    for (int r = 0; r < kRepeats; ++r) {
        auto t0 = Clock::now();
        std::vector<char> buf(raw_blob.size());
        size_t rc = ZSTD_decompress(buf.data(), buf.size(), zstd_raw_l3.data(), zstd_raw_l3.size());
        size_t count = 0;
        size_t pos = 0;
        while (pos + 2 <= rc) {
            uint16_t len; std::memcpy(&len, buf.data() + pos, 2);
            pos += 2;
            std::string_view row_str(buf.data() + pos, len);
            pos += len;
            count += (row_str == X) ? 1 : 0;
        }
        zstd_filter_ms += ms_since(t0);
        zstd_matches = count;
    }
    zstd_filter_ms /= kRepeats;

    if (dict_matches != zstd_matches) {
        printf("WARNING: match count mismatch! dict=%zu zstd=%zu\n", dict_matches, zstd_matches);
    }

    // =========================================================================
    // Results
    // =========================================================================
    auto ratio = [&](size_t sz) { return static_cast<double>(raw_size) / static_cast<double>(sz); };

    printf("\n================ SIZES ================\n");
    printf("Raw (length-prefixed strings):        %10zu bytes  (ratio 1.00x)\n", raw_size);
    printf("Dictionary encoding total:             %10zu bytes  (ratio %.2fx)\n", dict_encoding_total, ratio(dict_encoding_total));
    printf("  - dict blob (sorted distinct values): %10zu bytes\n", dict_blob_size);
    printf("  - dict offsets ((%zu+1) x u32):        %10zu bytes\n", dict_sorted.size(), dict_offsets_size);
    printf("  - id array (%zu x u16):          %10zu bytes\n", kNumRows, id_array_size);
    printf("Zstd(raw) level 3:                     %10zu bytes  (ratio %.2fx)\n", zstd_raw_l3_size, ratio(zstd_raw_l3_size));
    printf("Zstd(raw) level 19:                    %10zu bytes  (ratio %.2fx)\n", zstd_raw_l19_size, ratio(zstd_raw_l19_size));
    printf("Zstd(id array) level 3:                %10zu bytes  (ratio %.2fx)\n", zstd_ids_l3_size, ratio(zstd_ids_l3_size));
    printf("Layered total (dict + zstd(ids)):       %10zu bytes  (ratio %.2fx)\n", layered_total, ratio(layered_total));

    printf("\n================ TIMINGS (ms, avg of %d) ================\n", kRepeats);
    printf("Build dictionary ID array (encode):    %8.3f ms\n", build_dict_encoding_ms);
    printf("Zstd(raw) compress level 3 (encode):   %8.3f ms\n", zstd_raw_l3_encode_ms);
    printf("Zstd(raw) compress level 19 (encode):  %8.3f ms (single run)\n", zstd_raw_l19_encode_ms);
    printf("Zstd(raw) decompress to strings:       %8.3f ms\n", zstd_raw_decode_ms);
    printf("Dictionary decode to strings:           %8.3f ms\n", dict_decode_ms);
    printf("\nEquality filter for X = \"%s\" (%zu matches):\n", X.c_str(), dict_matches);
    printf("  Dictionary-encoded (binary search + int scan): %8.3f ms\n", dict_filter_ms);
    printf("  Zstd-compressed (decompress + string scan):    %8.3f ms\n", zstd_filter_ms);
    printf("  Ratio (zstd / dict):                            %8.2fx\n", zstd_filter_ms / dict_filter_ms);

    printf("\n(sink=%lu, to prevent dead-code elimination)\n", (unsigned long)g_sink);
    return 0;
}

Frame-of-Reference vs. zstd on a Numeric Column