Markdown

Gorilla: in-memory time series database and its compression¶

Scope: how the monitoring-metrics store behind a large fleet is built, using the Facebook Gorilla paper (VLDB 2015) as the primary source. This page covers the lossless timestamp and float compression scheme, the 26-hour in-memory write-through cache design, the availability-over-consistency failure model, and which parts carry over to GPU fleet telemetry (DCGM-style counters at a fixed scrape interval). It is the storage-internals companion to the runnable stack in telemetry, monitoring and alerting; it is not a GPU paper, and its fit to this KB is as a design reference for the metrics backend.

The encoder and decoder below are a from-scratch reimplementation of the paper's Section 4.1 bit layout, executed and asserted on this page. The telemetry series are synthetic; the byte-per-point figures are properties of that synthetic data, not measurements of a real cluster or of Facebook's.

What it is¶

Gorilla is an in-memory time series database (TSDB) that Facebook built as a write-through cache in front of its HBase-backed Operational Data Store (ODS). Each datum is a string key, a 64-bit timestamp, and a double. The paper states the workload as 2 billion unique series and about 12 million points per second (over 1 trillion per day) as of spring 2015, held for the most recent 26 hours.

The contribution that survived in other systems is the compression. Per series, timestamps and values are encoded separately into bit-packed blocks covering two hours:

Timestamps store the delta of deltas. A series scraped every 60 s with a one-second jitter yields deltas 60, 60, 59, 61 and delta-of-deltas 0, -1, 2. A zero is one bit; otherwise a control prefix selects a 7, 9, 12, or 32 bit payload (table below).
Values are XORed with the previous value. Identical values cost one bit. Otherwise the leading and trailing zero counts of the XOR locate the meaningful bits, and if they fit inside the previous window only the window contents are stored.

Timestamp delta-of-delta `D`	Control bits	Payload bits	Total bits
0	`0`	0	1
-63 to 64	`10`	7	9
-255 to 256	`110`	9	12
-2047 to 2048	`1110`	12	16
otherwise	`1111`	32	36

XOR with previous value	Bits written
zero	`0`
meaningful bits inside the previous window	`10` + window contents
otherwise	`11` + 5 bits leading zeros + 6 bits length + meaningful bits

The paper reports 1.37 bytes per point on ODS data against 16 bytes raw, a 12x reduction, and that about 96% of timestamps compress to a single bit and about 51% of values to a single bit (the 30% and 19% remainder average 26.6 and 36.9 bits).

Why use it¶

Query latency is what changes user behaviour. The paper reports query volume growing from 450 to over 5,000 steady-state queries per second, peaking at 40,000, once reads became fast enough for interactive and automated use.
Memory becomes the tier. At 16 bytes per point, the paper computes 16 TB of RAM as impractical; at 1.37 bytes, 26 hours fit in 1.3 TB across 20 machines at launch (80 machines after two doublings).
Recent data is the valuable data. The paper states at least 85% of ODS queries targeted the past 26 hours, which is what justifies a cache-shaped design.
Cheap whole-dataset scans enable new tooling. The paper builds a brute-force Pearson correlation search over up to 1 million series and moves the roll-up job from map-reduce over HBase to a scan of Gorilla.

When to use it (and when not)¶

Use the ideas when the workload is many series at a fixed scrape interval with slowly changing or integer values: DCGM clocks, temperatures, utilization percentages, ECC counters, link state.
Expect little gain on continuously varying floating-point values. The executed block below shows power-draw-like floats staying near 6.8 bytes per point because their mantissa bits are effectively random; rounding to 0.1 W does not help, since a decimal value is not a short binary value.
Do not copy the failure model blindly. Gorilla accepts losing seconds of data on a crash and keeps two regional copies with no consistency attempt. That suits threshold alerting on aggregates; it does not suit billing or chargeback data.
Do not build one. For a GPU cluster the practical use is understanding the encoding inside the TSDB already in use. Prometheus' chunk encoder credits the same lineage, see below.

Architecture¶

flowchart LR
  W["Write clients"] -->|"stream to both regions"| A["Gorilla region A"]
  W --> B["Gorilla region B"]
  A --> H["HBase: long-term store"]
  B --> H
  Q["Query clients"] -->|"closest healthy region"| A
  Q -.->|"partial or failed: retry"| B
  A --> D["GlusterFS: logs + 2 h block files"]

Structure, as described in the paper's Section 4:

Sharding. Series are assigned to shards by hash of the string key; the ShardMap points at a per-shard Timeseries Map (TSmap) of roughly 1 million entries. A Paxos-based ShardManager assigns shards to hosts.
Blocks. Each series holds closed two-hour blocks plus one append-only open block. Closed blocks are immutable and copied into large slabs to limit fragmentation. A read copies whole blocks out and the client decompresses, so the server never decodes.
Persistence. One append-only log per shard (buffered to 64 kB, explicitly not a write-ahead log), a complete block file every two hours, and a checkpoint file written only after the block file is fully flushed. A missing checkpoint means the block file is distrusted and only the log is replayed.
Failure handling. Write clients buffer one minute of data during shard movement and drop the oldest first. Reads from a recovering node return newest blocks first and are flagged partial, and the client retries the other region.

How to use it¶

Nothing in the paper ships as a product to install; the usable takeaways are at the level of the metrics pipeline.

Keep scrape timestamps regular. Timestamp cost is driven by jitter. In the executed block a constant-valued series still costs 0.44 bytes per point, almost entirely from 5% of scrapes arriving 1 s late and 1% missing.
Expose integer-valued or low-entropy metrics as such. Temperature, utilization, and clock counters compress 13 to 36 times in the block below; decimal-scaled power floats compress 2.3 times.
Query by recent window. The design assumes most reads are the last day, and archived data is rolled up to coarser granularity (a lossy step the paper states is worth the lost precision).

Back-of-envelope for sizing (the inputs are assumptions of this page, only the per-point sizes come from the paper): 1,000 nodes x 8 GPUs x 40 fields = 320,000 series at a 15 s scrape is 5,760 points per series per day, 1.84 billion points per day. Raw at 16 bytes is 29.5 GB per day; at the paper's 1.37 bytes it is 2.5 GB per day; at the 6.8 bytes per point measured below for float-heavy series it is 12.5 GB per day. Mixed fleets land between those, so measure the real mix.

How to develop with it¶

The reference implementation below is the encoder and decoder. Design points to preserve if reimplementing it:

The first timestamp delta is 14 bits, enough for a block just over 4 hours (16,384 s); the paper notes it would need to grow for larger blocks.
The smallest bucket is asymmetric, [-63, 64]: the n-bit two's complement value with the most negative pattern unused. Decoding must treat values above 2^(n-1) as negative, and the 32-bit escape is full two's complement. A first draft of the block below failed the round trip at -2^31 until the 32-bit case was special-cased.
The 6-bit length field cannot hold 64. A XOR that uses all 64 bits writes length 0, and the decoder must read 0 as 64. The block exercises that case.
Leading zeros are capped at 31 because the field is 5 bits.
Floats are compared bitwise (-0.0, NaN, subnormals), never with ==.

import math
import struct
import numpy as np

def f2b(x: float) -> int:
    return struct.unpack(">Q", struct.pack(">d", x))[0]

def b2f(b: int) -> float:
    return struct.unpack(">d", struct.pack(">Q", b))[0]

# Timestamp buckets from the paper: (control prefix, payload bits); the lowest
# bucket covers [-63, 64], i.e. n-bit two's complement with -2**(n-1) unused.
BUCKETS = [("10", 7), ("110", 9), ("1110", 12)]

def put_dod(d: int) -> str:
    if d == 0:
        return "0"
    for prefix, n in BUCKETS:
        if -(2 ** (n - 1)) + 1 <= d <= 2 ** (n - 1):
            return prefix + format(d & ((1 << n) - 1), f"0{n}b")
    return "1111" + format(d & 0xFFFFFFFF, "032b")

def encode(ts: list[int], vs: list[float], start: int) -> str:
    assert len(ts) == len(vs) and ts and ts[0] - start < 2 ** 14
    out = [format(start, "064b"), format(ts[0] - start, "014b"), format(f2b(vs[0]), "064b")]
    pd, pv, pl, pt = ts[0] - start, f2b(vs[0]), 99, 99
    pt_prev = ts[0]
    for t, v in zip(ts[1:], vs[1:]):
        d = t - pt_prev
        assert -(2 ** 31) <= d - pd <= 2 ** 31 - 1, "delta of delta exceeds 32 bits"
        out.append(put_dod(d - pd))
        pd, pt_prev = d, t
        x = f2b(v) ^ pv
        pv = f2b(v)
        if x == 0:
            out.append("0")
            continue
        lead = min(64 - x.bit_length(), 31)
        trail = (x & -x).bit_length() - 1
        if lead >= pl and trail >= pt:
            out.append("10" + format(x >> pt, f"0{64 - pl - pt}b"))
        else:
            n = 64 - lead - trail
            out.append("11" + format(lead, "05b") + format(n & 63, "06b") + format(x >> trail, f"0{n}b"))
            pl, pt = lead, trail
    return "".join(out)

class Reader:
    def __init__(self, bits: str): self.b, self.i = bits, 0
    def take(self, n: int) -> int:
        if self.i + n > len(self.b): raise EOFError("truncated block")
        v = int(self.b[self.i:self.i + n], 2); self.i += n; return v
    def signed(self, n: int) -> int:
        v = self.take(n)
        top = (1 << (n - 1)) - (1 if n == 32 else 0)   # 32-bit escape is full two's complement
        return v - (1 << n) if v > top else v

def decode(bits: str, count: int) -> tuple[list[int], list[float]]:
    r = Reader(bits)
    start = r.take(64); d = r.take(14); t = start + d
    pv = r.take(64); ts, vs = [t], [b2f(pv)]
    pl, pt = 99, 99
    for _ in range(count - 1):
        if r.take(1) == 0: dod = 0
        elif r.take(1) == 0: dod = r.signed(7)
        elif r.take(1) == 0: dod = r.signed(9)
        elif r.take(1) == 0: dod = r.signed(12)
        else: dod = r.signed(32)
        d += dod; t += d; ts.append(t)
        if r.take(1) == 0:
            vs.append(b2f(pv)); continue
        if r.take(1) == 1:
            pl = r.take(5); n = r.take(6) or 64; pt = 64 - pl - n
        else:
            n = 64 - pl - pt
        pv ^= r.take(n) << pt
        vs.append(b2f(pv))
    return ts, vs

def same(a, b): return [f2b(x) for x in a] == [f2b(x) for x in b]
def rt(ts, vs, start):
    bits = encode(ts, vs, start)
    t2, v2 = decode(bits, len(ts))
    assert t2 == ts and same(vs, v2), "round trip mismatch"
    return len(bits)

# (1) paper worked examples
assert put_dod(-2) == "10" + "1111110" and len(put_dod(-2)) == 9          # Figure 2: 9 bits
assert [put_dod(d) for d in (0,)] == ["0"]
ts = [0, 60, 120, 179, 240]                                              # deltas 60,60,59,61
assert [b - a for a, b in zip(ts, ts[1:])] == [60, 60, 59, 61]
d = [b - a for a, b in zip(ts, ts[1:])]
assert [y - x for x, y in zip(d, d[1:])] == [0, -1, 2]
d = [60, 60, 121, 59]                                                    # one missing point
dod = [y - x for x, y in zip(d, d[1:])]
assert dod == [0, 61, -62] and all(len(put_dod(x)) == 9 for x in dod[1:])
assert f2b(12.0) ^ f2b(24.0) == 0x0010000000000000                       # one meaningful bit
print("paper examples OK")

# (2) bucket boundaries, both sides
for dd in (-63, 64, -64, 65, -255, 256, -256, 257, -2047, 2048, -2048, 2049, 2**31 - 1, -2**31):
    pd = 100
    ts = [0, pd, pd + pd + dd]       # third delta gives dod == dd
    rt(ts, [1.0, 2.0, 3.0], 0)
print("lengths:", {dd: len(put_dod(dd)) for dd in (63, 64, 65, 255, 256, 257, 2047, 2048, 2049)})

# (3) special floats, 64-bit meaningful window, repeated and alternating values
sp = [0.0, -0.0, math.inf, -math.inf, math.nan, 5e-324, 1.7976931348623157e308, b2f(0x8000000000000001), b2f(1), 1.0, 1.0, 1.0000000000000002]
rt(list(range(0, 15 * len(sp), 15)), sp, 0)
rt([0, 15], [b2f(0), b2f(0xFFFFFFFFFFFFFFFF)], 0)                       # xor uses all 64 bits: length field wraps to 0
rt([0], [3.5], 0)
print("edge floats OK")

# (4) compression on synthetic GPU-like telemetry, 15 s scrape, 2 h block (480 points)
rng = np.random.default_rng(7)
n = 480
ts = (np.arange(n) * 15).tolist()
jit = ts[:]
for i in range(1, n):                     # 5% of scrapes late by 1 s, 1% missing
    if rng.random() < 0.05: jit[i] += 1
ts_j = [t for t in jit if rng.random() > 0.01]
series = {
    "sm_clock_mhz (constant)": [1410.0] * len(ts_j),
    "gpu_temp_c (integer walk)": np.clip(60 + np.cumsum(rng.integers(-1, 2, len(ts_j))), 30, 85).astype(float).tolist(),
    "gpu_util_pct (noisy integer)": rng.integers(85, 100, len(ts_j)).astype(float).tolist(),
    "power_w (0.1 W resolution)": np.round(300 + rng.normal(0, 8, len(ts_j)), 1).tolist(),
    "power_w (full-precision float)": (300 + rng.normal(0, 8, len(ts_j))).tolist(),
}
res = {}
for k, v in series.items():
    b = rt(ts_j, v, 0) if ts_j[0] < 2**14 else None
    res[k] = b / 8 / len(ts_j)
    print(f"{k:34s} {res[k]:6.2f} bytes/point  ({16 / res[k]:5.1f}x vs 16 B raw)")
ks = list(res)
assert res[ks[0]] < 0.6 < res[ks[1]] < res[ks[2]] < 2 < 6 < res[ks[3]] < 16
assert res[ks[4]] > 6 and abs(res[ks[3]] - res[ks[4]]) < 0.5     # rounding to 0.1 W does not help: decimals are not binary-short
print("compression ordering OK")

# (5) corruption and truncation
bits = encode(ts_j, series[ks[3]], 0)
try: decode(bits[: len(bits) // 2], len(ts_j)); raise SystemExit("truncation undetected")
except EOFError: print("truncation raises EOFError")
flip = bits[:200] + ("1" if bits[200] == "0" else "0") + bits[201:]
try:
    t2, v2 = decode(flip, len(ts_j)); assert (t2, [f2b(x) for x in v2]) != (ts_j, [f2b(x) for x in series[ks[3]]])
    print("bit flip: decoded without error but output differs (no checksum in the format)")
except (EOFError, ValueError, OverflowError, struct.error): print("bit flip: decode raised")

# (6) block size: header cost amortizes; gains flatten after a few hundred points
sizes = {}
for blk in (8, 120, 480, 1920):
    t = (np.arange(blk) * 15).tolist()
    v = (60 + rng.integers(-1, 2, blk)).astype(float).tolist()
    sizes[blk] = len(encode(t, v, 0)) / 8 / blk
    print(f"block of {blk:5d} points: {sizes[blk]:5.2f} bytes/point")
assert sizes[8] > 2 * sizes[480] and abs(sizes[480] - sizes[1920]) < 0.15

Executed output (Python 3 with numpy 2.4.6, 2026-09-30, byte-for-byte):

paper examples OK
lengths: {63: 9, 64: 9, 65: 12, 255: 12, 256: 12, 257: 16, 2047: 16, 2048: 16, 2049: 36}
edge floats OK
sm_clock_mhz (constant)              0.44 bytes/point  ( 36.0x vs 16 B raw)
gpu_temp_c (integer walk)            0.96 bytes/point  ( 16.6x vs 16 B raw)
gpu_util_pct (noisy integer)         1.25 bytes/point  ( 12.8x vs 16 B raw)
power_w (0.1 W resolution)           6.93 bytes/point  (  2.3x vs 16 B raw)
power_w (full-precision float)       6.83 bytes/point  (  2.3x vs 16 B raw)
compression ordering OK
truncation raises EOFError
bit flip: decoded without error but output differs (no checksum in the format)
block of     8 points:  3.30 bytes/point
block of   120 points:  0.78 bytes/point
block of   480 points:  0.62 bytes/point
block of  1920 points:  0.60 bytes/point

What the assertions prove, and what they do not:

Round trip is bit-exact for the paper's worked examples, every bucket boundary on both sides, special floats, a full 64-bit XOR window, and a single-point block.
Truncation is detected as EOFError. A flipped bit is not detected: the decoder returned different data without an error, so the format alone has no integrity protection and needs a checksum around each block.
Block length matters mostly at the small end: 8 points cost 3.30 bytes per point because the 64-bit header and first value dominate, and 480 to 1,920 points are within 0.02 bytes of each other, consistent with the paper's statement that blocks past two hours give diminishing returns.
Not validated: the paper's 1.37 bytes per point, which depends on Facebook's data and is not reproducible from public data; the paper's query latency and throughput figures; any concurrency, sharding, or recovery behaviour.

How to maintain it¶

Watch partial results. Gorilla flags reads from a node that is still loading blocks from disk. A dashboard that silently shows partial data after a restart reads as a real dip; propagate the flag.
Treat block files and logs differently. Only the checkpoint file says a block file is trustworthy; restore logic must honour it.
Roll up old data deliberately. Coarser granularity for old data is lossy and irreversible.
Re-measure the compression ratio whenever the metric mix changes; a move from integer counters to float ratios changes bytes per point severalfold, as the executed block shows.

How to run it in production¶

Redundancy over efficiency. The paper keeps two full in-memory copies in different regions "despite the efficiency hit" and names fault tolerance the most time-consuming part of the project.
Rolling upgrades are modelled as single-node failures, so upgrade safety and crash safety are one mechanism. The paper reports zero data drops during software upgrades.
Recovery time. A host reads its needed data from GlusterFS in about 5 minutes, at about 16 GB of on-disk storage per shard, and accepts new writes into a queue meanwhile.
Failover policy. After a regional loss longer than one minute, data is dropped from that region, and reads stay off it until it has been healthy for 26 hours.

Relation to Prometheus¶

Prometheus' XOR chunk encoder states that its code was largely written by Damian Gryski in go-tsz, the Go implementation of the Gorilla scheme, and its source comments that "Gorilla has a max resolution of seconds, Prometheus milliseconds". It is not a byte-for-byte port: the current tsdb/chunkenc/xor.go uses 14, 17, and 20 bit delta-of-delta buckets and a 64-bit fallback, against the paper's 7, 9, 12, and 32. Numbers from the paper therefore do not transfer to Prometheus storage without measurement.

Failure modes¶

Symptom	Cause	Action
Dashboard dips to zero right after a node restart	Partial read served while blocks load from disk	Honour the partial flag; retry the other region
Last seconds of data missing after a crash	64 kB log buffer and no write-ahead log, by design	Accept, or add write-ahead logging if data must be complete
Bytes per point far above the paper's 1.37	Float-heavy series with random mantissas, or heavy timestamp jitter	Measure per-series; fix scrape regularity; quantize at source if acceptable
Decoded data wrong with no error	Bit corruption; the format has no checksum	Checksum each block outside the format
Decode fails on all-bits-changing values	6-bit length field wraps at 64	Decode length 0 as 64
Stale region serves reads after recovery	Reads re-enabled before the 26 hour window	Keep reads off until 26 hours healthy

Open questions and source inconsistencies¶

Headline speedups differ across the paper. The abstract states 73x lower query latency and 14x higher throughput against HBase; the Figure 8 caption states 73x to 350x depending on query size; the conclusion states "over 70x". The text examined does not explain the 14x throughput figure beyond the abstract. The figures were not reproduced here.
Requirements versus measurement. Section 2.2 requires reads under one millisecond; Section 6 reports a 90th-percentile query time of 10 ms and the introduction aims for "tens of milliseconds". These measure different things and the paper does not reconcile them.
The 1.37 bytes per point is an average over Facebook ODS data in 2015; nothing on this page confirms it for GPU telemetry.
Figures and plots in the PDF extract as unreadable glyphs, so figure-only numbers (for example Figure 6 block-size curve values) were not read.

References¶

Pelkonen, Franklin, Teller, Cavallaro, Huang, Meza, Veeraraghavan, Gorilla: A Fast, Scalable, In-Memory Time Series Database, PVLDB 8(12), 2015: https://www.vldb.org/pvldb/vol8/p1816-teller.pdf
Prometheus XOR chunk encoder (tsdb/chunkenc/xor.go): https://github.com/prometheus/prometheus/blob/main/tsdb/chunkenc/xor.go
go-tsz, Gorilla compression in Go (credited in the Prometheus source header): https://github.com/dgryski/go-tsz