Snapshot Plus Delta Streaming for Dashboards

Part of Live Dashboards & Metrics Feeds, under Real-Time Application Patterns.

Sending only what changed is the obvious way to keep a dashboard stream small. Sending only what changed, and nothing else, is the reason so many dashboards show wrong numbers after a reconnect. This guide builds the protocol that fixes it: a versioned snapshot at the start of every connection, deltas that name the version they apply to, and a client that detects the one case where it must ask for help.

Symptom & Developer Intent

Delta-only dashboards fail in recognisable ways:

  • After Wi-Fi drops and returns, some tiles show values from before the drop until that metric happens to change again — sometimes for hours on a quiet metric.
  • Two people looking at the same dashboard see different numbers for the same tile.
  • A tile that was removed on the server stays on the client forever, because “removed” was never a delta.
  • A newly opened tab shows empty tiles until each metric changes for the first time.
  • Occasionally a value jumps backwards: an old delta arrived after a newer one following a server-side reorder.

The intent is a stream where every client, at any moment, holds state equal to some version the server actually had — never a mixture of two.

Root Cause Analysis

A delta is only meaningful relative to the state it was computed from. {"rps": 1302} means “change rps to 1302 starting from version 1842”. A client that missed versions 1843 to 1850 and applies version 1851’s delta ends up with a state that never existed on the server: some fields current, others eight versions old.

Client states in a snapshot-plus-delta protocol State machine with four client states — empty, in sync, gap detected and resyncing — and the transitions between them on snapshot, delta, gap and reconnect. Client states in a snapshot-plus-delta protocol EMPTY no state yet IN SYNC version v GAP base ≠ v RESYNC await snapshot snapshot stale delta reconnect snapshot
The only way into the in-sync state is a snapshot. A delta whose base version is not the client's version is a gap, and a gap is resolved by another snapshot, never by guessing.

EventSource makes the gap window real. When the connection drops, events published during the retry delay are simply never sent to that client. Without a snapshot on reconnect, and without a version check on each delta, the client has no way to know it is wrong. The browser’s Last-Event-ID resend gives the server the information it needs — the client’s last version — but the server must use it.

Deletions are the second root cause. A delta format of “fields and their new values” cannot express “this field no longer exists”. A tile for a decommissioned host keeps its last value until the page reloads.

Step-by-Step Resolution

Step 1 — Version the whole view, not individual fields

Keep a monotonically increasing version for the view. Every applied change increments it once, however many fields changed.

// view-store.js — the single source of truth on each SSE node.
export class ViewStore {
  constructor() { this.version = 0; this.values = new Map(); }

  apply(changes, removed = []) {
    const set = {};
    for (const [k, v] of Object.entries(changes)) {
      if (this.values.get(k) !== v) { this.values.set(k, v); set[k] = v; }
    }
    const del = removed.filter((k) => this.values.delete(k));
    if (!Object.keys(set).length && !del.length) return null;   // nothing changed, no version bump
    const base = this.version;
    this.version += 1;
    return { base, version: this.version, set, del };
  }

  snapshot() { return { version: this.version, values: Object.fromEntries(this.values) }; }
}

Step 2 — Make deltas carry their base version and deletions

The delta names the version it was computed from (base) and the version it produces. Deletions travel explicitly.

event: snapshot
id: 1842
data: {"version":1842,"values":{"cpu":38.2,"rps":1290,"host-7":"up"}}

event: delta
id: 1843
data: {"base":1842,"set":{"rps":1302},"del":[]}

event: delta
id: 1844
data: {"base":1843,"set":{},"del":["host-7"]}

Putting the version in id: as well means the browser echoes it back as Last-Event-ID on reconnect with no extra client code.

Step 3 — Decide on the server whether a reconnect needs a snapshot 

app.get('/dash/stream', (req, res) => {
  openStream(res);
  const clientVersion = Number(req.get('Last-Event-ID') ?? -1);
  const snap = store.snapshot();

  // A client exactly at the current version needs nothing; anyone else gets a snapshot.
  if (clientVersion !== snap.version) {
    res.write(`event: snapshot\nid: ${snap.version}\ndata: ${JSON.stringify(snap)}\n\n`);
  }

  const onDelta = (d) => {
    res.write(`event: delta\nid: ${d.version}\ndata: ${JSON.stringify(d)}\n\n`);
  };
  bus.on('delta', onDelta);
  req.on('close', () => bus.off('delta', onDelta));
});

Take the snapshot and subscribe to deltas without yielding in between. In Node.js, both lines run in the same synchronous turn, so no delta can slip between them. In a multi-threaded server, take a lock or read the snapshot and the subscription position atomically; otherwise a delta applied between the two is either lost or applied twice.

Step 4 — Apply deltas only when the base matches 

// client reducer: apply in order, or detect the gap and resync.
export function applyFrame(state, type, frame, resync) {
  if (type === 'snapshot') {
    return { version: frame.version, values: { ...frame.values } };
  }
  if (!state || frame.base !== state.version) {
    resync();                       // never apply a delta to the wrong base
    return state;
  }
  const values = { ...state.values, ...frame.set };
  for (const k of frame.del) delete values[k];
  return { version: frame.version, values };
}

The resync callback closes and reopens the EventSource. Because the last successfully applied version is still the Last-Event-ID, the server sees a stale version and sends a snapshot. In practice gaps are rare once step 3 is in place — they indicate a server bug or a message reordered by a broker — so treating them as “start again” is both simple and safe.

A gap is detected and repaired with one snapshot Sequence diagram in which the client receives delta 1843, misses 1844, receives 1845 whose base does not match, reconnects and receives snapshot 1845. A gap is detected and repaired with one snapshot Client SSE node delta base 1842 → 1843 delta 1844 lost in a reorder delta base 1844 → 1845 base 1844 ≠ 1843, resync reconnect Last-Event-ID: 1843 snapshot version 1845
The client never applies a delta to a base it does not hold. The cost of a gap is one reconnect and one snapshot, not a wrong dashboard.

Step 5 — Keep snapshots cheap for large views

A view with thousands of tiles makes the snapshot the most expensive frame. Two techniques keep it cheap. Serialise the snapshot once per version and share the string across all connections that need it, instead of calling JSON.stringify per connection. And for very large views, let the client request a partial snapshot — only the tiles on screen — by passing the tile set as a query parameter when it opens the stream.

let cached = { version: -1, text: '' };
function snapshotFrame() {
  if (cached.version !== store.version) {
    const snap = store.snapshot();
    cached = { version: snap.version, text: `event: snapshot\nid: ${snap.version}\ndata: ${JSON.stringify(snap)}\n\n` };
  }
  return cached.text;              // one serialisation per version, however many reconnects
}

Validation & Monitoring

Test the reconnect path explicitly, because it is the path that is broken in delta-only designs.

# 1. Open the stream, record the last id you saw.
curl -sN https://app.example.com/dash/stream | grep -m3 '^id:'

# 2. Reconnect claiming an old version: expect a snapshot as the first event.
curl -sN -H 'Last-Event-ID: 1' https://app.example.com/dash/stream | head -3

# 3. Reconnect claiming the current version: expect no snapshot, only deltas.
curl -sN -H "Last-Event-ID: $(curl -s https://app.example.com/dash/version)" \
  https://app.example.com/dash/stream | head -3

A unit test for the reducer should cover four cases: snapshot replaces everything, matching delta applies, deletion removes a key, and mismatched base triggers resync without changing state.

Reducer test cases and expected outcomes Matrix of four input cases against three expectations — state changes, version advances, resync is requested. Reducer test cases and expected outcomes Input frame State changes Version advances Resync requested Snapshot replaced to snapshot no Delta, base matches merged to delta no Delta with del key removed to delta no Delta, base mismatch unchanged unchanged yes
The gap case is the one that protects correctness, and the one most delta implementations never test.

In production, count snapshots sent per reason — new_connection, stale_reconnect, client_resync — as a labelled counter. client_resync should be near zero; a rising rate means deltas are being reordered or dropped between the store and the socket, often by a broker that does not preserve order across partitions.

Production Checklist

Frequently Asked Questions

Why put the version in the SSE id field?

Because the browser automatically sends the last id back in the Last-Event-ID header when it reconnects. Using the version as the id gives the server the client's exact position with no custom client code.

Can deltas be sent without a base version?

Only if the transport can never drop or reorder them, which SSE across reconnects cannot guarantee. Without a base the client cannot detect that it has missed something, so it silently shows a state that never existed.

How big can a snapshot be?

There is no protocol limit, but proxies and client parsers have practical ones and large frames delay everything behind them. Keep snapshots under a few hundred kilobytes, and for larger views send only the tiles the client is displaying.

What about multiple SSE nodes with different versions?

Versions must come from one source — a single writer or the broker's sequence — so that version 1843 means the same state on every node. If each node counts independently, a client that reconnects to a different node compares incompatible numbers.