Market Data & Live Scoreboards

Part of Real-Time Application Patterns.

Price tickers, crypto order books, betting odds, sports scoreboards and auction countdowns share a shape: a small set of keys whose values change very often, watched by a very large audience. A popular match or a volatile market produces hundreds of updates per second upstream and tens of thousands of viewers downstream — the worst combination of rate and fan-out any real-time feature faces. Server-Sent Events handles it well, because the traffic is entirely server-to-client and every viewer receives the same data, but only with a design built around one insight: intermediate values are disposable. A viewer who misses a price that was superseded fifty milliseconds later has lost nothing. This guide covers conflation (keeping only the latest value per key and flushing on a timer), sequence numbers and gap detection so a viewer knows when its view is wrong, snapshot recovery, and the fan-out architecture that serves a crowd. It is for engineers building trading front ends, sports and betting products, auction sites and live leaderboards.

How It Works

Upstream, a feed handler receives updates from an exchange, a data vendor or an internal scoring service. Each update is keyed — by instrument symbol, by match id, by lot number — and carries a sequence number from the source. Downstream, each viewer subscribes to the keys it displays.

From upstream feed to thousands of screens Flow from an upstream feed through a normaliser, a per-key conflation buffer flushed every 100 milliseconds, a broker, and SSE edge nodes that write to viewers. From upstream feed to thousands of screens Upstream feed 800 updates / s raw Normaliser key, seq, value typed Conflator latest per key flush Broker 100 ms batches fan-out SSE edges 40k viewers
Conflation sits before fan-out, so it reduces the work of every downstream hop. By the time frames reach the edge, each key changes at most ten times a second.

On the wire, a frame carries a batch of changed keys with their source sequence numbers, and the SSE id is the conflator’s own batch sequence:

retry: 1000

event: snapshot
id: 90210
data: {"seq":90210,"quotes":{"ACME":{"bid":101.20,"ask":101.24,"s":5512},"GLOBX":{"bid":44.10,"ask":44.13,"s":2210}}}

event: batch
id: 90211
data: {"prev":90210,"quotes":{"ACME":{"bid":101.21,"ask":101.25,"s":5519}}}

event: batch
id: 90212
data: {"prev":90211,"quotes":{"ACME":{"bid":101.19,"ask":101.23,"s":5530},"GLOBX":{"bid":44.11,"ask":44.14,"s":2214}}}

Every batch names the batch it follows (prev). A client that receives a batch whose prev is not the last batch it applied knows it missed something and must resynchronise. The per-key s field is the source sequence, which lets the client ignore a stale value for one key even when batches arrive in order — a guard against feed handlers that reorder across partitions.

Order books are state with structure

An order book — the ladder of bids and asks at each price level — is still current state, but it is too large to snapshot on every change and too structured for a flat key map. The standard encoding is a snapshot of the top N levels followed by level updates, where a size of zero deletes the level:

event: book
id: 5510
data: {"sym":"ACME","snap":true,"bids":[[101.20,900],[101.19,1500]],"asks":[[101.24,400],[101.25,2200]]}

event: book
id: 5511
data: {"sym":"ACME","prev":5510,"bids":[[101.20,0],[101.21,300]],"asks":[]}

The second frame removes the 101.20 bid and adds a new best bid at 101.21. Book updates cannot be conflated by simply keeping the latest frame — two consecutive updates may touch different levels — but they can be conflated by merging: keep a map from price level to latest size per side, and flush the map. A zero size must survive the merge, because it is the instruction to delete the level. And because a book assembled from updates is only correct if no update was missed, prev gap detection is mandatory here, not optional; the repair is a fresh snap: true frame for that symbol.

// Merge level updates for one symbol between flushes; zero sizes are kept as deletions.
function mergeBook(pending, update) {
  for (const [px, sz] of update.bids) pending.bids.set(px, sz);
  for (const [px, sz] of update.asks) pending.asks.set(px, sz);
}

Clients should also cap the depth they render. A phone showing ten levels does not need the server to send fifty, so accept a depth parameter and trim both the snapshot and the updates to it on the edge.

Scoreboards are the same shape at lower rates, with one difference that matters: some events are not disposable. A goal, a wicket, a red card or a trade execution must be seen even if the score it produced is superseded. Those travel as their own named event type, excluded from conflation. Streaming live sports scores with SSE builds that split.

Server-Side Implementation

The conflator is a map from key to latest value plus a timer. Updates overwrite; the timer flushes whatever changed.

// conflator.js — latest value per key, flushed every intervalMs.
export function createConflator({ intervalMs = 100, publish }) {
  let dirty = new Map();                         // key → latest update since last flush
  let batchSeq = 0;
  const current = new Map();                     // key → latest value ever (for snapshots)

  function update(key, value, sourceSeq) {
    const prev = current.get(key);
    if (prev && prev.s >= sourceSeq) return;     // stale or duplicate from the feed
    const v = { ...value, s: sourceSeq };
    current.set(key, v);
    dirty.set(key, v);                           // overwrites any earlier unflushed value
  }

  setInterval(() => {
    if (!dirty.size) return;
    const quotes = Object.fromEntries(dirty);
    dirty = new Map();
    const prev = batchSeq;
    batchSeq += 1;
    publish({ id: batchSeq, prev, quotes });
  }, intervalMs);

  return {
    update,
    snapshot: () => ({ seq: batchSeq, quotes: Object.fromEntries(current) }),
  };
}

The edge node subscribes to batches, keeps its own copy of current for snapshots, and writes to each viewer only the keys they subscribed to:

app.get('/api/quotes/stream', async (req, res) => {
  const symbols = new Set(String(req.query.s ?? '').split(',').filter(Boolean).slice(0, 200));
  if (!symbols.size) return res.status(400).end();
  openStream(res, { retryMs: 1000 });

  const pick = (quotes) => {
    const out = {};
    for (const k of Object.keys(quotes)) if (symbols.has(k)) out[k] = quotes[k];
    return out;
  };

  const snap = edgeState.snapshot();                             // always snapshot on connect
  res.write(`event: snapshot\nid: ${snap.seq}\ndata: ${JSON.stringify({ seq: snap.seq, quotes: pick(snap.quotes) })}\n\n`);
  let last = snap.seq;

  const onBatch = (b) => {
    const quotes = pick(b.quotes);
    const frame = { prev: last, quotes };
    last = b.id;
    if (!Object.keys(quotes).length) return;                     // nothing this viewer watches
    if (res.writableNeedDrain) { skipped.add(res); return; }     // slow viewer: resnapshot later
    res.write(`event: batch\nid: ${b.id}\ndata: ${JSON.stringify(frame)}\n\n`);
  };
  edgeBus.on('batch', onBatch);
  req.on('close', () => edgeBus.off('batch', onBatch));
});

Note that prev is computed per viewer, as the last batch id this viewer was sent or skipped past because it contained none of its keys. A viewer watching two symbols never sees batches that only touched other symbols, and it must not treat those as gaps.

A slow viewer is added to a skipped set instead of being queued. When its socket drains, send a fresh snapshot of its keys: because the data is state-shaped, a snapshot is always a complete repair. Conflating price ticks for slow clients implements that per-viewer conflation in full.

In Go, the conflator maps naturally onto a goroutine that owns the map and selects between an input channel and a ticker, which removes the need for locks:

func Conflate(in <-chan Update, out chan<- Batch, every time.Duration) {
	dirty := map[string]Quote{}
	var seq uint64
	t := time.NewTicker(every)
	defer t.Stop()
	for {
		select {
		case u, ok := <-in:
			if !ok {
				return
			}
			if prev, seen := dirty[u.Key]; !seen || prev.S < u.S {
				dirty[u.Key] = u.Quote
			}
		case <-t.C:
			if len(dirty) == 0 {
				continue
			}
			seq++
			out <- Batch{ID: seq, Prev: seq - 1, Quotes: dirty}
			dirty = map[string]Quote{}
		}
	}
}

Client-Side Consumption

The client applies batches only in sequence, resyncs on a gap, and renders on animation frames with change highlighting.

// quotes-client.js
export function watchQuotes(symbols, render) {
  const url = `/api/quotes/stream?s=${encodeURIComponent(symbols.join(','))}`;
  let es, last = null;
  const quotes = new Map();
  let changed = new Set();

  function connect() {
    es = new EventSource(url);
    es.addEventListener('snapshot', (e) => {
      const d = JSON.parse(e.data);
      quotes.clear();
      for (const [k, v] of Object.entries(d.quotes)) { quotes.set(k, v); changed.add(k); }
      last = d.seq;
    });
    es.addEventListener('batch', (e) => {
      const d = JSON.parse(e.data);
      if (d.prev !== last) { es.close(); last = null; return connect(); }   // gap: resnapshot
      for (const [k, v] of Object.entries(d.quotes)) {
        const cur = quotes.get(k);
        if (!cur || v.s > cur.s) { quotes.set(k, v); changed.add(k); }
      }
      last = Number(e.lastEventId);
    });
  }
  connect();

  (function frame() {
    if (changed.size) { render(quotes, changed); changed = new Set(); }
    requestAnimationFrame(frame);
  })();
  return () => es.close();
}

The changed set lets the renderer flash only the cells that moved — the familiar green/red tick highlight — without diffing the whole board. Keep flashes short and use a shape or arrow as well as colour, so colour-blind users can read direction.

Edge Cases & Network Interference

Firehose failure modes Two panels listing failures specific to high-rate market data streams, split between the network path and the client. Firehose failure modes On the network path proxy buffering batches frames gzip holds small writes mobile links fall behind HTTP/1.1 six-stream cap In the client render per frame, not per msg stale value after reorder silent gap after sleep memory from history arrays
At hundreds of frames per second, problems that are invisible on a quiet stream become the dominant cost.
  • Compression. A text stream of numbers compresses extremely well, which tempts teams to enable gzip. Compressors buffer; unless the server flushes the compressor after each batch, prices arrive late in clumps. If you compress, flush per batch and measure latency, not just bandwidth.
  • Falling behind on mobile. A phone on a congested cell cannot receive 20 KB per second indefinitely. Per-viewer conflation turns “falling further behind every second” into “fewer, fresher updates”.
  • Sleep and resume. A tab that was frozen for ten minutes resumes with a socket that may still be open and delivering. The gap check catches the discontinuity; without it the board silently shows ten-minute-old prices until each symbol ticks again.
  • Market hours. Outside trading hours the stream goes quiet, and idle timeouts start closing connections. Heartbeats every 15 seconds prevent it.
  • Entitlements. Real-time data is often licensed per user. Enforce entitlements when filtering keys on the edge, and re-check periodically on long-lived streams.

Mitigation checklist:

Performance & Scale Considerations

Conflation is the lever that makes the economics work. Its effect grows with the upstream rate: the busier the market, the higher the share of updates that are superseded within one interval.

Frames per viewer as upstream rate rises Line chart of downstream frames per second for one viewer watching 20 symbols, without conflation and with a 100-millisecond conflation interval, as the upstream update rate rises. Frames per viewer as upstream rate rises no conflation 100 ms conflation 0 250 500 750 1000 0 200 400 600 800 1000 upstream updates per second for the watched symbols frames per second to one viewer
Without conflation the viewer's rate tracks the market. With a 100 ms interval it can never exceed ten frames per second, however wild the market gets.

Beyond conflation:

  • Serialise once per batch per key set, not per viewer. Viewers of a popular board watch identical symbol sets. Cache the serialised frame by symbol-set hash and reuse it:
// Per batch: at most one JSON.stringify per distinct symbol set, shared by all its viewers.
function frameFor(batch, symbolSetKey, symbols, cache) {
  let text = cache.get(symbolSetKey);
  if (text === undefined) {
    const quotes = pick(batch.quotes, symbols);
    text = Object.keys(quotes).length
      ? `event: batch\nid: ${batch.id}\ndata: ${JSON.stringify({ quotes })}\n\n`
      : null;
    cache.set(symbolSetKey, text);
  }
  return text;
}
// cache is a new Map() per batch, discarded after the batch is written to every viewer.

On a board where 30,000 viewers watch one of a dozen default watchlists, this turns 30,000 serialisations per batch into twelve.

  • Shard edges by audience, not by symbol. Every edge node receives every batch; each serves a share of viewers. Adding viewers adds edges linearly.
  • Keep payloads small. Short keys, numbers rather than strings, and no unchanged fields. A board of 50 symbols at ten batches per second is well under 10 KB per second per viewer with compact JSON.
Bytes per batch for 20 changed symbols Bar chart comparing batch payload sizes for verbose JSON with full objects, compact JSON with short keys, and compact JSON with only changed fields. Bytes per batch for 20 changed symbols Verbose objects, all fields 3.9 KB Short keys, all fields 1.7 KB Short keys, changed fields 0.9 KB bytes per batch before transport compression
Payload discipline is multiplied by every viewer and every batch. Sending only changed fields with short keys cuts the stream to under a quarter of its naive size.
  • Measure tick-to-screen latency. Stamp the upstream receive time in each batch and report the render time from a sample of clients. That number, not throughput, is what traders and bettors notice.
  • Separate the firehose from everything else. Run market data edges as their own process pool. A burst that saturates them should never delay notifications, chat or account updates served by the rest of the product, and a deploy of the quote edges should not reconnect every other stream.
  • Plan the reconnect wave. When an edge restarts, all of its viewers reconnect and each receives a snapshot. Jitter the retry: value and cache snapshots per symbol set for a second, so the wave costs a few serialisations rather than thousands.

Validation & Debugging 

# Rate and gap check: batches should arrive ~10 per second with contiguous prev links.
curl -sN 'https://md.example.com/api/quotes/stream?s=ACME,GLOBX' \
  | grep --line-buffered '^data:' | head -200 \
  | jq -r 'select(.prev != null) | .prev' \
  | awk 'NR>1 && $1 != last+0 { gaps++ } { last=$1 } END { print "gaps:", gaps+0 }'

# Slow viewer: throttle the client and confirm it receives snapshots, not a backlog.
curl -sN --limit-rate 2k 'https://md.example.com/api/quotes/stream?s=ACME' | grep -c '^event: snapshot'

The gap count in the first command counts discontinuities for a viewer that is keeping up; it should be zero (the edge only skips batches that did not touch the viewer’s symbols, and prev accounts for those). In the browser, the Performance panel should show one render per frame during a burst.

Production Checklist

Frequently Asked Questions

What conflation interval should I use?

100 milliseconds suits most human-facing boards: faster than people can read, slow enough to collapse bursts. Trading screens for professionals sometimes use 50 ms; scoreboards and auctions are fine at 250 ms to 1 s.

Is SSE fast enough for trading applications?

For display to humans, yes — the transport adds little beyond one network round trip. For machine-to-machine order routing, dedicated binary protocols remain the right tool; SSE is for the screen.

How do viewers subscribe to different symbols without reconnecting?

Either reconnect with a new symbol list — cheap, since the snapshot repairs state — or keep one stream per session and change subscriptions with a POST that updates the server-side filter for that stream id.

Should prices be sent as numbers or strings?

Strings, when exact decimal representation matters and the client must not reformat, as with many currency and crypto pairs; JSON numbers are binary floating point in JavaScript. Numbers are fine for display-only boards where a price is rounded to a fixed number of decimals before rendering.

How do I test a feed handler and conflator without live market data?

Record an hour of the upstream feed during a volatile session and replay it at one, five and twenty times speed against staging. The replay reproduces real burst shapes, which synthetic random generators rarely do.

Should trades and goals be conflated too?

No. Discrete events that a viewer must see individually travel as their own event type with their own ids and replay, while the prices or scores they produce are conflated like any other state.

Deep Dives

Conflating Price Ticks for Slow Clients Keep slow SSE viewers current instead of hopelessly behind — per-viewer latest-value buffers, drain-driven flushes and snapshot repair for high-rate price feeds. Streaming Live Sports Scores with SSE Build a live scoreboard over SSE that never misses a goal — conflated match state plus replayable key events, CDN-friendly fan-out and big-match traffic spikes.