Streaming Time Series into Charts

Part of Live Dashboards & Metrics Feeds, under Real-Time Application Patterns.

A live chart is harder than a live number. A number only needs its latest value; a chart needs a continuous history, which means the stream must fill in what the client missed, the client must reject what it already has, and something must stop the history growing forever. This guide builds a chart feed that handles all three, with code that works for any charting library that accepts arrays of points.

Symptom & Developer Intent

Typical failures of a naively streamed chart:

  • A flat line or a straight diagonal segment appears across every period the connection was down.
  • After a reconnect, the same points are plotted twice, producing a visible zig-zag or a doubled spike.
  • Points arrive slightly out of order and the line doubles back on itself.
  • The chart gets slower every hour it is open and eventually freezes the tab.
  • A newly opened chart starts empty and fills in over the following minutes.

The intent is a chart that opens with the recent window already drawn, extends smoothly in real time, repairs gaps after reconnects, and uses constant memory.

Root Cause Analysis

Time series have the append-only shape, not the current-state shape, so the snapshot-only reconnect used for gauges is not enough. A client reconnecting after twenty seconds needs the twenty seconds of points it missed, or the chart shows a gap. Unlike a notification feed, though, it does not need them all forever — only the ones inside the visible window.

Where a chart gap comes from Timeline of one minute showing a stream connected, dropped for twelve seconds, and reconnected; points during the drop are missing unless backfilled. Where a chart gap comes from Connection Points plotted Backfill needed connected retrying connected 1 per second 1 per second 12 points 0 12 24 36 48 60 seconds drop reconnect
EventSource never delivers events published while it was reconnecting. The server has to send them — or the chart draws a straight line across the hole.

Three mechanics cause the other symptoms. Duplicates come from overlap: the server’s backfill starts from the client’s cursor, but the client may have already received a point with a timestamp equal to the cursor. Out-of-order points come from multiple producers or brokers that do not preserve order. Unbounded growth comes from pushing every point into the chart’s data array, which charting libraries then re-process in full on every update.

Step-by-Step Resolution

Step 1 — Use the point timestamp as the event id

A time series already has a natural, seekable cursor: the timestamp of the last point. Using it as the SSE id means the browser returns it as Last-Event-ID on reconnect with no extra code.

event: points
id: 1726650042000
data: {"series":"rps","points":[[1726650041000,1290],[1726650042000,1302]]}

Batching several points into one event, keyed by the newest timestamp, keeps frame overhead low at high sample rates.

Step 2 — Backfill from the cursor on every connection 

// Server: on connect, send the visible window (new client) or the gap (reconnect).
const WINDOW_MS = 15 * 60 * 1000;

app.get('/charts/:series/stream', async (req, res) => {
  openStream(res);
  const now = Date.now();
  const cursor = Number(req.get('Last-Event-ID') ?? 0);
  // Never backfill further than the visible window, however old the cursor.
  const from = Math.max(cursor + 1, now - WINDOW_MS);

  const history = await tsdb.range(req.params.series, from, now);   // [[t, v], ...]
  for (const chunk of chunks(history, 500)) {
    const last = chunk[chunk.length - 1][0];
    res.write(`event: points\nid: ${last}\ndata: ${JSON.stringify({ series: req.params.series, points: chunk })}\n\n`);
  }

  const off = live.subscribe(req.params.series, (pts) => {
    const fresh = pts.filter(([t]) => t > cursorOf(res));          // skip anything already backfilled
    if (!fresh.length) return;
    setCursor(res, fresh[fresh.length - 1][0]);
    res.write(`event: points\nid: ${cursorOf(res)}\ndata: ${JSON.stringify({ series: req.params.series, points: fresh })}\n\n`);
  });
  req.on('close', off);
});

Subscribe to the live feed before reading history, and filter the live points by the per-connection cursor. That order guarantees no point published during the history query is lost, and the cursor filter guarantees none is sent twice.

Step 3 — Merge points into a bounded, sorted buffer on the client 

// ring-series.js — fixed-capacity, sorted, de-duplicated by timestamp.
export class RingSeries {
  constructor(windowMs) { this.windowMs = windowMs; this.t = []; this.v = []; }

  add(points) {
    for (const [t, v] of points) {
      const last = this.t.length ? this.t[this.t.length - 1] : -Infinity;
      if (t > last) { this.t.push(t); this.v.push(v); continue; }      // common fast path
      const i = lowerBound(this.t, t);
      if (this.t[i] === t) { this.v[i] = v; continue; }                // duplicate: overwrite
      this.t.splice(i, 0, t); this.v.splice(i, 0, v);                  // late point: insert
    }
    const cutoff = this.t[this.t.length - 1] - this.windowMs;
    const drop = lowerBound(this.t, cutoff);
    if (drop > 0) { this.t.splice(0, drop); this.v.splice(0, drop); }  // trim to the window
  }
}

function lowerBound(a, x) {
  let lo = 0, hi = a.length;
  while (lo < hi) { const m = (lo + hi) >> 1; if (a[m] < x) lo = m + 1; else hi = m; }
  return lo;
}

The fast path handles the overwhelmingly common case — a point newer than everything held — in constant time. Duplicates overwrite, late points are inserted in order, and the window trim keeps memory flat.

Step 4 — Update the chart once per frame, with the library’s cheapest API 

const series = new RingSeries(15 * 60 * 1000);
let dirty = false;

es.addEventListener('points', (e) => {
  series.add(JSON.parse(e.data).points);
  dirty = true;
});

(function frame() {
  if (dirty) {
    dirty = false;
    // Most libraries accept parallel arrays or a typed array; avoid rebuilding objects per point.
    chart.setData([series.t, series.v]);
  }
  requestAnimationFrame(frame);
})();

Libraries differ in what an update costs. Canvas-based libraries that accept columnar arrays redraw tens of thousands of points in a few milliseconds; SVG-based libraries that create one DOM node per point become slow in the low thousands. For streaming use, pick a canvas renderer or downsample before rendering, as the frame-rate throttling guide explains for the general case.

Redraw time for a 15-minute window at 10 points per second Bar chart comparing redraw times for nine thousand points across an SVG renderer, a canvas renderer with object points, and a canvas renderer with columnar arrays. Redraw time for a 15-minute window at 10 points per second SVG, one node per point 180 ms Canvas, object points 22 ms Canvas, columnar arrays 4 ms milliseconds per full redraw, 9,000 points
The renderer choice matters more than the stream. With columnar data on canvas, a full redraw fits comfortably inside one 16 ms frame.

Step 5 — Downsample on the server for long windows

A 24-hour window at one point per second is 86,400 points — more than there are horizontal pixels on any screen. Let the client state its pixel width and have the server downsample the backfill (min/max per bucket preserves spikes better than averages), while streaming live points at full resolution for the most recent minutes.

// ?width=1200 → at most 1,200 buckets, each carrying the min and max of its range.
const buckets = Math.min(Number(req.query.width) || 1000, 4000);
const history = await tsdb.rangeMinMax(series, from, now, buckets);

Validation & Monitoring

Prove the three properties — no gaps, no duplicates, bounded memory — separately.

# Gap repair: reconnect with a cursor 30 s old and count the backfilled points.
ts=$(( ($(date +%s) - 30) * 1000 ))
curl -sN -H "Last-Event-ID: $ts" https://app.example.com/charts/rps/stream \
  | grep -m1 '^data:' | jq '.points | length'
# Expect ~30 at 1 point/s.

# Duplicate check: every id must be strictly greater than the previous one.
curl -sN https://app.example.com/charts/rps/stream | grep --line-buffered '^id:' \
  | awk '{ if ($2 <= prev) print "NON-MONOTONIC", prev, $2; prev = $2 }'

In the browser, toggle DevTools’ network throttling to Offline for twenty seconds and back: the chart should close the gap within one frame of the reconnect. For memory, take heap snapshots an hour apart with the chart open — retained size should be flat.

Chart memory over two hours of streaming Line chart of tab heap size over two hours for a chart that appends every point and a chart using a windowed ring buffer. Chart memory over two hours of streaming append every point 15 min ring buffer 0 100 200 300 400 0 24 48 72 96 120 minutes open heap (MB)
The windowed buffer plateaus as soon as the first fifteen minutes are full; the unbounded version grows until the tab is discarded.

Production Checklist

Frequently Asked Questions

Should each data point be its own SSE event?

At one point per second, it does not matter. Above a few points per second per series, batch them: one event carrying an array of points, with the newest timestamp as the id, cuts frame overhead and client dispatch work proportionally.

What if two points share the same timestamp?

Treat the timestamp as the key and let the later arrival overwrite. If the source can legitimately produce distinct points at the same millisecond, use a sequence number as the id and cursor instead of the timestamp.

How far back should the server backfill after a long disconnection?

Only as far as the chart displays. A client that was away for a day needs the current fifteen-minute window, not a day of points it will immediately trim.

Can the server send pre-aggregated points for live data too?

Yes, and at high source rates it should. Aggregate to the resolution the chart can show — for example one point per second — on the server, and stream the aggregates. Clients should never receive more resolution than they can plot.