Deduplicating Events on the Client with Event IDs

Part of Idempotent Event ID Generation, under Backend Stream Generation & Connection Management.

At-least-once delivery is the realistic guarantee for a resumable Server-Sent Events stream. A replay after reconnect may resend the event the client received just before the drop; a server that subscribes before replaying may deliver an event through both paths; a user with two tabs sharing state may see the same event arrive twice. The server should minimise duplicates, but the client must tolerate them. This guide makes client state idempotent, so a duplicate is a no-op rather than a double-counted order, a repeated toast or a list item shown twice.

Symptom & Developer Intent

  • After a flaky connection, the same notification appears twice in the list.
  • Counters (unread items, cart totals, “3 new comments”) drift upward over a session.
  • A celebratory animation or sound plays twice for one event.
  • Two open tabs each append the same item when syncing through BroadcastChannel.
  • Duplicates appear only on mobile, where reconnects are frequent.

The intent is a client where applying the same event twice produces exactly the same state as applying it once, with bounded memory and no dependence on the server being perfect.

Root Cause Analysis

The browser’s Last-Event-ID records the id of the last event dispatched, not the last event the application finished processing. If the connection drops in the window between the server writing an event and the browser dispatching it — or if the server’s replay query is inclusive of the cursor — the event can arrive again after reconnect.

How a duplicate arises across a reconnect Sequence diagram in which the server writes event 41, the connection drops before the client records it, the client reconnects with Last-Event-ID 40, and the server replays event 41 a second time. How a duplicate arises across a reconnect Client Network Server id 41 written dropped in flight reconnect Last-Event-ID: 40 replay 41 in a variant, 41 was dispatched but the app crashed before rendering; it arrives again
Neither side did anything wrong. At-least-once delivery means the client must treat event 41 arriving twice as normal.

Applying events as increments or appends is what turns a harmless duplicate into corrupted state. Applying them as assignments keyed by id makes duplicates harmless by construction.

Step-by-Step Resolution

Step 1 — Key state by entity id, not by arrival

The strongest form of idempotence needs no seen-set at all: store entities in a map keyed by their own id, and make every event an upsert or delete.

// Notifications: a Map keyed by notification id — a duplicate overwrites itself.
const notifications = new Map();

es.addEventListener('notification', (e) => {
  const n = JSON.parse(e.data);
  notifications.set(n.id, n);             // idempotent
  render();
});

es.addEventListener('notification-deleted', (e) => {
  notifications.delete(JSON.parse(e.data).id);   // idempotent too
  render();
});

Step 2 — Replace increments with authoritative values

Counters are where duplicates do the most damage. Have the server send the value, not the change:

: fragile — a duplicate increments twice
event: comment-added
data: {"postId":7,"delta":1}

: robust — a duplicate sets the same value twice
event: comment-count
data: {"postId":7,"count":14,"version":88}

If the event must carry a delta (to avoid a read on the server), include the entity’s version and ignore deltas whose version is not exactly one greater than the one held — the same versioning used for snapshot plus delta streams.

Step 3 — Guard side effects with a bounded seen-set

Some reactions cannot be expressed as state: a toast, a sound, an analytics beacon. Guard them with a set of recently seen event ids, bounded so it cannot grow without limit:

class SeenSet {
  constructor(limit = 2000) { this.limit = limit; this.set = new Set(); this.order = []; }
  firstTime(id) {
    if (!id || this.set.has(id)) return !id ? true : false;   // events without ids cannot be deduped
    this.set.add(id); this.order.push(id);
    if (this.order.length > this.limit) this.set.delete(this.order.shift());
    return true;
  }
}

const seen = new SeenSet();
es.addEventListener('notification', (e) => {
  if (seen.firstTime(e.lastEventId)) toast(JSON.parse(e.data));   // side effect once
});

The limit only needs to cover the replay window — the number of events a reconnect could resend — not the whole session. Two thousand ids is generous for most feeds.

Three kinds of client reaction and how each stays idempotent Three panels describing entity state, counters and side effects, and the technique that makes each idempotent. Three kinds of client reaction and how each stays idempotent Entity state Map keyed by entity id events are upserts/deletes duplicates overwrite Counters server sends the value or delta + version check never blind increments Side effects toast, sound, beacon bounded seen-set by id fire only on first sight
Most duplicate bugs disappear once state is keyed by id and counters are sent as values. The seen-set is only for the remainder.

Step 4 — Deduplicate across tabs too

When one tab holds the stream and broadcasts events to others through BroadcastChannel, or when each tab holds its own stream, the same event can reach shared state (IndexedDB, a shared store) more than once. Apply the same rules at the shared layer: upsert by entity id, and record handled event ids in the shared store if side effects run there. Sharing one SSE connection across tabs avoids most of this by having one tab own the stream.

Step 5 — Help from the server

Clients should not depend on it, but the server can reduce duplicates: replay with an exclusive range (strictly after the cursor), subscribe-then-replay with a monotonic filter at the seam, and give every event an id. Events without ids cannot be deduplicated by id at all, which is a good reason never to send them for anything that matters. Generating monotonic event IDs covers the server side.

Validation & Monitoring 

// Test: feeding every event twice must produce the same state as feeding it once.
test('store is idempotent under duplicate delivery', () => {
  const events = loadFixture('session-with-reconnects.json');
  const once = events.reduce(apply, initialState());
  const twice = events.flatMap((e) => [e, e]).reduce(apply, initialState());
  expect(twice).toEqual(once);
});
Unread counter error after a simulated flaky session Bar chart of counter drift after 200 events with 30 reconnects, comparing blind increments, deltas with version checks and server-sent values. Unread counter error after a simulated flaky session Blind increments +17 Delta + version check 0 Server-sent value 0 difference between displayed and true unread count after the session
Blind increments drift by every duplicate. Version checks and authoritative values stay exact.

In production, count duplicates the seen-set suppresses and report them as a client metric. A steady low rate is the normal cost of at-least-once delivery; a sudden rise indicates a server change that broke exclusive replay or the replay seam.

Production Checklist

Frequently Asked Questions

Can the server guarantee exactly-once delivery instead?

Not over a connection that can fail at any point. The server can minimise duplicates with exclusive replay and careful seams, but the only robust guarantee is at-least-once delivery plus idempotent clients.

Should I deduplicate by event id or entity id?

Entity id for state, because it also collapses different events about the same thing into the latest version. Event id for side effects, because each event should trigger its reaction once.

How large should the seen-set be?

Large enough to cover every event a reconnect could replay — typically the server's maximum replay batch. A few thousand ids cost a few hundred kilobytes at most.

What about events without an id?

They cannot be deduplicated by id and do not advance Last-Event-ID. Use them only for data where duplicates are harmless, such as heartbeats and ephemeral presence signals.