Tailing Server Logs to the Browser with SSE

Part of Log Tailing & CI Output Streaming, under Real-Time Application Patterns.

Internal admin panels eventually grow a “live logs” tab: the output of a service, a container or a pod, following in real time. The quickest implementation spawns tail -f, journalctl -f or docker logs -f and pipes stdout into a Server-Sent Events response. That works in a demo and leaks processes, file descriptors and memory in production. This guide builds the robust version for the three common sources, with native cursors so reconnects resume exactly, and a guaranteed kill of the follower process when the viewer leaves.

Symptom & Developer Intent

  • ps on the admin host shows hundreds of orphaned journalctl -f or docker logs -f processes.
  • After a reconnect the log restarts from the last 100 lines, duplicating what the viewer already saw.
  • A service that logs heavily makes the admin API’s memory climb until it is killed.
  • A user can view logs for services they should not see by editing a query parameter.
  • Output stops appearing after a minute of silence, then arrives in a clump.

The intent is a live log view per service or pod that resumes exactly after reconnects, never outlives its viewer, stays within a memory budget and enforces access control.

Root Cause Analysis

Each follower is a child process holding a pipe. If the HTTP handler does not kill it when the client disconnects, it runs until the log source ends — which for a long-running service is never. Node.js does not kill children when a request closes; Python’s asyncio subprocesses behave the same way.

What stays alive when a viewer closes the tab Stack of the browser tab, the SSE response, the child process and its pipe, showing which layers are released automatically and which leak. What stays alive when a viewer closes the tab Browser tab closed released SSE response socket closed released by the framework journalctl -f child process keeps running, forever stdout pipe 64 KB buffer fills, then blocks the child
The response is cleaned up by the server framework. The child process and its pipe are not, unless the close handler kills them.

The resume problem comes from ignoring the sources’ own cursors. journalctl has a cursor per entry (__CURSOR), Docker and Kubernetes accept --since timestamps and support per-line timestamps; each can resume precisely if the SSE id carries that cursor.

Memory growth comes from writing to the response without respecting backpressure: when the viewer’s connection is slower than the log, res.write queues output in memory. Pausing the child’s stdout when the response needs draining pushes the backpressure all the way back to the source.

Step-by-Step Resolution

Step 1 — Authorise the source on the server

Never accept a unit name, container id or pod name straight from the query string. Map an opaque id the user is allowed to see to the real target.

async function resolveTarget(user, serviceId) {
  const svc = await services.get(serviceId);
  if (!svc || !user.canViewLogs(svc)) return null;
  return svc.kind === 'systemd' ? { kind: 'journal', unit: svc.unit }
       : svc.kind === 'docker'  ? { kind: 'docker', container: svc.containerId }
       : { kind: 'k8s', namespace: svc.namespace, pod: svc.pod, container: svc.container };
}

Step 2 — Build the follower command with native cursors 

function followerArgs(t, cursor) {
  switch (t.kind) {
    case 'journal':
      // JSON output carries __CURSOR for every entry; --after-cursor resumes exactly.
      return ['journalctl', ['-u', t.unit, '-f', '-o', 'json',
        ...(cursor ? ['--after-cursor', cursor] : ['-n', '200'])]];
    case 'docker':
      return ['docker', ['logs', '-f', '--timestamps',
        ...(cursor ? ['--since', cursor] : ['--tail', '200']), t.container]];
    case 'k8s':
      return ['kubectl', ['logs', '-f', '--timestamps', '-n', t.namespace, t.pod, '-c', t.container,
        ...(cursor ? ['--since-time', cursor] : ['--tail', '200'])]];
  }
}

Arguments are passed as an array, never through a shell, so a crafted service name cannot inject commands. For Docker and Kubernetes the cursor is the RFC 3339 timestamp of the last line; timestamps are not unique, so on resume skip lines whose timestamp equals the cursor and whose text was the last one sent.

Step 3 — Stream stdout with backpressure and a guaranteed kill 

import { spawn } from 'node:child_process';
import readline from 'node:readline';

app.get('/api/services/:id/logs', requireSession, async (req, res) => {
  const target = await resolveTarget(req.user, req.params.id);
  if (!target) return res.status(404).end();

  const [cmd, args] = followerArgs(target, req.get('Last-Event-ID'));
  const child = spawn(cmd, args, { stdio: ['ignore', 'pipe', 'pipe'] });
  const kill = () => { if (child.exitCode === null) child.kill('SIGTERM'); };
  req.on('close', kill);                               // the line that prevents the leak
  child.on('error', () => res.end());

  openStream(res, { retryMs: 1000 });
  const rl = readline.createInterface({ input: child.stdout });
  let batch = [], lastId = null, timer = null;

  const flush = () => {
    timer = null;
    if (!batch.length) return;
    const ok = res.write(`event: lines\nid: ${lastId}\ndata: ${JSON.stringify({ lines: batch })}\n\n`);
    batch = [];
    if (!ok) { child.stdout.pause(); res.once('drain', () => child.stdout.resume()); }
  };

  rl.on('line', (raw) => {
    const { id, text } = parseLine(target.kind, raw);   // journal JSON → __CURSOR + MESSAGE
    lastId = id;
    batch.push(text);
    if (batch.length >= 500) flush();
    else timer ??= setTimeout(flush, 100);               // cut by time as well as size
  });
  child.on('exit', (code) => { flush(); res.write(`event: end\ndata: {"code":${code}}\n\n`); res.end(); });
});

function parseLine(kind, raw) {
  if (kind === 'journal') {
    const e = JSON.parse(raw);
    return { id: e.__CURSOR, text: String(e.MESSAGE ?? '') };
  }
  const sp = raw.indexOf(' ');                          // "2026-09-18T09:12:03.123Z message"
  return { id: raw.slice(0, sp), text: raw.slice(sp + 1) };
}

Pausing child.stdout when the response signals backpressure stops reading the pipe; the pipe fills; the kernel blocks the child’s writes. Nothing accumulates in the Node.js heap.

Backpressure from a slow viewer back to the log source Flow showing a slow browser connection causing res.write to return false, which pauses the child's stdout, fills the pipe and blocks the follower process. Backpressure from a slow viewer back to the log source Slow viewer 3G link slow drain res.write returns false pause stdout.pause() stop reading fills Pipe full 64 KB blocks Follower write blocks
Each link hands the pressure to the one before it. The admin API's memory stays flat however fast the service logs.

Step 4 — Cap concurrent followers per user and per host

Every viewer costs a process. Limit them so one user with twenty tabs cannot exhaust the host:

const perUser = new Map();
function acquire(userId, max = 5) {
  const n = perUser.get(userId) ?? 0;
  if (n >= max) return false;
  perUser.set(userId, n + 1);
  return true;
}
function release(userId) { perUser.set(userId, Math.max(0, (perUser.get(userId) ?? 1) - 1)); }

Return HTTP 429 when the limit is hit. A non-200 response makes EventSource stop reconnecting, which is correct here: the user should close a tab, not have the browser retry forever. The general pattern is in limiting SSE connections per user.

Step 5 — Keep quiet streams alive

A service that logs nothing for minutes will have its stream closed by idle timeouts. Add a heartbeat comment every 15 seconds alongside the follower. When several viewers watch the same service, a single shared follower per service per host, fanning out to viewers, is cheaper than a process each; the log tailing topic covers that fan-out.

Validation & Monitoring 

# Open a stream, close it, and confirm the follower process exits.
curl -sN -b s.txt https://admin.example.com/api/services/api-7/logs > /dev/null &
sleep 3; kill %1; sleep 1
pgrep -fa 'journalctl -u api' || echo "no leaked followers"

# Resume with a journal cursor: the first line must be the entry after it.
curl -sN -b s.txt -H "Last-Event-ID: $CURSOR" https://admin.example.com/api/services/api-7/logs | head -3

Export a gauge of live follower processes and alert when it exceeds the number of open log streams — any difference is a leak. A soak test that opens and closes a thousand streams should end with the gauge back at zero.

Follower processes left after 1,000 opened and closed streams Bar chart comparing leaked follower processes after a soak test, with no close handler, with a close handler, and with a close handler plus a follower gauge alert. Follower processes left after 1,000 opened and closed streams No close handler 1,000 Kill on request close 0 processes still running after the test
One missing line — killing the child on request close — is the difference between a thousand leaked processes and none.

Production Checklist

Frequently Asked Questions

Is it safe to spawn journalctl or kubectl from a web handler?

It is safe if the target is resolved server-side from an authorised id, arguments are passed as an array without a shell, and every child is killed on disconnect. Many teams prefer calling the runtime's log API directly, which removes the process but not the other requirements.

How do I resume Docker or Kubernetes logs exactly?

Request timestamps with every line, use the last timestamp as the SSE id, and resume with a since option. Because several lines can share a timestamp, skip lines at the cursor timestamp that were already sent.

Why did output stop and then arrive in a clump?

Either an intermediary buffered the response or the follower buffered its own stdout because it was not attached to a terminal. Disable proxy buffering and, for tools that buffer when piped, run them with line buffering enabled.

Should one viewer's stream use one process?

For a handful of viewers, yes. When many people watch the same service, share one follower per service per host and fan its output to all viewers, so process count scales with services rather than viewers.