Reporting File Upload and Processing Progress

Part of Progress Streaming for Long-Running Jobs, under Real-Time Application Patterns.

Uploading a large file is two jobs that users experience as one: sending the bytes, then waiting while the server validates, transcodes, parses or indexes them. The browser can measure the first half itself; only the server knows the second. This guide joins the two into a single progress bar — upload progress from the request, processing progress from a Server-Sent Events stream — with an ETA that does not bounce around.

Symptom & Developer Intent

  • The bar reaches 100 % when the upload finishes, then sits there for two minutes while the server processes the file.
  • Users re-upload because they think the first attempt hung.
  • The ETA swings from “5 seconds” to “3 minutes” and back as batch sizes vary.
  • fetch() gives no upload progress at all, so the team shows an indeterminate spinner for the whole operation.
  • Processing progress starts from zero on reload, because the processing stream is keyed to the upload request rather than to a durable id.

The intent is one bar with two clearly labelled phases, a smoothed ETA, and a processing phase that survives reloads.

Root Cause Analysis

The browser exposes upload progress through XMLHttpRequest.upload’s progress events. fetch() has no equivalent for request bodies in production browsers, so fetch-based upload code has no way to report bytes sent. After the last byte, the upload request typically blocks until processing finishes — and during that time neither the request nor anything else reports progress.

Upload and processing on one timeline Timeline of a 400 megabyte upload showing the upload phase measured by the browser, a short handoff, and a longer processing phase measured by the server. Upload and processing on one timeline Browser knows Server knows upload: bytes sent queued, then transcode + index 0 36 72 108 144 180 seconds last byte ready
The browser can see only the first bar. Without a stream, the second phase is two minutes of silence at "100 %".

Two design changes fix it. First, the upload request should return as soon as the bytes are stored, with a job id, rather than holding the connection through processing. Second, processing progress should be a separate stream keyed by that job id, which the browser opens as soon as the upload completes and which any later page load can reopen.

ETA instability is a statistics problem: estimating time remaining from the instantaneous rate of the most recent update amplifies every variation in batch size or network throughput. An exponentially weighted moving average of the rate is far steadier.

Step-by-Step Resolution

Step 1 — Upload with XHR to get byte progress 

// upload.js — XHR because fetch has no upload progress events.
export function uploadFile(file, onBytes) {
  return new Promise((resolve, reject) => {
    const xhr = new XMLHttpRequest();
    xhr.open('POST', '/api/uploads');
    xhr.responseType = 'json';
    xhr.upload.onprogress = (e) => { if (e.lengthComputable) onBytes(e.loaded, e.total); };
    xhr.onload = () => (xhr.status === 202 ? resolve(xhr.response) : reject(new Error(`upload ${xhr.status}`)));
    xhr.onerror = () => reject(new Error('network'));
    const form = new FormData();
    form.append('file', file);
    xhr.send(form);
  });
}

For files above a few hundred megabytes, split into chunks and upload them sequentially or with a resumable protocol; the byte progress is then the sum across chunks, and a failed chunk is retried without restarting the whole file.

Step 2 — Return a job id as soon as the bytes are stored 

app.post('/api/uploads', requireSession, upload.single('file'), async (req, res) => {
  const jobId = `up_${crypto.randomUUID()}`;
  await jobs.create(jobId, { owner: req.user.id, file: req.file.path, size: req.file.size });
  await processingQueue.add('process-upload', { jobId }, { jobId });
  res.status(202).json({ jobId, eventsUrl: `/api/jobs/${jobId}/events` });   // do not wait
});

The 202 response is the handoff: from here on, progress is the server’s to report.

Step 3 — Report processing progress with a smoothed rate 

// worker: EWMA of throughput gives a stable ETA.
function etaTracker(total, alpha = 0.2) {
  let rate = null, lastT = Date.now(), lastDone = 0;
  return (done) => {
    const now = Date.now();
    const inst = (done - lastDone) / Math.max(1, (now - lastT) / 1000);   // units per second
    rate = rate == null ? inst : alpha * inst + (1 - alpha) * rate;
    lastT = now; lastDone = done;
    return rate > 0 ? Math.round((total - done) / rate) : null;
  };
}

const eta = etaTracker(totalFrames);
for (const segment of segments) {
  framesDone += await transcode(segment);
  await reporter.progress(framesDone, totalFrames, 'transcoding', { etaS: eta(framesDone) });
}
ETA shown to the user during processing Line chart over ninety seconds comparing an ETA computed from the instantaneous rate, which swings widely, with an ETA from an exponentially weighted average, which declines smoothly. ETA shown to the user during processing instantaneous rate EWMA rate 0 50 100 150 200 0 18 36 54 72 90 seconds into processing ETA shown (s)
Both estimates end at zero at the same time. Only one of them is something a user can plan around.

Step 4 — Present one bar with two phases

Map upload and processing onto a single scale, weighted by how long each phase usually takes, and label the phase so users understand what is happening.

// Weight phases by typical duration: here upload ≈ 35 %, processing ≈ 65 %.
const UPLOAD_WEIGHT = 0.35;

async function uploadWithProgress(file, ui) {
  const { jobId, eventsUrl } = await uploadFile(file, (loaded, total) => {
    ui.set(UPLOAD_WEIGHT * (loaded / total) * 100, `Uploading — ${formatBytes(loaded)} of ${formatBytes(total)}`);
  });
  sessionStorage.setItem('activeUpload', jobId);            // survive reloads

  const es = new EventSource(eventsUrl, { withCredentials: true });
  es.addEventListener('progress', (e) => {
    const p = JSON.parse(e.data);
    const pct = UPLOAD_WEIGHT * 100 + (1 - UPLOAD_WEIGHT) * p.pct;
    ui.set(pct, p.etaS != null ? `Processing — about ${formatEta(p.etaS)} left` : 'Processing');
  });
  es.addEventListener('done', (e) => { es.close(); sessionStorage.removeItem('activeUpload'); ui.done(JSON.parse(e.data)); });
  es.addEventListener('failed', (e) => { es.close(); ui.fail(JSON.parse(e.data)); });
}

Estimate the weight from your own data: median upload time over median total time. It does not need to be exact, only good enough that the bar does not stall visibly at the boundary.

Step 5 — Reattach after a reload 

const pending = sessionStorage.getItem('activeUpload');
if (pending) watchProcessing(`/api/jobs/${pending}/events`, ui);   // bytes are already on the server

Because the processing stream opens with the job’s current state, a reload during processing shows the bar at its real position immediately — only the upload phase, which lives in the browser, cannot be resumed this way.

If the upload is still in flight when the user tries to leave, warn them: a beforeunload handler that returns a prompt only while bytes are being sent protects the one phase that cannot be recovered, and costs nothing once the job id exists.

let uploading = false;
window.addEventListener('beforeunload', (e) => {
  if (!uploading) return;               // processing survives a reload; the upload does not
  e.preventDefault();
  e.returnValue = '';
});

Set uploading to true before uploadFile and back to false in both its resolve and reject paths.

Two transports, one progress bar Flow from the file input through an XHR upload with byte progress to a 202 response with a job id, then an EventSource stream with processing progress, to a single progress bar. Two transports, one progress bar File input 400 MB bytes XHR upload upload.onprogress handoff 202 + jobId stored open EventSource processing events render One bar two labelled phases
The job id is the hinge between the two phases, and the only thing that needs to survive a reload.

Validation & Monitoring 

# Upload a large file and confirm the response returns before processing completes.
time curl -s -F file=@big.mov https://app.example.com/api/uploads
# {"jobId":"up_…","eventsUrl":"/api/jobs/up_…/events"}   real 0m58s  (upload time only)

# Then watch the processing stream, including the ETA field.
curl -sN https://app.example.com/api/jobs/up_…/events | grep -o '"etaS":[0-9]*'

In the browser, throttle the network to “Fast 3G” to lengthen the upload phase and check that the bar moves continuously through the phase boundary. Reload during processing and confirm the bar reappears at the right position. Track in analytics how often users start a second upload of the same file within five minutes; a drop after this change is the clearest evidence it worked.

Production Checklist

Frequently Asked Questions

Why not use fetch for the upload?

Fetch has no upload progress events in production browsers. Streaming request bodies exist in some browsers but still do not report bytes acknowledged by the network. XHR remains the reliable way to measure upload progress.

Could the upload request itself stream processing progress back?

A single request could respond with text/event-stream after reading the body, but a reload or network change during processing would lose it. A separate stream keyed by a job id is resumable and can be shared with other tabs.

How should the bar behave if processing time is unpredictable?

Report stages and counts, and switch the processing phase to an indeterminate style with stage text. Keep the upload phase determinate, since it is always measurable.

What smoothing factor works for the ETA?

An alpha between 0.1 and 0.3 on updates every quarter second is a good start. Lower values are steadier but slower to react when the rate genuinely changes, such as when a transcode moves from a simple scene to a complex one.