JSON Streaming
JSON streaming is a technique for processing large amounts of JSON data piece by piece, without loading the entire dataset into memory at once. Standard JSON parsing waits for the full document before doing anything. Streaming JSON processing starts working on the data as it arrives.
Why JSON Streaming Is Important
Imagine a truck delivering 10,000 boxes to a warehouse. One approach: the truck unloads all 10,000 boxes into the parking lot, then workers carry them into the warehouse one by one. Another approach: workers start carrying boxes inside as soon as the truck begins unloading. The second approach is faster and uses less parking lot space.
JSON streaming works the same way. Instead of loading a 500 MB JSON file completely into memory before processing, you process each record as it comes in. This keeps memory usage low and allows you to start producing results immediately.
When Standard JSON Parsing Fails
Standard JSON.parse() workflow:
--------------------------------
[Read ENTIRE file into memory]
|
v
[Parse ENTIRE string into object]
|
v
[Process the data]
Problem: A 500 MB JSON file requires 500 MB+ of RAM just to load
before you can do anything with it.
Types of JSON Streaming
1. Newline-Delimited JSON (NDJSON)
NDJSON is the most common streaming format. Each line is a complete, valid JSON object. There is no outer array wrapping everything. Each line is independent.
NDJSON File Example (logs.ndjson)
{"timestamp":"2026-09-01T08:00:01Z","level":"INFO","message":"Server started"}
{"timestamp":"2026-09-01T08:00:05Z","level":"INFO","message":"User 42 logged in"}
{"timestamp":"2026-09-01T08:01:12Z","level":"WARN","message":"High memory usage"}
{"timestamp":"2026-09-01T08:02:45Z","level":"ERROR","message":"DB connection failed"}
{"timestamp":"2026-09-01T08:02:50Z","level":"INFO","message":"DB reconnected"}
Each line is a self-contained JSON object. You can process them one by one, stop at any line, or start from any line. This format is used for log files, data exports, and real-time event streams.
2. JSON Lines (JSONL)
JSONL is essentially the same as NDJSON. The file extension is .jsonl. Each line is one JSON value. It is popular in machine learning for training datasets because you can easily shuffle lines or process a subset.
3. Streaming Array (Chunked)
Some APIs stream a JSON array by sending chunks. The response starts with [, sends each item one by one, and ends with ]. The client processes each item as it arrives without waiting for the closing bracket.
Diagram: Standard vs Streaming JSON Processing
STANDARD JSON (Load All First)
--------------------------------
Server sends all data
↓
Client stores 500 MB in RAM
↓
Parse complete document
↓
Start processing record 1
↓
Process record 2...
↓
Process record 500,000
STREAMING JSON (Process As It Arrives)
---------------------------------------
Server sends record 1
↓ ← Process record 1 immediately
Server sends record 2
↓ ← Process record 2
...continues...
↓
Server sends record 500,000
↓ ← Process record 500,000
RAM usage: Only 1 record at a time instead of 500,000
Reading NDJSON in Node.js
Line-by-Line Processing with readline
const fs = require('fs');
const readline = require('readline');
const fileStream = fs.createReadStream('./logs.ndjson');
const rl = readline.createInterface({
input: fileStream,
crlfDelay: Infinity // Handle Windows line endings
});
let errorCount = 0;
rl.on('line', (line) => {
// Each line is a complete JSON object
if (line.trim() === '') return; // Skip empty lines
const logEntry = JSON.parse(line);
if (logEntry.level === 'ERROR') {
errorCount++;
console.log('Error found:', logEntry.message);
}
});
rl.on('close', () => {
console.log(`Total errors found: ${errorCount}`);
console.log('Processing complete.');
});
Writing NDJSON in Node.js
const fs = require('fs');
const logs = [
{ timestamp: "2026-09-01T08:00:01Z", level: "INFO", message: "Server started" },
{ timestamp: "2026-09-01T08:00:05Z", level: "INFO", message: "User logged in" },
{ timestamp: "2026-09-01T08:02:45Z", level: "ERROR", message: "DB failed" }
];
// Write each object as one line
const writer = fs.createWriteStream('./output.ndjson');
logs.forEach(log => {
writer.write(JSON.stringify(log) + '\n');
});
writer.end(() => {
console.log('NDJSON file written successfully');
});
Reading NDJSON in Python
error_count = 0
with open('logs.ndjson', 'r', encoding='utf-8') as file:
for line in file:
line = line.strip()
if not line:
continue # Skip empty lines
log_entry = json.loads(line) # Parse one line at a time
if log_entry['level'] == 'ERROR':
error_count += 1
print('Error:', log_entry['message'])
print(f'Total errors: {error_count}')
Python processes one line at a time from the file. The entire 500 MB file never loads into memory.
Streaming JSON from an HTTP API in Node.js
When an API streams data, you process each chunk as it arrives instead of buffering everything.
const https = require('https');
let buffer = '';
https.get('https://api.example.com/events/stream', (response) => {
response.on('data', (chunk) => {
buffer += chunk.toString();
// Process complete lines from the buffer
const lines = buffer.split('\n');
buffer = lines.pop(); // Keep any incomplete last line in buffer
lines.forEach(line => {
if (line.trim()) {
const event = JSON.parse(line);
console.log('Received event:', event.type, event.data);
}
});
});
response.on('end', () => {
console.log('Stream ended');
});
});
Server-Sent Events (SSE) with JSON
Server-Sent Events is a browser standard for receiving real-time updates from a server. Each event typically contains JSON data. This is how live sports scores, chat messages, and notifications work.
Server (Node.js with Express)
const express = require('express');
const app = express();
app.get('/scores', (req, res) => {
res.setHeader('Content-Type', 'text/event-stream');
res.setHeader('Cache-Control', 'no-cache');
res.setHeader('Connection', 'keep-alive');
// Send a score update every 3 seconds
const interval = setInterval(() => {
const scoreUpdate = {
team: "India",
score: Math.floor(Math.random() * 300),
wickets: Math.floor(Math.random() * 10),
timestamp: new Date().toISOString()
};
res.write(`data: ${JSON.stringify(scoreUpdate)}\n\n`);
}, 3000);
// Clean up when client disconnects
req.on('close', () => {
clearInterval(interval);
res.end();
});
});
app.listen(3000);
Browser Client
const eventSource = new EventSource('/scores');
eventSource.onmessage = (event) => {
const scoreUpdate = JSON.parse(event.data);
document.getElementById('score').textContent =
`${scoreUpdate.team}: ${scoreUpdate.score}/${scoreUpdate.wickets}`;
};
eventSource.onerror = () => {
console.log('Connection error or stream ended');
};
Real-World Use Cases for JSON Streaming
- Log processing — Parse millions of log lines without loading everything into RAM.
- Data export tools — Export database tables as NDJSON for transfer or backup.
- Machine learning datasets — JSONL files store training data line by line for easy shuffling.
- Live chat and notifications — SSE streams JSON events to the browser in real time.
- Large API responses — Stream a list of 100,000 products without waiting for all 100,000 before showing any.
- Financial data feeds — Stream stock ticks or trade events as they happen.
NDJSON vs JSON Array
Feature NDJSON JSON Array
------- ------ ----------
Can stream Yes - process line by line No - need full array first
File size limit No practical limit Limited by available RAM
Append new items Just add a new line Must re-parse the whole file
Validate one item Parse just that line Must parse entire array
Shuffle/split data Easy - rearrange lines Complex without full parse
Summary
JSON streaming processes data piece by piece instead of loading everything at once. Newline-Delimited JSON (NDJSON) and JSON Lines (JSONL) store one JSON object per line, making them easy to stream and process in any order. Node.js and Python both have built-in tools to process NDJSON line by line with minimal memory usage. Server-Sent Events stream JSON events from server to browser in real time. Use streaming whenever your JSON dataset is too large to load into memory all at once, or when you need real-time data delivery.