Large JSON Payloads: Streaming Parsers vs In-Memory Deserialization

Overview and Problem Statement

Modern backend services increasingly ingest JSON payloads that exceed what naive parsing strategies can handle safely. A batch export from a data warehouse, a webhook delivering a full order history, or a third-party API returning an uncapped result set can easily reach hundreds of megabytes. The way a service chooses to deserialize that payload, either loading it entirely into memory or processing it incrementally, has direct consequences for stability, latency, and infrastructure cost.

In Node.js specifically, this choice stops being optional once payload size crosses a certain threshold. V8's default heap size is capped (historically around 1.5 GB to 2 GB for the old generation on 64-bit systems, adjustable via the --max-old-space-size flag), and a single JSON.parse() call on a sufficiently large string will throw FATAL ERROR: Reached heap limit, Allocation failed - JavaScript heap out of memory and crash the process. The JVM heap and Python's interpreter memory face comparable pressure under the same workload, though the failure mode and thresholds differ.

This article compares in-memory (DOM-style) deserialization against streaming (SAX-style) parsing, and covers where the newline-delimited JSON (NDJSON) format fits into that architecture.

Core Architectural Concepts

In-Memory (DOM-style) Deserialization

The default behavior of most JSON libraries, including JSON.parse() in JavaScript, encoding/json.Unmarshal() in Go, and Jackson's data-binding ObjectMapper.readValue() in Java, is to read the entire input, build a complete in-memory representation (an object graph or document tree), and return it in one call.

This model is simple to reason about: once the call returns, the full structure is available for random access, and no partial-state handling is required. The tradeoff is memory. Peak usage during deserialization is typically several times the size of the raw payload (the raw bytes, the decoded string, and the resulting object graph with per-key overhead all coexist briefly), so a 300 MB JSON file can realistically require 1 GB or more of heap during the parse.

Streaming (SAX-style) Parsers

A streaming parser reads the input incrementally and emits low-level tokens (start of object, field name, string value, end of array, and so on) as it encounters them, rather than materializing a full tree. The caller's code reacts to each token as it arrives and decides what to keep in memory and what to discard.

Three concrete implementations illustrate this pattern across ecosystems:

• Jackson's Streaming API (com.fasterxml.jackson.core.JsonParser) in Java exposes a nextToken() method that advances the parser one token at a time, independent of Jackson's higher-level data-binding layer.
• ijson in Python wraps a C-based YAJL parser and yields (prefix, event, value) tuples through a generator, so a caller can iterate over a multi-gigabyte file with constant memory overhead.
• json.Decoder in Go's standard library reads from any io.Reader and supports Decode() in a loop for a stream of top-level values, or Token() for full SAX-style token access within a single large object or array.

Because none of these approaches build a full document tree, memory usage during parsing is bounded by nesting depth and whatever fields the caller retains, not by total payload size.

NDJSON: A Streaming-Friendly Payload Format

Newline-delimited JSON (NDJSON) represents a sequence of independent JSON values, one per line, with no enclosing array and no separating commas. Each line is a complete, self-contained JSON document:

{"user_id": 4821, "event": "login", "ts": 1758099200}
{"user_id": 4821, "event": "page_view", "ts": 1758099214}
{"user_id": 9013, "event": "logout", "ts": 1758099230}

This format matters for streaming architecture for two reasons. First, a consumer can process the stream line by line using a plain buffered reader, decoding each line as an independent, complete JSON value, without needing a stateful parser that tracks array bracket depth across chunks. Second, it composes naturally with Unix-style tooling and log pipelines: lines can be split, filtered, or piped between processes without special JSON-aware handling at the transport layer.

A single large JSON array containing the same records forces a different tradeoff: a streaming consumer still needs a parser capable of recognizing array-element boundaries mid-stream, since a naive line-by-line read would not align with JSON value boundaries. Closing that gap is exactly what token-based parsers like json.Decoder or ijson are designed to do.

Implementation Caveats: Malformed Lines and Unbounded Buffers

Streaming does not eliminate error handling, it relocates it. Two issues surface in production NDJSON consumers that a single blocking JSON.parse() call never has to deal with.

First, a single malformed line (a truncated write, an encoding error, a producer crash mid-line) should not necessarily abort the entire stream. Most NDJSON consumers catch the decode error per line, log the offending line with its byte offset, and continue to the next line, rather than propagating the error and discarding everything already processed.

Second, a line reader still needs a maximum line-length limit. If a producer emits one pathologically large line (a single record with a multi-gigabyte embedded field, for example), an unbounded bufio.Scanner or readline() call will buffer that entire line before yielding it, reintroducing the same memory spike that streaming was meant to avoid. Setting an explicit buffer cap (bufio.Scanner.Buffer() in Go, for instance) and treating an oversized line as a rejected record, rather than growing the buffer indefinitely, keeps the memory guarantee intact.

Practical Comparison Table

The table below summarizes the practical differences between the two approaches.

Code Implementation

The following Go example uses json.Decoder to process an NDJSON stream one record at a time, keeping memory usage constant regardless of how many lines the stream contains.

package main

import (
	"encoding/json"
	"fmt"
	"io"
	"os"
)

type Event struct {
	UserID int    `json:"user_id"`
	Event  string `json:"event"`
	Ts     int64  `json:"ts"`
}

func main() {
	file, err := os.Open("events.ndjson")
	if err != nil {
		panic(err)
	}
	defer file.Close()

	decoder := json.NewDecoder(file)
	var count int

	for {
		var evt Event
		if err := decoder.Decode(&evt); err != nil {
			if err == io.EOF {
				break
			}
			panic(err)
		}
		// Process one record at a time; nothing accumulates in memory.
		count++
		if evt.Event == "login" {
			fmt.Printf("login detected for user %d at %d\n", evt.UserID, evt.Ts)
		}
	}

	fmt.Printf("processed %d records\n", count)
}

Each call to decoder.Decode(&evt) advances the underlying reader just far enough to fill one Event struct, then discards its internal buffer state for that value. Memory usage stays flat whether the file contains a thousand lines or a hundred million.

Conclusion & Next Steps

In-memory deserialization remains the right default for small, bounded payloads where implementation simplicity and immediate random field access matter more than memory footprint. Streaming parsers become necessary once payload size is large or unbounded, particularly for NDJSON-based log ingestion, ETL pipelines, and any service where a single oversized request should not be able to crash the process. Jackson's streaming API, ijson, and Go's json.Decoder all provide this without external infrastructure changes, just a different parsing loop.

Validate Your JSON Payloads with Yurlie

Whether your service deserializes in memory or streams through records one at a time, validating structure and syntax before it reaches production code catches malformed payloads early. Check formatting and RFC 8259 compliance instantly with our free Yurlie JSON Formatter and Validator.