Streaming Mode Execution in PDAL

Every PDAL pipeline you write has two possible execution strategies, and by default you get the one that reads the entire point cloud into memory before the first filter sees a point. That is the right choice for a stage that has to look at the whole cloud — a ground classifier building a minimum surface cannot decide about one point without seeing its neighbours. It is the wrong choice for a chain of per-point operations on a 40 GB tile, where it turns a small job into an out-of-memory failure. Streaming mode is the alternative: points move through the chain in fixed-size chunks, each chunk is filtered and written before the next is read, and peak memory is set by the chunk size instead of the file size. This topic belongs to PDAL Pipeline Architecture and Execution, and it is the single largest lever on what hardware a given pipeline needs.

# Prerequisites

Streaming is not a flag you can bolt onto an arbitrary pipeline. It is a property the whole chain either has or does not, and the work of adopting it is almost entirely the work of arranging your stages so that it does.

# Core Workflow Architecture

Streaming execution runs as a pull loop rather than a sequence of phases. The writer asks for points, the request travels back up the chain to the reader, and each stage transforms whatever arrives before passing it on.

1. Capability negotiation. Before any point moves, PDAL walks the chain and asks each stage whether it can operate in streaming mode. The answer is a property of the stage type and sometimes of its options. If any stage says no, the pipeline as a whole is not streamable.
2. Buffer allocation. A single PointView sized to chunk_size is allocated. Unlike standard mode, this buffer is reused for every chunk rather than reallocated, so allocation cost is paid once.
3. Fill. The reader decodes points into the buffer until it holds chunk_size points or the source is exhausted.
4. Traverse. Each filter in order receives the buffer, mutates it in place or marks points for removal, and hands it on. A filter that drops points shrinks the buffer’s occupied count rather than allocating a new one.
5. Drain. The writer consumes the surviving points and appends them to the output. For LAS this is a straight append; for writers.gdal the raster accumulator absorbs them and the raster itself is written at the end.
6. Repeat or finish. Control returns to step three until the reader reports exhaustion, at which point every stage is given a chance to flush anything it was holding.

The consequence worth internalising is in step four: a streaming filter sees each point exactly once and cannot look at points it has already released or has not yet received. That is precisely why some filters cannot stream, and understanding which is the subject of which PDAL filters break streaming mode.

# Full Implementation

The module below runs a streaming pipeline over a tile, reports whether streaming was actually used, and measures peak resident memory so the claim is verifiable rather than assumed.

"""Stream a LAZ tile through a filter chain with bounded memory."""
from __future__ import annotations

import json
import logging
import resource
from pathlib import Path

import pdal

LOG = logging.getLogger("stream_tile")


def build_pipeline(src: Path, dst: Path, epsg: str = "EPSG:6318") -> str:
    """A chain of strictly per-point stages, so the whole thing streams."""
    stages = [
        {"type": "readers.las", "filename": str(src)},
        # Elevation sanity limits: a pure per-point predicate.
        {"type": "filters.range", "limits": "Z[-30:5000]"},
        # Drop points already flagged as noise upstream.
        {"type": "filters.range", "limits": "Classification![7:7]"},
        # Coordinate transform: each point is transformed independently.
        {"type": "filters.reprojection", "out_srs": epsg},
        {
            "type": "writers.las",
            "filename": str(dst),
            "compression": "laszip",
            "minor_version": 4,
            "dataformat_id": 6,
            "forward": "all",
        },
    ]
    return json.dumps({"pipeline": stages})


def run(src: Path, dst: Path, chunk_size: int = 250_000) -> dict:
    pipeline = pdal.Pipeline(build_pipeline(src, dst))

    if not pipeline.streamable:
        raise RuntimeError(
            "pipeline is not streamable — a stage in the chain requires the whole cloud"
        )

    LOG.info("streaming %s with chunk_size=%d", src.name, chunk_size)
    count = pipeline.execute_streaming(chunk_size=chunk_size)

    peak_kb = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
    stats = {
        "points_written": count,
        "peak_rss_mb": round(peak_kb / 1024, 1),
        "chunk_size": chunk_size,
        "output": str(dst),
    }
    LOG.info("wrote %d points, peak RSS %.1f MB", count, stats["peak_rss_mb"])
    return stats


if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
    result = run(Path("tile_0431.laz"), Path("tile_0431_clean.laz"))
    print(json.dumps(result, indent=2))

# Code Breakdown

pipeline.streamable is checked before executing, not after. The property is cheap — it only asks each stage about its capability — and turning a silent fallback into an explicit exception is the difference between a job that quietly uses 30 GB and one that tells you why it cannot.

Every stage in the chain is per-point on purpose. filters.range tests one point’s dimensions. filters.reprojection transforms one coordinate. Neither needs a neighbour. Had the chain included filters.smrf, pipeline.streamable would be False and the guard would fire.

chunk_size is passed explicitly rather than defaulted. The default of 10,000 points is conservative; a quarter of a million costs about 10 MB of working set on a typical point layout and removes most of the per-chunk overhead. The trade-off is quantified under Performance Tuning below.

execute_streaming returns the point count, not a PointView. This is not an oversight: there is no complete array to return, because no complete array ever existed. Pipelines that need pipeline.arrays afterwards are, by construction, not streaming pipelines.

forward: "all" keeps the header records. Streaming does not change what a writer preserves, but it is easy to lose sight of the ordinary options while focusing on execution mode. The attribute mapping guide covers what forward carries.

# Parameter Reference Table

# Validation and Integrity Checks

A streaming run is only useful if it produced the same answer more cheaply. Three checks establish that.

Point counts agree. Run the pipeline conventionally on a small tile and streaming on the same tile. The returned counts must match exactly; a mismatch means a stage behaved differently, not that streaming rounded something.

plain = pdal.Pipeline(build_pipeline(src, Path("a.laz")))
plain.execute()
streamed = pdal.Pipeline(build_pipeline(src, Path("b.laz")))
n = streamed.execute_streaming(chunk_size=100_000)
assert len(plain.arrays[0]) == n, "streaming changed the surviving point count"

Memory actually stayed bounded. Peak RSS should be roughly constant as the input grows. Run the same pipeline against a 200 MB tile and a 2 GB tile; if peak memory scales with the file, something in the chain is accumulating despite reporting itself streamable.

The header survived. Streaming writers still recompute counts and bounding boxes, but it is worth confirming — pdal info --metadata on the output, compared against the input, is the check described in metadata and header sync.

# Performance Tuning

Streaming trades a small amount of throughput for a large amount of memory headroom, and chunk_size is the dial that sets the exchange rate.

Two further levers matter in production. First, streaming and file-level parallelism compose well: because each worker’s memory is bounded, you can run many more concurrent tile processes than standard mode allows on the same box — see parallel execution for how to size that pool. Second, streaming makes cloud reads far more attractive, since a ranged read feeding a bounded buffer never needs the whole object locally; that pattern is covered in streaming LAZ from S3 with PDAL.

# Common Errors and Troubleshooting

pdal pipeline --stream exits with “Pipeline is not streamable”. One stage in the chain blocks. Run with --verbose 8 and PDAL names it. The fix is either to remove the stage, move it into a second pipeline that runs on a much smaller input, or accept standard mode for that step.

Peak memory did not drop. Confirm execute_streaming was actually called, not execute. It is easy to build the pipeline correctly, check streamable, and then execute the conventional way — the guard passes and nothing changes.

Output point count is zero. A filters.crop or filters.range predicate that matches nothing behaves identically in both modes, but the failure is more visible when streaming because there is no intermediate array to inspect. Test the predicate on a small tile with execute() and check pipeline.arrays[0].shape.

Throughput collapsed compared with standard mode. The chunk size is too small for the work being done, so per-chunk overhead dominates. Raise it tenfold and re-measure before changing anything else.

writers.gdal refuses to stream. Rasterization accumulates cells across the whole input by nature. Split the job: stream the point-domain filtering to an intermediate LAZ, then rasterize that in a second, conventional pipeline.

Streaming from a network source stalls. A bounded buffer only helps if the source can feed it. Reading a LAZ object over a virtual filesystem with the default settings issues a directory listing on every open and fetches small ranges, so the pipeline spends its time waiting rather than filtering. The symptom is a flat, low memory trace with dismal throughput — the opposite of the standard-mode failure, and easy to misread as a streaming problem when it is a network one.

A second pipeline in the same process inherits the memory. Python does not necessarily return freed pages to the operating system, so ru_maxrss after a streaming run that followed a conventional run still shows the earlier peak. When benchmarking, run each mode in its own process, or the numbers will suggest streaming did nothing.

A last point about where streaming pays off most. The obvious case is the tile too large for the worker, but the more common one in production is concurrency: because each streaming worker has a small, predictable footprint, a sixteen-core machine can run sixteen tile jobs at once instead of the three that standard mode’s memory appetite allowed. The throughput gain there comes not from any one pipeline running faster — it does not — but from the machine running many more of them at the same time. That is usually the argument that decides whether it is worth restructuring a chain to keep it streamable.

# Frequently Asked Questions

How do I know whether a PDAL pipeline can stream?

Ask the pipeline itself. In Python, pipeline.streamable returns True only when every stage in the chain supports streaming. On the command line, pdal pipeline --stream fails with a message naming the blocking stage. Never infer it from the stage list alone, because whether a filter streams can depend on the options you gave it.

Why does streaming mode use more time but less memory?

Streaming processes a fixed number of points at a time, so the pipeline pays per-chunk overhead repeatedly instead of once. In exchange the resident set is bounded by the chunk rather than the file, which is what lets a 40 GB tile run on a worker with 4 GB of memory.

Does streaming change the result of a pipeline?

For stages that stream, no. A streamable filter by definition decides using only the point in front of it, so the output is identical. The risk is not wrong numbers but a stage you assumed was streaming turning out not to be.

What chunk size should I use?

Start at 100,000 and raise it toward 500,000 only if profiling shows per-chunk overhead dominating. Past half a million points the throughput gain flattens while memory keeps climbing.

# Related

• PDAL Pipeline Architecture and Execution — the section this execution mode belongs to
• Running a PDAL Pipeline in Streaming Mode — the hands-on walkthrough with a real tile
• Which PDAL Filters Break Streaming Mode — the capability rules and how to work around them
• Splitting a Blocking Pipeline into Two Passes — keeping most of a workflow streamable when one stage cannot be
• Memory Management in PDAL Pipelines — the buffer model streaming is an alternative to
• PDAL Stage Chaining — how stages hand points to one another in either mode