Python LiDAR

Which PDAL Filters Break Streaming Mode

TL;DR: A filter streams when its decision about a point depends on nothing but that point — filters.range, filters.assign, filters.ferry, filters.crop and filters.reprojection all qualify. Anything that needs neighbours, a global ordering or a completed raster — filters.smrf, filters.outlier, filters.hag_nn, filters.sort, filters.sample, writers.gdal — blocks, and one blocking stage makes the whole pipeline non-streamable.

# Context and Motivation

This guide is part of Streaming Mode Execution in PDAL, which describes the chunked pull loop that keeps memory bounded. Here we answer the question that decides whether you can use it at all: given a chain of stages, which ones will refuse?

The rule is not arbitrary and it is not a list to memorise. A streaming stage receives one chunk, must produce its answer for those points, and then loses access to them forever. A filter can only work under that constraint if the decision for each point is a pure function of that point’s own dimensions. The moment a filter needs to know something about a different point — the elevation of a neighbour, the rank of this point in a global sort, the number of returns within a radius — it must have the whole cloud, and PDAL marks it non-streamable. Once you see the rule that way, you can classify a filter you have never used before by asking a single question.

One question decides it A decision diagram. Starting from a filter, the question is whether the answer for one point depends on any other point. If no, the filter streams and sees each point once. If yes, it needs the whole cloud in memory and blocks streaming for the entire pipeline, no matter how many streamable stages surround it. does the answer for this point need another point? no yes streams one chunk at a time, memory bounded blocks whole cloud resident before it decides and one blocking stage anywhere in the chain makes every other stage block too — capability is a property of the pipeline

# Prerequisites and Assumptions

Requirement Detail
PDAL 2.3+ — earlier versions stream fewer stages
Python pdal bindings for pipeline.streamable
A pipeline to classify the rules below apply to the chain, not to any one stage
pdal --options <stage> prints the options a stage accepts, some of which change its capability

Assume nothing from a stage’s family name. filters.range streams and filters.sample does not, even though both are “filters that remove points”.

# Step-by-Step Implementation

# Step 1 — Ask PDAL rather than guessing

python
import pdal
p = pdal.Pipeline(spec)
print(p.streamable)

That single boolean is the ground truth for your exact chain with your exact options.

# Step 2 — Find the culprit by bisection

If the answer is False, remove stages from the end until it flips. The last stage you removed is the blocker. On the command line the same information arrives faster:

bash
pdal pipeline chain.json --stream --verbose 8 2>&1 | grep -i stream

# Step 3 — Classify what you found

Blocking stages fall into four families, and the family tells you the workaround.

  • Neighbourhood filters — filters.outlier, filters.hag_nn, filters.neighborclassifier, filters.cluster. Each needs a spatial index over all points.
  • Ordering filters — filters.sort, filters.mortonorder. A global sort is definitionally not per-point.
  • Surface builders — filters.smrf, filters.pmf, filters.dem. Each constructs a raster from every point before classifying any.
  • Accumulating writers — writers.gdal when it must hold the raster, writers.ogr when it aggregates.

# Step 4 — Choose a workaround

Three exist, in order of preference: replace the stage with a streamable equivalent, move it to a second pass over much less data, or accept standard mode for that stage and bound the input by tiling.

# Complete Working Example

classify_chain.py reports which stages in a pipeline block streaming, by testing prefixes of the chain.

python
"""Report the first stage in a PDAL pipeline that prevents streaming."""
from __future__ import annotations

import json
from typing import Any

import pdal


def first_blocking_stage(stages: list[dict[str, Any]]) -> int | None:
    """Return the index of the first stage that makes the chain non-streamable."""
    for cut in range(1, len(stages) + 1):
        prefix = stages[:cut]
        # A chain must end in something PDAL can execute; a bare reader is fine.
        pipeline = pdal.Pipeline(json.dumps({"pipeline": prefix}))
        if not pipeline.streamable:
            return cut - 1
    return None


def report(spec: dict[str, Any]) -> None:
    stages = spec["pipeline"]
    idx = first_blocking_stage(stages)
    if idx is None:
        print("streamable: every stage in the chain supports streaming")
        return

    blocker = stages[idx]
    print(f"blocked at stage {idx}: {blocker.get('type', '<inferred>')}")
    print(json.dumps(blocker, indent=2))
    print("\nstreamable prefix:")
    for s in stages[:idx]:
        print("  -", s.get("type", "<inferred>"))


if __name__ == "__main__":
    spec = {
        "pipeline": [
            {"type": "readers.las", "filename": "tile_0431.laz"},
            {"type": "filters.range", "limits": "Z[-30:5000]"},
            {"type": "filters.reprojection", "out_srs": "EPSG:6318"},
            {"type": "filters.outlier", "method": "statistical", "mean_k": 12},
            {"type": "writers.las", "filename": "out.laz"},
        ]
    }
    report(spec)

Output names the offender and, just as usefully, the prefix that would still stream:

text
blocked at stage 3: filters.outlier
streamable prefix:
  - readers.las
  - filters.range
  - filters.reprojection
What --verbose 8 actually tells you Six log lines from a verbose PDAL run. Each stage reports its streaming capability in order, and one line stands out: the statistical outlier filter reports that it is not streamable. The final line is the pipeline-level verdict, which follows from that single stage regardless of what the other four said. readers.las streamable filters.range streamable filters.reprojection streamable filters.outlier NOT streamable writers.las streamable pipeline not streamable pdal pipeline chain.json --stream --verbose 8, one line per stage the fourth row is the answer; the sixth is only its consequence

# Key Parameter Table

Stage Streams Why
filters.range yes Tests one point’s dimensions against fixed limits
filters.expression yes Same, with a richer predicate syntax
filters.assign yes Writes a value into one point
filters.ferry yes Copies a dimension within one point
filters.crop yes Tests one coordinate against a fixed geometry
filters.reprojection yes Transforms one coordinate independently
filters.outlier no Needs the k nearest neighbours of each point
filters.smrf / filters.pmf no Builds a minimum surface from every point first
filters.hag_nn no Searches for nearby ground points
filters.sort no A global ordering is not a per-point decision
filters.sample no Poisson sampling depends on points already kept
writers.gdal no Accumulates raster cells across the whole input

# Verification

Assert the property in the code that builds the pipeline, not in a comment:

python
pipeline = pdal.Pipeline(spec)
assert pipeline.streamable, "a stage was added that blocks streaming"

Put that assertion in a unit test alongside the pipeline definition and a future edit that adds filters.sample fails the test rather than the production job. The same idea, applied to the whole pipeline rather than one property, is the subject of validating PDAL pipelines in CI.

# Gotchas and Edge Cases

Options can change capability. writers.gdal and a handful of others advertise different capability depending on how they are configured. Always test the chain you will actually run rather than a simplified version.

A reader can block too. Most stream, but a reader that has to sort or index its source before yielding points does not. If bisection points at stage zero, the reader is the problem, and converting the source to LAZ or COPC usually solves it.

Four families of blocker, four workarounds Blocking stages grouped into four families with the workaround for each. Neighbourhood filters can often move to a second pass over a decimated cloud. Ordering filters can be dropped when downstream stages do not depend on order. Surface builders belong in their own pass over a cropped tile. Accumulating writers should be separated from the point-domain filtering entirely. family the workaround it allows neighbourhood — outlier, hag_nn needs a spatial index over all points second pass over a cropped or decimated cloud ordering — sort, mortonorder a global rank is not per-point drop it unless a later stage truly needs the order surface builders — smrf, pmf rasterizes everything before deciding its own pass, on a tile small enough to fit accumulating writers — gdal holds the raster until the end split point filtering from rasterization

Streamable does not mean cheap. filters.reprojection streams happily and is still one of the more expensive per-point operations in PDAL. Capability and cost are separate axes, and the filtering logic guide covers the second one.

A stage that streams today may not tomorrow. Capability has broadened across PDAL releases and occasionally narrowed. Pin the version — the version drift problem applies here as much as anywhere.

# Frequently Asked Questions

What single rule decides whether a filter streams?

Whether the answer for one point depends on any other point. A filter that tests one point’s own dimensions streams; a filter that needs a neighbour, a global ordering or a completed raster does not. Every entry in the blocking list follows from that one rule.

Can I make a blocking filter stream by changing its options?

Occasionally. A few stages advertise different capability under different configurations, so it is always worth testing the exact chain you will run. But for the neighbourhood and surface-building filters the answer is structural — no option makes SMRF able to classify a point without seeing its surroundings.

Does one blocking stage really disable streaming for the whole pipeline?

Yes. Capability is negotiated across the chain before any point moves, so a single blocking stage forces the entire pipeline into standard mode. That is why splitting a workflow into a streaming pass and a small blocking pass is usually more effective than trying to optimise the mixed chain.

Is filters.crop streamable even with a complex polygon?

Yes. Testing whether one coordinate falls inside a fixed geometry is a per-point decision however complicated the geometry is. The polygon is loaded once at pipeline start and reused for every chunk, so complexity costs CPU rather than memory.

Related in Streaming Mode Execution in PDAL

↑ Streaming Mode Execution in PDAL overview --- title: "Streaming Mode Execution in PDAL" description: "How PDAL's streaming execution model moves points through a pipeline in... Running a PDAL Pipeline in Streaming Mode --- title: "Running a PDAL Pipeline in Streaming Mode" description: "Step-by-step recipe for executing a PDAL pipeline with executestreaming... Splitting a Blocking Pipeline into Two Passes --- title: "Splitting a Blocking Pipeline into Two Passes" description: "How to cut a PDAL pipeline at its blocking stage so the per-point...