Python LiDAR

Splitting a Blocking Pipeline into Two Passes

TL;DR: Cut the chain at the blocking stage: run everything above it as a streaming pass that writes an intermediate LAZ, run the blocking stage alone over that much smaller file, then stream the tail. Two passes over reduced data almost always beat one pass that has to materialise the original cloud.

# Context and Motivation

This guide is part of Streaming Mode Execution in PDAL. The parent explains why a single blocking stage makes an entire pipeline non-streamable; this page is what to do about it when the blocking stage is one you actually need.

The situation is common enough to be the normal case. A production chain reads a tile, applies elevation limits, reprojects, classifies ground with filters.smrf, keeps the ground returns and writes a DTM. Four of those six stages stream. The classifier does not, and neither does the raster writer, so the whole pipeline runs conventionally and the reader materialises 18 million points before anything else happens. Splitting the chain does not make SMRF stream — nothing does — but it changes what SMRF has to hold. If the streaming pass has already discarded noise and cropped to the area of interest, the blocking stage starts from a third of the points, and a job that needed 12 GB now needs 4.

Cutting the chain at the blocking stage Above, a single pipeline of six stages where one blocking classifier forces the whole chain into standard mode and 12 gigabytes of peak memory. Below, the same six stages split into three runs: a streaming pass that filters and reprojects into an intermediate file, a short conventional pass that classifies it, and a streaming tail that selects and writes. Peak memory drops to 3.9 gigabytes because the blocking stage now starts from a third of the points. one pipeline — 12.1 GB peak read 18.4 M range + crop reproject smrf — blocks write DTM three runs — 3.9 GB peak pass 1 — streaming range, crop, reproject · 40 MB pass 2 — conventional smrf on 6.1 M points · 3.9 GB pass 3 — streaming select ground, write · 40 MB the blocking stage did not change — what changed is that it now starts from 6.1 M points instead of 18.4 M the intermediate file is the cost: one extra write and one extra read, usually a few seconds against gigabytes of headroom

# Prerequisites and Assumptions

Requirement Detail
PDAL 2.3+ with the Python bindings
Scratch space room for one intermediate LAZ, typically 20–40% of the input
A known blocker identified as in which filters break streaming
Ordering freedom the reducing stages must be safe to run before the blocking one

That last row is the real constraint. Moving filters.crop ahead of filters.smrf is safe if the crop boundary is generous enough that the classifier still sees the terrain context it needs. Moving it ahead with a tight boundary starves SMRF of the surroundings at the tile edge and produces a different, worse answer. Buffer the crop — the same reasoning as tile buffering in DTM raster generation.

# Step-by-Step Implementation

# Step 1 — Identify the cut point

Bisect the chain until pipeline.streamable flips. Everything before the blocker is pass one; the blocker is pass two; everything after is pass three.

# Step 2 — Make pass one reduce as much as it safely can

This is where the benefit comes from. Elevation limits, noise-class rejection and a buffered crop all stream, and each one shrinks what pass two must hold.

json
{"pipeline": [
  {"type": "readers.las", "filename": "tile_0431.laz"},
  {"type": "filters.range", "limits": "Z[-30:5000]"},
  {"type": "filters.range", "limits": "Classification![7:7]"},
  {"type": "filters.crop", "polygon": "POLYGON((...))"},
  {"type": "filters.reprojection", "out_srs": "EPSG:6318"},
  {"type": "writers.las", "filename": "/scratch/pass1.laz", "compression": "laszip", "forward": "all"}
]}

# Step 3 — Run the blocking stage alone

json
{"pipeline": [
  {"type": "readers.las", "filename": "/scratch/pass1.laz"},
  {"type": "filters.smrf", "window": 18, "slope": 0.15, "threshold": 0.5, "cell": 1.0},
  {"type": "writers.las", "filename": "/scratch/pass2.laz", "compression": "laszip", "forward": "all"}
]}

# Step 4 — Stream the tail

Selecting ground and writing the output are both per-point operations, so pass three streams like pass one.

# Step 5 — Delete the intermediates deliberately

Scratch files that survive a crash will be picked up by the next run and silently reused. Write them under a run-specific prefix and remove them on success.

# Complete Working Example

python
"""Split a pipeline at its blocking stage and run the three passes in order."""
from __future__ import annotations

import json
import logging
import shutil
import tempfile
from pathlib import Path

import pdal

LOG = logging.getLogger("split_passes")


def run_streaming(stages: list[dict], chunk: int = 100_000) -> int:
    pipeline = pdal.Pipeline(json.dumps({"pipeline": stages}))
    if not pipeline.streamable:
        raise RuntimeError("pass was expected to stream but does not")
    return pipeline.execute_streaming(chunk_size=chunk)


def run_standard(stages: list[dict]) -> int:
    return pdal.Pipeline(json.dumps({"pipeline": stages})).execute()


def process(src: Path, dst: Path, polygon: str, out_srs: str = "EPSG:6318") -> dict:
    scratch = Path(tempfile.mkdtemp(prefix="pdal_split_"))
    p1 = scratch / "pass1.laz"
    p2 = scratch / "pass2.laz"
    try:
        n1 = run_streaming([
            {"type": "readers.las", "filename": str(src)},
            {"type": "filters.range", "limits": "Z[-30:5000]"},
            {"type": "filters.range", "limits": "Classification![7:7]"},
            {"type": "filters.crop", "polygon": polygon},
            {"type": "filters.reprojection", "out_srs": out_srs},
            {"type": "writers.las", "filename": str(p1),
             "compression": "laszip", "forward": "all"},
        ])
        LOG.info("pass 1 (streaming) kept %d points", n1)

        n2 = run_standard([
            {"type": "readers.las", "filename": str(p1)},
            {"type": "filters.smrf", "window": 18, "slope": 0.15,
             "threshold": 0.5, "cell": 1.0},
            {"type": "writers.las", "filename": str(p2),
             "compression": "laszip", "forward": "all"},
        ])
        LOG.info("pass 2 (standard, blocking) classified %d points", n2)

        n3 = run_streaming([
            {"type": "readers.las", "filename": str(p2)},
            {"type": "filters.range", "limits": "Classification[2:2]"},
            {"type": "writers.las", "filename": str(dst),
             "compression": "laszip", "forward": "all"},
        ])
        LOG.info("pass 3 (streaming) wrote %d ground points", n3)
        return {"pass1": n1, "pass2": n2, "ground": n3}
    finally:
        shutil.rmtree(scratch, ignore_errors=True)


if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO)
    poly = "POLYGON((512000 4783000, 513000 4783000, 513000 4784000, 512000 4784000, 512000 4783000))"
    print(json.dumps(process(Path("tile_0431.laz"), Path("ground.laz"), poly), indent=2))
What the split costs on disk Disk usage over the run. The source tile occupies 1.4 gigabytes throughout. Pass one adds a 480 megabyte intermediate. Pass two adds a second intermediate of similar size while the first is still present, which is the peak. Pass three writes the final output and both intermediates are deleted, leaving only the source and the result. start 1.4 GB after pass 1 1.9 GB after pass 2 — peak 2.4 GB after cleanup 1.7 GB scratch space needed is the peak, not the final state — size the volume for the middle bar deleting pass 1 before pass 3 runs saves 480 MB and costs nothing, because pass 3 only reads pass 2

# Key Parameter Table

Choice Options Guidance
Intermediate format LAZ, LAS LAZ unless the intermediate is read many times; see LAZ vs uncompressed LAS
Scratch location tmpfs, local SSD, network Local disk; tmpfs re-introduces the memory problem you are solving
Crop buffer 0–50 m At least the classifier’s window, so edge points keep their context
forward on intermediates all Header records must survive both hand-offs or the final file loses them
Cleanup on success only Keep the scratch files when a pass fails — they are the debugging material

# Verification

The three passes agree with one. On a tile small enough to run both ways, the single-pipeline output and the three-pass output should contain the same number of ground points. A difference means a reducing stage moved ahead of a stage that depended on what it removed.

Peak memory dropped where you expected. Measure each pass separately. Passes one and three should be flat and small; pass two should be noticeably below the original peak. If pass two is unchanged, the streaming pass is not actually reducing anything.

The intermediate is not silently stale. Assert the intermediate’s modification time is newer than the source before pass two reads it, or generate it under a unique prefix per run.

# Gotchas and Edge Cases

Not every reduction is safe to move earlier Four reducing stages assessed for whether they can move ahead of a ground classifier. Dropping noise-classified points and applying generous elevation limits are safe. Cropping is safe only with a buffer at least as wide as the classifier window. Poisson sampling is not safe at all, because thinning the cloud changes which points the classifier sees as the local minimum. can this stage move ahead of filters.smrf? drop Classification 7 safe — noise was never ground Z limits, generous safe — nothing near terrain is removed crop to the area of interest only with a buffer ≥ window poisson sampling unsafe — changes the local minima the test: would the blocking stage have made a different decision if it had seen the removed points? when the answer is uncertain, run both orders on one tile and difference the ground rasters — the disagreement is always concentrated at the boundary, which tells you immediately whether the buffer is wide enough.

The intermediate write can dominate. If pass one removes almost nothing, you have added a full write and read for no benefit. Measure the reduction before committing to the split; below about 30% it is rarely worth it.

Two processes, two memory peaks. Splitting only helps if the passes run sequentially. Launching them concurrently on the same worker re-creates the ceiling you were avoiding.

Metadata has to be carried across each hand-off. Every intermediate write is an opportunity to lose VLRs and extra dimensions. forward: "all" on every writer in the chain, and a check on the final header as described in metadata and header sync.

# Frequently Asked Questions

Does splitting a pipeline change the result?

It should not, and verifying that is part of the job. Run both forms on a tile small enough to afford it and compare the surviving point counts. A difference means a reducing stage moved ahead of a stage whose decision depended on the points it removed — most often a crop without enough buffer.

How much does the intermediate file cost?

One extra write and one extra read, typically a few seconds per tile against gigabytes of memory headroom. The trade is only bad when the streaming pass removes very little; below roughly a thirty percent reduction the split stops paying for itself.

Can I keep the intermediate in memory instead of on disk?

You can pass arrays between pipelines in Python, but doing so re-creates exactly the whole-cloud residency that the split was meant to avoid. Local disk is the right place for the intermediate; tmpfs is not, because it is memory wearing a filesystem interface.

Which stages are safe to move ahead of the blocking one?

Ask whether the blocking stage would have decided differently had it seen the removed points. Dropping noise and applying generous elevation limits are safe. Cropping is safe with a buffer at least as wide as the classifier window. Thinning the cloud is not safe, because it changes which points are local minima.

Related in Streaming Mode Execution in PDAL

↑ Streaming Mode Execution in PDAL overview --- title: "Streaming Mode Execution in PDAL" description: "How PDAL's streaming execution model moves points through a pipeline in... Running a PDAL Pipeline in Streaming Mode --- title: "Running a PDAL Pipeline in Streaming Mode" description: "Step-by-step recipe for executing a PDAL pipeline with executestreaming... Which PDAL Filters Break Streaming Mode --- title: "Which PDAL Filters Break Streaming Mode" description: "The rule that decides whether a PDAL stage can stream, the four families...