Python LiDAR

Parallel Tile Processing with ProcessPoolExecutor

TL;DR: One process per tile, OMP_NUM_THREADS=1 in every worker, a pool sized to the physical cores, and results returned as small dictionaries rather than point arrays — that combination scales close to linearly to the core count and then flattens as the disk saturates.

# Context and Motivation

This guide is part of Parallel Execution in PDAL, which explains the two axes of parallelism available. This page is the concrete pattern for the axis that scales best in practice: file-level parallelism with Python’s ProcessPoolExecutor.

The reason processes beat threads here has nothing to do with the GIL and everything to do with isolation. Each tile is an independent unit of work with its own memory peak, its own failure mode and its own output. A worker that dies on a corrupt tile takes one tile with it. A worker that needs 1.5 GB does not affect its siblings’ allocation. And because PDAL’s own stages already use OpenMP internally, giving each process a single thread avoids the oversubscription that makes a sixty-four-thread request on an eight-core machine slower than doing nothing at all.

Fan out tiles, collect records A tile manifest feeds a pool of four worker processes. Each worker runs the whole pipeline for one tile and returns a small dictionary — tile identifier, point count, elapsed seconds, output path — rather than the point data itself. The parent collects those records into a run summary, so the memory held by the parent never grows with the number of tiles. tile manifest 1,240 paths read once, in order worker 1 · OMP=1 worker 2 · OMP=1 worker 3 · OMP=1 worker 4 · OMP=1 run summary one small record per tile, not points returning arrays instead of records would pickle gigabytes through the pool and undo the whole design

# Prerequisites and Assumptions

Requirement Detail
Python 3.9+, concurrent.futures from the standard library
PDAL 2.4+ with the Python bindings importable in the worker
A tile manifest one line per input path, frozen before the run
Memory budget workers × per-tile peak must fit; see diagnosing OOM failures
Start method spawn on macOS and Windows, fork on Linux — both work, spawn re-imports the module

# Step-by-Step Implementation

# Step 1 — Pin threads before importing PDAL

OpenMP reads its environment when the library loads, so setting the variable after import pdal in the worker is too late.

python
import os
os.environ.setdefault("OMP_NUM_THREADS", "1")
import pdal  # noqa: E402 — must come after the env var

# Step 2 — Make the worker a plain function of a path

No shared state, no globals holding pipelines, no partially-applied closures over large objects. The worker takes a path and returns a dictionary.

# Step 3 — Size the pool to physical cores

os.cpu_count() reports logical processors, which on a hyperthreaded machine is twice the useful number for CPU-bound geometry work. Start at physical cores and measure before going higher.

# Step 4 — Submit, then consume as futures complete

as_completed lets you record failures and progress as they happen rather than at the end, which matters when a run takes hours.

# Step 5 — Treat a failed tile as data, not as an exception

One corrupt tile in twelve hundred should not end the run. Catch, record, continue, and report the failures at the end.

# Complete Working Example

python
"""Process a tile manifest with one PDAL pipeline per worker process."""
from __future__ import annotations

import os

os.environ.setdefault("OMP_NUM_THREADS", "1")

import json  # noqa: E402
import logging  # noqa: E402
import time  # noqa: E402
from concurrent.futures import ProcessPoolExecutor, as_completed  # noqa: E402
from pathlib import Path  # noqa: E402

import pdal  # noqa: E402

LOG = logging.getLogger("tilepool")


def process_tile(src: str, out_dir: str, out_srs: str = "EPSG:6318") -> dict:
    """Run the pipeline for one tile. Returns a small record, never point data."""
    src_path = Path(src)
    dst = Path(out_dir) / f"{src_path.stem}_ground.laz"
    spec = json.dumps({"pipeline": [
        {"type": "readers.las", "filename": str(src_path)},
        {"type": "filters.range", "limits": "Z[-30:5000]"},
        {"type": "filters.reprojection", "out_srs": out_srs},
        {"type": "filters.smrf", "window": 18, "slope": 0.15, "threshold": 0.5, "cell": 1.0},
        {"type": "filters.range", "limits": "Classification[2:2]"},
        {"type": "writers.las", "filename": str(dst),
         "compression": "laszip", "forward": "all"},
    ]})
    started = time.perf_counter()
    kept = pdal.Pipeline(spec).execute()
    return {
        "tile": src_path.name,
        "ground_points": kept,
        "seconds": round(time.perf_counter() - started, 2),
        "output": str(dst),
    }


def run(manifest: Path, out_dir: Path, workers: int) -> dict:
    out_dir.mkdir(parents=True, exist_ok=True)
    tiles = [line.strip() for line in manifest.read_text().splitlines() if line.strip()]
    done, failed = [], []

    with ProcessPoolExecutor(max_workers=workers) as pool:
        futures = {pool.submit(process_tile, t, str(out_dir)): t for t in tiles}
        for i, future in enumerate(as_completed(futures), start=1):
            tile = futures[future]
            try:
                record = future.result()
                done.append(record)
                LOG.info("[%d/%d] %s — %d ground points in %.1fs",
                         i, len(tiles), record["tile"], record["ground_points"],
                         record["seconds"])
            except Exception as exc:  # one bad tile must not end the run
                failed.append({"tile": tile, "error": repr(exc)})
                LOG.error("[%d/%d] %s FAILED: %r", i, len(tiles), tile, exc)

    return {"completed": len(done), "failed": failed,
            "total_seconds": round(sum(r["seconds"] for r in done), 1)}


if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO, format="%(asctime)s %(message)s")
    summary = run(Path("tiles.txt"), Path("out"), workers=8)
    print(json.dumps(summary, indent=2))
What a healthy pool looks like Four worker lanes with tile jobs drawn as blocks over time. The blocks vary in length because tiles vary in density, and each worker picks up the next tile the moment it finishes rather than waiting for its siblings. The tail at the end is one unusually dense tile finishing alone, which is why the last few percent of a run is always the slowest. four workers, 21 tiles, no barrier between them w1 w2 w3 w4 one very dense tile three workers idle from here submitting the largest tiles first shortens this tail, which is worth doing when tile sizes vary by more than about 3×

# Key Parameter Table

Setting Value Why
OMP_NUM_THREADS 1 Prevents each worker from claiming every core for itself
max_workers physical cores Logical cores over-subscribe on hyperthreaded machines
Return type small dict Anything larger is pickled through the pool on every tile
Failure policy catch and record One corrupt tile in a thousand should not end a twelve-hour run
Start method spawn or fork spawn is safer with native libraries; fork starts faster

# Verification

Every tile is accounted for. len(done) + len(failed) must equal the manifest length. A shortfall means a worker died without raising, which usually means the kernel killed it for memory.

Speedup is close to the worker count until it is not. Time the run at one, two, four and eight workers. Near-linear scaling up to the core count, then flat, is healthy; a curve that peaks at three workers on an eight-core box means the threads were never pinned.

Outputs are independent. Two workers must never write the same path. Deriving the output name from the input stem, as above, guarantees it.

What the parent ends up holding Two traces of the parent process memory across a 1,240-tile run. Returning small dictionaries keeps the parent flat at about 40 megabytes for the whole run. Returning point arrays makes the parent grow with every completed tile, passing eight gigabytes before the run is halfway through. returning pipeline.arrays — grows with every tile returning a dict — flat at 40 MB 0 8 GB tiles completed the workers were memory-safe in both runs — it is the parent that fails, several hours in

# Gotchas and Edge Cases

Setting OMP_NUM_THREADS in the parent only. With fork the child inherits it, with spawn it does not unless it is in the environment before the interpreter starts. Set it in both places.

Returning pipeline.arrays from the worker. It pickles the whole tile back to the parent, which then holds every tile’s points at once. This single mistake converts a memory-safe design into the original problem.

A pool larger than the disk can feed. Eight workers reading eight LAZ files saturate a spinning disk long before the CPUs. Watch the read throughput, not just the core utilisation.

# Frequently Asked Questions

Why processes rather than threads?

Isolation. Each tile has its own memory peak and its own failure mode, and a process boundary contains both. PDAL stages also use OpenMP internally, so processes with threads pinned to one give predictable core usage where threads inside one process would fight over the same cores.

What should the worker return?

A small dictionary: tile name, point count, elapsed time, output path. Returning pipeline.arrays pickles the entire point cloud back to the parent for every tile, which recreates exactly the whole-campaign memory footprint the design exists to avoid.

How many workers is too many?

When the speedup stops improving, which is usually at the physical core count and sometimes earlier if the disk saturates first. Multiply the per-tile memory peak by the worker count and check it against the machine before raising the number.

Why set OMP_NUM_THREADS before importing pdal?

OpenMP reads its configuration when the library loads. Setting the variable after the import has no effect, so each worker claims every core and eight workers on an eight-core machine ask for sixty-four threads.

Related in Parallel Execution in PDAL: Multi-Core Point Cloud Processing with Python

↑ Parallel Execution in PDAL: Multi-Core Point Cloud Processing with Python overview --- title: "Parallel Execution in PDAL: Multi-Core Point Cloud Processing with Python" description: "Distribute PDAL-driven LiDAR workflows... Optimizing PDAL for Multi-Core Processing --- title: "Optimizing PDAL for Multi-Core Processing" description: "Learn how to tune PDAL for multi-core throughput: set OMPNUMTHREADS,...