Python LiDAR

Diagnosing PDAL Out-of-Memory Failures

TL;DR: Estimate the requirement before you run — points × bytes-per-point × live buffers — then attack it in this order: drop unused dimensions, stream the chain, lower chunk_size, tile the input. A std::bad_alloc almost never means the machine is too small; it means the pipeline asked for the whole file when it did not have to.

# Context and Motivation

This guide belongs to Memory Management in PDAL Pipelines, which explains the buffer model. Here we work backwards from the failure: a job that ran fine on last week’s tiles has just died with std::bad_alloc, or the container was killed by the out-of-memory reaper with no message at all, and you need to know why before you can decide what to change.

The first thing worth internalising is that PDAL’s memory use is predictable to within about twenty percent. It is not a mystery to be profiled; it is an arithmetic expression with three terms. Points is what the reader delivers. Bytes-per-point is the sum of the dimension widths in the point layout. Live buffers is how many stages are holding a view at the same moment, which for a linear chain peaks at two during each hand-off. Multiply those and you have the requirement. When the answer is larger than the machine, no amount of tuning fixes it — the input has to shrink or the execution mode has to change.

Three terms, three levers Peak memory is the product of three terms. The point count is reduced by cropping, tiling or streaming. The bytes per point are reduced by dropping dimensions the pipeline never reads. The live buffer count is reduced by shortening the chain or splitting it into passes. Each term has a different lever and they multiply, so halving two of them quarters the requirement. points 18,400,000 × bytes per point 40 B for a typical layout × live buffers 2 during each hand-off crop · tile · stream the biggest lever drop unused dimensions often a third, for free split the chain smallest effect, most work 18.4 M × 40 B × 2 = 1.47 GB before PDAL, GDAL and PROJ have allocated anything of their own a ground classifier adds its own raster on top of this, which is why SMRF jobs fail first

# Prerequisites and Assumptions

Requirement Detail
PDAL any recent version; the arithmetic does not change
pdal info --schema to read the point layout and its dimension widths
/usr/bin/time -v or cgroup stats to see the real peak rather than the reported one
A reproducible failure one tile that fails the same way every time

# Step-by-Step Implementation

# Step 1 — Compute the requirement before profiling

bash
pdal info tile_0431.laz --schema | python -c "
import json,sys
s=json.load(sys.stdin)['schema']['dimensions']
print(sum(d['size'] for d in s), 'bytes/point', len(s), 'dimensions')"

Multiply by the point count from pdal info --summary and by two. If that number is close to the machine’s memory, you have your answer without running anything.

# Step 2 — Confirm what actually killed the process

std::bad_alloc is a PDAL-level allocation failure and will appear in the log. A container killed by the kernel leaves no message at all — check dmesg or the orchestrator’s termination reason. The two look identical from the application’s point of view and have different fixes.

# Step 3 — Drop dimensions the pipeline never reads

The cheapest win. A tile carrying RGB, GPS time and four vendor extras costs twice what the pipeline needs if it only ever touches X, Y, Z and Classification.

json
{"type": "writers.las", "filename": "out.laz", "extra_dims": "none"}

Better still, avoid materialising them at all by selecting dimensions at the reader when the format allows it.

# Step 4 — Make the chain stream

If every stage is per-point, streaming mode removes the point count from the expression entirely. This is the change that turns an impossible job into a routine one.

# Step 5 — Tile only when the first four fail

Retiling is real work and adds edge-effect problems. It is the correct answer when a genuinely blocking stage must see a cloud larger than the machine, and the wrong answer when a dimension you never use was doubling the footprint.

# Complete Working Example

python
"""Estimate PDAL memory for a tile and pipeline, and say what to change."""
from __future__ import annotations

import json
from pathlib import Path

import pdal

BYTES_PER_GB = 1024 ** 3


def layout(path: Path) -> tuple[int, int]:
    """Return (point_count, bytes_per_point) from the file's own schema."""
    p = pdal.Pipeline(json.dumps({"pipeline": [
        {"type": "readers.las", "filename": str(path), "count": 1}
    ]}))
    p.execute()
    meta = p.quickinfo["readers.las"]
    dims = json.loads(p.schema)["schema"]["dimensions"]
    return int(meta["num_points"]), sum(int(d["size"]) for d in dims)


def estimate(path: Path, live_buffers: int = 2, headroom_gb: float = 0.4) -> dict:
    points, per_point = layout(path)
    core = points * per_point * live_buffers / BYTES_PER_GB
    return {
        "points": points,
        "bytes_per_point": per_point,
        "live_buffers": live_buffers,
        "estimated_gb": round(core + headroom_gb, 2),
    }


def advise(est: dict, available_gb: float) -> list[str]:
    out = []
    if est["estimated_gb"] <= available_gb:
        return ["fits — no change needed"]
    if est["bytes_per_point"] > 32:
        out.append("drop unused dimensions: the layout is wider than X/Y/Z/Intensity/Classification")
    out.append("stream the chain if every stage is per-point")
    out.append(f"or tile the input to about {available_gb / est['estimated_gb']:.0%} of its current extent")
    return out


if __name__ == "__main__":
    est = estimate(Path("tile_0431.laz"))
    print(json.dumps(est, indent=2))
    for line in advise(est, available_gb=4.0):
        print(" -", line)
The estimate is good enough to plan with Five tiles plotted with the arithmetic estimate on one axis and the measured peak resident set on the other. Every point sits slightly above the one-to-one line, by between eight and ten percent, which is the fixed overhead of PDAL, GDAL and PROJ. The relationship is close enough that the estimate can be used to size a worker without running the job. one-to-one 0 4.5 GB 9 GB 0 4 GB 8 GB estimated from points × width × buffers measured peak

# Key Parameter Table

Lever Where Typical saving
extra_dims: "none" writers.las 10–50% of the point width
streaming execution execute_streaming removes the point count from the expression
chunk_size execute_streaming linear — halve it, halve the buffer
filters.crop early pipeline order proportional to what it removes
split the chain pipeline design one buffer instead of two at the peak
smaller tiles upstream proportional, at the cost of edge effects

# Verification

The estimate matched the measurement. Run /usr/bin/time -v and compare Maximum resident set size against the estimate. Agreement within about twenty percent means the model holds and you can trust it for other tiles.

The fix moved the right term. If you dropped dimensions, the per-point width should fall in pdal info --schema. If you streamed, peak memory should stop tracking the file size at all.

Nothing else changed. Compare output point counts before and after. Memory work should never alter the result.

# Gotchas and Edge Cases

Four symptoms, four different causes Four out-of-memory presentations. A bad_alloc exception in the log is PDAL failing to allocate. A silent kill with exit code 137 is the kernel or the container runtime. Memory that climbs across tiles in one process is a leak in the loop, not in PDAL. A failure only on some tiles points at those tiles being denser, not at the pipeline. std::bad_alloc in the log PDAL asked for more than malloc could give exit 137, no message the kernel or container limit killed it memory climbs tile after tile your loop holds references, not PDAL only some tiles fail those tiles are denser — size for the worst one

Peak memory is set by the worst tile, not the average. A campaign whose tiles average 12 million points may contain one urban tile with 40. Size the worker for that tile or the job fails at three in the morning on the one file nobody looked at.

Python holds arrays longer than you think. pipeline.arrays keeps a reference for as long as the pipeline object lives. In a loop over tiles, delete both, or run each tile in its own process — the pattern described in parallel execution.

Concurrency multiplies the requirement. Four workers at 1.5 GB is 6 GB, and the container limit applies to the whole cgroup. Memory tuning and pool sizing are the same decision.

# Frequently Asked Questions

How do I estimate PDAL memory before running anything?

Multiply three numbers: the point count from pdal info --summary, the sum of the dimension widths from pdal info --schema, and the number of buffers alive at once, which is two for a linear chain during each hand-off. Add a few hundred megabytes for PDAL, GDAL and PROJ themselves and the estimate is usually within twenty percent.

What is the difference between bad_alloc and exit code 137?

A bad_alloc means PDAL asked the allocator for memory and was refused, and it appears in the log. Exit 137 means something outside the process — the kernel OOM killer or a container memory limit — terminated it, and there is no application-level message at all. The first is fixed by asking for less; the second may also be fixed by raising the limit.

Why does memory climb across tiles in a loop?

Because your loop is holding references. pipeline.arrays keeps the point data alive as long as the pipeline object exists, so a list of results accumulates every tile. Delete the pipeline and its arrays each iteration, or process each tile in its own worker process.

Does lowering chunk_size always reduce memory?

In streaming mode, yes, and roughly linearly. In standard mode chunk_size has no effect at all, because the reader materialises the whole cloud regardless — which is why checking pipeline.streamable comes before tuning the chunk.

Related in Memory Management in Python LiDAR & Point Cloud Processing Workflows

↑ Memory Management in Python LiDAR & Point Cloud Processing Workflows overview --- title: "Memory Management in Python LiDAR & Point Cloud Processing Workflows" description: "Control RAM usage in Python PDAL pipelines:... LAZ vs Uncompressed LAS for Iterative Processing --- title: "LAZ vs Uncompressed LAS for Iterative Processing" description: "A decision guide on when to keep LiDAR as compressed LAZ versus...