Python LiDAR

Adding a New Dimension from a Python Filter

TL;DR: Three declarations must agree or the dimension vanishes: add_dimension: "Name=type" on filters.python, an outs["Name"] array of that exact dtype in the function, and extra_dims: "Name=type" on a LAS 1.4 writer with point format 6 or above.

# Context and Motivation

This guide is part of Programmable Python Filters in PDAL. Modifying an existing dimension is straightforward; creating one is where the three-way agreement between the pipeline, the function and the writer starts to matter, and where a mismatch produces a file that looks fine and has quietly lost your work.

The reason it takes three declarations is that a PDAL dimension is three different things at three moments. Inside the pipeline it is a column in the point layout, allocated before any stage runs. Inside your function it is a NumPy array with a dtype. Inside the LAS file it is an entry in the extra-bytes descriptor record with a documented type and name. Nothing propagates automatically between those three, and each one fails differently: a missing add_dimension throws, a dtype mismatch truncates, and a missing extra_dims writes a perfectly valid file with no trace of the dimension at all.

Three declarations, three different failures A new dimension has to be declared in three places. Omitting add_dimension raises an invalid dimension error at pipeline start. Getting the NumPy dtype wrong truncates or widens the values silently. Omitting extra_dims on the writer produces a valid file in which the dimension simply does not appear, with no warning of any kind. add_dimension on filters.python outs dtype in the function extra_dims on the writer omitted → error "Invalid dimension" at start wrong → truncated values change, no message omitted → gone valid file, no dimension the third failure is the dangerous one: the pipeline exits zero, the file opens, and the work is not in it

# Prerequisites and Assumptions

Requirement Detail
PDAL 2.4+ with Python support
Output format LAS 1.4, point format 6 or above — earlier formats have no standard extra-bytes mechanism
A type decision make it before writing code; changing it later means rewriting every consumer
laspy for reading the extra-bytes descriptor back

# Step-by-Step Implementation

# Step 1 — Choose the narrowest type that holds the values

A confidence score in 0–100 is uint8, one byte per point. The same score as double is eight, and on an 18 million point tile that difference is 126 MB in every copy of the file forever. Types available include uint8, int8, uint16, int16, uint32, int32, float, double.

# Step 2 — Declare it on the filter

json
{"type": "filters.python", "script": "score.py", "function": "confidence",
 "module": "score", "add_dimension": "Confidence=uint8"}

Several dimensions are declared as a list: "add_dimension": ["Confidence=uint8", "Roughness=float"].

# Step 3 — Produce exactly that dtype

python
outs["Confidence"] = score.astype(np.uint8)

# Step 4 — Declare it on the writer

json
{"type": "writers.las", "filename": "out.laz", "minor_version": 4,
 "dataformat_id": 6, "extra_dims": "Confidence=uint8", "forward": "all"}

"extra_dims": "all" also works and is blunter: it writes every non-standard dimension in the layout, including any a previous stage added that you did not want persisted.

# Step 5 — Read it back before believing it

python
import laspy
with laspy.open("out.laz") as fh:
    print([d.name for d in fh.header.point_format.extra_dimensions])

# Complete Working Example

python
"""Add a per-point confidence dimension derived from local return geometry."""
import numpy as np

# Weights are configuration, not logic — pdalargs can override them per run.
DEFAULTS = {"w_single": 40, "w_last": 30, "w_intensity": 30}


def confidence(ins, outs):
    args = {**DEFAULTS, **globals().get("pdalargs", {})}

    ret = ins["ReturnNumber"].astype(np.int16)
    n_ret = ins["NumberOfReturns"].astype(np.int16)
    intensity = ins["Intensity"].astype(np.float64)

    single = (n_ret == 1)
    last = (ret == n_ret) & ~single

    # Intensity contributes on a normalised 0-1 scale, robust to outliers.
    lo, hi = np.percentile(intensity, [5, 95])
    span = max(hi - lo, 1.0)
    inten_score = np.clip((intensity - lo) / span, 0.0, 1.0)

    score = (single * float(args["w_single"])
             + last * float(args["w_last"])
             + inten_score * float(args["w_intensity"]))

    outs["Confidence"] = np.clip(score, 0, 255).astype(np.uint8)
    return True


def test_single_return_scores_highest():
    n = 3
    ins = {
        "ReturnNumber": np.array([1, 1, 2], dtype=np.uint8),
        "NumberOfReturns": np.array([1, 3, 3], dtype=np.uint8),
        "Intensity": np.array([5000, 5000, 5000], dtype=np.uint16),
    }
    outs = {}
    confidence(ins, outs)
    assert outs["Confidence"].dtype == np.uint8
    assert len(outs["Confidence"]) == n
    assert outs["Confidence"][0] > outs["Confidence"][2]

The pipeline that uses it:

json
{
  "pipeline": [
    {"type": "readers.las", "filename": "tile_0431.laz"},
    {"type": "filters.python", "script": "score.py", "function": "confidence",
     "module": "score", "add_dimension": "Confidence=uint8",
     "pdalargs": {"w_intensity": 20}},
    {"type": "writers.las", "filename": "tile_0431_scored.laz",
     "compression": "laszip", "minor_version": 4, "dataformat_id": 6,
     "extra_dims": "Confidence=uint8", "forward": "all"}
  ]
}

# Key Parameter Table

Type Bytes Range Use for
uint8 1 0–255 flags, scores, small class codes
int8 1 −128–127 signed offsets, deltas
uint16 2 0–65,535 counts, scaled ratios
int16 2 ±32,767 scan angles, signed residuals
float 4 ~7 digits heights above ground, distances
double 8 ~16 digits rarely justified for a derived dimension
One name, four namespaces The dimension name Confidence appears in four places: the add_dimension declaration, the key written into outs, the writer extra_dims option, and the extra-bytes descriptor stored in the file. All four must agree exactly, including case, and none of them checks the others. add_dimension Confidence outs key Confidence extra_dims Confidence extra-bytes VLR Confidence confidence · CONFIDENCE · Confidence_ — three different dimensions as far as PDAL is concerned define the name once as a Python constant and interpolate it into the JSON when you generate the pipeline this is the single most common cause of an empty dimension, and no layer of the stack warns about it

# Verification

The descriptor exists. laspy’s point_format.extra_dimensions is the authoritative list; if your name is absent, extra_dims was wrong.

The type is what you asked for. Read arr["Confidence"].dtype after a round trip. A uint8 that returns as float64 means the two declarations disagreed and PDAL widened it.

The values are not all zero. A dimension that exists and is uniformly zero means the function never ran, ran on an empty buffer, or wrote under a different key.

What one extra dimension costs per tile Added file size for one extra dimension on an 18.4 million point tile, by type. A single byte per point adds 18 megabytes. Two bytes adds 37. A four-byte float adds 74 and a double adds 147. The choice is made once and paid for in every copy of the file thereafter. added size on an 18.4 M point tile, uncompressed uint8 / int8 18 MB uint16 / int16 37 MB float 74 MB double 147 MB a 0–100 score stored as a double is eight times the file growth of the same score stored as a byte

# Gotchas and Edge Cases

LAS 1.2 cannot carry it. Writing a file with minor_version: 2 discards extra dimensions without an error. Point format matters too: formats 0 to 5 have no standard extra-bytes mechanism, so use format 6 or above — the version differences are laid out in LAS/LAZ file structure.

extra_dims: "all" writes more than you meant. Intermediate dimensions added by other stages come along, inflating the file with values nobody downstream reads.

A name collision with a standard dimension is silently accepted. Adding a dimension called Intensity shadows the real one. Prefix your own names if there is any doubt.

Compression does not save you. LAZ compresses extra dimensions poorly compared with coordinates, because they lack the spatial correlation the encoder exploits — one of the effects visible in converting LAS to LAZ.

# Frequently Asked Questions

Why does my new dimension not appear in the output file?

Almost always because extra_dims was not set on the writer. add_dimension makes the dimension exist inside the pipeline; extra_dims makes the writer store it. Without the second, the pipeline succeeds, the file is valid, and the dimension is simply absent.

Which type should I choose?

The narrowest one that holds your values. A flag or a 0 to 100 score fits in a uint8 at one byte per point; the same value stored as a double costs eight, which on an 18 million point tile is 147 megabytes instead of 18 — in every copy of the file, forever.

Can I add a dimension to a LAS 1.2 file?

Not in a way any other tool will read. LAS 1.2 and point formats 0 to 5 have no standard extra-bytes mechanism, so the writer discards the dimension without an error. Write LAS 1.4 with point format 6 or above.

What happens if the declared type and the NumPy dtype disagree?

PDAL reconciles them, and the reconciliation is not always the one you wanted — values may be truncated or the dimension widened. Cast explicitly in the function so the two declarations state the same thing.

Related in Programmable Python Filters in PDAL

↑ Programmable Python Filters in PDAL overview --- title: "Programmable Python Filters in PDAL" description: "How filters.python hands a PDAL point buffer to a NumPy function, the rules... Debugging and Profiling filters.python --- title: "Debugging and Profiling filters.python" description: "Getting output out of a PDAL Python stage, counting the calls to catch... Writing a filters.python Stage with NumPy --- title: "Writing a filters.python Stage with NumPy" description: "From empty file to running stage: the (ins, outs) signature, casting...