PDAL Pipeline Templating and Parameterization

Every LiDAR team eventually finds a directory holding dtm_projectA.json, dtm_projectA_v2.json, dtm_projectB_fixed.json and a dozen more — nearly identical pipelines that differ in an EPSG code, a resolution and a file path. Each copy drifts: a bug fixed in one is still present in three others, and nobody can say which version produced last year’s deliverable. Parameterization solves this by separating the shape of a pipeline — which stages, in which order — from the values that change per tile or per project. This topic in the PDAL Pipeline Architecture and Execution section compares the three practical ways to do that — PDAL’s own command-line overrides, Jinja2 templates, and the Python stage API — and shows how to validate whatever you render before it touches data.

# Prerequisites

• PDAL 2.4+ on the command line and the Python bindings (python-pdal 3.x) for the stage API.
• Python 3.10+ with Jinja2 3.x, and optionally jsonschema for structural validation.
• A working pipeline you want to generalize. Start from something that runs correctly for one tile; parameterize second.
• A parameter inventory. Before templating, list what actually varies: input and output paths, CRS, resolution, filter thresholds, classes kept. Anything not on the list stays literal.
• Version control for templates and parameter files, so every rendered pipeline can be traced to a commit.

# Core Workflow Architecture

1. Identify the variable parts. Separate values that change per tile (paths), per project (CRS, resolution, thresholds) and never (stage order, writer format).
2. Choose a mechanism. Command-line overrides for one or two per-run values; Jinja2 templates for per-project configuration in a batch system; the Python stage API when pipelines are assembled conditionally in code.
3. Keep parameters typed and in one file. A small YAML or TOML per project, loaded and validated in Python, rather than values scattered across shell scripts.
4. Render deterministically. The same template and parameters must always produce byte-identical JSON; sort keys and avoid timestamps in the output.
5. Validate before execution. Parse the rendered JSON, check it against your own schema, and run pdal pipeline --validate or construct pdal.Pipeline without executing.
6. Record provenance. Store the rendered pipeline JSON beside each output, or its hash in the output’s metadata, so the exact pipeline behind any deliverable is recoverable.

# Full Implementation

The implementation renders a Jinja2 template from a per-project YAML file, validates the result structurally and with PDAL, and runs it for a list of tiles, writing the rendered JSON next to each output.

"""Render, validate and run a templated PDAL pipeline for a list of tiles."""
from __future__ import annotations

import hashlib
import json
import logging
import subprocess
from dataclasses import dataclass
from pathlib import Path

import jinja2
import pdal
import yaml

log = logging.getLogger("tpl")


@dataclass(frozen=True)
class ProjectParams:
    crs: str
    resolution: float
    smrf_slope: float
    smrf_window: float
    out_dir: Path

    @classmethod
    def load(cls, path: Path) -> "ProjectParams":
        raw = yaml.safe_load(path.read_text())
        p = cls(crs=str(raw["crs"]), resolution=float(raw["resolution"]),
                smrf_slope=float(raw["smrf"]["slope"]), smrf_window=float(raw["smrf"]["window"]),
                out_dir=Path(raw["out_dir"]))
        if not p.crs.startswith("EPSG:"):
            raise ValueError(f"crs must be an EPSG code, got {p.crs!r}")
        if not 0.1 <= p.resolution <= 10:
            raise ValueError(f"resolution {p.resolution} outside 0.1–10 m")
        return p


ENV = jinja2.Environment(loader=jinja2.FileSystemLoader("templates"),
                         undefined=jinja2.StrictUndefined, autoescape=False,
                         trim_blocks=True, lstrip_blocks=True)


def render(template: str, tile: Path, params: ProjectParams) -> dict:
    text = ENV.get_template(template).render(
        src=str(tile), dst=str(params.out_dir / f"{tile.stem}_dtm.tif"),
        crs=params.crs, res=params.resolution,
        slope=params.smrf_slope, window=params.smrf_window)
    spec = json.loads(text)                       # fails fast on broken JSON
    stages = spec["pipeline"]
    types = [s["type"] if isinstance(s, dict) else "reader" for s in stages]
    if types[-1].split(".")[0] != "writers":
        raise ValueError(f"last stage must be a writer, got {types[-1]}")
    return spec


def pdal_validate(spec: dict, scratch: Path) -> None:
    scratch.write_text(json.dumps(spec, indent=2, sort_keys=True))
    res = subprocess.run(["pdal", "pipeline", "--validate", str(scratch)],
                         capture_output=True, text=True)
    if res.returncode != 0:
        raise RuntimeError(f"pdal --validate failed: {res.stderr.strip() or res.stdout.strip()}")


def run_tile(template: str, tile: Path, params: ProjectParams) -> str:
    spec = render(template, tile, params)
    rendered = params.out_dir / f"{tile.stem}.pipeline.json"
    params.out_dir.mkdir(parents=True, exist_ok=True)
    pdal_validate(spec, rendered)
    digest = hashlib.sha256(rendered.read_bytes()).hexdigest()[:12]
    n = pdal.Pipeline(json.dumps(spec)).execute()
    log.info("%s: %d points, pipeline %s", tile.name, n, digest)
    return digest


if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")
    params = ProjectParams.load(Path("projects/county_north.yaml"))
    for tile in sorted(Path("tiles").glob("*.laz")):
        run_tile("dtm.json.j2", tile, params)

The template it renders:

{
  "pipeline": [
    { "type": "readers.las", "filename": "{{ src }}", "default_srs": "{{ crs }}" },
    { "type": "filters.range", "limits": "Classification![7:7],Classification![18:18]" },
    { "type": "filters.smrf", "slope": {{ slope }}, "window": {{ window }},
      "threshold": 0.45, "scalar": 1.2, "cell": 1.0 },
    { "type": "filters.range", "limits": "Classification[2:2]" },
    { "type": "writers.gdal", "filename": "{{ dst }}", "resolution": {{ res }},
      "output_type": "idw", "window_size": 6, "data_type": "float32",
      "gdalopts": "COMPRESS=DEFLATE,TILED=YES" }
  ]
}

# Code Breakdown

StrictUndefined. Jinja2’s default renders a missing variable as an empty string, which produces "slope": , — broken JSON if you are lucky, or a silently empty filename if the placeholder was inside quotes. Strict mode raises instead, so a typo in a parameter name fails at render time.

Typed parameters with range checks. Loading YAML into a frozen dataclass converts types once and rejects nonsense before rendering. A resolution of 100 or a CRS of "6347" without the prefix fails with a readable message rather than as a PDAL error three stages in.

json.loads immediately after rendering. The rendered text must be valid JSON before anything else runs. Quoting mistakes in templates — a numeric placeholder wrapped in quotes, or a string placeholder without them — surface here.

Structural check, then PDAL’s check. The code asserts a writer is last, which PDAL would also catch, but the custom check can encode team rules PDAL cannot, such as “every DTM pipeline must remove noise first”. pdal pipeline --validate then confirms that every stage exists and every option is recognized by the installed PDAL.

Sorted, indented rendered JSON beside the output. Writing the exact pipeline next to its product, with sorted keys, makes the file reproducible and diff-able. The short SHA-256 digest logged per tile is enough to tell later whether two outputs came from the same pipeline.

default_srs rather than override_srs. The template assigns the project CRS only when the file lacks one; a file that declares its own CRS keeps it. Switch to override_srs deliberately when a vendor’s CRS is known to be wrong, and put that decision in the parameter file, not the template.

# Parameter Reference Table

# Validation and Integrity Checks

Validation of templated pipelines has three layers, and each catches a different class of mistake.

1. Render-time. Strict undefined variables and immediate json.loads catch missing parameters and quoting errors.
2. Structure. A JSON Schema or a few assertions encode team conventions: required stages, allowed writers, output extensions. See schema-validating pipeline JSON before execution.
3. PDAL. pdal pipeline --validate checks stage names and options against the installed version. Run it in CI against every template with a representative parameter file, as in validating PDAL pipelines in CI.

The order matters for the quality of error messages as much as for correctness. A missing parameter reported as “undefined variable ‘res’” is immediately actionable; the same mistake reported by PDAL as a failure to parse an empty resolution three stages into execution is not. Running the cheap, specific checks first means each failure is reported by the layer that understands it best.

It is also worth being explicit about what none of the layers catch: a pipeline that is valid but wrong. A slope of 0.15 where 0.5 was intended passes every check. The defence there is a small regression test on a reference tile — run the rendered pipeline and compare a few summary statistics of the output with stored values — which belongs in the same CI job as the validation.

A fourth, cheap check is determinism: render the same template and parameters twice and compare hashes. Anything that differs — a timestamp, an unordered set rendered into a list — will make provenance useless.

def test_render_is_deterministic(tmp_path: Path) -> None:
    params = ProjectParams.load(Path("projects/county_north.yaml"))
    a = json.dumps(render("dtm.json.j2", Path("tiles/t1.laz"), params), sort_keys=True)
    b = json.dumps(render("dtm.json.j2", Path("tiles/t1.laz"), params), sort_keys=True)
    assert a == b

# Organizing Templates in a Repository

A layout that scales from one project to dozens keeps three things apart: templates, parameters and rendered output.

pipelines/
  templates/
    dtm.json.j2
    dsm.json.j2
    classify.json.j2
  projects/
    county_north.yaml
    county_south.yaml
  schema/
    pipeline.schema.json
  tests/
    test_render.py

Templates change rarely and are reviewed like code. Parameter files change per project and are reviewed by whoever owns that project’s specification. Rendered pipelines are never committed; they are build artefacts written beside outputs. Keeping the three apart means a pull request that edits a template is visibly different from one that edits a project, and a reviewer knows which kind of risk they are looking at.

Resist the temptation to put every conceivable option into the template. A template with forty placeholders is just the pipeline JSON with extra steps, and every placeholder is another value a parameter file can get wrong. Parameterize what has actually varied between projects; leave the rest literal until a real project needs it changed.

# Performance Tuning

Templating adds almost nothing to run time — rendering and validating a pipeline takes milliseconds, and pdal pipeline --validate well under a second. The performance levers are the ones parameterization makes easier to apply consistently.

• Validate once per template and parameter set, not per tile. Paths differ per tile but the stages and options do not; validating one rendered pipeline per project and then only substituting paths saves thousands of subprocess calls on a large batch.
• Push per-tile values to CLI overrides. For very large batches, render the project pipeline once to a file and loop pdal pipeline dtm.json --readers.las.filename=... --writers.gdal.filename=.... That avoids Python start-up per tile in shell-driven batch systems.
• Parameterize tuning knobs. Putting chunk_size, threads and resolution in the parameter file makes it trivial to run a quick benchmark across settings, as described in optimizing PDAL for multi-core processing.

# Common Errors and Troubleshooting

json.decoder.JSONDecodeError: Expecting value after rendering. A placeholder rendered empty or a numeric value was quoted inconsistently. Print the rendered text around the reported column; with StrictUndefined a missing variable raises earlier, so this is almost always quoting.

PDAL: Unexpected argument 'windw' from --validate. A misspelled option in the template. Validation in CI catches it before a batch run does.

CLI override ignored. --readers.las.filename applies to every readers.las stage; if the pipeline has two, both are overridden. Tag stages and use --stage.<tag>.filename to target one.

Different outputs from the same template. Parameters loaded from YAML as strings in one project and floats in another render differently ("1" versus 1.0). Type-convert in the loader, as the dataclass does.

Template changes break old projects. A template edit applies to every project on the next run. Version templates, and pin each project’s parameter file to a template version when deliveries must be reproducible.

# Frequently Asked Questions

Does PDAL support variables in pipeline JSON?

Not as a templating language. PDAL supports overriding stage options from the command line, which covers per-run values like file names. For anything richer — computed values, conditionals, per-project defaults — render the JSON with a template engine or build it in Python.

When should I use the Python stage API instead of templates?

When the set of stages depends on logic: add a reprojection only if the CRS differs, add a noise filter only for certain sensors. Code expresses conditionals more clearly than template syntax, and the result can still be serialized to JSON for provenance.

How do I know which pipeline produced an old output?

Store the rendered, key-sorted pipeline JSON beside each output, or record its hash in the output’s metadata. Combined with the template and parameter files in version control, that makes any deliverable traceable.

Should parameter files be YAML, TOML or JSON?

Any format that loads into typed values works. YAML is the most common for hand-edited project configuration because it allows comments, which is where the reason for an unusual threshold belongs. Whatever you choose, load it through one function that converts types and checks ranges, so every project is held to the same rules.

Can I override an option on only one of two readers?

Yes. Give each stage a tag in the JSON and override with the stage.tag.option form on the command line. Without tags, a type-based override applies to every stage of that type.

# Related

• Parameterizing Pipelines with Jinja Templates — template design, filters and conditionals
• Overriding Stage Options from the Command Line — PDAL’s built-in per-run parameters
• Building Pipelines with the Python Stage API — composing stages in code
• Pipeline Validation — checking what you render
• PDAL Stage Chaining — the pipeline shape being parameterized