Parameterizing Pipelines with Jinja Templates

TL;DR: Render every value through Jinja’s tojson filter ("filename": {{ src | tojson }}) so quoting and escaping are always correct, use StrictUndefined so a missing parameter raises, express optional stages with {% if %} blocks that emit a whole stage object, and pin rendered output with golden-file tests.

# Context and Motivation

This guide is part of PDAL Pipeline Templating and Parameterization. Jinja2 is the most common way to template PDAL pipelines because it is already in most Python environments, it is familiar from Airflow and Ansible, and it handles both simple substitution and conditional structure. It also makes it easy to produce broken JSON: a Windows path with backslashes, a string placeholder without quotes, a trailing comma after a conditional stage that did not render.

Every one of those failure modes has a mechanical fix, and applying the fixes consistently turns templates from a source of mysterious PDAL errors into something you can trust in a batch system.

# Prerequisites and Assumptions

• Python 3.10+ with Jinja2 3.x.
• A working PDAL pipeline to generalize, and the PDAL CLI for validation.
• Templates stored as files (templates/*.json.j2) rather than strings in code, so they can be reviewed and diffed.

# Step-by-Step Implementation

# Step 1 — Configure a strict environment

Create one jinja2.Environment with StrictUndefined, autoescape=False (HTML escaping would corrupt JSON), and keep_trailing_newline=True.

# Step 2 — Render every value with tojson

{{ value | tojson }} emits a JSON literal of the right type: strings quoted and escaped, numbers bare, booleans as true/false, lists as arrays. Never write quotes around a placeholder yourself.

# Step 3 — Build optional stages as whole objects

Wrap an entire stage object and its separating comma in one {% if %} block. The cleanest way to avoid comma bugs is to build the stage list with a loop over a Jinja list and loop.last.

# Step 4 — Loop for variable-length inputs

For merge pipelines, loop over a list of input files to emit one reader per tile, followed by filters.merge.

# Step 5 — Test rendered output against golden files

For each template, render with a fixed parameter set and compare with a committed expected JSON. Any template edit shows up as a reviewable diff of the golden file.

# Complete Working Example

templates/clean_and_grid.json.j2:

{% set stages = [] %}
{% for f in inputs %}
  {% set _ = stages.append({"type": "readers.las", "filename": f, "default_srs": crs}) %}
{% endfor %}
{% if inputs | length > 1 %}
  {% set _ = stages.append({"type": "filters.merge"}) %}
{% endif %}
{% set _ = stages.append({"type": "filters.range",
                          "limits": "Classification![7:7],Classification![18:18]"}) %}
{% if reproject_to %}
  {% set _ = stages.append({"type": "filters.reprojection", "out_srs": reproject_to}) %}
{% endif %}
{% if outlier %}
  {% set _ = stages.append({"type": "filters.outlier", "method": "statistical",
                            "mean_k": outlier.mean_k, "multiplier": outlier.multiplier}) %}
{% endif %}
{% set _ = stages.append({"type": "writers.gdal", "filename": dst, "resolution": res,
                          "output_type": "idw", "data_type": "float32"}) %}
{{ {"pipeline": stages} | tojson(indent=2) }}

The template builds the stage list as a Jinja data structure and serializes it once with tojson, which removes the comma problem entirely: there is no hand-written JSON punctuation left in the template.

render.py:

"""Render and sanity-check a PDAL pipeline from a Jinja2 template."""
from __future__ import annotations

import json
from pathlib import Path

import jinja2

ENV = jinja2.Environment(
    loader=jinja2.FileSystemLoader("templates"),
    undefined=jinja2.StrictUndefined,
    autoescape=False,
    keep_trailing_newline=True,
    extensions=["jinja2.ext.do"],
)


def render(template: str, **params) -> dict:
    text = ENV.get_template(template).render(**params)
    spec = json.loads(text)
    types = [s["type"] for s in spec["pipeline"]]
    assert types[0].startswith("readers."), types
    assert types[-1].startswith("writers."), types
    return spec


if __name__ == "__main__":
    spec = render(
        "clean_and_grid.json.j2",
        inputs=["tiles/t_0431.laz", "tiles/t_0432.laz"],
        crs="EPSG:6347",
        reproject_to=None,
        outlier={"mean_k": 12, "multiplier": 2.5},
        dst="out/block_043_dtm.tif",
        res=1.0,
    )
    Path("out").mkdir(exist_ok=True)
    Path("out/block_043.pipeline.json").write_text(json.dumps(spec, indent=2, sort_keys=True))
    print(" -> ".join(s["type"] for s in spec["pipeline"]))

Output:

readers.las -> readers.las -> filters.merge -> filters.range -> filters.outlier -> writers.gdal

# Key Parameter Table

# Verification

Golden-file tests are the verification step that pays for itself fastest.

import json
from pathlib import Path

import pytest

from render import render

CASES = {
    "two_tiles_outlier": dict(inputs=["a.laz", "b.laz"], crs="EPSG:6347", reproject_to=None,
                              outlier={"mean_k": 12, "multiplier": 2.5}, dst="o.tif", res=1.0),
}


@pytest.mark.parametrize("name", CASES)
def test_golden(name: str) -> None:
    got = render("clean_and_grid.json.j2", **CASES[name])
    want = json.loads(Path(f"tests/golden/{name}.json").read_text())
    assert got == want

Follow it with pdal pipeline --validate on each golden file in CI, which confirms the stages and options exist in the installed PDAL.

# Gotchas and Edge Cases

autoescape=True corrupts expressions. PDAL expressions such as Classification == 2 && Z > 10 contain &. HTML autoescaping renders &&, which PDAL cannot parse. JSON templates must disable it.

Numbers rendered as strings. YAML loaded without type conversion yields "1.0" strings; tojson then quotes them faithfully, and PDAL may reject "resolution": "1.0" or interpret it differently. Convert types in the parameter loader, not in the template.

Whitespace in hand-written JSON templates. If you do write JSON by hand around placeholders, trim_blocks and lstrip_blocks keep the output tidy, but comma handling around conditionals remains fragile. Building a data structure and serializing once, as the example does, avoids the whole class of bug.

Secrets in templates. Credentials for S3 readers do not belong in templates or parameter files. Let GDAL and PDAL pick them up from the environment or an instance role, as described in configuring GDAL /vsis3/ for fast point cloud reads.

# Frequently Asked Questions

Why use tojson instead of quoting placeholders myself?

Because tojson knows the value’s type and escapes it correctly. Hand-quoted placeholders break on backslashes, embedded quotes and non-ASCII characters, and they quote numbers that should be bare.

How do I include a stage only when a parameter is set?

Build the stage list as a Jinja list, append the optional stage inside an if block, and serialize the whole object with tojson at the end. That avoids comma handling entirely.

Can templates call Python functions?

Yes, by registering them as globals or filters on the environment, for example a function that picks a UTM zone from a bounding box. Keep such functions small and tested; complex logic is usually clearer in the Python that prepares the parameters.

How do I test a template?

Render it with fixed parameters and compare against a committed golden JSON file, then run pdal pipeline with the validate flag on the golden file. Both belong in CI.

# Related

• PDAL Pipeline Templating and Parameterization — choosing a mechanism
• Overriding Stage Options from the Command Line — lighter-weight per-run values
• Building Pipelines with the Python Stage API — the code-first alternative
• Schema-Validating Pipeline JSON Before Execution — structural checks on rendered output
• Testing PDAL Pipelines with pytest — running rendered pipelines on fixtures