Overriding Stage Options from the Command Line

TL;DR: Keep one pipeline JSON with placeholder paths, then run pdal pipeline dtm.json --readers.las.filename=in.laz --writers.gdal.filename=out.tif per tile. When a pipeline has two stages of the same type, give each a "tag" and override with --stage.<tag>.<option>=value so only that stage changes.

# Context and Motivation

This guide is part of PDAL Pipeline Templating and Parameterization. Before reaching for a template engine, check whether PDAL already does what you need. The pdal pipeline command accepts option overrides for any stage on its command line, which covers the most common kind of parameter — a different input and output path for every tile — without rendering anything. The pipeline file stays a plain, valid JSON document that you can validate once and reuse thousands of times.

Overrides are also the natural fit for shell-driven batch tools: GNU parallel, a Slurm array job, or an AWS Batch array where the tile path arrives as an environment variable.

# Prerequisites and Assumptions

• The PDAL command-line application (2.x). Overrides are a feature of pdal pipeline, not of the Python bindings.
• A pipeline JSON that runs correctly for one tile.
• For parallel runs, GNU parallel or xargs -P.

# Step-by-Step Implementation

# Step 1 — Write the pipeline with representative values

Use a real tile path so the file validates on its own. Overrides replace the value at run time.

{
  "pipeline": [
    { "type": "readers.las", "filename": "tiles/t_0431.laz" },
    { "type": "filters.range", "limits": "Classification![7:7],Classification![18:18]" },
    { "type": "filters.smrf", "slope": 0.15, "window": 18, "threshold": 0.5 },
    { "type": "filters.range", "limits": "Classification[2:2]" },
    { "type": "writers.gdal", "filename": "out/t_0431_dtm.tif", "resolution": 1.0,
      "output_type": "idw", "data_type": "float32" }
  ]
}

# Step 2 — Override by stage type

--<stage type>.<option>=<value> applies to every stage of that type in the pipeline.

pdal pipeline dtm.json \
  --readers.las.filename=tiles/t_0432.laz \
  --writers.gdal.filename=out/t_0432_dtm.tif

# Step 3 — Tag stages that repeat

This pipeline has two filters.range stages. To change only the second, tag it and override the tag.

{ "type": "filters.range", "tag": "keep_ground", "limits": "Classification[2:2]" }

pdal pipeline dtm.json --stage.keep_ground.limits="Classification[2:2],Z[-50:3000]"

# Step 4 — Loop over tiles in parallel

Hand the tile list to GNU parallel, deriving output names from input names.

# Step 5 — Record what actually ran

Pass --verbose 4 and keep the log, or add --pipeline-serialization to write the effective pipeline with overrides applied, for provenance.

# Complete Working Example

#!/usr/bin/env bash
# Run one pipeline over every tile, four at a time, with per-tile overrides.
set -euo pipefail

PIPELINE=dtm.json
mkdir -p out logs

pdal pipeline --validate "$PIPELINE"

find tiles -name '*.laz' | sort | parallel -j 4 --halt now,fail=1 --joblog logs/jobs.tsv '
  base=$(basename {} .laz)
  pdal pipeline '"$PIPELINE"' \
    --readers.las.filename={} \
    --writers.gdal.filename=out/${base}_dtm.tif \
    --pipeline-serialization=out/${base}.pipeline.json \
    --verbose 4 > logs/${base}.log 2>&1
'

awk -F'\t' 'NR > 1 && $7 != 0 { bad++ } END { print (bad ? bad : 0), "failed tiles" }' logs/jobs.tsv

--halt now,fail=1 stops the batch on the first failure, which is what you want while developing; switch to --halt never in production and read the job log for tiles to retry.

# Overrides in Batch Schedulers

The same pattern carries directly into schedulers that hand each task an index rather than a filename. An AWS Batch array job exposes AWS_BATCH_JOB_ARRAY_INDEX; a Slurm array exposes SLURM_ARRAY_TASK_ID. A two-line wrapper maps the index to a tile from a manifest and calls pdal pipeline with overrides, so the container image carries one pipeline file and never needs rebuilding for a new tile list.

#!/usr/bin/env bash
set -euo pipefail
idx="${AWS_BATCH_JOB_ARRAY_INDEX:-${SLURM_ARRAY_TASK_ID:-0}}"
tile=$(sed -n "$((idx + 1))p" manifest.txt)
base=$(basename "$tile" .laz)
exec pdal pipeline /opt/pipelines/dtm.json \
  --readers.las.filename="/vsis3/lidar-in/$tile" \
  --writers.gdal.filename="/vsis3/lidar-out/dtm/${base}.tif"

Because the manifest is the only thing that changes between runs, retrying a failed index reproduces exactly the same command. The full pattern, including how the manifest is uploaded and how failures are collected, is in array jobs for LiDAR tiles in AWS Batch.

# Key Parameter Table

# Verification

• Check the serialized pipeline. Open one out/*.pipeline.json and confirm the filenames and any overridden options are what you expected.
• Count outputs. The number of rasters should equal the number of tiles minus failures in the job log.
• Spot-check a raster. gdalinfo -stats out/t_0432_dtm.tif should show a plausible elevation range, not the range of the tile named in the JSON file.

jq -r '.pipeline[] | select(.type=="readers.las") | .filename' out/t_0432.pipeline.json
ls out/*_dtm.tif | wc -l

# Gotchas and Edge Cases

Overrides do not exist in the Python bindings. pdal.Pipeline takes JSON only. In Python, modify the parsed JSON dict or use the stage API — see building pipelines with the Python stage API.

Shell quoting of expressions. Range limits and expressions contain brackets, commas and sometimes &&. Quote the whole --option=value argument, and inside parallel commands, double-check the layers of quoting with --dry-run first.

An override for a missing option adds it. Overriding an option the stage does not set in the JSON adds it — useful, but a misspelled option name is then rejected at run time rather than ignored. That is a feature: typos fail loudly.

Readers inferred from filenames. A pipeline whose first element is a bare filename string has an inferred reader type. Override it with the concrete type — --readers.las.filename — which only works if the inferred type is readers.las. Explicit stage objects avoid the ambiguity.

# Frequently Asked Questions

Can I override options when running PDAL from Python?

Not with the command-line syntax. In Python, load the JSON into a dict and change the option, or build the pipeline with the stage API. Both produce the same result as a command-line override.

What happens if two stages share a type and I override that type?

The override applies to every stage of that type. Tag the stages and use the stage.tag.option form to change only one of them.

How do I see the pipeline that actually ran after overrides?

Pass the pipeline-serialization option with a filename; PDAL writes the effective pipeline, including overridden values, to that file.

Are command-line overrides enough for multi-project batches?

For paths and one or two values, yes. When the set of stages or many options differ per project, a template or the Python stage API is easier to maintain.

# Related

• PDAL Pipeline Templating and Parameterization — comparing mechanisms
• Parameterizing Pipelines with Jinja Templates — when overrides are not enough
• Building Pipelines with the Python Stage API — the Python equivalent
• Array Jobs for LiDAR Tiles in AWS Batch — overrides driven by an array index
• Validating PDAL Pipelines in CI — validating the base JSON once