Python LiDAR

Making Tile Outputs Idempotent

TL;DR: Give every output a key derived only from the tile and the processing version, write it to a temporary name and move it into place only after success, and store a small marker recording a hash of the input and the pipeline parameters. A task then checks the marker first: if it matches, skip; otherwise redo the tile and overwrite. Running the same job once or five times gives the same result, which is what makes retries, partial reruns and backfills safe.

# Context and Motivation

This guide is part of Tile Indexing and Merging. Distributed tile processing always involves reruns: spot instances disappear mid-write, a scheduler retries a task it believes failed, someone reruns yesterday’s job after fixing one tile, or a backfill overlaps a scheduled run. If outputs are not idempotent, each rerun risks a corrupt half-written raster that looks complete, a merged product containing a tile twice, or an appended LAZ with duplicated points — errors that are expensive to find later because nothing failed loudly.

An idempotent task is one whose effect is the same no matter how many times it runs. For tile work that comes down to a few simple rules about names, writes and markers.

Check, write, publish A tile task first reads the completion marker. If the marker's input and parameter hashes match, it skips. Otherwise it writes the output to a temporary key, validates it, moves it to the final deterministic key, and writes the marker last. read markerhashes match? skip write totmp key validate move tofinal key writemarker yes no

# Prerequisites and Assumptions

  • Tiles with stable identifiers, for example from retiling flightline files into a grid.
  • Output storage that is either a filesystem (atomic rename) or S3 (atomic single-object PUT, with strong read-after-write consistency).
  • A version string for your processing — pipeline version, image tag or lock hash.

# Step-by-Step Implementation

# Step 1 — Derive output keys deterministically

Build the output key from the product, the processing version and the tile ID only, for example dtm/v3/571000_4190000.tif. Never include timestamps, run IDs or random suffixes in final keys.

# Step 2 — Write to a temporary location

Write to …/_tmp/<tile>.<uuid>.tif, or a local scratch file, so a crash never leaves a partial file at the final key.

# Step 3 — Validate before publishing

Open the temporary output and check it has the expected size, CRS and a plausible value range.

# Step 4 — Publish atomically

On a filesystem, os.replace the temp file onto the final path. On S3, upload the complete file with a single PUT or a completed multipart upload, which becomes visible all at once, and then delete the temporary object if you used one.

# Step 5 — Write the marker last

Write a small JSON marker with the input ETag or hash, the parameter hash and the output checksum. Tasks check it first and skip when it matches.

# Complete Working Example

python
"""Idempotent per-tile DTM task for local disk or S3."""
import hashlib
import json
import os
import tempfile

import boto3
import pdal
import rasterio

s3 = boto3.client("s3")
BUCKET = "lidar-out"
VERSION = "v3"
PARAMS = {"slope": 0.15, "window": 18.0, "threshold": 0.5, "resolution": 1.0}
PARAM_HASH = hashlib.sha256(json.dumps(PARAMS, sort_keys=True).encode()).hexdigest()[:16]


def marker_key(tile):
    return f"dtm/{VERSION}/_markers/{tile}.json"


def is_done(tile, input_etag):
    try:
        m = json.loads(s3.get_object(Bucket=BUCKET, Key=marker_key(tile))["Body"].read())
    except s3.exceptions.NoSuchKey:
        return False
    return m["input_etag"] == input_etag and m["param_hash"] == PARAM_HASH


def process(tile, in_bucket="lidar-in"):
    in_key = f"tiles/{tile}.laz"
    etag = s3.head_object(Bucket=in_bucket, Key=in_key)["ETag"].strip('"')
    if is_done(tile, etag):
        return "skipped"

    final_key = f"dtm/{VERSION}/{tile}.tif"
    with tempfile.TemporaryDirectory() as tmp:
        local = os.path.join(tmp, f"{tile}.tif")
        spec = {"pipeline": [
            f"/vsis3/{in_bucket}/{in_key}",
            {"type": "filters.smrf", "slope": PARAMS["slope"], "window": PARAMS["window"],
             "threshold": PARAMS["threshold"]},
            {"type": "filters.range", "limits": "Classification[2:2]"},
            {"type": "writers.gdal", "filename": local, "resolution": PARAMS["resolution"],
             "output_type": "idw", "data_type": "float32", "nodata": -9999},
        ]}
        pdal.Pipeline(json.dumps(spec)).execute()

        with rasterio.open(local) as src:                         # validate before publishing
            z = src.read(1, masked=True)
            assert src.crs is not None and z.count() > 0, f"{tile}: empty or no CRS"
            assert -500 < float(z.min()) and float(z.max()) < 9000, f"{tile}: implausible z"

        digest = hashlib.sha256(open(local, "rb").read()).hexdigest()
        s3.upload_file(local, BUCKET, final_key)                  # atomic: whole object or none

    s3.put_object(Bucket=BUCKET, Key=marker_key(tile), Body=json.dumps({
        "tile": tile, "input_etag": etag, "param_hash": PARAM_HASH,
        "version": VERSION, "output_sha256": digest}).encode())
    return "written"

Because the marker is written after the output, a crash between the two leaves an output without a marker; the next run redoes the tile and overwrites the same key, which is harmless. The reverse — a marker without an output — cannot happen.

Crash at any point is safe A timeline of the task with three crash points. A crash during processing leaves only a temporary file, ignored by readers. A crash after upload but before the marker leaves a correct output without a marker, so the next run redoes and overwrites it. A crash after the marker leaves a completed tile that the next run skips. process to tmp upload final write marker only a tmp filereaders never see it output, no markerrerun overwrites complete: rerun skips

# Versioning Instead of Overwriting History

Idempotency and reproducibility pull slightly in different directions: idempotent tasks overwrite, but you may want to keep the outputs of the previous processing version. Put the version in the key prefix — dtm/v3/ — and bump it whenever parameters or the software environment change in a way that alters results. Reruns within a version overwrite identical results; a new version writes alongside the old one, and consumers switch by prefix. Including the parameter hash in the marker, not just the version, catches the case where someone edits parameters but forgets to bump the version: the marker no longer matches, so every tile is reprocessed rather than silently mixing old and new results within one version.

Merged products need the same care. Build mosaics and merged LAZ files from the tile outputs listed in markers, not from a directory listing that might include temporary objects, and write the merged product to a key that includes the version and a hash of the input list, as in merging processed tiles into one LAZ.

# Key Parameter Table

Rule Implementation Prevents
Deterministic keys product/version/tile Duplicates across reruns
Temp then publish os.replace or single PUT Partial files at final keys
Validate first open and range-check Publishing empty or broken outputs
Marker last JSON with input and param hashes Skipping stale results
Version prefix v3/ Mixing processing versions
No appends overwrite whole tile outputs Duplicated points

# Verification

  • Run twice. Run a batch, then run it again unchanged; the second run skips every tile and the output checksums are identical.
  • Kill mid-run. Stop workers partway through, rerun, and confirm there are no truncated outputs and no tile is missing.
  • Change a parameter. Edit one parameter without bumping the version; every tile’s marker mismatches and the tiles are reprocessed.

# Gotchas and Edge Cases

Appending is never idempotent. Writers that append to an existing LAZ or GeoPackage layer duplicate data on retry. Write per-tile outputs and merge them afterwards from the marker list.

ETags of multipart uploads. An S3 ETag for a multipart object is not an MD5 of the content, but it is stable for the same object, which is all the marker needs. If inputs may be rewritten with identical content, use a content hash instead.

Clock-based names. A key such as dtm_2026-09-18T10:22.tif creates a new object on every retry. Put timestamps in marker metadata, never in output keys.

Append duplicates, overwrite does not Two outcomes after a task runs twice. Appending to a shared output leaves the tile's points in it twice. Overwriting a per-tile output leaves exactly one copy, identical to a single run. append on retry tile points written twice density doubles silently overwrite per tile one copy, same bytes retries are harmless merge afterwards from the marker list

# Frequently Asked Questions

What does idempotent mean for LiDAR tile processing?

Running a tile task once or many times produces the same outputs. Retries, reruns and overlapping backfills then cannot create duplicates, partial files or mixed results.

How do I avoid partial output files when a job crashes?

Write to a temporary file or key, validate it, and only then move it to the final key. A rename on a filesystem or a single-object upload to S3 makes the complete file appear at once.

How can a task tell whether a tile is already done?

Store a marker after publishing the output, recording a hash of the input and of the processing parameters. The task compares these with the current input and parameters and skips only when both match.

Should output file names include timestamps?

No. Output keys should depend only on the product, processing version and tile, so a rerun overwrites the same object. Record timestamps inside the marker instead.

Related in Tile Indexing, Buffering and Merging

↑ Tile Indexing, Buffering and Merging overview --- title: "Tile Indexing, Buffering and Merging" description: "Building a queryable tile index from LiDAR headers, using it to find... Buffered Tiling to Avoid Edge Artefacts --- title: "Buffered Tiling to Avoid Edge Artefacts" description: "Read the tile plus its neighbours, process the buffered extent, write... Building a Tile Index with pdal tindex --- title: "Building a Tile Index with pdal tindex" description: "The one-line command that turns a directory of LAZ files into a queryable... Retiling Flightline Files into a Grid --- title: "Retiling Flightline Files into a Grid" description: "Convert LiDAR delivered as long flightline swaths into a regular square... Merging Processed Tiles into One LAZ --- title: "Merging Processed Tiles into One LAZ" description: "Crop each tile to its nominal extent before merging or the overlap strips...