Retrying Failed Tiles in Airflow

TL;DR: Give the per-tile task retries, retry_delay and retry_exponential_backoff=True so throttling and spot interruptions heal themselves, but raise AirflowFailException for errors that will never succeed — a corrupt LAZ or a missing CRS — so they fail at once without wasting retries. Afterwards, clear only the failed mapped instances rather than rerunning the DAG, and make outputs idempotent so reruns are safe.

# Context and Motivation

This guide is part of Airflow DAG Orchestration. In a run of thousands of tiles, some tasks will fail. Most failures in cloud LiDAR processing are transient: S3 returning SlowDown, a worker pod evicted, a spot instance reclaimed. A few are permanent: a truncated LAZ, a tile with no ground points, a file whose header says it has a CRS it does not. Treating both kinds the same is costly either way. Too few retries and transient failures need manual attention; too many and every corrupt tile burns an hour of retries before anyone looks at it.

# Prerequisites and Assumptions

• A per-tile Airflow task, ideally dynamically mapped as in dynamic task mapping for LiDAR tiles.
• Outputs written to deterministic keys, so a rerun overwrites rather than duplicates.
• Access to the Airflow UI or CLI to clear task instances.

# Step-by-Step Implementation

# Step 1 — Set retry policy on the tile task

retries=3, retry_delay=timedelta(minutes=2), retry_exponential_backoff=True and max_retry_delay=timedelta(minutes=30).

# Step 2 — Classify exceptions

Wrap the PDAL call; for known permanent errors raise AirflowFailException, which fails the instance immediately with no retries. Let everything else propagate normally and use the retry budget.

# Step 3 — Skip tiles that should not be processed

Raise AirflowSkipException for tiles outside the project area or already complete, so they show as skipped instead of failed.

# Step 4 — Add a failure callback

on_failure_callback records the tile and error in a table or sends an alert, giving a list of failed tiles without trawling logs.

# Step 5 — Rerun only failures

After the fix, clear the failed mapped instances with airflow tasks clear -s ... --only-failed, or in the grid view, and let the reduce task run again.

# Complete Working Example

from __future__ import annotations

import json
from datetime import timedelta

from airflow.decorators import task
from airflow.exceptions import AirflowFailException, AirflowSkipException

PERMANENT = ("Invalid LAZ", "Unable to read", "No points", "readers.las: Invalid")


def record_failure(context):
    ti = context["ti"]
    print(json.dumps({"dag_run": context["run_id"], "task": ti.task_id,
                      "map_index": ti.map_index, "error": str(context.get("exception"))}))


@task(retries=3, retry_delay=timedelta(minutes=2), retry_exponential_backoff=True,
      max_retry_delay=timedelta(minutes=30), pool="pdal_slots",
      on_failure_callback=record_failure)
def process_tile(key: str, out_prefix: str) -> dict:
    import boto3
    import pdal

    stem = key.rsplit("/", 1)[-1].removesuffix(".laz")
    out_key = f"{out_prefix}/{stem}.tif"

    s3 = boto3.client("s3")
    if s3.list_objects_v2(Bucket="lidar-out", Prefix=out_key).get("KeyCount"):
        raise AirflowSkipException(f"{out_key} already exists")

    spec = {"pipeline": [
        f"/vsis3/lidar-in/{key}",
        {"type": "filters.smrf"},
        {"type": "filters.range", "limits": "Classification[2:2]"},
        {"type": "writers.gdal", "filename": f"/vsis3/lidar-out/{out_key}",
         "resolution": 1.0, "output_type": "idw"},
    ]}
    try:
        n = pdal.Pipeline(json.dumps(spec)).execute()
    except RuntimeError as exc:
        if any(p in str(exc) for p in PERMANENT):
            raise AirflowFailException(f"{key}: permanent error: {exc}") from exc
        raise                      # transient: let Airflow retry with backoff
    if n == 0:
        raise AirflowFailException(f"{key}: no ground points after SMRF")
    return {"tile": stem, "ground_points": n}

Clearing only the failed instances of one run from the CLI:

airflow tasks clear lidar_dtm_daily \
  --task-regex '^process_tile$' --only-failed \
  --start-date 2026-09-17 --end-date 2026-09-17 --yes

# Backoff Arithmetic

With retry_delay of two minutes and exponential backoff, the waits grow roughly as 2, 4 and 8 minutes, with jitter, capped at max_retry_delay. Three retries therefore span about a quarter of an hour, long enough for S3 throttling to subside or for a replacement spot node to join the cluster, and short enough that a run does not stall for hours on one tile. If failures are mostly spot interruptions, a longer first delay helps more than more retries, because capacity often returns in bursts.

# Where Retries Belong: Task or Operator

When PDAL runs inside a container launched by KubernetesPodOperator or an AWS Batch operator, there are two retry layers: Airflow’s task retries and the platform’s own (Batch job retryStrategy, Kubernetes pod restarts). Pick one owner per failure type. Letting both retry multiplies attempts — three Airflow retries of a Batch job with three attempts each is up to sixteen runs of a corrupt tile. A workable split is to let the platform absorb infrastructure failures such as spot reclamation, which it detects precisely through exit reasons, and let Airflow own application failures, where the exception classification above applies.

Record the attempt number in the output metadata or the failure log (ti.try_number inside the task). A tile that only succeeds on its third attempt, run after run, points to a resource problem — memory too tight, a slow disk — that retries are hiding rather than fixing.

# Key Parameter Table

# Verification

• Inject a corrupt file. Put a truncated LAZ in the input list; its instance fails once, with no retries, and the callback logs it.
• Inject throttling. Temporarily lower S3 request limits or kill a worker mid-task; the instance retries and eventually succeeds.
• Rerun count. After clearing failures, only those instances run; completed tiles stay green.

# Gotchas and Edge Cases

Idempotency is a precondition. Retrying a task that wrote half an output and then crashed must produce a correct file, not append to a broken one. Write to a temporary key and rename it on success, or overwrite deterministically — see making tile outputs idempotent.

Matching error strings is brittle. PDAL error messages change between versions. Keep the list of permanent patterns short, and back it with a pre-check (for example pdal info --summary in a quick upstream task) that validates headers before the heavy work.

Timeouts count as failures. Set execution_timeout so a hung read eventually fails and retries, rather than holding a pool slot forever.

# Frequently Asked Questions

How do I stop Airflow retrying a tile that will never succeed?

Raise AirflowFailException from the task. The instance is marked failed immediately and its remaining retries are not used, which suits corrupt files or tiles with no usable points.

What retry settings suit cloud LiDAR tasks?

Two to four retries with a base delay of a couple of minutes and exponential backoff capped around thirty minutes. That covers S3 throttling and node loss without stalling runs for hours.

How do I rerun only the failed tiles?

Clear the failed mapped instances, either in the grid view or with airflow tasks clear and the only-failed flag. Successful instances keep their state, and downstream tasks rerun once the cleared ones finish.

Why must tile outputs be idempotent for retries?

A retried task may run after a partial write. If outputs are written to a temporary key and moved into place on success, or overwritten deterministically, retries cannot leave corrupt or duplicated results.

# Related

• Airflow DAG Orchestration — Airflow for LiDAR pipelines
• Dynamic Task Mapping for LiDAR Tiles — the mapped task retried here
• Making Tile Outputs Idempotent — safe reruns
• Handling Spot Interruptions in PDAL Batch Jobs — the most common transient failure
• Tracking Tile Progress and Failures in Dask — the same idea in Dask