Python LiDAR

Handling Spot Interruptions in PDAL Batch Jobs

TL;DR: Trap SIGTERM, flush whatever is complete, write the marker only after the output object is durable, and make the job’s first action a check for that marker — then a spot reclaim costs one sub-tile instead of the whole job.

# Context and Motivation

This guide is part of AWS Batch Processing for PDAL. Spot capacity is roughly a third of the on-demand price, and at a typical interruption rate of a few percent the arithmetic is not close — provided an interruption is cheap. Making it cheap is a design property, not a setting.

The mechanism is straightforward. When capacity is reclaimed, the instance receives a termination notice and the container gets SIGTERM, followed about two minutes later by SIGKILL. Two minutes is ample to finish writing a sub-tile and record that it is done. It is nowhere near enough to finish a forty-minute pipeline, so the design has to be built from units that fit inside the warning.

Two minutes is enough for a checkpoint, not for a job Two timelines of the same 40-minute tile job interrupted at 28 minutes. Without checkpoints all 28 minutes are discarded and the retry starts from zero. With sub-tile checkpoints the completed sub-tiles are already durable, so the retry resumes and loses only the four minutes of work in flight. no checkpoints 28 minutes of work, all discarded never reached sub-tile checkpoints saved saved saved saved in flight retry continues here SIGTERM — about two minutes of warning the unit of work has to be smaller than the warning, or the warning cannot be used for anything four minutes lost against twenty-eight is the difference between spot being cheap and spot being a false economy

# Prerequisites and Assumptions

Requirement Detail
AWS Batch on a spot compute environment
A decomposable job sub-tiles, or a natural checkpoint boundary
Durable output S3, with the marker written after the object
Signal handling in the entrypoint, not in a wrapper script that ignores it
Retries enabled attempts above 1 in the job definition

# Step-by-Step Implementation

# Step 1 — Decompose the tile into sub-tiles

Choose a size that completes well inside two minutes. On a typical pipeline that is a few hundred metres square.

# Step 2 — Write the marker after the object

python
write_output(sub_tile)      # object is durable first
put_marker(sub_tile)        # only then is it recorded as done

Reversing these turns an interruption into permanent missing data — the point made in scaling PDAL tile processing with AWS Batch.

# Step 3 — Skip completed sub-tiles on entry

The first action of every attempt is to list the markers and remove those sub-tiles from the work list.

# Step 4 — Trap SIGTERM and stop cleanly

python
signal.signal(signal.SIGTERM, lambda *_: stop.set())

Check the flag between sub-tiles rather than trying to abort mid-write.

# Step 5 — Exit with a retryable status

Exit non-zero on interruption so Batch retries the job; the retry will skip everything already marked.

# Complete Working Example

python
"""A Batch entrypoint that survives spot reclamation."""
from __future__ import annotations

import json
import logging
import os
import signal
import sys
import threading
from pathlib import Path

import boto3

LOG = logging.getLogger("worker")
S3 = boto3.client("s3")
BUCKET = os.environ["OUTPUT_BUCKET"]
PREFIX = os.environ["OUTPUT_PREFIX"]

stop = threading.Event()


def _on_term(signum, frame):  # noqa: ARG001
    LOG.warning("received signal %s — finishing the current sub-tile and stopping", signum)
    stop.set()


def marker_key(sub_tile: str) -> str:
    return f"{PREFIX}/_markers/{sub_tile}.done"


def already_done(sub_tile: str) -> bool:
    try:
        S3.head_object(Bucket=BUCKET, Key=marker_key(sub_tile))
        return True
    except S3.exceptions.ClientError:
        return False


def process_sub_tile(sub_tile: str) -> None:
    """Run the pipeline for one sub-tile and upload the result."""
    out = Path(f"/scratch/{sub_tile}.laz")
    # ... pdal.Pipeline(...).execute() writes `out` ...
    S3.upload_file(str(out), BUCKET, f"{PREFIX}/{sub_tile}.laz")
    # Marker last: the object is durable before anything claims it is.
    S3.put_object(Bucket=BUCKET, Key=marker_key(sub_tile), Body=b"")
    out.unlink(missing_ok=True)


def main() -> int:
    logging.basicConfig(level=logging.INFO, format="%(asctime)s %(message)s")
    signal.signal(signal.SIGTERM, _on_term)
    signal.signal(signal.SIGINT, _on_term)

    sub_tiles = json.loads(Path("/work/sub_tiles.json").read_text())
    pending = [s for s in sub_tiles if not already_done(s)]
    LOG.info("%d of %d sub-tiles remain", len(pending), len(sub_tiles))

    for sub_tile in pending:
        if stop.is_set():
            LOG.warning("stopping early; %d sub-tiles left for the retry", len(pending))
            return 75  # non-zero: Batch retries, and the retry skips what is done
        process_sub_tile(sub_tile)
        LOG.info("completed %s", sub_tile)

    return 0


if __name__ == "__main__":
    sys.exit(main())
Where spot stops being cheaper Effective cost per completed tile on spot capacity, at four interruption rates, compared with the on-demand price. With sub-tile checkpoints the wasted work per interruption is small, so spot stays about a third of on-demand even at a fifteen percent interruption rate. Without checkpoints the same rate makes spot more expensive than on-demand. on-demand $0.100 — the baseline spot, 2% interrupted, checkpointed $0.034 spot, 15% interrupted, checkpointed $0.039 spot, 15% interrupted, no checkpoints $0.118 — worse than on-demand effective cost per completed tile, including work discarded by interruptions the whole spot argument rests on the fourth row not being the one you are running

# Key Parameter Table

Setting Where Guidance
sub-tile size your decomposition Completes well inside the two-minute warning
marker order entrypoint Object first, marker second, always
attempts job definition 3 is typical on spot
exit code entrypoint Non-zero on interruption so Batch retries
SIGTERM handler entrypoint Sets a flag; never aborts a write in progress

# Verification

A killed job resumes. Send SIGTERM to a running container and confirm the retry starts from the first unmarked sub-tile.

Markers never precede objects. Delete an output object but leave its marker, re-run, and confirm the gap is visible — because it will not be filled.

Retries are cheap. A re-run of a completed job should finish in seconds, having found every marker.

Every hop is a place the signal can be dropped The termination notice travels from the spot service to the ECS agent, to the container runtime, to PID 1 inside the container, and only then to your handler. A shell wrapper that is PID 1 and does not forward signals absorbs it, and the handler never runs — the single most common reason a correctly written trap appears to do nothing. spot service ECS agent PID 1 in container shell, or your process your SIGTERM handler sh -c "python worker.py" — signal stops here use exec, or an ENTRYPOINT in exec form, so the Python process is PID 1 and receives the signal directly rather than inheriting it from a shell that has already exited.

# Gotchas and Edge Cases

A wrapper script that swallows the signal. sh -c "python worker.py" may not forward SIGTERM. Use exec, or make the Python process PID 1.

Sub-tiles that are too large. If one takes five minutes, the warning cannot save it and checkpointing buys nothing.

Markers in the same prefix as outputs. A later listing then treats markers as data. Keep them under their own prefix.

Assuming the warning always arrives. It usually does; occasionally an instance simply disappears. The design must be correct without it, which it is — the marker is the only source of truth.

# Frequently Asked Questions

How long does a spot instance give me?

About two minutes between the termination notice and the kill. That is ample to finish writing one sub-tile and record it, and nowhere near enough to finish a forty-minute pipeline — which is why the unit of work has to be smaller than the warning for the warning to be useful at all.

Why must the marker be written after the output?

Because a marker written first turns a crash into permanent missing data. A later run finds the marker, concludes the tile is done, and skips it forever. Writing the object first means the worst case is redundant work, which costs money rather than correctness.

Is spot actually worth it?

At roughly a third of the on-demand price and a few percent interruption rate, yes, provided an interruption is cheap. With sub-tile checkpoints an interruption costs a few minutes; without them it costs the whole job, and at that point the arithmetic can reverse.

What if the termination notice never arrives?

It occasionally does not, and the design has to be correct without it. Because the marker is written only after the object is durable, an instance that simply disappears leaves an unmarked sub-tile that the retry picks up — the same path as an orderly shutdown.

Related in AWS Batch Processing for PDAL Point Clouds

↑ AWS Batch Processing for PDAL Point Clouds overview --- title: "AWS Batch Processing for PDAL Point Clouds" description: "Running thousands of PDAL tile jobs on AWS Batch — job definitions,... Scaling PDAL Tile Processing with AWS Batch --- title: "Scaling PDAL Tile Processing with AWS Batch" description: "A worked example of fanning out a PDAL ground-to-DTM pipeline across...