Python LiDAR

Array Jobs for LiDAR Tiles in AWS Batch

TL;DR: Write the tile list to a manifest in S3, submit one array job with arrayProperties.size equal to the tile count (up to 10,000), and have the container read AWS_BATCH_JOB_ARRAY_INDEX to pick its line from the manifest. Each child is an independent job with its own retries and logs, and a second array job can depend on the first with N_TO_N so tile i of stage two starts as soon as tile i of stage one finishes.

# Context and Motivation

This guide is part of AWS Batch Processing. Submitting one Batch job per tile works, but at thousands of tiles it means thousands of SubmitJob calls, API throttling, and a job list nobody can read. An array job is a single submission that fans out into child jobs, each told its index through an environment variable. The submission is instant, the console groups the children under one parent, and the retry strategy applies to each child independently.

For LiDAR the pattern maps cleanly: index i is tile i of a manifest, and the container is the same PDAL image used for single-tile runs, as described in scaling PDAL tile processing with AWS Batch.

Index to tile through a manifest A manifest file in S3 lists tile keys on numbered lines 0 to n. One array job submission creates child jobs 0 to n. Each child reads AWS_BATCH_JOB_ARRAY_INDEX and fetches the matching line from the manifest, then runs PDAL on that tile. manifest.txt 0 tiles/571_4190.laz 1 tiles/571_4191.laz 2 tiles/572_4190.laz n tiles/598_4222.laz ⋮ child 0 → line 0 child 1 → line 1 child n → line n ⋮ AWS_BATCH_JOB_ARRAY_INDEX selects the line

# Prerequisites and Assumptions

  • An AWS Batch compute environment and job queue (EC2 or Fargate), and a job definition using a PDAL image in ECR.
  • An IAM job role with read access to the input bucket and write access to the output bucket.
  • A tile list, for example from a tile index built with pdal tindex or an S3 listing.

# Step-by-Step Implementation

# Step 1 — Write the manifest

One tile key per line, sorted, uploaded next to the run’s outputs so every run records exactly what it processed.

# Step 2 — Write the entrypoint

The container script reads AWS_BATCH_JOB_ARRAY_INDEX and MANIFEST_URI, fetches the manifest, and processes line index.

# Step 3 — Size the child job

Set vcpus and memory from a measured tile: PDAL memory scales with points per tile, so measure the largest tile rather than the average.

# Step 4 — Submit the array job

submit_job with arrayProperties={"size": n} and a retryStrategy that retries on host termination.

# Step 5 — Chain stages with N_TO_N

A second array job of the same size with dependsOn=[{"jobId": first, "type": "N_TO_N"}] starts child i as soon as child i of the first job succeeds.

# Complete Working Example

Entrypoint inside the container:

python
#!/usr/bin/env python3
"""entrypoint.py: process one tile chosen by the array index."""
import json
import os
import sys

import boto3
import pdal

s3 = boto3.client("s3")
idx = int(os.environ.get("AWS_BATCH_JOB_ARRAY_INDEX", "0"))
bucket, key = os.environ["MANIFEST_URI"].removeprefix("s3://").split("/", 1)
tiles = s3.get_object(Bucket=bucket, Key=key)["Body"].read().decode().split()
tile = tiles[idx]
stem = tile.rsplit("/", 1)[-1].removesuffix(".laz")
out = f"{os.environ['OUT_PREFIX']}/{stem}.tif"

spec = {"pipeline": [
    f"/vsis3/lidar-in/{tile}",
    {"type": "filters.smrf", "slope": 0.15, "window": 18},
    {"type": "filters.range", "limits": "Classification[2:2]"},
    {"type": "writers.gdal", "filename": f"/vsis3/lidar-out/{out}",
     "resolution": 1.0, "output_type": "idw", "data_type": "float32"},
]}
n = pdal.Pipeline(json.dumps(spec)).execute()
print(json.dumps({"index": idx, "tile": tile, "ground_points": n}))
sys.exit(0 if n else 3)

Submission from a workstation or an orchestrator:

python
import boto3

batch, s3 = boto3.client("batch"), boto3.client("s3")
tiles = sorted(o["Key"] for p in s3.get_paginator("list_objects_v2")
               .paginate(Bucket="lidar-in", Prefix="tiles/") for o in p.get("Contents", [])
               if o["Key"].endswith(".laz"))
s3.put_object(Bucket="lidar-out", Key="runs/2026-09-18/manifest.txt",
              Body="\n".join(tiles).encode())

env = [{"name": "MANIFEST_URI", "value": "s3://lidar-out/runs/2026-09-18/manifest.txt"},
       {"name": "OUT_PREFIX", "value": "runs/2026-09-18/dtm"}]

dtm = batch.submit_job(
    jobName="dtm-2026-09-18", jobQueue="lidar-spot", jobDefinition="pdal-dtm:7",
    arrayProperties={"size": len(tiles)},
    containerOverrides={"environment": env,
                        "resourceRequirements": [{"type": "VCPU", "value": "2"},
                                                 {"type": "MEMORY", "value": "8192"}]},
    retryStrategy={"attempts": 3, "evaluateOnExit": [
        {"onStatusReason": "Host EC2*", "action": "RETRY"},
        {"onExitCode": "3", "action": "EXIT"},
        {"onReason": "*", "action": "EXIT"}]},
)["jobId"]

hillshade = batch.submit_job(
    jobName="hillshade-2026-09-18", jobQueue="lidar-spot", jobDefinition="gdal-hillshade:2",
    arrayProperties={"size": len(tiles)},
    dependsOn=[{"jobId": dtm, "type": "N_TO_N"}],
    containerOverrides={"environment": env},
)["jobId"]
print(dtm, hillshade)
N_TO_N chaining Two rows of child jobs. The top row is the DTM array job with children 0 to 4, the bottom row is the hillshade array job with children 0 to 4. Each DTM child has an arrow to the hillshade child with the same index, so hillshade 2 can start as soon as DTM 2 finishes, without waiting for the whole DTM job. DTM job hillshade job 0 1 2 3 4 0 1 2 3 4 child i of stage two waits only for child i of stage one

# Sizing Children From the Largest Tile

Batch places child jobs by their declared vcpus and memory. If the declaration is too small, the container is killed with OutOfMemoryError: Container killed due to memory usage and retries fail the same way; if it is too large, fewer children fit per instance and the run takes longer and costs more. Tiles in a project are rarely uniform — dense urban tiles and tiles covering overlap between flightlines can hold several times the points of a rural tile — so size the job definition from the largest tile, not the median. Measure peak memory on that tile locally as described in measuring peak memory of a PDAL pipeline, then add around 25% headroom.

When the spread is extreme, split the manifest into two array jobs — normal tiles with a small memory request and a handful of dense tiles with a large one. The cost saving from packing the common case tightly usually outweighs the small complexity of two submissions.

# Key Parameter Table

Setting Typical Notes
arrayProperties.size 2–10,000 One child per tile; split larger lists
AWS_BATCH_JOB_ARRAY_INDEX 0…size−1 Set in each child
resourceRequirements VCPU 1–4 PDAL is mostly single-threaded per tile
resourceRequirements MEMORY from largest tile +25% headroom
retryStrategy.attempts 2–3 Per child
evaluateOnExit retry on Host EC2* Exit early on permanent codes
dependsOn type N_TO_N / sequential Per-index vs whole-job dependency

# Verification

  • Child count. The parent job’s array status counts total children equal to the manifest length.
  • Outputs. The output prefix contains one raster per manifest line; list both and diff.
  • Chaining. Hillshade children start while DTM children are still running, which is visible in the start times.

# Gotchas and Edge Cases

The 10,000 limit. An array job holds at most 10,000 children. For larger projects, split the manifest into chunks and submit one array job per chunk, or have each child process a small group of tiles.

Manifest immutability. Children read the manifest when they start, possibly hours after submission. Never overwrite a manifest during a run; write each run’s manifest to a new key.

Failure of the parent. The parent array job is marked failed if any child fails after its retries, and an N_TO_N dependant child whose upstream child failed never runs. Collect failures from the child statuses and resubmit only those tiles with a smaller manifest.

Chunking beyond 10,000 tiles A 24,000-tile manifest is split into three chunk manifests of 10,000, 10,000 and 4,000 lines. Each chunk is submitted as its own array job, keeping every job within the 10,000-child limit. 24,000 tiles one manifest array job A10,000 array job B10,000 array job C4,000 each chunk gets its own manifest key

# Frequently Asked Questions

How does a child job know which tile to process?

AWS Batch sets AWS_BATCH_JOB_ARRAY_INDEX in every child. The container reads it and takes that line of a manifest file listing the tiles, so the same image serves every child.

How many tiles can one array job handle?

Up to 10,000 children. Split larger tile lists into several manifests and array jobs, or let each child process a small group of tiles.

What does an N_TO_N dependency do?

It links two array jobs of the same size index by index: child i of the second job starts once child i of the first succeeds, instead of waiting for the entire first job.

How should I set memory for PDAL array children?

Measure peak memory on the densest tile and add about a quarter as headroom. If tile sizes vary widely, submit dense tiles as a separate array job with a larger memory request.

Related in AWS Batch Processing for PDAL Point Clouds

↑ AWS Batch Processing for PDAL Point Clouds overview --- title: "AWS Batch Processing for PDAL Point Clouds" description: "Running thousands of PDAL tile jobs on AWS Batch — job definitions,... Scaling PDAL Tile Processing with AWS Batch --- title: "Scaling PDAL Tile Processing with AWS Batch" description: "A worked example of fanning out a PDAL ground-to-DTM pipeline across... Handling Spot Interruptions in PDAL Batch Jobs --- title: "Handling Spot Interruptions in PDAL Batch Jobs" description: "Decompose the tile, write the marker after the object, trap... Estimating Cloud Cost per LiDAR Tile --- title: "Estimating Cloud Cost per LiDAR Tile" description: "Estimate what a cloud LiDAR processing run will cost before launching it:...