Python LiDAR

Estimating Cloud Cost per LiDAR Tile

TL;DR: Run the pipeline on a sample of tiles spanning the density range and record wall time, CPU time, peak memory and bytes read and written for each. Fit cost per tile as compute (instance price × instance-seconds ÷ tiles per instance) plus requests plus storage, then multiply by the tile count with a margin for retries. Compute usually dominates, and it scales with points, not with tile area.

# Context and Motivation

This guide is part of AWS Batch Processing. “How much will it cost to process the county?” is a question to answer before submitting 8,000 array children, not after the bill arrives. The answer is not hard to estimate, but it needs measurement rather than guesses: PDAL runtime varies by an order of magnitude between a sparse rural tile and a dense urban tile covered by several overlapping flightlines, and ground filters such as SMRF scale worse than linearly with point count.

A per-tile model built from twenty or thirty measured tiles predicts the full run well enough to choose between instance types, between spot and on-demand, and between processing everything and processing only what changed.

What a tile costs A stacked breakdown of cost per tile into compute, the largest share, then S3 requests, S3 storage of outputs, and data transfer, which is zero within a region. Compute depends on points per tile and instance price; the others depend on bytes and request counts. compute: price × instance-seconds requests storage xfer scales with points per tile GET / PUT GB-month ≈0 in-region widths illustrate a typical DTM run; measure your own

# Prerequisites and Assumptions

  • A working per-tile pipeline and container, as in array jobs for LiDAR tiles in AWS Batch.
  • A tile index with point counts per tile (from pdal info --summary or the tile index), so the sample can be stratified.
  • Current prices for your region from the AWS pricing pages. Prices below are placeholders for the arithmetic, not quotes.

# Step-by-Step Implementation

# Step 1 — Sample tiles across the density range

Sort tiles by point count and pick evenly spaced tiles, plus the densest few.

# Step 2 — Measure each run

Record wall time, CPU seconds, peak RSS, and bytes in and out. /usr/bin/time -v or Python’s resource.getrusage gives CPU time and peak memory.

# Step 3 — Fit time against points

A linear fit of seconds against millions of points is usually adequate; check the residuals on the densest tiles.

# Step 4 — Convert to instance cost

From peak memory and vCPUs per child, work out how many children fit per instance, then cost per tile = instance price per second × seconds ÷ children per instance.

# Step 5 — Add storage and requests, then project

Add S3 request and output storage costs per tile, sum over the tile index using each tile’s point count, and add a retry and idle margin.

# Complete Working Example

python
"""Measure sample tiles, fit a time model, and project the cost of a full run."""
import json
import resource
import subprocess
import time

import numpy as np

SAMPLE = ["tiles/571_4190.laz", "tiles/580_4201.laz", "tiles/598_4222.laz"]  # stratified
PIPE = "dtm.json"   # the production pipeline, reading from FILENAME


def measure(tile):
    t0 = time.time()
    r0 = resource.getrusage(resource.RUSAGE_CHILDREN)
    subprocess.run(["pdal", "pipeline", PIPE, f"--readers.las.filename=/vsis3/lidar-in/{tile}",
                    "--writers.gdal.filename=/tmp/out.tif"], check=True)
    r1 = resource.getrusage(resource.RUSAGE_CHILDREN)
    info = json.loads(subprocess.run(["pdal", "info", "--summary", f"/vsis3/lidar-in/{tile}"],
                                     capture_output=True, text=True, check=True).stdout)
    return {"tile": tile, "mpts": info["summary"]["num_points"] / 1e6,
            "wall_s": time.time() - t0,
            "cpu_s": (r1.ru_utime + r1.ru_stime) - (r0.ru_utime + r0.ru_stime),
            "peak_gb": r1.ru_maxrss / 1024**2}          # ru_maxrss is KiB on Linux


rows = [measure(t) for t in SAMPLE]
mpts = np.array([r["mpts"] for r in rows])
wall = np.array([r["wall_s"] for r in rows])
slope, intercept = np.polyfit(mpts, wall, 1)
peak = max(r["peak_gb"] for r in rows)
print(f"seconds ≈ {slope:.1f} × Mpts + {intercept:.1f};  peak {peak:.1f} GB")

# ---- projection (placeholder prices: replace with your region's current rates) ----
INSTANCE_PER_HOUR = 0.34      # e.g. an 8 vCPU / 32 GB instance, spot or on-demand
VCPU, MEM_GB = 8, 32
children_per_instance = min(VCPU // 1, int(MEM_GB // (peak * 1.25)))
PUT_PER_1000, GET_PER_1000 = 0.005, 0.0004
STORAGE_GB_MONTH = 0.023

index = json.load(open("tile_index_counts.json"))       # {"tile": point_count, ...}
total_s = sum(slope * (n / 1e6) + intercept for n in index.values())
compute = total_s / 3600 / children_per_instance * INSTANCE_PER_HOUR
requests = len(index) * (20 / 1000 * GET_PER_1000 + 2 / 1000 * PUT_PER_1000)
storage = len(index) * 0.05 * STORAGE_GB_MONTH           # ~50 MB output per tile
total = (compute + requests + storage) * 1.15             # retries, idle, stragglers
print(f"{len(index)} tiles: compute ${compute:,.0f}, requests ${requests:,.2f}, "
      f"storage ${storage:,.2f}/month -> estimate ${total:,.0f}")

ru_maxrss from RUSAGE_CHILDREN reports the largest child so far, which is what matters for sizing when you measure one tile per call. On macOS it is in bytes rather than KiB.

Fitting seconds against points A scatter plot of wall-clock seconds against millions of points for about a dozen sample tiles, with a fitted straight line. Points lie close to the line at low and medium density and slightly above it for the densest tiles, showing mild super-linear growth from the ground filter. million points per tile sec dense tiles run slightly above the line

# Where Estimates Go Wrong

Three things commonly push real costs above the model. First, stragglers: compute environments scale in instances, and the last hour of a run is often a few long tiles keeping otherwise idle instances alive; sorting the manifest so dense tiles start first, and allowing instances to scale down, trims this. Second, retries: spot reclamation repeats work, and a 5–10% margin is realistic on busy instance families. Third, hidden I/O: reading LAZ over /vsis3/ with a small block size multiplies GET requests, and writing temporary files to EBS adds volume costs. Configure GDAL’s S3 options as in configuring GDAL /vsis3/ for fast point cloud reads.

Things that make costs lower than feared are also worth checking: in-region transfer between S3 and EC2 is free, so moving terabytes of LAZ to compute in the same region does not add transfer charges; and when only a few tiles changed, processing only those — possible when outputs are idempotent and keyed by tile — reduces the run to a fraction of the full cost.

# Key Parameter Table

Input Source Notes
Seconds per Mpts Sample fit Pipeline- and instance-specific
Peak memory per tile ru_maxrss Sets children per instance
Instance price AWS pricing Spot often well below on-demand
Children per instance min(vCPU, memory ÷ peak) With ~25% memory headroom
Requests per tile S3 access logs or estimate Depends on block size
Output GB per tile Measured Storage per month
Margin 10–20% Retries, stragglers, idle

# Verification

  • Holdout tiles. Predict the runtime of five tiles not in the fit and compare with measured runtimes; a consistent bias means the model needs another term.
  • Small real run. Run 1% of the project and compare actual cost from Cost Explorer or tagged billing with the projection for that subset.
  • Tag resources. Tag the compute environment and bucket by project so actual spend can be read back per run.

# Gotchas and Edge Cases

Area is not a proxy. Two 1 km tiles can differ tenfold in points. Always model on point counts.

Instance choice changes the slope. A newer CPU generation can cut seconds per million points noticeably; re-measure rather than scaling old results.

Fargate prices differently. Fargate charges per vCPU and GB of memory per second requested, so the memory headroom you declare is billed whether used or not.

The straggler tail A curve of running instances over time rises quickly, stays flat through most of the run, then drops to a long thin tail where a few dense tiles keep several instances busy. Starting dense tiles first shortens this tail. full fleet tail: few dense tiles time → (sort dense tiles first to shrink the tail)

# Frequently Asked Questions

What drives the cost of processing a LiDAR tile in the cloud?

Compute time, which grows with the number of points in the tile and the filters applied. S3 requests and storage are usually small in comparison, and transfer within one region is free.

How many tiles should I measure to build a cost model?

Twenty to thirty tiles chosen across the full range of point counts, including the densest few, is usually enough to fit runtime against points and check the fit on holdout tiles.

Why do real runs cost more than the estimate?

Retries after spot interruptions, straggling dense tiles that keep instances alive at the end, and extra S3 requests from small read block sizes. A 10 to 20 percent margin covers most of this.

Does tile area predict processing cost?

Poorly. Point density varies greatly between tiles because of land cover and flightline overlap, so model cost on point counts from the tile index rather than on area.

Related in AWS Batch Processing for PDAL Point Clouds

↑ AWS Batch Processing for PDAL Point Clouds overview --- title: "AWS Batch Processing for PDAL Point Clouds" description: "Running thousands of PDAL tile jobs on AWS Batch — job definitions,... Scaling PDAL Tile Processing with AWS Batch --- title: "Scaling PDAL Tile Processing with AWS Batch" description: "A worked example of fanning out a PDAL ground-to-DTM pipeline across... Array Jobs for LiDAR Tiles in AWS Batch --- title: "Array Jobs for LiDAR Tiles in AWS Batch" description: "Process thousands of LiDAR tiles with one AWS Batch array job: write a... Handling Spot Interruptions in PDAL Batch Jobs --- title: "Handling Spot Interruptions in PDAL Batch Jobs" description: "Decompose the tile, write the marker after the object, trap...