Dask vs ProcessPoolExecutor for PDAL

TL;DR: Stay with concurrent.futures.ProcessPoolExecutor while a batch finishes in acceptable time on one machine and you can live with hand-written retries and progress logging. Move to Dask distributed when you need more than one machine, automatic memory-based worker restarts, retries, a live dashboard, or elastic cloud scaling. Write the per-tile function so it works unchanged with both, and the switch is a few lines.

# Context and Motivation

This guide is part of Dask Distributed Processing for LiDAR. Teams often adopt a distributed framework before they need one, and pay for it in operational complexity; others stretch a single machine far past sensible limits. The decision is not about PDAL — both run PDAL identically, one process per tile — but about scale, failure handling and visibility. Framing it as a few concrete questions makes the answer clear for most projects.

# Prerequisites and Assumptions

• A per-tile function that takes a path and returns a small summary, with outputs written to storage.
• An estimate of total work: tile count × median time per tile ÷ cores available.
• An honest view of how often the batch will run and who will operate it.

# Step-by-Step Implementation

# Step 1 — Estimate single-machine wall time

Multiply tile count by median tile time and divide by the number of workers that fit in memory. If that is overnight or less, one machine may be enough.

# Step 2 — List the operational needs

Retries on transient errors, restarts when a tile blows memory, progress visibility, per-task timing — note which you need.

# Step 3 — Price the alternatives

A process pool costs nothing to operate. Dask costs a scheduler, a shared image and network configuration — small, but not zero.

# Step 4 — Write executor-agnostic code

Keep the per-tile function pure (path in, summary out) and pass the executor in, so the same driver works with both.

# Step 5 — Migrate when a threshold is crossed

Switch when a batch no longer fits a working day on one machine, or when the hand-written retry and memory logic starts to grow.

# Complete Working Example

A driver that runs the same task with either executor:

"""One driver, two executors: ProcessPoolExecutor locally, Dask when scaled out."""
from __future__ import annotations

import os
from concurrent.futures import ProcessPoolExecutor, as_completed as cf_as_completed
from pathlib import Path

from lidar_tasks import process_tile          # path in, summary dict out


def run_local(keys: list[str], workers: int | None = None) -> list[dict]:
    results = []
    with ProcessPoolExecutor(max_workers=workers or os.cpu_count()) as pool:
        futs = {pool.submit(process_tile, k): k for k in keys}
        for f in cf_as_completed(futs):
            try:
                results.append(f.result())
            except Exception as exc:              # noqa: BLE001 — record and continue
                results.append({"tile": Path(futs[f]).stem, "error": repr(exc)})
    return results


def run_dask(keys: list[str], address: str) -> list[dict]:
    from dask.distributed import Client, as_completed
    client = Client(address)
    futs = client.map(process_tile, keys, retries=2, pure=False)
    results = []
    for f in as_completed(futs):
        results.append({"tile": f.key, "error": repr(f.exception())} if f.status == "error"
                       else f.result())
    client.close()
    return results


if __name__ == "__main__":
    keys = [k.strip() for k in Path("tile_list.txt").read_text().splitlines() if k.strip()]
    address = os.environ.get("DASK_SCHEDULER_ADDRESS")
    out = run_dask(keys, address) if address else run_local(keys)
    print(f"{sum('error' not in r for r in out)} ok, {sum('error' in r for r in out)} failed")

The process-pool branch records failures but does not retry or restart on memory exhaustion — a worker killed by the kernel raises BrokenProcessPool and ends the pool. Dask handles both.

# A Middle Path

There is a useful halfway house between the two: Dask’s LocalCluster on a single machine. It gives the operational features — memory limits with restarts, retries, the dashboard — without any multi-host setup, and the same code later points at a remote scheduler unchanged. For teams that expect to grow, starting with LocalCluster instead of a raw process pool costs one extra dependency and avoids rewriting the driver later. For teams that do not, the standard library remains the simplest thing that works.

Whichever you choose, keep the per-tile function free of executor-specific code: no Dask imports inside it, no assumptions about which process it runs in, and all outputs written to storage by the function itself. That discipline is what makes the executor a configuration choice rather than an architecture.

# Key Parameter Table

# Verification

• Same results. Run twenty tiles with both executors and compare outputs; they must be identical, because the task is the same.
• Failure behaviour. Feed one tile that exceeds memory. The process pool should fail the batch or the task visibly; Dask should restart the worker and eventually mark the task failed after retries.
• Throughput. On one machine, throughput should be nearly identical; Dask’s overhead per multi-minute task is negligible.

# Gotchas and Edge Cases

BrokenProcessPool. When the kernel kills a pool worker, every pending future in that pool fails. Resubmitting the rest requires a new pool — code you now own. This is often the moment teams move to Dask.

Memory contention on one machine. Neither executor prevents too many large tiles running at once, unless you add a semaphore (pool) or resources (Dask).

Pickling. Both pickle the function and its arguments. Pass paths, not open file handles or pipeline objects.

Premature distribution. A Dask cluster for a 200-tile batch that finishes in an hour on a laptop adds failure modes without adding value. Start simple.

# Frequently Asked Questions

Is Dask faster than ProcessPoolExecutor on one machine?

Not meaningfully for PDAL tile work. Both run one process per tile, and per-task overhead is negligible compared with minutes of processing. Dask’s advantages are scaling out and operational features, not single-machine speed.

What happens when a PDAL worker runs out of memory?

In a process pool, the kernel kills the worker and the pool breaks, failing pending tasks. In Dask, the worker’s nanny restarts it, and the task is retried or marked failed while other tasks continue.

Can I switch from a process pool to Dask later?

Yes, if the per-tile function takes a path and returns a small summary. The driver changes from pool.submit to client.map; the task code stays the same.

Do I need Dask for Airflow or AWS Batch?

No. Those systems distribute tiles themselves, one task or container per tile. Dask is an alternative orchestration layer, most useful when you want Python-native control and interactive visibility.

# Related

• Dask Distributed Processing for LiDAR — Dask for PDAL in depth
• Parallel Tile Processing with ProcessPoolExecutor — the single-machine baseline
• Threads vs Processes for PDAL Workloads — why processes
• Processing LiDAR Tiles with Dask Distributed — first multi-host deployment
• Scaling PDAL Tile Processing with AWS Batch — the managed-queue alternative