Configuring GDAL /vsis3/ for Fast Point Cloud Reads

TL;DR: Set GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR before anything else — on a fan-out that opens thousands of objects it is usually worth a factor of three on its own — then VSI_CACHE=TRUE, a curl chunk size of 1 MB, and a cache large enough to hold the header and index of the object you are reading.

# Context and Motivation

This guide is part of S3 and Cloud Storage I/O. The pipeline is rarely the problem when a cloud read is slow. GDAL’s virtual filesystem sits between PDAL and the object store, and its defaults are tuned for a desktop user opening one raster, not for a worker opening ten thousand LAZ objects.

The most expensive default is the one nobody expects. When GDAL opens /vsis3/bucket/tiles/tile_0431.laz, it lists the containing prefix first, because for many formats a dataset is several files — a shapefile’s siblings, a world file, an auxiliary XML. For a LAZ tile there are no siblings, and on a prefix holding twenty thousand objects that listing is paginated, slow and repeated for every single open. Disabling it changes nothing about correctness for self-contained formats and removes an entire class of latency.

# Prerequisites and Assumptions

That last row matters more than it looks. GDAL reads most of these settings once, when the driver initialises, so exporting them from inside Python after import pdal is too late.

# Step-by-Step Implementation

# Step 1 — Disable the directory listing

export GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR

EMPTY_DIR rather than YES: it disables the listing while still allowing an explicit sibling open if something needs one.

# Step 2 — Turn on the read cache

export VSI_CACHE=TRUE
export VSI_CACHE_SIZE=25000000        # per-file, bytes
export CPL_VSIL_CURL_CACHE_SIZE=200000000   # process-wide, bytes

The per-file cache is what stops a COPC reader re-fetching the header and index for every query.

# Step 3 — Size the range requests

export CPL_VSIL_CURL_CHUNK_SIZE=1048576

One megabyte is a good default: small enough not to over-fetch, large enough that per-request latency does not dominate.

# Step 4 — Set the region and any bucket policy flags

export AWS_DEFAULT_REGION=us-east-1
export AWS_REQUEST_PAYER=requester    # only for requester-pays buckets

# Step 5 — Put them in the container definition

Environment, not code. The same settings then apply to pdal, gdalinfo and anything else in the image.

# Complete Working Example

"""Read a LAZ object from S3 with the virtual filesystem tuned, and time it."""
from __future__ import annotations

import json
import os
import time

# Must be set before the GDAL driver initialises, which happens on import.
os.environ.setdefault("GDAL_DISABLE_READDIR_ON_OPEN", "EMPTY_DIR")
os.environ.setdefault("VSI_CACHE", "TRUE")
os.environ.setdefault("VSI_CACHE_SIZE", "25000000")
os.environ.setdefault("CPL_VSIL_CURL_CACHE_SIZE", "200000000")
os.environ.setdefault("CPL_VSIL_CURL_CHUNK_SIZE", "1048576")

import pdal  # noqa: E402


def timed_read(url: str, bounds: str | None = None) -> dict:
    reader: dict = {"type": "readers.las", "filename": url}
    if bounds:
        reader["bounds"] = bounds
    started = time.perf_counter()
    pipeline = pdal.Pipeline(json.dumps({"pipeline": [reader]}))
    n = pipeline.execute()
    return {
        "url": url,
        "points": n,
        "seconds": round(time.perf_counter() - started, 2),
    }


def settings_in_effect() -> dict:
    keys = ["GDAL_DISABLE_READDIR_ON_OPEN", "VSI_CACHE", "VSI_CACHE_SIZE",
            "CPL_VSIL_CURL_CACHE_SIZE", "CPL_VSIL_CURL_CHUNK_SIZE",
            "AWS_DEFAULT_REGION"]
    return {k: os.environ.get(k, "<unset>") for k in keys}


if __name__ == "__main__":
    print(json.dumps(settings_in_effect(), indent=2))
    result = timed_read("/vsis3/lidar-archive/tiles/tile_0431.laz")
    print(json.dumps(result, indent=2))
    # A second read of the same object should be much faster if the cache is on.
    again = timed_read("/vsis3/lidar-archive/tiles/tile_0431.laz")
    print(json.dumps({"first": result["seconds"], "second": again["seconds"]}, indent=2))

# Key Parameter Table

# Verification

The second read is much faster. The example prints both. If they are the same, VSI_CACHE is not in effect — usually because it was set after import pdal.

Request counts fall. Check the bucket access log or a proxy. Opens should show ranged GETs and no LIST.

Nothing broke. The settings change performance, not semantics, for self-contained formats. Compare a point count against a local copy.

# Gotchas and Edge Cases

Setting the variables after importing PDAL. GDAL reads most of them at driver initialisation. Set them in the container environment, or before the import if you must do it in Python.

EMPTY_DIR with a sidecar format. A shapefile or a raster with an auxiliary XML genuinely needs the listing. Scope the setting to the pipelines that read self-contained formats.

Cross-region reads. No environment tuning recovers the latency or the egress cost. Colocate the worker with the bucket.

A cache larger than the container’s memory limit. CPL_VSIL_CURL_CACHE_SIZE is real memory. On a worker with a 2 GB limit running eight processes, a 200 MB cache each is most of the budget.

Credentials resolved differently than you expect. GDAL has its own resolution order and does not always agree with boto3. If a read fails with access denied while boto3 succeeds, that is why — and the Docker containers guide covers pinning the whole environment so it stops varying.

# Frequently Asked Questions

Why is opening a small object from S3 so slow?

Because GDAL lists the containing prefix first, in case the dataset has sibling files. For a self-contained format such as LAZ there are none, and on a prefix holding twenty thousand objects that listing is paginated and repeated for every open. Setting GDAL_DISABLE_READDIR_ON_OPEN to EMPTY_DIR removes it.

Why did setting the variables in Python change nothing?

GDAL reads most of them once, when the driver initialises, which happens as PDAL is imported. Set them in the container environment, or at the very top of the module before the import, and the same settings then apply to every tool in the image.

How large should the caches be?

The per-file cache needs to hold whatever gets re-read — for COPC that is the header and the octree index, so around twenty-five megabytes is comfortable. The process-wide cache is real memory and must fit the container limit divided by the number of worker processes.

Is EMPTY_DIR ever wrong?

Yes, for formats with sidecar files. A shapefile needs its siblings and a raster may need an auxiliary XML, and disabling the listing hides them. Scope the setting to the pipelines that read self-contained formats such as LAZ, COPC and GeoTIFF.

# Related

• S3 and Cloud Storage I/O — the parent guide to reading and writing through the virtual filesystem
• Streaming LAZ from S3 with PDAL — the read pattern these settings make fast
• Writing Cloud Optimized GeoTIFFs to S3 — the same filesystem on the write path
• Querying a COPC File by Bounds and Resolution — the query whose latency these settings dominate
• PDAL Docker Containers — where the environment belongs so it stops varying