Writing Cloud-Optimized GeoTIFFs to S3

TL;DR: Rasterise ground points with writers.gdal using gdaldriver: "COG" and creation options (COMPRESS=DEFLATE, BLOCKSIZE=512, OVERVIEWS=AUTO), write to local scratch, then push the finished file to s3://survey-deliverables/cog/tile.tif with a boto3 upload_file — validate with rio cogeo validate before and gdalinfo /vsis3/... after.

# Context and Motivation

This guide is part of S3 Cloud Storage I/O with PDAL, which covers the full range of reading and writing point-cloud data against Amazon S3.

A terrain raster that lives in a bucket is only useful to downstream web maps and tiling services if it is a Cloud-Optimized GeoTIFF: internally tiled, carrying overviews, and laid out so a client can pull a small window with an HTTP range request instead of downloading the whole file. PDAL’s writers.gdal stage rasterises a classified point cloud into exactly this kind of surface, and GDAL’s COG driver structures the bytes correctly. The remaining question — the one this guide answers — is how the finished raster gets into the bucket: a direct /vsis3/ write from the writer, or a two-step “write local, upload with boto3” that trades a little code for atomicity and resumability. The upstream rasterisation is the same one covered in Generating a DTM GeoTIFF with writers.gdal; here the focus is COG structure and the S3 handoff.

# Prerequisites and Assumptions

The point cloud should already carry ground labels. If it does not, run filters.smrf first as shown below; the ground-classification background lives in the broader terrain-model material referenced under DTM Raster Generation.

# Step-by-Step Implementation

# Step 1 — Rasterise a DTM with the COG driver

The COG driver builds tiling and overviews in a single pass, so the writer output is a valid Cloud-Optimized GeoTIFF with no separate overview step:

{
  "type": "writers.gdal",
  "filename": "/tmp/dtm_tile_0421.tif",
  "resolution": 0.5,
  "output_type": "idw",
  "data_type": "float32",
  "nodata": -9999,
  "gdaldriver": "COG",
  "gdalopts": "COMPRESS=DEFLATE,BLOCKSIZE=512,OVERVIEWS=AUTO,RESAMPLING=BILINEAR"
}

output_type: "idw" interpolates across small gaps between ground points; BLOCKSIZE=512 sets the internal tile size the COG driver uses, and OVERVIEWS=AUTO lets GDAL pick the overview levels.

# Step 2 — Choose direct write or local-plus-upload

For a direct write, swap the local filename for a /vsis3/ path — GDAL buffers the raster and issues one PutObject on close:

"filename": "/vsis3/survey-deliverables/cog/dtm_tile_0421.tif"

For the more robust two-step path, keep the local filename, validate the file, then upload it. This is preferred for large rasters and unattended batch jobs because a mid-run crash never leaves a half-written object, and boto3’s multipart transfer resumes cleanly.

# Step 3 — Upload with boto3 (two-step path)

import boto3
from boto3.s3.transfer import TransferConfig

def upload_cog(local_path: str, bucket: str, key: str) -> None:
    """Multipart-upload a finished COG with the correct content type."""
    cfg = TransferConfig(multipart_threshold=8 * 1024 * 1024)  # 8 MB parts
    boto3.client("s3").upload_file(
        local_path, bucket, key,
        ExtraArgs={"ContentType": "image/tiff", "ServerSideEncryption": "AES256"},
        Config=cfg,
    )

# Step 4 — Validate the COG structure

Confirm the file is genuinely cloud-optimized before or after upload:

rio cogeo validate /tmp/dtm_tile_0421.tif
# or, structurally:
gdalinfo /tmp/dtm_tile_0421.tif | grep -iE "block|overview|layout"

A valid COG reports an internally tiled layout with overviews present.

# Complete Working Example

Save as dtm_cog_to_s3.py. It classifies ground, rasterises a COG DTM to scratch, validates it, and multipart-uploads it to S3.

#!/usr/bin/env python3
"""
dtm_cog_to_s3.py
Rasterise a Cloud-Optimized GeoTIFF DTM with PDAL and upload it to S3.

Usage:
    python dtm_cog_to_s3.py input.laz survey-deliverables cog/dtm_tile_0421.tif

Requirements:
    conda install -c conda-forge pdal python-pdal   # GDAL 3.1+ for the COG driver
    pip install boto3 rio-cogeo
"""

import os
import sys
import json
import subprocess
import tempfile

import pdal
import boto3
from boto3.s3.transfer import TransferConfig


def build_cog_dtm(src: str, dst_local: str, resolution: float = 0.5) -> pdal.Pipeline:
    """Classify ground, keep it, and rasterise a COG DTM to a local path."""
    stages = [
        {"type": "readers.las", "filename": src},
        {"type": "filters.smrf", "slope": 0.2, "window": 16.0, "threshold": 0.45},
        {"type": "filters.range", "limits": "Classification[2:2]"},
        {
            "type": "writers.gdal",
            "filename": dst_local,
            "resolution": resolution,
            "output_type": "idw",
            "data_type": "float32",
            "nodata": -9999,
            "gdaldriver": "COG",
            "gdalopts": "COMPRESS=DEFLATE,BLOCKSIZE=512,OVERVIEWS=AUTO,RESAMPLING=BILINEAR",
        },
    ]
    return pdal.Pipeline(json.dumps({"pipeline": stages}))


def validate_cog(path: str) -> None:
    """Raise if the file is not a valid Cloud-Optimized GeoTIFF."""
    result = subprocess.run(
        ["rio", "cogeo", "validate", path],
        capture_output=True, text=True,
    )
    if "is a valid cloud optimized GeoTIFF" not in result.stdout:
        raise RuntimeError(f"COG validation failed:\n{result.stdout}\n{result.stderr}")
    print(result.stdout.strip())


def upload_cog(local_path: str, bucket: str, key: str) -> None:
    cfg = TransferConfig(multipart_threshold=8 * 1024 * 1024)
    boto3.client("s3").upload_file(
        local_path, bucket, key,
        ExtraArgs={"ContentType": "image/tiff", "ServerSideEncryption": "AES256"},
        Config=cfg,
    )
    print(f"Uploaded s3://{bucket}/{key}")


def confirm_remote(bucket: str, key: str) -> None:
    head = boto3.client("s3").head_object(Bucket=bucket, Key=key)
    print(f"Confirmed s3://{bucket}/{key}: {head['ContentLength']:,} bytes")


def main() -> None:
    if len(sys.argv) != 4:
        print("Usage: python dtm_cog_to_s3.py <input.laz> <bucket> <key>")
        sys.exit(1)

    src, bucket, key = sys.argv[1], sys.argv[2], sys.argv[3]
    os.environ.setdefault("AWS_REGION", "us-west-2")

    with tempfile.TemporaryDirectory() as tmp:
        local_cog = os.path.join(tmp, "dtm.tif")
        pipeline = build_cog_dtm(src, local_cog)
        pipeline.validate()
        n = pipeline.execute()
        print(f"Rasterised {n:,} ground points into {local_cog}")

        validate_cog(local_cog)
        upload_cog(local_cog, bucket, key)
        confirm_remote(bucket, key)


if __name__ == "__main__":
    main()

# Key Parameter Table

For the plain GTiff driver, the equivalent block option is BLOCKXSIZE/BLOCKYSIZE with TILED=YES, and you must add overviews separately with gdaladdo before the file counts as a COG.

# Verification

Validate the COG locally with rio cogeo validate, which checks tiling, overview presence, and IFD ordering:

rio cogeo validate /tmp/dtm.tif
# -> ".../dtm.tif is a valid cloud optimized GeoTIFF"

Confirm the object landed with a head_object and check the size is plausible for the raster dimensions:

import boto3
head = boto3.client("s3").head_object(Bucket="survey-deliverables", Key="cog/dtm_tile_0421.tif")
assert head["ContentLength"] > 0
print(head["ContentLength"], head["ETag"])

Read it back through /vsis3/ to prove the uploaded COG opens remotely and reports the tiled structure — the ultimate proof that a client can range-read it:

AWS_REGION=us-west-2 gdalinfo /vsis3/survey-deliverables/cog/dtm_tile_0421.tif | grep -iE "block|overview|epsg"

# Gotchas and Edge Cases

1. COG driver versus GTiff + manual overviews. The dedicated COG driver is the reliable choice: it emits tiling, overviews, and the metadata-before-data layout in one write. If you must use GTiff (for a driver-specific option the COG driver does not expose), you have to set TILED=YES and run gdaladdo -r bilinear file.tif 2 4 8 16 afterward, then re-validate. Skipping the overview step yields a tiled GeoTIFF that fails COG validation.

2. Direct /vsis3/ writes are not resumable. A direct write holds the whole raster in memory and PUTs it once on close. That is atomic from a reader’s perspective — no partial object appears — but if the process dies mid-run you lose the compute and get no object at all. For large tiles, the local-then-upload path lets boto3 resume a failed multipart transfer and lets you validate the file before it ever reaches the bucket.

3. Overviews inflate small tiles. For tiny rasters, overviews and internal tiling can add more overhead than they save. The COG structure pays off for rasters large enough that a client benefits from range-reading a window; for a 200×200 pixel tile the benefit is marginal. Consider your consumer’s access pattern before mandating COG on every output.

4. Content type and encryption on upload. boto3 does not infer ContentType from the extension, so without ExtraArgs the object may be served as binary/octet-stream, which some tile viewers reject. Set ContentType="image/tiff", and if the bucket policy denies unencrypted PUTs, pass ServerSideEncryption explicitly — a direct /vsis3/ write cannot easily attach these headers, which is another reason to prefer the boto3 path for governed buckets.

# Frequently Asked Questions

Should I use the GDAL COG driver or GTiff with COG options in writers.gdal?

Prefer the dedicated COG driver (gdaldriver: "COG"). It builds internal tiling, overviews, and the correct IFD ordering in one pass, so the output is a valid Cloud-Optimized GeoTIFF by construction. GTiff with TILED=YES plus a manual gdaladdo can produce an equivalent file but requires the extra overview step and careful option matching.

Is a direct /vsis3/ write atomic?

Effectively yes for the object itself — GDAL buffers the raster and issues the PutObject only on close, so readers never see a partial file. But a crash mid-run leaves no object at all and wastes the compute. Writing to local scratch then uploading with boto3 gives you a verifiable file plus multipart resumability.

Do Cloud-Optimized GeoTIFFs need internal overviews?

Yes. Overviews let a client fetch a low-resolution preview with a small range request instead of downloading the full raster. The COG driver generates them automatically; with the plain GTiff driver you must add them with gdaladdo or the OVERVIEWS creation option before the file qualifies as a COG.

How do I validate that my GeoTIFF is actually a valid COG?

Run rio cogeo validate on the file, or inspect it with gdalinfo and confirm it reports a tiled layout with overviews and the COG structure. A valid COG has internal tiles, overviews ordered for range reads, and the image data positioned after the metadata.

# Related

• S3 Cloud Storage I/O with PDAL — parent guide to reading and writing point clouds against S3
• Streaming LAZ from S3 with PDAL — the read-side counterpart to this write guide
• Generating a DTM GeoTIFF with writers.gdal — the upstream rasterisation this guide sends to the cloud
• DTM Raster Generation — surface-model interpolation options and gap handling
• Batch Automation and Cloud Integration for PDAL — the wider cloud automation picture