Serverless Geospatial Pipeline Recipes

The three neighbouring sections of this reference each isolate one hard problem: what a serverless runtime will and will not let you do, how to package heavy native geospatial dependencies, and how to wire event-driven triggers between functions. This section stitches those constraints into complete, runnable pipelines. Every recipe here is a full path from an ingestion event to a cataloged product — a 10 GB GeoTIFF reprojected and tiled, a live AIS vessel feed aggregated into spatial windows, Sentinel-2 scenes turned into NDVI tiles, or a vector dataset rendered to Mapbox Vector Tiles — assembled so that no single function invocation ever breaches a platform ceiling. The goal is not another survey of limits but a set of blueprints you can lift, parameterise, and deploy.

Foundational Architecture Patterns

Every recipe on this site is a variation on the same five-stage skeleton. An event announces new data, a planner turns that data into a manifest of bounded work items, an orchestrator fans those items out across many short-lived function invocations, a merge stage reassembles the partial outputs, and a catalog stage makes the result discoverable. The diagram below traces that reference pipeline end to end.

1. Ingestion Event — An object-storage notification fires when a new scene lands. The S3 and GCS event triggers for Shapefiles patterns generalise to any raster or vector payload; the trigger function stays tiny and does no pixel work.
2. Metadata + Planner — A lightweight function reads the file header, derives dimensions, block size, and CRS, and computes a manifest of bounded work items. This is where the raster is cut into tiles that individually fit the compute stage’s memory and timeout.
3. Orchestrator — Step Functions (AWS), Cloud Workflows (GCP), or Durable Functions (Azure) own execution state, retries, and parallel fan-out. The orchestrator never touches pixels; it dispatches manifest items and collects results.
4. Tiled Compute Fan-Out — One short-lived function invocation per tile, each reading only its window through chunked I/O for large satellite imagery. Failures are isolated to a single tile and routed to a dead-letter queue for failed vector jobs rather than aborting the run.
5. Merge + Catalog — A final function reassembles the tiled outputs into a Cloud Optimized GeoTIFF or tile pyramid, builds overviews, and registers a STAC item so downstream consumers can discover the product.

The flagship worked example of this skeleton is the process a 10 GB GeoTIFF with Step Functions and Lambda recipe, which carries a full 10 GB scene through all five stages while keeping every invocation inside Lambda’s ceilings. Its two focused companions cover the two stages that break most often: partitioning a GeoTIFF into Step Functions Map-state tiles and merging tiled Lambda outputs into a COG.

The other three recipes bend the same skeleton toward different data shapes. The real-time AIS vessel tracking pipeline replaces the batch planner with a streaming windowed aggregation; the serverless NDVI tiling from Sentinel-2 recipe fans multi-band raster algebra out per tile; and the vector tile pipeline with Cloud Run and Pub/Sub swaps Lambda for containers when Tippecanoe’s memory profile exceeds function limits.

Platform Constraints Reference Table

A recipe is only correct if every stage respects the ceilings of the provider you deploy it on. These are the quotas that decide how finely the planner must tile and whether an orchestrator step can run at all.

The single most consequential figure for these recipes is the memory-backed nature of GCP’s /tmp: writing a 6 GB intermediate VRT there consumes 6 GB of your memory allocation, whereas Lambda’s /tmp is separate disk-backed storage up to 10 GB. That difference decides whether a merge stage can run in one invocation or must itself be tiled. The full breakdown of these boundaries lives in Serverless Geospatial Architecture & Platform Limits, and the per-tile packaging of GDAL that every compute stage depends on is covered in Packaging & Dependency Management for Serverless GIS.

Which Orchestrator Fits Which Recipe

The orchestrator choice is rarely about preference — it follows from which limit the workload hits first and where the source data lives. This table maps each recipe to the provider and orchestrator that fit it without a container fallback.

Cloud Workflows and Durable Functions become the natural choice the moment a stage needs more than Lambda’s 15 minutes or 10 GB — for example a whole-country vector build or a 32 GB in-memory raster mosaic. When even those ceilings are not enough, every recipe degrades to a container fallback (Fargate or Cloud Run) rather than failing.

Runtime Optimization for Geospatial Libraries

Recipe compute stages share one performance profile: a large native dependency graph imported cold, then a burst of CPU-bound array work. Two levers dominate.

First, cold-start amortisation. Because a Map state can launch hundreds of tile workers within seconds, each paying the Python GDAL cold-start cost, the marginal saving from a smaller, pre-warmed layer is multiplied across the whole fan-out. For latency-sensitive recipes, provisioned concurrency on the tile-worker function pays back quickly. Always pin the runtime paths explicitly so GDAL never falls back to nonexistent compile-time defaults:

import os

# Set in the tile-worker function configuration, applied before importing rasterio.
os.environ.setdefault("GDAL_DATA", "/opt/share/gdal")
os.environ.setdefault("PROJ_LIB", "/opt/share/proj")
os.environ.setdefault("LD_LIBRARY_PATH", "/opt/lib:" + os.environ.get("LD_LIBRARY_PATH", ""))
# COG-friendly GDAL tuning: only fetch the bytes a window needs.
os.environ.setdefault("GDAL_DISABLE_READDIR_ON_OPEN", "EMPTY_DIR")
os.environ.setdefault("CPL_VSIL_CURL_ALLOWED_EXTENSIONS", ".tif,.tiff")
os.environ.setdefault("VSI_CACHE", "TRUE")

Second, memory-proportional CPU. On AWS and GCP, vCPU scales linearly with the memory you allocate, so a tile worker doing raster algebra is often faster and cheaper at a higher memory tier even when it never uses the extra RAM — the extra cores finish the NumPy work sooner. Size the tier against the memory and CPU allocation model for raster workloads rather than guessing, and keep intermediate arrays out of /tmp where ephemeral storage limits apply.

Security, IAM, and Data Governance

A fan-out pipeline multiplies the blast radius of an over-broad role: one leaked worker credential with s3:* exposes every scene in the bucket. Each stage should carry only the permissions it exercises. The planner needs s3:GetObject on the ingestion prefix to read headers; the tile workers need s3:GetObject on the source and s3:PutObject on a per-run output prefix; the merge stage needs read on that output prefix and write on the catalog prefix. The IAM security boundaries for Cloud GIS discipline covers per-stage scoping, and its least-privilege policies for Azure Blob geospatial access companion translates the same boundaries to Azure Managed Identities.

Additional governance controls that matter at recipe scale:

• Per-run output prefixes — Write each pipeline run under a deterministic prefix derived from the input URI and parameters. This isolates partial outputs, makes cleanup atomic, and prevents one run from overwriting another’s tiles.
• VPC endpoints — Route the high-volume tile reads and writes over S3/Storage private endpoints. At thousands of tiles per run, public-path NAT egress cost is significant and avoidable.
• KMS / CMEK at rest — Encrypt source and product buckets with customer-managed keys scoped to the pipeline roles, so a compromised worker role still cannot decrypt data outside its stage.
• Signed manifests — When the planner writes a tile manifest to S3, treat it as trusted input to the orchestrator; validate its schema in the first worker step so a corrupted manifest cannot drive workers to arbitrary URIs.

Observability, Cost Control, and Fallback Patterns

The dominant cost in these recipes is GB-seconds across the fan-out, so per-tile instrumentation is where cost control lives. Emit a structured record from every worker with the tile index, bytes read, peak memory, and duration, and aggregate it at the orchestrator boundary.

When a tile consistently trips the memory or timeout alert — a denser region, a larger block, an extra band — the correct response is not a bigger tile-worker tier but a finer manifest. The planner re-tiles the offending window into sub-tiles on retry. Only when a single indivisible unit of work genuinely exceeds the function ceiling does the circuit breaker route it to a container:

import boto3, os

sfn = boto3.client("stepfunctions")
ecs = boto3.client("ecs")

def dispatch_tile(tile, ctx):
    """Run a tile on Lambda; fall back to Fargate only for oversized indivisible tiles."""
    if tile["est_pixels"] > ctx["lambda_pixel_ceiling"]:
        # A single tile too large to subdivide further — send to a container task.
        return ecs.run_task(
            cluster=ctx["cluster_arn"],
            taskDefinition=ctx["tile_task_def"],
            launchType="FARGATE",
            overrides={"containerOverrides": [{
                "name": "tile-processor",
                "environment": [{"name": "TILE_JSON", "value": os.path.basename(tile["uri"])}],
            }]},
        )
    # Normal path: the Map state already dispatched this tile to Lambda.
    return {"tile": tile["index"], "mode": "lambda"}

This keeps the serverless cost profile for the 99% of tiles that fit, while guaranteeing the run completes even when one tile does not.

Operational Checklist

Before promoting any recipe on this page to production, confirm:

• Bounded work items — The planner proves, not assumes, that the largest manifest item fits the compute stage’s memory, timeout, and /tmp ceilings on the target provider.
• Manifest offloaded to storage — Large manifests live in S3/GCS and are passed by URI, keeping every state transition under the 256 KB (AWS) or 512 KB (GCP) payload limit.
• Idempotent tile writes — Each worker writes to a deterministic per-tile key so retries overwrite rather than duplicate. A re-run produces identical output.
• Dead-letter routing — Every fan-out branch has a DLQ; a non-zero DLQ depth alerts within five minutes and the manifest item is recoverable.
• Merge /tmp guard — The merge stage checks free /tmp before building the VRT and streams to a COG rather than materialising the full mosaic in memory.
• Catalog write is terminal and idempotent — The STAC item write is the last step and keyed by product ID, so a retried catalog step does not create duplicates.
• Explicit GDAL environment — GDAL_DATA, PROJ_LIB, and LD_LIBRARY_PATH are set in every function that imports GDAL, never inherited implicitly.
• Container fallback wired — The circuit breaker path to Fargate or Cloud Run is tested, not aspirational, so an oversized tile completes instead of failing the run.

Frequently Asked Questions

Which orchestrator should I use for a serverless geospatial pipeline?

On AWS use Step Functions when you need per-tile fan-out with a Map state and native retry; on GCP use Cloud Workflows with parallel branches driving Cloud Functions or Cloud Run; on Azure use Durable Functions fan-out/fan-in. Choose by which compute limit your workload hits first — AWS caps invocations at 15 minutes and 10 GB memory, GCP at 60 minutes and 32 GB, Azure Consumption at 10 minutes and 1.5 GB.

How do I process a raster larger than the Lambda memory ceiling?

Never load the full raster. Read windows through range requests against a Cloud Optimized GeoTIFF, fan out one function invocation per tile so each stays under the 10,240 MB memory and 10 GB /tmp ceilings, and reassemble the tiled outputs in a final merge step. The 10 GB GeoTIFF recipe walks the full path.

How do I keep the Step Functions payload under the 256 KB state limit?

Emit the tile manifest to S3 and pass only the object URI between states, or use the Map state ItemReader to stream items directly from an S3 JSON or CSV manifest. Never inline thousands of tile windows into the state payload — see partitioning a GeoTIFF into Map-state tiles.

Can a single serverless pipeline span more than one cloud provider?

It can, but avoid it for tightly coupled tiling stages because cross-cloud egress cost and latency dominate. A pragmatic split keeps ingestion and orchestration on the provider that hosts the data, with a container fallback for tiles that exceed function limits.

Related

• Process a 10 GB GeoTIFF with AWS Step Functions and Lambda — the flagship end-to-end recipe: upload, tile fan-out, merge, and catalog within Lambda limits
• Serverless Geospatial Architecture & Platform Limits — the hard quotas and cold-start behaviour every recipe stage must respect
• Packaging & Dependency Management for Serverless GIS — building the GDAL/rasterio layers the compute stages import
• Event-Driven Geospatial Processing Patterns — the triggers, queues, and chunked-I/O reads the ingestion and fan-out stages are built on
• Chunked I/O for Large Satellite Imagery — window and range-read techniques that let each tile worker stay within memory