Serverless Geospatial Pipeline Recipes

The three neighbouring sections of this reference each isolate one hard problem: what a serverless runtime will and will not let you do, how to package heavy native geospatial dependencies, and how to wire event-driven triggers between functions. This section stitches those constraints into complete, runnable pipelines. Every recipe here is a full path from an ingestion event to a cataloged product — a 10 GB GeoTIFF reprojected and tiled, a live AIS vessel feed aggregated into spatial windows, Sentinel-2 scenes turned into NDVI tiles, or a vector dataset rendered to Mapbox Vector Tiles — assembled so that no single function invocation ever breaches a platform ceiling. The goal is not another survey of limits but a set of blueprints you can lift, parameterise, and deploy.

Foundational Architecture Patterns

Every recipe on this site is a variation on the same five-stage skeleton. An event announces new data, a planner turns that data into a manifest of bounded work items, an orchestrator fans those items out across many short-lived function invocations, a merge stage reassembles the partial outputs, and a catalog stage makes the result discoverable. The diagram below traces that reference pipeline end to end.

1. Ingestion Event — An object-storage notification fires when a new scene lands. The S3 and GCS event triggers for Shapefiles patterns generalise to any raster or vector payload; the trigger function stays tiny and does no pixel work.
2. Metadata + Planner — A lightweight function reads the file header, derives dimensions, block size, and CRS, and computes a manifest of bounded work items. This is where the raster is cut into tiles that individually fit the compute stage’s memory and timeout.
3. Orchestrator — Step Functions (AWS), Cloud Workflows (GCP), or Durable Functions (Azure) own execution state, retries, and parallel fan-out. The orchestrator never touches pixels; it dispatches manifest items and collects results.
4. Tiled Compute Fan-Out — One short-lived function invocation per tile, each reading only its window through chunked I/O for large satellite imagery. Failures are isolated to a single tile and routed to a dead-letter queue for failed vector jobs rather than aborting the run.
5. Merge + Catalog — A final function reassembles the tiled outputs into a Cloud Optimized GeoTIFF or tile pyramid, builds overviews, and registers a STAC item so downstream consumers can discover the product.

What makes this a skeleton rather than a diagram is the contract between the planner and the fan-out. The planner’s job is not to divide work evenly but to prove an upper bound: the largest item it emits must fit the compute stage’s memory, timeout, and /tmp ceilings on the provider you deploy to, with margin for the working buffers GDAL allocates around the decoded array. For a raster that means deriving tile dimensions from the file’s own internal block size rather than from a round number — a 2,048-pixel window over a COG with 512-pixel blocks is sixteen whole blocks, so every read fetches complete blocks and nothing is fetched twice. For a vector dataset it means partitioning by feature count or by a spatial index cell — a quadkey, an H3 cell — rather than by file offset, because geometry sizes vary by orders of magnitude within a single layer. Get that bound right and the compute stage never needs a retry policy for resource exhaustion, only for transient I/O.

The second invariant is idempotency, and it is far cheaper to build in than to add. Derive the run identifier deterministically from the input URI plus the processing parameters, write every work item’s output to a key computed from that run ID and the item’s index, and a retried item overwrites its own previous attempt instead of appending a duplicate. Object-storage notifications are at-least-once on every provider, so the same upload will occasionally start two executions; deterministic keys make the second one converge on the same product rather than corrupt it. It also makes partial failure recoverable: if eleven of four hundred windows dead-letter, the merge can still run over the 389 that succeeded, and a targeted re-run of the eleven completes the product without reprocessing anything else.

The third invariant is that the orchestrator holds state, not data. Every recipe here passes URIs between steps and keeps payloads in the low kilobytes, because the alternative — threading tile geometry, band statistics or, worse, pixel buffers through the state machine — hits a hard transition limit on AWS and GCP and makes the execution history unreadable everywhere. When a stage needs to hand something large to the next one, it writes it to object storage and passes the key.

The flagship worked example of this skeleton is the process a 10 GB GeoTIFF with Step Functions and Lambda recipe, which carries a full 10 GB scene through all five stages while keeping every invocation inside Lambda’s ceilings. Its two focused companions cover the two stages that break most often: partitioning a GeoTIFF into Step Functions Map-state tiles and merging tiled Lambda outputs into a COG.

The other three recipes bend the same skeleton toward different data shapes. The real-time AIS vessel tracking pipeline replaces the batch planner with a streaming windowed aggregation; the serverless NDVI tiling from Sentinel-2 recipe fans multi-band raster algebra out per tile; and the vector tile pipeline with Cloud Run and Pub/Sub swaps Lambda for containers when Tippecanoe’s memory profile exceeds function limits.

What does not change between them is the shape of the failure. In every recipe a work item can be too large, too slow, or malformed, and in every recipe the answer is the same ladder: retry the item, subdivide it, and only then escalate it to a container. The recipes differ in what a work item is — a raster window, a five-minute time bucket of AIS positions, one MGRS tile, one vector layer — and in which provider hosts the data. They do not differ in how they degrade.

Platform Constraints Reference Table

A recipe is only correct if every stage respects the ceilings of the provider you deploy it on. These are the quotas that decide how finely the planner must tile and whether an orchestrator step can run at all.

The single most consequential figure for these recipes is the memory-backed nature of GCP’s /tmp: writing a 6 GB intermediate VRT there consumes 6 GB of your memory allocation, whereas Lambda’s /tmp is separate disk-backed storage up to 10 GB. That difference decides whether a merge stage can run in one invocation or must itself be tiled. The full breakdown of these boundaries lives in Serverless Geospatial Architecture & Platform Limits, and the per-tile packaging of GDAL that every compute stage depends on is covered in Packaging & Dependency Management for Serverless GIS.

Three of the remaining rows carry arithmetic worth doing before you deploy rather than after. Concurrency first: a 400-item fan-out at a MaxConcurrency of 500 will happily request four hundred simultaneous Lambda invocations, which is 40% of the default 1,000 regional concurrency quota — a quota shared with every other function in that account and region. Two such runs overlapping put you at 80%, and a third throttles something that is not this pipeline. Reserve concurrency on the worker function so the fan-out cannot starve unrelated workloads, and cap MaxConcurrency below that reservation so the orchestrator throttles itself rather than discovering the quota through a storm of TooManyRequestsException retries.

Deployment size is the second. A rasterio, shapely, and pyproj stack unzipped runs past 200 MB against Lambda’s 250 MB unzipped limit, which counts the function package and every attached layer together. That leaves under 50 MB for application code, coefficient tables, and any model weights — enough, but not by much, and it is why the packaging discipline matters more here than it would in a container world. GCP’s 100 MB compressed source limit looks tighter still on paper, which is exactly why 2nd-gen functions built from Artifact Registry images side-step it entirely.

The third is the state payload. A tile record serialises to roughly 200 bytes, so a manifest of more than about 1,200 items overflows the 256 KB transition limit if it is inlined. Because manifest length is a function of scene size, that threshold gets crossed by someone uploading a bigger file, not by a code change — the failure arrives in production, on a Tuesday, from an input nobody reviewed. That is why every recipe here streams the manifest from object storage through ItemReader even in the cases where inlining would have fitted.

Which Orchestrator Fits Which Recipe

The orchestrator choice is rarely about preference — it follows from which limit the workload hits first and where the source data lives. This table maps each recipe to the provider and orchestrator that fit it without a container fallback.

Cloud Workflows and Durable Functions become the natural choice the moment a stage needs more than Lambda’s 15 minutes or 10 GB — for example a whole-country vector build or a 32 GB in-memory raster mosaic. When even those ceilings are not enough, every recipe degrades to a container fallback (Fargate or Cloud Run) rather than failing.

Two secondary properties decide the rest. The first is workflow flavour. Step Functions Standard workflows are durable, replayable, and billed per state transition, while Express workflows are billed by duration and memory and keep no queryable execution history beyond CloudWatch. A 400-tile raster run makes a few thousand transitions, costs cents in Standard, and leaves an execution history that is genuinely worth having the morning a tile fails. A streaming AIS pipeline making millions of very short transitions an hour would be ruinous in Standard, which is why the real-time recipe uses Express and accepts the loss of per-execution history in exchange.

The second is where the data already sits. Cross-cloud fan-out is technically straightforward and economically poor: every tile read crosses an egress boundary, and at four hundred windows against a 10 GB scene the transfer line on the bill overtakes the compute line quickly. Keep the orchestrator, the workers, and the object storage inside one region of one provider, and reach for that provider’s container fallback rather than for another cloud’s larger ceiling.

Runtime Optimization for Geospatial Libraries

Recipe compute stages share one performance profile: a large native dependency graph imported cold, then a burst of CPU-bound array work. Two levers dominate.

First, cold-start amortisation. Because a Map state can launch hundreds of tile workers within seconds, each paying the Python GDAL cold-start cost, the marginal saving from a smaller, pre-warmed layer is multiplied across the whole fan-out. For latency-sensitive recipes, provisioned concurrency on the tile-worker function pays back quickly. Always pin the runtime paths explicitly so GDAL never falls back to nonexistent compile-time defaults:

import os

# Set in the tile-worker function configuration, applied before importing rasterio.
os.environ.setdefault("GDAL_DATA", "/opt/share/gdal")
os.environ.setdefault("PROJ_LIB", "/opt/share/proj")
os.environ.setdefault("LD_LIBRARY_PATH", "/opt/lib:" + os.environ.get("LD_LIBRARY_PATH", ""))
# COG-friendly GDAL tuning: only fetch the bytes a window needs.
os.environ.setdefault("GDAL_DISABLE_READDIR_ON_OPEN", "EMPTY_DIR")
os.environ.setdefault("CPL_VSIL_CURL_ALLOWED_EXTENSIONS", ".tif,.tiff")
os.environ.setdefault("VSI_CACHE", "TRUE")

Second, memory-proportional CPU. On AWS and GCP, vCPU scales linearly with the memory you allocate, so a tile worker doing raster algebra is often faster and cheaper at a higher memory tier even when it never uses the extra RAM — the extra cores finish the NumPy work sooner. Size the tier against the memory and CPU allocation model for raster workloads rather than guessing, and keep intermediate arrays out of /tmp where ephemeral storage limits apply.

A third lever is smaller but very nearly free: how GDAL reads. The environment above disables directory listing on open, restricts the extensions the virtual filesystem will consider, and turns on the read cache — together these cut a COG window read from a dozen HTTP requests to two or three. Beyond that, GDAL_CACHEMAX governs the block cache and defaults to a proportion of available memory, so on a large tier GDAL will happily retain hundreds of megabytes of blocks a single-window worker will never touch again. Setting it explicitly — 256 MB is generous for a worker that reads one window — leaves that memory for the decoded array instead and makes the peak footprint predictable enough to size a tier against. GDAL_HTTP_MULTIRANGE=YES and GDAL_HTTP_MERGE_CONSECUTIVE_RANGES=YES let the driver coalesce adjacent block requests into one, which pays off most for exactly the block-aligned windows a good planner emits.

Security, IAM, and Data Governance

A fan-out pipeline multiplies the blast radius of an over-broad role: one leaked worker credential with s3:* exposes every scene in the bucket. Each stage should carry only the permissions it exercises. The planner needs s3:GetObject on the ingestion prefix to read headers; the tile workers need s3:GetObject on the source and s3:PutObject on a per-run output prefix; the merge stage needs read on that output prefix and write on the catalog prefix. The IAM security boundaries for Cloud GIS discipline covers per-stage scoping, and its least-privilege policies for Azure Blob geospatial access companion translates the same boundaries to Azure Managed Identities.

Additional governance controls that matter at recipe scale:

• Per-run output prefixes — Write each pipeline run under a deterministic prefix derived from the input URI and parameters. This isolates partial outputs, makes cleanup atomic, and prevents one run from overwriting another’s tiles.
• VPC endpoints — Route the high-volume tile reads and writes over S3/Storage private endpoints. At thousands of tiles per run, public-path NAT egress cost is significant and avoidable.
• KMS / CMEK at rest — Encrypt source and product buckets with customer-managed keys scoped to the pipeline roles, so a compromised worker role still cannot decrypt data outside its stage.
• Signed manifests — When the planner writes a tile manifest to S3, treat it as trusted input to the orchestrator; validate its schema in the first worker step so a corrupted manifest cannot drive workers to arbitrary URIs.

One control is specific to fan-out and easy to miss: the worker role is exercised hundreds of times per run, on paths driven by a manifest that some earlier function wrote. Treat that manifest as the trust boundary it actually is. Validate each item against a schema and, more importantly, assert that its source URI and output key fall under the prefixes this run owns before the worker opens anything. Without that assertion, a manifest derived from an attacker-influenced object key becomes a way to turn several hundred trusted invocations loose on whatever the role permits — and the role permits a whole prefix, because it has to.

Observability, Cost Control, and Fallback Patterns

The dominant cost in these recipes is GB-seconds across the fan-out, so per-tile instrumentation is where cost control lives. Emit a structured record from every worker with the tile index, bytes read, peak memory, and duration, and aggregate it at the orchestrator boundary.

The arithmetic behind those metrics is what makes the cost lever obvious. A run of 400 windows at 24 seconds each on a 2 GB allocation is 400 × 24 × 2 ≈ 19,200 GB-seconds, plus a few thousand state transitions: cents, not dollars. Doubling the tile size halves the invocation count but roughly doubles each invocation’s duration and pushes up its memory tier, so the GB-second total barely moves. What actually moves the bill is the total number of pixels processed and how many times each pixel is read — which is why a second pass over the source, or a merge stage that downloads tiles it could have referenced in place, costs far more than any tuning of the tier ever recovers. Track GB-seconds per megapixel as a derived metric and a regression shows up as a step change in it, weeks before the invoice says anything.

When a tile consistently trips the memory or timeout alert — a denser region, a larger block, an extra band — the correct response is not a bigger tile-worker tier but a finer manifest. The planner re-tiles the offending window into sub-tiles on retry. Only when a single indivisible unit of work genuinely exceeds the function ceiling does the circuit breaker route it to a container:

import boto3, os

sfn = boto3.client("stepfunctions")
ecs = boto3.client("ecs")

def dispatch_tile(tile, ctx):
    """Run a tile on Lambda; fall back to Fargate only for oversized indivisible tiles."""
    if tile["est_pixels"] > ctx["lambda_pixel_ceiling"]:
        # A single tile too large to subdivide further — send to a container task.
        return ecs.run_task(
            cluster=ctx["cluster_arn"],
            taskDefinition=ctx["tile_task_def"],
            launchType="FARGATE",
            overrides={"containerOverrides": [{
                "name": "tile-processor",
                "environment": [{"name": "TILE_JSON", "value": os.path.basename(tile["uri"])}],
            }]},
        )
    # Normal path: the Map state already dispatched this tile to Lambda.
    return {"tile": tile["index"], "mode": "lambda"}

This keeps the serverless cost profile for the 99% of tiles that fit, while guaranteeing the run completes even when one tile does not.

Operational Checklist

Before promoting any recipe on this page to production, confirm:

• Bounded work items — The planner proves, not assumes, that the largest manifest item fits the compute stage’s memory, timeout, and /tmp ceilings on the target provider.
• Manifest offloaded to storage — Large manifests live in S3/GCS and are passed by URI, keeping every state transition under the 256 KB (AWS) or 512 KB (GCP) payload limit.
• Idempotent tile writes — Each worker writes to a deterministic per-tile key so retries overwrite rather than duplicate. A re-run produces identical output.
• Dead-letter routing — Every fan-out branch has a DLQ; a non-zero DLQ depth alerts within five minutes and the manifest item is recoverable.
• Merge /tmp guard — The merge stage checks free /tmp before building the VRT and streams to a COG rather than materialising the full mosaic in memory.
• Catalog write is terminal and idempotent — The STAC item write is the last step and keyed by product ID, so a retried catalog step does not create duplicates.
• Explicit GDAL environment — GDAL_DATA, PROJ_LIB, and LD_LIBRARY_PATH are set in every function that imports GDAL, never inherited implicitly.
• Container fallback wired — The circuit breaker path to Fargate or Cloud Run is tested, not aspirational, so an oversized tile completes instead of failing the run.

Frequently Asked Questions

Which orchestrator should I use for a serverless geospatial pipeline?

On AWS use Step Functions when you need per-tile fan-out with a Map state and native retry; on GCP use Cloud Workflows with parallel branches driving Cloud Functions or Cloud Run; on Azure use Durable Functions fan-out/fan-in. Choose by which compute limit your workload hits first — AWS caps invocations at 15 minutes and 10 GB memory, GCP at 60 minutes and 32 GB, Azure Consumption at 10 minutes and 1.5 GB.

How do I process a raster larger than the Lambda memory ceiling?

Never load the full raster. Read windows through range requests against a Cloud Optimized GeoTIFF, fan out one function invocation per tile so each stays under the 10,240 MB memory and 10 GB /tmp ceilings, and reassemble the tiled outputs in a final merge step. The 10 GB GeoTIFF recipe walks the full path.

How do I keep the Step Functions payload under the 256 KB state limit?

Emit the tile manifest to S3 and pass only the object URI between states, or use the Map state ItemReader to stream items directly from an S3 JSON or CSV manifest. Never inline thousands of tile windows into the state payload — see partitioning a GeoTIFF into Map-state tiles.

Can a single serverless pipeline span more than one cloud provider?

It can, but avoid it for tightly coupled tiling stages because cross-cloud egress cost and latency dominate. A pragmatic split keeps ingestion and orchestration on the provider that hosts the data, with a container fallback for tiles that exceed function limits.

How large a scene can this skeleton handle before it stops being serverless?

The skeleton stops scaling at the fan-out ceiling rather than at the scene size. Holding tile pixels constant, a 100 GB scene simply yields ten times the windows, and the distributed Map state will run up to 10,000 concurrent child executions. Past roughly ten thousand items the answer is a larger tile or a batched sub-manifest per band of rows, not a smaller tile. The stage that actually breaks first is usually the merge: it reads every tile once and builds overviews single-threaded, so somewhere above a few hundred gigabytes of reassembled product it approaches the 15-minute Lambda ceiling and has to be split into partial merges or moved to a container.

What should the planner do with a source that is not a Cloud Optimized GeoTIFF?

Convert it first, as an explicit stage. Windowed reads against a striped, non-tiled GeoTIFF pull whole scanlines, so a 2,048-pixel window costs the same bytes as a 40,000-pixel one and the fan-out loses its entire advantage. Validate the source with rio cogeo validate in the planner and, when validation fails, route the object through a one-off translation to COG — in a container if the file is too large for a function — before any manifest is emitted. Rewriting the file once is far cheaper than paying for full-width reads four hundred times.

Do these recipes need provisioned concurrency?

Usually not on the worker. A fan-out hides cold starts behind parallelism: four hundred workers starting at once each pay their own 3–8 second GDAL initialisation, but they pay it concurrently, so the run’s wall-clock grows by roughly one cold start rather than four hundred. Provisioned concurrency earns its keep in latency-sensitive recipes — an interactive tile request, a streaming aggregation with a delivery deadline — and on the trigger and planner functions, which are single invocations sitting on the critical path where a cold start is the entire latency.

Topics in this section

• Process a 10 GB GeoTIFF with AWS Step Functions and Lambda — A complete serverless recipe: S3 upload triggers metadata extraction, Step Functions Map fans out per-tile…
• Real-Time AIS Vessel Tracking Pipeline on Serverless — Build a serverless AIS vessel tracking pipeline: ingest NMEA messages through Kinesis or Pub/Sub, parse…
• Serverless NDVI Tiling from Sentinel-2 Imagery — Fan out per-tile Lambdas on a new Sentinel-2 scene: window-read the B04 and B08 bands from the source COG…
• STAC Cataloging and Metadata Publishing — Publish the output of a serverless tiling job as a SpatioTemporal Asset Catalog: write Items with pystac and…
• Vector Tile Pipeline with Cloud Run and Pub/Sub — Build a GCP vector tile pipeline: GeoJSON or GeoParquet lands in GCS, Pub/Sub fires a Cloud Run service that…

Related

• Process a 10 GB GeoTIFF with AWS Step Functions and Lambda — the flagship end-to-end recipe: upload, tile fan-out, merge, and catalog within Lambda limits
• Serverless Geospatial Architecture & Platform Limits — the hard quotas and cold-start behaviour every recipe stage must respect
• Packaging & Dependency Management for Serverless GIS — building the GDAL/rasterio layers the compute stages import
• Event-Driven Geospatial Processing Patterns — the triggers, queues, and chunked-I/O reads the ingestion and fan-out stages are built on
• Chunked I/O for Large Satellite Imagery — window and range-read techniques that let each tile worker stay within memory