Serverless Geospatial Architecture & Platform Limits
Modern geospatial processing has shifted from monolithic, always-on GIS servers to event-driven, ephemeral compute. For cloud GIS engineers, Python backend developers, DevOps practitioners, and platform architects, this transition unlocks substantial scalability and cost efficiency — but spatial workloads are inherently resource-intensive. Raster tiling, vector topology validation, coordinate transformations, and spatial joins routinely push against the hard boundaries of serverless execution environments. Designing a resilient serverless geospatial architecture requires deliberate orchestration, memory-aware data streaming, strict IAM scoping, and fallback patterns that degrade gracefully under platform quotas.
Foundational Architecture Patterns
Serverless geospatial processing thrives on event-driven decomposition. Rather than processing an entire scene in a single invocation, mature architectures break workflows into discrete, stateless stages:
- Ingestion Trigger — Object storage events (S3, GCS, Azure Blob) fire when new GeoTIFFs, Shapefiles, or GeoParquet geometries land in a watched prefix.
- Metadata Extraction — Lightweight functions read file headers, extract bounding boxes, CRS, and band counts without loading pixel data into memory.
- Orchestration Layer — Step Functions (AWS), Cloud Workflows (GCP), or Durable Functions (Azure) manage execution state, retries, and parallel fan-out across tiles.
- Compute Execution — Memory-intensive operations (tiling, resampling, vectorization, reprojection) run in fully allocated function environments or containerised serverless endpoints.
- Output & Cataloging — Processed artifacts are written to object storage, registered in spatial catalogs (STAC), and indexed for downstream query.
This decomposition enforces idempotency and isolates failures at the step boundary. If a tiling job fails mid-process, the orchestrator retries only the failed step, preserving partial outputs and avoiding redundant compute. State machines should track job progression using deterministic identifiers derived from input URIs and processing parameters — this prevents duplicate S3 notifications or transient network failures from triggering redundant spatial transformations. Pairing exponential backoff with jitter on retry policies, alongside dead-letter queues for failed vector jobs, creates a self-healing pipeline that requires minimal operator intervention.
By decoupling metadata parsing from pixel processing you also prevent cold-start latency from cascading into downstream tile compute. Reading a 50 GB GeoTIFF via HTTP range requests against a Cloud Optimized GeoTIFF (COG) layout requires careful chunked I/O — the patterns for that are covered in Chunked I/O for Large Satellite Imagery.
Sizing the Fan-Out
The orchestrator’s real job is to convert one scene into a number of invocations that each fit inside the quota envelope, and that number is arithmetic rather than judgement. A Sentinel-2 L2A 10 m band is 10,980 × 10,980 pixels of uint16 — 241 MB decompressed for a single band, and roughly 3.1 GB if you materialise all thirteen. Cut at 512 × 512 that is 460 tiles per band; cut at 2,048 × 2,048 it is 36. The fine split finishes inside a 15-minute Lambda window with enormous headroom but pays 460 invocation overheads and up to 460 cold starts; the coarse split pays five and risks an out-of-memory kill the moment a resampling kernel needs a second copy of the array in float64. The workable rule is to choose the largest tile whose peak working set — input window, output window, and any intermediate type promotion — sits below roughly 60% of allocated memory, then let concurrency rather than tile size absorb the remaining throughput requirement.
Fan-out width matters as much as tile size. Step Functions’ inline Map state runs at most 40 concurrent iterations, which quietly serialises a 460-tile job into twelve waves; a Distributed Map runs up to 10,000 child executions and will saturate a 1,000-invocation regional concurrency quota within seconds unless MaxConcurrency is set deliberately. State payloads are capped at 256 KB, so tile lists must be passed by reference — an S3 manifest URI — rather than inlined, and the same discipline applies to the 256 KB asynchronous Lambda event limit and the 6 MB synchronous response limit. A pipeline that tries to return pixel data through the orchestrator will hit one of those three ceilings on its first real scene, and the failure surfaces as a truncated state payload rather than an obvious size error.
Platform Constraints Reference Table
Understanding the hard limits of each provider is non-negotiable for production spatial pipelines. The following table captures the constraints that most directly affect geospatial workloads.
| Constraint | AWS Lambda | GCP Cloud Functions (2nd gen) | Azure Functions (Consumption) |
|---|---|---|---|
| Max timeout | 15 min | 60 min | 10 min |
| Memory ceiling | 10,240 MB | 32,768 MB | 1,536 MB |
Ephemeral storage (/tmp) |
10,240 MB (512 MB default) | ~8 GB (tmpfs) | ~1.5 GB shared |
| Deployment package | 250 MB zipped / 250 MB unzipped | 100 MB compressed | 1 GB (zip) |
| Concurrency quota | 1,000 (soft, regional) | 3,000 (per project) | 200 (per function) |
| CPU scaling | Linear with memory | Linear with memory | Fixed per plan |
| VPC support | Yes (ENI-based) | Yes (Serverless VPC) | Yes (VNET integration) |
| Provisioned concurrency | Yes (additional cost) | Yes (min instances) | Yes (Premium plan only) |
Direct geospatial impact:
- Timeout — Global DEM mosaicking, large-scale network analysis, and ML inference on satellite imagery routinely require more than 10–15 minutes. Tile-based chunking is not optional on AWS or Azure Consumption; it is a hard architectural requirement.
- Memory — Memory and CPU allocation for raster workloads is the primary lever for throughput; CPU scales linearly with memory on both AWS and GCP.
- Ephemeral storage — Ephemeral storage limits in AWS Lambda can exhaust
/tmpbefore GDAL registers its first driver when intermediate VRTs, unpacked shapefiles, or tile caches accumulate. Azure’s shared pool makes this even tighter. - Deployment package — A standard
rasterio+shapely+pyprojstack unzipped exceeds 200 MB. On AWS this must be split across Lambda Layers; on GCP, Artifact Registry container images side-step the zip limit entirely. - Concurrency — Burst tiling jobs that fan out per-tile can saturate regional concurrency quotas within seconds. Throttling triggers silent retry storms without a properly tuned orchestrator.
Reading the Table as a Decision Procedure
Used in order, the five rows answer the platform question without a proof-of-concept. First, does one unit of work complete inside the timeout? Azure Functions on the Consumption plan gives you 10 minutes, AWS Lambda 15, and GCP Cloud Functions 2nd gen 60 — so a job that needs 25 minutes per unit is a GCP job, a container job, or a job that has not been decomposed finely enough yet. Second, does the peak working set fit the memory ceiling? At 1,536 MB, Azure Consumption cannot hold a single full-resolution Sentinel-2 band plus an output buffer, which is why its recipes chunk earlier than the AWS equivalents; Lambda’s 10,240 MB and GCP’s 32,768 MB both do. Third, does the dependency stack fit the deployment surface — 250 MB unzipped on Lambda, which a rasterio + shapely + pyproj build already approaches before application code. Fourth, does the intermediate footprint fit /tmp, which is 512 MB by default on Lambda and configurable to 10,240 MB. Only then does concurrency come into it.
Concurrency deserves separate attention because its failure mode is the least visible. Lambda’s 1,000 regional limit is shared across every function in the account and region, so a tiling fan-out can throttle an unrelated API handler; the platform adds execution environments in bursts rather than instantly, so a step change from zero to a thousand tiles produces throttles even when the account is nowhere near its ceiling. Reserved concurrency on the tile processor and a bounded MaxConcurrency on the orchestrator are what turn that from a retry storm into a queue.
Runtime Optimization for Geospatial Libraries
Packaging and initialising spatial dependencies in serverless environments introduces performance characteristics that differ sharply from containerised workloads. Deployment archive size, native binary compatibility, and initialisation overhead all directly affect latency and reliability.
Cold Starts and Dependency Packaging
Geospatial Python packages are large. A standard rasterio, shapely, and pyproj stack can exceed 200 MB unzipped, pushing Lambda deployment packages against the 250 MB limit before application code is even added. Cold starts occur when the platform provisions a new execution environment, unpacks the archive, resolves shared libraries, and imports Python modules. For Python-based GIS workloads this adds 3–8 seconds of latency before the first line of business logic executes.
The cold start mapping for Python GDAL sequence begins with shared-library resolution: libgdal, libproj, and libgeos must each be found, loaded, and linked before import rasterio completes. Mitigation strategies:
- Provisioned concurrency — Pre-warms a fixed number of execution environments. The reducing Python GDAL cold starts with provisioned concurrency guide documents exact initialisation timelines and cost trade-offs.
- Lambda Layers — Separate the geospatial binary stack (
GDAL,PROJ,GEOS) from application code. Layers are cached across deployments, reducing the unpack surface on each cold start. - Lightweight alternatives —
pyogriofor vector I/O andxarray+rioxarrayfor raster workflows reduce import chains without sacrificing functionality.pyarrowwith GeoParquet can replacefionafor many read-only use cases. - Always set runtime environment variables explicitly:
import os
os.environ["GDAL_DATA"] = "/opt/share/gdal"
os.environ["PROJ_LIB"] = "/opt/share/proj"
os.environ["LD_LIBRARY_PATH"] = "/opt/lib:" + os.environ.get("LD_LIBRARY_PATH", "")
Omitting these causes GDAL to fall back to compile-time paths that do not exist in the Lambda execution environment, producing cryptic CPLE_OpenFailed errors during CRS resolution.
Two packaging escape hatches are worth knowing before you spend a week trimming wheels. A function can carry at most five layers, and the combined unzipped size of the function package plus all its layers is what the 250 MB limit applies to — splitting a 260 MB stack across two layers does not help. Deploying the same stack as a container image raises the ceiling to 10 GB, which removes the packaging problem entirely at the cost of a slower first pull; on GCP that is the only model available anyway. Where the runtime supports it, a snapshot-based start such as Lambda SnapStart restores a pre-initialised execution environment rather than replaying the import, which converts the shared-library resolution cost into a restore cost — but it also freezes any state captured at snapshot time, so a cached PROJ context or an open HTTP connection must be re-established in a post-restore hook.
GDAL Configuration That Actually Moves the Number
Most of the tuning that matters for serverless GDAL is environment configuration, not code. Set these deliberately rather than inheriting whatever the base image chose:
GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR— stops GDAL listing the whole prefix before opening a remote file. On an S3 prefix with tens of thousands of objects this alone removes seconds per open, and it is the single highest-yield setting for/vsis3and/vsicurlreads.GDAL_CACHEMAX— the block cache defaults to a percentage of physical RAM, which on a 10,240 MB function is a very large number that competes with your own arrays. Pin it explicitly, in megabytes, to a value that leaves room for the working set you sized above.VSI_CACHE=TRUEandVSI_CACHE_SIZE— a per-file read cache in front of range requests. Worth enabling when several windows fall inside the same COG block; wasted memory when each invocation touches one tile once.GDAL_PAM_ENABLED=NO— suppresses the.aux.xmlsidecar. In a read-only function the sidecar write fails or lands in/tmp, where it silently consumes ephemeral storage across warm invocations.CPL_TMPDIR=/tmp— makes the scratch location explicit so that intermediate VRTs and unzipped shapefiles land where you have quota rather than in a read-only path.
The /tmp interaction is the one most teams discover in production. Ephemeral storage persists for the life of the execution environment, not the invocation, so a warm function that writes a 40 MB intermediate per call exhausts the 512 MB default after roughly a dozen invocations and then fails with a disk-full error that looks nothing like a quota breach. Delete intermediates in a finally block, or stream through /vsimem and never touch the disk at all.
GIL Contention and Parallel Processing
The Global Interpreter Lock in CPython prevents true multi-threading for CPU-bound tasks. Geospatial operations — raster algebra, spatial indexing, topology validation — are heavily CPU-bound. To bypass the GIL:
- Use
multiprocessing.ProcessPoolExecutorrather thanThreadPoolExecutor. - Allocate maximum available memory (10 GB on Lambda), spawn 2–4 worker processes, and exchange large arrays via
numpy.memmapto avoid duplication. - Prefer C-extensions that release the GIL internally: GDAL’s C bindings,
shapely’s GEOS calls, andpyproj3.x all release the GIL during I/O-intensive operations. - Profile with
tracemallocormemory_profilerbefore deploying — uncontrolled process forking is the leading cause of serverless out-of-memory (OOM) errors in spatial workloads.
import os
import numpy as np
from concurrent.futures import ProcessPoolExecutor
import rasterio
from rasterio.windows import Window
def process_tile(args):
src_path, col_off, row_off, width, height = args
os.environ["GDAL_DATA"] = "/opt/share/gdal"
os.environ["PROJ_LIB"] = "/opt/share/proj"
with rasterio.open(src_path) as src:
window = Window(col_off, row_off, width, height)
data = src.read(1, window=window)
return data.mean() # replace with real transform
def lambda_handler(event, context):
src_path = f"/vsicurl/{event['url']}"
tile_size = 512
tiles = [
(src_path, c, r, tile_size, tile_size)
for r in range(0, 4096, tile_size)
for c in range(0, 4096, tile_size)
]
with ProcessPoolExecutor(max_workers=3) as pool:
results = list(pool.map(process_tile, tiles))
return {"tile_count": len(results), "mean": float(np.mean(results))}
Security, IAM, and Data Governance
Spatial data frequently contains sensitive location intelligence, proprietary survey results, or regulated environmental datasets. Serverless architectures must enforce least-privilege access at every stage.
Least-Privilege Execution Roles
Functions must never run with broad s3:* or storage.admin permissions. IAM security boundaries for Cloud GIS scopes each pipeline stage to the minimum required S3 prefix and action set — for example, the metadata extraction function needs only s3:GetObject on the ingestion bucket prefix, while the cataloging function requires s3:PutObject on the output prefix and dynamodb:PutItem for the spatial index. The least-privilege IAM policies for Azure Blob geospatial access page covers the equivalent role assignments for Azure Managed Identities.
Additional controls for production spatial pipelines:
- VPC endpoints — Route S3 and DynamoDB traffic over private endpoints rather than the public internet. This prevents data exfiltration and eliminates NAT Gateway data-transfer costs for high-throughput tiling jobs.
- KMS encryption — Encrypt all raster and vector outputs at rest with customer-managed keys. Scope key policies to the function’s execution role.
- Resource-based policies — Complement identity-based policies with S3 bucket policies that deny
s3:*from all principals except the designated pipeline roles. - Compliance frameworks — FedRAMP, ISO 27001, and GDPR all require demonstrable access control audit trails; CloudTrail (AWS), Cloud Audit Logs (GCP), and Azure Monitor Activity Logs satisfy this if structured logging is configured.
Two details are specific to spatial pipelines rather than generic serverless hygiene. The first is that a tiling fan-out multiplies every access-control mistake by the tile count: a compute role with s3:GetObject on the whole bucket does not leak one object when it is compromised, it leaks the archive, and the CloudTrail record of that leak is buried in the hundreds of thousands of legitimate reads the same pipeline makes every night. Scope the compute role to the scratch and ingest prefixes and it becomes possible to alert on any read outside them. The second is data residency. Imagery is frequently licensed per region, and a function that reads from an eu-west-1 bucket while running in us-east-1 has moved regulated data across a boundary as a side effect of a deployment default. Pin the function region to the bucket region, deny cross-region access in the bucket policy, and treat any cross-region read as a pipeline defect rather than a latency inconvenience.
Note also that VPC attachment no longer carries the cold-start penalty it once did — network interfaces are provisioned per function configuration rather than per execution environment — so routing spatial traffic through private endpoints is now a security decision rather than a latency trade-off.
Data Lineage and Audit Trails
Every transformation, reprojection, and aggregation should emit structured JSON logs that capture input/output URIs, CRS transformations applied, processing timestamps, and the function version. Plain-text logs are insufficient for root-cause analysis in distributed spatial pipelines.
import json, logging, time
logger = logging.getLogger()
logger.setLevel(logging.INFO)
def log_transform(input_uri, output_uri, src_crs, dst_crs, duration_ms):
logger.info(json.dumps({
"event": "crs_transform",
"input_uri": input_uri,
"output_uri": output_uri,
"src_crs": src_crs,
"dst_crs": dst_crs,
"duration_ms": duration_ms,
"timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
}))
Implement OpenTelemetry for distributed tracing across orchestrator steps and compute functions. Span IDs that propagate from the ingestion trigger through to the STAC catalog write enable rapid root-cause analysis when spatial outputs deviate from expected bounding boxes or CRS.
Observability, Cost Control, and Fallback Patterns
Production spatial pipelines require continuous monitoring and well-defined fallback behaviour when platform quotas are reached.
Structured Logging and Distributed Tracing
Emit cost-per-tile metrics alongside spatial quality indicators (feature count, area, CRS authority code) in every log record. CloudWatch Metric Filters (AWS), Cloud Monitoring (GCP), and Application Insights (Azure) can ingest structured JSON and surface per-stage cost dashboards without custom log parsers.
Key metrics to track per pipeline stage:
| Metric | Source | Alert threshold |
|---|---|---|
Duration (ms) |
Lambda / Cloud Functions | > 80% of timeout ceiling |
MemoryUsed (MB) |
Lambda Insights | > 85% of allocated memory |
ConcurrentExecutions |
CloudWatch | > 70% of regional quota |
tmp_used_bytes |
Custom metric from os.statvfs |
> 80% of provisioned /tmp |
tiles_failed |
Custom metric | > 0 (trigger DLQ investigation) |
The thresholds matter more than the metric list. Alerting at 80% of the timeout ceiling gives roughly three minutes of warning on a 15-minute Lambda and two on a 10-minute Azure function — enough to raise memory or shrink the tile before the first hard failure, and not so tight that a slow object-store read pages someone at 3 a.m. The memory threshold is deliberately higher at 85% because raster memory use is spiky by nature; the signal you actually want is a sustained breach across several invocations, not a single peak. Emit these as structured metric records rather than parsing them out of log text: an embedded-metric format payload carries the dimensions (pipeline_stage, crs_authority, tile_size) alongside the value, so a per-stage cost dashboard falls out of the same log line that carries the trace ID.
One caution about out-of-memory failures specifically. A function killed for exceeding its memory allocation does not run an exception handler, does not flush a buffered log, and in many runtimes reports only a terse termination message. If the only record of a tile’s parameters is written at the end of the handler, an OOM leaves you with a failed invocation and no way to reproduce it. Log the input URI, window offsets, and allocated memory at the start of the handler, and the post-mortem becomes a query rather than an archaeology exercise.
Circuit Breakers for OOM and Timeout Fallback
When platform limits are reached, circuit breakers should automatically route workloads to managed container endpoints without breaking orchestration state. A practical pattern for AWS:
import boto3, os
lambda_client = boto3.client("lambda")
ecs_client = boto3.client("ecs")
def invoke_with_fallback(payload, task_def_arn, cluster_arn):
try:
response = lambda_client.invoke(
FunctionName=os.environ["TILE_PROCESSOR_FUNCTION"],
Payload=payload,
)
if response.get("FunctionError"):
raise RuntimeError(response["FunctionError"])
return response
except (RuntimeError, lambda_client.exceptions.TooManyRequestsException):
# Fall back to Fargate for heavy or throttled jobs
return ecs_client.run_task(
cluster=cluster_arn,
taskDefinition=task_def_arn,
overrides={"containerOverrides": [{"name": "processor", "command": [payload]}]},
launchType="FARGATE",
)
This hybrid approach preserves serverless cost benefits for bursty workloads while guaranteeing SLA compliance for heavy spatial transformations.
When to Stop Using Functions
Serverless is a good default for spatial work and a poor universal answer, and the boundary is unusually easy to state. Four signals say the workload has outgrown the model.
The first is chunking that has stopped being meaningful. Decomposition is legitimate when each chunk is an independent unit of the problem — a tile, a scene, a partition. When a job has been split only to fit under 15 minutes, so that chunk n must read the state chunk n−1 wrote to /tmp or to a scratch prefix, the orchestrator is emulating a long-running process and paying object-store latency for every variable it passes. That job belongs on Fargate or Cloud Run.
The second is a working set that will not fit. At 10,240 MB, Lambda covers most tile-level raster work, but a global mosaic, a large triangulation, or an ML model whose weights alone are several gigabytes does not decompose into memory-bounded pieces without changing the algorithm. GCP’s 32,768 MB ceiling buys headroom here; Azure Consumption’s 1,536 MB does not.
The third is sustained utilisation. Serverless pricing rewards idle time. A pipeline that runs continuously at high concurrency loses the economic argument well before it loses the technical one, and warm-keeping mechanisms — provisioned concurrency, minimum instances, always-ready instances — narrow that gap only by making the function look more like a container that costs more than one.
The fourth is a dependency stack that fights the packaging model. If the build spends more effort defeating the 250 MB unzipped limit than implementing the transformation, a container image is the honest answer. The useful test is whether removing the constraint would change the design: if it would, the constraint is doing harm rather than enforcing decomposition.
None of these argue for abandoning the event-driven structure. The most durable production designs keep the ingestion trigger, orchestration, and cataloguing stages serverless — where they are cheap, bursty, and stateless — and move only the compute stage to a container. The pipeline shape survives; the execution substrate for one stage changes.
Operational Checklist
Use this checklist before promoting a spatial pipeline to production:
- Chunking strategy — Validate that raster inputs split into tiles that fit within memory and timeout ceilings. Use 256×256 or 512×512 tile boundaries for rasters; apply H3 hexagons or quadkeys for vector spatial partitioning.
- Idempotency keys — Derive deterministic job IDs from input URI + processing parameters. Duplicate S3 notifications must not trigger redundant transformations.
- Environment variables —
GDAL_DATA,PROJ_LIB, andLD_LIBRARY_PATHmust be set explicitly in every function, not assumed from the runtime image. - CRS validation at ingestion — Reject or flag datasets with deprecated or ambiguous EPSG codes before they enter the pipeline. Silent projection errors compound across stages.
- Dead-letter queues — Attach DLQs to every SQS queue or SNS topic feeding compute functions. Alert on non-zero DLQ depth within five minutes.
- Concurrency reservation — Reserve a minimum concurrency allocation for tile-processor functions so burst fan-out from the orchestrator cannot starve them.
- Graceful degradation — Implement the circuit-breaker pattern so OOM or timeout failures route to Fargate/Cloud Run without breaking orchestration state.
- Cost tagging — Tag every resource with
pipeline_stageandspatial_job_type. Set budget alerts for unexpected GB-second or invocation-count spikes per stage. - Load testing — Run load tests with production-scale datasets (real GeoTIFF scenes, real Shapefile collections) using
locustork6, simulating concurrent ingestion events and realistic network latency. - Dependency pinning — Pin
rasterio,pyproj,shapely, andGDALto specific versions in all Lambda Layers and container images. Version drift between the layer and the application package is a leading cause of silent CRS resolution failures.
Frequently Asked Questions
What is the maximum timeout for AWS Lambda geospatial jobs?
AWS Lambda enforces a hard ceiling of 15 minutes per invocation. Jobs exceeding this must be decomposed into tile-sized chunks orchestrated by Step Functions, or offloaded to AWS Fargate for batch execution.
How much ephemeral /tmp storage does AWS Lambda provide?
The default is 512 MB, configurable up to 10,240 MB (10 GB) at an additional cost of $0.0000000309 per GB-second beyond the first 512 MB. See managing /tmp storage limits for GeoTIFF extraction for strategies to stay within the default allocation.
Why do cold starts take 3–8 seconds for Python GDAL stacks?
The platform must unpack the deployment archive, resolve shared libraries (libgdal, libproj, libgeos), and import Python modules before any business logic executes. Provisioned concurrency eliminates this by keeping warm execution environments pre-allocated at a fixed hourly cost.
How do I prevent the Python GIL from limiting raster throughput?
Use multiprocessing.ProcessPoolExecutor rather than threading. Allocate maximum Lambda memory (10 GB on AWS), spawn 2–4 worker processes, and pass data via memory-mapped numpy arrays to avoid duplication overhead.
Which provider should I choose for long-running raster jobs?
Timeout is usually the deciding row. GCP Cloud Functions 2nd gen allows 60 minutes per invocation, AWS Lambda 15, and Azure Functions on the Consumption plan 10. If a single unit of work genuinely cannot be decomposed below 15 minutes, GCP is the only one of the three that fits without moving to containers — and if it cannot be decomposed below 60, none of them do.
Why does my Lambda fail with a disk-full error only after it has been running fine for an hour?
Ephemeral /tmp storage persists for the life of the execution environment rather than the invocation, so intermediates accumulate across warm calls. A function writing 40 MB per invocation exhausts the 512 MB default after about a dozen calls on the same environment. Delete intermediates in a finally block, raise the allocation toward the 10,240 MB ceiling, or write through /vsimem instead of the filesystem.
Can I split a large geospatial stack across multiple Lambda Layers?
Not to escape the size limit. A function may attach at most five layers, and the 250 MB unzipped ceiling applies to the function package and all its layers combined. Layers help with build time, caching, and reuse across functions — not with total size. To exceed 250 MB you must deploy a container image, which raises the limit to 10 GB.
How do I stop a tiling fan-out from throttling the rest of my account?
Lambda’s default 1,000 concurrent executions is a regional quota shared by every function in the account, so a burst fan-out starves unrelated handlers. Set reserved concurrency on the tile processor so it cannot consume more than its share, bound MaxConcurrency on the orchestrator’s Map state, and put a queue between the trigger and the compute stage so backpressure produces a growing queue rather than a retry storm.
Topics in this section
- Cold Start Comparison: AWS vs GCP vs Azure for Python GDAL — Side-by-side cold-start behaviour for a Python GDAL/rasterio stack on AWS Lambda, GCP Cloud Functions 2nd…
- Cold Start Mapping for Python GDAL — Measure, isolate, and eliminate the 4–12 second initialization overhead of Python GDAL in serverless…
- Concurrency and Throttling for Tile Fan-Out — A single Sentinel-2 scene fans out into 4,096 tiles against a 1,000-invocation regional default
- Cost Modelling for Serverless Raster Pipelines — Build a defensible cost model for serverless raster processing: the cost-per-tile equation, why a bigger…
- Ephemeral Storage Comparison Across Serverless Platforms — AWS Lambda vs GCP Cloud Functions vs Azure Functions scratch disk for geospatial temp files: /tmp quotas…
- Ephemeral Storage Limits in AWS Lambda for Geospatial Processing — AWS Lambda’s /tmp directory caps at 10 GB
- IAM Security Boundaries for Cloud GIS — How to scope, enforce, and validate IAM permission boundaries for serverless geospatial pipelines on AWS…
- Memory and CPU Allocation for Raster Workloads — Size memory and CPU correctly for serverless raster processing on AWS Lambda, GCP Cloud Run, and Azure…
- Timeout Ceiling Comparison for Long-Running Geospatial Jobs — Compare the max execution time of AWS Lambda (15 min), GCP Cloud Functions 2nd gen (60 min), and Azure…
Related
- Cold Start Mapping for Python GDAL — initialisation sequence, shared-library resolution timings, and provisioned concurrency configuration
- Ephemeral Storage Limits in AWS Lambda —
/tmpquota management for GeoTIFF extraction and intermediate VRTs - Memory and CPU Allocation for Raster Workloads — tuning the memory/CPU ratio for windowed raster reads
- IAM Security Boundaries for Cloud GIS — per-stage role scoping, VPC endpoints, and KMS key policies
- SQS and Pub/Sub Queue Routing Strategies — queue-level fan-out patterns and dead-letter queue configuration for spatial jobs