Concurrency and Throttling for Tile Fan-Out
A single Sentinel-2 scene tiled at 512×512 produces roughly 4,096 windows, and every one of them wants its own function invocation the moment the orchestrator opens the fan-out — against an AWS Lambda default of 1,000 concurrent executions per account per region. The arithmetic is not close: the fan-out demands four times what the account can serve, so three quarters of those invocations are refused with TooManyRequestsException before any GDAL code runs. Every serverless tiling pipeline eventually meets this wall, and the ones that survive it do so because the fan-out width was chosen deliberately rather than inherited from the tile grid.
Throttling is the least visible failure mode in a spatial pipeline because it does not look like failure. A throttled invocation never starts, so there is no function log, no traceback, and no Errors datapoint. What you see instead is a job that used to finish in six minutes taking forty, a Step Functions execution history full of retries, and a bill that grew without a corresponding increase in output. This page maps where the ceilings sit on each platform, what the throttle looks like when it hits, and the three structural fixes — reserved concurrency, an orchestrator cap, and a queue in the middle — that turn an uncontrolled fan-out into a rate-limited one.
Why Fan-Out Width Is a Geospatial Problem, Not a Generic One
Most serverless workloads scale with user traffic, which arrives smoothly and is shaped by human behaviour. Spatial workloads scale with the geometry of the data, which arrives in one lump. When a satellite ground station delivers a day’s acquisitions, or a survey contractor uploads a 40 GB LiDAR delivery, the fan-out width is decided by a raster’s dimensions divided by a tile size — a number nobody chose with concurrency in mind.
Three properties make this worse than an ordinary traffic spike:
The fan-out is instantaneous. A Step Functions Distributed Map does not ramp. It reads the manifest, and it starts child executions as fast as the service allows — up to 10,000 in parallel. An S3 event storm behaves the same way: 4,096 ObjectCreated notifications land in the same second, and the Lambda service attempts to create 4,096 execution environments. There is no natural smoothing anywhere in the chain unless you put it there.
Each invocation is expensive to start. A tile worker carrying rasterio, pyproj and a native GDAL stack pays the cold start documented at 4–12 seconds unoptimised. When 1,000 environments spin up simultaneously, that is 1,000 simultaneous cold starts consuming GB-seconds while producing nothing. Throttling and cold starts compound: the throttled invocations retry, the retries land on environments that are still initialising, and the effective throughput of the fan-out drops below what a single sequential worker would achieve.
The tiles are not independent at the storage layer. Tiles from one scene write to one output prefix, index into one catalog table, and often read from one source COG. The fan-out is wide at the compute layer and narrow at the persistence layer, which means the concurrency that the compute quota allows is frequently more than the store behind it can absorb.
The practical consequence is that fan-out width must be a configured parameter of the pipeline, derived from the ceiling you are willing to occupy, and never a derived property of the tile grid. A 10,980×10,980 Sentinel-2 band tiled at 512 px yields 484 windows per band; at 256 px it yields 1,849. Multiply by 13 bands and the same scene demands either 6,292 or 24,037 invocations. The tile size chosen for I/O efficiency — see chunked I/O for large satellite imagery — silently sets the concurrency demand, and the two decisions are almost never made by the same person.
Where the Ceiling Sits on Each Platform
Every provider imposes at least two distinct limits: how many function instances can run at once, and how quickly you are allowed to reach that number. Confusing the two is the most common diagnostic error in throttling investigations — a workload can be far below its concurrency ceiling and still be throttled because it tried to get there too fast.
| Constraint | AWS Lambda | GCP Cloud Functions (2nd gen) | Azure Functions (Consumption) |
|---|---|---|---|
| Concurrency ceiling | 1,000 per account per region (soft, raisable) | 3,000 instances per project | 200 instances per function app |
| Scope of the ceiling | Shared by every function in the region | Shared by every function in the project | Per function app |
| Burst / ramp allowance | 500–3,000 initial burst by region, then +1,000 per 10 s per function | Instance-based autoscaling, tunable via --max-instances per function |
Scale controller adds instances gradually; no configurable burst |
| Per-instance concurrency | 1 request per environment | Up to 1,000 requests per instance (--concurrency) |
1 per function invocation on Consumption |
| Reserved allocation | Reserved concurrency (carved from the account pool) | --min-instances and --max-instances per service |
Premium plan only |
| Pre-warmed allocation | Provisioned concurrency (billed hourly) | --min-instances (billed while idle) |
preWarmedInstances, Premium plan only |
| Throttle signature | TooManyRequestsException, HTTP 429, Rate Exceeded |
HTTP 429, no available instance |
HTTP 429 with Retry-After, host threshold messages |
| Orchestrator ceiling | Step Functions Distributed Map: 10,000 parallel child executions | Workflows: parallel branches capped per execution | Durable Functions fan-out bounded by plan instances |
Three consequences follow directly from that grid.
On AWS the orchestrator out-scales the compute by an order of magnitude. Step Functions Distributed Map will happily run 10,000 parallel child executions; the Lambda function each child invokes has 1,000 concurrent executions available across the entire region, shared with every other function in the account. Left at defaults, the orchestrator’s job becomes generating throttles. MaxConcurrency on the Map state is not an optimisation — it is the mechanism by which the two limits are reconciled.
On GCP the ceiling is higher but the unit is different. Cloud Functions 2nd gen runs on the Cloud Run substrate, so one instance can serve many concurrent requests. Three thousand instances at a concurrency setting of 1 is 3,000 in-flight tiles; the same 3,000 instances at a concurrency setting of 4 is 12,000. For GDAL workloads the higher setting is usually wrong — each concurrent request wants its own GDAL block cache and its own share of the 32,768 MB memory ceiling — but the knob exists, and it means GCP’s quota is not directly comparable to the AWS number without stating the per-instance concurrency alongside it.
On Azure Consumption the ceiling is the binding constraint on every realistic fan-out. Two hundred instances against a 4,096-tile scene means the scene runs in 21 sequential waves at best. Combined with the 10-minute timeout and the 1,536 MB memory ceiling documented in timeout ceiling comparison for geospatial jobs, Azure Consumption is a plan for narrow fan-outs; wide ones belong on Premium or in containers.
Reserved, Provisioned, and Unreserved Concurrency
AWS exposes three concurrency concepts that are routinely conflated, and the distinction decides what a throttle means when you see one.
Unreserved concurrency is the default: every function draws from the same regional pool. It is efficient when functions burst at different times and catastrophic when they burst together. A tiling fan-out that consumes 1,000 concurrent executions leaves nothing for the metadata extractor, the catalog writer, or the API handler that a user is waiting on. The throttle lands on whichever function asks next, which is rarely the one causing the problem.
Reserved concurrency carves a fixed slice out of the account pool for one function. It does two things at once: it guarantees that function can always reach its reservation, and it caps that function so it can never exceed it. Setting reserved concurrency to 400 on the tile worker means the worker is both protected and contained — the remaining 600 stay available to everything else. AWS requires at least 100 unreserved concurrent executions to remain in the account, so reservations cannot sum to the full quota. The mechanics and the trade-offs are covered in reserved concurrency for geospatial Lambda fan-out.
Provisioned concurrency pre-initialises execution environments so they skip the cold start entirely. It is a latency control, not a capacity control — provisioned environments still count against the account quota, and requests beyond the provisioned count spill over to on-demand environments that cold-start normally. For a batch tile fan-out, provisioned concurrency is usually the wrong lever: the fan-out is not latency sensitive, and paying an hourly rate for warm environments that sit idle between scene deliveries is expensive. It earns its keep on the synchronous tile-serving path, as reducing Python GDAL cold starts with provisioned concurrency sets out.
The rule of thumb: reserved concurrency answers “how much of the account may this function take?”, provisioned concurrency answers “how fast does this function respond when it is called?” A batch fan-out needs the first. A tile server needs the second. Almost no function needs both.
Step-by-Step Implementation
Step 1: Measure the Fan-Out Width Before You Build It
The width is computable from the raster header alone, without reading a pixel. Do this at ingestion and record it, so the orchestrator knows how wide it is about to go before it goes.
import math
import os
import json
os.environ.setdefault("GDAL_DATA", "/opt/share/gdal")
os.environ.setdefault("PROJ_LIB", "/opt/share/proj")
os.environ.setdefault("LD_LIBRARY_PATH", "/opt/lib")
import rasterio
REGIONAL_QUOTA = 1000 # AWS Lambda default, per account per region
WORKER_RESERVATION = 400 # what this pipeline is allowed to occupy
def plan_fanout(src_uri: str, tile_px: int = 512) -> dict:
"""Return the fan-out width and the wave count it implies."""
with rasterio.open(src_uri) as src:
cols = math.ceil(src.width / tile_px)
rows = math.ceil(src.height / tile_px)
bands = src.count
tiles = cols * rows * bands
waves = math.ceil(tiles / WORKER_RESERVATION)
plan = {
"source": src_uri,
"tile_px": tile_px,
"tiles": tiles,
"grid": f"{cols}x{rows}x{bands}",
"reservation": WORKER_RESERVATION,
"regional_quota": REGIONAL_QUOTA,
"waves": waves,
"unbounded_demand_exceeds_quota": tiles > REGIONAL_QUOTA,
}
print(json.dumps(plan))
return plan
# A 10,980 x 10,980 Sentinel-2 band at 512 px:
# {"tiles": 484, "grid": "22x22x1", "waves": 2, ...}
# The same scene across all 13 bands:
# {"tiles": 6292, "grid": "22x22x13", "waves": 16, ...}
The waves number is the one to argue about in review. Sixteen waves of 400 tiles, each wave dominated by a cold start, is a very different pipeline from four waves of 1,600 — and the second one does not fit the account quota at all. Deciding this at ingestion also lets you route oversized deliveries to a container path rather than discovering the problem in the Map state, which is the same escape hatch the process a 10 GB GeoTIFF with Step Functions and Lambda recipe uses.
Step 2: Cap the Orchestrator and Reserve the Worker
Two settings, applied together. MaxConcurrency on the Map state bounds what the orchestrator asks for; reserved concurrency on the function bounds what it can be given. Setting only one of them leaves a failure mode open — a cap with no reservation can still be starved by a neighbouring function, and a reservation with no cap still generates throttles that must be retried.
from aws_cdk import Stack, Duration
from aws_cdk import aws_lambda as lambda_
from aws_cdk import aws_stepfunctions as sfn
from constructs import Construct
class TileFanoutStack(Stack):
def __init__(self, scope: Construct, cid: str, **kw) -> None:
super().__init__(scope, cid, **kw)
tile_worker = lambda_.Function(
self,
"TileWorker",
runtime=lambda_.Runtime.PYTHON_3_12,
handler="worker.handler",
code=lambda_.Code.from_asset("src/worker"),
memory_size=3008,
timeout=Duration.minutes(5),
# The cap AND the guarantee. 400 of the account's 1,000.
reserved_concurrent_executions=400,
environment={
"GDAL_DATA": "/opt/share/gdal",
"PROJ_LIB": "/opt/share/proj",
"LD_LIBRARY_PATH": "/opt/lib",
"GDAL_DISABLE_READDIR_ON_OPEN": "EMPTY_DIR",
"GDAL_CACHEMAX": "512",
},
)
# Distributed Map can run 10,000 children. We ask for 380 — deliberately
# under the 400 reservation, so the worker always has headroom for the
# retries of its own throttled tiles.
definition = {
"StartAt": "TileFanout",
"States": {
"TileFanout": {
"Type": "Map",
"MaxConcurrency": 380,
"ToleratedFailurePercentage": 2,
"ItemReader": {
"Resource": "arn:aws:states:::s3:getObject",
"ReaderConfig": {"InputType": "JSON"},
"Parameters": {
"Bucket.$": "$.manifest_bucket",
"Key.$": "$.manifest_key",
},
},
"ItemProcessor": {
"ProcessorConfig": {
"Mode": "DISTRIBUTED",
"ExecutionType": "STANDARD",
},
"StartAt": "ProcessTile",
"States": {
"ProcessTile": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"Parameters": {
"FunctionName": tile_worker.function_arn,
"Payload.$": "$",
},
"Retry": [
{
"ErrorEquals": [
"Lambda.TooManyRequestsException",
"Lambda.ServiceException",
],
"IntervalSeconds": 2,
"MaxAttempts": 6,
"BackoffRate": 2.0,
"JitterStrategy": "FULL",
}
],
"End": True,
}
},
},
"End": True,
}
},
}
sfn.CfnStateMachine(
self,
"TilePipeline",
role_arn=self.node.try_get_context("sfn_role_arn"),
definition_string=sfn.JsonPath.json_to_string(definition),
)
Asking for slightly less than the reservation is deliberate. If MaxConcurrency equals the reservation exactly, a retry of a throttled tile has nowhere to go — the function is saturated by the tiles that succeeded. Leaving 5% of headroom converts a self-perpetuating retry loop into a queue that drains. The retry and tolerance settings themselves are covered in detail in handling throttling errors in Step Functions Map state.
Step 3: Put a Queue Between the Orchestrator and the Workers
A MaxConcurrency cap works when one orchestrator owns the fan-out. It does not work when the fan-out comes from an S3 event storm, from several concurrent executions of the same state machine, or from two pipelines sharing one worker. In those cases the only reliable back-pressure is a queue: the orchestrator writes tile jobs at whatever rate it likes, and the event source mapping decides how fast they are consumed.
import boto3
sqs = boto3.client("sqs")
lam = boto3.client("lambda")
QUEUE_URL = "https://sqs.eu-west-1.amazonaws.com/123456789012/tile-jobs"
def attach_bounded_consumer(function_name: str, queue_arn: str) -> dict:
"""Create an SQS event source mapping that can never exceed 200 in flight."""
return lam.create_event_source_mapping(
EventSourceArn=queue_arn,
FunctionName=function_name,
BatchSize=1, # one tile per invocation
MaximumBatchingWindowInSeconds=0,
# The back-pressure control. Valid range is 2-1000; it caps how many
# concurrent invocations the poller will create, independent of how
# many messages are waiting.
ScalingConfig={"MaximumConcurrency": 200},
FunctionResponseTypes=["ReportBatchItemFailures"],
)
def enqueue_tiles(tiles: list[dict]) -> int:
"""Write the fan-out to the queue in batches of 10. Never throttles."""
sent = 0
for i in range(0, len(tiles), 10):
batch = tiles[i : i + 10]
sqs.send_message_batch(
QueueUrl=QUEUE_URL,
Entries=[
{
"Id": str(i + n),
"MessageBody": __import__("json").dumps(t),
# Deduplicate identical tile jobs from duplicate S3 events
"MessageAttributes": {
"tile_key": {"DataType": "String", "StringValue": t["key"]}
},
}
for n, t in enumerate(batch)
],
)
sent += len(batch)
return sent
With MaximumConcurrency set to 200, a 6,292-tile fan-out becomes a queue that drains at 200 tiles in flight regardless of how fast the messages arrived. The visible symptom of overload changes from a throttle to a queue depth, which is a far better signal: depth is monotonic, alarmable, and tells you exactly how far behind you are. Size the visibility timeout to at least six times the worker’s expected duration, as sizing SQS visibility timeout for long-running raster jobs explains, and attach a dead-letter queue so a poison tile does not recirculate forever.
Step 4: Match the Downstream Store to the Same Ceiling
Getting 400 workers running is only useful if what they write can accept 400 concurrent writers. In practice the downstream store is the real bottleneck more often than the function quota is.
- DynamoDB catalog writes. A provisioned table partition sustains 1,000 write capacity units per second; a single hot partition key — say,
scene_idas the partition key withtile_idas the sort key — puts every tile from one scene on one partition. Four hundred workers each writing a 2 KB STAC item is 800 WCU against a per-partition ceiling of 1,000, and the margin disappears the moment two scenes process together. Prefix the partition key with a hash bucket, or batch the catalog write into one item per wave. - A relational database. PostGIS with a 100-connection limit meets 400 concurrent workers and refuses 300 of them with
FATAL: sorry, too many clients already. Serverless functions and connection pools interact badly; use RDS Proxy, or have workers write to object storage and load the catalog in one batched step. - An external tile API. Third-party elevation, geocoding, or basemap APIs enforce their own rate limits, typically far below 400 requests per second. Their throttle arrives as an HTTP 429 inside your worker, which means it is billed as function duration while it waits and retries.
- S3 itself. S3 sustains 3,500
PUTrequests per second per prefix. A fan-out writing every tile unders3://bucket/scene-id/shares one prefix; at 400 concurrent workers each writing several artefacts,503 SlowDownbecomes reachable. Spread writes across prefixes derived from the tile index rather than the scene.
The general rule: the narrowest quota in the chain sets the fan-out width, and it is almost never the one printed in the compute service’s documentation.
Measurement and Verification
Throttling is diagnosed from three metrics that must be read together. Any one alone is misleading.
import boto3
from datetime import datetime, timedelta, timezone
cw = boto3.client("cloudwatch")
def throttle_report(function_name: str, minutes: int = 60) -> dict:
end = datetime.now(timezone.utc)
start = end - timedelta(minutes=minutes)
def total(metric: str, namespace: str = "AWS/Lambda", dims=None) -> float:
resp = cw.get_metric_statistics(
Namespace=namespace,
MetricName=metric,
Dimensions=dims if dims is not None
else [{"Name": "FunctionName", "Value": function_name}],
StartTime=start,
EndTime=end,
Period=60,
Statistics=["Sum"],
)
return sum(p["Sum"] for p in resp["Datapoints"])
invocations = total("Invocations")
throttles = total("Throttles")
errors = total("Errors")
report = {
"function": function_name,
"window_minutes": minutes,
"invocations": int(invocations),
"throttles": int(throttles),
"errors": int(errors),
"throttle_rate_pct": round(
100 * throttles / max(invocations + throttles, 1), 2
),
}
assert report["throttle_rate_pct"] < 1.0, (
f"Throttle rate {report['throttle_rate_pct']}% exceeds the 1% budget — "
f"reduce MaxConcurrency or raise the reservation."
)
return report
# Expected output from a correctly bounded fan-out:
# {'function': 'tile-worker', 'window_minutes': 60, 'invocations': 6292,
# 'throttles': 3, 'errors': 0, 'throttle_rate_pct': 0.05}
A CloudWatch Logs Insights query over the Step Functions execution history surfaces the same picture from the orchestrator’s side:
fields @timestamp, details.error, details.cause
| filter type = "TaskFailed" or type = "LambdaFunctionScheduleFailed"
| filter details.error like /TooManyRequests/
| stats count() as throttled_tasks by bin(1m)
| sort @timestamp desc
What to expect after applying the caps: ConcurrentExecutions should sit flat at just under the reservation for the duration of the fan-out, not spike and collapse. A sawtooth pattern in ConcurrentExecutions paired with a non-zero Throttles sum is the signature of an uncapped orchestrator fighting the quota — it climbs to the ceiling, gets refused, backs off, and climbs again. A flat plateau is what a bounded fan-out looks like.
Track four numbers per run and store them alongside the job record:
| Metric | Source | Healthy value |
|---|---|---|
Throttles |
AWS/Lambda, per function |
0, or under 1% of invocations |
ConcurrentExecutions (max) |
AWS/Lambda, per function |
90–98% of the reservation |
ApproximateAgeOfOldestMessage |
AWS/SQS, per queue |
Under 2× worker duration |
ExecutionThrottled |
AWS/States, per state machine |
0 |
Failure Modes and Debugging
1. Lambda.TooManyRequestsException — Rate Exceeded (HTTP 429)
The account-level invocation throttle. The Step Functions execution history records it as a LambdaFunctionScheduleFailed event with "error": "Lambda.TooManyRequestsException" and a cause containing Rate Exceeded.. No environment was created, so nothing appears in the function’s own log group — this is the error that looks like nothing happened. Fix: lower MaxConcurrency, raise the reservation, or add the retry block with JitterStrategy: FULL. Confirm by checking whether the account’s total ConcurrentExecutions was at 1,000 at the timestamp, which is a different metric from the function’s own.
2. States.ThrottledEvent — the orchestrator itself is throttled
Step Functions meters state transitions independently of Lambda invocations. Standard workflows allow a bucket of state transitions per second per account; a Distributed Map running thousands of children with several states each can exhaust it. The symptom is a Map state that stalls with no failing child. Fix: reduce the number of states inside the ItemProcessor — collapse a Pass plus a Task into one Task — or move the inner workflow to an EXPRESS execution type, which is not metered against the standard transition bucket.
3. ProvisionedThroughputExceededException from the catalog table
Raised by DynamoDB when a partition’s write capacity is exhausted. In a tile pipeline this almost always means a hot partition rather than an undersized table: every tile from one scene shares the same partition key. The AWS SDK retries this internally, so the first visible symptom is worker duration climbing from 900 ms to 8 s while doing the same work — the retries are billed as function time. Fix: hash-prefix the partition key, or aggregate catalog writes so one worker writes one item for a batch of tiles.
4. An error occurred (SlowDown) when calling the PutObject operation
S3 returns 503 SlowDown when request rate against a prefix exceeds what the partition can serve. Wide fan-outs that write every output under a single scene prefix reach it at a few hundred concurrent writers. Fix: distribute outputs across prefixes by tile index (/tiles/{z}/{x}/{y}/) rather than by scene, which also improves read performance for the tile server later. The GDAL-side symptom is a CPLE_AppDefined error from /vsis3/ rather than a boto3 exception.
5. The request was aborted because there was no available instance (GCP)
Cloud Functions 2nd gen and Cloud Run return HTTP 429 with this message when a burst exceeds the instance ceiling — either the per-function --max-instances or the 3,000-instance project quota. Unlike the AWS throttle, this one is visible in the request log with a 429 status, so it is easier to spot. Fix: raise --max-instances on the function if project headroom exists, or lower the publisher rate. The Azure equivalent on Consumption is a 429 with a Retry-After header, or a host-level Host thresholds exceeded: Connections message when the 200-instance app ceiling is reached.
Cost and Scaling
Throttling is not free. Every throttled invocation that is retried costs a request charge, and every retry that lands on a cold environment costs the full initialisation duration before doing any work.
Take a 6,292-tile fan-out on a 3,008 MB function with a 4-second cold start and a 900 ms warm execution. Run uncapped against a 1,000-concurrency account with five other functions active, and roughly 40% of invocations throttle on the first attempt. Six retry attempts with exponential backoff eventually land all of them, but the pipeline has now issued around 10,400 invocation requests instead of 6,292, and a disproportionate share of them cold-started because the backoff spread them across fresh environments. At $0.20 per million requests the extra requests are negligible; the extra 4,100 cold starts at 4 s × 3,008 MB is roughly 48,000 GB-seconds, or about $0.80 per scene at $0.0000166667 per GB-second — on top of a wall-clock time that has roughly tripled.
Capped at 380 concurrent, the same fan-out issues 6,292 invocations, holds environments warm across 17 waves so only the first wave cold-starts, and completes in a predictable window. The compute cost falls by around 35% and the variance in completion time nearly disappears — which matters more than the money, because a predictable duration is what lets you set a meaningful alarm.
Two scaling notes:
Raising the quota is legitimate and usually free. The 1,000-concurrency limit is a soft quota; a Service Quotas request for 5,000 or 10,000 is routinely approved for accounts with a billing history. Raise it before you architect around it — but raise the ceiling and still cap the fan-out, because an unbounded fan-out against a 10,000 ceiling simply moves the collision to the downstream store.
Concurrency and memory trade against each other. Every concurrent environment holds its own memory allocation, and cost is GB-seconds. Four hundred workers at 3,008 MB is 1,203 GB in flight; 200 workers at 6,016 MB is the same. If per-tile work is CPU-bound, the second configuration finishes sooner for the same money because CPU scales with memory — the mechanics are in memory and CPU allocation for raster workloads. Halving the fan-out width and doubling the memory is frequently the cheapest way out of a throttling problem, and it is the one nobody tries first.
For a full side-by-side of what each provider will let you do here, see comparing concurrency quotas across AWS, GCP, and Azure.
Frequently Asked Questions
What is the default concurrency limit for AWS Lambda?
1,000 concurrent executions per account per region. It is a soft limit and can be raised through Service Quotas. On top of it sits a burst allowance of 500–3,000 concurrent executions depending on the region, which governs how fast you may reach the ceiling rather than how high the ceiling is. A workload can be well under 1,000 and still be throttled because it tried to get there in one step.
How many parallel child executions can a Step Functions Distributed Map run?
Up to 10,000. That is ten times the default Lambda concurrency in the same region, so a Distributed Map left at its default MaxConcurrency will throttle the function it invokes long before it reaches its own ceiling. Set MaxConcurrency to slightly less than the function’s reserved concurrency so retries have headroom.
Does provisioned concurrency prevent throttling?
No. Provisioned concurrency removes cold starts for a fixed number of environments, but those environments still count against the account’s concurrent execution quota, and requests beyond the provisioned count spill over to on-demand environments. Use reserved concurrency to control capacity and provisioned concurrency to control latency.
Why do my throttled invocations produce no error logs?
Because a throttled invocation is refused before an execution environment is created. There is no handler, no log stream, and no Errors metric datapoint — only a Throttles datapoint and, if an orchestrator issued the call, a LambdaFunctionScheduleFailed event in its execution history. Alarm on Throttles explicitly; an error-rate alarm will never fire for this failure.
What is the fastest way to stop a retry storm in progress?
Set reserved concurrency on the offending function to a small number, such as 10. This caps it immediately and returns the rest of the account pool to every other function, which stops the collateral throttling of unrelated workloads. The fan-out will run slowly, but it will run, and the pipeline stops taking down its neighbours while you fix the root cause.
Guides in this topic
- Comparing Concurrency Quotas Across AWS, GCP, and Azure — The same 6,292-tile spatial fan-out costed against AWS Lambda’s 1,000 regional executions, GCP Cloud…
- Handling Throttling Errors in Step Functions Map State — Retry Lambda.TooManyRequestsException with jittered exponential backoff, let ToleratedFailurePercentage…
- Reserved Concurrency for Geospatial Lambda Fan-Out — Carve a guaranteed, capped slice of the 1,000-execution account pool for a tile worker so a raster fan-out…
Related
- Reserved Concurrency for Geospatial Lambda Fan-Out — carving a guaranteed, capped slice of the account pool for the tile worker
- Handling Throttling Errors in Step Functions Map State — retry blocks,
ToleratedFailurePercentage, and the 10,000-child ceiling - Comparing Concurrency Quotas Across AWS, GCP, and Azure — the same spatial fan-out costed against all three providers
- SQS and Pub/Sub Queue Routing Strategies — the queue layer that converts a throttle into a queue depth
- Partitioning a GeoTIFF into Step Functions Map Tiles — the manifest generation step that decides the fan-out width
Back to Serverless Geospatial Architecture & Platform Limits