Reserved Concurrency for Geospatial Lambda Fan-Out

Set ReservedConcurrentExecutions to 400 on the tile worker and the function is both guaranteed 400 of the account’s 1,000 concurrent executions and prevented from ever exceeding them. The remaining 600 stay available to the metadata extractor, the catalog writer, and the synchronous tile API, none of which can now be starved by a scene delivery. AWS enforces a floor of 100 unreserved executions, so reservations across all functions can total at most 900 on a default quota — plan the budget as a whole, not one function at a time.

Why a Reservation Is the Right Control

Without a reservation, every function in an AWS account draws from one shared regional pool on a first-come basis. That works while functions burst at different times. It fails the moment a raster fan-out starts, because a fan-out is not a burst — it is a sustained occupation of every available execution slot for as long as the scene takes to process. The concurrency and throttling for tile fan-out overview shows the arithmetic: one Sentinel-2 scene across thirteen bands wants 6,292 invocations against a pool of 1,000.

The damage is rarely to the fan-out itself. The tile worker gets its 1,000 slots, throttles the excess, retries, and eventually finishes. What breaks is everything else: the metadata extractor that should have run on the next upload is refused, the catalog writer cannot record what has already been produced, and a user-facing map request times out because the tile server could not get an execution environment. A batch job took down an interactive service, and nothing in the batch job’s own metrics says so.

A reservation resolves this by making the allocation explicit and symmetric. The reserved slice is unavailable to any other function, so the worker is protected from noisy neighbours; and the worker cannot draw outside its slice, so the neighbours are protected from it. That symmetry is the whole point — a control that only guaranteed capacity would just move the starvation somewhere else.

Reserved concurrency budget across the functions of a tiling pipelineFive meters against the same 1,000-execution regional pool: the tile worker reserved at 400, the synchronous tile API at 200, the metadata extractor at 60, the catalog writer at 40, and an unreserved remainder of 300 shared by every other function in the account.The 1,000-execution account pool as named line itemsTile workerreserved — the raster fan-out400 / 1,000Tile APIreserved — synchronous, user-facing200 / 1,000Metadata extractorreserved — header reads on ingest60 / 1,000Catalog writerreserved — STAC item writes40 / 1,000Unreserved pooleverything else; AWS floor is 100300 / 1,000Reservations may total at most 900 against a default 1,000 quota, because AWS keeps 100 unreserved executions in the account at all times.
Every slice is both a floor and a ceiling. The unreserved 300 is not headroom for the fan-out — it is what every function without a reservation shares, including ones other teams deploy.

Note the shape of that budget. The unreserved remainder is not spare capacity for the fan-out to grow into; it is the pool that every unreserved function in the account shares, including ones deployed by other teams. Treating the account quota as a budget with named line items — rather than as a number that nobody owns — is what makes throttling diagnosable when it does occur.

Prerequisites

  • A measured peak demand. Compute the fan-out width from the raster header before choosing a number, as the partitioning a GeoTIFF into Step Functions Map tiles step does. Reserving a number you guessed produces throttles you cannot explain.
  • Deploy-time IAM permissions: lambda:PutFunctionConcurrency, lambda:GetFunctionConcurrency, and lambda:DeleteFunctionConcurrency for the deploying principal. lambda:GetAccountSettings is needed for the verification step.
  • An inventory of existing reservations. Run aws lambda list-functions and check each function’s concurrency before adding another; the account floor is enforced across all of them together, and the error surfaces at deploy time on whichever function happens to be last.
  • A memory decision already made. Reserved concurrency multiplies memory: 400 workers at 3,008 MB hold 1,203 GB in flight. Settle the memory tier first using memory and CPU allocation for raster workloads, because doubling memory to halve duration also halves the reservation you need.
  • Environment variables pinned in the function configuration, not in code: GDAL_DATA=/opt/share/gdal, PROJ_LIB=/opt/share/proj, LD_LIBRARY_PATH=/opt/lib. A reservation guarantees environments; it does not guarantee they initialise correctly, and 400 simultaneous CPLE_OpenFailed failures are an expensive way to discover a missing path.
  • A downstream store sized to the same number. The reservation you can serve is bounded by what the catalog table, the database pool, or the external API can accept concurrently.

Sizing the Reservation

The reservation is not “as much as the fan-out wants”. It is the smallest number that keeps the pipeline inside its completion window, because every execution slot it holds is a slot no other function can use.

Work it out from three measurements you already have. Let T be the tiles in a scene, d the mean warm duration of one tile, and W the wall-clock window the scene must finish in. The reservation R must satisfy R >= T * d / W. A 6,292-tile scene at 900 ms per tile that must complete in 20 minutes needs 6292 * 0.9 / 1200, or about 5 concurrent workers — which is a startling result the first time you compute it, and it is usually correct. Most tile fan-outs are given hundreds of execution slots to satisfy an impatience nobody wrote down.

Two corrections push that floor upward. Cold starts are not in d: each wave that adds new environments pays the 4–12 seconds mapped in cold start mapping for Python GDAL, so a reservation small enough to produce many waves spends a large fraction of its budget initialising. And scenes rarely arrive alone — a morning delivery of eight scenes multiplies T by eight unless the orchestrator serialises them.

A workable procedure: compute the floor from the formula, multiply by the number of scenes that can plausibly overlap, round up to a round number, and then check the result against what the catalog table and the source bucket will accept. Reserve that. Revisit it when the tile size changes, because halving the tile edge quadruples T while roughly quartering d, leaving the floor unchanged but multiplying the cold-start overhead by four.

Reserved Versus Provisioned: The Distinction That Matters

Reserved concurrency compared with provisioned concurrencyTwo panels. Reserved concurrency is an allocation control that is free, guarantees a floor and enforces a ceiling, belongs to the function as a whole, and suits a batch tile fan-out. Provisioned concurrency is a latency control billed hourly, removes cold starts, attaches to a published version or alias, and suits a synchronous tile server.Two settings, two different problemsReserved concurrencyAllocation: how much of the account this function may takeFree — you pay only for invocations that actually runGuarantees a floor and enforces a ceiling at the samevalueBelongs to the function, shared by every version and aliasThe lever for a batch tile fan-outProvisioned concurrencyLatency: how fast this function answers a cold requestBilled hourly whether or not any request arrivesRemoves cold starts; requests beyond the count spill overAttaches to a published version or an alias, never to$LATESTThe lever for a synchronous tile serverSet both and the provisioned environments come out of the reservation: 50 provisioned on a function reserved at 400leaves 350 on-demand slots, not 400.
Applying both to the same function is almost always a mistake made after one search returned two answers. A batch fan-out needs the first; a tile server needs the second.

The two settings are often applied together by mistake, usually after a search for “Lambda concurrency” returned both. They solve different problems. Reserved concurrency is an allocation control and costs nothing; provisioned concurrency is a latency control and bills hourly whether or not the environments are used. For a batch fan-out triggered by scene deliveries a few times a day, provisioned concurrency spends money keeping environments warm through the many hours when no scene arrives — the case for it, on the synchronous path, is made in reducing Python GDAL cold starts with provisioned concurrency.

One interaction is worth knowing: provisioned concurrency is drawn from a function’s reserved concurrency when a reservation exists. Provisioning 50 environments on a function reserved at 400 leaves 350 on-demand slots, not 400. Configuring provisioned concurrency higher than the reservation is rejected outright.

Implementation

The reservation belongs in infrastructure code alongside the function, so the budget is reviewable in a diff. This Terraform module allocates the account pool explicitly and fails the plan if the arithmetic does not leave the unreserved floor intact.

hcl 
# tile_fanout.tf — the account's concurrency budget as one reviewable object.

variable "account_concurrency_quota" {
  description = "Regional concurrent execution quota. 1000 is the AWS default."
  type        = number
  default     = 1000
}

locals {
  # Named line items. Everything not listed here draws from the unreserved pool.
  reservations = {
    tile_worker        = 400 # the raster fan-out
    metadata_extractor = 60  # header reads on ingest
    catalog_writer     = 40  # STAC item writes
    tile_api           = 200 # synchronous, user-facing
  }

  reserved_total = sum(values(local.reservations))
  unreserved     = var.account_concurrency_quota - local.reserved_total
}

# AWS refuses any reservation that would leave fewer than 100 unreserved
# executions in the account. Catch it at plan time, not at apply time.
resource "null_resource" "concurrency_budget_guard" {
  lifecycle {
    precondition {
      condition     = local.unreserved >= 100
      error_message = "Reservations total ${local.reserved_total} of ${var.account_concurrency_quota}; only ${local.unreserved} unreserved would remain and AWS requires 100."
    }
  }
}

resource "aws_lambda_function" "tile_worker" {
  function_name = "tile-worker"
  role          = aws_iam_role.tile_worker.arn
  handler       = "worker.handler"
  runtime       = "python3.12"
  filename      = "build/worker.zip"

  memory_size = 3008
  timeout     = 300

  # The guarantee and the cap, in one attribute.
  reserved_concurrent_executions = local.reservations.tile_worker

  layers = [aws_lambda_layer_version.gdal.arn]

  environment {
    variables = {
      GDAL_DATA                    = "/opt/share/gdal"
      PROJ_LIB                     = "/opt/share/proj"
      LD_LIBRARY_PATH              = "/opt/lib"
      GDAL_DISABLE_READDIR_ON_OPEN = "EMPTY_DIR"
      GDAL_CACHEMAX                = "512"
      PROJ_NETWORK                 = "OFF"
      # Publish the reservation to the handler so it can log the ratio it
      # is running at without a control-plane API call per invocation.
      WORKER_RESERVATION           = local.reservations.tile_worker
    }
  }
}

resource "aws_cloudwatch_metric_alarm" "tile_worker_saturated" {
  alarm_name          = "tile-worker-at-reservation"
  namespace           = "AWS/Lambda"
  metric_name         = "ConcurrentExecutions"
  dimensions          = { FunctionName = aws_lambda_function.tile_worker.function_name }
  statistic           = "Maximum"
  period              = 60
  evaluation_periods  = 5
  # Fire when the worker sits at 98% of its slice for five minutes: the
  # fan-out is wider than the reservation and tiles are being refused.
  threshold           = local.reservations.tile_worker * 0.98
  comparison_operator = "GreaterThanOrEqualToThreshold"
  treat_missing_data  = "notBreaching"
}

resource "aws_cloudwatch_metric_alarm" "tile_worker_throttled" {
  alarm_name          = "tile-worker-throttles"
  namespace           = "AWS/Lambda"
  metric_name         = "Throttles"
  dimensions          = { FunctionName = aws_lambda_function.tile_worker.function_name }
  statistic           = "Sum"
  period              = 300
  evaluation_periods  = 1
  threshold           = 10
  comparison_operator = "GreaterThanThreshold"
  treat_missing_data  = "notBreaching"
}
Five steps to apply reserved concurrency to a tile workerFive ordered steps: inventory every existing reservation in the account, size the slice from peak tiles in flight, apply the value in infrastructure code rather than the console, cap the orchestrator at about ninety-five per cent of the reservation, and alarm on both the throttle count and sustained saturation.Rolling the reservation out without a surprise1Inventory the account poolEvery existing reservation, across all functionsGetAccountSettings2Size the sliceFrom peak tiles in flight, not tiles per scene400 of 1,0003Apply in code, not consoleThe budget belongs in a reviewable diffterraform apply4Cap the orchestrator below itMaxConcurrency at about 95% of the reservationMaxConcurrency 3805Alarm on both signalsThrottle count, and sitting at the reservation5 min at 98%
Step four is the one most often skipped, and it is the one that decides whether a retried tile has anywhere to land.

Two alarms rather than one is deliberate. Throttles tells you tiles were refused; ConcurrentExecutions at the reservation tells you the fan-out is running exactly as wide as it is allowed to and would go wider if permitted. The second fires before the first when the orchestrator is correctly capped just under the reservation, which gives you warning rather than incident.

Verification

Reserved concurrency is applied on the control plane, so verify it there rather than inferring it from behaviour. This probe reads the account settings and every function’s reservation, then asserts the budget adds up.

python 
import boto3

lam = boto3.client("lambda", region_name="eu-west-1")


def audit_concurrency_budget() -> dict:
    settings = lam.get_account_settings()["AccountLimit"]
    quota = settings["ConcurrentExecutions"]
    unreserved_floor = settings["UnreservedConcurrentExecutions"]

    reserved = {}
    paginator = lam.get_paginator("list_functions")
    for page in paginator.paginate():
        for fn in page["Functions"]:
            name = fn["FunctionName"]
            cfg = lam.get_function_concurrency(FunctionName=name)
            if "ReservedConcurrentExecutions" in cfg:
                reserved[name] = cfg["ReservedConcurrentExecutions"]

    total = sum(reserved.values())
    report = {
        "account_quota": quota,
        "reserved_total": total,
        "unreserved_remaining": quota - total,
        "unreserved_reported_by_aws": unreserved_floor,
        "reservations": dict(sorted(reserved.items(), key=lambda kv: -kv[1])),
    }

    assert report["unreserved_remaining"] >= 100, (
        f"Only {report['unreserved_remaining']} unreserved executions remain; "
        "AWS requires 100 and unreserved functions will throttle."
    )
    return report


if __name__ == "__main__":
    import json
    print(json.dumps(audit_concurrency_budget(), indent=2))

Expected output on a correctly budgeted account:

code 
{
  "account_quota": 1000,
  "reserved_total": 700,
  "unreserved_remaining": 300,
  "unreserved_reported_by_aws": 300,
  "reservations": {
    "tile-worker": 400,
    "tile-api": 200,
    "metadata-extractor": 60,
    "catalog-writer": 40
  }
}

Then confirm behaviour during a real fan-out. ConcurrentExecutions for tile-worker should form a flat plateau just under 400 for the duration of the run and return to zero afterwards, with Throttles at or near zero. A sawtooth instead of a plateau means the orchestrator is asking for more than the reservation and backing off repeatedly — lower MaxConcurrency on the Map state to about 95% of the reservation, as handling throttling errors in Step Functions Map state describes.

Gotchas

  • Setting a reservation of zero disables the function. ReservedConcurrentExecutions = 0 is valid and means “this function may never run” — every invocation is throttled. It is a genuinely useful emergency switch for stopping a runaway fan-out without deleting anything, but a Terraform variable that defaults to 0 because it was left unset will silently take a production function offline. Distinguish unset from zero explicitly in any module that exposes the value.

  • The reservation applies to the function, not to a version or alias. Unlike provisioned concurrency, which attaches to a published version or alias, reserved concurrency is a property of the function as a whole and is shared across every version and alias of it. A blue/green deployment does not get two independent slices — the old and new versions compete inside the same reservation during the cut-over, which is worth accounting for if the cut-over happens mid-fan-out.

  • Reservations do not cross regions or accounts. The 1,000-execution quota is per account per region. A multi-region tiling deployment has an independent budget in each region, and raising the quota in eu-west-1 does nothing for us-east-1. Conversely, moving a fan-out to a second region is a legitimate way to double capacity without a quota request — provided the source imagery is readable from both, which the tuning HTTP range requests for COG reads on S3 guidance affects directly through cross-region read latency.

  • An SQS event source mapping respects the reservation but reports the throttle as its own. When a queue-triggered worker is capped by its reservation, the poller backs off and the messages return to the queue, so the visible symptom is a rising ApproximateAgeOfOldestMessage rather than a function-level error. Set ScalingConfig.MaximumConcurrency on the mapping to a value below the reservation so the poller self-limits instead of discovering the cap by being refused.

Frequently Asked Questions

Does reserved concurrency cost anything?

No. It is a free allocation control; you pay only for invocations that actually run, at the usual per-request and GB-second rates. Provisioned concurrency is the one with an hourly charge, because it holds initialised execution environments whether or not requests arrive.

Can I reserve all 1,000 concurrent executions?

No. AWS requires at least 100 unreserved concurrent executions to remain in the account, so reservations can total at most 900 against a default 1,000 quota. Exceeding that fails with InvalidParameterValueException and a message naming the shortfall. Raising the account quota through Service Quotas raises the ceiling but not the floor — the 100 unreserved executions are required regardless of quota size.

What happens to invocations beyond a function’s reserved concurrency?

They are throttled with TooManyRequestsException, exactly as if the account quota had been hit — the reservation caps as well as guarantees. Synchronous callers see the 429 immediately; asynchronous invocations and event-source mappings are retried by the invoking service, so the fan-out still completes, just more slowly.

Should the tile worker’s reservation match the Map state’s MaxConcurrency?

No — set MaxConcurrency to roughly 95% of the reservation. If they are equal, a retry of a throttled tile has no free slot to land in, because the tiles that succeeded are occupying the entire reservation. The 5% gap is what lets the retry queue drain instead of recirculating.

Back to Concurrency and Throttling for Tile Fan-Out