Sizing SQS Visibility Timeout for Long-Running Raster Jobs

Q: How do I handle raster jobs whose duration varies widely?

Use a heartbeat: run a background timer that calls ChangeMessageVisibility to extend the invisibility window while the job reports progress. This lets you keep a modest base timeout for typical jobs while still protecting the rare large scene that runs much longer, up to the 15-minute Lambda ceiling.

Set the SQS visibility timeout to at least six times the Lambda function timeout when SQS is an event source mapping, and for jobs whose duration varies, extend the window at runtime with ChangeMessageVisibility heartbeats instead of over-provisioning a fixed value. A raster job that runs longer than its visibility timeout is silently redelivered mid-processing, so a second worker begins reprojecting the same scene while the first is still writing — doubling cost and corrupting mosaic output. Because a Lambda function cannot exceed the 15-minute ceiling, size visibility from the chunk duration and let maxReceiveCount route a truly stuck message to the dead-letter queue.

Context

Raster processing is the workload most likely to outlast a naively chosen visibility timeout. Reprojecting a Sentinel-2 granule, warping a 10 GB GeoTIFF, or resampling overviews with rasterio and GDAL routinely runs for minutes, not milliseconds — and the duration swings with scene size, band count, and cold-start penalty. The SQS and Pub/Sub Queue Routing Strategies layer feeds these jobs from a queue, and the visibility timeout is the single setting that decides whether a slow job runs once or twice.

When an SQS message is received, it becomes invisible for the visibility timeout window rather than being deleted. The consumer is expected to finish its work and explicitly delete the message before that window closes. If processing runs long, the message reappears in the queue, a second poll delivers it to another worker, and now two invocations warp the same raster. Beyond wasted compute, the duplicate write can interleave with the first and corrupt a partially written Cloud-Optimized GeoTIFF — the ordering hazard that guaranteeing order with SQS FIFO for sequential tile jobs is meant to prevent, reintroduced through a sizing mistake.

Every redelivery also increments ApproximateReceiveCount. A job that is merely slow, not broken, marches toward maxReceiveCount and is eventually dead-lettered as if it had failed — one of the more confusing failure modes to debug in a raster pipeline.

Prerequisites

Runtime: Python 3.11+ on AWS Lambda with an SQS event source mapping, or a self-managed poller running the same boto3 calls.
Dependencies: boto3>=1.34.0; rasterio>=1.3.9 for the raster work being timed.
Baseline metric: A measured p99 processing duration for the slowest scene, including cold start. Sizing from an average rather than the tail is the most common cause of intermittent duplicates.
IAM: sqs:ReceiveMessage, sqs:DeleteMessage, sqs:ChangeMessageVisibility, and sqs:GetQueueAttributes on the queue ARN. The heartbeat pattern is impossible without sqs:ChangeMessageVisibility.
Environment variables: RASTER_QUEUE_URL and FUNCTION_TIMEOUT_SECONDS so the handler can compute its own heartbeat interval rather than hardcoding it.

How the Ceiling Constrains the Timeout

Sizing does not happen in a vacuum — the platform’s own limits bound both ends. Two Lambda constraints interact directly with the visibility timeout: the function timeout can never exceed 15 minutes, and the SQS visibility timeout has a hard maximum of 12 hours. The practical rule for an event source mapping is to keep visibility at six times the function timeout, capped by the 12-hour maximum.

Platform	Redelivery control	Max in-flight window	Function/execution ceiling
AWS Lambda + SQS	Visibility timeout	12 hours	15 min (10,240 MB, 10 GB /tmp)
GCP Cloud Functions 2nd gen + Pub/Sub	Ack deadline	10 min per extension, 1 h total	60 min (32 GB)
Azure Functions Consumption + Service Bus	Lock duration	5 min per renewal	10 min (1.5 GB)

The asymmetry is the whole point: AWS lets a message stay invisible for 12 hours, but the function that processes it dies at 15 minutes. You cannot solve a slow raster job by raising the function timeout past 15 minutes — instead you chunk raster jobs to fit the 15-minute Lambda ceiling, size each chunk to run comfortably under the function timeout, then set visibility from the chunk duration. GCP’s Pub/Sub ack deadline behaves similarly, extendable up to a total of an hour, while its function ceiling is a more generous 60 minutes, as the timeout ceiling comparison for geospatial jobs lays out in full.

Implementation

For jobs with predictable duration, the six-times rule and a chunking strategy are enough. For jobs whose duration varies with scene size, add a heartbeat: a background thread that periodically extends the message’s invisibility while the main thread makes progress. The handler below keeps a modest base timeout for typical scenes and only extends for the occasional large granule, up to a hard cap that respects the function ceiling.

python

import os
import threading

import boto3
import rasterio  # the raster work whose duration we are protecting

sqs = boto3.client("sqs")
RASTER_QUEUE_URL = os.environ["RASTER_QUEUE_URL"]
FUNCTION_TIMEOUT = int(os.environ["FUNCTION_TIMEOUT_SECONDS"])  # e.g. 900 (15 min)

# Extend the window in generous steps, but never claim more than we could use.
HEARTBEAT_INTERVAL = 30          # seconds between visibility extensions
HEARTBEAT_EXTENSION = 120        # seconds of invisibility added per heartbeat


class VisibilityHeartbeat:
    """Extend a message's visibility timeout on a timer until the job completes.

    Guards a long raster job from redelivery without over-provisioning the
    base visibility timeout for every message in the queue.
    """

    def __init__(self, receipt_handle: str):
        self._receipt_handle = receipt_handle
        self._stop = threading.Event()
        self._thread = threading.Thread(target=self._run, daemon=True)

    def _run(self) -> None:
        # Stop extending once we near the function ceiling — a job that has run
        # this long should be chunked, not kept alive indefinitely.
        while not self._stop.wait(HEARTBEAT_INTERVAL):
            try:
                sqs.change_message_visibility(
                    QueueUrl=RASTER_QUEUE_URL,
                    ReceiptHandle=self._receipt_handle,
                    VisibilityTimeout=HEARTBEAT_EXTENSION,
                )
            except sqs.exceptions.ClientError:
                # Receipt handle expired or message already deleted — stop quietly
                break

    def __enter__(self):
        self._thread.start()
        return self

    def __exit__(self, *exc):
        self._stop.set()
        self._thread.join(timeout=5)


def handler(event, context):
    for record in event.get("Records", []):
        receipt_handle = record["receiptHandle"]

        # Heartbeat runs for the lifetime of this raster job only
        with VisibilityHeartbeat(receipt_handle):
            with rasterio.open(record_source_uri(record)) as src:
                warp_and_write(src)  # the long-running reproject / resample

        # Delete only after a clean completion; an exception skips this and the
        # message returns to the queue for a bounded retry via maxReceiveCount.
        sqs.delete_message(QueueUrl=RASTER_QUEUE_URL, ReceiptHandle=receipt_handle)

The heartbeat and maxReceiveCount play complementary roles: the heartbeat protects a healthy-but-slow job from redelivery, while maxReceiveCount guarantees a genuinely stuck job still terminates in the dead-letter queue rather than heartbeating forever. Configure both together, as the dead letter queue setup for failed jobs describes, so slow and broken are handled by different mechanisms.

Verification

Set the base timeout to six times the function timeout and confirm it took effect, then run a scene near your p99 duration and check that it is received exactly once.

bash

# Function timeout 900s (15 min) -> visibility 5400s (90 min), capped at 12h max
aws sqs set-queue-attributes \
  --queue-url "$RASTER_QUEUE_URL" \
  --attributes VisibilityTimeout=5400

# Confirm the setting and the current in-flight count during a test run
aws sqs get-queue-attributes \
  --queue-url "$RASTER_QUEUE_URL" \
  --attribute-names VisibilityTimeout ApproximateNumberOfMessagesNotVisible

Expected output while a single large scene is processing — one message in flight, correct timeout:

json

{
  "Attributes": {
    "VisibilityTimeout": "5400",
    "ApproximateNumberOfMessagesNotVisible": "1"
  }
}

If ApproximateNumberOfMessagesNotVisible climbs above the number of scenes you actually submitted, the timeout is too short and messages are being redelivered mid-flight. Cross-check the per-message ApproximateReceiveCount — a value above 1 for a scene you sent once confirms duplicate processing rather than genuine failure.

Gotchas and Edge Cases

The six-times rule is specific to event source mappings. When Lambda polls SQS on your behalf it batches and retries internally, so a single function run is not the whole in-flight lifetime — hence six times, not one. A hand-rolled poller that receives, processes, and deletes one message synchronously can use a tighter multiple, but must still exceed the true p99 duration.
A heartbeat cannot rescue a job past the 15-minute function ceiling. ChangeMessageVisibility extends how long SQS waits, not how long Lambda runs. If the raster work legitimately needs more than 15 minutes, no visibility setting helps — split it into chunks sized to fit the ceiling and size visibility from the chunk.
Raising the timeout to mask slowness inflates recovery time. A 12-hour visibility timeout does hide redelivery, but if a worker crashes silently, that message now sits invisible for 12 hours before anyone retries it. Size for the real p99 plus margin, not the theoretical maximum.
ChangeMessageVisibility needs a live receipt handle. Once the original visibility window lapses even briefly, the receipt handle can become invalid and the heartbeat call fails — which is why the base timeout must already cover a typical job and the heartbeat only extends beyond it, never rescues an already-expired message.

Frequently Asked Questions

For an SQS event source mapping, Lambda may retry a batch internally and processes messages in windows, so a message must stay invisible well beyond a single function run. Six times the function timeout gives enough headroom that the poller never redelivers a message that is still being processed, which is why the console warns when the ratio drops below that.

How do I handle raster jobs whose duration varies widely?

Use a heartbeat: run a background timer that calls ChangeMessageVisibility to extend the invisibility window while the job reports progress. This keeps a modest base timeout for typical scenes while still protecting the rare large granule that runs much longer, up to the 15-minute Lambda ceiling.

What happens if the visibility timeout is shorter than the raster job?

The message becomes visible again while the first invocation is still running, so a second consumer picks it up and reprocesses the same scene in parallel. This double-processing corrupts mosaic writes, inflates cost, and drives ApproximateReceiveCount toward maxReceiveCount, eventually dead-lettering a message that never actually failed.

SQS and Pub/Sub Queue Routing Strategies — how raster jobs reach the queue whose visibility timeout you are sizing
Guaranteeing Order with SQS FIFO for Sequential Tile Jobs — why a mid-job redelivery is especially damaging to ordered mosaic writes
Implementing Dead Letter Queues for Failed Vector Jobs — pairing maxReceiveCount with the heartbeat so slow and broken diverge
Chunking Raster Jobs to Fit the 15-Minute Lambda Ceiling — the alternative to over-long timeouts when a scene cannot finish in one run
Memory and CPU Allocation for Raster Workloads — right-sizing compute so the p99 duration you size visibility from stays predictable

Back to SQS and Pub/Sub Queue Routing Strategies