Implementing Dead Letter Queues for Failed Vector Jobs

Set maxReceiveCount to 3–5 on your primary SQS queue or Pub/Sub subscription, bind a dedicated dead-letter destination, and route failed vector messages there with structured diagnostic metadata. This decouples your primary ingestion pipeline from a failure-routing layer, preventing toxic spatial payloads from poisoning downstream consumers. Across AWS, GCP, and Azure, this pattern replaces blind retry loops with deterministic failure isolation.

Dead Letter Queue Flow for Failed Vector Jobs Diagram showing a vector message flowing from the primary queue to a Lambda consumer. On success the message is deleted; after maxReceiveCount failures it routes to the dead letter queue where a triage function handles auto-repair, quarantine, or warehouse export. Primary Queue SQS / Pub/Sub Vector Consumer shapely / pyproj CRS transform ACK / Delete success path max retries exceeded Dead Letter Queue error_type · geometry_type · trace_id Triage Function repair / quarantine warehouse export maxReceiveCount: 3–5

Why Vector Processing Needs Explicit Failure Routing

Geospatial vector workloads fail in predictable, repeating categories that generic retry logic cannot distinguish. Coordinate transformations, topology validation, and spatial joins routinely hit edge cases that surface as distinct exception types inside shapely, pyproj, or GEOS:

  • Invalid geometry: Self-intersecting polygons, unclosed rings, or missing CRS definitions break shape() and Transformer calls with TopologicalError or GEOSException before any meaningful work begins.
  • Resource exhaustion: Oversized multipart geometries or dense point clouds exceed Lambda’s 10 GB memory ceiling or Cloud Functions 2nd gen’s 32 GB limit, triggering OOM kills that produce no useful stack trace.
  • Transient rate limits: Tile servers, elevation APIs, or basemap providers return 429 responses, causing requests.exceptions.HTTPError or timeout exceptions that are genuinely worth retrying.
  • Serialization failures: Malformed GeoJSON or WKT payloads fail schema validation with json.JSONDecodeError or Pydantic ValidationError before processing begins — these should never be retried.

Without explicit failure routing, these errors either silently drop or trigger infinite redelivery loops that exhaust compute quotas. Routing failed messages to a DLQ is the fundamental fault-tolerance primitive in SQS and Pub/Sub Queue Routing Strategies, where each message category gets a deterministic outcome rather than an unbounded retry.

Prerequisites

Before configuring DLQs for vector jobs, verify the following:

  • Runtime: Python 3.11+ (Lambda) or Python 3.12 (Cloud Functions 2nd gen). Earlier runtimes have known shapely 2.x incompatibilities with GEOS 3.12.
  • Dependencies: shapely>=2.0.0, pyproj>=3.6.0, boto3>=1.28.0 (AWS) or google-cloud-pubsub>=2.21.0 (GCP).
  • IAM permissions (AWS): sqs:SendMessage, sqs:ReceiveMessage, sqs:DeleteMessage, sqs:GetQueueAttributes on both the primary queue and the DLQ ARN.
  • IAM permissions (GCP): pubsub.subscriptions.consume on the source subscription, pubsub.topics.publish on the dead-letter topic, plus pubsub.subscriptions.create to bind the policy.
  • Environment variables: Set PRIMARY_QUEUE_URL, DLQ_URL (AWS) or PROJECT_ID, DLQ_TOPIC_ID (GCP) as Lambda environment variables or Cloud Functions runtime configuration — never hardcode queue URLs.

Cross-Cloud DLQ Configuration

Each platform implements dead-letter routing differently. The architectural contract is identical: set a receive threshold, bind a secondary destination, and have your consumer explicitly acknowledge or reject messages.

AWS SQS — RedrivePolicy

Attach a RedrivePolicy to the source queue. Set maxReceiveCount to 3–5 for heavy geometry workloads. Configure the visibility timeout to exceed your longest expected spatial operation (coordinate reprojection of a dense multipolygon can take 8–12 seconds on a cold Lambda). For FIFO queues, preserve MessageDeduplicationId and MessageGroupId in the DLQ to prevent duplicate reprocessing.

bash 
aws sqs set-queue-attributes \
  --queue-url "$PRIMARY_QUEUE_URL" \
  --attributes '{
    "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:us-east-1:123456789012:vector-dlq\",\"maxReceiveCount\":\"4\"}",
    "VisibilityTimeout": "120"
  }'

GCP Pub/Sub — deadLetterPolicy

Pub/Sub requires a separate topic and subscription pair for the DLQ. The minimum maxDeliveryAttempts is 5; Pub/Sub will not accept lower values. The Pub/Sub service account ([email protected]) needs pubsub.publisher on the dead-letter topic and pubsub.subscriber on the source subscription.

bash 
gcloud pubsub subscriptions modify-push-config geo-vector-sub \
  --dead-letter-topic=projects/$PROJECT_ID/topics/vector-dlq \
  --max-delivery-attempts=5

Azure Service Bus — MaxDeliveryCount

Configure MaxDeliveryCount on the queue or subscription. Service Bus automatically moves exhausted messages to the $DeadLetterQueue sub-queue. Read from queue_name/$DeadLetterQueue explicitly. Azure also routes messages to the DLQ when DeadLetteringOnMessageExpiration is enabled and TTL expires.

Platform Config key Minimum Recommended for vector DLQ destination type
AWS SQS maxReceiveCount 1 4 Secondary SQS queue
GCP Pub/Sub maxDeliveryAttempts 5 5 Secondary Pub/Sub topic
Azure Service Bus MaxDeliveryCount 1 4 $DeadLetterQueue sub-queue

Implementation

The consumer below handles all four failure categories, writes structured diagnostic metadata to the DLQ, then explicitly deletes the original message to prevent redelivery. It targets AWS Lambda but the exception-handling logic applies unchanged to GCP Cloud Functions 2nd gen (swap the SQS calls for Pub/Sub nack()/ack()).

python 
import json
import logging
import os

import boto3
from botocore.exceptions import ClientError
from pyproj import Transformer
from pyproj.exceptions import CRSError
from shapely.errors import GEOSException, TopologicalError
from shapely.geometry import shape

logger = logging.getLogger()
logger.setLevel(logging.INFO)

# Read from Lambda environment variables — never hardcode queue URLs
PRIMARY_QUEUE_URL = os.environ["PRIMARY_QUEUE_URL"]
DLQ_URL = os.environ["DLQ_URL"]

sqs = boto3.client("sqs")


def _send_to_dlq(body: dict, error_type: str, error_message: str,
                 geometry_type: str, trace_id: str) -> None:
    """Write a structured failure record to the DLQ."""
    dlq_payload = {
        "original_message": body,
        "error_type": error_type,
        "error_message": error_message,
        "geometry_type": geometry_type,
        "trace_id": trace_id,
    }
    sqs.send_message(QueueUrl=DLQ_URL, MessageBody=json.dumps(dlq_payload))


def handler(event, context):
    for record in event.get("Records", []):
        receipt_handle = record["receiptHandle"]
        message_id = record["messageId"]

        try:
            body = json.loads(record["body"])
        except json.JSONDecodeError as exc:
            # Malformed envelope — no point retrying; route directly to DLQ
            logger.error("JSON parse failure on %s: %s", message_id, exc)
            _send_to_dlq(
                body={"raw": record["body"]},
                error_type="JSONDecodeError",
                error_message=str(exc),
                geometry_type="unknown",
                trace_id=context.aws_request_id,
            )
            sqs.delete_message(QueueUrl=PRIMARY_QUEUE_URL, ReceiptHandle=receipt_handle)
            continue

        payload = body.get("geometry", {})
        geometry_type = payload.get("type", "unknown")

        try:
            # 1. Parse and validate geometry
            geom = shape(payload)
            if not geom.is_valid:
                raise TopologicalError(
                    f"Invalid geometry ({geom.geom_type}): {geom.is_valid_reason}"
                )

            # 2. Reproject from source CRS to Web Mercator
            source_crs = body.get("crs", "EPSG:4326")
            transformer = Transformer.from_crs(source_crs, "EPSG:3857", always_xy=True)
            projected_coords = [
                transformer.transform(x, y) for x, y in geom.exterior.coords
            ]

            # 3. Downstream spatial work (topology checks, attribute joins, etc.)
            logger.info(
                "Processed %s geometry: %d vertices projected from %s",
                geometry_type, len(projected_coords), source_crs,
            )

            # Explicit delete — SQS won't remove on Lambda success automatically
            sqs.delete_message(QueueUrl=PRIMARY_QUEUE_URL, ReceiptHandle=receipt_handle)

        except MemoryError as exc:
            # OOM: serverless runtimes don't always surface these in logs
            logger.error("OOM on %s (%s): %s", message_id, geometry_type, exc)
            _send_to_dlq(body, "MemoryError", str(exc), geometry_type, context.aws_request_id)
            sqs.delete_message(QueueUrl=PRIMARY_QUEUE_URL, ReceiptHandle=receipt_handle)

        except (TopologicalError, GEOSException) as exc:
            # Invalid geometry — may be auto-repairable via make_valid()
            logger.warning("Geometry error on %s: %s", message_id, exc)
            _send_to_dlq(body, type(exc).__name__, str(exc), geometry_type, context.aws_request_id)
            sqs.delete_message(QueueUrl=PRIMARY_QUEUE_URL, ReceiptHandle=receipt_handle)

        except CRSError as exc:
            # Bad or unsupported CRS definition — not worth retrying
            logger.error("CRS error on %s: %s", message_id, exc)
            _send_to_dlq(body, "CRSError", str(exc), geometry_type, context.aws_request_id)
            sqs.delete_message(QueueUrl=PRIMARY_QUEUE_URL, ReceiptHandle=receipt_handle)

        except ClientError as exc:
            # SQS API failure — let Lambda's retry mechanism handle this
            logger.error("SQS ClientError on %s: %s", message_id, exc)
            raise  # Do NOT delete; allow visibility timeout to expire for retry

Key implementation decisions:

  • DLQ write before delete: Send to the DLQ first, then delete the original. Reversing the order risks permanent message loss if the DLQ send fails after deletion.
  • Raise on ClientError: If the SQS API itself is unavailable, re-raising lets the message return to the queue via visibility timeout expiry. Swallowing this exception would silently drop messages.
  • Separate exception branches: Separating TopologicalError from MemoryError from CRSError gives the downstream triage function an actionable error_type field — automated repair workflows can match on this without reparsing stack traces.

Pre-Flight Schema Validation

DLQs catch runtime failures, but upstream validation reduces DLQ volume significantly. A lightweight Pydantic model at the API Gateway or ingress Lambda rejects structurally invalid payloads before they enter the primary queue — keeping the DLQ reserved for genuinely unprocessable spatial data.

python 
from pydantic import BaseModel, field_validator
from typing import Any

class GeoPayload(BaseModel):
    source_uri: str
    crs: str = "EPSG:4326"
    processing_mode: str
    geometry: dict[str, Any]

    @field_validator("geometry")
    @classmethod
    def geometry_must_have_type_and_coords(cls, v):
        if "type" not in v or "coordinates" not in v:
            raise ValueError("geometry must contain 'type' and 'coordinates'")
        return v

# At ingress — reject before enqueuing
try:
    validated = GeoPayload(**incoming_payload)
except Exception as exc:
    return {"statusCode": 400, "body": f"Schema validation failed: {exc}"}

Verification Step

After deploying the consumer, confirm that a synthetic bad geometry reaches the DLQ within one delivery cycle:

bash 
# Send a self-intersecting polygon (bowtie) to the primary queue
aws sqs send-message \
  --queue-url "$PRIMARY_QUEUE_URL" \
  --message-body '{
    "source_uri": "s3://my-bucket/test.geojson",
    "crs": "EPSG:4326",
    "processing_mode": "vector",
    "geometry": {
      "type": "Polygon",
      "coordinates": [[[0,0],[1,1],[0,1],[1,0],[0,0]]]
    }
  }'

# Wait for maxReceiveCount cycles, then check DLQ depth
aws sqs get-queue-attributes \
  --queue-url "$DLQ_URL" \
  --attribute-names ApproximateNumberOfMessages

Expected output after one Lambda invocation:

json 
{
  "Attributes": {
    "ApproximateNumberOfMessages": "1"
  }
}

A count of 0 means the message is still in the primary queue (Lambda not triggered or consumer raising before the DLQ write). A count above 1 for a single test message indicates duplicate processing — check that sqs.delete_message() is reached in all exception branches.

For GCP, inspect DLQ depth via Cloud Monitoring:

bash 
gcloud pubsub subscriptions describe vector-dlq-sub \
  --format="value(messageRetentionDuration,expirationPolicy)"

Triage and Reprocessing Workflows

Once messages land in the DLQ, bind a triage consumer that routes based on error_type:

  1. Auto-repair: For TopologicalError, attempt shapely.make_valid() on the geometry, re-validate, and if the result passes is_valid, re-enqueue to the primary queue. Tag the re-sent message with "repaired": true so downstream consumers can log the repair event.
  2. Quarantine and alert: For MemoryError or oversized multipart geometries, publish a structured alert to PagerDuty or Slack with geometry_type, source_uri, and approximate vertex count. Do not retry automatically.
  3. Warehouse export for systemic analysis: For CRSError or JSONDecodeError, stream DLQ records to BigQuery, Redshift, or Azure Synapse. Aggregate by source_uri prefix to identify providers consistently shipping malformed payloads — this data feeds the upstream schema validation rules described above.

This triage model aligns with the event-driven patterns where failure routing is a first-class data quality signal rather than a reactive afterthought. It also pairs naturally with batch vs stream geospatial processing decisions — auto-repaired geometries can re-enter the stream path, while systemic failures accumulate for offline batch repair against the source dataset.

Gotchas and Edge Cases

  • Pub/Sub minimum maxDeliveryAttempts is 5, not 1. Attempting to set it to 3 via the API returns a INVALID_ARGUMENT error. Plan your retry budget assuming at least 5 deliveries before a message routes to the dead-letter topic.
  • SQS FIFO DLQs must also be FIFO. If your primary queue is FIFO, the deadLetterTargetArn must point to a FIFO DLQ — SQS will reject a standard queue as the target. Preserve MessageGroupId in your DLQ write to maintain ordering metadata for forensic analysis.
  • Lambda’s built-in retry does not count toward maxReceiveCount. Lambda retries the same ReceiptHandle on function error. Each attempt increments the ApproximateReceiveCount attribute. If your Lambda raises an unhandled exception (rather than writing to the DLQ and deleting), the message stays in the primary queue and the counter ticks toward maxReceiveCount — which is fine for transient ClientError, but means invalid geometries will burn retries before reaching the DLQ.
  • [Ephemeral Storage Limits in AWS Lambda](/serverless-geospatial-architecture-platform-limits/ephemeral-storage-limits-in-aws-lambda/) can mask OOM as timeout. When shapely processes large multipolygons that spill intermediate data, the function may timeout (at 15 minutes maximum) rather than raising MemoryError. Add ApproximateNumberOfMessagesNotVisible monitoring to distinguish timeout-induced redelivery from genuine geometry failures.

Frequently Asked Questions

What maxReceiveCount should I set for vector geometry jobs?

Set maxReceiveCount to 3–5. Values above 5 waste compute retrying fundamentally broken geometries; values below 3 risk discarding transient failures caused by rate-limited tile servers or brief network partitions.

Why does GCP Pub/Sub require a separate topic for the DLQ?

Pub/Sub routes exhausted messages to a dead-letter topic (not a subscription), so you need a dedicated topic plus a subscription on that topic to consume failures. Unlike SQS, Pub/Sub does not retry from the dead-letter topic automatically — your consumer must drive reprocessing.

Should I delete the original message before or after writing to the DLQ?

Write to the DLQ first, then delete. If you delete first and the DLQ send fails, the message is permanently lost with no recovery path.

Back to SQS and Pub/Sub Queue Routing Strategies