Aggregating Multipart Shapefile Uploads Before Processing

A shapefile is not one file — it is a bundle of .shp, .shx, .dbf, and usually .prj objects, and S3 fires a separate ObjectCreated event for each. Do not process on the first event. Instead, record each arriving component in a DynamoDB manifest keyed by the shared basename, and invoke GDAL only when the mandatory set {.shp, .shx, .dbf} is complete — flipping a dispatched flag in the same conditional write so the bundle fires exactly once. When uploads are near-simultaneous and you would rather not run a table, a short SQS DelaySeconds requeue that re-polls the prefix on redelivery achieves the same gate.

Aggregating shapefile component events before dispatch Four separate S3 events for .shp, .shx, .dbf, and .prj each update a DynamoDB manifest keyed by the bundle basename. Only when the mandatory .shp/.shx/.dbf set is complete does a conditional dispatched flag flip and trigger GDAL processing. parcels.shp parcels.shx parcels.dbf parcels.prj DynamoDB manifest key = "parcels" seen = ADD .shp .shx .dbf .prj required ⊆ seen ? flip dispatched Complete? conditional first-only GDAL / Fiona once Per-component events accumulate in one manifest → one dispatch when the set is complete

Context

A client uploading a shapefile with aws s3 cp parcels.* s3://bucket/incoming/ produces three to seven distinct PutObject operations, and S3 emits one notification per object. Nothing guarantees ordering: the .dbf can land before the .shp, the .prj can trail by seconds, and on a slow uplink the geometry file may still be uploading when the index arrives. If you point a Lambda straight at every ObjectCreated:* event and call fiona.open(), most invocations fail with fiona.errors.DriverError: unable to open ... .shp because a required companion is not yet present — a race, not a real error.

The S3 and GCS Event Triggers for Shapefiles overview frames this as the multi-file atomicity problem. The GCS-side treatment triggers only on .shp and lists the prefix to confirm companions — viable because clients usually upload the geometry file last. On S3 that assumption is fragile, so the robust approach is an explicit manifest that tolerates any arrival order. This pattern is distinct from — and composes with — the deduplication of duplicate notifications: aggregation waits for enough different events, while deduplication suppresses repeat events. A production ingester wants both.

Prerequisites

  • Runtime: Python 3.11 on AWS Lambda. AWS Lambda offers a 15-minute timeout, up to 10,240 MB memory, and 10 GB of /tmp, so a completed bundle downloads and validates comfortably in one invocation.
  • S3 notification config: all shapefile extensions under the incoming prefix routed to the aggregator. Configure the bucket notification with a prefix filter (e.g. incoming/) and no suffix filter, so .shp, .shx, .dbf, and .prj all reach the same function.
  • DynamoDB manifest table: partition key basename (String), a String Set attribute seen, and a Boolean dispatched. On-demand billing; add an expires_at TTL Number to garbage-collect abandoned partial bundles.
  • Optional SQS queue for the delay-and-poll fallback, with DelaySeconds up to 900 (15 minutes) per message.
  • IAM — execution role:
    • dynamodb:UpdateItem and dynamodb:GetItem on the manifest table ARN
    • s3:GetObject and s3:ListBucket (scoped to the incoming prefix) to confirm and read components
    • sqs:SendMessage on the fallback queue ARN, if used
  • Environment variables:
    code 
    MANIFEST_TABLE=shapefile-manifest
REQUIRED_EXTS=.shp,.shx,.dbf
MANIFEST_TTL_SECONDS=86400
GDAL_DATA=/opt/python/lib/python3.11/site-packages/rasterio/gdal_data
PROJ_LIB=/opt/python/lib/python3.11/site-packages/pyproj/proj_dir/share/proj

Implementation

The aggregator updates the manifest and reads the resulting seen set back in a single UpdateItem (ReturnValues="ALL_NEW"). It then tests completeness and, if complete, flips dispatched with a conditional write so exactly one event triggers GDAL — even if the last two components land concurrently.

python 
"""aggregate_shapefile.py — gate GDAL on a complete shapefile bundle.

Each component event ADDs its extension to a basename-keyed manifest.
The event that both completes the required set AND wins the conditional
flip of `dispatched` is the single event that invokes processing.
"""
import os
import urllib.parse
from pathlib import PurePosixPath

import boto3
from botocore.exceptions import ClientError

TABLE = os.environ["MANIFEST_TABLE"]
REQUIRED = frozenset(os.environ["REQUIRED_EXTS"].split(","))  # {'.shp','.shx','.dbf'}
TTL = int(os.environ.get("MANIFEST_TTL_SECONDS", "86400"))

_ddb = boto3.resource("dynamodb")
_table = _ddb.Table(TABLE)


def _split(key: str) -> tuple[str, str]:
    """Return (basename-with-prefix, lowercased extension).

    Basename includes the S3 prefix so parcels in different folders never
    collide: 'incoming/a/parcels.shp' -> ('incoming/a/parcels', '.shp').
    """
    p = PurePosixPath(key)
    return str(p.with_suffix("")), p.suffix.lower()


def _record_component(basename: str, ext: str) -> set[str]:
    """ADD the extension to the manifest and return the full seen set."""
    import time
    resp = _table.update_item(
        Key={"basename": basename},
        UpdateExpression="ADD seen :e SET expires_at = :t, dispatched = if_not_exists(dispatched, :f)",
        ExpressionAttributeValues={
            ":e": {ext},                       # String Set add is idempotent
            ":t": int(time.time()) + TTL,
            ":f": False,
        },
        ReturnValues="ALL_NEW",
    )
    return set(resp["Attributes"]["seen"])


def _claim_dispatch(basename: str) -> bool:
    """Flip dispatched False->True atomically. True means we won the race."""
    try:
        _table.update_item(
            Key={"basename": basename},
            UpdateExpression="SET dispatched = :t",
            ConditionExpression="dispatched = :f",
            ExpressionAttributeValues={":t": True, ":f": False},
        )
        return True
    except ClientError as exc:
        if exc.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return False   # another event already dispatched this bundle
        raise


def _process_bundle(bucket: str, basename: str, seen: set[str]) -> None:
    """Download the complete bundle and validate with Fiona."""
    import tempfile
    import fiona
    s3 = boto3.client("s3")
    stem = PurePosixPath(basename).name
    with tempfile.TemporaryDirectory() as tmp:
        for ext in seen:
            local = os.path.join(tmp, f"{stem}{ext}")
            s3.download_file(bucket, f"{basename}{ext}", local)
        shp = os.path.join(tmp, f"{stem}.shp")
        with fiona.open(shp) as src:
            print(f"Bundle {basename}: {len(src)} features, CRS={src.crs}")
    # ... route validated dataset downstream (PostGIS load, tiling, etc.) ...


def handler(event, _context):
    dispatched = []
    for record in event.get("Records", []):
        bucket = record["s3"]["bucket"]["name"]
        key = urllib.parse.unquote_plus(record["s3"]["object"]["key"])
        basename, ext = _split(key)

        seen = _record_component(basename, ext)

        # Not complete yet — wait for the remaining component events.
        if not REQUIRED.issubset(seen):
            print(f"{basename}: have {sorted(seen)}, need {sorted(REQUIRED - seen)}")
            continue

        # Complete. Exactly one event wins the flip and processes.
        if _claim_dispatch(basename):
            _process_bundle(bucket, basename, seen)
            dispatched.append(basename)
        else:
            print(f"{basename}: already dispatched by a sibling event")

    return {"dispatched": dispatched}

For the fallback variant, replace _claim_dispatch with an SQS requeue: on an incomplete manifest, send_message(..., DelaySeconds=30) a message carrying the basename, and on redelivery re-run head_object for each required extension — dispatching when all resolve, or requeuing again until a max-attempts ceiling routes the partial bundle to a dead-letter queue. The delay-and-poll path trades precision for simplicity: it needs no DynamoDB table and no String Set semantics, but it wakes the function on a fixed cadence rather than the instant the last component lands, so it wastes a few invocations on bundles that were already complete. Keep the attempt counter in the message body or a message attribute so each redelivery can back off and eventually give up rather than looping forever on a bundle whose .dbf upload silently failed.

Verification

Upload the components out of order, one at a time, and confirm dispatch fires only on the completing event:

bash 
# Upload .dbf and .shx first, then the .shp last — worst-case ordering.
aws s3 cp parcels.dbf s3://geo-uploads/incoming/parcels.dbf
aws s3 cp parcels.shx s3://geo-uploads/incoming/parcels.shx
aws s3 cp parcels.prj s3://geo-uploads/incoming/parcels.prj
aws s3 cp parcels.shp s3://geo-uploads/incoming/parcels.shp   # completes the set

aws logs tail /aws/lambda/aggregate-shapefile --since 2m --format short

Expected log sequence — the first three events wait, the fourth dispatches:

code 
incoming/parcels: have ['.dbf'], need ['.shp', '.shx']
incoming/parcels: have ['.dbf', '.shx'], need ['.shp']
incoming/parcels: have ['.dbf', '.prj', '.shx'], need ['.shp']
Bundle incoming/parcels: 14823 features, CRS=EPSG:4326

Inspect the manifest to confirm dispatched flipped exactly once:

bash 
aws dynamodb get-item --table-name shapefile-manifest \
  --key '{"basename": {"S": "incoming/parcels"}}' \
  --query 'Item.{seen: seen.SS, dispatched: dispatched.BOOL}'
# Expected: {"seen": [".dbf", ".prj", ".shp", ".shx"], "dispatched": true}

Gotchas and Edge Cases

  • Include the prefix in the basename key. Two uploads named parcels.shp in incoming/county-a/ and incoming/county-b/ must not share a manifest row. Keying on the full prefixed path (PurePosixPath.with_suffix("")) keeps them isolated; keying on just the filename silently merges unrelated bundles.

  • A never-completing bundle must expire. If a client uploads .shp and .shx but the .dbf upload fails, the manifest row sits half-complete forever. The expires_at TTL garbage-collects it after the grace period. For observability, run a scheduled sweep that alerts on rows older than an hour with dispatched=false, and consider routing them to a dead-letter queue for failed vector jobs.

  • Overwrites reset expectations, not the manifest. Re-uploading parcels.shp with new content fires a fresh event whose ETag differs. Aggregation alone will not reprocess it because dispatched is already true. Pair this gate with content-addressed deduplication so a genuine re-upload creates a new logical job while duplicates stay suppressed.

  • SQS delay caps at 15 minutes. The delay-and-poll fallback cannot wait longer than a single message’s 900-second DelaySeconds. For uploads that can span longer — very large .dbf attribute tables over a slow link — the DynamoDB manifest is the only correct choice, since it dispatches immediately on completion rather than polling on a fixed cadence.

Frequently Asked Questions

Which shapefile components are mandatory before GDAL can open the file?

GDAL’s OGR shapefile driver needs .shp (geometry), .shx (index), and .dbf (attributes) to open a dataset. The .prj (CRS), .cpg (encoding), and spatial-index .sbn/.sbx files are optional: GDAL opens the file without them, reporting CRS as None when .prj is absent. Gate on the three mandatory components and treat the rest as best-effort — download them if present, but never block dispatch waiting for them.

Manifest in DynamoDB or an SQS delay — which should I use?

Use a DynamoDB manifest when uploads can span more than a few seconds or arrive out of order: it dispatches the instant the last required component lands, with no wasted waiting. Use an SQS delay-and-poll when uploads are near-simultaneous and you would rather not run an extra table — requeue with a short DelaySeconds and re-check the prefix on redelivery. The manifest is more precise; the delay is simpler to stand up.

How do I avoid dispatching the same bundle twice?

Make the completion transition atomic. The completing event flips a dispatched flag in a conditional UpdateItem guarded by dispatched = false. Only the write that flips the flag invokes GDAL; concurrent completing events fail the condition and return. This is the same first-writer-wins primitive used for notification deduplication, applied to the completion edge rather than the arrival edge.

Does this pattern work the same on GCP?

The mechanism ports directly: Firestore or a marker blob replaces the DynamoDB manifest, and a transaction replaces the conditional write. On GCS, though, the simpler trigger-on-.shp-and-verify approach is often enough because clients commonly upload the geometry file last, letting a single prefix listing confirm the companions.

Back to S3 and GCS Event Triggers for Shapefiles