EventBridge vs S3 Notifications for Spatial Ingestion

Use direct S3 notifications when a single consumer needs every object under a prefix, and EventBridge when the ingest predicate needs prefix and suffix together, several suffixes, a size test, or more than one independent consumer. Enable it by adding "EventBridgeConfiguration": {} to the bucket’s notification configuration, then match with {"detail": {"object": {"key": [{"wildcard": "incoming/*.shp"}], "size": [{"numeric": [">", 0]}]}}} — a predicate that no FilterRule can express. Both paths are at-least-once and neither guarantees ordering, so the completion gate downstream is unchanged either way.

Context

The S3 and GCS event trigger pattern for shapefiles turns on one awkward fact: a dataset is three to seven objects, arriving in no guaranteed order, and the trigger has to see all of them before GDAL can open anything. That constraint pushes most teams to the same configuration — a prefix filter on the staging area, no suffix filter, and every extension routed to one aggregator. It works. It also means the aggregator is invoked for every object anyone writes under that prefix, including .tif deliveries, .zip archives, checksum sidecars, and the zero-byte folder markers the S3 console creates.

The reason the suffix filter is dropped rather than tightened is a hard limit in the notification API, not a stylistic choice. An S3 key filter accepts at most one prefix rule and one suffix rule, both literal strings, and the two are ANDed. There is no OR, no wildcard, and no repetition. “All five shapefile extensions under incoming/” is not expressible. You can declare five separate configurations with five different suffixes, but they must not overlap for the same event type — S3 rejects the entire PutBucketNotificationConfiguration call with Configurations overlap. Configurations on the same bucket cannot share a common event type — and you have to enumerate in advance every extension a desktop GIS client might emit.

There is a second, less discussed constraint. A bucket has exactly one notification configuration document, and PutBucketNotificationConfiguration replaces it wholesale. Two teams managing two Terraform stacks against the same ingest bucket will silently delete each other’s triggers, and the failure presents as “the pipeline stopped firing” with nothing in any log to explain it.

EventBridge removes both constraints by moving the filtering out of the bucket. The bucket is opted in once and then publishes Object Created events on the default bus with source: "aws.s3"; from there, any number of independent rules match on whatever they like. Rules are separate resources, so two teams add two rules without touching each other. Patterns support prefix, suffix, wildcard, anything-but, numeric and exists, which is enough to express the predicate you actually meant.

Prerequisites

• Runtime: Python 3.11+ on AWS Lambda, or any EventBridge target — the choice of transport does not constrain the consumer.
• Dependencies: boto3>=1.34.0 for the configuration calls below; no GDAL dependency at this layer, since the trigger sees keys and sizes rather than geometry.
• A bucket you control the notification configuration for. Enabling EventBridge is a modification to that same document, so you need the ability to read it, merge, and write it back without clobbering existing entries.
• IAM: s3:PutBucketNotificationConfiguration and s3:GetBucketNotificationConfiguration on the bucket; events:PutRule, events:PutTargets and events:CreateArchive on the bus. EventBridge invokes a Lambda target through a resource policy on the function, not through the rule’s own role — lambda:AddPermission with principal="events.amazonaws.com" is the step people forget.
• A decided consumer count. One consumer and one predicate is a direct notification. Anything past that is the case this page is about.

Where the Two Diverge

Six axes matter for a spatial ingest, and only three of them favour one option decisively.

Delivery semantics are the row that surprises people: EventBridge does not upgrade the guarantee. Both paths are at-least-once, so the deterministic job id and conditional-put lock from deduplicating S3 event notifications for idempotent ingestion are required under both, and become more important under EventBridge because two matching rules can legitimately deliver the same upload to the same downstream twice.

Replay is the row that has no equivalent on the other side at all. An archive keeps every event the bus saw for a retention you set, and StartReplay re-emits a time range into a chosen rule. For an ingest pipeline that is a genuine capability: a parser bug that mangled a week of .dbf attribute tables is fixed by replaying that week, without listing the bucket and without re-uploading anything. It is bounded by the archive retention, which makes it a companion to — not a substitute for — a manifest-driven backfill from archived scenes.

The cost row is close to a tie and is usually argued about for the wrong reason. Events that AWS services publish to the default bus are not charged, so the transport itself is free on both sides. The money is in invocations you did not need.

An ingest prefix taking 2.4 million objects a month of which 400,000 are shapefile components invokes the aggregator 2.4 million times under a prefix-only notification. The wasted two million invocations cost only a few dollars in Lambda and DynamoDB, which is why the argument gets dismissed — but they are also two million draws against the 1,000-slot regional concurrency quota that the tiling fleet is competing for, and that is not a rounding error.

Implementation

Enable EventBridge without destroying the existing configuration — read, merge, write — then create the rule that expresses the real predicate:

import json

import boto3

s3 = boto3.client("s3")
events = boto3.client("events")
BUCKET = "gis-ingest-prod"

# The notification document is replaced wholesale, so read the current one and
# merge. Writing a bare {"EventBridgeConfiguration": {}} deletes every existing
# Lambda, SQS and SNS trigger on the bucket without warning.
current = s3.get_bucket_notification_configuration(Bucket=BUCKET)
current.pop("ResponseMetadata", None)
current["EventBridgeConfiguration"] = {}
s3.put_bucket_notification_configuration(
    Bucket=BUCKET, NotificationConfiguration=current
)

# wildcard is what makes prefix AND suffix expressible in one term. Members of
# the key array are ORed, so five wildcards cover all five shapefile components
# — the exact predicate an S3 FilterRule cannot carry.
pattern = {
    "source": ["aws.s3"],
    "detail-type": ["Object Created"],
    "detail": {
        "bucket": {"name": [BUCKET]},
        "object": {
            "key": [
                {"wildcard": "incoming/*.shp"},
                {"wildcard": "incoming/*.shx"},
                {"wildcard": "incoming/*.dbf"},
                {"wildcard": "incoming/*.prj"},
                {"wildcard": "incoming/*.cpg"},
            ],
            # Drops the zero-byte folder markers the console creates, which a
            # notification filter has no way to see at all.
            "size": [{"numeric": [">", 0]}],
        },
        # CompleteMultipartUpload is how any client uploads a large .shp; a rule
        # that omits it silently ignores every dataset over the part threshold.
        "reason": ["PutObject", "CompleteMultipartUpload", "CopyObject"],
    },
}

events.put_rule(
    Name="shapefile-components-to-gate",
    EventPattern=json.dumps(pattern),
    State="ENABLED",
    EventBusName="default",
)

# Fan-out is the rule's own property: the gate and the audit sink are peers, and
# neither knows the other exists.
events.put_targets(
    Rule="shapefile-components-to-gate",
    EventBusName="default",
    Targets=[
        {"Id": "completion-gate", "Arn": GATE_LAMBDA_ARN},
        {"Id": "ingest-audit", "Arn": AUDIT_FIREHOSE_ARN, "RoleArn": FIREHOSE_ROLE_ARN},
    ],
)

Note what the pattern buys beyond the extension list. size is not present in an S3 notification filter at all, so a prefix-only notification invokes the aggregator on every zero-byte directory placeholder. reason distinguishes a PutObject from a CompleteMultipartUpload, which matters because a large .shp arrives as the latter and a rule that lists only the former ignores exactly the datasets that are expensive to reprocess.

The added hop is real and worth measuring rather than assuming:

Roughly a fifth of the budget is EventBridge, about half is the consumer’s own initialisation, and the whole thing is dwarfed by the wait for the second and third components of the dataset to arrive. For a completion gate that is a non-issue. For the sub-second alerting path in when to use batch vs streaming for real-time AIS tracking it would not be.

Verification

Confirm the bucket is opted in and that the pattern matches the keys you expect, before pointing anything expensive at it:

# 1. The bucket must report an EventBridgeConfiguration key
aws s3api get-bucket-notification-configuration --bucket gis-ingest-prod \
  --query 'EventBridgeConfiguration'

# 2. Test the pattern against a real event shape without uploading anything
aws events test-event-pattern \
  --event-pattern file://shapefile-pattern.json \
  --event '{"source":"aws.s3","detail-type":"Object Created",
            "account":"123456789012","region":"eu-west-1",
            "time":"2026-08-07T09:14:02Z","resources":[],
            "detail":{"version":"0","bucket":{"name":"gis-ingest-prod"},
            "object":{"key":"incoming/county-a/parcels.dbf","size":184320,
            "etag":"9b2cf5d1a0e34b77"},"reason":"PutObject"}}'

Expected output — an empty object from the first call means the bucket is not opted in, and false from the second means the pattern will never fire:

{}
{
    "Result": true
}

The first response is the one to read carefully: {} is the failure case and {"EventBridgeConfiguration": {}} is success, which is an unhelpfully small difference. Re-run the second call with incoming/county-a/ as the key to confirm the size matcher rejects a zero-byte folder marker; it must return false.

Gotchas and Edge Cases

• A bare PutBucketNotificationConfiguration deletes everything else. The API replaces the whole document, so enabling EventBridge with a one-key payload removes every Lambda, SQS and SNS trigger on the bucket. The read-merge-write above is not defensive style, it is the only correct sequence — and it is why two IaC stacks managing one bucket’s triggers will eventually break each other.
• wildcard and suffix are both case-sensitive. A desktop GIS export named PARCELS.SHP matches neither {"suffix": ".shp"} nor {"wildcard": "incoming/*.shp"}. Either normalise keys at upload time or list both cases explicitly; the failure is silent on both transports, because a non-matching event is simply an event nobody consumes.
• Array members are ORed, never ANDed. "key": [{"prefix": "incoming/"}, {"suffix": ".shp"}] matches anything under incoming/ or anything ending .shp anywhere in the bucket — which is almost certainly not the rule you meant, and it will look like it works because most test uploads satisfy both. wildcard is the way to get an AND on a single field.
• EventBridge does not deduplicate across rules. Two rules that both match an upload and both target the same function deliver it twice, on top of the at-least-once guarantee. This is a new failure mode relative to S3 notifications, where overlapping filters are rejected at configuration time rather than tolerated at runtime.
• Replay re-emits into a rule, not into history. A StartReplay over last Tuesday delivers those events to the current targets with current code, and the events carry their original time field. Anything downstream that partitions by ingest wall-clock will file a replayed week under today unless it reads time from the envelope.

Frequently Asked Questions

Why can an S3 suffix filter not select all shapefile components at once?

Because a key filter accepts at most one prefix rule and one suffix rule, both literal strings with no wildcards, and the rules inside one filter are ANDed. There is no way to write “.shp or .shx or .dbf”. Several configurations with different suffixes are legal only while they do not overlap for a shared event type, and they require enumerating every extension a client might send. Most teams therefore drop the suffix filter and invoke the aggregator on every object under the prefix.

Does EventBridge add meaningful latency to a spatial ingest?

It adds a hop worth a couple of hundred milliseconds against a total ingest budget of roughly a second, most of which is the consumer’s own initialisation. Against a completion gate that already waits for three or more objects to arrive, that is invisible. Against a sub-second alerting path it is not. Measure it against the pipeline’s stated staleness budget rather than deciding in the abstract.

Can I run S3 notifications and EventBridge on the same bucket?

Yes, and it is the sensible migration path. EventBridgeConfiguration sits alongside the LambdaFunctionConfigurations and QueueConfigurations in the same document, so the bucket can publish to the bus while an existing direct notification keeps working. Just remember that a consumer wired to both now receives every upload through two independent at-least-once paths, so its idempotency key must be derived from the object rather than from the delivery.

How does this compare with the GCP equivalent?

The split is different. On GCP there is no “direct versus bus” choice — Cloud Storage publishes to Pub/Sub or Eventarc either way — and Eventarc filters only on event type and bucket, as triggering GCP Cloud Functions on new shapefile uploads describes. GCP’s filtering is therefore weaker than an EventBridge pattern and its fan-out is stronger than an S3 notification’s, since additional Pub/Sub subscriptions on the same topic are free to add.

Related

• S3 and GCS Event Triggers for Shapefiles — the completion gate both transports feed, unchanged by the choice
• Aggregating Multipart Shapefile Uploads Before Processing — the manifest that has to see every component, which is why filtering matters
• Deduplicating S3 Event Notifications for Idempotent Ingestion — the at-least-once defence that both transports still require
• Triggering GCP Cloud Functions on New Shapefile Uploads — the GCP side, where Eventarc filters on event type and bucket only
• SQS and Pub/Sub Queue Routing Strategies — where a matched event goes once the trigger has decided it is worth processing

Back to S3 and GCS Event Triggers for Shapefiles