Triggering GCP Cloud Functions on New Shapefile Uploads

Deploy a Cloud Functions 2nd gen handler with --trigger-event-filters="type=google.cloud.storage.object.v1.finalized" scoped to your bucket, filter inside the handler to .shp objects only, verify that the mandatory companion files (.shx, .dbf) exist at the same prefix, check a .processed marker for idempotency, then validate with Fiona. Cloud Functions 2nd gen supports up to 60 minutes of execution time and 32 GB of memory, which comfortably covers large municipal shapefile bundles. GCS delivers finalized events via Eventarc within seconds of the upload completing.

Context: Why a Naive Trigger Fails for Shapefiles

Google Cloud Storage fires one event per object, not per logical dataset. A shapefile is a bundle of at least three files — .shp (geometry), .shx (index), .dbf (attributes) — and optionally .prj, .cpg, .sbn, and .sbx. If you attach a Cloud Function to every finalized event without filtering, the function invokes on the .shx arrival before the geometry file is present, or on the .dbf before the index exists. Both scenarios produce a fiona.errors.DriverError: Failed to open datasource that looks like a transient failure but is actually a race condition.

This is the core atomicity problem described in S3 and GCS Event Triggers for Shapefiles, where a completion gate prevents premature dispatch. On Cloud Functions specifically, the preferred gate is to trigger only on .shp finalization (the last large file in a typical upload sequence) and immediately list the bucket prefix to confirm the companions have arrived. Unlike AWS Lambda where SQS and Pub/Sub Queue Routing Strategies add a Pub/Sub buffer before the compute layer, a 2nd gen Cloud Function connected directly to Eventarc can self-validate and skip processing when companions are absent — the next upload event (or a retry from the client) will trigger a fresh, complete run.

Prerequisites

Before deploying, confirm all of the following are in place:

• Python runtime: python311 (Cloud Functions 2nd gen; Python 3.12 is also available but Fiona wheels are more reliable on 3.11 as of mid-2026)
• IAM — service account roles on the source bucket:
  • roles/storage.objectViewer — to list the prefix and download the shapefile bundle
  • roles/storage.objectCreator — to write the .processed marker blob
• IAM — Eventarc agent: roles/eventarc.eventReceiver on the project (granted automatically when you enable the Eventarc API, but confirm it exists)
• APIs enabled: Cloud Functions, Cloud Build, Eventarc, Cloud Storage, Artifact Registry
• Environment variables set in the function config:
  • GDAL_DATA — path to the GDAL data directory inside the runtime (Cloud Functions 2nd gen bundles GDAL; the runtime sets this automatically, but verify with import os; os.environ.get("GDAL_DATA"))
  • PROJ_LIB — path to the PROJ datum grids (also set automatically; confirm with a test invocation)
• Python dependencies (requirements.txt):
  code Copy

  functions-framework==3.*
  google-cloud-storage==2.*
  fiona==1.9.*
  

• Quota: Cloud Functions 2nd gen supports up to 60 min timeout and 32 GB memory per instance. For large shapefiles approaching hundreds of MB, allocate at least 2 GiB and set --timeout=300s.

Implementation

Deploy the function

gcloud functions deploy process-shapefile \
  --gen2 \
  --runtime=python311 \
  --region=us-central1 \
  --source=. \
  --entry-point=process_shapefile \
  --trigger-event-filters="type=google.cloud.storage.object.v1.finalized" \
  --trigger-event-filters="bucket=YOUR_GCS_BUCKET" \
  --trigger-location=us-central1 \
  --memory=2Gi \
  --timeout=300s \
  --service-account=shp-ingest-sa@YOUR_PROJECT_ID.iam.gserviceaccount.com \
  --set-env-vars="SOURCE_BUCKET=YOUR_GCS_BUCKET,DESTINATION_BUCKET=YOUR_DEST_BUCKET"

The --trigger-event-filters flag uses Eventarc’s native filtering rather than legacy Pub/Sub topic routing. Separate type and bucket filters must be passed as independent flags (not comma-joined) with --gen2.

Python handler (main.py)

import os
import tempfile
import logging
from pathlib import Path

from google.cloud import storage
from google.cloud.functions_v2 import CloudEvent
import fiona

# Cloud Functions initialises GDAL_DATA and PROJ_LIB automatically in the
# 2nd gen runtime. Confirm here so any misconfiguration surfaces early.
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

logger.info("GDAL_DATA=%s", os.environ.get("GDAL_DATA", "NOT SET — check runtime"))
logger.info("PROJ_LIB=%s",  os.environ.get("PROJ_LIB",  "NOT SET — check runtime"))

SOURCE_BUCKET = os.environ["SOURCE_BUCKET"]
DESTINATION_BUCKET = os.environ.get("DESTINATION_BUCKET", SOURCE_BUCKET)

# Minimum companions required before processing begins.
REQUIRED_EXTS = {".shp", ".shx", ".dbf"}
# Optional files downloaded if present; absence is logged but not fatal.
OPTIONAL_EXTS = {".prj", ".cpg", ".sbn", ".sbx"}
# Marker blob name suffix written after a successful run.
PROCESSED_SUFFIX = ".processed"

# Reuse the storage client across warm invocations.
_storage_client = storage.Client()


def process_shapefile(event: CloudEvent) -> None:
    """Entry point for Cloud Functions 2nd gen (CloudEvent signature)."""
    data = event.data
    bucket_name: str = data["bucket"]
    object_name: str = data["name"]

    # 1. Filter: only react to the primary geometry file.
    if not object_name.lower().endswith(".shp"):
        logger.info("Ignoring non-.shp object: %s", object_name)
        return

    bucket = _storage_client.bucket(bucket_name)
    stem = Path(object_name).stem
    # Preserve any directory prefix (e.g. "uploads/2026/parcels").
    prefix = str(Path(object_name).parent).rstrip("/")
    prefix_sep = f"{prefix}/" if prefix != "." else ""

    marker_path = f"{prefix_sep}{stem}{PROCESSED_SUFFIX}"

    # 2. Idempotency gate: skip if already processed.
    if bucket.blob(marker_path).exists():
        logger.info("Already processed — skipping: %s", stem)
        return

    # 3. Verify all required companion files are present at the same prefix.
    required_blobs = {
        ext: f"{prefix_sep}{stem}{ext}" for ext in REQUIRED_EXTS
    }
    missing = [
        path for path in required_blobs.values()
        if not bucket.blob(path).exists()
    ]
    if missing:
        # Companions have not arrived yet; the final upload will re-trigger.
        logger.warning(
            "Companion files missing for %s — deferring: %s", stem, missing
        )
        return

    # 4. Collect optional files that happen to be present.
    optional_blobs = {
        ext: f"{prefix_sep}{stem}{ext}"
        for ext in OPTIONAL_EXTS
        if bucket.blob(f"{prefix_sep}{stem}{ext}").exists()
    }
    if ".prj" not in optional_blobs:
        logger.warning(
            ".prj absent for %s — CRS will be None; downstream must reproject.", stem
        )

    # 5. Download the full bundle to ephemeral /tmp storage.
    #    Cloud Functions 2nd gen provides 512 MiB of /tmp by default;
    #    raise --ephemeral-storage-size if your shapefiles exceed that.
    all_blobs = {**required_blobs, **optional_blobs}
    with tempfile.TemporaryDirectory() as tmp_dir:
        local_paths: dict[str, str] = {}
        for ext, blob_path in all_blobs.items():
            local_file = os.path.join(tmp_dir, f"{stem}{ext}")
            bucket.blob(blob_path).download_to_filename(local_file)
            local_paths[ext] = local_file
            logger.debug("Downloaded %s → %s", blob_path, local_file)

        # 6. Validate geometry and attribute schema with Fiona.
        shp_local = local_paths[".shp"]
        try:
            with fiona.open(shp_local, "r") as src:
                feature_count = len(src)
                crs = src.crs
                geom_type = src.schema["geometry"]
                logger.info(
                    "Validated %s: %d features, geometry=%s, CRS=%s",
                    stem, feature_count, geom_type, crs,
                )
        except fiona.errors.DriverError as exc:
            logger.error("Fiona validation failed for %s: %s", stem, exc)
            # Re-raise so Cloud Functions marks the invocation as failed
            # and Eventarc applies its built-in retry policy.
            raise

    # 7. Write idempotency marker and signal downstream routing.
    bucket.blob(marker_path).upload_from_string(
        "processed",
        content_type="text/plain",
    )
    logger.info("Marker written: %s", marker_path)

    # Emit a Pub/Sub message or update Firestore here to trigger
    # downstream ETL (BigQuery load, PostGIS ingest, Data Catalog registration).
    # The validated metadata is: stem, feature_count, geom_type, crs, prefix_sep.
    logger.info("Ready for downstream routing: %s/%s", prefix_sep, stem)

Verification

After deploying, upload a complete shapefile bundle and confirm the function executed correctly:

# Upload the full bundle (all four files at once)
gsutil cp parcels.{shp,shx,dbf,prj} gs://YOUR_GCS_BUCKET/test/

# Tail the Cloud Logging output for the function
gcloud functions logs read process-shapefile \
  --gen2 \
  --region=us-central1 \
  --limit=50

Expected log output (in order of emission):

Ignoring non-.shp object: test/parcels.shx
Ignoring non-.shp object: test/parcels.dbf
Ignoring non-.shp object: test/parcels.prj
Downloaded gs://…/test/parcels.shp → /tmp/…/parcels.shp
Downloaded gs://…/test/parcels.shx → /tmp/…/parcels.shx
Downloaded gs://…/test/parcels.dbf → /tmp/…/parcels.dbf
Downloaded gs://…/test/parcels.prj → /tmp/…/parcels.prj
Validated parcels: 14823 features, geometry=Polygon, CRS=EPSG:4326
Marker written: test/parcels.processed
Ready for downstream routing: test/ / parcels

Confirm the marker exists:

gsutil ls gs://YOUR_GCS_BUCKET/test/parcels.processed
# Expected: gs://YOUR_GCS_BUCKET/test/parcels.processed

Upload the same bundle a second time and verify the idempotency log line fires:

gsutil cp parcels.shp gs://YOUR_GCS_BUCKET/test/
# Logs should show: Already processed — skipping: parcels

Frequently Asked Questions

Why does my function fire three times for the same shapefile?

Cloud Functions 2nd gen with Eventarc uses at-least-once delivery. GCS may also emit a duplicate finalized event if the upload completes during a transient network partition and the client retries. The .processed marker handles all of these: the second and third invocations hit the idempotency check at step 2 and return immediately without reading the bundle.

Why trigger only on .shp and not on .shx or .dbf uploads?

GCS emits one event per file. Triggering on .shx or .dbf means the function fires before the geometry file (.shp) is present, causing GDAL to throw OGRERR_NOT_ENOUGH_DATA. Scoping to .shp gives you a deterministic entry point, after which the handler verifies that the companions have arrived.

What IAM roles does the Cloud Functions service account need?

roles/storage.objectViewer on the source bucket (to list and read the shapefile bundle) and roles/storage.objectCreator on the same or a destination bucket (to write the .processed marker). Never grant roles/storage.admin in production. If you write processed output to a separate bucket, scope objectCreator to that bucket only.

How do I handle a .prj file that is missing?

Treat .prj as optional but log a warning when absent. Fiona will still open the shapefile and report CRS as None. Your handler should reproject to a known CRS (e.g. EPSG:4326) with a default assumption logged to Cloud Logging, then flag the output dataset with a crs_assumed=true metadata attribute so downstream consumers know the projection was inferred.

Gotchas and Edge Cases

• Function invoked before .shp has fully finalized on GCS: Very large .shp files (>1 GB) can trigger the finalized event from the storage layer milliseconds before the object is fully consistent across GCS replicas. Add a time.sleep(2) after confirming the marker is absent but before listing companions — this is the one acceptable use of sleep here because GCS eventual consistency is real and usually resolves within 1–2 seconds.

• Ephemeral storage exhaustion: Cloud Functions 2nd gen provides 512 MiB of /tmp by default. A shapefile bundle with a large .dbf (attribute-heavy parcel datasets can exceed 400 MiB) will exhaust this. Set --ephemeral-storage-size=2Gi on the gcloud functions deploy command. The cold start behavior for Python GDAL also increases when memory is bumped; consider provisioned concurrency if latency matters.

• Trigger location mismatch: --trigger-location must match the region of the GCS bucket. Cross-region Eventarc triggers are not supported for Cloud Storage events. If your bucket is us-east1, set --trigger-location=us-east1 and --region=us-east1.

• fiona.errors.DriverError on valid files after cold start: On the first invocation after a cold start, GDAL’s shared-library registration can fail if GDAL_DATA or PROJ_LIB are not exported before Fiona imports. The handler above logs both variables at module load time, which surfaces this immediately. If you see PROJ: proj_create_from_database: Cannot find proj.db, the PROJ database path is wrong — check that the pyproj wheel bundled with your requirements.txt includes the data files (pyproj ships proj.db internally; do not pin pyproj below 3.4).

Related

• S3 and GCS Event Triggers for Shapefiles — parent page covering the multi-file atomicity problem and the completion-gate pattern for both AWS and GCP
• SQS and Pub/Sub Queue Routing Strategies — add a Pub/Sub layer between Cloud Functions and downstream ETL for fan-out routing and backpressure control
• Implementing Dead-Letter Queues for Failed Vector Jobs — route permanently failed shapefile ingestion events to a DLQ for inspection without blocking the main pipeline
• Reducing Python GDAL Cold Starts with Provisioned Concurrency — mitigate the GDAL/Fiona shared-library initialization latency that appears on the first invocation after deployment
• When to Use Batch vs Streaming for Real-Time AIS Tracking — contrasts event-per-file triggers (used here) with continuous stream processing for high-frequency vector data

Back to S3 and GCS Event Triggers for Shapefiles