VPC Endpoints and Private Access for Spatial Buckets

Attach an S3 gateway endpoint to the route tables of the subnets your function runs in, add aws:SourceVpce as a Condition on the bucket policy, and every /vsis3/ range request leaves through a private route at zero data-processing cost — while the same bucket becomes unreachable from the public internet even with valid credentials. Gateway endpoints cost nothing, remove NAT gateway charges of roughly $0.045 per GB on read-heavy raster traffic, and add about 100–250 ms to a cold start once the Hyperplane ENI for that subnet and security-group pair exists. Interface endpoints, GCP Private Service Connect and Azure Private Endpoints solve the same problem with different bills and different failure modes.

What a Private Path Buys a Raster Pipeline

IAM Security Boundaries for Cloud GIS scopes who may call the storage API. A private endpoint scopes from where, and the two compose into a boundary neither achieves alone. A leaked access key for a role scoped to s3:GetObject on /imagery/* is a data exfiltration event; the same key against a bucket whose policy carries "Condition": {"StringNotEquals": {"aws:SourceVpce": "vpce-0abc…"}} with an explicit Deny is inert, because the request arrives from the internet gateway and is refused before the role is even evaluated.

The economics reinforce the security argument, which is unusual and worth exploiting. A function in a private subnet without an endpoint reaches S3 through a NAT gateway, and NAT is billed both hourly and per gigabyte processed — around $0.045/GB in us-east-1. Raster pipelines move a lot of gigabytes: a tiling fan-out that reads 40 GB of COG per run, four times a day, is 4.8 TB a month through NAT. A gateway endpoint removes that charge entirely because it is not a device — it is a route table entry that sends the S3 prefix list to the endpoint instead of the NAT.

There is a third benefit that matters specifically for streaming reads. Range requests over /vsis3/ are numerous and small; the pattern described in Streaming COGs Without Touching /tmp issues several per window. Every one of those crosses the NAT gateway’s connection table when there is no endpoint, and a fan-out of 200 concurrent functions each holding a dozen connections is a well-known way to exhaust NAT ports and start seeing ErrorPortAllocation. The endpoint path has no such table.

The Three Providers

AWS offers two shapes. A gateway endpoint supports S3 and DynamoDB only, is free, and works by adding the service’s prefix list to a route table — so it is invisible to DNS and only serves resources inside the VPC. An interface endpoint (AWS PrivateLink) creates an ENI with a private IP in each subnet you choose, is billed per AZ-hour plus per GB processed, and does work across VPC peering, Direct Connect and from on-premises. For a function reading a same-region bucket, gateway is correct and interface is a needless bill. Note that S3 interface endpoints do not by default override the public s3.amazonaws.com name — you either enable private DNS or address the endpoint-specific hostname, and GDAL needs AWS_S3_ENDPOINT set for the latter.

GCP also offers two. Private Google Access is a per-subnet boolean that lets instances without external IPs reach Google APIs including storage.googleapis.com; it is free and is the equivalent of a gateway endpoint. Private Service Connect goes further, allocating an address inside your VPC that a DNS zone maps storage.googleapis.com onto, which is what you need for hybrid connectivity or for VPC Service Controls perimeters. Cloud Functions and Cloud Run reach both through a Serverless VPC Access connector, which is billed per instance-hour and is the component that actually costs money.

Azure has one mechanism, the Private Endpoint: a NIC in your VNet with a private IP, plus a privatelink.blob.core.windows.net private DNS zone that overrides the public name. It is billed per endpoint-hour plus per GB. The function app reaches it through VNet integration, and — the detail that catches people — outbound VNet integration alone is not enough; WEBSITE_VNET_ROUTE_ALL=1 must be set or only RFC1918 destinations are routed through the VNet, and privatelink.blob.core.windows.net resolves to a public address for everything else.

Prerequisites

• Runtime: Python 3.11/3.12 with rasterio 1.3.9+ / GDAL 3.6+, or the Terraform 1.6+ / AWS provider 5.x used below.
• Networking that must exist first: at least two private subnets in different availability zones, a route table associated with each, and a security group whose egress allows 443. A Lambda in a public subnet with an internet gateway has no internet access at all — it needs a NAT or an endpoint regardless.
• IAM for the deploying principal: ec2:CreateVpcEndpoint, ec2:ModifyVpcEndpoint, ec2:DescribeVpcEndpoints, s3:PutBucketPolicy. The function’s own execution role additionally needs ec2:CreateNetworkInterface, ec2:DescribeNetworkInterfaces and ec2:DeleteNetworkInterface — normally supplied by AWSLambdaVPCAccessExecutionRole — or the function will fail to initialise with EC2AccessDeniedException and no clearer message.
• Environment variables in the function configuration:
  • GDAL_DATA=/opt/share/gdal, PROJ_LIB=/opt/share/proj, LD_LIBRARY_PATH=/opt/lib
  • PROJ_NETWORK=OFF — mandatory here rather than merely advisable. A private subnet has no route to the PROJ CDN, so a datum-grid fetch does not fail fast; it hangs until GDAL_HTTP_TIMEOUT expires.
  • AWS_STS_REGIONAL_ENDPOINTS=regional — the global STS endpoint is not reachable through an S3 gateway endpoint, and credential refresh in a long-running container will hang without this.
  • GDAL_HTTP_TIMEOUT=20 and GDAL_HTTP_MAX_RETRY=3 — so a misrouted request surfaces as an error inside the invocation instead of consuming the whole timeout.
  • GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR, CPL_VSIL_CURL_CACHE_SIZE=67108864
• A test object in the bucket you can read, plus a second identity outside the VPC to prove the deny works.

Implementation

# ---------------------------------------------------------------------------
# S3 gateway endpoint + a bucket policy that makes the VPC the only way in.
# Gateway endpoints are free: no hourly rate, no per-GB data processing.
# ---------------------------------------------------------------------------
resource "aws_vpc_endpoint" "s3" {
  vpc_id            = var.vpc_id
  service_name      = "com.amazonaws.${var.region}.s3"
  vpc_endpoint_type = "Gateway"

  # The endpoint is a ROUTE, so it must be attached to the route table of every
  # subnet the function's ENIs live in. A missing association is the single
  # most common cause of "it works in AZ a and times out in AZ b".
  route_table_ids = var.private_route_table_ids

  # Endpoint policy: a second boundary, independent of the bucket policy and of
  # the execution role. Traffic through this endpoint can reach these two
  # buckets and nothing else, whatever the caller's IAM permits.
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = "*"
      Action    = ["s3:GetObject", "s3:PutObject", "s3:ListBucket"]
      Resource = [
        aws_s3_bucket.imagery.arn,
        "${aws_s3_bucket.imagery.arn}/*",
        aws_s3_bucket.tiles.arn,
        "${aws_s3_bucket.tiles.arn}/*",
      ]
    }]
  })

  tags = { Name = "geo-s3-gateway" }
}

# ---------------------------------------------------------------------------
# Deny everything that does not arrive through the endpoint. This is what makes
# a leaked key inert: the request is refused before the role is evaluated.
# ---------------------------------------------------------------------------
resource "aws_s3_bucket_policy" "imagery_vpce_only" {
  bucket = aws_s3_bucket.imagery.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Sid       = "DenyOutsideTheVpcEndpoint"
      Effect    = "Deny"
      Principal = "*"
      Action    = "s3:*"
      Resource = [
        aws_s3_bucket.imagery.arn,
        "${aws_s3_bucket.imagery.arn}/*",
      ]
      Condition = {
        StringNotEquals = { "aws:SourceVpce" = aws_vpc_endpoint.s3.id }
        # Without this exception the policy locks out the console session that
        # would fix it. Keep exactly one break-glass principal.
        ArnNotLike = { "aws:PrincipalArn" = var.breakglass_role_arn }
      }
    }]
  })
}

# ---------------------------------------------------------------------------
# The function. Placing it in private subnets is what makes the endpoint apply;
# a Lambda outside a VPC never traverses your route tables at all.
# ---------------------------------------------------------------------------
resource "aws_lambda_function" "cog_reader" {
  function_name = "cog-reader"
  role          = aws_iam_role.cog_reader.arn
  package_type  = "Image"
  image_uri     = var.image_uri
  memory_size   = 3008
  timeout       = 300

  vpc_config {
    # Two AZs: the Hyperplane ENI is created per unique subnet + security-group
    # pair, so keeping this list small keeps the one-off ENI setup small too.
    subnet_ids         = var.private_subnet_ids
    security_group_ids = [aws_security_group.egress_443.id]
  }

  environment {
    variables = {
      GDAL_DATA                    = "/opt/share/gdal"
      PROJ_LIB                     = "/opt/share/proj"
      LD_LIBRARY_PATH              = "/opt/lib"
      GDAL_DISABLE_READDIR_ON_OPEN = "EMPTY_DIR"
      CPL_VSIL_CURL_CACHE_SIZE     = "67108864"
      GDAL_HTTP_TIMEOUT            = "20"
      GDAL_HTTP_MAX_RETRY          = "3"
      # No route to the PROJ CDN from a private subnet — fail closed, not slow.
      PROJ_NETWORK                 = "OFF"
      # Global STS is not reachable via the S3 gateway endpoint.
      AWS_STS_REGIONAL_ENDPOINTS   = "regional"
    }
  }
}

The endpoint policy and the bucket policy do different jobs and you want both. The endpoint policy constrains what may be reached through this door regardless of who is knocking; the bucket policy constrains which doors are acceptable regardless of what is behind them. Together with the per-stage execution roles from IAM Security Boundaries for Cloud GIS, a compromise has to defeat three independent controls.

Verification

Prove three things: that the private path works, that the public path is refused, and that the cold-start cost is what you budgeted.

"""Run inside the VPC-attached function. Asserts the path, not just the read."""
import json, socket, time
import rasterio
from rasterio.session import AWSSession

def handler(event, context):
    t0 = time.perf_counter()
    # An S3 gateway endpoint does NOT change DNS: the name still resolves to a
    # public S3 address. The proof of the private path is the route, so check
    # reachability and latency rather than expecting a 10.x.x.x answer.
    resolved = socket.gethostbyname("s3.us-east-1.amazonaws.com")

    with rasterio.Env(session=AWSSession(), GDAL_DISABLE_READDIR_ON_OPEN="EMPTY_DIR",
                      GDAL_HTTP_TIMEOUT="20"):
        with rasterio.open("/vsis3/geo-imagery/probe.tif") as src:
            shape = (src.width, src.height)
            blocks = src.block_shapes[0]

    return {"statusCode": 200, "body": json.dumps({
        "resolved_s3_ip": resolved,
        "shape": shape,
        "blocks": list(blocks),
        "read_ms": round((time.perf_counter() - t0) * 1000, 1),
    })}

# 1. From inside the VPC — must succeed.
aws lambda invoke --function-name cog-reader --payload '{}' \
  --cli-binary-format raw-in-base64-out /dev/stdout | jq -r '.body'

# 2. From your laptop with a fully-permitted role — must be denied.
aws s3api get-object --bucket geo-imagery --key probe.tif /dev/null

# 3. Confirm the endpoint is on every relevant route table.
aws ec2 describe-vpc-endpoints --vpc-endpoint-ids "$VPCE" \
  --query 'VpcEndpoints[0].RouteTableIds' --output text

Expected output — a successful private read, an explicit denial from outside, and the endpoint present on both route tables:

{"resolved_s3_ip":"52.216.xxx.xx","shape":[10980,10980],"blocks":[512,512],"read_ms":143.7}

An error occurred (AccessDenied) when calling the GetObject operation:
User: arn:aws:iam::…:role/admin is not authorized to perform: s3:GetObject
with an explicit deny in a resource-based policy

rtb-0aa11bb22cc33dd44	rtb-0ee55ff66aa77bb88

The denial message naming an explicit deny in a resource-based policy is the specific string that confirms the aws:SourceVpce condition fired, rather than an ordinary permissions gap. If you get AccessDenied without that phrase, the role is wrong and the endpoint is untested.

What It Costs

The ENI cost is real but small and frequently overstated. Before 2019 a VPC-attached Lambda created an ENI per execution environment and cold starts ran to 8–10 seconds. Hyperplane ENIs changed that: one ENI is created per unique subnet-plus-security-group combination and shared across every execution environment of every function using it. The one-off creation takes tens of seconds and happens at first deploy; after that the per-cold-start addition is roughly 100–250 ms. Measure it with the instrumentation from Measuring Cold Starts with CloudWatch and Cloud Trace rather than trusting the number — it varies with subnet size and AZ count.

Against that, a 4.8 TB/month read volume through NAT is about $216 in data processing alone, before the roughly $32/month hourly charge per NAT gateway. The gateway endpoint replaces both with zero. On GCP the connector’s instance-hours are the cost and Private Google Access itself is free; on Azure the Private Endpoint’s hourly and per-GB charges apply but replace the same egress.

Gotchas

• A gateway endpoint changes routing, not DNS. s3.us-east-1.amazonaws.com still resolves to a public IP address. Engineers who expect a private IP conclude the endpoint is not working and add an interface endpoint they do not need. Verify by checking the route tables and the bucket-policy denial, never by nslookup.

• The endpoint must be on the route table of every subnet the ENIs use. Attach a function to three subnets, associate the endpoint with two route tables, and roughly a third of your invocations time out — non-deterministically, because which AZ serves an invocation is not under your control.

• Global STS and the PROJ CDN are not reachable through an S3 gateway endpoint. The endpoint carries S3 traffic only. Anything else your function calls — STS for credential refresh, Secrets Manager, the PROJ grid CDN — needs its own interface endpoint or a NAT. AWS_STS_REGIONAL_ENDPOINTS=regional and PROJ_NETWORK=OFF remove the two that bite geospatial functions specifically.

• WEBSITE_VNET_ROUTE_ALL=1 is required on Azure, and its absence fails silently. With VNet integration but without that setting, only RFC1918 destinations route through the VNet; privatelink.blob.core.windows.net resolves publicly and the Private Endpoint is bypassed while everything appears to work. The tell is that reads succeed from a network you expected to be blocked.

Frequently Asked Questions

Does an S3 gateway endpoint cost anything?

No — no hourly charge and no per-GB charge. It is a route table entry pointing the S3 prefix list at the endpoint. An interface endpoint is billed per availability-zone-hour plus per GB processed, so gateway is the right default for a same-region bucket and interface is for cross-VPC or hybrid access.

How much cold start does attaching a Lambda to a VPC add?

Roughly 100–250 ms once the Hyperplane ENI for that subnet and security-group pair exists, not the 8–10 seconds of the pre-2019 behaviour. The ENI is created once and shared across every execution environment, so only the first deployment of a new configuration pays a delay measured in tens of seconds.

What is the GCP equivalent of an S3 gateway endpoint?

Private Google Access — a free per-subnet flag that lets instances without external IPs reach storage.googleapis.com. Private Service Connect is the heavier option, giving you an address inside your VPC that the storage hostname resolves to, which you need for hybrid connectivity or VPC Service Controls perimeters.

Can I use a private endpoint with a public open-data bucket?

Not usefully. Buckets such as the Sentinel-2 open archive are reached over the public S3 service, and a gateway endpoint’s route covers the whole S3 prefix list, so those reads do traverse it — but you cannot apply a aws:SourceVpce condition to a bucket you do not own. If you need both private control and open data, copy the scenes you need into a bucket you control at ingest.

Related

• IAM Security Boundaries for Cloud GIS — the identity half of the boundary this page completes
• Least-Privilege IAM Policies for Azure Blob Geospatial Access — the role scoping a Private Endpoint sits in front of
• Streaming COGs Without Touching /tmp — the range-request traffic whose volume makes NAT charges material
• Measuring Cold Starts with CloudWatch and Cloud Trace — measuring the ENI attachment cost rather than assuming it
• Serverless Geospatial Architecture & Platform Limits — where private networking sits among the other platform constraints

Back to IAM Security Boundaries for Cloud GIS