Writing STAC Items from a Lambda Tiling Job

Write the Item inside the tile worker, immediately after the COG upload returns, using an id derived from (scene_id, row_off, col_off) so the object key is deterministic and a retried invocation overwrites rather than duplicates. The Item costs about 3.6 KB of JSON and 40 ms of wall clock on a worker that already spent twenty seconds on pixels, and it carries the two extension blocks that matter for raster: proj:epsg / proj:shape / proj:transform on the Item, and raster:bands on the data asset. The footprint goes into the Item as EPSG:4326 longitude and latitude no matter what the tile’s native CRS is.

Context

The STAC cataloging and metadata publishing page covers the object model and the choice of backend. This page is the one function that runs inside a worker in a fan-out — the tile Lambdas in serverless NDVI tiling from Sentinel-2, or the map-state workers in processing a 10 GB GeoTIFF with Step Functions and Lambda.

The worker is the only process that can write this Item cheaply. It holds the Window it was given, so the tile’s affine transform is one function call away; it just wrote the array, so the nodata count and value range are already in memory; and it knows the source scene id from its own payload. A later crawler would have to re-open every tile to recover all three, which for a 240,000-tile day is 240,000 header requests it does not need to make.

Everything below therefore runs at the tail of an existing handler. Nothing here fans out, and nothing here needs orchestration.

One consequence of that placement is worth stating up front, because it shapes the code: the worker has no global view. It does not know how many sibling tiles exist, whether they succeeded, or what the Collection’s final extent will be. So the handler must never try to update a shared document — appending to a Collection’s Item list from a thousand concurrent workers is a lost-update race with no winner. The only write it performs is to a key nobody else will ever write, and the aggregation is deferred to a single finaliser that runs once the fan-out has drained.

The five steps a tile worker performs to build one STAC ItemFive sequential steps inside the tile Lambda: read the written tile's own affine transform rather than reusing the scene's; reproject the native UTM bounds into EPSG:4326 for the STAC geometry and bbox; derive an Item id from the scene id and the window row and column offsets; attach proj:epsg, proj:shape and proj:transform to the Item and raster:bands to the data asset; and validate then PutObject the Item to a deterministic key beside the COG.Five things the handler does after the COG upload returns1Read the tile's own transformThe window's affine, not the scene's — a shared origin makes every tile claim the same cornerrasterio.open2Reproject the footprint to EPSG:4326geometry and bbox are always lon/lat degrees, whatever proj:epsg saystransform_bounds3Derive the Item idscene id + product version + zero-padded row and column offsets — no clock, no uuid4..._r04096_c020484Attach the extension fieldsproj:epsg, proj:shape, proj:transform on the Item; raster:bands with nodata and statistics on the asset+1.5 KB5Validate, then writeitem.validate() raises before the PutObject, so a malformed Item never reaches the bucketPutObject
Only step one touches the raster, and it reads the IFD alone. Everything after it is arithmetic on six affine coefficients and four bounds.

Prerequisites

  • Runtime: Python 3.11 on AWS Lambda with pystac==1.11.0, rasterio==1.4.3 and boto3. pystac is pure Python and adds about 1.4 MB unzipped against the 250 MB package limit; if the worker already carries rasterio for the tiling work, no new native library is introduced. Layer construction is covered in building rasterio Lambda layers on Amazon Linux 2023.
  • Memory and timeout: whatever the raster work already requires — typically 1,769 MB and 300 seconds for a 2048 × 2048 window. The Item build adds no measurable memory against the 10,240 MB ceiling.
  • IAM: s3:PutObject on the catalogue prefix. Nothing else. Keep it off the ingest bucket entirely, in line with IAM security boundaries for cloud GIS.
  • Input contract: the message carries scene_id, acquired (an ISO 8601 UTC timestamp), the window dict of col_off / row_off / width / height, and the source scene’s transform and CRS.
  • Output layout: assets and Items share a prefix — s3://out/ndvi-10m-v1/2026/07/14/<item_id>.tif and <item_id>.json — so one deterministic key derivation serves both.

Implementation

The handler below assumes the NDVI array has already been computed by the kernel in computing NDVI per tile with rasterio and NumPy and written to tile_uri. It picks up from there.

python 
# stac_writer.py — build and publish one STAC Item from a tile worker
import json
import os
from datetime import datetime, timezone

import boto3
import numpy as np
import pystac
import rasterio
from pystac.extensions.projection import ProjectionExtension
from pystac.extensions.raster import DataType, RasterBand, RasterExtension
from rasterio.warp import transform_bounds

s3 = boto3.client("s3")

CATALOG_BUCKET = os.environ["CATALOG_BUCKET"]
COLLECTION_ID = "ndvi-10m-v1"
PRODUCT_VERSION = "v1"


def item_id(scene_id: str, row_off: int, col_off: int) -> str:
    """Pure function of the tile's identity. No uuid4, no clock, no request id.

    A 10,980 x 10,980 Sentinel-2 10 m band tiled at 2,048 px gives offsets up to
    10,240, so five zero-padded digits sort lexicographically as well as
    numerically -- which matters when a listing is the fallback enumeration.
    """
    return f"{scene_id}_{PRODUCT_VERSION}_r{row_off:05d}_c{col_off:05d}"


def item_key(acquired: datetime, iid: str) -> str:
    """Deterministic key: the same tile always lands in the same place."""
    return (f"{COLLECTION_ID}/{acquired:%Y/%m/%d}/{iid}.json")


def build_item(tile_uri: str, scene_id: str, acquired: datetime,
               row_off: int, col_off: int, ndvi: np.ndarray,
               nodata: float) -> pystac.Item:
    with rasterio.open(tile_uri) as ds:      # IFD only, no pixel fetch
        native_bounds = list(ds.bounds)
        epsg = ds.crs.to_epsg()
        shape = [ds.height, ds.width]
        # rasterio's Affine serialises as (a, b, c, d, e, f); STAC wants the
        # first six coefficients of the 3x3 matrix, in that order.
        affine = list(ds.transform)[:6]

    # STAC geometry and bbox are EPSG:4326. proj:bbox keeps the native one.
    w, s, e, n = transform_bounds(f"EPSG:{epsg}", "EPSG:4326", *native_bounds)

    iid = item_id(scene_id, row_off, col_off)
    item = pystac.Item(
        id=iid,
        geometry={
            "type": "Polygon",
            "coordinates": [[[w, s], [e, s], [e, n], [w, n], [w, s]]],
        },
        bbox=[w, s, e, n],
        datetime=acquired.astimezone(timezone.utc),
        collection=COLLECTION_ID,
        properties={
            "gsd": 10,
            "constellation": "sentinel-2",
            "tile:row_off": row_off,
            "tile:col_off": col_off,
        },
    )

    proj = ProjectionExtension.ext(item, add_if_missing=True)
    proj.epsg = epsg
    proj.shape = shape
    proj.transform = affine
    proj.bbox = native_bounds

    valid = ndvi[ndvi != nodata]
    asset = pystac.Asset(
        href=tile_uri,
        media_type=pystac.MediaType.COG,
        roles=["data"],
        title="NDVI, float32",
    )
    item.add_asset("ndvi", asset)

    RasterExtension.ext(asset, add_if_missing=True).bands = [
        RasterBand.create(
            nodata=nodata,
            data_type=DataType.FLOAT32,
            spatial_resolution=10,
            statistics={
                "minimum": round(float(valid.min()), 4) if valid.size else None,
                "maximum": round(float(valid.max()), 4) if valid.size else None,
                "mean": round(float(valid.mean()), 4) if valid.size else None,
            },
        )
    ]

    # Absolute links: the tree is served over HTTP from a CDN, and a relative
    # self link resolved across a redirect boundary builds the wrong URL.
    root = f"https://catalog.example.com/stac/{COLLECTION_ID}"
    item.set_self_href(f"{root}/{acquired:%Y/%m/%d}/{iid}.json")
    item.add_link(pystac.Link("collection", f"{root}/collection.json",
                              media_type=pystac.MediaType.JSON))
    item.add_link(pystac.Link("parent", f"{root}/{acquired:%Y/%m/%d}/catalog.json",
                              media_type=pystac.MediaType.JSON))
    return item


def publish_item(item: pystac.Item, acquired: datetime) -> str:
    item.validate()                      # raises before anything is written
    body = json.dumps(item.to_dict(), separators=(",", ":")).encode()
    key = item_key(acquired, item.id)
    s3.put_object(
        Bucket=CATALOG_BUCKET,
        Key=key,
        Body=body,
        ContentType="application/geo+json",
        CacheControl="public, max-age=300",
    )
    return key

Three details in that code do the real work.

item_id() never reads a clock or a random source. That is what makes the write idempotent: the address is derived from the tile’s identity, so the second attempt computes the same key and overwrites the first. There is no conditional put, no lock and no read-modify-write anywhere in the path — the same reasoning behind idempotency and exactly-once spatial processing.

transform_bounds converts the native UTM bounds into longitude and latitude for geometry and bbox, while proj.bbox retains the metres. Skipping the conversion produces an Item that passes schema validation — the schema only counts the numbers — and is spatially wrong by millions of metres.

The raster:bands statistics are computed from the array the worker already holds, which is the whole reason this belongs in the worker rather than in a crawler. Recovering the minimum, maximum and mean of a tile later means downloading and decompressing 16 MB of float32 pixels; here it is three NumPy reductions over an array that is already resident, costing a couple of milliseconds. The same argument applies to the valid-pixel count: it is free at write time and expensive forever after.

item.validate() runs before the put_object, not after. A malformed Item that reaches the bucket is invisible: consumers skip it and the tile silently disappears from the archive. Failing the invocation instead sends the message to a dead-letter queue where somebody sees it, which is what dead-letter queues for failed jobs exist for.

What the Item actually contains

Byte composition of a single NDVI tile STAC ItemA proportional stack of the five parts of one Item's JSON, totalling about 3.6 kilobytes: the raster extension band block with nodata, data type and statistics is the largest at 1.1 kilobytes; the core Item fields of type, stac_version, id, bbox, geometry and datetime take 0.9 kilobytes; the assets block of hrefs, media types and roles takes 0.7 kilobytes; the links block of self, parent and collection takes 0.5 kilobytes; and the projection extension fields take 0.4 kilobytes.Where the 3.6 KB of one Item goesraster: extension on the assetnodata, data_type, spatial_resolution, statistics1.1 KBCore Item fieldstype, stac_version 1.0.0, id, bbox, geometry, datetime0.9 KBassets blockndvi href, media type image/tiff; application=geotiff; profile=cloud-optimized, roles0.7 KBlinks blockself, parent, collection — absolute, because the tree is served from a CDN0.5 KBproj: extensionproj:epsg 32633, proj:shape, proj:transform, proj:bbox in metres0.4 KBOne million such Items is 3.6 GB of JSON — about two dollars a month of object storage, against roughly $120 for the database that would indexthem.
The two extension blocks are 1.5 KB of the 3.6 KB — and they are what remove an HTTP range request from every downstream reader that would otherwise open the COG to learn its grid and nodata.

What a retry does

The window that makes retries dangerous is the gap between writing the COG and writing the Item. A worker that times out at Lambda’s 15-minute ceiling in that gap leaves an orphan tile; the retry closes it.

How a deterministic Item id turns a retry into an overwriteSequence diagram with three participants: the Step Functions map state, the tile Lambda and the S3 catalogue prefix. The map state invokes the worker for window row 4096 column 2048; the worker computes its Item id as a pure function of the scene and offsets; it writes the COG and then the Item to a derived key; the first attempt times out at the 900 second Lambda ceiling after the COG write; the map state retries with the same payload; the worker computes the same id and writes the Item to the same key, overwriting rather than duplicating.A tile that times out at 900 s, retriedStep Functions mapTile LambdaS3 catalogue prefixinvoke, attempt 1window row 4096, col 2048item_id(scene, row, col)pure function — no uuid4, no clockPutObject the COGasset first: the Item promises it existstimeout at 900 sdied before the Item write — orphan tileinvoke, attempt 2identical payloadPutObject the COGsame derived key — overwritePutObject item.jsonsame derived key — one Item, not two
Nothing in this diagram is a lock, a conditional put or a read-modify-write. The retry is safe purely because the address is derived from the tile rather than allocated at write time.

Note the ordering: the asset is written first, the Item second. An Item is a promise that its asset exists, so publishing it first opens a window in which a consumer resolves an href to a 404. The reverse failure — an orphan tile with no Item — is recoverable, because a catalogue rebuild that enumerates the prefix finds it.

Verification

Assert the two properties the schema cannot check: that the id is stable across repeated builds, and that the footprint is in degrees rather than metres.

python 
# verify_item.py — the two invariants that matter
from datetime import datetime, timezone

import numpy as np

from stac_writer import build_item, item_id, item_key

acquired = datetime(2026, 7, 14, 10, 21, 9, tzinfo=timezone.utc)
ndvi = np.full((2048, 2048), 0.42, dtype="float32")

a = build_item("/tmp/t.tif", "S2B_33UUP_20260714", acquired, 4096, 2048,
               ndvi, -9999.0)
b = build_item("/tmp/t.tif", "S2B_33UUP_20260714", acquired, 4096, 2048,
               ndvi, -9999.0)

assert a.id == b.id, "id is not deterministic — something in it is allocated"
assert item_key(acquired, a.id) == item_key(acquired, b.id)
assert all(-180 <= v <= 180 for v in (a.bbox[0], a.bbox[2])), "bbox in metres"
assert all(-90 <= v <= 90 for v in (a.bbox[1], a.bbox[3])), "bbox in metres"
assert a.properties["proj:epsg"] == 32633
a.validate()

print(a.id)
print(item_key(acquired, a.id))
print([round(v, 4) for v in a.bbox])
print("stable id, degrees bbox, valid against STAC 1.0.0")

Expected output:

code 
S2B_33UUP_20260714_v1_r04096_c02048
ndvi-10m-v1/2026/07/14/S2B_33UUP_20260714_v1_r04096_c02048.json
[14.7743, 48.9021, 15.0521, 49.0862]
stable id, degrees bbox, valid against STAC 1.0.0

The bbox values are single- and double-digit degrees. If they come back as [499980.0, 5390220.0, ...] the reprojection was skipped, and the assertion above is the only thing between that and a catalogue nobody can search. To keep the gate honest, add a case that must fail — build an Item with bbox=native_bounds and assert the check rejects it.

Gotchas and Edge Cases

  • A tile of pure nodata still needs an Item — or a decision. Scene edges produce windows with no valid pixels at all, and valid.min() on an empty array raises. The code above guards with if valid.size else None, but the better answer is usually to skip the tile entirely: do not write the COG, do not write the Item, and record the skip. A catalogue full of empty Items makes every spatial search return tiles with nothing in them.
  • proj:transform must be the tile’s affine, not the scene’s. rasterio.windows.transform(window, src.transform) shifts the origin to the window’s top-left corner. Reusing the scene transform gives every tile in the scene the same origin, so proj:transform disagrees with bbox and any client that trusts it reads the wrong pixels. Reading the transform back off the written tile — as the handler above does — makes this mistake structurally impossible.
  • datetime must be timezone-aware and UTC. pystac serialises a naive datetime without an offset, and consumers then disagree about what it means. Parse the acquisition time with an explicit timezone and call .astimezone(timezone.utc) before it reaches the Item.
  • item.validate() reaches the network on a cold start. pystac fetches the JSON schemas for the core spec and every declared extension the first time it validates, then caches them for the life of the process. In a Lambda that means a few hundred milliseconds added to the first invocation of each execution environment, and a hard failure if the function sits in a VPC with no egress. Vendor the schemas into the deployment package and point pystac’s validator at the local copies — the same version-pinning discipline as pinning GDAL and PROJ versions across build and runtime.
  • Setting Content-Type: application/json on an Item is a downgrade, not an error. An Item is GeoJSON; application/geo+json is the correct media type and some browsers and clients branch on it. Set it on the put_object call, because S3 will otherwise infer binary/octet-stream from a key with no recognised extension.

Frequently Asked Questions

Why derive the Item id from the window instead of using uuid4?

Because the object key is derived from the id, and an idempotent write needs a derived address rather than an allocated one. A tile identified by scene id plus row and column offsets always produces the same key, so a retried invocation overwrites its own earlier Item. With uuid4() the retry writes a second Item pointing at the same asset, and the catalogue permanently records two observations where one exists.

Should the Item be written before or after the COG?

After, always. The Item is a promise that the asset exists, so publishing it first creates a window in which a consumer can resolve an href to a 404. Write the COG, confirm the upload returned, then build the Item from the written file’s own header. If the invocation dies between the two writes you have an orphan tile, which a catalogue rebuild can discover and fix; the reverse ordering leaves a broken Item that only a consumer discovers.

How large is a STAC Item for one tile, and does it affect the Lambda budget?

About 3.6 KB of JSON with the projection and raster extensions attached, and roughly 40 ms of wall clock including the PutObject. Against a worker that already holds two 2048 × 2048 arrays and spends twenty seconds on pixels, that is noise. The only budget it touches meaningfully is the deployment package: pystac adds about 1.4 MB unzipped against the 250 MB Lambda limit.

Back to STAC Cataloging and Metadata Publishing