STAC Cataloging and Metadata Publishing

A pipeline that writes a tile and stops has produced a file nobody can find. Cataloguing it as a SpatioTemporal Asset Catalog Item costs roughly 3.6 KB of JSON and about 40 ms of the worker’s runtime, and it is the difference between an S3 prefix of 240,000 anonymous GeoTIFFs and a queryable archive. Every other recipe on this site — NDVI tiling from Sentinel-2, the 10 GB GeoTIFF Step Functions fan-out, the Cloud Run vector tile pipeline — ends at the moment bytes land in object storage. This page covers the stage after that: turning those bytes into Items, Collections and a browsable tree, with the STAC specification pinned at version 1.0.0.

Why Cataloguing Matters for Geospatial Workloads

Object storage has no concept of geometry. A bucket holding one Sentinel-2 derived product per day for a decade is 3,650 scene prefixes, each holding perhaps 30 tiles, and the only structure available to a consumer is the key naming convention that whoever wrote the pipeline happened to invent. Answering “which tiles cover this farm on the third week of June” means listing prefixes and parsing filenames — an operation that is O(everything) and breaks the first time the naming convention changes.

STAC fixes this by making the metadata a first-class artifact rather than a comment in the key. An Item is a GeoJSON Feature: it carries a geometry in EPSG:4326, a bounding box, a datetime, arbitrary properties, and a set of Assets that are hrefs to real bytes. Because it is GeoJSON, every spatial tool already reads it. Because it is a plain file, it can be published from a Lambda with a single PutObject and served from a CDN with no server behind it.

The serverless angle matters more than it looks. In a fan-out pipeline, the worker that produced a tile is the only process that knows the tile’s window transform, its valid-pixel count and its nodata statistics — the orchestrator saw only offsets, and any later scan would have to re-open the file. Writing the Item inside the worker turns metadata publishing into a constant-cost side effect of work already done. Deferring it to a nightly crawler turns it into a full re-read of the archive, which for a 240,000-tile day is 240,000 header requests against object storage.

There is a second, less obvious payoff in the Collection. A Collection carries summaries — the range or set of values each property takes across its Items — and that block is what lets a client decide whether a Collection is worth walking at all before it fetches a single Item. A summaries entry saying this product only exists between 2019 and 2026, only over MGRS zones 32 and 33, and only at 10 m ground sample distance answers most “do you have data for me” questions in one 4 KB request. Pipelines that skip summaries force every consumer to discover the same facts by exhaustive traversal.

The cost of not cataloguing compounds silently. A pipeline that reprocesses under at-least-once delivery writes duplicate tiles; without a catalogue you cannot tell a duplicate from a legitimate revision, and without idempotent Item ids the catalogue records both. The discipline that makes a tiling pipeline correct is the same discipline that makes its catalogue correct, which is why this stage belongs in the pipeline rather than beside it.

The four object types

STAC has exactly four, and only the last one contains data.

A Catalog is a node with links and a description. A Collection is a Catalog that additionally declares a spatial and temporal extent, a licence and summaries of its Items’ properties — one product family, such as ndvi-10m-v1. An Item is one observation of one area at one time. An Asset is an href plus a media type. Everything above the Asset is link plumbing, which is exactly why a catalogue can be repartitioned or republished without touching a single pixel.

Platform-by-Platform: Where a Catalogue Lives

A static catalogue needs three things from a cloud: an object store to hold the JSON, a CDN to serve it with the CORS headers a browser needs, and — if you want search — a managed place to run the index. Every major provider has all three, but they are not equivalent.

The Item writer’s ceilings almost never bind, because writing one Item is a few kilobytes of JSON serialisation. The tree rebuild is where the platform column starts to matter: regenerating the Catalog nodes for a million-Item archive means holding a link list in memory and writing a few thousand small objects, and on Azure Functions Consumption the 10-minute ceiling and 1,536 MB memory make that a container job rather than a function. The same escape hatch as everywhere else on this site applies — the timeout ceiling comparison for geospatial jobs lays out when a job has outgrown its function runtime.

One AWS-specific detail is worth pinning down: Lambda’s 250 MB unzipped deployment limit. pystac is pure Python and adds about 1.4 MB, but pyproj — which you need to reproject the tile footprint into EPSG:4326 — brings its own PROJ data directory. If the tiling worker already carries rasterio, both are present and the Item build is free; if you split cataloguing into its own function, keep it to pystac plus a hand-rolled transform rather than pulling the whole geo stack in, as covered in stripping unnecessary Python packages from AWS Lambda layers.

Static Catalogue or STAC API

This is the decision that determines everything downstream, and it is usually made backwards — teams reach for the API because it sounds more capable, then operate a PostgreSQL cluster for a workload that never issues a spatial query.

The rule is straightforward. If every consumer already knows which partition it wants, a static catalogue is correct. A dashboard that renders “yesterday’s NDVI for tile 33UUP” knows the date and the tile; it needs a URL, not a query engine. If a consumer needs to ask a question whose answer is a set the tree does not enumerate — intersect this polygon, between these dates, cloud cover under 10% — that is an index query and you need a STAC API. Walking a static tree to answer it means downloading every Item.

pgstac is the standard answer for the API side: a PostgreSQL schema plus PL/pgSQL functions that store Items in partitioned tables and implement the STAC API search endpoints, usually fronted by stac-fastapi-pgstac on Cloud Run, Fargate or Lambda behind an API Gateway. The operational cost is real. A pool-less Lambda front end opening one connection per invocation will exhaust PostgreSQL’s connection limit long before it exhausts Lambda’s 1,000 regional concurrency, so an API on Lambda needs RDS Proxy or an equivalent pooler in front of the database — the same fan-out pressure described in concurrency and throttling for tile fan-out, pointed at a database instead of a function.

There is a compatibility argument for the API that is worth weighing honestly. pystac-client and most off-the-shelf tooling speak the STAC API search interface, so exposing one means consumers write three lines instead of a traversal loop. But the same library reads a static tree perfectly well through Client.open() on a root Catalog — it simply walks links instead of issuing a search. The convenience gap only becomes a capability gap when the query is genuinely a filter over the whole archive rather than a path through it, and that distinction, not tooling familiarity, is what should decide the column.

The hybrid worth knowing about: publish the static tree as the source of truth, and load the same Items into pgstac asynchronously as a search index. Consumers that know their partition read JSON straight from the CDN at zero database cost; consumers that need search hit the API. If the index is lost you rebuild it from the tree, because the tree — not the database — is authoritative. This is how most large public catalogues are actually shaped.

Step-by-Step Implementation

Step 1: Where the write happens in the pipeline

The Item is built by the worker that produced the tile, immediately after the COG is written and before the invocation returns.

The header read in stage two is not a re-download. rasterio.open() on a freshly written local file, or on the object just uploaded, fetches the IFD and nothing else — the same header-only access pattern used to enumerate windows in partitioning a GeoTIFF into Step Functions map tiles.

Step 2: Build the Item with pystac

The core of the whole page. This function takes a written tile and returns a validated Item carrying both the projection and raster extensions.

# build_item.py — a STAC Item for one tile from a serverless tiling job
from datetime import datetime, timezone

import pystac
import rasterio
from pystac.extensions.projection import ProjectionExtension
from pystac.extensions.raster import DataType, RasterBand, RasterExtension
from rasterio.warp import transform_bounds

COLLECTION_ID = "ndvi-10m-v1"


def item_id(scene_id: str, row_off: int, col_off: int) -> str:
    """Deterministic: the same tile always yields the same id, forever."""
    return f"{scene_id}_r{row_off:05d}_c{col_off:05d}"


def build_item(tile_uri: str, scene_id: str, acquired: datetime,
               row_off: int, col_off: int) -> pystac.Item:
    with rasterio.open(tile_uri) as ds:          # header read, no pixels
        native_bbox = list(ds.bounds)
        epsg = ds.crs.to_epsg()
        shape = [ds.height, ds.width]
        transform = list(ds.transform)[:6]
        nodata = ds.nodata
        dtype = ds.dtypes[0]

    # STAC geometry and bbox are ALWAYS EPSG:4326, whatever proj:epsg says.
    wgs84 = transform_bounds(f"EPSG:{epsg}", "EPSG:4326", *native_bbox)
    west, south, east, north = wgs84
    geometry = {
        "type": "Polygon",
        "coordinates": [[
            [west, south], [east, south], [east, north],
            [west, north], [west, south],
        ]],
    }

    item = pystac.Item(
        id=item_id(scene_id, row_off, col_off),
        geometry=geometry,
        bbox=[west, south, east, north],
        datetime=acquired.astimezone(timezone.utc),
        collection=COLLECTION_ID,
        properties={
            "gsd": 10,
            "constellation": "sentinel-2",
            "processing:level": "L3",
            "tile:row_off": row_off,
            "tile:col_off": col_off,
        },
    )

    # --- projection extension: the native grid, so a client can window-read ---
    proj = ProjectionExtension.ext(item, add_if_missing=True)
    proj.epsg = epsg
    proj.shape = shape
    proj.transform = transform
    proj.bbox = native_bbox

    asset = pystac.Asset(
        href=tile_uri,
        media_type=pystac.MediaType.COG,
        roles=["data"],
        title="NDVI, float32",
    )
    item.add_asset("ndvi", asset)

    # --- raster extension: what stops a renderer painting -9999 as a value ---
    RasterExtension.ext(asset, add_if_missing=True).bands = [
        RasterBand.create(
            nodata=nodata,
            data_type=DataType(dtype),
            spatial_resolution=10,
            unit="dimensionless",
        )
    ]
    return item

Two things in that function are load-bearing. The first is transform_bounds — a STAC geometry and bbox are defined in EPSG:4326 regardless of the data’s native CRS, and writing UTM metres into bbox is the single most common way to produce an Item that validates cleanly and is spatially wrong by six million metres. The second is that proj:epsg, proj:shape and proj:transform preserve the native grid the bbox just discarded, so a client can still compute an exact pixel window without opening the COG.

The mechanics of writing that Item from inside a Lambda handler — the S3 key layout, the parent link, the error paths — are worked through in writing STAC Items from a Lambda tiling job.

Step 3: Rebuild the Collection and the tree once, at the end

Items are written by many workers; the tree is rebuilt by exactly one finaliser after the fan-out completes.

# finalise_catalog.py — rebuild Collection and Catalog nodes after a run
from datetime import datetime, timezone

import pystac

ROOT_HREF = "https://catalog.example.com/stac"


def build_collection(items: list[pystac.Item]) -> pystac.Collection:
    boxes = [it.bbox for it in items]
    spatial = pystac.SpatialExtent([[
        min(b[0] for b in boxes), min(b[1] for b in boxes),
        max(b[2] for b in boxes), max(b[3] for b in boxes),
    ]])
    times = sorted(it.datetime for it in items)
    temporal = pystac.TemporalExtent([[times[0], times[-1]]])

    collection = pystac.Collection(
        id="ndvi-10m-v1",
        description="Per-tile NDVI at 10 m from Sentinel-2 L2A.",
        extent=pystac.Extent(spatial=spatial, temporal=temporal),
        license="CC-BY-4.0",
    )
    collection.add_items(items)
    return collection


def publish(collection: pystac.Collection, out_dir: str) -> None:
    catalog = pystac.Catalog(id="root", description="NDVI archive root")
    catalog.add_child(collection)
    catalog.normalize_hrefs(ROOT_HREF)
    # ABSOLUTE_PUBLISHED writes absolute self links — required when the tree
    # is served over HTTP from a CDN rather than opened from a local clone.
    catalog.save(catalog_type=pystac.CatalogType.ABSOLUTE_PUBLISHED,
                 dest_href=out_dir)

CatalogType.ABSOLUTE_PUBLISHED is the right choice for anything served from a CDN, because a STAC browser resolving relative links across a redirect boundary will silently build wrong URLs. The serving side of that decision — bucket layout, cache headers, invalidation — is covered in serving a static STAC catalog from S3 and CloudFront.

Step 4: Keep the catalogue consistent across retries

Every queue on every cloud delivers at least once. A tile job that times out at 900 seconds and is retried will produce its output twice, and the catalogue is where that duplication becomes permanent.

The fix is not deduplication — it is making the second write indistinguishable from the first. Because item_id() above is a pure function of (scene_id, row_off, col_off), the Item’s object key is deterministic, so attempt two overwrites attempt one. There is no read-modify-write, no conditional, and no lock; the operation is idempotent because the address is derived rather than allocated. This is the catalogue-side application of the pattern in idempotency and exactly-once spatial processing and its ingestion-side counterpart, deduplicating S3 event notifications.

The rule extends to the tree. If the finaliser enumerates Items by listing the prefix rather than by consuming the fan-out’s result list, a partial rerun rebuilds a correct tree from whatever is actually present, instead of a tree describing what one particular execution happened to produce.

Step 5: Partition so the tree stays browsable

A catalogue is browsable when opening any node is cheap. That property survives to a million Items only if the fan-out per node stays bounded.

Partition by time first, because time is the axis along which the archive grows without limit. Root → year → month → day gives four hops to any Item, with the leaf day node holding a few thousand links. A flat root with a million child links is a 190 MB JSON document; no browser opens it, and neither does pystac, which will try to hold the whole thing in memory and take the function’s heap with it.

Add a second axis — MGRS tile, orbit, region — only if a consumer genuinely browses that way. Every extra level multiplies the number of small objects in the tree, and rebuilding 40,000 tiny Catalog nodes costs more in PutObject requests than the Items themselves.

Measurement and Verification

Validation is not optional, because a malformed Item fails silently: consumers skip it, and the tile it describes vanishes from the archive without an error anywhere. stac-validator checks an Item or a whole tree against the 1.0.0 schemas plus every extension the Item declares.

# validate_catalog.py — validate a published tree against STAC 1.0.0
import sys

from stac_validator import stac_validator

ROOT = "https://catalog.example.com/stac/catalog.json"

stac = stac_validator.StacValidate(ROOT, recursive=True, max_depth=5,
                                   extensions=True)
stac.run()

failures = [r for r in stac.message if r.get("valid_stac") is not True]
for f in failures:
    print(f"FAIL {f['path']}: {f.get('error_message', 'schema mismatch')}")

print(f"checked {len(stac.message)} objects, {len(failures)} invalid")
sys.exit(1 if failures else 0)

Expected output on a healthy tree:

checked 2714 objects, 0 invalid

Wire that into CI on every catalogue change, the same way GDAL and PROJ versions are pinned in CI/CD pipeline sync for geo dependencies. Two additions make the gate meaningful rather than decorative. First, validate a deliberately broken Item and assert that the run fails — an assertion that has never rejected anything proves nothing. Second, check the invariant that no schema encodes: that every Asset href actually resolves, which is one HEAD request per Item and catches the common case of an Item published before its COG finished uploading.

# check_hrefs.py — the invariant the JSON schema cannot express
import urllib.request

def asset_hrefs_resolve(item: dict) -> list[str]:
    broken = []
    for name, asset in item["assets"].items():
        req = urllib.request.Request(asset["href"], method="HEAD")
        try:
            urllib.request.urlopen(req, timeout=10)
        except Exception as exc:
            broken.append(f"{item['id']}/{name}: {exc}")
    return broken

Failure Modes and Debugging

bbox in metres, geometry in the wrong hemisphere. The Item validates — the schema only checks that bbox has four or six numbers — but every spatial search misses it, because a UTM easting of 499,980 is read as 499,980 degrees of longitude and clamped or discarded. Signature: a catalogue whose Items all report a bbox near [180, 90], or a search that returns nothing for an area you can see in the data. Always run transform_bounds into EPSG:4326 and keep the native box in proj:bbox.

Duplicate Items after a retry. One Item per attempt, each with a different random id, both pointing at the same asset href. Signature: the Item count exceeds the tile count, and the surplus matches the retry count in the queue metrics. Fix by deriving the id from the tile’s coordinates, never from uuid4() or the invocation request id.

Extension schema not found / validation fails only in CI. The Item lists an extension URL in stac_extensions that the validator cannot fetch, usually because the CI runner has no egress or because a schema host is rate-limiting. Signature: intermittent failures on unchanged Items. Cache the schemas in the repository and point the validator at the local copies rather than making CI depend on a third-party host being up.

Root catalog.json times out in a browser. The tree was never partitioned, so the root node lists every Item. Signature: a multi-hundred-megabyte catalog.json and a browser tab that hangs before rendering anything. Partition by time as in Step 5; the fix is a rebuild of the tree, not of the Items.

Assets 404 for a browser but not for curl. This is a CORS failure, not a missing object. Signature: the network tab shows the request completing with a 200 but the JavaScript reading it fails with an opaque error, or the preflight OPTIONS returns a 403. The bucket and the CDN each need their own CORS configuration — worked through in serving a static STAC catalog from S3 and CloudFront.

Cost and Scaling

The static path is close to free. A million Items at 3.6 KB each is 3.6 GB of JSON, about $0.08 per month of S3 Standard storage, plus roughly $5 in PutObject requests to write them once. The tree adds a few thousand small objects on top. CDN egress dominates only if consumers walk the tree repeatedly, and that is exactly what a long Cache-Control on Item JSON prevents.

The API path is a different order of magnitude. A small Aurora Serverless v2 PostgreSQL instance with pgstac plus a container running stac-fastapi lands around $90–150 per month before any traffic, and it never scales to zero the way the JSON tree does. That is not an argument against it — it is an argument for only paying it when a consumer actually issues spatial queries.

The scaling limit that surprises people is neither storage nor compute: it is the tree rebuild. Regenerating Catalog nodes is O(number of Items) if you enumerate naively, and at a million Items that is a listing operation with a five-figure request count every time a single tile is republished. The fix is to make the rebuild incremental — only the day node containing a changed Item, and its ancestors, need rewriting, which is four objects rather than forty thousand. Partitioning is what makes that possible, which is the real reason to do it before the archive gets large rather than after.

Request counts are the other place where a catalogue quietly gets expensive. Writing one Item per tile is one PutObject, but a naive finaliser that reads every Item back to compute the Collection’s extent adds one GetObject per tile on top — and at a million Items that is a five-figure request bill for arithmetic the workers could have returned in their own results. Have each worker return its bbox and datetime to the orchestrator, and let the finaliser fold those into the extent without reading anything back. The pattern is the same one that keeps state payloads small in partitioning a GeoTIFF into Step Functions map tiles: pass summaries forward, not documents.

Per-Item, the work is genuinely negligible: the JSON build is a few hundred microseconds and the PutObject is a single round trip, so cataloguing adds roughly 40 ms to a worker that already spent 20 seconds computing pixels. There is no memory pressure — the memory and CPU allocation model for raster workloads is set by the raster arrays, and an Item is noise against a 2048 × 2048 float32 window.

Frequently Asked Questions

Do I need a STAC API, or is a static catalogue enough?

A static catalogue is enough whenever every consumer already knows which partition it wants — a date, a scene, a product. It is a tree of JSON files on object storage, costs a couple of dollars a month for a million Items, and has no database to operate. You need a STAC API the moment somebody asks a question the tree cannot answer, such as “every Item intersecting this polygon between two dates with cloud cover under 10%”. That is an index query, and answering it from a static tree means downloading the whole catalogue.

What makes a STAC Item id idempotent across a retried tile job?

The id must be a pure function of the tile’s identity — source scene id, row offset, column offset, product version — and nothing else. No uuid4(), no time.time(), no invocation request id. When the id is deterministic the Item’s object key is deterministic too, so a retried worker overwrites its own previous Item rather than adding a second one. Catalogues built on random ids accumulate one duplicate per retry, and because both Items point at the same asset href the duplication stays invisible until somebody counts.

How should I partition a catalogue that will hold a million Items?

Partition by time first, because time is how the archive grows. A root Catalog linking to year Catalogs, each linking to month Catalogs, each linking to day Catalogs of a few thousand Items, keeps every node under about half a megabyte and reaches any Item in four HTTP requests. A flat root with a million child links is a 190 MB JSON document that no browser will open. Add a second axis such as MGRS tile only if a consumer genuinely browses spatially.

Which STAC extensions matter for a raster tiling pipeline?

Two carry almost all the value. The projection extension records the native CRS, pixel shape and affine transform, letting a client compute a pixel window without opening the COG. The raster extension records nodata, data type, scale and offset per band, which is what stops a consumer rendering a -9999 nodata sentinel as a real measurement. Both cost a few hundred bytes per Item and remove an HTTP range request from every downstream reader.

Where should catalogue writing happen in a fan-out pipeline?

Each tile worker writes its own Item, because only that worker knows the window transform, the nodata statistics and whether the tile contained any valid pixels at all. The parent Collection and the Catalog nodes are rebuilt once, by a single finaliser after the fan-out completes. Split this way, the per-tile write is one idempotent PutObject and the tree rebuild — the only part needing a global view — runs exactly once.

Guides in this topic

• Serving a Static STAC Catalog from S3 and CloudFront — The bucket CORS rules, response-headers policy and per-path cache TTLs a STAC browser needs to walk a…
• Writing STAC Items from a Lambda Tiling Job — A production Lambda handler that writes one STAC Item per tile: a deterministic id derived from the window…

Related

• Writing STAC Items from a Lambda Tiling Job — the handler that builds and writes one Item, with the retry semantics worked out
• Serving a Static STAC Catalog from S3 and CloudFront — the CORS, caching and invalidation configuration a STAC browser needs
• Serverless NDVI Tiling from Sentinel-2 Imagery — the pipeline whose output this catalogue describes
• Idempotency and Exactly-Once Spatial Processing — why a derived id beats an allocated one everywhere in the pipeline
• Merging Tiled Lambda Outputs into a COG — the other thing a finaliser step does after a fan-out

Back to Serverless Geospatial Pipeline Recipes