Measuring Cold Starts with CloudWatch and Cloud Trace

Cold start latency is not a metric on any of the three clouds by default — it is a field in a log line on AWS, a distribution on a container metric on GCP, and a boolean column in a KQL table on Azure. Extract it once and it becomes a percentile you can alert on: a CloudWatch metric filter on Init Duration from the REPORT line, an EMF record carrying your own gdal_init_ms, a run.googleapis.com/container/startup_latencies query on Cloud Run, and requests | where customDimensions.ColdStart == "True" in Application Insights. Without that extraction the Cold Start Comparison numbers stay anecdotes, and you cannot tell whether a layer change helped.

The Problem With Anecdotal Cold Starts

Every team that runs a Python GDAL function has a story about a 12-second first request. Almost none of them can say what the p99 initialisation time was last Tuesday, or whether trimming 40 MB off the GDAL layer moved it. The reason is structural: the platforms report initialisation cost through a channel that is not the metrics channel, and nobody wires the two together.

On AWS Lambda the REPORT line at the end of every invocation carries Init Duration: 4231.45 ms — but only on a cold invocation, and only in the log stream. It is not a CloudWatch metric, so it cannot be graphed, percentiled or alarmed on until you extract it. On GCP, Cloud Run publishes container/startup_latencies as a distribution, which is closer to useful but attributes the whole container start to one number with no idea which part was import rasterio. On Azure, Application Insights records a ColdStart custom dimension on the request telemetry, which is a flag rather than a duration.

This matters more for geospatial functions than for most workloads because the initialisation cost is both large and controllable. The breakdown in Cold Start Mapping for Python GDAL shows shared-library resolution and module import dominating a 6-second cold start, and both respond to packaging changes. If you cannot measure the split, the packaging work in Python Layer Management and Size Reduction is uncosted guesswork.

Phases inside one Lambda REPORT line's Init DurationA 4,602 millisecond initialisation window split into five segments: runtime bootstrap and archive unpack at 1,370 milliseconds, importing rasterio at 2,184 milliseconds, importing pyproj at 613 milliseconds, the first PROJ transform at 431 milliseconds, and GDAL environment validation at 4 milliseconds.What a 4,602 ms Init Duration is actually made ofRuntime bootstrap + unpackimport rasterioimport pyprojFirstPROJtransform1,370 ms2,184 ms613 ms431 msGDAL env validation — 4 msThe platform reports only the 4,602 ms total. The four right-hand segments come from the module-scope _phase() timers, emitted as one EMFrecord per container.
Only the first segment is the platform's; the remaining 3,232 ms is your import graph, and it is the part the EMF record breaks out.

Prerequisites

  • Runtime: Python 3.11 or 3.12 with GDAL 3.6+ available through a layer, a container image, or the GCP buildpack.
  • AWS: the execution role needs logs:CreateLogGroup, logs:CreateLogStream and logs:PutLogEvents; the deploying principal needs logs:PutMetricFilter and cloudwatch:PutMetricAlarm. Set AWS_LAMBDA_LOG_FORMAT=JSON if you want structured logs, but note the caveat below — it changes the REPORT line’s shape.
  • GCP: enable cloudtrace.googleapis.com and monitoring.googleapis.com. The runtime service account needs roles/cloudtrace.agent and roles/monitoring.metricWriter. Install opentelemetry-sdk 1.24+ and opentelemetry-exporter-gcp-trace 1.6+.
  • Azure: an Application Insights resource with APPLICATIONINSIGHTS_CONNECTION_STRING set in the function app configuration, and azure-monitor-opentelemetry 1.2+ in requirements.txt.
  • Environment variables that must be explicit, because they change what initialisation costs:
    • GDAL_DATA=/opt/share/gdal, PROJ_LIB=/opt/share/proj, LD_LIBRARY_PATH=/opt/lib — if any of these is unset, GDAL falls back to a filesystem search that adds seconds to init and makes your measurements meaningless.
    • GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR — otherwise the first Open() inside the handler issues a list call whose latency shows up in your handler timing rather than in init.
    • POWERTOOLS_METRICS_NAMESPACE=GeoPipeline and POWERTOOLS_SERVICE_NAME=cog-reader if you use the AWS Lambda Powertools EMF helper.
  • A comparable baseline. Measure the same layer, the same memory tier and the same input on every platform, or you are comparing packaging decisions rather than platforms.

The Signals, Platform by Platform

Cold-start signals and their queries on AWS, GCP and AzureComparison grid across AWS Lambda, GCP Cloud Run and Azure Functions covering the raw signal name, where the signal lives, the extraction mechanism needed to make it alertable, whether percentiles are available without extra work, and whether the geospatial import cost can be separated from the platform's own startup.Where each cloud hides the number, and what extracts itAWS LambdaGCP Cloud RunAzure FunctionsRaw signalInit DurationREPORT log linestartup_latenciescontainer metricColdStartcustom dimension, booleanQuery surfaceLogs Insights /metric filtergcloud monitoringtime-seriesApplication InsightsKQLAlertable without extra workNoneeds a metric filterYesalready a distributionNoflag, not a durationWarm invocations excludedYesfield absent when warmYesemitted on start onlyFilter neededwhere cold == trueSeparates GDAL import costEMF recordgdal_init_ms dimensiongdal.init spannested in request traceOTel spanazure-monitor-opentelemetryProvisioned concurrency, min-instances and always-ready instances each remove the raw signal rather than lowering it — alarms must treatmissing data as not breaching.
Only GCP publishes a startup number as a real metric, and it is the one that excludes your import graph — which is why every platform needs a span or an EMF record on top.

Each cloud puts the number somewhere different, and each requires a different extraction. The AWS route is a metric filter over the log group, which converts a log field into a real metric with no code change. The GCP route is a built-in distribution metric plus an explicit trace span for the part you control. The Azure route is a KQL query over the requests table, joined against customDimensions.

AWS: a metric filter over the REPORT line

bash 
# Convert the Init Duration field of every REPORT line into a CloudWatch metric.
# The filter pattern binds the fields positionally; $initDuration is the value.
aws logs put-metric-filter \
  --log-group-name "/aws/lambda/cog-reader" \
  --filter-name "InitDuration" \
  --filter-pattern '[report="REPORT", requestIdLabel, requestId, durationLabel, duration, durationUnit, billedLabel, billedDurationLabel, billedDuration, billedUnit, memoryLabel, memorySizeLabel, memorySize, memoryUnit, maxMemoryLabel, maxMemoryUsedLabel, maxMemoryUsed, maxMemoryUnit, initLabel, initDurationLabel, initDuration, initUnit]' \
  --metric-transformations \
      metricName=InitDurationMs,metricNamespace=GeoPipeline,metricValue='$initDuration',unit=Milliseconds

# Alarm on the p99, not the average — cold starts are a tail phenomenon.
aws cloudwatch put-metric-alarm \
  --alarm-name "cog-reader-cold-start-p99" \
  --namespace GeoPipeline --metric-name InitDurationMs \
  --extended-statistic p99 --period 300 --evaluation-periods 3 \
  --threshold 8000 --comparison-operator GreaterThanThreshold

Because the pattern only matches REPORT lines that have the init fields, warm invocations are silently excluded — the metric is a clean population of cold starts only. Counting the difference between total invocations and metric sample count gives you the cold-start rate for free.

AWS: an EMF record that splits GDAL out of the total

The metric filter gives you the platform’s number. It cannot tell you how much of it was import rasterio versus the runtime bootstrap. For that, time the phases yourself at module scope and emit an Embedded Metric Format record — a JSON log line CloudWatch parses into metrics with no filter at all.

python 
"""Module-scope instrumentation: time each init phase, emit one EMF record."""
import json
import os
import time

# Wall clock at import. Everything before this belongs to the runtime bootstrap.
_MODULE_T0 = time.perf_counter()

_phases = {}


def _phase(name):
    """Context manager that records the duration of one initialisation phase."""
    class _P:
        def __enter__(self):
            self.t = time.perf_counter()
            return self

        def __exit__(self, *exc):
            _phases[name] = round((time.perf_counter() - self.t) * 1000, 2)
            return False
    return _P()


with _phase("import_rasterio_ms"):
    import rasterio                                  # noqa: E402
    from rasterio.session import AWSSession          # noqa: E402

with _phase("import_pyproj_ms"):
    import pyproj                                    # noqa: E402

with _phase("gdal_env_ms"):
    # Validate the paths that, when wrong, silently add seconds to every init.
    for var in ("GDAL_DATA", "PROJ_LIB", "LD_LIBRARY_PATH"):
        if not os.environ.get(var):
            raise RuntimeError(f"{var} is unset — init timings will not be comparable")
    _GDAL_ENV = dict(
        GDAL_DISABLE_READDIR_ON_OPEN="EMPTY_DIR",
        GDAL_CACHEMAX="512",
        CPL_VSIL_CURL_CACHE_SIZE="67108864",
    )
    _SESSION = AWSSession()

with _phase("proj_first_transform_ms"):
    # The first transform pays for loading proj.db. Doing it here moves that
    # cost out of the first request and into a phase you can see.
    pyproj.Transformer.from_crs("EPSG:4326", "EPSG:3857", always_xy=True).transform(0, 0)

_phases["module_total_ms"] = round((time.perf_counter() - _MODULE_T0) * 1000, 2)

# One EMF record per container, emitted at import. CloudWatch turns the fields
# named in Metrics[] into real metrics; the rest ride along as context.
print(json.dumps({
    "_aws": {
        "Timestamp": int(time.time() * 1000),
        "CloudWatchMetrics": [{
            "Namespace": "GeoPipeline",
            "Dimensions": [["service", "gdal_version"]],
            "Metrics": [{"Name": n, "Unit": "Milliseconds"} for n in _phases],
        }],
    },
    "service": os.environ.get("AWS_LAMBDA_FUNCTION_NAME", "cog-reader"),
    "gdal_version": rasterio.__gdal_version__,
    **_phases,
}))

_IS_COLD = True


def handler(event, context):
    global _IS_COLD
    cold = _IS_COLD
    _IS_COLD = False   # every later invocation on this container is warm

    with rasterio.Env(session=_SESSION, **_GDAL_ENV):
        with rasterio.open(event["uri"].replace("s3://", "/vsis3/", 1)) as src:
            profile = {"width": src.width, "height": src.height,
                       "blocks": list(src.block_shapes[0])}

    return {"statusCode": 200,
            "body": json.dumps({"cold": cold, "init_phases": _phases,
                                "profile": profile})}

The EMF record is emitted once per container, at import time. On a warm container it never fires again, so the metric population is exactly the set of cold starts, and each sample carries the four-way split. gdal_version as a dimension means a layer upgrade shows up as a new series rather than a step change in an existing one.

GCP: the built-in distribution and an explicit span

bash 
# Startup latency distribution for a Cloud Run service, aligned to 5 minutes.
gcloud monitoring time-series list \
  --project "$PROJECT" \
  --filter='metric.type="run.googleapis.com/container/startup_latencies"
            AND resource.labels.service_name="cog-reader"' \
  --format='table(points[0].value.distributionValue.mean,
                  points[0].value.distributionValue.count)'

For the geospatial share of that number, open a span around the same module-scope block and export it to Cloud Trace:

python 
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.cloud_trace import CloudTraceSpanExporter

trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(
    BatchSpanProcessor(CloudTraceSpanExporter())
)
tracer = trace.get_tracer("cog-reader")

with tracer.start_as_current_span("gdal.init") as span:
    import rasterio
    span.set_attribute("gdal.version", rasterio.__gdal_version__)
    span.set_attribute("gdal.cachemax_mb", 512)
    span.set_attribute("proj.network", "OFF")

The span nests inside the request trace of the first request on that instance, so Cloud Trace’s waterfall shows container start, then gdal.init, then the handler — which is the picture that tells you whether min-instances is worth paying for, as covered in Reducing Cloud Functions 2nd Gen Cold Starts with Min Instances.

Azure: KQL over Application Insights

kusto 
// Cold-start rate and cold-only latency for one function over 24 hours.
// duration on the requests table is milliseconds.
requests
| where timestamp > ago(24h)
| where cloud_RoleName == "cog-reader"
| extend cold = tobool(customDimensions["ColdStart"])
| summarize
      invocations  = count(),
      cold_starts  = countif(cold),
      p50_all_ms   = percentile(duration, 50),
      p95_cold_ms  = percentile(iff(cold, duration, real(null)), 95),
      p99_cold_ms  = percentile(iff(cold, duration, real(null)), 99)
  by bin(timestamp, 1h)
| extend cold_rate_pct = round(100.0 * cold_starts / invocations, 2)
| order by timestamp asc

percentile ignores nulls, so projecting warm rows to real(null) inside iff() gives a cold-only percentile in the same summarize as the overall rate. The ColdStart dimension is populated by the Azure Functions host, not by your code — if it is absent, the worker is on an older runtime and you need azure-monitor-opentelemetry to emit it yourself.

Verification

Force a cold start and read back what each pipeline captured. On AWS, updating an environment variable is enough to retire every warm container without redeploying code:

bash 
aws lambda update-function-configuration \
  --function-name cog-reader \
  --environment "Variables={GDAL_DATA=/opt/share/gdal,PROJ_LIB=/opt/share/proj,LD_LIBRARY_PATH=/opt/lib,GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR,COLD_NONCE=$(date +%s)}" >/dev/null

aws lambda invoke --function-name cog-reader \
  --payload '{"uri":"s3://eo-scenes/S2B_33UUP_20260714.tif"}' \
  --cli-binary-format raw-in-base64-out /dev/stdout | jq '.body|fromjson|.init_phases'

Expected output — the four phases sum to roughly the platform’s own Init Duration, and the imports dominate:

json 
{
  "import_rasterio_ms": 2184.31,
  "import_pyproj_ms": 612.77,
  "gdal_env_ms": 3.94,
  "proj_first_transform_ms": 431.06,
  "module_total_ms": 3232.08
}

Cross-check against the platform’s own figure with a Logs Insights query over the same window:

code 
filter @type = "REPORT"
| stats count() as invocations,
        count(@initDuration) as cold_starts,
        pct(@initDuration, 50) as p50_init_ms,
        pct(@initDuration, 99) as p99_init_ms

If module_total_ms is 3,232 and p50_init_ms is 4,600, the ~1,370 ms difference is the runtime bootstrap and archive unpack — the part packaging work in Stripping Unnecessary Python Packages from AWS Lambda Layers attacks. If the two numbers are nearly equal, your imports are the whole problem and layer trimming will not help.

Path from module-scope timers to a cold-start alarmFive stages left to right: module-scope phase timers wrapping each import, a single Embedded Metric Format JSON record printed once per container, CloudWatch parsing that record into named metrics, a p99 extended statistic computed over the metric, and an alarm that fires above eight seconds.From an import statement to a p99 you can alarm onPhase timers_phase() atmodule scopeEMF recordone JSON lineper containerCloudWatchparseNamespaceGeoPipelinedims: service,gdal_versionp99 statistic300 s period3 evaluation periodsAlarm at 8 smissing data =notBreaching
The record is printed at import, so it fires exactly once per container — the metric population is the set of cold starts with no filtering required.

Gotchas

  • AWS_LAMBDA_LOG_FORMAT=JSON breaks positional metric filter patterns. In JSON log mode the REPORT line becomes a structured record and the space-delimited [report="REPORT", ...] pattern matches nothing — silently, with the metric simply reporting no data. Switch the filter to { $.record.metrics.initDurationMs = * } when you change log format.

  • Provisioned concurrency removes Init Duration from the REPORT line entirely. Initialisation happens outside any request, so there is no field to extract and your cold-start metric goes to zero samples. That is correct behaviour, but an alarm configured to fire on missing data will page you for a successful optimisation. Set the alarm’s treat-missing-data to notBreaching.

  • The first pyproj.Transformer costs more than the import. Loading proj.db is lazy, so a naive measurement attributes it to whichever request happens to transform first. Forcing one transform at module scope — as the code above does — moves it into the init window where it belongs and stops it polluting handler percentiles.

  • Cloud Run’s startup_latencies covers the container, not your code. A trimmed image lowers it; a trimmed Python import does not, because imports run after the container is reported started. Always pair the metric with the gdal.init span or you will conclude that import cost is free.

Frequently Asked Questions

Is Init Duration included in the billed duration on AWS Lambda?

Not for on-demand invocations — the REPORT line prints it separately and Billed Duration covers only the handler. On a provisioned-concurrency instance initialisation happens outside the request entirely, so no Init Duration field appears at all, which makes its absence a reliable warm-start detector.

Why does CloudWatch have no built-in cold start metric?

Init Duration is a field in a log line, not a published metric. To percentile or alarm on it you must extract it, either with a metric filter over the log group or by emitting your own EMF record from module scope. Both are shown above; the EMF route additionally splits GDAL’s share out of the total.

How do I see cold starts on Cloud Run?

Query run.googleapis.com/container/startup_latencies for the container-side cost, and wrap module-scope GDAL initialisation in an explicit OpenTelemetry span named gdal.init so the Python share appears as its own bar in the Cloud Trace waterfall.

What is a reasonable p99 cold start target for a Python GDAL function?

Under 6 seconds on a 2,048 MB function with a trimmed layer, and under 1 second on a warm-kept instance. If p99 sits above 10 seconds the usual causes are an untrimmed layer, unset GDAL_DATA/PROJ_LIB forcing a filesystem search, or a first transform pulling in a datum grid over the network.

Back to Cold Start Comparison: AWS vs GCP vs Azure