Spatial IaC Terraform & Pulumi for GIS

Orchestrating COG Generation Pipelines with Step Functions

You want a raw GeoTIFF dropped into an S3 prefix to become a validated Cloud-Optimized GeoTIFF (COG) in the tile bucket automatically, with no operator in the loop and no half-written objects when a conversion fails. The reliable way to build that is an AWS Step Functions state machine that validates the upload, runs gdal_translate with overviews on a Lambda or AWS Batch task, verifies the result, and only then publishes. This task-specific guide belongs to Raster Data Pipeline Provisioning within the broader Geospatial Resource Provisioning discipline, and it focuses on the orchestration itself: the ASL definition, the retry and catch semantics, the Map state for parallel tiles, and the cost of getting it wrong.

Symptom Identification and Triage

The failures that push teams toward a state machine are recognizable, and each one points at a different missing piece of orchestration. Before writing any ASL, classify what is actually going wrong:

  • Corrupt or non-COG tiles reaching the serving path: rio cogeo validate fails on objects in tiles/, or a tile client reads garbage. A conversion crashed and left a truncated file where a reader could see it. This is a missing validate-before-publish gate.
  • Conversions that silently never happen: an upload lands but no COG appears, and there is no error anywhere. A fire-and-forget Lambda trigger swallowed the failure. This needs a durable orchestrator with a catch clause, not a raw S3-to-Lambda notification.
  • States.Timeout or Lambda 15-minute limit hit: a large orthomosaic exceeds Lambda’s ceiling mid-gdal_translate. The compute target is wrong for the payload size — the job belongs on Batch.
  • Retry storms and duplicated outputs: the same source is converted repeatedly and produces duplicate objects. The state machine is not idempotent and its retry policy does not distinguish transient from permanent errors.
COG pipeline triage: find which state failed before changing the definition A top-down decision tree. The symptom, an upload did not become a valid COG, leads to checking the Step Functions execution history and CloudWatch logs. A decision asks where it failed. Three outcomes branch out: a failure in the validate state means a bad or non-COG source and routes to fixing the input or the gdalinfo check; a task timeout or exit 137 means the payload is too large for Lambda and routes to moving the job to AWS Batch; and no execution at all means the S3 event notification never triggered the state machine and routes to fixing the event wiring. Upload did not become a valid COG Check execution history + CloudWatch task logs Where did it fail? Validate state bad / non-COG source fix input · gdalinfo Task timeout / 137 payload too large move job to Batch No execution event never fired fix S3 → SQS wiring validate error States.Timeout history empty

Prerequisites and Environment Assumptions

This guide assumes an AWS target with a raw-upload prefix and a tile bucket already provisioned per Object Storage for Raster & Vector. To deploy the orchestration you will need:

  • Terraform >= 1.7.0 with the AWS provider pinned to ~> 5.60. Unpinned providers cause phantom diffs on the state machine definition, so pin the version explicitly.
  • A GDAL container image referenced by digest for the Batch path (a rebuilt :latest changes COG block size and predictor and breaks readers), plus a Lambda packaged with GDAL for the small-file path.
  • An SQS queue with a dead-letter queue receiving the S3 ObjectCreated event, so uploads are a durable trigger rather than a fire-and-forget notification. The task role must read the ingest prefix, write staging, and promote to tiles/, following IAM Role Mapping for GIS.
  • rio cogeo and gdalinfo available in the validation task and on the operator’s machine for the verification step below.

Step-by-Step Remediation

Build the state machine so that no object reaches the serving path unless it was validated both before and after conversion.

  1. Validate the upload before spending compute. The first state runs gdalinfo on the raw object and asserts it is a readable, georeferenced raster with a known projection. Rejecting a bad upload here costs a few hundred milliseconds instead of a multi-minute Batch job that would fail anyway. The geospatial reason: a source with no CRS or a broken IFD will produce a COG that validates structurally but is spatially meaningless.

  2. Branch by payload size to the right compute. A Choice state routes small rasters to a Lambda task and large ones to an AWS Batch job. Lambda’s 15-minute ceiling and 10 GB memory cannot build overviews on a large orthomosaic, so anything above the threshold must go to Batch. Route on estimated decompressed pixel count from gdalinfo, not on the S3 object size, because a highly compressed source expands far larger than its byte count.

  3. Convert to COG with overviews. The task runs gdal_translate with the COG driver and internal overviews, writing to a staging/ key derived deterministically from the source key and version so a replay is idempotent:

    gdal_translate "/vsis3/${SRC_BUCKET}/${SRC_KEY}" \
  "/vsis3/${STAGING_BUCKET}/${OUT_KEY}" \
  -of COG \
  -co COMPRESS=DEFLATE \
  -co PREDICTOR=2 \
  -co OVERVIEWS=AUTO \
  -co BLOCKSIZE=512 \
  -co NUM_THREADS=ALL_CPUS

  4. Validate the output, then publish atomically. A post-conversion state runs rio cogeo validate on the staging object and only on success copies it to tiles/ and updates the manifest. The manifest write is last, so a reader never sees a tile whose entry does not yet exist, and never sees an object the converter did not confirm.

Here is the core of the state machine as pinned Amazon States Language, with retry and catch on every task and a Map state that builds per-region tile sets in parallel. The ${...} tokens are Terraform templatefile interpolations resolved at apply time:

{
  "Comment": "Ingest a raw GeoTIFF, convert to COG with overviews, validate, publish",
  "StartAt": "ValidateUpload",
  "States": {
    "ValidateUpload": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "${validate_lambda_arn}",
        "Payload.$": "$"
      },
      "Retry": [
        {
          "ErrorEquals": ["Lambda.ServiceException", "Lambda.TooManyRequestsException"],
          "IntervalSeconds": 2,
          "MaxAttempts": 4,
          "BackoffRate": 2.0
        }
      ],
      "Catch": [
        { "ErrorEquals": ["BadRasterError"], "Next": "RejectUpload" }
      ],
      "ResultPath": "$.validation",
      "Next": "RouteBySize"
    },
    "RouteBySize": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.validation.megapixels",
          "NumericGreaterThan": ${size_threshold_mp},
          "Next": "ConvertOnBatch"
        }
      ],
      "Default": "ConvertOnLambda"
    },
    "ConvertOnLambda": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "${cog_lambda_arn}",
        "Payload.$": "$"
      },
      "Retry": [
        {
          "ErrorEquals": ["States.Timeout", "Lambda.ServiceException"],
          "IntervalSeconds": 5,
          "MaxAttempts": 2,
          "BackoffRate": 2.0
        }
      ],
      "Catch": [
        { "ErrorEquals": ["States.ALL"], "Next": "SendToDLQ" }
      ],
      "ResultPath": "$.cog",
      "Next": "ValidateCog"
    },
    "ConvertOnBatch": {
      "Type": "Task",
      "Resource": "arn:aws:states:::batch:submitJob.sync",
      "Parameters": {
        "JobDefinition": "${batch_job_defn}",
        "JobQueue": "${batch_job_queue}",
        "JobName": "cog-convert",
        "Parameters": {
          "input_key.$": "$.detail.object.key",
          "output_key.$": "$.cog.out_key"
        }
      },
      "Retry": [
        {
          "ErrorEquals": ["Batch.AWSBatchException"],
          "IntervalSeconds": 10,
          "MaxAttempts": 3,
          "BackoffRate": 2.0
        }
      ],
      "Catch": [
        { "ErrorEquals": ["States.ALL"], "Next": "SendToDLQ" }
      ],
      "ResultPath": "$.cog",
      "Next": "ValidateCog"
    },
    "ValidateCog": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "${validate_cog_lambda_arn}",
        "Payload.$": "$"
      },
      "Catch": [
        { "ErrorEquals": ["States.ALL"], "Next": "SendToDLQ" }
      ],
      "Next": "PublishTiles"
    },
    "PublishTiles": {
      "Type": "Map",
      "ItemsPath": "$.cog.regions",
      "MaxConcurrency": 8,
      "Iterator": {
        "StartAt": "PromoteRegion",
        "States": {
          "PromoteRegion": {
            "Type": "Task",
            "Resource": "arn:aws:states:::lambda:invoke",
            "Parameters": {
              "FunctionName": "${publish_lambda_arn}",
              "Payload.$": "$"
            },
            "End": true
          }
        }
      },
      "Next": "UpdateManifest"
    },
    "UpdateManifest": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "${manifest_lambda_arn}", "Payload.$": "$" },
      "End": true
    },
    "RejectUpload": { "Type": "Fail", "Error": "BadRasterError", "Cause": "Source failed gdalinfo validation" },
    "SendToDLQ": {
      "Type": "Task",
      "Resource": "arn:aws:states:::sqs:sendMessage",
      "Parameters": { "QueueUrl": "${dlq_url}", "MessageBody.$": "$" },
      "Next": "FailRun"
    },
    "FailRun": { "Type": "Fail", "Error": "PipelineError", "Cause": "Conversion failed; message sent to DLQ" }
  }
}

The Terraform that ships this definition pins the provider and templates the ARNs so the identical ASL deploys to every environment:

terraform {
  required_version = ">= 1.7.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.60"
    }
  }
}

resource "aws_sfn_state_machine" "cog" {
  name     = "${var.environment}-cog-generation"
  role_arn = var.pipeline_role_arn

  definition = templatefile("${path.module}/cog.asl.json", {
    validate_lambda_arn     = var.validate_lambda_arn
    cog_lambda_arn          = var.cog_lambda_arn
    validate_cog_lambda_arn = var.validate_cog_lambda_arn
    publish_lambda_arn      = var.publish_lambda_arn
    manifest_lambda_arn     = var.manifest_lambda_arn
    batch_job_defn          = var.batch_job_defn_arn
    batch_job_queue         = var.batch_job_queue_arn
    dlq_url                 = var.dlq_url
    size_threshold_mp       = var.size_threshold_mp # megapixels, not bytes
  })

  logging_configuration {
    log_destination        = "${aws_cloudwatch_log_group.cog.arn}:*"
    include_execution_data = true
    level                  = "ALL" # full history: COG builds are expensive to reproduce
  }
}

resource "aws_cloudwatch_log_group" "cog" {
  name              = "/aws/states/${var.environment}-cog-generation"
  retention_in_days = 30
}

Note: If the Lambda COG task reads from an RDS-backed catalog, keep any parameter-group memory setting (for example shared_buffers) expressed in AWS integer units — 8 kB blocks — never a percentage string, so the value resolves identically across instance classes.

Verification

Confirm the pipeline produces genuinely valid COGs, not just objects that exist.

  1. Validate the published object as a COG. rio cogeo validate is the authoritative check — it asserts the internal tiling and overview structure a naive gdal_translate can omit:

    rio cogeo validate s3://prod-tiles/tiles/scene-2026-07/cog.tif
# expect: "…is a valid cloud optimized GeoTIFF"

  2. Inspect the structure with gdalinfo. Confirm the overviews and block size the pipeline was supposed to build actually landed:

    gdalinfo /vsis3/prod-tiles/tiles/scene-2026-07/cog.tif | grep -E "Overviews|Block"
# expect internal overviews (e.g. 4096x4096, 2048x2048, …) and Block=512x512

  3. Confirm the execution path. In the Step Functions console or via aws stepfunctions get-execution-history, verify the run took the expected branch (Lambda for small, Batch for large), that ValidateCog succeeded before PublishTiles, and that UpdateManifest was the final state. An execution that reached SendToDLQ means the object was never promoted, which is the correct fail-closed behavior.

Preventing Recurrence

Encode the guarantees so a future change cannot reintroduce corrupt tiles:

  • Validate-before-publish is non-negotiable. Keep ValidateCog as a required predecessor of PublishTiles in the ASL, and add a CI check that fails any state-machine change where a publish path bypasses the COG validation state.
  • Pin the GDAL image by digest. A policy-as-code rule should reject any Batch job definition referencing a floating tag, mirroring the digest discipline in Raster Data Pipeline Provisioning.
  • Right-size the Lambda-to-Batch threshold. Route on decompressed megapixels and review the threshold whenever a new high-resolution source class is onboarded, so large payloads never hit the Lambda timeout. The provisioning trade-offs for the Lambda side are covered in Provisioning Lambda for On-the-Fly Raster Reprojection.
  • Scheduled drift check. Run terraform plan on a cron and fail the build on any non-empty diff, so a console edit to the state machine or the Batch sizing surfaces immediately.

Frequently Asked Questions

Should the conversion run on Lambda or AWS Batch?

Both, chosen by a Choice state on payload size. Lambda handles small rasters cheaply with fast cold-to-result latency, but its 15-minute timeout and 10 GB memory cannot build overviews on a large orthomosaic. Route anything above your megapixel threshold to Batch, which can request tens of gigabytes of memory and hours of runtime.

How do I stop a failed conversion from publishing a corrupt COG?

Convert to a staging/ prefix, run rio cogeo validate in a dedicated ValidateCog state, and only copy to tiles/ on success. The publish step must never write directly to the serving path, and the manifest update must be the last write so readers never see an object the validator did not confirm.

What does the Map state buy me here?

The Map state fans out per-region (or per-tile-set) publishing in parallel with a bounded MaxConcurrency, so a scene covering several output regions promotes concurrently without you writing a parallel loop. Keeping concurrency bounded avoids overwhelming the S3 request rate on the tiles/ prefix.

How is the pipeline cost dominated?

By Batch compute time on large mosaics and by S3 requests during publish, not by Step Functions state transitions, which are fractions of a cent. Size Batch memory from the tile-pyramid math and keep small files on Lambda so you pay for a multi-vCPU job only when the payload genuinely needs it.