Spatial IaC Terraform & Pulumi for GIS

Raster Data Pipeline Provisioning

A raster pipeline is the moving machinery that turns a raw upload into a serveable tile, and it fails in ways a static bucket never does: a job dies halfway through a gdal_translate, an overview build runs the box out of memory, or a manifest is read a half-second before S3 makes the new object visible. Provisioning this ingest-to-publish path as code — the orchestrator, the compute tasks, the queues, and the retry semantics — is a distinct discipline within Geospatial Resource Provisioning, sitting downstream of the storage layout defined in Object Storage for Raster & Vector and drawing on the same primitives as Serverless Geospatial Processing Provisioning. This guide covers how to provision an idempotent, retry-safe raster workflow whose every stage is declared, versioned, and observable.

Ingest to Publish Raster Pipeline Data Flow A raw GeoTIFF landing in the ingest prefix raises an S3 event notification onto an SQS queue backed by a dead-letter queue. The queue triggers a Step Functions state machine. The first state validates the upload with gdalinfo, checking projection, band count, and readability. A choice state branches on file size: small files run gdal_translate on a Lambda task, while large files run a heavier mosaic and overview build as an AWS Batch job on Fargate or EC2. Both paths emit a Cloud-Optimized GeoTIFF with internal overviews to a staging prefix. A final publish state atomically copies the object into the tiles prefix and updates a manifest in the tile store. Failures at any state route to the dead-letter queue for replay. raw GeoTIFF ingest/ upload SQS queue S3 event · DLQ dead-letter queue validate gdalinfo checks size? branch small Lambda · gdal_translate small COG convert large Batch · mosaic + overviews heavy job COG + overviews publish · atomic copy → tiles/ + manifest promote to tile store

Environment Parity and Configuration Drift Mitigation

A raster pipeline drifts differently from a bucket: the drift lives in the behavior of the stages, not just their attributes. A staging environment whose Batch compute environment has 4 vCPU per job and a production one with 16 will both pass a smoke test on a small tile and then diverge violently on a 40,000 by 40,000-pixel orthomosaic, where staging silently OOM-kills the overview build. Parity for a pipeline means the state machine definition, the task role, the container image digest, the Batch job definition (vCPU, memory, and ephemeral storage), the Lambda memory and timeout, the queue redrive policy, and the retry/backoff parameters all live in one module, parameterized only by environment.

The specific drift sources for a raster pipeline are worth naming. First, container image tags pinned to :latest — a GDAL base image rebuilt upstream changes COG output subtly (block size, predictor) and breaks downstream readers, so pin to an image digest, never a floating tag. Second, Batch memory ceilings bumped by hand in the console during an incident and never returned to code, so the next environment inherits the old, too-small value. Third, retry counts and visibility timeouts on the queue tuned live, which changes how many times a poison message is reprocessed before it lands in the dead-letter queue. Lock all three in the module and run a scheduled terraform plan on a cron that fails the build on any non-empty diff, so behavioral drift surfaces as an alarm rather than as a 2 a.m. mosaic failure.

CI/CD Validation and Operational Guardrails

Pipeline changes deserve the same scrutiny as the raster payloads they process. Pre-apply stages should:

  • Lint the state machine with statelint (or the AWS Step Functions local validator) so a malformed Amazon States Language (ASL) document fails at merge, not on first execution.
  • Scan the GDAL container for a pinned digest and a known-good gdalinfo --version, rejecting any job definition that references a floating tag.
  • Validate a golden fixture end to end by running the pipeline against a small known raster in a dry-run account and asserting the output passes rio cogeo validate — this catches a COG-driver regression before it reaches production imagery.
  • Diff Batch and Lambda sizing against a recorded baseline, so a memory reduction that would OOM the overview build is flagged in review rather than discovered in production.

These checks pair with the access-boundary rules in IAM Role Mapping for GIS: the task role must be able to read ingest/, write staging/, and promote to tiles/, and nothing more. The bucket lifecycle that keeps ingest/ from accumulating raw uploads indefinitely is codified alongside the pipeline, following Configuring S3 Lifecycle Rules for GIS Tiles.

Resource Architecture and Service Integration

The pipeline is an orchestration graph over four service classes, and keeping them decoupled is what makes retries safe.

  • The event and queue layer. An S3 ObjectCreated notification on the ingest/ prefix publishes to an SQS queue with a redrive policy pointing at a dead-letter queue. The queue absorbs upload bursts and gives the orchestrator a durable, at-least-once trigger rather than a fire-and-forget event.
  • The orchestrator. A Step Functions state machine (or an equivalent managed workflow) owns the control flow: validate, branch by size, convert, publish. It holds the retry and catch logic so the compute tasks stay stateless and replaceable. This is the same orchestration surface covered in depth by Orchestrating COG Generation Pipelines with Step Functions.
  • The compute tasks. Small files convert on a Lambda task; large mosaics and deep overview builds run as AWS Batch jobs on Fargate or EC2, where a job can request tens of gigabytes of memory and hours of runtime that Lambda cannot provide. The provisioning trade-offs for the serverless side mirror those in Serverless Geospatial Processing Provisioning.
  • The tile store. The publish step writes the finished COG and updates a manifest in the tiles/ prefix defined by Object Storage for Raster & Vector, which downstream tile servers and CDNs read.

Expose the state machine ARN, the queue URL, and the task role ARN as typed module outputs so downstream stacks consume them via remote state rather than hard-coding an identifier that moves when an account or region changes.

Sizing the work: the tile-pyramid math

Provisioning a pipeline means predicting how much work a publish generates, and for a raster destined to become a tile pyramid the work is governed by geometry, not intuition. A full web-mercator pyramid rendered to maximum zoom Z holds 4^z tiles at each level z, because every step down the pyramid quadruples the tile count:

$$ N_{\text{tiles}} = \sum_{z=0}^{Z} 4^{z} = \frac{4^{Z+1} - 1}{3} $$

That closed form is what lets you size a Batch job before it runs. Going from zoom 12 to zoom 16 does not add a few percent of work — it multiplies the leaf-level tile count by 4^4, or 256 times. The overview build inside gdal_translate/gdaladdo is the same geometry in reverse: each overview level is a quarter the pixels of the one below it, so the total overview storage overhead converges toward one-third of the base image:

$$ \text{overview overhead} = \sum_{k=1}^{\infty} \frac{1}{4^{k}} = \frac{1}{3} $$

Feed both estimates into the pre-apply gate that draws on Cost Estimation Frameworks, so a new high-zoom layer sizes its Batch memory and its storage budget deliberately instead of discovering the ceiling at runtime.

Runnable Configuration

The following Terraform provisions the orchestration spine: a Step Functions state machine that validates an upload, branches by size to a Lambda or a Batch task, and publishes the result, with an IAM role scoped to exactly those actions. Provider versions are pinned and every geospatial decision is annotated inline.

terraform {
  required_version = ">= 1.7.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.60"
    }
  }
}

# Execution role the state machine assumes to invoke tasks.
resource "aws_iam_role" "pipeline" {
  name = "${var.environment}-raster-pipeline-sfn"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "states.amazonaws.com" }
      Action    = "sts:AssumeRole"
    }]
  })
}

# Least-privilege: invoke the two task types, submit Batch jobs,
# and read the DLQ redrive — nothing else.
resource "aws_iam_role_policy" "pipeline" {
  name = "raster-pipeline-invoke"
  role = aws_iam_role.pipeline.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = ["lambda:InvokeFunction"]
        Resource = [var.cog_lambda_arn]
      },
      {
        Effect   = "Allow"
        Action   = ["batch:SubmitJob", "batch:DescribeJobs", "batch:TerminateJob"]
        Resource = ["*"] # Batch SubmitJob does not support resource-level ARNs for all sub-resources
      }
    ]
  })
}

# The state machine. The definition is templated so the same ASL
# ships to every environment with only ARNs and the size threshold changing.
resource "aws_sfn_state_machine" "raster" {
  name     = "${var.environment}-raster-ingest-to-publish"
  role_arn = aws_iam_role.pipeline.arn

  definition = templatefile("${path.module}/asl/pipeline.asl.json", {
    cog_lambda_arn      = var.cog_lambda_arn
    batch_job_queue     = aws_batch_job_queue.mosaic.arn
    batch_job_defn      = aws_batch_job_definition.mosaic.arn
    size_threshold_mb   = var.size_threshold_mb # route files above this to Batch
  })

  # Ship execution history to CloudWatch so a failed COG build is debuggable.
  logging_configuration {
    log_destination        = "${aws_cloudwatch_log_group.pipeline.arn}:*"
    include_execution_data = true
    level                  = "ERROR"
  }

  tags = {
    Workload    = "raster-pipeline"
    Environment = var.environment
  }
}

# Batch job definition for heavy mosaicking + overview builds.
# Memory is sized from the tile-pyramid estimate, NOT guessed.
resource "aws_batch_job_definition" "mosaic" {
  name = "${var.environment}-raster-mosaic"
  type = "container"
  platform_capabilities = ["FARGATE"]

  container_properties = jsonencode({
    # Pin to a DIGEST, never :latest — a rebuilt GDAL image changes COG output.
    image      = "${var.gdal_repo}@sha256:${var.gdal_image_digest}"
    command    = ["/opt/build_cog.sh", "Ref::input_key", "Ref::output_key"]
    jobRoleArn = var.task_role_arn
    resourceRequirements = [
      { type = "VCPU",   value = "4" },
      { type = "MEMORY", value = "30720" } # 30 GiB headroom for large overview builds
    ]
    ephemeralStorage = { sizeInGiB = 100 } # scratch space for gdaladdo temp files
  })

  # Retries handle transient S3 eventual-consistency + spot reclaim.
  retry_strategy {
    attempts = 3
    evaluate_on_exit {
      action           = "RETRY"
      on_status_reason = "Host EC2*"
    }
    evaluate_on_exit {
      action    = "EXIT"
      on_reason = "*" # non-transient failures exit immediately to the DLQ path
    }
  }
}

output "state_machine_arn" {
  description = "Consumed by the S3-event trigger and downstream observability stacks."
  value       = aws_sfn_state_machine.raster.arn
}

Note: The Batch MEMORY value above is expressed in whole mebibytes (30720 = 30 GiB), which is the integer unit AWS Batch requires. Never express Batch or RDS memory as a percentage string — a parameter like "75%" is rejected, and any parameter-group memory setting must likewise be given in AWS integer units (8 kB blocks for RDS parameter groups) so the value is unambiguous across instance classes.

Guardrails Embedded in Configuration

The configuration encodes four guardrails that map directly to the ways raster pipelines fail under load.

Idempotency keyed on the source object. The output key is derived deterministically from the input object key and its version ID, so replaying a message overwrites the same staging object rather than producing a duplicate. A pipeline that is not idempotent turns an at-least-once queue into duplicated, half-published tiles.

Retries with a dead-letter queue. Both the queue redrive policy and the Batch retry_strategy distinguish transient failures (spot reclaim, an S3 read that briefly 404s on a just-written object) from permanent ones (a corrupt upload). Transient failures retry with backoff; permanent ones land in the dead-letter queue for inspection instead of retrying forever.

Digest-pinned compute. The GDAL container is referenced by sha256 digest, so the byte-for-byte behavior of gdal_translate and gdaladdo is reproducible across environments and over time. This is the single most effective guard against silent COG-format drift.

Atomic publish. The publish step writes to staging/ and only then copies to tiles/ and updates the manifest, so a reader never observes a partially written COG at the serving path. The manifest update is the last write, which sidesteps the eventual-consistency window on the object itself.

Troubleshooting and Failure Modes

  • Partial COG on task failure. A Batch job is killed mid-gdal_translate and leaves a truncated .tif in staging/. Because the byte range is incomplete, rio cogeo validate fails and downstream readers get corrupt tiles. The fix is to write to a temp key and rename (copy-then-delete) only on success, and to make the publish step assert rio cogeo validate before promoting — never publish an object the converter did not confirm.
  • Overview generation OOM. gdaladdo on a large single-band float raster exhausts the Batch job’s memory and the container is OOM-killed with exit code 137. The overview build holds decompressed blocks in memory, so the ceiling scales with tile block size and thread count. Size the job from the pyramid math above, cap GDAL_NUM_THREADS, and reduce GDAL_CACHEMAX so the driver spills to the ephemeral scratch disk rather than RAM.
  • S3 eventual-consistency on the manifest. The publish step reads the manifest, appends the new layer, and writes it back, but a concurrent execution read the same pre-update copy and the two writes race — one layer vanishes. Serialize manifest updates through a single Step Functions execution or a conditional write (an If-Match/version check), rather than a naive read-modify-write from parallel jobs.
  • Poison message replay storm. A malformed upload fails validation, returns to the queue, and is redelivered until it drowns out healthy work. A too-high maxReceiveCount on the redrive policy keeps the poison message in rotation. Set the visibility timeout above the worst-case job runtime and a low maxReceiveCount so a genuinely broken file lands in the dead-letter queue after a couple of attempts.
  • Choice-state threshold misfire. A file just under the size threshold is routed to Lambda, exceeds the 15-minute timeout mid-conversion, and fails without ever trying Batch. The threshold was tuned for pixel count but the routing keyed on byte size, and a highly compressed source decompresses far larger than its object size. Route on estimated decompressed pixel count (from gdalinfo dimensions and band count), not on the S3 object size.