Spatial IaC Terraform & Pulumi for GIS

Event-Driven COG Ingestion with S3 Notifications

Imagery arrives on its own schedule. A satellite tasking order completes at 04:12, a drone survey uploads in a burst of three hundred frames, a partner drops a quarterly orthophoto set overnight. A polling ingestion job either wastes most of its runs finding nothing or discovers new imagery hours after it landed; an event-driven pipeline starts converting a scene to a Cloud Optimized GeoTIFF within seconds of the upload completing. The difficulty is that object-storage notifications are at-least-once, unordered, and fire on every object including the ones your own pipeline writes — so a naive wiring produces duplicate conversions and, in the worst case, an infinite loop. This guide extends Raster Pipeline Provisioning within Geospatial Resource Provisioning.

Symptom identification and triage

If you are moving from polling, the symptoms that justify the change are latency and waste: imagery visible on the map hours after upload, and a scheduled job that runs 288 times a day to do useful work on four of them. If you already have an event pipeline misbehaving, three signatures dominate.

Duplicate conversions of the same scene. Two conversion jobs for one upload. Notifications are at-least-once, so a duplicate delivery is normal and expected; a pipeline without idempotency treats it as a second scene and pays twice, or worse, produces two outputs that race to write the same key.

A recursion loop. Compute cost climbs without bound and the object count doubles repeatedly. The classic cause is a notification configured on the whole bucket, so the COG the pipeline writes triggers the pipeline again. Prefix and suffix filters are not tidiness — they are the loop break.

Silently dropped scenes. An upload appears in the bucket and nothing happens. Usually a multipart upload whose completion event type is not the one the notification subscribes to, or a scene that failed conversion and vanished because there is no dead-letter queue.

Event-driven COG ingestion with the loop break and the dead letter path An upload lands under the raw prefix and emits an object-created notification, filtered by both prefix and file suffix so that only source imagery qualifies. The event enters a queue which buffers bursts and absorbs duplicate deliveries. The conversion job checks an idempotency record keyed by object identity and version before doing any work, so a redelivered event is a cheap no-op. The converted Cloud Optimized GeoTIFF is written to a separate output prefix that no notification watches, which is what prevents the pipeline from triggering itself. A message that exhausts its retries moves to a dead-letter queue where it is visible, rather than being silently discarded. Upload raw/ prefix Filtered event prefix + suffix Queue buffers the burst Convert idempotency check cog/ prefix unwatched Dead-letter queue after the retry limit The output prefix is watched by nothing. That single fact is what stops the pipeline triggering itself.

Prerequisites and environment assumptions

Terraform 1.6 or later with hashicorp/aws pinned at ~> 5.60. A bucket layout that separates source imagery from derived products by prefix, following the conventions in Object Storage for Raster and Vector Data — this guide’s loop break depends on it. A conversion runtime with GDAL available; for scenes under a few gigabytes a Lambda with a GDAL layer is sufficient, and beyond that the batch approach in Provisioning Batch Compute for Raster Mosaicking is the right target.

Decide the idempotency key before writing any code. Object key alone is insufficient because a re-upload under the same key is a legitimately new scene; object key plus version identifier, or key plus ETag, identifies a specific object generation and is what makes redelivery cheap and re-upload effective.

Step-by-step implementation

  1. Filter the notification to source imagery only. Configure the event on the raw/ prefix with the source suffix, so the converted output under cog/ cannot match. This is the recursion break, and it is worth stating in a comment in the code because it looks like an optimisation and is not.

  2. Put a queue between the notification and the work. Direct invocation couples the notification’s concurrency to your conversion capacity, and a drone upload of three hundred frames will exceed it. A queue absorbs the burst, gives you a retry policy, and provides a dead-letter destination.

  3. Make the conversion idempotent. Record the object’s identity and version in a conditional write before starting work, and treat a conditional-write failure as “already handled” rather than as an error. Without this, at-least-once delivery means paying twice for a proportion of every day’s imagery.

  4. Write output to an unwatched prefix, atomically. Convert to a temporary key and copy to the final key on success, so a consumer never sees a partially written COG. A half-written GeoTIFF that a tile renderer opens is a confusing failure precisely because the file format tolerates truncation until it does not.

  5. Configure a dead-letter queue and alarm on its depth. A scene that fails conversion must be visible. The single most common cause of “we lost a day’s imagery” is a message that exhausted retries into a destination nobody monitors.

terraform {
  required_version = ">= 1.6.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.60"
    }
  }
}

resource "aws_sqs_queue" "ingest_dlq" {
  name                      = "cog-ingest-dlq"
  message_retention_seconds = 1209600 # 14 days: long enough to notice and replay
}

resource "aws_sqs_queue" "ingest" {
  name = "cog-ingest"
  # Visibility must exceed the longest conversion, or a slow scene is redelivered
  # while it is still being converted and the work is duplicated.
  visibility_timeout_seconds = 900
  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.ingest_dlq.arn
    maxReceiveCount     = 3
  })
}

resource "aws_s3_bucket_notification" "imagery" {
  bucket = aws_s3_bucket.imagery.id

  queue {
    queue_arn = aws_sqs_queue.ingest.arn
    # Both event types: a large scene arrives as a multipart upload, and
    # subscribing only to Put silently drops every scene above the threshold.
    events = ["s3:ObjectCreated:Put", "s3:ObjectCreated:CompleteMultipartUpload"]

    # THE LOOP BREAK. The pipeline writes to cog/, which is not raw/, so the
    # output cannot re-trigger the pipeline. Removing either filter turns this
    # into an unbounded recursion.
    filter_prefix = "raw/"
    filter_suffix = ".tif"
  }
}

# Idempotency records, keyed by object identity AND version so a genuine
# re-upload is processed while a redelivery is not.
resource "aws_dynamodb_table" "processed" {
  name         = "cog-ingest-processed"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "object_version"

  attribute {
    name = "object_version"
    type = "S"
  }

  ttl {
    attribute_name = "expires_at"
    enabled        = true
  }
}

resource "aws_cloudwatch_metric_alarm" "dlq_depth" {
  alarm_name          = "cog-ingest-dlq-not-empty"
  comparison_operator = "GreaterThanThreshold"
  threshold           = 0
  evaluation_periods  = 1
  metric_name         = "ApproximateNumberOfMessagesVisible"
  namespace           = "AWS/SQS"
  period              = 300
  statistic           = "Maximum"
  dimensions          = { QueueName = aws_sqs_queue.ingest_dlq.name }
  alarm_actions       = [var.oncall_topic_arn]
}

variable "oncall_topic_arn" { type = string }

The conversion itself is a short GDAL invocation, and the flags are worth being deliberate about because they determine whether the output is genuinely cloud-optimized:

# Convert to COG. The overviews and the internal tiling are what make range
# reads cheap for a tile renderer — a GeoTIFF without them is just a GeoTIFF
# in a bucket, and every tile request reads far more than it needs.
gdal_translate "/vsis3/${SRC_BUCKET}/${SRC_KEY}" /tmp/out.tif \
  -of COG \
  -co COMPRESS=DEFLATE \
  -co PREDICTOR=2 \
  -co BLOCKSIZE=512 \
  -co OVERVIEWS=AUTO \
  -co NUM_THREADS=ALL_CPUS

# Write to a temporary key, then copy to the final key, so a consumer never
# opens a partially written raster.
aws s3 cp /tmp/out.tif "s3://${DST_BUCKET}/cog/.staging/${SCENE_ID}.tif"
aws s3 mv "s3://${DST_BUCKET}/cog/.staging/${SCENE_ID}.tif" \
          "s3://${DST_BUCKET}/cog/${SCENE_ID}.tif"
Idempotency by object version, not by key Three deliveries of similar-looking events resolve differently. The first delivery of a scene attempts a conditional write of a record keyed by object key and version; the write succeeds and the conversion runs. A duplicate delivery of the same event attempts the same conditional write, which fails because the record already exists, so the job exits immediately having done no work and paid almost nothing. A genuine re-upload of the same key produces a different version identifier, so its conditional write succeeds and the conversion runs again — which is correct, because the pixels are new. First delivery key + version v1 conditional write ok convert Redelivery key + version v1 again write rejected exit, no work done Genuine re-upload same key, version v2 conditional write ok convert — the pixels are new Keying on the object KEY alone would make the third case a no-op, and the new imagery would never appear.

Verification

Upload a test scene to the raw prefix and confirm a COG appears under the output prefix within the expected latency, then confirm the loop break by checking that the output object generated no second queue message — the queue’s received-message count should increment by one, not two. Deliver the same event twice deliberately and confirm the second produces no conversion and no second output.

Then verify the output is genuinely cloud-optimized rather than merely a GeoTIFF: gdalinfo should report internal tiling and overviews, and rio cogeo validate returns a definitive answer. Finally test the failure path — upload a corrupt file, confirm it retries the configured number of times and lands in the dead-letter queue, and confirm the alarm fires. A dead-letter queue that has never received a message is untested infrastructure.

What makes the output cloud-optimized rather than merely stored Three properties separate a Cloud Optimized GeoTIFF from an ordinary GeoTIFF that happens to live in a bucket. Internal tiling means a request for a small area reads one block rather than a whole scanline-organised file. Embedded overviews mean a low-zoom request reads a small pyramid level instead of decimating full resolution. And a header laid out at the front lets a client discover both of those with one small range request before deciding what else to fetch. A conversion that omits them produces a file that opens correctly and makes every tile request read far more than it needs, which shows up as latency and egress rather than as an error. Internal tiling 512 by 512 blocks a small area reads one block not a whole scanline file Embedded overviews a pyramid inside the file low zoom reads a small level instead of decimating full res Header at the front one small range request locates everything else before anything is downloaded Omit these and the file still opens — and every tile request reads far more than it needs, as latency and egress.

Preventing recurrence

  • Assert the notification filters in policy. A plan-contract rule that fails any bucket notification without both a prefix and a suffix filter closes the recursion class permanently, using the assertion approach in Testing and Validation for Spatial IaC.
  • Alarm on dead-letter depth and on ingest freshness. Depth catches failures; freshness — the age of the newest successfully converted scene — catches the case where notifications stopped arriving at all and there is therefore nothing to fail.
  • Keep visibility timeout above the longest conversion. When scene sizes grow, this ceiling is the first thing to become wrong, and its symptom is duplicate work rather than an error.
  • Replay from the dead-letter queue deliberately. A documented replay procedure turns a bad day’s imagery into a re-run rather than a manual reconstruction.

Frequently Asked Questions

Why not invoke the conversion directly from the notification?

Because a burst of three hundred frames arrives faster than the conversion capacity, and direct invocation gives you throttling errors instead of a backlog. A queue converts a burst into a backlog, which is the failure mode you want, and it is also where retries and the dead-letter destination live.

How do I stop the pipeline processing its own output?

Prefix and suffix filters on the notification, with the output written to a prefix the notification does not match. Nothing else is reliable — a check inside the conversion job runs after the invocation has already been billed, and one deployment that forgets it produces the loop.

Are notifications ever lost?

Delivery is at-least-once, not exactly-once, and the practical implication is that duplicates are normal and gaps are rare but possible. Treat freshness monitoring as the backstop: a periodic reconciliation that compares source objects against converted outputs catches the rare gap without polling for every upload.

Should the COG output land in the same bucket?

A separate prefix in the same bucket is sufficient for the loop break and simpler to manage. A separate bucket is preferable when the lifecycle policies genuinely differ — source imagery archived aggressively, derived COGs kept hot — because it removes the risk that a lifecycle rule intended for one applies to the other.