Spatial IaC Terraform & Pulumi for GIS

Configuring S3 Lifecycle Rules for GIS Tiles: Incident Response and IaC Remediation

A drifted Amazon S3 lifecycle policy on a tile bucket fails loudly: edge CDNs start returning HTTP 404/410 for tiles that existed an hour ago, rendering workers burn through their retry budgets, and the storage-cost dashboard shows an unexplained drop as objects are silently expired or transitioned out from under live readers. This guide walks through triaging that incident, recovering the lost tiles, and re-encoding the lifecycle rules so the failure cannot recur — within the broader Geospatial Resource Provisioning discipline and the specific patterns covered in Object Storage for Raster/Vector Workloads. The root cause is almost always an over-broad Expiration or Transition rule applied during a routine infrastructure change.

Symptom Identification and Triage

The defining signal is a sudden rise in GetObject failures concentrated on a single prefix — typically tiles/ — while unrelated prefixes stay healthy. Confirm the lifecycle hypothesis before touching anything by cross-checking three sources:

  • CDN / origin status codes. A wave of 404 NoSuchKey (object expired) or 403/410 on tiles that recently served 200 indicates objects were removed or transitioned to a class the origin role cannot read. Pan-and-zoom amplifies this into hundreds of failures per session.
  • CloudWatch 4xxErrors and worker retries. A step-change in the bucket’s 4xxErrors metric, paired with rendering-worker retry spikes, confirms consumers are repeatedly refetching tiles that no longer resolve.
  • CloudTrail management events. Filter for PutBucketLifecycleConfiguration and DeleteObject/lifecycle expiration events. The eventTime, userIdentity, and requestParameters.LifecycleConfiguration pin the exact change and the principal (usually a CI/CD service role) that applied it.
Triage Decision Tree for a Tile 404/410 Spike A tile 404/410 spike paired with rising bucket 4xxErrors routes through a CloudTrail management-events check. A recent PutBucketLifecycleConfiguration on the active prefix means lifecycle drift, handled by freezing the pipeline, suspending rules, and recovering versions. No lifecycle event points to an IAM or storage-class read failure. Origin 5xx or connection errors point to a routing or CDN issue covered elsewhere. Tile 404 / 410 spike on tiles/ bucket 4xxErrors rising · worker retries spike CloudTrail check management events rule applied none found origin 5xx PutBucketLifecycle- Configuration targets active prefix No lifecycle change in trail 5xx / connection errors at origin Lifecycle drift this guide applies IAM / storage-class read failure Routing / CDN issue Freeze pipeline → suspend rules → recover versions Verify origin role & object storage class Check VPC routing outside this guide

The decision is binary: if CloudTrail shows a recent PutBucketLifecycleConfiguration whose rules target an active prefix, treat this as lifecycle drift and proceed below. If not, the 404s are more likely an IAM or VPC Routing for Tile Servers problem and this guide does not apply.

Prerequisites and Environment Assumptions

This procedure assumes:

  • Provider versions: Terraform >= 1.6 with the AWS provider ~> 5.60, or Pulumi @pulumi/aws ^6.0. The aws_s3_bucket_lifecycle_configuration resource and the filter-per-rule schema below require these minimums; older AWS-provider releases used the deprecated inline lifecycle_rule block and will silently reshape your rules on upgrade.
  • State backend: a remote, locked backend — S3 + DynamoDB for Terraform, or the Pulumi Service / S3 backend — so you can inspect the live plan without racing the CI/CD pipeline. State reconciliation patterns are covered in Managing Terraform State Locks for Spatial Data.
  • Bucket versioning: enabled before the incident. Without versioning, expired tiles are unrecoverable and the recovery step degrades to a full cache rebuild.
  • IAM permissions for the operator: s3:GetLifecycleConfiguration, s3:PutLifecycleConfiguration, s3:ListBucketVersions, s3:GetObjectVersion, and s3:RestoreObject on the affected bucket, plus read access to the state backend.

Step-by-Step Remediation

1. Freeze the pipeline. Disable the CI/CD workflow that applies this stack so no further apply/up re-asserts the bad rule while you work. This is the single most important step — every minute the rule stays active, more tiles age past the expiration threshold.

2. Back up and suspend the live rules. Capture the current configuration, then apply an empty rule set to stop further data loss immediately. This is a console/CLI break-glass action, intentionally out-of-band from IaC because the IaC source is what produced the bad state:

# Snapshot the live lifecycle config before changing anything
aws s3api get-bucket-lifecycle-configuration \
  --bucket prod-gis-tiles-production \
  --output json > lifecycle-backup-$(date +%s).json

# Suspend ALL rules so no further tiles expire or transition
aws s3api put-bucket-lifecycle-configuration \
  --bucket prod-gis-tiles-production \
  --lifecycle-configuration '{"Rules":[]}'

3. Identify the offending rule from state, not memory. Inspect the declared configuration so the corrected version is grounded in the real resource, never reconstructed by hand. Do not run terraform state rm or hand-edit the state file:

terraform state show aws_s3_bucket_lifecycle_configuration.tile_lifecycle
# or, for Pulumi:
pulumi stack export | jq '.deployment.resources[]
  | select(.type=="aws:s3/bucketLifecycleConfigurationV2:BucketLifecycleConfigurationV2")'

Look for any rule whose filter is empty (matches the whole bucket) or whose prefix overlaps tiles/ while carrying an expiration or aggressive transition. That is the blast radius.

4. Recover expired tiles from versioned history. With versioning enabled, an expiration created delete markers rather than destroying data. List the affected versions and remove the delete markers to restore current objects. The geospatial reason this matters: tile pyramids are addressed by deterministic z/x/y keys, so a missing current version returns a hard 404 — there is no graceful fallback the renderer can use.

aws s3api list-object-versions \
  --bucket prod-gis-tiles-production --prefix tiles/ \
  --query "DeleteMarkers[?IsLatest==\`true\`].[Key,VersionId]" \
  --output text | while read -r key vid; do
    aws s3api delete-object --bucket prod-gis-tiles-production \
      --key "$key" --version-id "$vid"   # removes the delete marker, exposing the prior version
done

5. Reconcile the spatial index. Misaligned transitions frequently orphan tile-index records that point at objects now in a colder class or gone entirely. Cross-reference the restored object manifest against the PostGIS Cluster Provisioning tile-index table and re-queue any rows whose objects could not be restored for regeneration, so the cache and the index agree before traffic resumes.

6. Re-encode and reapply corrected rules. Replace the blanket rule with tag- and prefix-scoped rules that only ever touch completed, non-active batches, then reapply through IaC (not the console). The intended lifecycle state machine is:

Intended S3 Storage-Class State Machine for Tile Objects A tile written into S3 Standard (hot) transitions to Intelligent-Tiering (warm) after 30 days once completed, then to Glacier Instant Retrieval (cold) after 180 days for the archive prefix, expiring at 1095 days. A noncurrent version in Standard expires after 90 days, which keeps a rollback window without touching live tiles. tile written S3 Standard hot 30 days completed Intelligent-Tiering warm 180 days archive/ Glacier IR cold · instant retrieval expire 1095 d noncurrent version · expire after 90 days (rollback window)

Minimal reproducible Terraform configuration

terraform {
  required_version = ">= 1.6"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.60"
    }
  }
}

resource "aws_s3_bucket" "gis_tile_cache" {
  bucket = "prod-gis-tiles-${var.environment}"
}

resource "aws_s3_bucket_versioning" "gis_tile_cache" {
  bucket = aws_s3_bucket.gis_tile_cache.id
  versioning_configuration { status = "Enabled" }
}

resource "aws_s3_bucket_lifecycle_configuration" "tile_lifecycle" {
  bucket = aws_s3_bucket.gis_tile_cache.id

  rule {
    id     = "hot-to-warm-transition"
    status = "Enabled"
    filter {
      tag {                       # scope by tag, never the whole bucket
        key   = "tile_status"
        value = "completed"       # only fully rendered batches age out
      }
    }
    transition {
      days          = 30
      storage_class = "INTELLIGENT_TIERING"
    }
    noncurrent_version_expiration {
      noncurrent_days = 90        # keep recovery window for rollbacks
    }
  }

  rule {
    id     = "archive-inactive-tiles"
    status = "Enabled"
    filter { prefix = "archive/" } # explicit prefix, disjoint from tiles/
    transition {
      days          = 180
      storage_class = "GLACIER_IR"
    }
    expiration { days = 1095 }
  }
}

The equivalent Pulumi resource (pin @pulumi/aws ^6.0 in package.json) mirrors the same scoping:

import * as aws from "@pulumi/aws"; // package.json: "@pulumi/aws": "^6.0.0"

const tileBucket = new aws.s3.BucketV2("gisTileCache", {
  bucket: `prod-gis-tiles-${process.env.ENVIRONMENT}`,
});

new aws.s3.BucketVersioningV2("gisTileCacheVersioning", {
  bucket: tileBucket.id,
  versioningConfiguration: { status: "Enabled" },
});

new aws.s3.BucketLifecycleConfigurationV2("tileLifecycle", {
  bucket: tileBucket.bucket,
  rules: [
    {
      id: "hot-to-warm-transition",
      status: "Enabled",
      filter: { tag: { key: "tile_status", value: "completed" } },
      transitions: [{ days: 30, storageClass: "INTELLIGENT_TIERING" }],
      noncurrentVersionExpiration: { noncurrentDays: 90 },
    },
    {
      id: "archive-inactive-tiles",
      status: "Enabled",
      filter: { prefix: "archive/" },
      transitions: [{ days: 180, storageClass: "GLACIER_IR" }],
      expiration: { days: 1095 },
    },
  ],
});

Verification

Confirm the fix is live before unfreezing the pipeline:

  • Inspect the applied rules: aws s3api get-bucket-lifecycle-configuration --bucket prod-gis-tiles-production should show every rule carrying a non-empty Filter and no Expiration on tiles/.
  • Probe the tile endpoint: request a previously failing tile through the CDN, e.g. curl -I https://tiles.example.org/tiles/12/2048/1361.png, and confirm a 200 with the expected Content-Type: image/png.
  • Watch the metrics settle: the bucket’s CloudWatch 4xxErrors should return to baseline and rendering-worker retry counts should fall to normal within one cache-fill cycle.
  • Audit the trail: a CloudTrail lookup on PutBucketLifecycleConfiguration should show your corrected apply as the most recent event, attributed to the CI/CD role rather than a human principal.

Preventing Recurrence

Encode the fix so the next routine change cannot reintroduce it. The cost trade-off the rules are tuned against is the per-tier monthly storage bill, where each prefix’s volume meets a different unit price:

$$ C_{\text{month}} = \sum_{t \in {\text{std}, \text{IT}, \text{glacier}}} S_t \cdot p_t $$

Driving cold batches toward GLACIER_IR lowers C only while the served tiles/ prefix stays on hot storage — which is exactly why the guardrails below forbid expiration there:

  • Policy-as-code gate. Add an Open Policy Agent / HashiCorp Sentinel (Terraform) or CrossGuard (Pulumi) rule to the pull-request pipeline that rejects any lifecycle rule with an empty filter, or any expiration/transition whose prefix overlaps tiles/. This is the same review discipline applied to access boundaries in IAM Role Mapping for GIS.
  • Scheduled drift detection. Run terraform plan -detailed-exitcode (or pulumi preview --expect-no-changes) on a cron and fail the build on any non-empty diff, converting silent console edits into a visible alarm before a promotion cycle.
  • Least-privilege IAM. Restrict s3:PutLifecycleConfiguration to the CI/CD service role only, so no human can hand-edit lifecycle rules in the console during an incident and leave the change un-codified.
  • Cross-environment parity. Keep staging and production rules generated from the same module, parameterized only by environment, so a tested rule set in staging is byte-for-byte what reaches the live tile cache feeding GeoServer Deployment Patterns.