Spatial IaC Terraform & Pulumi for GIS

Object Storage for Raster/Vector Workloads in Spatial IaC

Object storage is the persistence layer that most often drifts out of sync between environments on a spatial platform, because buckets accumulate ad-hoc lifecycle rules, CORS exceptions, and IAM grants long after the original deployment. Treating a bucket as an undifferentiated data dump is what breaks tile caches, leaks credentials, and inflates egress bills. This guide covers how to provision raster, vector, and tile storage as codified infrastructure within the broader Geospatial Resource Provisioning discipline, so that the hot path feeding PostGIS Cluster Provisioning and the compute fleet defined in Compute Node Orchestration consume exactly the assets they are authorized to, in every region, from the same commit.

Codified Object Store Feeding Three Spatial Consumers by Prefix An IaC state backend provisions the bucket configuration and a KMS customer-managed key supplies server-side encryption; both attach off to the side. The bucket is encrypted, versioned, and blocks public access, and exposes three prefixes as an interface contract. The rasters/ prefix holds source Cloud-Optimized GeoTIFFs read by GDAL compute workers over hot byte-range requests. The vectors/ prefix holds GeoParquet and GeoJSON extracts that hydrate a PostGIS cluster through foreign data wrappers or COPY loaders. The tiles/ prefix holds rendered tile pyramids served by a CDN-fronted GeoServer cache and drives the dominant egress and request load. A lifecycle and intelligent-tiering rule transitions only cold tile objects down to an archive tier, leaving rasters and vectors on the hot path. IaC state backend locked · provisions config KMS customer key SSE-KMS · bucket key Object store bucket encrypted · versioned · no public access rasters/ — source COGs hot byte-range reads vectors/ — GeoParquet · GeoJSON FDW / COPY loaders tiles/ — rendered pyramids dominant egress + requests GDAL compute workers stream rasters/ PostGIS cluster FDW / ETL hydrate GeoServer tile cache CDN-fronted lifecycle: cool → archive Archive tier intelligent tiering · cold tiles only

Environment Parity and Configuration Drift Mitigation

Environment parity is the primary guardrail for object storage, and it fails quietly. A staging bucket that lacks the production KMS key, omits a tiles/ lifecycle transition, or carries a looser public-access posture will pass functional tests and then corrupt behavior only under production load. Parity must be enforced structurally: bucket name, storage class defaults, versioning state, encryption key, public-access block, CORS rules, and event-notification subscriptions all belong in the same module, parameterized only by environment and region.

The recurring drift sources for spatial buckets are specific. First, CORS exceptions added by hand so a browser map client can fetch tiles — these never make it back into code and break the next apply. Second, lifecycle transitions edited in the console to relieve a cost spike, which then archive objects that a time-series raster job still reads. Third, prefix-scoped IAM widened to s3:* during an incident and never narrowed again. Lock all three in the module and treat the bucket configuration as immutable infrastructure: changes flow through a pull request or they do not happen. A scheduled terraform plan (or pulumi preview) on a cron, failing the build on any non-empty diff, converts silent drift into a visible alarm before it reaches a promotion cycle.

CI/CD Validation and Operational Guardrails

Object-storage changes deserve the same scrutiny as application code, with checks tuned to the geospatial failure modes. Pre-apply pipeline stages should:

  • Validate uploaded raster integrity by running gdalinfo against staged sample objects, confirming projection, band count, and overview presence before the asset becomes addressable to consumers.
  • Lint policies-as-code with Open Policy Agent (OPA) or HashiCorp Sentinel to reject any plan that disables encryption, drops the public-access block, or grants wildcard s3:* actions.
  • Diff lifecycle rules against the access-log-derived retention model, so a transition that would archive still-hot tiles fails review rather than silently shipping.
  • Verify CORS and event-notification subscriptions are present and scoped to the expected origins and prefixes, catching the most common manual-exception regressions.

These checks belong in pull-request workflows alongside the access-boundary rules described in IAM Role Mapping for GIS, and they pair with automated rollback triggers: if a post-apply synthetic read/write against the tiles/ prefix fails, the pipeline reverts the bucket policy to the last known-good state from the State Backend Selection backend.

Resource Architecture and Service Integration

Object storage is the data plane for nearly every other spatial resource, and its prefix layout is an interface contract, not an implementation detail. A disciplined three-prefix scheme — rasters/, vectors/, tiles/ — lets IAM, lifecycle, and event rules attach to access patterns rather than to individual objects:

  • rasters/ holds source imagery and Cloud-Optimized GeoTIFFs (COGs). Consumers stream byte ranges directly, so these objects stay on hot storage and are read by GDAL-based workers in the compute fleet and by raster ETL that hydrates spatial indexes.
  • vectors/ holds GeoParquet, GeoJSON, and FlatGeobuf extracts. These feed PostGIS Cluster Provisioning through foreign data wrappers or COPY-based loaders.
  • tiles/ holds rendered raster and vector tile pyramids served by GeoServer Deployment Patterns behind a CDN. This prefix is the dominant egress and request driver.

Because the bucket endpoint and IAM role names become inputs to those downstream stacks, expose them as typed module outputs and consume them via remote state or stack references — never hard-code an ARN. This keeps the dependency graph explicit and lets a region or account move without editing every consumer.

Sizing the tile pyramid

Storage planning for the tiles/ prefix is driven by the pyramid geometry, not by guesswork. A full web-mercator pyramid rendered to maximum zoom Z contains a predictable object count, since each zoom level holds 4^z tiles:

$$ N_{\text{tiles}} = \sum_{z=0}^{Z} 4^{z} = \frac{4^{Z+1} - 1}{3} $$

Multiplying N_tiles by the mean encoded tile size yields the cache footprint; most platforms render densely only over populated areas, so apply a coverage fraction per level rather than assuming a full pyramid. Feeding this estimate into the Cost Estimation Frameworks pre-apply gate keeps a new high-zoom layer from quietly multiplying storage cost.

Runnable Configuration

The following Terraform provisions a hardened raster/vector/tile bucket with pinned providers, KMS encryption, versioning, an account-aligned public-access block, intelligent tiering for the tiles/ prefix, and prefix-scoped least-privilege access. Every geospatial-specific decision is annotated inline.

terraform {
  required_version = ">= 1.7.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.60"
    }
  }
}

resource "aws_s3_bucket" "spatial_assets" {
  bucket = "${var.environment}-spatial-assets-${var.region}"

  tags = {
    Workload    = "raster-vector-pipeline"
    DataClass   = "internal-spatial"
    Environment = var.environment
  }
}

# Versioning protects against partial tile-pyramid overwrites and
# lets a corrupted ETL load be rolled back to the prior object version.
resource "aws_s3_bucket_versioning" "spatial_assets" {
  bucket = aws_s3_bucket.spatial_assets.id
  versioning_configuration {
    status = "Enabled"
  }
}

# Customer-managed KMS, not SSE-S3: spatial imagery is often
# licensed/regulated, so the key boundary must be auditable.
resource "aws_s3_bucket_server_side_encryption_configuration" "spatial_assets" {
  bucket = aws_s3_bucket.spatial_assets.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = "aws:kms"
      kms_master_key_id = var.spatial_kms_key_id
    }
    bucket_key_enabled = true # cuts KMS request cost on high-fanout tile reads
  }
}

# Deny public access at the bucket level even if a stray ACL slips through.
resource "aws_s3_bucket_public_access_block" "spatial_assets" {
  bucket                  = aws_s3_bucket.spatial_assets.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

# Auto-tier only the tile cache: rendered tiles cool predictably,
# while source rasters and vectors stay on the hot path.
resource "aws_s3_bucket_intelligent_tiering_configuration" "tiles" {
  bucket = aws_s3_bucket.spatial_assets.id
  name   = "tiles-archive"
  filter {
    prefix = "tiles/"
  }
  tiering {
    access_tier = "ARCHIVE_ACCESS"
    days        = 90
  }
}

# Least-privilege read scoped to the three spatial prefixes only.
resource "aws_iam_policy" "dataset_prefix_scoped" {
  name        = "${var.environment}-spatial-prefix-access"
  description = "Least-privilege access scoped to rasters, vectors, and tiles"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = ["s3:GetObject", "s3:ListBucket"]
        Resource = [
          aws_s3_bucket.spatial_assets.arn,
          "${aws_s3_bucket.spatial_assets.arn}/rasters/*",
          "${aws_s3_bucket.spatial_assets.arn}/vectors/*",
          "${aws_s3_bucket.spatial_assets.arn}/tiles/*"
        ]
      }
    ]
  })
}

output "spatial_bucket_arn" {
  description = "Consumed by PostGIS, GeoServer, and compute stacks via remote state."
  value       = aws_s3_bucket.spatial_assets.arn
}
Fail-Closed Provisioning Sequence Under a State Lock A pull request carrying a storage change enters fail-closed pre-apply gates: gdalinfo validates projection, band count, and overview presence on staged objects, and OPA rejects any plan that disables KMS encryption, drops the public-access block, or grants wildcard s3 actions. If any gate fails the run halts with an audit record and no partial apply. On pass the state lock backed by S3 and DynamoDB is acquired and an ordered apply runs: create bucket, enable versioning, configure KMS encryption with a bucket key, set the public-access block, attach intelligent tiering scoped to the tiles prefix, and create prefix-scoped IAM. The lock is then released and typed outputs are published for downstream consumers to read. Pull request · storage change Pre-apply gates · fail-closed gdalinfo: projection · bands · overviews OPA: deny no-KMS · public · s3:* gate fails Halt + audit no partial apply acquire lock (S3+DynamoDB) state lock held · ordered apply create bucket versioning KMS encryption · bucket key public-access block intelligent tiering · tiles/ prefix-scoped IAM release lock Outputs published · consumers read

Guardrails Embedded in Configuration

The configuration above encodes four guardrails that exist precisely because they are the parts teams forget under pressure.

State locking. The bucket definition lives in a remote backend with mandatory locking (S3 + DynamoDB, or the Pulumi Service), aligned with State Backend Selection. Concurrent applies on the same state are what produce half-written lifecycle rules and orphaned policies; the lock makes that race impossible rather than merely unlikely.

Secret and key management. No access keys appear in the module. Compute and middleware assume the prefix-scoped role at runtime, and the encryption key is referenced by ARN (var.spatial_kms_key_id) so key rotation is a key-policy change, never an infrastructure rewrite.

Network isolation. Internal consumers reach the bucket exclusively through a VPC gateway endpoint, keeping raster and tile traffic off the public internet and removing egress as an exfiltration path. The endpoint policy itself should restrict the same three prefixes, so a compromised node cannot enumerate the whole account.

Prefix scope as a blast-radius control. Because IAM, lifecycle, and intelligent tiering all bind to rasters/, vectors/, and tiles/, a mistake is contained to one access class. Widening a grant to s3:* or to the bucket root defeats every other guardrail at once, which is exactly why the OPA gate rejects it.

Troubleshooting and Failure Modes

  • S3 prefix scope mismatch. A GeoServer node returns 403 AccessDenied on tiles while raster reads succeed. The role was scoped to rasters/* and vectors/* but the tiles/* ARN was dropped during a refactor. Diff the rendered policy against the three-prefix contract and restore the missing resource line.
  • VPC endpoint policy gap. Reads work from a developer laptop but time out from inside the VPC. The gateway endpoint exists but its policy omits s3:GetObject for the bucket, so in-VPC requests are silently denied while public DNS still resolves. Align the endpoint policy with the prefix-scoped IAM policy.
  • Premature lifecycle archival. A time-series raster job throws on objects that moved to ARCHIVE_ACCESS. The transition window was shorter than the analytics retention SLA. Validate transitions against real access logs before applying, and exclude rasters/ from any archive rule.
  • KMS throttling on tile fanout. A pan-and-zoom storm produces ThrottlingException from KMS because each tile read decrypts independently. Confirm bucket_key_enabled = true so a single bucket-level data key serves the high-fanout reads.
  • CORS regression after redeploy. Browser map clients fail with opaque CORS errors after an apply overwrites a hand-added exception. Move the CORS rule into the module and verify it in CI, mirroring the approach in CORS & CSP Configuration.