Spatial IaC Terraform & Pulumi for GIS

Object Storage for Raster/Vector Workloads in Spatial IaC

Object storage is the persistence layer that most often drifts out of sync between environments on a spatial platform, because buckets accumulate ad-hoc lifecycle rules, CORS exceptions, and IAM grants long after the original deployment. Treating a bucket as an undifferentiated data dump is what breaks tile caches, leaks credentials, and inflates egress bills. This guide covers how to provision raster, vector, and tile storage as codified infrastructure within the broader Geospatial Resource Provisioning discipline, so that the hot path feeding PostGIS Cluster Provisioning and the compute fleet defined in Compute Node Orchestration consume exactly the assets they are authorized to, in every region, from the same commit.

Codified Object Store Feeding Three Spatial Consumers by Prefix An IaC state backend provisions the bucket configuration and a KMS customer-managed key supplies server-side encryption; both attach off to the side. The bucket is encrypted, versioned, and blocks public access, and exposes three prefixes as an interface contract. The rasters/ prefix holds source Cloud-Optimized GeoTIFFs read by GDAL compute workers over hot byte-range requests. The vectors/ prefix holds GeoParquet and GeoJSON extracts that hydrate a PostGIS cluster through foreign data wrappers or COPY loaders. The tiles/ prefix holds rendered tile pyramids served by a CDN-fronted GeoServer cache and drives the dominant egress and request load. A lifecycle and intelligent-tiering rule transitions only cold tile objects down to an archive tier, leaving rasters and vectors on the hot path. IaC state backend locked · provisions config KMS customer key SSE-KMS · bucket key Object store bucket encrypted · versioned · no public access rasters/ — source COGs hot byte-range reads vectors/ — GeoParquet · GeoJSON FDW / COPY loaders tiles/ — rendered pyramids dominant egress + requests GDAL compute workers stream rasters/ PostGIS cluster FDW / ETL hydrate GeoServer tile cache CDN-fronted lifecycle: cool → archive Archive tier intelligent tiering · cold tiles only

Environment Parity and Configuration Drift Mitigation

Environment parity is the primary guardrail for object storage, and it fails quietly. A staging bucket that lacks the production KMS key, omits a tiles/ lifecycle transition, or carries a looser public-access posture will pass functional tests and then corrupt behavior only under production load. Parity must be enforced structurally: bucket name, storage class defaults, versioning state, encryption key, public-access block, CORS rules, and event-notification subscriptions all belong in the same module, parameterized only by environment and region.

The recurring drift sources for spatial buckets are specific. First, CORS exceptions added by hand so a browser map client can fetch tiles — these never make it back into code and break the next apply. Second, lifecycle transitions edited in the console to relieve a cost spike, which then archive objects that a time-series raster job still reads. Third, prefix-scoped IAM widened to s3:* during an incident and never narrowed again. Lock all three in the module and treat the bucket configuration as immutable infrastructure: changes flow through a pull request or they do not happen. A scheduled terraform plan (or pulumi preview) on a cron, failing the build on any non-empty diff, converts silent drift into a visible alarm before it reaches a promotion cycle.

CI/CD Validation and Operational Guardrails

Object-storage changes deserve the same scrutiny as application code, with checks tuned to the geospatial failure modes. Pre-apply pipeline stages should:

  • Validate uploaded raster integrity by running gdalinfo against staged sample objects, confirming projection, band count, and overview presence before the asset becomes addressable to consumers.
  • Lint policies-as-code with Open Policy Agent (OPA) or HashiCorp Sentinel to reject any plan that disables encryption, drops the public-access block, or grants wildcard s3:* actions.
  • Diff lifecycle rules against the access-log-derived retention model, so a transition that would archive still-hot tiles fails review rather than silently shipping.
  • Verify CORS and event-notification subscriptions are present and scoped to the expected origins and prefixes, catching the most common manual-exception regressions.

These checks belong in pull-request workflows alongside the access-boundary rules described in IAM Role Mapping for GIS, and they pair with automated rollback triggers: if a post-apply synthetic read/write against the tiles/ prefix fails, the pipeline reverts the bucket policy to the last known-good state from the State Backend Selection backend.

Resource Architecture and Service Integration

Object storage is the data plane for nearly every other spatial resource, and its prefix layout is an interface contract, not an implementation detail. A disciplined three-prefix scheme — rasters/, vectors/, tiles/ — lets IAM, lifecycle, and event rules attach to access patterns rather than to individual objects:

  • rasters/ holds source imagery and Cloud-Optimized GeoTIFFs (COGs). Consumers stream byte ranges directly, so these objects stay on hot storage and are read by GDAL-based workers in the compute fleet and by raster ETL that hydrates spatial indexes.
  • vectors/ holds GeoParquet, GeoJSON, and FlatGeobuf extracts. These feed PostGIS Cluster Provisioning through foreign data wrappers or COPY-based loaders.
  • tiles/ holds rendered raster and vector tile pyramids served by GeoServer Deployment Patterns behind a CDN. This prefix is the dominant egress and request driver.

Because the bucket endpoint and IAM role names become inputs to those downstream stacks, expose them as typed module outputs and consume them via remote state or stack references — never hard-code an ARN. This keeps the dependency graph explicit and lets a region or account move without editing every consumer.

Sizing the tile pyramid

Storage planning for the tiles/ prefix is driven by the pyramid geometry, not by guesswork. A full web-mercator pyramid rendered to maximum zoom Z contains a predictable object count, since each zoom level holds 4^z tiles:

$$ N_{\text{tiles}} = \sum_{z=0}^{Z} 4^{z} = \frac{4^{Z+1} - 1}{3} $$

Multiplying N_tiles by the mean encoded tile size yields the cache footprint; most platforms render densely only over populated areas, so apply a coverage fraction per level rather than assuming a full pyramid. Feeding this estimate into the Cost Estimation Frameworks pre-apply gate keeps a new high-zoom layer from quietly multiplying storage cost.

Runnable Configuration

The following Terraform provisions a hardened raster/vector/tile bucket with pinned providers, KMS encryption, versioning, an account-aligned public-access block, intelligent tiering for the tiles/ prefix, and prefix-scoped least-privilege access. Every geospatial-specific decision is annotated inline.

terraform {
  required_version = ">= 1.7.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.60"
    }
  }
}

resource "aws_s3_bucket" "spatial_assets" {
  bucket = "${var.environment}-spatial-assets-${var.region}"

  tags = {
    Workload    = "raster-vector-pipeline"
    DataClass   = "internal-spatial"
    Environment = var.environment
  }
}

# Versioning protects against partial tile-pyramid overwrites and
# lets a corrupted ETL load be rolled back to the prior object version.
resource "aws_s3_bucket_versioning" "spatial_assets" {
  bucket = aws_s3_bucket.spatial_assets.id
  versioning_configuration {
    status = "Enabled"
  }
}

# Customer-managed KMS, not SSE-S3: spatial imagery is often
# licensed/regulated, so the key boundary must be auditable.
resource "aws_s3_bucket_server_side_encryption_configuration" "spatial_assets" {
  bucket = aws_s3_bucket.spatial_assets.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = "aws:kms"
      kms_master_key_id = var.spatial_kms_key_id
    }
    bucket_key_enabled = true # cuts KMS request cost on high-fanout tile reads
  }
}

# Deny public access at the bucket level even if a stray ACL slips through.
resource "aws_s3_bucket_public_access_block" "spatial_assets" {
  bucket                  = aws_s3_bucket.spatial_assets.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

# Auto-tier only the tile cache: rendered tiles cool predictably,
# while source rasters and vectors stay on the hot path.
resource "aws_s3_bucket_intelligent_tiering_configuration" "tiles" {
  bucket = aws_s3_bucket.spatial_assets.id
  name   = "tiles-archive"
  filter {
    prefix = "tiles/"
  }
  tiering {
    access_tier = "ARCHIVE_ACCESS"
    days        = 90
  }
}

# Least-privilege read scoped to the three spatial prefixes only.
resource "aws_iam_policy" "dataset_prefix_scoped" {
  name        = "${var.environment}-spatial-prefix-access"
  description = "Least-privilege access scoped to rasters, vectors, and tiles"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = ["s3:GetObject", "s3:ListBucket"]
        Resource = [
          aws_s3_bucket.spatial_assets.arn,
          "${aws_s3_bucket.spatial_assets.arn}/rasters/*",
          "${aws_s3_bucket.spatial_assets.arn}/vectors/*",
          "${aws_s3_bucket.spatial_assets.arn}/tiles/*"
        ]
      }
    ]
  })
}

output "spatial_bucket_arn" {
  description = "Consumed by PostGIS, GeoServer, and compute stacks via remote state."
  value       = aws_s3_bucket.spatial_assets.arn
}
Fail-Closed Provisioning Sequence Under a State Lock A pull request carrying a storage change enters fail-closed pre-apply gates: gdalinfo validates projection, band count, and overview presence on staged objects, and OPA rejects any plan that disables KMS encryption, drops the public-access block, or grants wildcard s3 actions. If any gate fails the run halts with an audit record and no partial apply. On pass the state lock backed by S3 and DynamoDB is acquired and an ordered apply runs: create bucket, enable versioning, configure KMS encryption with a bucket key, set the public-access block, attach intelligent tiering scoped to the tiles prefix, and create prefix-scoped IAM. The lock is then released and typed outputs are published for downstream consumers to read. Pull request · storage change Pre-apply gates · fail-closed gdalinfo: projection · bands · overviews OPA: deny no-KMS · public · s3:* gate fails Halt + audit no partial apply acquire lock (S3+DynamoDB) state lock held · ordered apply create bucket versioning KMS encryption · bucket key public-access block intelligent tiering · tiles/ prefix-scoped IAM release lock Outputs published · consumers read

Guardrails Embedded in Configuration

The configuration above encodes four guardrails that exist precisely because they are the parts teams forget under pressure.

State locking. The bucket definition lives in a remote backend with mandatory locking (S3 + DynamoDB, or the Pulumi Service), aligned with State Backend Selection. Concurrent applies on the same state are what produce half-written lifecycle rules and orphaned policies; the lock makes that race impossible rather than merely unlikely.

Secret and key management. No access keys appear in the module. Compute and middleware assume the prefix-scoped role at runtime, and the encryption key is referenced by ARN (var.spatial_kms_key_id) so key rotation is a key-policy change, never an infrastructure rewrite.

Network isolation. Internal consumers reach the bucket exclusively through a VPC gateway endpoint, keeping raster and tile traffic off the public internet and removing egress as an exfiltration path. The endpoint policy itself should restrict the same three prefixes, so a compromised node cannot enumerate the whole account.

Prefix scope as a blast-radius control. Because IAM, lifecycle, and intelligent tiering all bind to rasters/, vectors/, and tiles/, a mistake is contained to one access class. Widening a grant to s3:* or to the bucket root defeats every other guardrail at once, which is exactly why the OPA gate rejects it.

Troubleshooting and Failure Modes

  • S3 prefix scope mismatch. A GeoServer node returns 403 AccessDenied on tiles while raster reads succeed. The role was scoped to rasters/* and vectors/* but the tiles/* ARN was dropped during a refactor. Diff the rendered policy against the three-prefix contract and restore the missing resource line.
  • VPC endpoint policy gap. Reads work from a developer laptop but time out from inside the VPC. The gateway endpoint exists but its policy omits s3:GetObject for the bucket, so in-VPC requests are silently denied while public DNS still resolves. Align the endpoint policy with the prefix-scoped IAM policy.
  • Premature lifecycle archival. A time-series raster job throws on objects that moved to ARCHIVE_ACCESS. The transition window was shorter than the analytics retention SLA. Validate transitions against real access logs before applying, and exclude rasters/ from any archive rule.
  • KMS throttling on tile fanout. A pan-and-zoom storm produces ThrottlingException from KMS because each tile read decrypts independently. Confirm bucket_key_enabled = true so a single bucket-level data key serves the high-fanout reads.
  • CORS regression after redeploy. Browser map clients fail with opaque CORS errors after an apply overwrites a hand-added exception. Move the CORS rule into the module and verify it in CI, mirroring the approach in CORS & CSP Configuration.

Storage class, and the object that is old but hot

The storage-class decision for spatial data is unusual because the correlation everyone relies on — old objects are cold objects — is false for exactly the assets that matter most. A tile pyramid generated eighteen months ago and read ten thousand times an hour is old by age and hot by access. Source imagery uploaded last week and never touched again is new by age and cold by access. A lifecycle policy that transitions on age alone will therefore move the wrong objects, and it will do so silently, because a transition produces no error and the symptom is a tile endpoint that has become inexplicably slow and expensive to serve.

The decision should follow access pattern, and access pattern is measurable rather than assumed. Object-level access logging or a storage analytics report tells you which prefixes are read and how often, and that measurement almost always contradicts at least one intuition the team held. The common findings are that a “cold” archive prefix is being scanned nightly by a job nobody remembered, and that a prefix believed to be hot has not been read in months because the product it fed was retired.

Three classes cover most spatial estates. Standard for anything on a serving path: tile pyramids, published Cloud Optimized GeoTIFFs, the catalogue. Infrequent access for source imagery that is reprocessed occasionally — the per-gigabyte storage saving is real and the retrieval charge is acceptable at the frequency these are actually read. Archive only for material with a retention obligation and no expectation of routine access, and only when someone has confirmed that no scheduled job touches it, because an archival retrieval on a job that runs quarterly turns a storage saving into a per-retrieval bill and a multi-hour delay.

Intelligent tiering deserves a mention as the option that removes the decision, at the cost of a small per-object monitoring charge. For an archive of large rasters that charge is negligible against the objects’ size and it is often the right default. For a tile pyramid of many millions of small objects it is not, because the per-object fee is charged regardless of size and the tiles are small — which is a good illustration of why the storage-class question has a different answer for raster and for tiles even within one platform.

Whatever the choice, scope the rule by prefix and assert that scope in the pipeline. A lifecycle rule with no prefix filter is a rule that will eventually reach a prefix it was never intended for, and the assertion that a transition excludes the serving prefixes is one line in the plan-contract layer described in Testing and Validation for Spatial IaC.

Prefix design is the interface

A bucket’s prefix scheme is not filing; it is the interface every consumer, lifecycle rule, access policy and cost report binds to. It is also the hardest thing to change later, because external scripts, published documentation and IAM conditions all encode it. Designing it deliberately at the start costs an hour, and redesigning it after publication costs a migration nobody wants to run.

Four properties make a scheme durable. It should be stable under growth, so that adding a new sensor, region or product does not require reorganising what already exists — which usually means the most stable dimension comes first and the most volatile last. It should be prunable by prefix, so that a lifecycle rule can address exactly one class of object without a filter that will silently stop matching. It should be scopable in a policy, so a grant can be written for one tenant, one product or one partner without wildcards that reach further than intended. And it should be attributable in a cost report, which in practice means the prefix boundaries line up with the tag boundaries described in FinOps Tagging Strategies for Geospatial Resources.

The scheme that satisfies all four for most spatial estates puts lifecycle class first, then product, then time, then space: raw/, cog/, tiles/, derived/ at the top, each carrying a product identifier, then a date partition, then a coarse spatial key. Putting lifecycle class first is the decision that repays itself, because it is what lets an archival rule address source imagery without any possibility of touching the tile pyramid a renderer reads continuously.

Why lifecycle class comes first in the prefix The prefix scheme reads lifecycle class, then product, then date, then a coarse spatial key. Lifecycle class occupies the first position because it is the boundary that retention rules, access grants and cost reports all need to address independently. With it first, an archival transition can be scoped to source imagery with no possibility of reaching the tile pyramid a renderer reads continuously, and a partner grant can be written for one product's published output without a wildcard. With lifecycle class deeper in the path, every one of those becomes a pattern that matches across classes, and a rule intended for cold source data eventually reaches hot derived data. lifecycle class raw · cog · tiles · derived product sentinel2 · orthophoto date year= / month= coarse spatial key utm zone or low-zoom quadkey With class first, a retention rule addresses exactly one class and a partner grant addresses exactly one product's published output With class deeper, every one of those becomes a wildcard and a rule written for cold source data eventually reaches hot derived data

Two anti-patterns are worth naming because both look organised. Putting the date first — 2026/07/sentinel2/cog/... — reads naturally and makes it impossible to write a lifecycle rule for one product or one class without enumerating every date. Putting the tenant or customer first in a multi-tenant archive achieves clean isolation and makes every cross-tenant operation, including the cost report and the compaction job, a full-bucket scan. Neither is fatal, and both are the kind of decision that is cheap now and expensive in three years.

The versioning question follows directly. A published prefix is an interface, so a change to it is a breaking change and should be treated as one: publish the new layout alongside the old, keep the old paths resolving for a stated period, and announce the retirement rather than discovering it through a support request from someone whose script stopped working. Where the archive is external-facing, as in Requester Pays Buckets for Public Raster Archives, the paths may be cited in published papers and the retirement window should be measured in years rather than weeks.

Finally, encode the scheme rather than documenting it. A module that derives object keys from typed inputs — class, product, date, spatial key — makes an inconsistent path impossible to write, and it gives the lifecycle rules, the policies and the cost tags a single definition to share. A convention that lives only in a wiki page is one that will be followed by everyone who read it and by nobody who joined afterwards.