Abstracting Object Storage Across AWS and GCP for Tiles

You need one reusable module that provisions either an S3 bucket on AWS or a GCS bucket on GCP to serve raster and vector tiles, so a customer tenant can be placed on either cloud without rewriting the tile-serving layer or breaking the URL template shipped to browsers. The hard part is not creating two buckets — it is holding a single variable contract (versioning, lifecycle, CORS, public-read-via-CDN) so that the /{z}/{x}/{y} fetch path is byte-for-byte identical no matter which backend answers. This guide implements the storage-tier contract from Multi-Cloud Abstraction Patterns for Spatial Platforms and sits within Spatial IaC Architecture & Fundamentals; it walks the module end to end and verifies a live tile fetch from each cloud.

Symptom Identification and Triage

The failure that sends people here is subtle: the module “works” on both clouds, but the tile-serving path behaves differently between them. Classify the divergence before editing the module, because the fix differs by symptom.

• Tiles 200 on AWS, 403 on GCP (or vice versa): the public-read posture is expressed for one cloud only. AWS needs a public-access-block plus CDN origin-access config; GCS needs uniform bucket-level access plus an IAM binding or a CDN backend bucket.
• CORS preflight passes on one backend, fails on the other: the CORS block was set on the S3 side but not translated to the GCS cors block, or the allowed headers differ, so a Mapbox GL client works against one tenant and throws on the other.
• Same tile, different content type: one backend returns application/octet-stream for a .pbf vector tile because its content-type inference differs, breaking the client decoder while the raster .png path looks fine.
• Cost divergence with identical inputs: the lifecycle rule transitioned cold pyramids on both, but GCS NEARLINE and S3 STANDARD_IA have different minimum-retention floors, so an aggressive transition age costs more on one cloud.

Prerequisites and Environment Assumptions

This walkthrough assumes a single Terraform configuration that can target either cloud from one variable and a CDN in front of each bucket so the browser never sees the raw storage host.

• Terraform >= 1.10.0 with both providers installed and pinned. Unpinned providers are the most common cause of a contract behaving differently between the two implementations.

  terraform {
    required_version = ">= 1.10.0"
    required_providers {
      aws    = { source = "hashicorp/aws",    version = "~> 5.60" }
      google = { source = "hashicorp/google", version = "~> 6.8" }
    }
  }

• A locking-capable state backend per target, so a plan against the AWS tenant never races a plan against the GCP tenant.

• IAM/credentials for whichever cloud the instance targets: on AWS, permission to manage the bucket, its public-access block, versioning, lifecycle, and a CloudFront origin-access control; on GCP, permission to manage the bucket, IAM bindings, and a Cloud CDN backend bucket.

• A stable tile path convention already agreed with clients: /{z}/{x}/{y}.png for raster and /{z}/{x}/{y}.pbf for vector, served from a CDN hostname you control — never the storage host directly.

Step-by-Step Remediation

Build the contract once, then let a cloud selector pick the implementation. Provision, then verify a real tile from each backend.

1. Define the single variable contract. One set of inputs drives both clouds. The tile path shape is not a variable — it is a fixed convention — so nothing here can make the URL diverge.

   variable "cloud" {
     type = string
     validation {
       condition     = contains(["aws", "gcp"], var.cloud)
       error_message = "cloud must be aws or gcp."
     }
   }
   variable "name"                { type = string }
   variable "versioning_enabled"  { type = bool,  default = true }
   variable "cold_transition_days"{ type = number, default = 30 }
   variable "cors_allowed_origins"{ type = list(string) }

   The cold_transition_days default of 30 is deliberate: it clears both the S3 STANDARD_IA and GCS NEARLINE minimum-retention floors, so the same value is safe on either cloud. This is the geospatial reason parity matters — tile pyramids go cold in deep zoom levels, and a transition age that is cheap on one cloud can incur early-deletion charges on the other.

2. Implement the AWS backend with CDN-only public read. The bucket stays private; CloudFront’s origin-access control is the only reader, so the tile URL is the CDN host, not the S3 host.

   resource "aws_s3_bucket" "tiles" {
     count  = var.cloud == "aws" ? 1 : 0
     bucket = "tiles-${var.name}"
   }
   resource "aws_s3_bucket_public_access_block" "tiles" {
     count                   = var.cloud == "aws" ? 1 : 0
     bucket                  = aws_s3_bucket.tiles[0].id
     block_public_acls       = true
     block_public_policy     = true
     ignore_public_acls      = true
     restrict_public_buckets = true
   }
   resource "aws_s3_bucket_versioning" "tiles" {
     count  = var.cloud == "aws" ? 1 : 0
     bucket = aws_s3_bucket.tiles[0].id
     versioning_configuration { status = var.versioning_enabled ? "Enabled" : "Suspended" }
   }
   resource "aws_s3_bucket_lifecycle_configuration" "tiles" {
     count  = var.cloud == "aws" ? 1 : 0
     bucket = aws_s3_bucket.tiles[0].id
     rule {
       id     = "cold-pyramids"
       status = "Enabled"
       transition {
         days          = var.cold_transition_days
         storage_class = "STANDARD_IA"
       }
     }
   }
   resource "aws_s3_bucket_cors_configuration" "tiles" {
     count  = var.cloud == "aws" ? 1 : 0
     bucket = aws_s3_bucket.tiles[0].id
     cors_rule {
       allowed_methods = ["GET", "HEAD"]
       allowed_origins = var.cors_allowed_origins
       allowed_headers = ["*"]
       max_age_seconds = 3600
     }
   }

3. Implement the GCP backend with the same posture. Uniform bucket-level access plus a Cloud CDN backend bucket give the identical CDN-only public-read behavior; the cors block mirrors the S3 rule field-for-field.

   resource "google_storage_bucket" "tiles" {
     count                       = var.cloud == "gcp" ? 1 : 0
     name                        = "tiles-${var.name}"
     location                    = "US"
     uniform_bucket_level_access = true
     versioning { enabled = var.versioning_enabled }
     lifecycle_rule {
       condition { age = var.cold_transition_days }
       action {
         type          = "SetStorageClass"
         storage_class = "NEARLINE"
       }
     }
     cors {
       origin          = var.cors_allowed_origins
       method          = ["GET", "HEAD"]
       response_header = ["Content-Type", "Cache-Control"]
       max_age_seconds = 3600
     }
   }

4. Emit one portable output and front it with a CDN. The module returns a single origin host; the CDN in front rewrites it to your stable tile hostname, so the /{z}/{x}/{y} template shipped to browsers never contains a provider host.

   output "tile_origin_host" {
     value = var.cloud == "aws" ?
       aws_s3_bucket.tiles[0].bucket_regional_domain_name :
       "storage.googleapis.com/${google_storage_bucket.tiles[0].name}"
   }

Verification

Confirm portability by fetching the same tile from each backend and asserting identical observable behavior.

1. Fetch a raster tile from each backend through its CDN and assert a 200 with the correct content type:

   # AWS-backed tenant
   curl -sI "https://tiles.example.com/10/301/384.png" | grep -Ei "HTTP/|content-type"
   # GCP-backed tenant (same path, different backend)
   curl -sI "https://tiles-gcp.example.com/10/301/384.png" | grep -Ei "HTTP/|content-type"

   Both must return HTTP/2 200 and content-type: image/png. A 403 on one side means step 2 or 3 left that cloud’s public-read posture incomplete.

2. Verify a vector tile content type on both, since .pbf inference is the most common divergence:

   curl -sI "https://tiles.example.com/10/301/384.pbf" | grep -i content-type

   Expect application/x-protobuf (or your agreed vector MIME) on both backends; a mismatch means the upload step, not the module, is setting content type inconsistently.

3. Confirm the CORS contract matches by replaying a preflight against each host and diffing the access-control-allow-origin and access-control-allow-methods headers — they must be identical.

4. Prove the URL is provider-agnostic by checking that neither the S3 nor the GCS host string appears in the client-facing tile template; it should only ever contain your CDN hostname.

Preventing Recurrence

• Contract conformance test in CI. Add a job that provisions the module with cloud=aws and cloud=gcp against ephemeral names, uploads a fixture tile to each, and asserts identical status, content type, and CORS headers. A divergence fails the build before merge.
• Policy-as-code on effective posture. A rule asserts CDN-only public read on both — public-access-block present on S3, uniform bucket-level access on GCS — rather than checking for one cloud’s specific resource.
• Lifecycle floor guard. Validate cold_transition_days >= 30 so no overlay sets a transition age that triggers early-deletion charges on either class. Pair this with Configuring S3 Lifecycle Rules for GIS Tiles for the AWS-side detail on tiering cold pyramids.
• Stable-path lint. A check greps client tile templates for any raw storage host and fails if one leaks, keeping the URL contract intact through migrations.

Frequently Asked Questions

Related

• Multi-Cloud Abstraction Patterns for Spatial Platforms — the parent contract this module implements.
• Spatial IaC Architecture & Fundamentals — the architectural context for portable spatial infrastructure.
• Object Storage for Raster & Vector — provisioning the storage tier this contract abstracts.
• Configuring S3 Lifecycle Rules for GIS Tiles — the AWS-side detail on tiering cold tile pyramids.