Spatial IaC Terraform & Pulumi for GIS

Requester Pays Buckets for Public Raster Archives

Publishing an open raster archive has an asymmetric economics problem. Storage of a few terabytes of Cloud Optimized GeoTIFFs is a predictable monthly cost you can budget. Egress is not: one research group running a continental analysis can transfer more data in a weekend than the archive costs to store for a year, and a mirroring bot can do it repeatedly. Requester-pays inverts that, billing data transfer and request charges to the account making the request while you continue to pay only for storage — which is what makes an archive publishable at all. It also changes how every client must call the bucket, and that is where the operational surprises live. This guide extends Object Storage for Raster and Vector Data within Geospatial Resource Provisioning.

What changes when requester-pays is on

Three things change, and each breaks a category of client until it is handled.

Anonymous access stops working entirely. Requester-pays requires an authenticated principal, because there must be an account to bill. A bucket that was publicly readable becomes unreadable to anyone without credentials, which is a deliberate trade and needs to be stated prominently in the archive’s documentation — it is the single most common support question a published archive receives.

Every request must carry an acknowledgement. A caller has to explicitly signal that they accept the charges: --request-payer requester on the CLI, RequestPayer='requester' in an SDK call, AWS_REQUEST_PAYER=requester for GDAL’s /vsis3/ driver. Without it the request fails with a 403 that says nothing about payment, so users conclude they lack permission. Documenting the exact incantation for GDAL, rasterio and the CLI saves more support time than anything else you can write.

Your own internal access changes too. Pipelines in your own account reading the archive now need the header as well, and a batch job that has been reading the bucket for a year will start failing at the moment the setting is enabled. Enable it in a non-production copy first and update your own consumers before flipping production.

Three ways a request to a requester-pays bucket resolves An anonymous request is refused outright, because requester-pays needs an authenticated principal to bill. An authenticated request that omits the requester-payer acknowledgement is also refused, with a 403 whose message does not mention payment at all, which is why users interpret it as a permissions problem. An authenticated request that carries the acknowledgement succeeds: data transfer and request charges are billed to the caller's account while the bucket owner continues to pay only for storage. Anonymous no credentials Refused no account to bill Authenticated no acknowledgement 403 — and it never says why users read it as a permissions error Authenticated request-payer: requester Served caller billed for transfer Owner still pays storage replication lifecycle transitions and the catalogue

Prerequisites and environment assumptions

Terraform 1.6 or later with hashicorp/aws pinned at ~> 5.60. An archive already laid out with a stable prefix scheme, because requester-pays makes the URL contract effectively permanent — external users will embed these paths in scripts and papers, and a reorganisation after publication breaks work you cannot see. An inventory of your own internal consumers of the bucket, since each will need the acknowledgement added.

Decide what stays free. A small metadata layer — a STAC catalogue, an index of scene footprints, a checksums file — is worth leaving outside requester-pays in a separate small bucket, because it is tiny, it is what makes the archive discoverable, and forcing authentication on discovery is what makes an open archive feel closed. The pattern is: free discovery, paid bulk.

Step-by-step implementation

  1. Split discovery from bulk. Put the catalogue and footprint index in a small public bucket with no requester-pays, and the imagery in the requester-pays bucket. Users can then browse and query freely and pay only when they pull pixels.

  2. Enable requester-pays and grant authenticated read. The bucket policy grants s3:GetObject and s3:ListBucket broadly to authenticated principals — the payment condition is the access control, not the policy.

  3. Publish the exact client incantations. Documentation must include the GDAL environment variable, the rasterio session argument and the CLI flag, because the failure mode without them is a misleading 403. This single documentation page prevents most of the archive’s support load.

  4. Update your own consumers before enabling it. Batch jobs, tile pipelines and analysis notebooks in your own accounts all need the acknowledgement. Enabling requester-pays without this step breaks your own platform first.

  5. Keep the storage class matched to the access pattern. Requester-pays shifts transfer cost but not retrieval charges from archival classes, and an archival class with a per-gigabyte retrieval fee produces a surprising bill for a downloader who expected only transfer. If the archive is meant to be used, keep it in a standard or infrequent-access class and say which in the documentation.

terraform {
  required_version = ">= 1.6.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.60"
    }
  }
}

resource "aws_s3_bucket" "archive" {
  bucket = "open-raster-archive"
  lifecycle { prevent_destroy = true }
}

resource "aws_s3_bucket_request_payment_configuration" "archive" {
  bucket = aws_s3_bucket.archive.id
  payer  = "Requester"
}

data "aws_iam_policy_document" "archive" {
  statement {
    sid    = "AuthenticatedRead"
    effect = "Allow"
    # Any authenticated principal. The payment requirement — not this policy —
    # is what bounds who actually pulls terabytes.
    principals {
      type        = "AWS"
      identifiers = ["*"]
    }
    actions   = ["s3:GetObject", "s3:ListBucket"]
    resources = [aws_s3_bucket.archive.arn, "${aws_s3_bucket.archive.arn}/*"]
    condition {
      test     = "Bool"
      variable = "aws:SecureTransport"
      values   = ["true"]
    }
  }
}

resource "aws_s3_bucket_policy" "archive" {
  bucket = aws_s3_bucket.archive.id
  policy = data.aws_iam_policy_document.archive.json
}

# Discovery stays free: a small catalogue bucket with no requester-pays, so
# browsing and searching the archive never requires an account.
resource "aws_s3_bucket" "catalog" {
  bucket = "open-raster-archive-catalog"
}

resource "aws_s3_bucket_request_payment_configuration" "catalog" {
  bucket = aws_s3_bucket.catalog.id
  payer  = "BucketOwner"
}
# The three incantations to publish. Without them every call returns a 403
# whose message says nothing about payment.

# GDAL and anything built on it (rio, gdalinfo, gdal_translate):
export AWS_REQUEST_PAYER=requester
gdalinfo /vsis3/open-raster-archive/cog/2026/s2_32633_0412.tif

# AWS CLI:
aws s3 cp s3://open-raster-archive/cog/2026/s2_32633_0412.tif . \
  --request-payer requester

# rasterio / boto3:
#   import rasterio
#   from rasterio.session import AWSSession
#   with rasterio.Env(AWSSession(requester_pays=True)):
#       with rasterio.open("s3://open-raster-archive/cog/...") as src:
#           profile = src.profile
Free discovery, paid bulk The archive is published as two buckets with different payment configurations. A small catalogue bucket, paid for by the owner and readable anonymously, holds the STAC index, the scene footprint index and the checksum manifests — everything needed to find out what exists and decide what to download. The imagery bucket, configured as requester-pays, holds the Cloud Optimized GeoTIFFs themselves. A user can therefore search the entire archive and inspect its metadata without an account, and pays only at the moment they pull pixels. Researcher no account yet Catalogue bucket — free STAC index · footprints · checksums anonymous read, owner pays tiny, and it is what makes the archive findable decide what to fetch Authenticate + acknowledge request-payer: requester Imagery bucket — requester pays Cloud Optimized GeoTIFFs Free discovery, paid bulk: the archive stays open, the egress stays bounded.

Verification

Attempt an anonymous GET and confirm it fails — that is the setting working, not a misconfiguration. Attempt an authenticated GET without the acknowledgement and confirm the 403, so you know exactly what your users will see and can quote it in the documentation. Then perform the three documented calls — CLI, GDAL and rasterio — and confirm each succeeds, because the point of publishing the incantations is that they are correct, and an untested snippet in an archive’s documentation generates support tickets for years.

Confirm the billing behaves as expected by checking, after a day of external access, that data-transfer charges on your account have not moved while storage has. Requester-pays configured but ineffective — because a legacy policy grants anonymous access that bypasses it — looks exactly like requester-pays working until the bill arrives.

The billing signal that proves the setting is effective After a day of external access, two lines on the owner's bill tell you whether requester-pays is genuinely in force. Storage cost continues as normal, because the owner always pays for storage. Data-transfer cost should stay flat, because every byte leaving the bucket is billed to the caller. A non-zero and rising owner-paid transfer figure means something is bypassing the setting — most often a legacy anonymous grant left in the bucket policy that satisfies the request before the payment requirement is ever considered. Requester-pays configured but ineffective looks exactly like requester-pays working, right up until the bill arrives. Storage cost continues as before the owner always pays for storage this line is expected to move Owner-paid transfer stays flat while downloads happen rising means something bypasses it usually a legacy anonymous grant Configured but ineffective looks exactly like working — until the bill arrives.

Preventing recurrence

  • Treat the prefix scheme as a published API. External scripts and papers cite these paths. Version the layout if it must change, and keep the old paths resolving.
  • Alarm on your own egress from the archive bucket. A non-zero owner-paid transfer figure means something is bypassing requester-pays, and the sooner you know the smaller the surprise.
  • Keep the catalogue in sync automatically. A catalogue that drifts from the imagery is worse than none, because users pay to download objects that do not exist or miss ones that do.
  • Document the storage class and any retrieval fee. A user who is billed a retrieval charge they were not warned about will not use the archive again, and the warning costs one sentence.

Frequently Asked Questions

Can I keep anonymous access for small files and requester-pays for large ones?

Not within one bucket — the setting is bucket-wide. Two buckets is the supported pattern, and it maps neatly onto the useful split: a free catalogue bucket for metadata, a requester-pays bucket for imagery.

Why do users report a 403 that mentions permissions, not payment?

Because that is what the service returns when the acknowledgement header is missing. The error is genuinely unhelpful, which is why publishing the exact CLI flag, GDAL variable and SDK argument in the archive documentation is the highest-value page you will write.

Does requester-pays work with the GDAL virtual file system?

Yes, through the AWS_REQUEST_PAYER=requester environment variable, which applies to /vsis3/ and therefore to gdalinfo, gdal_translate, rasterio and anything else built on GDAL. This is the variable most users have never heard of, so document it first.

What happens to my own pipelines when I enable it?

They break, immediately, unless they were updated first. Every internal consumer needs the acknowledgement added. Enable requester-pays on a copy, fix your consumers against it, and only then change production.