Spatial IaC Terraform & Pulumi for GIS

Migrating GIS Terraform State from Local to S3

A spatial platform almost always begins on a laptop. Someone provisions a PostGIS instance and a raster bucket to test an idea, it works, and eighteen months later that terraform.tfstate file — still on one machine, still unlocked, still the only record of which resources exist — is the authoritative description of production geometry storage. Migrating it to a locked remote backend is the single highest-value piece of infrastructure hygiene available to a GIS team, and it is also the operation with the most alarming failure mode, because a botched migration can leave the engine believing that a live PostGIS cluster does not exist. This guide is the operational companion to State Backend Selection within Spatial IaC Architecture and Fundamentals, and it walks the migration in an order designed so that every step before the point of no return is reversible.

Symptom identification and triage

You need this migration if any of the following is true, and the first two are the ones that turn into incidents.

  • The state file exists in exactly one place. No versioning, no backup, and the machine holding it is a workstation. A disk failure here does not destroy the infrastructure, but it destroys the only map of it, and rebuilding that map means importing every resource by hand.
  • Two people can apply at once. Without a lock, two concurrent applies against the same resources interleave writes and produce a state file that describes neither run. On a spatial estate this typically surfaces as a duplicate security group or an orphaned read replica that no configuration claims.
  • State is committed to version control. This is the variant that looks safe and is not: the file contains resource attributes including, for a database, connection endpoints and sometimes generated passwords, and it is now in every clone of the repository and in its entire history.
  • terraform plan proposes to create things that already exist. Someone ran an apply from a different copy of the state. The engine is not wrong; it genuinely does not know about those resources.

Before touching anything, establish what the state actually contains. terraform state list enumerates every managed resource, and for a spatial estate you should expect to recognise every entry — a cluster, its parameter group, its subnet group, the buckets, the security groups. An entry you cannot account for, or a resource you know exists that is missing from the list, is a discrepancy to resolve before migrating rather than during.

Migration order and the point of no return Five ordered steps. First, inventory the existing state and resolve any discrepancy between what it lists and what exists. Second, provision the state bucket and lock table using a separate configuration whose own state stays local, avoiding a chicken-and-egg problem. Third, take a timestamped backup of the local state file and store it away from the working directory. Fourth, add the backend block and run init with migrate-state, which is the point of no return because the engine rewrites the local file. Fifth, verify by running a plan that must report no changes. Everything before step four is reversible; after step four, recovery depends on the backup taken in step three. 1 · Inventory state list 2 · Provision bucket + lock table 3 · Back up timestamped copy 4 · init -migrate point of no return 5 · Verify plan: no changes reversible — nothing has been rewritten yet recovery depends on the step 3 backup Resolve any discrepancy between the inventory and reality in step 1 — never during step 4. A plan that proposes to CREATE an existing PostGIS cluster means the state did not arrive. Stop and restore.

Prerequisites and environment assumptions

Terraform 1.6 or later with the hashicorp/aws provider pinned at ~> 5.60; provider versions matter here because the backend block’s supported arguments have changed across releases and a mismatched example is a confusing first failure. An IAM principal that can create a bucket, a DynamoDB table and a key, and separately a principal for day-to-day use that can read and write only the state object and the lock item. Exclusive access to the estate for the duration — announce the migration and ensure nobody applies from another copy while it is in progress, because the one thing this procedure cannot survive is a concurrent write from the state file you are moving.

The bucket and lock table must be created by a separate configuration whose own state is small and can stay local or be committed deliberately. Trying to have one configuration create the backend it also stores itself in is the chicken-and-egg problem that produces half-migrated estates, and the workaround of applying it twice is more fragile than simply keeping a small bootstrap configuration.

Step-by-step migration

  1. Inventory and reconcile. Run terraform state list and terraform plan. The plan must be clean before you begin. If it proposes changes, decide deliberately whether to apply them first or to fix the configuration — migrating a state that already disagrees with reality means you cannot use “plan reports no changes” as your success signal afterwards, and that signal is the whole verification strategy.

  2. Provision the backend resources. A bucket with versioning enabled and a customer-managed key, plus a DynamoDB lock table. Versioning is not optional: it is what makes a bad state write recoverable, and it costs nothing at state-file volumes. Encryption follows the domain-key approach in Encryption and Key Management for Spatial Data, because a state file for a spatial estate contains endpoints, ARNs and occasionally generated credentials.

  3. Back up the local state. Copy terraform.tfstate to a timestamped file outside the working directory, and verify the copy parses as JSON and lists the resources you expect. This backup is the entire recovery plan for step four.

  4. Add the backend block and migrate. Add the backend "s3" block and run terraform init -migrate-state. The engine detects the local state, asks for confirmation, and copies it to the bucket. Answer yes only once you have completed step three.

  5. Verify, then remove the local file. Run terraform plan and require No changes. Only after that succeeds should the local terraform.tfstate and terraform.tfstate.backup be deleted from the working directory, and only then should the state file be removed from version control history if it was ever committed.

# bootstrap/main.tf — a separate, deliberately small configuration.
terraform {
  required_version = ">= 1.6.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.60"
    }
  }
}

resource "aws_s3_bucket" "state" {
  bucket = "gis-terraform-state-prod"
  # A state bucket must never be destroyed by a refactor of this configuration.
  lifecycle { prevent_destroy = true }
}

resource "aws_s3_bucket_versioning" "state" {
  bucket = aws_s3_bucket.state.id
  # Versioning is what makes a bad state write recoverable. It is the reason
  # this bucket is not just any bucket.
  versioning_configuration { status = "Enabled" }
}

resource "aws_s3_bucket_public_access_block" "state" {
  bucket                  = aws_s3_bucket.state.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_dynamodb_table" "locks" {
  name         = "gis-terraform-state-locks"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"
  attribute {
    name = "LockID"
    type = "S"
  }
  lifecycle { prevent_destroy = true }
}
# The backend block added to the real configuration in step 4.
terraform {
  backend "s3" {
    bucket         = "gis-terraform-state-prod"
    key            = "platform/postgis/terraform.tfstate"
    region         = "eu-west-1"
    dynamodb_table = "gis-terraform-state-locks"
    encrypt        = true
  }
}
How the remote backend serialises two concurrent applies After migration the working directory holds no state file. Each operation reads the state object from the versioned, encrypted bucket and takes a lock item in the DynamoDB table keyed by the state path. A second engineer running apply at the same moment attempts to take the same lock item, fails, and blocks with a message naming who holds it and since when, rather than writing over the first run's state. When the first operation finishes it writes a new object version and releases the lock. Engineer A terraform apply Engineer B blocks on the lock Lock table one item per state path State object versioned, encrypted B blocking with a named holder is the desired outcome — the alternative is two interleaved writes and a state describing neither.

Verification

terraform plan reporting No changes. Your infrastructure matches the configuration. is the primary signal and it is close to conclusive: it means every resource in the migrated state was found in the account with matching attributes. Confirm the state object exists in the bucket with a version identifier, and confirm the lock mechanism works by starting a plan in one terminal and a second in another — the second must block with a message naming the holder, not proceed.

Then verify the spatial estate itself is untouched, because a migration should change nothing about it: the PostGIS cluster still accepts a connection and SELECT postgis_full_version() still answers, the raster bucket still lists, and the tile endpoint still returns a tile. State migration does not touch running resources, so any change here indicates that something other than the migration happened during the window.

Five post-migration checks and what each one proves A plan reporting no changes proves every resource in the migrated state was found in the account with matching attributes, which is the closest thing to a conclusive signal. A state object carrying a version identifier proves the write reached the bucket rather than an error being swallowed. A second plan blocking with a named lock holder proves the locking mechanism is active rather than merely configured. A successful PostGIS connection and version query proves the migration disturbed nothing running, which it should not have. An absent local state file proves the next run in this directory cannot silently diverge from the remote copy. Check What it proves plan reports No changes every resource carried across state object has a version id the write reached the bucket a second plan blocks, naming the holder locking is active, not just configured postgis_full_version() answers nothing running was disturbed no local terraform.tfstate remains the next run cannot diverge

Preventing recurrence

  • Delete the local state and remove it from history. A leftover terraform.tfstate in a working directory is the seed of the next divergence, because someone will eventually run against it. If it was committed, purge it from repository history and rotate any credential it contained — it was readable by everyone with a clone.
  • Make the backend mandatory in the module template. New configurations should start from a template that already carries a backend block, so the local-state stage never happens again for the next component.
  • Enforce it in the pipeline. A policy rule that fails any configuration without a backend block, run in the gate described in Policy as Code for Spatial Resources, turns this from a convention into a constraint.
  • Add the state bucket to the backup and drill schedule. A versioned bucket is recoverable in principle; a team that has never restored a prior state version has not verified that in practice.

Frequently Asked Questions

Will migrating state change or restart my PostGIS cluster?

No. State migration copies a file describing resources; it does not call any API against the resources themselves. If a plan after migration proposes changes to the cluster, the cause is a configuration drift that predates the migration, not the migration.

What do I do if the post-migration plan proposes to create resources that already exist?

Stop and do not apply. That plan means the remote state is empty or partial — the migration did not carry everything across. Restore the backup taken in step three, re-run terraform init -migrate-state, and confirm terraform state list against the remote backend returns the same inventory as before.

Do I still need DynamoDB for locking?

Recent Terraform releases support S3-native conditional-write locking, which removes the separate table. Whichever mechanism your pinned version supports, the requirement is unchanged: some lock must exist, because the failure it prevents is concurrent writes to spatial infrastructure state. Verify which mechanism your version implements rather than copying a configuration written for a different one.

Should each environment have its own bucket or its own key?

Separate keys in one bucket per account is the common arrangement and is sufficient when IAM scopes access per key prefix. Separate buckets per environment give a cleaner blast radius and simpler policies at the cost of more bootstrap. The trade-off is developed in State Backend Selection.