Spatial IaC Terraform & Pulumi for GIS

Troubleshooting Terraform VPC Peering for Distributed GeoServer Deployments

Your distributed GeoServer cluster was healthy yesterday, and today half the WMS nodes return 502 on tile requests while JDBC pools to a central PostGIS database time out on port 5432 — yet nothing in the application changed. This is the signature of a VPC peering link that has drifted from its declared Terraform state: a missing reciprocal route, a peering connection stuck in pending-acceptance, or an overlapping CIDR that silently blackholes inter-node traffic. This guide is the incident-response companion to VPC Routing for Tile Servers and sits within the broader Network Security & Access Control framework; it covers how to isolate a peering-layer failure, reconcile Terraform state without tearing down live routes, and rebuild a symmetric, DNS-resolving peering configuration that keeps GeoServer reaching its data tier over private paths.

Symptom identification and triage

Peering failures in distributed GIS environments almost never surface as application stack traces — they surface as routing anomalies that masquerade as application bugs. Before editing any route or peering resource, classify the failure against concrete, observable signals:

  • Connection timeouts on 8080 or 5432, no RST: a GeoServer node cannot reach a sibling node or the central PostGIS instance, and the TCP SYN simply disappears. This is asymmetric routing or a missing route-table entry on one side of the peering boundary — confirm with VPC Flow Logs showing a SYN egressing the requester VPC but no return packet from the accepter.
  • Load balancer 502/504 on tile requests: the renderer resolved but could not complete a backend fetch. When this correlates with cross-VPC service names failing to resolve, the private hosted zone is associated with only one VPC — a DNS problem, not a routing one.
  • terraform apply errors PeeringConnectionHasWrongState or InvalidVpcPeeringConnectionID.NotFound: the connection is stuck in pending-acceptance (cross-account acceptance never happened) or was deleted out of band, so state references a connection that no longer exists.
  • RouteAlreadyExists or InvalidRouteTableId.NotFound: overlapping CIDR allocations between the peered VPCs, or a route table managed in the console that Terraform no longer recognises. Overlapping CIDRs cannot be peered at all and must be re-allocated before any route will install.

Establishing this baseline keeps diagnostics from mutating infrastructure prematurely: a 502 driven by CORS and CSP Configuration or an ingress gap from Security Group Hardening looks identical at the edge but has nothing to do with peering. Route the failure to the right layer before touching the peering connection.

Triage Tree: Routing a GeoServer 502 / PostGIS Timeout to Its Peering-Layer Cause The symptom — a 502 on tile requests or a JDBC timeout on port 5432 — branches on a single diagnostic signal into four distinct causes. If VPC Flow Logs show an outbound SYN with no return packet, the cause is a missing reciprocal route (asymmetric routing) on one side of the peering boundary. If cross-VPC service names fail to resolve, the cause is a private hosted zone associated with only one VPC — a DNS problem, not a routing one. If terraform apply errors with PeeringConnectionHasWrongState or InvalidVpcPeeringConnectionID.NotFound, the cause is peering state drift: a connection stuck in pending-acceptance or deleted out of band. If apply errors with RouteAlreadyExists, the cause is an overlapping CIDR allocation or a route managed in the console that Terraform no longer recognises. Symptom: 502 on tiles / JDBC timeout on 5432 Flow Logs: SYN out, no return packet no RST seen Cross-VPC name fails to resolve route exists, DNS does not apply: WrongState / ...ID.NotFound connection unresolved apply: RouteAlreadyExists install refused Missing reciprocal route asymmetric routing — add route on failing side Hosted zone on one VPC only DNS, not routing — associate zone + remote DNS Peering state drift pending-acceptance or deleted out of band — accept / re-import Overlapping CIDR / console route re-allocate CIDR or terraform import the route Classify on the observable signal before mutating any peering or route resource.

Prerequisites and environment assumptions

This guide assumes an AWS target running distributed GeoServer nodes across two or more VPCs that must reach a shared PostGIS data tier and synchronise tile caches over private addressing. To execute the remediation you will need:

  • Terraform >= 1.6 with the AWS provider pinned. Unpinned providers are a frequent cause of phantom peering and route diffs, so pin the provider in required_providers:

    terraform {
  required_version = ">= 1.6.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.60"
    }
  }
}

  • A locking-capable state backend (S3 + DynamoDB, or Terraform Cloud) already configured. Concurrent apply runs are the same hazard described in Managing Terraform State Locks for Spatial Data — never run two applies against one peering state without a lock.

  • Non-overlapping CIDR blocks across every VPC in the mesh. AWS rejects a peering connection between VPCs with overlapping ranges, so verify allocations before troubleshooting routes.

  • IAM permissions for the operator: ec2:Describe*, ec2:CreateVpcPeeringConnection, ec2:AcceptVpcPeeringConnection, ec2:ModifyVpcPeeringConnectionOptions, and ec2:CreateRoute/ec2:ReplaceRoute. Cross-account acceptance also requires the accepter-account principal to hold ec2:AcceptVpcPeeringConnection, scoped through IAM Role Mapping for GIS.

Step-by-step remediation

Reconcile state first, then repair routing. Applying new route declarations on top of drifted state risks a destructive recreation that blackholes every GeoServer-to-PostGIS connection at once.

  1. Reconcile state before changing anything. Confirm the peering connection, route tables, and routes are mapped to real cloud identifiers, then run a refresh-only plan to surface drift without writing it:

    terraform state list | grep -E 'peering|route'
terraform plan -refresh-only

    A refresh-only plan reveals manually provisioned routes that a normal apply would overwrite — review it before proceeding, because a blind apply -refresh-only can clobber a route someone added under incident pressure.

  2. Accept a stuck cross-account connection. If the connection sits in pending-acceptance, auto_accept cannot resolve it across accounts. Validate the accepter principal’s permissions, then accept explicitly before re-running provisioning:

    aws ec2 accept-vpc-peering-connection \
  --vpc-peering-connection-id pcx-0abc123def456 \
  --region eu-west-1

  3. Re-align state for an existing route instead of recreating it. When Terraform reports a missing route that actually exists in the console, import it rather than letting apply destroy and recreate — recreation transiently drops the path:

    terraform import aws_route.peering_requester \
  rtb-0aaa111bbb222ccc_10.20.0.0/16

  4. Declare symmetric routes and cross-VPC DNS in code. Rebuild the peering with reciprocal routes on both sides and remote DNS resolution enabled, so GeoServer resolves the PostGIS private endpoint instead of paying for recursive cross-VPC lookups. Each side must hold a route to the opposite VPC’s CIDR — a missing entry on either route table blackholes the connection. The configuration below targets this exact scenario for a same-account mesh:

    terraform {
  required_version = ">= 1.6.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.60"
    }
  }
}

# Same-account peering: auto_accept resolves the connection in one apply.
resource "aws_vpc_peering_connection" "geoserver_peering" {
  vpc_id      = var.requester_vpc_id
  peer_vpc_id = var.accepter_vpc_id
  auto_accept = true # valid only when both VPCs share one account

  tags = {
    Name      = "geoserver-cluster-peering"
    ManagedBy = "terraform"
  }
}

# Enable name resolution across the link so GeoServer resolves PostGIS privately.
resource "aws_vpc_peering_connection_options" "geoserver_peering_opts" {
  vpc_peering_connection_id = aws_vpc_peering_connection.geoserver_peering.id
  requester { allow_remote_vpc_dns_resolution = true }
  accepter  { allow_remote_vpc_dns_resolution = true }
}

# Reciprocal routes — both sides are mandatory or traffic is blackholed.
resource "aws_route" "peering_requester" {
  route_table_id            = var.requester_route_table_id
  destination_cidr_block    = var.accepter_vpc_cidr
  vpc_peering_connection_id = aws_vpc_peering_connection.geoserver_peering.id
}

resource "aws_route" "peering_accepter" {
  route_table_id            = var.accepter_route_table_id
  destination_cidr_block    = var.requester_vpc_cidr
  vpc_peering_connection_id = aws_vpc_peering_connection.geoserver_peering.id
}

    For cross-account peering, drop auto_accept and accept the connection in the accepter account with a separate aws_vpc_peering_connection_accepter resource. This pattern aligns with established VPC Routing for Tile Servers practice and the Terraform AWS VPC peering resource documentation for version-specific attributes.

  5. Apply with a clean plan. Run terraform apply only after terraform plan shows in-place route additions and the peering connection moving to active — never when it proposes replacing the connection or a route.

The data path the reciprocal routes enable looks like this:

Symmetric Peering Routes Carrying the GeoServer-to-PostGIS Data Path The requester VPC contains the GeoServer rendering nodes and a route table whose entry points at the accepter VPC's CIDR. The accepter VPC contains the central PostGIS database and a route table whose entry points at the requester VPC's CIDR. A single VPC peering connection joins the two route tables, and traffic only flows when both reciprocal routes are present — a missing entry on either side blackholes the link. With both routes in place, GeoServer opens a JDBC connection to PostGIS on port 5432 over a private IP path that never traverses the public internet. Requester VPC GeoServer nodes WMS / WMTS renderers Route table → accepter CIDR Accepter VPC Central PostGIS shared data tier Route table → requester CIDR VPC peering connection 5432 over private IP Both reciprocal routes are mandatory — a missing entry on either side blackholes the link.

Verification

Confirm the fix at the routing, DNS, and application layers before closing the incident.

  1. Confirm the connection is active and DNS is enabled. Prove the peering link resolved and carries remote DNS resolution:

    aws ec2 describe-vpc-peering-connections \
  --vpc-peering-connection-ids pcx-0abc123def456 \
  --query 'VpcPeeringConnections[0].[Status.Code,AccepterVpcInfo.PeeringOptions]'

    The status code must read active and both peering-options blocks must show AllowDnsResolutionFromRemoteVpc: true.

  2. Verify reciprocal routes exist. Check that each route table holds a route to the opposite CIDR pointing at the peering connection — a missing route on either side is the most common cause of a one-way blackhole.

  3. Probe the real data path. From a GeoServer node, open a TCP connection to PostGIS over the private address (nc -zv <postgis-private-ip> 5432), then issue a live WMS GetMap through the load balancer and assert a 200 with image/png. Cross-VPC name resolution should return the PostGIS private IP, confirming the DNS option took effect.

Preventing recurrence

Encode the fix so the same drift cannot recur:

  • Policy-as-code gate. Add a Checkov or Conftest rule to the pipeline that fails any peering module missing a reciprocal aws_route for each VPC CIDR or missing allow_remote_vpc_dns_resolution. This blocks the two highest-frequency regressions — one-way routes and recursive cross-VPC DNS — before merge.
  • Scheduled drift detection. Run terraform plan -detailed-exitcode on a nightly schedule; a non-zero exit signals an out-of-band console route change and should page the owning team rather than wait for the next deploy.
  • State discipline. Keep locking enabled and never run concurrent applies against the peering state, the same way Managing Terraform State Locks for Spatial Data protects spatial state.
  • CIDR governance and identity alignment. Maintain a central non-overlapping CIDR allocation table for every GIS VPC, scope cross-account peering acceptance through IAM Role Mapping for GIS, and keep security-group ingress for 5432/8080 scoped to peer CIDRs per Security Group Hardening so reachability and identity enforce the same least-privilege posture.

Frequently Asked Questions

Why does my peering connection stay in pending-acceptance even with auto_accept = true?

Because auto_accept only works when both VPCs belong to the same AWS account. For cross-account peering the accepter account must accept the connection explicitly — through a separate aws_vpc_peering_connection_accepter resource, the aws ec2 accept-vpc-peering-connection CLI call, or the console — before any route will carry traffic.

GeoServer can reach PostGIS from one VPC but not the other — what is wrong?

This is a one-way blackhole caused by a missing reciprocal route. Peering only carries traffic when both route tables hold a route to the opposite VPC’s CIDR pointing at the peering connection. Add the missing aws_route on the side that fails and confirm with VPC Flow Logs that return packets now appear.

Can I peer two VPCs that share part of their CIDR range?

No. AWS rejects a peering connection between VPCs with overlapping CIDR blocks, and even partial overlap makes routing ambiguous. Re-allocate one VPC’s range from a central allocation table before attempting to peer.

terraform apply reports RouteAlreadyExists — should I delete the route and re-apply?

No. The route exists in the cloud but is absent from state, usually after a console edit. Import it with terraform import aws_route.<name> <route-table-id>_<destination-cidr> so state aligns without a destructive recreation that would transiently drop the path.