Spatial IaC Terraform & Pulumi for GIS

Troubleshooting Blackholed Routes Between Tile-Server VPCs

A tile server that rendered fine an hour ago now hangs on every geometry fetch, its PostGIS connection pool draining to connection timed out with no RST in sight, and the only change was a peering or Transit Gateway edit someone applied to the shared network — that is the signature of a blackholed route, and this guide traces it from the blackhole state stamped on a route-table entry back to the missing return path or overlapping CIDR that caused it. It sits within VPC Routing for Tile Servers and the wider Network Security & Access Control framework, and it treats the fix as reproducible Terraform — symmetric static routes reviewed in a plan diff — rather than a console edit that will drift the moment the next attachment lands.

Symptom identification and triage

A blackholed route drops packets on the floor: the requester sends a SYN, nothing comes back, and the socket eventually times out. That silence is the tell that separates a routing blackhole from an application fault or a security-group deny, which usually fails faster or leaves a log line. Before you touch a route table, classify the failure against concrete, observable signals — the target ENI or gateway of a route has disappeared and AWS has flipped the route’s State to blackhole, or a return path is simply missing.

  • A route in state blackhole. aws ec2 describe-route-tables shows an entry whose State is blackhole — the destination CIDR still points at a NAT gateway, peering connection, or TGW attachment that was deleted or detached, so every packet to that CIDR is silently discarded. This is the unambiguous case and the fastest to confirm.
  • Timeout with no RST, one direction only. The tile server’s SYN to PostGIS on 5432 egresses its VPC — visible in VPC Flow Logs as an ACCEPT on the outbound record — but no return packet appears. The request route exists; the return route on the database VPC’s route table does not, so the reply is blackholed. This is asymmetric routing.
  • Works from one subnet, fails from another. Renderers in subnet A reach PostGIS while subnet B times out. The two subnets are associated with different route tables and only one carries the peering or TGW route to the data-tier CIDR.
  • Overlapping CIDRs after a merge. A newly attached VPC shares a CIDR range with the tile-server VPC, so the most-specific local route always wins and the cross-VPC route is never consulted — traffic to the overlap disappears with no error and no blackhole state, because from the local table’s view there is nothing wrong.

Triage has to separate routing from the layers that look identical at the socket. A connection timed out can equally be a security-group or NACL deny, so read the Flow Logs verdict before blaming a route: a Flow Log REJECT is a security-group or NACL problem, while an ACCEPT on egress with no return record is a routing blackhole. Only the latter is fixed in a route table; a REJECT routes to Security Group Hardening, and a browser-side failure with no server contact at all is a CORS and CSP Configuration problem, not a network one.

Triage tree for a blackholed route between tile-server VPCs The symptom — a PostGIS or tile fetch timing out with no RST after a peering or Transit Gateway change — leads to reading the VPC Flow Logs verdict. If the verdict is REJECT, the cause is a security-group or NACL deny, not a route. If the verdict is ACCEPT on egress with no matching return record, the cause is a routing blackhole, which splits three ways: a route whose State is blackhole because its target gateway or attachment was deleted; a missing reciprocal return route on the other VPC causing asymmetric routing; or an overlapping CIDR that makes the most-specific local route win so the cross-VPC route is never consulted. Timeout, no RST PostGIS 5432 / tile fetch Read VPC Flow Logs verdict ACCEPT egress vs REJECT REJECT SG / NACL deny not a route problem ACCEPT, no return Routing blackhole State = blackhole target deleted / attachment detached Missing return route asymmetric routing — reply blackholed Overlapping CIDR local route shadows the cross-VPC route Re-declare route to Add reciprocal route on both route tables Re-IP one VPC live target

Prerequisites and environment assumptions

This guide assumes a distributed tile-serving topology: a renderer fleet in one VPC reaching a central PostGIS catalog in another over VPC peering or a Transit Gateway, provisioned as code. To diagnose and fix a blackhole you need:

  • Terraform 1.6+ with the AWS provider pinned to ~> 5.40, so the route, peering, and TGW route-table-association resources render identical semantics across environments. Unpinned providers are a frequent cause of a route that plans differently than it applied.
  • VPC Flow Logs enabled on both VPCs (or at least the requester and accepter subnets), because the ACCEPT-egress-with-no-return signature is the fastest way to confirm asymmetric routing. Enable them ahead of the incident — you cannot retroactively capture a packet you did not log.
  • Access to VPC Reachability Analyzer, which evaluates the route tables, security groups, NACLs, and peering state along a path without sending a packet, and names the exact hop that drops traffic. IAM permissions: ec2:CreateNetworkInsightsPath, ec2:StartNetworkInsightsAnalysis, and ec2:Describe*.
  • A locking-capable remote backend. Route-table edits under incident pressure race badly; a backend with locking, per State Backend Selection, stops two responders leaving a half-written table live.

The cross-VPC topology and the peering-state failures adjacent to this one are covered in Terraform VPC Peering for Distributed GeoServer; this page focuses specifically on the blackhole and asymmetric-route cases once the peering connection itself is active.

Step-by-step remediation

Confirm the blackhole before mutating a route — re-declaring a route to a stale target just recreates the blackhole. Work from the observable state outward.

  1. Find the blackholed route. Enumerate the route tables and filter for the blackhole state, then note the destination CIDR and the dead target so you know exactly which entry to replace and which live target it must point at.

    aws ec2 describe-route-tables \
  --query 'RouteTables[].Routes[?State==`blackhole`].[DestinationCidrBlock,TransitGatewayId,VpcPeeringConnectionId,NatGatewayId]' \
  --output table

  2. Run Reachability Analyzer requester-to-accepter, then reverse it. Analyze the path from a renderer ENI to the PostGIS ENI, then run it in the opposite direction. A path that is reachable forward and not reachable backward pinpoints the missing return route — the asymmetric case the socket timeout hides.

    aws ec2 create-network-insights-path \
  --source eni-renderer --destination eni-postgis \
  --protocol tcp --destination-port 5432
# then start-network-insights-analysis on the returned path id; repeat with src/dst swapped

  3. Declare symmetric static routes on both route tables. The durable fix is reciprocal routes: the renderer VPC’s private route table needs a route to the database CIDR via the peering or TGW attachment, and the database VPC’s table needs the matching return route to the renderer CIDR. Static declaration keeps the data plane reproducible instead of leaning on BGP propagation, which can reintroduce a transient asymmetric path under load.

  4. Resolve any overlapping CIDR before expecting a route to install. If the analyzer or a RouteAlreadyExists-style conflict reveals an overlap, no route will carry the traffic — the local route always wins. Re-IP one VPC onto a non-overlapping range; there is no route-table fix for an overlap.

  5. Apply through a plan diff and verify the state flips off blackhole. Review the plan so the route delta is visible, apply with create_before_destroy on associations so active tile jobs are not dropped mid-swap, and re-query the route to confirm its State is now active.

The minimal, reviewable Terraform that installs the symmetric pair — the request route on the renderer side and the return route on the database side — is two aws_route resources across the two route tables:

terraform {
  required_version = ">= 1.6.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.40" # route + TGW-association semantics are version-sensitive
    }
  }
}

variable "renderer_rt_id"   { type = string } # renderer VPC private route table
variable "postgis_rt_id"    { type = string } # database VPC route table
variable "renderer_cidr"    { type = string } # e.g. 10.20.0.0/16 — must NOT overlap
variable "postgis_cidr"     { type = string } # e.g. 10.30.0.0/16
variable "tgw_id"           { type = string }

# Request path: renderer -> PostGIS, via the Transit Gateway.
resource "aws_route" "renderer_to_postgis" {
  route_table_id         = var.renderer_rt_id
  destination_cidr_block = var.postgis_cidr
  transit_gateway_id     = var.tgw_id
}

# Return path: PostGIS -> renderer. Missing this is the classic blackhole —
# the SYN arrives, the reply is dropped, and the socket times out with no RST.
resource "aws_route" "postgis_to_renderer" {
  route_table_id         = var.postgis_rt_id
  destination_cidr_block = var.renderer_cidr
  transit_gateway_id     = var.tgw_id
}

# Guardrail: fail the plan if the two CIDRs overlap — an overlap cannot be routed.
resource "null_resource" "cidr_non_overlap_gate" {
  lifecycle {
    precondition {
      condition     = cidrhost(var.renderer_cidr, 0) != cidrhost(var.postgis_cidr, 0)
      error_message = "renderer_cidr and postgis_cidr overlap; re-IP one VPC before routing."
    }
  }
}

Verification

Prove the path is symmetric and the state is clean, not just that one direction works.

  • No route remains in blackhole. Re-run the step-1 describe-route-tables filter; it must return empty for the affected tables, and the specific route’s State must read active.
  • Reachability Analyzer is green both ways. The requester-to-accepter and accepter-to-requester analyses both return reachable. A one-directional pass is exactly the asymmetric trap that leaves a socket hanging.
  • Flow Logs show a matched pair. After a live connection attempt, the outbound ACCEPT on the renderer subnet now has a corresponding return record on the database subnet — the missing half of the conversation is present.
  • The spatial path works end to end. A psql connection to PostGIS on 5432 from a renderer completes its handshake, and a probe to a tile endpoint that forces a geometry fetch (e.g. a /{z}/{x}/{y}.pbf request against the PostGIS Cluster Provisioning backend) returns tiles rather than timing out.

Preventing recurrence

Encode the fix so the next attachment change cannot silently blackhole the data tier again:

  • Declare routes statically and in reciprocal pairs. Keep both directions in the same module so a request route can never merge without its return route. A review that shows only one aws_route for a new peering is the regression to catch in the pull request.
  • CIDR non-overlap assertion in CI. Add a checkov, Conftest, or plan-time precondition rule that fails when any peering or TGW destination intersects a local VPC CIDR — the overlap case has no route-table remedy, so it must be blocked before merge.
  • Scheduled drift detection on route tables. A nightly terraform plan -detailed-exitcode across the network stacks surfaces a route that an out-of-band change flipped to blackhole, the same estate-wide discipline as Drift Detection and Remediation for Geospatial Estates. Exit code 2 should page the owning team rather than wait for the next deploy.
  • Sequence attachment teardown safely. When removing a peering or TGW attachment, remove the routes that depend on it in the same change, so a deleted target never leaves a live route stranded in blackhole. Pair route-table association changes with create_before_destroy so an association always exists during a swap.

Frequently asked questions

What exactly does a route in `blackhole` state mean?

The route’s destination CIDR points at a target — a NAT gateway, peering connection, or Transit Gateway attachment — that no longer exists, because it was deleted or detached. AWS marks the route blackhole and silently discards every packet matching that CIDR. Re-declare the route to a live target, and remove routes that depend on an attachment in the same change that removes the attachment.

Connections time out with no `RST` — is that a route or a firewall problem?

Read the VPC Flow Logs verdict. An ACCEPT on the outbound record with no matching return record is a routing blackhole, usually a missing return route on the other VPC. A REJECT is a security-group or NACL deny, which is fixed in Security Group Hardening, not in a route table.

Why does the tile server reach PostGIS but PostGIS can't reply?

Asymmetric routing. The request route exists on the renderer VPC, but the database VPC’s route table has no return route to the renderer CIDR, so the reply is blackholed. Declare the reciprocal route on the database route table and confirm both directions with Reachability Analyzer.

Can I fix an overlapping CIDR with a more-specific route?

No. When two peered VPCs share a range, the local route for that range always wins and the cross-VPC route is never consulted; AWS also refuses to peer overlapping CIDRs in the first place. The only remedy is to re-IP one VPC onto a non-overlapping range, then install the symmetric routes.